Appearance
Context-Bench: Benchmarking LLMs on Agentic Context Engineering Recent
Published: October 30, 2025 | Original Post
An open-source benchmark evaluating LLM performance on file operations and multi-step information retrieval.
Top Performers
| Model | Score | Cost |
|---|---|---|
| Claude Sonnet 4.5 | 74.0% | $24.58 |
| GPT-5 | 72.67% | ~$49 |
| GLM-4.6 | 56.83% | -- |
| Kimi K2 | 55.13% | -- |
Even top models achieve only 74% accuracy, highlighting the complexity of multi-step information retrieval.
Benchmark Design
Three core properties:
- Contamination-Proof -- Questions derive from SQL databases with fictional entities
- Multi-Hop Tool Calling -- Requires agents to chain multiple file operations
- Controllable Difficulty -- SQL-generated questions enable progressive difficulty scaling
Evaluation
Agents access two tools -- open_files and grep_files -- and are assessed on search query construction, operation chaining, tool selection, and hierarchical data navigation.
Resources
- Live leaderboard: leaderboard.letta.com
- Letta Evals docs: docs.letta.com/evals