Appearance
Letta Evals: Evaluating Agents that Learn Recent
Published: October 23, 2025 | Original Post
An open-source evaluation framework for systematically testing stateful agents.
Core Problem
Agent design changes must be evaluated both for new agents and existing agents. Traditional evaluation approaches fail to capture how stateful agents behave differently as they accumulate context.
Framework Components
| Component | Description |
|---|---|
| Datasets | JSONL files containing test cases with inputs, outputs, and metadata |
| Targets | Agent Files (.af) defining agent configuration |
| Graders | Scoring mechanisms (exact match to LLM-as-judge) |
| Gates | Pass/fail thresholds preventing regressions |
CI/CD Integration
The framework integrates into GitHub workflows. Failed evaluations return non-zero status codes, enabling teams to block pull requests that compromise agent behavior.
Real-World Usage
Bilt Rewards operates over one million personalized stateful agents and uses Letta Evals to validate architectural changes across their agent fleet before production deployment.