evalloop
Tests for LLM agents that run every case more than once.
evalloop is a small open source toolkit I built for LLM agents. It records every model call and tool call, runs each case several times, checks the whole trace, and fails CI when behavior regresses.
A 50 second walkthrough: run it five times, check every step, fix the agent, gate the merge.
01The problem
One green run proves very little for an LLM agent. The same case can pass once and fail the next time. A final answer can look right while the agent called the wrong tool, invented an amount, or followed an instruction hidden in a document.
Most test setups check a single run and the last message. Behavior also changes after a prompt tweak or a model swap, and nobody notices until a user does.
02What the report shows
03What it checks
Wrong tool, arguments or order
tool_called tool_args tool_orderHallucinated numbers
grounded()Prompt injection leaking a secret
no_secret_leakLoops and wasted steps
no_loop max_stepsCost and latency creep
cost_under latency_underFlaky behavior
pass^k reliability04How it works
- 01
Trace
Every run records each model call, tool call and argument.
- 02
Check
Checks run on the trace, not only on the final answer.
- 03
Repeat
Each case runs k times with stable seeds.
- 04
Diff
A run is compared against a saved baseline.
- 05
Gate
CI exits with code 1 when any case got worse.
from evalloop import Case, run_suite, tool_args, grounded, no_loop
cases = [
Case("refund_small", "Refund order #1042",
[tool_args("issue_refund", order_id="1042"), grounded(), no_loop(2)]),
]
result = run_suite(my_agent, cases, trials=5)
print(result.pass_rate, result.reliability)evalloop run suite:cases --agent my_module:my_agent --trials 5 \
--baseline baseline.json --fail-on-regression --fail-under 0.9505On a real model
| Pass rate | 94% |
| Reliability (pass^3) | 83% |
| Cases with a failed trial | 1 of 6 |
The failing case skipped its knowledge base search in one of three trials. The final answer still sounded right, which is why a check on the trace caught it and a single run would not.
06Decisions
Trace first
Asserting on behavior needs the steps, so the tracer is the core and the checks are small functions over it.
Zero dependencies
Standard library only. The report is a single HTML file with no JavaScript, safe to attach to a pull request.
Reliability over accuracy
An agent that passes 80% of the time still looks fine on one run. Counting a case as reliable only when every trial passes shows what users feel.
Keys never stored
In Python the key comes from an environment variable. In the browser demo it lives in memory only and goes only to the provider you chose.
07Honest limits
- The demo numbers come from a simulated agent with planted bugs. They show the method, not a benchmark.
- The real model run is one run on one free tier model. Results change between runs, so treat 94% and 83% as an example.
- Real models work through any OpenAI compatible endpoint. A native Anthropic adapter, parallel trials and an LLM judge check are on the roadmap.
Want this on your agent?
I build and test LLM agents, RAG and backend systems. If your agent works in a demo and wobbles in production, I can help.

