Open source · Agent evals

evalloop

Tests for LLM agents that run every case more than once.

evalloop is a small open source toolkit I built for LLM agents. It records every model call and tool call, runs each case several times, checks the whole trace, and fails CI when behavior regresses.

A 50 second walkthrough: run it five times, check every step, fix the agent, gate the merge.

Demo support agent, simulated with planted bugs, 5 trials per case
Pass rate33%100%
Reliability0%100%
p95 latency3.1s1.0s
Cost per run$0.0105$0.0026
Role
Designed and built end to end
Stack
Python, Zero dependencies, pass@k and pass^k
Status
Live, 2026

01The problem

One green run proves very little for an LLM agent. The same case can pass once and fail the next time. A final answer can look right while the agent called the wrong tool, invented an amount, or followed an instruction hidden in a document.

Most test setups check a single run and the last message. Behavior also changes after a prompt tweak or a model swap, and nobody notices until a user does.

02What the report shows

evalloop report for the first version of the support agent, showing 33 percent pass rate and the failing trace
The report for v1: 33% pass rate, and the exact trial and step where it went wrong.
evalloop diff report showing zero regressed and six improved cases after the fix
The diff after the fix: 0 regressed, 6 improved. This is what CI reads.

03What it checks

Wrong tool, arguments or order

tool_called tool_args tool_order

Hallucinated numbers

grounded()

Prompt injection leaking a secret

no_secret_leak

Loops and wasted steps

no_loop max_steps

Cost and latency creep

cost_under latency_under

Flaky behavior

pass^k reliability

04How it works

  1. 01

    Trace

    Every run records each model call, tool call and argument.

  2. 02

    Check

    Checks run on the trace, not only on the final answer.

  3. 03

    Repeat

    Each case runs k times with stable seeds.

  4. 04

    Diff

    A run is compared against a saved baseline.

  5. 05

    Gate

    CI exits with code 1 when any case got worse.

Test your agent
from evalloop import Case, run_suite, tool_args, grounded, no_loop

cases = [
    Case("refund_small", "Refund order #1042",
         [tool_args("issue_refund", order_id="1042"), grounded(), no_loop(2)]),
]
result = run_suite(my_agent, cases, trials=5)
print(result.pass_rate, result.reliability)
Gate your pull requests
evalloop run suite:cases --agent my_module:my_agent --trials 5 \
  --baseline baseline.json --fail-on-regression --fail-under 0.95

05On a real model

Real model run: Groq gpt-oss-120b, 3 trials per case
Pass rate94%
Reliability (pass^3)83%
Cases with a failed trial1 of 6

The failing case skipped its knowledge base search in one of three trials. The final answer still sounded right, which is why a check on the trace caught it and a single run would not.

06Decisions

Trace first

Asserting on behavior needs the steps, so the tracer is the core and the checks are small functions over it.

Zero dependencies

Standard library only. The report is a single HTML file with no JavaScript, safe to attach to a pull request.

Reliability over accuracy

An agent that passes 80% of the time still looks fine on one run. Counting a case as reliable only when every trial passes shows what users feel.

Keys never stored

In Python the key comes from an environment variable. In the browser demo it lives in memory only and goes only to the provider you chose.

07Honest limits

  • The demo numbers come from a simulated agent with planted bugs. They show the method, not a benchmark.
  • The real model run is one run on one free tier model. Results change between runs, so treat 94% and 83% as an example.
  • Real models work through any OpenAI compatible endpoint. A native Anthropic adapter, parallel trials and an LLM judge check are on the roadmap.

Want this on your agent?

I build and test LLM agents, RAG and backend systems. If your agent works in a demo and wobbles in production, I can help.