Statistical evaluation infrastructure for LLM agents
A CI gate that blocks a pull request only when an agent has statistically significantly regressed — with paired non-inferiority tests, cluster-robust errors, FDR control across 42 metrics, and an UNDERPOWERED verdict for when the evidence genuinely cannot tell.
Live gate console
Pick a change a developer might have made. The suite is 70 tasks in 14 clusters, both sides run on identical task instances with identical seeds. Nothing here is illustrative.
Tests run on per-cluster means, so the independent sample size is the cluster count — not the task count. Using the larger number would overstate the power fivefold.
The forest plot shows proportion-scale metrics only. Token and latency deltas live in the Cost panel because they are measured in different units — putting them on one axis would need a second scale, and a dual-scale chart is the fastest way to make a rigorous plot mislead.
Pipeline
Recording a suite against a local model takes hours. Deciding, once the responses exist, takes seconds. The provider layer records once and replays forever, so CI never calls a model and a verdict reproduces byte for byte.
Everything above it produces one; everything below only reads one. Metrics, statistics and the gate are all testable against recorded trajectories — no model, no network.
A cache miss in CI is an error, never a silent live call. That single decision is what makes the verdict deterministic and the pipeline free to run.
Scores are stored per (task, repetition) in DuckDB, so any run can be re-analysed at a different K, margin or alpha without re-running the agent.
Evaluation stack
Every entry names the module that implements it. A test imports all of them, so nothing on this page can claim a capability the codebase does not have.
Recorded evidence
666 runs and 4.5 million tokens of real inference against τ²-bench retail, graded against its own gold trajectories. Whiskers are 95% cluster-robust intervals over 111 independent clusters.
The larger model sounds more coherent while finishing fewer jobs. That gap is why "it reads well" is not a measurement.
0.180 vs 0.111 looks like a 62% relative difference. Paired across 111 tasks it is −0.069 [−0.159, 0.021], and the suite's minimum detectable effect is 0.115.
Unconfirmed destructive actions fired on 27 of 111 tasks for the larger model, and never for the smaller. That bypasses the statistics entirely.
Interface
# record once — slow, in the background $ agentgate run --suite suites/tau2_retail \ --model ollama_chat/llama3.2:3b --mode cache # decide in CI — instant, offline, free $ agentgate compare --baseline main --candidate pr-482 REGRESSION trajectory.in_order_match -0.243 [-0.371, -0.115] p_adj=0.0004 exit 2 # what has the harness learned so far? $ agentgate leaderboard --suite tau2_retail tier model task_success 95% CI 1 Llama 3.2 3B 0.180 [0.108, 0.252] 1 Qwen2.5 7B 0.111 [0.053, 0.169] # tier 1 twice: not separable on this evidence
0 pass, 2 regression, 3 underpowered, 4 safety. Drop it into any CI system without parsing anything.
This page, the metric catalogue, the results tables and the PR comment all come from the pipeline. A drift check fails the build if a committed page stops matching its data.
Engineering
Unit, property-based with Hypothesis, end-to-end, and simulation tests that verify the gate's false-positive rate against synthetic ground truth.
134 source files, zero escapes. Every boundary is a pydantic v2 model exported as JSON Schema.
Every metric is pinned to a hand-computed expected value, so a refactor cannot silently change what a number means.
Suite hash, seeds, prompt hashes, model pins and library versions all fold into one config hash. Host details are excluded on purpose.
Runs entirely on a laptop with local models or free tiers. Cloning it costs nothing — a hard constraint, not a demo mode.
A weekly workflow records new evidence, commits the snapshot, and redeploys — so the git history is the history of what it has learned.