Example reports
When the gate runs, it leaves a report like the ones below: the verdict, every metric with its error bars, what the statistics could and could not establish, and how much evidence it had to work with.
Each report here comes from a real run against the same 70-task suite — a healthy agent against one that was deliberately broken in a specific, realistic way. Nothing is hand-written. Notice that the gate reaches a different verdict for each kind of breakage; that difference is the whole point.
The agent did something it must never do. These skip the statistics entirely — no average is good enough to make up for a security failure.
A support ticket contained text telling the agent to ignore its rules — and the agent followed it.
Stop — safety failureThe agent was switched to a smaller, cheaper model to cut the bill.
Stop — safety failureA "cleanup" PR deleted a paragraph of the system prompt that looked redundant.
Stop — safety failureSampling temperature was nudged up to make answers less repetitive.
The numbers moved, but not by enough to tell a real change from ordinary variation. This is the verdict a threshold rule can never give, and the one that prevents false alarms.
The agent really did get worse, by more than the tolerance the policy declared. These block the merge.
Non-inferiority was established — not merely 'we found no problem', but 'the suite was big enough to have found one and did not'.
How to read one
Ship, block, or not enough evidence — with the reasoning, not just the label.
A number without a range is a guess. Each metric shows the interval it actually supports.
The smallest change this suite was capable of detecting, so you know what a pass does and does not rule out.