PASS: non-inferiority established for outcome.task_success, reliability.pass_hat_k@2, trajectory.f1, trajectory.argument_correctness, efficiency.latency_ms
A fixed −3% threshold gate would have PASSED this comparison. AgentGate's verdict: PASS.
The industry-standard rule — fail if any metric drops more than
3% — never consults the sample size. On 70 tasks a 3-point
move is well inside sampling noise.
Reliability: pass^k decay
pass^k is the probability that all k repetitions succeed. A single-run
pass rate cannot show this curve, which is why every task runs K times (E2).
Power and minimum detectable effect
This suite can detect a change of 0.000 in outcome.task_success with 80% power at n=14; your configured margin is delta=0.030, for which achieved power is 100% (n=2 pairs would be needed for 80%).
This suite can detect a change of 0.000 in reliability.pass_hat_k@2 with 80% power at n=14; your configured margin is delta=0.050, for which achieved power is 100% (n=2 pairs would be needed for 80%).
This suite can detect a change of 0.000 in trajectory.f1 with 80% power at n=14; your configured margin is delta=0.030, for which achieved power is 100% (n=2 pairs would be needed for 80%).
This suite can detect a change of 0.000 in trajectory.argument_correctness with 80% power at n=14; your configured margin is delta=0.050, for which achieved power is 100% (n=2 pairs would be needed for 80%).
This suite can detect a change of 0.000 in efficiency.latency_ms with 80% power at n=14; your configured margin is delta=8.571, for which achieved power is 100% (n=2 pairs would be needed for 80%).
Why each metric ruled the way it did
outcome.task_success — PASS
test: paired_t — all differences identical; normality test is undefined
non-inferiority established: the confidence bound on the change sits above -0.03 (p=1.145e-202 <= alpha=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
McNemar exact on discordant pairs (b=0, c=0): p=1.0000 at margin 0. The gate's test uses the shifted paired differences, which is what carries the margin (C3).
reliability.pass_hat_k@2 — PASS
test: paired_t — all differences identical; normality test is undefined
non-inferiority established: the confidence bound on the change sits above -0.05 (p=7.687e-208 <= alpha=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
trajectory.f1 — PASS
test: paired_t — all differences identical; normality test is undefined
non-inferiority established: the confidence bound on the change sits above -0.03 (p=1.145e-202 <= alpha=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
trajectory.argument_correctness — PASS
test: paired_t — all differences identical; normality test is undefined
non-inferiority established: the confidence bound on the change sits above -0.05 (p=7.687e-208 <= alpha=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
efficiency.latency_ms — PASS
test: paired_t — all differences identical; normality test is undefined
non-inferiority established: the confidence bound on the change sits above -8.571 (p=0 <= alpha=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).