PASS with warning: outcome.task_success, reliability.pass_hat_k@2 is UNDERPOWERED — this suite cannot detect the configured margin, so its silence is not evidence of no regression
A fixed −3% threshold gate would have FAILED on outcome.task_success. AgentGate's verdict: UNDERPOWERED.
The industry-standard rule — fail if any metric drops more than
3% — never consults the sample size. On 70 tasks a 3-point
move is well inside sampling noise.
Reliability: pass^k decay
pass^k is the probability that all k repetitions succeed. A single-run
pass rate cannot show this curve, which is why every task runs K times (E2).
Power and minimum detectable effect
This suite can detect a change of 0.299 in outcome.task_success with 80% power at n=14; your configured margin is delta=0.030, for which achieved power is 8% (n=1247 pairs would be needed for 80%).
This suite can detect a change of 0.299 in reliability.pass_hat_k@2 with 80% power at n=14; your configured margin is delta=0.050, for which achieved power is 11% (n=450 pairs would be needed for 80%).
This suite can detect a change of 0.000 in trajectory.f1 with 80% power at n=14; your configured margin is delta=0.030, for which achieved power is 100% (n=2 pairs would be needed for 80%).
This suite can detect a change of 0.000 in trajectory.argument_correctness with 80% power at n=14; your configured margin is delta=0.050, for which achieved power is 100% (n=2 pairs would be needed for 80%).
This suite can detect a change of 0.000 in efficiency.latency_ms with 80% power at n=14; your configured margin is delta=8.571, for which achieved power is 100% (n=2 pairs would be needed for 80%).
Why each metric ruled the way it did
outcome.task_success — UNDERPOWERED
test: permutation — Shapiro-Wilk p=0.0000 < 0.05: non-normal; switched to the sign-flip permutation test because a non-inferiority margin is defined on the mean while Wilcoxon tests a median shift
neither test rejects, and this suite has only 8% power to detect a 0.03 change (target 80%). About 1247 paired tasks would be needed. The gate cannot tell.
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
McNemar exact on discordant pairs (b=15, c=0): p=0.0000 at margin 0. The gate's test uses the shifted paired differences, which is what carries the margin (C3).
reliability.pass_hat_k@2 — UNDERPOWERED
test: permutation — Shapiro-Wilk p=0.0000 < 0.05: non-normal; switched to the sign-flip permutation test because a non-inferiority margin is defined on the mean while Wilcoxon tests a median shift
neither test rejects, and this suite has only 11% power to detect a 0.05 change (target 80%). About 450 paired tasks would be needed. The gate cannot tell.
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
trajectory.f1 — PASS
test: paired_t — all differences identical; normality test is undefined
non-inferiority established: the confidence bound on the change sits above -0.03 (p=1.145e-202 <= alpha=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
trajectory.argument_correctness — PASS
test: paired_t — all differences identical; normality test is undefined
non-inferiority established: the confidence bound on the change sits above -0.05 (p=7.687e-208 <= alpha=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
efficiency.latency_ms — PASS
test: paired_t — all differences identical; normality test is undefined
non-inferiority established: the confidence bound on the change sits above -8.571 (p=0 <= alpha=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).