A fixed −3% threshold gate would have FAILED on outcome.task_success, trajectory.argument_correctness, trajectory.f1. AgentGate's verdict: REGRESSION.
The industry-standard rule — fail if any metric drops more than
3% — never consults the sample size. On 70 tasks a 3-point
move is well inside sampling noise.
Reliability: pass^k decay
pass^k is the probability that all k repetitions succeed. A single-run
pass rate cannot show this curve, which is why every task runs K times (E2).
Power and minimum detectable effect
This suite can detect a change of 0.255 in outcome.task_success with 80% power at n=14; your configured margin is delta=0.030, for which achieved power is 9% (n=908 pairs would be needed for 80%).
This suite can detect a change of 0.255 in reliability.pass_hat_k@2 with 80% power at n=14; your configured margin is delta=0.050, for which achieved power is 12% (n=328 pairs would be needed for 80%).
This suite can detect a change of 0.157 in trajectory.f1 with 80% power at n=14; your configured margin is delta=0.030, for which achieved power is 12% (n=346 pairs would be needed for 80%).
This suite can detect a change of 0.219 in trajectory.argument_correctness with 80% power at n=14; your configured margin is delta=0.050, for which achieved power is 14% (n=242 pairs would be needed for 80%).
This suite can detect a change of 4.100 in efficiency.latency_ms with 80% power at n=14; your configured margin is delta=8.571, for which achieved power is 100% (n=5 pairs would be needed for 80%).
Why each metric ruled the way it did
outcome.task_success — REGRESSION
test: permutation — Shapiro-Wilk p=0.0000 < 0.05: non-normal; switched to the sign-flip permutation test because a non-inferiority margin is defined on the mean while Wilcoxon tests a median shift
regression established: outcome.task_success fell by 0.8571, beyond the 0.03 margin (adjusted p=0.0001875 <= q=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
McNemar exact on discordant pairs (b=60, c=0): p=0.0000 at margin 0. The gate's test uses the shifted paired differences, which is what carries the margin (C3).
reliability.pass_hat_k@2 — REGRESSION
test: permutation — Shapiro-Wilk p=0.0000 < 0.05: non-normal; switched to the sign-flip permutation test because a non-inferiority margin is defined on the mean while Wilcoxon tests a median shift
regression established: reliability.pass_hat_k@2 fell by 0.8571, beyond the 0.05 margin (adjusted p=0.0001875 <= q=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
trajectory.f1 — REGRESSION
test: permutation — Shapiro-Wilk p=0.0001 < 0.05: non-normal; switched to the sign-flip permutation test because a non-inferiority margin is defined on the mean while Wilcoxon tests a median shift
regression established: trajectory.f1 fell by 0.5143, beyond the 0.03 margin (adjusted p=0.0001875 <= q=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
trajectory.argument_correctness — REGRESSION
test: permutation — Shapiro-Wilk p=0.0004 < 0.05: non-normal; switched to the sign-flip permutation test because a non-inferiority margin is defined on the mean while Wilcoxon tests a median shift
regression established: trajectory.argument_correctness fell by 0.6701, beyond the 0.05 margin (adjusted p=0.0001875 <= q=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
efficiency.latency_ms — PASS
test: permutation — Shapiro-Wilk p=0.0093 < 0.05: non-normal; switched to the sign-flip permutation test because a non-inferiority margin is defined on the mean while Wilcoxon tests a median shift
non-inferiority established: the confidence bound on the change sits above -8.571 (p=5e-05 <= alpha=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).