A fixed −3% threshold gate would have FAILED on efficiency.latency_ms, outcome.task_success, trajectory.f1. AgentGate's verdict: REGRESSION.
The industry-standard rule — fail if any metric drops more than
3% — never consults the sample size. On 70 tasks a 3-point
move is well inside sampling noise.
Reliability: pass^k decay
pass^k is the probability that all k repetitions succeed. A single-run
pass rate cannot show this curve, which is why every task runs K times (E2).
Flakiest tasks (succeeded sometimes, not always)
Task
Successes
K
address_delft-01
3
4
address_langley-01
3
4
address_langley-02
3
4
address_langley-03
3
4
escalation_delivery-03
3
4
escalation_delivery-05
3
4
refund_large_compiler-01
3
4
refund_large_compiler-03
3
4
refund_large_compiler-05
3
4
refund_large_orbital-05
3
4
Power and minimum detectable effect
This suite can detect a change of 0.038 in outcome.task_success with 80% power at n=14; your configured margin is delta=0.030, for which achieved power is 63% (n=22 pairs would be needed for 80%).
This suite can detect a change of 0.075 in reliability.pass_hat_k@2 with 80% power at n=14; your configured margin is delta=0.050, for which achieved power is 50% (n=30 pairs would be needed for 80%).
This suite can detect a change of 0.020 in trajectory.f1 with 80% power at n=14; your configured margin is delta=0.030, for which achieved power is 98% (n=8 pairs would be needed for 80%).
This suite can detect a change of 0.030 in trajectory.argument_correctness with 80% power at n=14; your configured margin is delta=0.050, for which achieved power is 99% (n=7 pairs would be needed for 80%).
This suite can detect a change of 186.129 in efficiency.latency_ms with 80% power at n=14; your configured margin is delta=8.571, for which achieved power is 6% (n=5917 pairs would be needed for 80%).
Why each metric ruled the way it did
outcome.task_success — UNDERPOWERED
test: permutation — Shapiro-Wilk p=0.0178 < 0.05: non-normal; switched to the sign-flip permutation test because a non-inferiority margin is defined on the mean while Wilcoxon tests a median shift
neither test rejects, and this suite has only 63% power to detect a 0.03 change (target 80%). About 22 paired tasks would be needed. The gate cannot tell.
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
McNemar exact on discordant pairs (b=0, c=0): p=1.0000 at margin 0. The gate's test uses the shifted paired differences, which is what carries the margin (C3).
reliability.pass_hat_k@2 — UNDERPOWERED
test: permutation — Shapiro-Wilk p=0.0178 < 0.05: non-normal; switched to the sign-flip permutation test because a non-inferiority margin is defined on the mean while Wilcoxon tests a median shift
neither test rejects, and this suite has only 50% power to detect a 0.05 change (target 80%). About 30 paired tasks would be needed. The gate cannot tell.
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
trajectory.f1 — REGRESSION
test: paired_t — Shapiro-Wilk p=0.8482 >= 0.05: normal enough
regression established: trajectory.f1 fell by 0.07377, beyond the 0.03 margin (adjusted p=9.58e-05 <= q=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
trajectory.argument_correctness — PASS
test: permutation — Shapiro-Wilk p=0.0002 < 0.05: non-normal; switched to the sign-flip permutation test because a non-inferiority margin is defined on the mean while Wilcoxon tests a median shift
non-inferiority established: the confidence bound on the change sits above -0.05 (p=0.04245 <= alpha=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).
efficiency.latency_ms — REGRESSION
test: paired_t — Shapiro-Wilk p=0.1422 >= 0.05: normal enough
regression established: efficiency.latency_ms fell by 784, beyond the 8.571 margin (adjusted p=1.566e-07 <= q=0.05)
clustered analysis: 70 tasks in 14 clusters. Tests and intervals both use per-cluster mean differences, so five paraphrases of one scenario count as one independent observation, not five (C1.3).