The score improved. Why did tool tasks get worse?
Can these reports justify the release? Design a valid comparison.
“Verified badges prove that the results can be reproduced.” (opens the source)
The question
KestrelOps reports a higher aggregate success rate after a release. The two reports used different mixes of lookup and action tasks. Each task has one final success label. Action tasks require a confirmed permitted write, not merely a plausible answer.
| Lookup correct / tasks | Action correct / tasks | Overall | |
|---|---|---|---|
| Previous | 40 / 50 | 36 / 50 | 76 / 100 |
| Current | 81 / 90 | 5 / 10 | 86 / 100 |
Lookup success rises from 80% to 90%; action success falls from 72% to 50%. The action task share shrank from 50% to 10%.
On a matched task set, action success must not fall below the previous release. Preserve dataset IDs, task definitions, grader versions, and environment fixtures.
“Can these reports justify the release? Design a valid comparison.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Separate capability from task mix
What would you do here? Open the move you’d make.
Celebrate the ten-point overall improvement.
Tempting, but it misses something. The aggregate combines a capability change with a population change.
Compare each slice before trusting the overall gain.
The strong move. A changed denominator can hide the product capability that regressed.
See how a strong candidate says it
“The overall rate rises from 76% to 86%, but the current test contains far fewer action tasks. Action success falls 22 percentage points.”
Why it works: A changed denominator can hide the product capability that regressed.
Step 2: Normalize the comparison
What would you do here? Open the move you’d make.
Compare the two totals as if task difficulty were fixed.
Tempting, but it misses something. Different tasks can explain a score difference without any model improvement.
Replay both systems on the same frozen task set.
The strong move. Matched inputs isolate the system change better than unrelated samples.
See how a strong candidate says it
“I would run both releases on identical cases, fixtures, and graders, report slice counts, and preserve run configuration IDs.”
Why it works: Matched inputs isolate the system change better than unrelated samples.
Step 3: Check the action outcomes
What would you do here? Open the move you’d make.
Ask a judge whether the replies sound complete.
Tempting, but it misses something. A polished completion reply does not establish a successful write.
Inspect the five current action failures against environment state.
The strong move. The action slice needs outcome evidence, not just language quality.
See how a strong candidate says it
“I would inspect target identity, permissions, tool errors, and final state for each failed action, then build targeted regressions on a separate development set.”
Why it works: The action slice needs outcome evidence, not just language quality.
Step 4: Use the contract’s slice gate
What would you do here? Open the move you’d make.
Use the higher overall rate to waive the action rule.
Tempting, but it misses something. A strong lookup score cannot prove an action capability recovered.
Hold the action release until the matched comparison passes.
The strong move. A critical task slice should have an explicit release boundary.
See how a strong candidate says it
“I would keep the prior action path and test a repaired candidate. Lookup improvements could ship separately only if the paths and release evidence are separable.”
Why it works: A critical task slice should have an explicit release boundary.
The answer, in short
- Decision
- The reports do not justify replacing the action path.
- Because
- The task mix changed, and observed action success fell from 72% to 50%.
- What would change it
- A matched replay that meets the action gate can support a release decision.
If the expected production mix is 70% lookup and 30% action, what weighted rate does each observed slice imply?
Common traps
- Comparing scores with different task mixes.
- Calling a reproducible run a representative workload.
Check your answer
- Did I report the changed denominators?
- Did I compute the 22-point action drop?
- Did I require a matched replay?
- Did I inspect actual write outcomes?
Last reviewed Nov 10, 2026