Why did last week’s winner fail today?
Contain the failure and design a test that detects this regression next time.
“Online evaluations score real sessions in production” (opens the source)
The question
KestrelOps updated its parts-return policy on Monday. The model and prompt did not change. A frozen regression set contains only the previous policy. The service can pin a known-good index and show a manual support path while it rebuilds evidence.
| Previous release correct | Current release correct | |
|---|---|---|
| Frozen old-policy cases / 80 | 76 | 76 |
| New-policy cases / 20 | 19 | 11 |
The old result used the current policy text at that time. The new policy needs a new expected-answer version.
| Version | |
|---|---|
| User request | Policy effective Monday |
| Retrieved passage | Previous Friday snapshot |
| Model and prompt | Unchanged |
| Index alias | parts-returns-latest |
The alias changed after the last evaluation. Preserve its resolved snapshot, not only its name.
“Contain the failure and design a test that detects this regression next time.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Locate the changing evidence
What would you do here? Open the move you’d make.
Upgrade the model because answer quality fell.
Tempting, but it misses something. The observed change is the policy/evidence path, not demonstrated model capability.
Inspect index snapshot, effective date, and cache lineage.
The strong move. A model upgrade cannot repair missing current evidence by itself.
See how a strong candidate says it
“The trace points to stale policy evidence. I would preserve the trace and resolved index snapshot, then check ingestion, alias resolution, and cache versioning.”
Why it works: A model upgrade cannot repair missing current evidence by itself.
Step 2: Contain the new-policy risk
What would you do here? Open the move you’d make.
Keep serving because the frozen score is unchanged.
Tempting, but it misses something. The old set does not test the policy that changed.
Route affected requests to current verified evidence or manual support.
The strong move. Containment should target the failure’s known scope.
See how a strong candidate says it
“I would stop automated new-policy answers until the current index is verified, while retaining unaffected scoped answers where evidence is current.”
Why it works: Containment should target the failure’s known scope.
Step 3: Keep comparable history
What would you do here? Open the move you’d make.
Rewrite the old labels and overwrite last week’s run.
Tempting, but it misses something. Overwriting labels destroys the basis for diagnosing the change.
Version expected answers and retain both old and new slices.
The strong move. A test result needs the evidence and expected-answer version it used.
See how a strong candidate says it
“I would keep the old-policy set for historical replay and add current-policy cases with dates and answer versions. Then compare like-for-like slices.”
Why it works: A test result needs the evidence and expected-answer version it used.
Step 4: Close the loop
What would you do here? Open the move you’d make.
Wait for another user complaint.
Tempting, but it misses something. A complaint is a late and incomplete failure detector.
Gate index changes and monitor sampled live outcomes.
The strong move. Evidence deployments need release checks too.
See how a strong candidate says it
“I would test the new index against current cases, canary the alias change, monitor evidence versions and failures, and keep a reversible index switch.”
Why it works: Evidence deployments need release checks too.
The answer, in short
- Decision
- Fix and verify the evidence path before restoring automated new-policy answers.
- Because
- The frozen slice stays 95%; the new-policy slice falls from 95% to 55%. The stale passage explains a testable cause.
- What would change it
- Fresh authorized evidence and a passing current-policy canary can reopen the affected scope.
What would distinguish a stale cache from a failed index rebuild?
Common traps
- Monitoring only a single aggregate.
- Saving an index alias without its resolved version.
Check your answer
- Did I compute both slice rates?
- Did I contain the affected scope?
- Did I preserve versions and history?
- Did I test the evidence deployment?
Last reviewed Nov 10, 2026