Your judge agrees 91% of the time. Is it useful?
Evaluate the judge and propose the next experiment.
“Use human feedback to calibrate automated scoring.” (opens the source)
The question
KestrelOps uses an automated judge to flag unsupported answers for review. Humans adjudicated a fresh sample of 100 responses using a written evidence rubric. Here, positive means an answer that should be flagged as a failure.
| Human failure | Human acceptable | |
|---|---|---|
| Judge flags failure | 2 | 1 |
| Judge passes | 8 | 89 |
100 total cases: 10 human failures and 90 acceptable answers. Each answer has one final human label.
The judge should catch at least 80% of human-labeled failures. Borderline labels go to human review; the same sample cannot be used to tune and certify the judge.
“Evaluate the judge and propose the next experiment.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Calculate the tempting number
What would you do here? Open the move you’d make.
Certify the judge from 91% agreement.
Tempting, but it misses something. A high headline can be one point above a useless baseline.
Compute agreement, then inspect the failure row.
The strong move. Class balance changes what agreement means.
See how a strong candidate says it
“The judge agrees on 2 + 89 = 91 cases, so agreement is 91%. But an always-pass judge already gets 90% on this imbalanced sample.”
Why it works: Class balance changes what agreement means.
Step 2: Measure missed failures
What would you do here? Open the move you’d make.
Use precision alone.
Tempting, but it misses something. Precision ignores failures that the judge passes.
Compute failure recall as 2 out of 10.
The strong move. Recall answers the stated question: which human failures the judge catches.
See how a strong candidate says it
“Failure recall is 20%, far below the 80% contract. Precision is 2/3, about 67%; that does not erase eight missed failures.”
Why it works: Recall answers the stated question: which human failures the judge catches.
Step 3: Repair the rubric against evidence
What would you do here? Open the move you’d make.
Tell the judge to be stricter without checking labels.
Tempting, but it misses something. A blanket stricter prompt can increase false alarms without finding the right failures.
Review missed failures and ambiguous human labels separately.
The strong move. A grader can fail from a weak rubric, missing evidence, or label disagreement.
See how a strong candidate says it
“I would inspect the eight misses, resolve rubric ambiguity with human reviewers, and test whether evidence-specific criteria improve failure detection.”
Why it works: A grader can fail from a weak rubric, missing evidence, or label disagreement.
Step 4: Validate on fresh cases
What would you do here? Open the move you’d make.
Report the training sample’s improved agreement.
Tempting, but it misses something. Tuning success on reused labels is not independent certification.
Freeze the revised grader and check a fresh adjudicated sample.
The strong move. A calibrated release test must stay separate from grader tuning.
See how a strong candidate says it
“I would measure failure recall, false alarms, and reviewer workload on fresh cases, including difficult slices and more than one run if the judge varies.”
Why it works: A calibrated release test must stay separate from grader tuning.
The answer, in short
- Decision
- Do not use this judge as the sole release gate.
- Because
- It misses 8 of 10 human failures despite 91% agreement.
- What would change it
- A frozen revised grader that meets recall and review-load limits on fresh labels could qualify.
How would the review queue change if false positives doubled?
Common traps
- Using aggregate agreement as failure detection.
- Tuning and certifying on the same examples.
Check your answer
- Did I name the positive class?
- Did I compute 20% recall?
- Did I compare the 90% baseline?
- Did I reserve fresh human labels?
Last reviewed Nov 10, 2026