Which pipeline meets the release contract?
Choose the pipeline, calculate its daily cost, and identify the closest limit.
“trade latency and cost for better task performance” (opens the source)
The question
KestrelOps must choose a pipeline for 10,000 maintenance questions a day. All candidates ran on the same 200 held-out cases. Cost and end-to-end latency were measured on a replay of the same workload, including each pipeline’s retries.
| Correct / 200 | Critical errors | Cost / request | End-to-end p95 | |
|---|---|---|---|---|
| Single call | 180 | 0 | $0.006 | 1.4 s |
| Retrieve + verify | 188 | 0 | $0.014 | 2.8 s |
| Planner agent | 190 | 1 | $0.031 | 5.9 s |
Observed p95s come from complete requests. They are not sums of component percentiles.
At least 93% correct, zero observed critical errors, p95 at most 3 seconds, and model/tool cost at most $150 per day at the stated traffic.
“Choose the pipeline, calculate its daily cost, and identify the closest limit.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Apply the quality gates
What would you do here? Open the move you’d make.
Pick the 95% planner agent.
Tempting, but it misses something. A higher accuracy does not waive a critical-error limit.
Reject candidates that miss either quality limit.
The strong move. A candidate must satisfy the whole contract.
See how a strong candidate says it
“The single call is 90% correct and fails the 93% gate. The planner is 95% correct but has a critical error. Retrieve + verify is 94% with no observed critical errors.”
Why it works: A candidate must satisfy the whole contract.
Step 2: Calculate the recurring cost
What would you do here? Open the move you’d make.
Compare only the nominal model price.
Tempting, but it misses something. Token list prices alone can omit tools, repeated calls, and retry patterns.
Multiply the measured request cost by daily traffic.
The strong move. The measured request cost already includes this workload’s retries.
See how a strong candidate says it
“Retrieve + verify costs $0.014 × 10,000 = $140 per day. That leaves $10, or about 7.1% additional traffic, before the daily cost cap.”
Why it works: The measured request cost already includes this workload’s retries.
Step 3: Use end-to-end latency
What would you do here? Open the move you’d make.
Add retrieval p95 to generation p95.
Tempting, but it misses something. Adding percentiles can produce a misleading latency estimate.
Compare the measured request p95 with the 3-second cap.
The strong move. Percentiles of components do not generally add to a request percentile.
See how a strong candidate says it
“Its measured end-to-end p95 is 2.8 seconds, leaving 0.2 seconds. I would check load and timeout behavior before increasing traffic.”
Why it works: Percentiles of components do not generally add to a request percentile.
Step 4: Release with headroom checks
What would you do here? Open the move you’d make.
Ship without limits because the sample passed.
Tempting, but it misses something. The sample does not guarantee the same workload, retry rate, or queue delay.
Choose retrieve + verify and monitor cost and latency at load.
The strong move. The candidate passes the observed contract with limited headroom.
See how a strong candidate says it
“I would release retrieve + verify with traffic and spend limits, verify the replay under expected concurrency, and stop expansion if either budget is exceeded.”
Why it works: The candidate passes the observed contract with limited headroom.
The answer, in short
- Decision
- Choose retrieve + verify for a bounded release.
- Because
- It reaches 94%, has zero observed critical failures, costs $140/day, and measures 2.8-second p95.
- What would change it
- A traffic, evidence-size, or retry increase can erase its budget; measure again before expanding.
Traffic rises to 11,000 requests a day. Which limit fails first if request behavior stays fixed?
Common traps
- Averaging or adding component p95 values.
- Ignoring retry costs or critical failures.
Check your answer
- Did I calculate $140/day?
- Did I reject the other candidates for the right reasons?
- Did I use end-to-end latency?
- Did I state the small headroom?
Last reviewed Nov 10, 2026