Which inference benchmark supports the speed claim?
Compare time to first token and output throughput on the same workload.
“You control the shape of the load, not just the volume.” (opens the source)
The question
KestrelOps compares two configurations for the same model and hardware. Both runs use the same output limits and quality checks. First-token p95 is the 95th percentile of the wait from request to first token. System output rate is total tokens across all requests divided by the time from first request to last response.
| First-token p95 (ms) | System output (tokens/s) | |
|---|---|---|
| A | 240 | 800 |
| B | 170 | 1000 |
The report leaves out the workload details.
“What would you check before saying B starts responses faster?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: What does each number measure?
What would you do here? Open the move you’d make.
Separate the request delay from the system output rate.
The strong move. A request can wait longer for its first token while the system produces more tokens across all requests.
Use the output rate to estimate how much sooner each response starts.
Tempting, but it misses something. System output counts concurrent responses. It does not give first-token times for individual requests.
See how a strong candidate says it
“B’s reported p95 wait to the first token is 70 ms lower. Its total output rate is 200 tokens/s higher. The first figure describes the upper end of request waits. The second counts output from the system as a whole.”
Why it works: The dashboard reports two different measures. A change in one does not determine the other.
Step 2: What has to match?
What would you do here? Open the move you’d make.
Match the workload, measure both configurations the same way, and verify completions.
The strong move. A fair comparison needs the same request trace, comparable output and the same timing boundary.
Compare the dashboard rows because the model and hardware match.
Tempting, but it misses something. The rows can still differ in request timing, prompt lengths or reusable work.
See how a strong candidate says it
“I’d compare A and B on the same prompts and arrival times. I’d check whether earlier processing can be reused, count completed requests and output tokens, and keep the timing and quality checks the same.”
Why it works: Request timing, reusable work and completion rules can all change what the benchmark measures.
Step 3: What changes after a matched run?
The opening A result used fresh prompts. B used repeated prefixes that could reuse earlier processing. In the matched runs, both configurations used the same request trace and passed the same output checks within each pair. They completed the same requests and produced the same token count. Fresh prompts: A 240 ms and 800 tokens/s. B 320 ms and 920 tokens/s. Repeated prefixes: A 190 ms and 880 tokens/s. B 170 ms and 1,000 tokens/s.
What would you do here? Open the move you’d make.
Report a higher output rate on both workloads and a slower first token on fresh prompts.
The strong move. The fresh and repeated runs each support a comparison. The opening report compares across workloads.
Keep the 70 ms improvement because the fresh A and repeated B rows stayed the same.
Tempting, but it misses something. Those two rows still use different workloads. The matched fresh result changes the conclusion.
Average the two p95 values for each configuration to get the mixed p95.
Tempting, but it misses something. The two group p95 values do not show the distribution of waits. Compute p95 from timings in a mixed run.
See how a strong candidate says it
“On fresh prompts, B’s first-token p95 is 80 ms slower and its output rate is 15% higher. On repeated prefixes, B’s first-token p95 is 20 ms faster and its output rate is about 13.6% higher.”
Why it works: B increases system output rate on both matched workloads. Its first-token result changes direction between the two workloads.
Step 4: Which claim can you defend?
What would you do here? Open the move you’d make.
Keep both matched results and ask for a representative mixed replay.
The strong move. The matched runs give separate results. A representative mixed replay could support a claim about that mix.
Publish one speed claim for all traffic and attribute it to prefix reuse.
Tempting, but it misses something. The tests do not establish a production-wide result or isolate prefix reuse as the cause of the difference.
See how a strong candidate says it
“I’d say B delivered more output per second in both tests. Its first-token p95 improved only on repeated prefixes. To make a mixed-workload claim, I’d run both configurations on one representative trace and compute p95 from the individual request times.”
Why it works: This states what the measured runs support and names the evidence needed for a broader claim.
The answer, in short
- Decision
- B raised system output rate in both tests. First-token p95 got worse on fresh prompts and improved on repeated prefixes.
- Because
- The opening dashboard compared different workloads. Each matched pair held its request trace and output checks fixed.
- What would change it
- A representative mixed replay with individual timings could support a claim about that traffic mix.
Common traps
- Reading total tokens per second as time to the first token.
- Comparing fresh and repeated requests as if they did the same work.
- Averaging two p95 values to get a mixed p95.
Check your answer
- Did I explain what each metric measures?
- Did I match workloads and check completion?
- Did I state the two first-token results separately?
- Did I ask for the evidence a mixed claim needs?
Last reviewed Nov 11, 2026