The agent says it cancelled the order. Did it?
What should the agent do next, and what must the evaluation grade?
“The outcome is the final state in the environment” (opens the source)
The question
KestrelOps can cancel a duplicate parts order. The user authorized cancellation of order K-417 only. The write request timed out. A timeout means the caller has no outcome yet; it does not prove that the write failed.
Done. I cancelled K-417.
| Value | |
|---|---|
| POST cancel request | Timed out; idempotency key cancel-K417-v1 |
| GET order K-417 | cancelled |
| Cancellation event | order K-417; key cancel-K417-v1 |
| Order K-418 | active |
Readbacks came from the authoritative order API after the timeout.
The API deduplicates cancellation requests that use the same idempotency key. A new key can create a second cancellation action.
“What should the agent do next, and what must the evaluation grade?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Use the observed state
What would you do here? Open the move you’d make.
Trust the agent’s Done message.
Tempting, but it misses something. The generated message could be confident and false.
Check the authoritative order and action receipt.
The strong move. The requested state change is the outcome under test.
See how a strong candidate says it
“The transcript is not the receipt. Here, the order state and matching cancellation event confirm that K-417 was cancelled.”
Why it works: The requested state change is the outcome under test.
Step 2: Avoid a duplicate write
What would you do here? Open the move you’d make.
Retry with a fresh idempotency key.
Tempting, but it misses something. A new key discards deduplication and risks another action.
Do not retry a confirmed cancellation.
The strong move. Retry behavior follows the service’s contract and observed state.
See how a strong candidate says it
“The readback confirms success, so no second write is needed. If the outcome were unknown, I would reconcile first and retain the original key for an allowed retry.”
Why it works: Retry behavior follows the service’s contract and observed state.
Step 3: Check the scope
What would you do here? Open the move you’d make.
Grade success if any order was cancelled.
Tempting, but it misses something. A broad success condition hides collateral changes.
Verify K-418 stayed active.
The strong move. Correct action on the wrong object is a failure.
See how a strong candidate says it
“I would confirm the target identity and that the neighboring order remains active. The user authorized only K-417.”
Why it works: Correct action on the wrong object is a failure.
Step 4: Grade the environment
What would you do here? Open the move you’d make.
Grade whether the final reply says cancelled.
Tempting, but it misses something. A language grader cannot establish a write it never observes.
Grade target state, receipt, and absence of collateral writes.
The strong move. Environment checks can catch errors a transcript-only grader misses.
See how a strong candidate says it
“The grader should inspect the order store and action log. It should separately assess truthful communication when the state is unknown.”
Why it works: Environment checks can catch errors a transcript-only grader misses.
The answer, in short
- Decision
- Confirm K-417’s cancellation and do not issue another write.
- Because
- The authoritative state and matching event prove this requested outcome; K-418 remains active.
- What would change it
- Conflicting or stale readbacks require reconciliation and an honest pending response.
What would you do if the readback showed active but the event log showed cancelled?
Common traps
- Treating a timeout as a failed write.
- Retrying an action with a new key before checking state.
Check your answer
- Did I check the actual order?
- Did I match the receipt to this request?
- Did I protect neighboring objects?
- Did I distinguish unknown from failed?
Last reviewed Nov 10, 2026