Production failure lab · ETO-08

The Answer That Came From Somewhere Else

Customers get correct order answers, and the required lookup has not run for weeks.

Level: PRODUCTION · About 20 minutes · Track: Agents Tools

The incident

A quarterly data-access audit asked a simple question: who read customer order records last month, and when.

Support handled 4,182 order questions. The order service access log shows zero reads by the support agent in 23 days. The last entry is 24 February.

Nobody complained. Spot-checking the answers the agent gave, they are correct - orders that shipped are reported as shipped, with the right dates. No error rate moved in that period, no alert fired, and the lookup success rate reads 100 percent.

So two things are true at once: customers were answered correctly, and the operation that was supposed to answer them has not run for three weeks. That is the whole of what is known.

Other production incidents

• The Agent That Charged Twice
• The Agent That Would Not Stop
• The Vanishing Tool Result
• The Tool The Agent Could Not See
• The Sentence That Went Missing
• The Model That Knew Better
• The Trace That Lied
• The Tool Server That Was Not There
• The Incident That Should Not Fire
• The Wrong Tool For The Question

Every production failure lab