When AI Systems Fail: The Architect's Incident Playbook
When an AI system fails in production, the first job is containment, not diagnosis: switch the workflow to a safe fallback (human review, a cached response, or a full stop) within minutes, before anyone tries to work out why it happened. Diagnosis comes second, and it depends entirely on whether the system was built to log its reasoning, not just its output. Most were not, which is why AI incidents take four times longer to resolve than they should.
That gap, between systems built to fail loudly and systems built to fail silently, is the difference between an incident that costs an afternoon and one that costs a quarter. It is also where most businesses discover, too late, that nobody designed for this moment at all.
What counts as an AI incident, and how is it different from a software outage?
A software outage has a clear signature: an error code, a stack trace, a service that stops responding. An AI incident often has none of these. The system runs, returns a plausible-looking answer, and the answer is simply wrong, or the agent takes an action nobody intended, and everything downstream of it looks normal until a human notices the consequence.
This is why AI incidents are undercounted. A 2026 retrospective of public AI incidents from January to mid-May catalogued more than fifty cases: hallucinations that reached customers, tool-misuse cascades that burned through budgets, prompt injections that exfiltrated data, and biased outputs that ended up in regulatory complaints. Hallucination was the single largest category, at roughly 35% of incidents, though its share is falling as tool-misuse and prompt-injection cases grow, evidence that hallucination has become a known, managed risk while the newer failure modes are still being discovered the hard way (Digital Applied, H1 2026 retrospective).
The architect’s definition is simple: an AI incident is any case where the system’s output or action diverges from what the business needed, badly enough that a human has to intervene. That includes wrong answers, unauthorised actions, data exposure, and silent underperformance that only shows up in aggregate.
What should happen in the first fifteen minutes?
The first fifteen minutes are not for finding the root cause. They are for stopping the damage from compounding while the system is still running unsupervised. A workable sequence looks like this.
- Detect. An alert fires, a support ticket spikes, or a human reviewer flags an output. The trigger matters less than having one at all; systems with no monitoring rely on customers to report the failure, which is the slowest and most expensive detection method available.
- Classify. Decide, in under five minutes, what kind of failure this is and how far it has spread: a single bad output, a pattern across many users, or an action already taken that needs to be undone.
- Contain. Flip the system into a safe mode: pause the agent, route new requests to a human, or roll back to the previous model or prompt version. This step exists specifically so nobody has to wait for executive sign-off mid-incident; without pre-authorised containment actions, teams routinely lose 45 to 90 minutes just getting permission to pull the switch.
- Communicate. Tell whoever is affected, internally and externally, what changed and what to expect next, before they ask.
Everything after this point, investigation, remediation, the formal report, happens on a slower clock. The point of the first fifteen minutes is simply to stop an isolated failure from becoming a systemic one.
Why do generic incident playbooks fail on AI systems specifically?
Most incident response playbooks were written for deterministic software: the same input produces the same output, so a bug can be reproduced, isolated, and patched. AI systems break that assumption. The same request can produce a different result after a model update, a change in retrieved context, a prompt edit, or nothing more than a shift in what the agent’s memory happened to contain that day. You cannot always replay an AI incident the way you would replay a server crash.
The deeper problem is that agents do not fail the way code fails. Code crashes; agents reason, and the reasoning is where the failure lives. A traditional postmortem asks “what line of code broke?” An AI postmortem has to ask “at what point did the system’s reasoning drift from the goal it was given?”, and answering that requires a record of the reasoning itself, not just the final output.
This is not a hypothetical gap. In an analysis of 73 production agent incidents gathered between January and May 2026, incidents without a logged reasoning trace took an average of 4.2 hours to resolve, more than four times longer than incidents where the full decision trace was captured, which typically resolved in under an hour (Sherlocks.ai). The same analysis found that in 61% of multi-layer incidents, a retrieval failure was the true upstream cause of what looked like a tool-calling error: the agent had called the right tool with the wrong context, not the wrong tool altogether. Without a trace, that distinction is invisible, and teams end up fixing the symptom (the tool call) while the actual defect (the retrieval step) ships again in the next release.
What should an architect build in before the incident happens?
Incident response for AI is mostly an architecture decision, made months before anything goes wrong, not a response written the night of the outage. The table below sets out what needs to exist beforehand and what it buys you when it does.
| Design-time control | What it does | What happens without it |
|---|---|---|
| Kill switch | Stops the agent or workflow instantly, without a deploy | Containment waits on an engineer being available and awake |
| Human-in-the-loop fallback | Routes uncertain or high-stakes decisions to a person automatically | The agent keeps acting on low-confidence outputs until someone notices |
| Reasoning trace logging | Records the agent’s intermediate steps, not just the final output | Postmortems guess at causes instead of reading them; MTTR roughly quadruples |
| Pre-authorised containment actions | Lets the incident commander act without seeking approval mid-incident | 45 to 90 minutes lost to sign-off before any fix can start |
| Version rollback capability | Reverts to the last known-good model, prompt or retrieval config in one step | Fixes are attempted live on the failing configuration |
None of these are exotic. They are the same discipline any mature engineering organisation applies to deployments, adapted to the fact that an AI system’s “bug” is often a reasoning error rather than a code defect. The architect’s job during diagnosis is to have made sure this instrumentation existed before diagnosis was ever needed, which is the same diagnose-first logic Bedrock AI applies to any AI build: map the failure modes the business actually faces before you deploy, not after the first one lands in a customer’s inbox.
How does a postmortem differ for an agent versus a piece of software?
A software postmortem traces a call stack back to the line that broke. An agent postmortem has to reconstruct a chain of decisions: what context the agent retrieved, what it inferred from that context, what tool it chose, and at which step its understanding of the goal diverged from what the business intended. Without a captured reasoning trace, this reconstruction is guesswork dressed up as analysis.
The output of a good agent postmortem is not just a fix; it is an update to the eval suite. If the failure case was not already being tested for, it becomes a permanent test, so the same failure mode cannot silently ship again in a future model or prompt version. Incident response and evaluation are the same discipline running on two different clocks: evals catch known failure modes before deployment; incident response catches the ones evals missed, and then feeds them back into the eval suite.
Who actually owns an AI incident: the architect, the vendor, or operations?
Ownership should be decided before the first incident, not negotiated during it. As a rule: the architect owns the design of the containment and logging systems, and defines what “safe mode” means for each workflow. Operations owns the moment-to-moment response, using the tools the architect built. The vendor owns defects in the underlying model or platform, which is why the vendor’s data processing agreement and support commitments need checking before a workflow goes live, not after it fails. A business that has not assigned these three roles in writing will spend the first hour of every incident arguing about whose job it is to press the button, which is exactly the delay a pre-authorised containment plan is meant to remove.
FAQ
Does every AI failure need a full incident response process? No. Minor, contained errors caught by a human reviewer before reaching a customer can be logged and fixed in the next release cycle. Full incident response is for failures that reached a customer, took an unauthorised action, or exposed data.
How is an AI incident different from a data breach? A data breach is a specific category of AI incident, one where the failure exposed information rather than producing a wrong decision or action. It follows the same detect-contain-diagnose sequence but adds mandatory legal and regulatory notification steps.
Can small businesses realistically build reasoning trace logging? Yes. Most agent frameworks can log intermediate steps with a configuration change, not custom engineering. The cost is storage and review time, both far cheaper than a 4-hour, untraceable incident.
Should the incident playbook live with IT, or with whoever owns the AI system? With whoever owns the AI system. IT teams are equipped for network and endpoint incidents; AI failures are judgement failures inside a specific workflow, and the people who understand that workflow need to be first responders, not IT.
What is the single highest-leverage thing to build before any incident happens? A kill switch paired with reasoning trace logging. The kill switch buys you time; the trace is what makes that time useful instead of just quiet.
Bedrock AI maps your systems, team and workflows to show where AI actually pays, before you spend a pound building. Book a strategy call.