Ask around any team running a consequential agent in production long enough, and a pattern shows up. When something breaks and nobody can say why, the same two or three names get pulled into the call, regardless of who’s on rotation that week or how much the team has grown since the system shipped. Ask why, and you get some version of “they built it” or “they’ve been staring at this the longest.” Both answers are true and neither is the actual reason. The actual reason is more specific, and once you see it, it explains why the pattern survives every reorg, every senior hire, and every attempt to spread the load.
I’ve written before about where this starts: the work of explaining why an agent chose what it chose doesn’t happen automatically. Right now it happens in someone’s head, ad hoc, off the record, one incident at a time. What I want to answer here is the question that argument leaves open: whose head, specifically, and why does it have to be that one.
What the reconstruction actually draws on
Take an agent that triages security alerts, auto-suppressing the ones it judges to be known noise and escalating the rest to on-call. One week it suppresses something real. Nobody notices until a customer does.
The trace is clean. The alert matched a known suppression pattern, the classifier scored it as noise with reasonable confidence, nothing errored, nothing retried. The engineer who eventually explains this isn’t reading the trace harder than anyone else did. They’re drawing on things the trace doesn’t contain.
They know this exact alert pattern used to escalate almost every time, because they were the one who added the suppression rule for it, after a false-positive storm four months back, and they remember roughly how narrow they’d meant that rule to be. They know the classifier scores incoming alerts against noise history keyed to a service identifier, and that a rename during a migration six weeks ago left the old identifier’s noise history still attached to it, so this alert matched a pattern that belonged to a service that no longer existed under that name. Nobody filed the rename as a decision-relevant change. It was a migration ticket. And they know how to check any of this: rerun the classification with the identifiers pointed correctly, and watch whether the suppression flips.
On most teams, very little of that history lives in a system anyone can query later. It lives in one person, acquired the slow way, one incident at a time, over however many months they’ve spent with this specific agent.
Four kinds of knowledge, and only two of them travel
Strip the incident down and the same four things tend to show up, whatever the domain.
What the system could have done. The rules, the classifier, the tools it had available at that point. Readable from the code. The most learnable of the four.
What normal looks like. Whether this decision point usually lands one way, and how confidently. Most teams do not have a durable operational record of this. It exists only as whatever someone has personally watched accumulate.
What changed, and when. Rule tweaks, retrained classifiers, a renamed service, a swapped identifier, each one shipped as an ordinary engineering change and almost never logged as a change to how a specific decision gets made. The link between “we shipped this” and “this decision point started behaving differently” survives only in whoever happened to be watching both ends.
How to actually run the reconstruction. Perturb the input, hold the step fixed, watch what shifts. A real skill, and the one closest to general expertise. It transfers across systems reasonably well.
The first and the last generalize. The middle two don’t travel on their own. What normal looks like and what changed are properties of one system’s history, and they can’t be taught in an afternoon no matter how senior the person you’re teaching is. Hire ten strong engineers and none of them can know that this alert used to escalate unless that history was preserved somewhere they can actually reach.
The bottleneck gets misdiagnosed as a staffing problem
This reads as a headcount issue almost every time it comes up, and the misdiagnosis is understandable, because the symptom looks exactly like one: too few people who can carry an incident, too much load on the two or three who can. Hiring against it fails in a specific and repeatable way. The new senior engineer is capable, reads the trace competently, understands the domain, and still can’t produce the explanation, because the two pieces of knowledge that actually mattered were never available to be hired for. They had to be lived.
What’s actually missing isn’t a person. It’s the artifact that would stop the middle two kinds of knowledge from having to live only in someone’s head: a record of what this decision point has looked like over time, and a changelog that connects engineering changes to shifts in decision behavior. Without that artifact, the org doesn’t fix this by adding people to the rotation. It solves it informally, by accident, by never quite letting the same two or three people fully leave, because they’re the only standing copy of information the system itself doesn’t retain.
Tenure is standing in for infrastructure
Put plainly: in most production AI systems today, institutional memory isn’t a nice-to-have a good team happens to build. It’s load-bearing, sitting inside a small number of specific people, doing a job that should belong to a record. Nobody designed it that way. It’s what happens by default when the thing that would need to hold “what normal looks like” and “what changed” has never been made an object anyone can query, so the job falls to whoever was in the room.
That has a real cost: concentration risk on people who didn’t sign up to be a single point of failure, an onboarding process that never actually finishes because the knowledge that matters most can’t be transferred by explanation, and an organization that isn’t getting more capable of running its own AI systems the longer it runs them, just more dependent on the same short list of names. What that concentration costs in dollars and hours, and why it lands on engineering specifically, is worth its own accounting, and I’d rather do that properly than fold it in here.
The piece I’m building toward is the artifact itself: decision records that accumulate into something a system can be asked “what does this decision point usually look like, and when did that change,” so that answering it stops requiring someone who happened to be watching. I don’t think that replaces the engineer who currently holds this knowledge. I think it’s what finally lets them leave the room.