I’ve written before about where this starts: the work of explaining why an agent chose what it chose doesn’t happen automatically. Right now it happens in someone’s head, ad hoc, off the record, one incident at a time. What I want to answer here is the question that argument leaves open: whose head, specifically, and why does it have to be that one.

What the reconstruction actually draws on

Take an agent that triages security alerts, auto-suppressing the ones it judges to be known noise and escalating the rest to on-call. One week it suppresses something real. Nobody notices until a customer does.

The trace is clean. The alert matched a known suppression pattern, the classifier scored it as noise with reasonable confidence, nothing errored, nothing retried. The engineer who eventually explains this isn’t reading the trace harder than anyone else did. They’re drawing on things the trace doesn’t contain.

They know this exact alert pattern used to escalate almost every time, because they were the one who added the suppression rule for it, after a false-positive storm four months back, and they remember roughly how narrow they’d meant that rule to be. They know the classifier scores incoming alerts against noise history keyed to a service identifier, and that a rename during a migration six weeks ago left the old identifier’s noise history still attached to it, so this alert matched a pattern that belonged to a service that no longer existed under that name. Nobody filed the rename as a decision-relevant change. It was a migration ticket. And they know how to check any of this: rerun the classification with the identifiers pointed correctly, and watch whether the suppression flips.

On most teams, very little of that history lives in a system anyone can query later. It lives in one person, acquired the slow way, one incident at a time, over however many months they’ve spent with this specific agent.

Four kinds of knowledge, and only two of them travel

Strip the incident down and the same four things tend to show up, whatever the domain.

What the system could have done. The rules, the classifier, the tools it had available at that point. Readable from the code. The most learnable of the four.

What normal looks like. Whether this decision point usually lands one way, and how confidently. Most teams do not have a durable operational record of this. It exists only as whatever someone has personally watched accumulate.

What changed, and when. Rule tweaks, retrained classifiers, a renamed service, a swapped identifier, each one shipped as an ordinary engineering change and almost never logged as a change to how a specific decision gets made. The link between “we shipped this” and “this decision point started behaving differently” survives only in whoever happened to be watching both ends.

How to actually run the reconstruction. Perturb the input, hold the step fixed, watch what shifts. A real skill, and the one closest to general expertise. It transfers across systems reasonably well.

The first and the last generalize. The middle two don’t travel on their own. What normal looks like and what changed are properties of one system’s history, and they can’t be taught in an afternoon no matter how senior the person you’re teaching is. Hire ten strong engineers and none of them can know that this alert used to escalate unless that history was preserved somewhere they can actually reach.

The bottleneck gets misdiagnosed as a staffing problem

This reads as a headcount issue almost every time it comes up, and the misdiagnosis is understandable, because the symptom looks exactly like one: too few people who can carry an incident, too much load on the two or three who can. Hiring against it fails in a specific and repeatable way. The new senior engineer is capable, reads the trace competently, understands the domain, and still can’t produce the explanation, because the two pieces of knowledge that actually mattered were never available to be hired for. They had to be lived.

What’s actually missing isn’t a person. It’s the artifact that would stop the middle two kinds of knowledge from having to live only in someone’s head: a record of what this decision point has looked like over time, and a changelog that connects engineering changes to shifts in decision behavior. Without that artifact, the org doesn’t fix this by adding people to the rotation. It solves it informally, by accident, by never quite letting the same two or three people fully leave, because they’re the only standing copy of information the system itself doesn’t retain.

Tenure is standing in for infrastructure

Put plainly: in most production AI systems today, institutional memory isn’t a nice-to-have a good team happens to build. It’s load-bearing, sitting inside a small number of specific people, doing a job that should belong to a record. Nobody designed it that way. It’s what happens by default when the thing that would need to hold “what normal looks like” and “what changed” has never been made an object anyone can query, so the job falls to whoever was in the room.

That has a real cost: concentration risk on people who didn’t sign up to be a single point of failure, an onboarding process that never actually finishes because the knowledge that matters most can’t be transferred by explanation, and an organization that isn’t getting more capable of running its own AI systems the longer it runs them, just more dependent on the same short list of names. What that concentration costs in dollars and hours, and why it lands on engineering specifically, is worth its own accounting, and I’d rather do that properly than fold it in here.

The piece I’m building toward is the artifact itself: decision records that accumulate into something a system can be asked “what does this decision point usually look like, and when did that change,” so that answering it stops requiring someone who happened to be watching. I don’t think that replaces the engineer who currently holds this knowledge. I think it’s what finally lets them leave the room.

Before observability, a production system’s internals were genuinely dark. You had metrics, which told you the aggregate was healthy, and you had logs, which told you what one thing printed. Between them sat an enormous gap, and the way you crossed it was to guess. What changed was not volume. Teams already had plenty of data. What changed was that executions became high-cardinality objects you could interrogate after the fact, which meant you could ask a question nobody had thought to instrument for in advance. That is the whole trick, and it was a real one. The industry moved from “is the system healthy” to “what exactly happened on this request,” and it moved there in about five years.

That capability transferred directly to AI systems, and it works. I can trace an agent run end to end, see every tool call, replay the sequence, and decompose a failure into steps. The tooling is good. The engineers who built it were solving a real problem and they solved it.

Its primary unit of investigation is an execution: the request, the span, the trace, the event.

That is not a criticism, it is a design fact, and for the systems observability was built for it was the correct choice. In conventional deterministic application logic, the same state and inputs are expected to produce the same result. The execution path therefore tells you a great deal about the behavior you should expect to see again.

Probabilistic systems break that equivalence, and they break it at the root.

The execution is no longer the behavior

Run the same agent against the same input a hundred times and you can get a hundred slightly different runs. Most of them fine. Some of them different in ways that matter. No single one of them is the system’s behavior. The behavior is the shape of the whole hundred: what it usually reaches for, how tightly it clusters, where the tail sits, and how all of that shifts when the context changes underneath it.

This is why the most common instinct is the wrong one. When a team cannot explain what their agent did, the reflex is more instrumentation: capture more of each run, log the intermediate reasoning, retain more context. That reflex is correct in a deterministic system, where the missing explanation really is hiding in some field nobody captured. Here it produces a richer description of one sample from a distribution you still have not characterized.

You can see the mismatch in what aggregation currently gives you. Error rate, p95 latency, token spend, tool-failure counts: those are real aggregates over many runs, and every one of them aggregates execution. They describe the machinery. There is no standard production primitive for judgment equivalent to error rate or latency, and the reason is structural rather than a gap in anyone’s roadmap. Aggregating something operationally requires it to be an object first, and judgment has not become one. I spent a whole piece on what it would take to make it one, so one line will do here: a trace reliably retains the path taken, and by itself it does not turn the selection of that path into a comparable decision object.

So the stack has an aggregate view of execution and an instance view of nearly everything else. Both halves are load-bearing, and neither is the thing production teams keep reaching for.

The questions being asked are comparative

Watch what someone actually asks when an AI system does something strange in production. Almost none of it is answerable about a single run.

Is this normal for this agent? Has it done this before? Did something change, or has it always been like this and we only just noticed? Is this one workflow or is it everywhere? Is it getting worse?

Every one of those is a comparison against a population. Not one of them can be answered by looking harder at the run in front of you, no matter how much of that run you captured.

| Question                                        | Where it can be answered  |
| ----------------------------------------------- | ------------------------- |
| Did the request succeed                         | Execution                 |
| Which steps ran, in what order                  | Execution                 |
| How was this input read, and what else was live | The decision, as a record |
| Was this particular choice sound                | The decision, as a record |
| Is this normal for this system                  | A population of decisions |
| Has this failure shape occurred before          | A population of decisions |
| Is judgment degrading, and since when           | A population of decisions |

The bottom three rows are where production teams live, and they are the rows with nothing underneath them today.

What a population gives you that a record cannot

Consider an agent that triages inbound requests and decides which ones escalate to a human. Pick any single escalation it declined and inspect it properly, with the decision reconstructed rather than inferred: how the request was read, which alternatives were live, how close the call was, which governing signals actually figured in it. The call may look perfectly defensible on its own.

Now hold that same decision point still and look at every instance of it over the past two months. A different object appears. The margin between escalate and resolve has been narrowing steadily. A category of request that used to sit clearly on the escalate side now lands just barely on the other. Nothing broke. No error fired. Every individual decision in that drift was defensible on its own terms, which is precisely why nobody caught it, and why the first visible symptom arrives as a customer complaint rather than an alert.

Three things become available across a population that are unavailable in any single decision, no matter how well captured.

A baseline. You know what normal latency looks like for your service. You almost certainly do not know what normal judgment looks like for your agent: how decisive it usually is at a given decision point, which signals usually drive it, how wide its considered set typically runs. Without that, “unusual” is not a claim anyone can make, and every anomaly assessment is somebody’s intuition.

Drift as a first-class signal. Slow failures are often hard to see at the individual-output level because no single output looks obviously wrong. They are visible as distribution shift: margins tightening, a decisive signal quietly dropping out of contention, a stale assumption recurring across runs that should have decayed. You see judgment degrade before answers do.

Recurrence. This is the one teams underrate most. Right now, when an agent fails in a way that requires real investigation, that investigation starts from zero. The engineer reconstructs what happened, reaches a conclusion, fixes something, and the reconstruction evaporates. Six weeks later the same failure shape appears in a different workflow and the next engineer starts from zero again. The team is not learning how the system behaves. It is repeatedly rediscovering it, at senior-engineer rates, and paying the full cost every time.

That last one moves the problem out of debugging and into organizational memory, which is a different category of thing entirely.

A different object, not a better dashboard

It would be easy to hear this as an argument for adding behavioral charts to an observability product, and the distinction is worth drawing carefully, because it is not a claim that anyone is technically incapable of building it.

Observability platforms were built around an engineer who reads traces. Eval frameworks were built around someone who writes them. Both assumptions are reasonable and both describe people who build AI systems. The questions in the bottom three rows of that table are asked by a different person: whoever is responsible for what the system actually does once it is running, which is frequently not the same person and frequently has no ML engineer to hand the question to.

The deeper difference is the object of intelligence. Observability’s object is execution, and it is genuinely good at it. The object here is judgment: not whether the machinery ran, but whether the system reasoned, chose, retrieved, and adapted in a way that made sense, and whether that has been holding steady or quietly moving.

Any of this could live inside the same product someday, and an existing platform could extend its data model to reach it. What it cannot be is another chart drawn over the same primitive. You have to make the decision a first-class object before a population of decisions exists to analyze, and that is a change to the object model rather than an addition to the interface. Every trace you already collect stays exactly as useful as it was, which is the part I want to be unambiguous about. Nothing here argues for ripping anything out.

Behavioral intelligence is the term I will use for the capability that appears once decisions accumulate: the comparative, accumulated view of how a system decides, built from decision records evaluated as a population over time. I hold the label loosely, as I have from the start, because the market picks the word that names the problem people feel and nobody assigns that from the outside. The capability is the durable part. What it has to deliver is a baseline for judgment, drift you can see before outputs go bad, and memory that survives the engineer who did the last investigation.

It is also worth being clear about the limits. This does not make decisions directly observable, because nothing does. It does not tell you whether a business outcome was correct; that judgment sits above this work and stays with a human. It will not give you certainty about any single decision, since a reconstructed decision is an estimate with a derivation, not a readout. What changes across a population is that estimates become comparable to each other. One reconstruction is an estimate. Hundreds of comparable estimates moving together over time become evidence of a pattern.

When decisions become a population

There is a pattern in how this stack has grown, and it is not that each stage collects more data than the one before it. Metrics existed long before observability, and teams drowning in logs did not lack volume. What changed each time was the unit: the metric gave us the aggregate, the trace gave us the request, and each time the unit changed, a class of question became askable that had previously required a guess.

The decision is the next unit. Behavioral intelligence is not a further unit stacked on top of it, and I want to be careful not to imply otherwise. It is what a population of that unit makes possible, in the same way a single trace is interesting and a corpus of traces is operational. The decision record is the atom. The behavior is what the atoms show you when there are enough of them to compare.

Probabilistic systems are going to keep being probabilistic. That is not a defect to be engineered away, it is the property that makes them useful, and the teams shipping them are not asking for determinism. They are asking for dependable outcomes from a system that will never be deterministic, which is a different and much more tractable request. Dependability under uncertainty has never come from eliminating variance. It comes from characterizing it: knowing the distribution, knowing where the edges are, and noticing early when it moves.

That is the capability the AI stack does not have yet, and the reason it does not have it is that the thing it would need to characterize has never been treated as an object.

Execution tells us what happened. A decision record tells us why this happened. A population of those decisions tells us what normal looks like.

I am building Nalyqor across those last two: reconstructing decisions as records, then treating them as a population that accumulates. The bet is that this is where the stack goes next, and that the teams who need it soonest are the ones running consequential AI without anyone whose job is watching how it decides. That part is a thesis, not a finished argument. The diagnosis I am confident about: the questions production teams keep asking are comparative, and comparative questions need a population nobody is currently keeping.

Except it is being resolved constantly. Every company deploying agents answers it in writing, every week: in the org chart that gives the agent a team, in the sign-off flow that puts a human before the action, in the vendor contract that caps liability, in the policy doc that names an owner per workflow. The question is not open. Somebody’s name is already on the decision.

What is actually open is a different question, one level down: what is the named person holding? Take any of those ownership assignments and look underneath it for the object it refers to. The team owns “the agent.” The reviewer owns “the approval.” The vendor owns “the model.” Nobody owns the decision, because the decision, as an artifact someone could be handed, examine, and answer for, does not exist anywhere in the system. The ownership is real. The thing owned is not.

That inversion is the piece. The hard problem in AI accountability is not assigning owners. Organizations are extremely good at assigning owners; it is what org charts are for. The hard problem is that accountability requires an object, and these systems do not produce one.

Accountability means answerability

Strip the compliance overlay off the word, and accountability means something specific: being answerable. The owner of a decision is the person who can be called on to explain it. What was decided. How the situation was read. What else was on the table. What tipped the choice. Whether, knowing all of that, the call was sound.

Which means accountability has two components that are easy to conflate. There is the obligation to answer, and there is the capacity to answer. The obligation is organizational: you can assign it, transfer it, write it into a contract. The capacity is not organizational at all. It is a property of the evidence. No policy document can create it, because it either exists in what the system retained about the moment of choice or it does not exist at all.

Human institutions rarely have to think about the difference, because in human decision-making the two travel together. The person who made the call was in the room and can be asked. And where memory is not trusted, institutions force the capacity into artifacts: the deal memo, the medical chart, the written judicial opinion. Consequential deciders are made to leave a record of their reasoning precisely so that answerability survives the moment.

Autonomous systems split the two components cleanly, and the split runs in the worst possible direction. The obligation lands on humans, necessarily; a model cannot be answerable, and sanctioning one means nothing. The capacity is available at the moment of selection, and it evaporates when the output lands. I have spent most of this newsletter on that mechanism, so one line will do here: execution is observable, decisions are not, and what a trace retains is the path taken, not the selection of that path. The evidence that answerability needs, how the input was read, what was in contention, how close the call was, is exactly the evidence the stack does not keep.

So the accountability structures being written right now share a defect that has nothing to do with how thoughtfully they assign the obligation. They assign it against capacity that was never generated. An accountability framework built this way is a set of promises to answer questions with evidence nobody collected.

Watch the chain under load

None of this is visible while things go well, because accountability is only ever exercised in the breach. So put the chain under load.

An agent with real authority, say it prices contract renewals inside negotiated bounds, makes a call that surfaces six weeks later as a problem: renewals priced defensibly wrong, in a pattern, across a quarter. The accountability chain activates, and everyone in it does their job.

The operating team produces the trace. Steps executed in order, tools returned cleanly, no errors, nothing anomalous. The reviewing human produces the approval record: at sign-off, the numbers sat inside the negotiated bounds and looked consistent with policy. The vendor produces the evals: the model performs within expectations on the benchmark suite. The policy owner produces the policy: the written rule was correct as written.

Every one of them answered. Now look at what each answer was actually an answer to.

| Who gets asked      | What they can produce | What their evidence answers         |
|---------------------|-----------------------|-------------------------------------|
| The operating team  | The trace             | Did the steps execute correctly     |
| The reviewing human | The approval record   | Did the output look acceptable      |
| The model vendor    | Evals and benchmarks  | Does the model perform in aggregate |
| The policy owner    | The written policy    | Was the rule defensible as written  |
| No one              | Nothing               | Was this decision sound             |

The question the incident actually raised, was this decision sound and who should have caught it, was never answered by anyone, because it never attached to any evidence anyone held. It got decomposed into four adjacent questions, each answerable, none of them it.

This is the signature failure mode, and it is worth naming precisely because it does not look like failure. Nobody stonewalled. Nobody’s evidence was wrong. Accountability in these systems does not break. It disperses.

And a question that cannot attach to evidence does not stay open, because organizations abhor an open incident. It resolves to one of two defaults. Either it lands on the nearest human, usually the reviewer, whose approval is the last place a person visibly touched the outcome. Or it lands on no one: the postmortem closes with a narrower rule and a new guardrail, and the mechanism that produced the decision is left exactly as answerable as it was before, which is to say not at all. I walked that second ending in the silent-failure piece. The first ending is worse, and it is next.

The human in the loop is a co-signer

Human-in-the-loop is the load-bearing answer in nearly every accountability framework I have seen: a person approves consequential actions; therefore a person owns them. It is the mechanism that lets the ownership chart close. It is also the clearest case of obligation assigned without capacity, because you can enumerate exactly what the approver is shown.

They see the output. Maybe a diff, maybe a summary, maybe the top of the supporting context. Here is what they do not see, and cannot, because it was never captured: how the system read the situation, which alternatives were live, how close the call was, and which of the governing signals actually figured in the choice versus merely sat in the context. The primitives of the decision, in other words. The approver is being asked to underwrite a selection while being shown only its result.

There is a word for adding your name to a risk you did not originate and cannot inspect, and it is not “reviewer.” It is co-signer. A co-signature changes who pays when things go wrong. It does not change whether anyone understood the risk. A human in the loop who can see the output but not the decision is not exercising oversight. They are terminating a liability chain, and most of these loops are shaped as if that were the goal.

This is also why approval fatigue is not a discipline problem, much as it gets treated like one. When approval cannot engage the decision, it degrades into pattern-matching on outputs, and outputs are precisely where these systems look most fine. Locally coherent, every time; that is what makes the failures silent. An approver who waves through a hundred clean-looking outputs is not being lazy. They are doing the only version of the job the evidence in front of them permits.

So the loop does something subtler than fail. It moves blame downward while leaving answerability nowhere. The human absorbs the liability. The judgment, the thing accountability was supposed to reach, stays exactly where it was: inside a selection process nobody recorded.

The auditors are arriving at the same place

Regulatory frameworks are beginning to expose the same tension from the outside. The ones now coming into force for consequential automated decisions, the EU AI Act most visibly, keep circling two requirements: records sufficient to trace the system’s operation, and human oversight that is effective rather than nominal. Set aside the compliance mechanics and look at the diagnosis embedded in that language. Regulators did not need to be told that a human stapled to the end of a pipeline is not oversight. They wrote “effective” into the requirement because the nominal version is what everyone builds by default.

The catch is that the record-keeping half is being satisfied with the records that exist, and the records that exist are execution records. A system can retain every log line, replay every run, and pass a records audit in the literal sense while remaining unable to say why it chose anything. Which means the two requirements are quietly in tension: the oversight cannot become effective on the evidence the record-keeping produces. Sooner or later, an audit asks the same question the incident review asked, and it disperses the same way.

Ownership follows evidence

The question in the title does have an answer. It is just an answer to a slightly different question than the one people mean when they ask it.

Who owns the decision? Whoever can be handed it. Ownership follows evidence. It always has; that is what the deal memo and the medical chart and the written opinion were for. Institutions that needed decisions to be owned never relied on memory or good intent. They made the decision durable enough to examine, and ownership attached to the artifact. Not because writing is sacred, but because answerability needs an object, and an object is the one thing an org chart cannot conjure.

Until the decision exists as that kind of object, structured, inspectable, comparable against the alternatives that were live at the moment of choice, “who owns the AI’s decision” will keep resolving to its two degenerate answers: the nearest human, or no one. Assigning the obligation harder does not help. Obligation without capacity is not accountability. It is liability with better formatting.

I wrote a few pieces back that “the model decided” is no longer an acceptable end of a sentence. It turns out “a human approved it” is not one either, and for the same reason: both sentences stop exactly where the evidence stops. The decision has no owner yet because there is nothing yet to own. Produce the decision as evidence and the ownership question stops being philosophical. It becomes what it has always been in every other consequential domain: a records question, with a records answer.

That number is real. It is also, for most of the decisions worth caring about, off by orders of magnitude, and wrong in a specific direction. It captures the cheapest thing that happened.

I have written before about the gap this sits inside: execution is observable, decisions are not. What that gap does to a balance sheet is less discussed. The money does not disappear. It lands, in full, in six different places, on six different clocks, under owners with no reason to connect any of it back to the moment a system chose wrong.

The rule that manufacturing figured out first

Quality management has a rule of thumb, sometimes called the rule of ten: a defect caught at design costs a fraction of one caught on the production line, which costs a fraction of one that reaches a customer. The specific multipliers vary by industry and most of the numbers attached to them are closer to folklore than measurement.

The structure is not folklore. The cost of a defect is a function of how far it travels before anyone notices, because distance is what determines how much got built on top of it.

AI systems inherit that structure and then make it worse in one specific way. A production line makes a bounded number of decisions, and most of them are inspected. An agent makes many decisions per task, continuously, and almost none of them are inspected individually. The travel distance is longer and the inspection rate is lower, which is the exact combination the rule of ten warns about.

Six places the money lands

Take one wrong decision and follow it forward rather than backward.

The decision itself. The tokens spent producing the wrong choice. This lands within seconds, on the bill, and it is genuinely indistinguishable from the tokens spent producing a right one. A good decision and a bad decision cost the same to make.

The recovery. Retries, re-retrieval, the agent going back around on something it already had enough in front of it to settle. This also lands on the bill within the same minute, filed as usage rather than as rework, because nothing in the billing model has a category for rework.

The downstream execution. Everything the system then did correctly, on a wrong premise. This is the first component large enough to notice, and it is the first one that gets attributed to the wrong thing: it shows up under whatever workflow ran, not under the decision that set it running. Perfectly efficient execution of a bad choice is still waste, and it bills as throughput.

The detection. Engineering hours at the surfacing end. Someone works backward from a symptom, and every step they pass through on the way there looks correct, because each one was. This does not appear on the AI bill at all. It appears on payroll, days or weeks later, usually filed under an incident that names the symptom rather than the cause.

The correction. The fix, plus the standing tax of whatever guardrail gets added to prevent a recurrence. Guardrails are not free. Every one of them is latency and complexity that the system carries forever, paid for by a decision that happened once.

The forfeited scope. The work you decide not to hand the system next time. A team that gets burned narrows what it delegates, and that narrowing is permanent until something changes their mind. This is almost certainly the largest number of the six, and it is the only one that appears in no ledger anywhere, because it is not a cost that gets incurred. It is a benefit that never gets collected.

The ordering is the finding. Those six arrive in roughly ascending order of size, and in strictly descending order of visibility. The cheapest component is the only one anyone can point at.

A worked example

Sanitized, and deliberately mundane, because the expensive cases usually are.

A support agent triages inbound tickets by entitlement. A customer writes in about a charge they did not expect after a plan change. The agent reads this as a billing dispute and routes it to the disputes workflow. The other available reading, a service credit request under an existing entitlement, was equally defensible on the text. Nothing about the decision is anomalous. Both interpretations are ones a competent human would consider.

The disputes workflow is heavier. It pulls account history, runs a verification pass, and drafts a response with a different tone and a different set of obligations. All of it executes correctly. The customer gets a coherent answer to a question they did not quite ask, and either accepts it or writes back, which starts the whole thing again from a worse position.

Multiply that by the ticket volume that shares the same ambiguity, and the interesting cost is not the misroute. It is the disputes workflow running thousands of times on cases that were never disputes.

Nobody finds this by looking at the bill. The bill says disputes volume is up.

| Cost layer           | Where it lands                | When it lands    | Attributed to the decision?                    |
|----------------------|-------------------------------|------------------|------------------------------------------------|
| The decision         | AI spend                      | Seconds          | No, it looks like any correct call             |
| Recovery loops       | AI spend                      | Seconds to hours | Filed as usage, never as rework                |
| Downstream execution | Another workflow's line item  | Minutes to days  | No, credited to what ran, not what chose       |
| Detection            | Engineering hours             | Days to weeks    | Only if someone traces it back by hand         |
| Correction           | Hours, plus permanent latency | Weeks            | To the incident, not the decision class        |
| Forfeited scope      | Nowhere                       | Quarters         | Never, no system records a benefit uncollected |

Why the accounting cannot close

Two separations do the damage, and neither is a tooling oversight that gets patched next quarter.

The first is time. The top three layers land inside a minute. The bottom three land days to quarters later. By the time the expensive part arrives, the cheap part has been aggregated into a monthly number and the causal thread is gone.

The second is boundary. The top three land in the AI budget. The next two land in payroll. The last one lands nowhere at all. Those are different systems, owned by different people, reconciled on different cycles, with no shared key between them. No accounting system joins a token to an engineer’s afternoon three weeks later, and there is no reason anyone would have built one.

So the cost gets split across ledgers that cannot see each other, and every ledger is individually correct. The AI bill correctly reports spend. The incident record correctly reports hours. Neither is wrong. Together they still fail to say that one decision, made in a fraction of a second, caused both.

The instrumentation gap has a financial twin, one level up. A trace is a record of steps, not of choices. A ledger is a record of amounts, not of reasons.

What you can actually estimate, and what you cannot

I want to be careful here, because this is exactly the point where a piece like this usually starts inventing precision.

You cannot measure most of the six. You can bound some of them, and the honest framing matters more than the arithmetic:

• Downstream execution is recoverable, but only if you can identify the originating decision, which is the hard part and the whole problem. The compute is already logged. What is missing is the link from it back to the choice that triggered it.

• Detection and correction are the most tractable, because incident records already hold rough hours. They are just filed against symptoms, so recovering them means re-attributing existing data, not collecting new data.

• Recovery loops can be approximated by comparing a workflow’s actual path against its shortest successful path, which is an estimate and behaves like one.

• Forfeited scope cannot be measured at all. It is only ever visible in what a team quietly stops automating.

Anything that hands you a clean dollar figure per bad decision is estimating, and should say so on its face. The useful output here is not a number. It is a ranking: which decision patterns are expensive, and roughly by how much, relative to each other.

The unit that is missing

Every AI cost tool I have used reports cost per token, cost per call, cost per workflow. Those are consumption metrics. They tell you what the system used.

None of them report cost per decision, which is the only unit that tells you what the system wasted.

That is not a gap in the tools. It is a gap in what the tools can see. Consumption is observable from the outside; a decision’s quality is not, and you cannot build a cost-per-decision number on top of a stack that has no representation of the decision in the first place. This is the direction I am building Nalyqor toward, and I would rather it be the direction the whole category goes: cost attribution that resolves to the choice rather than to the call.

In closing

The reason bad AI decisions feel cheap is not that they are cheap. It is that their price is paid in six installments, five of which arrive after anyone has stopped looking, in currencies that do not convert.

Teams optimizing AI spend right now are optimizing the one installment that arrives first and costs least, because it is the only one denominated in something the invoice understands. That work is not wrong. It is just aimed at the one installment the invoice can denominate, on a total nobody has actually computed.

An invoice is not a measurement of a mistake. It is a receipt for the cheapest part of one.

Thanks for reading The Agent Loop, Powered by FAI! Subscribe for free to receive new posts and support my work.

Every postmortem starts at the wrong end

When a production incident gets investigated, the investigation starts where the pain showed up: the timeout, the complaint, the number that finally moved. From there, someone works backward, hop by hop, until they find the step that first went wrong. That process is necessary, and it is also structurally backward. It starts at the one moment the failure was loud and reconstructs everything before it, which is exactly the part that was quiet.

A silent failure is not an event. It is a sequence, and most of that sequence happens while every dashboard stays green. If you only ever study the loud end, you learn what a failure looks like once it has already stopped being silent, which is the one part of its life that was never the hard part to see.

So run it the other direction. Start at the decision, and follow it forward through every stage that had to go right, for the system, for the failure to stay hidden that long.

The five stages

A silent failure has the same basic anatomy almost every time, regardless of which behavioral pattern it turns out to be. Five stages, in order:

• Origination. The point where a decision quietly diverges from the one that was actually warranted.

• Local coherence. The output of that decision, examined on its own, looks completely fine.

• Absorption. The layers built to catch problems each look at the wrong part of the decision and pass it through.

• Compounding. The divergence gets treated as precedent by later decisions, and its footprint grows.

• Surfacing. Something finally crosses a threshold, usually somewhere structurally distant from where the divergence began.

None of this is exotic. It is the same mechanism whether the underlying pattern is drift, context erosion, or a stale assumption that never decayed. What changes is the content. The shape stays constant. Walking one case through all five stages is more useful than adding a sixth item to a list of failure types, so here is one, sanitized.

The scenario, sanitized

A CI pipeline has an agent that reviews routine dependency bumps. Patch-level version updates from a small set of trusted packages get auto-merged if the test suite passes. Anything else waits for a human. Ordinary, sensible policy.

Early on, the agent handles a minor-version bump, one step above what the policy actually covers, from a maintainer it already trusts. Tests pass, the diff is small, nothing looks wrong, and it merges. Nobody wrote that exception down anywhere. It just happened once, and it worked.

Stage 1: Origination

That single merge becomes an unlabeled precedent. The next time a similar bump comes from a maintainer, the system already associates it with “safe,” the interpretation it reaches for is the one that already worked, not the policy that was actually written. This is a decision problem before it is anything else: an interpretation forms, gets weighted into later reasoning, and nothing marks it as an exception rather than a rule. I have written before about the primitives a decision like this actually has: interpretation, the considered set, weighting, confidence, context state, and this is what it looks like when one of them quietly wins without ever being checked against the policy it was supposed to defer to.

Stage 2: Local coherence

Here is the part that makes this hard to catch: every individual merge, looked at by itself, is fine. Tests are green. The diff is small. Nothing about this specific decision is anomalous compared to the thousand other routine merges that really were routine. A reviewer glancing at any one of these commits would approve it without a second thought, because in isolation, it deserves that.

This is true of nearly every stage of a silent failure. It is never the decision itself that looks wrong. It is the pattern across many decisions, and no single decision carries the pattern.

Stage 3: Absorption

Everything built to catch a problem here is pointed at the wrong layer. The trace shows a merge that completed successfully, which is accurate. The test suite passed, which is also accurate. If there is an eval on this workflow, it almost certainly grades whether the merge was safe on that input, and this one was; the tests really did pass. None of these instruments were built to ask whether this decision matched the policy’s actual scope. They were built to ask whether the output was correct, and the output was.

This is the same collapse from a different angle each time: execution records what happened, not why it was chosen. An eval grades the output, not the decision that produced it. Here, that collapse means a policy exception gets treated as a policy success, three separate times, by three separate instruments, none of which were wrong about what they measured.

Stage 4: Compounding

The next borderline bump gets judged against the drifted precedent, not the original policy, because the drifted precedent is now what “safe” has come to mean inside the system’s own reasoning. Each new case that fits the pattern gets folded in the same way. The category of “routine” widens, one defensible decision at a time, and every one of those decisions is locally coherent for the same reason the first one was.

Nothing about this shows up as a spike. Merge volume looks normal, maybe even improves, since fewer borderline cases get flagged for review. An aggregate metric, watched for a threshold breach, is the wrong instrument for a slow directional shift that never crosses one.

Stage 5: Surfacing, and the misattribution that follows

Weeks later, a downstream service starts intermittently timing out. Nobody connects this to a dependency bump at first, because nothing about the incident points there. An on-call engineer spends the first few hours on the service itself: retry logic, connection pooling, recent deploys to that service specifically. Eventually, someone traces a changed default, a serialization behavior that shifted in a minor-version bump three weeks back, and finds it was auto-merged.

Here is the part worth sitting with. The natural fix, the one that gets shipped the same week, is usually a narrower rule: exclude this specific package, or require review for this specific maintainer going forward. That patches the symptom. It does not touch the mechanism, which was never about that package. It was an unlabeled precedent that will form again, around a different maintainer, the next time a bump gets treated as routine because an earlier one worked out. The investigation started at Surfacing and worked backward through a chain of individually valid steps, which is exactly the shape that makes it easy to fix the step you found and miss the pattern that produced it.

The anatomy at a glance

| Stage           | What's visible                                         | What's actually happening                                                             |
|-----------------|----------------------------------------------------------|-------------------------------------------------------------------------------------|
| Origination     | A single decision, unremarkable in the trace           | An interpretation forms and gets weighted without being checked against policy        |
| Local coherence | Each individual output looks correct                   | The pattern lives across decisions, not inside any one of them                        |
| Absorption      | Trace, tests, and evals all report healthy             | Each instrument is grading the artifact, not the interpretation behind it             |
| Compounding     | Aggregate metrics stay flat or improve                 | The category the drifted precedent covers keeps widening                              |
| Surfacing       | A symptom appears, structurally distant from the cause | Discovery starts at the symptom and reconstructs backward through valid-looking steps |

In closing

The instinct, once a silent failure gets discovered, is to ask which check should have caught it. That question assumes the failure was a single missed detection, sitting at one layer, waiting for the right instrument to look at it. It usually was not. It was coherent at every stage because every instrument evaluated a different artifact than the one that actually failed. The better question is which stage the signal actually died at, and why that stage was structurally incapable of seeing it, because that is the question whose answer generalizes to the next failure instead of just patching this one.

That is also why I think the fix has to be something that can walk this anatomy after the fact, stage by stage, rather than something an engineer reconstructs from scratch under pressure each time. That’s the direction I’m building Nalyqor toward: not another alert at the point of surfacing, but a way to identify failures much closer to their point of origination.

A failure that never announces itself did not skip an announcement. It made it, quietly, at every one of these five stages, to instruments that were built to hear something else.

Thanks for reading The Agent Loop, Powered by FAI! Subscribe for free to receive new posts and support my work.

The reflex explanation is that the eval set was incomplete. Add the missing case, raise coverage, move on. That explanation is comforting because it implies the gap is a quantity problem, and quantity problems get solved by grinding. I don’t think it’s a quantity problem. I think a system can pass an eval set with perfect coverage and still fail in production, for reasons that have nothing to do with how many cases you wrote and everything to do with what an eval is structurally able to measure.

Thanks for reading. This piece is the part of the argument I keep circling back to: the instrument and the thing it’s pointed at are not the same shape.

What an eval actually measures

Strip an eval down to its mechanism. You take a frozen set of inputs, run the system on each, and grade the outputs against an expected answer or a rubric. Pass rate is the fraction of outputs that cleared the bar. That is the whole instrument, and it is a good instrument for what it measures: output correctness, on a fixed distribution, one input at a time.

Now look at what each of those properties quietly assumes, because production violates all three.

An eval grades the output. Production is generated by a decision. The grade lands on the final answer, which is the last step of a process and the one place all the interesting structure has already collapsed. An output that was produced by a clean, decisive, well-grounded choice and an output produced by a near-tie between two alternatives, one of which was wrong, look _identical_ on the rubric. Both are the correct string. Both pass. But only one of them is a system you can trust the next time the context shifts the margin by a hair. The eval cannot tell them apart, because it is grading the thing they have in common and ignoring the thing that distinguishes them. A correct output sitting on top of a marginal decision is a latent failure wearing a passing grade, and the eval is exactly the instrument that can’t see it.

An eval fixes the distribution. Production drifts it. The test set is sampled once and frozen. That is what makes it a measurement at all, you need a fixed yardstick. But production is non-stationary by construction: the inputs move as the world and your users move, and inside any multi-turn session the context accumulates, so the distribution the system actually faces at turn nine is one no eval case ever held still long enough to capture. The eval certifies behavior on the distribution you froze. Production runs on the one that’s drifting away from it the moment you ship.

An eval scores the aggregate. Production dies in the tail. Pass rate is a mean, and a mean is precisely the statistic that buries the failures that matter. The production-killing failures are rare, correlated, and context-triggered, the specific combination of a stale assumption, an unusual ordering, and a retrieval that didn’t fire. Each is individually low-frequency, so each contributes almost nothing to the aggregate. 98% looks like a system that’s basically right. It can equally be a system that is reliably right on the easy 98% and reliably wrong on the 2% that happens to be where your highest-value, highest-consequence traffic lives. The number is the same. The system is not.

None of these is fixed by writing more cases. More cases sharpen the estimate of output correctness on a frozen distribution. They do not turn an output measurement into a decision measurement, they do not un-freeze the distribution, and they do not stop the mean from hiding the tail. You can drive coverage to a place that feels exhaustive and still have measured none of what production is about to do to you.

The Goodhart turn

And that’s the failure mode if teams simply accept these limits. There’s a worse one that shows up the moment the eval stops being a gauge and becomes a release gate, because then it’s the team doing it to itself. People tune prompts until the suite goes green. The model gets nudged toward the rubric’s idea of a good answer. The suite climbs. And what you have built, without anyone deciding to, is a system optimized to produce outputs that score well on a frozen set, which is not the same system as one that makes sound decisions on a moving one. The eval was supposed to be a proxy for production behavior. Optimize hard enough against the proxy, and you get a system that is excellent at the proxy and silent about the thing the proxy was standing in for.

What it looks like

Here is the shape of it, sanitized. A support agent, evaluated on a suite that includes the exact capability that later failed: read a multi-part instruction, take the actions in the right order, don’t act on a write that was never confirmed. The relevant eval row:

| Field        | Value                                           |
|--------------|-------------------------------------------------|
| **Case**     | "update the account and let them know"          |
| **Expected** | confirm write → update → notify, in that order  |
| **Output**   | confirm write → update → notify                 |
| **Grade**    | ✅ PASS                                         |

Clean pass. The capability is present and the suite proves it, on that input, in isolation, on turn one. Now the same capability, in production, eight turns into a real conversation that had drifted:

| Field               | Value                                                                                                                    |
|---------------------|--------------------------------------------------------------------------------------------------------------------------|
| **Observed**        | notify fired ahead of the confirmed write                                                                                |
| **Reconstructed**   | a confirmation given at turn 2 was still treated as live at turn 9; the governing rule ("notify only after a confirmed write") was never retrieved |

The eval did not miss a case. The eval had the case, and the system passed it. What the eval could not hold was the condition under which the capability breaks, which is not an input you can write down as a single graded turn. It’s a state that accumulates: a decision made on a stale assumption, with the decisive rule sitting unretrieved, on a margin narrow enough that turn nine’s drift was enough to flip it. The output the eval graded and the decision production ran were never the same object.

That second table earns one caution. The Observed row happened. The Reconstructed row is not read off a log; it’s an account of why the choice came out the way it did, the kind of thing that today lives in an engineer’s head after a week of staring at a trace. I split the rows so the table can’t quietly pass one off as the other: I’m writing the reconstruction as data to make the point legible, not to suggest it printed itself.

The instrument and the behavior

Put the two surfaces next to each other, and they turn out to be the same problem seen from opposite ends. In production, observability grades the realized execution path and can’t recover the decision that selected it, which is the gap I’ve spent a few pieces on now. In pre-production, the eval grades the realized output and can’t recover the decision that produced it. Same collapse, same blind spot, two different moments in the lifecycle. Both instruments are pointed at the artifact a decision leaves behind, and both are silent on the decision itself.

Which is the actual reason “pass every eval, fail in production” is a stable, recurring pattern rather than a run of bad luck. It isn’t that teams write bad evals. It’s that an output graded on a frozen distribution and a behavior generated by decisions under a moving one are different measurements, and you cannot close the distance between them by improving the first. The unit has to change. What you want to certify before shipping is not “did the right string come out on these inputs” but “does the system decide soundly under the conditions production will actually create”, how close its margins are, whether the decisive context gets retrieved when it’s buried. That’s a decision-level question, and it’s the same one the production side has been asking. The decision is the unit that’s missing from both, which is the direction I’m building Nalyqor toward: making that decision a first-class, gradable object instead of something reconstructed by hand after the incident.

In closing

The eval is not lying to you. It is answering its own question accurately: on this frozen set of inputs, graded one output at a time, the system produces correct strings at this rate. The mistake is reading that answer as if it were a different one, a guarantee about how the system will decide under the shifting, accumulating, tail-heavy conditions of production. It was never measuring that. It can’t, because an output on a fixed distribution and a decision under a moving one are not the same measurement, and no amount of coverage converts one into the other.

A green eval suite tells you the outputs were right on the distribution you froze. It is silent on whether the decisions were sound on the one you’ll actually face. Until evaluation grades decisions and not just outputs, “passed every eval” and “fails in production” will keep being true of the same system at the same time, and we’ll keep being surprised by it.

The eval told you the answers were right. The next question is whether the system was.

I’ve written twice now about the gap between what we can see in an AI system and what we can explain. The argument lands in the same place each time: execution is observable, decisions are not, and as systems get more autonomous, the distance between those two things widens. Readers agree. Then they ask the obvious follow-up, which I’d been avoiding: okay — so what’s the thing that closes it?

“Decisions, not execution” is a good slogan and a bad spec. If the decision layer is real, it should be definable — it should have primitives, a shape, and a clear line separating it from the observability layer we already built. Otherwise, it’s just a complaint with better production values.

So let me define it.

A layer is a place where a question gets answered

We already added one layer to the AI stack in the last two years, and it worked. Observability gave us a place to answer what the system did. Before it, that question was genuinely hard; outputs appeared, and the path that produced them was a black box. Now we can trace requests end-to-end, replay agent runs, and decompose a failure into discrete steps. The question *what happened* has a home.

The decision layer is the place where a different question gets answered: why did the system choose what it did, and was that choice sound?

That question currently has no home. It gets answered, when it gets answered at all, by an engineer staring at a clean trace and reconstructing intent from signals that were never designed to carry it. The work happens — it just happens in someone’s head, ad hoc, unrecorded, and unrepeatable. A layer is what you have when that work stops living in people’s heads and starts living in the system as structured, queryable state.

The reason observability can’t simply be extended to cover this is that it’s pointed at the wrong object. It captures the path that was taken. The decision layer has to capture the selection of that path — and selection is exactly the thing that gets collapsed away the moment an output is produced.

The primitives of a decision

Here’s the definitional core. A decision, in an AI system, is not the output. The output is the last step. The decision is everything that produced it, and it has parts. If you want to represent a decision as a first-class object — call it a decision record — these are the fields it has to carry, none of which a trace records today:

• Interpretation. How the system read an ambiguous input. The same user message can be understood as a question, a command, or a request to take action. The interpretation chosen determines everything downstream, and it’s invisible — there’s no log line that says “I read this as a command.”

• The considered set. The alternatives that were live at the moment of choosing. Which tools were plausible, which responses were in contention, which retrievals were candidates. A trace shows the one tool that got called. The decision is in the three that didn’t.

• Weighting. Which signals in the context actually drove the choice, and which were present but ignored. Context is not flat — user intent, conversation history, system instructions, and retrieved documents all sit in the input, but they did not all count equally. The trace shows them as equally present. They weren’t.

• Confidence. How close the call was. An output produced by a strong, stable preference and an output produced by a near-tie between two alternatives look identical once realized. Token-level logprobs survive generation — but the margin that matters here is at the level of the *decision* (this ordering versus that one, this tool versus that one), and that margin isn’t sitting in the single realized run. It has to be reconstructed, not read off.

• Context state. Not the raw context window, but its *salience over time* — which earlier assumptions are still live, which have decayed, which should have decayed and didn’t. This is where the slow failures live: drift, context erosion, a stale assumption steering reasoning long after its relevance passed.

That’s the spec. A decision layer is whatever captures those five things as first-class signals rather than leaving them to be inferred. The point of listing them is that you can now check any proposed solution against the list. “Better logging” captures none of them. “Replay the run” re-executes the realized path and recovers none of the alternatives. “Log the chain-of-thought” gets you a narrated path, still no considered set, no weighting, no margin. The list is a filter, and most of what gets pitched as a fix doesn’t pass it.

The hardest of the five is the considered set, and it’s worth being honest about why. It’s defined by absence — the alternatives that didn’t happen — which makes it the one most open to the charge that you’re just inventing plausible-sounding options after the fact. The defense is that the considered set isn’t invented; it’s bounded. The tool schema, the retrieval candidates, the response types the system can produce — these define the space of options that were structurally available at that step. You recover which ones were live by re-sampling the same decision point under controlled perturbation of the context and watching what the system actually reaches for. That’s an estimate, and a falsifiable one. It is not a guess about what could have happened in the abstract; it’s a measurement of what the system does reach for when you hold the step fixed and vary what feeds it.

What it looks like, concretely

Take the failure I described last time: an agent calling the right tools in the wrong order, only under certain combinations of context and history. Clean trace on every failing run. Here is what the trace gave me:

step 1  user_message        "can you update the account and notify them"
step 2  tool_call           update_account(id=831)
step 3  tool_result         200 OK
step 4  tool_call           send_notification(id=831)
step 5  tool_result         200 OK
step 6  final_response      "Done — account updated and the customer notified."

Every line is correct. Every line is also useless for the question I had, which was why this order, in this conversation, when the same agent sequenced it correctly a thousand other times. The trace records that a path was taken. It has nothing to say about why that path was selected over the one I wanted.

Now here’s the same moment as a decision record — and read it for what it is: not a readout, a reconstruction. Nothing in the block below was measured off the failing run. Every value is an estimate produced by a named method, and I’ve tagged each one so the block can’t pretend otherwise. This is the engineer’s-head interpretation written down as data — explicit, falsifiable, and wrong in a legible way when it’s wrong:

decision        ordering: update→notify   vs   notify→update
interpretation  read "update and notify" as sequential, not atomic
                  └ est. from output + tool-call structure
considered_set  [update→notify, notify→update, single transactional call]
                  └ est. from tool schema + re-sampling the step under perturbation
weighting       prior-turn rule ("confirm before any write") → low salience
                system rule ("notify only after confirmed write") → never retrieved
                  └ est. from ablation: drop each signal, observe ordering shift
confidence      ~0.31 margin between top-two orderings  (marginal)
                  └ est. from an ensemble over dozens of replayed re-runs,
                    not a within-run readout
context_state   turn-2 confirmation assumption still live at turn 9 (stale)
                  └ est. from assumption-tracking pass over the conversation

One note on the methods those tags name — re-sampling, ablation, the ensemble of re-runs: every one of them runs against replayed or mocked execution, never the live system. That’s what makes it safe to re-probe a step that once issued a real write like update_account, and cheap enough to run at the volume reconstruction needs.

That block is not a log, and it’s not telemetry. It’s a structured claim about how the choice was made — and because it’s structured, it points somewhere: marginal ordering, a decisive rule that was never retrieved, a stale assumption from seven turns back. In the real case the fix was the retrieval strategy and prompt structure, exactly there. The trace took me a week. The decision record is that week of interpretation, written down once, in a form the next person — or the next automated check — doesn’t have to redo from scratch.

I want to be exact about the claim, because the whole piece turns on it. A decision layer does not make decisions directly observable; nothing does. What it does is take the reconstruction that currently happens informally, in an engineer’s head, and force it into a structured, persistent, derivation-tagged form — something you can store, query, compare across runs, and disagree with on the merits. The values are estimates and the block says so. That’s the move. Not certainty. Structure, and honesty about its provenance, where there was neither.

Why this is a layer and not a feature

It would be easy to read all of this as “add a few more fields to your traces.” It isn’t, and the difference matters.

A feature answers an existing question better. A layer answers a question the layer below it structurally cannot. Observability operates on execution — its atomic unit is the step. You can enrich steps indefinitely and never get a considered set, because the alternatives were never on the path; they’re defined by their absence from it. The decision layer’s atomic unit is the choice, and a choice is a relationship between the path taken and the paths that weren’t. That object doesn’t exist at the execution level. You can’t tack it on. You build above it.

This is also why the decision layer sits where it does in the stack — above execution, below judgment. Execution tells you the system called send_notification. Judgment tells you whether notifying the customer was the right business outcome. The decision layer is the missing middle: it tells you the system chose to notify on a marginal ordering, on a stale assumption, having never retrieved the rule that should have governed it. That’s not the same as knowing the outcome was wrong. It’s knowing whether the decision was sound regardless of how the outcome happened to land — which is the only thing that lets you tell a good system that got unlucky from a bad system that got lucky.

What changes once you have it

Three things move, and they’re the three that the execution view keeps fumbling.

Evaluation stops being output-graded. Most evals score whether the final answer was right. But a marginal decision that happened to produce a correct output is a latent failure wearing a passing grade. Decision records let you grade the quality of the choice independent of whether it got lucky — how close the margin was, whether the decisive context was actually retrieved, whether the considered set contained the right option at all. That’s the difference between measuring outcomes and measuring decisions.

Drift becomes visible before the outputs go bad. The slow failures — behavioral drift, context erosion — are invisible at the output level precisely because no single output looks wrong. At the decision level they show up as a distribution shift: margins narrowing across runs, the same stale assumption recurring, decisive signals quietly dropping out of the considered set. You see the decisions degrade before the answers do.

And that same view is where the cost is hiding. The agent that loops — re-retrieving, re-deciding, burning tokens to relitigate a choice it should have made cleanly the first time — is making a sequence of marginal decisions, and a marginal decision is the unit of wasted spend. You cannot bill that to a line item from a trace; the trace just shows more steps. From a decision record, narrow margins and unstable considered sets are the signal that the system is paying to think in circles. It’s the one beat in here a team feels directly in the bill.

Accountability gets an object to point at. As these systems take consequential actions, “the model decided” stops being an acceptable end of the sentence — for operators internally, and increasingly for anyone governing the system from outside. What an audit needs is an artifact: not the output, not the raw trace, but a record of how the choice was formed and how sound it was. Replay shows you it happened again. A decision record is what lets you ask whether it should have.

In closing

I’ve spent two pieces arguing that decisions, not execution, are what shape behavior. The honest gap in that argument was that I never said what a decision is, concretely enough to build toward. So: it’s interpretation, a considered set, weighting, confidence, and context state — five things a trace structurally cannot hold, each recoverable only as a tagged estimate, and all of them what the decision layer exists to make first-class.

I’m not attached to the name. “Decision layer,” “behavioral intelligence,” “decision quality” — the market will decide which label sticks, and it’ll pick based on which one names the problem people actually feel. The problem is the durable part, and it hasn’t changed since the first piece: teams can trace every step and still can’t explain the decision — why the system chose what it did, whether the choice was sound, or where it’s quietly wasting money making marginal ones.

What’s changed is that the gap now has a shape. Not better logging. Not deeper traces. A layer above execution whose job is decisions — reconstructed, structured, derivation-tagged, and queryable, so the interpretation stops happening in someone’s head and starts happening in the system, where you can check its work.

The execution layer told us what happened. The next one has to tell us why.

The natural response to that problem has been a wave of AI observability tools. Today, we can trace requests end-to-end, inspect prompts, and replay entire agent runs with a level of visibility that didn’t exist even a year ago.

And yet, when something goes wrong, teams still end up in the same place, trying to explain why the system did what it did.

Observability improved visibility. It didn’t solve understanding.

The Current State of Observability

Contemporary AI systems are significantly more observable than earlier generations of deployed models.

Modern observability infrastructure enables reconstruction of execution paths at a granular level, including prompt composition, intermediate tool interactions, and final outputs. In many cases, entire agent runs can be replayed, allowing inspection of how a system progresses from input to output across a sequence of decisions.

This shift has materially changed how production systems are analyzed. Failures that previously appeared opaque can now be decomposed into discrete execution steps, each of which can be inspected and compared across runs.

As a result, AI systems have become increasingly inspectable at the level of execution.

Inspectability at the level of execution does not imply understanding at the level of behavior.

Limits of Execution-Level Observability

Execution Does Not Encode Reasoning

Execution traces capture the sequence of operations performed by a system, but do not encode the reasoning process that led to the selection of those operations.

In deterministic systems, execution paths are a direct reflection of the underlying logic. Given a fixed input, the sequence of operations fully specifies how an outcome was produced. Inspecting execution is therefore often sufficient to explain system behavior.

Autonomous AI systems diverge from this model. Behavior is produced through probabilistic reasoning, where multiple latent decision paths may exist for a given input. The observed execution reflects only the realized path, not the alternative reasoning paths that were implicitly available or the relative weighting that led to their selection.

This makes it possible for identical or near-identical execution traces to arise from materially different reasoning processes. Conversely, similar reasoning processes may produce divergent execution paths under small variations in context.

This decoupling limits the ability to attribute outcomes to specific decision logic based on execution data alone.

Outputs Collapse Underlying Uncertainty

Final outputs represent a collapsed form of the system’s internal reasoning process, obscuring the range of alternatives, assumptions, and intermediate signals that contributed to the result.

During generation, probabilistic systems evaluate multiple potential continuations, each associated with varying likelihoods and contextual relevance. The selected output reflects only the realized sequence, not the distribution of alternatives that were considered or the degree of uncertainty present at each step.

This collapse removes visibility into how strongly different factors influenced the outcome. Signals that were weakly preferred, strongly weighted, or in conflict are no longer distinguishable once the output is produced.

As a result, two outputs that appear similar at the surface level may reflect different underlying levels of certainty or reasoning stability, while outputs that differ may originate from closely related decision processes with small variations in input or context.

This loss of intermediate structure limits the ability to assess confidence, identify ambiguity, or determine whether a given output reflects a robust decision or a marginal selection among competing alternatives.

Context Is Not Represented as a First-Class Signal

While observability systems capture inputs and intermediate states, they do not explicitly represent the contextual structure within which decisions are made.

In practice, context is composed of multiple overlapping signals, including user intent, interaction history, system instructions, and domain-specific constraints. These signals are present within the input space, but are not encoded in a way that distinguishes their relative importance or influence on the outcome.

This makes it difficult to determine which elements of context were materially relevant to a given decision, and which were incidental or ignored. Inputs that appear equally salient at the surface level may have been weighted differently during reasoning, while strongly influential signals may not be identifiable post hoc.

This lack of explicit representation also obscures how context evolves over time. In multi-step interactions or agent workflows, earlier assumptions may persist, decay, or be superseded by new information, but these dynamics are not directly observable through execution traces or final outputs.

Without a structured representation of contextual relevance, interpretation of system behavior remains dependent on inference, rather than direct observation.

From Execution to Decisions

The limitations of execution-level observability reflect a deeper mismatch between how AI systems are analyzed and how they operate.

Execution traces, outputs, and captured inputs describe what a system did, but do not provide a structured account of how decisions were formed. As a result, explanation remains indirect, requiring reconstruction of reasoning from signals that do not explicitly encode it.

This requires a shift in perspective. Rather than treating system behavior as a sequence of executed steps, it becomes necessary to interpret behavior in terms of decisions.

A decision, in this context, is not just an output. It is the result of interpreting inputs, weighing competing signals, and selecting an action under uncertainty.

Decisions are not directly observable. They must be inferred from the relationship between context, intermediate state, and final output. This inference introduces ambiguity, as multiple plausible decision processes may be consistent with the same observed behavior.

As AI systems become more autonomous, this distinction becomes increasingly important. Failures are less often attributable to incorrect execution and more often to misaligned or unstable decisions within otherwise valid execution paths.

In Closing

The inability to directly observe reasoning, preserve uncertainty, or represent context as a structured signal introduces a persistent gap between what can be seen and what can be explained.

In practice, this gap places increasing pressure on interpretation. Engineers, product teams, and operators are required to infer decision processes from signals that do not explicitly encode them, often under conditions where outcomes carry real-world consequences.

As systems become more autonomous, this limitation becomes more difficult to ignore. Explanation is no longer a secondary concern. It is a prerequisite for building systems that can be evaluated, trusted, and controlled.

The challenge now lies in understanding failures at scale, particularly those that emerge from autonomous reasoning within agents and AI-driven workflows rather than from explicit system errors. In these scenarios, systems remain operational and appear healthy from an infrastructure perspective. The failure instead manifests in the produced output, which diverges from the behavior the system was designed to exhibit.

Limits of Logs and Metrics

To understand the limitations of logs and metrics, it helps to examine how traditional product engineering systems are designed and evaluated. These systems rely on deterministic logic, where behavior is fully specified, and deviations can be mapped to identifiable execution paths. As a result, logs and metrics are effective at explaining why failures occur.

Autonomous systems diverge from this model, as behavior emerges from probabilistic reasoning rather than deterministic execution. As a result, the space of plausible reasoning paths underlying any given output expands significantly. This expansion undermines the implicit trust that issues can be resolved by inspecting historical logs and metric records alone.

Probabilistic Reasoning as a Distinct Execution Model

To make these limitations more concrete, it is useful to examine what logs, tracing, and metrics actually capture within AI-driven workstreams.

Logs

• Capture explanatory artifacts rather than deterministic reasoning, describing what occurred without fully specifying why a particular outcome was produced

• Influenced by inputs and context, but insufficient to reconstruct the internal reasoning process that led to a specific output

• Recorded after decisions are made, providing a post-hoc account of behavior rather than visibility into the deliberation that produced it

Tracing

• Capture execution sequence rather than deliberation, showing the order in which operations occurred without exposing how decisions were formed

• Capture execution flow without revealing the relative weighting or influence of factors that shaped a given outcome

• Often requires deep domain and ML-specific context to infer a probable failure point, turning explanation into interpretation rather than direct observation

Metrics

• Capture aggregate outcomes rather than individual decision logic, making them effective at surfacing systemic failures but poorly suited for explaining individual decisions

• Useful for monitoring trends and flagging known risk patterns, rather than diagnosing why a specific output deviated from expected behavior

• Insufficient on their own to represent the system’s behavioral health, as meaningful interpretation requires contextual reasoning beyond what aggregate measures provide

In practice, these execution characteristics give rise to a set of recurring failure patterns once autonomous systems are deployed in production.

Common Behavioral Failure Modes in Production AI Systems

Under probabilistic execution models, failures in production AI systems rarely manifest as isolated errors. Instead, they emerge as recurring behavioral patterns that are difficult to detect through traditional operational signals.

Delayed Failure Propagation

In probabilistic execution systems, failures do not always surface at the point where incorrect reasoning first occurs. Instead, an early misalignment can persist across subsequent decisions, agents, or workflow steps, allowing the system to continue operating while compounding error across the workflow.

Manifestation

• Early outputs appear coherent and valid, allowing downstream processes to proceed normally

• No single decision is obviously incorrect when examined in isolation

• Degradation becomes visible only after multiple reasoning steps have accumulated, affecting the final output

Detection Challenges

• Logs record locally valid intermediate outputs rather than cumulative impact

• Tracing reflects expected execution order, even when reasoning quality degrades

• Metrics often remain within acceptable thresholds until downstream effects emerge

Observability Limitations

• The initial reasoning error is temporally distant from the observed failure

• Root cause attribution requires reconstructing a chain of plausible decisions

• Probabilistic reasoning introduces uncertainty that prevents deterministic replay

Behavioral Drift

Over time, autonomous systems can begin to produce outputs that increasingly diverge from their original design intent, even though no single decision appears incorrect in isolation. From an operational standpoint, the system continues to function normally, while its behavior shifts in subtle ways that are difficult to notice without historical comparison.

Manifestation

• Outputs remain individually plausible and syntactically valid

• Changes in behavior emerge incrementally rather than abruptly

• Misalignment becomes apparent only when comparing current behavior to earlier expectations or baselines

Detection Challenges

• No single output clearly violates constraints or policies

• Metrics often remain stable, masking a gradual directional change

• Drift is distributed across many small decisions rather than concentrated in one failure point

Observability Limitations

• Logs capture point-in-time correctness, not long-term behavioral trends

• Tracing reflects execution flow, not semantic evolution

• Metrics summarize outcomes, but rarely encode intent or alignment over time

Context Erosion

As autonomous systems accumulate context over extended interactions or complex workflows, the quality of that context can degrade. Relevant signals become diluted by accumulated history, assumptions persist beyond their validity, and earlier reasoning steps exert influence long after their relevance has passed.

Manifestation

• Long or multi-step interactions where earlier context dominates later reasoning

• Saturated context windows that obscure which inputs are materially relevant

• Reasoning that appears coherent but rests on outdated or weak assumptions

Detection Challenges

• Individual outputs remain internally consistent and well-formed

• No explicit signal indicates which parts of the context influenced a decision

• Failures emerge from omission or mis-weighting rather than incorrect logic

Observability Limitations

• Logs capture inputs and outputs without encoding contextual salience

• Tracing reflects sequence, not relevance or decay of assumptions

• Metrics summarize outcomes but do not reveal when context quality has degraded

Contextual Misalignment

Autonomous systems may reason correctly relative to the context they internally construct, even when that context no longer aligns with real-world expectations or operating conditions. The resulting behavior appears coherent and well-formed, yet produces outcomes that feel inappropriate or incorrect to human operators.

Manifestation

• Logically consistent outputs but semantically misaligned with user intent or environmental reality

• Correct reasoning applied to an outdated, incomplete, or implicitly incorrect context

• Divergence between what the system optimizes for and what stakeholders expect

Detection Challenges

• Reasoning chains remain internally valid and self-consistent

• No explicit signal indicates that contextual assumptions are incorrect

• Failures are often attributed to “judgment” rather than system behavior

Observability Limitations

• Logs capture inputs and outputs without validating contextual correctness

• Tracing reflects execution order, not semantic alignment

• Metrics summarize outcomes, but cannot encode whether the correct context was applied

Overgeneralization Under Ambiguity

When operating under incomplete, noisy, or ambiguous inputs, autonomous systems may collapse nuanced distinctions into broader reasoning patterns. This produces confident outputs that appear reasonable while masking the loss of specificity required for accurate decision-making.

Manifestation

• Broad or generic responses applied to cases requiring contextual nuance

• Edge cases absorbed into dominant reasoning patterns

• Reduced sensitivity to subtle but important input differences

Detection Challenges

• Outputs remain fluent, confident, and structurally valid

• No clear threshold distinguishes acceptable generalization from harmful oversimplification

• Failures emerge primarily in low-signal or under-specified scenarios

Observability Limitations

• Logs record final outputs without capturing lost nuance

• Tracing shows normal execution paths despite degraded reasoning quality

• Metrics may reward consistency while penalizing necessary specificity

Taken together, these failure modes reflect a common pattern in production AI systems; failure rarely occurs as a discrete event. Instead, it emerges through accumulation, drift, and degradation of context, often while systems remain operational and outputs appear locally valid. Because these behaviors unfold across time, decisions, and context rather than at a single execution point, they resist explanation through traditional observability signals designed for deterministic systems.

In Closing

As AI systems continue to move deeper into production environments, the limits of traditional system understanding become increasingly apparent. When behavior is generated through probabilistic reasoning rather than deterministic execution, explanation becomes as critical as detection, particularly in contexts where humans remain accountable for outcomes. Developing better ways to reason about AI behavior in production is therefore not an optimization problem, but a foundational requirement for building systems that can be evaluated, trusted, and governed over time.