That number is real. It is also, for most of the decisions worth caring about, off by orders of magnitude, and wrong in a specific direction. It captures the cheapest thing that happened.

I have written before about the gap this sits inside: execution is observable, decisions are not. What that gap does to a balance sheet is less discussed. The money does not disappear. It lands, in full, in six different places, on six different clocks, under owners with no reason to connect any of it back to the moment a system chose wrong.

The rule that manufacturing figured out first

Quality management has a rule of thumb, sometimes called the rule of ten: a defect caught at design costs a fraction of one caught on the production line, which costs a fraction of one that reaches a customer. The specific multipliers vary by industry and most of the numbers attached to them are closer to folklore than measurement.

The structure is not folklore. The cost of a defect is a function of how far it travels before anyone notices, because distance is what determines how much got built on top of it.

AI systems inherit that structure and then make it worse in one specific way. A production line makes a bounded number of decisions, and most of them are inspected. An agent makes many decisions per task, continuously, and almost none of them are inspected individually. The travel distance is longer and the inspection rate is lower, which is the exact combination the rule of ten warns about.

Six places the money lands

Take one wrong decision and follow it forward rather than backward.

The decision itself. The tokens spent producing the wrong choice. This lands within seconds, on the bill, and it is genuinely indistinguishable from the tokens spent producing a right one. A good decision and a bad decision cost the same to make.

The recovery. Retries, re-retrieval, the agent going back around on something it already had enough in front of it to settle. This also lands on the bill within the same minute, filed as usage rather than as rework, because nothing in the billing model has a category for rework.

The downstream execution. Everything the system then did correctly, on a wrong premise. This is the first component large enough to notice, and it is the first one that gets attributed to the wrong thing: it shows up under whatever workflow ran, not under the decision that set it running. Perfectly efficient execution of a bad choice is still waste, and it bills as throughput.

The detection. Engineering hours at the surfacing end. Someone works backward from a symptom, and every step they pass through on the way there looks correct, because each one was. This does not appear on the AI bill at all. It appears on payroll, days or weeks later, usually filed under an incident that names the symptom rather than the cause.

The correction. The fix, plus the standing tax of whatever guardrail gets added to prevent a recurrence. Guardrails are not free. Every one of them is latency and complexity that the system carries forever, paid for by a decision that happened once.

The forfeited scope. The work you decide not to hand the system next time. A team that gets burned narrows what it delegates, and that narrowing is permanent until something changes their mind. This is almost certainly the largest number of the six, and it is the only one that appears in no ledger anywhere, because it is not a cost that gets incurred. It is a benefit that never gets collected.

The ordering is the finding. Those six arrive in roughly ascending order of size, and in strictly descending order of visibility. The cheapest component is the only one anyone can point at.

A worked example

Sanitized, and deliberately mundane, because the expensive cases usually are.

A support agent triages inbound tickets by entitlement. A customer writes in about a charge they did not expect after a plan change. The agent reads this as a billing dispute and routes it to the disputes workflow. The other available reading, a service credit request under an existing entitlement, was equally defensible on the text. Nothing about the decision is anomalous. Both interpretations are ones a competent human would consider.

The disputes workflow is heavier. It pulls account history, runs a verification pass, and drafts a response with a different tone and a different set of obligations. All of it executes correctly. The customer gets a coherent answer to a question they did not quite ask, and either accepts it or writes back, which starts the whole thing again from a worse position.

Multiply that by the ticket volume that shares the same ambiguity, and the interesting cost is not the misroute. It is the disputes workflow running thousands of times on cases that were never disputes.

Nobody finds this by looking at the bill. The bill says disputes volume is up.

| Cost layer           | Where it lands                | When it lands    | Attributed to the decision?                    |
|----------------------|-------------------------------|------------------|------------------------------------------------|
| The decision         | AI spend                      | Seconds          | No, it looks like any correct call             |
| Recovery loops       | AI spend                      | Seconds to hours | Filed as usage, never as rework                |
| Downstream execution | Another workflow's line item  | Minutes to days  | No, credited to what ran, not what chose       |
| Detection            | Engineering hours             | Days to weeks    | Only if someone traces it back by hand         |
| Correction           | Hours, plus permanent latency | Weeks            | To the incident, not the decision class        |
| Forfeited scope      | Nowhere                       | Quarters         | Never, no system records a benefit uncollected |

Why the accounting cannot close

Two separations do the damage, and neither is a tooling oversight that gets patched next quarter.

The first is time. The top three layers land inside a minute. The bottom three land days to quarters later. By the time the expensive part arrives, the cheap part has been aggregated into a monthly number and the causal thread is gone.

The second is boundary. The top three land in the AI budget. The next two land in payroll. The last one lands nowhere at all. Those are different systems, owned by different people, reconciled on different cycles, with no shared key between them. No accounting system joins a token to an engineer’s afternoon three weeks later, and there is no reason anyone would have built one.

So the cost gets split across ledgers that cannot see each other, and every ledger is individually correct. The AI bill correctly reports spend. The incident record correctly reports hours. Neither is wrong. Together they still fail to say that one decision, made in a fraction of a second, caused both.

The instrumentation gap has a financial twin, one level up. A trace is a record of steps, not of choices. A ledger is a record of amounts, not of reasons.

What you can actually estimate, and what you cannot

I want to be careful here, because this is exactly the point where a piece like this usually starts inventing precision.

You cannot measure most of the six. You can bound some of them, and the honest framing matters more than the arithmetic:

• Downstream execution is recoverable, but only if you can identify the originating decision, which is the hard part and the whole problem. The compute is already logged. What is missing is the link from it back to the choice that triggered it.

• Detection and correction are the most tractable, because incident records already hold rough hours. They are just filed against symptoms, so recovering them means re-attributing existing data, not collecting new data.

• Recovery loops can be approximated by comparing a workflow’s actual path against its shortest successful path, which is an estimate and behaves like one.

• Forfeited scope cannot be measured at all. It is only ever visible in what a team quietly stops automating.

Anything that hands you a clean dollar figure per bad decision is estimating, and should say so on its face. The useful output here is not a number. It is a ranking: which decision patterns are expensive, and roughly by how much, relative to each other.

The unit that is missing

Every AI cost tool I have used reports cost per token, cost per call, cost per workflow. Those are consumption metrics. They tell you what the system used.

None of them report cost per decision, which is the only unit that tells you what the system wasted.

That is not a gap in the tools. It is a gap in what the tools can see. Consumption is observable from the outside; a decision’s quality is not, and you cannot build a cost-per-decision number on top of a stack that has no representation of the decision in the first place. This is the direction I am building Nalyqor toward, and I would rather it be the direction the whole category goes: cost attribution that resolves to the choice rather than to the call.

In closing

The reason bad AI decisions feel cheap is not that they are cheap. It is that their price is paid in six installments, five of which arrive after anyone has stopped looking, in currencies that do not convert.

Teams optimizing AI spend right now are optimizing the one installment that arrives first and costs least, because it is the only one denominated in something the invoice understands. That work is not wrong. It is just aimed at the one installment the invoice can denominate, on a total nobody has actually computed.

An invoice is not a measurement of a mistake. It is a receipt for the cheapest part of one.

Thanks for reading The Agent Loop, Powered by FAI! Subscribe for free to receive new posts and support my work.

Every postmortem starts at the wrong end

When a production incident gets investigated, the investigation starts where the pain showed up: the timeout, the complaint, the number that finally moved. From there, someone works backward, hop by hop, until they find the step that first went wrong. That process is necessary, and it is also structurally backward. It starts at the one moment the failure was loud and reconstructs everything before it, which is exactly the part that was quiet.

A silent failure is not an event. It is a sequence, and most of that sequence happens while every dashboard stays green. If you only ever study the loud end, you learn what a failure looks like once it has already stopped being silent, which is the one part of its life that was never the hard part to see.

So run it the other direction. Start at the decision, and follow it forward through every stage that had to go right, for the system, for the failure to stay hidden that long.

The five stages

A silent failure has the same basic anatomy almost every time, regardless of which behavioral pattern it turns out to be. Five stages, in order:

• Origination. The point where a decision quietly diverges from the one that was actually warranted.

• Local coherence. The output of that decision, examined on its own, looks completely fine.

• Absorption. The layers built to catch problems each look at the wrong part of the decision and pass it through.

• Compounding. The divergence gets treated as precedent by later decisions, and its footprint grows.

• Surfacing. Something finally crosses a threshold, usually somewhere structurally distant from where the divergence began.

None of this is exotic. It is the same mechanism whether the underlying pattern is drift, context erosion, or a stale assumption that never decayed. What changes is the content. The shape stays constant. Walking one case through all five stages is more useful than adding a sixth item to a list of failure types, so here is one, sanitized.

The scenario, sanitized

A CI pipeline has an agent that reviews routine dependency bumps. Patch-level version updates from a small set of trusted packages get auto-merged if the test suite passes. Anything else waits for a human. Ordinary, sensible policy.

Early on, the agent handles a minor-version bump, one step above what the policy actually covers, from a maintainer it already trusts. Tests pass, the diff is small, nothing looks wrong, and it merges. Nobody wrote that exception down anywhere. It just happened once, and it worked.

Stage 1: Origination

That single merge becomes an unlabeled precedent. The next time a similar bump comes from a maintainer, the system already associates it with “safe,” the interpretation it reaches for is the one that already worked, not the policy that was actually written. This is a decision problem before it is anything else: an interpretation forms, gets weighted into later reasoning, and nothing marks it as an exception rather than a rule. I have written before about the primitives a decision like this actually has: interpretation, the considered set, weighting, confidence, context state, and this is what it looks like when one of them quietly wins without ever being checked against the policy it was supposed to defer to.

Stage 2: Local coherence

Here is the part that makes this hard to catch: every individual merge, looked at by itself, is fine. Tests are green. The diff is small. Nothing about this specific decision is anomalous compared to the thousand other routine merges that really were routine. A reviewer glancing at any one of these commits would approve it without a second thought, because in isolation, it deserves that.

This is true of nearly every stage of a silent failure. It is never the decision itself that looks wrong. It is the pattern across many decisions, and no single decision carries the pattern.

Stage 3: Absorption

Everything built to catch a problem here is pointed at the wrong layer. The trace shows a merge that completed successfully, which is accurate. The test suite passed, which is also accurate. If there is an eval on this workflow, it almost certainly grades whether the merge was safe on that input, and this one was; the tests really did pass. None of these instruments were built to ask whether this decision matched the policy’s actual scope. They were built to ask whether the output was correct, and the output was.

This is the same collapse from a different angle each time: execution records what happened, not why it was chosen. An eval grades the output, not the decision that produced it. Here, that collapse means a policy exception gets treated as a policy success, three separate times, by three separate instruments, none of which were wrong about what they measured.

Stage 4: Compounding

The next borderline bump gets judged against the drifted precedent, not the original policy, because the drifted precedent is now what “safe” has come to mean inside the system’s own reasoning. Each new case that fits the pattern gets folded in the same way. The category of “routine” widens, one defensible decision at a time, and every one of those decisions is locally coherent for the same reason the first one was.

Nothing about this shows up as a spike. Merge volume looks normal, maybe even improves, since fewer borderline cases get flagged for review. An aggregate metric, watched for a threshold breach, is the wrong instrument for a slow directional shift that never crosses one.

Stage 5: Surfacing, and the misattribution that follows

Weeks later, a downstream service starts intermittently timing out. Nobody connects this to a dependency bump at first, because nothing about the incident points there. An on-call engineer spends the first few hours on the service itself: retry logic, connection pooling, recent deploys to that service specifically. Eventually, someone traces a changed default, a serialization behavior that shifted in a minor-version bump three weeks back, and finds it was auto-merged.

Here is the part worth sitting with. The natural fix, the one that gets shipped the same week, is usually a narrower rule: exclude this specific package, or require review for this specific maintainer going forward. That patches the symptom. It does not touch the mechanism, which was never about that package. It was an unlabeled precedent that will form again, around a different maintainer, the next time a bump gets treated as routine because an earlier one worked out. The investigation started at Surfacing and worked backward through a chain of individually valid steps, which is exactly the shape that makes it easy to fix the step you found and miss the pattern that produced it.

The anatomy at a glance

| Stage           | What's visible                                         | What's actually happening                                                             |
|-----------------|----------------------------------------------------------|-------------------------------------------------------------------------------------|
| Origination     | A single decision, unremarkable in the trace           | An interpretation forms and gets weighted without being checked against policy        |
| Local coherence | Each individual output looks correct                   | The pattern lives across decisions, not inside any one of them                        |
| Absorption      | Trace, tests, and evals all report healthy             | Each instrument is grading the artifact, not the interpretation behind it             |
| Compounding     | Aggregate metrics stay flat or improve                 | The category the drifted precedent covers keeps widening                              |
| Surfacing       | A symptom appears, structurally distant from the cause | Discovery starts at the symptom and reconstructs backward through valid-looking steps |

In closing

The instinct, once a silent failure gets discovered, is to ask which check should have caught it. That question assumes the failure was a single missed detection, sitting at one layer, waiting for the right instrument to look at it. It usually was not. It was coherent at every stage because every instrument evaluated a different artifact than the one that actually failed. The better question is which stage the signal actually died at, and why that stage was structurally incapable of seeing it, because that is the question whose answer generalizes to the next failure instead of just patching this one.

That is also why I think the fix has to be something that can walk this anatomy after the fact, stage by stage, rather than something an engineer reconstructs from scratch under pressure each time. That’s the direction I’m building Nalyqor toward: not another alert at the point of surfacing, but a way to identify failures much closer to their point of origination.

A failure that never announces itself did not skip an announcement. It made it, quietly, at every one of these five stages, to instruments that were built to hear something else.

Thanks for reading The Agent Loop, Powered by FAI! Subscribe for free to receive new posts and support my work.

The reflex explanation is that the eval set was incomplete. Add the missing case, raise coverage, move on. That explanation is comforting because it implies the gap is a quantity problem, and quantity problems get solved by grinding. I don’t think it’s a quantity problem. I think a system can pass an eval set with perfect coverage and still fail in production, for reasons that have nothing to do with how many cases you wrote and everything to do with what an eval is structurally able to measure.

Thanks for reading. This piece is the part of the argument I keep circling back to: the instrument and the thing it’s pointed at are not the same shape.

What an eval actually measures

Strip an eval down to its mechanism. You take a frozen set of inputs, run the system on each, and grade the outputs against an expected answer or a rubric. Pass rate is the fraction of outputs that cleared the bar. That is the whole instrument, and it is a good instrument for what it measures: output correctness, on a fixed distribution, one input at a time.

Now look at what each of those properties quietly assumes, because production violates all three.

An eval grades the output. Production is generated by a decision. The grade lands on the final answer, which is the last step of a process and the one place all the interesting structure has already collapsed. An output that was produced by a clean, decisive, well-grounded choice and an output produced by a near-tie between two alternatives, one of which was wrong, look _identical_ on the rubric. Both are the correct string. Both pass. But only one of them is a system you can trust the next time the context shifts the margin by a hair. The eval cannot tell them apart, because it is grading the thing they have in common and ignoring the thing that distinguishes them. A correct output sitting on top of a marginal decision is a latent failure wearing a passing grade, and the eval is exactly the instrument that can’t see it.

An eval fixes the distribution. Production drifts it. The test set is sampled once and frozen. That is what makes it a measurement at all, you need a fixed yardstick. But production is non-stationary by construction: the inputs move as the world and your users move, and inside any multi-turn session the context accumulates, so the distribution the system actually faces at turn nine is one no eval case ever held still long enough to capture. The eval certifies behavior on the distribution you froze. Production runs on the one that’s drifting away from it the moment you ship.

An eval scores the aggregate. Production dies in the tail. Pass rate is a mean, and a mean is precisely the statistic that buries the failures that matter. The production-killing failures are rare, correlated, and context-triggered, the specific combination of a stale assumption, an unusual ordering, and a retrieval that didn’t fire. Each is individually low-frequency, so each contributes almost nothing to the aggregate. 98% looks like a system that’s basically right. It can equally be a system that is reliably right on the easy 98% and reliably wrong on the 2% that happens to be where your highest-value, highest-consequence traffic lives. The number is the same. The system is not.

None of these is fixed by writing more cases. More cases sharpen the estimate of output correctness on a frozen distribution. They do not turn an output measurement into a decision measurement, they do not un-freeze the distribution, and they do not stop the mean from hiding the tail. You can drive coverage to a place that feels exhaustive and still have measured none of what production is about to do to you.

The Goodhart turn

And that’s the failure mode if teams simply accept these limits. There’s a worse one that shows up the moment the eval stops being a gauge and becomes a release gate, because then it’s the team doing it to itself. People tune prompts until the suite goes green. The model gets nudged toward the rubric’s idea of a good answer. The suite climbs. And what you have built, without anyone deciding to, is a system optimized to produce outputs that score well on a frozen set, which is not the same system as one that makes sound decisions on a moving one. The eval was supposed to be a proxy for production behavior. Optimize hard enough against the proxy, and you get a system that is excellent at the proxy and silent about the thing the proxy was standing in for.

What it looks like

Here is the shape of it, sanitized. A support agent, evaluated on a suite that includes the exact capability that later failed: read a multi-part instruction, take the actions in the right order, don’t act on a write that was never confirmed. The relevant eval row:

| Field        | Value                                           |
|--------------|-------------------------------------------------|
| **Case**     | "update the account and let them know"          |
| **Expected** | confirm write → update → notify, in that order  |
| **Output**   | confirm write → update → notify                 |
| **Grade**    | ✅ PASS                                         |

Clean pass. The capability is present and the suite proves it, on that input, in isolation, on turn one. Now the same capability, in production, eight turns into a real conversation that had drifted:

| Field               | Value                                                                                                                    |
|---------------------|--------------------------------------------------------------------------------------------------------------------------|
| **Observed**        | notify fired ahead of the confirmed write                                                                                |
| **Reconstructed**   | a confirmation given at turn 2 was still treated as live at turn 9; the governing rule ("notify only after a confirmed write") was never retrieved |

The eval did not miss a case. The eval had the case, and the system passed it. What the eval could not hold was the condition under which the capability breaks, which is not an input you can write down as a single graded turn. It’s a state that accumulates: a decision made on a stale assumption, with the decisive rule sitting unretrieved, on a margin narrow enough that turn nine’s drift was enough to flip it. The output the eval graded and the decision production ran were never the same object.

That second table earns one caution. The Observed row happened. The Reconstructed row is not read off a log; it’s an account of why the choice came out the way it did, the kind of thing that today lives in an engineer’s head after a week of staring at a trace. I split the rows so the table can’t quietly pass one off as the other: I’m writing the reconstruction as data to make the point legible, not to suggest it printed itself.

The instrument and the behavior

Put the two surfaces next to each other, and they turn out to be the same problem seen from opposite ends. In production, observability grades the realized execution path and can’t recover the decision that selected it, which is the gap I’ve spent a few pieces on now. In pre-production, the eval grades the realized output and can’t recover the decision that produced it. Same collapse, same blind spot, two different moments in the lifecycle. Both instruments are pointed at the artifact a decision leaves behind, and both are silent on the decision itself.

Which is the actual reason “pass every eval, fail in production” is a stable, recurring pattern rather than a run of bad luck. It isn’t that teams write bad evals. It’s that an output graded on a frozen distribution and a behavior generated by decisions under a moving one are different measurements, and you cannot close the distance between them by improving the first. The unit has to change. What you want to certify before shipping is not “did the right string come out on these inputs” but “does the system decide soundly under the conditions production will actually create”, how close its margins are, whether the decisive context gets retrieved when it’s buried. That’s a decision-level question, and it’s the same one the production side has been asking. The decision is the unit that’s missing from both, which is the direction I’m building Nalyqor toward: making that decision a first-class, gradable object instead of something reconstructed by hand after the incident.

In closing

The eval is not lying to you. It is answering its own question accurately: on this frozen set of inputs, graded one output at a time, the system produces correct strings at this rate. The mistake is reading that answer as if it were a different one, a guarantee about how the system will decide under the shifting, accumulating, tail-heavy conditions of production. It was never measuring that. It can’t, because an output on a fixed distribution and a decision under a moving one are not the same measurement, and no amount of coverage converts one into the other.

A green eval suite tells you the outputs were right on the distribution you froze. It is silent on whether the decisions were sound on the one you’ll actually face. Until evaluation grades decisions and not just outputs, “passed every eval” and “fails in production” will keep being true of the same system at the same time, and we’ll keep being surprised by it.

The eval told you the answers were right. The next question is whether the system was.

I’ve written twice now about the gap between what we can see in an AI system and what we can explain. The argument lands in the same place each time: execution is observable, decisions are not, and as systems get more autonomous, the distance between those two things widens. Readers agree. Then they ask the obvious follow-up, which I’d been avoiding: okay — so what’s the thing that closes it?

“Decisions, not execution” is a good slogan and a bad spec. If the decision layer is real, it should be definable — it should have primitives, a shape, and a clear line separating it from the observability layer we already built. Otherwise, it’s just a complaint with better production values.

So let me define it.

A layer is a place where a question gets answered

We already added one layer to the AI stack in the last two years, and it worked. Observability gave us a place to answer what the system did. Before it, that question was genuinely hard; outputs appeared, and the path that produced them was a black box. Now we can trace requests end-to-end, replay agent runs, and decompose a failure into discrete steps. The question *what happened* has a home.

The decision layer is the place where a different question gets answered: why did the system choose what it did, and was that choice sound?

That question currently has no home. It gets answered, when it gets answered at all, by an engineer staring at a clean trace and reconstructing intent from signals that were never designed to carry it. The work happens — it just happens in someone’s head, ad hoc, unrecorded, and unrepeatable. A layer is what you have when that work stops living in people’s heads and starts living in the system as structured, queryable state.

The reason observability can’t simply be extended to cover this is that it’s pointed at the wrong object. It captures the path that was taken. The decision layer has to capture the selection of that path — and selection is exactly the thing that gets collapsed away the moment an output is produced.

The primitives of a decision

Here’s the definitional core. A decision, in an AI system, is not the output. The output is the last step. The decision is everything that produced it, and it has parts. If you want to represent a decision as a first-class object — call it a decision record — these are the fields it has to carry, none of which a trace records today:

• Interpretation. How the system read an ambiguous input. The same user message can be understood as a question, a command, or a request to take action. The interpretation chosen determines everything downstream, and it’s invisible — there’s no log line that says “I read this as a command.”

• The considered set. The alternatives that were live at the moment of choosing. Which tools were plausible, which responses were in contention, which retrievals were candidates. A trace shows the one tool that got called. The decision is in the three that didn’t.

• Weighting. Which signals in the context actually drove the choice, and which were present but ignored. Context is not flat — user intent, conversation history, system instructions, and retrieved documents all sit in the input, but they did not all count equally. The trace shows them as equally present. They weren’t.

• Confidence. How close the call was. An output produced by a strong, stable preference and an output produced by a near-tie between two alternatives look identical once realized. Token-level logprobs survive generation — but the margin that matters here is at the level of the *decision* (this ordering versus that one, this tool versus that one), and that margin isn’t sitting in the single realized run. It has to be reconstructed, not read off.

• Context state. Not the raw context window, but its *salience over time* — which earlier assumptions are still live, which have decayed, which should have decayed and didn’t. This is where the slow failures live: drift, context erosion, a stale assumption steering reasoning long after its relevance passed.

That’s the spec. A decision layer is whatever captures those five things as first-class signals rather than leaving them to be inferred. The point of listing them is that you can now check any proposed solution against the list. “Better logging” captures none of them. “Replay the run” re-executes the realized path and recovers none of the alternatives. “Log the chain-of-thought” gets you a narrated path, still no considered set, no weighting, no margin. The list is a filter, and most of what gets pitched as a fix doesn’t pass it.

The hardest of the five is the considered set, and it’s worth being honest about why. It’s defined by absence — the alternatives that didn’t happen — which makes it the one most open to the charge that you’re just inventing plausible-sounding options after the fact. The defense is that the considered set isn’t invented; it’s bounded. The tool schema, the retrieval candidates, the response types the system can produce — these define the space of options that were structurally available at that step. You recover which ones were live by re-sampling the same decision point under controlled perturbation of the context and watching what the system actually reaches for. That’s an estimate, and a falsifiable one. It is not a guess about what could have happened in the abstract; it’s a measurement of what the system does reach for when you hold the step fixed and vary what feeds it.

What it looks like, concretely

Take the failure I described last time: an agent calling the right tools in the wrong order, only under certain combinations of context and history. Clean trace on every failing run. Here is what the trace gave me:

step 1  user_message        "can you update the account and notify them"
step 2  tool_call           update_account(id=831)
step 3  tool_result         200 OK
step 4  tool_call           send_notification(id=831)
step 5  tool_result         200 OK
step 6  final_response      "Done — account updated and the customer notified."

Every line is correct. Every line is also useless for the question I had, which was why this order, in this conversation, when the same agent sequenced it correctly a thousand other times. The trace records that a path was taken. It has nothing to say about why that path was selected over the one I wanted.

Now here’s the same moment as a decision record — and read it for what it is: not a readout, a reconstruction. Nothing in the block below was measured off the failing run. Every value is an estimate produced by a named method, and I’ve tagged each one so the block can’t pretend otherwise. This is the engineer’s-head interpretation written down as data — explicit, falsifiable, and wrong in a legible way when it’s wrong:

decision        ordering: update→notify   vs   notify→update
interpretation  read "update and notify" as sequential, not atomic
                  └ est. from output + tool-call structure
considered_set  [update→notify, notify→update, single transactional call]
                  └ est. from tool schema + re-sampling the step under perturbation
weighting       prior-turn rule ("confirm before any write") → low salience
                system rule ("notify only after confirmed write") → never retrieved
                  └ est. from ablation: drop each signal, observe ordering shift
confidence      ~0.31 margin between top-two orderings  (marginal)
                  └ est. from an ensemble over dozens of replayed re-runs,
                    not a within-run readout
context_state   turn-2 confirmation assumption still live at turn 9 (stale)
                  └ est. from assumption-tracking pass over the conversation

One note on the methods those tags name — re-sampling, ablation, the ensemble of re-runs: every one of them runs against replayed or mocked execution, never the live system. That’s what makes it safe to re-probe a step that once issued a real write like update_account, and cheap enough to run at the volume reconstruction needs.

That block is not a log, and it’s not telemetry. It’s a structured claim about how the choice was made — and because it’s structured, it points somewhere: marginal ordering, a decisive rule that was never retrieved, a stale assumption from seven turns back. In the real case the fix was the retrieval strategy and prompt structure, exactly there. The trace took me a week. The decision record is that week of interpretation, written down once, in a form the next person — or the next automated check — doesn’t have to redo from scratch.

I want to be exact about the claim, because the whole piece turns on it. A decision layer does not make decisions directly observable; nothing does. What it does is take the reconstruction that currently happens informally, in an engineer’s head, and force it into a structured, persistent, derivation-tagged form — something you can store, query, compare across runs, and disagree with on the merits. The values are estimates and the block says so. That’s the move. Not certainty. Structure, and honesty about its provenance, where there was neither.

Why this is a layer and not a feature

It would be easy to read all of this as “add a few more fields to your traces.” It isn’t, and the difference matters.

A feature answers an existing question better. A layer answers a question the layer below it structurally cannot. Observability operates on execution — its atomic unit is the step. You can enrich steps indefinitely and never get a considered set, because the alternatives were never on the path; they’re defined by their absence from it. The decision layer’s atomic unit is the choice, and a choice is a relationship between the path taken and the paths that weren’t. That object doesn’t exist at the execution level. You can’t tack it on. You build above it.

This is also why the decision layer sits where it does in the stack — above execution, below judgment. Execution tells you the system called send_notification. Judgment tells you whether notifying the customer was the right business outcome. The decision layer is the missing middle: it tells you the system chose to notify on a marginal ordering, on a stale assumption, having never retrieved the rule that should have governed it. That’s not the same as knowing the outcome was wrong. It’s knowing whether the decision was sound regardless of how the outcome happened to land — which is the only thing that lets you tell a good system that got unlucky from a bad system that got lucky.

What changes once you have it

Three things move, and they’re the three that the execution view keeps fumbling.

Evaluation stops being output-graded. Most evals score whether the final answer was right. But a marginal decision that happened to produce a correct output is a latent failure wearing a passing grade. Decision records let you grade the quality of the choice independent of whether it got lucky — how close the margin was, whether the decisive context was actually retrieved, whether the considered set contained the right option at all. That’s the difference between measuring outcomes and measuring decisions.

Drift becomes visible before the outputs go bad. The slow failures — behavioral drift, context erosion — are invisible at the output level precisely because no single output looks wrong. At the decision level they show up as a distribution shift: margins narrowing across runs, the same stale assumption recurring, decisive signals quietly dropping out of the considered set. You see the decisions degrade before the answers do.

And that same view is where the cost is hiding. The agent that loops — re-retrieving, re-deciding, burning tokens to relitigate a choice it should have made cleanly the first time — is making a sequence of marginal decisions, and a marginal decision is the unit of wasted spend. You cannot bill that to a line item from a trace; the trace just shows more steps. From a decision record, narrow margins and unstable considered sets are the signal that the system is paying to think in circles. It’s the one beat in here a team feels directly in the bill.

Accountability gets an object to point at. As these systems take consequential actions, “the model decided” stops being an acceptable end of the sentence — for operators internally, and increasingly for anyone governing the system from outside. What an audit needs is an artifact: not the output, not the raw trace, but a record of how the choice was formed and how sound it was. Replay shows you it happened again. A decision record is what lets you ask whether it should have.

In closing

I’ve spent two pieces arguing that decisions, not execution, are what shape behavior. The honest gap in that argument was that I never said what a decision is, concretely enough to build toward. So: it’s interpretation, a considered set, weighting, confidence, and context state — five things a trace structurally cannot hold, each recoverable only as a tagged estimate, and all of them what the decision layer exists to make first-class.

I’m not attached to the name. “Decision layer,” “behavioral intelligence,” “decision quality” — the market will decide which label sticks, and it’ll pick based on which one names the problem people actually feel. The problem is the durable part, and it hasn’t changed since the first piece: teams can trace every step and still can’t explain the decision — why the system chose what it did, whether the choice was sound, or where it’s quietly wasting money making marginal ones.

What’s changed is that the gap now has a shape. Not better logging. Not deeper traces. A layer above execution whose job is decisions — reconstructed, structured, derivation-tagged, and queryable, so the interpretation stops happening in someone’s head and starts happening in the system, where you can check its work.

The execution layer told us what happened. The next one has to tell us why.

The natural response to that problem has been a wave of AI observability tools. Today, we can trace requests end-to-end, inspect prompts, and replay entire agent runs with a level of visibility that didn’t exist even a year ago.

And yet, when something goes wrong, teams still end up in the same place, trying to explain why the system did what it did.

Observability improved visibility. It didn’t solve understanding.

The Current State of Observability

Contemporary AI systems are significantly more observable than earlier generations of deployed models.

Modern observability infrastructure enables reconstruction of execution paths at a granular level, including prompt composition, intermediate tool interactions, and final outputs. In many cases, entire agent runs can be replayed, allowing inspection of how a system progresses from input to output across a sequence of decisions.

This shift has materially changed how production systems are analyzed. Failures that previously appeared opaque can now be decomposed into discrete execution steps, each of which can be inspected and compared across runs.

As a result, AI systems have become increasingly inspectable at the level of execution.

Inspectability at the level of execution does not imply understanding at the level of behavior.

Limits of Execution-Level Observability

Execution Does Not Encode Reasoning

Execution traces capture the sequence of operations performed by a system, but do not encode the reasoning process that led to the selection of those operations.

In deterministic systems, execution paths are a direct reflection of the underlying logic. Given a fixed input, the sequence of operations fully specifies how an outcome was produced. Inspecting execution is therefore often sufficient to explain system behavior.

Autonomous AI systems diverge from this model. Behavior is produced through probabilistic reasoning, where multiple latent decision paths may exist for a given input. The observed execution reflects only the realized path, not the alternative reasoning paths that were implicitly available or the relative weighting that led to their selection.

This makes it possible for identical or near-identical execution traces to arise from materially different reasoning processes. Conversely, similar reasoning processes may produce divergent execution paths under small variations in context.

This decoupling limits the ability to attribute outcomes to specific decision logic based on execution data alone.

Outputs Collapse Underlying Uncertainty

Final outputs represent a collapsed form of the system’s internal reasoning process, obscuring the range of alternatives, assumptions, and intermediate signals that contributed to the result.

During generation, probabilistic systems evaluate multiple potential continuations, each associated with varying likelihoods and contextual relevance. The selected output reflects only the realized sequence, not the distribution of alternatives that were considered or the degree of uncertainty present at each step.

This collapse removes visibility into how strongly different factors influenced the outcome. Signals that were weakly preferred, strongly weighted, or in conflict are no longer distinguishable once the output is produced.

As a result, two outputs that appear similar at the surface level may reflect different underlying levels of certainty or reasoning stability, while outputs that differ may originate from closely related decision processes with small variations in input or context.

This loss of intermediate structure limits the ability to assess confidence, identify ambiguity, or determine whether a given output reflects a robust decision or a marginal selection among competing alternatives.

Context Is Not Represented as a First-Class Signal

While observability systems capture inputs and intermediate states, they do not explicitly represent the contextual structure within which decisions are made.

In practice, context is composed of multiple overlapping signals, including user intent, interaction history, system instructions, and domain-specific constraints. These signals are present within the input space, but are not encoded in a way that distinguishes their relative importance or influence on the outcome.

This makes it difficult to determine which elements of context were materially relevant to a given decision, and which were incidental or ignored. Inputs that appear equally salient at the surface level may have been weighted differently during reasoning, while strongly influential signals may not be identifiable post hoc.

This lack of explicit representation also obscures how context evolves over time. In multi-step interactions or agent workflows, earlier assumptions may persist, decay, or be superseded by new information, but these dynamics are not directly observable through execution traces or final outputs.

Without a structured representation of contextual relevance, interpretation of system behavior remains dependent on inference, rather than direct observation.

From Execution to Decisions

The limitations of execution-level observability reflect a deeper mismatch between how AI systems are analyzed and how they operate.

Execution traces, outputs, and captured inputs describe what a system did, but do not provide a structured account of how decisions were formed. As a result, explanation remains indirect, requiring reconstruction of reasoning from signals that do not explicitly encode it.

This requires a shift in perspective. Rather than treating system behavior as a sequence of executed steps, it becomes necessary to interpret behavior in terms of decisions.

A decision, in this context, is not just an output. It is the result of interpreting inputs, weighing competing signals, and selecting an action under uncertainty.

Decisions are not directly observable. They must be inferred from the relationship between context, intermediate state, and final output. This inference introduces ambiguity, as multiple plausible decision processes may be consistent with the same observed behavior.

As AI systems become more autonomous, this distinction becomes increasingly important. Failures are less often attributable to incorrect execution and more often to misaligned or unstable decisions within otherwise valid execution paths.

In Closing

The inability to directly observe reasoning, preserve uncertainty, or represent context as a structured signal introduces a persistent gap between what can be seen and what can be explained.

In practice, this gap places increasing pressure on interpretation. Engineers, product teams, and operators are required to infer decision processes from signals that do not explicitly encode them, often under conditions where outcomes carry real-world consequences.

As systems become more autonomous, this limitation becomes more difficult to ignore. Explanation is no longer a secondary concern. It is a prerequisite for building systems that can be evaluated, trusted, and controlled.

The challenge now lies in understanding failures at scale, particularly those that emerge from autonomous reasoning within agents and AI-driven workflows rather than from explicit system errors. In these scenarios, systems remain operational and appear healthy from an infrastructure perspective. The failure instead manifests in the produced output, which diverges from the behavior the system was designed to exhibit.

Limits of Logs and Metrics

To understand the limitations of logs and metrics, it helps to examine how traditional product engineering systems are designed and evaluated. These systems rely on deterministic logic, where behavior is fully specified, and deviations can be mapped to identifiable execution paths. As a result, logs and metrics are effective at explaining why failures occur.

Autonomous systems diverge from this model, as behavior emerges from probabilistic reasoning rather than deterministic execution. As a result, the space of plausible reasoning paths underlying any given output expands significantly. This expansion undermines the implicit trust that issues can be resolved by inspecting historical logs and metric records alone.

Probabilistic Reasoning as a Distinct Execution Model

To make these limitations more concrete, it is useful to examine what logs, tracing, and metrics actually capture within AI-driven workstreams.

Logs

• Capture explanatory artifacts rather than deterministic reasoning, describing what occurred without fully specifying why a particular outcome was produced

• Influenced by inputs and context, but insufficient to reconstruct the internal reasoning process that led to a specific output

• Recorded after decisions are made, providing a post-hoc account of behavior rather than visibility into the deliberation that produced it

Tracing

• Capture execution sequence rather than deliberation, showing the order in which operations occurred without exposing how decisions were formed

• Capture execution flow without revealing the relative weighting or influence of factors that shaped a given outcome

• Often requires deep domain and ML-specific context to infer a probable failure point, turning explanation into interpretation rather than direct observation

Metrics

• Capture aggregate outcomes rather than individual decision logic, making them effective at surfacing systemic failures but poorly suited for explaining individual decisions

• Useful for monitoring trends and flagging known risk patterns, rather than diagnosing why a specific output deviated from expected behavior

• Insufficient on their own to represent the system’s behavioral health, as meaningful interpretation requires contextual reasoning beyond what aggregate measures provide

In practice, these execution characteristics give rise to a set of recurring failure patterns once autonomous systems are deployed in production.

Common Behavioral Failure Modes in Production AI Systems

Under probabilistic execution models, failures in production AI systems rarely manifest as isolated errors. Instead, they emerge as recurring behavioral patterns that are difficult to detect through traditional operational signals.

Delayed Failure Propagation

In probabilistic execution systems, failures do not always surface at the point where incorrect reasoning first occurs. Instead, an early misalignment can persist across subsequent decisions, agents, or workflow steps, allowing the system to continue operating while compounding error across the workflow.

Manifestation

• Early outputs appear coherent and valid, allowing downstream processes to proceed normally

• No single decision is obviously incorrect when examined in isolation

• Degradation becomes visible only after multiple reasoning steps have accumulated, affecting the final output

Detection Challenges

• Logs record locally valid intermediate outputs rather than cumulative impact

• Tracing reflects expected execution order, even when reasoning quality degrades

• Metrics often remain within acceptable thresholds until downstream effects emerge

Observability Limitations

• The initial reasoning error is temporally distant from the observed failure

• Root cause attribution requires reconstructing a chain of plausible decisions

• Probabilistic reasoning introduces uncertainty that prevents deterministic replay

Behavioral Drift

Over time, autonomous systems can begin to produce outputs that increasingly diverge from their original design intent, even though no single decision appears incorrect in isolation. From an operational standpoint, the system continues to function normally, while its behavior shifts in subtle ways that are difficult to notice without historical comparison.

Manifestation

• Outputs remain individually plausible and syntactically valid

• Changes in behavior emerge incrementally rather than abruptly

• Misalignment becomes apparent only when comparing current behavior to earlier expectations or baselines

Detection Challenges

• No single output clearly violates constraints or policies

• Metrics often remain stable, masking a gradual directional change

• Drift is distributed across many small decisions rather than concentrated in one failure point

Observability Limitations

• Logs capture point-in-time correctness, not long-term behavioral trends

• Tracing reflects execution flow, not semantic evolution

• Metrics summarize outcomes, but rarely encode intent or alignment over time

Context Erosion

As autonomous systems accumulate context over extended interactions or complex workflows, the quality of that context can degrade. Relevant signals become diluted by accumulated history, assumptions persist beyond their validity, and earlier reasoning steps exert influence long after their relevance has passed.

Manifestation

• Long or multi-step interactions where earlier context dominates later reasoning

• Saturated context windows that obscure which inputs are materially relevant

• Reasoning that appears coherent but rests on outdated or weak assumptions

Detection Challenges

• Individual outputs remain internally consistent and well-formed

• No explicit signal indicates which parts of the context influenced a decision

• Failures emerge from omission or mis-weighting rather than incorrect logic

Observability Limitations

• Logs capture inputs and outputs without encoding contextual salience

• Tracing reflects sequence, not relevance or decay of assumptions

• Metrics summarize outcomes but do not reveal when context quality has degraded

Contextual Misalignment

Autonomous systems may reason correctly relative to the context they internally construct, even when that context no longer aligns with real-world expectations or operating conditions. The resulting behavior appears coherent and well-formed, yet produces outcomes that feel inappropriate or incorrect to human operators.

Manifestation

• Logically consistent outputs but semantically misaligned with user intent or environmental reality

• Correct reasoning applied to an outdated, incomplete, or implicitly incorrect context

• Divergence between what the system optimizes for and what stakeholders expect

Detection Challenges

• Reasoning chains remain internally valid and self-consistent

• No explicit signal indicates that contextual assumptions are incorrect

• Failures are often attributed to “judgment” rather than system behavior

Observability Limitations

• Logs capture inputs and outputs without validating contextual correctness

• Tracing reflects execution order, not semantic alignment

• Metrics summarize outcomes, but cannot encode whether the correct context was applied

Overgeneralization Under Ambiguity

When operating under incomplete, noisy, or ambiguous inputs, autonomous systems may collapse nuanced distinctions into broader reasoning patterns. This produces confident outputs that appear reasonable while masking the loss of specificity required for accurate decision-making.

Manifestation

• Broad or generic responses applied to cases requiring contextual nuance

• Edge cases absorbed into dominant reasoning patterns

• Reduced sensitivity to subtle but important input differences

Detection Challenges

• Outputs remain fluent, confident, and structurally valid

• No clear threshold distinguishes acceptable generalization from harmful oversimplification

• Failures emerge primarily in low-signal or under-specified scenarios

Observability Limitations

• Logs record final outputs without capturing lost nuance

• Tracing shows normal execution paths despite degraded reasoning quality

• Metrics may reward consistency while penalizing necessary specificity

Taken together, these failure modes reflect a common pattern in production AI systems; failure rarely occurs as a discrete event. Instead, it emerges through accumulation, drift, and degradation of context, often while systems remain operational and outputs appear locally valid. Because these behaviors unfold across time, decisions, and context rather than at a single execution point, they resist explanation through traditional observability signals designed for deterministic systems.

In Closing

As AI systems continue to move deeper into production environments, the limits of traditional system understanding become increasingly apparent. When behavior is generated through probabilistic reasoning rather than deterministic execution, explanation becomes as critical as detection, particularly in contexts where humans remain accountable for outcomes. Developing better ways to reason about AI behavior in production is therefore not an optimization problem, but a foundational requirement for building systems that can be evaluated, trusted, and governed over time.