Somewhere between a pilot and a fleet, the number of decisions an AI system makes passes the number anyone on the team can look at. What has to exist after that point is different in kind from what came before.
The invoice agent had been running for seven months when someone asked how it was doing, and the honest answer was that nobody knew.
It had started narrow. Match supplier invoices against purchase orders, approve the clean ones, route anything unusual to the AP team. That worked, so it grew. It began handling partial deliveries, then credit notes, then invoices from suppliers with no purchase order at all, on the reasoning that a human was reviewing the exceptions anyway. By month seven it was touching most of what came through the inbox, and the exceptions queue had stayed roughly the size it was in month one.
That sounds like success. It is also exactly what the problem looks like from the inside.
I wrote recently about why only certain people can explain an AI failure, and why that concentration survives every attempt to hire against it. That piece assumed something it never examined: that a failure had surfaced, and the question was who could account for it. The constraint I want to look at here sits upstream of that one, and it binds earlier. Before anyone asks why a decision was made, somebody has to notice the decision was worth asking about.
Two things grow at different rates
When a team adds an agent, it is usually replacing a step a person used to do. The instinct is to reason about it that way: the agent does the thing, the person checks the thing, capacity roughly holds.
What actually happens is that the scope creeps in a particular direction. An agent that works gets handed adjacent cases, and each new case type expands the number and variety of decision points inside a run. The invoice agent did not go from a hundred decisions to two hundred. It went from one kind of judgment made many times to several kinds of judgment, compounding, with the harder ones added last precisely because the easy ones had gone well.
Human review does not move like that. It is bounded by attention, and attention is bounded by headcount, and headcount moves slowly and imperfectly for the reasons the expertise piece covered. So the ratio between decisions made and decisions looked at does not drift. It opens.
The part that gets missed is what fills the gap. Nothing stops being reviewed on purpose. There is no meeting where a team decides to stop watching the credit-note path. The reviewed set simply stops being chosen and starts being whatever surfaced: what a customer complained about, what threw an error, what someone happened to open. The team still feels like it is reviewing its AI system. It is reviewing the part of its AI system that made noise.
The decisions that don’t make noise
Go back to the invoice agent.
A supplier changes its billing entity after an acquisition. Same goods, same delivery addresses, new legal name on the invoice, new tax identifier. The agent has no purchase order match on the new entity, which is one of the cases it was extended to handle, so it falls back to matching on delivery history and line-item similarity. It finds a strong match. It approves.
Every one of those approvals completes. No error, no retry, no timeout, nothing that a monitor is watching for. The supplier gets paid, which is the outcome everyone wanted, so no human on either side has a reason to look. Whether that behavior was right depends on something the run does not contain: whether this company treats a post-acquisition entity as the same counterparty or a new one requiring re-onboarding. If it is the second, the agent has been quietly routing payments outside a control for weeks, successfully.
None of this is hard to explain. Hand it to the engineer who built the fallback and they will have an answer quickly. The expertise is available. What was never available was the prompt to use it. That decision had no property that would cause anyone to pull it out of the stream, because the properties production systems select on are failure, latency, and cost, and this was fine on all three.
Sampling finds the typical, not the consequential
The obvious answer is to look at a sample, and it is worth being precise about why that helps less than it seems to.
Small random samples are good at telling you how the system usually behaves. That is a real question, and it is not the one being asked here. The decisions that matter are rare by construction: a new entity type, an edge in a rule that only fires under a condition that shows up occasionally, a path that opened when someone shipped a change nobody connected to decision behavior. A sample sized to be reviewable by a team is unlikely to contain them, and if it does, there is nothing marking the one that matters as different from the others in the sample.
Spot-checking has the same shape with a worse selection rule, because the human picks what to open, and people open what they already suspect.
Neither is a bad practice. They answer a different question than the one a team crossing this threshold actually has, which is not how does this usually go but which of the things I did not look at should I have looked at.
What has to exist instead
That question has a structural requirement hiding in it. To rank decisions by whether they deserve attention, something has to have formed a view on every decision, including the ones nobody asked about, at the time they happened.
That is a different posture from investigation. Investigation is a thing you start, which means it presupposes the noticing that has already failed. What the situation calls for is an assessment that exists by default: a standing judgment on each decision, produced whether or not a human ever opens it, so that when someone does have an hour, the thing they spend it on was selected on the merits rather than by whatever escalated.
This does not need a new tier of the stack. The decision record from the earlier pieces already holds the material; what changes is that it has to be doing the work continuously rather than on request. A record you can query is only useful to someone who knows to query it. The same record, assessed as it is created and held against what that decision point has looked like historically, can put itself forward.
Two properties matter more than the mechanism, and they are the ones I would evaluate any approach on.
The basis has to be inspectable. If a system tells a team which decisions deserve a second look and cannot show what about them earned that, it has replaced a judgment nobody was making with a judgment nobody can examine, which is worse, because now there is a number and the number feels like an answer.
And it has to be honest about what it does not know. A standing assessment on a decision point with no history behind it is a weaker claim than one on a path that has run thousands of times, and a system that reports those with the same confidence is manufacturing certainty at exactly the moment a team is deciding where to spend its scarcest resource. The most valuable output in the invoice case is the observation that this path is new, that it began firing after a specific change, and that nothing in the system’s history says what normal looks like for it yet. That is worth more than a verdict.
The conclusion I’d draw
The gap that opens when a workflow outgrows its team is usually described as a resourcing problem, and resourcing is genuinely part of it. But the part hiring does not solve by itself is the selection step. An organization can employ every person capable of explaining an AI decision and still be examining an essentially arbitrary subset of what its systems do, because the mechanism that decides what reaches a human was never designed. It accumulated.
Execution tells us what happened. A decision record tells us why this happened. A population of those decisions tells us what normal looks like. The thing this piece adds is that in a system nobody has the hours to read, all of that has to be produced without being asked for, or it is produced for the wrong decisions.
I am building Nalyqor toward that, on the belief that standing assessment is what the record turns into once volume makes querying it impractical. The diagnosis I am more confident about than the bet: the constraint on most teams is no longer who can explain what the system did. It is that nothing currently decides what they should be looking at, and by the time a workflow has outgrown the team running it, that decision has already been made by default.