What Manufacturing Taught Me About Agent Failure Modes

Software people discussing autonomous systems tend to talk as though the problem is new. On a shop floor it is not new at all. Manufacturing has been running systems that act on the physical world, without a human confirming each step, for a very long time, and it has developed a specific and unsentimental discipline about how those systems fail.

Most of that discipline transfers directly to agents, and almost none of it appears in the AI conversation.

The floor's core assumption is that the system's record and physical reality will drift apart, and that this is normal rather than exceptional. Everything else follows from taking that seriously.

Drift is assumed, not treated as a defect

An inventory system says there are four hundred units. There are three hundred and ninety one. Nobody on the floor finds this shocking. Scrap happened, a count was off, something was consumed against the wrong order, something is physically present and logically committed elsewhere.

The response is not to demand a system that never drifts. It is cycle counting: a scheduled, resourced, permanent ritual of comparing the record to the thing, with a tolerance, an investigation threshold, and an owner.

Software culture treats reconciliation as a sign that something is broken. Manufacturing treats it as a cost of operating in a world where the record is a model and the model is always slightly behind. That framing is more accurate for agents too. An agent's understanding of the system's state is a model of a model, one degree further from the world, and it will drift for reasons nobody did anything wrong to cause.

The practical consequence is that an agent touching operational data needs a scheduled comparison against ground truth, with somebody who owns the discrepancies. Not an alert. A ritual.

Stopping is a designed outcome

Andon is the practice of stopping the line and signaling for help. What makes it interesting is not the cord, it is that the authority to stop was deliberately given to the person nearest the problem, and that using it is treated as correct behavior rather than as failure.

This is close to the exact opposite of how autonomous software is usually built. We reward completion. An agent that finishes is a success, an agent that halts is a case to be engineered away, and the pressure across releases runs steadily toward fewer stops.

That pressure is the failure mode. On a line, a system that continues while wrong produces defective units at speed, and the cost of the run is far higher than the cost of the stop. The same arithmetic holds in an ERP: an agent that proceeds through ambiguity produces a batch of consistent, confidently wrong records, and consistency makes them harder to find later than a single obvious error would have been.

So the design question is not how to make the agent stop less. It is what a good stop looks like: what it signals, who receives it, what context it carries, and how quickly a human can resolve it and resume. A stop that pages a person with no context at two in the morning is not an andon, it is an alarm, and teams learn to ignore alarms.

Make the wrong action impossible rather than unlikely

Poka yoke is mistake proofing: designing the fixture so the part cannot be inserted backward, rather than training the operator to be careful. The physical world rewards this because parts get inserted backward regardless of training, at three in the morning, in the last hour of a long shift.

The software equivalent is constraining at the interface, not at the agent. If an agent must never post to a closed period, the enforcement belongs in the permission set and the interface, not in the prompt. Prompts are guidance. Permissions are physics.

This one matters more for agents than it did for people, because an agent will attempt the same wrong thing at machine speed and without the hesitation that makes a human pause and ask. Every boundary that exists only as instruction is a boundary that will eventually be crossed politely and repeatedly.

Genealogy beats logging

In a regulated plant you can take a finished unit and reconstruct it: which lot each component came from, which machine, which operator, which shift, which process parameters. Genealogy exists so that when a defect surfaces months later, you can determine the blast radius precisely rather than recalling everything.

Application logging usually does not do this. It records that an action occurred and often what the result was. It rarely records what the system believed at the time.

For an agent, the equivalent of genealogy is the reasoning inputs: the data it read, the version of the rules it applied, the model and prompt in use, and the alternative it rejected. Store that with the output. When something is discovered to be wrong six weeks later, the question is never "what did it do." It is "what else did it do for the same reason," and only genealogy answers that.

This is also the honest answer to why a plant's traceability requirements feel excessive until the first recall, at which point they are the only thing standing between a contained problem and an uncontained one.

Where the analogy stops

Manufacturing's physical constraints impose a natural rate limit. A machine can only run so fast, a line only produces so many units an hour, and that ceiling has quietly bounded the damage of every bad decision the system has ever made.

Software has no such ceiling. An agent can make ten thousand wrong writes in the time a line makes one bad part, and it will do so without any of the sensory signals, the noise, the smell, the pile of scrap, that tell a floor something has gone wrong before the report does.

So the floor's disciplines transfer, but the margin does not. Which argues for the rate limits being explicit and designed rather than inherited from physics: batch sizes, throttles, daily caps, staged rollouts by entity or site. Not because the agent is expected to be wrong, but because everything else in this article assumes somebody gets to notice in time.

The part that stays with me

The floor's real lesson is cultural rather than technical. Manufacturing accepted long ago that its systems are approximations of a physical world that does not consult them, and it built rituals, authority, and tooling around that acceptance instead of around the hope of eliminating it.

Enterprise software has generally not accepted this about itself, and agents are about to make the gap between the record and the world expensive enough to notice.