All posts

We tried RPA and it failed, and why agents are a different mechanism

Operators who burned a budget on robotic process automation are right to be skeptical of the next automation pitch. The honest answer is to explain why a model-based agent fails differently, not to insist it cannot fail at all.

Javier Leguina

Javier Leguina

Co-founder & CTO of flowscope, previously a founding engineer at ModelML (YC W24).

· Buyer objections, answered

The most useful objection an operator can raise is that they have done this before. They bought robotic process automation, a vendor wired bots to their screens, the program ran for a while, and then it broke and stopped paying for itself. When an agent vendor walks in with a similar promise, the operator's prior is grounded in a real failure rate. EY reported in 2016 that it had seen as many as thirty to fifty percent of initial RPA projects fail. Dismissing that history is the wrong move. The useful move is to explain why a model-based agent is a different mechanism, with a different failure mode, so that the prior failure becomes evidence about brittleness rather than a verdict on whether automation can work here at all.

What robotic process automation actually does

A robotic process automation bot does not understand the work. It records a sequence of operations against a user interface and replays them: click the field at this position, read the cell in this column, copy that value, paste it into that other application, press submit. The instructions are tied to selectors, screen coordinates, fixed field names, and an assumed document layout. Within the exact conditions it was recorded against, this is fast and reliable. It is deterministic in the literal sense that the same inputs produce the same steps every time, which is why it passed its initial demo and why the early savings were real.

The fragility is structural and it follows directly from how the bot is built. Because the script is pinned to the surface of the system rather than to the meaning of the task, any change to that surface breaks it. A vendor ships a portal redesign and the button moves. A supplier reformats its invoice and the total lands two rows lower. The ERP adds a confirmation dialog that was not there when the bot was recorded. None of these are changes to what the work means, yet all of them stop the bot. Someone then has to re-record the path or patch the selector, and that maintenance recurs every time any upstream system changes, which in a mid-market environment running several vendors' software is constant.

Why the maintenance burden outran the savings

This is the mechanism behind the EY range. RPA savings are realized once, at deployment, and then decay as the surrounding systems drift. Maintenance cost, by contrast, scales with the number of integration points and the rate at which they change, and it is incurred for the life of the program. A bot that automates ten steps across four applications is exposed to every interface change in all four. Add more bots and the surface area of things that can break grows with them. The program does not fail in one event. It erodes as the team that was supposed to be freed up gets pulled back into keeping the bots running, until the controller quietly decides the manual process was less trouble. The operator who lived through that is describing brittleness, and brittleness is a property of fixed-script automation specifically.

What changes when a model reads intent

A model-based agent is built on the opposite assumption. Instead of replaying a recorded path, it reads the input, infers what is being asked, and decides what to do. Given an invoice, it does not look for the total at a memorized coordinate. It reads the document, identifies which number is the amount due regardless of where the vendor put it, and proceeds. The variance that breaks a script, a new layout, a relabeled field, a wording change, is the variance a model tolerates, because it was trained on enough variation to generalize past it. This is why the long tail of formats that defeats template-based extraction is tractable for an agent, a point worth reading in full in the long tail of document variability, and it is the same reason the common worry that the inputs are too inconsistent to automate, addressed in the objection that the data is too messy, turns out to be the wrong frame.

The agent reading intent rather than position is also what makes it possible to write results back into systems that were never designed to be automated against. A bot pinned to a brittle screen path and an agent that can operate the same interface adaptively are not the same tool pointed at the same problem, as writing back into a system with no usable API lays out.

The new failure mode, named honestly

None of this means the agent is safe by default, and pretending otherwise would repeat the original error of the RPA pitch. The agent trades determinism for tolerance, and the cost of that trade is non-determinism. The same input can produce a different answer, and the model can produce a confident, well-formatted answer that is wrong. A fixed script fails loudly by halting, while a model can fail silently by proceeding. That is a worse failure mode if it is left unbounded, because a halted bot gets noticed and a wrong number posted to the ledger may not.

So the agent has to be bounded by construction. It is grounded against the actual source documents and system state rather than asked to recall, its outputs are checked against an evaluation set before it runs unattended, and the cases it is not confident about are routed to a person on a short, ranked exception list instead of guessed at. Where the consequences of a wrong write are serious, the rule for what stays under human control is the subject of when not to automate. The agent does the mechanical majority, and the exceptions go to a human. That boundary is the whole engineering problem, and a vendor who cannot describe theirs is selling the unbounded version.

Why prior failure is not the verdict

A reasonable counter is that the wider record on this newer technology is no better than RPA's, so the operator's pessimism should extend straight through. The numbers are real. Gartner has warned about agent-washing and forecast that over forty percent of agentic AI projects will be canceled by the end of 2027, and MIT's NANDA initiative found that about ninety-five percent of enterprise generative-AI pilots delivered no measurable production value. Both figures should temper any promise. They do not, however, collapse the distinction. The RPA failures trace to a mechanism that breaks on surface change. The agent failures trace overwhelmingly to projects that were never bounded, never grounded, and never integrated into the real workflow, the same gap MIT identified. Those are correctable engineering choices rather than properties of the mechanism. The operator who saw RPA fail learned something true about brittle scripts. What remains open is whether this vendor has built the bounded version or the agent-washed one, and that is answerable by inspection rather than by analogy to the last thing that broke.

Common questions

Why did our RPA program work in the demo but break down over time?
An RPA bot replays a recorded sequence of operations tied to selectors, screen coordinates, fixed field names, and an assumed document layout, so it performs reliably under the exact conditions it was recorded against and its early savings were real. The fragility is structural because the script is pinned to the surface of the system rather than to the meaning of the task. Any change to that surface, such as a portal redesign that moves a button, a reformatted invoice, or a new ERP confirmation dialog, stops the bot even though the work itself has not changed. Maintenance then recurs with every upstream change, which in a mid-market environment running several vendors' software is constant.
How is a model-based agent actually different from the RPA bots we already tried?
Instead of replaying a recorded path, a model-based agent reads the input, infers what is being asked, and decides what to do. Given an invoice it does not look for the total at a memorized coordinate; it reads the document and identifies which number is the amount due regardless of where the vendor placed it. The variance that breaks a fixed script, such as a new layout, a relabeled field, or a wording change, is the variance a model tolerates because it was trained on enough variation to generalize past it. That tolerance also makes it possible to write results back into systems that were never designed to be automated against.
If agents are non-deterministic, what stops one from quietly posting a wrong number to our ledger?
An agent trades determinism for tolerance, and the cost is non-determinism, which means it can produce a confident, well-formatted answer that is wrong and fail silently by proceeding rather than halting. To prevent that it has to be bounded by construction: it is grounded against the actual source documents and system state rather than asked to recall, its outputs are checked against an evaluation set before it runs unattended, and cases it is not confident about are routed to a person on a short, ranked exception list instead of guessed at. The agent handles the mechanical majority while exceptions go to a human, and a vendor who cannot describe that boundary is selling the unbounded version.