"Automate data entry" is a phrase behind more failed back-office projects than almost any other, because it names only the first eighty percent of the work. A company licenses a template-based OCR product, points it at a stack of invoices, and watches it read the clean ones well, with invoice numbers, line items, totals, and PO references all landing in the right fields, and the demo wins the deal. Then the system meets the documents that actually arrive every day, at which point the queue of exceptions a human has to fix grows until the savings disappear into the salary of the person hired to handle the machine's misreads. The eighty percent of documents that look like the template was never the work. The long tail that does not look like the template always was.
The seventy-percent ceiling is structural, not a tuning problem
The industry has a number for how far rules and template OCR get on their own, and it is not flattering. Across the intelligent document processing market analyses from GMInsights, Market.us, and Precedence Research, traditional rules-and-template extraction tops out at straight-through processing in the region of seventy percent, while model-assisted pipelines push that figure into the high nineties. Straight-through processing is the share of documents that move from arrival to posted record with no human touching them, and seventy percent sounds respectable until you do the arithmetic on the other thirty. If three documents in ten kick out to a person, and a person can clear one every minute or two once you count the context-switching and the lookups, you have not removed the clerk. You have reorganized the clerk's day around the documents the machine cannot post.
The reason the ceiling holds is worth stating plainly, because it gets blamed on bad configuration when it is really a property of the method. A template encodes a fixed expectation, that the invoice number lives here, the total lives there, the dates match this format, and that expectation is correct exactly as long as the documents conform to it, which real documents do not. A new vendor sends a layout the template has never seen, an approval gets scrawled in the margin by hand, or a field that was top-right last quarter shows up in a footer because someone changed their accounting software. Each of these is individually small and collectively unbounded, which is why every additional vendor, format, and exception path adds work the template was never going to absorb. You can tune a template against the documents you have, but you cannot tune it against the ones you have not received yet.
This is the eighty-to-ninety-nine problem at the document level
We have written before about the 80-to-99% chasm, where the last sliver of capability costs far more to reach than the easy majority that precedes it. Document extraction is the same gap at the scale of a single page, where the clean invoice is the easy majority and the handwritten note, the misplaced field, and the vendor nobody onboarded are the long tail that costs almost everything. Template OCR was always going to stall there for the same reason every model demo stalls at the happy path, which is that the variance lives in the cases the method was never built to see.
A language model changes the economics of the tail specifically because it does not need the field to be where it expected. Given an invoice it has never encountered, it can reason from the surrounding text that a number adjacent to "Balance Due" is the total even when no template anchors that position, and it can read intent out of a handwritten margin note rather than rejecting the page as malformed. The model is not better than the template on the clean eighty percent, where both work fine and the template is cheaper; it earns its place on the thirty percent the template hands to a human, and that is where straight-through processing climbs from the seventies toward the high nineties.
The cost you are paying while the tail stays manual
The case for closing this gap is not abstract, because manual data entry is an expensive and error-prone cost that operating businesses pay every month without ever booking it as a line item. IBM's frequently cited estimate put the cost of poor data quality at roughly $3.1 trillion a year for US businesses, a figure that dates to around 2016 and should be read as a measure of scale rather than a precise current number, though the structure it points at has not changed: a person keying figures from a document into a system of record makes mistakes, those mistakes propagate into reconciliations and reports and payments, and the cost of catching them downstream far exceeds the cost of the keystroke. Every document that exits the automated path and lands in a manual queue adds to that cost, which is why the point of lifting straight-through processing is not the elegance of the pipeline but the shrinking of the population of documents that any human ever has to retype.
Why the human stays in the loop, and why that is the design
The honest version of this argument does not pretend the tail disappears, because a model in the high nineties still kicks out a small share of genuinely ambiguous documents, and the right response is not to chase a hundred percent that does not exist but to design the human-in-the-loop fallback as a first-class part of the system. The clerk who used to retype thirty percent of the stack now reviews the handful the model flags as low-confidence, which is a different job at a different scale, and the model learns from those corrections so the flagged share keeps shrinking over time. Extraction off the long tail is only half the work, of course, because the figure still has to land in the ledger, and writing it back into a system with no usable API is its own discipline; getting the number right is the part the variance makes hard in the first place.
A reasonable counter is that the right move is not better extraction but document standardization, making every vendor send a structured file so the long tail goes away. There is something to this, and where you control both ends of the exchange, structured data beats any amount of clever reading. The trouble is that an operating business does not control its vendors, its customers, or the handwritten note a foreman left on a delivery slip, and it cannot wait for the world to standardize before it stops paying clerks to retype. The variance is not a defect in the documents. It is how businesses actually communicate, which is exactly why the discovery has to start from how the work really runs rather than from the tidy template the brochure promised. The long tail of documents that do not match the template was never an edge case at the margin of the job. It is the job.