Reading documents reliably

Start with the pile, not the software. A month of real intake for one process, supplier invoices say, will hold documents generated by accounting systems, spreadsheets exported to PDF, paper invoices scanned at a branch and photographs taken on a delivery dock. The invoice number sits top right on one, in a footer on another and inside a sentence on a third. None of this is unusual. It is what a document process looks like, and it is the reason an agent is being considered rather than a template.

The first distinction that matters is how the text got onto the page. A native PDF produced by software carries a text layer: the characters are stored as characters, and reading them is exact. A scan or a photograph carries only pixels, so the software must first recognise the characters, which is optical character recognition, before the reader, the part of the software that extracts the values, can use them. Some files mix the two: a native invoice with a handwritten amendment, or an old scan that was run through recognition years ago and saved with its unreliable text layer hidden underneath the image.

Reading from pixels produces errors of a particular shape, and it is worth knowing them by name. Similar characters swap: zero and the letter O, one and a lower-case L, five and S. A minus sign or a decimal separator vanishes into a speck. Table columns run together, so a quantity attaches itself to the wrong line. A received stamp lands on the total. Quality also varies with things nobody controls: the resolution of the scanner, a fold across the page, a photograph taken at an angle under warehouse lighting. Each of these produces a value that looks entirely plausible, which is exactly what makes it dangerous.

Layout causes the rest. Tables that break across pages, running subtotals repeated at the foot of each page, two documents in one file, a remittance advice stapled behind the invoice it pays, pages that come out in the wrong reading order, dates written day first by one sender and month first by another. An agent reads meaning rather than fixed positions, which is why it copes with variety a template cannot. It does not follow that it copes with every variety equally well, and you only find out which ones it struggles with by measuring them separately.

So the first piece of work is a census. Take a month of real intake, not the clean examples someone saved in a folder, and sort it: native, scanned, photographed or mixed; which senders; which layouts; how many pages; what share carries handwriting or stamps. It is tedious work, and the rest of this module depends on it. That census becomes the shape of your validation set in lesson four. Any accuracy figure quoted across the whole mix is an average of kinds that read well and kinds that do not, and the kind that fails is usually the one that matters.

Every extraction system will offer you a confidence score. The first question is what it attaches to. A score for the whole document tells you very little you can act on, because a document is not right or wrong as a unit. The supplier name can be certain while the bank account number is a guess. One number for both averages away exactly the value you most need to see. The decision you actually make, whether to post, check or ask, is made field by field, so the confidence has to be field by field as well.

The second question is where the number comes from. Ask a language model how sure it is and it will answer, fluently, with a figure. That figure is text the model generated in the same way it generated the value. It is not a measurement of how often values like this one turn out to be correct. Asking it to be more careful, or to rate itself more strictly, changes the wording of the figure, not what it measures. It can be high on a value the model misread and low on a value it got right, and nothing in the output tells you which is which.

The property you want is calibration. A score is calibrated when, across many values, the ones scored at a given level are correct about that often: of the values marked very likely, very few are wrong; of the values marked doubtful, a good number are. Calibration is not something a model has in the abstract. It is something you measure, on your own documents, by comparing scores with known-correct values, and it can differ from one field to the next and from one layout to the next.

The better sources of confidence are signals you can inspect. Did recognition report low certainty on these characters? Does the value appear verbatim in the text, or was it inferred? Did two independent reads, using a different method, a different crop or a different reader, produce the same value? Does it pass its format rule and the checks in lesson three? Each of these is concrete, each can be logged beside the value, and together they give you a score you can test against your validation set.

Then set the thresholds with the validation set in front of you, one field at a time. For each field, read off what error rate would post at each score and what share of documents would go to a person. A bank account number deserves a stricter threshold than a free-text description, because an error that escapes costs more. Where no threshold gives an acceptable error rate at a volume people can review, the field is not ready to run unattended. That is the oversight arithmetic from Module 01 applied to a single field, and like any threshold it is a parameter you revisit rather than a setting you forget.

Confidence tells you how sure the reading is. A check tells you whether the value is consistent with something else. The second is worth more, because it does not depend on the thing being checked. Documents carry a good deal of internal redundancy, and your own systems hold most of the rest. Checks are also easier to explain: a reviewer or an auditor understands a total that disagrees with the order without knowing anything about how the reading was done. An extraction design that uses neither is asking the reader to be right on its own word, which is precisely the thing you cannot verify.

The cheapest checks are arithmetic. Quantity times unit price should equal each line amount; the line amounts should sum to the subtotal; subtotal plus tax should equal the total; the tax should correspond to a rate that is actually in use. When recognition turns an 8 into a 3, one of these usually breaks. None of this needs a model; it is ordinary arithmetic, run on every document. Format rules do the same work for single values: a date must exist, an account number must pass its check digits, a currency code must be real, a reference must match its known pattern.

The strongest checks use a second source. The purchase order says what was ordered and at what price. The supplier master record says which bank account belongs to this supplier. The contract states the rate. The goods-received record says what actually arrived. A total that agrees with the order is evidence. A bank account that differs from the one on file is not a reading problem at all, and it should go to a person whatever the confidence says, because the document itself may be the fault.

Checks have failure modes of their own. The commonest is the check that cannot fail: if the software computes the total by adding the lines it extracted, the total always reconciles and nothing has been tested. The second is the matching value that is wrong, such as an order number that exists but belongs to another supplier. The third is a tolerance so wide that misreads pass through it. A check that has never failed in live running deserves a second look for exactly this reason. For each check, write down the two sides of the comparison and where each one came from.

Record the outcome of every check against the value, not only the final decision. Knowing that a total passed its arithmetic and matched the order, or passed its arithmetic with no order to compare against, is what lets a reviewer decide in seconds and an auditor follow the reasoning months later. It also shows where coverage is thin. Fields with no independent check are fields where you rely on confidence alone, and they should carry stricter thresholds or go to a person.

The validation set for extraction is the set of known-outcome cases from Module 01, applied to reading. It is a collection of real documents, each paired with the correct value for every field, keyed by people who know the documents. It is also the only thing that lets you say whether a change to the software made reading better or merely different. Its shape should follow the census from lesson one: the same senders, the same layouts, the same share of scans and photographs as the intake. A set assembled from clean examples measures clean examples, and nothing else.

Getting the expected values right takes more care than it sounds. Have two people key each document independently, then compare. Where they disagree, you have found either a keying slip or a genuine ambiguity, such as a document carrying two dates that could each be the invoice date, and an ambiguity needs a written rule before it can be scored. Decide too what counts as a match: whether spacing matters in a reference, how dates and amounts are normalised, when a blank is the correct answer. This work is slow and unglamorous, and most of the value of the set comes from it.

Then score per field, and per field within each kind of document. An overall accuracy figure is an average over fields that do not matter equally and are not equally hard. A system can read supplier names and dates almost perfectly while misreading the account number on one scanned layout, and still report an excellent overall figure. The field that decides where money goes deserves its own line in the report, its own threshold and a named owner who acts when it moves.

Once the system is live, the set gradually stops describing the world. Suppliers redesign their invoices, a branch buys a new scanner, a new customer sends a format nobody has seen, a sender switches from native files to scans. Nobody warns you in advance: the supplier does not write to say its invoice has a new layout, and the first sign is the documents themselves. Drift in extraction rarely arrives across the board. It arrives one sender or one layout at a time, and a change affecting a single sender barely moves the overall figure while it quietly corrupts every document that sender sends.

So monitor at the level where drift happens. Track, for each sender and each layout, the share of fields below threshold, the share failing checks and the share a person corrected. A sudden rise for one sender should be treated as a template change until the documents show otherwise. An alarm nobody reads is not an alarm, so name the person who receives it. Feed corrected documents back into the validation set so that it keeps pace with the intake, and re-score the whole set before and after any change to the software, the model or the thresholds.

Every extraction design needs a written answer to one question: what happens to a value that falls below its threshold or fails a check. There are three good answers and one bad one. The bad one is the commonest: fill the field with the likeliest reading, or with a default, and carry on. That turns uncertainty into a figure that looks exactly like a certain one, and it is how misreads reach the ledger without anyone having decided anything. Write the answer down field by field, because the right route for a missing order number is not the right route for a smudged digit.

The first good answer is to re-read. A different method often settles a character-level misread: recognition at a higher resolution, a tighter crop around the region, a second reader that works differently. The re-read is only useful if it is independent of the first; asking the same reader the same question again tends to return the same answer. Do it once. If you retry until two reads agree, you are no longer measuring the document, you are selecting the answer you hoped for.

The second is to ask. Some values are not on the document at all, or are illegible on the paper itself: a missing order reference, a smudged account number, a total hidden under a stamp. No amount of reading recovers what was never written. The right move is a request to the sender that names the missing field and says why it is needed, with the case held until the answer arrives. Say exactly what is missing and what the sender should send back; a vague request tends to produce a second document as hard to read as the first. It is slower, and it is correct.

The third is to route to a person, and how you route decides whether the review works. Send the reviewer the page with the region highlighted, the text as read, the candidate values and the check that failed, not the whole document and an empty form. A reviewer who can see why the software doubted a value decides in seconds. One who has to search the document is redoing the extraction, and will soon start approving without looking. The reviewer's decision, and the reason for it, is then recorded against the value like any other step.

All of this depends on keeping the evidence. For every value that posts, store the page, the region on it, the raw text as read, the normalised value, the method, the confidence signals and the outcome of each check. That record lets a reviewer confirm quickly and an auditor trace any figure back to paper. It turns every correction into a new case for the validation set. And when an error is found, it tells you which other documents from the same layout to reopen.