Document extraction: what to verify before you trust it

Never post extracted data straight into a system of record. Validate it in three layers first: field-level format and type checks, cross-field arithmetic that must reconcile, and a lookup against data you already hold. Route anything that fails to a human queue. A pipeline reporting 96% accuracy is usually reporting an average that hides the one field failing 20% of the time.

Extraction is the easy half. Deciding what to trust is the half that determines whether the pipeline saves work or creates a different kind of work.

Key takeaways

Whole-document accuracy is a useless metric. Measure per field.
Arithmetic that must reconcile is your cheapest and strongest check.
Matching against data you already hold catches what no format check can.
A low-confidence field should route to review, not fail the document.
Measure how often review changes something. That number is the real accuracy.

Who this applies to

You are processing invoices, purchase orders, delivery notes, contracts or forms through an extraction pipeline and deciding what has to be checked before the data reaches your accounting, ERP or CRM system.

Why one accuracy number is misleading

A vendor reports 96% accuracy. Three questions make that number mean something.

Accuracy of what? Whole documents, or fields? If a document has 12 fields and each is 96% accurate, the chance all 12 are right is about 61%. Whole-document accuracy is always dramatically lower than field accuracy and it is the number that matters operationally.

Which fields? Averages hide distribution. A pipeline can be 99% on invoice number and 78% on line-item quantities, and average to something respectable while being unusable for the thing you cared about.

On what documents? Clean digital PDFs from three suppliers, or the actual mix including the scanned one, the photographed one and the one that comes as a spreadsheet renamed to .pdf?

Insist on per-field accuracy on a representative sample of your own documents. Anything else is a demo statistic.

The three validation layers

1. Field-level checks

Cheap, deterministic, and they catch a surprising share.

Type and format. Dates parse and are plausible - an invoice dated 2019 or next March is suspect. Currency amounts are numbers. Tax identifiers match the expected pattern for the country.
Range. A unit price three orders of magnitude off the historical range for that item is almost certainly a decimal error.
Required presence. If a field is mandatory and empty, that is a failure, not a blank.
Enumerations. Currency codes, country codes, units of measure must be in the allowed set.

None of this needs a model. It is validation code, and it should run on every extraction.

2. Cross-field arithmetic

The strongest check available on financial documents, and the most frequently skipped.

On an invoice: line items sum to the subtotal, tax computes correctly from the subtotal at the stated rate, subtotal plus tax equals the total. If any of those fail, something was misread - and critically, the check tells you so without you knowing the correct answer.

That is what makes arithmetic checks so valuable. Most validation asks “does this look plausible”. Reconciliation asks “is this internally consistent”, and consistency is checkable without ground truth.

The same idea generalises: quantities times unit price equals line total; delivered quantity does not exceed ordered quantity; date ranges do not run backwards.

A document that reconciles is not guaranteed correct, but a document that does not reconcile is guaranteed wrong somewhere.

3. Match against what you already hold

The layer that catches what format and arithmetic cannot.

Supplier exists in your vendor master, and the tax ID matches the one on file.
Purchase order exists, and the referenced line items are on it.
Bank details match the ones you hold for that supplier. This one deserves emphasis: a changed bank account on an otherwise perfect invoice is the classic invoice-fraud pattern, and no extraction check catches it. A mismatch here should never auto-approve, regardless of extraction confidence.
Not a duplicate - same supplier, same invoice number, or same amount and date.

Confidence routing

Extraction tools return per-field confidence. Use it to route rather than to fail.

Situation	Route to
All fields high confidence, all validations pass	Auto-post
One or two low-confidence fields	Review those fields only
Validation failure or reconciliation mismatch	Full human review
Bank details differ from file	Human review, always
Duplicate suspected	Human review, always

The middle row is where the value is. Presenting a reviewer with two uncertain fields and the source image takes seconds. Presenting them with the whole document takes minutes. The difference across a few thousand documents a month is most of the business case.

A caution: extraction confidence is calibrated on character recognition, not on correctness. A cleanly printed wrong number can come back highly confident. Confidence is a routing signal, not a correctness guarantee, and the validation layers are what actually protect you.

Confidence is one input to the route, not the route. The bottom rule has no confidence threshold on purpose: a changed bank account is the fraud pattern, and no score overrides it.

The metric to actually track

Not extraction accuracy. Review change rate: of the documents a human reviewed, what fraction did they change?

High rate means routing is working - you are sending the right documents for review.
Very low rate means you are reviewing too much. Loosen the thresholds and save the time.
Changes on auto-posted documents discovered later means thresholds are too loose, and this is the number that should alarm you.

Sample auto-posted documents deliberately - perhaps 2% - and have someone check them. That is how you find out whether the auto-post path is actually safe rather than assuming it from the absence of complaints.

Build the evaluation set for this the same way you would for any AI system: from real documents, stratified deliberately, including the ugly ones. See building an eval set from real tickets - the method transfers directly.

Where these pipelines actually fail

A new supplier with an unusual layout. Accuracy on the top ten suppliers says little about the long tail.

A format change nobody announced. A supplier’s system upgrade moves the invoice number, and extraction silently starts reading a different field. Validation catches it; nothing else does.

Scanned quality drift. A new scanner, a different setting, and documents that used to read cleanly stop doing so. Watch confidence distributions over time, not just averages.

Success with nothing extracted. The pipeline reports processed, zero fields populated, no error. The same failure mode described in why automations fail silently, and the same defence: assert on output volume.

What we insist on

Every document pipeline we build posts nothing without validation, and the human review queue is in scope from the first version rather than added when problems appear.

The reasoning: extraction errors are silent and cumulative. A wrong support answer is visible to a customer within minutes. A wrong invoice total enters your ledger, gets paid, and is discovered at reconciliation weeks later, by which time there are forty more like it. The asymmetry justifies a validation layer that occasionally routes something for review unnecessarily.

The rule we will not bend is the bank-details one. Changed payment details do not auto-approve at any confidence level, and we have had that pushed back on as over-cautious because it adds a manual step to otherwise clean invoices. It is the single cheapest control against a fraud pattern that costs real money, and one prevented incident pays for years of the friction.

The honest limitation of our own claims here: we publish a starting price for document processing based on one clean document type, and the number that actually matters for your project is the review change rate after tuning - which nobody can quote before seeing your documents.

When not to build this

Low volume. Under roughly 200 documents a month, a person with a keyboard is cheaper than a pipeline plus its maintenance.

When the supplier could just send data. If they can provide a structured feed or a portal export, take it. Extracting from a PDF a system generated from a database is a self-inflicted problem.

When every document is different. Extraction works on recurring structures. One-off bespoke documents are a poor fit.

When there is nobody to staff the review queue. A queue nobody works becomes a backlog, and a backlog becomes an argument for turning off the validation.

Frequently asked questions

What accuracy should we expect?

For clean, recurring document types after tuning, high per-field accuracy on the common fields and noticeably lower on line items and handwriting. Insist on per-field figures measured on your own document mix rather than a headline number.

Should we use a specialist extraction service or a general model?

Specialist services are often stronger on structured, high-volume document types and cheaper at scale. General models are more flexible on varied or unusual layouts. Many production pipelines use both, routed by document type.

How do we handle handwriting?

Expect materially lower accuracy and route those fields to review by default. Do not build a business case that assumes handwritten fields will be automated.

Can we skip validation if confidence is high?

No. Confidence measures recognition certainty, not correctness. A clearly printed wrong value returns high confidence.

What is the single most valuable check?

Arithmetic reconciliation on financial documents. It requires no ground truth and catches a large share of real errors.

Next step

If you have a pipeline posting data without a validation layer, that is the gap worth closing before increasing volume. The document processing engagement includes validation against your systems and a human review queue as part of the first version.