Invoice OCR Automation: Classic OCR, Vision LLMs, or Both?

Classic OCR: predictable, cheap, and brittle on layout

Engines like Tesseract recognise characters on a page image. On their own they do not know what an invoice is; you add the meaning with parsing rules tuned to each document layout: "the amount due is the number to the right of this label", "the account number matches this pattern".

Where it works well:

• A small number of stable layouts, such as bills from a few utility providers or one vendor's invoices.
• Clean, digitally generated PDFs. Scans need preparation; the Tesseract docs on improving quality recommend images of at least 300 DPI and cover deskewing, binarisation, and noise removal.
• Data that must not leave your infrastructure: it runs locally, with no per-page API cost.
• Deterministic behaviour: the same page gives the same output every time, which makes failures easy to reproduce.

Where it breaks:

• Many vendors with different layouts. Every new layout is new parsing rules.
• Layout changes. When a provider redesigns its bill, the rules quietly stop matching.
• Tables, multi-column pages, and line items that wrap across rows.

Vision LLMs and document models: layout understanding, new risks

Vision-capable models read the page the way a person skims it: they understand that a label and a number belong together even when the layout is new. Claude, for example, accepts PDFs directly and, per Anthropic's PDF support docs, can extract text and interpret charts and other visual content, processing each page as an image. Dedicated open document models are moving fast too; I wrote about one in Baidu's Unlimited OCR, which outputs structured Markdown with tables and reading order preserved.

Ask for structured output and you get JSON you can load directly. OpenAI's Structured Outputs guarantees the response follows the JSON Schema you supply, so required keys are present and enum values are valid.

The risks you have to design around:

• A valid shape is not a correct value. Schema enforcement means the total field exists and is a number. It does not mean it is the right number. A model can return a plausible amount that is not on the page.
• Cost scales with pages. Every page sent to an API costs tokens. Multi-page statements where you need one figure from page one are worth trimming before sending.
• Data handling. Invoices carry bank details, addresses, and account numbers. Decide up front which provider may see them, under which data retention terms, or whether a self-hosted model is required.
• Non-determinism. The same page can produce slightly different outputs across runs, so failures are harder to reproduce than with rules.

The architecture that holds up: extract, then verify

The pattern I recommend treats the model (or the OCR rules) as a fast first reader and plain code as the auditor:

1. Get clean text and layout. Native text from digital PDFs, OCR or a document model for scans.
2. Extract fields into a fixed schema. Rules for the few stable layouts you see most, an LLM with structured output for the long tail. Same schema either way: vendor, invoice number, dates, currency, subtotal, tax, total, line items.
3. Validate deterministically. Line items sum to the subtotal; subtotal plus tax equals the total; dates parse and the due date is after the issue date; the vendor exists in your vendor list; the invoice number plus vendor is not already in the system.
4. Route by confidence. Anything that passes every check goes straight through. Anything that fails a check, or comes from a new vendor, goes to a person who sees the original page next to the extracted fields.
5. Keep the source. Archive the original PDF and link it to the record so any figure can be traced back.

The validation step is what makes the LLM safe to use. A hallucinated total almost never also makes the line items add up, so arithmetic checks catch most extraction errors from either approach.

Measure accuracy on your own documents

Vendor benchmarks will not tell you how a pipeline performs on your bills. Before rollout, build a labelled sample: a set of real documents across your vendors, with the correct values typed in by someone who knows them. Run the pipeline and measure, per field:

• How often the value is correct and passes straight through.
• How often a wrong value is caught by validation and routed to review (annoying but safe).
• How often a wrong value passes validation (the number that matters most, and the one to drive down).

Keep the sample and rerun it whenever you change a prompt, a model, or a parsing rule.

Often the real bottleneck: getting the PDFs at all

Plenty of bills never arrive by email. They sit behind supplier and utility portals, one login and one download per account. That is an RPA problem before it is an extraction problem.

For one client I built two unattended Robomotion bots for PG&E and SFWater bills. Each bot reads its account list from an Excel configuration the client already maintained, logs in to the portal, walks every account, downloads the PDF bills, and runs Tesseract OCR with layout-aware parsing tuned to each provider's bill format. Extracted fields are type-checked and validated before they are inserted into one shared PostgreSQL schema, every source PDF is archived to Google Drive and DigitalOcean Spaces, and Discord notifications report every start, resume, and error, with the bot resuming where it stopped after an interruption.

Two providers with stable layouts was exactly the case where classic OCR plus rules is the right tool. With dozens of vendors and changing layouts, I would put an LLM extraction step behind the same validation and archival. The full write-up is in the invoice processing case study.

Which should you use?

• Few stable layouts, sensitive data: classic OCR and rules, run locally.
• Many vendors, varied layouts: an LLM or document model with structured output for extraction.
• Both, which is most real businesses: rules for the top vendors, a model for the rest, one schema, and the same validation and review queue behind both.

Frequently asked questions