Multimodal models on invoices and delivery notes: at last
Ten years of per-vendor OCR rules replaced by a model that reads the document the way a person would.
Invoicing pipelines have always struggled with edge cases and unusual document formats that don't fit standard templates. Most normal documents flow through existing systems without problems. However, the genuinely odd formats including bad scans, handwritten amendments, split tables, and various stamps represent only fifteen percent of total volume yet they create seventy percent of support load and maintenance burden.
The legacy system was built entirely on hand-tuned optical character recognition rules customized per vendor type. Whenever a new vendor arrived, someone had to extract their format requirements into code, debug unexpected edge cases nobody anticipated, and accept that more variations would inevitably emerge requiring additional patches and maintenance work.
Multimodal models in 2024 handle those exceptions without requiring any template or vendor-specific code. The model reads handwritten dates positioned beside printed fields correctly, extracts split tables that span multiple pages into sensible structures, and detects when confused, flagging output for human review before processing.
We did not replace the entire pipeline because the bulk of documents are already straightforward. The predictable eighty-five percent still runs through classic optical character recognition because it's demonstrably cheaper, much faster, and completely deterministic without cloud API costs or latency concerns.
The final architecture consists of: deterministic optical character recognition on the normal path for routine documents, multimodal fallback for genuine exceptions, and validation against purchase orders to catch errors from either system before they escape into production business processes.
Eight vendors were onboarded without writing a single new template or hiring additional people to understand their format variations. That dramatic reduction in operational drag and maintenance burden justified the integration cost immediately and delivered clear business value.