Building an AI Invoice OCR Pipeline That Doesn't Fall Over
FastAPI, Celery, and Claude working together to turn messy scanned invoices into clean structured data.
We built an invoice OCR pipeline for a client drowning in 2,000+ scanned invoices a month. The demos looked great. Production was a different story. Here's the architecture that finally held up, and the traps we hit along the way.
The stack: FastAPI + Celery + Claude
FastAPI for the API layer, Celery with Redis for the async job queue, and Claude for the actual extraction. We tried open-source models first (Tesseract + layout parsers) — they hit ~60% accuracy on messy scans. Claude took us to 94% on the same set. The cost per invoice is higher, but the manual review savings more than cover it.
Why a queue is non-negotiable
OCR is slow (2-8 seconds per page) and bursty. If you process synchronously, a 50-invoice upload will time out the API and block your workers. Celery lets you accept the upload, enqueue jobs, and process in parallel. The frontend polls or gets a webhook when done.
The trap: confidence without grounding
The model would confidently extract wrong numbers from blurry regions. Our fix: a validation layer that checks extracted totals against line-item sums, flags mismatches, and routes low-confidence extractions to a human review queue. The AI does 94%, humans do the last 6% — and that 6% teaches the next round of prompts.
Structured output is the whole game
Don't ask the model for prose. Ask for a strict JSON schema (vendor, invoice number, date, line items with tax). Use a schema validator, retry on violation. This turns an LLM into a reliable data pipeline component instead of a chatbot that sometimes returns data.
Observability saves your job
Log every extraction: the scan, the prompt, the raw model output, the validated result, the human verdict. When accuracy drops (and it will, after a model update), these logs are how you debug. Without them, you're flying blind.
The pipeline now handles 3,000 invoices a month with one part-time reviewer. The lesson: applied AI is 20% model and 80% plumbing. Spend your time on the queue, the validation, and the observability — not on prompt-tuning theatre.
Want thinking like this on your project?
We don't just write about this stuff — we do it, every day, for brands like yours.
Start a project
