Document OCR – Card Core (LISA AI)
Company project at AETHER NOVA AI: a document extraction service that reads Vietnamese personal, job, financial and travel documents for a visa assistant, combining OCR, rule-based validation and a Vision LLM fallback.
Overview
Document OCR (Card Core) is the shared extraction layer behind LISA AI, a visa assistant. Customers upload passports, ID cards, labour contracts, bank statements, tickets and other documents; the service turns them into structured fields that the visa workflow can use. A request passes a synchronous precheck, is queued on TaskIQ/Redis, and runs through a cascade: image/PDF normalization, OCR, a quality gate, document-specific rules, and a Vision LLM fallback only where rules fail. Every field carries its source, confidence, validation results and a bounding box so reviewers can verify it quickly. On top of the service I built the benchmark, regression tests and fine-tuning work used to choose and improve OCR engines.
Architecture Design
Async FastAPI service with a TaskIQ/Redis job queue and a pluggable cascade of OCR, rule and LLM providers
Key Features
Technical Implementation
Extraction Cascade
- Normalizes images and PDFs (EXIF rotation, RGB, PDF rendering), then extracts text through a pluggable provider: Google Vision, pdfplumber/PyMuPDF, domestic eKYC OCR, or local Tesseract/PaddleOCR
- An OCR quality gate rejects unreadable files and asks the customer to upload again
- Per-document card rules (regex, checksum, MRZ parsing, format checks) fill and validate fields
- A Vision LLM (pydantic-ai with a typed schema) is called only for required fields the rules could not fill, and only for document types that allow it
- Each document ends as auto, needs_review or rejected so staff only review what needs attention
Prompt Quality & LLM Policy
- Fixed 111 prompt bugs across the LLM agent and document extraction, re-evaluating output quality after each fix
- Per-document LLM policy decides which fields only the LLM may fill and which fields must never be sent to it
- Table reconstruction from OCR coordinates for bank statements and itineraries
OCR Benchmark & Fine-tuning
- Benchmark tool with three modes: accuracy (Exact Match, completeness, CER/WER), performance (latency) and concurrency
- Ground-truth images grouped by condition: clear, scanned, skewed, blurred and glare
- Compared local engines (PaddleOCR variants, PP-StructureV3, PaddleOCR-VL, Tesseract) with Google Vision and fixed the engine per document type
- Prepared Vietnamese datasets and fine-tuned PP-OCRv5 recognition, then tested and evaluated it against the base model
Production & Data Safety
- Precheck runs before a job is created: blank or monochrome pages, minimum size, page and batch budget
- Three submit modes: always async, sync with timeout, and dynamic (wait when idle, return a job id when busy)
- Files are kept in private storage and served only through an internal proxy endpoint; raw field values are never logged
- Regression CLI and a QA web app (test sets, test runs, bulk tests, bbox overlay) to catch quality drops before release
Screenshots & Previews
Technologies Used
System Pattern
Async FastAPI service with a TaskIQ/Redis job queue and a pluggable cascade of OCR, rule and LLM providers