colin mitts

documents · text + vision

Order Form Extraction

input — scanned form

Degraded scan of a synthetic packing slip: skewed, blurred, and shadowed — order 4821 for John Smith, two line items

output — extracted records, verbatim

order4821
date06/12/2024 → ships 06/14/2024
customerJohn Smith
emailjohn.smith@example.com
ships to1420 Cedar Ave, Portland, OR 97214
itemGUMM-3 · Botanical Gummies — 3-Pack · 1 × $64.99
itemCHOC-1 · Dark Chocolate Bar — 1-Pack · 2 × $28.00
total$120.99 + $7.26 fee + $12.00 shipping = $140.25
sourcesample_order_form_scan.jpg · vision path · name re-verified against the image

real pipeline output. the form is the repo's synthetic sample, degraded to match the scans the system actually got — which forces the vision path, second-pass verification and all. run it yourself: app/extraction.py

What it does

A small member-based collective took orders on paper — hundreds of low-quality scans a week, retyped by hand. This system reads the scans automatically: hybrid text-and-vision extraction on Claude, 240+ OCR product-code variants standardized into a clean catalog, 5,800+ line items in a searchable database, and a dashboard for fulfillment and business insight. Customer data is anonymized non-reversibly before anything leaves the building.

How it works

scanned PDF ──▶ embedded text rich enough?
                 │                │
              yes ▼             no ▼
      claude: text        render page hi-DPI
      extraction          claude vision extraction
                 │                │
                 │        second-pass vision check
                 │        on name + address
                 │                │
                 └───────┬────────┘
                         ▼
        standardize codes · dedupe line items
                         ▼
          chromadb ──▶ dashboard · search · analytics

The fields most expensive to get wrong — customer name, shipping address — get a focused second look against the image before anything is written. Python · ChromaDB · Streamlit.

What broke + what I changed

The vision path's verification step would occasionally return JSON that didn't parse — fine in testing, fatal in a batch run. The fix wasn't a better prompt; it was switching to structured outputs so the model can't return anything but the schema.