Shipping Document Extraction Pipeline

Extracts structured data from freight-forwarder PDFs (bills of lading, air waybills, tax invoices, etc.) using a local vision model, validates it, maps it into the format your ops system expects, and serves a review UI where clerks correct mistakes before handoff.

Runs entirely on this PC. No cloud, no API keys.

What it does

    PDF file                   Review UI (browser)            Ops system (desktop app)
       │                              ▲                               ▲
       ▼                              │                               │
  ┌───────────┐   extract    ┌───────────┐  clerk      ┌───────────────┐
  │  raw JSON │ ─────────▶   │ mapped JSON │ ────▶   reads JSON, fills fields
  │(our shape)│              │(target shape)│         
  └───────────┘              └─────────────┘          

The handoff to the desktop app is the mapped JSON file at out/mapped/<name>.<tag>.json. The desktop team reads that file and fills their own form fields using their own logic — we don't drive the desktop UI.

Pipeline stages

System requirements

Software (must be installed)

Optional / upgrade path

Hardware

Tested on: i9-13900KF / 64 GB RAM / RTX 4090 24 GB VRAM. Minimum workable: any CPU + 16 GB RAM + 16 GB VRAM for qwen3-vl:8b (or CPU-only Ollama — much slower).

Python packages

Install once:

pip install fastapi uvicorn jinja2 python-multipart pdfplumber pillow requests pytesseract jsonschema

One-time setup

1. Install Ollama, then pull the model: cmd ollama pull qwen3-vl:8b

2. Install Tesseract. Run the installer; accept defaults. Check it worked: cmd "C:\Program Files\Tesseract-OCR\tesseract.exe" --version

3. (Optional) Tesseract language packs for Chinese / Vietnamese etc. Download .traineddata files from https://github.com/tesseract-ocr/tessdata_fast and drop them into C:\Program Files\Tesseract-OCR\tessdata\.

4. Install Python packages (see list above).

5. (Optional) Firewall — if you want LAN access to the review UI from other PCs: cmd netsh advfirewall firewall add rule name="ExtractionReviewUI" dir=in action=allow protocol=TCP localport=8000

Daily use

Option A — Web UI (recommended)

python serve.py

Open http://localhost:8000 (or http://<this-pc-ip>:8000 from another LAN machine).

• Drop a PDF on the upload zone
• Wait 2-5 minutes while extraction runs (PDF visible on the left during processing)
• When done, the page switches to the review form with fields filled in
• Edit any wrong fields; click Save (or Ctrl+S)
• The mapped JSON at out/mapped/<name>.<tag>.json is updated on every save

Features in the review UI: - Split view: source PDF on the left, editable form on the right - Empty fields get an orange tint; format-invalid fields get a red tint with tooltip - Validator flags at the top are clickable — jump straight to the affected field - Add / delete rows on line-item tables (containers, invoice lines, dimensions) - Sticky Save button + Ctrl+S shortcut - Unsaved-changes guard on page unload - Home page filter box (type to narrow the file list) - "Only review_required" toggle to focus on docs that need attention

Option B — CLI only (batch / scripting)

REM Single PDF
python extract_cli.py "MBL OBD.pdf"

REM Every PDF in the working directory
python extract_cli.py --all

REM Separate override for a specific extraction
set EXTRACT_MODEL=qwen3.5:35b-a3b-q4_K_M
python extract_cli.py "EBS - Internal.pdf"
set EXTRACT_MODEL=

REM Validate existing JSON files
python validate_cli.py                      REM validates every out/*.json
python validate_cli.py "out/MBL OBD.json"   REM one file
python validate_cli.py "out/*BL*.json"      REM glob

REM Map raw JSONs into target-shaped records
python map_cli.py                           REM maps every out/*.json
python map_cli.py "out/MBL OBD.json"

File layout

c:\tmp\
  README.md              ← this file
  extract_cli.py         ← CLI: PDF → raw JSON (classify + extract)
  validators.py          ← Step-5 validators (ISO 6346, date, tax math, etc.)
  validate_cli.py        ← CLI: runs validators on out/*.json
  map_cli.py             ← CLI: raw JSON → mapped JSON (target shape)
  serve.py               ← FastAPI app: upload, processing, review UI
  schema.json            ← (optional) jsonschema for raw extractions

  prompts/               ← One prompt per document type, read by extract_cli
    classifier.txt
    sea_bill_of_lading.txt
    air_waybill.txt
    tax_invoice.txt
    commercial_invoice.txt
    packing_list.txt
    delivery_order.txt
    purchase_order.txt
    credit_note.txt
    debit_note.txt
    proforma_invoice.txt
    statement_of_accounts.txt
    unknown.txt

  mappers/               ← raw JSON → target-system record shape
    __init__.py
    common.py            (address splitter, party block, numbered-lines helpers)
    plvi.py              (tax_invoice / commercial_invoice → Vendor Invoice)
    sibl.py              (sea_bill_of_lading → Sea Import BL)
    aiaw.py              (air_waybill → Air Import AWB)

  push/                  ← Browser-automation demo (optional — not used in prod)
    push_cli.py
    selectors.py

  templates/             ← Jinja2 HTML templates for the web UI
    base.html
    index.html           (home: file list + upload zone + filter bar)
    plvi.html            (Vendor Invoice review form)
    sibl.html            (Sea Import review form)
    aiaw.html            (Air Import review form)
    processing.html      (upload progress page with PDF preview)
    raw.html             (view raw extraction JSON)
    demo_target.html     (fake target form — Playwright demo only)
    _macros.html         (shared template macros)

  static/
    app.js               (client-side: drag-drop, validation, animate-fill, tabs)
    style.css            (shared styles)

  out/                   ← Pipeline output — both stages
    <stem>.json          (raw extraction per PDF)
    mapped/
      <stem>.<tag>.json  (target-shaped record; tag = plvi|sibl|aiaw)

How each part works

extract_cli.py — classification + extraction

1. Renders every PDF page to PNG
2. Classify the document (fastest to slowest path):
3. Text rules: regex over the PDF's native text layer — runs in <1 ms when the PDF has text
4. OCR: runs Tesseract on the page image, then re-runs the text rules on the OCR output (~500-1000 ms)
5. Vision classifier: last resort — one Ollama call on page 1 (~15-30 s)
6. Extract: one Ollama call with ALL pages as image inputs and prompts/<doc_type>.txt as the system prompt. Output token budget scales with page count (8k base + 4k per extra page).
7. Writes out/<stem>.json with _meta.classified_via set to text / ocr+text / vision.

Classification rules are in TEXT_CLASSIFIER_RULES at the top of the file. Ordered — more specific first. Support English + Chinese (Simplified/Traditional) + Vietnamese. Extend by adding regex patterns per doc type.

Doc type hint: if the caller passes doc_type_hint="tax_invoice", classification is skipped entirely. The web upload form exposes this as a dropdown.

validators.py — Step 5 integrity checks

Runs after extraction. Walks the extracted JSON looking for integrity problems — bad container checksums, dates that don't parse, line-item totals that don't match the subtotal, etc. Emits:

• _meta.flags: list of short codes like "bad_container_number", "line_total_mismatch:5"
• _meta.validator_details: structured [{severity, code, msg}, ...] for the review UI
• _meta.extraction_confidence: high / medium / low (derived from flag severities)
• review_required: boolean — true if any error-severity flag fired

Known validator codes and the fields they map to in the review UI are in static/app.js : CODE_TO_SELECTORS.

map_cli.py + mappers/ — target-shape transformation

Each doc type dispatches to a mapper:

Writes out/mapped/<stem>.<tag>.json. The tag determines which HTML template the web UI uses.

Target field names match what the ops-system desktop app expects (e.g. BlNo, ContainerNo, ShipperAddress1..4). See mappers/*.py for the exact mappings.

serve.py — FastAPI web app

Binds to 0.0.0.0:8000. Routes:

All mutations are synchronous file writes — no database.

static/app.js — client-side behaviour

• Drag-and-drop upload on the home page
• Animate-fill: when arriving at /doc/<stem>?animate=1, clears all inputs, then re-fills them one by one with a green flash (so the clerk sees "the system just filled this")
• Per-field validation: flags empty fields with an orange tint, format-invalid fields (bad dates, bad container numbers, etc.) with a red tint + tooltip
• Validator-chip click-to-jump: clicking a flag at the top of the page scrolls to the affected field(s) and flashes them yellow
• Add / delete row on line-item tables — re-indexes and syncs a hidden __rowcount.PLVI2 marker so the server can truncate the array properly on save
• Unsaved-changes guard via beforeunload
• Ctrl+S saves

Configuration

All via environment variables — no config files:

Extending the system

Add a new language to classification

1. Install the Tesseract language pack (drop .traineddata into tessdata/).
2. Add regex patterns to the relevant entries in TEXT_CLASSIFIER_RULES (top of extract_cli.py).
3. No restart needed if running just extract_cli.py from CLI; restart serve.py for the web path.

Add a new document type

1. Create prompts/<new_type>.txt with a JSON schema + rules.
2. Add "<new_type>" to KNOWN_DOC_TYPES in extract_cli.py.
3. Add regex patterns for it in TEXT_CLASSIFIER_RULES.
4. Add a classification rule in prompts/classifier.txt.
5. If it needs a target-shape mapping, add a new file in mappers/, export it from mappers/__init__.py, and add a dispatch entry in map_cli.py : DISPATCH and serve.py : DISPATCH.
6. (For the UI) create templates/<newtag>.html using plvi.html as a template.

Change the target-system field names

Each mapper (mappers/plvi.py etc.) is the single source of truth for the target-shape record. Change field names there; re-run map_cli.py to regenerate all mapped files.

Use a different Ollama model

set EXTRACT_MODEL=qwen3.5:35b-a3b-q4_K_M
python serve.py

Or edit the DEFAULT_MODEL constant at the top of extract_cli.py.

Push to a real desktop/web target system

Two options:

• Desktop app (our current integration model): the desktop team reads out/mapped/<stem>.<tag>.json on-disk and fills their form with their own logic. They open the review UI (/doc/<stem>) from within the desktop app (WebView2 or external browser) so the clerk can review before the desktop reads the file.
• Web target system: edit push/selectors.py to map JSON paths to your form's CSS selectors, then run python push/push_cli.py <mapped-json> --url https://your-target/form. Uses Playwright to drive the browser.

Troubleshooting

Known limitations

• One doc type per PDF — the pipeline assumes a PDF is a single logical document (possibly multi-page). True mixed-content PDFs (BL + invoice + packing list in one file) are not auto-segmented; split them manually first.
• Fixed row count in UI minus the add/delete buttons — you can add and delete rows but the server doesn't re-classify the doc if the row structure fundamentally changes.
• No user accounts or access control — anyone on LAN with the URL can read and edit. Fine for a trusted internal network; add auth for anything else.
• Validator rules are English-centric — the regex rules for things like "Tax Invoice" labels are English/CJK/Vietnamese only. Some tax ID format checks (SG GST, AU ABN, etc.) are country-specific.
• Extraction is CPU/GPU-bound — 2-5 minutes per doc on an RTX 4090 with qwen3-vl:8b. Not suitable for high-volume automation without batching or a dedicated GPU server.

Contact points to hand over to the desktop team

When handing off to the team that'll integrate with the ops system:

1. URL shape: http://<this-pc>:8000/doc/<stem> — show this to clerks inside the desktop app (WebView2 or an external browser window).
2. Output file path: C:\tmp\out\mapped\<stem>.<tag>.json — the desktop app reads this after the clerk clicks Save.
3. Tag values: plvi (Vendor Invoice), sibl (Sea Import), aiaw (Air Import) — determines which screen to populate in the ops system.
4. Review signal: today, there's no explicit "reviewed" flag; the desktop app reads the file whenever it wants. If you need a "clerk finished reviewing" signal, add a button + _meta.reviewed_at field — see the backlog below.

Backlog / next steps

• "Mark as Reviewed" button that stamps _meta.reviewed_at in the mapped JSON so the desktop app knows when to pull.
• JSON API endpoints GET /api/doc/{stem} (returns mapped JSON) and GET /api/status/{stem} (returns review state) for programmatic handoff.
• True multi-document PDF segmentation — classify every page, group into sub-documents, emit one JSON per sub-doc. Option B scaffolding exists but was simplified for speed; re-enable with code from earlier commits if needed.
• Batch push to a real target system once it's known (web form via Playwright, or desktop app via pywinauto).
• Auth + audit log if the review UI grows beyond internal trust.