ClearPath — AI Document Intelligence Platform

High-stakes documents like immigration notices or benefit letters are notoriously difficult to understand, especially for non-native speakers. A simple misunderstanding can lead to missed deadlines or legal peril. Standard LLM wrappers are too risky for this because they hallucinate facts and invent URLs. On top of accuracy, a multi-minute analysis pipeline must survive process crashes and network drops without losing state — and the user must be able to verify the extracted text before any LLM tokens are spent on it. Performance is also a feature: every stage that runs inline on the hot path is wall-clock time the user is waiting, and every model loaded in the hot-path worker is memory the result-producing stages have to fight for.

Zero tolerance for lost processing jobs if any process crashes mid-pipeline. Strict requirement for LLM accuracy/grounding — every trusted URL must come from a real Tavily search result. Protection against malicious prompt injections inside user-uploaded PDFs. Long-running multi-minute analyses that must stream real-time progress to the client and recover gracefully on reconnect. OCR is a Python service (Docling/RapidOCR); the Node worker and Python service must share one outbox without claiming each other's jobs. Hot-path performance and memory: chunking + embedding are necessary for retrieval quality but they were the slowest stages on the user-visible path, and loading the embedding model was a meaningful share of worker memory.

Full-Stack Engineer & AI Pipeline Architect — designed and implemented the monorepo, the 12-file database migration set, custom RS256 JWT auth, the Outbox Dispatcher, the three BullMQ queues (hot path, background, Python OCR), the Node worker orchestration, the Python/Docling OCR service, the SSE streaming layer, and the 5-stage AI pipeline.

ClearPath is an event-driven, decoupled, multi-runtime system with two parallel Node paths and one Python path. The Next.js frontend talks to the Express API for uploads, auth, confirm-extraction, history, and SSE. The API writes all state changes to Postgres in transactions that also INSERT into document_pipeline_outbox. The Outbox Dispatcher (separate process) subscribes to Postgres NOTIFY on outbox inserts and polls as a safety net, then enqueues the next stage's BullMQ job. Three queues exist: ANALYSIS_QUEUE_NAME (Node hot path: initialization → OCR → cleaning → structuring → completion), BACKGROUND_QUEUE_NAME (Node background path: chunking → embedding — runs in parallel, never blocks the user), and OCR_QUEUE_NAME (Python Docling). The Python OCR service writes its document.extracted outbox event via the Supabase client, so the same NOTIFY triggers the next Node stage regardless of which runtime wrote the row. Each worker stage persists its result and publishes progress to Redis Pub/Sub for live SSE clients while also writing a row to document_pipeline_events so reconnecting clients can replay from their Last-Event-ID. The API's SSE endpoint keeps one Redis subscriber per active client, replays any missed events from Postgres, and emits periodic heartbeats. Stage transitions are guarded by isStageCompleteOrPast() so BullMQ retries are idempotent. Idempotency on analyze requests is enforced both at the DB level (unique index on idempotency_key) and at the queue level (BullMQ job ID = analysis:<requestId>). The hot-path result ships the moment the completion stage fires, while the background path keeps running until chunks and embeddings land in Postgres for retrieval — net 30–35% faster time-to-result with the same output quality.