The text read and the rasterizer each opened pdf.js on their own: the text
read with bare options, the rasterizer through src/lib/documents/pdfjs.ts
with the standard font, CMap and wasm directories. The loader now serves
both, so a CID font with a predefined CMap is mapped to characters on the
text path too, and there is one set of options to keep in step.
The loader is typed with pdf.js' own declarations instead of a hand-written
shape. That drops isEvalSupported, which pdf.js 6 no longer has and which did
nothing. Only the render slice the DOM-typed declarations cannot describe
(the native canvas factory) keeps a named local type in the rasterizer.
The run joiner with its named steps moves into the loader as joinTextRuns,
and readPdfText is its one page walk; the second layout copy that mimicked
the previous library's tabs and blank lines is gone. extractPdfText keeps
its outcomes and page cap on top of it, and a failed read still says what
pdf.js threw (documents.local_extract.failed). A test draws a word in two
fonts into a real PDF and reads it back whole.
Refs #1277
pageText worked on raw text-matrix slots and bare numbers. Each step now
has a name: positionOf reads where a run starts and ends and its font
size, startsNewLine says when the baseline moved by more than half a font
size, and leavesWordGap says when the gap is wider than a tenth of the
font size. The output is the same on every test PDF.
extractPdfText cast the dynamic import to a hand-written shape
(`as unknown as`), so the compiler could not check the call against
pdfjs. It now uses pdfjs' own types: PDFDocumentLoadingTask, TextItem and
TextMarkedContent, all type-only imports that are erased at build.
That check found a dead option at once: pdfjs 6 has no isEvalSupported.
The option went away with the font-program eval path it used to switch
off, so it did nothing here; it is removed.
A structural guard fails when any runtime file under src/ imports
pdf-parse again, which would bring its second pdfjs back into the server
process. The next.config comment on the tracing includes now describes the
text path as it is, and the extractor no longer turns an empty
end-of-line run into a blank line of its own.
Local text extraction went through pdf-parse, which pins its own
pdfjs-dist (5.4.296) beside the app's (6.4.299), and both copies take
their worker from globalThis.pdfjsWorker. Whichever ran first in a process
broke the other: after a text extraction every raster failed with "The
API version "6.4.299" does not match the Worker version "5.4.296""
(thumbnails raster-failed, vision reads of a PDF 422 extractFailed), and
after a raster the text extraction failed and the local content index
stayed empty without a trace.
extractPdfText now reads the text layer with getTextContent on the
pdfjs-dist the rasterizer uses, so only one pdfjs runs in the server.
Pointing pdf-parse at 6.4 instead is not enough: its destroy() calls
PDFDocumentProxy.destroy(), which pdfjs 6 no longer has, so every
document would skip its teardown. Two tests run text and raster in one
process, each order in its own file.
The local content-extraction reader imported pdf-parse at module top level.
pdf-parse pulls in pdfjs, whose module evaluation references the browser-only
DOMMatrix global; on the server that reference throws during chunk
instantiation and takes down every route sharing the chunk — the document
list, upload, and search among them, so old and new documents alike fail to
load. Move the parser to a lazy import() inside the extraction function
(surface type kept as a type-only import). Verified under plain Node that the
lazy path still extracts a real PDF text layer, so local indexing is
unaffected.
Index a document for content search automatically the moment it is stored,
instead of requiring a manual index action. A fire-and-forget per-document
job runs an AI-first decision tree: when a vision-capable provider is
configured and consented, transcribe the stored original (rich, handles
scanned PDFs and images), budget-reserved exactly like the interactive index
route; otherwise fall back to server-side text-layer extraction via pdf-parse
(no provider, no egress, milliseconds) so content search works at a baseline
even with no AI. Budget exhaustion, a provider that cannot read the file
(a text-layer PDF on a non-Anthropic account), and provider errors all fall
through to the free local path rather than leave a document unsearchable.
Both paths write the same blind, encrypted content index (AES-256-GCM text +
opaque HMAC token tags) — nothing readable at rest. The local path adds the
'local-pdf' / 'local-ocr' source values (free String column, no migration).
Server-side OCR for scanned images and scanned PDFs is left as a documented
seam. Pin pdf-parse + pdfjs-dist into the standalone trace so the wasm text
extractor ships in the runtime image.
Marc-André Bombeck