Commit Graph
5 Commits
Author SHA1 Message Date
Marc-André Bombeck c47f61f02a feat(documents): read a PDF's text layer page by page without rendering
The reader child gains a `page-texts` request: the first pages' text
layers, each read the way a rendered page's is (cut and marked past the
per-page cap), plus the document's page count. Nothing is rendered, so it
works where the page renderer cannot run.

`readPdfTextLayer` judges what that read is worth: a page counts only with
at least 40 characters of its own besides white space, so a scanner's
stamped page number or fax header does not pass for a page of text; a PDF
with no such page is a scan and is reported as one.
2026-11-03 22:07:11 +01:00
Marc-André Bombeck 4d970561fa fix(documents): decode thumbnails in the reader child, share it fairly, judge exits by cause
Three defects in the bounded document reader.

Image uploads were decoded into their thumbnails in the server process. A
header check is all that stood between a decoder and an allocation the file
chooses, and the GIF check showed how that fails. The image is now decoded in
the same child process PDFs run in, under its memory ceiling and time limit;
the header check stays in front of it.

The reader's one slot went to whoever asked first, and a waiting item waited
without end, so a few heavy PDFs from one account held everyone's lab scans
behind them. The slot now goes to a read a person is waiting on before
background work, to accounts in turn, to a few waiting items per account, and
an item that waits past its lane's deadline is answered busy, which callers
treat as retryable.

Any child that ended without an answer counted as a document past its limits,
so a kernel kill caused by the container's overall memory marked a good PDF
as permanently failed. Only a bound the reader measured (the sampled memory,
the time limit, the child's heap cap) is final now; any other end is run once
more and then answered interrupted, which the thumbnail job retries within its
ordinary attempts.
2026-10-25 17:38:25 +01:00
Marc-André Bombeck 518ce2fd37 fix(documents): read and render PDFs in a bounded child process
pdf.js decodes an image at the size the file declares, whatever size the
page is drawn at, and the server opened documents without an image limit.
A small upload declaring a very large image made the web process allocate
gigabytes until the container was killed, for every user at once, and the
thumbnail job ran the same document again after each restart. An image
whose soft mask declares the size does the same past pdf.js' own limit, and
nothing bounded how long a text read could take.

pdf.js is now opened with maxImageSize (20 million pixels, which keeps one
image inside the reader's ceiling and admits a 300 dpi A3 scan), and every
read and render runs in a child process, one at a time: its V8 heap is
capped, its resident memory is sampled and the child killed past 640 MB, on
Linux it volunteers to the out-of-memory killer first, a time limit ends a
read that runs on, and it starts with an almost empty environment. The
loader and the rendering moved into pdf-engine.mjs, which the child runs
with plain Node and which is traced into the standalone image.

A document that crosses a bound answers resource-limit. The thumbnail job
records it as a final failure, so the backfill does not queue it again.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck 540c6bfaeb refactor(documents): read the text layer through the one shared loader
The text read and the rasterizer each opened pdf.js on their own: the text
read with bare options, the rasterizer through src/lib/documents/pdfjs.ts
with the standard font, CMap and wasm directories. The loader now serves
both, so a CID font with a predefined CMap is mapped to characters on the
text path too, and there is one set of options to keep in step.

The loader is typed with pdf.js' own declarations instead of a hand-written
shape. That drops isEvalSupported, which pdf.js 6 no longer has and which did
nothing. Only the render slice the DOM-typed declarations cannot describe
(the native canvas factory) keeps a named local type in the rasterizer.

The run joiner with its named steps moves into the loader as joinTextRuns,
and readPdfText is its one page walk; the second layout copy that mimicked
the previous library's tabs and blank lines is gone. extractPdfText keeps
its outcomes and page cap on top of it, and a failed read still says what
pdf.js threw (documents.local_extract.failed). A test draws a word in two
fonts into a real PDF and reads it back whole.

Refs #1277
2026-10-24 20:16:01 +02:00
Marc-André Bombeck 0b1757ff6f fix(documents): read the text layer through the same pdf.js that renders
Under Node, pdf.js runs its worker in the server process and keeps it on
a global, and whichever copy of pdf.js opens a document first owns that
global until a restart. The local text read went through pdf-parse, which
pins its own pdfjs-dist 5.4.296; the rasterizer uses the app's 6.4.299. A
text read first made every later rasterization fail with "The API version
does not match the Worker version" (thumbnails raster-failed, the vision
read of a scan 422), and a rasterization first made every later local text
read fail, which left the document out of the search index with nothing in
the logs.

src/lib/documents/pdfjs.ts is now the one place pdf.js is loaded and
opened, with one set of options (asset directories, no eval, no system
fonts). The text read walks the pages with getTextContent and lays the
text out the way pdf-parse did with the options it was called with: line
breaks, tabs between far-apart items, a blank line after each page, the
200-page cap. Text extracted from the PDF fixtures and from generated lab
tables, encrypted files and doctor reports is identical before and after.

A failed text read now emits documents.local_extract.failed with the
error pdf.js threw, and annotates the index event with it, instead of
returning a bare error.

The report tests read PDF text through the same reader. Two tests, one per
order, run a text read and a rasterization in one process; both failed
before this change.

Refs #1277
2026-10-24 19:32:13 +02:00