Commit Graph
177 Commits
Author SHA1 Message Date
Marc-André Bombeck 253176d01a fix(documents): index every page of a PDF read by its text layer
"Read with AI" and the index backfill wrote a text-layer read as a complete
`local-pdf` index although it held only the first ten pages (the render
cap), while the local text index reads up to 200. `prepareVisionInput` takes
the number of pages a text-layer read may take, and both index writers ask
for the local index's own cap, so the index is complete.

The `partial` flag on a text read had no reader and is gone; the
extraction and the lab scan report their missing pages themselves.

Refs #1311
2026-11-03 23:24:49 +01:00
Marc-André Bombeck 763d6c600f fix(labs): bound each text-layer read of an unrenderable PDF
`planTextLayerReads` grouped whole pages into the reads the deadline
allows, with no bound on a read's size, while the text path refuses a part
past `OCR_TEXT_READ_MAX_CHARS`. A dense report in a request's three reads
went out as reads of 12 000 characters and more, which ends at the output
limit or loses rows.

`planTextLayerChunks` now cuts a page longer than one read at line
boundaries, keeps a page per read while the deadline allows that many, and
otherwise packs pages into reads of at most twice the surface's target
(8 000 characters for the lab scan, 5 000 for the document extraction).
What the reads cannot take is read from the start, and the pages left over
are reported with the pages without a text layer: `unreadPages` on the lab
scan, `extractionIncomplete` on a document.

Refs #1311
2026-11-03 23:24:38 +01:00
Marc-André Bombeck 119431dc15 fix(documents): fence document text in the summary and the filing suggestion
Both text modes spliced the text after a bare `OCR TEXT:` label, while the
extraction, the lab scan and the label read fence it. A PDF read by its
text layer on a server that cannot render now reaches these prompts, and
the automatic summary reads imported PDFs that way, so text the page does
not show (white on white) could write past its own end. The text is now
fenced with `fenceDocumentText`, the browser-OCR path included, and a marker
forged inside it is scrubbed.

Refs #1311
2026-11-03 23:24:26 +01:00
Marc-André Bombeck 144840bab9 test(documents): type the fixtures the text-layer cases added instead of casting them
The stored bytes are already a Buffer and the MIME literal already fits
its union, so neither needs a cast; the external pick goes through one
helper. The as-never ratchet tightens accordingly.
2026-11-03 22:36:18 +01:00
Marc-André Bombeck b1cb387062 Merge branch 'fix/v147-pdf-unrenderable' into release/v1.47.0
# Conflicts:
#	.env.production.example
2026-11-03 22:10:14 +01:00
Marc-André Bombeck df6d20b696 fix(documents): read a PDF the server cannot render by its text layer
On a host whose CPU cannot run the page renderer (x64 without AVX2, arm64
musl without LSE/RCpc), every AI read of a PDF for a provider that only
takes images failed with "Scanning PDFs needs a Claude vision provider",
which says nothing about the cause. Most medical PDFs are produced
digitally and carry every word in their text layer, which pdf.js reads
without rendering.

`prepareVisionInput` now answers a text read (`mode: "text"`) when the
renderer is unavailable on this host, the provider cannot take the PDF
whole and the PDF has a usable text layer. Summary, suggest and the
background summary send that text through their existing text mode;
"Read with AI" and the index backfill write it as the local text index it
is, with no provider call; the automatic index leaves it to its local path.
`planVisionReads` does the same for the extraction and the lab scan: text
reads by page group, with anchored provenance, and the pages without a text
layer reported as `extractionIncomplete` on a document and as `unreadPages`
on a lab scan. A supplement label PDF is read from its text layer too.

A scan on such a host now answers `documents.inbound.pdfRendererUnavailable`
and `labs.ocr.pdfRendererUnavailable` (422): this server's processor cannot
run the page renderer, so a scanned PDF can only be read by a provider that
takes PDF files directly, or as photos. The `pdfNeedsAnthropic` codes are no
longer emitted, since with a working renderer every provider reads a PDF;
the recorded index outcome keeps its `pdf-needs-anthropic` value with
corrected copy.

The extraction path no longer folds a render failure of one document into
the capability error: like `prepareVisionInput` since #776 it reports
"The PDF pages couldn't be rendered for scanning." (`extractFailed`), and
the lab scan answers `labs.ocr.extractFailed`.

Hosts that render, and providers that take PDFs whole, are unchanged.

Refs #1311
2026-11-03 22:07:22 +01:00
Marc-André Bombeck c47f61f02a feat(documents): read a PDF's text layer page by page without rendering
The reader child gains a `page-texts` request: the first pages' text
layers, each read the way a rendered page's is (cut and marked past the
per-page cap), plus the document's page count. Nothing is rendered, so it
works where the page renderer cannot run.

`readPdfTextLayer` judges what that read is worth: a page counts only with
at least 40 characters of its own besides white space, so a scanner's
stamped page number or fax header does not pass for a page of text; a PDF
with no such page is a scan and is reported as one.
2026-11-03 22:07:11 +01:00
Marc-André Bombeck 5d36afb487 build: run the image on a Wolfi runner
The dependency and build stages stay on node:22-trixie-slim, where pnpm
picks the glibc native packages. The runner moves to wolfi-base, pinned
by its index digest: glibc, about fifty packages, and every one of them
known to the image scan, Node and the PostgreSQL client included.

nodejs-22, tzdata and postgresql-18-client come from apk, so the pgclient
stage and its pin guard go. The builder writes its Node module ABI and
glibc version out and the runner refuses to build on another ABI or an
older glibc. Wolfi's npm is major 12 and runs no install scripts unless
package.json allows them, so the Prisma CLI install names the three
packages that have one, and the build checks that the schema engine file
is there before asking the CLI about it. npm is installed for that step
and removed in the same layer; the check that nothing is left asks a
fresh shell, whose command cache cannot remember the deleted binary.
2026-11-03 21:55:30 +01:00
Marc-André Bombeck 3d0ae05192 docs: say the published image renders documents on a Raspberry Pi 4
The image runs the glibc canvas build now, which needs no ARMv8.2
features, so the Pi 4 gets thumbnails and scanned-PDF rendering back
without a code change: the gate only ever closed for musl on arm64. The
gate's comment, the env example and the getting-started prerequisites
say so, and keep NATIVE_CANVAS as the override. The x64 AVX2 gate stays:
the glibc x64 build rendered a PDF page under an emulated core without
AVX2, but it still carries AVX2 code and that one run does not cover
every path.
2026-11-03 20:53:28 +01:00
Marc-André Bombeck a822aa5a0e fix(documents): send a read that states no unit itself to review
A unit of "1" is the explicit no-unit choice a person makes. When a
document read returns it on its own, the value is staged for review
instead of passing straight through, so a unit the read dropped never
lands as a dimensionless value.
2026-11-02 20:31:33 +01:00
Marc-André Bombeck 79ac5670c7 feat(labs): offer an explicit no-unit choice for ratios and indices
A ratio (cholesterol/HDL, albumin/globulin) or an index such as INR has
no unit, and every lab write path refuses a numeric value without one.
That refusal stays: an empty unit usually means the extraction dropped
it. Instead the value is stored as the UCUM unity "1", chosen on purpose.

The unit field when a marker is defined, on a scanned report's row and
in the document review edit now carries a "No unit" checkbox that sets
"1". The CSV import takes 1 in the unit column, as the template and the
import sheet say; an empty cell is still refused, and other spellings
such as "ratio" or "-" stay the literal unit they always were.

"1" compares as a unit of its own: it matches only itself, so it is
refused under a marker kept in mg/dL and the other way round.

Refs #1313
2026-11-02 20:28:23 +01:00
Marc-André Bombeck 773b943bde fix(documents): keep PDF reading alive on CPUs the canvas build skips
The arm64 musl build of the native canvas uses LSE atomics and RCpc
loads throughout, so loading it on an ARMv8.0 core such as the
Raspberry Pi 4 ends the process with SIGILL. The CPU gate only knew
the x64 AVX2 case, and pdf.js loads the canvas as soon as it starts,
so on such a host every PDF read died in the child: no text layer, no
index, and the manual read reported a busy reader.

The gate now also closes on linux/arm64 under musl when the CPU lacks
the atomics or lrcpc feature. A closed gate travels to the PDF child,
which refuses to resolve the canvas module, so pdf.js carries on
without it and the text layer reads; rendering and thumbnails keep
their existing fail-soft path. A child that ends without an answer
now logs its exit code and signal.

Refs #1311
2026-11-01 01:24:37 +01:00
Marc-André Bombeck d88313e929 fix(labs): move document lab values filed under their confirmation day
Before the confirm step dated an undated value by its document, a lab
value confirmed from a report that prints its collection date only in the
header was stored with takenAt set to the moment of confirmation. The fix
covers new confirms; the values already filed that way stay on the wrong
day until something moves them.

A one-shot maintenance job, sent at boot only while there is something to
repair, moves exactly those values to the document's report date (else its
filing date) at noon UTC, the same rule the confirm step now uses. A value
qualifies only when the chain is certain: written by the document confirm
path, named by exactly one approved observation fact, takenAt still within
five seconds of its own createdAt, no calendar date of its own in the
decrypted fact, and a document day that differs from the day the value is
filed under in the owner's zone. A value re-dated since, one really taken
on its confirmation day, or one whose document has no date is left alone.
A moved value no longer sits at its creation moment, so the pass converges
and a second run changes nothing.

The job definition lives in a new registry file beside maintenance.ts so
that file stays under the size ratchet.
2026-10-29 08:49:40 +01:00
haritowa 58141cb48b fix(documents): date an undated lab value by its document
A lab report usually prints the collection date once, in its header. The
extraction puts that date on the document (reportDate) and leaves each
value's own date empty, and the confirm step filed every such value under
the moment it was confirmed, so an old report showed up as today's
results.

The confirm route now reads the document's report date and filing date
with the document it already loads, and an undated value takes the report
date, else the filing date, else now: the order the day view places a
document by.

A stated date, a lab value's or a condition's onset, now counts only when
it names a real calendar day. 2021-02-30 used to be stored as 2 March, and
2021-13-01 became an invalid date that failed the whole confirm.
2026-10-26 01:08:43 +00:00
Marc-André Bombeck f63d5d5819 fix(ai-runs): do not count time in the document reader against a run's deadline
A background AI run's deadline is armed in model time plus a fixed margin.
Time spent waiting in the document reader's queue, and the read itself, came
out of that margin, so a person's lab scan or document read could be timed
out by the reaper because other work held the reader. When the reader grants
the slot it now moves the run's deadline by the wait plus the most the read
may take, and gives back the unused part when the read ends, never past the
job's own expiry and only while the run is running. The interactive queue
wait stays within the margin, which covers the run until the slot is
granted.
2026-10-25 18:02:45 +01:00
Marc-André Bombeck d3257a1b6b fix(documents): tell people when the document reader is busy, and attribute its work
A read a person started now runs as interactive work of their account, and
the document jobs as background work of theirs, so the reader's queue can
order and cap them. When the reader is busy or stopped before it answered,
a lab scan, label read or document read fails with aiRuns.readerBusy (503,
nothing read or charged) instead of a misleading unreadable-file error, and
the index and summary jobs throw so the queue runs them again rather than
recording an outcome about the document.
2026-10-25 17:38:25 +01:00
Marc-André Bombeck 4d970561fa fix(documents): decode thumbnails in the reader child, share it fairly, judge exits by cause
Three defects in the bounded document reader.

Image uploads were decoded into their thumbnails in the server process. A
header check is all that stood between a decoder and an allocation the file
chooses, and the GIF check showed how that fails. The image is now decoded in
the same child process PDFs run in, under its memory ceiling and time limit;
the header check stays in front of it.

The reader's one slot went to whoever asked first, and a waiting item waited
without end, so a few heavy PDFs from one account held everyone's lab scans
behind them. The slot now goes to a read a person is waiting on before
background work, to accounts in turn, to a few waiting items per account, and
an item that waits past its lane's deadline is answered busy, which callers
treat as retryable.

Any child that ended without an answer counted as a document past its limits,
so a kernel kill caused by the container's overall memory marked a good PDF
as permanently failed. Only a bound the reader measured (the sampled memory,
the time limit, the child's heap cap) is final now; any other end is run once
more and then answered interrupted, which the thumbnail job retries within its
ordinary attempts.
2026-10-25 17:38:25 +01:00
Marc-André Bombeck 933be858c5 fix(images): size a GIF by its largest frame, not only its logical screen
The header check before a thumbnail decode read only the GIF logical screen.
A decoder sizes its buffer from the frame's image descriptor, so a file with a
small screen and a huge frame passed the pixel cap and allocated the full
frame. Walk every block, read each image descriptor, size the image by the
largest of screen and frames, and refuse a file that ends inside a header or
descriptor, carries an unknown block, or has no frame.
2026-10-25 16:53:31 +01:00
Marc-André Bombeck 6b9b9a3f8d test(documents): type the index fixtures instead of casting them away
The resource-limit cases and two older mocks now pass typed values, so the
file carries fewer as-never casts than before; the outcome list's comment
is a line shorter to keep its file at its length. Both ratchet baselines
are written down, the source size one also for offhost-backup.ts, which
had shrunk without its number following.
2026-10-25 16:24:15 +01:00
Marc-André Bombeck 8471e0edc9 fix(documents): say a PDF past the reader's limits is too large to read
A text read that crossed the isolated reader's memory or time bound came
back as a plain read error, which the index job records as decrypt-error,
so the document said its stored file could not be opened. The local
extractor now answers resource-limit for PdfResourceLimitError, the index
records it under that name, and the detail view says the PDF is too large
or complex to read here, in every locale. Like the thumbnail's refusal of
the same bytes it is final: nothing retries it on its own.
2026-10-25 16:00:35 +01:00
Marc-André Bombeck ba22932101 fix(documents): read a JPEG's size past fill bytes and refuse an unreadable one
The thumbnail's header walk took the byte after any 0xFF as a marker. A
single 0xFF fill byte after the start-of-image marker, which the JPEG
standard allows and every decoder skips, threw it off; it read no size, and
an unread size let the image through to a decode that allocates the full
bitmap. A small JPEG declaring 20000 x 20000 pixels allocated 1.6 GB in the
web process before the post-decode check could see it.

The avatar upload already had a walker that skips fill bytes. Both now use
one reader (lib/image-dimensions) that follows the marker rules: fill bytes,
the length-less markers, and no size when a scan or the end of the image
comes before a frame header. The thumbnail treats an unreadable size as a
refusal (size-unreadable) for every format it decodes, and the backfill
counts that refusal as final.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck 518ce2fd37 fix(documents): read and render PDFs in a bounded child process
pdf.js decodes an image at the size the file declares, whatever size the
page is drawn at, and the server opened documents without an image limit.
A small upload declaring a very large image made the web process allocate
gigabytes until the container was killed, for every user at once, and the
thumbnail job ran the same document again after each restart. An image
whose soft mask declares the size does the same past pdf.js' own limit, and
nothing bounded how long a text read could take.

pdf.js is now opened with maxImageSize (20 million pixels, which keeps one
image inside the reader's ceiling and admits a 300 dpi A3 scan), and every
read and render runs in a child process, one at a time: its V8 heap is
capped, its resident memory is sampled and the child killed past 640 MB, on
Linux it volunteers to the out-of-memory killer first, a time limit ends a
read that runs on, and it starts with an almost empty environment. The
loader and the rendering moved into pdf-engine.mjs, which the child runs
with plain Node and which is traced into the standalone image.

A document that crosses a bound answers resource-limit. The thumbnail job
records it as a final failure, so the backfill does not queue it again.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck 8fee138256 fix(backup): type the alias order as Prisma's mutable order list 2026-10-25 03:35:14 +01:00
Marc-André Bombeck 84064ea045 Merge branch 'fix/bound-sync-lab-reads' into fix/review-meds-docs 2026-10-25 03:02:57 +01:00
Marc-André Bombeck 3853ce7f1a fix(backup): order the source aliases a deleted document's keys are read from
The backup export is totally ordered so two exports of the same account are
byte-identical; the alias relation the import-key ledger reads was the one
nested read without an order.
2026-10-25 03:02:56 +01:00
Marc-André Bombeck fd299cad61 fix(labs): plan every report read against a schedule
The lab scan runs only in the background worker, which hands it the
run's deadline, but its body still took the schedule as optional and
planned a missing one without a bound: a twenty-page report as forty
model calls. Nothing calls it that way today; the signature allowed the
next caller to, and so did every read planner beneath it.

The scan and the planners (maxReadsFor, planTextReads, planVisionReads,
planInboundTextReads, planInboundReads) now require a schedule. A run
with room for every read says so with an unbounded one; a request uses
SYNCHRONOUS_READ_SCHEDULE. No route, wire or response changes.
2026-10-25 02:58:35 +01:00
Marc-André Bombeck 474fc08649 fix(documents): bound the lab staging an index request continues into
The index route without Prefer: respond-async, which the iPhone app
calls, continues into the lab staging with no run schedule. The staging
then planned a long report's text as up to twenty reads with a retry
each: forty model calls inside one HTTP request, the same class the
extract route was bounded against.

The staging now plans a missing schedule as one request does
(SYNCHRONOUS_READ_SCHEDULE, at most three reads): a longer report has
its text shared out between them, and what does not fit is left out and
reported as extractionIncomplete on the document. The request bound
moves into ocr-plan beside the schedule it bounds, so the extract run
and the staging share one definition. Background runs and the automatic
index job keep planning against their own deadline.
2026-10-25 02:52:10 +01:00
Marc-André Bombeck d6c2760415 Merge remote-tracking branch 'origin/release/v1.46.0' into fix/review-meds-docs
# Conflicts:
#	src/__tests__/baselines/source-function-lines.json
#	src/__tests__/baselines/version-stamp-comments.json
#	src/__tests__/sharing-surface-guard.test.ts
2026-10-25 02:45:23 +01:00
Marc-André Bombeck 205d8f257c fix(backup): carry the import ledger, and the keys of deleted imports
The backup carried neither the import ledger nor the tombstones of
deleted documents, so after a restore onto a fresh instance a token
importer stored every imported document the person had deleted on
purpose again.

The document filing section now carries documentImportKeys: the
ledger's rows plus the own and alias keys of the documents waiting in
Recently deleted, which still do not travel as documents. The restore
writes them back, minus any key a restored live document holds, and
counts them in the restore summary and the preview. DocumentImportKey
moves from the excluded models to the two-ended ones, and the round
trip, the column audit and the parity goldens carry a ledger row.
2026-10-25 02:42:09 +01:00
Marc-André Bombeck da4056ce05 fix(documents): key an undated fact on its report's date in the merge
The extraction merged facts across reads on a key whose only date was
the fact's own effective date. Facts without one collapsed across reads
of different reports, so the same value on two days staged once. An
undated fact is now keyed on its read's report date, falling back to
the document's, as the lab scan keys a row.

Refs #1257
2026-10-25 02:03:27 +01:00
Marc-André Bombeck c9481b672c fix(documents): bound an extraction that runs inside the request
Without Prefer: respond-async the extract route ran the read with no
schedule, so a twenty-page report read a page at a time, with a retry
each, could make forty model calls in one HTTP request. The iPhone app
sends no Prefer header and reads the document detail from the answer,
so the request keeps its 200 and its body.

An extraction without a run deadline now plans at most three reads (six
calls): a longer document has its pages or text shared out between
them, and what does not fit is reported as extractionIncomplete on the
document. The background run keeps reading it a page at a time.

Refs #1257
2026-10-25 01:01:13 +02:00
Marc-André Bombeck 958d230b9f fix(documents): keep an incomplete extraction on the document
A long document read in parts can end with a part cut at the provider's
output limit, a part never read, or more facts than one review takes.
That was told once, as a notice when the run finished; a review opened
later looked complete.

InboundDocument.extractionIncomplete (migration 0483) is written with
the facts by both stagings, the extract route or run and lab
auto-staging. The document detail publishes it and the review shows the
same note whenever it is opened. The disaster-recovery backup carries
it; an older file restores it as false.

Refs #1257
2026-10-25 01:01:13 +02:00
Marc-André Bombeck b0ea3b1654 fix(ai): charge what failed hops spent when a later hop answers
A chain whose first provider reasoned without answering and whose second
one answered billed only the answer: result.tokensUsed is the answering
call's alone, and the failed hop's spend, already reported on its
FallbackHop, was dropped by every caller that reconciles the day's ledger.

The raw runners now report tokensSpent on success too, the sum the failed
hops before the answer reported. The briefing, status, Coach (no-tools path
and every tool-loop round) and medication-extract reconciles charge it on
top of the answer. On failure, the Coach turn ledger's close keeps what the
failed chain reported spending instead of refunding everything, and the
medication-extract and document-chat routes charge it too, as the status
and briefing paths already did. Document indexing resolves a single provider
and already charged its failed call.

Refs #1279
2026-10-25 00:33:18 +02:00
Marc-André Bombeck 3f17b40092 Merge remote-tracking branch 'origin/release/v1.46.0' into fix/answerless-retry-bound
# Conflicts:
#	src/__tests__/baselines/source-file-lines.json
#	src/__tests__/baselines/source-function-lines.json
2026-10-24 21:31:24 +02:00
Marc-André Bombeck 94658e9af2 Merge remote-tracking branch 'origin/release/v1.46.0' into feat/documents-recently-deleted 2026-10-24 20:57:49 +02:00
Marc-André Bombeck 52fa5a06b1 feat(documents): let the picker bring back a deleted document
The document picker showed a document deleted in HealthLog as "Deleted
in HealthLog" and would not let it be ticked, so a person who deleted an
imported document by mistake met a dead end exactly where they would look
for it again.

Such a row can now be ticked, and the import sends revive: true for it.
A document still in Recently deleted is restored through the shared
restore, nothing is fetched, and the row reads "Restored from Recently
deleted". One the purge already removed is fetched and stored again, and
only once it is stored does the ledger forget its key, so a failed fetch
leaves the deletion standing. Without revive the route answers "deleted"
as before, and the token upload has no such switch: a Paperless workflow
or the import script re-sending a deleted document is still refused,
whether it is in Recently deleted or long purged.

The search names the document a deleted key still points at (documentId,
null once purged), so the row can say "In Recently deleted" rather than
"Deleted in HealthLog". The outcome gains "restored", sent only in answer
to revive. The row logic moves to picker-rows.ts.

Refs #1275
2026-10-24 20:55:19 +02:00
Marc-André Bombeck 4752115d7d feat(documents): add Recently deleted for deleted documents
Deleting a document only stamps a tombstone and the purge removes it
after 30 days, but the only way back was the undo toast, gone a few
seconds after the delete. Recently deleted lists the deleted documents
of the record with type, filing date, where an import came from, when
each was deleted and how long it has left, and offers Restore, Restore
all and Delete now.

GET /api/documents/inbound/deleted lists them (owner-only, capped at 200
with the list-cap meta). POST /api/documents/inbound/{id}/purge removes
one deleted document for good through the same function the nightly
purge runs, so its import keys and aliases still move into the ledger
and an importer re-sending it is still answered "deleted"; a live
document answers 409 documents.inbound.notDeleted. The usage read gains
deletedCount, 0 under a grant. The restore route and bulk restore
share one restore, which now also clears ledger entries for the
document's own keys.

The vault header shows a quiet "Recently deleted (n)" link only while
something is in it, and the delete toasts and confirmations say that
deleted documents stay there for 30 days. The purge's queue facts and
grace window move to document-purge-queue.ts so the worker registry
still loads none of the work.

Refs #1275
2026-10-24 20:51:35 +02:00
Marc-André Bombeck ba4774e48a fix(ai): keep background work off a benched provider and charge failed replies
The health ledger already counted every failed call and paused a provider
that kept failing, with a pause that grows. A paused provider only moved to
the tail of the chain, though, and the tail is still walked: a chain whose
only entry answered every call with reasoning and no answer was called by
every background job regardless.

A walk for background work (surface "job") now leaves out every provider
the ledger has benched until its window lifts; the local floor only once
it is paused, past a brief backoff. One ai.chain.background_paused line
per window says so. A person's own call still reaches the provider at the
tail and reports its own error. The ledger is in Postgres, so the bench
holds across workers and restarts.

The status, briefing and document-index paths refunded a failed call in
full, so background spend on replies without an answer never reached the
day's token ceiling. A failed chain now carries the tokens its hops
reported (tokensSpent) and those paths charge them.

The two clients share one answerless step (askAgainUnreasoned), which
keeps both inside the source size ratchet; its baselines go down.

Refs #1279
2026-10-24 20:42:42 +02:00
Marc-André Bombeck 1fd34e4d74 test(documents): keep one pair of one-process order tests
Two pairs of tests ran a text read and a rasterization in one process,
one file per order. The pdfjs-one-version pair stays; the other pair and
its shared helper go. The kept pair now first asserts that nothing in the
process has claimed the pdf.js worker yet, so a file that lands in a shared
process fails instead of passing on an order it never ran, and it mocks the
failure signal the text read can emit.

Refs #1277
2026-10-24 20:16:02 +02:00
Marc-André Bombeck 540c6bfaeb refactor(documents): read the text layer through the one shared loader
The text read and the rasterizer each opened pdf.js on their own: the text
read with bare options, the rasterizer through src/lib/documents/pdfjs.ts
with the standard font, CMap and wasm directories. The loader now serves
both, so a CID font with a predefined CMap is mapped to characters on the
text path too, and there is one set of options to keep in step.

The loader is typed with pdf.js' own declarations instead of a hand-written
shape. That drops isEvalSupported, which pdf.js 6 no longer has and which did
nothing. Only the render slice the DOM-typed declarations cannot describe
(the native canvas factory) keeps a named local type in the rasterizer.

The run joiner with its named steps moves into the loader as joinTextRuns,
and readPdfText is its one page walk; the second layout copy that mimicked
the previous library's tabs and blank lines is gone. extractPdfText keeps
its outcomes and page cap on top of it, and a failed read still says what
pdf.js threw (documents.local_extract.failed). A test draws a word in two
fonts into a real PDF and reads it back whole.

Refs #1277
2026-10-24 20:16:01 +02:00
Marc-André Bombeck d3621617c6 Merge pull request #1278
Read a PDF's text with the app's own pdfjs (#1277).
2026-10-24 20:10:04 +02:00
Marc-André Bombeck 0b1757ff6f fix(documents): read the text layer through the same pdf.js that renders
Under Node, pdf.js runs its worker in the server process and keeps it on
a global, and whichever copy of pdf.js opens a document first owns that
global until a restart. The local text read went through pdf-parse, which
pins its own pdfjs-dist 5.4.296; the rasterizer uses the app's 6.4.299. A
text read first made every later rasterization fail with "The API version
does not match the Worker version" (thumbnails raster-failed, the vision
read of a scan 422), and a rasterization first made every later local text
read fail, which left the document out of the search index with nothing in
the logs.

src/lib/documents/pdfjs.ts is now the one place pdf.js is loaded and
opened, with one set of options (asset directories, no eval, no system
fonts). The text read walks the pages with getTextContent and lays the
text out the way pdf-parse did with the options it was called with: line
breaks, tabs between far-apart items, a blank line after each page, the
200-page cap. Text extracted from the PDF fixtures and from generated lab
tables, encrypted files and doctor reports is identical before and after.

A failed text read now emits documents.local_extract.failed with the
error pdf.js threw, and annotates the index event with it, instead of
returning a bare error.

The report tests read PDF text through the same reader. Two tests, one per
order, run a text read and a rasterization in one process; both failed
before this change.

Refs #1277
2026-10-24 19:32:13 +02:00
Marc-André Bombeck a1ee2ac003 fix(documents): keep document reads inside their job's expiry
Since the lab staging reads a long report in parts, one auto-index could
make the transcription plus up to twenty reads of two calls each at the
person's response timeout, hours past pg-boss's fifteen-minute default,
which failed the job and retried it beside the reads still running. The
staging now takes the run's deadline as a schedule and plans only the
reads that fit before it. The index job is long, with an expiry above the
calls that run before the plan can say no, a stop on jobDeadline and a
lock per document; the document run hands the same staging its own
deadline, which moves the run's expiry as the reads are planned. The
document summary, one call and its reasoning repeat, gets an expiry above
that worst case.

ocrReadCallMs in the read core is the one per-call timeout both deadlines
divide by. The expiry guard now names every importer of the document and
lab read entry points with the queues that reach it.

The runtime guard's planted cases stripped every file's comments again on
each rescan, which took over a second alone and timed out under a full
parallel run; the stripped text is now memoised.

Refs #1274
2026-10-24 11:28:57 +02:00
Marc-André Bombeck 5a7bd95c7b fix(documents): read a long document through the lab scan's read core
Extracting lab values from a stored document still sent the whole text
in one call. A multi-page report produced more JSON than one call's
output ceiling, the answer was cut mid-fact, the parse failed twice, and
the read came back as 422 "Couldn't read the document. Try a clearer
copy." The lab scan had already learned to read the same report a page
or a part at a time, but each surface carried its own copy of the read,
so the document path never got it. It also ran under the shared 60 s
timeout, dropped every fact past 120 (or every fact when one did not
match the schema), and charged a flat reservation.

The read now lives once, in lib/documents/ocr-read.ts and ocr-plan.ts:
the per-read limits, the corrective retry, the salvage of a cut
answer's complete rows, the wire re-check and budget reservation before
each read, and the planning of a PDF a page at a time and text in parts
at line boundaries. The lab scan and the document extraction (the
extract route, its background run and the lab auto-staging) both read
through it, and the background run plans its reads on the same deadline
as a lab scan. The per-read budget moves to a shared metered budget.

The document path validates the answer fact by fact, keeps the lab
scan's row cap (400), and says why nothing came back: 422
documents.inbound.outputLimit when the reads stopped at the output
limit, 429 when the budget refused them, extractFailed only for answers
that were not the schema. A run whose reads were cut or not reached
reports `incomplete` on its result.

A guard holds the class: the output ceiling is set only in the core,
both surfaces read through runOcrReads, both runs share the deadline,
and a lab read and a document read go out under the same limits.

Refs #1257
2026-10-24 10:34:31 +02:00
haritowa ce6c752864 refactor(documents): name the steps of the page text joiner
pageText worked on raw text-matrix slots and bare numbers. Each step now
has a name: positionOf reads where a run starts and ends and its font
size, startsNewLine says when the baseline moved by more than half a font
size, and leavesWordGap says when the gap is wider than a tenth of the
font size. The output is the same on every test PDF.
2026-10-23 18:37:23 +00:00
haritowa 2c5e9a80b9 refactor(documents): type the text read with pdfjs' own declarations
extractPdfText cast the dynamic import to a hand-written shape
(`as unknown as`), so the compiler could not check the call against
pdfjs. It now uses pdfjs' own types: PDFDocumentLoadingTask, TextItem and
TextMarkedContent, all type-only imports that are erased at build.

That check found a dead option at once: pdfjs 6 has no isEvalSupported.
The option went away with the font-program eval path it used to switch
off, so it did nothing here; it is removed.
2026-10-23 18:35:17 +00:00
haritowa 85bef667dd test(documents): read files the way the realism ratchet expects
The test-realism ratchet counts a test that calls readFileSync and names
a source file. The guard now reads source through readSourceText, and the
two one-process tests no longer name their twin file in a comment. Their
doc comments are rewritten in plainer words.
2026-10-23 17:56:38 +00:00
haritowa 275121391b test(documents): keep pdf-parse out of the server's runtime
A structural guard fails when any runtime file under src/ imports
pdf-parse again, which would bring its second pdfjs back into the server
process. The next.config comment on the tracing includes now describes the
text path as it is, and the extractor no longer turns an empty
end-of-line run into a blank line of its own.
2026-10-23 17:28:28 +00:00
haritowa bb762135df fix(documents): read a PDF's text with the app's own pdfjs
Local text extraction went through pdf-parse, which pins its own
pdfjs-dist (5.4.296) beside the app's (6.4.299), and both copies take
their worker from globalThis.pdfjsWorker. Whichever ran first in a process
broke the other: after a text extraction every raster failed with "The
API version "6.4.299" does not match the Worker version "5.4.296""
(thumbnails raster-failed, vision reads of a PDF 422 extractFailed), and
after a raster the text extraction failed and the local content index
stayed empty without a trace.

extractPdfText now reads the text layer with getTextContent on the
pdfjs-dist the rasterizer uses, so only one pdfjs runs in the server.
Pointing pdf-parse at 6.4 instead is not enough: its destroy() calls
PDFDocumentProxy.destroy(), which pdfjs 6 no longer has, so every
document would skip its teardown. Two tests run text and raster in one
process, each order in its own file.
2026-10-23 17:28:28 +00:00
Marc-André Bombeck 91a5464aa3 refactor(documents): move the vault upload reading and the list read out of their route 2026-10-23 15:10:26 +02:00