A lab report's reference range reaches the record by four paths. Three of
them validate that the floor does not exceed the ceiling; the document
path did not, and the catalog store writes what it is handed. An
integration run against a real Postgres confirmed the consequence: a
100/30 pair landed on the row and on the freshly minted biomarker, so
every later reading of that analyte inherited a window that cannot be
true and a high value came back reassuring.
Dropping beats swapping. A transposed pair says the transcription is
wrong, not which of the two numbers is, and inventing an order would
state something the report never printed. The verbatim string survives
either way, so nothing the document actually said is lost.
Three points, answering according to what each one is:
- the extraction schema drops the pair, because every field there is
catch-guarded so one bad value cannot sink a 120-fact document
- the review-screen edit refuses with the wording the lab schemas
already use, because a person can see both numbers and fix them
- the commit path drops it as the last gate, which is what covers facts
staged before this rule existed and confirmed afterwards
parseReferenceRange has answered a transposed PRINTED window this way
since it was written, and the commit path already counts a window it
could not read at all. This extends both to the case that was readable
and cannot be true.
Registering `ko` in the `Locale` union left seven locale-keyed tables
without an entry, so the build could not type-check and the safety
screens, the citation-coverage grader and the reference-range parser
read nothing at all for a Korean reader.
Three failures were not missing keys but wrong assumptions:
- The shared fold decomposes with NFD and strips U+0300-U+036F. Latin
accents become plain letters, but Hangul decomposes into jamo, which
are not in that range and survived — so folded Korean text was a jamo
sequence no precomposed table entry could ever match. It recomposes to
NFC now; every Latin, Polish and German fold is byte-identical.
- `normaliseLabKey` kept `[a-z0-9]`, which reduced every Korean analyte
name to the empty string, so the derived index held none of them and a
Korean cholesterol result reached a FHIR export uncoded.
- The reference-range parser had the hyphen and the dashes but not the
tilde a Korean lab actually prints (`3.5~5.0`).
The Korean safety banks carry no `\b` next to a Hangul token — `\b` is
an ASCII `\w` boundary and Hangul is not `\w`, so `/\b증량/` matches
nothing — and the clause order is the mirror of the Latin banks, since
Korean closes with the verb. The dose patterns require a target-marking
particle for the same reason the Latin banks require "to"/"by": without
it the permitted restatement "7.5mg을 복용 중이고, 체중은 내려가고
있어요" tripped the lowering pattern.
The Coach and Insights prompts route `ko` down the reviewed English body
plus a Korean reply directive; there is no `safety-contracts.ko.yaml`, so
the native builder has no Korean ground rules to compose from.
The derived names extended the analyte class by list index. Reordering the
signal list would have re-pointed them at the units or the report header with
nothing to notice, so the class is a named constant and the check compares
against it.
The gate that decides whether a transcribed document is worth an extraction
call scored four classes of evidence and needed two. Three of them — the
reference-range label, the analyte names, the report header — were written in
English and German, so a French, Spanish, Italian or Polish report scored at
most one: the units, which are the same everywhere. Auto-staging never fired
for it. The document was lost from the flow silently, because the manual
extract button stays available and nothing reads as broken.
The text is folded once — lower case, accents dropped — and every pattern is
matched against that, which on its own recovers a good deal: "Cholestérol",
"Créatinine" and "Emoglobina" reach the existing stems as soon as the accents
are out of the way. The reference-range labels and report headers of the four
locales are added by hand, because they are prose a lab printed and the app
owns no string for them.
The analyte names ARE derived, from `labs.catalog.<slug>` across every shipped
bundle: those are names the app itself displays, and a new locale should not
need an edit here to be recognised. Names under four folded characters are
dropped — "ALT" is German for "old", and a three-letter alternative in a gate
this cheap would make every German letter a lab report. The acronyms worth
having are in the hand-written half, where each was considered on its own.
Which way this errs, stated at the code: a false negative loses a whole
document from auto-staging, while a false positive costs one extraction call
and stages facts a person then declines — nothing commits either way. So the
vocabulary is widened, and the two-distinct-class rule holds the other side.
The derived names extend the analyte class rather than forming a fifth one, so
a document naming one analyte in two languages still scores one. Tests pin
each addition on a case that fails the gate without it, and pin the prose that
must keep failing it.
The free-text reference-range parser decided what a printed window meant from
an alternation of German and English words. The app ships six locales, and a
lab prints in the language of the country it stands in, so `jusqu'à 5,0`,
`hasta 5,0`, `fino a 5,0` and `do 5,0` all fell through to `low: null,
high: null`. The string was stored, the reading was then judged against the
biomarker's own band or against nothing at all, and no surface anywhere said
that the lab had printed a window the app could not read.
The vocabulary is TABULATED here, not derived from `messages/<locale>.json`
the way an app-owned word list would be. These words are prose a lab printed
on paper; the app owns no string they could be read out of, and the language
of a report has nothing to do with the language of the UI. So the table is
hand-written, keyed by the shipped `Locale` union so that adding a locale
fails to compile until someone has answered how that language prints a range,
and every locale's words are tried against every input at once. The cost is
written down at the table: a report in a language absent from it stays
text-only.
Three things had to change beside the word lists. Introducing words are now
stripped only on a second pass, because Italian "da", Polish "od" and French
"de" also state a floor on their own and stripping first read "da 3,5" as a
bare number. Input and table are diacritic-folded and their apostrophes
normalised, so a table row written "jusqu'a" meets a report's "jusqu'à" and
the curled apostrophe a word processor emits. And one-letter separators ("a",
"e", "y") are only accepted with whitespace on both sides, so "3.5e5" is not
read as a window. Decimal commas already worked and are pinned by the new
cases.
An unreadable window now has a name. `isUnreadableRange` separates "the report
stated nothing" from "the report stated something nobody could read" — the two
were indistinguishable at every call site, which is what let the French case
pass unnoticed. The manual form already showed the reader that the text was
kept but not read; the two ingest paths now count it into their wide event, so
an unsupported notation surfaces as a number rather than as silence.
The German and English lists are the previous alternation split word for word,
and the existing suite covering them passes unchanged.
Document search does whole-token equality over a blind index, so
'Blutdruck' never found the discharge letter that says 'blood pressure'
and 'hypertension' missed every German Arztbrief. A curated German/English
medical synonym list (vitals, common labs, imaging, specialties, document
words — static, no vendor names) now expands the QUERY tokens before they
are hashed: the stored index, its tokenizer version and the schema are
untouched, the expansion happens ahead of the one-way hash, and recall
simply widens. Typed tokens always survive the expansion cap.
- The provider-chain walker now checks the day's recorded spend against
OPERATOR_COST_CAP before invoking an operator-funded admin-* entry. A
request whose primary provider runs on the user's own key reserves
under the generous user-plan ceiling; a fallback hop (or health-ledger
reorder) onto the operator's key could then spend past the operator's
cap unchecked. An exhausted cap skips the candidate like a failed hop.
- summaryState PENDING gets a giver-upper: the detail serialiser
degrades a PENDING older than an hour to UNAVAILABLE (which offers the
manual generate action), and an hourly reaper persists the same heal
following the orphan-ImportJob reconcile precedent — a job dead past
its retry budget no longer leaves the sheet saying "generating"
forever.
The residual scan from the report is a CCITT Group 4 single-image PDF,
the codec class pdfjs rasterization is known to fail on. Chain checks:
a pdfjs open failure or throw already lands as render-failed with the
error text annotated; the zero-pages shape now annotates too instead
of returning silently. An end-to-end test drives the REAL rasterizer
through prepareVisionInput and pins the rasterFailed mapping; the test
header documents why a tiny in-repo CCITT fixture is deliberately not
checked in (probed against this pdfjs version, it renders the page as
blank instead of throwing, so the fixture cannot deterministically
fail and would be flaky across upgrades). The detail view's
raster-failed line now names scanned fax compression formats as a
known cause, in all six locales.
Refs #776
A content-index attempt that failed (raster failure, a scan with no
text layer, a provider error) recorded its reason only in the server's
observability stream. In the vault, hasContentIndex false looked
identical for a document nobody ever tried and one that failed three
times, and the detail sheet had nothing to say about either.
Apply the summaryState pattern to the index. Two additive columns on
InboundDocument (migration 0333) record when the last attempt finished
and why it produced no index, as a closed reason set: local-empty,
local-unsupported, decrypt-error, raster-failed, provider-error,
pdf-needs-anthropic, empty-transcription. The writer sits at the choke
point the auto job and the corpus backfill share, and in the manual
index route; a success clears the reason. The provider path's specific
failure now survives the local fallback, since "no text" is less
actionable than "the PDF would not render". rasterizePdf names its
failure class so a build that cannot rasterize at all is distinguished
from a PDF that would not render. New guard in both index paths: an
empty provider transcription never becomes a successful empty index.
The detail sheet renders the reason in plain words (muted, six
locales). The DTO and OpenAPI carry the two fields; the disaster
recovery backup and restore carry them verbatim like their summary
siblings.
Refs #776
The document-AI actions (read/index, extract, suggest, summarise) shared a
hard-coded 6/hour bucket that lied twice: the 429 said "try again in a few
minutes" while the window is an hour, and several failure paths burned a
slot before any provider was dispatched.
- DOCUMENT_AI_LIMIT_PER_HOUR configures the shared per-user ceiling
(default 6, clamped 1-1000), wired end to end: compose environment
whitelist, .env.production.example, env manifest, operator docs.
- INSIGHTS_RATE_LIMIT_PER_HOUR joins the compose whitelist and manifest;
it was documented but never listed, so an operator value could not reach
the container.
- The 429 mirrors the bucket's reset instant into meta.retryAt (plus the
existing X-RateLimit-Reset header); the client derives "try again in
about N minutes" from it in all six locales, with a plural-aware key.
- A slot is consumed only when a request actually reaches the provider (or
the local index/echo work): malformed bodies, unreadable originals,
refused consent, and exhausted daily budgets refund the charged slot via
a window-scoped refundRateLimit primitive. Consistent across all four
routes sharing the bucket.
Every arm here is the same shape as the one that already exists for
conditions, because the vault's filing model is settled and a second shape
would be the thing to explain rather than the thing to use: encounterIds on
upload and on PATCH with replace-set semantics, linkEncounter / unlinkEncounter
beside the episode pair in the bulk action, and an encounterId list filter.
Both directions ride ONE table. The visit's sheet writes encounter_document_links
from its end and the document reads it from the other, declared as a second
direction in the link service rather than a fourth table. A visit's kind
crosses the wire as the enum constant rather than a rendered name: unlike a
condition label, which is the person's own words, a kind is one of eight closed
values whose human name the reader's own bundle owns.
The suggestion is offered where the document is being looked at, resolved by
the shared rule rather than by this surface: one candidate pre-selects with an
undo, two offer a picker with nothing pre-selected, none offers nothing at all.
It resolves with no AI provider configured, and it renders nothing once the
document already names a visit.
The bulk verb has a caller — the vault's selection bar files a whole selection
against a visit, mirroring the condition verb and hiding itself on an account
with no visits. Nothing here is one-ended.
OpenAPI regenerated in the same commit.
Three narrow link tables landed with the visit record, which is a step
away from the one-shared-service direction the polymorphic-linking work
argues for. This bounds that step to a single module: every caller now
speaks one vocabulary — a source, a target kind, a list of target ids —
so replacing the storage underneath later is a change in one file rather
than a sweep across every route that files something.
The document-condition links move behind the same signature, so there
are two families and one call surface. Behaviour is unchanged and the
existing vault tests are the proof; where a test double had to follow
the new query shape, it says why in the diff.
The rules that used to live at each call site now live in the module and
are proven against real Postgres rather than a fake that ignores a
where clause: both ends re-read scoped to the owner before a row is
written, a repeated link is a success that changes nothing, a call is
capped at a hundred targets, and a tombstoned record is not linkable on
either end. Unlinking one deliberately still works after a delete, or a
filing nobody can clear would be left behind.
Reads return a resolved label and date rather than a bare id, so a
client rendering a list never re-derives what the server already knows.
The restore is the one documented exemption and writes the tables
directly: it rebuilds an account rather than filing something in one.
A lab runs its own method on its own device, so the window a physician
evaluates a value against is the one printed next to it on that report,
not a catalog band averaged over every lab. Reading a report pulled the
value and dropped that range: the extraction asked for it, the commit
handed it to the biomarker resolver, and the resolver only uses it when
it mints a NEW marker. For every analyte the user already had — the
normal case after a first panel — the printed range was read and then
thrown away.
Add `sourceReferenceLow` / `sourceReferenceHigh` / `sourceReferenceText`
to `LabResult` and carry the range from the report to the reading. The
text is stored whether or not numbers can be read out of it, because
"bis 5,0" and "negativ" state a window too; a range that could not be
parsed is kept verbatim, never replaced with invented bounds.
`resolveEffectiveReferenceRange` is the one place the precedence lives:
the source window wins for that single reading, and the catalog stays
the net for readings without one. A printed string with no derivable
bound does not displace the catalog. Every surface that judges a
reading routes through it — the API verdict, the doctor report, the
insights blocks and per-marker assessment, the coach snapshot, the MCP
read and the FHIR `referenceRange` — so a value cannot read in range on
one surface and out on another.
The API keeps `referenceLow` / `referenceHigh` as the window in force,
so a client that reads only those two renders the right range and the
right verdict with no change of its own; the catalog band, the source
window, its origin and a divergence flag ride alongside for clients
that want to show both.
Read side: the chart draws the source window as a second, visually
distinct band with the reading it came from named beneath it, the
reading rows name the printed range and the catalog band where the two
disagree, and the doctor-report PDF prints the report's own string with
a footnote naming the saved range. Entry side: one optional field on
the lab form, folded away until asked for, and the printed range is
editable on the scan review screen.
The range parser covers the notations German and English reports
actually print — two-sided, one-sided, comma or dot decimals, thousands
separators, trailing units — reads units without ever converting them,
and refuses to guess on anything ambiguous.
The chain gains a seventh entry. It builds the OpenAI client against the
base URL the user saved, with the bearer they saved for it and the model
they named, and it reads none of the OpenAI columns — the pinned
api.openai.com arm stays pinned.
Host policy is the Local provider's, unchanged: a public host always, a
private one only where the operator allowlisted it, which is what a LAN
LiteLLM or vLLM needs. Egress keeps riding safeFetch.
The JSON-mode dialect the Local client learns per endpoint moves into a
shared module and now serves both clients, so a gateway that rejects
response_format is detected once and retried without it.
The entry stays out of the default chain: it can only resolve against a
base URL somebody configured.
Replace the internal "[unverified]" token the grounding guard left in a Coach
reply with the editorial elision mark […], and add a quiet muted caption
under the bubble explaining that some figures could not be checked against the
user's data. The mark holds the slot grammar-safely mid-sentence, before a
surviving unit, and inside a range; the notice carries the meaning. The withheld
count rides the existing provenance envelope (unverifiedFigures) through the
stream, persistence, and reload, and the fenced document-chat path carries a
minimal envelope so its notice survives too. New i18n key across all six locales.
Also fix an adjacent reload gap: provenanceFromJson restored only a subset of the
serialised envelope, so a persisted suggestion card, confirm action card, and the
tool trace were dropped on conversation reload despite the render layer expecting
them to survive. Restore them defensively alongside the new field.
Replace the untyped magnitude bag in the Coach numeric verifier with a typed
tokenizer and a normal-form reconciler, and narrow the outbound risk and dose
banks so correct figures and correct refusals stop being blocked while every
fabrication shape still blocks.
Grounding verifier (coach-prose-grounding.ts):
- Classify each prose number before grading — ISO/written dates, clock times,
bare years, ordinals, list markers, hyphenated time spans, ranges, percents,
thousands-separated integers, unit-suffixed measurements. Benign types are
never graded or stripped; ranges decompose into per-endpoint magnitudes.
- Grade against typed authoritative entries (value + kind + aggregation).
Sign-insensitive, canonical rounding, ratio to percent in percent space, and
minutes to hours widen the match only for an aggregate, kinded entry; kind
scoping stops a weight value grounding a blood-pressure claim.
- Thousands/decimal separators follow the reply language.
- Boundary-safe strip so a flagged token never clips a larger grounded number.
Outbound screen (outbound-screen.ts):
- Risk bank requires an asserted figure or a categorical engine result; a bare
engine or horizon mention (education/refusal) passes. Spelled-out percents
count as figures. No refusal-context exemption. EN + DE.
- Dose bank keeps a maintenance-continuation restatement, voided by any change
stem. Taper suffix fixed.
Route + fenced-chat thread the reply locale into the verifier.
Every ingest path in the product can now emit a typed, salience-classified
event into pg-boss, consumed by one worker that claims a durable per-day
reaction marker and fans out to the reaction surfaces.
The recency classifier runs at the seam, before anything is enqueued, so a
mass backfill — a multi-year Apple Health export, a provider re-sync, a
30-day catch-up — emits nothing at all. Non-salient writes are annotated and
dropped.
The worker makes no provider call by construction: it imports no provider
module, writes only markers, and enqueues nothing that is not budget-gated
downstream. Business refusals return a status rather than throwing, so a
refusal is never retried against a ceiling that will not move until the local
day rolls over.
Seams: sleep (one hook, inherited by all seven transports), workouts (the
batch route, the four provider syncs, the export importer), weight and blood
pressure (both measurement routes), labs (manual, OCR commit, document fact
commit).
The provider workout syncs could not previously tell a created row from an
updated one, so a re-sync of a recent session would have re-fired on every
poll; they now recover the distinction from the row the upsert already
returns.
Reaction lines are AES-256-GCM at rest, following the InsightNarrative
precedent, and the column is registered for key rotation. Markers are pruned
after 14 days by a daily maintenance job.
Every staged fact carried a sourceText the model echoed back, stored verbatim
as 'the exact verbatim span of the document the fact came from'. Nothing
checked that the span existed. A paraphrase, a normalised rendering, or an
outright invention was persisted and shown to the reviewer as a transcription
of their own clinical record — an anchor pointing at nothing, or at a span the
document never contained.
Treat the echo as a lookup key instead. anchorSourceText resolves it against
the extracted text (exact, then whitespace-collapsed and case-folded to absorb
OCR reflow) and stores the span the document actually carries, with its offset.
A quote that matches nothing is dropped and the fact is flagged unanchored;
vision mode has no extracted text at all, so its facts are unanchored by
construction. Rows written before this resolve to unanchored on read — their
quote is preserved but never presented as checked.
A READY row whose ciphertext will not open reports UNAVAILABLE, so the view
does not render a heading over nothing. Nothing asserted that, and removing the
degrade left the suite green.
Opening a document said "the summary is being prepared" and kept saying it.
The view had one fallback for a null summary column and used it for every
outcome: a document nobody ever enqueued, a missing vision provider, a spent
budget, a withdrawn consent, an unreadable file, a provider error, and a
summary the outbound screen withheld all rendered as work in progress.
The common case is not the screen. The background job is enqueued at exactly
one place — a fresh upload with the auto-read opt-in already on — and no
backfill reaches anything else. Every document uploaded before the toggle went
on has a null column no job will ever fill, and the view promised all of them a
summary that was never coming.
Record the outcome as `summary_state` (migration 0254, additive, defaults to
NONE which is truthful for every existing row) and branch on it. Only PENDING,
set when a job genuinely exists, may claim generation is running. NONE,
WITHHELD and UNAVAILABLE each say what happened and offer to generate one.
The withhold keeps its rule and loses its dead end. Blocked prose is still
never shown and never stored — only the fact of the refusal is kept, and it is
no longer terminal, so asking again re-runs the gauntlet. That was the original
intent; leaving the column null defeated it, because null read as "generating"
and the guard meant nothing would ever overwrite it.
The on-demand summary now persists what it produces. It is the only repair path
for the documents the job never ran for, and re-deriving the same paragraph on
every open costs a provider call for nothing. Generation is still only ever a
click — nothing runs because a view rendered. Transcription stays session-only.
Also drop the "read by AI" line from the detail footer.
The Coach SNAPSHOT was spliced into the prompt bare. Its JSON leaves are not
all server-computed: lab analyte, panel and unit names are transcribed by a
model out of an uploaded PDF and stored verbatim as the biomarker name, so a
hostile document chooses those strings. The about-me block has been fenced
since v1.16.1; extend the same pattern rather than adding per-field escapes.
- Move the fence markers and the wrapping primitive into a shared data-fence
module. Content is scrubbed against every known marker, not just its own
pair, so text in one block cannot forge another block's boundary.
- Fence the SNAPSHOT with an explicit data/instruction frame.
- Fence the OCR text handed to the document extractor; the previous
'OCR TEXT:' label was a label, not a boundary.
- Sanitise analyte, panel and unit at the labs source, which feeds both the
snapshot prompt and the labs_read tool result, and the medication label in
the adherence storyline.
- Capture a ---REMEMBER--- sentinel as proposed rather than active. It was
the only model-driven write with no confirm step, and the reminder recall
block fed it back into every later turn. Proposed reminders are already
excluded from that block and already carry a confirm control.
The coverage check hung off generateInsight, whose only caller runWithFallback
has no production caller left -- every live surface routes through
runRawCompletionWithFallback instead. The admin quality dashboard was reading a
metric nothing emitted. Widen the input to the structural shape both payloads
share and call it from the comprehensive post-parse step, so the annotation
reflects real generations.
Writing the locale tests found three gaps the English-only banks hid: Polish
verb suffixes fell outside the ASCII word class, a trailing word boundary after
an accented character never closed, and German subordinate clauses put the verb
last so the natural phrasing of a dose instruction never matched. The payload
collector also skipped legacy bare-string recommendations entirely.
The summary is model prose about the document and is contractually
descriptive-only, but it was returned to the browser verbatim with no guard.
Screen it against the insights contracts and let each caller apply its own
policy: the on-demand route replaces the text with a short honest statement
because a user is waiting on it, while the background job persists nothing --
its updateMany is guarded on a null summary, so a stored refusal would stamp
the document permanently with no path to regenerate.
Transcription stays unscreened by construction. It reproduces the user's own
document verbatim; a discharge letter that says to increase a dose is the
prescriber's instruction, and censoring it would delete the user's own record
from their own view. Locale is required on the summary input type and absent
from the transcription input so a caller cannot silently fall back to the
English banks.
The outbound safety screen ran on the Coach only. A dose-change imperative or
a fabricated clinical risk score written onto a per-metric, biomarker or
derived-score status card persisted as that day's cached assessment, while the
identical sentence in the Coach was caught and replaced -- the only transform
between provider and persistence was whitespace normalisation.
Move the pattern banks into a shared screen that takes the reader's locale.
The old signature took no locale, so the dose-refusal and risk-score contracts
had no enforcement for fr/es/it/pl even though the ground rules ship in all six
contract files. Banks now cover all six locales and each screen runs the
reader's bank plus English, since a provider often answers in English
regardless of the language directive.
Status cards withhold on a trip: serve the deterministic stub and persist no
model text. Nobody waits on a background-generated card, and persisting a
refusal would pin it to the card for the day.
When the auto-AI-read opt-in is on, a freshly uploaded document is summarised
once in the background and the encrypted summary is stored and shown on the
document detail view.
- A background step runs after upload alongside the existing content-index and
thumbnail steps. It generates a short (3-4 sentence) plain-language summary
only when the user's documentsAutoAiRead opt-in is on (that flag is also the
standing AI-egress consent), skips a document that already has one, and
no-ops when no vision provider is configured. It reuses the same provider
resolution, egress-consent re-assertion, and AI-budget reserve/reconcile
helpers as the on-demand route; a provider error refunds the reservation and
never fails the upload.
- The summary is stored encrypted (summaryEncrypted Bytes) with a
summaryGeneratedAt timestamp on inbound_documents, registered for key
rotation, and returned decrypted in the document detail response.
- The on-demand transient summary route is unchanged.
Gated by the same consent, budget, and module checks.
Two directions, folding the vault and labs pipelines into one linking path
without weakening either manual flow.
Vault document → lab values (auto-stage). After the per-document auto-index
job transcribes a stored document, a lab-looking document is run through the
same inbound extraction the manual "Extract" button uses, but the facts land
PENDING for the existing human review — nothing is committed. Gated on both
the documents and labs modules plus the auto-AI-read consent gate the
auto-index external path already uses; a text-structuring pass over the
transcription the index just produced, so no second vision call is spent. The
staging is a guarded STORED→EXTRACTED transition, and a keyword heuristic (or
an explicit LAB_RESULT kind) keeps an arbitrary photo from burning a call.
Everyone without the opt-in keeps the manual extract button.
Lab scan → vault. A lab report scanned in the labs area is now also filed
into the Documents vault (kind LAB_RESULT) through the existing upload —
encrypted at rest, EXIF-stripped, thumbnailed, sha256-deduped — and the
committed labs are cross-linked back to it with the same APPROVED-observation
provenance the manual vault → labs path produces. Text-mode scans keep the
image on-device and file nothing. Idempotent: a re-commit inserts no rows and
links nothing new, and the vault dedupe means no duplicate document.
Owner-scoped and module-gated on both sides; the cross-link is best-effort so
it never fails the authoritative write.
Replace the scalar document_id discriminator on a coach conversation with a
CoachConversationDocument join table and a sticky documentScoped flag, so a
conversation can be grounded on several stored documents at once while the
prompt-injection fence stays intact across the data-model change.
The fence is re-established at both enforcement points, fail-closed:
- Client: resolveCoachSendTarget routes any turn on a fenced conversation (or a
fresh chat with staged attachments) to the new /api/insights/chat/fenced
endpoint; the fenced body is strict and never carries scope, guided-question,
or prefill hints.
- Server: the tool route fetches with documentScoped:false AND 404s when the
fetched row still carries an attachment (flag/join drift), emitting a
fence_drift audit alarm. Document text now enters an LLM prompt in exactly one
code path — the fenced pipeline — with no tools and no health snapshot; a
module-graph test pins that the tool loop is unreachable from it.
The fenced turn fences each document separately with a server-built, scrubbed
header line (title and filename are untrusted text), runs a per-document egress
consent check on the single picked provider, and grounds numerically over the
live attachment union only. Attachments are capped at five documents and 192 KB
of combined text with honest truncation.
Attach and detach ride dedicated endpoints: attaching validates the document is
owned, live, indexed, and within cap, then flips the conversation fenced (the
one legal, privilege-reducing transition, audit-logged); detaching removes the
document but never clears the flag. The document-sheet chat keeps its public
wire contract and shares the same fenced pipeline over the join.
Migration 0245 adds the join table and the flag, backfills every legacy
single-document chat into a one-attachment fenced conversation, and drops the
scalar column in the same transaction — no dual-truth window.
The bundled Skia x64 build (document thumbnails, scanned-PDF rasterization)
uses AVX2 instructions; on an x86-64 host without them — Celeron/Atom-class
NAS CPUs, pre-2013 Xeons, VMs masking CPU flags — the first canvas call is an
uncatchable SIGILL that kills the process and crash-loops the container (live
incident: a Synology DS1520+, exit 132 ~18 s after every boot, reported with
kernel traps and the correct root cause in hand).
The renderer is now gated on a one-time CPU-feature check before the module
ever loads: linux/x64 requires the avx2 cpuinfo flag; arm64 and non-Linux are
unaffected; unreadable cpuinfo fails open. Behind a closed gate, thumbnail
generation and PDF rasterization take their existing fail-soft paths (vault
shows type icons; PDFs read as text) and the boot-time thumbnail backfill
skips enqueueing. NATIVE_CANVAS=off|on overrides detection, documented in the
env example and whitelisted in compose.
Dashboard layout changes take effect immediately: the save now evicts the
locale-keyed home-screen snapshot it previously missed, so a re-ordered or
toggled tile set no longer reverts to the prior selection for the SWR window.
A share link built with a narrowed report-section selection now serves exactly
that selection. The stored selection is the grouped export shape; the shared
view read it through the flat parser, which dropped every grouped toggle and
fell back to the full defaults — an over-share against the owner's choice. The
view now folds the grouped shape down faithfully. Documents-only shares were
never affected.
Hardening: the shared-document download path now sanitises the filename
identically to the owner download path, and thumbnail generation re-checks the
pixel ceiling after decode so a malformed image that slips past the header
sniff still cannot exhaust the worker.
Document vault shows a per-file thumbnail: a server-rendered preview,
downscaled and re-encoded so location and camera metadata are dropped,
served only to the file's owner over an authenticated request. Files that
cannot be previewed keep their type icon. Generation is bounded — a header
sniff refuses an over-large source before decode, so a crafted image cannot
exhaust the preview worker.
Also folds in admin-managed central AI access — an administrator connects
one access on the server and each user opts in or keeps their own provider,
off until an admin turns it on — and trims the redundant documents-search
caption and per-file searchable/read badges.
Render a small JPEG preview for each stored document and show it at the
card's leading edge, falling back to the kind icon when none exists.
- Generate previews in the background on upload (new document-thumbnail
pg-boss queue) plus a boot-time corpus backfill for existing documents;
upload never blocks on or fails because of it. Images are downscaled via
@napi-rs/canvas, a PDF's first page is rasterised then downscaled, every
other type is skipped. Generation never throws.
- Store the preview encrypted at rest in a 1:1 DocumentThumbnail side table
(migration 0236) so the list/detail queries that omit the original blob
never drag the preview either. Same AES-256-GCM string codec + fail-closed
decrypt as the content index; wired into the key-rotation walk.
- Serve it through an owner-scoped GET /api/documents/inbound/[id]/thumbnail
(auth + module gate + userId-scoped query + fail-closed decrypt), always
image/jpeg + nosniff + private, no-store; 404 on miss. The card renders an
authed lazy <img>, onError falls back to the icon.
- Add hasThumbnail to the document DTO so the card only fetches a preview
that exists.
Also drop the "search looks inside your documents" caption from the overview
(the index-all prompt stays) and the per-card searchable marker; both were
noise. Remove the now-unused i18n keys across all six locales.
Introduce an operator-level ChatGPT-subscription (Codex) credential that the
operator connects once and shares server-wide, plus a per-user switch to route
AI egress through it.
- Schema (migration 0235): three encrypted columns on AppSettings for the
operator's central Codex access/refresh tokens and the ChatGPT-Account-ID
claim, plus expiry/connected-at/status; a per-user useCentralCodex flag
(default false). Register the new encrypted columns in the rotation registry
and script.
- Resolution: add the admin-codex chain type; append it to a user's resolved
chain only when they opted in AND the operator connected it. Tokens refresh
server-side against the singleton row (CAS-style), same primitives as the
per-user Codex path.
- Budget: bill an admin-codex primary against the operator cost cap, not the
user plan — it drains the operator's account.
- Consent: admin-codex is server-managed external egress, gated by the existing
consent machinery (self-snapshot and document surfaces alike). No existing
consent path is weakened.
- Document order: rank admin-codex with the subscription tier (last), so the
document class stays local-first.
- UI: admin device-code connect/disconnect card (cookie-only admin routes) with
honest copy on account pooling, shared rate limits, and training-by-default;
a per-user opt-in switch shown only once the operator has connected it, off by
default, behind a one-time honesty confirm.
- i18n across all six locales; unit tests for the chain type, budget cap,
consent gate, vision capability, and document order; a mocked e2e for both
the admin connect surface and the per-user switch.
The central credential's tokens are encrypted at rest and never returned to a
non-admin API; the opt-in only takes effect once the operator has connected the
shared account.
The bundled (Turbopack) copy of pdfjs-dist failed its render path in the
standalone image with a bare Error, so a scanned PDF on a non-Anthropic
provider fell back to 'PDF scanning needs a Claude vision provider' — while the
identical un-bundled module rasterizes the same document fine (verified in the
container end to end on the real document). Externalize pdfjs-dist and
@napi-rs/canvas so the server loads the real modules, and hoist pdfjs-dist to
the top level in the image so the runtime import resolves. Also log the render
error message (not just its name) and drop the redundant 'not saved' note from
the document summary panel.
Only Anthropic accepts a native PDF block; codex and other image-only-wire
providers dropped it. Add a bounded, best-effort server-side rasterizer that
renders a PDF's pages to JPEG images those providers read as ordinary image
input, so a scanned/image-only PDF is legible on the auto-read path.
- @napi-rs/canvas promoted to a direct dependency; its musl prebuilts + the
pdfjs loader pinned into the standalone trace (next.config.ts)
- rasterize-pdf.ts: lazy pdfjs import (keeps DOMMatrix out of the eager chunk),
first 10 pages, 1600px long edge, JPEG q80, never throws
- prepareVisionInput rasterizes a PDF for a non-Anthropic vision provider and
keeps the native PDF block for Anthropic; the document capability reports
PDFs readable via raster so the UI offers the read
The auto-index-on-upload job resolved over the cost-first app-wide chain
(codex priority 1) and gated with the chain-based consent check, so a
pure-codex chain could silently egress a freshly uploaded document to the
subscription backend on upload. Reconcile it onto resolveDocumentVisionProvider
(local-first, codex last) and govern external egress with the
documentsAutoAiRead opt-in: a local pick always runs (no egress); an external
pick runs only when the toggle is ON, otherwise the job stays strictly local.
This tightens the default and makes the opt-in path deliberate.
Add the documentsAutoAiRead per-user Boolean (migration 0231, default
false) that authorises ambient AI reading of uploaded vault documents.
OFF by default keeps the vault local-first for every self-hoster.
- schema column + migration 0231_documents_auto_ai_read
- documentAutoReadEnabled(userId) read helper
- GET/PATCH /api/auth/me/documents-auto-ai-read toggle route; the ON
write mints an ai_full consent receipt for the audit trail
- the document consent gate short-circuits an external pick when the
toggle is ON; a local pick stays ungated and OFF still requires a
receipt for any external egress
The local content-extraction reader imported pdf-parse at module top level.
pdf-parse pulls in pdfjs, whose module evaluation references the browser-only
DOMMatrix global; on the server that reference throws during chunk
instantiation and takes down every route sharing the chunk — the document
list, upload, and search among them, so old and new documents alike fail to
load. Move the parser to a lazy import() inside the extraction function
(surface type kept as a type-only import). Verified under plain Node that the
lazy path still extracts a real PDF text layer, so local indexing is
unaffected.
Marc-André Bombeck