Seven findings from the adversarial pass before v1.36.1 was tagged. One
blocking defect shipped in that release; these did not, and they are all places
where somebody looking after another person's health record meets a wall with
no explanation, or where a guard cannot fail.
The document vault was the largest. `/documents` is marked as a shared
destination, so the shell offers it under a switch, and every read behind it
resolved the caller instead of the record: the vault painted a query-error card
and the illness view painted another beside it. That is the same failure the
release describes as fixed on the dashboard. Five GET arms move to the record
resolver now, and only the GET arms: the list, one document, its file, its
thumbnail and the vault's usage payload, which the filter bar is built from.
Uploading, retyping, deleting, linking a document to an illness and every AI
verb stay with the owner, and the two reads that spend decryption CPU key
their quota on the person calling rather than the record, the rule
`medications/compliance` already follows.
Logging a value on a metric you track yourself leaves the admitted set. It was
admitted, named on the consent screen in six languages, and unreachable: the
only form that posts one sits on a page a shared record does not open, and the
component that renders it says so in its own comment while returning null. It
is the third route the rule that withdrew allergies and family history fitted
exactly, and it was applied to two of the three. Reading those values stays.
Two guards could not fail, and both were proved by mutation before and after.
The prefetch guard checked rule three with two regexes: one was satisfied by
the doc comment every page carries, the other missed an aliased import, and
the pair stayed green with the dashboard genuinely prefetching off the
switch-blind session. It reads the AST now, resolves aliases through the
module's own import table, and pins the five prefetching pages by name so a
page that stops being detected is loud. The activity verb map decided
membership on a literal `requireRecordAuth("write")`, so putting the argument
on its own line, which is what the formatter does the moment the call grows,
dropped a route out of the scan and turned a genuinely-red run green. Both
counts are of matches now rather than of files walked.
The rest are single instances of rules applied everywhere else. "Take all due"
was withheld on a note saying it rides the bulk intake route; it does not, and
its own docblock says why not, so a helper with five morning tablets tapped
five times for nothing. Marking a dose skipped is offered wherever marking it
taken is. A form could be opened in the frame before the app knew whose record
was on screen and then stood there once the answer arrived, on the capture
sheet and on the dashboard's quick-entry sheets. Two more copies of the intake
toast were still building their own, one of them with an Undo pointing at a
route no delegate may call. The customise link on the mood page led to a page
that exists to say it is not part of what was shared. And the access-level
radios on the invitation screen are hidden by design with no focus ring on
their label, so keyboard focus moved invisibly across the one screen where
somebody grants write access to their record.
Three new structural guards, each watched fail first: the customise link sits
inside a capability branch on every surface a delegate can reach, every intake
surface confirms through the one shared decision, and the batch sweep is
offered at the level that admits it.
Replace the internal "[unverified]" token the grounding guard left in a Coach
reply with the editorial elision mark […], and add a quiet muted caption
under the bubble explaining that some figures could not be checked against the
user's data. The mark holds the slot grammar-safely mid-sentence, before a
surviving unit, and inside a range; the notice carries the meaning. The withheld
count rides the existing provenance envelope (unverifiedFigures) through the
stream, persistence, and reload, and the fenced document-chat path carries a
minimal envelope so its notice survives too. New i18n key across all six locales.
Also fix an adjacent reload gap: provenanceFromJson restored only a subset of the
serialised envelope, so a persisted suggestion card, confirm action card, and the
tool trace were dropped on conversation reload despite the render layer expecting
them to survive. Restore them defensively alongside the new field.
Opening a document said "the summary is being prepared" and kept saying it.
The view had one fallback for a null summary column and used it for every
outcome: a document nobody ever enqueued, a missing vision provider, a spent
budget, a withdrawn consent, an unreadable file, a provider error, and a
summary the outbound screen withheld all rendered as work in progress.
The common case is not the screen. The background job is enqueued at exactly
one place — a fresh upload with the auto-read opt-in already on — and no
backfill reaches anything else. Every document uploaded before the toggle went
on has a null column no job will ever fill, and the view promised all of them a
summary that was never coming.
Record the outcome as `summary_state` (migration 0254, additive, defaults to
NONE which is truthful for every existing row) and branch on it. Only PENDING,
set when a job genuinely exists, may claim generation is running. NONE,
WITHHELD and UNAVAILABLE each say what happened and offer to generate one.
The withhold keeps its rule and loses its dead end. Blocked prose is still
never shown and never stored — only the fact of the refusal is kept, and it is
no longer terminal, so asking again re-runs the gauntlet. That was the original
intent; leaving the column null defeated it, because null read as "generating"
and the guard meant nothing would ever overwrite it.
The on-demand summary now persists what it produces. It is the only repair path
for the documents the job never ran for, and re-deriving the same paragraph on
every open costs a provider call for nothing. Generation is still only ever a
click — nothing runs because a view rendered. Transcription stays session-only.
Also drop the "read by AI" line from the detail footer.
The summary is model prose about the document and is contractually
descriptive-only, but it was returned to the browser verbatim with no guard.
Screen it against the insights contracts and let each caller apply its own
policy: the on-demand route replaces the text with a short honest statement
because a user is waiting on it, while the background job persists nothing --
its updateMany is guarded on a null summary, so a stored refusal would stamp
the document permanently with no path to regenerate.
Transcription stays unscreened by construction. It reproduces the user's own
document verbatim; a discharge letter that says to increase a dose is the
prescriber's instruction, and censoring it would delete the user's own record
from their own view. Locale is required on the summary input type and absent
from the transcription input so a caller cannot silently fall back to the
English banks.
When the auto-AI-read opt-in is on, a freshly uploaded document is summarised
once in the background and the encrypted summary is stored and shown on the
document detail view.
- A background step runs after upload alongside the existing content-index and
thumbnail steps. It generates a short (3-4 sentence) plain-language summary
only when the user's documentsAutoAiRead opt-in is on (that flag is also the
standing AI-egress consent), skips a document that already has one, and
no-ops when no vision provider is configured. It reuses the same provider
resolution, egress-consent re-assertion, and AI-budget reserve/reconcile
helpers as the on-demand route; a provider error refunds the reservation and
never fails the upload.
- The summary is stored encrypted (summaryEncrypted Bytes) with a
summaryGeneratedAt timestamp on inbound_documents, registered for key
rotation, and returned decrypted in the document detail response.
- The on-demand transient summary route is unchanged.
Gated by the same consent, budget, and module checks.
Replace the scalar document_id discriminator on a coach conversation with a
CoachConversationDocument join table and a sticky documentScoped flag, so a
conversation can be grounded on several stored documents at once while the
prompt-injection fence stays intact across the data-model change.
The fence is re-established at both enforcement points, fail-closed:
- Client: resolveCoachSendTarget routes any turn on a fenced conversation (or a
fresh chat with staged attachments) to the new /api/insights/chat/fenced
endpoint; the fenced body is strict and never carries scope, guided-question,
or prefill hints.
- Server: the tool route fetches with documentScoped:false AND 404s when the
fetched row still carries an attachment (flag/join drift), emitting a
fence_drift audit alarm. Document text now enters an LLM prompt in exactly one
code path — the fenced pipeline — with no tools and no health snapshot; a
module-graph test pins that the tool loop is unreachable from it.
The fenced turn fences each document separately with a server-built, scrubbed
header line (title and filename are untrusted text), runs a per-document egress
consent check on the single picked provider, and grounds numerically over the
live attachment union only. Attachments are capped at five documents and 192 KB
of combined text with honest truncation.
Attach and detach ride dedicated endpoints: attaching validates the document is
owned, live, indexed, and within cap, then flips the conversation fenced (the
one legal, privilege-reducing transition, audit-logged); detaching removes the
document but never clears the flag. The document-sheet chat keeps its public
wire contract and shares the same fenced pipeline over the join.
Migration 0245 adds the join table and the flag, backfills every legacy
single-document chat into a one-attachment fenced conversation, and drops the
scalar column in the same transaction — no dual-truth window.
Render a small JPEG preview for each stored document and show it at the
card's leading edge, falling back to the kind icon when none exists.
- Generate previews in the background on upload (new document-thumbnail
pg-boss queue) plus a boot-time corpus backfill for existing documents;
upload never blocks on or fails because of it. Images are downscaled via
@napi-rs/canvas, a PDF's first page is rasterised then downscaled, every
other type is skipped. Generation never throws.
- Store the preview encrypted at rest in a 1:1 DocumentThumbnail side table
(migration 0236) so the list/detail queries that omit the original blob
never drag the preview either. Same AES-256-GCM string codec + fail-closed
decrypt as the content index; wired into the key-rotation walk.
- Serve it through an owner-scoped GET /api/documents/inbound/[id]/thumbnail
(auth + module gate + userId-scoped query + fail-closed decrypt), always
image/jpeg + nosniff + private, no-store; 404 on miss. The card renders an
authed lazy <img>, onError falls back to the icon.
- Add hasThumbnail to the document DTO so the card only fetches a preview
that exists.
Also drop the "search looks inside your documents" caption from the overview
(the index-all prompt stays) and the per-card searchable marker; both were
noise. Remove the now-unused i18n keys across all six locales.
Only Anthropic accepts a native PDF block; codex and other image-only-wire
providers dropped it. Add a bounded, best-effort server-side rasterizer that
renders a PDF's pages to JPEG images those providers read as ordinary image
input, so a scanned/image-only PDF is legible on the auto-read path.
- @napi-rs/canvas promoted to a direct dependency; its musl prebuilts + the
pdfjs loader pinned into the standalone trace (next.config.ts)
- rasterize-pdf.ts: lazy pdfjs import (keeps DOMMatrix out of the eager chunk),
first 10 pages, 1600px long edge, JPEG q80, never throws
- prepareVisionInput rasterizes a PDF for a non-Anthropic vision provider and
keeps the native PDF block for Anthropic; the document capability reports
PDFs readable via raster so the UI offers the read
Add GET /api/documents/inbound/capability (document provider order + vendor-blind
egress class), repoint the vault capability hook to it, and show a per-egress
notice in the detail sheet before a document read leaves the machine. Cover the
document provider order, the egress consent gate, and the notice with tests.
i18n x6 + OpenAPI regenerated.
Co-Authored-By: Marc-André Bombeck
Repoint suggest/summary/extract/index/reindex routes and the content-index
backfill job to the document-scoped resolvers (local-first, codex last) and the
pick-based document-egress consent gate. Labs OCR and Coach/insights untouched.
Co-Authored-By: Marc-André Bombeck
The upload route awaited an unguarded enqueueDocumentIndex after the
document was already committed. A transient boss.send rejection (a DB
hiccup) would then surface as a 500 for an upload that in fact stored the
file. Guard the send inside the enqueue (a failure annotates and no-ops,
matching the missing-boss path) and drop the await at the call site so the
response never blocks on or fails because of background indexing.
The PATCH detail route now reads the content-index provenance, so its route
test mocks documentContentIndex.findUnique; the searchable status spinner and
the Read-with-AI pulse pair with motion-reduce:animate-none per the coverage
guard.
Surface the content-search index's source (vision vs local extraction) on
the list + detail DTO so the vault UI can tell an AI-read document from a
locally-indexed one. Threads the source through both read paths and the
serialisers, regenerates the OpenAPI contract, and adds a narrowing helper
for the free-String column.
Index a document for content search automatically the moment it is stored,
instead of requiring a manual index action. A fire-and-forget per-document
job runs an AI-first decision tree: when a vision-capable provider is
configured and consented, transcribe the stored original (rich, handles
scanned PDFs and images), budget-reserved exactly like the interactive index
route; otherwise fall back to server-side text-layer extraction via pdf-parse
(no provider, no egress, milliseconds) so content search works at a baseline
even with no AI. Budget exhaustion, a provider that cannot read the file
(a text-layer PDF on a non-Anthropic account), and provider errors all fall
through to the free local path rather than leave a document unsearchable.
Both paths write the same blind, encrypted content index (AES-256-GCM text +
opaque HMAC token tags) — nothing readable at rest. The local path adds the
'local-pdf' / 'local-ocr' source values (free String column, no migration).
Server-side OCR for scanned images and scanned PDFs is left as a documented
seam. Pin pdf-parse + pdfjs-dist into the standalone trace so the wasm text
extractor ships in the runtime image.
The usage endpoint counted every DocumentContentIndex row for the user,
including those whose parent document has been soft-deleted (the index
row survives until the hard purge). Against totalCount, which filters
live documents, the gauge could read past 100 %. Scope the indexed count
to live documents so coverage never exceeds the corpus.
Also drop a redundant identical-branch ternary in the per-document
content-index status row.
P2-1. The list GET now issues a documentContentIndex lookup and unions a
content-token OR branch when a search term is present, so the existing route
test gains the mock + the third OR branch in its where assertion. Fix the
backfill test's findFirst mock signature under strict tsc.
P2-1. Unit tests for the token scheme (whole-word match, stopword drop,
de-accent, no plaintext at rest, subkey follows the active key), HKDF subkey
derivation, the suggest route (drafts only, writes nothing, provider-gated),
the summary route (session-only, budget+audit only), the index route (vision
transcribe→upsert, text posts OCR text, provider/local-OCR gated), and the
backfill (no-provider/no-consent no-op, indexes, budget-reached resume).
P2-1-T7/T8/T9.
- document-content-index-backfill: on-demand per-user pg-boss job that indexes
a user's not-yet-indexed image/PDF documents via one provider transcription
each. Consent + budget gated, id-cursor forward walk, capped per run, stops
on budget exhaustion, resumable. Queue + worker registered in the maintenance
registrar (no cron — deliberate action only). POST /reindex triggers it.
- rotation: register DocumentContentIndex.textEncrypted and add a dedicated
block that re-encrypts the text AND re-tokenises the blind index under the new
active HKDF subkey in one update (P2-D7).
- OpenAPI: suggest / summary / index / reindex paths; hasContentIndex on the
document schema; assistAvailable + contentIndex on usage. Regenerated
docs/api/openapi.yaml.
P2-1-T2/T3/T4/T5/T6.
- assist.ts + [id]/suggest: one provider call proposes a { title, kind,
documentDate } filing draft over the stored original or browser-OCR text.
Runs the full extract gauntlet (module → provider → consent → rate-limit →
budget) but writes nothing — never touches the fact rail or status.
- describe.ts + [id]/summary?mode=summary|text: session-only summary or raw
transcribed text. Descriptive only, no diagnosis; persists only the budget
ledger + audit.
- content-index.ts + [id]/index: normalise → tokenise (de-accent, stopword
drop, whole-word) → HMAC-SHA256 under an HKDF-derived subkey → store the
token array plus AES-256-GCM ciphertext of the text. Vision path transcribes
via the provider (consent + budget); text path indexes posted local-OCR text
(no egress). Gated on the existing AI consent — no new per-user toggle.
- crypto.deriveSubkey: HKDF-SHA256 purpose-bound subkey off the active key.
- list search now unions title/filename ILIKE with a GIN-accelerated
whole-word content-token match; DTO gains hasContentIndex.
- usage gains assistAvailable + contentIndex { enabled, indexedCount,
totalCount }; detail gains hasContentIndex.
The filter bar derived its condition chips from the loaded corpus, so an
episode whose only linked documents sat pages deep in the timeline never
surfaced a chip — the condition route into an older document silently
disappeared at four-digit corpus sizes. The usage endpoint (already read
on page mount) now returns `linkedEpisodes` — every episode carrying at
least one live document link, one indexed grouped query, no blobs — and
the view unions it with the active-filter fallback. Additive DTO change;
OpenAPI regenerated.
Concurrent uploads all read the same pre-insert usage SUM, so a burst
could overshoot the per-user quota by rate-limit x cap in one window. A
transaction-scoped advisory lock keyed on the user id serialises the
sum-and-insert; uploads by different users never queue on each other.
Integration probes pin both races: four parallel uploads against a
two-file quota land exactly two, and two racing identical uploads
collapse onto one row via the partial unique index.
Also aligns the round-trip integration test with the serve-route CSP
posture: the proxy carve-out owns the response CSP, the route sets none.
The one-shot deep-link consumption moves to the render-phase derived-
state pattern the view already uses for its search draft (the effect
variant trips the set-state-in-effect rule), gated on resolved auth so
a deep link never burns before the module flag is known.
Refs: document-vault plan W3-T4
The proxy's serve-route carve-out owns the response CSP (default-src
'none'; frame-ancestors 'self'; no sandbox — Chromium force-downloads
sandboxed PDFs), and middleware headers overwrite route headers, so the
route-level sandbox value never reached a client. Remove it and point
both the route and its header-matrix test at the proxy pin.
Refs: document-vault plan W1-T10 posture reconciliation
POST [id]/restore clears the soft-delete tombstone inside the 30-day undo
grace; it answers 409 once the tombstone is purged (the undo window is
over) and 409 with meta.reason=duplicateExists when an identical live
document would collide with the live-dedupe index. Restoring an
already-live row is a no-op success, so a late undo toast never errors.
POST /bulk applies one action (setKind / linkEpisode / unlinkEpisode /
delete / restore) to up to 100 owner-scoped documents and reports per-id
outcomes — a partial failure never aborts the batch. GET /usage returns
usedBytes (tombstone-inclusive), the effective quota, the per-file cap,
and the accepted-extension list for picker accept attributes.
The serve route now derives its posture from the serving class: inline
types keep their true Content-Type and gain Content-Security-Policy:
sandbox as defence-in-depth for the PDF viewer context; attachment types
ALWAYS go out as application/octet-stream + Content-Disposition:
attachment. nosniff and private,no-store on both; RFC 5987 filename
handling unchanged; decrypt dispatches on the row codec and stays
fail-closed. Header matrix pinned per class in the route tests.
Tasks: W1-T8, W1-T9, W1-T10.
POST /api/documents/inbound now runs the policy layer end to end: bounded
read aborts at the admin-set cap (413 + meta.reason=fileTooLarge with the
limit), magic-byte classification (415 + unsupportedType), per-user quota
checked inside the insert transaction over every non-purged row —
tombstones keep counting, so deleted bytes are never invisible weight
(413 + quotaExceeded with the figures), sha256 dedupe (a same-bytes
re-upload answers 200 + meta.duplicate with the existing live row; the
partial unique index closes the race), an honoured Idempotency-Key, the
binary2 codec on write, and optional episodeIds[] pre-linking narrowed to
the caller's live episodes.
GET stops fetching the encrypted blob per row (omit: contentEncrypted —
a 50-row page dragged up to 50 × cap ciphertext before), learns kind[] /
episodeId / year facets on top of q/from/to, and returns conditionLinks +
servingClass on every DTO via one grouped link query (no N+1). PATCH gains
replace-set episodeIds with the same 404-shaped refusal for a foreign
episode. The extract path decrypts through the codec dispatcher.
The document-kind labels gain REFERRAL in all six locales.
Tasks: W1-T5, W1-T6, W1-T7.
Default documentDate at store time. A store-only upload no longer lands
with a NULL filing date: when the client omits one, default it to the
upload day so the column is always set on a stored row. The library list
sorts, groups, and range-filters on the same effective date the UI shows,
so undated uploads no longer mis-sort under nulls-last, break date-group
headers, or fall through the range filter. The date stays user-editable
and may be cleared back to NULL via PATCH.
Block re-extraction once any fact is APPROVED. A partially-confirmed
document stays EXTRACTED, which the CONFIRMED guard does not catch;
re-extracting would delete the approved facts (severing the committed
record's provenance) and re-stage them as PENDING, letting a second
approval duplicate the committed lab / condition / medication. The
extract route now refuses with 409 (documents.inbound.alreadyPartlyConfirmed)
when approved facts exist; the user must finish reviewing or discard first.
RFC 5987-encode the original-download filename. The Content-Disposition
now carries both an ASCII filename= fallback and a filename*=UTF-8'' value
so a non-ASCII name survives intact; the stored set is inline-safe by
construction, so the dead attachment branch is removed.
Add a light per-user rate-limit backstop to the original-read, PATCH, and
DELETE routes (generous; self-DoS only — the routes are owner-scoped).
Lean the list counts: tally facts for the whole page in one grouped count
instead of materialising fact-id rows, and exclude REJECTED facts from
factCount. Extend the documentDate index with deleted_at to cover the
list scan. Document the deliberate plaintext-title PHI tradeoff at the
column and the store site.
Invert the inbound-documents surface from an OCR inbox into a document
library: a file can be stored without any AI provider, and extraction
becomes an opt-in action on top.
- Migration 0222: add `title` + `document_date` columns, a
`(user_id, document_date)` index, the richer kind taxonomy
(lab/imaging/prescription/insurance/vaccination), and a STORED status
made the column default. The status enum is recreated so the new value
can be set as the default within the migration transaction; kind values
use ADD VALUE. Nullable columns, no backfill.
- POST /api/documents/inbound is now store-only: bounded multipart read,
magic-byte MIME sniff, AES-256-GCM at rest, persisted as STORED with no
provider/consent/budget call and no egress.
- POST /api/documents/inbound/[id]/extract carries the relocated
extraction gauntlet (resolve provider, consent, rate-limit, budget, run,
reconcile, stage facts) against an already-stored row; absent a provider
it 422s the enhancement only and leaves the stored document intact.
- PATCH /api/documents/inbound/[id] edits title/kind/documentDate
field-by-field, owner-scoped by session userId.
- GET /api/documents/inbound gains search, kind filter, documentDate
range, sort, and keyset pagination via the parameter-bound query builder.
- Extend the validation schemas + DTO, regenerate the OpenAPI contract,
and add route tests for the store/extract/edit/list paths.
The inbound-document store keeps the uploaded doctor letter encrypted at
rest (InboundDocument.contentEncrypted, AES-256-GCM), but there was no way
to read the original back — the decrypt helper had been dropped as unused.
- Restore decryptDocumentFromBytes in the document store, fail-closed on a
bad / missing key id.
- Add GET /api/documents/inbound/[id]/original: owner-scoped (userId-narrowed
query), module-gated, returns the decrypted bytes with the stored MIME
type. A PDF / image is served inline so the browser can render it; other
types download as an attachment. The filename is sanitised before it
reaches the Content-Disposition header. The bytes are PHI — never logged,
never served cross-user, and a decrypt error fails closed (500) rather than
returning ciphertext. Cache-Control is private, no-store.
- Surface a 'View original' action in the document review screen.
- Document the route in the OpenAPI registry and regenerate the spec.
- Cover the route with a test: owner-scoped success, attachment fallback,
404 for another user's document, and fail-closed on decrypt error.
i18n: viewOriginal across all six locales.
The inbound-document store encrypts the raw upload
(`inbound_documents.content_encrypted`) because it is sensitive clinical
data, but the per-fact staging columns persisted the same class of data in
the clear: `extracted_facts.data_json` held the transcribed clinical values
(diagnosis text, lab values, medication names, stated codes) and
`provenance_json` held the verbatim source span (up to 2000 chars copied
straight out of the report). The at-rest guarantee was defeated for staged
facts, which persist through PENDING and are retained after the decision.
Convert both columns to the AES-256-GCM `*_encrypted` BYTEA convention shared
with the note columns, going through the same Bytes codec. The upload and
fact-edit paths encrypt on write; the confirm path decrypts at commit time to
write into the normal structured stores, so the review-then-confirm flow is
unchanged. Reads are fail-closed (a bad key id throws).
Register both columns in the encrypted-columns registry and the key-rotation
script so a dropped legacy key cannot strand them (the guard test enforces
both). Migration 0220 swaps the plaintext JSONB columns for the encrypted
BYTEA columns; pre-confirm staged rows are ephemeral and cannot be
re-encrypted in SQL, so any left by a pre-release build are dropped rather
than retained as plaintext (their parent document keeps its encrypted
original).
Ingest an uploaded doctor report / discharge letter through the existing
dedicated OCR/vision provider rails (the v1.22 document-scan provider/key,
reused — no new provider) and transcribe structured facts into staging.
The safety line is extract, never interpret: the prompt frames the document
as untrusted data to reproduce verbatim. Diagnoses map to Condition,
measured values to Observation, medications to MedicationStatement — with
STATED status only. A code survives only when the document writes it and its
system matches the resource (SNOMED/ICD-10 for a diagnosis, LOINC for a
value, RxNorm/ATC for a medication); nothing is inferred, no value is
range-flagged, no condition is linked to a medication. Every fact carries
per-field provenance and a confidence score; a fact below the confidence
floor fails closed — it cannot be approved until the user edits it.
Routes (apiHandler + Zod + session-narrowed userId, opt-in module-gated):
- POST/GET /api/documents/inbound — upload+extract (vision or local-OCR
text), and list. The raw document is stored encrypted at rest.
- GET/DELETE /api/documents/inbound/[id] — review detail; soft-delete.
- PATCH .../facts/[factId] — correct a staged fact before approval.
- POST .../confirm — the only write path: approved facts flow into the
labs / conditions / medications stores through their normal create paths;
rejected facts are discarded; low-confidence facts are refused.
Marc-André Bombeck