A narrow `documents:write` token, minted cookie-only at
`POST /api/tokens/documents` and accepted by the vault upload route alone.
A scoped caller draws on its own hourly bucket per token
(`DOCUMENT_UPLOAD_LIMIT_PER_HOUR`, default 120) and gets a receipt
`{ id, duplicate }` instead of the stored row.
Uploads take optional `sourceSystem` / `sourceId` fields. The source key is
unique per account across live and deleted documents, and the tombstone
purge carries it to a small ledger, so re-sending a document the owner
deleted answers 200 `deleted: true` and stores nothing. `aiRead=defer`
indexes locally and skips the summary and lab staging.
The detail sheet shows where an imported document came from; backup and
restore carry both fields; Delete All Data clears the ledger.
Refs #1038, #938
Reading a stored document (suggest, summary, extract, index, chat,
reindex), scanning a lab report and turning medication text into a
schedule now answer under documentAi, labsOcr and medicationExtract. The
operator's master switch and the new reading-documents switch stop the
work and the egress; a module off for the record, a delegate inside
somebody else's record, and a missing receipt refuse with the capability
envelope. Existing no-provider codes keep their status.
The document pick and the labs pick re-check the capability for the
provider they picked, immediately before anything is sent, so a job
enqueued before a switch flipped finds no provider and falls back to the
provider-free text layer. Consent follows the document rule for all
three: any provider that leaves the machine needs an ai_extraction or
ai_full receipt, answered for the provider actually picked.
The fenced Coach chat and attachments answer under coach and documentAi.
Uploads are always stored and indexed; the background summary is only
enqueued while documentAi is open. The capability probes report the
capability beside the provider answer and keep a read offered when only
the receipt is missing. medications/extract keeps its 503 for no provider
and adds ai.provider.none, and its budget refusal gains the
coach.budget.exceeded code.
Fifty-nine 422s raised off a Zod `safeParse` answered with one opaque
sentence and nothing a client could map onto a field, while the other
166 sent `details.issues` — the multi-issue envelope that exists so a
caller with three bad fields fixes all three in one round-trip. Twelve
of the fifty-nine put a dotted machine token in `error` where the
sentence belongs, so `error` was sometimes text to show a person and
sometimes a token that must not be shown, with nothing in the payload
to tell them apart.
Every one of them now answers through `apiValidationError`, which keeps
the existing `error` string and the existing `meta` byte for byte and
adds the issue list beside them. The dotted tokens keep their place in
`error` for the clients that already match on them and are published in
`meta.errorCode` as well, which is the field a machine code belongs in
and the one a new client should read.
A guard reads the route tree per refusal site rather than per file —
per file is what let this grow, since a route that used the helper once
looked compliant to a `grep -l` sweep — and asserts it still finds the
sites it judges, so a matcher that stops matching fails instead of
reporting a clean zero.
- The measurement-reminder tick claims its slot in the record-event
ledger BEFORE any provider egress (the documented claim-first pattern
from reminder-dedup.ts); a crash between dispatch and the nextDueAt
advance can no longer double-send.
- mood-entries/bulk classifies a lost P2002 race as the duplicate it is
(with the winning row id) instead of a skip carrying a raw Prisma
message — the row IS stored, the racing twin wrote it. The workouts
batch already re-derives per-entry status from RETURNING.
- The vault list's content-index and thumbnail probes ride the same
Promise.all as the link loads: four independent grouped queries, no
serialized round-trips on a 4s-polled list.
Every arm here is the same shape as the one that already exists for
conditions, because the vault's filing model is settled and a second shape
would be the thing to explain rather than the thing to use: encounterIds on
upload and on PATCH with replace-set semantics, linkEncounter / unlinkEncounter
beside the episode pair in the bulk action, and an encounterId list filter.
Both directions ride ONE table. The visit's sheet writes encounter_document_links
from its end and the document reads it from the other, declared as a second
direction in the link service rather than a fourth table. A visit's kind
crosses the wire as the enum constant rather than a rendered name: unlike a
condition label, which is the person's own words, a kind is one of eight closed
values whose human name the reader's own bundle owns.
The suggestion is offered where the document is being looked at, resolved by
the shared rule rather than by this surface: one candidate pre-selects with an
undo, two offer a picker with nothing pre-selected, none offers nothing at all.
It resolves with no AI provider configured, and it renders nothing once the
document already names a visit.
The bulk verb has a caller — the vault's selection bar files a whole selection
against a visit, mirroring the condition verb and hiding itself on an account
with no visits. Nothing here is one-ended.
OpenAPI regenerated in the same commit.
Three narrow link tables landed with the visit record, which is a step
away from the one-shared-service direction the polymorphic-linking work
argues for. This bounds that step to a single module: every caller now
speaks one vocabulary — a source, a target kind, a list of target ids —
so replacing the storage underneath later is a change in one file rather
than a sweep across every route that files something.
The document-condition links move behind the same signature, so there
are two families and one call surface. Behaviour is unchanged and the
existing vault tests are the proof; where a test double had to follow
the new query shape, it says why in the diff.
The rules that used to live at each call site now live in the module and
are proven against real Postgres rather than a fake that ignores a
where clause: both ends re-read scoped to the owner before a row is
written, a repeated link is a success that changes nothing, a call is
capped at a hundred targets, and a tombstoned record is not linkable on
either end. Unlinking one deliberately still works after a delete, or a
filing nobody can clear would be left behind.
Reads return a resolved label and date rather than a bare id, so a
client rendering a list never re-derives what the server already knows.
The restore is the one documented exemption and writes the tables
directly: it rebuilds an account rather than filing something in one.
A grant has meant one thing since it shipped: the whole record, to read or to
add to. This lays the words for the two shapes the rest of the sharing work
needs — part of a record, and management of one — while changing what no
existing grant does.
`ShareDomain` is a closed set of eight sections in `src/lib/sharing/scope.ts`,
clustered from the delegable read surfaces rather than invented, plus the
distinguished `record` for the routes that read across them. It never converts
to a `ModuleKey` and never imports one: two masks, two questions, no shared
spelling, because the day they share a name is the day somebody makes one
imply the other, and a module the owner enabled is not a section the owner
shared.
`account_grants.scope_json` is NULL on every row that exists, and NULL is the
entire record — the answer those rows were consented as, so no backfill and no
re-consent. A non-NULL value is read through one fail-closed normaliser: a
blob it cannot parse opens nothing, including a section the blob happened to
name. That is the deliberate inverse of the module preference normaliser,
which fails open on purpose; both do the safe thing with a value nobody can
read, and the safe thing differs because one answers an authorization question
and the other a presentation question.
`MANAGE` joins the level enum, ordered above WRITE so a write grant can never
satisfy it, and carries no scope at all: management of part of a record
promises a boundary that cannot survive an edit, so the invite path refuses to
write one.
`requireRecordAuth` takes the section as a second required argument. The
requirement is the lever — a delegable route without a classification does not
typecheck, so the set of classified routes cannot fall behind the set of
delegable ones. All fifty-seven modules carry theirs. The structural guard
freezes each against the production vocabulary, fails when a module's own arms
disagree, and pins a non-zero count of resolved sections so a matcher that
stopped reading the argument fails instead of agreeing with everything.
`requireGuardianAuth` is exported and reaches nothing yet. It gates on the
managed-profile marker rather than on the grant, so an adult who granted
management of their own record cannot arrive at that record's settings by any
argument about what MANAGE means: widening MANAGE would widen the record
resolver and would not touch this one. Its frozen list lands empty, with the
declaration site as the positive control that keeps "nothing reaches it" a
measurement rather than a renamed symbol.
`users.managed_profile_at` says since when a record is one somebody else runs.
A timestamp and not a boolean, for a reason the schema has to keep possible:
clearing it is the handover, and the grants and the activity trail span it
untouched.
Seven findings from the adversarial pass before v1.36.1 was tagged. One
blocking defect shipped in that release; these did not, and they are all places
where somebody looking after another person's health record meets a wall with
no explanation, or where a guard cannot fail.
The document vault was the largest. `/documents` is marked as a shared
destination, so the shell offers it under a switch, and every read behind it
resolved the caller instead of the record: the vault painted a query-error card
and the illness view painted another beside it. That is the same failure the
release describes as fixed on the dashboard. Five GET arms move to the record
resolver now, and only the GET arms: the list, one document, its file, its
thumbnail and the vault's usage payload, which the filter bar is built from.
Uploading, retyping, deleting, linking a document to an illness and every AI
verb stay with the owner, and the two reads that spend decryption CPU key
their quota on the person calling rather than the record, the rule
`medications/compliance` already follows.
Logging a value on a metric you track yourself leaves the admitted set. It was
admitted, named on the consent screen in six languages, and unreachable: the
only form that posts one sits on a page a shared record does not open, and the
component that renders it says so in its own comment while returning null. It
is the third route the rule that withdrew allergies and family history fitted
exactly, and it was applied to two of the three. Reading those values stays.
Two guards could not fail, and both were proved by mutation before and after.
The prefetch guard checked rule three with two regexes: one was satisfied by
the doc comment every page carries, the other missed an aliased import, and
the pair stayed green with the dashboard genuinely prefetching off the
switch-blind session. It reads the AST now, resolves aliases through the
module's own import table, and pins the five prefetching pages by name so a
page that stops being detected is loud. The activity verb map decided
membership on a literal `requireRecordAuth("write")`, so putting the argument
on its own line, which is what the formatter does the moment the call grows,
dropped a route out of the scan and turned a genuinely-red run green. Both
counts are of matches now rather than of files walked.
The rest are single instances of rules applied everywhere else. "Take all due"
was withheld on a note saying it rides the bulk intake route; it does not, and
its own docblock says why not, so a helper with five morning tablets tapped
five times for nothing. Marking a dose skipped is offered wherever marking it
taken is. A form could be opened in the frame before the app knew whose record
was on screen and then stood there once the answer arrived, on the capture
sheet and on the dashboard's quick-entry sheets. Two more copies of the intake
toast were still building their own, one of them with an Undo pointing at a
route no delegate may call. The customise link on the mood page led to a page
that exists to say it is not part of what was shared. And the access-level
radios on the invitation screen are hidden by design with no focus ring on
their label, so keyboard focus moved invisibly across the one screen where
somebody grants write access to their record.
Three new structural guards, each watched fail first: the customise link sits
inside a capability branch on every surface a delegate can reach, every intake
surface confirms through the one shared decision, and the batch sweep is
offered at the level that admits it.
When the auto-AI-read opt-in is on, a freshly uploaded document is summarised
once in the background and the encrypted summary is stored and shown on the
document detail view.
- A background step runs after upload alongside the existing content-index and
thumbnail steps. It generates a short (3-4 sentence) plain-language summary
only when the user's documentsAutoAiRead opt-in is on (that flag is also the
standing AI-egress consent), skips a document that already has one, and
no-ops when no vision provider is configured. It reuses the same provider
resolution, egress-consent re-assertion, and AI-budget reserve/reconcile
helpers as the on-demand route; a provider error refunds the reservation and
never fails the upload.
- The summary is stored encrypted (summaryEncrypted Bytes) with a
summaryGeneratedAt timestamp on inbound_documents, registered for key
rotation, and returned decrypted in the document detail response.
- The on-demand transient summary route is unchanged.
Gated by the same consent, budget, and module checks.
Render a small JPEG preview for each stored document and show it at the
card's leading edge, falling back to the kind icon when none exists.
- Generate previews in the background on upload (new document-thumbnail
pg-boss queue) plus a boot-time corpus backfill for existing documents;
upload never blocks on or fails because of it. Images are downscaled via
@napi-rs/canvas, a PDF's first page is rasterised then downscaled, every
other type is skipped. Generation never throws.
- Store the preview encrypted at rest in a 1:1 DocumentThumbnail side table
(migration 0236) so the list/detail queries that omit the original blob
never drag the preview either. Same AES-256-GCM string codec + fail-closed
decrypt as the content index; wired into the key-rotation walk.
- Serve it through an owner-scoped GET /api/documents/inbound/[id]/thumbnail
(auth + module gate + userId-scoped query + fail-closed decrypt), always
image/jpeg + nosniff + private, no-store; 404 on miss. The card renders an
authed lazy <img>, onError falls back to the icon.
- Add hasThumbnail to the document DTO so the card only fetches a preview
that exists.
Also drop the "search looks inside your documents" caption from the overview
(the index-all prompt stays) and the per-card searchable marker; both were
noise. Remove the now-unused i18n keys across all six locales.
The upload route awaited an unguarded enqueueDocumentIndex after the
document was already committed. A transient boss.send rejection (a DB
hiccup) would then surface as a 500 for an upload that in fact stored the
file. Guard the send inside the enqueue (a failure annotates and no-ops,
matching the missing-boss path) and drop the await at the call site so the
response never blocks on or fails because of background indexing.
Surface the content-search index's source (vision vs local extraction) on
the list + detail DTO so the vault UI can tell an AI-read document from a
locally-indexed one. Threads the source through both read paths and the
serialisers, regenerates the OpenAPI contract, and adds a narrowing helper
for the free-String column.
Index a document for content search automatically the moment it is stored,
instead of requiring a manual index action. A fire-and-forget per-document
job runs an AI-first decision tree: when a vision-capable provider is
configured and consented, transcribe the stored original (rich, handles
scanned PDFs and images), budget-reserved exactly like the interactive index
route; otherwise fall back to server-side text-layer extraction via pdf-parse
(no provider, no egress, milliseconds) so content search works at a baseline
even with no AI. Budget exhaustion, a provider that cannot read the file
(a text-layer PDF on a non-Anthropic account), and provider errors all fall
through to the free local path rather than leave a document unsearchable.
Both paths write the same blind, encrypted content index (AES-256-GCM text +
opaque HMAC token tags) — nothing readable at rest. The local path adds the
'local-pdf' / 'local-ocr' source values (free String column, no migration).
Server-side OCR for scanned images and scanned PDFs is left as a documented
seam. Pin pdf-parse + pdfjs-dist into the standalone trace so the wasm text
extractor ships in the runtime image.
P2-1-T2/T3/T4/T5/T6.
- assist.ts + [id]/suggest: one provider call proposes a { title, kind,
documentDate } filing draft over the stored original or browser-OCR text.
Runs the full extract gauntlet (module → provider → consent → rate-limit →
budget) but writes nothing — never touches the fact rail or status.
- describe.ts + [id]/summary?mode=summary|text: session-only summary or raw
transcribed text. Descriptive only, no diagnosis; persists only the budget
ledger + audit.
- content-index.ts + [id]/index: normalise → tokenise (de-accent, stopword
drop, whole-word) → HMAC-SHA256 under an HKDF-derived subkey → store the
token array plus AES-256-GCM ciphertext of the text. Vision path transcribes
via the provider (consent + budget); text path indexes posted local-OCR text
(no egress). Gated on the existing AI consent — no new per-user toggle.
- crypto.deriveSubkey: HKDF-SHA256 purpose-bound subkey off the active key.
- list search now unions title/filename ILIKE with a GIN-accelerated
whole-word content-token match; DTO gains hasContentIndex.
- usage gains assistAvailable + contentIndex { enabled, indexedCount,
totalCount }; detail gains hasContentIndex.
Concurrent uploads all read the same pre-insert usage SUM, so a burst
could overshoot the per-user quota by rate-limit x cap in one window. A
transaction-scoped advisory lock keyed on the user id serialises the
sum-and-insert; uploads by different users never queue on each other.
Integration probes pin both races: four parallel uploads against a
two-file quota land exactly two, and two racing identical uploads
collapse onto one row via the partial unique index.
Also aligns the round-trip integration test with the serve-route CSP
posture: the proxy carve-out owns the response CSP, the route sets none.
POST /api/documents/inbound now runs the policy layer end to end: bounded
read aborts at the admin-set cap (413 + meta.reason=fileTooLarge with the
limit), magic-byte classification (415 + unsupportedType), per-user quota
checked inside the insert transaction over every non-purged row —
tombstones keep counting, so deleted bytes are never invisible weight
(413 + quotaExceeded with the figures), sha256 dedupe (a same-bytes
re-upload answers 200 + meta.duplicate with the existing live row; the
partial unique index closes the race), an honoured Idempotency-Key, the
binary2 codec on write, and optional episodeIds[] pre-linking narrowed to
the caller's live episodes.
GET stops fetching the encrypted blob per row (omit: contentEncrypted —
a 50-row page dragged up to 50 × cap ciphertext before), learns kind[] /
episodeId / year facets on top of q/from/to, and returns conditionLinks +
servingClass on every DTO via one grouped link query (no N+1). PATCH gains
replace-set episodeIds with the same 404-shaped refusal for a foreign
episode. The extract path decrypts through the codec dispatcher.
The document-kind labels gain REFERRAL in all six locales.
Tasks: W1-T5, W1-T6, W1-T7.
Default documentDate at store time. A store-only upload no longer lands
with a NULL filing date: when the client omits one, default it to the
upload day so the column is always set on a stored row. The library list
sorts, groups, and range-filters on the same effective date the UI shows,
so undated uploads no longer mis-sort under nulls-last, break date-group
headers, or fall through the range filter. The date stays user-editable
and may be cleared back to NULL via PATCH.
Block re-extraction once any fact is APPROVED. A partially-confirmed
document stays EXTRACTED, which the CONFIRMED guard does not catch;
re-extracting would delete the approved facts (severing the committed
record's provenance) and re-stage them as PENDING, letting a second
approval duplicate the committed lab / condition / medication. The
extract route now refuses with 409 (documents.inbound.alreadyPartlyConfirmed)
when approved facts exist; the user must finish reviewing or discard first.
RFC 5987-encode the original-download filename. The Content-Disposition
now carries both an ASCII filename= fallback and a filename*=UTF-8'' value
so a non-ASCII name survives intact; the stored set is inline-safe by
construction, so the dead attachment branch is removed.
Add a light per-user rate-limit backstop to the original-read, PATCH, and
DELETE routes (generous; self-DoS only — the routes are owner-scoped).
Lean the list counts: tally facts for the whole page in one grouped count
instead of materialising fact-id rows, and exclude REJECTED facts from
factCount. Extend the documentDate index with deleted_at to cover the
list scan. Document the deliberate plaintext-title PHI tradeoff at the
column and the store site.
Invert the inbound-documents surface from an OCR inbox into a document
library: a file can be stored without any AI provider, and extraction
becomes an opt-in action on top.
- Migration 0222: add `title` + `document_date` columns, a
`(user_id, document_date)` index, the richer kind taxonomy
(lab/imaging/prescription/insurance/vaccination), and a STORED status
made the column default. The status enum is recreated so the new value
can be set as the default within the migration transaction; kind values
use ADD VALUE. Nullable columns, no backfill.
- POST /api/documents/inbound is now store-only: bounded multipart read,
magic-byte MIME sniff, AES-256-GCM at rest, persisted as STORED with no
provider/consent/budget call and no egress.
- POST /api/documents/inbound/[id]/extract carries the relocated
extraction gauntlet (resolve provider, consent, rate-limit, budget, run,
reconcile, stage facts) against an already-stored row; absent a provider
it 422s the enhancement only and leaves the stored document intact.
- PATCH /api/documents/inbound/[id] edits title/kind/documentDate
field-by-field, owner-scoped by session userId.
- GET /api/documents/inbound gains search, kind filter, documentDate
range, sort, and keyset pagination via the parameter-bound query builder.
- Extend the validation schemas + DTO, regenerate the OpenAPI contract,
and add route tests for the store/extract/edit/list paths.
The inbound-document store encrypts the raw upload
(`inbound_documents.content_encrypted`) because it is sensitive clinical
data, but the per-fact staging columns persisted the same class of data in
the clear: `extracted_facts.data_json` held the transcribed clinical values
(diagnosis text, lab values, medication names, stated codes) and
`provenance_json` held the verbatim source span (up to 2000 chars copied
straight out of the report). The at-rest guarantee was defeated for staged
facts, which persist through PENDING and are retained after the decision.
Convert both columns to the AES-256-GCM `*_encrypted` BYTEA convention shared
with the note columns, going through the same Bytes codec. The upload and
fact-edit paths encrypt on write; the confirm path decrypts at commit time to
write into the normal structured stores, so the review-then-confirm flow is
unchanged. Reads are fail-closed (a bad key id throws).
Register both columns in the encrypted-columns registry and the key-rotation
script so a dropped legacy key cannot strand them (the guard test enforces
both). Migration 0220 swaps the plaintext JSONB columns for the encrypted
BYTEA columns; pre-confirm staged rows are ephemeral and cannot be
re-encrypted in SQL, so any left by a pre-release build are dropped rather
than retained as plaintext (their parent document keeps its encrypted
original).
Ingest an uploaded doctor report / discharge letter through the existing
dedicated OCR/vision provider rails (the v1.22 document-scan provider/key,
reused — no new provider) and transcribe structured facts into staging.
The safety line is extract, never interpret: the prompt frames the document
as untrusted data to reproduce verbatim. Diagnoses map to Condition,
measured values to Observation, medications to MedicationStatement — with
STATED status only. A code survives only when the document writes it and its
system matches the resource (SNOMED/ICD-10 for a diagnosis, LOINC for a
value, RxNorm/ATC for a medication); nothing is inferred, no value is
range-flagged, no condition is linked to a medication. Every fact carries
per-field provenance and a confidence score; a fact below the confidence
floor fails closed — it cannot be approved until the user edits it.
Routes (apiHandler + Zod + session-narrowed userId, opt-in module-gated):
- POST/GET /api/documents/inbound — upload+extract (vision or local-OCR
text), and list. The raw document is stored encrypted at rest.
- GET/DELETE /api/documents/inbound/[id] — review detail; soft-delete.
- PATCH .../facts/[factId] — correct a staged fact before approval.
- POST .../confirm — the only write path: approved facts flow into the
labs / conditions / medications stores through their normal create paths;
rejected facts are discarded; low-confidence facts are refused.
Marc-André Bombeck