A document uploaded with aiRead=defer is now indexed locally by any
automatic path, because the indexer reads the marker from the row rather
than from the job. Index all for search leaves such documents out and no
longer counts them. Only an explicit Read with AI or Generate summary ends
the hold.
Uploads that send a body keep their slot of the hourly limit, duplicates
included; handing it back let a token send the same bytes under id after
id for free. One document takes at most 20 further source ids, beyond
which the upload answers 409 documents.inbound.sourceAliasLimit. The
query-string key check is metered on the lookup allowance it shares with
GET /api/documents/inbound/source.
Refs #1038
A keyed upload that matched a stored document by its bytes never recorded
its key, so deleting that document let the next import run store it again.
Such keys are now kept as aliases of the document, the source-key check
reads them, and the purge carries them into the ledger with the row's own.
A source key may ride the query string and is then answered before the
body is read or the hourly allowance charged; an upload answered as a
duplicate hands its slot back. GET /api/documents/inbound/source answers
whether a key is held, under the same documents:write scope, so an
importer can skip a download.
aiRead=defer is stored on the row. The summary catch-up, the summary job
and lab auto-staging skip such documents until an explicit Read with AI
or Generate summary clears the marker; backup and restore carry it.
A documents:write caller no longer sees the quota figures on a full vault.
Refs #1038
A narrow `documents:write` token, minted cookie-only at
`POST /api/tokens/documents` and accepted by the vault upload route alone.
A scoped caller draws on its own hourly bucket per token
(`DOCUMENT_UPLOAD_LIMIT_PER_HOUR`, default 120) and gets a receipt
`{ id, duplicate }` instead of the stored row.
Uploads take optional `sourceSystem` / `sourceId` fields. The source key is
unique per account across live and deleted documents, and the tombstone
purge carries it to a small ledger, so re-sending a document the owner
deleted answers 200 `deleted: true` and stores nothing. `aiRead=defer`
indexes locally and skips the summary and lab staging.
The detail sheet shows where an imported document came from; backup and
restore carry both fields; Delete All Data clears the ledger.
Refs #1038, #938
A document could show its visits read-only and its vaccinations not at
all, so a childhood record covering a dozen doses had to be attached
from a dozen dose forms. The document's detail sheet now lists the
visits and vaccinations it is filed against and, for someone who may
manage the vault, links and unlinks them in one place, with the records
dated near the document on top. Both ends write the same link tables
through the link service, so the dose's form and the document's sheet
cannot disagree about what is filed where.
The document detail response carries the linked doses, named by
catalogue slug or the person's own wording. It is null rather than
empty for a grant that opens the vault but not the health background,
so a shared page never claims to record no doses. The update route
takes a replace-set of vaccination ids, up to 100 so one record can
cover a whole primary schedule; ownership is re-checked, a foreign id
answers 404 and the change lands in the audit log beside the other link
fields. Linking from the dose suggestion now refreshes the sheet.
Refs #1024, #1005
The document, labs-scan, medication-extraction, fenced-chat, attachment,
connection-test and auto-read operations document the capability they
answer under, the refusal envelope and codes, and the consent rule. The
two capability probes publish the capability beside the provider answer.
The shared 429 named `Retry-After` and the `X-RateLimit` triple in prose, so a
generated client got no typed accessor for the field the same sentence tells
it to back off on. They are a `headers` block now: `Retry-After` and the cap
and remainder as integers, the reset instant as a date-time. The text also
says what a 429 from a ceiling that is not one of these buckets carries —
nothing, because there is no bucket to describe.
Two route descriptions still listed the pre-change header set: the ingest 429
named only `X-RateLimit-Remaining` / `X-RateLimit-Reset`, and the document
suggest 429 named only the reset instant beside `meta.retryAt`. Both are short
of the wire in exactly the direction this work exists to close.
Zod 4's `.meta()` clones the schema and registers the clone, so a bare
`someSchema.meta({ id })` statement registers nothing at all. Thirty-nine of
those across eleven route modules published neither the name nor the
description written for it: the schema inlined anonymously at every use site,
and a client generated from the contract got a duplicated nameless type where
one named model belongs. That last part is the cost that matters, because the
native client is the reason the contract is published.
Bind each annotated clone to a const and point the route table at that const.
Twenty-nine ids now emit under `components.schemas`, referenced from their
thirty use sites. Seven stay absent by design: they annotate
`requestParams.query` objects, which the emitter expands into inline
`parameters` rather than a component. `MoodTagLayout` stays absent too, for
the reason `Medication` and `CoachPrefs` already carry a forced registration
in the components block: its only consumer is an `.extend()`, and extending
builds a new object rather than referencing the base.
Two annotations in the illness module named ids that no route ever referenced,
`GetIllnessDayLogQuery` and `ListIllnessDayLogsQuery` — the day-log path
publishes the merged `GetIllnessDayLogsQuery` instead. They were dead twice
over, so they go rather than get wired to nothing.
`docs/api/openapi.yaml` grows the twenty-nine components and their refs. The
only other movement in it is thirty descriptions appearing where the text had
never reached the document. No request or response shape changes.
`withIdempotency` has made writes safe to retry since v1.5.x. It was described
in route comments and in no schema, so an app generated against
`docs/api/openapi.yaml` could not learn that a retry was safe. The iOS side
asked for it; the mechanism does not change, only what it says about itself.
34 operations now declare `Idempotency-Key` and the `X-Idempotent-Replay`
header, plus the 409 that refuses a duplicate still in flight. The description
leads with the one silent failure in the mechanism: a key that misses
`^[A-Za-z0-9_\-:.]{8,128}$` is IGNORED rather than refused, so the write
proceeds unprotected and looks exactly like a request that carried no key.
## What the work turned up
Three things the contract had wrong or missing, none of them the thing being
built:
- `/api/insights/chat` describes `withIdempotency()` in its header comment and
applies it nowhere. The Coach chat is not retry-safe. Its prose says it is.
- `/api/measurements` exports GET, POST, PUT and DELETE and wraps only POST.
Publishing the header per PATH would have promised a safe retry on two
methods that do not honour one — the dangerous direction, so the guard works
per OPERATION.
- `/api/insights/feedback` and `/api/admin/backups/{id}/restore` are not in the
document at all. Both are named in the guard's exemption list with a reason,
not filtered out silently.
## The guard
`idempotency-contract-publication.test.ts` reads the expected set out of the
route tree on every run — nothing is hand-listed, so it cannot fall behind the
code. It fails in both directions: a new wrapped route that forgets to publish,
and a published claim no handler honours.
A fourth assertion checks the exemption list itself is still true. It earned
its place immediately: the first draft exempted the whole `/api/medications`
family on the strength of a grep for path literals that came back empty. The
paths are not written as literals there; the emitted document carries 19 of
them. The test went red, the claim was wrong, the list is two entries now.
Measure the artefact, not the source that builds it.
A content-index attempt that failed (raster failure, a scan with no
text layer, a provider error) recorded its reason only in the server's
observability stream. In the vault, hasContentIndex false looked
identical for a document nobody ever tried and one that failed three
times, and the detail sheet had nothing to say about either.
Apply the summaryState pattern to the index. Two additive columns on
InboundDocument (migration 0333) record when the last attempt finished
and why it produced no index, as a closed reason set: local-empty,
local-unsupported, decrypt-error, raster-failed, provider-error,
pdf-needs-anthropic, empty-transcription. The writer sits at the choke
point the auto job and the corpus backfill share, and in the manual
index route; a success clears the reason. The provider path's specific
failure now survives the local fallback, since "no text" is less
actionable than "the PDF would not render". rasterizePdf names its
failure class so a build that cannot rasterize at all is distinguished
from a PDF that would not render. New guard in both index paths: an
empty provider transcription never becomes a successful empty index.
The detail sheet renders the reason in plain words (muted, six
locales). The DTO and OpenAPI carry the two fields; the disaster
recovery backup and restore carry them verbatim like their summary
siblings.
Refs #776
The index-all trigger returned Boolean(jobId) while the client typed
the field as a number, so the toast read "Indexing true document(s)".
The route now answers with the count of live documents still lacking a
content index, computed from the same live-scoped counts the usage
gauge reads, and an honest 0 when no job was created. OpenAPI schema
updated to the integer shape; the hook's type carries a contract note
so the wire and the toast cannot drift apart again. Route test pins
the number, a component test pins that the toast renders the count and
never the word "true".
Refs #776
The server chain existed end to end (extract stages PENDING facts, per-fact
edit clears the low-confidence gate, confirm is the only write into the
structured stores) but the document detail offered no control over any of
it: it counted observation facts and linked to Labs. A user whose automatic
staging run was skipped had to re-upload the same PDF through Labs > Scan a
report.
- The detail sheet gains a staged-facts block with honest states: pending
facts open an inline review (approve/reject per fact; a low-confidence
fact is checked and saved first via the existing per-fact edit); a
lab-filed, already-read document with nothing staged offers "Extract lab
values" over its stored text; otherwise nothing renders.
- The extract route gains { mode: "stored" }: the same text-structuring
pass the automatic worker runs, over the document's own content index.
No re-upload, no second read of the original; full gauntlet including
the shared document-AI bucket with charge-at-dispatch honesty; 422
notIndexed when no stored text exists.
- A manual "Read with AI" / "Read again" now continues into the same
lab auto-staging the background worker performs; the response reports
labFactsStaged and the toast names the values waiting for review.
- Unchecked values are rejected and the copy says so; an unverified
low-confidence fact receives no decision and stays pending rather than
being silently discarded. Confirmation remains the only write into
Labs, conditions, and medications; approved facts carry the
committedRecordId link back to the source document (existing model, no
schema change).
- The "lab values from this document" link now counts APPROVED facts
only — it claims values that actually exist in Labs.
- Copy in all six locales; OpenAPI regenerated for the new extract mode
and the index response field.
The document-AI actions (read/index, extract, suggest, summarise) shared a
hard-coded 6/hour bucket that lied twice: the 429 said "try again in a few
minutes" while the window is an hour, and several failure paths burned a
slot before any provider was dispatched.
- DOCUMENT_AI_LIMIT_PER_HOUR configures the shared per-user ceiling
(default 6, clamped 1-1000), wired end to end: compose environment
whitelist, .env.production.example, env manifest, operator docs.
- INSIGHTS_RATE_LIMIT_PER_HOUR joins the compose whitelist and manifest;
it was documented but never listed, so an operator value could not reach
the container.
- The 429 mirrors the bucket's reset instant into meta.retryAt (plus the
existing X-RateLimit-Reset header); the client derives "try again in
about N minutes" from it in all six locales, with a plural-aware key.
- A slot is consumed only when a request actually reaches the provider (or
the local index/echo work): malformed bodies, unreadable originals,
refused consent, and exhausted daily budgets refund the charged slot via
a window-scoped refundRateLimit primitive. Consistent across all four
routes sharing the bucket.
Every arm here is the same shape as the one that already exists for
conditions, because the vault's filing model is settled and a second shape
would be the thing to explain rather than the thing to use: encounterIds on
upload and on PATCH with replace-set semantics, linkEncounter / unlinkEncounter
beside the episode pair in the bulk action, and an encounterId list filter.
Both directions ride ONE table. The visit's sheet writes encounter_document_links
from its end and the document reads it from the other, declared as a second
direction in the link service rather than a fourth table. A visit's kind
crosses the wire as the enum constant rather than a rendered name: unlike a
condition label, which is the person's own words, a kind is one of eight closed
values whose human name the reader's own bundle owns.
The suggestion is offered where the document is being looked at, resolved by
the shared rule rather than by this surface: one candidate pre-selects with an
undo, two offer a picker with nothing pre-selected, none offers nothing at all.
It resolves with no AI provider configured, and it renders nothing once the
document already names a visit.
The bulk verb has a caller — the vault's selection bar files a whole selection
against a visit, mirroring the condition verb and hiding itself on an account
with no visits. Nothing here is one-ended.
OpenAPI regenerated in the same commit.
Every route that resolves a record through the sharing fence can answer 403
`sharing.access.denied`, and until now that fact was published on the selector
header and on none of the 120 operations that produce it. All 120 carry it now,
byte for byte, from one constant: the refusal is deliberately indistinguishable
across a missing account, a missing grant, a scope that does not reach the
surface and a read grant on a write, so 120 paraphrases would invite a client to
tell apart what the server refuses to.
Where a delegable route already answered 403 for its own reason — a module
gate, the cycle gate, AI consent — the two share one description rather than one
of them going unpublished, since OpenAPI allows a single response per status.
The guard recomputes the delegable set from the route sources on every run and
classifies per verb, which matters: seventeen modules have a delegable read
beside a write that refuses a switch outright.
Every staged fact carried a sourceText the model echoed back, stored verbatim
as 'the exact verbatim span of the document the fact came from'. Nothing
checked that the span existed. A paraphrase, a normalised rendering, or an
outright invention was persisted and shown to the reviewer as a transcription
of their own clinical record — an anchor pointing at nothing, or at a span the
document never contained.
Treat the echo as a lookup key instead. anchorSourceText resolves it against
the extracted text (exact, then whitespace-collapsed and case-folded to absorb
OCR reflow) and stores the span the document actually carries, with its offset.
A quote that matches nothing is dropped and the fact is flagged unanchored;
vision mode has no extracted text at all, so its facts are unanchored by
construction. Rows written before this resolve to unanchored on read — their
quote is preserved but never presented as checked.
Opening a document said "the summary is being prepared" and kept saying it.
The view had one fallback for a null summary column and used it for every
outcome: a document nobody ever enqueued, a missing vision provider, a spent
budget, a withdrawn consent, an unreadable file, a provider error, and a
summary the outbound screen withheld all rendered as work in progress.
The common case is not the screen. The background job is enqueued at exactly
one place — a fresh upload with the auto-read opt-in already on — and no
backfill reaches anything else. Every document uploaded before the toggle went
on has a null column no job will ever fill, and the view promised all of them a
summary that was never coming.
Record the outcome as `summary_state` (migration 0254, additive, defaults to
NONE which is truthful for every existing row) and branch on it. Only PENDING,
set when a job genuinely exists, may claim generation is running. NONE,
WITHHELD and UNAVAILABLE each say what happened and offer to generate one.
The withhold keeps its rule and loses its dead end. Blocked prose is still
never shown and never stored — only the fact of the refusal is kept, and it is
no longer terminal, so asking again re-runs the gauntlet. That was the original
intent; leaving the column null defeated it, because null read as "generating"
and the guard meant nothing would ever overwrite it.
The on-demand summary now persists what it produces. It is the only repair path
for the documents the job never ran for, and re-deriving the same paragraph on
every open costs a provider call for nothing. Generation is still only ever a
click — nothing runs because a view rendered. Transcription stays session-only.
Also drop the "read by AI" line from the detail footer.
When the auto-AI-read opt-in is on, a freshly uploaded document is summarised
once in the background and the encrypted summary is stored and shown on the
document detail view.
- A background step runs after upload alongside the existing content-index and
thumbnail steps. It generates a short (3-4 sentence) plain-language summary
only when the user's documentsAutoAiRead opt-in is on (that flag is also the
standing AI-egress consent), skips a document that already has one, and
no-ops when no vision provider is configured. It reuses the same provider
resolution, egress-consent re-assertion, and AI-budget reserve/reconcile
helpers as the on-demand route; a provider error refunds the reservation and
never fails the upload.
- The summary is stored encrypted (summaryEncrypted Bytes) with a
summaryGeneratedAt timestamp on inbound_documents, registered for key
rotation, and returned decrypted in the document detail response.
- The on-demand transient summary route is unchanged.
Gated by the same consent, budget, and module checks.
Render a small JPEG preview for each stored document and show it at the
card's leading edge, falling back to the kind icon when none exists.
- Generate previews in the background on upload (new document-thumbnail
pg-boss queue) plus a boot-time corpus backfill for existing documents;
upload never blocks on or fails because of it. Images are downscaled via
@napi-rs/canvas, a PDF's first page is rasterised then downscaled, every
other type is skipped. Generation never throws.
- Store the preview encrypted at rest in a 1:1 DocumentThumbnail side table
(migration 0236) so the list/detail queries that omit the original blob
never drag the preview either. Same AES-256-GCM string codec + fail-closed
decrypt as the content index; wired into the key-rotation walk.
- Serve it through an owner-scoped GET /api/documents/inbound/[id]/thumbnail
(auth + module gate + userId-scoped query + fail-closed decrypt), always
image/jpeg + nosniff + private, no-store; 404 on miss. The card renders an
authed lazy <img>, onError falls back to the icon.
- Add hasThumbnail to the document DTO so the card only fetches a preview
that exists.
Also drop the "search looks inside your documents" caption from the overview
(the index-all prompt stays) and the per-card searchable marker; both were
noise. Remove the now-unused i18n keys across all six locales.
Add GET /api/documents/inbound/capability (document provider order + vendor-blind
egress class), repoint the vault capability hook to it, and show a per-egress
notice in the detail sheet before a document read leaves the machine. Cover the
document provider order, the egress consent gate, and the notice with tests.
i18n x6 + OpenAPI regenerated.
Co-Authored-By: Marc-André Bombeck
Surface the content-search index's source (vision vs local extraction) on
the list + detail DTO so the vault UI can tell an AI-read document from a
locally-indexed one. Threads the source through both read paths and the
serialisers, regenerates the OpenAPI contract, and adds a narrowing helper
for the free-String column.
P2-1. The list GET now issues a documentContentIndex lookup and unions a
content-token OR branch when a search term is present, so the existing route
test gains the mock + the third OR branch in its where assertion. Fix the
backfill test's findFirst mock signature under strict tsc.
P2-1-T7/T8/T9.
- document-content-index-backfill: on-demand per-user pg-boss job that indexes
a user's not-yet-indexed image/PDF documents via one provider transcription
each. Consent + budget gated, id-cursor forward walk, capped per run, stops
on budget exhaustion, resumable. Queue + worker registered in the maintenance
registrar (no cron — deliberate action only). POST /reindex triggers it.
- rotation: register DocumentContentIndex.textEncrypted and add a dedicated
block that re-encrypts the text AND re-tokenises the blind index under the new
active HKDF subkey in one update (P2-D7).
- OpenAPI: suggest / summary / index / reindex paths; hasContentIndex on the
document schema; assistAvailable + contentIndex on usage. Regenerated
docs/api/openapi.yaml.
The filter bar derived its condition chips from the loaded corpus, so an
episode whose only linked documents sat pages deep in the timeline never
surfaced a chip — the condition route into an older document silently
disappeared at four-digit corpus sizes. The usage endpoint (already read
on page mount) now returns `linkedEpisodes` — every episode carrying at
least one live document link, one indexed grouped query, no blobs — and
the view unions it with the active-filter fallback. Additive DTO change;
OpenAPI regenerated.
The documents route table picks up every delta: conditionLinks +
servingClass on the document DTO (render/download by servingClass, never
by MIME guess; unknown kind values decode as OTHER), the broadened upload
accept-set with the 413/415 error contract and the 200+meta.duplicate
dedupe outcome, episodeIds on upload and PATCH (replace-set), the kind[] /
episodeId / year list facets, the class-aware /original posture, and the
new /restore, /bulk, and /usage endpoints. Regenerated docs/api/openapi.yaml
committed alongside; openapi:check is clean.
Task: W1-T14.
Default documentDate at store time. A store-only upload no longer lands
with a NULL filing date: when the client omits one, default it to the
upload day so the column is always set on a stored row. The library list
sorts, groups, and range-filters on the same effective date the UI shows,
so undated uploads no longer mis-sort under nulls-last, break date-group
headers, or fall through the range filter. The date stays user-editable
and may be cleared back to NULL via PATCH.
Block re-extraction once any fact is APPROVED. A partially-confirmed
document stays EXTRACTED, which the CONFIRMED guard does not catch;
re-extracting would delete the approved facts (severing the committed
record's provenance) and re-stage them as PENDING, letting a second
approval duplicate the committed lab / condition / medication. The
extract route now refuses with 409 (documents.inbound.alreadyPartlyConfirmed)
when approved facts exist; the user must finish reviewing or discard first.
RFC 5987-encode the original-download filename. The Content-Disposition
now carries both an ASCII filename= fallback and a filename*=UTF-8'' value
so a non-ASCII name survives intact; the stored set is inline-safe by
construction, so the dead attachment branch is removed.
Add a light per-user rate-limit backstop to the original-read, PATCH, and
DELETE routes (generous; self-DoS only — the routes are owner-scoped).
Lean the list counts: tally facts for the whole page in one grouped count
instead of materialising fact-id rows, and exclude REJECTED facts from
factCount. Extend the documentDate index with deleted_at to cover the
list scan. Document the deliberate plaintext-title PHI tradeoff at the
column and the store site.
Invert the inbound-documents surface from an OCR inbox into a document
library: a file can be stored without any AI provider, and extraction
becomes an opt-in action on top.
- Migration 0222: add `title` + `document_date` columns, a
`(user_id, document_date)` index, the richer kind taxonomy
(lab/imaging/prescription/insurance/vaccination), and a STORED status
made the column default. The status enum is recreated so the new value
can be set as the default within the migration transaction; kind values
use ADD VALUE. Nullable columns, no backfill.
- POST /api/documents/inbound is now store-only: bounded multipart read,
magic-byte MIME sniff, AES-256-GCM at rest, persisted as STORED with no
provider/consent/budget call and no egress.
- POST /api/documents/inbound/[id]/extract carries the relocated
extraction gauntlet (resolve provider, consent, rate-limit, budget, run,
reconcile, stage facts) against an already-stored row; absent a provider
it 422s the enhancement only and leaves the stored document intact.
- PATCH /api/documents/inbound/[id] edits title/kind/documentDate
field-by-field, owner-scoped by session userId.
- GET /api/documents/inbound gains search, kind filter, documentDate
range, sort, and keyset pagination via the parameter-bound query builder.
- Extend the validation schemas + DTO, regenerate the OpenAPI contract,
and add route tests for the store/extract/edit/list paths.
The inbound-document store keeps the uploaded doctor letter encrypted at
rest (InboundDocument.contentEncrypted, AES-256-GCM), but there was no way
to read the original back — the decrypt helper had been dropped as unused.
- Restore decryptDocumentFromBytes in the document store, fail-closed on a
bad / missing key id.
- Add GET /api/documents/inbound/[id]/original: owner-scoped (userId-narrowed
query), module-gated, returns the decrypted bytes with the stored MIME
type. A PDF / image is served inline so the browser can render it; other
types download as an attachment. The filename is sanitised before it
reaches the Content-Disposition header. The bytes are PHI — never logged,
never served cross-user, and a decrypt error fails closed (500) rather than
returning ciphertext. Cache-Control is private, no-store.
- Surface a 'View original' action in the document review screen.
- Document the route in the OpenAPI registry and regenerate the spec.
- Cover the route with a test: owner-scoped success, attachment fallback,
404 for another user's document, and fail-closed on decrypt error.
i18n: viewOriginal across all six locales.
Document the /api/documents/inbound/* surface in the OpenAPI registry and
regenerate the spec. Add the inbound-documents copy across all six locales,
including the non-diagnostic disclaimer and the module label/description.
Marc-André Bombeck