An account of 1.25 million measurements could not export its data or be
restored from any backup (#1031). Measured in a 1 GB container:
- The full export and the passphrase export built the whole payload and
its JSON string in memory; the app ran out of heap and restarted.
- The weekly backup's JSON is 662 MB, longer than any string V8 can hold,
so `unpackBackupBlob` threw on every read. Restore, download and the
restore preview all answered "could not be decrypted".
- Upload accepted 10 MB, a fraction of a large export.
Every reader and writer of a backup file now streams it:
- `scanBackupJson` reads a backup's bytes once, keeps every section but
the named bulk arrays, and hands their elements out one at a time.
`readStreamedBackup` uses it twice: a first pass that validates each
measurement against the element schema and keeps the rest of the file,
and a second pass that hands the measurements to the writer in batches.
Stored copies and off-host objects are authenticated whole on their
compressed ciphertext before any plaintext is read; only the
decompression is streamed.
- Restore writes the measurements with one `INSERT … FROM unnest` per
batch. Prisma caches a query plan per `createMany` row count, and one
1 000-row plan kept about 145 MB alive, which alone put the restore past
the heap limit. The transaction budget grows with the measurement count.
The awards and ECG sections are handed the ids they reference that were
written, not every written id.
- The export routes write the file into the response as it is produced.
The passphrase export spools its ciphertext to a temporary file because
the HLX1 format puts the tag first; the format is unchanged, so existing
archives open as before.
- Upload accepts the raw body, JSON or gzip, up to the 512 MB request
ceiling, checks it as it arrives and stores it in the same pass; the admin
page compresses the file in the browser before sending it. Multipart stays
for small files.
- Download streams the stored JSON; the preview counts measurements from
the stream. The monthly restore drill and `scripts/restore-backup.ts`
read the off-host object the same way; the script can write `.json.gz`.
On the seeded account in a 1 GB container: export 25 s at 389 MB peak,
passphrase export 25 s, both uploads 19 s, restore of the weekly backup
94 s at 554 MB peak, and export, upload and restore round-trip to exactly
the same rows. The encrypted-column guard now also recognises the raw SQL
statement that writes `DataBackup.data`.
Migration 0343 moved the operator's Coach answer from module availability
into the Coach switch and folded every live row. Restoring the instance
settings from an older file wrote the dead coach key back into module
availability and left the Coach switch on, so a Coach the operator had off
came back.
The restore now applies the same fold: coach false in the file's module
availability switches the Coach off, and the key is dropped either way.
The operator switches become master, Coach, Daily Briefing, Status
notes and Reading documents. Correlations goes: it gated statistics,
not anything a model writes. Reading documents covers the document
vault, lab scans and medication extraction.
The Coach module's operator availability now follows the Coach
switch, so there is one place to turn the Coach off server-wide. The
migration folds a stored Coach availability of false into the switch,
and gives everyone who hid the Coach the AI analysis opt-out, since
hiding the Coach no longer stops nightly briefing and status notes.
A switch read that fails now fails closed. The insights module is
described as AI analysis in every locale, and the switch guard now
keeps both retired switches out and freezes the callers of the old
operator-only gate.
The admin test response carried a code and status per channel that nothing rendered. The toast now lists each failing channel with the translated cause and the HTTP status or SMTP code. The relay reply is no longer part of that response, and the ntfy error no longer echoes the topic, which the sender treats as a secret.
Refs #947
The webhook, ntfy and email test routes turned every delivery failure into a bare 500, so the settings card could only say "Test failed" while the server knew the relay had answered 400. A failure at the other end, or on the way to it, is now a 502 with meta.errorCode, the upstream HTTP status or SMTP reply code, and for webhook and ntfy what the relay answered when that is short and holds no credential. The codes reuse the test card's vocabulary; upstream_rejected is new, for a relay that accepted the connection and refused the payload. The private-origin 422 is unchanged, and only a failure the server cannot name stays a 500.
The guard that pinned the bare 500 now asserts the opposite, with the reason in its name.
The admin fan-out test reports the same code and status per channel, and tests the webhook and email channels, which it used to list as an unknown channel type.
Refs #947
CLAUDE.md forbids planning shorthand in committed artefacts, and this branch
introduced 74 lines of it across source, tests, a schema field and two
migrations. Someone reading the migration SQL in a year has no way to resolve a
wave letter or a round number, and a comment that points at something outside
the repository names no reason at all.
Three passes. A version followed by a parenthetical keeps the version and loses
the parenthetical, because the version says since when something holds. A
sentence that leaned on the vocabulary to carry its meaning is rewritten to
state the reason directly: the arrival reservation now says it was refused over
money the operator never spent, rather than naming the defect a wave existed to
remove. Three test titles carried the marker in the name itself and are renamed
to what they pin.
Comments only. No behaviour, no renames beyond those three titles.
The provider-health endpoint showed delivery health only. When a surface
refuses on budget the operator's next question is which ceiling he hit,
and one mixed token figure cannot answer it: a day at 1.24 M tokens looks
alarming until you see that 151 200 of them ran on the instance's own key
and the rest on the users' plans.
`spendToday` carries the day's key and both counters, aggregated across
accounts. The operator ceiling is enforced against `operatorTokens`, so
that is the figure the readout puts beside the total.
Nothing recomputes an answer. The module map they produced survives on the
account row, but the map says what is switched on, not that the person told the
flow they take medication daily and have a visit next month — so a restore
without the row opens a blank questionnaire for somebody who already answered
it, and leaves the checklist ordering itself from nothing.
Both ends live in one file, the arrangement the visits and the score already
use, because a reader who greps only the restore route gets a false negative
when the route delegates.
`modulesDerivedAt` is the field that makes this more than history and it is
asserted rather than defaulted: it is the once-only latch on the module
derivation, so a restore that let it default to null would hand the next confirm
permission to re-apply the questionnaire over every module decision taken since,
silently. The round trip seeds it non-null and reads it back.
The file's schema validates the three answer objects loosely and the restore
narrows them through the same fail-soft readers the account payload uses. Being
stricter there would refuse a whole account's restore over one retired chip.
The wipe takes the row for the reason the reset already takes the wizard's own
columns: the record the answers were about is gone, so the answers describe
nothing.
Four smaller things around the same card.
The configured flag came from the environment of whichever process serves the
request. A hand-rolled split that gives BACKUP_S3_* to the worker and not to
the web process would have shown an operator "nothing leaves this host" over a
ledger full of fresh rows. The ledger is queried either way now, and a row in
it outranks an environment read: a row is proof an upload happened.
Rows sorted by username put the one account that needs attention wherever the
alphabet leaves it, which on a hundred-account cohort is the crowd problem the
card exists to end, moved from a count into a list. Worst verdict first,
alphabetical within a verdict, decided server-side so the tests and the
accessibility fixture see the order the operator does.
The card's docblock claimed the worker holds PutObject and GetObject. The
runbook is right and the multipart path needs AbortMultipartUpload too, so the
prose no longer documents a grant that would break the uploader.
And the byte formatter moved to lib. The off-host card was importing it out of
a thousand-line client component, which costs nothing while both cards render
on one route and becomes a cost the moment either moves.
The badge read the enabled flag plus a DSN that parses. Neither says anything
about the target accepting a report: a wrong public key, a wrong project id and
a host that stopped answering all parse, and every one of them showed green
with a line naming the host. That is the false reassurance an operator finds
out about when they go looking for a crash that was never reported.
Nothing in the tree recorded an outcome, so there was nothing to read. The
sender now stamps one next to the DSN it used — instant on a success, a short
classification on a failure. Never the response body: that is text from a
remote host and has no business in this database or on this page.
The badge is green only while a report has left inside the last fortnight,
amber while a target is set and nothing has come back, and it names the reason
underneath. The clock lives with the ledger, so the verdict is resolved in the
route and the card paints what it is handed. A failure that a later success
superseded is not shown at all. Pressing Test records like any other send, so
the button proves the target on demand.
On the release that adds the ledger the table is empty, so on a host that has
been uploading nightly for months every account read Never, under a sentence
saying no copy had ever reached the bucket. That sentence is untrue on exactly
the morning an operator opens the new card for the first time, and it is a
claim the host cannot make: what is empty is the record, not the bucket.
The run now records every account it walks, not only the ones an object came
of. A walk that produced nothing moves the attempt instant and leaves the
success columns alone, so the two cases separate: no row at all is no record
yet, a row without a success is never. Never now means what the word says —
a run looked at this account and has never put anything there.
The card names the last check on the never row, so the state is a fact with a
timestamp rather than an assertion. Runbook and all seven bundles carry the
same two sentences. The integration file proves both arms against real
Postgres, including that a failed run leaves the previous good copy on the
row.
The threshold was the schedule period with no budget on top of it. Two things
spend that budget on a host where nothing is wrong. The cron runs 02:30
Europe/Berlin, so on the DST fall-back Sunday the gap between two runs is 25
hours and every account on every host would cross one period on the same
morning. And the ledger instant is when that account's object landed, by
design, so on a cohort walked one account at a time an account reached an hour
later tonight than last night drifts past 24 with two clean runs behind it.
The schedule period and the verdict threshold are now separate numbers. The
period is still what the card quotes; the threshold is the period plus six
hours, applied to both steps, so due still means one run produced nothing and
stale still means two. The fall-back night is pinned in the unit file by its
real dates rather than by an arithmetic 25, and a genuinely missed night at 31
hours still reads due. The runbook says why the slack is there.
The column was INTEGER. A single object may occupy 8 MiB across 10 000
multipart parts, which is the 80 GB the uploader explicitly permits, and
Postgres INTEGER stops at 2.1 GB. An account between the two uploads fine and
then cannot be recorded, so the biggest account on the host is the one the
console would have to report on and could not.
BIGINT in the table, BigInt in the model, Number() where the route builds the
wire row so the payload type stays a plain number. The migration is edited in
place rather than followed by a second one: the table has not shipped, and a
column widened before its first release is one column, not a history.
The comparison with the in-database backup does not carry over, for the
record: that size is octet_length over a bytea Postgres caps at 1 GB, while
this object never touches the database at all.
The nightly off-host job reports counts — uploaded, failed, total — and a
count cannot name the account it is failing for. A cohort where one object
has been refused every night for a month reads as "99 uploaded, 1 failed"
each time, which looks like weather rather than like an account with no
disaster-recovery copy at all.
The worker now writes a ledger row when an object lands (account, instant,
size), and Admin → Backups gains a second card that rates every account
against the nightly schedule: fresh inside one run, due when one run
produced nothing, stale when two did not, never when no object has ever
landed. On a host without BACKUP_S3_* the card says off-host backup is not
configured and stops there.
The verdicts are computed from the ledger, never from a bucket listing: the
worker's grant is deliberately PutObject + GetObject, and an admin page must
not put the operator's credentials on its render path to state a number the
database already holds.
Fifty-nine 422s raised off a Zod `safeParse` answered with one opaque
sentence and nothing a client could map onto a field, while the other
166 sent `details.issues` — the multi-issue envelope that exists so a
caller with three bad fields fixes all three in one round-trip. Twelve
of the fifty-nine put a dotted machine token in `error` where the
sentence belongs, so `error` was sometimes text to show a person and
sometimes a token that must not be shown, with nothing in the payload
to tell them apart.
Every one of them now answers through `apiValidationError`, which keeps
the existing `error` string and the existing `meta` byte for byte and
adds the issue list beside them. The dotted tokens keep their place in
`error` for the clients that already match on them and are published in
`meta.errorCode` as well, which is the field a machine code belongs in
and the one a new client should read.
A guard reads the route tree per refusal site rather than per file —
per file is what let this grow, since a route that used the helper once
looked compliant to a `grep -l` sweep — and asserts it still finds the
sites it judges, so a matcher that stops matching fails instead of
reporting a clean zero.
`POST /api/admin/notifications/reminder-check` walked every active
medication on the instance and dispatched a reminder for each overdue
slot it found. That is the operator button's intended behaviour and it
stays the default, but it left any other caller — a browser journey, a
smoke check — writing into every account's delivery ledger at once with
no way to say otherwise.
The body now accepts an optional `userId` that narrows the sweep. It can
only narrow: the route is `requireAdmin()`, cookie-only by construction,
so a Bearer caller never reaches it however wide its scope, and the
selector feeds the Prisma `where` rather than the authorisation. A
bodyless POST — what the admin panel sends — is unchanged. An unknown key
is a 422 instead of a silently instance-wide run.
Published in the contract alongside the notification diagnostic, so the
scope is readable rather than folded into the console-only exemption.
Two places claimed a stored payload cannot leave the page: the card's own
description ("payloads are not downloadable from this page") and the listing
route's comment ("admins can trigger a re-snapshot but not download another
user's payload from the UI"). Both have been false since the download button
shipped — every row carries one, and the route hands back the decrypted JSON.
The description now says what happens instead: sizes are the encrypted blob,
Download decrypts the snapshot for the admin who asks, the ciphertext stays in
the database. Seven catalogues, and the route comment points at the route that
owns the payload.
A `DataBackup` row that fails to decrypt answered 500 on the restore, the
download and the preview. It is not a server fault: either the key that wrote
the copy is no longer in `ENCRYPTION_KEYS` — a rotation that dropped the legacy
entry before the script reported zero rows, the case the rotation runbook warns
about — or the stored bytes are not the ones that were written. Every other
bad-input arm on those routes already answers 4xx, and the 500 also paged the
error reporter as if the process had broken.
All three now answer 422 with `meta.errorCode = backup.payload.undecryptable`
and one shared sentence, so the three routes cannot drift into three wordings
for one condition. The download's second 500 stays: a payload this instance
wrote and cannot parse really is a fault here.
The two routes that open a stored copy move from the unpublished list into the
contract, because the refusal is now a promise an operator can look up rather
than whatever the handler happened to answer. The restore publishes its
`Idempotency-Key` header with them.
A disaster-recovery snapshot carries the singleton `app_settings` row —
registration, the MFA requirement, the default locale and timezone, module
availability, the notification and AI configuration, the document size cap and
quota — and the restore wrote it back as a side effect of restoring one
account. An operator putting a single record back on Wednesday reconfigured the
whole host from Monday's file, silently, with nothing on screen saying so.
Rebuilding a host from a snapshot is a real case, so the write stays: it is now
`restoreInstanceSettings: true` on the request and a checkbox beside the typed
gate, off unless the operator ticks it. Restoring an account and reconfiguring
the installation are two decisions, so they are two answers.
The 422 confirmation sentence and the dialog copy drop the instance-wide claim;
the opt-in carries it instead, in all seven catalogues. Integration coverage
runs both arms: without the flag the settings row keeps its own values, with it
the file's values land.
The provider-health ledger has recorded the HTTP status of every failure
since v1.11.0 and nothing has ever read it. The operator card could say
three of five chains were failing without saying whose problem that was,
which is the one thing the operator needs: a 401 is a credential only they
can replace, a 429 or a 503 is the provider's afternoon and waiting is the
right move.
The fold carries the status of the failure whose instant the row already
shows, not an aggregate over the column. A MAX of two HTTP statuses is
arithmetic on a label, and a status describing a different failure than
the one on screen would send the operator to the wrong place. The column
joins the grouping key instead, and the fold picks the group belonging to
the newest failure.
Blank where there was nothing to record: a connection that never completed
has no status, and a chain that has since recovered had its status cleared
along with the rest of its failure state.
The compression that landed last release was real and was measured in the
wrong place. On a development machine with a multi-gigabyte heap the pass
finished; the container the job runs in is capped at 1 GB, which gives V8
a 524 MB limit, and there the same pass died of memory about fifteen
seconds in. Because the job shares the application process, that was not
one failed backup — it was a restart, and every signed-in session on the
host went with it.
Reproduced before touching anything, under `--max-old-space-size=450`
against a seeded account of 445 000 measurements, 30 000 mood entries and
60 000 intake events: `FATAL ERROR: Reached heap limit` at 34 s. The same
fixture doubled to 890 000 measurements now completes inside a 296 MB
limit, holding 16 MB of live heap where the old writer held 168 MB.
The document is written incrementally now. `buildFullBackupPayload` takes
a `deferBulk` mode that declares the three unbounded tables instead of
reading them, and `streamFullBackupJson` walks the payload in key order,
pulls those tables through page by page, and releases every other section
as soon as its JSON reaches the sink. `packBackupBlobStreaming` feeds that
into gzip and an AES-256-GCM stream, so nothing between the database rows
and the stored value ever exists whole. Both writers go through the same
per-row serialisers, and the round-trip suite pins their output byte for
byte rather than trusting that they agree.
The intake and mood reads are ordered totally now. A keyset cursor needs
it, and a partial order also let two exports of one account differ over a
tie, which every multi-dose day has.
The streamed ciphertext is a new shape at rest, because GCM only produces
its tag once the last block is in and a leading tag is what makes the old
layout unstreamable. It carries a marker no older writer can produce, so
the two families are disjoint by construction rather than by sniffing.
All three stored shapes — plain, compressed, streamed — restore, proved
both ways.
The writer also watches its own live heap and stops at 80 % of V8's
limit. An account too large for its host is now one failed backup,
counted in the run's meta, instead of a restart for everybody on it.
The stored column is the ceiling that is left: `data_backups.data` is a
single `text` value, so the finished artifact has to exist as one string,
and that is the only part of the job still growing with the record. The
operator runbook says so, with the number at which it starts to matter.
A `z.object()` without `.strict()` strips an unknown key and parses
clean, so a caller that misspells a field name gets a 200 and loses the
value with nothing to tell it. The worst of them was the Coach
question-dismiss endpoint, where an omitted `question` means "dismiss
all": a typo in that key wiped every pending question and answered as
though one had been dismissed.
Twenty-four inline schemas across twenty-two endpoints are strict now —
the settings toggles, the restore and bulk-delete endpoints, message
feedback, the admin user update, the onboarding acknowledgements, the
provider chain, the chart overlay preferences, the medication extractor
and the two client-side reporters.
The strip stays where it carries weight. Sync ingest, device
registration and the layout PUTs are built out of fields that shipped as
additive and optional, on the promise that a server which does not know
one ignores it; an app running ahead of the install it talks to is
routine here rather than exceptional, and refusing would turn a dropped
field into a refused batch — a year of history lost to one row, a
dashboard that will not save, a phone that quietly stops registering for
push. A read filter discards nothing when a stray query parameter rides
along, the OAuth endpoints are required by RFC 6749 and RFC 7591 to
ignore parameters they do not recognise, and an import file comes from
another version or another tool by definition. Each is registered with
its reason in a new inventory test, so a schema cannot join them without
someone making the call. That test parses the tree rather than grepping
it: fourteen of these schemas are written `z\n .object({…})` and a
`z.object(` sweep never saw them.
`sanitiseZodIssues` renders `unrecognized_keys` instead of passing it
through. Zod leaves `path` empty and puts the names in `issue.keys`, so
the sanitised issue named nothing at all and the caller was told only
that its body had failed. The names now ride a bounded `keys` field and
the message is rebuilt from them, capped at ten keys of 64 characters
and restricted to identifier characters, so a caller-supplied key cannot
echo back verbatim through the error path or into the audit ledger. The
names survive `stripValuesFromMessage` because a field name is
structure, not user-typed content.
The weekly pass can stop and leave no trace where anyone looks. The
handler's own error never runs when the job is killed rather than thrown;
`recordError()` increments an unnamed counter; and the admin status page's
failing-queue panel looks back 72 hours, which for a weekly queue means a
permanent failure is on screen three days in seven. The backups page
itself showed an ordinary row with an ordinary timestamp, because a copy
made six weeks ago reads exactly like Sunday's until somebody subtracts
the dates.
The listing now carries two facts that fail at different times: the age of
the newest scheduled copy, which never leaves any retention window, and
the terminal state of the last scheduled run, which is the only place the
reason is written down. `readLastQueueRun` takes the newest run rather
than a window, so a weekly queue's outcome stays readable until the next
one replaces it, and it unwraps the value pg-boss stores a thrown error
under, so the page says "job timed out" instead of raw JSON. The
failing-queue reader gets the same unwrap.
Manual uploads are excluded from the cadence judgement by the shape of the
input, so an ad-hoc upload cannot make a dead cron look alive, and a fresh
instance is not called overdue for having no copy yet.
The listing also stops reading every blob into the web process to measure
it; `octet_length` answers that in the database, which on an instance with
large records was its own way of running out of memory.
The weekly pass built the whole account in memory, serialised it whole,
encrypted the string whole, and handed the result to the database whole.
That is four full copies of the record at the same moment — the object
graph, the JSON, the base64 ciphertext at 1.33x the JSON, and the copy the
driver makes of that parameter. Measured against a seeded account of
445 000 readings, the JSON alone is 242 MB and the run dies of heap
exhaustion; under a modest heap it never reaches the insert. The pass was
not slow, it could not finish.
A health record's JSON is extremely repetitive, so gzip takes that 242 MB
to 14 MB and every copy after it shrinks with it. The same run completes
in about twenty seconds with a peak heap of 434 MB and a stored row of
25 MB instead of 322 MB — which also matters because the copy lives inside
the database it would be needed to restore. Both backup jobs now hand the
serialised form back from a frame of its own, so the payload graph is
released before the compress step allocates anything.
The marker sits inside the ciphertext, so the stored column keeps the one
shape `decrypt()` already understands and rows written before this still
restore. The off-host object gets the same treatment through the version
byte its envelope always carried; version 1 objects still read back, which
both the restore drill and the restore script depend on.
The obvious saving — dropping soft-deleted readings, four rows in five on
a real record — is not taken, and the round-trip test now seeds a
tombstone and asserts it makes the journey. A tombstone inside the
retention window is what tells a synced client the row is gone and what
makes a re-sent sample a no-op; a restore without them brings back every
measurement the account ever deleted on the next device sync.
The `data-backup` queue was registered without send options, so every job
it minted took pg-boss's queue defaults: a fifteen-minute expiration and
two redeliveries. A snapshot that serialises every account's whole record
does not fit in fifteen minutes on a large one, so the pass was killed
part-way through, redelivered twice against the same record, and reached
`failed` with "job timed out" about three quarters of an hour after it
started — having never once completed.
Two hours, on the schedule rather than on the queue: `create_queue()` ends
in ON CONFLICT DO NOTHING, so a queue-level default would apply to a fresh
database and be inert on every instance that already has the queue. The
admin run-now route sends with the same options, or an ad-hoc snapshot
would silently keep the default the cron was rescued from.
The default two retries stay. The old redeliveries were pointless because
the first attempt could not succeed either; a transient failure genuinely
benefits from a second go, and the alternative is losing the week.
`adminSettingsSchema` validates five `assistant*` booleans and the strict
schema refuses anything else, but the route's `booleanFields` list never
carried them. Sending one alone answered 422 "No valid fields"; sending one
alongside a real field answered 200 and dropped it. The schema comment says
the entry exists so an operator scripting their settings over the single
route can carry the values, so the route is brought up to the contract
rather than the contract cut down to the route.
The flags are echoed in the GET and PUT payloads as raw column values — the
master-over-sub resolution stays with `/settings/assistant-flags`, which
owns that shape and was never affected.
The comprehensive-briefing cache (User.insightsCachedText) is one slot
per account with no record of the language it was written in. Every
reader served whatever was there, so a German-language account whose
nightly warm had written English kept seeing English on the dashboard
lead and the Today briefing. The request path only replaced it once the
row was a day old and the provider answered, which with a provider at
its usage limit was never.
Add User.insightsCachedLocale (migration 0335). Both writers store the
locale they generated in; the timestamp-only refresh on an unchanged
snapshot tags the row too, since the fingerprint covers the generation
locale. Every reset that nulls the text nulls the tag.
The snapshot briefing lift and GET /api/insights/generate treat a tag
that differs from the reader's locale as an empty cache: no prose, no
timestamp, the preparing / no-provider state, and the out-of-band warm
enqueued in the reader's locale through the existing path. The
generator's own short-circuit and the POST short-circuit refuse to call
a foreign-language row fresh. A NULL tag (rows written before this
change) is served as before until the next write tags it. The Today
digest derives from the snapshot and follows.
Covered by unit tests on the lift, the route and the generator, and an
integration test that writes the row through Prisma and reads the
snapshot for a reader in another language.
The per-user retry ledger has recorded every AI delivery outcome for a
long time, but no surface ever read it for the operator: a central
provider could fail on every call for weeks and the admin console said
nothing. The AI settings section now carries a health card, backed by a
new admin-only endpoint that folds the ledger into one row per provider
type: users tracked, users currently failing, the worst uninterrupted
failure run, and the last success and failure instants. Operator-managed
types sort first because their failure takes the shared fallback down
for everyone whose chain relies on it. Counts and timestamps only; the
response never names a user.
Register `ko` in the locale list, language names, and labels, and ship
messages/ko.json so the whole UI — dashboard, settings, share views,
clinician PDF, notifications, and AI prompts — renders in Korean.
Extend the locale-parity, English-leak, plural-completeness, and copy
guard tests to cover the new locale, and update README/CONTRIBUTING to
list Korean among the supported languages.
A restore's first act is to delete the class of records it is about to
rebuild, so a payload whose documents section is simply absent did not
"restore everything else". It emptied the vault, reported success, and left
the operator to find the hole later.
The whole file is refused instead, above the transaction that wipes the
account, and the refusal names the section and says nothing was changed —
an operator holding the only copy of an account cannot act on "restore
failed". The sections ride the error envelope as meta.errorCode
backup.section.missing, and the admin console holds them in a panel beside
the skip report rather than a toast that outlives nothing.
The manifest is the discriminator, and only the manifest. A portable export
leaves the mental-health screeners and the consent receipts out on purpose
and declares both as omitted; those files still restore. So does a file
whose section is present and empty, which is what an account with no
documents honestly writes, and so does every backup written before the
manifest existed. The integration test proves the distinction with the same
builder and the same deleted keys, differing only in what the manifest said.
The route is deliberately outside the published contract — requireAdmin is
cookie-only — so the OpenAPI document is unchanged.
The register said a restored medication keeps today's schedule and loses
the record of when the dose or the timing moved. Measured, it costs more
than a record: the band minter reads each past day against the schedule
that was live THEN, so an account that came back without its eras had its
own compliance history rewritten against a plan it was never on.
The hard part was supersededByRevisionId, the pointer from a corrected
ARCHIVED era to the MANUAL row that replaced it. Three things about it
had to be settled.
It is a bare id column with no relation, so nothing in the database
refuses a value that points at nothing. That makes losing it quieter and
worse than a broken foreign key would be: every era consumer SKIPS a row
that carries the pointer, so a corrected era whose link came back null
goes live again beside its own correction, and the timeline shows one
window twice with past days minted against whichever the query read
first.
A portable restore mints fresh ids, so the pointer cannot travel as one.
It travels as a POSITION in the drug's own ordered era list. That list is
the only address space both ends share: the builder sorts it totally
(validFrom, then createdAt, then id) so two eras sharing an instant
cannot swap places between two exports of the same account, and the
pointer never crosses a drug, because every route that writes it scopes
the update to one medication. The same mechanism serves both purposes
rather than one carrying ids and the other positions.
Writing it needs two passes, because the row a link addresses is as often
written after the row addressing it as before. Pass one creates every era
with the link null; pass two patches the links once every position has an
id.
A hand-edited file that names a position out of range, or an era's own
position, gets the era and not the pointer, and the position is reported
as scheduleRevisionLink with the JSON path into the file. Refusing the
file would cost an operator every era of every drug over one bad integer,
and writing the link would point a correction at whatever row happened to
be there. A row that supersedes itself is treated the same way: every
consumer would skip it, so the era would vanish from the timeline while
still sitting in the table.
The round trip seeds three eras with one supersede among them and asserts
the shape of the chain, then asserts that exactly two are live. Three
rows with every link null satisfies any count and has thrown away the
only thing the pointer says.
The register said the drug comes back with no statement of what it was
for, which is exactly what a pinned efficacy target is: a row exists ONLY
where the person overrode the derived ATC / name resolution, so it is a
decision and never a computation. Nothing recomputes it after a restore
because there is nothing to recompute it from.
The lab arm travels as a biomarker NAME rather than an id, the way a lab
result's cross-reference already does. A portable restore mints a fresh
catalogue id, and `Biomarker` is unique on `(userId, name)`, so the name
is the only end of the reference that survives the trip.
The obstacle was ordering, not machinery: the remap map already existed,
but the drugs are rebuilt several hundred lines before the analytes are.
The targets are collected against the id each drug actually got and
written in a second pass once the biomarkers exist.
The alternative was hoisting the biomarker restore above the medications,
and it loses on two counts. `biomarkerByName` and `restoredBiomarkerIds`
are read by the lab-result loop immediately below the block, so moving it
would separate a map from its only other consumer and leave a later edit
free to re-order them back with nothing noticing. And the dependency
would read backwards: a pinned target is a leaf of the medication tree,
and letting a leaf dictate where the clinical record gets rebuilt inverts
what the file is about. The second pass states the constraint where it
applies instead of hiding it in a line order.
A named analyte the restore did not put back is dropped and reported as
medicationTarget rather than written. `biomarkerId` is a real foreign
key, so a dangling value would fail the constraint and cost the whole
account over one override. Dropping beats nulling here because the
override IS the reference: a row with neither arm set is what the
resolver already reads as "no override", so keeping it would restore
something that means nothing while still holding the primary slot.
The round trip asserts the restored target points at the biomarker the
restore itself wrote, resolved from the database rather than remembered
from the fixture. Counting would have passed on a row whose biomarkerId
came back null, and nothing else would have complained: the resolver
falls back to deriving a target and the page still renders.
The register said the originating device may still hold them. That holds
for about as long as the phone beside the watch does, and not at all once
a Withings connection is revoked: the signal id stops resolving with it.
So a self-hoster who exported and wiped had a file with no traces in it
and a restore that reported success.
Both ends live in src/lib/export/ecg-backup.ts, the arrangement the
reminders, visits and Coach sections already use.
Three things needed deciding.
The waveform follows the note contract: ciphertext verbatim in a
disaster-recovery payload, the micro-volt samples decrypted in a portable
one. There is no placeholder for a strip that will not decrypt the way
the Coach has one for a turn, because waveformEncrypted is a required
column. Such a recording is left out of a portable file with a warning on
the wide event, so the number of strips in the file is the number of
strips that can be shown.
measurementId is a REAL foreign key against the event row a strip was
filed with, unlike the Coach references next door. A value that will not
resolve does not dangle quietly here, it violates the constraint and
rolls the whole restore back over one cross-reference. A portable export
omits soft-deleted measurements, so that case is reachable rather than
theoretical. The reference is resolved against the measurements the
restore wrote, nulled when it cannot be, and reported as ecgReference.
The strips are wiped and rewritten by the section itself. The wipe at the
top of the restore never reached them: a recording hangs off the account,
not off a measurement, so deleting the measurements only nulled the
pointer and left the row behind.
The round trip asserts the row column by column rather than counting it.
A recording is three things at once and each is the whole value of the
row on its own: a strip with no rhythm verdict is a squiggle nobody has
read, a strip with no recordedAt belongs to no day, and a strip with no
samples is a claim that an ECG exists which nothing can draw.
Four models leave the coverage-pending register with both ends and a
round-trip proof: PersonalRecord, UserAchievement, EnvironmentContext and
EnvironmentTravelLocation. Two new section modules hold reader and restore
beside each other, the arrangement the visits, vaccinations, reminders and
Coach sections already use.
The register understated two of them. The bests are "recomputable in
principle", which is true and does not help: nothing recomputes them, and a
recomputation over a restored history would find different rows, because
sample series and GPS routes never travel, a portable file omits deleted
readings, and a record set in a reading that has since been corrected cannot
be found twice. The badge's unlock date is worse. The evaluator prefers a
persisted date over a derived one, and several of the dates it can still
derive come from counters no backup carries, so a dropped row does not relock
the badge, it re-earns it today.
PersonalRecord.sourceMeasurementId is the reference that needed care, and it
is the dangerous kind. The schema declares no relation for it; migration 0054
created the column as a real foreign key against measurements. Writing it
blind is not a dangling pointer, it is a constraint violation that rolls the
whole restore back and hands the operator an empty account. Measured, with the
resolution removed: "Foreign key constraint violated on the constraint:
personal_records_source_measurement_id_fkey", HTTP 500, nothing restored. So
the restore runs after the measurements, resolves the pointer against the rows
it actually wrote, nulls what it cannot resolve and reports it under a new
personalRecordReference skip kind.
An unlocked badge names a definition in the code catalogue, which drifts
between the release that writes a file and the one that reads it. The id is
carried and written back verbatim rather than judged, the same answer the
vaccination restore gives an antigen slug it cannot resolve.
The readings and the location periods travel as a pair because carrying the
readings alone is worse than losing them. The environment resolver reads the
periods live, so with them gone the nightly seven-day lookback re-resolves
every trip day to the home location and upserts over the row: an account would
come back with a fortnight abroad and then have it quietly rewritten as a
fortnight at home, days after a restore that reported success. Day keys stay
strings end to end, and fetchedAt is restored verbatim so a two-year-old
provisional reading does not claim it was confirmed today.
The round trip seeds bests that share a metric type and an instant and differ
only in the sport slot, two badges earned years apart (one of them under an id
this build no longer defines), and a trip day beside a home day, then reads
each back column by column. Counting alone passes every one of the failures
worth fearing here: slots dropped so a 5 km best is presented as a 10 km best,
unlock dates stamped at restore time, a trip day re-attributed to the home
city, a location period whose bounds no longer cover the reading it explains.
Each of those was applied on purpose and each failed on its own assertion.
Two schema comments claimed no read path selects a column that this backup now
selects; both say what reads it instead of staying wrong.
`DocumentConditionLink` and `ExtractedFact` were classified as backed up and
came back from no restore. The register said a restored vault is unsorted and
that the facts are re-derivable by re-running the extraction. The first is
right; the second understates it. What a re-run cannot produce is the REVIEW
DECISION on each fact, and every column holding one defaults to "nobody has
looked at this yet". `POST /api/documents/inbound/[id]/confirm` acts only on a
PENDING fact and commits it into the structured store, so a restore that let
those columns default would offer an already-committed reading for approval a
second time and write a second lab result behind it.
Both ends live in `src/lib/export/document-filing-backup.ts`, beside each other
like the Coach, reminder and vaccination sections. `status`, `needsReview`,
`confidence` and the commitment pair are carried and restored verbatim, and the
wire schema requires the first three rather than defaulting them.
Ordering: the section runs after the documents, the condition episodes, the lab
results and the medications. The first two are real foreign keys, unlike the
Coach's bare id columns, so an unresolvable value there does not cost one edge,
it aborts the transaction and the operator gets none of the account back. The
last two are what `committedRecordId` resolves against. The builder therefore
carries a filing or a fact only when both of its ends are carried, and the
restore still checks and drops what a hand-edited file leaves unresolvable.
`committedRecordId` is nulled together with its type, because a type with no id
claims a commitment the row cannot name.
The round trip seeds a second condition and files the page under it, so a
restore that kept the count and lost the pairing fails rather than handing back
a lab report filed under a head cold. A third case breaks both references in a
saved file to prove the drop path runs and still answers 200.
Three new skip catalogues arrive with it, and with them labels for the three
that had none: a dropped Coach attachment had been reaching the operator's
screen labelled "mood factor" since it was added. The label chain is exhaustive
now, so the next catalogue without one stops the build.
Two models come off the coverage register together, and neither is an
ordering or an id-remap problem. Both hang off the account and nothing
else. What kept them off was a question the code never wrote down, so
this change writes it down.
The screeners hold completed WHO-5, PHQ and GAD administrations, and for
PHQ-9 the encrypted blob contains item 9, the question about thoughts of
being better off dead. The application already draws a line around that
blob: it keeps a coarse item9Flagged boolean so the history view can
re-surface crisis resources without decrypting anything. The intent here
was to follow that line and carry the administration while leaving the
answers encrypted. The schema settles it more firmly. responsesEncrypted
is Bytes and NOT NULL, so there is no half-row to write and the choice
is the whole thing or none of it. A plaintext export in a downloads
folder is the wrong home for that answer, so the row rides in a
disaster-recovery file only.
The consent receipts are not sensitive in that way. Their problem is
that a consent is given to an operator. A receipt restored onto a
different instance asserts an agreement that operator never obtained,
with a signature timestamp to make it look settled. A disaster-recovery
file goes back to the same instance, so it travels there and not in a
portable one.
In both cases the portable file SAYS so in its own manifest rather than
reporting an account that has neither. The manifest is merged rather
than added beside the records one, because two keys of the same name
would shadow each other depending on spread order.
item9Flagged is restored from the file rather than recomputed.
Recomputing would mean decrypting every administration during a restore,
which is the handling this data is kept away from, and the stored flag
is what the history view has been reading all along.
Verified by breaking it: dropping revokedAt on the way back fails the
round trip with a withdrawn consent coming back as an active one.
Two entries sat side by side on the coverage register, described as "the
same shape of loss". They are not the same shape at all, and the
difference is worth writing down.
ReminderPhaseConfig hangs off a medication, one row per drug, nested
inside something that already travels. It needed nothing new and it
travels now. The round trip tunes all four thresholds away from their
defaults and puts one of them in PERCENT rather than MINUTES, because a
restore that quietly fell back to the schema defaults would hand back
four plausible numbers and change when the drug turns orange.
NotificationPreference cannot travel, and the register was wrong to
imply it was merely unbuilt. A preference addresses ONE notification
channel by a hard foreign key. The channel model is deliberately
excluded from every backup because it carries channel secrets, and the
restore deletes every channel it finds. A carried preference would point
at a row that cannot exist: the same constraint violation the custom
mood categories were causing, waiting for somebody to implement it in
good faith.
So it moves to the deliberate exclusions with a reason that names the
dependency instead of the symptom. Re-applying somebody's tuning to a
channel they add after a restore is a feature about channel types, not a
backup contract about rows, and it should be argued for on its own
terms rather than smuggled in as debt repayment.
An account that ever created its own mood category could not be restored
at all. Not partially, not with a gap: the restore violated
mood_tags_category_id_fkey and rolled the whole file back, so the
operator got a 500 naming a constraint and an empty account.
The coverage register described this as custom tags landing in a
grouping that no longer exists, which reads like a cosmetic loss.
Measured, it is the same shape the retired-catalogue-key work already
removed once: one row nobody thought about taking down everything else
with it.
MoodTag.categoryId is a real foreign key. Seeded categories belong to
the instance and are deliberately still not carried, but a category the
person made is theirs, cascades away with them, and has to travel. It
now does, and it is written before the tags that point at it.
The hidden set travels by key rather than id, because what people hide
is almost always a seeded tag whose id differs on every instance. A key
this instance no longer knows costs one hidden row and is reported: a
silent drop would un-hide a factor somebody had deliberately taken out
of their mood editor, and they would have no way to know why it came
back.
Found by experiment rather than by reading: seeding a user-owned
category into the round trip turned it red with the constraint name, and
the same test is green now.
A restored account reported no supply at all. Every pack was gone, and
with it every low-stock reminder, on a feature that became first-class
when medication moved to one stock per drug.
The two travel together because the register itself paired them: without
the events the count "cannot be rebuilt even in principle", and without
the packs the events are a list of deltas against nothing.
The count comes back exactly as recorded, not recalculated from the
ledger. It is the value the server had resolved and the person was
shown, and every reminder already sent was computed from it. The fixture
makes the difference visible on purpose: thirty in, eight consumed, and
a remaining count of twenty-one, because somebody dropped one without
recording it. A restore that "corrected" that to twenty-two would
disagree with the account's own history and look like a bug fix while
doing it.
Both decimal columns cross the wire as strings. Half tablets are a real
prescription and a JSON number would round them.
The note follows the contract the side effects and dose changes beside
it already use, and the packs nest inside their drug, so a portable
restore minting fresh ids needs no remap.
Verified by breaking it: recomputing the remaining count from the ledger
fails the round trip with twenty-one against twenty-two.
The transcript came back in the previous change; what the Coach keeps
BETWEEN threads did not. A restored account had every word of every
conversation and a Coach that has to be told its own history again — the
register said as much for all three: "a restore makes the account
re-tell its history", "keeps the talk and loses the commitment", "a
restored account silently stops sending them".
Tombstones ride only in a disaster-recovery file. All three soft-delete,
and a deleted fact is hidden from injection while kept for audit — a
distinction only the owning instance can act on. A portable file
carrying them would hand a person rows their own app refuses to show
them, and a restore that resurrected one would put a fact back into the
Coach's mouth that the person had deliberately taken out of it.
Neither reference here is a foreign key. sourceConversationId, on all
three, and relatedPlanId on the reminder are bare id columns, so nothing
in the database refuses a value pointing at nothing. That makes them
more dangerous than a real key rather than less: a dangling reference
costs no error, it just quietly stops meaning anything while the row
goes on looking complete. The restore resolves both against what it
actually wrote, nulls what it cannot resolve, and reports the count.
Nulling rather than dropping is the point. A live reminder must not be
lost because the plan it mentions was tombstoned out of a portable file.
The reminder is still the person's and it still fires; what it loses is
a pointer that was already going nowhere.
Order follows from that: threads, then plans, then reminders. Running
the memory before the transcript would cut every fact loose from the
conversation it came out of and still report success.
Verified by breaking it: making the conversation lookup always return
null fails the round trip on the fact's own source field.
A restored account got a Coach that had never spoken to it. The
conversations, every turn in them, and the record of which vault
documents a thread was grounded in were all classified as backed up and
travelled with nothing. The register's own entry for the messages called
them the largest volume of free text an account owns after its
documents.
The three travel together because two of them mean nothing alone: a turn
without its thread has nowhere to hang, and an attachment names a
pairing rather than a thing.
documentScoped is the field that makes this more than lost history. It
is a permanent fence — set when a conversation is created through the
fenced endpoint or gets its first document, and lowered by no code path
anywhere, not by detaching every document and not by deleting one. A
restore that let it default to false would hand the tool loop back to
precisely the conversations the fence exists to keep away from it, and
nothing in the interface would look wrong afterwards. So it is carried
verbatim, restored verbatim, and asserted per thread rather than in
aggregate: a restore that wrote false everywhere would still return both
threads and every turn and satisfy every count.
Prose follows the contract the rest of the payload already uses. A
disaster-recovery file carries the stored ciphertext; a portable file
carries the text, because a portable export exists to be readable by the
person who owns it, exactly as it already decrypts medication notes,
mood notes and document summaries. A single row under a key the instance
no longer holds is named rather than thrown, so one unreadable turn
cannot cost someone the rest of the transcript.
Ids ride in both modes, as the reminders do, so an attachment resolves
and the facts, plans and reminders still owed have something real to
point at. The restore runs after the documents for the same reason, and
an attachment naming a document the file does not carry is dropped and
reported instead of failing the whole file over one link.
Verified by breaking it: forcing documentScoped to false on the way back
fails the round trip on that field by name.
The titration history was classified as backed up and travelled with
nothing. An account restored from a backup showed the current dose and
no record of how it got there: the ramp, the reductions, the note the
prescriber gave for each step. For a long-running drug that history is
often the more clinically interesting half.
Export writes the rows nested inside their drug, ordered by
effectiveFrom, so a portable restore that mints fresh ids needs no
remap. The note follows the contract the side effects already use: a
portable export decrypts it for a human reader, a disaster-recovery
export carries the ciphertext verbatim, and a restore never writes it
into the legacy plaintext column.
Order is the content here, not decoration. Two rows on the same drug in
the same unit, 5 mg then 10 mg, describe a ramp; the same two rows
returned in the other order describe a taper. Counting them back cannot
tell those apart, so the round-trip test asserts the sequence and the
values, not the count.
MedicationDoseChange moves from the coverage-pending register into the
two-ended list, which makes the round-trip registry incomplete until a
row is seeded and counted. That is a compile error, not a review note.
`MedicationPauseEra` was classified as backed up and travelled nowhere. The
spans where a drug was deliberately paused are the difference between "stopped
taking it" and "was told to stop taking it", and nothing else in the payload can
reconstruct them: an era is a decision, not an observation, and no intake row
records its own absence.
Without them the compliance recomputation reads a deliberate pause as missed
doses, so a restored account looks non-adherent for a stretch during which it
did exactly what it was told.
Both ends land together. The eras ride nested inside their medication, like the
schedules and side effects beside them, so a portable file that mints fresh
medication ids needs no remap — Prisma binds each child to whatever id the
parent actually got.
## The open era is the point
`resumedAt` is nullable end to end and the writer always emits it. An open era
means the drug is paused right now, and readers run it to the present. The
restore passes the null through rather than closing it at restore time: a pause
that was still running when the backup was taken is still running afterwards,
and closing it would silently turn an ongoing decision into a historical one.
Counting alone cannot catch that — a restore that closed the era would still
hand back two rows. So the round-trip test seeds one closed era and one open
one and asserts both field by field.
## What is proven and what is not
Moving the model out of `COVERAGE_PENDING` and onto `TWO_ENDED_MODELS` makes
the round-trip test's registry incomplete, which is a compile error. Verified
by watching `tsc` fail with TS2741 before the count-back was written: the
coupling the plan's comment describes does hold.
The round-trip test itself needs Postgres via testcontainers and has not been
run here. Typecheck, lint, format and the 21458-test unit suite are green; the
integration proof comes from CI.
Marc-André Bombeck