GET /api/auth/me gains coachReasoning: the level a turn runs at, the
person's own pick, the highest level the settings may offer, whether
reasoning can run at all, whether the lowest option is a real off, and
the layer that decided. Resolved on the server so the web settings and
the native client read the same answer.
The web reads it through useCoachReasoning. The Coach preferences route
names the stored level on its wide event, and a unit test round-trips
the level through the account-settings backup section.
Taking the weekly preview from the writer dropped the schema check the
old scanner ran at the end, so a copy that violates the schema was
described as restorable. The collector now checks each section in its
JSON form as the writer shows it, an array section and a bulk table one
element at a time, plus the schema's one whole-document rule (a
canonical copy names every measurement by id). A copy that fails gets
no stored preview, and the preview route reads the copy itself, as it
did before. Nothing is kept beyond one element.
Measured on the preview memory test under --max-old-space-size=524:
held after the producer 3 MB with and without the check, raw peak
heapUsed 293 MB with it (335 MB without). The test now loads the
backup schema before its baseline: a module loaded once per process,
not memory that grows with the record.
backupPayloadSchema is built when validations/backup.ts is evaluated and
holds about 12 MB of heap. The restore job, the stored-backup readers and
the export writer import that module, and they sit on the boot path, so
every server process built the schema at start whether or not it ever
touched a backup.
The schema version, the compatibility check and the per-section summary
move to validations/backup-summary.ts, which never builds the schema. The
call sites that validate a file (upload, restore, the summary and download
routes, and the streamed reader's element check) load backup.ts with a
dynamic import at that moment; everything else takes only types. The
OpenAPI registry keeps its static import, since only the spec generator
loads it.
After warm-up on a production build under --max-old-space-size=524, the
used heap drops by about 29 MB, and the schema no longer appears in a heap
snapshot. A structural test refuses a new static value import of the
module outside the registry.
Since v1.39.6 the preview scanner parsed the JSON on its way into the
pieces, kept every section but the three bulk tables, and validated the
whole document at the end: a second copy of the record beside the
writer's own, inside the weekly job. On an account with a document vault
that is most of the record.
The writer now shows each section and each bulk row to an observer just
before writing it. The preview collects key uses from that and takes its
counts from the writer's own FullBackupCounts; nothing is parsed or kept.
An upload hands over the summary and key uses it has already validated.
The round trip pins the writer's counts and key uses to a full read of
the copy on an account with a row in every model, and a memory test holds
what the pass keeps once the producer is done.
Measured on 70,000 measurements, 3,000 mood entries and 40 one-megabyte
documents under --max-old-space-size=524: held after the producer
56 MB before, 3 MB after; peak heapUsed 398 MB before, 334 MB after.
A text column written before key ids existed holds the bare base64 of
nonce, tag and ciphertext. The key check read such a value as an encoded
Bytes column, decoded it once too often, and accused the v1 key of not
opening a copy it opens. The instance's stored AI key has exactly that
shape and is the shortest v1 value in every disaster-recovery copy, so it
was picked as the probe and refused every restore of an account with v1
values, even with the instance settings left out of the verdict.
A legacy value now reads as a string unless its decoded head is base64
text. Samples are kept per section and only the sections the verdict
counts are probed; a key is unreadable when none of its probes opens.
Binary values open with the label their column is sealed under, taken
from the encrypted-column registry, and are not probed when no label is
known. A preview stored by an earlier release is still read, and its
sample's layout is derived again from the value.
Reasoning models on Ollama and similar servers think by default and can
spend the whole max_tokens budget before the visible answer starts, so
the call comes back empty or times out. The Local and OpenAI-compatible
chain entries now carry an optional reasoning setting: Default sends
nothing, Off sends reasoning_effort "none", Low / Medium / High send
their own value. It is stored on the entry in aiProviderChain, so a
chain can mix a thinking model with one that does not know the field.
- Every exported resolver binds the entry's value onto the client, the
connection test included, keyed on the instance type so the operator
provider, OpenAI, Anthropic and Codex never carry it.
- PATCH /api/user/ai-provider accepts localReasoningEffort and
compatReasoningEffort and GET reports them. A provider absent from
the chain holds its value on a disabled entry; restating Default on a
chain that was never customised writes nothing.
- PUT /api/insights/provider-chain accepts reasoningEffort on local and
openai-compatible entries, refuses it on any other, and keeps the
stored value when the body leaves it out. Its schema is now shared
with the OpenAPI registry.
- A Default request body is byte-identical to the one sent before.
Refs #1126
A portable file carried the day-log note, the sensitive-category envelope
and custom symptom labels as the stored ciphertext, so a restore onto a
host with another key either refused the file or wrote values nobody
could open. The portable file now carries them readable and the restore
seals them under the receiving host's key; a disaster-recovery file keeps
the ciphertext.
A file written before this still restores: each ciphertext value is
opened with this host's keys first, and one that does not open is kept
out of the row and reported in the skip summary as cycleCiphertext. The
key preflight leaves the cycle sections to that per-value check.
A medication taken now and then (#1024) keeps one row and gains course
rows: calendar spans, both ends inclusive, an optional encrypted note, no
schedule or dose of their own. Courses never overlap and every course but
the latest has ended, so the latest course is always the one the
medication's startsOn/endsOn project. src/lib/medications/courses.ts is
the only writer of those two columns for a medication with courses; it
writes under a per-medication advisory lock and re-projects in the same
transaction, so reminders, the next-due line and intakeActionable keep
reading the columns and stay right.
- Migration 0368 adds medication_courses and backfills one course per
medication window (an end-only window starts on the creation day, or
on its end day when that lies earlier); a medication without a window
stays course-less. Idempotent, data-preserving.
- GET/POST /api/medications/{id}/courses and PATCH/DELETE
.../courses/{courseId} (owner-only); create seeds the first course
from a window; the medication PUT edits the latest course and refuses
a window that would break the rules (medications.course.*).
- List, detail, create and update carry courses, courseCount,
previousCourseEndedOn and canStartCourse, resolved on the server.
- Compliance expands over the span of all courses and drops the slots
between them; every compliance read selects the courses. A rolling
cadence is not anchored by an intake from a previous course.
- The card names the last course's end and the running course's number;
the detail Lifecycle group lists the courses with start, edit and
delete. The Coach compliance block, the MCP resource, the doctor
report window and table, the clinician view and FHIR (one
MedicationStatement per course with effectivePeriod and status) read
the courses.
- Backup carries courses per medication and the restore writes them and
re-projects; an older file derives the one course its window
describes. Wipe plan, encrypted columns, rotation and OpenAPI follow.
A medication can be filed under a category the person named, next to the
built-in ones (#1041). The labels live in medication_category_labels,
encrypted at rest, capped at 20 per account; a medication filed under one
carries its custom:<uuid> key in the category column.
- GET/POST /api/medications/categories (POST idempotent, 30/min per actor,
admitted at MANAGE like the cycle symptom vocabulary) and owner-only
PATCH/DELETE /api/medications/categories/{key}. DELETE moves every
medication filed under the category to OTHER in the same transaction
and answers movedCount.
- category is widened to a built-in value or custom:<uuid> on the wire,
the create and update bodies refuse a key that is not the caller's
(422 medications.category.unknown), and list, detail, the insights
medication tiles and the MCP medications resource carry categoryLabel.
A key with no owned label reads as OTHER and is counted on the event.
- The wizard's class step lists the person's categories under the
built-in rows with an inline add row; the list page gains a category
filter (session only) and a manage sheet for rename, hide and delete.
- Backup carries customMedicationCategories (decrypted labels) and the
restore writes them before the medications; wipe plan, encrypted
column registry, rotation script, sharing guards and OpenAPI follow.
A symptom of one's own ("aura", "headache") is a definition with an
encrypted name and an icon from the cycle custom-symptom allowlist; each
occurrence carries a time, a 0-10 intensity (the pain score's scale), an
optional encrypted note and an optional link to one of the record's open
illness episodes. Both ride the illness module: every route answers
illness.disabled when it is off.
- Migration 0369 (idempotent): symptom_definitions and symptom_events,
intensity CHECK 0-10, episode link ON DELETE SET NULL.
- /api/symptoms/definitions{,/[id]} and /api/symptoms/events{,/[id]}:
eight active definitions per record (a hidden one frees its slot,
showing it again re-checks), hide or purge, events create/list/edit/
delete, a new link must name an open episode of the same record
(422 symptoms.episode.notOpen). Error group symptoms.*.
- Sharing: reads at READ, logging an occurrence at WRITE, defining and
editing at MANAGE; C1/C3/C4 audit details; frozen lists, the handler
matrix and the activity verbs updated.
- Both encrypted columns registered and rotated; both models in the wipe
plan; both carried at both backup ends with occurrences nested in their
definition, plaintext in a portable file and ciphertext in a recovery
file, and a link to an episode the file lacks dropped and named
(symptomEpisodeReference).
- OpenAPI regenerated.
The clinical category lived in a side table the app created at runtime,
after migration 0004 had dropped it as orphaned. It is now the
MedicationCategoryAssignment model. Migration 0368 creates the table
IF NOT EXISTS with the runtime DDL, so an existing table converges with
its rows untouched (verified with prisma migrate diff against a database
holding the runtime table and against a fresh one).
The helpers keep their names and run on the Prisma delegate; the runtime
CREATE TABLE and the pre-transaction ensure call in the restore are gone.
The backup plan judges the table as a schema model (backed up, two-ended,
seeded in the round trip) instead of under RAW_SQL_TABLES, which is now
empty; its guard proves its matcher on a fixture string instead.
Both backup purposes now carry an accountSettings section. The restore writes
it to the account being restored inside the same transaction, names every
refused value under the new accountSetting skip catalogue, and re-warms the
stored texts after the transaction when a unit preference changed, the way the
unit settings routes do. A file without the section leaves the account row as
it is, which is how every older backup restores. The instance settings
restore stays its own opt-in.
Every column of the account row now has a backup verdict in
USER_COLUMN_BACKUP_CLASS: SETTING, CREDENTIAL, IDENTITY or OPERATIONAL. A
schema-derived guard fails on an unclassified column, and the setting columns
each get a codec in account-settings-backup.ts, so a column classified as a
setting without a way to travel does not compile.
The restore end checks each value the way the settings route that writes it
does: AI endpoints against this host's private-origin policy, languages, units
and time zones against the known vocabularies, threshold bands against their
bounds, the avatar against the upload's sniff and limits. A refused value is
left out and named; credentials, identity and the host's own cursors are never
admitted.
The restore dialog asked GET /api/admin/backups/{id}/summary for the
contents of a copy, and the route read the whole stored copy to answer:
every piece opened, every measurement parsed, the rest validated. On an
account of 1.8 million readings on slow hardware that took about two
minutes. The dialog stopped waiting after 15 s and said the contents
could not be read, and its automatic retry made the server read the
whole copy a second time.
The copy passes through the process once anyway while it is written, so
the preview is worked out there and kept on the row (DataBackup.preview,
migration 0364): the summarizeBackup counts, the schema version, and
the inner key ids with the sections they sit in and one sample value
each. Every creation path goes through storeBackupBlob (the weekly and
manual pass, and an upload), so every new copy carries one. The column
is plain JSON: counts and dates only, and the samples are ciphertext
that stands for what is inside the copy, which a key rotation must not
re-seal.
The route answers from it without opening the copy. Both verdicts are
taken again on every read, from what the preview keeps: whether this
release restores the schema version, and whether this server holds the
keys the copy needs. The pieces are still counted against the row, so
a copy that lost one is refused as the restore and the download refuse
it. A preview names the copy it describes and is ignored once the copy
changes; converting a single stored value into pieces carries it over.
A copy written before this has no preview. It is read whole once, as
before, and the result is kept on the row, so it is slow only the first
time; concurrent requests for the same copy share the one read.
Only a file that fails the schema answers 422 "Backup payload failed
schema validation". Read refusals keep their codes; anything else is
the server failing and answers 500.
The dialog no longer retries the preview, waits up to 55 s, and when it
stops waiting on an older copy it says the copy is still being read and
to open the dialog again in a few minutes.
Refs #1031
The clinical category a person picks for a medication lives in a side table
created at runtime, outside the schema. The backup builder and the restore
only walk schema models, so no backup ever carried it: after a restore every
medication read as "Other".
Both backup formats now carry the category inside its medication, and the
restore writes it back in the same transaction as the medication. A file
from before this change restores as before; a category this server does not
know reads as Other instead of refusing the file.
The backup plan gains a list for tables created with raw SQL, and a check
that every such table in the source either maps to a schema model or has a
verdict there, so the next one cannot drop out of backups unnoticed.
A restore writes rows back with their original updatedAt, behind every
paired client's watermarks, and removes rows without leaving tombstones.
A client that kept its cursor would miss both on its next incremental
pull. The same holds after the person deletes all their data.
`users.sync_reset_at` (migration 0363) records when the record was last
replaced: the restore stamps it as its last write and again right after
the commit, the per-account and the admin wipe stamp it inside their
transactions. Cursors now carry the time they were issued, and
/api/sync/changes answers its existing `cursorExpired` to a cursor issued
before the reset, or one without an issue time on an account that has
one. The comparison is on the issue time, because after a restore a
freshly drained cursor still carries old watermarks.
Sealing the practitioner's address left the appointment arm reading the old
readable column, which is now empty, so a booked visit's push lost the address
on booking, rescheduling and restoring a deleted visit.
The reminder no longer keeps its own copy of the address. The reminder tick
resolves it from the owning visit's practitioner when it builds the push and
decrypts it there, so an edit of the address also reaches a nudge that is
already booked. MeasurementReminder.location stays NULL on appointment rows:
booking writes none, a re-anchor clears one an earlier release wrote, and a
backup neither carries nor restores one. A checkup's own free-text location is
unchanged.
A restore of an account with 1.9 million readings, most of them deleted
by the nightly consolidation and still awaiting the purge, was cancelled
in its clearing step: `DELETE FROM measurements` ran past the 60 s
statement timeout every pooled connection carries, and the whole restore
rolled back with a message that named no cause (#1031).
Most of that statement was not the delete. Two columns point into
`measurements` with ON DELETE SET NULL and had no index, so Postgres
scanned `personal_records` and `ecg_recordings` once per deleted reading:
38 of 40 s on a seeded account with 300 personal records. Migration 0362
indexes both, which also speeds up the tombstone purge and account
deletion.
The restore transaction now sets its own statement and idle-in-transaction
limits to the transaction budget with SET LOCAL, and the budget counts
the rows the account holds as well as the rows the file carries. A
rolled-back restore names what the database refused it over (timeout,
lock, constraint, storage, connection, other) on the job row, the
console, the audit row and the wide event.
The shortest value either codec writes is 40 base64 characters. A
shorter string under an ...Encrypted member is not ciphertext, and
reading it as a legacy value would have made a restore demand the v1
key for a field that holds none.
Off-host objects are now written as envelope version 4: the header names
the off-host key by fingerprint, and the object's key in the bucket
(<date>/user-<id>.json.enc) is authenticated as GCM associated data, so
a copy moved to another account or another night no longer opens.
Objects of versions 1 to 3 still read. BACKUP_ENCRYPTION_PREVIOUS_KEYS
(comma-separated, on the compose whitelist and in the env manifest)
keeps retired off-host keys readable, which is the rotation path the
off-host key did not have.
The admin backup upload counted the bytes that arrived and none after
the gunzip, so a small gzip could expand without limit into the JSON
scanner and take the app down. The inflated bytes are now held to 100
times the compressed bytes read (at least 256 MB, at most 4 GB) and a
file past that is refused with 413 before anything is stored.
Sealing a readable Coach title or custom-metric note while building a
disaster-recovery file gave each export a fresh IV, so the streaming
writer and the materialising builder no longer produced the same file.
A row the free-text backfill has not reached yet now rides as stored,
readable, inside the encrypted backup; the restore seals it, and a
migrated row carries its ciphertext verbatim as before.
The title of a Coach conversation is the first 80 characters of the
opening message, and the note on a custom-metric reading is free text;
both were stored readable while every other note was encrypted.
Migration 0357 adds coach_conversations.title_encrypted and
custom_metric_entries.note_encrypted (the title loses its NOT NULL) plus
partial indexes on the backfill's discovery predicate. Every writer now
stores ciphertext only: conversation create, proactive nudge, reminder
surfacing, rename, entry create and edit, and both restore formats.
Readers prefer the ciphertext and fall back to the old column for rows
not yet migrated; a ciphertext that does not open throws instead of
falling back. The history title search decrypts in memory, since SQL can
no longer match the title.
free-text-encryption-backfill is a boot-time, idempotent pg-boss job
(staggered past the startup storm, daily discovery tick) that seals each
remaining readable value and nulls it in one transaction per row. A pass
that sees a row twice stops with an error instead of looping.
Both columns are registered for key rotation (registry, CLI script, and
the in-app rotation that walks the registry). A disaster-recovery backup
carries ciphertext only, sealing a not-yet-migrated row on the way out;
a portable export carries readable text. The old columns are marked for
removal in a later release.
A backup carries every encrypted column as the stored ciphertext, and a
restore writes it back verbatim. Key rotation re-seals the stored copy's
envelope but cannot reach inside it, so a copy taken before a rotation
still needs the retired key. Without it the restore went through and
left rows no reader can open.
The restore, the restore preview, the admin upload and the monthly
restore drill now collect the key ids of every inner ciphertext value,
hold them against ENCRYPTION_KEYS and decrypt one value per key. A
missing key, or an id configured with different key material, is
refused before anything is deleted, with the key id named. The restore
status and the preview say which key to put back.
The restore also checks, inside its transaction, that it added no
reference from the account's rows to rows another account owns. The
check is derived from the schema's foreign keys rather than written per
section, so the tag and category upserts keyed on a global key are
covered along with every link table. The illness parent update is
scoped to the owner as well.
The off-host listing now follows every page: the first page held only
the oldest dates, so the drill could read an old object and report the
chain as stale.
Key rotation re-sealed a single-value backup in place, which holds the
stored string, its bytes, the plaintext and the new ciphertext at once; a
copy can be over a hundred megabytes and the in-app rotation walked 25 of
them per page, enough to run a 1 GB container out of memory. Rotation now
converts each such copy into pieces under the active key instead, one row
at a time. The ~hlgcm1. stream of v1.38.6 to v1.39.1 is read a megabyte
at a time with its tag checked at the end, inside one transaction, so a
copy that fails the check is left untouched; the copy keeps its date. The
scan reads only the first characters of each value. Measured on a copy of
a 2.6 million-reading account (126 MB stored): converted in 25 s with the
heap flat at 57 MB under a 200 MB limit, where re-sealing in place ran out
of heap.
The weekly and manual backup leave an account alone while a restore of
its current copy is queued or running, and count it as skipped, not
failed. A reader whose copy is replaced mid-read answers 409
backup_changed, "replaced by a newer one while it was being read", rather
than the undecryptable error; a restore rolls back with the same code.
Pieces are sealed with a fixed associated-data label, so a value the same
key sealed for another purpose never opens as a piece; rotation re-seals
under the same label. A row holding both a single value and pieces, which
only an older release writing after a downgrade produces, is refused with
a message saying so, and the next weekly run replaces it. Rotation treats a
row deleted between its read and its write as gone rather than an error.
A stored backup was one value in data_backups.data. v1.39.1 wrote it a
piece at a time but still joined the pieces into that one value, and one
value has to pass through the app whole, so a copy was capped at a fifth of
the heap: 105 MB in the default 1 GB container. An account with 1.75
million readings is past that and could not be backed up at all (#1031).
A copy is now kept as ordered pieces of about a megabyte in the new
data_backup_chunks table (migration 0352). Each piece is sealed on its own
with AES-256-GCM, and its sealed header names the copy (a random stream id
kept on the row), its position and whether it is the last. A reader checks
every piece before it releases a byte, so a piece that is missing, moved,
altered, taken from another copy, or a copy cut short is refused with
backup.payload.undecryptable before a restore deletes anything, and the
second read during the restore checks each piece again.
The write is one transaction, so the previous copy stays whole and readable
until the new one commits. Pieces are written through the model API: a raw
query with a large parameter is kept alive by Prisma's query plan cache,
which left 160 MB of heap behind per hundred one-megabyte inserts.
Restore, preview, download, the restore job and the admin list read both
forms; copies written before this change keep their single value and still
restore. The heap-derived limit is gone. The one limit left is on storage,
BACKUP_MAX_STORED_MB (default 2048), and its message names the setting. Key
rotation covers the pieces, the wipe and account deletion take them with
their backup, and the backup plan lists the table as not carried.
Measured on an account of 2.6 million readings in a 1 GB container with web
and worker in one process: backup 87 s, 97 MiB in 96 pieces, preview 20 s,
download 25 s, restore 264 s with every reading back exactly, peak 567 MiB,
no restart.
A condition (illness episode) can carry where on the body it sits, in the
person's own words and encrypted at rest, plus an optional side (LEFT,
RIGHT, BOTH). Same enum, codec and reasoning as the procedure's site.
- migration 0354: two nullable columns on illness_episodes, reusing the
laterality type from 0349
- create and edit accept bodySite and laterality; on PATCH an absent key
keeps the stored value, so an edit from the shipped iPhone app, which
never sends either field, leaves them untouched
- GET /api/body-sites: the sites the record holds across visits and
conditions, folded after the decrypt and bounded per kind; with site
(and laterality) the visits and conditions there, one hop to their
documents, lab results and visits. Read in the profile section;
conditions only under an illness grant with the module on, and links
into ungranted sections come back as unlabelled placeholders
- the link service reads document and visit links from the condition's
end of the two existing tables
- /checkups gains a Body sites tab; the visit and condition forms share
one body-site field pair that suggests sites already used; a
condition's page links its site into the view
- the doctor report, the clinician page and the MCP doctor-visit summary
name the site on conditions; the Coach illness block carries it
sanitised, and MCP get_illness_recovery returns it fenced
- backup and restore carry both fields in both file kinds; the site is
registered as ciphertext and rotates with the key
- i18n in all seven locales, OpenAPI regenerated
Refs #1025
A per-medication trackIntake flag (migration 0351, default on for every
existing and new row). With it off the dose, dates and schedule stay stored,
but nothing derives a due dose from them: the reminder worker, the manual
reminder check, the today projector, the auto-miss pass, the low-stock pass,
the doses card, the today list, the outstanding-dose badge, compliance and
its heatmap, the doctor report, the Coach, the BP-status gate, insights,
targets, achievements, the efficacy view and the MCP schedule tool all skip
it.
The medication wire serves such a medication with schedules: [] and the
stored rows in an additive recordedSchedules field, so a client that arms
local reminders from schedules stays silent without an update. Switching
tracking off or back on archives a schedule revision at that instant, the
untracked stretch with an empty payload, so it never counts as missed and
the projector does not put a dose from before the switch back on the card.
A structural guard freezes the set of schedule readers: each one filters on
the flag, derives from dueSchedules, shapes the wire with scheduleWireFields,
or is allowlisted with the reason it derives nothing due. The flag travels
through backup, restore and the medication CSV export.
Portable files exported before v1.39.1 carry no creation date for a
medication, so the restore stamped the restore time. The dose history reads
that date as the day the medication began to exist, and every miss before
it, including the day before the restore, dropped out of the record.
When the file has no creation date, the restore now uses the earliest thing
the file records for the medication: its first dose (scheduled, taken or
written), the start of its first schedule revision, its course start, its
first pause, dose change or side effect, and never a date after the restore.
A file that records nothing about the medication keeps the old default.
Restoring a large account was one request: 94 s for 1.25 million readings.
A reverse proxy that cuts a request at 60 s (nginx's default) showed the
operator an error for a restore that went on to finish on the server.
The restore request now checks the typed confirmation, that the backup
exists and that its stored copy opens, admits one job for the account and
answers 202 with its id. The `backup-restore` queue runs the restore and
records its phase, counts and outcome on a `backup_restore_jobs` row
(migration 0347), which `GET /api/admin/backups/restores` and
`GET /api/admin/backups/restores/{id}` read back.
What a restore checks and writes is unchanged. The body moved from the
route into `restoreBackup` (src/lib/export/restore-backup.ts): every
refusal still happens before the first delete and the replacement is still
one transaction, so a failure part-way leaves the account as it was. While
it runs the account keeps showing its current data, and other writes are
neither refused nor held, as before.
- One restore per account: a partial unique index admits one queued or
running job, and a second request gets 409 `backup.restore.active` naming
the running job. A per-account advisory lock inside the transaction holds
even if a worker thought gone is still writing.
- Time budget: the job's expiry is two hours and a file whose transaction
limit would run past three quarters of it is refused before anything is
deleted (`jobDeadline`, beside `jobBudget`).
- Resuming: the job writes a heartbeat every two seconds. A boot sweep
queues a job whose worker stopped again from the start, at most twice,
then fails it as interrupted. The admission check releases an abandoned
job so it never holds the account's slot.
- The staged file is the stored `DataBackup` row, already encrypted at
rest. It stays whatever the outcome, so a retry needs no new upload. The
job records a digest of the copy and refuses one that was replaced while
it waited.
- Failures carry a stable code and a sentence that says what to do.
- The console follows the job: phase and counts while it runs, the outcome
after, and after a reload it finds the job again from the server. A poll
that fails on the network is never shown as a failed restore. A web
process separate from the worker evicts the account's caches when it
first reports the restore finished. The success line goes through the
outcome module, so an empty file is not drawn as a tick.
The model is registered in the backup plan (not carried) and the wipe plan
(wiped with the account); the queue in the maintenance registrar, the
subsystem surface and the job-fact allowlist; the new code in the error
catalogue; the routes in the OpenAPI contract. The tests that pin restore
semantics now drive the real route and the real job through one helper, and
`backup-restore-job.test.ts` pins the asynchronous contract itself.
An account of 1.25 million measurements could not export its data or be
restored from any backup (#1031). Measured in a 1 GB container:
- The full export and the passphrase export built the whole payload and
its JSON string in memory; the app ran out of heap and restarted.
- The weekly backup's JSON is 662 MB, longer than any string V8 can hold,
so `unpackBackupBlob` threw on every read. Restore, download and the
restore preview all answered "could not be decrypted".
- Upload accepted 10 MB, a fraction of a large export.
Every reader and writer of a backup file now streams it:
- `scanBackupJson` reads a backup's bytes once, keeps every section but
the named bulk arrays, and hands their elements out one at a time.
`readStreamedBackup` uses it twice: a first pass that validates each
measurement against the element schema and keeps the rest of the file,
and a second pass that hands the measurements to the writer in batches.
Stored copies and off-host objects are authenticated whole on their
compressed ciphertext before any plaintext is read; only the
decompression is streamed.
- Restore writes the measurements with one `INSERT … FROM unnest` per
batch. Prisma caches a query plan per `createMany` row count, and one
1 000-row plan kept about 145 MB alive, which alone put the restore past
the heap limit. The transaction budget grows with the measurement count.
The awards and ECG sections are handed the ids they reference that were
written, not every written id.
- The export routes write the file into the response as it is produced.
The passphrase export spools its ciphertext to a temporary file because
the HLX1 format puts the tag first; the format is unchanged, so existing
archives open as before.
- Upload accepts the raw body, JSON or gzip, up to the 512 MB request
ceiling, checks it as it arrives and stores it in the same pass; the admin
page compresses the file in the browser before sending it. Multipart stays
for small files.
- Download streams the stored JSON; the preview counts measurements from
the stream. The monthly restore drill and `scripts/restore-backup.ts`
read the off-host object the same way; the script can write `.json.gz`.
On the seeded account in a 1 GB container: export 25 s at 389 MB peak,
passphrase export 25 s, both uploads 19 s, restore of the weekly backup
94 s at 554 MB peak, and export, upload and restore round-trip to exactly
the same rows. The encrypted-column guard now also recognises the raw SQL
statement that writes `DataBackup.data`.
The backup streamed its JSON through gzip and the cipher, but kept the
stored copy as one string and handed that to Postgres in one parameter.
On a seeded account of 1.25 million measurements the copy is 64 MB of
base64, and the pieces, the joined string and the driver's copies of it
grew the process from 309 MB to 946 MB resident. A bare process needed a
heap limit above 400 MB to finish it (the new path finishes under 220 MB).
With the dense retention running beside it in a 1 GB container the
process died of heap exhaustion, and the backup job sat "active" until its
two-hour limit marked it "job timed out" (#1031).
`packBackupBlobInto` hands the stored copy on in pieces of up to 4 MB as
it is produced, and `storeBackupBlob` writes them into a temporary table
on its own transaction and assembles the row with one `string_agg`. The
process never holds the whole copy. The account's previous copy stays in
place until the new one is complete: a failure rolls back to it.
Measured on the seeded account in a 1 GB container, web and worker in one
process: the backup completes in about 30 s with the dense retention
running beside it, where v1.39.0 crashed within 30 s.
The streamed JSON also reports its progress to the job every 50 000 rows.
The operator switches become master, Coach, Daily Briefing, Status
notes and Reading documents. Correlations goes: it gated statistics,
not anything a model writes. Reading documents covers the document
vault, lab scans and medication extraction.
The Coach module's operator availability now follows the Coach
switch, so there is one place to turn the Coach off server-wide. The
migration folds a stored Coach availability of false into the switch,
and gives everyone who hid the Coach the AI analysis opt-out, since
hiding the Coach no longer stops nightly briefing and status notes.
A switch read that fails now fails closed. The insights module is
described as AI analysis in every locale, and the switch guard now
keeps both retired switches out and freezes the callers of the old
operator-only gate.
Nothing recomputes an answer. The module map they produced survives on the
account row, but the map says what is switched on, not that the person told the
flow they take medication daily and have a visit next month — so a restore
without the row opens a blank questionnaire for somebody who already answered
it, and leaves the checklist ordering itself from nothing.
Both ends live in one file, the arrangement the visits and the score already
use, because a reader who greps only the restore route gets a false negative
when the route delegates.
`modulesDerivedAt` is the field that makes this more than history and it is
asserted rather than defaulted: it is the once-only latch on the module
derivation, so a restore that let it default to null would hand the next confirm
permission to re-apply the questionnaire over every module decision taken since,
silently. The round trip seeds it non-null and reads it back.
The file's schema validates the three answer objects loosely and the restore
narrows them through the same fail-soft readers the account payload uses. Being
stricter there would refuse a whole account's restore over one retired chip.
The wipe takes the row for the reason the reset already takes the wizard's own
columns: the record the answers were about is gone, so the answers describe
nothing.
The writer added in v1.38.6 compared the process's live heap usage against
80 % of V8's heap limit and aborted above it. That reading is not the
backup's footprint; it is the whole process's, garbage included. A
long-running app sits at 400 MB of largely collectable heap, so on a 1 GB
container (a 419 MB budget) the weekly pass aborted the first flush of every
account: four out of four in seven seconds, the smallest of them a demo
record whose entire stored copy is 1.2 MB, each told its record was too large
and its operator to buy memory.
Drop the gauge. What the writer holds is bounded by its shape and needs no
reading to prove: the three unbounded tables are pulled a page at a time,
every other section is released as soon as its JSON is in the sink, and at
most one flush of text is pending. The one copy that does grow with the
record is the stored blob, so the bound moves there — counted in the
ciphertext bytes actually produced, against a fifth of this process's heap
limit, which is what two copies of the answer plus a live set have to fit
inside. The message names both numbers.
A pass that wrote nothing for anybody now fails its job instead of reporting
`ok: true` with `backed: 0`, so the failing-queue panel and the backups page
read the same thing an operator would. A pass where some accounts were
written keeps passing with the failures as counts.
Measured under --max-old-space-size=450 from a heap deliberately dirtied
first: 445 000 measurements peak at 28 MB held by the backup, a one-row
record at under 1 MB.
The compression that landed last release was real and was measured in the
wrong place. On a development machine with a multi-gigabyte heap the pass
finished; the container the job runs in is capped at 1 GB, which gives V8
a 524 MB limit, and there the same pass died of memory about fifteen
seconds in. Because the job shares the application process, that was not
one failed backup — it was a restart, and every signed-in session on the
host went with it.
Reproduced before touching anything, under `--max-old-space-size=450`
against a seeded account of 445 000 measurements, 30 000 mood entries and
60 000 intake events: `FATAL ERROR: Reached heap limit` at 34 s. The same
fixture doubled to 890 000 measurements now completes inside a 296 MB
limit, holding 16 MB of live heap where the old writer held 168 MB.
The document is written incrementally now. `buildFullBackupPayload` takes
a `deferBulk` mode that declares the three unbounded tables instead of
reading them, and `streamFullBackupJson` walks the payload in key order,
pulls those tables through page by page, and releases every other section
as soon as its JSON reaches the sink. `packBackupBlobStreaming` feeds that
into gzip and an AES-256-GCM stream, so nothing between the database rows
and the stored value ever exists whole. Both writers go through the same
per-row serialisers, and the round-trip suite pins their output byte for
byte rather than trusting that they agree.
The intake and mood reads are ordered totally now. A keyset cursor needs
it, and a partial order also let two exports of one account differ over a
tie, which every multi-dose day has.
The streamed ciphertext is a new shape at rest, because GCM only produces
its tag once the last block is in and a leading tag is what makes the old
layout unstreamable. It carries a marker no older writer can produce, so
the two families are disjoint by construction rather than by sniffing.
All three stored shapes — plain, compressed, streamed — restore, proved
both ways.
The writer also watches its own live heap and stops at 80 % of V8's
limit. An account too large for its host is now one failed backup,
counted in the run's meta, instead of a restart for everybody on it.
The stored column is the ceiling that is left: `data_backups.data` is a
single `text` value, so the finished artifact has to exist as one string,
and that is the only part of the job still growing with the record. The
operator runbook says so, with the number at which it starts to matter.
The weekly pass built the whole account in memory, serialised it whole,
encrypted the string whole, and handed the result to the database whole.
That is four full copies of the record at the same moment — the object
graph, the JSON, the base64 ciphertext at 1.33x the JSON, and the copy the
driver makes of that parameter. Measured against a seeded account of
445 000 readings, the JSON alone is 242 MB and the run dies of heap
exhaustion; under a modest heap it never reaches the insert. The pass was
not slow, it could not finish.
A health record's JSON is extremely repetitive, so gzip takes that 242 MB
to 14 MB and every copy after it shrinks with it. The same run completes
in about twenty seconds with a peak heap of 434 MB and a stored row of
25 MB instead of 322 MB — which also matters because the copy lives inside
the database it would be needed to restore. Both backup jobs now hand the
serialised form back from a frame of its own, so the payload graph is
released before the compress step allocates anything.
The marker sits inside the ciphertext, so the stored column keeps the one
shape `decrypt()` already understands and rows written before this still
restore. The off-host object gets the same treatment through the version
byte its envelope always carried; version 1 objects still read back, which
both the restore drill and the restore script depend on.
The obvious saving — dropping soft-deleted readings, four rows in five on
a real record — is not taken, and the round-trip test now seeds a
tombstone and asserts it makes the journey. A tombstone inside the
retention window is what tells a synced client the row is gone and what
makes a re-sent sample a no-op; a restore without them brings back every
measurement the account ever deleted on the next device sync.
A lab note or a biomarker context written under a key this instance no
longer holds came out of a portable export as `null` — byte-identical to a
column that was never written. The restore importer then wrote nothing, so
the text was gone and nothing said it had ever been there. The whole loss
fit inside a fail-soft `catch` written for a list view, where returning null
is the right answer.
Both now export a short marker naming what happened, and the marker survives
a restore. The Coach transcript already did this with a placeholder sentence
and the doctor report with a rendered flag; the marker is now one constant
the transcript shares, and its comment says why an empty string will not do.
The unit test that required `note: null` here pinned the loss. It keeps the
half that matters — one bad row must never abort the export — and asserts
the marker instead. The round trip is proven end to end against a real
restore.
A restore's first act is to delete the class of records it is about to
rebuild, so a payload whose documents section is simply absent did not
"restore everything else". It emptied the vault, reported success, and left
the operator to find the hole later.
The whole file is refused instead, above the transaction that wipes the
account, and the refusal names the section and says nothing was changed —
an operator holding the only copy of an account cannot act on "restore
failed". The sections ride the error envelope as meta.errorCode
backup.section.missing, and the admin console holds them in a panel beside
the skip report rather than a toast that outlives nothing.
The manifest is the discriminator, and only the manifest. A portable export
leaves the mental-health screeners and the consent receipts out on purpose
and declares both as omitted; those files still restore. So does a file
whose section is present and empty, which is what an account with no
documents honestly writes, and so does every backup written before the
manifest existed. The integration test proves the distinction with the same
builder and the same deleted keys, differing only in what the manifest said.
The route is deliberately outside the published contract — requireAdmin is
cookie-only — so the OpenAPI document is unchanged.
Marc-André Bombeck