Key rotation rewrote every ciphertext column through a plain Prisma update,
in the in-app corpus walk and in the rotation script alike, so a rotation
stamped the run time on every encrypted row: the Coach panel then listed
old conversations as recent, and every client re-pulled measurements and
mood entries whose sync cursor had moved. Both now read each row's
updatedAt and write it back, matching on it, so a row edited since the
read is left for the next run instead of being stamped. The models that
carry the column are listed beside the registry and checked against the
schema, and the updatedAt guard now covers the rotation writes.
pg-boss gives a job fifteen minutes unless the send says otherwise,
then fails it and retries it while the handler keeps running. Several
passes can legitimately need longer and had no protection at all: the
seven nightly status passes and the pre-generation (a provider call per
account), the period-narrative warm, the record-detection cron (every
account's history, every thirty minutes), the key rotation (every
encrypted column), the content-index backfill (up to 200
transcriptions) and the on-host backup.
Each of them now carries an explicit expiry on every send and schedule,
stops between units once three quarters of it is spent (jobBudget), and
runs under a session advisory lock keyed on queue and identity
(lockedPass), so the retry pg-boss starts after the expiry does nothing
beside a run that is still going. The lock is the one the full-history
imports already held, moved into job-lock.ts.
Where a pass stops early it says so (stopped_early) and the next run
continues: the status passes and the narrative warm leave the rest to
the next night, the pre-generation and the narratives walk the stalest
accounts first, the on-host backup now backs up the oldest copies
first, the key rotation reports a partial run in its audit row and
skips rotated rows when run again, and the dose-history import fails
the delivery so its retry resumes from the committed chunk. The
off-host backup, which already stopped on its budget, gets a four-hour
expiry and the lock.
The password-reset runbook tells an operator to run
`docker compose exec app node scripts/reset-password.mjs`, and the
intake-repair runbook runs `healthlog-tsx scripts/repair-intake-anomalies.ts`
inside the container. The runner stage copied nothing from scripts/, so
both failed with "Cannot find module", the reset at the moment an admin
is locked out.
The runner stage now copies both scripts and the Argon2 parameter module
the reset imports, and proves at build time that each reaches its own
first check with every import resolved. A structural test holds every
in-container command in docs/ and the Dockerfile copies in step.
Key rotation is documented the way it works on the published image: the
Rotate now button under Admin > Encryption is the primary path. The
rotation script imports the application graph and runs from a source
checkout of the same release; the runbook, the admin hint in all seven
languages and the decrypt error now say so.
Key rotation re-sealed a single-value backup in place, which holds the
stored string, its bytes, the plaintext and the new ciphertext at once; a
copy can be over a hundred megabytes and the in-app rotation walked 25 of
them per page, enough to run a 1 GB container out of memory. Rotation now
converts each such copy into pieces under the active key instead, one row
at a time. The ~hlgcm1. stream of v1.38.6 to v1.39.1 is read a megabyte
at a time with its tag checked at the end, inside one transaction, so a
copy that fails the check is left untouched; the copy keeps its date. The
scan reads only the first characters of each value. Measured on a copy of
a 2.6 million-reading account (126 MB stored): converted in 25 s with the
heap flat at 57 MB under a 200 MB limit, where re-sealing in place ran out
of heap.
The weekly and manual backup leave an account alone while a restore of
its current copy is queued or running, and count it as skipped, not
failed. A reader whose copy is replaced mid-read answers 409
backup_changed, "replaced by a newer one while it was being read", rather
than the undecryptable error; a restore rolls back with the same code.
Pieces are sealed with a fixed associated-data label, so a value the same
key sealed for another purpose never opens as a piece; rotation re-seals
under the same label. A row holding both a single value and pieces, which
only an older release writing after a downgrade produces, is refused with
a message saying so, and the next weekly run replaces it. Rotation treats a
row deleted between its read and its write as gone rather than an error.
A stored backup was one value in data_backups.data. v1.39.1 wrote it a
piece at a time but still joined the pieces into that one value, and one
value has to pass through the app whole, so a copy was capped at a fifth of
the heap: 105 MB in the default 1 GB container. An account with 1.75
million readings is past that and could not be backed up at all (#1031).
A copy is now kept as ordered pieces of about a megabyte in the new
data_backup_chunks table (migration 0352). Each piece is sealed on its own
with AES-256-GCM, and its sealed header names the copy (a random stream id
kept on the row), its position and whether it is the last. A reader checks
every piece before it releases a byte, so a piece that is missing, moved,
altered, taken from another copy, or a copy cut short is refused with
backup.payload.undecryptable before a restore deletes anything, and the
second read during the restore checks each piece again.
The write is one transaction, so the previous copy stays whole and readable
until the new one commits. Pieces are written through the model API: a raw
query with a large parameter is kept alive by Prisma's query plan cache,
which left 160 MB of heap behind per hundred one-megabyte inserts.
Restore, preview, download, the restore job and the admin list read both
forms; copies written before this change keep their single value and still
restore. The heap-derived limit is gone. The one limit left is on storage,
BACKUP_MAX_STORED_MB (default 2048), and its message names the setting. Key
rotation covers the pieces, the wipe and account deletion take them with
their backup, and the backup plan lists the table as not carried.
Measured on an account of 2.6 million readings in a 1 GB container with web
and worker in one process: backup 87 s, 97 MiB in 96 pieces, preview 20 s,
download 25 s, restore 264 s with every reading back exactly, peak 567 MiB,
no restart.
A weekly or uploaded backup written by v1.39.x is a single ~hlgcm1. stream
in data_backups.data. Its key id sits behind the stream marker, which the
string codec's parser does not read, so key rotation filed every such row
as legacy, failed to decrypt it and counted it as an error. On a host that
ran v1.39.x that was every weekly copy, and the rotation never finished
clean.
Rotation now reads the key id of a streamed value and re-seals it as a
stream under the active key, without touching what it holds.
The corpus walk and the rotation script paged with Prisma's cursor and skip.
Prisma resolves that cursor by reading the page's last row again, so when
that row was hard-deleted before the next page (a backup replaced, a cached
response expired), the next page came back empty and the walk ended early.
The scan then counted fewer rows than the database holds, which could read
as nothing left on the old key while rows it never reached still were.
Both walks now ask for the rows whose id is greater than the last one read,
which needs no row to still exist.
Key rotation and the rotation status read every column that was not
flagged as a blob with one `findMany` over the whole table.
`Measurement.notesEncrypted` and `MoodEntry.noteEncrypted` are such
columns, so rotation loaded every measurement of the instance: 1.25
million rows on one large account (#1031), and the rotation status page
did the same on every view.
Every column is now walked with the id-cursor pages the blob columns
already used: 25 rows a page for blobs, 5 000 for everything else. The
maintenance script's own per-column walks page the same way. On the seeded
account with 1 460 notes to rotate, peak heap fell from 272 MB to 153 MB,
and it no longer grows with the table.
`scanned` now counts the rows that hold ciphertext for every column, as
it already did for the blob columns, rather than every row of the table
for some columns and only the ciphertext rows for others. The runbook
says so.
`DataBackup.data` holds every weekly disaster-recovery snapshot and every
uploaded pack, AES-256-GCM at rest like everything else, under a column
called `data`. The rotation registry was keyed on the `*Encrypted` naming
convention, so the script never read a backup, reported zero rows
remaining, and the runbook read that zero as permission to retire the
previous key. Following the documented procedure destroyed every stored
backup — the copy an operator reaches for precisely when nothing else
works.
Register it, and register the idempotent-replay response cache the same
sweep turned up. The backup column walks in bounded batches (one row is a
whole compressed account) and re-seals the ciphertext without reading the
plaintext, so both stored envelopes come back byte-identical and so will
any later one. The cache is marked disposable: a row it cannot read is
dropped rather than counted as an error, because keeping it only
guarantees the next replay gets a body it cannot parse.
Running the script against a seeded database rather than reading it turned
up two more. It died on the mood day-context note, whose table is keyed on
the entry rather than on an `id`, and abandoned every column after it —
the backups included. And the document index's verbatim text rotated
without ever being named in the summary.
Make the naming assumption stop carrying weight. A new guard traces the
encryption helpers through their wrappers, walks every Prisma write
payload, and fails when a column receives ciphertext without being
registered. It asserts an anchored, non-zero match count so a matcher that
stops matching fails instead of passing. A second check pins each model's
primary key against the schema.
The script now ends by naming the registered columns it did NOT walk and
exiting non-zero when any is missing: a zero that means "nothing left" and
a zero that means "never looked" must not print the same. The runbook's
retire-the-key step was rewritten around that signal, and tells an
operator who already rotated on an older release how to get their backups
back.
Eighteen modules carried a convenience re-export of a symbol defined
elsewhere. In each case the callers the comment describes now import the
concrete module directly, so the line had no importer left. Where the
whole re-export block went, the comment explaining it went with it —
doctor-report-pdf-core, trend-card and medication-extract-prompt each
claimed to be a single import surface that nothing was using.
Definitions are untouched; imports that existed only to feed a removed
re-export go too.
The registry entry for InboundDocument.contentEncrypted gains a codecField
marker: the column now holds two layouts (base64v1 string codec | binary2
binary codec), recorded per row, and its rows are up to cap-sized blobs.
scan/rotate walk such a column with id-cursor pagination (25 rows per
page — never an unbounded blob findMany) and re-encrypt each row under its
OWN codec; an unknown codec value is counted as an error and the row left
untouched, same fail-closed posture as a missing key. The walk is
idempotent, so an interrupted run resumes on re-invocation with the
already-active rows skipped.
Both rotation paths share the logic: the admin-triggered pg-boss job via
rotateCorpus() and the CLI (scripts/rotate-encryption-key.ts), which now
calls the shared batched walk for this column. The registry guard test
keeps schema, registry, and script in lock-step as before.
Unit coverage over a mocked 500-row corpus asserts the batch boundedness
(take + advancing cursor per page), both-codec rotation, per-row
fail-closed on unknown codecs, and interrupted-run resume.
Task: W1-T11.
Add a read-only admin Encryption section that surfaces what the rotation
tooling computes: per encrypted-column rows-per-key-id, legacy-key row
counts, and overall rotation progress, plus a 'safe to drop the legacy
key' state once every column sits on the active key. GET
/api/admin/encryption/status drives it and returns only key ids and
counts — never key material. A shared scan over the canonical encrypted-
column registry backs both this view and the rotation run. Cookie-only
admin.
Marc-André Bombeck