Key rotation re-sealed a single-value backup in place, which holds the
stored string, its bytes, the plaintext and the new ciphertext at once; a
copy can be over a hundred megabytes and the in-app rotation walked 25 of
them per page, enough to run a 1 GB container out of memory. Rotation now
converts each such copy into pieces under the active key instead, one row
at a time. The ~hlgcm1. stream of v1.38.6 to v1.39.1 is read a megabyte
at a time with its tag checked at the end, inside one transaction, so a
copy that fails the check is left untouched; the copy keeps its date. The
scan reads only the first characters of each value. Measured on a copy of
a 2.6 million-reading account (126 MB stored): converted in 25 s with the
heap flat at 57 MB under a 200 MB limit, where re-sealing in place ran out
of heap.
The weekly and manual backup leave an account alone while a restore of
its current copy is queued or running, and count it as skipped, not
failed. A reader whose copy is replaced mid-read answers 409
backup_changed, "replaced by a newer one while it was being read", rather
than the undecryptable error; a restore rolls back with the same code.
Pieces are sealed with a fixed associated-data label, so a value the same
key sealed for another purpose never opens as a piece; rotation re-seals
under the same label. A row holding both a single value and pieces, which
only an older release writing after a downgrade produces, is refused with
a message saying so, and the next weekly run replaces it. Rotation treats a
row deleted between its read and its write as gone rather than an error.
A stored backup was one value in data_backups.data. v1.39.1 wrote it a
piece at a time but still joined the pieces into that one value, and one
value has to pass through the app whole, so a copy was capped at a fifth of
the heap: 105 MB in the default 1 GB container. An account with 1.75
million readings is past that and could not be backed up at all (#1031).
A copy is now kept as ordered pieces of about a megabyte in the new
data_backup_chunks table (migration 0352). Each piece is sealed on its own
with AES-256-GCM, and its sealed header names the copy (a random stream id
kept on the row), its position and whether it is the last. A reader checks
every piece before it releases a byte, so a piece that is missing, moved,
altered, taken from another copy, or a copy cut short is refused with
backup.payload.undecryptable before a restore deletes anything, and the
second read during the restore checks each piece again.
The write is one transaction, so the previous copy stays whole and readable
until the new one commits. Pieces are written through the model API: a raw
query with a large parameter is kept alive by Prisma's query plan cache,
which left 160 MB of heap behind per hundred one-megabyte inserts.
Restore, preview, download, the restore job and the admin list read both
forms; copies written before this change keep their single value and still
restore. The heap-derived limit is gone. The one limit left is on storage,
BACKUP_MAX_STORED_MB (default 2048), and its message names the setting. Key
rotation covers the pieces, the wipe and account deletion take them with
their backup, and the backup plan lists the table as not carried.
Measured on an account of 2.6 million readings in a 1 GB container with web
and worker in one process: backup 87 s, 97 MiB in 96 pieces, preview 20 s,
download 25 s, restore 264 s with every reading back exactly, peak 567 MiB,
no restart.
A weekly or uploaded backup written by v1.39.x is a single ~hlgcm1. stream
in data_backups.data. Its key id sits behind the stream marker, which the
string codec's parser does not read, so key rotation filed every such row
as legacy, failed to decrypt it and counted it as an error. On a host that
ran v1.39.x that was every weekly copy, and the rotation never finished
clean.
Rotation now reads the key id of a streamed value and re-seals it as a
stream under the active key, without touching what it holds.
`DataBackup.data` holds every weekly disaster-recovery snapshot and every
uploaded pack, AES-256-GCM at rest like everything else, under a column
called `data`. The rotation registry was keyed on the `*Encrypted` naming
convention, so the script never read a backup, reported zero rows
remaining, and the runbook read that zero as permission to retire the
previous key. Following the documented procedure destroyed every stored
backup — the copy an operator reaches for precisely when nothing else
works.
Register it, and register the idempotent-replay response cache the same
sweep turned up. The backup column walks in bounded batches (one row is a
whole compressed account) and re-seals the ciphertext without reading the
plaintext, so both stored envelopes come back byte-identical and so will
any later one. The cache is marked disposable: a row it cannot read is
dropped rather than counted as an error, because keeping it only
guarantees the next replay gets a body it cannot parse.
Running the script against a seeded database rather than reading it turned
up two more. It died on the mood day-context note, whose table is keyed on
the entry rather than on an `id`, and abandoned every column after it —
the backups included. And the document index's verbatim text rotated
without ever being named in the summary.
Make the naming assumption stop carrying weight. A new guard traces the
encryption helpers through their wrappers, walks every Prisma write
payload, and fails when a column receives ciphertext without being
registered. It asserts an anchored, non-zero match count so a matcher that
stops matching fails instead of passing. A second check pins each model's
primary key against the schema.
The script now ends by naming the registered columns it did NOT walk and
exiting non-zero when any is missing: a zero that means "nothing left" and
a zero that means "never looked" must not print the same. The runbook's
retire-the-key step was rewritten around that signal, and tells an
operator who already rotated on an older release how to get their backups
back.
Marc-André Bombeck