docs(backup): describe backups stored in pieces and the storage limit

The weekly-backup section still described one stored value capped at a
fifth of the heap and advised more memory. It now says how a copy is stored
and checked, that its size no longer depends on memory, what
BACKUP_MAX_STORED_MB limits and how to raise it, how older copies are read,
and how key rotation handles both forms. It also states that a 512 MB
container is below the app's own working set during a backup.

docs/ops/backup-restore.md

@@ -254,69 +254,78 @@ that account's object being refused, and the reason is in the run's
 ## The weekly in-database backup (`data-backup`)
 
 Separate from the off-host job above, and easy to confuse with it. A second
-pg-boss job writes one `WEEKLY_AUTO` row per user into `data_backups.data` —
-the same JSON document, gzipped and then encrypted under `ENCRYPTION_KEY` /
+pg-boss job writes one `WEEKLY_AUTO` copy per user into the database, the same
+JSON document, gzipped and then encrypted under `ENCRYPTION_KEY` /
 `ENCRYPTION_KEYS`, staying inside the instance. It is what
-`/api/admin/backups/<id>/restore` reads.
+`/api/admin/backups/<id>/restore` reads, and an uploaded backup file is stored
+the same way.
 
-### Container memory
+### How a copy is stored
 
-This is the part that bites. The job runs inside the app process, so V8's heap
-limit is the app's heap limit, and a container capped at 1 GB gives Node a
-524 MB old-space limit by default. A long-lived record is bigger than it looks:
-several hundred thousand measurements serialise to a JSON document of a few
-hundred megabytes, and the writer used to need the object graph and that
-document resident at the same time. On a seeded account of 445 000 measurements
-under a 546 MB limit that is `FATAL ERROR: Reached heap limit` about thirty
-seconds in — and since the job shares the process, the whole instance restarted
-and every signed-in session on it went with it.
+From v1.39.2 a copy is kept as ordered pieces of about a megabyte in
+`data_backup_chunks`, with the list of pieces on its `data_backups` row. The
+writer produces the JSON a page at a time, gzips it as it goes and seals each
+megabyte of gzip output on its own with AES-256-GCM. Reading it back (restore,
+preview, download) takes the pieces one at a time. Nothing on either path holds
+the whole copy, so the size of a copy has nothing to do with the app's memory.
+Measured on an account of 2.6 million readings in the default 1 GB container,
+with the web server and the worker in one process: the backup took 87 seconds
+and stored 97 MB in 96 pieces, the preview 20 seconds, the download of the
+1.4 GB file 25 seconds, and the restore 264 seconds, which gave back every
+reading exactly. The container peaked at 567 MB and did not restart. What the
+backup, preview and download hold on top of the running app stays at a few tens
+of megabytes and does not grow with the account. A 512 MB container is too
+small for the app itself: its heap limit is 259 MB and a backup of any account,
+even one with 130 000 readings, runs it out of heap.
 
-Two things changed, and both matter to an operator:
+Each piece carries, inside its encryption, which copy it belongs to, its
+position, and whether it is the last. Before a restore deletes anything, every
+piece is checked: a piece that is missing, moved, altered, taken from another
+copy, or a copy that stops short of its last piece is refused with
+`backup.payload.undecryptable`, and the account is left as it was.
 
-- **The writer streams.** The three tables that grow without bound —
-  measurements, intake events, mood entries — are read a page at a time and
-  serialised straight into gzip and the cipher, and every other section is
-  released as soon as its JSON exists. On the same fixture doubled to 890 000
-  measurements, the pass completes inside a 296 MB heap limit; before the
-  change, half that record did not fit in 546 MB.
-- **It stops itself, on its own size.** The envelope writer counts the
-  encrypted bytes it has produced for an account and gives up on the account
-  whose stored copy would not fit this process — a fifth of V8's heap limit,
-  which is 105 MB on the 524 MB limit a 1 GB container gets. That backup then
-  fails for that one account, is counted in the run's `users_failed` and
-  `records_oversized` meta, and the pass carries on with everybody else. A
-  memory failure is a failed job rather than a restart for every user on the
-  host.
+The write is one transaction. Until the new copy is complete the previous one
+stays in place and readable; a run that fails halfway leaves it untouched.
 
-If `records_oversized` is non-zero in `job.data_backup`, the answer is more
-memory, not a retry: raise the container's limit, or set
-`NODE_OPTIONS=--max-old-space-size=<MB>` to something under it. The limit is
-derived from the heap limit, so it rises with it.
+Copies written before v1.39.2 are a single value in `data_backups.data` and
+still restore, preview and download as they did. Nothing converts them: the
+next weekly run replaces the weekly copy in the new form, and an uploaded copy
+keeps its old form. Those older copies were limited by the app's memory when
+they were written, and reading one still needs that much.
 
-The check deliberately reads bytes it produced and not the process's heap. A
-version that read the heap shipped in v1.38.6 and compared the whole process's
-live usage — garbage included — against 80 % of the limit. A Next.js server
-that has been up for a week sits at 400 MB of largely collectable heap, so the
-weekly pass aborted every account on the first chunk, including one whose whole
-stored copy is 1.2 MB, and reported success while doing it. If you are on
-v1.38.6, a `data-backup` run that finishes in seconds having backed up nothing
-is that defect and not your record.
+### The size limit
 
-### The stored column is the remaining ceiling
+The one limit left is on storage: a copy of one account may take at most
+`BACKUP_MAX_STORED_MB` megabytes (default 2048). It exists so a runaway copy
+fails with a message instead of filling the database volume, which holds the
+old and the new copy side by side while a backup runs. An account past it fails
+for that account alone, is counted in the run's `users_failed` and
+`records_oversized` meta, and keeps its previous copy; the pass carries on with
+everybody else. The message names the setting. Raise it in `.env`, recreate the
+app container, and make sure the database volume has room for two copies of
+that size.
 
-`data_backups.data` is a single `text` column, so the finished artifact has to
-exist as one value before it can be written — that copy cannot be streamed
-away. It is the only thing left in the job that grows with the record: at a
-compressed blob of ~42 MB the pass peaks around 236 MB, and the growth from
-there is roughly twice the blob's size (the base64 answer, plus the driver's
-copy of it on the way to the wire). That is the ceiling the size check is set
-against — a fifth of the heap limit, so two copies plus the process's own live
-set still fit — and an account that reaches it fails as a job with a message
-naming both numbers. The fix at that point is to stop storing the artifact in
-one column: chunk it across rows, or keep only the off-host copy. Reading it back has the same
-shape, and worse: a restore parses the whole document, so the read path needs
-several times the blob in heap. An operator restoring a very large account
-should give the container more memory for the duration.
+Before v1.39.2 the limit was a fifth of the app's heap, 105 MB in a 1 GB
+container, because the copy had to pass through the app as one value. An
+account with 1.75 million readings was past it, and the only way round was more
+memory. That is no longer the case: `records_oversized` means the storage limit,
+never memory.
+
+The writer deliberately reads no heap gauge. A version that did shipped in
+v1.38.6 and compared the whole process's live usage, garbage included, against
+80 % of the limit. A Next.js server that has been up for a week sits at 400 MB
+of largely collectable heap, so the weekly pass aborted every account on the
+first chunk, including one whose whole stored copy is 1.2 MB, and reported
+success while doing it. If you are on v1.38.6, a `data-backup` run that finishes
+in seconds having backed up nothing is that defect and not your record.
+
+### Key rotation
+
+`scripts/rotate-encryption-key.ts` re-seals every piece under the active key,
+in small batches, without reading the backup inside. The link between a piece,
+its copy and its position is inside the encryption, so it comes through
+rotation unchanged. Copies in the older single-value form, including the
+`~hlgcm1.` form v1.39.x wrote, are re-sealed in their own form.
 
 ### What a restore replaces