diff --git a/docs/ops/backup-restore.md b/docs/ops/backup-restore.md index 4130a2277..97d614d29 100644 --- a/docs/ops/backup-restore.md +++ b/docs/ops/backup-restore.md @@ -254,69 +254,78 @@ that account's object being refused, and the reason is in the run's ## The weekly in-database backup (`data-backup`) Separate from the off-host job above, and easy to confuse with it. A second -pg-boss job writes one `WEEKLY_AUTO` row per user into `data_backups.data` — -the same JSON document, gzipped and then encrypted under `ENCRYPTION_KEY` / +pg-boss job writes one `WEEKLY_AUTO` copy per user into the database, the same +JSON document, gzipped and then encrypted under `ENCRYPTION_KEY` / `ENCRYPTION_KEYS`, staying inside the instance. It is what -`/api/admin/backups//restore` reads. +`/api/admin/backups//restore` reads, and an uploaded backup file is stored +the same way. -### Container memory +### How a copy is stored -This is the part that bites. The job runs inside the app process, so V8's heap -limit is the app's heap limit, and a container capped at 1 GB gives Node a -524 MB old-space limit by default. A long-lived record is bigger than it looks: -several hundred thousand measurements serialise to a JSON document of a few -hundred megabytes, and the writer used to need the object graph and that -document resident at the same time. On a seeded account of 445 000 measurements -under a 546 MB limit that is `FATAL ERROR: Reached heap limit` about thirty -seconds in — and since the job shares the process, the whole instance restarted -and every signed-in session on it went with it. +From v1.39.2 a copy is kept as ordered pieces of about a megabyte in +`data_backup_chunks`, with the list of pieces on its `data_backups` row. The +writer produces the JSON a page at a time, gzips it as it goes and seals each +megabyte of gzip output on its own with AES-256-GCM. Reading it back (restore, +preview, download) takes the pieces one at a time. Nothing on either path holds +the whole copy, so the size of a copy has nothing to do with the app's memory. +Measured on an account of 2.6 million readings in the default 1 GB container, +with the web server and the worker in one process: the backup took 87 seconds +and stored 97 MB in 96 pieces, the preview 20 seconds, the download of the +1.4 GB file 25 seconds, and the restore 264 seconds, which gave back every +reading exactly. The container peaked at 567 MB and did not restart. What the +backup, preview and download hold on top of the running app stays at a few tens +of megabytes and does not grow with the account. A 512 MB container is too +small for the app itself: its heap limit is 259 MB and a backup of any account, +even one with 130 000 readings, runs it out of heap. -Two things changed, and both matter to an operator: +Each piece carries, inside its encryption, which copy it belongs to, its +position, and whether it is the last. Before a restore deletes anything, every +piece is checked: a piece that is missing, moved, altered, taken from another +copy, or a copy that stops short of its last piece is refused with +`backup.payload.undecryptable`, and the account is left as it was. -- **The writer streams.** The three tables that grow without bound — - measurements, intake events, mood entries — are read a page at a time and - serialised straight into gzip and the cipher, and every other section is - released as soon as its JSON exists. On the same fixture doubled to 890 000 - measurements, the pass completes inside a 296 MB heap limit; before the - change, half that record did not fit in 546 MB. -- **It stops itself, on its own size.** The envelope writer counts the - encrypted bytes it has produced for an account and gives up on the account - whose stored copy would not fit this process — a fifth of V8's heap limit, - which is 105 MB on the 524 MB limit a 1 GB container gets. That backup then - fails for that one account, is counted in the run's `users_failed` and - `records_oversized` meta, and the pass carries on with everybody else. A - memory failure is a failed job rather than a restart for every user on the - host. +The write is one transaction. Until the new copy is complete the previous one +stays in place and readable; a run that fails halfway leaves it untouched. -If `records_oversized` is non-zero in `job.data_backup`, the answer is more -memory, not a retry: raise the container's limit, or set -`NODE_OPTIONS=--max-old-space-size=` to something under it. The limit is -derived from the heap limit, so it rises with it. +Copies written before v1.39.2 are a single value in `data_backups.data` and +still restore, preview and download as they did. Nothing converts them: the +next weekly run replaces the weekly copy in the new form, and an uploaded copy +keeps its old form. Those older copies were limited by the app's memory when +they were written, and reading one still needs that much. -The check deliberately reads bytes it produced and not the process's heap. A -version that read the heap shipped in v1.38.6 and compared the whole process's -live usage — garbage included — against 80 % of the limit. A Next.js server -that has been up for a week sits at 400 MB of largely collectable heap, so the -weekly pass aborted every account on the first chunk, including one whose whole -stored copy is 1.2 MB, and reported success while doing it. If you are on -v1.38.6, a `data-backup` run that finishes in seconds having backed up nothing -is that defect and not your record. +### The size limit -### The stored column is the remaining ceiling +The one limit left is on storage: a copy of one account may take at most +`BACKUP_MAX_STORED_MB` megabytes (default 2048). It exists so a runaway copy +fails with a message instead of filling the database volume, which holds the +old and the new copy side by side while a backup runs. An account past it fails +for that account alone, is counted in the run's `users_failed` and +`records_oversized` meta, and keeps its previous copy; the pass carries on with +everybody else. The message names the setting. Raise it in `.env`, recreate the +app container, and make sure the database volume has room for two copies of +that size. -`data_backups.data` is a single `text` column, so the finished artifact has to -exist as one value before it can be written — that copy cannot be streamed -away. It is the only thing left in the job that grows with the record: at a -compressed blob of ~42 MB the pass peaks around 236 MB, and the growth from -there is roughly twice the blob's size (the base64 answer, plus the driver's -copy of it on the way to the wire). That is the ceiling the size check is set -against — a fifth of the heap limit, so two copies plus the process's own live -set still fit — and an account that reaches it fails as a job with a message -naming both numbers. The fix at that point is to stop storing the artifact in -one column: chunk it across rows, or keep only the off-host copy. Reading it back has the same -shape, and worse: a restore parses the whole document, so the read path needs -several times the blob in heap. An operator restoring a very large account -should give the container more memory for the duration. +Before v1.39.2 the limit was a fifth of the app's heap, 105 MB in a 1 GB +container, because the copy had to pass through the app as one value. An +account with 1.75 million readings was past it, and the only way round was more +memory. That is no longer the case: `records_oversized` means the storage limit, +never memory. + +The writer deliberately reads no heap gauge. A version that did shipped in +v1.38.6 and compared the whole process's live usage, garbage included, against +80 % of the limit. A Next.js server that has been up for a week sits at 400 MB +of largely collectable heap, so the weekly pass aborted every account on the +first chunk, including one whose whole stored copy is 1.2 MB, and reported +success while doing it. If you are on v1.38.6, a `data-backup` run that finishes +in seconds having backed up nothing is that defect and not your record. + +### Key rotation + +`scripts/rotate-encryption-key.ts` re-seals every piece under the active key, +in small batches, without reading the backup inside. The link between a piece, +its copy and its position is inside the encryption, so it comes through +rotation unchanged. Copies in the older single-value form, including the +`~hlgcm1.` form v1.39.x wrote, are re-sealed in their own form. ### What a restore replaces