mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-12 06:06:10 +00:00
fix(backup): stream the weekly backup instead of building it whole
The compression that landed last release was real and was measured in the wrong place. On a development machine with a multi-gigabyte heap the pass finished; the container the job runs in is capped at 1 GB, which gives V8 a 524 MB limit, and there the same pass died of memory about fifteen seconds in. Because the job shares the application process, that was not one failed backup — it was a restart, and every signed-in session on the host went with it. Reproduced before touching anything, under `--max-old-space-size=450` against a seeded account of 445 000 measurements, 30 000 mood entries and 60 000 intake events: `FATAL ERROR: Reached heap limit` at 34 s. The same fixture doubled to 890 000 measurements now completes inside a 296 MB limit, holding 16 MB of live heap where the old writer held 168 MB. The document is written incrementally now. `buildFullBackupPayload` takes a `deferBulk` mode that declares the three unbounded tables instead of reading them, and `streamFullBackupJson` walks the payload in key order, pulls those tables through page by page, and releases every other section as soon as its JSON reaches the sink. `packBackupBlobStreaming` feeds that into gzip and an AES-256-GCM stream, so nothing between the database rows and the stored value ever exists whole. Both writers go through the same per-row serialisers, and the round-trip suite pins their output byte for byte rather than trusting that they agree. The intake and mood reads are ordered totally now. A keyset cursor needs it, and a partial order also let two exports of one account differ over a tie, which every multi-dose day has. The streamed ciphertext is a new shape at rest, because GCM only produces its tag once the last block is in and a leading tag is what makes the old layout unstreamable. It carries a marker no older writer can produce, so the two families are disjoint by construction rather than by sniffing. All three stored shapes — plain, compressed, streamed — restore, proved both ways. The writer also watches its own live heap and stops at 80 % of V8's limit. An account too large for its host is now one failed backup, counted in the run's meta, instead of a restart for everybody on it. The stored column is the ceiling that is left: `data_backups.data` is a single `text` value, so the finished artifact has to exist as one string, and that is the only part of the job still growing with the record. The operator runbook says so, with the number at which it starts to matter.
This commit is contained in:
1 parent
fc1ebeae9e
commit
3118cf240c
14 files changed
+1292
-227
No files matched your search
@@ -122,6 +122,59 @@ The full per-model reasoning lives in `src/lib/export/backup-plan.ts`, where
|
||||
every excluded model carries a written verdict and a structural test refuses to
|
||||
let a new model land without one.
|
||||
|
||||
## The weekly in-database backup (`data-backup`)
|
||||
|
||||
Separate from the off-host job above, and easy to confuse with it. A second
|
||||
pg-boss job writes one `WEEKLY_AUTO` row per user into `data_backups.data` —
|
||||
the same JSON document, gzipped and then encrypted under `ENCRYPTION_KEY` /
|
||||
`ENCRYPTION_KEYS`, staying inside the instance. It is what
|
||||
`/api/admin/backups/<id>/restore` reads.
|
||||
|
||||
### Container memory
|
||||
|
||||
This is the part that bites. The job runs inside the app process, so V8's heap
|
||||
limit is the app's heap limit, and a container capped at 1 GB gives Node a
|
||||
524 MB old-space limit by default. A long-lived record is bigger than it looks:
|
||||
several hundred thousand measurements serialise to a JSON document of a few
|
||||
hundred megabytes, and the writer used to need the object graph and that
|
||||
document resident at the same time. On a seeded account of 445 000 measurements
|
||||
under a 546 MB limit that is `FATAL ERROR: Reached heap limit` about thirty
|
||||
seconds in — and since the job shares the process, the whole instance restarted
|
||||
and every signed-in session on it went with it.
|
||||
|
||||
Two things changed, and both matter to an operator:
|
||||
|
||||
- **The writer streams.** The three tables that grow without bound —
|
||||
measurements, intake events, mood entries — are read a page at a time and
|
||||
serialised straight into gzip and the cipher, and every other section is
|
||||
released as soon as its JSON exists. On the same fixture doubled to 890 000
|
||||
measurements, the pass completes inside a 296 MB heap limit; before the
|
||||
change, half that record did not fit in 546 MB.
|
||||
- **It stops itself.** The writer watches its own live heap and aborts at 80 %
|
||||
of V8's limit. That backup then fails for that one account, is counted in the
|
||||
run's `users_failed` and `heap_budget_trips` meta, and the pass carries on
|
||||
with everybody else. A memory failure is now a failed job rather than a
|
||||
restart for every user on the host.
|
||||
|
||||
If `heap_budget_trips` is non-zero in `job.data_backup`, the answer is more
|
||||
memory, not a retry: raise the container's limit, or set
|
||||
`NODE_OPTIONS=--max-old-space-size=<MB>` to something under it.
|
||||
|
||||
### The stored column is the remaining ceiling
|
||||
|
||||
`data_backups.data` is a single `text` column, so the finished artifact has to
|
||||
exist as one value before it can be written — that copy cannot be streamed
|
||||
away. It is the only thing left in the job that grows with the record: at a
|
||||
compressed blob of ~42 MB the pass peaks around 236 MB, and the growth from
|
||||
there is roughly twice the blob's size (the base64 answer, plus the driver's
|
||||
copy of it on the way to the wire). A blob past roughly 150 MB — an account
|
||||
several times larger than any seen so far — will hit a 524 MB limit again, and
|
||||
the fix at that point is to stop storing the artifact in one column: chunk it
|
||||
across rows, or keep only the off-host copy. Reading it back has the same
|
||||
shape, and worse: a restore parses the whole document, so the read path needs
|
||||
several times the blob in heap. An operator restoring a very large account
|
||||
should give the container more memory for the duration.
|
||||
|
||||
## Monthly restore drill (automatic)
|
||||
|
||||
Since v1.16.4 a pg-boss job (`data-restore-drill`, cron `11 4 1 * *` —
|
||||
|
||||
Reference in new issue
Block a user