mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 12:32:43 +00:00
Marc-André Bombeck
3118cf240c
The compression that landed last release was real and was measured in the wrong place. On a development machine with a multi-gigabyte heap the pass finished; the container the job runs in is capped at 1 GB, which gives V8 a 524 MB limit, and there the same pass died of memory about fifteen seconds in. Because the job shares the application process, that was not one failed backup — it was a restart, and every signed-in session on the host went with it. Reproduced before touching anything, under `--max-old-space-size=450` against a seeded account of 445 000 measurements, 30 000 mood entries and 60 000 intake events: `FATAL ERROR: Reached heap limit` at 34 s. The same fixture doubled to 890 000 measurements now completes inside a 296 MB limit, holding 16 MB of live heap where the old writer held 168 MB. The document is written incrementally now. `buildFullBackupPayload` takes a `deferBulk` mode that declares the three unbounded tables instead of reading them, and `streamFullBackupJson` walks the payload in key order, pulls those tables through page by page, and releases every other section as soon as its JSON reaches the sink. `packBackupBlobStreaming` feeds that into gzip and an AES-256-GCM stream, so nothing between the database rows and the stored value ever exists whole. Both writers go through the same per-row serialisers, and the round-trip suite pins their output byte for byte rather than trusting that they agree. The intake and mood reads are ordered totally now. A keyset cursor needs it, and a partial order also let two exports of one account differ over a tie, which every multi-dose day has. The streamed ciphertext is a new shape at rest, because GCM only produces its tag once the last block is in and a leading tag is what makes the old layout unstreamable. It carries a marker no older writer can produce, so the two families are disjoint by construction rather than by sniffing. All three stored shapes — plain, compressed, streamed — restore, proved both ways. The writer also watches its own live heap and stops at 80 % of V8's limit. An account too large for its host is now one failed backup, counted in the run's meta, instead of a restart for everybody on it. The stored column is the ceiling that is left: `data_backups.data` is a single `text` value, so the finished artifact has to exist as one string, and that is the only part of the job still growing with the record. The operator runbook says so, with the number at which it starts to matter.