mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-07 04:12:04 +00:00
Marc-André Bombeck
13a9aacf20
The off-host job was said to benefit from the streaming work the weekly in-database backup got, and it did not. Only the payload builder was shared: `runOffhostBackup` called `buildFullBackupJson` for the whole document, `gzipSync` over that whole string, a whole-buffer `createCipheriv` pass over the result, and handed the finished buffer to a single `PutObject`. Four full copies of the record alive at once, on the same arithmetic the weekly pass had already been fixed for. Configuring an S3 target on a long-lived record therefore restarted the container. On an account of 445 000 measurements the JSON alone is 242 MB, and against a 1 GB container's 524 MB heap the run died of heap exhaustion seventeen seconds in — taking every signed-in session with it, because the job shares the app process. The write path is a stream end to end now: `streamFullBackupJson` produces the document a page at a time, gzip and an incremental AES-256-GCM writer consume it as it arrives, and `@aws-sdk/lib-storage` puts it up in two 8 MB parts. What the process holds is fixed by the pipeline's shape rather than by the record. Measured on that same 445 000-measurement account under `--max-old-space-size=450`, with the heap pre-loaded to what a warm server holds: the old path dies with `Ineffective mark-compacts near heap limit`, the new one finishes with a peak 79 MB above its baseline and writes a 9.1 MB object. The stored envelope gains version 3, which moves the GCM tag from in front of the ciphertext to the end. That is the only change the stream required — the tag exists only once the last block is in, so a leading one means the whole object must exist before its first byte can be sent. Nothing about the authentication moves: the tag still covers every ciphertext byte and is verified before a byte of plaintext comes back. Versions 1 and 2 still read, so every object already in a bucket restores through the same reader with no flag to tell them apart. The incremental writer is the crypto module's, split into its byte-level half so the base64 form the in-database blob uses and the raw form an object needs are one implementation under two framings. The off-host key handling is unchanged: a separate `BACKUP_ENCRYPTION_KEY`, 64 hex or 32-byte base64, passed in rather than looked up. A run that uploaded nothing for anybody is a failed job now, not `ok: true` with an empty bucket behind it. Wrong credentials, a missing bucket and an unreachable endpoint all fail every account rather than one, and the target's own sentence rides out as the cause. A run where some account got a copy still succeeds; retrying the whole cohort over one object would re-upload everybody's. The one ceiling left is structural rather than a memory bound, and it is counted rather than discovered: a multipart upload carries 10 000 parts, so an object past 80 GB is refused for that account with a clear message instead of failing halfway through with an SDK error about part numbers. `tests/integration/offhost-backup-streaming-memory.test.ts` pins the budget from both sides — what the streaming uploader holds, and that the materialising path does not fit the same budget on the same fixture in the same process, so the number cannot pass by measuring nothing.