fix(backup): stream every large section of an uploaded backup

The admin upload streamed only the measurements while it checked a file,
so a document-heavy account copy was held whole in memory and an upload of
a few hundred megabytes could take the process down.

The sections a reader takes an element at a time are now named once
(BACKUP_STREAMED_SECTIONS in streamed-backup.ts): measurements, intake
events, mood entries, documents and ECG recordings. The upload reads its
file through readStreamedBackup like the restore and the preview, and the
restore drill and the restore script use the same key set.

A behavioural test pushes a 271 MB document-heavy file through the real
upload route and fails when the heap grows with the file. The stream guard
now asserts that every reader goes through the shared reader or key set
and that no file declares its own.
This commit is contained in:
Marc-André Bombeck committed 2026-10-25 12:25:15 +01:00
1 parent 27c8533f40
commit 41c6eea2a1
7 files changed
+464 -68

No files matched your search

+2 -1
View File
@@ -28,6 +28,7 @@ import { pipeline } from "node:stream/promises";
import { createGzip } from "node:zlib";
import { scanBackupJson } from "@/lib/export/backup-json-scan";
import { BACKUP_STREAM_KEYS } from "@/lib/export/streamed-backup";
import {
loadOffhostConfig,
getS3Client,
@@ -115,7 +116,7 @@ async function main() {
// Checked first, in one streamed pass that keeps only the small sections,
// so nothing is written for an object that is not a backup.
const { document, streamedCounts } = await scanBackupJson(source(), {
streamKeys: new Set(["measurements", "intakeEvents", "moodEntries"]),
streamKeys: BACKUP_STREAM_KEYS,
});
const body = validateBackupShape({
...document,