mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-11 05:42:19 +00:00
fix(backup): stream every large section of an uploaded backup
The admin upload streamed only the measurements while it checked a file, so a document-heavy account copy was held whole in memory and an upload of a few hundred megabytes could take the process down. The sections a reader takes an element at a time are now named once (BACKUP_STREAMED_SECTIONS in streamed-backup.ts): measurements, intake events, mood entries, documents and ECG recordings. The upload reads its file through readStreamedBackup like the restore and the preview, and the restore drill and the restore script use the same key set. A behavioural test pushes a 271 MB document-heavy file through the real upload route and fails when the heap grows with the file. The stream guard now asserts that every reader goes through the shared reader or key set and that no file declares its own.
This commit is contained in:
1 parent
27c8533f40
commit
41c6eea2a1
7 files changed
+464
-68
No files matched your search
@@ -28,6 +28,7 @@ import { pipeline } from "node:stream/promises";
|
||||
import { createGzip } from "node:zlib";
|
||||
|
||||
import { scanBackupJson } from "@/lib/export/backup-json-scan";
|
||||
import { BACKUP_STREAM_KEYS } from "@/lib/export/streamed-backup";
|
||||
import {
|
||||
loadOffhostConfig,
|
||||
getS3Client,
|
||||
@@ -115,7 +116,7 @@ async function main() {
|
||||
// Checked first, in one streamed pass that keeps only the small sections,
|
||||
// so nothing is written for an object that is not a backup.
|
||||
const { document, streamedCounts } = await scanBackupJson(source(), {
|
||||
streamKeys: new Set(["measurements", "intakeEvents", "moodEntries"]),
|
||||
streamKeys: BACKUP_STREAM_KEYS,
|
||||
});
|
||||
const body = validateBackupShape({
|
||||
...document,
|
||||
|
||||
Reference in new issue
Block a user