fix(backup): export, upload and restore a large account as streams

An account of 1.25 million measurements could not export its data or be
restored from any backup (#1031). Measured in a 1 GB container:

- The full export and the passphrase export built the whole payload and
  its JSON string in memory; the app ran out of heap and restarted.
- The weekly backup's JSON is 662 MB, longer than any string V8 can hold,
  so `unpackBackupBlob` threw on every read. Restore, download and the
  restore preview all answered "could not be decrypted".
- Upload accepted 10 MB, a fraction of a large export.

Every reader and writer of a backup file now streams it:

- `scanBackupJson` reads a backup's bytes once, keeps every section but
  the named bulk arrays, and hands their elements out one at a time.
  `readStreamedBackup` uses it twice: a first pass that validates each
  measurement against the element schema and keeps the rest of the file,
  and a second pass that hands the measurements to the writer in batches.
  Stored copies and off-host objects are authenticated whole on their
  compressed ciphertext before any plaintext is read; only the
  decompression is streamed.
- Restore writes the measurements with one `INSERT … FROM unnest` per
  batch. Prisma caches a query plan per `createMany` row count, and one
  1 000-row plan kept about 145 MB alive, which alone put the restore past
  the heap limit. The transaction budget grows with the measurement count.
  The awards and ECG sections are handed the ids they reference that were
  written, not every written id.
- The export routes write the file into the response as it is produced.
  The passphrase export spools its ciphertext to a temporary file because
  the HLX1 format puts the tag first; the format is unchanged, so existing
  archives open as before.
- Upload accepts the raw body, JSON or gzip, up to the 512 MB request
  ceiling, checks it as it arrives and stores it in the same pass; the admin
  page compresses the file in the browser before sending it. Multipart stays
  for small files.
- Download streams the stored JSON; the preview counts measurements from
  the stream. The monthly restore drill and `scripts/restore-backup.ts`
  read the off-host object the same way; the script can write `.json.gz`.

On the seeded account in a 1 GB container: export 25 s at 389 MB peak,
passphrase export 25 s, both uploads 19 s, restore of the weekly backup
94 s at 554 MB peak, and export, upload and restore round-trip to exactly
the same rows. The encrypted-column guard now also recognises the raw SQL
statement that writes `DataBackup.data`.
This commit is contained in:
Marc-André Bombeck committed 2026-09-25 02:05:19 +02:00
1 parent 79ed62b7f9
commit 237acc0002
38 files changed
+2250 -341

No files matched your search

@@ -130,10 +130,10 @@ describe("BackupsSection — empty state", () => {
const className = helpElement?.match(/\bclass="([^"]*)"/)?.[1] ?? "";
expect(english).toContain(
"JSON file matching the current backup schema. Max 10 MB.",
"A HealthLog backup as .json or .json.gz. The file is compressed before upload; compressed, it may be up to 512 MB.",
);
expect(german).toContain(
"JSON-Datei passend zum aktuellen Backup-Schema. Max. 10 MB.",
"HealthLog-Backup als .json oder .json.gz. Die Datei wird vor dem Hochladen komprimiert; komprimiert darf sie bis zu 512 MB groß sein.",
);
expect(helpElement).toBeDefined();
expect(className.split(/\s+/)).toContain("text-muted-foreground");
+28 -7
View File
@@ -454,6 +454,20 @@ function missingSectionLabel(
* the server may reword, and matching on it would turn a copy edit into a
* silent loss of the panel.
*/
/**
* The upload body for a backup file: the file itself when it is already
* gzip, otherwise the file compressed in the browser. The server accepts
* both; compressing keeps a large export well inside the request size limit.
*/
async function uploadBody(file: File): Promise<Blob> {
const head = new Uint8Array(await file.slice(0, 2).arrayBuffer());
if (head[0] === 0x1f && head[1] === 0x8b) return file;
return new Response(
file.stream().pipeThrough(new CompressionStream("gzip")),
).blob();
}
export function missingSectionsOf(err: unknown): MissingBackupSection[] {
if (!(err instanceof ApiError)) return [];
if (err.meta?.errorCode !== "backup.section.missing") return [];
@@ -697,10 +711,11 @@ export function BackupsSection() {
const fileInputRef = useRef<HTMLInputElement | null>(null);
const upload = useMutation({
mutationFn: async (file: File) => {
const fd = new FormData();
fd.append("file", file);
// apiFetch with a raw init: the multipart body must NOT be JSON
// re-encoded, so the verb helper (which JSON-stringifies) is out.
// Sent as the raw body, compressed on the way unless it already is: the
// server reads it as a stream, and a large export (hundreds of MB of
// JSON) shrinks about tenfold. No request timeout, because a file of
// that size takes longer to check and store than the default allows.
const body = await uploadBody(file);
return apiFetch<{
id: string;
valid: true;
@@ -734,7 +749,12 @@ export function BackupsSection() {
customMetricEntries?: number;
intradayProfiles?: number;
};
}>("/api/admin/backups/upload", { method: "POST", body: fd });
}>("/api/admin/backups/upload", {
method: "POST",
body,
headers: { "Content-Type": "application/gzip" },
signal: null,
});
},
onSuccess: (data) => {
const total =
@@ -818,7 +838,8 @@ export function BackupsSection() {
return apiPost<{ restored: true; skipped?: RestoreSkipSummary }>(
`/api/admin/backups/${row.id}/restore`,
{ confirm: "RESTORE", restoreInstanceSettings },
{ headers: { "Idempotency-Key": idempotencyKey } },
// No request timeout: rebuilding a large account takes minutes.
{ headers: { "Idempotency-Key": idempotencyKey }, signal: null },
);
},
onSuccess: (data) => {
@@ -959,7 +980,7 @@ export function BackupsSection() {
<input
ref={fileInputRef}
type="file"
accept="application/json,.json"
accept="application/json,.json,application/gzip,.gz"
className="sr-only"
onChange={handleFileChange}
aria-label={t("admin.section.backups.uploadButton")}