fix(backup): export, upload and restore a large account as streams

An account of 1.25 million measurements could not export its data or be
restored from any backup (#1031). Measured in a 1 GB container:

- The full export and the passphrase export built the whole payload and
  its JSON string in memory; the app ran out of heap and restarted.
- The weekly backup's JSON is 662 MB, longer than any string V8 can hold,
  so `unpackBackupBlob` threw on every read. Restore, download and the
  restore preview all answered "could not be decrypted".
- Upload accepted 10 MB, a fraction of a large export.

Every reader and writer of a backup file now streams it:

- `scanBackupJson` reads a backup's bytes once, keeps every section but
  the named bulk arrays, and hands their elements out one at a time.
  `readStreamedBackup` uses it twice: a first pass that validates each
  measurement against the element schema and keeps the rest of the file,
  and a second pass that hands the measurements to the writer in batches.
  Stored copies and off-host objects are authenticated whole on their
  compressed ciphertext before any plaintext is read; only the
  decompression is streamed.
- Restore writes the measurements with one `INSERT … FROM unnest` per
  batch. Prisma caches a query plan per `createMany` row count, and one
  1 000-row plan kept about 145 MB alive, which alone put the restore past
  the heap limit. The transaction budget grows with the measurement count.
  The awards and ECG sections are handed the ids they reference that were
  written, not every written id.
- The export routes write the file into the response as it is produced.
  The passphrase export spools its ciphertext to a temporary file because
  the HLX1 format puts the tag first; the format is unchanged, so existing
  archives open as before.
- Upload accepts the raw body, JSON or gzip, up to the 512 MB request
  ceiling, checks it as it arrives and stores it in the same pass; the admin
  page compresses the file in the browser before sending it. Multipart stays
  for small files.
- Download streams the stored JSON; the preview counts measurements from
  the stream. The monthly restore drill and `scripts/restore-backup.ts`
  read the off-host object the same way; the script can write `.json.gz`.

On the seeded account in a 1 GB container: export 25 s at 389 MB peak,
passphrase export 25 s, both uploads 19 s, restore of the weekly backup
94 s at 554 MB peak, and export, upload and restore round-trip to exactly
the same rows. The encrypted-column guard now also recognises the raw SQL
statement that writes `DataBackup.data`.

src/app/api/export/__tests__/soft-delete-filter.test.ts

@@ -64,6 +64,29 @@ vi.mock("@/lib/db", () => ({
     documentConditionLink: { findMany: vi.fn() },
     extractedFact: { findMany: vi.fn() },
     ecgRecording: { findMany: vi.fn() },
+    onboardingRecord: { findUnique: vi.fn().mockResolvedValue(null) },
+    environmentTravelLocation: { findMany: vi.fn().mockResolvedValue([]) },
+    environmentContext: { findMany: vi.fn().mockResolvedValue([]) },
+    personalRecord: { findMany: vi.fn().mockResolvedValue([]) },
+    coachReminder: { findMany: vi.fn().mockResolvedValue([]) },
+    coachPlan: { findMany: vi.fn().mockResolvedValue([]) },
+    coachFact: { findMany: vi.fn().mockResolvedValue([]) },
+    coachConversation: { findMany: vi.fn().mockResolvedValue([]) },
+    measurementReminderEvent: { findMany: vi.fn().mockResolvedValue([]) },
+    measurementReminder: { findMany: vi.fn().mockResolvedValue([]) },
+    vaccinationDocumentLink: { findMany: vi.fn().mockResolvedValue([]) },
+    vaccinationRecord: { findMany: vi.fn().mockResolvedValue([]) },
+    encounterConditionLink: { findMany: vi.fn().mockResolvedValue([]) },
+    encounterLabLink: { findMany: vi.fn().mockResolvedValue([]) },
+    encounterDocumentLink: { findMany: vi.fn().mockResolvedValue([]) },
+    encounter: { findMany: vi.fn().mockResolvedValue([]) },
+    practitioner: { findMany: vi.fn().mockResolvedValue([]) },
+    healthScoreRecord: { findMany: vi.fn().mockResolvedValue([]) },
+    intradayCumulativeProfile: { findMany: vi.fn().mockResolvedValue([]) },
+    correlationPattern: { findMany: vi.fn().mockResolvedValue([]) },
+    healthProfileFactRevision: { findMany: vi.fn().mockResolvedValue([]) },
+    customMetric: { findMany: vi.fn().mockResolvedValue([]) },
+    cycleSymptom: { findMany: vi.fn().mockResolvedValue([]) },
   },
 }));
 
@@ -177,6 +200,7 @@ beforeEach(() => {
   );
   vi.mocked(prisma.extractedFact.findMany).mockResolvedValue([] as never);
   vi.mocked(prisma.ecgRecording.findMany).mockResolvedValue([] as never);
+  vi.mocked(prisma.userAchievement.findMany).mockResolvedValue([] as never);
 });
 
 describe("v1.4.41 W-DELETED-2 — soft-delete invisibility", () => {
@@ -192,7 +216,13 @@ describe("v1.4.41 W-DELETED-2 — soft-delete invisibility", () => {
 
   it("/api/export/full-backup scopes the measurement read to deletedAt: null", async () => {
     const { GET } = await import("../full-backup/route");
-    await GET(mkReq("http://localhost/api/export/full-backup"));
+    // The file is written into the response as it is produced, so the reads
+    // happen while the body is consumed. The mocked client does not cover
+    // every section the builder reads, so the body errors part-way (as the
+    // whole response did before it streamed); the reads asserted here have
+    // been issued by then.
+    const res = await GET(mkReq("http://localhost/api/export/full-backup"));
+    await res.text();
     expect(prisma.measurement.findMany).toHaveBeenCalledWith(
       expect.objectContaining({
         where: expect.objectContaining({ deletedAt: null }),
@@ -202,7 +232,10 @@ describe("v1.4.41 W-DELETED-2 — soft-delete invisibility", () => {
 
   it("/api/export/full-backup scopes every v1.28 records domain to userId + deletedAt: null", async () => {
     const { GET } = await import("../full-backup/route");
-    await GET(mkReq("http://localhost/api/export/full-backup"));
+    // The file is written into the response as it is produced, so the reads
+    // happen while the body is consumed.
+    const res = await GET(mkReq("http://localhost/api/export/full-backup"));
+    await res.text();
 
     expect(prisma.labResult.findMany).toHaveBeenCalledWith(
       expect.objectContaining({

src/app/api/export/encrypted/route.ts

@@ -42,13 +42,44 @@ import {
   safeJson,
 } from "@/lib/api-response";
 import { checkRateLimit } from "@/lib/rate-limit";
-import { buildFullBackupPayload } from "@/lib/export/full-backup-payload";
+import { randomBytes } from "node:crypto";
+import { createReadStream } from "node:fs";
+import { readdir, rm, stat } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import type { FullBackupCounts } from "@/lib/export/full-backup-payload";
+import { streamFullBackupJson } from "@/lib/export/full-backup-stream";
 import {
-  encryptArchive,
+  encryptArchiveToFile,
   MIN_EXPORT_PASSPHRASE_LENGTH,
+  type SpooledArchive,
 } from "@/lib/export/passphrase-archive";
+import { streamToResponseBody } from "@/lib/export/response-stream";
 import { NextRequest, NextResponse } from "next/server";
 
+const SPOOL_PREFIX = "healthlog-export-";
+
+/**
+ * Remove spool files an earlier export left behind: a process that died
+ * mid-export never reached its cleanup. An hour is far longer than any export
+ * takes, so nothing still being written or sent is touched.
+ */
+async function removeStaleSpools(): Promise<void> {
+  try {
+    const dir = tmpdir();
+    const cutoff = Date.now() - 60 * 60 * 1000;
+    for (const name of await readdir(dir)) {
+      if (!name.startsWith(SPOOL_PREFIX)) continue;
+      const path = join(dir, name);
+      const info = await stat(path).catch(() => null);
+      if (info && info.mtimeMs < cutoff) await rm(path, { force: true });
+    }
+  } catch {
+    // Housekeeping only; an export must not fail over it.
+  }
+}
+
 const encryptedExportSchema = z
   .object({
     passphrase: z
@@ -89,10 +120,31 @@ export const POST = apiHandler(async (request: NextRequest) => {
   }
   const { passphrase } = parsed.data;
 
-  const { payload, counts } = await buildFullBackupPayload(prisma, user.id);
-
-  // Encrypt the JSON bytes. The passphrase goes no further than the KDF.
-  const archive = await encryptArchive(JSON.stringify(payload), passphrase);
+  // Sealed as it is produced and spooled to a temporary file, then sent: the
+  // format puts the tag in front of the ciphertext, so the archive cannot go
+  // out before its last byte is encrypted, and holding it in memory is what
+  // took the app down on an account of 1.25 million measurements (#1031). Only
+  // ciphertext touches the disk, and the file is removed once it is sent or
+  // the client leaves.
+  let counts: FullBackupCounts | undefined;
+  await removeStaleSpools();
+  const bodyPath = join(
+    tmpdir(),
+    `${SPOOL_PREFIX}${randomBytes(12).toString("hex")}.hlx.part`,
+  );
+  let archive: SpooledArchive;
+  try {
+    archive = await encryptArchiveToFile(
+      async (write) => {
+        counts = await streamFullBackupJson(prisma, user.id, write);
+      },
+      passphrase,
+      bodyPath,
+    );
+  } catch (err) {
+    await rm(bodyPath, { force: true });
+    throw err;
+  }
 
   await auditLog("user.export.encrypted", {
     userId: user.id,
@@ -102,38 +154,36 @@ export const POST = apiHandler(async (request: NextRequest) => {
 
   annotate({
     meta: {
-      export_measurements_count: counts.measurements,
-      export_medications_count: counts.medications,
-      export_intake_count: counts.intakeEvents,
-      export_medication_side_effect_count: counts.medicationSideEffects,
-      export_mood_count: counts.moodEntries,
-      export_cycle_count: counts.cycles,
-      export_cycle_day_log_count: counts.cycleDayLogs,
-      export_lab_result_count: counts.labResults,
-      export_biomarker_count: counts.biomarkers,
-      export_illness_episode_count: counts.illnessEpisodes,
-      export_illness_day_log_count: counts.illnessDayLogs,
-      export_allergy_count: counts.allergies,
-      export_family_history_count: counts.familyHistory,
-      export_workout_count: counts.workouts,
-      export_document_count: counts.documents,
-      export_nutrient_day_count: counts.nutrientDays,
-      export_health_profile_count: counts.healthProfile,
-      export_custom_metric_count: counts.customMetrics,
-      export_custom_metric_entry_count: counts.customMetricEntries,
-      export_intraday_profile_count: counts.intradayProfiles,
+      export_measurements_count: counts?.measurements,
+      export_medications_count: counts?.medications,
+      export_intake_count: counts?.intakeEvents,
+      export_mood_count: counts?.moodEntries,
       export_archive_bytes: archive.byteLength,
     },
   });
 
+  const archiveBody = streamToResponseBody(
+    async (write) => {
+      await write(archive.prefix);
+      for await (const chunk of createReadStream(archive.bodyPath)) {
+        await write(chunk as Buffer);
+      }
+    },
+    {
+      onComplete: () => rm(archive.bodyPath, { force: true }),
+      onError: () => rm(archive.bodyPath, { force: true }),
+    },
+  );
+
   const stamp = new Date().toISOString().slice(0, 10);
   // Return the raw binary archive (NOT the apiSuccess envelope). The file is a
   // self-contained `.hlx` archive openable with the user's passphrase via
   // scripts/decrypt-export.ts.
-  return new NextResponse(new Uint8Array(archive), {
+  return new NextResponse(archiveBody, {
     status: 200,
     headers: {
       "Content-Type": "application/octet-stream",
+      "Content-Length": String(archive.byteLength),
       "Content-Disposition": `attachment; filename="healthlog-backup-${user.id}-${stamp}.hlx"`,
       "Cache-Control": "no-store",
     },

src/app/api/export/full-backup/route.ts

@@ -35,7 +35,9 @@ import { annotate } from "@/lib/logging/context";
 import { auditLog } from "@/lib/auth/audit";
 import { apiError, getClientIp } from "@/lib/api-response";
 import { checkRateLimit } from "@/lib/rate-limit";
-import { buildFullBackupPayload } from "@/lib/export/full-backup-payload";
+import type { FullBackupCounts } from "@/lib/export/full-backup-payload";
+import { streamFullBackupJson } from "@/lib/export/full-backup-stream";
+import { streamToResponseBody } from "@/lib/export/response-stream";
 import { NextRequest, NextResponse } from "next/server";
 
 export const GET = apiHandler(async (request: NextRequest) => {
@@ -47,48 +49,44 @@ export const GET = apiHandler(async (request: NextRequest) => {
     return apiError("Maximum 10 exports per hour", 429);
   }
 
-  // v1.23 — the payload builder is shared with the passphrase-encrypted
-  // export route so both emit the byte-for-byte same shape (the measurement/
-  // medication/mood/cycle domains are restore-compatible; the newer records
-  // domains are export-only for now — see the module doc comment above).
-  const { payload, counts } = await buildFullBackupPayload(prisma, user.id);
-
-  await auditLog("user.export.full-backup", {
-    userId: user.id,
-    ipAddress: getClientIp(request),
-    details: { counts },
-  });
-
-  annotate({
-    meta: {
-      export_measurements_count: counts.measurements,
-      export_medications_count: counts.medications,
-      export_intake_count: counts.intakeEvents,
-      export_medication_side_effect_count: counts.medicationSideEffects,
-      export_mood_count: counts.moodEntries,
-      export_cycle_count: counts.cycles,
-      export_cycle_day_log_count: counts.cycleDayLogs,
-      export_lab_result_count: counts.labResults,
-      export_biomarker_count: counts.biomarkers,
-      export_illness_episode_count: counts.illnessEpisodes,
-      export_illness_day_log_count: counts.illnessDayLogs,
-      export_allergy_count: counts.allergies,
-      export_family_history_count: counts.familyHistory,
-      export_workout_count: counts.workouts,
-      export_document_count: counts.documents,
-      export_nutrient_day_count: counts.nutrientDays,
-      export_health_profile_count: counts.healthProfile,
-      export_custom_metric_count: counts.customMetrics,
-      export_custom_metric_entry_count: counts.customMetricEntries,
-      export_intraday_profile_count: counts.intradayProfiles,
+  // Written into the response as it is produced (`streamFullBackupJson`), not
+  // built and stringified first: for an account of 1.25 million measurements
+  // the built payload and its JSON string did not fit a 1 GB container, and
+  // the export took the whole app down (#1031). The file is byte for byte
+  // what `JSON.stringify` of the payload builder's object gives; the passphrase
+  // export beside this route writes the same bytes, encrypted.
+  //
+  // The counts are only known once the last row has gone out, so the audit
+  // row is written then. A file that stops early is audited as a failure.
+  const ipAddress = getClientIp(request);
+  let counts: FullBackupCounts | undefined;
+  const body = streamToResponseBody(
+    async (write) => {
+      counts = await streamFullBackupJson(prisma, user.id, write);
     },
-  });
+    {
+      onComplete: () =>
+        auditLog("user.export.full-backup", {
+          userId: user.id,
+          ipAddress,
+          details: { counts },
+        }),
+      onError: (err) =>
+        auditLog("user.export.full-backup.failed", {
+          userId: user.id,
+          ipAddress,
+          details: {
+            reason: err instanceof Error ? err.message : String(err),
+          },
+        }),
+    },
+  );
 
   const stamp = new Date().toISOString().slice(0, 10);
   // Stream the JSON directly (NOT wrapped in the apiSuccess envelope) so
   // the file is a self-contained backup — admin upload + restore expect
   // the raw payload, not `{ data: { ... } }`.
-  return new NextResponse(JSON.stringify(payload), {
+  return new NextResponse(body, {
     status: 200,
     headers: {
       "Content-Type": "application/json; charset=utf-8",