fix(db-backup): keep instances apart in a shared bucket, and no decrypted dump behind

Off-host dumps were named by their time alone. Two instances sharing a bucket
(nothing forbids it, and it was safe for the account objects) overwrote each
other's dump at the same second of the same default schedule, and
latest --from-bucket restored whichever instance's dump was newest. Each
instance now uploads under database/<host of APP_URL>/, the object key (and so
the encryption's associated data) carries it, and the restore takes only this
instance's dumps: it refuses another's and names the instances it found,
unless --instance=<host> or the dump's full key says which one on purpose.

A restore killed outright (out of memory, docker kill) left its .restore- or
.download- directory, a whole database decrypted, in the backups directory,
where nothing removed it. A restore now removes them before it starts and on
SIGINT and SIGTERM, and the nightly retention sweeps any older than twelve
hours.

docs/ops/backup-restore.md

@@ -113,8 +113,9 @@ gzipped and sealed with AES-256-GCM under `BACKUP_ENCRYPTION_KEY`, with the
 object's own name as associated data and the key's id in the header.
 
 The nightly database dump goes into the same bucket, as
-`database/healthlog-<time>-daily.dump.enc` with its manifest beside it, sealed
-the same way. It is the copy that rebuilds a lost server
+`database/<host of APP_URL>/healthlog-<time>-daily.dump.enc` with its manifest
+beside it, sealed the same way. The folder keeps the dumps of instances that
+share a bucket apart. It is the copy that rebuilds a lost server
 ([disaster-recovery.md](disaster-recovery.md)).
 
 **Permissions.** The worker needs `PutObject`, `GetObject`, `ListBucket`,

docs/ops/disaster-recovery.md

@@ -6,11 +6,11 @@ onto a new server, and how the procedure is tested.
 
 ## What is backed up, and where
 
-| Copy                    | When                                                                        | Where                                                               | Kept                            |
-| ----------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------- | ------------------------------- |
-| Nightly dump            | 01:30 Europe/Berlin, the zone every schedule runs in (`DB_BACKUP_SCHEDULE`) | `DB_BACKUP_DIR`, the `backups` volume of the bundled compose file   | the newest 7 (`DB_BACKUP_KEEP`) |
-| Pre-migration dump      | on start, when the new image brings migrations, before them                 | the same directory                                                  | the newest 2                    |
-| Off-host copy of a dump | right after each nightly dump, if the bucket is configured                  | your bucket, under `database/`, sealed with `BACKUP_ENCRYPTION_KEY` | your bucket's lifecycle rule    |
+| Copy                    | When                                                                        | Where                                                                                 | Kept                            |
+| ----------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------- |
+| Nightly dump            | 01:30 Europe/Berlin, the zone every schedule runs in (`DB_BACKUP_SCHEDULE`) | `DB_BACKUP_DIR`, the `backups` volume of the bundled compose file                     | the newest 7 (`DB_BACKUP_KEEP`) |
+| Pre-migration dump      | on start, when the new image brings migrations, before them                 | the same directory                                                                    | the newest 2                    |
+| Off-host copy of a dump | right after each nightly dump, if the bucket is configured                  | your bucket, under `database/<host of APP_URL>/`, sealed with `BACKUP_ENCRYPTION_KEY` | your bucket's lifecycle rule    |
 
 A dump is `pg_dump --format=custom` of the whole database: every account,
 sign-ins, passkeys, sharing grants and managed profiles, documents, the job
@@ -103,9 +103,21 @@ different.
    docker compose run --rm db-restore latest --from-bucket
    ```
 
-   `latest --from-bucket` downloads the newest `database/…dump.enc` whose
+   `latest --from-bucket` downloads the newest dump of this instance whose
    manifest is in the bucket too, opens both with `BACKUP_ENCRYPTION_KEY`
    and restores. Name a dump instead of `latest` to take an older one.
+   Each instance uploads under its own folder, `database/<host of APP_URL>/`,
+   so set `APP_URL` as on the old server. A bucket shared with another
+   instance holds its dumps too: `latest` refuses rather than take one of
+   those, and `list --from-bucket` marks them. To restore one on purpose
+   (after moving to a new domain, the old one), add
+   `--instance=<old host>`, or name the dump by its full key,
+   `database/<host>/healthlog-….dump.enc`.
+
+   The opened copy is a whole database, decrypted, in a scratch directory
+   beside the backups. The restore removes it when it ends, on Ctrl-C and
+   `docker compose stop` too. One left by a restore that was killed outright
+   is removed when the next restore starts, and by the nightly retention.
 
    With the files in hand instead (copied off the host, or downloaded from
    the bucket with your own tools: both `<name>.dump.enc` and

scripts/__tests__/db-restore.test.ts

@@ -18,12 +18,21 @@ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
 const tablesInTarget = vi.fn(async (): Promise<string[]> => []);
 const runTool = vi.fn(async () => "");
 const end = vi.fn(async () => {});
+/** What the stand-in bucket lists under `database/`; downloads are refused. */
+const bucketKeys = vi.fn(async (): Promise<string[]> => []);
+const downloads = vi.fn(async (key: string) => {
+  throw new Error(`no download in this test: ${key}`);
+});
 
 vi.mock("../../src/lib/database-backup/dump-core.mjs", async (original) => ({
   ...(await original<Record<string, unknown>>()),
   connect: async () => ({ query: async () => ({ rows: [] }), end }),
   listUserTables: () => tablesInTarget(),
   runTool: (...args: unknown[]) => runTool(...(args as [])),
+  openBucket: async () => ({
+    list: () => bucketKeys(),
+    download: (key: string) => downloads(key),
+  }),
 }));
 
 import { buildManifest } from "../../src/lib/database-backup/dump-core.mjs";
@@ -153,6 +162,50 @@ describe("db-restore refuses before it writes", () => {
     expect(runTool).not.toHaveBeenCalled();
   });
 
+  // Two instances may share a bucket. `latest --from-bucket` took the newest
+  // dump in it, whichever instance wrote it.
+  it("does not take another instance's dump from a shared bucket unless told", async () => {
+    const bucketEnv = env({
+      APP_URL: "https://health.example.org",
+      BACKUP_S3_ENDPOINT: "https://bucket.example",
+      BACKUP_S3_BUCKET: "shared",
+      BACKUP_S3_ACCESS_KEY: "a",
+      BACKUP_S3_SECRET_KEY: "s",
+      BACKUP_ENCRYPTION_KEY: KEY_B,
+    });
+    const other =
+      "database/staging.example.org/healthlog-20261025T013000Z-daily";
+    bucketKeys.mockResolvedValue([
+      `${other}.dump.enc`,
+      `${other}.manifest.json.enc`,
+    ]);
+    expect(await main(["latest", "--from-bucket"], bucketEnv)).toBe(1);
+    expect(errors.join("\n")).toMatch(
+      /no database dump of this instance \(health\.example\.org\).*holds dumps of: staging\.example\.org.*--instance=/,
+    );
+    expect(downloads).not.toHaveBeenCalled();
+
+    // Told: the other instance's newest dump is fetched.
+    await main(
+      ["latest", "--from-bucket", "--instance=staging.example.org"],
+      bucketEnv,
+    );
+    expect(downloads).toHaveBeenCalledWith(`${other}.manifest.json.enc`);
+  });
+
+  it("removes a decrypted copy a killed restore left behind before it starts", async () => {
+    const { mkdirSync, existsSync } = await import("node:fs");
+    mkdirSync(join(dir, ".restore-abc123"));
+    writeFileSync(join(dir, ".restore-abc123", `${BASE}.dump`), "PGDMP");
+    mkdirSync(join(dir, ".download-def456"));
+    writeDump();
+    tablesInTarget.mockResolvedValue(["public.users"]);
+    await main(["latest"], env());
+    expect(existsSync(join(dir, ".restore-abc123"))).toBe(false);
+    expect(existsSync(join(dir, ".download-def456"))).toBe(false);
+    expect(lines.join("\n")).toMatch(/removed \.restore-abc123/);
+  });
+
   it("lists what is there", async () => {
     writeDump();
     expect(await main(["list"], env())).toBe(0);

scripts/compose-dr-drill.sh

@@ -323,8 +323,9 @@ else
     const [port, base, work] = process.argv.slice(1);
     const bucket = await core.openBucket({ ...core.bucketConfigFromEnv(process.env), endpoint: `http://127.0.0.1:${port}` });
     const name = `${base}.manifest.json.enc`;
-    await bucket.download(core.offhostObjectKey(name), join(work, name));
-    await core.decryptOffhostFile({ inPath: join(work, name), outPath: join(work, "manifest.json"), objectKey: core.offhostObjectKey(name), keys: core.offhostKeysFromEnv(process.env) });
+    const key = core.offhostObjectKey(name, core.instanceIdFromEnv(process.env));
+    await bucket.download(key, join(work, name));
+    await core.decryptOffhostFile({ inPath: join(work, name), outPath: join(work, "manifest.json"), objectKey: key, keys: core.offhostKeysFromEnv(process.env) });
   ' "$BUCKET_PORT" "$BASE" "$WORK") || fail "could not read the manifest from the bucket"
 fi
 node -e '

scripts/db-restore.mjs

@@ -3,7 +3,7 @@
  *
  *   docker compose run --rm db-restore list [directory]
  *   docker compose run --rm db-restore list --from-bucket
- *   docker compose run --rm db-restore latest [--from-bucket]
+ *   docker compose run --rm db-restore latest [--from-bucket] [--instance=<host>]
  *   docker compose run --rm db-restore <file> [--ignore-key-mismatch]
  *
  * Runs in the app image with plain `node` (no `tsx`, no source checkout), as
@@ -14,7 +14,16 @@
  * (`.manifest.json` or `.manifest.json.enc`) beside it. With `--from-bucket`
  * it fetches the pair from the bucket the `BACKUP_S3_*` variables name (the
  * newest for `latest`), which is the restore after the host and its local
- * dumps are gone.
+ * dumps are gone. Only this instance's dumps count there: the ones under
+ * `database/<host of APP_URL>/`. A bucket shared with another instance holds
+ * its dumps too, and `latest` refuses rather than restore one of them; name
+ * the other instance with `--instance=<host>` (after a move to a new
+ * domain, the old one) or the dump by its full key to restore it on purpose.
+ *
+ * A copy opened from the bucket is a whole database, decrypted, in a scratch
+ * directory beside the backups. It is removed when the restore ends, on
+ * Ctrl-C and `docker stop` too; one left by a restore that was killed outright
+ * is removed when the next restore starts, and by the nightly retention.
  *
  * In this order, and it stops at the first thing that is wrong, before it
  * writes anything:
@@ -32,6 +41,7 @@
  *
  * Exit 0 restored (or listed), 1 refused or failed, 2 usage.
  */
+import { rmSync } from "node:fs";
 import { mkdtemp, readFile, rm } from "node:fs/promises";
 import { tmpdir } from "node:os";
 import { basename, dirname, isAbsolute, join } from "node:path";
@@ -62,6 +72,9 @@ import {
   openBucket,
   pickBucketDump,
   OFFHOST_DATABASE_PREFIX,
+  instanceIdFromEnv,
+  listBucketDumps,
+  removeStaleScratch,
 } from "../src/lib/database-backup/dump-core.mjs";
 import { shippedMigrations } from "./pending-migrations.mjs";
 
@@ -70,6 +83,33 @@ const say = (line) => console.log(`db-restore: ${line}`);
 
 class Refusal extends Error {}
 
+/**
+ * The scratch directories this process holds, removed on the way out when a
+ * signal ends it before its own `finally` can.
+ */
+const liveScratch = new Set();
+
+async function makeScratch(dir, prefix) {
+  const path = await mkdtemp(
+    join(isWritableDirectory(dir) ? dir : tmpdir(), prefix),
+  );
+  liveScratch.add(path);
+  return path;
+}
+
+async function dropScratch(path) {
+  liveScratch.delete(path);
+  await rm(path, { recursive: true, force: true });
+}
+
+/** The instance whose bucket dumps count: `--instance=<host>`, else APP_URL's. */
+function bucketInstance(flags, env) {
+  const named = [...flags].find((f) => f.startsWith("--instance="));
+  return named
+    ? named.slice("--instance=".length) || null
+    : instanceIdFromEnv(env);
+}
+
 function mb(bytes) {
   return `${(bytes / 1024 / 1024).toFixed(1)} MB`;
 }
@@ -116,7 +156,7 @@ function listCommand(dir) {
  * The dump and manifest to restore: a local pair, or an encrypted pair
  * opened into `scratch`.
  */
-async function resolveInput(arg, dir, env) {
+async function resolveInput(arg, dir, env, objectKeys = null) {
   if (arg === "latest") {
     const set = listDumpSets(dir).find(
       (s) =>
@@ -163,19 +203,39 @@ async function resolveInput(arg, dir, env) {
       "this is an off-host copy and BACKUP_ENCRYPTION_KEY is not set; it cannot be opened without it",
     );
   }
-  const scratch = await mkdtemp(
-    join(isWritableDirectory(dir) ? dir : tmpdir(), ".restore-"),
-  );
+  const scratch = await makeScratch(dir, ".restore-");
+  // The name the copy had in the bucket is part of what its encryption is
+  // bound to. Fetched from the bucket here, it is known; a file downloaded by
+  // hand was under this instance's folder, or at the top of `database/`.
+  const instance = instanceIdFromEnv(env);
+  const keyCandidates = (fileName, fetched) =>
+    fetched
+      ? [fetched]
+      : [
+          ...(instance ? [offhostObjectKey(fileName, instance)] : []),
+          offhostObjectKey(fileName),
+        ];
+  const open = async (inPath, outPath, candidates) => {
+    let last;
+    for (const objectKey of candidates) {
+      try {
+        await decryptOffhostFile({ inPath, outPath, objectKey, keys });
+        return;
+      } catch (error) {
+        last = error;
+      }
+    }
+    throw last;
+  };
   try {
     let manifest;
     if (await statOrNull(encManifest)) {
       const opened = join(scratch, `${parsed.base}.manifest.json`);
-      await decryptOffhostFile({
-        inPath: encManifest,
-        outPath: opened,
-        objectKey: offhostObjectKey(basename(encManifest)),
-        keys,
-      });
+      await open(
+        encManifest,
+        opened,
+        keyCandidates(basename(encManifest), objectKeys?.manifest),
+      );
       manifest = parseManifest(JSON.parse(await readFile(opened, "utf8")));
     } else if (await statOrNull(plainManifest)) {
       manifest = parseManifest(
@@ -188,15 +248,10 @@ async function resolveInput(arg, dir, env) {
     }
     say(`opening the off-host copy ${name}...`);
     const dumpPath = join(scratch, `${parsed.base}.dump`);
-    await decryptOffhostFile({
-      inPath: path,
-      outPath: dumpPath,
-      objectKey: offhostObjectKey(name),
-      keys,
-    });
+    await open(path, dumpPath, keyCandidates(name, objectKeys?.dump));
     return { dumpPath, manifest, scratch };
   } catch (error) {
-    await rm(scratch, { recursive: true, force: true });
+    await dropScratch(scratch);
     throw error instanceof Refusal ? error : new Refusal(error.message);
   }
 }
@@ -206,7 +261,7 @@ async function resolveInput(arg, dir, env) {
  * a scratch directory: the newest pair for `latest`, the named one otherwise.
  * Answers the downloaded `.dump.enc` and the directory to remove afterwards.
  */
-async function downloadFromBucket(arg, dir, env) {
+async function downloadFromBucket(arg, dir, env, instance) {
   const config = bucketConfigFromEnv(env);
   if (!config) {
     throw new Refusal(
@@ -215,17 +270,34 @@ async function downloadFromBucket(arg, dir, env) {
   }
   const bucket = await openBucket(config);
   const keys = await bucket.list(OFFHOST_DATABASE_PREFIX);
-  const pick = pickBucketDump(keys, arg === "latest" ? null : basename(arg));
+  const pick = pickBucketDump(
+    keys,
+    arg === "latest"
+      ? null
+      : arg.startsWith(OFFHOST_DATABASE_PREFIX)
+        ? arg
+        : basename(arg),
+    instance,
+  );
   if (!pick) {
+    const folder = `${OFFHOST_DATABASE_PREFIX}${instance ? `${instance}/` : ""}`;
+    const others = [
+      ...new Set(
+        listBucketDumps(keys)
+          .filter((c) => c.instance !== instance)
+          .map((c) => c.instance ?? "(no instance, top of database/)"),
+      ),
+    ];
     throw new Refusal(
-      arg === "latest"
-        ? `no database dump with its manifest under ${OFFHOST_DATABASE_PREFIX} in bucket ${config.bucket}`
-        : `${basename(arg)} and its manifest are not both under ${OFFHOST_DATABASE_PREFIX} in bucket ${config.bucket}`,
+      (arg === "latest"
+        ? `no database dump of this instance (${instance ?? "APP_URL is not set"}) with its manifest under ${folder} in bucket ${config.bucket}`
+        : `${basename(arg)} and its manifest are not both under ${folder} in bucket ${config.bucket}`) +
+        (others.length > 0
+          ? `. The bucket holds dumps of: ${others.join(", ")}. To restore one of those on purpose, run again with --instance=<host> or name the dump by its full key.`
+          : ""),
     );
   }
-  const scratch = await mkdtemp(
-    join(isWritableDirectory(dir) ? dir : tmpdir(), ".download-"),
-  );
+  const scratch = await makeScratch(dir, ".download-");
   try {
     say(
       `downloading ${pick.dumpKey} and its manifest from ${config.bucket}...`,
@@ -236,28 +308,36 @@ async function downloadFromBucket(arg, dir, env) {
       join(scratch, basename(pick.manifestKey)),
     );
     await bucket.download(pick.dumpKey, dumpPath);
-    return { dumpPath, scratch };
+    return {
+      dumpPath,
+      scratch,
+      objectKeys: { dump: pick.dumpKey, manifest: pick.manifestKey },
+    };
   } catch (error) {
-    await rm(scratch, { recursive: true, force: true });
+    await dropScratch(scratch);
     throw new Refusal(
       `could not download from the bucket: ${error instanceof Error ? error.message : String(error)}`,
     );
   }
 }
 
-async function listBucketCommand(env) {
+async function listBucketCommand(flags, env) {
   const config = bucketConfigFromEnv(env);
   if (!config)
     throw new Refusal("--from-bucket needs the BACKUP_S3_* variables");
-  const keys = (await (await openBucket(config)).list(OFFHOST_DATABASE_PREFIX))
-    .filter((k) => k.endsWith(".dump.enc"))
-    .sort()
-    .reverse();
-  out(
-    `Dumps in bucket ${config.bucket} under ${OFFHOST_DATABASE_PREFIX}, newest first:`,
+  const instance = bucketInstance(flags, env);
+  const dumps = listBucketDumps(
+    await (await openBucket(config)).list(OFFHOST_DATABASE_PREFIX),
   );
-  for (const key of keys) out(`  ${key}`);
-  if (keys.length === 0) say("none");
+  out(
+    `Dumps in bucket ${config.bucket} under ${OFFHOST_DATABASE_PREFIX}, newest first (this instance: ${instance ?? "APP_URL is not set"}):`,
+  );
+  for (const dump of dumps) {
+    out(
+      `  ${dump.dumpKey}${dump.instance === instance ? "" : "  (another instance)"}`,
+    );
+  }
+  if (dumps.length === 0) say("none");
   return 0;
 }
 
@@ -266,18 +346,35 @@ async function restoreCommand(arg, flags, env) {
   const databaseUrl = env.DATABASE_URL;
   if (!databaseUrl) throw new Refusal("DATABASE_URL is not set");
 
+  // A decrypted copy a killed restore left behind goes before anything else.
+  for (const path of await removeStaleScratch(dir, { olderThanMs: 0 })) {
+    say(
+      `removed ${basename(path)}, left behind by a restore that did not finish`,
+    );
+  }
+
   let input = arg;
   let download = null;
   if (flags.has("--from-bucket")) {
-    download = await downloadFromBucket(arg, dir, env);
+    download = await downloadFromBucket(
+      arg,
+      dir,
+      env,
+      bucketInstance(flags, env),
+    );
     input = download.dumpPath;
   }
   let resolved;
   try {
-    resolved = await resolveInput(input, dir, env);
+    resolved = await resolveInput(
+      input,
+      dir,
+      env,
+      download ? download.objectKeys : null,
+    );
   } finally {
     // The sealed copy is not needed once it is opened (or failed to open).
-    if (download) await rm(download.scratch, { recursive: true, force: true });
+    if (download) await dropScratch(download.scratch);
   }
   const { dumpPath, manifest, scratch } = resolved;
   try {
@@ -380,7 +477,7 @@ async function restoreCommand(arg, flags, env) {
     );
     return 0;
   } finally {
-    if (scratch) await rm(scratch, { recursive: true, force: true });
+    if (scratch) await dropScratch(scratch);
   }
 }
 
@@ -389,13 +486,13 @@ export async function main(argv = process.argv.slice(2), env = process.env) {
   const [command] = argv.filter((a) => !a.startsWith("--"));
   if (!command) {
     console.error(
-      "db-restore: usage: db-restore list [directory] | latest | <file> [--from-bucket] [--ignore-key-mismatch]",
+      "db-restore: usage: db-restore list [directory] | latest | <file> [--from-bucket [--instance=<host>]] [--ignore-key-mismatch]",
     );
     return 2;
   }
   try {
     if (command === "list" && flags.has("--from-bucket")) {
-      return await listBucketCommand(env);
+      return await listBucketCommand(flags, env);
     }
     if (command === "list") {
       const [, other] = argv.filter((a) => !a.startsWith("--"));
@@ -411,5 +508,18 @@ export async function main(argv = process.argv.slice(2), env = process.env) {
 }
 
 if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) {
+  // Ctrl-C or `docker stop` ends the process before its own `finally`: take
+  // the decrypted copy with it.
+  for (const [signal, code] of [
+    ["SIGINT", 130],
+    ["SIGTERM", 143],
+  ]) {
+    process.on(signal, () => {
+      for (const path of liveScratch) {
+        rmSync(path, { recursive: true, force: true });
+      }
+      process.exit(code);
+    });
+  }
   process.exit(await main());
 }

src/lib/database-backup/__tests__/dump-core.test.ts

@@ -52,6 +52,8 @@ import {
   resolveDailyKeep,
   selectForDeletion,
   verifyArchive,
+  instanceIdFromEnv,
+  listBucketDumps,
   type DirectoryProbe,
   type DumpSet,
   type Env,
@@ -784,6 +786,81 @@ describe("finding a dump in the bucket", () => {
   });
 });
 
+describe("dumps of more than one instance in one bucket", () => {
+  it("names the instance by the host of its public URL", () => {
+    expect(instanceIdFromEnv({ APP_URL: "https://Health.Example.org/" })).toBe(
+      "health.example.org",
+    );
+    expect(
+      instanceIdFromEnv({ NEXT_PUBLIC_APP_URL: "http://10.0.0.5:3000" }),
+    ).toBe("10.0.0.5-3000");
+    expect(instanceIdFromEnv({ APP_URL: "not a url" })).toBeNull();
+    expect(instanceIdFromEnv({})).toBeNull();
+    expect(offhostObjectKey("x.dump.enc", "health.example.org")).toBe(
+      "database/health.example.org/x.dump.enc",
+    );
+    expect(offhostObjectKey("x.dump.enc")).toBe("database/x.dump.enc");
+  });
+
+  it("takes the newest dump of this instance, never another's, unless named", () => {
+    const pair = (folder: string, stamp: string) => [
+      `database/${folder}healthlog-${stamp}-daily.dump.enc`,
+      `database/${folder}healthlog-${stamp}-daily.manifest.json.enc`,
+    ];
+    const keys = [
+      ...pair("prod.example/", "20261024T013000Z"),
+      // Newer, and another instance's.
+      ...pair("staging.example/", "20261025T013000Z"),
+      ...pair("", "20261023T013000Z"),
+    ];
+    expect(pickBucketDump(keys, null, "prod.example")?.dumpKey).toBe(
+      "database/prod.example/healthlog-20261024T013000Z-daily.dump.enc",
+    );
+    expect(pickBucketDump(keys, null, "nobody.example")).toBeNull();
+    // The dumps of an instance with no public URL, at the top.
+    expect(pickBucketDump(keys, null, null)?.dumpKey).toBe(
+      "database/healthlog-20261023T013000Z-daily.dump.enc",
+    );
+    // Named by its full key: on purpose.
+    expect(
+      pickBucketDump(
+        keys,
+        "database/staging.example/healthlog-20261025T013000Z-daily.dump.enc",
+        "prod.example",
+      )?.manifestKey,
+    ).toBe(
+      "database/staging.example/healthlog-20261025T013000Z-daily.manifest.json.enc",
+    );
+    expect(listBucketDumps(keys).map((d) => d.instance)).toEqual([
+      "staging.example",
+      "prod.example",
+      null,
+    ]);
+  });
+});
+
+describe("a restore's decrypted scratch", () => {
+  it("is swept by retention once it is older than a restore can run", async () => {
+    const dir = tempDir();
+    const { mkdirSync, existsSync } = await import("node:fs");
+    for (const name of [".restore-old", ".download-old", ".restore-new"]) {
+      mkdirSync(join(dir, name));
+      writeFileSync(join(dir, name, "x.dump"), "PGDMP");
+    }
+    const old = new Date(Date.now() - 13 * 60 * 60 * 1000);
+    utimesSync(join(dir, ".restore-old"), old, old);
+    utimesSync(join(dir, ".download-old"), old, old);
+    mkdirSync(join(dir, "not-ours"));
+    utimesSync(join(dir, "not-ours"), old, old);
+
+    await applyRetention(dir, { dailyKeep: 7 });
+    expect(existsSync(join(dir, ".restore-old"))).toBe(false);
+    expect(existsSync(join(dir, ".download-old"))).toBe(false);
+    expect(existsSync(join(dir, ".restore-new"))).toBe(true);
+    expect(existsSync(join(dir, "not-ours"))).toBe(true);
+  });
+});
+
 describe("whether the dump directory survives the container", () => {
   // /proc/self/mountinfo of the app container under the bundled compose
   // file, trimmed to the lines that matter, with and without the volume.

src/lib/database-backup/dump-core.mjs

@@ -1201,6 +1201,51 @@ export function selectForDeletion(sets, options) {
   return remove;
 }
 
+/**
+ * The directories a restore opens an off-host copy into (`.restore-…`) and
+ * downloads one into (`.download-…`). What is in them is a whole database,
+ * decrypted. A restore removes its own when it ends; one that was killed
+ * (out of memory, `docker kill`) cannot, so they are swept: by the next
+ * restore before it starts, and by the nightly retention once they are older
+ * than a restore could still be running.
+ */
+export const SCRATCH_DIR_PATTERN = /^\.(?:restore|download)-/;
+
+/**
+ * Remove the restore scratch directories in `dir` older than `olderThanMs`.
+ * Answers the paths removed. Never throws.
+ *
+ * @param {string} dir
+ * @param {{ olderThanMs: number; now?: number }} options
+ */
+export async function removeStaleScratch(dir, options) {
+  const now = options.now ?? Date.now();
+  /** @type {string[]} */
+  const removed = [];
+  let names = [];
+  try {
+    names = readdirSync(dir);
+  } catch {
+    return removed;
+  }
+  for (const name of names) {
+    if (!SCRATCH_DIR_PATTERN.test(name)) continue;
+    const path = join(dir, name);
+    try {
+      const info = statSync(path);
+      if (!info.isDirectory()) continue;
+      if (options.olderThanMs > 0 && now - info.mtimeMs < options.olderThanMs) {
+        continue;
+      }
+      await rm(path, { recursive: true, force: true });
+      removed.push(path);
+    } catch {
+      // Gone already, or not ours to remove.
+    }
+  }
+  return removed;
+}
+
 /**
  * Apply retention to `dir`. Answers the paths removed.
  *
@@ -1208,15 +1253,20 @@ export function selectForDeletion(sets, options) {
  * @param {{ dailyKeep?: number; now?: number }} [options]
  */
 export async function applyRetention(dir, options = {}) {
+  const now = options.now ?? Date.now();
   const doomed = selectForDeletion(listDumpSets(dir), {
     keep: {
       daily: options.dailyKeep ?? DEFAULT_DAILY_KEEP,
       "pre-migration": PRE_MIGRATION_KEEP,
     },
-    now: options.now ?? Date.now(),
+    now,
   });
   for (const path of doomed) await rm(path, { force: true });
-  return doomed;
+  const scratch = await removeStaleScratch(dir, {
+    olderThanMs: STALE_LEFTOVER_MS,
+    now,
+  });
+  return [...doomed, ...scratch];
 }
 
 /** @param {string} path */
@@ -1227,14 +1277,65 @@ export async function readManifestFile(path) {
 // ── Off-host copies ────────────────────────────────────────────────────────
 
 /**
- * The bucket key of an off-host database object. It is also the associated
- * data its encryption is bound to, so a copy cannot be passed off under
- * another name.
+ * Which instance wrote a dump into a shared bucket: the host of its public
+ * URL (`APP_URL`, else `NEXT_PUBLIC_APP_URL`), lowercased, with anything but
+ * letters, digits, dots and dashes as a dash. Null when neither is set.
+ *
+ * A bucket may hold the copies of more than one instance; nothing forbids it,
+ * and it was safe for the per-account objects. Database dumps were named by
+ * their time alone, so two instances dumping at the same second (the default
+ * schedule is the same everywhere) overwrote each other, and `latest
+ * --from-bucket` restored whichever instance's dump was newest.
+ *
+ * @param {Env} env
+ * @returns {string | null}
+ */
+export function instanceIdFromEnv(env = process.env) {
+  for (const name of ["APP_URL", "NEXT_PUBLIC_APP_URL"]) {
+    const raw = (env[name] ?? "").trim();
+    if (!raw) continue;
+    try {
+      const host = new URL(raw).host.toLowerCase().replace(/[^a-z0-9.-]/g, "-");
+      if (host) return host;
+    } catch {
+      // Not a URL; try the next one.
+    }
+  }
+  return null;
+}
+
+/**
+ * The bucket key of an off-host database object: under the instance's own
+ * folder (`database/<instance>/…`), or at the top of `database/` for an
+ * instance with no public URL set. It is also the associated data its
+ * encryption is bound to, so a copy cannot be passed off under another name,
+ * or as another instance's.
  *
  * @param {string} fileName e.g. `healthlog-…-daily.dump.enc`
+ * @param {string | null} [instanceId]
  */
-export function offhostObjectKey(fileName) {
-  return `${OFFHOST_DATABASE_PREFIX}${fileName}`;
+export function offhostObjectKey(fileName, instanceId = null) {
+  return instanceId
+    ? `${OFFHOST_DATABASE_PREFIX}${instanceId}/${fileName}`
+    : `${OFFHOST_DATABASE_PREFIX}${fileName}`;
+}
+
+/**
+ * A database object key read back: the instance folder it is in (null at the
+ * top of `database/`) and its file name, or null for anything else.
+ *
+ * @param {string} key
+ * @returns {{ instance: string | null; fileName: string } | null}
+ */
+export function parseOffhostObjectKey(key) {
+  if (!key.startsWith(OFFHOST_DATABASE_PREFIX)) return null;
+  const rest = key.slice(OFFHOST_DATABASE_PREFIX.length);
+  const parts = rest.split("/");
+  if (parts.length === 1) return { instance: null, fileName: parts[0] };
+  if (parts.length === 2 && parts[0]) {
+    return { instance: parts[0], fileName: parts[1] };
+  }
+  return null;
 }
 
 const ENVELOPE_MAGIC = "HLBK";
@@ -1364,45 +1465,76 @@ export function bucketConfigFromEnv(env = process.env) {
 }
 
 /**
- * The newest database dump in a bucket listing whose manifest is there too,
- * or the one `name` asks for. Pure. `name` may carry the `.enc` suffix or
- * not. Answers both object keys, or null.
+ * Every database dump in a bucket listing whose manifest is there too, with
+ * the instance that wrote it, newest first. Pure.
  *
  * @param {string[]} keys every key under the `database/` prefix
- * @param {string | null} name
- * @returns {{ dumpKey: string; manifestKey: string } | null}
+ * @returns {Array<{ instance: string | null; base: string; timestamp: number; dumpKey: string; manifestKey: string }>}
  */
-export function pickBucketDump(keys, name = null) {
+export function listBucketDumps(keys) {
   const present = new Set(keys);
-  const wanted = name
-    ? name.replace(/\.enc$/, "").replace(/^database\//, "")
-    : null;
-  const candidates = keys
-    .map((key) => ({
-      key,
-      parsed: parseDumpFileName(key.slice(OFFHOST_DATABASE_PREFIX.length)),
-    }))
+  return keys
+    .map((key) => {
+      const where = parseOffhostObjectKey(key);
+      const parsed = where ? parseDumpFileName(where.fileName) : null;
+      return where && parsed ? { where, parsed } : null;
+    })
     .filter(
-      ({ key, parsed }) =>
-        key.startsWith(OFFHOST_DATABASE_PREFIX) &&
-        parsed !== null &&
-        parsed.role === "dump" &&
-        parsed.variant === "encrypted" &&
-        (wanted === null || `${parsed.base}.dump` === wanted),
+      (c) =>
+        c !== null &&
+        c.parsed.role === "dump" &&
+        c.parsed.variant === "encrypted",
     )
-    .map(({ parsed }) => {
-      const p = /** @type {ParsedDumpName} */ (parsed);
+    .map((c) => {
+      const { where, parsed } =
+        /** @type {{ where: { instance: string | null }; parsed: ParsedDumpName }} */ (
+          c
+        );
       return {
-        timestamp: p.timestamp.getTime(),
-        dumpKey: offhostObjectKey(`${p.base}.dump.enc`),
-        manifestKey: offhostObjectKey(`${p.base}.manifest.json.enc`),
+        instance: where.instance,
+        base: parsed.base,
+        timestamp: parsed.timestamp.getTime(),
+        dumpKey: offhostObjectKey(`${parsed.base}.dump.enc`, where.instance),
+        manifestKey: offhostObjectKey(
+          `${parsed.base}.manifest.json.enc`,
+          where.instance,
+        ),
       };
     })
     .filter((c) => present.has(c.manifestKey))
     .sort((a, b) => b.timestamp - a.timestamp);
-  return candidates[0]
-    ? { dumpKey: candidates[0].dumpKey, manifestKey: candidates[0].manifestKey }
-    : null;
+}
+
+/**
+ * The newest database dump of `instance` in a bucket listing whose manifest
+ * is there too, or the one `name` asks for. Pure. `name` may carry the
+ * `.enc` suffix or not; a name with a folder
+ * (`database/<instance>/healthlog-….dump.enc` or `<instance>/…`) names that
+ * instance's dump, which is how an operator restores another instance's dump
+ * on purpose. Answers both object keys, or null.
+ *
+ * @param {string[]} keys every key under the `database/` prefix
+ * @param {string | null} name
+ * @param {string | null} [instance] the instance whose dumps count; null for
+ *   the dumps at the top of `database/`
+ * @returns {{ dumpKey: string; manifestKey: string } | null}
+ */
+export function pickBucketDump(keys, name = null, instance = null) {
+  let wanted = null;
+  let folder = instance;
+  if (name) {
+    const path = name
+      .replace(/\.enc$/, "")
+      .replace(new RegExp(`^${OFFHOST_DATABASE_PREFIX}`), "");
+    const slash = path.lastIndexOf("/");
+    if (slash >= 0) folder = path.slice(0, slash) || null;
+    wanted = path.slice(slash + 1);
+  }
+  const pick = listBucketDumps(keys).find(
+    (c) =>
+      c.instance === folder && (wanted === null || `${c.base}.dump` === wanted),
+  );
+  return pick ? { dumpKey: pick.dumpKey, manifestKey: pick.manifestKey } : null;
 }
 
 /**

src/lib/database-backup/run.ts

@@ -41,6 +41,7 @@ import {
   backupDirectoryStatus,
   createDatabaseDump,
   databaseBackupEnabled,
+  instanceIdFromEnv,
   offhostObjectKey,
   resolveBackupDir,
   resolveDailyKeep,
@@ -93,10 +94,11 @@ async function uploadFile(
   s3: S3Like,
   key: Buffer,
   path: string,
+  instanceId: string | null,
 ): Promise<number> {
   return uploadEncryptedBackup(
     s3,
-    offhostObjectKey(`${basename(path)}.enc`),
+    offhostObjectKey(`${basename(path)}.enc`, instanceId),
     key,
     async (write) => {
       for await (const chunk of createReadStream(path)) {
@@ -196,8 +198,21 @@ export async function runDatabaseBackup(
           const s3 = options.s3 ?? (await getS3Client(offhostConfig));
           // The manifest goes up first: a dump object in the bucket always
           // has the manifest a restore checks it against.
-          await uploadFile(s3, offhostConfig.encryptionKey, dump.manifestPath);
-          await uploadFile(s3, offhostConfig.encryptionKey, dump.dumpPath);
+          // Under this instance's own folder: a bucket may hold the copies
+          // of more than one instance (`instanceIdFromEnv`).
+          const instanceId = instanceIdFromEnv(env);
+          await uploadFile(
+            s3,
+            offhostConfig.encryptionKey,
+            dump.manifestPath,
+            instanceId,
+          );
+          await uploadFile(
+            s3,
+            offhostConfig.encryptionKey,
+            dump.dumpPath,
+            instanceId,
+          );
           report.offhost = "uploaded";
         } catch (err) {
           uploadError = `off-host upload failed: ${reason(err)}`;