fix(backup): refuse a restore that needs a dropped key or reaches into another account

A backup carries every encrypted column as the stored ciphertext, and a
restore writes it back verbatim. Key rotation re-seals the stored copy's
envelope but cannot reach inside it, so a copy taken before a rotation
still needs the retired key. Without it the restore went through and
left rows no reader can open.

The restore, the restore preview, the admin upload and the monthly
restore drill now collect the key ids of every inner ciphertext value,
hold them against ENCRYPTION_KEYS and decrypt one value per key. A
missing key, or an id configured with different key material, is
refused before anything is deleted, with the key id named. The restore
status and the preview say which key to put back.

The restore also checks, inside its transaction, that it added no
reference from the account's rows to rows another account owns. The
check is derived from the schema's foreign keys rather than written per
section, so the tag and category upserts keyed on a global key are
covered along with every link table. The illness parent update is
scoped to the owner as well.

The off-host listing now follows every page: the first page held only
the oldest dates, so the drill could read an old object and report the
chain as stale.
This commit is contained in:
Marc-André Bombeck committed 2026-09-26 19:22:03 +02:00
1 parent 4af6b87a2a
commit 405c996b6a
24 files changed
+1249 -25

No files matched your search

+4 -1
View File
@@ -7309,7 +7309,10 @@
"restoreFailureNotStarted": "Der Hintergrunddienst hat die Wiederherstellung nicht übernommen. Prüfe, ob er läuft.",
"restoreFailureEnqueue": "Die Wiederherstellung konnte nicht an den Hintergrunddienst übergeben werden. Prüfe, ob er läuft.",
"restoreFailureAfterCommit": "Die Daten wurden wiederhergestellt, aber ein Schritt danach ist fehlgeschlagen. Diagramme holen das womöglich erst beim nächsten nächtlichen Lauf nach.",
"restoreFailureUnexpected": "Die Wiederherstellung ist an einem unerwarteten Fehler gescheitert und wurde zurückgerollt. Das Server-Log nennt den Fehler."
"restoreFailureUnexpected": "Die Wiederherstellung ist an einem unerwarteten Fehler gescheitert und wurde zurückgerollt. Das Server-Log nennt den Fehler.",
"restoreFailureKeyMissing": "Die Sicherung braucht den Verschlüsselungsschlüssel {keys}, mit dem dieser Server sie nicht öffnen kann. Nimm den Schlüssel wieder in ENCRYPTION_KEYS auf, starte neu und starte die Wiederherstellung erneut.",
"restoreFailureForeignReference": "Die Datei verweist auf Einträge, die zu einem anderen Konto gehören. Es wurde nichts geschrieben.",
"previewKeyMissing": "Diese Sicherung lässt sich auf diesem Server nicht wiederherstellen: Sie braucht den Verschlüsselungsschlüssel {keys}. Nimm den Schlüssel zuerst wieder in ENCRYPTION_KEYS auf und starte neu."
},
"danger-zone": {
"title": "Gefahrenzone",
+4 -1
View File
@@ -7309,7 +7309,10 @@
"restoreFailureNotStarted": "The background worker did not pick the restore up. Check that the worker is running.",
"restoreFailureEnqueue": "The restore could not be handed to the background worker. Check that the worker is running.",
"restoreFailureAfterCommit": "The data was restored, but a step after it failed. Charts may take until the next nightly run to catch up.",
"restoreFailureUnexpected": "The restore stopped on an unexpected error and was rolled back. The server log names the error."
"restoreFailureUnexpected": "The restore stopped on an unexpected error and was rolled back. The server log names the error.",
"restoreFailureKeyMissing": "The backup needs encryption key {keys}, which this server cannot open it with. Add the key back to ENCRYPTION_KEYS, restart, and start the restore again.",
"restoreFailureForeignReference": "The file points at records that belong to another account. Nothing was written.",
"previewKeyMissing": "This backup cannot be restored on this server: it needs encryption key {keys}. Add the key back to ENCRYPTION_KEYS and restart first."
},
"danger-zone": {
"title": "Danger Zone",
+4 -1
View File
@@ -7309,7 +7309,10 @@
"restoreFailureNotStarted": "El proceso en segundo plano no recogió la restauración. Comprueba que está en marcha.",
"restoreFailureEnqueue": "No se pudo entregar la restauración al proceso en segundo plano. Comprueba que está en marcha.",
"restoreFailureAfterCommit": "Los datos se restauraron, pero falló un paso posterior. Puede que los gráficos no se pongan al día hasta la próxima ejecución nocturna.",
"restoreFailureUnexpected": "La restauración se detuvo por un error inesperado y se deshizo. El registro del servidor indica el error."
"restoreFailureUnexpected": "La restauración se detuvo por un error inesperado y se deshizo. El registro del servidor indica el error.",
"restoreFailureKeyMissing": "La copia necesita la clave de cifrado {keys}, con la que este servidor no puede abrirla. Vuelve a añadir la clave a ENCRYPTION_KEYS, reinicia e inicia de nuevo la restauración.",
"restoreFailureForeignReference": "El archivo apunta a registros que pertenecen a otra cuenta. No se escribió nada.",
"previewKeyMissing": "Esta copia no se puede restaurar en este servidor: necesita la clave de cifrado {keys}. Vuelve a añadir la clave a ENCRYPTION_KEYS y reinicia primero."
},
"danger-zone": {
"title": "Zona de peligro",
+4 -1
View File
@@ -7309,7 +7309,10 @@
"restoreFailureNotStarted": "Le processus en arrière-plan n'a pas pris la restauration en charge. Vérifie qu'il tourne.",
"restoreFailureEnqueue": "La restauration n'a pas pu être transmise au processus en arrière-plan. Vérifie qu'il tourne.",
"restoreFailureAfterCommit": "Les données ont été restaurées, mais une étape suivante a échoué. Les graphiques ne seront peut-être à jour qu'après la prochaine exécution nocturne.",
"restoreFailureUnexpected": "La restauration s'est arrêtée sur une erreur inattendue et a été annulée. Le journal du serveur indique l'erreur."
"restoreFailureUnexpected": "La restauration s'est arrêtée sur une erreur inattendue et a été annulée. Le journal du serveur indique l'erreur.",
"restoreFailureKeyMissing": "La sauvegarde a besoin de la clé de chiffrement {keys}, avec laquelle ce serveur ne peut pas l'ouvrir. Remettez la clé dans ENCRYPTION_KEYS, redémarrez, puis relancez la restauration.",
"restoreFailureForeignReference": "Le fichier renvoie à des éléments qui appartiennent à un autre compte. Rien n'a été écrit.",
"previewKeyMissing": "Cette sauvegarde ne peut pas être restaurée sur ce serveur : elle a besoin de la clé de chiffrement {keys}. Remettez d'abord la clé dans ENCRYPTION_KEYS et redémarrez."
},
"danger-zone": {
"title": "Zone de danger",
+4 -1
View File
@@ -7309,7 +7309,10 @@
"restoreFailureNotStarted": "Il processo in background non ha preso in carico il ripristino. Controlla che sia in esecuzione.",
"restoreFailureEnqueue": "Non è stato possibile passare il ripristino al processo in background. Controlla che sia in esecuzione.",
"restoreFailureAfterCommit": "I dati sono stati ripristinati, ma un passaggio successivo non è riuscito. I grafici potrebbero aggiornarsi solo con la prossima esecuzione notturna.",
"restoreFailureUnexpected": "Il ripristino si è fermato per un errore imprevisto ed è stato annullato. Il log del server indica l'errore."
"restoreFailureUnexpected": "Il ripristino si è fermato per un errore imprevisto ed è stato annullato. Il log del server indica l'errore.",
"restoreFailureKeyMissing": "Il backup richiede la chiave di cifratura {keys}, con cui questo server non riesce ad aprirlo. Aggiungi di nuovo la chiave a ENCRYPTION_KEYS, riavvia e avvia di nuovo il ripristino.",
"restoreFailureForeignReference": "Il file rimanda a voci che appartengono a un altro account. Non è stato scritto nulla.",
"previewKeyMissing": "Questo backup non può essere ripristinato su questo server: richiede la chiave di cifratura {keys}. Aggiungi prima di nuovo la chiave a ENCRYPTION_KEYS e riavvia."
},
"danger-zone": {
"title": "Zona pericolosa",
+4 -1
View File
@@ -7288,7 +7288,10 @@
"restoreFailureNotStarted": "백그라운드 작업자가 복원을 가져가지 않았어요. 작업자가 실행 중인지 확인해 주세요.",
"restoreFailureEnqueue": "복원을 백그라운드 작업자에게 넘기지 못했어요. 작업자가 실행 중인지 확인해 주세요.",
"restoreFailureAfterCommit": "데이터는 복원됐지만 그다음 단계가 실패했어요. 차트는 다음 야간 실행 때 따라잡을 수 있어요.",
"restoreFailureUnexpected": "예상하지 못한 오류로 복원이 멈췄고 되돌려졌어요. 서버 로그에 오류가 나와 있어요."
"restoreFailureUnexpected": "예상하지 못한 오류로 복원이 멈췄고 되돌려졌어요. 서버 로그에 오류가 나와 있어요.",
"restoreFailureKeyMissing": "이 백업에는 암호화 키 {keys}가 필요하지만 이 서버는 그 키로 백업을 열 수 없습니다. 키를 ENCRYPTION_KEYS에 다시 추가하고 재시작한 뒤 복원을 다시 시작하세요.",
"restoreFailureForeignReference": "파일이 다른 계정에 속한 기록을 가리키고 있습니다. 아무것도 기록되지 않았습니다.",
"previewKeyMissing": "이 서버에서는 이 백업을 복원할 수 없습니다. 암호화 키 {keys}가 필요합니다. 먼저 키를 ENCRYPTION_KEYS에 다시 추가하고 재시작하세요."
},
"danger-zone": {
"title": "위험 구역",
+4 -1
View File
@@ -7309,7 +7309,10 @@
"restoreFailureNotStarted": "Proces w tle nie przejął przywracania. Sprawdź, czy działa.",
"restoreFailureEnqueue": "Nie udało się przekazać przywracania do procesu w tle. Sprawdź, czy działa.",
"restoreFailureAfterCommit": "Dane zostały przywrócone, ale krok po nich się nie powiódł. Wykresy mogą nadrobić to dopiero przy następnym nocnym przebiegu.",
"restoreFailureUnexpected": "Przywracanie zatrzymało się na nieoczekiwanym błędzie i zostało wycofane. Dziennik serwera podaje błąd."
"restoreFailureUnexpected": "Przywracanie zatrzymało się na nieoczekiwanym błędzie i zostało wycofane. Dziennik serwera podaje błąd.",
"restoreFailureKeyMissing": "Kopia wymaga klucza szyfrowania {keys}, którym ten serwer nie może jej otworzyć. Dodaj klucz z powrotem do ENCRYPTION_KEYS, uruchom ponownie i rozpocznij przywracanie jeszcze raz.",
"restoreFailureForeignReference": "Plik wskazuje na wpisy należące do innego konta. Nic nie zostało zapisane.",
"previewKeyMissing": "Tej kopii nie da się przywrócić na tym serwerze: wymaga klucza szyfrowania {keys}. Najpierw dodaj klucz z powrotem do ENCRYPTION_KEYS i uruchom ponownie."
},
"danger-zone": {
"title": "Strefa zagrożenia",
@@ -22,9 +22,15 @@ import {
storedBackupRefusal,
STORED_BACKUP_SELECT,
} from "@/lib/export/stored-backup";
import {
assessBackupKeys,
BACKUP_KEY_MISSING_CODE,
describeBackupKeyProblem,
} from "@/lib/export/backup-key-ids";
import {
readStreamedBackup,
type BackupSource,
type StreamedBackup,
} from "@/lib/export/streamed-backup";
import { annotate } from "@/lib/logging/context";
import {
@@ -72,8 +78,9 @@ export const GET = apiHandler(
let payload;
let measurementCount: number;
let streamed: StreamedBackup;
try {
const streamed = await readStreamedBackup(source);
streamed = await readStreamedBackup(source);
measurementCount = streamed.measurementCount;
payload = parseBackupPayload(streamed.raw);
} catch (err) {
@@ -93,6 +100,22 @@ export const GET = apiHandler(
);
}
// The preview is where the operator decides, so it says what the restore
// would refuse: a key the file's inner ciphertext needs and this host
// does not have. The instance settings are left out here, as the restore
// leaves them out unless asked; the restore itself checks them when they
// are asked for.
const keyVerdict = assessBackupKeys(streamed.keys, {
ignoreSections: new Set(["appSettings"]),
});
const keyProblem = describeBackupKeyProblem(keyVerdict);
if (keyProblem) {
return apiError(keyProblem, 422, {
errorCode: BACKUP_KEY_MISSING_CODE,
keyIds: [...keyVerdict.missing, ...keyVerdict.unreadable],
});
}
const summary = {
...summarizeBackup(payload),
measurements: measurementCount,
+29
View File
@@ -38,6 +38,12 @@ import { apiHandler, HttpError, requireAdmin } from "@/lib/api-handler";
import { apiError, apiSuccess, getClientIp } from "@/lib/api-response";
import { auditLog } from "@/lib/auth/audit";
import { BackupJsonError, scanBackupJson } from "@/lib/export/backup-json-scan";
import {
assessBackupKeys,
BACKUP_KEY_MISSING_CODE,
BackupKeyIdCollector,
describeBackupKeyProblem,
} from "@/lib/export/backup-key-ids";
import {
BACKUP_UPLOAD_TOO_LARGE_CODE,
BackupBlobTooLargeError,
@@ -235,6 +241,7 @@ export const POST = apiHandler(async (request: NextRequest) => {
}
}
let firstWithoutId: number | null = null;
const keys = new BackupKeyIdCollector();
let scanned;
try {
scanned = await scanBackupJson(storedAsRead(), {
@@ -251,6 +258,7 @@ export const POST = apiHandler(async (request: NextRequest) => {
if (!parsed.data.id && firstWithoutId === null) {
firstWithoutId = index;
}
keys.visit(element, "measurements");
},
});
} catch (err) {
@@ -319,6 +327,27 @@ export const POST = apiHandler(async (request: NextRequest) => {
);
}
// A file whose inner ciphertext needs a key this host does not have
// can be stored but never restored: the restore would refuse it, or
// worse, a restore on a host that had lost the key would write rows
// nobody can open. Refused here, where the operator is looking, with
// the key named. The instance settings are left to the restore, which
// checks them only when they are asked for back.
keys.visit(scanned.document);
const keyVerdict = assessBackupKeys(keys, {
ignoreSections: new Set(["appSettings"]),
});
const keyProblem = describeBackupKeyProblem(keyVerdict);
if (keyProblem) {
const keyIds = [...keyVerdict.missing, ...keyVerdict.unreadable];
throw new UploadRefused(
422,
keyProblem,
{ reason: "key_missing", keyIds },
{ errorCode: BACKUP_KEY_MISSING_CODE, keyIds },
);
}
// Make sure the userId on the file points at a user that exists in
// this DB. Restore would fail later anyway, but failing here gives
// the admin a precise error before any side-effect.
+10 -1
View File
@@ -71,6 +71,7 @@ function restoredCount(job: BackupRestoreJobView): number {
function failureText(
code: BackupRestoreFailureCode,
t: ReturnType<typeof useTranslations>["t"],
keyIds: readonly string[] = [],
): string {
switch (code) {
case "backup_not_found":
@@ -91,6 +92,12 @@ function failureText(
return t("admin.section.backups.restoreFailureSectionMissing");
case "document_ciphertext_missing":
return t("admin.section.backups.restoreFailureDocument");
case "backup.key.missing":
return t("admin.section.backups.restoreFailureKeyMissing", {
keys: keyIds.join(", "),
});
case "backup.foreign_reference":
return t("admin.section.backups.restoreFailureForeignReference");
case "time_budget":
return t("admin.section.backups.restoreFailureTimeBudget");
case "transaction_failed":
@@ -289,7 +296,9 @@ export function RestoreJobStatus({
<p className="font-medium">
{t("admin.section.backups.restoreJobFailedTitle", { username })}
</p>
<p className="text-xs">{failureText(code, t)}</p>
<p className="text-xs">
{failureText(code, t, job.failure?.keyIds)}
</p>
{changedNothing ? (
<p className="text-muted-foreground mt-1 text-xs">
{t("admin.section.backups.restoreJobFailedKept")}
+17 -5
View File
@@ -233,11 +233,23 @@ function RestoreRowDialog({
{t("admin.section.backups.previewLoading")}
</p>
)}
{preview.isError && (
<p className="text-muted-foreground">
{t("admin.section.backups.previewUnavailable")}
</p>
)}
{preview.isError &&
(preview.error instanceof ApiError &&
preview.error.meta?.errorCode === "backup.key.missing" ? (
// The one preview failure the restore would repeat: say which
// key, so the operator can put it back before trying.
<p role="alert" className="text-destructive">
{t("admin.section.backups.previewKeyMissing", {
keys: Array.isArray(preview.error.meta.keyIds)
? preview.error.meta.keyIds.map(String).join(", ")
: "",
})}
</p>
) : (
<p className="text-muted-foreground">
{t("admin.section.backups.previewUnavailable")}
</p>
))}
{preview.isSuccess && (
<ul className="grid list-none grid-cols-1 gap-x-4 gap-y-0.5 p-0 sm:grid-cols-2">
{previewRows.map((entry) => (
@@ -0,0 +1,173 @@
/**
* The key ids a backup's inner ciphertext needs, and the verdict against the
* keys this host has.
*
* The values are produced by the real codecs under real keys, then the host's
* key set is changed underneath them, which is exactly what an operator does
* when a rotation is followed by dropping the old key.
*/
import { Buffer } from "node:buffer";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { _resetCryptoCacheForTests, encrypt, encryptBytes } from "@/lib/crypto";
import {
assessBackupKeys,
BackupKeyIdCollector,
createBackupKeyIdTextScanner,
describeBackupKeyProblem,
innerCiphertextKeyId,
} from "../backup-key-ids";
const K1 = "11".repeat(32);
const K2 = "22".repeat(32);
const K3 = "33".repeat(32);
function useKeys(keys: Record<string, string>, active: string) {
delete process.env.ENCRYPTION_KEY;
process.env.ENCRYPTION_KEYS = JSON.stringify(keys);
process.env.ENCRYPTION_ACTIVE_KEY_ID = active;
_resetCryptoCacheForTests();
}
const saved = {
key: process.env.ENCRYPTION_KEY,
keys: process.env.ENCRYPTION_KEYS,
active: process.env.ENCRYPTION_ACTIVE_KEY_ID,
};
beforeEach(() => useKeys({ old: K1, cur: K2 }, "old"));
afterEach(() => {
for (const [name, value] of [
["ENCRYPTION_KEY", saved.key],
["ENCRYPTION_KEYS", saved.keys],
["ENCRYPTION_ACTIVE_KEY_ID", saved.active],
] as const) {
if (value === undefined) delete process.env[name];
else process.env[name] = value;
}
_resetCryptoCacheForTests();
});
/** A Bytes column holding the string codec, as a DR payload carries it. */
const bytesString = (plain: string) =>
Buffer.from(encrypt(plain), "utf8").toString("base64");
/** A Bytes column holding the binary codec, as a document's content. */
const binary = (plain: string) =>
encryptBytes(Buffer.from(plain)).toString("base64");
describe("innerCiphertextKeyId", () => {
it("reads the key id of each stored layout", () => {
expect(innerCiphertextKeyId(encrypt("note"))).toEqual({
keyId: "old",
form: "string",
});
expect(innerCiphertextKeyId(bytesString("note"))).toEqual({
keyId: "old",
form: "bytes-string",
});
expect(innerCiphertextKeyId(binary("document"))).toEqual({
keyId: "old",
form: "binary",
});
});
it("reads a value with no key id as the legacy v1 key", () => {
const legacy = Buffer.from("A".repeat(40)).toString("base64");
expect(innerCiphertextKeyId(legacy)?.keyId).toBe("v1");
});
it("ignores an empty value and one that is not base64", () => {
expect(innerCiphertextKeyId("")).toBeNull();
expect(innerCiphertextKeyId("not ciphertext at all!")).toBeNull();
});
});
describe("assessBackupKeys", () => {
function fileWrittenUnderOld() {
return {
userId: "u1",
moodEntries: [{ noteEncrypted: bytesString("a mood note") }],
documents: [{ contentEncrypted: binary("%PDF-1.4 …") }],
appSettings: { adminAiKeyEncrypted: encrypt("sk-something") },
};
}
it("passes a file whose keys are all configured and open", () => {
const collector = new BackupKeyIdCollector();
collector.visit(fileWrittenUnderOld());
collector.visit({ notesEncrypted: bytesString("x") }, "measurements");
const verdict = assessBackupKeys(collector);
expect(verdict).toEqual({
keyIds: ["old"],
missing: [],
unreadable: [],
affectedValues: 0,
});
expect(describeBackupKeyProblem(verdict)).toBeNull();
});
it("names a key the file needs and the host no longer has", () => {
const file = fileWrittenUnderOld();
useKeys({ cur: K2 }, "cur");
const collector = new BackupKeyIdCollector();
collector.visit(file);
const verdict = assessBackupKeys(collector);
expect(verdict.missing).toEqual(["old"]);
expect(verdict.affectedValues).toBe(3);
expect(describeBackupKeyProblem(verdict)).toContain("'old'");
});
it("leaves out sections the caller will not write", () => {
useKeys({ old: K1, cur: K2 }, "cur");
const file = {
moodEntries: [{ noteEncrypted: bytesString("fine") }],
appSettings: { adminAiKeyEncrypted: "gone.QUJDRA==" },
};
const collector = new BackupKeyIdCollector();
collector.visit(file);
expect(assessBackupKeys(collector).missing).toEqual(["gone"]);
expect(
assessBackupKeys(collector, { ignoreSections: new Set(["appSettings"]) })
.missing,
).toEqual([]);
});
it("catches an id that is configured with different key material", () => {
const file = fileWrittenUnderOld();
useKeys({ old: K3, cur: K2 }, "cur");
const collector = new BackupKeyIdCollector();
collector.visit(file);
const verdict = assessBackupKeys(collector);
expect(verdict.missing).toEqual([]);
expect(verdict.unreadable).toEqual(["old"]);
});
});
describe("createBackupKeyIdTextScanner", () => {
it("finds every key id in JSON fed in arbitrary pieces", () => {
const early = bytesString("written before the rotation");
useKeys({ old: K1, cur: K2 }, "cur");
const late = binary("written after it");
const json = JSON.stringify({
measurements: [{ id: "m1", notesEncrypted: early }],
documents: [{ contentEncrypted: late, note: "plain" }],
plain: "noteEncrypted",
});
for (const size of [1, 7, 33, 4096]) {
const scanner = createBackupKeyIdTextScanner();
for (let i = 0; i < json.length; i += size) {
scanner.feed(Buffer.from(json.slice(i, i + size), "utf8"));
}
expect(scanner.keyIds()).toEqual(["cur", "old"]);
}
});
it("reads pretty-printed JSON", () => {
const scanner = createBackupKeyIdTextScanner();
scanner.feed(
JSON.stringify({ a: { noteEncrypted: bytesString("x") } }, null, 2),
);
expect(scanner.keyIds()).toEqual(["old"]);
});
});
@@ -420,9 +420,15 @@ function recordingTx(
return new Proxy({} as Record<string, unknown>, {
// `$executeRaw` and friends: the restore's per-account lock, and the
// measurement insert, which answer with a row count.
// measurement insert, which answer with a row count. `$queryRaw` is the
// tenant check's read of the schema's foreign keys, which a mock has none
// of (the integration suite runs it against the real catalogue).
get: (_t, model: string) =>
model.startsWith("$") ? async () => 0 : delegate(model),
model === "$queryRaw"
? async () => []
: model.startsWith("$")
? async () => 0
: delegate(model),
});
}
+334
View File
@@ -0,0 +1,334 @@
/**
* Which application keys the ciphertext INSIDE a backup was written under.
*
* A disaster-recovery backup carries every encrypted column the way the
* database holds it: a note, a document, a coach message travel as the stored
* ciphertext, base64-encoded, and a restore writes that ciphertext back
* verbatim. The envelope around the whole copy is sealed separately, and key
* rotation re-seals that envelope, but it cannot reach inside it, and it
* cannot reach a copy in the off-host bucket or a file an operator keeps on a
* disk at all. So a backup taken before a rotation still needs the key the
* rotation retired, and a restore without that key writes rows back that no
* reader can open. Nothing in the restore used to notice: the envelope opened,
* the schema passed, and the damage showed up later as notes that would not
* decrypt.
*
* This module answers the question the restore has to ask first: which key
* ids does this file need, and are they all here. Every encrypted column in a
* backup payload is named `…Encrypted`, and each value is one of:
*
* - the string codec as stored in a text column: `<keyId>.<base64>`;
* - a Bytes column, base64-encoded, holding the string codec as UTF-8;
* - a Bytes column, base64-encoded, holding the binary codec
* (`0x02 | keyIdLen | keyId | iv | tag | ct`);
* - a legacy value with no key id, which only the `v1` key opens.
*
* Two readers share the classification: a walk over a parsed document (the
* restore, the preview, the upload and the restore drill already hold one),
* and a text scanner for a backup that is being written, which never exists
* as a document (`createBackupKeyIdTextScanner`).
*/
import { Buffer } from "node:buffer";
import {
decrypt,
decryptBytes,
extractKeyIdFromBytes,
getConfiguredKeyIds,
} from "@/lib/crypto";
/** The key id a value with no key id prefix was written under. */
export const LEGACY_INNER_KEY_ID = "v1";
const KEY_ID = /^[A-Za-z0-9_-]{1,32}$/;
const VERSIONED = /^([A-Za-z0-9_-]{1,32})\.[A-Za-z0-9+/=]+$/;
const BASE64 = /^[A-Za-z0-9+/]+={0,2}$/;
const BASE64_TAIL = /^[A-Za-z0-9+/=]*$/;
const BYTES_CODEC_VERSION = 0x02;
/** How an inner value is laid out, which decides how a sample is opened. */
export type InnerCiphertextForm = "string" | "bytes-string" | "binary";
export interface InnerCiphertext {
keyId: string;
form: InnerCiphertextForm;
}
/**
* The key id one inner ciphertext value needs, or null for something that is
* not ciphertext at all (an empty string). Decodes only a prefix, so a
* multi-megabyte document costs the same as a note.
*/
export function innerCiphertextKeyId(value: string): InnerCiphertext | null {
if (value.length === 0) return null;
const versioned = VERSIONED.exec(value.slice(0, 64));
if (versioned && value.indexOf(".") === versioned[1].length) {
return { keyId: versioned[1], form: "string" };
}
// A Bytes column, base64-encoded. 48 characters decode to 36 bytes, which
// covers the longest header either codec can have (2 + 32 for the binary
// one, 33 for `<keyId>.`).
const head = value.slice(0, 48);
if (!BASE64.test(head)) return null;
const bytes = Buffer.from(head, "base64");
if (bytes.length >= 3 && bytes[0] === BYTES_CODEC_VERSION) {
const keyIdLen = bytes[1];
if (keyIdLen >= 1 && keyIdLen <= 32 && bytes.length >= 2 + keyIdLen) {
const keyId = bytes.subarray(2, 2 + keyIdLen).toString("latin1");
if (KEY_ID.test(keyId)) return { keyId, form: "binary" };
}
}
const text = bytes.toString("latin1");
const dot = text.indexOf(".");
// The rest of the prefix has to read as base64 too, or a legacy value's
// random bytes could pass for `<keyId>.` about once in a thousand values.
if (
dot > 0 &&
KEY_ID.test(text.slice(0, dot)) &&
BASE64_TAIL.test(text.slice(dot + 1))
) {
return { keyId: text.slice(0, dot), form: "bytes-string" };
}
// No key id anywhere: the legacy layout, which `decrypt` opens with `v1`.
return { keyId: LEGACY_INNER_KEY_ID, form: "bytes-string" };
}
/** Whether a member name holds inner ciphertext. */
export function isEncryptedMember(name: string): boolean {
return name.endsWith("Encrypted") && name.length > "Encrypted".length;
}
interface KeyUse {
/** How many values need this key. */
count: number;
/** The top-level sections they sit in (`measurements`, `appSettings`, …). */
sections: Set<string>;
/** The shortest value seen, kept to prove the key actually opens it. */
sample: { value: string; form: InnerCiphertextForm } | null;
}
/** A sample longer than this is not kept; opening it proves nothing more. */
const MAX_SAMPLE_CHARS = 64 * 1024;
/**
* Collects the key ids a backup's inner ciphertext needs, with the section
* each one appears in and one sample per key.
*/
export class BackupKeyIdCollector {
private readonly uses = new Map<string, KeyUse>();
/** Record one inner value found in `section`. */
add(section: string, value: string): void {
const found = innerCiphertextKeyId(value);
if (!found) return;
let use = this.uses.get(found.keyId);
if (!use) {
use = { count: 0, sections: new Set(), sample: null };
this.uses.set(found.keyId, use);
}
use.count += 1;
use.sections.add(section);
if (
value.length <= MAX_SAMPLE_CHARS &&
(!use.sample || value.length < use.sample.value.length)
) {
use.sample = { value, form: found.form };
}
}
/**
* Walk a parsed value and record every `…Encrypted` string in it. `section`
* names where it sits in the file; for a whole document, pass nothing and
* each top-level member is its own section.
*/
visit(value: unknown, section?: string): void {
if (section === undefined) {
if (value && typeof value === "object" && !Array.isArray(value)) {
for (const [key, child] of Object.entries(value)) {
this.walk(child, key, key);
}
}
return;
}
this.walk(value, section, null);
}
private walk(value: unknown, section: string, member: string | null): void {
if (typeof value === "string") {
if (member !== null && isEncryptedMember(member)) {
this.add(section, value);
}
return;
}
if (Array.isArray(value)) {
// Elements of an array under an `…Encrypted` member are ciphertext too.
for (const element of value) this.walk(element, section, member);
return;
}
if (value && typeof value === "object") {
for (const [key, child] of Object.entries(value)) {
this.walk(child, section, key);
}
}
}
/** Every key id seen, sorted. */
keyIds(): string[] {
return [...this.uses.keys()].sort();
}
/** How many values need `keyId`. */
count(keyId: string): number {
return this.uses.get(keyId)?.count ?? 0;
}
/** @internal for `assessBackupKeys`. */
entries(): ReadonlyMap<string, KeyUse> {
return this.uses;
}
}
export interface BackupKeyAssessment {
/** Every key id the file's inner ciphertext was written under. */
keyIds: string[];
/** Needed and not in `ENCRYPTION_KEYS` on this host. */
missing: string[];
/**
* Configured, but the sample written under it does not open: the id is
* there with different key material behind it.
*/
unreadable: string[];
/** Values that need a missing or unreadable key. */
affectedValues: number;
}
function openSample(sample: { value: string; form: InnerCiphertextForm }) {
switch (sample.form) {
case "string":
decrypt(sample.value);
return;
case "bytes-string":
decrypt(Buffer.from(sample.value, "base64").toString("utf8"));
return;
case "binary":
decryptBytes(Buffer.from(sample.value, "base64"));
return;
}
}
/**
* Hold what the file needs against what this host has, and open one value per
* key to prove the key under that id is the right one.
*
* `ignoreSections` leaves out sections the caller will not write: the
* instance settings, when the operator has not asked for them back.
*/
export function assessBackupKeys(
collector: BackupKeyIdCollector,
options: { ignoreSections?: ReadonlySet<string> } = {},
): BackupKeyAssessment {
const ignore = options.ignoreSections ?? new Set<string>();
const keyIds: string[] = [];
const missing: string[] = [];
const unreadable: string[] = [];
let affectedValues = 0;
// Read lazily: a file with no inner ciphertext needs no key, and must not
// fail on a host (or a test) that has none configured.
let configured: Set<string> | null = null;
for (const [keyId, use] of [...collector.entries()].sort(([a], [b]) =>
a.localeCompare(b),
)) {
if ([...use.sections].every((section) => ignore.has(section))) continue;
keyIds.push(keyId);
configured ??= new Set(getConfiguredKeyIds());
if (!configured.has(keyId)) {
missing.push(keyId);
affectedValues += use.count;
continue;
}
if (!use.sample) continue;
try {
openSample(use.sample);
} catch {
unreadable.push(keyId);
affectedValues += use.count;
}
}
return { keyIds, missing, unreadable, affectedValues };
}
/** The refusal code every surface answers a key problem with. */
export const BACKUP_KEY_MISSING_CODE = "backup.key.missing";
/** One sentence an operator can act on, naming the key ids. */
export function describeBackupKeyProblem(
assessment: BackupKeyAssessment,
): string | null {
const parts: string[] = [];
if (assessment.missing.length > 0) {
const ids = assessment.missing.map((id) => `'${id}'`).join(", ");
parts.push(
`This backup holds ${assessment.affectedValues} encrypted ${
assessment.affectedValues === 1 ? "value" : "values"
} written under encryption key ${ids}, which is not in ENCRYPTION_KEYS on this server. ` +
`Add ${assessment.missing.length === 1 ? "that key" : "those keys"} back to ENCRYPTION_KEYS and restart, then try again. Nothing was changed.`,
);
}
if (assessment.unreadable.length > 0) {
const ids = assessment.unreadable.map((id) => `'${id}'`).join(", ");
parts.push(
`Encryption key ${ids} is configured, but it does not open the values in this backup written under that id: ` +
`the key material behind the id differs from the one the backup was written with. Nothing was changed.`,
);
}
return parts.length > 0 ? parts.join(" ") : null;
}
/**
* Key ids found in backup JSON as it is written, for a copy that never exists
* as one document. Fed the text in pieces; a member split across two pieces is
* read once the rest of it arrives.
*/
export interface BackupKeyIdTextScanner {
feed(piece: string | Uint8Array): void;
keyIds(): string[];
}
const MEMBER_VALUE =
/"([A-Za-z0-9_]*Encrypted)"\s*:\s*"([A-Za-z0-9+/=._-]{0,64})(")?/g;
/** Enough of the previous piece to finish any member that straddles it. */
const CARRY_CHARS = 256;
export function createBackupKeyIdTextScanner(): BackupKeyIdTextScanner {
const found = new Set<string>();
let carry = "";
return {
feed(piece) {
// latin1, not utf-8: every character this looks for is ASCII, and a
// piece may end in the middle of a multi-byte character.
const text =
carry +
(typeof piece === "string"
? piece
: Buffer.from(
piece.buffer,
piece.byteOffset,
piece.byteLength,
).toString("latin1"));
MEMBER_VALUE.lastIndex = 0;
let match: RegExpExecArray | null;
while ((match = MEMBER_VALUE.exec(text)) !== null) {
const [, member, value, closed] = match;
if (!isEncryptedMember(member)) continue;
// A value cut off by the end of the piece is read again next time.
if (!closed && value.length < 64) continue;
const inner = innerCiphertextKeyId(value);
if (inner) found.add(inner.keyId);
}
carry = text.slice(-CARRY_CHARS);
},
keyIds() {
return [...found].sort();
},
};
}
+69 -1
View File
@@ -46,6 +46,17 @@ import {
type BackupSource,
type StreamedBackup,
} from "@/lib/export/streamed-backup";
import {
assessBackupKeys,
BACKUP_KEY_MISSING_CODE,
describeBackupKeyProblem,
} from "@/lib/export/backup-key-ids";
import {
assertNoNewForeignReferences,
countForeignReferences,
ForeignReferenceError,
listTenantEdges,
} from "@/lib/export/tenant-boundary";
import { encryptNote } from "@/lib/crypto/note-cipher";
import { encryptToBytes } from "@/lib/ai/coach/bytes-codec";
import { encryptContextToBytes } from "@/lib/labs/biomarker-store";
@@ -239,6 +250,8 @@ export type RestoreFailureCode =
| "owner_not_found"
| "backup.section.missing"
| "document_ciphertext_missing"
| "backup.key.missing"
| "backup.foreign_reference"
| "time_budget"
| "transaction_failed"
| "interrupted"
@@ -474,6 +487,40 @@ export async function restoreBackup(
);
}
// The ciphertext inside the file is written back verbatim, so every key it
// was written under has to be on this host, and has to be the key that
// wrote it. A copy taken before a key rotation still needs the key the
// rotation retired: the rotation re-seals the stored copy's envelope but
// cannot reach inside it. Refused here, above the first delete, because the
// alternative is an account whose notes, documents and coach history come
// back as rows no reader can open. The instance settings count only when
// the operator asked for them back.
const keyVerdict = assessBackupKeys(streamed.keys, {
ignoreSections: restoreInstanceSettings
? new Set<string>()
: new Set(["appSettings"]),
});
const keyProblem = describeBackupKeyProblem(keyVerdict);
if (keyProblem) {
const keyIds = [...keyVerdict.missing, ...keyVerdict.unreadable];
await auditLog("admin.backups.restore.failed", {
userId: input.actorUserId,
ipAddress: input.ipAddress,
details: {
backupId: backup.id,
ownerId,
reason: "key_missing",
missingKeyIds: keyVerdict.missing,
unreadableKeyIds: keyVerdict.unreadable,
affectedValues: keyVerdict.affectedValues,
},
});
return refused(422, BACKUP_KEY_MISSING_CODE, keyProblem, {
errorCode: BACKUP_KEY_MISSING_CODE,
keyIds,
});
}
// Audit the *intent* before the transaction begins so the trail
// describing "an admin is about to restore <user>" survives even
// if the operation crashes midway.
@@ -604,6 +651,17 @@ export async function restoreBackup(
// commit or roll back, then replaces the account again from its own
// file, which is what it was asked to do.
await tx.$executeRaw`SELECT pg_advisory_xact_lock(hashtext(${`backup-restore:${ownerId}`}))`;
// Every reference this restore writes stays inside the account: the
// references that leave it are counted now and again after the last
// write, and any that grew rolls the whole transaction back
// (`tenant-boundary.ts` says why the check is derived from the schema
// rather than written per section).
const tenantEdges = await listTenantEdges(tx);
const foreignBefore = await countForeignReferences(
tx,
ownerId,
tenantEdges,
);
// Declared INSIDE the transaction so a rollback takes the report with
// it. An accumulator that outlived a failed attempt would carry drops
// that never happened into the next one.
@@ -1834,7 +1892,7 @@ export async function restoreBackup(
for (const episode of payload.illnessEpisodes) {
if (episode.parentConditionId) {
await tx.illnessEpisode.update({
where: { id: episode.id },
where: { id: episode.id, userId: ownerId },
data: { parentConditionId: episode.parentConditionId },
});
}
@@ -2279,6 +2337,10 @@ export async function restoreBackup(
environmentCleared.environmentTravelLocations,
ecgRecordings: ecgCleared.ecgRecordings,
};
assertNoNewForeignReferences(
foreignBefore,
await countForeignReferences(tx, ownerId, tenantEdges),
);
progress.sectionsDone = RESTORE_SECTION_STEPS.length;
report("sections", true);
return { cleared, skipped: summarizeRestoreSkips(skips) };
@@ -2307,6 +2369,12 @@ export async function restoreBackup(
},
});
annotate({ meta: { restoreFailReason: verbose } });
if (err instanceof ForeignReferenceError) {
return refused(422, "backup.foreign_reference", err.message, {
errorCode: "backup.foreign_reference",
references: err.edges,
});
}
if (isStoredBackupReadError(err)) {
// The second read, inside the transaction, found the copy replaced or
// altered. The transaction rolled back, so nothing was changed.
+12
View File
@@ -21,6 +21,7 @@
* check reads). Run it through `parseBackupPayload` for the rest of the
* validation; together that is the whole-file check.
*/
import { BackupKeyIdCollector } from "@/lib/export/backup-key-ids";
import { scanBackupJson } from "@/lib/export/backup-json-scan";
import {
BACKUP_SCHEMA_VERSION,
@@ -51,6 +52,12 @@ export interface StreamedBackup {
measurementCount: number;
/** The file's size in bytes. */
bytes: number;
/**
* The application key ids the file's inner ciphertext was written under,
* gathered on the first read (`assessBackupKeys` holds them against this
* host's keys).
*/
keys: BackupKeyIdCollector;
/**
* Read the file again and hand its measurements over in file order, in
* batches of at most `size`, each row parsed by the element schema.
@@ -93,6 +100,7 @@ export async function readStreamedBackup(
): Promise<StreamedBackup> {
let firstWithoutId: number | null = null;
let bytes = 0;
const keys = new BackupKeyIdCollector();
async function* counted() {
for await (const chunk of source()) {
bytes += chunk.byteLength;
@@ -104,6 +112,7 @@ export async function readStreamedBackup(
onElement: (_key, element, index) => {
const row = parseMeasurement(element, index);
if (!row.id && firstWithoutId === null) firstWithoutId = index;
keys.visit(element, MEASUREMENTS);
options.onMeasurementChecked?.(index + 1);
},
});
@@ -120,6 +129,8 @@ export async function readStreamedBackup(
);
}
keys.visit(document);
const skipKeys = new Set(
Object.keys(document).filter((key) => key !== MEASUREMENTS),
);
@@ -128,6 +139,7 @@ export async function readStreamedBackup(
raw: document,
measurementCount: streamedCounts[MEASUREMENTS] ?? 0,
bytes,
keys,
async forEachMeasurementBatch(size, fn) {
let batch: BackupMeasurement[] = [];
await scanBackupJson(source(), {
+178
View File
@@ -0,0 +1,178 @@
/**
* Every reference a restore writes has to stay inside the account it writes.
*
* A backup file names ids: a custom mood tag names its category, an illness
* names its parent condition, a link names the document and the episode it
* joins. The restore writes those ids back as given, and a foreign key only
* proves the row they point at exists, not whose it is. So a file edited by
* hand, or an upload whose owner is simply what the file says it is, could
* attach the restored account's rows to rows of another account on the same
* host: a custom tag filed under somebody else's category, a note linked to
* somebody else's document. Several of the restore's writes (the tag and
* category upserts keyed on a globally unique `key`, for one) never even look
* at the owner.
*
* Checking each write site by hand is the approach that keeps missing one, so
* the check is derived from the schema instead. Every single-column foreign key
* is read from the database catalogue, and two shapes are held:
*
* - a table with `user_id` pointing at another table with `user_id`: a row of
* the restored account may not point at a row another account owns;
* - a link table without `user_id` joining two such tables: a link whose one
* end is the restored account's may not have its other end in another
* account.
*
* A NULL owner on the far end is a shared catalogue row (a seeded tag) and is
* allowed. The counts are taken twice inside the restore transaction, before
* the first delete and after the last write, and the restore is refused when
* any count went up: the host may have legitimate cross-account links of its
* own (a shared record), and the question is only whether THIS restore added
* one. Refusing throws, which rolls the whole transaction back.
*/
import type { Prisma } from "@/generated/prisma/client";
type Db = Pick<Prisma.TransactionClient, "$queryRawUnsafe" | "$queryRaw">;
/** One foreign key, or one pair of them through a link table. */
export interface TenantEdge {
/** Readable name for the refusal: `child.column -> parent`. */
name: string;
/** The counting query; `$1` is the owner id. */
sql: string;
}
/**
* Identifiers read from the catalogue are spliced into SQL, so each is held
* to this before use (the whitelist-splice rule for `$queryRawUnsafe`).
*/
const IDENT = /^[a-z_][a-z0-9_]*$/;
interface ForeignKeyRow {
child: string;
child_col: string;
parent: string;
parent_col: string;
child_owned: boolean;
parent_owned: boolean;
}
/** Thrown inside the restore transaction; rolls it back. */
export class ForeignReferenceError extends Error {
readonly edges: string[];
constructor(edges: string[]) {
super(
`The backup references records that belong to another account (${edges.join(", ")}). Nothing was changed.`,
);
this.name = "ForeignReferenceError";
this.edges = edges;
}
}
export async function listTenantEdges(db: Db): Promise<TenantEdge[]> {
const rows = await db.$queryRaw<ForeignKeyRow[]>`
SELECT
child.relname AS child,
ca.attname AS child_col,
parent.relname AS parent,
pa.attname AS parent_col,
EXISTS (
SELECT 1 FROM pg_attribute x
WHERE x.attrelid = c.conrelid AND x.attname = 'user_id'
AND NOT x.attisdropped
) AS child_owned,
EXISTS (
SELECT 1 FROM pg_attribute x
WHERE x.attrelid = c.confrelid AND x.attname = 'user_id'
AND NOT x.attisdropped
) AS parent_owned
FROM pg_constraint c
JOIN pg_class child ON child.oid = c.conrelid
JOIN pg_class parent ON parent.oid = c.confrelid
JOIN pg_namespace n ON n.oid = child.relnamespace
JOIN pg_attribute ca ON ca.attrelid = c.conrelid AND ca.attnum = c.conkey[1]
JOIN pg_attribute pa ON pa.attrelid = c.confrelid AND pa.attnum = c.confkey[1]
WHERE c.contype = 'f'
AND array_length(c.conkey, 1) = 1
AND n.nspname = current_schema()
ORDER BY child.relname, ca.attname
`;
const safe = rows.filter((row) =>
[row.child, row.child_col, row.parent, row.parent_col].every((id) =>
IDENT.test(id),
),
);
if (safe.length !== rows.length) {
// Never silently narrower than the schema: a check that quietly skips a
// table is the kind that stays green because it matched nothing.
throw new Error(
"Tenant check: a foreign key has an identifier outside [a-z0-9_]",
);
}
const edges: TenantEdge[] = [];
for (const fk of safe) {
if (!fk.child_owned || !fk.parent_owned) continue;
edges.push({
name: `${fk.child}.${fk.child_col} -> ${fk.parent}`,
sql:
`SELECT count(*)::int AS n FROM "${fk.child}" c ` +
`JOIN "${fk.parent}" p ON p."${fk.parent_col}" = c."${fk.child_col}" ` +
`WHERE c.user_id = $1 AND p.user_id IS NOT NULL AND p.user_id <> $1`,
});
}
// Link tables: no owner of their own, two or more keys into owned tables.
const byChild = new Map<string, ForeignKeyRow[]>();
for (const fk of safe) {
if (fk.child_owned || !fk.parent_owned) continue;
const list = byChild.get(fk.child) ?? [];
list.push(fk);
byChild.set(fk.child, list);
}
for (const [child, fks] of byChild) {
if (fks.length < 2) continue;
for (const near of fks) {
for (const far of fks) {
if (near === far) continue;
edges.push({
name: `${child}.${near.child_col} / ${child}.${far.child_col} -> ${far.parent}`,
sql:
`SELECT count(*)::int AS n FROM "${child}" l ` +
`JOIN "${near.parent}" a ON a."${near.parent_col}" = l."${near.child_col}" ` +
`JOIN "${far.parent}" b ON b."${far.parent_col}" = l."${far.child_col}" ` +
`WHERE a.user_id = $1 AND b.user_id IS NOT NULL AND b.user_id <> $1`,
});
}
}
}
return edges;
}
/** How many references from `ownerId`'s rows leave the account, per edge. */
export async function countForeignReferences(
db: Db,
ownerId: string,
edges: readonly TenantEdge[],
): Promise<Map<string, number>> {
const counts = new Map<string, number>();
for (const edge of edges) {
const [row] = await db.$queryRawUnsafe<Array<{ n: number }>>(
edge.sql,
ownerId,
);
counts.set(edge.name, row?.n ?? 0);
}
return counts;
}
/** Throw when any edge carries more foreign references than it did before. */
export function assertNoNewForeignReferences(
before: ReadonlyMap<string, number>,
after: ReadonlyMap<string, number>,
): void {
const grown = [...after]
.filter(([name, n]) => n > (before.get(name) ?? 0))
.map(([name]) => name);
if (grown.length > 0) throw new ForeignReferenceError(grown);
}
@@ -37,6 +37,8 @@ const CODE_TABLE: Record<BackupRestoreFailureCode, true> = {
owner_not_found: true,
"backup.section.missing": true,
document_ciphertext_missing: true,
"backup.key.missing": true,
"backup.foreign_reference": true,
time_budget: true,
transaction_failed: true,
interrupted: true,
@@ -1,5 +1,6 @@
import type { Readable } from "node:stream";
import { describe, it, expect, beforeEach, vi } from "vitest";
import { _resetCryptoCacheForTests, encrypt } from "@/lib/crypto";
import { encryptBackup } from "../offhost-backup";
import {
handleRestoreDrill,
@@ -115,6 +116,43 @@ describe("runRestoreDrill", () => {
expect(report.stale).toBe(true);
});
it("decrypts a value per inner key id and names a key the host lacks", async () => {
vi.stubEnv("ENCRYPTION_KEY", "");
vi.stubEnv(
"ENCRYPTION_KEYS",
JSON.stringify({ old: "11".repeat(32), cur: "22".repeat(32) }),
);
vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "old");
_resetCryptoCacheForTests();
const note = Buffer.from(encrypt("a note"), "utf8").toString("base64");
const s3 = makeS3Mock({
"2026-06-01/user-abc.json.enc": backupObject({
measurements: [{ id: "m1", notesEncrypted: note }],
}),
});
const now = new Date("2026-06-02T04:11:00Z");
const report = await runRestoreDrill(s3, now);
expect(report.innerKeyIds).toEqual(["old"]);
// The rotation ran, and the operator dropped the old key.
vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ cur: "22".repeat(32) }));
vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "cur");
_resetCryptoCacheForTests();
await expect(runRestoreDrill(s3, now)).rejects.toThrow(/'old'/);
// Same id, different key material behind it.
vi.stubEnv(
"ENCRYPTION_KEYS",
JSON.stringify({ old: "33".repeat(32), cur: "22".repeat(32) }),
);
_resetCryptoCacheForTests();
await expect(runRestoreDrill(s3, now)).rejects.toThrow(
/does not open the values/,
);
vi.unstubAllEnvs();
_resetCryptoCacheForTests();
});
it("throws when the bucket holds no backup-shaped objects", async () => {
const s3 = makeS3Mock({ "_healthcheck/1.bin": Buffer.from([0x42]) });
await expect(runRestoreDrill(s3)).rejects.toThrow(/no backup objects/);
+8
View File
@@ -120,12 +120,15 @@ export interface BackupRestoreFailure {
message: string;
/** The sections a file's manifest names and it does not carry. */
sections?: string[];
/** The encryption key ids a file needs and this host cannot open it with. */
keyIds?: string[];
}
const failureSchema = z.object({
code: z.string(),
message: z.string(),
sections: z.array(z.string()).optional(),
keyIds: z.array(z.string()).optional(),
});
const progressSchema = z.object({
@@ -308,6 +311,7 @@ function failureJson(failure: BackupRestoreFailure) {
code: failure.code,
message: failure.message,
...(failure.sections ? { sections: failure.sections } : {}),
...(failure.keyIds ? { keyIds: failure.keyIds } : {}),
});
}
@@ -632,11 +636,15 @@ export async function runBackupRestoreJob(
const sections = Array.isArray(outcome.meta?.sections)
? (outcome.meta.sections as unknown[]).map(String)
: undefined;
const keyIds = Array.isArray(outcome.meta?.keyIds)
? (outcome.meta.keyIds as unknown[]).map(String)
: undefined;
await finish("failed", {
failure: {
code: outcome.code,
message: outcome.message,
...(sections ? { sections } : {}),
...(keyIds ? { keyIds } : {}),
},
});
// A file refused by its checks is the operator's to fix and fails the
+27 -6
View File
@@ -244,6 +244,9 @@ export function openBackupObject(
const UPLOAD_PART_BYTES = 8 * 1024 * 1024;
const UPLOAD_CONCURRENCY = 2;
/** Pages of 1 000 keys one listing may walk: 200 000 objects. */
const MAX_LIST_PAGES = 200;
/**
* The largest object one multipart upload can carry: S3 and every compatible
* target cap a multipart upload at 10 000 parts.
@@ -567,13 +570,31 @@ export async function getS3Client(cfg: OffhostBackupConfig): Promise<S3Like> {
}
},
listObjects: async (prefix) => {
const out = await client.send(
new mod.ListObjectsV2Command({ Bucket: cfg.bucket, Prefix: prefix }),
// Every page, not the first. A bucket answers at most 1 000 keys per
// call, in key order, so the first page of a bucket with a few accounts
// and a month of retention holds only the OLDEST dates: the restore
// drill read an old object and called the chain stale, and a purge
// would have missed every newer copy. Bounded, so a bucket that holds
// far more than this job put there cannot keep the loop going forever.
const found: Array<{ key: string; lastModified?: Date }> = [];
let token: string | undefined;
for (let page = 0; page < MAX_LIST_PAGES; page++) {
const out = await client.send(
new mod.ListObjectsV2Command({
Bucket: cfg.bucket,
Prefix: prefix,
ContinuationToken: token,
}),
);
for (const c of out.Contents ?? []) {
found.push({ key: c.Key ?? "", lastModified: c.LastModified });
}
if (!out.IsTruncated || !out.NextContinuationToken) return found;
token = out.NextContinuationToken;
}
throw new Error(
`Bucket listing under "${prefix}" did not end within ${MAX_LIST_PAGES} pages`,
);
return (out.Contents ?? []).map((c) => ({
key: c.Key ?? "",
lastModified: c.LastModified,
}));
},
deleteObject: async (key) => {
await client.send(
+27
View File
@@ -23,6 +23,11 @@
*/
import type { Job } from "pg-boss";
import { BackupJsonError, scanBackupJson } from "@/lib/export/backup-json-scan";
import {
assessBackupKeys,
BackupKeyIdCollector,
describeBackupKeyProblem,
} from "@/lib/export/backup-key-ids";
import {
openBackupObject,
getS3Client,
@@ -54,6 +59,8 @@ export interface RestoreDrillReport {
stale: boolean;
ciphertextBytes: number;
plaintextBytes: number;
/** Application key ids the object's inner ciphertext needs, each proven to open. */
innerKeyIds: string[];
recordCounts: {
measurements: number;
medications: number;
@@ -115,10 +122,17 @@ export async function runRestoreDrill(
yield chunk;
}
}
// The envelope opening proves the off-host key. It proves nothing about
// the application keys the ciphertext INSIDE was written under, and that
// is the half a key rotation breaks: a restore of this object writes those
// values back verbatim. So every inner key id is collected, and one value
// per key is actually decrypted.
const keys = new BackupKeyIdCollector();
let scanned;
try {
scanned = await scanBackupJson(counted(), {
streamKeys: new Set(["measurements", "intakeEvents", "moodEntries"]),
onElement: (key, element) => keys.visit(element, key),
});
} catch (err) {
if (err instanceof BackupJsonError) {
@@ -140,6 +154,17 @@ export async function runRestoreDrill(
);
}
keys.visit(payload);
const keyVerdict = assessBackupKeys(keys, {
ignoreSections: new Set(["appSettings"]),
});
const keyProblem = describeBackupKeyProblem(keyVerdict);
if (keyProblem) {
throw new Error(
`Restore drill: backup object "${objectKey}" opens, but its content could not be restored on this server. ${keyProblem}`,
);
}
const ageDays = Math.floor(
(now.getTime() - Date.parse(`${dateKey}T00:00:00Z`)) /
(24 * 60 * 60 * 1000),
@@ -152,6 +177,7 @@ export async function runRestoreDrill(
stale: ageDays > MAX_BACKUP_AGE_DAYS,
ciphertextBytes: ciphertext.length,
plaintextBytes,
innerKeyIds: keyVerdict.keyIds,
recordCounts: {
measurements: streamedCounts.measurements,
medications: countArray(payload.medications),
@@ -193,6 +219,7 @@ export async function handleRestoreDrill(
report.recordCounts.moodEntries,
);
evt.addMeta("restore_drill_stale", report.stale);
evt.addMeta("restore_drill_inner_key_ids", report.innerKeyIds.join(","));
if (report.stale) {
await reportWorkerError(
RESTORE_DRILL_QUEUE,
+3 -2
View File
@@ -86,15 +86,16 @@ const restoreFailure = z
.object({
code: z.string().meta({
description:
"Stable reason: `backup_not_found`, `backup_changed`, `backup.payload.undecryptable`, `schema_invalid`, `incompatible_schema_version`, `owner_mismatch`, `owner_not_found`, `backup.section.missing`, `document_ciphertext_missing`, `time_budget`, `transaction_failed`, `interrupted`, `not_started`, `enqueue_failed`, `failed_after_commit`, `unexpected`. Every code but `failed_after_commit` means the account was not changed.",
"Stable reason: `backup_not_found`, `backup_changed`, `backup.payload.undecryptable`, `schema_invalid`, `incompatible_schema_version`, `owner_mismatch`, `owner_not_found`, `backup.section.missing`, `document_ciphertext_missing`, `backup.key.missing`, `backup.foreign_reference`, `time_budget`, `transaction_failed`, `interrupted`, `not_started`, `enqueue_failed`, `failed_after_commit`, `unexpected`. Every code but `failed_after_commit` means the account was not changed.",
}),
message: z.string(),
sections: z.array(z.string()).optional(),
keyIds: z.array(z.string()).optional(),
})
.meta({
id: "AdminBackupRestoreFailure",
description:
"Why the restore did not happen, in a sentence an operator can act on. `sections` names the missing sections for `backup.section.missing`.",
"Why the restore did not happen, in a sentence an operator can act on. `sections` names the missing sections for `backup.section.missing`; `keyIds` names the encryption key ids the file needs and this server cannot open it with, for `backup.key.missing`.",
});
const restoreJob = z
@@ -0,0 +1,262 @@
/**
* Two refusals a restore has to make before it changes anything.
*
* 1. The file's inner ciphertext was written under a key this host no
* longer has. The envelope opens (it is sealed under the current key),
* the schema passes, and without the check the restore would write back
* rows no reader can open. Refused with the key named; the account keeps
* what it had.
*
* 2. The file attaches the account's rows to rows another account owns: a
* custom mood tag filed under another account's category. The foreign
* key is satisfied, so only the tenant check stands in the way. Refused;
* the transaction rolls back and the account keeps what it had.
*
* Mutations that must turn these red: drop the `keyProblem` refusal in
* `restoreBackup` (case 1 restores and the note comes back unreadable), or the
* `assertNoNewForeignReferences` call (case 2 commits the foreign link).
*/
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { _resetCryptoCacheForTests, encrypt } from "@/lib/crypto";
import { STORED_BACKUP_SELECT } from "@/lib/export/stored-backup";
import { restoreBackup } from "@/lib/export/restore-backup";
import { getPrismaClient, truncateAllTables } from "./setup";
const saved = {
key: process.env.ENCRYPTION_KEY,
keys: process.env.ENCRYPTION_KEYS,
active: process.env.ENCRYPTION_ACTIVE_KEY_ID,
};
function useKeys(keys: Record<string, string>, active: string) {
process.env.ENCRYPTION_KEY = "";
process.env.ENCRYPTION_KEYS = JSON.stringify(keys);
process.env.ENCRYPTION_ACTIVE_KEY_ID = active;
_resetCryptoCacheForTests();
}
beforeEach(async () => {
await truncateAllTables(getPrismaClient());
});
afterEach(() => {
process.env.ENCRYPTION_KEY = saved.key ?? "";
process.env.ENCRYPTION_KEYS = saved.keys ?? "";
process.env.ENCRYPTION_ACTIVE_KEY_ID = saved.active ?? "";
_resetCryptoCacheForTests();
});
async function seedOwner(username: string) {
return getPrismaClient().user.create({
data: { username, email: `${username}@example.test` },
});
}
async function storeBackup(userId: string, payload: unknown) {
const prisma = getPrismaClient();
const row = await prisma.dataBackup.create({
data: {
userId,
type: "MANUAL_UPLOAD_BOUNDARY_TEST",
data: encrypt(JSON.stringify(payload)),
},
});
return prisma.dataBackup.findUniqueOrThrow({
where: { id: row.id },
select: STORED_BACKUP_SELECT,
});
}
function basePayload(userId: string) {
return {
schemaVersion: "1",
exportedAt: "2026-05-09T10:00:00.000Z",
userId,
measurements: [] as unknown[],
medications: [],
intakeEvents: [],
moodEntries: [],
};
}
describe("restore refuses before it changes anything", () => {
it("refuses a file whose inner ciphertext needs a key the host dropped", async () => {
const prisma = getPrismaClient();
useKeys({ old: "11".repeat(32), cur: "22".repeat(32) }, "old");
const owner = await seedOwner("key-owner");
const note = Buffer.from(encrypt("written before the rotation"), "utf8");
const payload = basePayload(owner.id);
payload.measurements = [
{
type: "WEIGHT",
value: 80,
unit: "kg",
measuredAt: "2026-05-08T07:00:00.000Z",
source: "MANUAL",
notesEncrypted: note.toString("base64"),
},
];
// The rotation ran and the operator dropped the old key; the stored copy's
// envelope is sealed under the key that remains.
useKeys({ cur: "22".repeat(32) }, "cur");
const backup = await storeBackup(owner.id, payload);
const kept = await prisma.measurement.create({
data: {
userId: owner.id,
type: "WEIGHT",
value: 99,
unit: "kg",
measuredAt: new Date("2026-05-01T07:00:00.000Z"),
source: "MANUAL",
},
});
const outcome = await restoreBackup({
backup,
actorUserId: owner.id,
ipAddress: null,
restoreInstanceSettings: false,
});
expect(outcome.ok).toBe(false);
if (outcome.ok) return;
expect(outcome.code).toBe("backup.key.missing");
expect(outcome.message).toContain("'old'");
expect(outcome.meta?.keyIds).toEqual(["old"]);
const rows = await prisma.measurement.findMany({
where: { userId: owner.id },
});
expect(rows.map((r) => r.id)).toEqual([kept.id]);
// With the key back, the same copy restores and the note opens.
useKeys({ old: "11".repeat(32), cur: "22".repeat(32) }, "cur");
const again = await restoreBackup({
backup,
actorUserId: owner.id,
ipAddress: null,
restoreInstanceSettings: false,
});
expect(again.ok).toBe(true);
const restored = await prisma.measurement.findFirstOrThrow({
where: { userId: owner.id },
});
expect(restored.notesEncrypted).not.toBeNull();
expect(
Buffer.from(restored.notesEncrypted!).toString("utf8").startsWith("old."),
).toBe(true);
});
it("refuses a file that files a tag under another account's category", async () => {
const prisma = getPrismaClient();
const owner = await seedOwner("tenant-owner");
const other = await seedOwner("tenant-other");
const foreign = await prisma.moodTagCategory.create({
data: {
userId: other.id,
key: "custom:other-category",
labelKey: "mood.tagCategory.custom",
},
});
const kept = await prisma.moodEntry.create({
data: {
userId: owner.id,
date: "2026-05-01",
mood: "OKAY",
score: 3,
moodLoggedAt: new Date("2026-05-01T20:00:00.000Z"),
},
});
const payload = {
...basePayload(owner.id),
moodEntries: [
{
date: "2026-05-08",
mood: "GUT",
score: 4,
tags: null,
source: "MOODLOG",
loggedAt: "2026-05-08T20:00:00.000Z",
structuredTags: ["custom:owner-tag"],
},
],
customMoodTags: [
{
key: "custom:owner-tag",
labelKey: "mood.tag.custom",
categoryId: foreign.id,
kind: "BINARY",
},
],
};
const backup = await storeBackup(owner.id, payload);
const outcome = await restoreBackup({
backup,
actorUserId: owner.id,
ipAddress: null,
restoreInstanceSettings: false,
});
expect(outcome.ok).toBe(false);
if (outcome.ok) return;
expect(outcome.code).toBe("backup.foreign_reference");
expect(
await prisma.moodTag.count({ where: { key: "custom:owner-tag" } }),
).toBe(0);
const moods = await prisma.moodEntry.findMany({
where: { userId: owner.id },
});
expect(moods.map((m) => m.id)).toEqual([kept.id]);
expect(outcome.message).toContain("mood_tags.category_id");
});
it("restores a file whose tag uses the account's own category", async () => {
const prisma = getPrismaClient();
const owner = await seedOwner("tenant-own");
const payload = {
...basePayload(owner.id),
moodEntries: [
{
date: "2026-05-08",
mood: "GUT",
score: 4,
tags: null,
source: "MOODLOG",
loggedAt: "2026-05-08T20:00:00.000Z",
structuredTags: ["custom:own-tag"],
},
],
customMoodTagCategories: [
{
id: "own-category-id",
key: "custom:own-category",
labelKey: "mood.tagCategory.custom",
},
],
customMoodTags: [
{
key: "custom:own-tag",
labelKey: "mood.tag.custom",
categoryId: "own-category-id",
kind: "BINARY",
},
],
};
const backup = await storeBackup(owner.id, payload);
const outcome = await restoreBackup({
backup,
actorUserId: owner.id,
ipAddress: null,
restoreInstanceSettings: false,
});
expect(outcome.ok).toBe(true);
const tag = await prisma.moodTag.findUniqueOrThrow({
where: { key: "custom:own-tag" },
});
expect(tag.userId).toBe(owner.id);
});
});