diff --git a/CHANGELOG.md b/CHANGELOG.md index a4e13ca4f..35a500464 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,14 @@ client that decoded it as a required field needs the field made optional or dropped; the `timezone` it sits beside is unchanged and is what the greeting should be derived from. +- The rotation script says what it did NOT look at. A zero that means "nothing + left" and a zero that means "never looked" read the same on a summary, and an + operator acts on that zero by dropping a key. Each run now ends with the + count of registered columns it walked, names any it skipped, and exits + non-zero when the two disagree. The runbook's retire-the-key step was + rewritten around that signal instead of around "the script ran cleanly", and + it tells an operator who already rotated on an older release how to get their + backups back. ### Fixed @@ -146,9 +154,40 @@ a backup that failed for that one account and is counted in the run's meta, not a restart for everybody. Backups written under either older shape still restore, which is checked both ways rather than asserted. +- **Backups were outside key rotation.** `DataBackup.data` holds every weekly + disaster-recovery snapshot and every uploaded pack, encrypted at rest like + everything else — but under a column called `data`, and the rotation registry + was keyed on the `*Encrypted` naming convention. So the rotation script never + read a backup, reported zero rows remaining, and the runbook told the + operator that zero meant the previous key was safe to retire. Following those + instructions to the letter made every stored backup permanently + undecryptable, which is the last thing that should break. The registry now + covers it, rotation walks it in bounded batches (one row is a whole + compressed account), and it re-seals the ciphertext without reading the + plaintext, so both stored envelopes — the plain JSON and the compressed one — + come back byte-identical, as will any later one. The same sweep found one + more: the idempotent-replay response cache, now rotated too. +- **Rotation crashed a third of the way through and said nothing.** Running the + script against a seeded database — rather than reading it — showed it dying + on the mood day-context note. That table is keyed on the entry it belongs to, + not on an `id`, and the walk asked for a column the database does not have. + Everything after it in the pass, the backups included, was never reached. The + walk now takes the model's actual key, and a check fails the build if a + registered column's key does not match the schema. +- The document index's verbatim text was rotating but never appeared in the + summary, so the new coverage check read it as skipped. It has its own line + now, with its own counts. ### Internal +- A column holding ciphertext is now recognised by what the code writes into + it, not by what it is called. A new guard traces the encryption helpers + through their wrappers, walks every Prisma write payload, and fails when a + column receives ciphertext without being registered for rotation — the + check that would have caught the backup blob nine releases ago. It asserts a + non-zero, anchored match count, so a matcher that quietly stops matching + fails instead of passing, and it was verified by breaking it three ways. + - The column-reader sweep was wrong in both directions and had been for as long as it existed. It answers "does this column have a consumer" for every column in the schema, and it is the tool this project leans on to catch a schema change that ships its writer and forgets its reader. Its idea of a write was any `field:` key anywhere, so `where: { ticketHash }` counted as writing `ticketHash`: all twelve columns it reported were a lookup key or a cron's discovery predicate being filtered on, every one a false alarm. And because it only ever reported a column that HAD a write, a column with no write at all was the one thing it could not see. It now understands which Prisma argument a key sits in, follows a payload assembled into a variable before the call, credits raw SQL against the mapped column name, and reports a column no code touches at all as its own category. Twelve false alarms became nineteen findings that each hold up, four of them worth acting on. The matcher underneath is pinned by a test that was checked by breaking it three ways. - **Two columns have a reader and no writer.** The glucose unit preference is read across twenty-nine files, the dashboard and both exports and the doctor report and the FHIR resources among them, and no surface anywhere writes it, so an account cannot leave mg/dL. An import job's export date is described in the schema as parsed during unpack, is returned by the import status endpoint and is in the published contract, but the parser skips Apple's `ExportDate` element outright: the field has always been null. - **Two columns have a writer and no reader.** The provider-health ledger records the HTTP status of a failed call and nothing reads it back, and the known-device ledger stamps a first-seen date that no query, page or export ever asks for. diff --git a/docs/ops/encryption-key-rotation.md b/docs/ops/encryption-key-rotation.md index 9c28dad2a..007dd4fb9 100644 --- a/docs/ops/encryption-key-rotation.md +++ b/docs/ops/encryption-key-rotation.md @@ -41,35 +41,76 @@ under `v1` (the synthetic id assigned to the existing `ENCRYPTION_KEY`). Restart the app. New writes are now encrypted under `v2`; existing `v1`-keyed and legacy bare rows still decrypt because the `v1` entry is retained. -3. **Run the rotation script** to re-encrypt every existing encrypted column +3. **Run the rotation script** to re-encrypt every registered encrypted column under the new active key: + ``` pnpm dlx tsx scripts/rotate-encryption-key.ts v2 ``` + The script is idempotent — running it again is a no-op for rows already - prefixed with `v2.`. It rotates every registered encrypted column across - all models, plus the two non-`*Encrypted` members: - `IntegrationStatus.lastError` and `CoachMessage.encryptedContent` - (Bytes). The canonical registry lives in - `src/lib/crypto/encrypted-columns.ts`; a guard test fails CI if any - `*Encrypted` column in the schema is missing from the registry or not - referenced by the script. The summary line at the - end shows `scanned`, `rotated`, and `errors` per table+field; treat - `errors > 0` as a hard failure and re-run after fixing the cause. -4. **Drop the old key (optional, recommended).** Once the script has run - cleanly: + prefixed with `v2.`. It rotates every column in the canonical registry + (`src/lib/crypto/encrypted-columns.ts`), which covers the `*Encrypted` + columns plus the ones whose names say nothing about their contents: + `IntegrationStatus.lastError`, `CoachMessage.encryptedContent`, + `NotificationChannel.config`, the OAuth `accessToken` / `refreshToken` + columns, the web-push `p256dh` / `auth` secrets, the idempotent-replay + `IdempotencyKey.responseBody`, and `DataBackup.data` — the whole-account + backup blob. + + Read three things off the output before going further: + + - the per-column line, `scanned` / `rotated` / `errors` / `dropped`. Treat + `errors > 0` as a hard failure and re-run after fixing the cause. A + `dropped` count is only ever a cache row the run could not read and + deleted rather than leave unreadable. + - `Columns walked: N/M registered`. The two numbers must match. + - a `NOT WALKED` block, if one appears. It lists registered columns this + run skipped, and the run exits non-zero. Rotation is incomplete; do not + go on to step 4. + + Two guard tests keep the registry honest in CI: one scans the schema for + `*Encrypted` columns, the other derives the ciphertext-bearing columns from + the Prisma write payloads, which is what catches a column that holds + ciphertext under an ordinary name. + +4. **Confirm nothing is left on the old key, THEN drop it.** This is the step + that destroys data if it is taken on a bad signal, so check the corpus + rather than the absence of complaints. Either re-run the script — a clean + second pass reports `rotated=0`, `errors=0` and a full `Columns walked` + line — or open **Admin → Encryption** and confirm the status view reports + zero rows outside the active key. Both read the same registry the rotation + walks. + + A zero only means "nothing left" when the run also says it walked every + registered column. A zero from a run that skipped columns means "never + looked", and dropping the old key on it makes those rows permanently + undecryptable — `decrypt()` is fail-closed and there is no recovery path. + Only once the corpus reads clean: + ``` ENCRYPTION_KEYS='{"v2":""}' ENCRYPTION_ACTIVE_KEY_ID="v2" # ENCRYPTION_KEY can now be removed ``` + Restart. The legacy single-key fallback is now disconnected; only `v2` exists. +> **Backups are covered, and were not always.** `DataBackup.data` holds every +> weekly disaster-recovery snapshot and every uploaded pack, encrypted like +> everything else but under a column called `data`. It was outside the +> registry until v1.38.6, so a rotation before that release reported zero +> rows remaining without ever reading a backup. If you rotated on an older +> release and dropped the previous key, the stored backups are encrypted +> under the key you removed: put that key back into `ENCRYPTION_KEYS` and +> re-run the rotation on this release before removing it again. + ## Adding a third key (v2 → v3) -Same procedure, just shift the labels — keep `v2` in the map until the -script reports zero `v2.`-prefixed rows remaining. +Same procedure, just shift the labels — keep `v2` in the map until a full +run (every registered column walked) reports zero `v2.`-prefixed rows +remaining. ``` ENCRYPTION_KEYS='{"v2":"","v3":""}' diff --git a/scripts/rotate-encryption-key.ts b/scripts/rotate-encryption-key.ts index 811f55c2b..ff2367ff7 100644 --- a/scripts/rotate-encryption-key.ts +++ b/scripts/rotate-encryption-key.ts @@ -6,13 +6,16 @@ * already carries the active key id prefix are skipped. * * The set of columns this script rotates is the canonical registry in - * `src/lib/crypto/encrypted-columns.ts`. The guard test - * `src/lib/crypto/__tests__/encrypted-columns.test.ts` fails CI if any - * encrypted column in `prisma/schema.prisma` is missing from the registry, - * or if any registry column is not referenced here — so a new `*Encrypted` - * column can never silently skip rotation (which would make those rows - * permanently undecryptable once the legacy key is dropped, since decrypt is - * fail-closed). + * `src/lib/crypto/encrypted-columns.ts`. Two guard tests keep it honest: + * `encrypted-columns.test.ts` fails CI if a `*Encrypted`-named schema column + * is missing from the registry or is not referenced here, and + * `encrypted-column-writers.test.ts` derives the ciphertext-bearing columns + * from the Prisma write payloads instead of from names — the check that would + * have caught `DataBackup.data`, ciphertext under a column called `data`. + * + * The run ends by naming every registered column it did NOT walk and exiting + * non-zero, because an operator reads a zero here as permission to drop the + * previous key, and "nothing left" and "never looked" must not print alike. * * Usage: * ENCRYPTION_KEYS='{"v1":"","v2":""}' \ @@ -28,6 +31,11 @@ import { rotateColumn, type CorpusClient, } from "@/lib/crypto/encryption-corpus"; +import { + ENCRYPTED_COLUMNS, + encryptedColumnKey, + type EncryptedColumn, +} from "@/lib/crypto/encrypted-columns"; import { tokeniseAndHash } from "@/lib/documents/content-index"; const DATABASE_URL = process.env.DATABASE_URL; @@ -55,6 +63,42 @@ interface RotationResult { scanned: number; rotated: number; errors: number; + /** Unreadable rows deleted from a `disposable` column (cache entries only). */ + dropped: number; +} + +/** + * Look a column up in the canonical registry so the walk honours the flags + * recorded there (`batched`, `disposable`, `codecField`) rather than a second + * hand-written copy of them here. + */ +function registryColumn(model: string, field: string): EncryptedColumn { + const col = ENCRYPTED_COLUMNS.find( + (c) => c.model === model && c.field === field, + ); + if (!col) { + throw new Error( + `${model}.${field} is not in the encrypted-column registry`, + ); + } + return col; +} + +/** Rotate one column through the shared registry-driven corpus walk. */ +async function rotateRegistryColumn( + model: string, + field: string, + client: CorpusClient, +): Promise { + const r = await rotateColumn(client, registryColumn(model, field)); + return { + table: r.model, + field: r.field, + scanned: r.scanned, + rotated: r.rotated, + errors: r.errors, + dropped: r.dropped, + }; } function shouldRotate(value: string | null): boolean { @@ -89,6 +133,7 @@ async function rotateStringColumn( scanned: rows.length, rotated: 0, errors: 0, + dropped: 0, }; for (const row of rows) { const v = row[field] as string | null; @@ -131,6 +176,7 @@ async function rotateBytesColumn( scanned: rows.length, rotated: 0, errors: 0, + dropped: 0, }; for (const row of rows) { const buf = row[field] as Uint8Array | null; @@ -487,13 +533,14 @@ async function main() { // v1.38 — the day-context note on a mood entry. Its own table, so its own // walk; a context row exists only where somebody added one, and a row with // no note carries NULL here and is skipped like every other nullable Bytes - // column. + // column. It goes through the registry-driven walk rather than the generic + // `id`-keyed one above: the table is keyed on `moodEntryId`, and selecting a + // column Prisma does not know threw mid-pass and abandoned every column + // after it — a rotation that reported nothing at all rather than a gap. results.push( - await rotateBytesColumn( - "MoodContext", - "notesEncrypted", - prisma.moodContext, - ), + await rotateRegistryColumn("MoodContext", "notesEncrypted", { + moodContext: prisma.moodContext, + } as unknown as CorpusClient), ); // ───── v1.25 medication free-text notes (Bytes columns) ───── @@ -601,6 +648,7 @@ async function main() { scanned: docResult.scanned, rotated: docResult.rotated, errors: docResult.errors, + dropped: docResult.dropped, }); } // The staged extracted-fact payloads: the FHIR-staged clinical values and the @@ -650,6 +698,19 @@ async function main() { scanned: 0, rotated: 0, errors: 0, + dropped: 0, + }; + // The verbatim sibling rides the same update, but it needs its own line in + // the summary: the coverage check reads the results to decide which + // registered columns this run actually looked at, and a column that + // rotates without ever being named there reads as skipped. + const verbatim: RotationResult = { + table: "DocumentContentIndex", + field: "verbatimTextEncrypted", + scanned: 0, + rotated: 0, + errors: 0, + dropped: 0, }; let cursor: string | null = null; // v1.27.33 (Document vault P4) — re-encrypt the string-shaped BYTEA into a @@ -676,18 +737,19 @@ async function main() { if (!buf || buf.byteLength === 0) continue; const asString = Buffer.from(buf).toString("utf8"); if (!shouldRotate(asString)) continue; + // The "verbatimTextEncrypted" column (nullable; written together with + // "textEncrypted" so it shares the same key) rotates in the same + // update when present. Read outside the try so the catch can count it. + const verbatimBuf = row.verbatimTextEncrypted as Uint8Array | null; + const hasVerbatim = Boolean(verbatimBuf && verbatimBuf.byteLength > 0); + if (hasVerbatim) verbatim.scanned++; try { const plaintext = decrypt(asString); const nextBytes = reEncryptBytes(asString); const searchTokens = tokeniseAndHash(plaintext); - // The "verbatimTextEncrypted" column (nullable; written together with - // "textEncrypted" so it shares the same key) rotates in the same - // update when present. - const verbatimBuf = row.verbatimTextEncrypted as Uint8Array | null; - const verbatimNext = - verbatimBuf && verbatimBuf.byteLength > 0 - ? reEncryptBytes(Buffer.from(verbatimBuf).toString("utf8")) - : undefined; + const verbatimNext = hasVerbatim + ? reEncryptBytes(Buffer.from(verbatimBuf!).toString("utf8")) + : undefined; await prisma.documentContentIndex.update({ where: { id: row.id }, data: { @@ -697,8 +759,10 @@ async function main() { }, }); result.rotated++; + if (verbatimNext) verbatim.rotated++; } catch (err) { result.errors++; + if (hasVerbatim) verbatim.errors++; console.error( `[DocumentContentIndex.textEncrypted] row ${row.id}: ${(err as Error).message}`, ); @@ -708,6 +772,7 @@ async function main() { if (rows.length < 100) break; } results.push(result); + results.push(verbatim); } // ───── Document preview thumbnails (Bytes column) ───── @@ -723,18 +788,71 @@ async function main() { ), ); + // ───── Whole-account backup blob (String, batched) ───── + // `DataBackup.data` holds `packBackupBlob()` output: AES-256-GCM ciphertext + // under a column that does NOT carry the `*Encrypted` suffix, which is why + // it sat outside rotation until v1.38.6. Missing it was the worst possible + // miss — the script reported zero rows remaining, the runbook told the + // operator that zero meant safe to drop the old key, and every stored backup + // became undecryptable. Walked in bounded id-cursor batches because one row + // is an entire compressed account, and re-sealed WITHOUT reading the + // plaintext, so both stored envelopes (the plain backup JSON and the + // `HLZ1:` gzip form) — and any later one — rotate unchanged. + results.push( + await rotateRegistryColumn("DataBackup", "data", { + dataBackup: prisma.dataBackup, + } as unknown as CorpusClient), + ); + + // ───── Idempotent-replay response cache (String, disposable) ───── + // `IdempotencyKey.responseBody` is the cached response body, encrypted + // because the PHI-returning creates echo their own decrypted DTO. Rotating + // it keeps a key drop from handing a replaying client a body it cannot + // parse; a row that cannot be read is deleted rather than counted as an + // error, because a cache miss only costs a re-run. + results.push( + await rotateRegistryColumn("IdempotencyKey", "responseBody", { + idempotencyKey: prisma.idempotencyKey, + } as unknown as CorpusClient), + ); + console.log("\n=== Rotation summary ==="); let totalRotated = 0; let totalErrors = 0; + let totalDropped = 0; for (const r of results) { console.log( - `${r.table}.${r.field}: scanned=${r.scanned} rotated=${r.rotated} errors=${r.errors}`, + `${r.table}.${r.field}: scanned=${r.scanned} rotated=${r.rotated} ` + + `errors=${r.errors} dropped=${r.dropped}`, ); totalRotated += r.rotated; totalErrors += r.errors; + totalDropped += r.dropped; + } + console.log( + `\nTOTAL rotated=${totalRotated} errors=${totalErrors} dropped=${totalDropped}`, + ); + + // ───── Coverage: what was NOT looked at ───── + // A zero that means "nothing left" and a zero that means "never looked" read + // the same on the summary above, and the operator acts on that zero by + // dropping the old key. So say which registered columns this run actually + // walked, and refuse to report success if any of them was skipped. + const walked = new Set(results.map((r) => `${r.table}.${r.field}`)); + const notWalked = ENCRYPTED_COLUMNS.filter( + (c) => !walked.has(encryptedColumnKey(c)), + ).map(encryptedColumnKey); + console.log( + `\nColumns walked: ${walked.size}/${ENCRYPTED_COLUMNS.length} registered`, + ); + if (notWalked.length > 0) { + console.error( + `\nNOT WALKED (${notWalked.length}) — rotation is INCOMPLETE, do not ` + + `drop the previous key:\n ${notWalked.join("\n ")}`, + ); } - console.log(`\nTOTAL rotated=${totalRotated} errors=${totalErrors}`); await prisma.$disconnect(); + if (notWalked.length > 0) process.exit(4); process.exit(totalErrors > 0 ? 3 : 0); } diff --git a/src/lib/crypto/__tests__/encrypted-column-writers.test.ts b/src/lib/crypto/__tests__/encrypted-column-writers.test.ts new file mode 100644 index 000000000..f12982cf2 --- /dev/null +++ b/src/lib/crypto/__tests__/encrypted-column-writers.test.ts @@ -0,0 +1,461 @@ +/** + * Derive the ciphertext-bearing columns FROM THE CODE, then assert the + * key-rotation registry covers every one of them. + * + * Why this exists next to `encrypted-columns.test.ts`. That guard scans + * `schema.prisma` for the `*Encrypted` suffix plus a hand-kept list of + * historically-named exceptions. The suffix is a naming convention, not a + * fact about the value, and the moment a column holding ciphertext is called + * something else the guard goes green while the column is unrotatable. + * `DataBackup.data` — the whole-account backup blob — sat outside rotation + * that way: encrypted on write, invisible to a name scan, and therefore + * reported as "zero rows remaining" by a run that never looked at it. The + * runbook then told the operator that zero meant the old key was safe to + * drop, which would have destroyed every backup in the deployment. + * + * How the derivation works, in three steps: + * + * 1. Start from the two primitives in `src/lib/crypto.ts` (`encrypt`, + * `encryptBytes`) and close over every function that RETURNS a value + * derived from one — `packBackupBlob`, `encryptNote`, the per-domain + * `encrypt*ToBytes` codecs, and so on. Taint propagates only through + * pure expressions; anything awaited or read back from Prisma is a + * round-trip, not a ciphertext this code produced. + * 2. Walk every `prisma|tx|db..(...)` call, descend through + * the Prisma envelope keys (`data` / `create` / `update`, and relation + * writes resolved against the schema) into the payload objects, and + * record `.` for each field whose value is a producer + * call or a variable derived from one. + * 3. Assert the result is a subset of the rotation registry. + * + * Known limit, stated so it is not mistaken for coverage: a payload assembled + * somewhere other than the call site (`data: updates` where `updates` is built + * field by field) resolves to nothing here. Those columns are covered by the + * name scan in the sibling guard; the two signals are deliberately + * independent, and a column has to escape BOTH to go unregistered. The scan + * asserts a non-zero, anchored match count, so a matcher that quietly stops + * matching fails instead of passing. + */ +import { describe, it, expect } from "vitest"; +import { readFileSync, readdirSync, statSync } from "node:fs"; +import { join } from "node:path"; +import { ENCRYPTED_COLUMNS, encryptedColumnKey } from "../encrypted-columns"; + +const ROOT = join(__dirname, "../../../.."); +const SRC = join(ROOT, "src"); +const SCHEMA_PATH = join(ROOT, "prisma", "schema.prisma"); + +/** Prisma write methods whose argument carries a column payload. */ +const WRITE_METHODS = + "create|createMany|createManyAndReturn|update|updateMany|upsert"; +/** Prisma envelope keys whose value is a payload object, not a column. */ +const ENVELOPE = new Set(["data", "create", "update"]); +/** Relation-write envelopes nested inside a payload value. */ +const RELATION_ENVELOPE = new Set([ + "create", + "update", + "upsert", + "createMany", + "connectOrCreate", +]); + +/** + * Columns this scan derives that are deliberately NOT in the rotation + * registry. Empty today. An entry here is a written decision with a reason, + * never a way to quiet the guard. + */ +const NOT_ROTATED: ReadonlySet = new Set(); + +function sourceFiles(dir: string, out: string[] = []): string[] { + for (const entry of readdirSync(dir)) { + const p = join(dir, entry); + if (statSync(p).isDirectory()) { + if (entry === "generated" || entry === "__tests__") continue; + sourceFiles(p, out); + } else if (/\.tsx?$/.test(entry) && !/\.test\.tsx?$/.test(entry)) { + out.push(p); + } + } + return out; +} + +/** Strip comments so prose about `encrypt()` never counts as a call. */ +function stripComments(src: string): string { + return src + .replace(/\/\*[\s\S]*?\*\//g, "") + .replace(/(^|[^:"'`\\])\/\/.*$/gm, "$1"); +} + +const callRe = (names: string[]) => + new RegExp(`(? + new RegExp(`(? + new RegExp(`(? { + const assigns = [ + ...body.matchAll(/(?:const|let|var)\s+(\w+)[^;=\n]*=\s*([^;]*)/g), + ]; + const tainted = new Set(); + for (let pass = 0; pass < 8; pass++) { + const before = tainted.size; + const producerCall = callRe(producers); + const strictToken = tainted.size ? tokenRe([...tainted]) : null; + const looseToken = tainted.size ? tokenReLoose([...tainted]) : null; + for (const [, name, init] of assigns) { + if (tainted.has(name)) continue; + if (producerCall.test(init)) { + tainted.add(name); + continue; + } + // Propagate through pure expressions only: a value awaited or read back + // from the database is a round-trip, not ciphertext this code made. + if (/\bawait\b|\b(?:prisma|tx|db)\./.test(init)) continue; + // A member read normally carries no taint (`creds.expiresAt` is a + // timestamp). The one exception is a buffer being CONSTRUCTED around a + // ciphertext — `new Uint8Array(encoded.byteLength)` then `.set(encoded)` + // is how every Bytes codec here builds its result, and reading the + // length off the ciphertext is the only link the initialiser shows. + const allowMemberRead = loose && /\bnew\s+[A-Z]|Buffer\./.test(init); + const token = allowMemberRead ? looseToken : strictToken; + if (token?.test(init)) tainted.add(name); + } + if (tainted.size === before) break; + } + return tainted; +} + +interface Fn { + name: string; + body: string; +} + +function declaredFunctions(src: string): Fn[] { + const out: Fn[] = []; + const heads = [ + ...src.matchAll( + /(?:^|\n)\s*(?:export\s+)?(?:async\s+)?function\s+(\w+)\s*[(<]/g, + ), + ]; + for (let i = 0; i < heads.length; i++) { + const end = i + 1 < heads.length ? heads[i + 1].index! : src.length; + out.push({ name: heads[i][1], body: src.slice(heads[i].index!, end) }); + } + for (const m of src.matchAll( + /(?:const|let)\s+(\w+)\s*(?::[^=\n]+)?=\s*(?:async\s*)?\([^)]*\)\s*(?::[^=\n]+)?=>/g, + )) { + // Bound the arrow at its own body. A fixed-size window instead would read + // the code AFTER the arrow as part of it, and a tiny id-resolving helper + // declared above a Prisma write would inherit that write's ciphertext. + const arrow = m.index! + m[0].length; + const brace = src.slice(arrow).search(/\S/) + arrow; + const end = + src[brace] === "{" ? balanced(src, brace) : endOfValue(src, brace); + out.push({ name: m[1], body: src.slice(m.index!, end + 1) }); + } + return out; +} + +/** Fixpoint: every function that returns something derived from `encrypt`. */ +function ciphertextProducers(sources: string[]): Set { + const funcs = sources.flatMap(declaredFunctions); + const producers = new Set(["encrypt", "encryptBytes"]); + for (let pass = 0; pass < 8; pass++) { + const before = producers.size; + for (const fn of funcs) { + if (producers.has(fn.name)) continue; + const names = [...producers]; + const tainted = taintedIdentifiers(fn.body, names, true); + const producerCall = callRe(names); + const taintedToken = tainted.size ? tokenReLoose([...tainted]) : null; + const returnsCiphertext = [ + ...fn.body.matchAll(/\breturn\b([^;]*)/g), + ].some((m) => producerCall.test(m[1]) || taintedToken?.test(m[1])); + if (returnsCiphertext) producers.add(fn.name); + } + if (producers.size === before) break; + } + return producers; +} + +// ── brace / paren walking ──────────────────────────────────────────────── + +/** Index of the top-level comma (or closer) ending the value starting at `i`. */ +function endOfValue(s: string, i: number): number { + let depth = 0; + while (i < s.length) { + const c = s[i]; + if (c === '"' || c === "'" || c === "`") { + const quote = c; + i++; + while (i < s.length && s[i] !== quote) { + if (s[i] === "\\") i++; + i++; + } + i++; + continue; + } + if ("({[".includes(c)) depth++; + else if (")}]".includes(c)) { + if (depth === 0) return i; + depth--; + } else if (c === "," && depth === 0) return i; + i++; + } + return i; +} + +/** Index of the closer matching the opener at `i`. */ +function balanced(s: string, i: number): number { + const open = s[i]; + const close = { "(": ")", "{": "}", "[": "]" }[open]!; + let depth = 0; + for (; i < s.length; i++) { + const c = s[i]; + if (c === '"' || c === "'" || c === "`") { + const quote = c; + i++; + while (i < s.length && s[i] !== quote) { + if (s[i] === "\\") i++; + i++; + } + continue; + } + if (c === open) depth++; + else if (c === close && --depth === 0) return i; + } + return s.length - 1; +} + +/** Top-level `key: value` pairs of an object literal (braces included). */ +function objectPairs(obj: string): Array<[string, string, number]> { + const out: Array<[string, string, number]> = []; + let i = 1; + while (i < obj.length - 1) { + const m = /^[\s,]*(\w+|"[^"]*"|'[^']*')\s*:/.exec(obj.slice(i)); + if (!m) { + const end = endOfValue(obj, i); + if (end <= i) break; + i = end + 1; + continue; + } + const start = i + m[0].length; + const end = endOfValue(obj, start); + out.push([m[1].replace(/^["']|["']$/g, ""), obj.slice(start, end), start]); + i = end + 1; + } + return out; +} + +/** Object literals inside a payload value: `{…}`, `[{…}]`, or `map(() => ({…}))`. */ +function payloadObjects(value: string): Array<[string, number]> { + const trimmed = value.trim(); + if (trimmed.length === 0) return []; + const at = value.indexOf(trimmed[0]); + if (trimmed.startsWith("{")) { + return [[value.slice(at, balanced(value, at) + 1), at]]; + } + if (trimmed.startsWith("[")) { + const arr = value.slice(at, balanced(value, at) + 1); + const out: Array<[string, number]> = []; + let i = 1; + while (i < arr.length - 1) { + while (i < arr.length && /[\s,]/.test(arr[i])) i++; + if (arr[i] === "{") { + const end = balanced(arr, i); + out.push([arr.slice(i, end + 1), at + i]); + i = end + 1; + } else { + const end = endOfValue(arr, i); + if (end <= i) break; + i = end + 1; + } + } + return out; + } + const out: Array<[string, number]> = []; + for (const m of value.matchAll(/=>\s*\(?\s*\{/g)) { + const brace = value.indexOf("{", m.index!); + out.push([value.slice(brace, balanced(value, brace) + 1), brace]); + } + return out; +} + +// ── schema ─────────────────────────────────────────────────────────────── + +interface Schema { + models: Set; + fieldType: Map; +} + +function parseSchema(): Schema { + const src = readFileSync(SCHEMA_PATH, "utf8"); + const models = new Set(); + const fieldType = new Map(); + let model: string | null = null; + for (const raw of src.split("\n")) { + const line = raw.trim(); + const head = /^model\s+(\w+)\s*\{/.exec(line); + if (head) { + model = head[1]; + models.add(model); + continue; + } + if (line === "}") { + model = null; + continue; + } + if (!model || line === "" || line.startsWith("//")) continue; + const field = /^(\w+)\s+([A-Za-z]\w*)(\?|\[\])?/.exec(line); + if (field) fieldType.set(`${model}.${field[1]}`, field[2]); + } + return { models, fieldType }; +} + +// ── the scan ───────────────────────────────────────────────────────────── + +function scanCiphertextWrites(): Map> { + const files = sourceFiles(SRC).map( + (f) => [f, stripComments(readFileSync(f, "utf8"))] as const, + ); + const producers = [...ciphertextProducers(files.map(([, s]) => s))]; + const schema = parseSchema(); + const found = new Map>(); + + for (const [path, src] of files) { + // The rotation machinery re-encrypts what other code wrote; it is not a + // writer of new columns, and scanning it would fold its own generic + // `data: { [col.field]: … }` into the result. + if (path.includes("/lib/crypto/")) continue; + const tainted = taintedIdentifiers(src, producers, false); + const producerCall = callRe(producers); + const taintedToken = tainted.size ? tokenRe([...tainted]) : null; + const isCiphertext = (expr: string) => + producerCall.test(expr) || Boolean(taintedToken?.test(expr)); + const rel = path.slice(ROOT.length + 1); + const lineAt = (idx: number) => src.slice(0, idx).split("\n").length; + + const record = (model: string, field: string, idx: number) => { + const key = `${model}.${field}`; + if (!found.has(key)) found.set(key, new Set()); + found.get(key)!.add(`${rel}:${lineAt(idx)}`); + }; + + const descend = ( + model: string, + value: string, + base: number, + depth = 0, + ): void => { + if (depth > 4) return; + for (const [obj, objOffset] of payloadObjects(value)) { + for (const [field, expr, exprOffset] of objectPairs(obj)) { + const at = base + objOffset + exprOffset; + const type = schema.fieldType.get(`${model}.${field}`); + if (type && schema.models.has(type)) { + // Relation field: the nested payload belongs to the child model. + descend(type, expr, at, depth + 1); + continue; + } + if (!type) { + if (RELATION_ENVELOPE.has(field)) + descend(model, expr, at, depth + 1); + continue; + } + if (isCiphertext(expr)) record(model, field, at); + } + } + }; + + const writes = new RegExp( + `\\b(?:prisma|tx|db)\\s*\\.\\s*(\\w+)\\s*\\.\\s*(?:${WRITE_METHODS})\\s*\\(`, + "g", + ); + for (const m of src.matchAll(writes)) { + const model = m[1][0].toUpperCase() + m[1].slice(1); + if (!schema.models.has(model)) continue; + const paren = m.index! + m[0].length - 1; + const arg = src.slice(paren, balanced(src, paren) + 1); + const brace = arg.indexOf("{"); + if (brace < 0) continue; + const obj = arg.slice(brace, balanced(arg, brace) + 1); + for (const [key, value, offset] of objectPairs(obj)) { + if (!ENVELOPE.has(key)) continue; + descend(model, value, paren + brace + offset); + } + } + } + return found; +} + +describe("ciphertext columns derived from the code", () => { + const producers = ciphertextProducers( + sourceFiles(SRC).map((f) => stripComments(readFileSync(f, "utf8"))), + ); + const written = scanCiphertextWrites(); + + it("closes over the encryption wrappers, not just the two primitives", () => { + // A closure that collapsed back to the seeds would make the write scan + // blind to every column written through a codec helper. + expect(producers.size).toBeGreaterThan(10); + for (const wrapper of [ + "packBackupBlob", + "encryptNote", + "encryptToBytes", + "encryptCachedBody", + ]) { + expect([...producers]).toContain(wrapper); + } + }); + + it("finds ciphertext writes, including the non-suffixed columns", () => { + // Anchored non-zero match count. An empty or collapsed result set is the + // failure mode this whole file exists to make impossible, so name the + // columns whose only signal is the write and not the column's name. + expect(written.size).toBeGreaterThan(30); + for (const anchor of [ + "DataBackup.data", + "IdempotencyKey.responseBody", + "NotificationChannel.config", + "PushSubscription.p256dh", + "IntegrationStatus.lastError", + "WithingsConnection.accessToken", + ]) { + expect( + [...written.keys()], + `${anchor} is written with ciphertext but the scan did not see it`, + ).toContain(anchor); + } + }); + + it("registers every column the code writes ciphertext into", () => { + const registered = new Set(ENCRYPTED_COLUMNS.map(encryptedColumnKey)); + const unregistered = [...written.keys()] + .filter((key) => !registered.has(key) && !NOT_ROTATED.has(key)) + .sort() + .map((key) => `${key} (written at ${[...written.get(key)!][0]})`); + expect( + unregistered, + "columns written with AES-256-GCM ciphertext but absent from the " + + "key-rotation registry — dropping a retired key makes these rows " + + "permanently undecryptable", + ).toEqual([]); + }); +}); diff --git a/src/lib/crypto/__tests__/encrypted-columns.test.ts b/src/lib/crypto/__tests__/encrypted-columns.test.ts index b46b2b445..62ad11d73 100644 --- a/src/lib/crypto/__tests__/encrypted-columns.test.ts +++ b/src/lib/crypto/__tests__/encrypted-columns.test.ts @@ -127,21 +127,29 @@ describe("encrypted-column registry", () => { }); it("covers EVERY encrypted column in prisma/schema.prisma", () => { - const schemaEncrypted = parseSchemaColumns() + const schema = parseSchemaColumns(); + const schemaEncrypted = schema .filter(isEncryptedColumn) .map((c) => `${c.model}.${c.field}`) .sort(); + const allSchemaColumns = new Set( + schema.map((c) => `${c.model}.${c.field}`), + ); const registered = ENCRYPTED_COLUMNS.map(encryptedColumnKey).sort(); // Any schema column the registry forgot is a rotation gap (data-loss // risk on key drop). Any registry column missing from the schema is a // stale entry. Both fail here. + // + // Staleness is checked against the WHOLE schema, not against the name + // scan: the registry legitimately carries columns this scan cannot see — + // `DataBackup.data` holds ciphertext under a name the convention does not + // reach — and the sibling guard `encrypted-column-writers.test.ts` is + // what finds those, by tracing the writers instead of the names. const missingFromRegistry = schemaEncrypted.filter( (k) => !registered.includes(k), ); - const staleInRegistry = registered.filter( - (k) => !schemaEncrypted.includes(k), - ); + const staleInRegistry = registered.filter((k) => !allSchemaColumns.has(k)); expect( missingFromRegistry, "encrypted schema columns NOT wired into the rotation registry", @@ -152,6 +160,42 @@ describe("encrypted-column registry", () => { ).toEqual([]); }); + it("declares the primary key for every model not keyed on `id`", () => { + // The rotation walk selects, orders by and addresses rows through this + // field. Get it wrong and Prisma rejects the select at runtime, which + // abandons the rest of the pass — a rotation that stops early looks the + // same from the outside as one that had nothing to do. + const src = readFileSync(SCHEMA_PATH, "utf8"); + const primaryKey = new Map(); + let model: string | null = null; + for (const rawLine of src.split("\n")) { + const line = rawLine.trim(); + const head = /^model\s+(\w+)\s*\{/.exec(line); + if (head) { + model = head[1]; + continue; + } + if (line === "}") { + model = null; + continue; + } + if (!model) continue; + const field = /^(\w+)\s+[A-Za-z]\w*\??\s.*@id\b/.exec(line); + if (field) primaryKey.set(model, field[1]); + } + + const wrong = (ENCRYPTED_COLUMNS as EncryptedColumn[]) + .filter((c) => (c.pkField ?? "id") !== primaryKey.get(c.model)) + .map( + (c) => + `${encryptedColumnKey(c)} (schema key: ${primaryKey.get(c.model) ?? "none"})`, + ); + expect( + wrong, + "registry entries whose pkField does not match the model's @id column", + ).toEqual([]); + }); + it("is fully referenced by the key-rotation script", () => { const script = readFileSync(ROTATION_SCRIPT_PATH, "utf8"); // Field names are NOT unique across models (e.g. `noteEncrypted` lives on diff --git a/src/lib/crypto/__tests__/encryption-corpus.test.ts b/src/lib/crypto/__tests__/encryption-corpus.test.ts index 51216365c..39d9e6551 100644 --- a/src/lib/crypto/__tests__/encryption-corpus.test.ts +++ b/src/lib/crypto/__tests__/encryption-corpus.test.ts @@ -45,9 +45,17 @@ function makeClient(seed: Record): { if (client[key]) continue; const model = col.model; client[key] = { - findMany: async ({ select }) => { + // `take` / `cursor` are honoured so the batched-column walk terminates + // here the way it does against Postgres. + findMany: async ({ select, take, cursor, skip }) => { const fields = Object.keys(select); - return store[model].map((row) => { + let rows = store[model]; + if (cursor) { + const at = rows.findIndex((r) => r.id === cursor.id); + rows = at < 0 ? [] : rows.slice(at + (skip ?? 0)); + } + if (take !== undefined) rows = rows.slice(0, take); + return rows.map((row) => { const out: Record = {}; for (const f of fields) out[f] = row[f]; return out; @@ -58,6 +66,10 @@ function makeClient(seed: Record): { if (row) Object.assign(row, data); return row; }, + delete: async ({ where }) => { + const at = store[model].findIndex((r) => r.id === where.id); + return at < 0 ? null : store[model].splice(at, 1)[0]; + }, }; } return { client, store }; diff --git a/src/lib/crypto/encrypted-columns.ts b/src/lib/crypto/encrypted-columns.ts index 9ecca647b..1133fcf8b 100644 --- a/src/lib/crypto/encrypted-columns.ts +++ b/src/lib/crypto/encrypted-columns.ts @@ -24,6 +24,17 @@ * reads/writes the value directly. * - "bytes" — the ciphertext is stored as a `Bytes` column; the script * goes through a UTF-8 Buffer round-trip (matching the persistence layer). + * + * The `*Encrypted` suffix is NOT what makes a column belong here. It is a + * naming convention that happens to hold for most of them, and treating it as + * the definition is how `DataBackup.data` — the whole-account backup blob — + * went unregistered until v1.38.6: encrypted on write, invisible to + * a registry keyed on names, and therefore reported as "zero rows remaining" + * by a rotation that never looked at it. An operator following the runbook + * would then have dropped the old key and lost every backup. The guard test + * `encrypted-column-writers.test.ts` now derives the ciphertext-bearing set + * from the Prisma write payloads instead, so a column earns its place here by + * what the code stores in it, not by what it is called. */ export type EncryptedColumnKind = "string" | "bytes"; @@ -43,6 +54,30 @@ export interface EncryptedColumn { * multi-megabyte blobs, so an unbounded `findMany` would balloon memory. */ readonly codecField?: string; + /** + * Walk this column in bounded id-cursor batches instead of one `findMany`. + * Set it on any column whose rows are blobs rather than short strings — a + * whole-account backup compresses to megabytes, and pulling every row of + * them into one array is how the weekly backup pass died before + * `packBackupBlob` existed. `codecField` implies batching. + */ + readonly batched?: boolean; + /** + * The column is a cache, not a record. Rotation still re-encrypts it, but a + * row it cannot read is DELETED rather than counted as an error: the value + * is reproducible, and leaving an undecryptable row behind would hand the + * next reader a body it cannot parse. Never set this on anything a user + * would notice the loss of. + */ + readonly disposable?: boolean; + /** + * The model's primary-key field, when it is not `id`. The rotation walk + * selects it, orders by it and addresses each row through it, so a model + * keyed on something else (`MoodContext.moodEntryId`) crashes the whole run + * without this — Prisma rejects the unknown `id` in the select, and every + * column after it in the pass is left un-rotated. + */ + readonly pkField?: string; } /** @@ -226,7 +261,12 @@ export const ENCRYPTED_COLUMNS: readonly EncryptedColumn[] = [ // v1.38 — the one free-text note on a mood entry's day context. Same codec // as the entry note beside it; one column for the whole context rather than // one per section, so there is one rotation entry rather than four. - { model: "MoodContext", field: "notesEncrypted", kind: "bytes" }, + { + model: "MoodContext", + field: "notesEncrypted", + kind: "bytes", + pkField: "moodEntryId", + }, // ───── v1.25 medication free-text notes (Bytes columns) ───── // Side-effect log note, dose-change titration note, and inventory-item @@ -318,6 +358,34 @@ export const ENCRYPTED_COLUMNS: readonly EncryptedColumn[] = [ // codec `DocumentContentIndex.textEncrypted` uses. A scanned medical preview // is PHI; rotation re-encrypts it on the standard Bytes walk. { model: "DocumentThumbnail", field: "thumbnailEncrypted", kind: "bytes" }, + + // ───── Whole-account backup blob (String, batched) ───── + // The weekly disaster-recovery snapshot and every manually uploaded pack. + // Named `data`, so the `*Encrypted` scan never saw it; the value is + // `packBackupBlob()` output, which is AES-256-GCM ciphertext like every + // other entry here. It is also the single worst column to miss: a backup + // that cannot be decrypted is the copy an operator reaches for precisely + // when nothing else works. + // + // Rotation re-encrypts the ciphertext WITHOUT touching the plaintext, so it + // is blind to the envelope inside: today a row is either the plain backup + // JSON or the `HLZ1:`-prefixed gzip form, and any further shape a future + // writer introduces rotates unchanged for the same reason. Batched, because + // a single row is megabytes. + { model: "DataBackup", field: "data", kind: "string", batched: true }, + + // ───── Idempotent-replay response cache (String, disposable) ───── + // The cached response body, encrypted because the PHI-returning creates echo + // their own decrypted DTO. Rows live 24 hours. It is registered so a key + // drop cannot leave a reader holding an undecryptable body, and marked + // disposable so an unreadable row is dropped instead of failing the run — + // the cache is an accelerator, and a miss only costs a re-run. + { + model: "IdempotencyKey", + field: "responseBody", + kind: "string", + disposable: true, + }, ] as const; /** Stable `Model.field` key for a registry entry. */ diff --git a/src/lib/crypto/encryption-corpus.ts b/src/lib/crypto/encryption-corpus.ts index cadeee22d..31816b691 100644 --- a/src/lib/crypto/encryption-corpus.ts +++ b/src/lib/crypto/encryption-corpus.ts @@ -47,9 +47,10 @@ import { export const LEGACY_BUCKET = "legacy"; /** - * Batch size for codec-dispatched blob columns (`codecField` present). The - * document vault's rows are up to cap-sized ciphertexts, so the walk is - * id-cursor paginated — at most this many blobs are in memory at once. + * Batch size for blob columns (`codecField` or `batched` set). The document + * vault's rows are up to cap-sized ciphertexts and a backup row is the whole + * account compressed, so the walk is id-cursor paginated — at most this many + * blobs are in memory at once. */ export const BLOB_ROTATION_BATCH_SIZE = 25; @@ -63,9 +64,11 @@ interface ColumnDelegate { skip?: number; }) => Promise>>; update: (args: { - where: { id: string }; + where: Record; data: Record; }) => Promise; + /** Only ever called for a `disposable` column's unreadable rows. */ + delete: (args: { where: Record }) => Promise; } /** The subset of the Prisma client we touch: one delegate per model. */ @@ -149,44 +152,75 @@ function reencryptBlob(value: Uint8Array, codec: string): Uint8Array { } /** - * Walk a codec-dispatched blob column in bounded id-cursor batches, invoking - * `onRow` per non-empty row. At most `BLOB_ROTATION_BATCH_SIZE` blobs are in - * memory per step; an interrupted run resumes safely on re-invocation because - * processing is idempotent (already-active rows are skipped by the callers). + * Walk a blob column in bounded id-cursor batches, invoking `onRow` per + * non-empty row. At most `BLOB_ROTATION_BATCH_SIZE` rows are in memory per + * step; an interrupted run resumes safely on re-invocation because processing + * is idempotent (already-active rows are skipped by the callers). + * + * `codec` is the row's own codec label for a codec-dispatched column and null + * for a plain `batched` one — a backup blob has one layout, just a large one. */ async function walkBlobColumn( delegate: ColumnDelegate, col: EncryptedColumn, onRow: (row: { id: string; - value: Uint8Array; - codec: string; + value: unknown; + codec: string | null; }) => Promise | void, ): Promise { - const codecField = col.codecField!; + const codecField = col.codecField; + const pk = pkField(col); let cursor: string | null = null; for (;;) { const rows = await delegate.findMany({ - select: { id: true, [col.field]: true, [codecField]: true }, - orderBy: { id: "asc" }, + select: { + [pk]: true, + [col.field]: true, + ...(codecField ? { [codecField]: true } : {}), + }, + orderBy: { [pk]: "asc" }, take: BLOB_ROTATION_BATCH_SIZE, - ...(cursor ? { cursor: { id: cursor }, skip: 1 } : {}), + ...(cursor ? { cursor: { [pk]: cursor }, skip: 1 } : {}), }); if (rows.length === 0) break; for (const row of rows) { - const value = row[col.field] as Uint8Array | null; - if (!value || value.byteLength === 0) continue; + const value = row[col.field]; + if (value == null) continue; + if (value instanceof Uint8Array && value.byteLength === 0) continue; + if (typeof value === "string" && value.length === 0) continue; await onRow({ - id: row.id as string, + id: row[pk] as string, value, - codec: String(row[codecField] ?? ""), + codec: codecField ? String(row[codecField] ?? "") : null, }); } - cursor = rows[rows.length - 1]!.id as string; + cursor = rows[rows.length - 1]![pk] as string; if (rows.length < BLOB_ROTATION_BATCH_SIZE) break; } } +/** The model's primary-key field: `id` unless the registry says otherwise. */ +function pkField(col: EncryptedColumn): string { + return col.pkField ?? "id"; +} + +/** True when the column's rows must be walked in bounded batches. */ +function isBatched(col: EncryptedColumn): boolean { + return Boolean(col.codecField ?? col.batched); +} + +/** The key id a walked row was written under, dispatching on its codec. */ +function walkedKeyId( + value: unknown, + codec: string | null, + kind: EncryptedColumn["kind"], +): string | null { + if (codec !== null) return blobKeyId(value as Uint8Array, codec); + const ciphertext = toCiphertext(value, kind); + return ciphertext == null ? null : extractKeyId(ciphertext); +} + export interface ColumnScan { model: string; field: string; @@ -208,16 +242,16 @@ export async function scanColumn( const byKeyId: Record = {}; let total = 0; - if (col.codecField) { - // Codec-dispatched blob column: bounded batches, per-row codec. + if (isBatched(col)) { + // Blob column: bounded batches, per-row codec where the column has one. await walkBlobColumn(delegate, col, ({ value, codec }) => { total += 1; - const id = blobKeyId(value, codec) ?? LEGACY_BUCKET; + const id = walkedKeyId(value, codec, col.kind) ?? LEGACY_BUCKET; byKeyId[id] = (byKeyId[id] ?? 0) + 1; }); } else { const rows = await delegate.findMany({ - select: { id: true, [col.field]: true }, + select: { [pkField(col)]: true, [col.field]: true }, }); for (const row of rows) { const ciphertext = toCiphertext(row[col.field], col.kind); @@ -283,6 +317,8 @@ export interface RotationResult { scanned: number; rotated: number; errors: number; + /** Unreadable rows deleted from a `disposable` column. */ + dropped: number; } /** Re-encrypt one column's stale rows to the active key. */ @@ -297,51 +333,78 @@ export async function rotateColumn( scanned: 0, rotated: 0, errors: 0, + dropped: 0, }; - if (col.codecField) { - // Codec-dispatched blob column: bounded id-cursor batches (never an - // unbounded blob findMany), re-encrypted under each row's OWN codec. - // Idempotent — rows already on the active key are skipped, so an - // interrupted run resumes cleanly on the next invocation. + /** + * A row that could not be re-encrypted. FAIL-CLOSED by default: count it and + * leave it untouched rather than dropping data. A `disposable` column takes + * the other branch — the value is a reproducible cache entry, and keeping an + * unreadable one only guarantees the next reader gets a body it cannot + * parse, so the row goes instead. + */ + const onUnreadable = async (id: string): Promise => { + if (!col.disposable) { + result.errors += 1; + return; + } + try { + await delegate.delete({ where: { [pkField(col)]: id } }); + result.dropped += 1; + } catch { + result.errors += 1; + } + }; + + if (isBatched(col)) { + // Blob column: bounded id-cursor batches (never an unbounded findMany), + // re-encrypted under each row's OWN codec where it has one. Idempotent — + // rows already on the active key are skipped, so an interrupted run + // resumes cleanly on the next invocation. await walkBlobColumn(delegate, col, async ({ id, value, codec }) => { result.scanned += 1; - if (blobKeyId(value, codec) === getActiveKeyId()) return; + if (walkedKeyId(value, codec, col.kind) === getActiveKeyId()) return; try { + // The plaintext is never inspected, only re-sealed, so every envelope + // a stored value can legitimately carry survives rotation untouched. + const next = + codec !== null + ? reencryptBlob(value as Uint8Array, codec) + : fromCiphertext( + encrypt(decrypt(toCiphertext(value, col.kind)!)), + col.kind, + ); await delegate.update({ - where: { id }, - data: { [col.field]: reencryptBlob(value, codec) }, + where: { [pkField(col)]: id }, + data: { [col.field]: next }, }); result.rotated += 1; } catch { - // FAIL-CLOSED: unknown codec or a no-longer-configured key throws; - // count it and leave the row untouched rather than dropping data. - result.errors += 1; + await onUnreadable(id); } }); return result; } + const pk = pkField(col); const rows = await delegate.findMany({ - select: { id: true, [col.field]: true }, + select: { [pk]: true, [col.field]: true }, }); result.scanned = rows.length; for (const row of rows) { const ciphertext = toCiphertext(row[col.field], col.kind); if (ciphertext == null || !shouldRotate(ciphertext)) continue; - const id = row.id as string; + const id = row[pk] as string; try { // ACTIVE-KEY-ONLY: encrypt() always writes the active key id. const reencrypted = encrypt(decrypt(ciphertext)); await delegate.update({ - where: { id }, + where: { [pk]: id }, data: { [col.field]: fromCiphertext(reencrypted, col.kind) }, }); result.rotated += 1; } catch { - // FAIL-CLOSED: a row under a no-longer-configured key throws on decrypt; - // count it and leave the row untouched rather than dropping data. - result.errors += 1; + await onUnreadable(id); } } return result; @@ -353,6 +416,8 @@ export interface CorpusRotation { totalScanned: number; totalRotated: number; totalErrors: number; + /** Unreadable rows deleted from `disposable` columns. */ + totalDropped: number; } /** Re-encrypt the whole corpus to the active key. Idempotent + active-key-only. */ @@ -367,10 +432,12 @@ export async function rotateCorpus( let totalScanned = 0; let totalRotated = 0; let totalErrors = 0; + let totalDropped = 0; for (const r of results) { totalScanned += r.scanned; totalRotated += r.rotated; totalErrors += r.errors; + totalDropped += r.dropped; } return { activeKeyId, @@ -378,5 +445,6 @@ export async function rotateCorpus( totalScanned, totalRotated, totalErrors, + totalDropped, }; } diff --git a/src/lib/jobs/encryption-key-rotate.ts b/src/lib/jobs/encryption-key-rotate.ts index e8ebefe5b..725659f55 100644 --- a/src/lib/jobs/encryption-key-rotate.ts +++ b/src/lib/jobs/encryption-key-rotate.ts @@ -48,6 +48,7 @@ export async function runEncryptionKeyRotation(): Promise<{ totalScanned: number; totalRotated: number; totalErrors: number; + totalDropped: number; }> { const prisma = getWorkerPrisma(); const out = await rotateCorpus(prisma as unknown as CorpusClient); @@ -56,6 +57,7 @@ export async function runEncryptionKeyRotation(): Promise<{ totalScanned: out.totalScanned, totalRotated: out.totalRotated, totalErrors: out.totalErrors, + totalDropped: out.totalDropped, }; } @@ -70,6 +72,7 @@ export async function handleEncryptionKeyRotate( evt.addMeta("rotate_scanned", result.totalScanned); evt.addMeta("rotate_rotated", result.totalRotated); evt.addMeta("rotate_errors", result.totalErrors); + evt.addMeta("rotate_dropped", result.totalDropped); await auditLog("admin.encryption.rotate.completed", { userId: requestedBy, details: { @@ -77,6 +80,7 @@ export async function handleEncryptionKeyRotate( scanned: result.totalScanned, rotated: result.totalRotated, errors: result.totalErrors, + dropped: result.totalDropped, }, }); // Per-row errors are the fail-closed skip of a row under a key the @@ -86,6 +90,7 @@ export async function handleEncryptionKeyRotate( rotate_scanned: result.totalScanned, rotate_rotated: result.totalRotated, rotate_errors: result.totalErrors, + rotate_dropped: result.totalDropped, }); } catch (err) { evt.addWarning(`encryption-key-rotate failed: ${err}`); diff --git a/src/lib/jobs/job-outcome.ts b/src/lib/jobs/job-outcome.ts index 673378a85..089c84e08 100644 --- a/src/lib/jobs/job-outcome.ts +++ b/src/lib/jobs/job-outcome.ts @@ -173,6 +173,7 @@ export const JOB_FACT_ALLOWLIST: ReadonlySet = new Set([ "retried", "reviewed", "rollup_failed", + "rotate_dropped", "rotate_errors", "rotate_rotated", "rotate_scanned", diff --git a/tests/integration/encryption-key-rotation-blobs.test.ts b/tests/integration/encryption-key-rotation-blobs.test.ts new file mode 100644 index 000000000..c2187cd2f --- /dev/null +++ b/tests/integration/encryption-key-rotation-blobs.test.ts @@ -0,0 +1,196 @@ +/** + * Key rotation over the two ciphertext columns that carry no `*Encrypted` in + * their name, against real Postgres. + * + * `DataBackup.data` is the one that matters. It holds the whole account, + * compressed then encrypted, and it sat outside the rotation registry until + * v1.38.6 — the script reported zero rows remaining because it never looked, + * and the runbook reads a zero as permission to retire the previous key. This + * pins the fix at the level the bug lived at: seed real rows under an old key, + * rotate, and read the backup back. + * + * Both stored envelopes are seeded on purpose. A row written before + * `packBackupBlob` existed is `encrypt(json)`; one written after is the + * `HLZ1:`-prefixed gzip form. Rotation re-seals the ciphertext without ever + * looking at the plaintext, so both come back byte-identical — and so will + * whatever envelope a later writer introduces. + * + * `IdempotencyKey.responseBody` rides along as the disposable case: a row + * under a key the deployment no longer configures is DELETED rather than + * counted as an error, because the value is a 24h cache entry and leaving it + * behind only guarantees the next replay gets a body it cannot parse. + */ +import { beforeEach, describe, expect, it } from "vitest"; + +process.env.ENCRYPTION_KEYS = JSON.stringify({ + v1: "1".repeat(64), + v2: "2".repeat(64), +}); +process.env.ENCRYPTION_ACTIVE_KEY_ID = "v1"; +delete process.env.ENCRYPTION_KEY; + +import { _resetCryptoCacheForTests, encrypt, extractKeyId } from "@/lib/crypto"; +import { + ENCRYPTED_COLUMNS, + encryptedColumnKey, + type EncryptedColumn, +} from "@/lib/crypto/encrypted-columns"; +import { + rotateColumn, + type CorpusClient, +} from "@/lib/crypto/encryption-corpus"; +import { packBackupBlob, unpackBackupBlob } from "@/lib/export/backup-blob"; +import { getPrismaClient, truncateAllTables } from "./setup"; + +const TEST_USER_ID = "user-rotation-blobs"; +const BACKUP_JSON = JSON.stringify({ + schemaVersion: 7, + measurements: [{ type: "WEIGHT", value: 80.4, unit: "kg" }], + note: "üäö — a non-ASCII payload, so a mangled round-trip shows up", +}); + +function column(model: string, field: string): EncryptedColumn { + const col = ENCRYPTED_COLUMNS.find( + (c) => c.model === model && c.field === field, + ); + if (!col) throw new Error(`${model}.${field} is not registered`); + return col; +} + +/** Encrypt while `keyId` is active, then restore the caller's active id. */ +function underKey(keyId: string, plaintext: string, pack = false): string { + const previous = process.env.ENCRYPTION_ACTIVE_KEY_ID; + process.env.ENCRYPTION_ACTIVE_KEY_ID = keyId; + _resetCryptoCacheForTests(); + const out = pack ? packBackupBlob(plaintext) : encrypt(plaintext); + process.env.ENCRYPTION_ACTIVE_KEY_ID = previous; + _resetCryptoCacheForTests(); + return out; +} + +beforeEach(async () => { + process.env.ENCRYPTION_KEYS = JSON.stringify({ + v1: "1".repeat(64), + v2: "2".repeat(64), + }); + process.env.ENCRYPTION_ACTIVE_KEY_ID = "v2"; + _resetCryptoCacheForTests(); + await truncateAllTables(getPrismaClient()); + await getPrismaClient().user.create({ + data: { + id: TEST_USER_ID, + username: "rotation-blobs", + email: "rotation-blobs@example.test", + timezone: "Europe/Berlin", + }, + }); +}); + +describe("key rotation over the non-suffixed ciphertext columns", () => { + it("rotates DataBackup.data in both stored envelopes and keeps it readable", async () => { + const prisma = getPrismaClient(); + const gzipped = await prisma.dataBackup.create({ + data: { + userId: TEST_USER_ID, + type: "WEEKLY_AUTO", + data: underKey("v1", BACKUP_JSON, true), + }, + }); + // The pre-envelope shape: `encrypt(json)` with no `HLZ1:` marker. + const plainEnvelope = await prisma.dataBackup.create({ + data: { + userId: TEST_USER_ID, + type: "MANUAL_UPLOAD_1", + data: underKey("v1", BACKUP_JSON), + }, + }); + + const result = await rotateColumn( + { dataBackup: prisma.dataBackup } as unknown as CorpusClient, + column("DataBackup", "data"), + ); + expect(result.scanned).toBe(2); + expect(result.rotated).toBe(2); + expect(result.errors).toBe(0); + expect(result.dropped).toBe(0); + + for (const id of [gzipped.id, plainEnvelope.id]) { + const row = await prisma.dataBackup.findUniqueOrThrow({ where: { id } }); + expect(extractKeyId(row.data)).toBe("v2"); + expect(unpackBackupBlob(row.data)).toBe(BACKUP_JSON); + } + + // Idempotent: a second pass finds nothing left to do. + const again = await rotateColumn( + { dataBackup: prisma.dataBackup } as unknown as CorpusClient, + column("DataBackup", "data"), + ); + expect(again.scanned).toBe(2); + expect(again.rotated).toBe(0); + }); + + it("survives dropping the retired key once rotation has run", async () => { + const prisma = getPrismaClient(); + const backup = await prisma.dataBackup.create({ + data: { + userId: TEST_USER_ID, + type: "WEEKLY_AUTO", + data: underKey("v1", BACKUP_JSON, true), + }, + }); + await rotateColumn( + { dataBackup: prisma.dataBackup } as unknown as CorpusClient, + column("DataBackup", "data"), + ); + + // What the runbook tells the operator to do next: retire v1 entirely. + process.env.ENCRYPTION_KEYS = JSON.stringify({ v2: "2".repeat(64) }); + _resetCryptoCacheForTests(); + + const row = await prisma.dataBackup.findUniqueOrThrow({ + where: { id: backup.id }, + }); + expect(unpackBackupBlob(row.data)).toBe(BACKUP_JSON); + }); + + it("drops an unreadable idempotency row instead of failing the run", async () => { + const prisma = getPrismaClient(); + const seed = async (key: string, body: string) => + prisma.idempotencyKey.create({ + data: { + userId: TEST_USER_ID, + key, + method: "POST", + path: "/api/measurements", + responseStatus: 201, + responseBody: body, + expiresAt: new Date(Date.now() + 86_400_000), + }, + }); + const readable = await seed("k-readable", underKey("v1", '{"data":1}')); + // A key id the deployment no longer configures: fail-closed on decrypt. + const unreadable = await seed("k-unreadable", "v9.QUJDREVGR0hJSktM"); + + const result = await rotateColumn( + { idempotencyKey: prisma.idempotencyKey } as unknown as CorpusClient, + column("IdempotencyKey", "responseBody"), + ); + expect(result.rotated).toBe(1); + expect(result.dropped).toBe(1); + expect(result.errors).toBe(0); + + const kept = await prisma.idempotencyKey.findUnique({ + where: { id: readable.id }, + }); + expect(extractKeyId(kept!.responseBody)).toBe("v2"); + expect( + await prisma.idempotencyKey.findUnique({ where: { id: unreadable.id } }), + ).toBeNull(); + }); + + it("keeps both columns in the registry the script and the job read", () => { + const keys = ENCRYPTED_COLUMNS.map(encryptedColumnKey); + expect(keys).toContain("DataBackup.data"); + expect(keys).toContain("IdempotencyKey.responseBody"); + }); +});