diff --git a/docs/ops/encryption-key-rotation.md b/docs/ops/encryption-key-rotation.md index 0396e2897..b48c64a72 100644 --- a/docs/ops/encryption-key-rotation.md +++ b/docs/ops/encryption-key-rotation.md @@ -103,6 +103,13 @@ under `v1` (the synthetic id assigned to the existing `ENCRYPTION_KEY`). a full `Columns walked` line. Both read the same registry the rotation walks. + A run that walked every column without an error also removes the boot + key check's record of each previous key id that no longer holds a single + value (`encryption_key_canaries`); the script prints which ids it removed + and, for an id that still holds values, the columns that do. With the record + gone, the id can later carry a different key without the start-up check + refusing it. + A zero only means "nothing left" when the run also says it walked every registered column. A zero from a run that skipped columns means "never looked", and dropping the old key on it makes those rows permanently diff --git a/docs/self-hosting/getting-started.md b/docs/self-hosting/getting-started.md index f67feb66f..cc0499067 100644 --- a/docs/self-hosting/getting-started.md +++ b/docs/self-hosting/getting-started.md @@ -267,6 +267,34 @@ every API request answers `503` with `meta.errorCode` Restore the original key, point `DATABASE_URL` at the matching database, or start with an empty database; then restart. +The check remembers the key it first saw for each key id in the table +`encryption_key_canaries`. Before it records a key, it reads the oldest +encrypted values in the database and records the key only if they open, or +if the database holds no encrypted value under that key id at all. When the +stored data proves neither (a single unreadable value, the oldest values +failing while newer ones open, or a probe that ran out of time), the app +keeps serving, records nothing, logs a warning that starts with +`[boot] Encryption key check inconclusive`, and checks again at the next +start. + +If you are certain the configured key is the one your data was written +with, and the check still refuses because it recorded a different key +earlier, reset the check for that key id and restart. The refusal in the log +prints the exact statement; for the default key id it is: + +```bash +docker compose exec -T db psql -U healthlog healthlog \ + -c "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';" +docker compose restart app +``` + +The next start probes the stored data again and records the key only if +that data opens with it. Do this only when you are sure: if the key is in +fact wrong, the app serves errors on every encrypted value instead of the +one clear refusal. After a key rotation the rotation script removes the +record of a retired key id itself, once it reports that no values remain +under that id. + ## Backup and restore Your database is the only stateful piece — back it up, and back up the diff --git a/docs/self-hosting/truenas.md b/docs/self-hosting/truenas.md index 8bebe7441..c39f6e016 100644 --- a/docs/self-hosting/truenas.md +++ b/docs/self-hosting/truenas.md @@ -80,6 +80,32 @@ Fix it one of three ways, then restart the app: 3. Start fresh with an empty database. The old data and its backups need the old key. +Before it records a key as the right one, the check reads the oldest +encrypted values in the database and records the key only if they open (or if +the database holds no encrypted value under that key id yet). When the data +proves neither, the app keeps serving, records nothing, writes a warning +starting with `[boot] Encryption key check inconclusive` to the log, and +checks again at the next start. + +### Resetting the check + +If you are certain the configured key is the one the data was written with, +and the app still refuses because the check recorded a different key earlier, +remove that record and restart. Open a shell on the database container (Apps, +HealthLog, the database container's Shell) and run, with the key id the log +names (`v1` unless you use `ENCRYPTION_KEYS`): + +```bash +psql -U healthlog healthlog \ + -c "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';" +``` + +Then restart the app. It probes the stored data again and records the key +only if that data opens with it. If the key is in fact wrong, the app then +fails on every encrypted value instead of refusing clearly, so only do this +when you are sure. After a key rotation you never need this for the old key +id: the rotation script removes its record once no values remain under it. + ## Updates Pick the new tag, take a snapshot of the database dataset first, and update. diff --git a/scripts/rotate-encryption-key.ts b/scripts/rotate-encryption-key.ts index a3cd1742a..6e054ba8b 100644 --- a/scripts/rotate-encryption-key.ts +++ b/scripts/rotate-encryption-key.ts @@ -17,6 +17,10 @@ * non-zero, because an operator reads a zero here as permission to drop the * previous key, and "nothing left" and "never looked" must not print alike. * + * After a complete, error-free run it checks every other configured key id + * for remaining values and removes the boot key check's record (canary) of + * each id that holds none, naming the columns of each id that still does. + * * Usage: * ENCRYPTION_KEYS='{"v1":"","v2":""}' \ * ENCRYPTION_ACTIVE_KEY_ID=v2 \ @@ -27,6 +31,7 @@ import { Buffer } from "node:buffer"; import { PrismaClient } from "@/generated/prisma/client"; import { PrismaPg } from "@prisma/adapter-pg"; import { decrypt, encrypt, extractKeyId, getActiveKeyId } from "@/lib/crypto"; +import { retireCanariesWithoutData } from "@/lib/crypto/canary"; import { rotateColumn, type CorpusClient, @@ -1034,6 +1039,30 @@ async function main() { `drop the previous key:\n ${notWalked.join("\n ")}`, ); } + + // ───── Key-check records of retired key ids ───── + // The boot key check keeps one canary per configured key id. Once no + // registered column holds a single value under an id, that id's key can + // leave ENCRYPTION_KEYS, and its canary goes with it: left behind, it would + // refuse a later, different key configured under the same id. Only after a + // complete, error-free run; the existence check walks every column itself + // rather than trusting the counts above. + if (notWalked.length === 0 && totalErrors === 0) { + const { removed, remaining } = await retireCanariesWithoutData(prisma); + for (const [keyId, columns] of Object.entries(remaining)) { + console.log( + `\nKey id '${keyId}': values remain in ${columns.length} column(s); ` + + `keep its key:\n ${columns.join("\n ")}`, + ); + } + for (const keyId of removed) { + console.log( + `\nKey id '${keyId}': no values remain. Its key-check record was ` + + `removed; its key can now leave ENCRYPTION_KEYS.`, + ); + } + } + await prisma.$disconnect(); if (notWalked.length > 0) process.exit(4); process.exit(totalErrors > 0 ? 3 : 0); diff --git a/src/lib/boot/key-check.ts b/src/lib/boot/key-check.ts index 12c0c22bb..c4deeb0b7 100644 --- a/src/lib/boot/key-check.ts +++ b/src/lib/boot/key-check.ts @@ -13,9 +13,23 @@ import { checkEncryptionKeyCanaries, keyMismatchLogBlock, type CanaryClient, + type InconclusiveReason, } from "@/lib/crypto/canary"; import { setKeyMismatchState } from "./key-mismatch-state"; +const INCONCLUSIVE_EXPLANATION: Record = { + "single-value": "the only stored value found under that id did not open.", + mixed: + "the oldest stored values under that id do not open, newer ones do. " + + "That is what a wrong key that served for a while leaves behind; check " + + "that this is the key the database was first written with.", + unsampled: + "the values stored under that id are too large to probe at start-up.", + incomplete: + "the probe of the stored data did not finish in time or a table could " + + "not be read.", +}; + export async function runBootKeyCheck(client: CanaryClient): Promise { const outcome = await checkEncryptionKeyCanaries(client); if (outcome.state === "mismatch") { @@ -34,11 +48,11 @@ export async function runBootKeyCheck(client: CanaryClient): Promise { `[boot] Encryption key check skipped: ${outcome.message.slice(0, 300)}`, ); } else { - if (outcome.inconclusive.length > 0) { + for (const { keyId, reason } of outcome.inconclusive) { console.warn( - `[boot] Encryption key check inconclusive for key id(s) ${outcome.inconclusive.join(", ")}: ` + - "the only stored value found under that id did not open. Serving; " + - "no canary written; the check runs again at the next start.", + `[boot] Encryption key check inconclusive for key id ${keyId}: ` + + `${INCONCLUSIVE_EXPLANATION[reason]} Serving; no key recorded; ` + + "the check runs again at the next start.", ); } if (outcome.written.length > 0) { diff --git a/src/lib/crypto/__tests__/canary.test.ts b/src/lib/crypto/__tests__/canary.test.ts index 170e1e5bf..bb70ee45b 100644 --- a/src/lib/crypto/__tests__/canary.test.ts +++ b/src/lib/crypto/__tests__/canary.test.ts @@ -1,17 +1,30 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; +import { PrismaClient } from "@/generated/prisma/client"; +import { PrismaPg } from "@prisma/adapter-pg"; import { _resetCryptoCacheForTests, candidateMatchesKey, encrypt, + encryptBytes, encryptUnderKeyId, + extractKeyIdFromBytes, fingerprintKeyBytes, getKeyFingerprint, } from "@/lib/crypto"; +import { encryptToBytes } from "@/lib/ai/coach/bytes-codec"; +import { + ENCRYPTED_COLUMNS, + WORKOUT_ROUTE_GEOMETRY_AAD, + encryptedColumnKey, +} from "@/lib/crypto/encrypted-columns"; import { canaryPlaintext, checkEncryptionKeyCanaries, keyMismatchLogBlock, + probeValueOpens, + resolveColumnLocation, type CanaryClient, + type ColumnSampler, } from "../canary"; const KEY_A = @@ -19,23 +32,42 @@ const KEY_A = const KEY_B = "fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210"; -function useKey(hex: string) { +function configureKey(hex: string) { vi.stubEnv("ENCRYPTION_KEYS", ""); vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", ""); vi.stubEnv("ENCRYPTION_KEY", hex); _resetCryptoCacheForTests(); } -/** An in-memory stand-in for the two raw statements and the probe delegates. */ +type Seeded = + | string + | Uint8Array + | { value: string | Uint8Array | null; ts?: string; codec?: string }; + +/** + * An in-memory stand-in for the raw statements, plus a sampler over seeded + * values keyed by `Model.field`. A seeded value's `ts` orders it; values + * without one keep their seeded order after every dated value, as the SQL + * sampler orders them. + */ function fakeClient(opts: { canaries?: Record; - /** delegate → field → stored values (in row order). */ - probe?: Record>; + probe?: Record; + /** Make the sampler throw for these columns. */ + unreadable?: string[]; + /** Another process seals this id (with this ciphertext) just before us. */ + race?: { keyId: string; ciphertext: string }; }) { const canaries = new Map(Object.entries(opts.canaries ?? {})); const inserts: string[] = []; + let raced = false; const client = { - async $queryRaw() { + async $queryRaw(_q: TemplateStringsArray, ...values: unknown[]) { + const [arg] = values; + if (typeof arg === "string") { + const c = canaries.get(arg); + return c ? [{ ciphertext: c }] : []; + } return [...canaries.entries()].map(([key_id, ciphertext]) => ({ key_id, ciphertext, @@ -43,104 +75,118 @@ function fakeClient(opts: { }, async $executeRaw(_q: TemplateStringsArray, ...values: unknown[]) { const [keyId, ciphertext] = values as [string, string]; + if (opts.race && !raced && opts.race.keyId === keyId) { + raced = true; + canaries.set(keyId, opts.race.ciphertext); + } inserts.push(keyId); - if (!canaries.has(keyId)) canaries.set(keyId, ciphertext); + if (canaries.has(keyId)) return 0; + canaries.set(keyId, ciphertext); return 1; }, - } as unknown as CanaryClient & Record; - for (const [delegate, fields] of Object.entries(opts.probe ?? {})) { - client[delegate] = { - async findMany(args: { - select: Record; - take: number; - skip?: number; - }) { - const field = Object.keys(args.select)[0]; - const skip = args.skip ?? 0; - return (fields[field] ?? []) - .slice(skip, skip + args.take) - .map((value) => ({ [field]: value })); - }, - }; - } - return { client, canaries, inserts }; + } as unknown as CanaryClient; + const sampler: ColumnSampler = async (column, keyId, limit) => { + const key = `${column.model}.${column.field}`; + if (opts.unreadable?.includes(key)) throw new Error("relation missing"); + const rows = (opts.probe?.[key] ?? []).map((v) => + typeof v === "string" || v instanceof Uint8Array + ? { value: v, ts: null as Date | null, codec: null } + : { + value: v.value, + ts: v.ts ? new Date(v.ts) : null, + codec: v.codec ?? null, + }, + ); + const under = rows.filter((r) => { + if (r.value === null) return true; + if (typeof r.value === "string") return r.value.startsWith(`${keyId}.`); + const b = Buffer.from(r.value); + return ( + b.toString("latin1").startsWith(`${keyId}.`) || + extractKeyIdFromBytes(b) === keyId + ); + }); + return under + .map((r, i) => ({ r, i })) + .sort((a, b) => { + const ta = a.r.ts?.getTime() ?? Infinity; + const tb = b.r.ts?.getTime() ?? Infinity; + return ta === tb ? a.i - b.i : ta - tb; + }) + .slice(0, limit) + .map(({ r }) => r); + }; + const check = (budget?: number) => + checkEncryptionKeyCanaries(client, { sampler, probeBudgetMs: budget }); + return { client, sampler, canaries, inserts, check }; } +const ok = (fields: { + written?: string[]; + verified?: string[]; + inconclusive?: Array<{ keyId: string; reason: string }>; +}) => ({ + state: "ok", + written: fields.written ?? [], + verified: fields.verified ?? [], + inconclusive: fields.inconclusive ?? [], +}); + describe("encryption key canary", () => { beforeEach(() => { vi.unstubAllEnvs(); - useKey(KEY_A); + configureKey(KEY_A); }); - it("writes a canary for a configured key with no row and no data", async () => { - const { client, canaries, inserts } = fakeClient({}); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ - state: "ok", - written: ["v1"], - verified: [], - inconclusive: [], - }); + it("writes a canary for a configured key when no column holds a value under its id", async () => { + const { canaries, inserts, check } = fakeClient({}); + expect(await check()).toEqual(ok({ written: ["v1"] })); expect(inserts).toEqual(["v1"]); expect(canaries.get("v1")?.startsWith("v1.")).toBe(true); }); it("verifies a canary written under the same key", async () => { - const { client, inserts } = fakeClient({ + const { inserts, check } = fakeClient({ canaries: { v1: encryptUnderKeyId(canaryPlaintext("v1"), "v1") }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ - state: "ok", - written: [], - verified: ["v1"], - inconclusive: [], - }); + expect(await check()).toEqual(ok({ verified: ["v1"] })); expect(inserts).toEqual([]); }); it("reports a mismatch when the key changed under the same id", async () => { const sealedUnderA = encryptUnderKeyId(canaryPlaintext("v1"), "v1"); - useKey(KEY_B); - const { client, inserts } = fakeClient({ canaries: { v1: sealedUnderA } }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + configureKey(KEY_B); + const { inserts, check } = fakeClient({ canaries: { v1: sealedUnderA } }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); expect(inserts).toEqual([]); }); it("reports a mismatch for a canary that opens to the wrong value", async () => { // A row from another key id copied over: opens, but says the wrong thing. - const { client } = fakeClient({ + const { check } = fakeClient({ canaries: { v1: encryptUnderKeyId(canaryPlaintext("v2"), "v1") }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome.state).toBe("mismatch"); + expect((await check()).state).toBe("mismatch"); }); it("refuses to seal a wrong key when existing data predates the canary", async () => { const existing = [encrypt("a stored token"), encrypt("another token")]; - useKey(KEY_B); - const { client, inserts } = fakeClient({ + configureKey(KEY_B); + const { inserts, check } = fakeClient({ probe: { - user: { - codexAccessTokenEncrypted: [existing[0]], - codexRefreshTokenEncrypted: [existing[1]], - }, + "User.codexAccessTokenEncrypted": [existing[0]], + "User.codexRefreshTokenEncrypted": [existing[1]], }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); expect(inserts).toEqual([]); }); it("writes the canary when existing data opens under the key", async () => { - const { client, inserts } = fakeClient({ - probe: { - user: { codexAccessTokenEncrypted: [encrypt("a stored token")] }, - }, + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [encrypt("a stored token")] }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome.state).toBe("ok"); + expect((await check()).state).toBe("ok"); expect(inserts).toEqual(["v1"]); }); @@ -148,46 +194,228 @@ describe("encryption key canary", () => { // A row that claims key id v1 but does not open (damaged, or sealed by // something else), found FIRST, then values that open. const damaged = "v1." + Buffer.alloc(40, 7).toString("base64"); - const { client, inserts } = fakeClient({ + const { inserts, check } = fakeClient({ probe: { - user: { - codexAccessTokenEncrypted: [damaged], - codexRefreshTokenEncrypted: [encrypt("a stored token")], - }, + "User.codexAccessTokenEncrypted": [damaged], + "User.codexRefreshTokenEncrypted": [encrypt("a stored token")], }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome.state).toBe("ok"); + expect((await check()).state).toBe("ok"); expect(inserts).toEqual(["v1"]); }); it("several values in one column count, so a wrong key with one busy column still refuses", async () => { const values = [encrypt("one"), encrypt("two"), encrypt("three")]; - useKey(KEY_B); - const { client, inserts } = fakeClient({ - probe: { user: { codexAccessTokenEncrypted: values } }, + configureKey(KEY_B); + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": values }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); expect(inserts).toEqual([]); }); it("a single value that does not open is inconclusive: no mismatch, no canary", async () => { const existing = encrypt("the only stored token"); - useKey(KEY_B); - const { client, inserts } = fakeClient({ - probe: { user: { codexAccessTokenEncrypted: [existing] } }, - }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ - state: "ok", - written: [], - verified: [], - inconclusive: ["v1"], + configureKey(KEY_B); + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [existing] }, }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "single-value" }] }), + ); expect(inserts).toEqual([]); }); + describe("Bytes columns", () => { + it("a wrong key over Bytes-only ciphertext refuses instead of sealing", async () => { + const notes = [encryptToBytes("note one"), encryptToBytes("note two")]; + configureKey(KEY_B); + const { inserts, check } = fakeClient({ + probe: { + "MoodEntry.noteEncrypted": [notes[0]], + "Measurement.notesEncrypted": [notes[1]], + }, + }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); + expect(inserts).toEqual([]); + }); + + it("opens the binary layout with its label, and the UTF-8 layout", async () => { + const route = encryptBytes(Buffer.from("{}"), WORKOUT_ROUTE_GEOMETRY_AAD); + const { inserts, check } = fakeClient({ + probe: { + "WorkoutRoute.geometryEncrypted": [ + { value: route, ts: "2024-01-01" }, + ], + "MoodEntry.noteEncrypted": [ + { value: encryptToBytes("a note"), ts: "2024-01-02" }, + ], + }, + }); + expect(await check()).toEqual(ok({ written: ["v1"] })); + expect(inserts).toEqual(["v1"]); + }); + + it("a binary value under the wrong label does not count as opening", () => { + const sealed = encryptBytes(Buffer.from("x"), "some/other/label"); + const column = ENCRYPTED_COLUMNS.find((c) => c.model === "WorkoutRoute")!; + expect(probeValueOpens(column, { value: sealed, ts: null }, "v1")).toBe( + false, + ); + }); + + it("dispatches a codec column by the row's codec", () => { + const column = ENCRYPTED_COLUMNS.find( + (c) => c.model === "InboundDocument" && c.field === "contentEncrypted", + )!; + const binary = encryptBytes(Buffer.from("%PDF")); + const text = encryptToBytes("JVBERg=="); + expect( + probeValueOpens( + column, + { value: binary, codec: "binary2", ts: null }, + "v1", + ), + ).toBe(true); + expect( + probeValueOpens( + column, + { value: text, codec: "base64v1", ts: null }, + "v1", + ), + ).toBe(true); + }); + }); + + describe("oldest values decide", () => { + function seedOldAThenNewB() { + const old = [ + encrypt("old 1"), + encrypt("old 2"), + encryptToBytes("old note 1"), + encryptToBytes("old note 2"), + ]; + configureKey(KEY_B); + const recent = [ + encrypt("new 1"), + encrypt("new 2"), + encryptToBytes("new note 1"), + encryptToBytes("new note 2"), + ]; + configureKey(KEY_A); + return fakeClient({ + probe: { + "User.codexAccessTokenEncrypted": [ + { value: recent[0], ts: "2026-09-01" }, + { value: old[0], ts: "2024-01-01" }, + ], + "User.codexRefreshTokenEncrypted": [ + { value: old[1], ts: "2024-01-02" }, + { value: recent[1], ts: "2026-09-02" }, + ], + "MoodEntry.noteEncrypted": [ + { value: recent[2], ts: "2026-09-03" }, + { value: old[2], ts: "2024-01-03" }, + ], + "Measurement.notesEncrypted": [ + { value: old[3], ts: "2024-01-04" }, + { value: recent[3], ts: "2026-09-04" }, + ], + }, + }); + } + + it("the key that wrote only the newer rows is never sealed", async () => { + const fake = seedOldAThenNewB(); + configureKey(KEY_B); + const outcome = await fake.check(); + expect(outcome.state).not.toBe("error"); + if (outcome.state === "ok") expect(outcome.written).toEqual([]); + expect(fake.canaries.size).toBe(0); + }); + + it("the key that wrote the oldest rows is sealed, newer unreadable rows notwithstanding", async () => { + const fake = seedOldAThenNewB(); + expect(await fake.check()).toEqual(ok({ written: ["v1"] })); + }); + + it("two oldest values that fail before one that opens are mixed: inconclusive", async () => { + const opens = encrypt("opens"); + configureKey(KEY_B); + const foreign = [encrypt("x"), encrypt("y")]; + configureKey(KEY_A); + const { inserts, check } = fakeClient({ + probe: { + "User.codexAccessTokenEncrypted": [ + { value: foreign[0], ts: "2024-01-01" }, + ], + "User.codexRefreshTokenEncrypted": [ + { value: foreign[1], ts: "2024-01-02" }, + ], + "MoodEntry.noteEncrypted": [{ value: opens, ts: "2025-01-01" }], + }, + }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "mixed" }] }), + ); + expect(inserts).toEqual([]); + }); + }); + + describe("nothing unproven seals", () => { + it("a probe out of time is inconclusive, even over data that opens", async () => { + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [encrypt("opens")] }, + }); + expect(await check(-1)).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "incomplete" }] }), + ); + expect(inserts).toEqual([]); + }); + + it("a column that cannot be read is inconclusive, never a fresh install", async () => { + const { inserts, check } = fakeClient({ + unreadable: ["MoodEntry.noteEncrypted"], + }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "incomplete" }] }), + ); + expect(inserts).toEqual([]); + }); + + it("values too large to fetch are present: inconclusive, not fresh", async () => { + const { inserts, check } = fakeClient({ + probe: { "InboundDocument.contentEncrypted": [{ value: null }] }, + }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "unsampled" }] }), + ); + expect(inserts).toEqual([]); + }); + }); + + describe("the first-seal race", () => { + it("losing it to a process with another key is a mismatch", async () => { + configureKey(KEY_B); + const otherCanary = encryptUnderKeyId(canaryPlaintext("v1"), "v1"); + configureKey(KEY_A); + const { check } = fakeClient({ + race: { keyId: "v1", ciphertext: otherCanary }, + }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); + }); + + it("losing it to a process with the same key verifies", async () => { + const { check } = fakeClient({ + race: { + keyId: "v1", + ciphertext: encryptUnderKeyId(canaryPlaintext("v1"), "v1"), + }, + }); + expect(await check()).toEqual(ok({ verified: ["v1"] })); + }); + }); + it("reports an error, not a mismatch, when the table cannot be read", async () => { const client = { $queryRaw: async () => { @@ -202,22 +430,49 @@ describe("encryption key canary", () => { it("reports an error when no key is configured", async () => { vi.stubEnv("ENCRYPTION_KEY", ""); _resetCryptoCacheForTests(); - const outcome = await checkEncryptionKeyCanaries(fakeClient({}).client); + const outcome = await fakeClient({}).check(); expect(outcome.state).toBe("error"); }); - it("names the key ids and never a key in the log block", () => { + it("names the key ids, the reset statement, and never a key in the log block", () => { const block = keyMismatchLogBlock(["v1"]); expect(block).toContain("'v1'"); expect(block).toContain("encryption.key_mismatch"); + expect(block).toContain( + "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';", + ); expect(block).not.toContain(KEY_A); }); }); +describe("the SQL sampler's column map", () => { + it("resolves every registered column to a table, a column and an order", () => { + // A throwaway client: never connects, only exposes the data model the + // probe reads table and column names from. + const client = new PrismaClient({ + adapter: new PrismaPg({ + connectionString: "postgres://x:y@127.0.0.1:1/x", + }), + }); + for (const column of ENCRYPTED_COLUMNS) { + const loc = resolveColumnLocation(client, column); + expect(loc.table, encryptedColumnKey(column)).toMatch(/^[a-z_0-9]+$/); + expect(loc.column, encryptedColumnKey(column)).toMatch(/^[a-z_0-9]+$/); + if (column.codecField) expect(loc.codec).not.toBeNull(); + } + const backupPieces = ENCRYPTED_COLUMNS.find( + (c) => c.model === "DataBackupChunk", + )!; + expect(resolveColumnLocation(client, backupPieces).ts).toBeNull(); + const user = ENCRYPTED_COLUMNS.find((c) => c.model === "User")!; + expect(resolveColumnLocation(client, user).ts).toBe("updated_at"); + }); +}); + describe("key fingerprint and copy check", () => { beforeEach(() => { vi.unstubAllEnvs(); - useKey(KEY_A); + configureKey(KEY_A); }); it("is the first 12 hex of SHA-256 over the raw key bytes", () => { diff --git a/src/lib/crypto/canary.ts b/src/lib/crypto/canary.ts index e4f77178e..60ee6077f 100644 --- a/src/lib/crypto/canary.ts +++ b/src/lib/crypto/canary.ts @@ -12,44 +12,85 @@ * * - row present: it must decrypt to the expected value, else the key is not * the one the row was written with; - * - row absent: before writing one, sample existing ciphertext under that - * key id from the registered string columns and try it. Without this - * probe, the first boot after an upgrade (or a re-keyed boot over a - * database that predates the canary) would seal the wrong key as the - * right one. + * - row absent: before writing one, probe the ciphertext already stored + * under that key id. Without the probe, the first boot after an upgrade + * (or a re-keyed boot over a database that predates the canary) would + * seal the wrong key as the right one, and from then on refuse the + * right key. * - * The probe samples up to `PROBE_SAMPLE_LIMIT` values, one per column first - * (different columns are different writers, so one bad writer cannot speak - * for the whole database), then further values from columns that had any. - * Its verdict, and what each one does: + * ## The probe * - * - `opens`: at least one value opened. A wrong key opens nothing, so the - * key is right and any value that failed is a damaged or foreign row, not - * a key problem. The canary is written. + * Every registered column is asked, string and Bytes alike (`ENCRYPTED_COLUMNS`), + * for its OLDEST values under the key id: the `encrypt()` string form + * (`.`, or a bare legacy base64 value for the legacy id `v1`), + * the streamed form (`~hlgcm1..`), the same string stored as UTF-8 in a + * Bytes column, and the binary layout (`0x02 `). The answer for a + * column also tells whether it holds ANY value under the id, which is the + * existence check below. + * + * Oldest means the row's `updatedAt`, else its `createdAt`, else (no timestamp + * on the model, e.g. backup pieces) the primary key, ordered after every + * timestamped row. `updatedAt` rather than `createdAt` because a value is + * sealed when its row is written, and a row updated yesterday may carry a + * value sealed yesterday on a row created years ago; `updatedAt` is a ceiling + * on when the value was sealed. The point of the order: a process that served + * under a wrong key without a canary (an inconclusive boot, a version that + * predates this check) wrote rows that open under that wrong key. Those rows + * are by construction newer than the data the database was built with, so + * reading the oldest values first means they are never what decides. + * + * Up to `PROBE_ROWS_PER_COLUMN` values per column are fetched, the merged set + * is ordered by that timestamp, and the oldest `PROBE_SAMPLE_LIMIT` are tried. + * Values larger than `PROBE_VALUE_MAX_BYTES` (stored documents, large backups) + * count as present but are not fetched. The verdict: + * + * - `opens`: the oldest value that opens is preceded by at most ONE older + * value that does not. That one is a damaged or foreign row, not a key + * problem: a wrong key opens nothing. Values newer than the first opening + * one do not count against it; they are what a wrong-key process would + * have left behind. The canary is written. * - `fails`: two or more values were tried and none opened. That is what a * wrong key looks like; the process refuses (mismatch). - * - `inconclusive`: exactly one value was found and it did not open. One - * row cannot tell a wrong key from one damaged row, and the two possible - * mistakes are not equal: declaring a mismatch takes a correctly keyed - * server down with 503 on every request, while serving keeps today's - * behaviour. So the process serves, but writes NO canary either (that - * would seal a key that may be wrong), logs a warning, and probes again - * at the next boot, by which time more data may decide it. - * - `none`: nothing found inside the time budget. The canary is written, - * which is exactly the behaviour before this check existed. + * - `none`: every column answered, and none holds a single value under the + * key id. A genuinely fresh install (or a new id added for rotation); the + * canary is written. + * - `inconclusive`, with a reason: anything not proven. Only one value and + * it did not open (`single-value`); two or more of the oldest values do + * not open but a newer one does (`mixed`); values exist but none could be + * fetched (`unsampled`); the time budget ran out or a column could not be + * read (`incomplete`). The process serves, as before this check existed, + * writes NO canary, logs a warning, and probes again at the next boot. + * Serving rather than refusing because a refusal takes a correctly keyed + * server down with 503 on every request; not sealing because that would + * lock the right key out later. * - * Writes are `ON CONFLICT DO NOTHING`, so two processes racing is fine. + * The first seal can race another process. `ON CONFLICT DO NOTHING` keeps the + * row the other process wrote; when this process's insert affected no row, + * the stored canary is read back and must open under this process's key, or + * the outcome is a mismatch. * * Never logs or returns key material. The canary is written by raw SQL and is - * deliberately not in the rotation registry: it belongs to its key id. + * deliberately not in the rotation registry: it belongs to its key id, and the + * rotation script removes it once no ciphertext remains under that id + * (`retireCanariesWithoutData`). */ import { decrypt, + decryptBytes, + decryptStream, encryptUnderKeyId, extractKeyId, + extractKeyIdFromBytes, + extractStreamKeyId, + getActiveKeyId, getConfiguredKeyIds, + isStreamCiphertext, } from "@/lib/crypto"; -import { ENCRYPTED_COLUMNS } from "@/lib/crypto/encrypted-columns"; +import { + ENCRYPTED_COLUMNS, + encryptedColumnKey, + type EncryptedColumn, +} from "@/lib/crypto/encrypted-columns"; export const CANARY_PREFIX = "healthlog-canary:"; @@ -57,12 +98,21 @@ export function canaryPlaintext(keyId: string): string { return `${CANARY_PREFIX}${keyId}`; } -/** How many existing values the probe tries at most per key id. */ -export const PROBE_SAMPLE_LIMIT = 5; +/** How many of the oldest values the probe tries at most per key id. */ +export const PROBE_SAMPLE_LIMIT = 8; -/** How long the existing-data probe may take before the canary is written anyway. */ +/** How many of its oldest values each column contributes to the pool. */ +export const PROBE_ROWS_PER_COLUMN = 2; + +/** Values larger than this count as present but are not fetched. */ +export const PROBE_VALUE_MAX_BYTES = 2 * 1024 * 1024; + +/** How long the probe may take before it gives up as `inconclusive`. */ export const PROBE_BUDGET_MS = 10_000; +/** The key id the legacy bare-base64 string format belongs to. */ +const LEGACY_KEY_ID = "v1"; + /** The slice of a Prisma client this module needs. */ export interface CanaryClient { $queryRaw( @@ -73,31 +123,53 @@ export interface CanaryClient { query: TemplateStringsArray, ...values: unknown[] ): Promise; + $queryRawUnsafe?( + query: string, + ...values: unknown[] + ): Promise; + $executeRawUnsafe?(query: string, ...values: unknown[]): Promise; } -type ProbeDelegate = { - findMany: (args: { - where: Record; - select: Record; - take: number; - skip?: number; - }) => Promise>>; -}; +/** One stored value under the key id, as the probe sees it. */ +export interface ProbeRow { + /** The stored value; null when it exists but is too large to fetch. */ + value: string | Uint8Array | null; + /** The row's codec column, for a codec-dispatched column. */ + codec?: string | null; + /** The row's write time (see the module comment); null when it has none. */ + ts: Date | null; +} -export type ProbeVerdict = "opens" | "fails" | "inconclusive" | "none"; +/** + * Reads up to `limit` of the oldest values under `keyId` in one column, oldest + * first. Throws when the column cannot be read; an empty array means the + * column holds no value under that id. + */ +export type ColumnSampler = ( + column: EncryptedColumn, + keyId: string, + limit: number, +) => Promise; + +export type InconclusiveReason = + "single-value" | "mixed" | "unsampled" | "incomplete"; + +export type ProbeResult = + | { verdict: "opens" | "fails" | "none" } + | { verdict: "inconclusive"; reason: InconclusiveReason }; export type KeyCheckOutcome = | { state: "ok"; /** Key ids whose canary this run wrote. */ written: string[]; - /** Key ids whose canary existed and opened. */ + /** Key ids whose canary existed (or another process wrote) and opened. */ verified: string[]; /** - * Key ids with no canary whose only existing value did not open: not a - * mismatch, no canary written, probed again at the next boot. + * Key ids with no canary whose stored data proved nothing either way: + * not a mismatch, no canary written, probed again at the next boot. */ - inconclusive: string[]; + inconclusive: Array<{ keyId: string; reason: InconclusiveReason }>; } | { state: "mismatch"; keyIds: string[] } | { state: "error"; message: string }; @@ -112,77 +184,295 @@ function opensAs(ciphertext: string, keyId: string, expected?: string) { } } -/** - * Sample existing values sealed under `keyId` and try them; see the module - * comment for what each verdict means and why. - */ -export async function probeExistingData( - client: unknown, +function isBinaryLayout(column: EncryptedColumn, row: ProbeRow): boolean { + if (column.codec === "binary2") return true; + return column.codecField !== undefined && row.codec === "binary2"; +} + +/** Does one stored value open under `keyId`? Never throws. */ +export function probeValueOpens( + column: EncryptedColumn, + row: ProbeRow, keyId: string, - deadline: number, -): Promise { - const delegates = client as Record; - const columns = ENCRYPTED_COLUMNS.filter( - (c) => c.kind === "string" && !c.codec && !c.codecField, - ); - let tried = 0; - const columnsWithData: typeof columns = []; - - const sample = async ( - column: (typeof columns)[number], - skip: number, - take: number, - ): Promise => { - const delegate = - delegates[column.model.charAt(0).toLowerCase() + column.model.slice(1)]; - if (!delegate || typeof delegate.findMany !== "function") return []; - try { - const rows = await delegate.findMany({ - where: { [column.field]: { startsWith: `${keyId}.` } }, - select: { [column.field]: true }, - take, - ...(skip > 0 ? { skip } : {}), - }); - return rows - .map((row) => row[column.field]) - .filter((v): v is string => typeof v === "string"); - } catch { - return []; +): boolean { + const value = row.value; + if (value === null) return false; + try { + if (typeof value === "string") { + if (isStreamCiphertext(value)) { + if (extractStreamKeyId(value) !== keyId) return false; + decryptStream(value); + return true; + } + const id = extractKeyId(value); + if (id === null) { + // Legacy bare base64: only ever the legacy id's. + if (keyId !== LEGACY_KEY_ID) return false; + decrypt(value); + return true; + } + return opensAs(value, keyId); } - }; - - // Pass 1: one value per column, so different writers are heard first. - for (const column of columns) { - if (tried >= PROBE_SAMPLE_LIMIT || Date.now() > deadline) break; - const values = await sample(column, 0, 1); - if (values.length === 0) continue; - columnsWithData.push(column); - tried += 1; - if (opensAs(values[0], keyId)) return "opens"; - } - // Pass 2: more values from the columns that had any, up to the limit. - for (const column of columnsWithData) { - if (tried >= PROBE_SAMPLE_LIMIT || Date.now() > deadline) break; - const values = await sample(column, 1, PROBE_SAMPLE_LIMIT - tried); - for (const value of values) { - tried += 1; - if (opensAs(value, keyId)) return "opens"; + const buf = Buffer.from(value.buffer, value.byteOffset, value.byteLength); + if (isBinaryLayout(column, row)) { + if (extractKeyIdFromBytes(buf) !== keyId) return false; + decryptBytes(buf, column.aad); + return true; } + return opensAs(buf.toString("utf8"), keyId); + } catch { + return false; } - - if (tried >= 2) return "fails"; - if (tried === 1) return "inconclusive"; - return "none"; } +/** + * Probe the stored values under `keyId`; see the module comment for what each + * verdict means and why. + */ +export async function probeExistingData( + sampler: ColumnSampler, + keyId: string, + deadline: number, + columns: readonly EncryptedColumn[] = ENCRYPTED_COLUMNS, +): Promise { + const pool: Array<{ column: EncryptedColumn; row: ProbeRow; order: number }> = + []; + let present = 0; + for (const column of columns) { + if (Date.now() > deadline) { + return { verdict: "inconclusive", reason: "incomplete" }; + } + let rows: ProbeRow[]; + try { + rows = await sampler(column, keyId, PROBE_ROWS_PER_COLUMN); + } catch { + return { verdict: "inconclusive", reason: "incomplete" }; + } + present += rows.length; + for (const row of rows) { + if (row.value !== null) pool.push({ column, row, order: pool.length }); + } + } + if (Date.now() > deadline) { + return { verdict: "inconclusive", reason: "incomplete" }; + } + if (present === 0) return { verdict: "none" }; + if (pool.length === 0) { + return { verdict: "inconclusive", reason: "unsampled" }; + } + + // Oldest first; rows without a timestamp after every row with one. + pool.sort((a, b) => { + const ta = a.row.ts?.getTime() ?? Number.POSITIVE_INFINITY; + const tb = b.row.ts?.getTime() ?? Number.POSITIVE_INFINITY; + return ta === tb ? a.order - b.order : ta - tb; + }); + const tried = pool.slice(0, PROBE_SAMPLE_LIMIT); + + let failedBeforeFirstOpen = 0; + for (const { column, row } of tried) { + if (probeValueOpens(column, row, keyId)) { + return failedBeforeFirstOpen <= 1 + ? { verdict: "opens" } + : { verdict: "inconclusive", reason: "mixed" }; + } + failedBeforeFirstOpen += 1; + } + return tried.length >= 2 + ? { verdict: "fails" } + : { verdict: "inconclusive", reason: "single-value" }; +} + +// ─── The SQL sampler ──────────────────────────────────────────────────────── + +interface ColumnLocation { + table: string; + column: string; + pk: string; + ts: string | null; + codec: string | null; +} + +type RuntimeDataModel = { + models: Record< + string, + { + dbName: string | null; + fields: Array<{ name: string; dbName?: string | null }>; + } + >; +}; + +const IDENTIFIER = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/; +const KEY_ID = /^[A-Za-z0-9_-]{1,32}$/; + +function quoteIdentifier(name: string): string { + if (!IDENTIFIER.test(name)) { + throw new Error(`Unexpected identifier in the column registry: ${name}`); + } + return `"${name}"`; +} + +/** + * Table and column names for a registry entry, read from the Prisma client's + * runtime data model (the same `@@map` / `@map` names the client queries + * with). `canary.test.ts` pins that every registry column resolves, so a + * Prisma upgrade that moves this fails a test rather than the probe. + */ +export function resolveColumnLocation( + client: unknown, + column: EncryptedColumn, +): ColumnLocation { + const dm = (client as { _runtimeDataModel?: RuntimeDataModel }) + ._runtimeDataModel; + const model = dm?.models?.[column.model]; + if (!model) { + throw new Error(`No data model for ${column.model}`); + } + const dbField = (name: string): string | null => { + const f = model.fields.find((x) => x.name === name); + return f ? (f.dbName ?? f.name) : null; + }; + const columnName = dbField(column.field); + const pk = dbField(column.pkField ?? "id"); + if (!columnName || !pk) { + throw new Error(`No column for ${encryptedColumnKey(column)}`); + } + return { + table: model.dbName ?? column.model, + column: columnName, + pk, + ts: dbField("updatedAt") ?? dbField("createdAt"), + codec: column.codecField ? dbField(column.codecField) : null, + }; +} + +/** + * The SQL that finds values under `keyId` in one column. Identifiers come from + * the Prisma data model and are checked against `IDENTIFIER` before they are + * spliced; the key id is checked against the key-id grammar and travels as a + * bound parameter. The prefix tests use `substr` / `substring`, which read + * only the head of a large out-of-line value instead of the whole of it. + */ +function columnQuery( + loc: ColumnLocation, + column: EncryptedColumn, + keyId: string, + mode: "sample" | "exists", +): { sql: string; params: unknown[] } { + if (!KEY_ID.test(keyId)) throw new Error("Malformed key id"); + const c = quoteIdentifier(loc.column); + const params: unknown[] = []; + const bind = (v: unknown) => { + params.push(v); + return `$${params.length}`; + }; + const tests: string[] = []; + if (column.kind === "string") { + const versioned = `${keyId}.`; + const streamed = `~hlgcm1.${keyId}.`; + tests.push( + `substr(${c}, 1, ${versioned.length}) = ${bind(versioned)}`, + `substr(${c}, 1, ${streamed.length}) = ${bind(streamed)}`, + ); + if (keyId === LEGACY_KEY_ID) { + // A legacy value is bare base64 of at least IV + tag + one block. + tests.push(`substr(${c}, 1, 40) ~ '^[A-Za-z0-9+/]{40}$'`); + } + } else { + const utf8 = Buffer.from(`${keyId}.`, "ascii"); + const id = Buffer.from(keyId, "ascii"); + const binary = Buffer.concat([Buffer.from([0x02, id.length]), id]); + if (column.codec !== "binary2") { + tests.push( + `substring(${c} from 1 for ${utf8.length}) = ${bind(utf8)}::bytea`, + ); + } + if (column.codec === "binary2" || column.codecField) { + tests.push( + `substring(${c} from 1 for ${binary.length}) = ${bind(binary)}::bytea`, + ); + } + } + const table = quoteIdentifier(loc.table); + const where = `${c} IS NOT NULL AND (${tests.join(" OR ")})`; + if (mode === "exists") { + return { + sql: `SELECT 1 AS one FROM ${table} WHERE ${where} LIMIT 1`, + params, + }; + } + const ts = loc.ts ? quoteIdentifier(loc.ts) : null; + const codec = loc.codec ? quoteIdentifier(loc.codec) : "NULL"; + const max = Math.floor(PROBE_VALUE_MAX_BYTES); + const sql = + `SELECT CASE WHEN octet_length(${c}) <= ${max} THEN ${c} END AS value, ` + + `${codec} AS codec, ${ts ?? "NULL::timestamp"} AS ts ` + + `FROM ${table} WHERE ${where} ` + + `ORDER BY ${ts ? `${ts} ASC NULLS LAST, ` : ""}${quoteIdentifier(loc.pk)} ASC ` + + `LIMIT ${bind(0)}`; + return { sql, params }; +} + +function requireUnsafe(client: CanaryClient) { + const query = client.$queryRawUnsafe; + if (typeof query !== "function") { + throw new Error("The database client cannot run the probe queries"); + } + return query.bind(client); +} + +/** The production sampler: one bounded query per column. */ +export function sqlColumnSampler(client: CanaryClient): ColumnSampler { + const query = requireUnsafe(client); + return async (column, keyId, limit) => { + const loc = resolveColumnLocation(client, column); + const { sql, params } = columnQuery(loc, column, keyId, "sample"); + params[params.length - 1] = limit; + const rows = await query< + Array<{ + value: string | Uint8Array | null; + codec: string | null; + ts: Date | null; + }> + >(sql, ...params); + return rows.map((r) => ({ + value: r.value, + codec: r.codec, + ts: r.ts instanceof Date ? r.ts : r.ts ? new Date(r.ts) : null, + })); + }; +} + +/** + * Every registered column that still holds a value under `keyId`, by + * `Model.field`. Exhaustive: one existence query per column, no budget. + */ +export async function columnsHoldingKeyId( + client: CanaryClient, + keyId: string, +): Promise { + const query = requireUnsafe(client); + const holding: string[] = []; + for (const column of ENCRYPTED_COLUMNS) { + const loc = resolveColumnLocation(client, column); + const { sql, params } = columnQuery(loc, column, keyId, "exists"); + const rows = await query(sql, ...params); + if (rows.length > 0) holding.push(encryptedColumnKey(column)); + } + return holding; +} + +// ─── The check ────────────────────────────────────────────────────────────── + /** * Run the check for every configured key id. Never throws: a failure to run - * the check at all is reported as `error`, and only a value that provably does - * not open is reported as `mismatch`. + * the check at all is reported as `error`, and only values that provably do + * not open are reported as `mismatch`. */ export async function checkEncryptionKeyCanaries( client: CanaryClient, - options: { probeBudgetMs?: number } = {}, + options: { probeBudgetMs?: number; sampler?: ColumnSampler } = {}, ): Promise { let keyIds: string[]; try { @@ -217,12 +507,21 @@ export async function checkEncryptionKeyCanaries( const deadline = Date.now() + (options.probeBudgetMs ?? PROBE_BUDGET_MS); const toWrite: string[] = []; - const inconclusive: string[] = []; - for (const keyId of missing) { - const probe = await probeExistingData(client, keyId, deadline); - if (probe === "fails") mismatched.push(keyId); - else if (probe === "inconclusive") inconclusive.push(keyId); - else toWrite.push(keyId); + const inconclusive: Array<{ keyId: string; reason: InconclusiveReason }> = []; + if (missing.length > 0) { + let sampler: ColumnSampler; + try { + sampler = options.sampler ?? sqlColumnSampler(client); + } catch (err) { + return { state: "error", message: (err as Error).message }; + } + for (const keyId of missing) { + const probe = await probeExistingData(sampler, keyId, deadline); + if (probe.verdict === "fails") mismatched.push(keyId); + else if (probe.verdict === "inconclusive") { + inconclusive.push({ keyId, reason: probe.reason }); + } else toWrite.push(keyId); + } } if (mismatched.length > 0) { @@ -231,16 +530,61 @@ export async function checkEncryptionKeyCanaries( return { state: "mismatch", keyIds: mismatched.sort() }; } + const written: string[] = []; try { for (const keyId of toWrite) { const ciphertext = encryptUnderKeyId(canaryPlaintext(keyId), keyId); - await client.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES (${keyId}, ${ciphertext}) ON CONFLICT (key_id) DO NOTHING`; + const inserted = + await client.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES (${keyId}, ${ciphertext}) ON CONFLICT (key_id) DO NOTHING`; + if (inserted > 0) { + written.push(keyId); + continue; + } + // Another process sealed this id first. Its canary has to open under + // this process's key, or one of the two holds the wrong key. + const [winner] = await client.$queryRaw>` + SELECT ciphertext FROM encryption_key_canaries WHERE key_id = ${keyId}`; + if (winner && opensAs(winner.ciphertext, keyId, canaryPlaintext(keyId))) { + verified.push(keyId); + } else { + mismatched.push(keyId); + } } } catch (err) { return { state: "error", message: (err as Error).message }; } + if (mismatched.length > 0) { + return { state: "mismatch", keyIds: mismatched.sort() }; + } - return { state: "ok", written: toWrite, verified, inconclusive }; + return { state: "ok", written, verified, inconclusive }; +} + +/** + * After a rotation: remove the canary of every configured key id other than + * the active one that no longer holds a single value in any registered + * column. The id's key can then leave `ENCRYPTION_KEYS`; and if the id is + * ever reused for a different key, no stale canary refuses it. + */ +export async function retireCanariesWithoutData(client: CanaryClient): Promise<{ + removed: string[]; + remaining: Record; +}> { + const active = getActiveKeyId(); + const removed: string[] = []; + const remaining: Record = {}; + for (const keyId of getConfiguredKeyIds()) { + if (keyId === active) continue; + const holding = await columnsHoldingKeyId(client, keyId); + if (holding.length > 0) { + remaining[keyId] = holding; + continue; + } + const deleted = + await client.$executeRaw`DELETE FROM encryption_key_canaries WHERE key_id = ${keyId}`; + if (deleted > 0) removed.push(keyId); + } + return { removed, remaining }; } /** @@ -266,6 +610,13 @@ export function keyMismatchLogBlock(keyIds: string[]): string { " 3. Start fresh: an empty database with this key. Stored data and", " in-database backups of the old database cannot be read without", " the old key.", + "", + " Only if you are CERTAIN the configured key is the one this data was", + " written with (the check recorded a different key earlier), reset the", + " check and restart; it then probes the stored data again:", + ...keyIds.map( + (id) => ` DELETE FROM encryption_key_canaries WHERE key_id = '${id}';`, + ), "============================================================", ].join("\n"); } diff --git a/src/lib/jobs/encryption-key-rotate.ts b/src/lib/jobs/encryption-key-rotate.ts index cd9c93eb8..cf1c52250 100644 --- a/src/lib/jobs/encryption-key-rotate.ts +++ b/src/lib/jobs/encryption-key-rotate.ts @@ -32,6 +32,10 @@ import { type CorpusClient, } from "@/lib/crypto/encryption-corpus"; import { getWorkerPrisma } from "@/lib/jobs/reminder/shared"; +import { + retireCanariesWithoutData, + type CanaryClient, +} from "@/lib/crypto/canary"; export const ENCRYPTION_KEY_ROTATE_QUEUE = "encryption-key-rotate"; export const ENCRYPTION_KEY_ROTATE_CONCURRENCY = 1; @@ -62,9 +66,20 @@ export async function runEncryptionKeyRotation( totalErrors: number; totalDropped: number; stoppedEarly: boolean; + /** Key ids whose boot key-check record this run removed. */ + retiredKeyIds: string[]; }> { const prisma = getWorkerPrisma(); const out = await rotateCorpus(prisma as unknown as CorpusClient, shouldStop); + // A complete, error-free pass: remove the boot key check's record of every + // other key id that no longer holds a value in any registered column, so + // its key can leave ENCRYPTION_KEYS and the id can never refuse a later, + // different key. The existence check walks every column itself. + const retiredKeyIds = + !out.stoppedEarly && out.totalErrors === 0 + ? (await retireCanariesWithoutData(prisma as unknown as CanaryClient)) + .removed + : []; return { activeKeyId: out.activeKeyId, totalScanned: out.totalScanned, @@ -72,6 +87,7 @@ export async function runEncryptionKeyRotation( totalErrors: out.totalErrors, totalDropped: out.totalDropped, stoppedEarly: out.stoppedEarly, + retiredKeyIds, }; } @@ -98,6 +114,7 @@ export async function handleEncryptionKeyRotate( rotated: result.totalRotated, errors: result.totalErrors, dropped: result.totalDropped, + retiredKeyIds: result.retiredKeyIds, }, }); // Per-row errors are the fail-closed skip of a row under a key the diff --git a/tests/integration/encryption-key-check.test.ts b/tests/integration/encryption-key-check.test.ts index 926221f86..01f2796af 100644 --- a/tests/integration/encryption-key-check.test.ts +++ b/tests/integration/encryption-key-check.test.ts @@ -47,9 +47,17 @@ vi.mock("@/lib/db-compat", () => ({ import { _resetCryptoCacheForTests, encrypt, + encryptUnderKeyId, getKeyFingerprint, } from "@/lib/crypto"; +import { encryptToBytes } from "@/lib/ai/coach/bytes-codec"; import { runBootKeyCheck } from "@/lib/boot/key-check"; +import { + canaryPlaintext, + checkEncryptionKeyCanaries, + retireCanariesWithoutData, + type CanaryClient, +} from "@/lib/crypto/canary"; import { isKeyMismatch, setKeyMismatchState, @@ -223,6 +231,214 @@ describe("boot key check probe over existing data (real Postgres)", () => { }); }); +/** Two users, one oldest, with ciphertext in string AND Bytes columns. */ +async function seedUser( + name: string, + writtenAt: Date, + seal: () => { text: string; bytes: Uint8Array }, +) { + const prisma = getPrismaClient(); + const a = seal(); + const b = seal(); + const user = await prisma.user.create({ + data: { + username: name, + email: `${name}@example.test`, + codexAccessTokenEncrypted: a.text, + codexRefreshTokenEncrypted: b.text, + createdAt: writtenAt, + updatedAt: writtenAt, + }, + }); + await prisma.userHealthProfile.create({ + data: { + userId: user.id, + aboutMeEncrypted: a.bytes, + conditionsEncrypted: b.bytes, + createdAt: writtenAt, + updatedAt: writtenAt, + }, + }); + return user.id; +} + +const sealNow = () => ({ + text: encrypt("a stored value"), + bytes: encryptToBytes("a stored note"), +}); + +describe("boot key check: what decides, and what may seal (real Postgres)", () => { + it("probes Bytes columns: an install whose ciphertext is only Bytes does not seal a wrong key", async () => { + const prisma = getPrismaClient(); + const user = await prisma.user.create({ + data: { username: "bytes-only", email: "bytes-only@example.test" }, + }); + await prisma.userHealthProfile.create({ + data: { + userId: user.id, + aboutMeEncrypted: encryptToBytes("about me"), + conditionsEncrypted: encryptToBytes("conditions"), + allergiesEncrypted: encryptToBytes("allergies"), + }, + }); + + useKey(KEY_B); + const errors = vi.spyOn(console, "error").mockImplementation(() => {}); + expect(await runBootKeyCheck(prisma)).toBe(true); + errors.mockRestore(); + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + + useKey(KEY_A); + expect(await runBootKeyCheck(prisma)).toBe(false); + expect(await prisma.encryptionKeyCanary.count()).toBe(1); + }); + + it("a probe that runs out of time seals nothing and keeps serving", async () => { + const prisma = getPrismaClient(); + await seedUser("slow", new Date("2024-01-01T00:00:00Z"), sealNow); + const outcome = await checkEncryptionKeyCanaries(prisma, { + probeBudgetMs: -1, + }); + expect(outcome.state).toBe("ok"); + if (outcome.state !== "ok") return; + expect(outcome.written).toEqual([]); + expect(outcome.inconclusive.map((i) => i.keyId)).toEqual(["v1"]); + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + }); + + it("a fresh database with no ciphertext under the key id seals it", async () => { + const prisma = getPrismaClient(); + // Ciphertext exists, but under another key id: not evidence about v1. + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_A, v2: KEY_B })); + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v2"); + vi.stubEnv("ENCRYPTION_KEY", ""); + _resetCryptoCacheForTests(); + await seedUser("v2-only", new Date("2024-01-01T00:00:00Z"), sealNow); + expect(await runBootKeyCheck(prisma)).toBe(false); + const ids = (await prisma.encryptionKeyCanary.findMany()) + .map((r) => r.keyId) + .sort(); + expect(ids).toEqual(["v1", "v2"]); + }); + + it("rows a wrong key wrote recently never decide: old rows under A, newer under B", async () => { + const prisma = getPrismaClient(); + // The original data, under A. + await seedUser("original", new Date("2024-01-01T00:00:00Z"), sealNow); + await seedUser("original-2", new Date("2024-02-01T00:00:00Z"), sealNow); + // A process holding B served for a while (no canary yet) and wrote more. + useKey(KEY_B); + for (let i = 0; i < 6; i++) { + await seedUser( + `recent-${i}`, + new Date(Date.UTC(2026, 8, 1 + i)), + sealNow, + ); + } + + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + const errors = vi.spyOn(console, "error").mockImplementation(() => {}); + await runBootKeyCheck(prisma); + warn.mockRestore(); + errors.mockRestore(); + // B is not sealed as the right key, whatever else the verdict is. + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + const outcome = await checkEncryptionKeyCanaries(prisma); + expect( + outcome.state === "mismatch" || + (outcome.state === "ok" && outcome.written.length === 0), + ).toBe(true); + + // The key the oldest data was written with is the right one. + useKey(KEY_A); + setKeyMismatchState(null); + expect(await runBootKeyCheck(prisma)).toBe(false); + expect(isKeyMismatch()).toBe(false); + expect(await prisma.encryptionKeyCanary.count()).toBe(1); + }); + + it("losing the first-seal race to another key is a mismatch, not ok", async () => { + const prisma = getPrismaClient(); + // The other process, holding A, wins the insert between our probe and + // our write. + const sealedUnderA = encryptUnderKeyId(canaryPlaintext("v1"), "v1"); + useKey(KEY_B); + let raced = false; + const racing = new Proxy(prisma, { + get(target, prop, receiver) { + if (prop === "$executeRaw") { + return async (q: TemplateStringsArray, ...values: unknown[]) => { + if (!raced) { + raced = true; + await target.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES ('v1', ${sealedUnderA})`; + } + return target.$executeRaw(q, ...values); + }; + } + const value = Reflect.get(target, prop, receiver); + return typeof value === "function" ? value.bind(target) : value; + }, + }) as unknown as CanaryClient; + const outcome = await checkEncryptionKeyCanaries(racing); + expect(raced).toBe(true); + expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + + // The same race lost to a process holding the SAME key is fine. + await prisma.$executeRaw`DELETE FROM encryption_key_canaries`; + raced = false; + useKey(KEY_A); + const same = await checkEncryptionKeyCanaries(racing); + expect(same.state).toBe("ok"); + }); +}); + +describe("retiring a key id after rotation (real Postgres)", () => { + it("removes the canary of a key id only once no ciphertext remains under it", async () => { + const prisma = getPrismaClient(); + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_A, v2: KEY_B })); + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v1"); + vi.stubEnv("ENCRYPTION_KEY", ""); + _resetCryptoCacheForTests(); + const userId = await seedUser( + "rotating", + new Date("2024-01-01T00:00:00Z"), + sealNow, + ); + expect(await runBootKeyCheck(prisma)).toBe(false); + expect(await prisma.encryptionKeyCanary.count()).toBe(2); + + // Rotation to v2 has not finished: one Bytes value still under v1. + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v2"); + _resetCryptoCacheForTests(); + await prisma.user.update({ + where: { id: userId }, + data: { + codexAccessTokenEncrypted: encrypt("rotated"), + codexRefreshTokenEncrypted: encrypt("rotated"), + }, + }); + await prisma.userHealthProfile.update({ + where: { userId }, + data: { aboutMeEncrypted: encryptToBytes("rotated") }, + }); + const partial = await retireCanariesWithoutData(prisma); + expect(partial.removed).toEqual([]); + expect(partial.remaining).toEqual({ + v1: ["UserHealthProfile.conditionsEncrypted"], + }); + expect(await prisma.encryptionKeyCanary.count()).toBe(2); + + await prisma.userHealthProfile.update({ + where: { userId }, + data: { conditionsEncrypted: encryptToBytes("rotated") }, + }); + const done = await retireCanariesWithoutData(prisma); + expect(done.removed).toEqual(["v1"]); + const left = await prisma.encryptionKeyCanary.findMany(); + expect(left.map((r) => r.keyId)).toEqual(["v2"]); + }); +}); + describe("encryption key backup step (real Postgres)", () => { it("is due, refuses a stale confirmation, records the right one, and is due again after a re-key", async () => { const prisma = getPrismaClient();