From 2c295bbea86354001e456de2ecbfafb45f91af88 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc-Andr=C3=A9=20Bombeck?= Date: Sat, 3 Oct 2026 23:29:55 +0200 Subject: [PATCH 1/3] Probe the oldest stored values before recording an encryption key The boot key check recorded whatever key was configured whenever its probe found nothing to try. It only read string columns, so an install whose ciphertext sat in Bytes columns (notes, documents, ECG, symptoms) read as empty; a probe that ran out of time did the same; and rows written by a process that served under a wrong key could open on the next boot and get that key recorded as the right one. The probe now reads every registered column, string and Bytes, in all four stored layouts, and tries the oldest values first, ordered by the row's updatedAt (else createdAt, else primary key). A key is recorded only when the oldest value that opens is preceded by at most one that does not, or when no column holds a single value under the key id. A time-out, an unreadable table, oversized values or mixed results are inconclusive: the process serves, records nothing and checks again at the next start. When the first insert loses a race, the stored record is read back and must open under this process's key, else the outcome is a mismatch. A rotation that walked every column without errors removes the record of each previous key id that no longer holds any value. The refusal log block and the self-hosting pages explain how to reset the check for a key id. --- docs/ops/encryption-key-rotation.md | 7 + docs/self-hosting/getting-started.md | 28 + docs/self-hosting/truenas.md | 26 + scripts/rotate-encryption-key.ts | 29 + src/lib/boot/key-check.ts | 22 +- src/lib/crypto/__tests__/canary.test.ts | 429 ++++++++++--- src/lib/crypto/canary.ts | 567 ++++++++++++++---- src/lib/jobs/encryption-key-rotate.ts | 17 + .../integration/encryption-key-check.test.ts | 216 +++++++ 9 files changed, 1142 insertions(+), 199 deletions(-) diff --git a/docs/ops/encryption-key-rotation.md b/docs/ops/encryption-key-rotation.md index 0396e2897..b48c64a72 100644 --- a/docs/ops/encryption-key-rotation.md +++ b/docs/ops/encryption-key-rotation.md @@ -103,6 +103,13 @@ under `v1` (the synthetic id assigned to the existing `ENCRYPTION_KEY`). a full `Columns walked` line. Both read the same registry the rotation walks. + A run that walked every column without an error also removes the boot + key check's record of each previous key id that no longer holds a single + value (`encryption_key_canaries`); the script prints which ids it removed + and, for an id that still holds values, the columns that do. With the record + gone, the id can later carry a different key without the start-up check + refusing it. + A zero only means "nothing left" when the run also says it walked every registered column. A zero from a run that skipped columns means "never looked", and dropping the old key on it makes those rows permanently diff --git a/docs/self-hosting/getting-started.md b/docs/self-hosting/getting-started.md index f67feb66f..cc0499067 100644 --- a/docs/self-hosting/getting-started.md +++ b/docs/self-hosting/getting-started.md @@ -267,6 +267,34 @@ every API request answers `503` with `meta.errorCode` Restore the original key, point `DATABASE_URL` at the matching database, or start with an empty database; then restart. +The check remembers the key it first saw for each key id in the table +`encryption_key_canaries`. Before it records a key, it reads the oldest +encrypted values in the database and records the key only if they open, or +if the database holds no encrypted value under that key id at all. When the +stored data proves neither (a single unreadable value, the oldest values +failing while newer ones open, or a probe that ran out of time), the app +keeps serving, records nothing, logs a warning that starts with +`[boot] Encryption key check inconclusive`, and checks again at the next +start. + +If you are certain the configured key is the one your data was written +with, and the check still refuses because it recorded a different key +earlier, reset the check for that key id and restart. The refusal in the log +prints the exact statement; for the default key id it is: + +```bash +docker compose exec -T db psql -U healthlog healthlog \ + -c "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';" +docker compose restart app +``` + +The next start probes the stored data again and records the key only if +that data opens with it. Do this only when you are sure: if the key is in +fact wrong, the app serves errors on every encrypted value instead of the +one clear refusal. After a key rotation the rotation script removes the +record of a retired key id itself, once it reports that no values remain +under that id. + ## Backup and restore Your database is the only stateful piece — back it up, and back up the diff --git a/docs/self-hosting/truenas.md b/docs/self-hosting/truenas.md index 8bebe7441..c39f6e016 100644 --- a/docs/self-hosting/truenas.md +++ b/docs/self-hosting/truenas.md @@ -80,6 +80,32 @@ Fix it one of three ways, then restart the app: 3. Start fresh with an empty database. The old data and its backups need the old key. +Before it records a key as the right one, the check reads the oldest +encrypted values in the database and records the key only if they open (or if +the database holds no encrypted value under that key id yet). When the data +proves neither, the app keeps serving, records nothing, writes a warning +starting with `[boot] Encryption key check inconclusive` to the log, and +checks again at the next start. + +### Resetting the check + +If you are certain the configured key is the one the data was written with, +and the app still refuses because the check recorded a different key earlier, +remove that record and restart. Open a shell on the database container (Apps, +HealthLog, the database container's Shell) and run, with the key id the log +names (`v1` unless you use `ENCRYPTION_KEYS`): + +```bash +psql -U healthlog healthlog \ + -c "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';" +``` + +Then restart the app. It probes the stored data again and records the key +only if that data opens with it. If the key is in fact wrong, the app then +fails on every encrypted value instead of refusing clearly, so only do this +when you are sure. After a key rotation you never need this for the old key +id: the rotation script removes its record once no values remain under it. + ## Updates Pick the new tag, take a snapshot of the database dataset first, and update. diff --git a/scripts/rotate-encryption-key.ts b/scripts/rotate-encryption-key.ts index a3cd1742a..6e054ba8b 100644 --- a/scripts/rotate-encryption-key.ts +++ b/scripts/rotate-encryption-key.ts @@ -17,6 +17,10 @@ * non-zero, because an operator reads a zero here as permission to drop the * previous key, and "nothing left" and "never looked" must not print alike. * + * After a complete, error-free run it checks every other configured key id + * for remaining values and removes the boot key check's record (canary) of + * each id that holds none, naming the columns of each id that still does. + * * Usage: * ENCRYPTION_KEYS='{"v1":"","v2":""}' \ * ENCRYPTION_ACTIVE_KEY_ID=v2 \ @@ -27,6 +31,7 @@ import { Buffer } from "node:buffer"; import { PrismaClient } from "@/generated/prisma/client"; import { PrismaPg } from "@prisma/adapter-pg"; import { decrypt, encrypt, extractKeyId, getActiveKeyId } from "@/lib/crypto"; +import { retireCanariesWithoutData } from "@/lib/crypto/canary"; import { rotateColumn, type CorpusClient, @@ -1034,6 +1039,30 @@ async function main() { `drop the previous key:\n ${notWalked.join("\n ")}`, ); } + + // ───── Key-check records of retired key ids ───── + // The boot key check keeps one canary per configured key id. Once no + // registered column holds a single value under an id, that id's key can + // leave ENCRYPTION_KEYS, and its canary goes with it: left behind, it would + // refuse a later, different key configured under the same id. Only after a + // complete, error-free run; the existence check walks every column itself + // rather than trusting the counts above. + if (notWalked.length === 0 && totalErrors === 0) { + const { removed, remaining } = await retireCanariesWithoutData(prisma); + for (const [keyId, columns] of Object.entries(remaining)) { + console.log( + `\nKey id '${keyId}': values remain in ${columns.length} column(s); ` + + `keep its key:\n ${columns.join("\n ")}`, + ); + } + for (const keyId of removed) { + console.log( + `\nKey id '${keyId}': no values remain. Its key-check record was ` + + `removed; its key can now leave ENCRYPTION_KEYS.`, + ); + } + } + await prisma.$disconnect(); if (notWalked.length > 0) process.exit(4); process.exit(totalErrors > 0 ? 3 : 0); diff --git a/src/lib/boot/key-check.ts b/src/lib/boot/key-check.ts index 12c0c22bb..c4deeb0b7 100644 --- a/src/lib/boot/key-check.ts +++ b/src/lib/boot/key-check.ts @@ -13,9 +13,23 @@ import { checkEncryptionKeyCanaries, keyMismatchLogBlock, type CanaryClient, + type InconclusiveReason, } from "@/lib/crypto/canary"; import { setKeyMismatchState } from "./key-mismatch-state"; +const INCONCLUSIVE_EXPLANATION: Record = { + "single-value": "the only stored value found under that id did not open.", + mixed: + "the oldest stored values under that id do not open, newer ones do. " + + "That is what a wrong key that served for a while leaves behind; check " + + "that this is the key the database was first written with.", + unsampled: + "the values stored under that id are too large to probe at start-up.", + incomplete: + "the probe of the stored data did not finish in time or a table could " + + "not be read.", +}; + export async function runBootKeyCheck(client: CanaryClient): Promise { const outcome = await checkEncryptionKeyCanaries(client); if (outcome.state === "mismatch") { @@ -34,11 +48,11 @@ export async function runBootKeyCheck(client: CanaryClient): Promise { `[boot] Encryption key check skipped: ${outcome.message.slice(0, 300)}`, ); } else { - if (outcome.inconclusive.length > 0) { + for (const { keyId, reason } of outcome.inconclusive) { console.warn( - `[boot] Encryption key check inconclusive for key id(s) ${outcome.inconclusive.join(", ")}: ` + - "the only stored value found under that id did not open. Serving; " + - "no canary written; the check runs again at the next start.", + `[boot] Encryption key check inconclusive for key id ${keyId}: ` + + `${INCONCLUSIVE_EXPLANATION[reason]} Serving; no key recorded; ` + + "the check runs again at the next start.", ); } if (outcome.written.length > 0) { diff --git a/src/lib/crypto/__tests__/canary.test.ts b/src/lib/crypto/__tests__/canary.test.ts index 170e1e5bf..bb70ee45b 100644 --- a/src/lib/crypto/__tests__/canary.test.ts +++ b/src/lib/crypto/__tests__/canary.test.ts @@ -1,17 +1,30 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; +import { PrismaClient } from "@/generated/prisma/client"; +import { PrismaPg } from "@prisma/adapter-pg"; import { _resetCryptoCacheForTests, candidateMatchesKey, encrypt, + encryptBytes, encryptUnderKeyId, + extractKeyIdFromBytes, fingerprintKeyBytes, getKeyFingerprint, } from "@/lib/crypto"; +import { encryptToBytes } from "@/lib/ai/coach/bytes-codec"; +import { + ENCRYPTED_COLUMNS, + WORKOUT_ROUTE_GEOMETRY_AAD, + encryptedColumnKey, +} from "@/lib/crypto/encrypted-columns"; import { canaryPlaintext, checkEncryptionKeyCanaries, keyMismatchLogBlock, + probeValueOpens, + resolveColumnLocation, type CanaryClient, + type ColumnSampler, } from "../canary"; const KEY_A = @@ -19,23 +32,42 @@ const KEY_A = const KEY_B = "fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210"; -function useKey(hex: string) { +function configureKey(hex: string) { vi.stubEnv("ENCRYPTION_KEYS", ""); vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", ""); vi.stubEnv("ENCRYPTION_KEY", hex); _resetCryptoCacheForTests(); } -/** An in-memory stand-in for the two raw statements and the probe delegates. */ +type Seeded = + | string + | Uint8Array + | { value: string | Uint8Array | null; ts?: string; codec?: string }; + +/** + * An in-memory stand-in for the raw statements, plus a sampler over seeded + * values keyed by `Model.field`. A seeded value's `ts` orders it; values + * without one keep their seeded order after every dated value, as the SQL + * sampler orders them. + */ function fakeClient(opts: { canaries?: Record; - /** delegate → field → stored values (in row order). */ - probe?: Record>; + probe?: Record; + /** Make the sampler throw for these columns. */ + unreadable?: string[]; + /** Another process seals this id (with this ciphertext) just before us. */ + race?: { keyId: string; ciphertext: string }; }) { const canaries = new Map(Object.entries(opts.canaries ?? {})); const inserts: string[] = []; + let raced = false; const client = { - async $queryRaw() { + async $queryRaw(_q: TemplateStringsArray, ...values: unknown[]) { + const [arg] = values; + if (typeof arg === "string") { + const c = canaries.get(arg); + return c ? [{ ciphertext: c }] : []; + } return [...canaries.entries()].map(([key_id, ciphertext]) => ({ key_id, ciphertext, @@ -43,104 +75,118 @@ function fakeClient(opts: { }, async $executeRaw(_q: TemplateStringsArray, ...values: unknown[]) { const [keyId, ciphertext] = values as [string, string]; + if (opts.race && !raced && opts.race.keyId === keyId) { + raced = true; + canaries.set(keyId, opts.race.ciphertext); + } inserts.push(keyId); - if (!canaries.has(keyId)) canaries.set(keyId, ciphertext); + if (canaries.has(keyId)) return 0; + canaries.set(keyId, ciphertext); return 1; }, - } as unknown as CanaryClient & Record; - for (const [delegate, fields] of Object.entries(opts.probe ?? {})) { - client[delegate] = { - async findMany(args: { - select: Record; - take: number; - skip?: number; - }) { - const field = Object.keys(args.select)[0]; - const skip = args.skip ?? 0; - return (fields[field] ?? []) - .slice(skip, skip + args.take) - .map((value) => ({ [field]: value })); - }, - }; - } - return { client, canaries, inserts }; + } as unknown as CanaryClient; + const sampler: ColumnSampler = async (column, keyId, limit) => { + const key = `${column.model}.${column.field}`; + if (opts.unreadable?.includes(key)) throw new Error("relation missing"); + const rows = (opts.probe?.[key] ?? []).map((v) => + typeof v === "string" || v instanceof Uint8Array + ? { value: v, ts: null as Date | null, codec: null } + : { + value: v.value, + ts: v.ts ? new Date(v.ts) : null, + codec: v.codec ?? null, + }, + ); + const under = rows.filter((r) => { + if (r.value === null) return true; + if (typeof r.value === "string") return r.value.startsWith(`${keyId}.`); + const b = Buffer.from(r.value); + return ( + b.toString("latin1").startsWith(`${keyId}.`) || + extractKeyIdFromBytes(b) === keyId + ); + }); + return under + .map((r, i) => ({ r, i })) + .sort((a, b) => { + const ta = a.r.ts?.getTime() ?? Infinity; + const tb = b.r.ts?.getTime() ?? Infinity; + return ta === tb ? a.i - b.i : ta - tb; + }) + .slice(0, limit) + .map(({ r }) => r); + }; + const check = (budget?: number) => + checkEncryptionKeyCanaries(client, { sampler, probeBudgetMs: budget }); + return { client, sampler, canaries, inserts, check }; } +const ok = (fields: { + written?: string[]; + verified?: string[]; + inconclusive?: Array<{ keyId: string; reason: string }>; +}) => ({ + state: "ok", + written: fields.written ?? [], + verified: fields.verified ?? [], + inconclusive: fields.inconclusive ?? [], +}); + describe("encryption key canary", () => { beforeEach(() => { vi.unstubAllEnvs(); - useKey(KEY_A); + configureKey(KEY_A); }); - it("writes a canary for a configured key with no row and no data", async () => { - const { client, canaries, inserts } = fakeClient({}); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ - state: "ok", - written: ["v1"], - verified: [], - inconclusive: [], - }); + it("writes a canary for a configured key when no column holds a value under its id", async () => { + const { canaries, inserts, check } = fakeClient({}); + expect(await check()).toEqual(ok({ written: ["v1"] })); expect(inserts).toEqual(["v1"]); expect(canaries.get("v1")?.startsWith("v1.")).toBe(true); }); it("verifies a canary written under the same key", async () => { - const { client, inserts } = fakeClient({ + const { inserts, check } = fakeClient({ canaries: { v1: encryptUnderKeyId(canaryPlaintext("v1"), "v1") }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ - state: "ok", - written: [], - verified: ["v1"], - inconclusive: [], - }); + expect(await check()).toEqual(ok({ verified: ["v1"] })); expect(inserts).toEqual([]); }); it("reports a mismatch when the key changed under the same id", async () => { const sealedUnderA = encryptUnderKeyId(canaryPlaintext("v1"), "v1"); - useKey(KEY_B); - const { client, inserts } = fakeClient({ canaries: { v1: sealedUnderA } }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + configureKey(KEY_B); + const { inserts, check } = fakeClient({ canaries: { v1: sealedUnderA } }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); expect(inserts).toEqual([]); }); it("reports a mismatch for a canary that opens to the wrong value", async () => { // A row from another key id copied over: opens, but says the wrong thing. - const { client } = fakeClient({ + const { check } = fakeClient({ canaries: { v1: encryptUnderKeyId(canaryPlaintext("v2"), "v1") }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome.state).toBe("mismatch"); + expect((await check()).state).toBe("mismatch"); }); it("refuses to seal a wrong key when existing data predates the canary", async () => { const existing = [encrypt("a stored token"), encrypt("another token")]; - useKey(KEY_B); - const { client, inserts } = fakeClient({ + configureKey(KEY_B); + const { inserts, check } = fakeClient({ probe: { - user: { - codexAccessTokenEncrypted: [existing[0]], - codexRefreshTokenEncrypted: [existing[1]], - }, + "User.codexAccessTokenEncrypted": [existing[0]], + "User.codexRefreshTokenEncrypted": [existing[1]], }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); expect(inserts).toEqual([]); }); it("writes the canary when existing data opens under the key", async () => { - const { client, inserts } = fakeClient({ - probe: { - user: { codexAccessTokenEncrypted: [encrypt("a stored token")] }, - }, + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [encrypt("a stored token")] }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome.state).toBe("ok"); + expect((await check()).state).toBe("ok"); expect(inserts).toEqual(["v1"]); }); @@ -148,46 +194,228 @@ describe("encryption key canary", () => { // A row that claims key id v1 but does not open (damaged, or sealed by // something else), found FIRST, then values that open. const damaged = "v1." + Buffer.alloc(40, 7).toString("base64"); - const { client, inserts } = fakeClient({ + const { inserts, check } = fakeClient({ probe: { - user: { - codexAccessTokenEncrypted: [damaged], - codexRefreshTokenEncrypted: [encrypt("a stored token")], - }, + "User.codexAccessTokenEncrypted": [damaged], + "User.codexRefreshTokenEncrypted": [encrypt("a stored token")], }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome.state).toBe("ok"); + expect((await check()).state).toBe("ok"); expect(inserts).toEqual(["v1"]); }); it("several values in one column count, so a wrong key with one busy column still refuses", async () => { const values = [encrypt("one"), encrypt("two"), encrypt("three")]; - useKey(KEY_B); - const { client, inserts } = fakeClient({ - probe: { user: { codexAccessTokenEncrypted: values } }, + configureKey(KEY_B); + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": values }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); expect(inserts).toEqual([]); }); it("a single value that does not open is inconclusive: no mismatch, no canary", async () => { const existing = encrypt("the only stored token"); - useKey(KEY_B); - const { client, inserts } = fakeClient({ - probe: { user: { codexAccessTokenEncrypted: [existing] } }, - }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ - state: "ok", - written: [], - verified: [], - inconclusive: ["v1"], + configureKey(KEY_B); + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [existing] }, }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "single-value" }] }), + ); expect(inserts).toEqual([]); }); + describe("Bytes columns", () => { + it("a wrong key over Bytes-only ciphertext refuses instead of sealing", async () => { + const notes = [encryptToBytes("note one"), encryptToBytes("note two")]; + configureKey(KEY_B); + const { inserts, check } = fakeClient({ + probe: { + "MoodEntry.noteEncrypted": [notes[0]], + "Measurement.notesEncrypted": [notes[1]], + }, + }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); + expect(inserts).toEqual([]); + }); + + it("opens the binary layout with its label, and the UTF-8 layout", async () => { + const route = encryptBytes(Buffer.from("{}"), WORKOUT_ROUTE_GEOMETRY_AAD); + const { inserts, check } = fakeClient({ + probe: { + "WorkoutRoute.geometryEncrypted": [ + { value: route, ts: "2024-01-01" }, + ], + "MoodEntry.noteEncrypted": [ + { value: encryptToBytes("a note"), ts: "2024-01-02" }, + ], + }, + }); + expect(await check()).toEqual(ok({ written: ["v1"] })); + expect(inserts).toEqual(["v1"]); + }); + + it("a binary value under the wrong label does not count as opening", () => { + const sealed = encryptBytes(Buffer.from("x"), "some/other/label"); + const column = ENCRYPTED_COLUMNS.find((c) => c.model === "WorkoutRoute")!; + expect(probeValueOpens(column, { value: sealed, ts: null }, "v1")).toBe( + false, + ); + }); + + it("dispatches a codec column by the row's codec", () => { + const column = ENCRYPTED_COLUMNS.find( + (c) => c.model === "InboundDocument" && c.field === "contentEncrypted", + )!; + const binary = encryptBytes(Buffer.from("%PDF")); + const text = encryptToBytes("JVBERg=="); + expect( + probeValueOpens( + column, + { value: binary, codec: "binary2", ts: null }, + "v1", + ), + ).toBe(true); + expect( + probeValueOpens( + column, + { value: text, codec: "base64v1", ts: null }, + "v1", + ), + ).toBe(true); + }); + }); + + describe("oldest values decide", () => { + function seedOldAThenNewB() { + const old = [ + encrypt("old 1"), + encrypt("old 2"), + encryptToBytes("old note 1"), + encryptToBytes("old note 2"), + ]; + configureKey(KEY_B); + const recent = [ + encrypt("new 1"), + encrypt("new 2"), + encryptToBytes("new note 1"), + encryptToBytes("new note 2"), + ]; + configureKey(KEY_A); + return fakeClient({ + probe: { + "User.codexAccessTokenEncrypted": [ + { value: recent[0], ts: "2026-09-01" }, + { value: old[0], ts: "2024-01-01" }, + ], + "User.codexRefreshTokenEncrypted": [ + { value: old[1], ts: "2024-01-02" }, + { value: recent[1], ts: "2026-09-02" }, + ], + "MoodEntry.noteEncrypted": [ + { value: recent[2], ts: "2026-09-03" }, + { value: old[2], ts: "2024-01-03" }, + ], + "Measurement.notesEncrypted": [ + { value: old[3], ts: "2024-01-04" }, + { value: recent[3], ts: "2026-09-04" }, + ], + }, + }); + } + + it("the key that wrote only the newer rows is never sealed", async () => { + const fake = seedOldAThenNewB(); + configureKey(KEY_B); + const outcome = await fake.check(); + expect(outcome.state).not.toBe("error"); + if (outcome.state === "ok") expect(outcome.written).toEqual([]); + expect(fake.canaries.size).toBe(0); + }); + + it("the key that wrote the oldest rows is sealed, newer unreadable rows notwithstanding", async () => { + const fake = seedOldAThenNewB(); + expect(await fake.check()).toEqual(ok({ written: ["v1"] })); + }); + + it("two oldest values that fail before one that opens are mixed: inconclusive", async () => { + const opens = encrypt("opens"); + configureKey(KEY_B); + const foreign = [encrypt("x"), encrypt("y")]; + configureKey(KEY_A); + const { inserts, check } = fakeClient({ + probe: { + "User.codexAccessTokenEncrypted": [ + { value: foreign[0], ts: "2024-01-01" }, + ], + "User.codexRefreshTokenEncrypted": [ + { value: foreign[1], ts: "2024-01-02" }, + ], + "MoodEntry.noteEncrypted": [{ value: opens, ts: "2025-01-01" }], + }, + }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "mixed" }] }), + ); + expect(inserts).toEqual([]); + }); + }); + + describe("nothing unproven seals", () => { + it("a probe out of time is inconclusive, even over data that opens", async () => { + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [encrypt("opens")] }, + }); + expect(await check(-1)).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "incomplete" }] }), + ); + expect(inserts).toEqual([]); + }); + + it("a column that cannot be read is inconclusive, never a fresh install", async () => { + const { inserts, check } = fakeClient({ + unreadable: ["MoodEntry.noteEncrypted"], + }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "incomplete" }] }), + ); + expect(inserts).toEqual([]); + }); + + it("values too large to fetch are present: inconclusive, not fresh", async () => { + const { inserts, check } = fakeClient({ + probe: { "InboundDocument.contentEncrypted": [{ value: null }] }, + }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "unsampled" }] }), + ); + expect(inserts).toEqual([]); + }); + }); + + describe("the first-seal race", () => { + it("losing it to a process with another key is a mismatch", async () => { + configureKey(KEY_B); + const otherCanary = encryptUnderKeyId(canaryPlaintext("v1"), "v1"); + configureKey(KEY_A); + const { check } = fakeClient({ + race: { keyId: "v1", ciphertext: otherCanary }, + }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); + }); + + it("losing it to a process with the same key verifies", async () => { + const { check } = fakeClient({ + race: { + keyId: "v1", + ciphertext: encryptUnderKeyId(canaryPlaintext("v1"), "v1"), + }, + }); + expect(await check()).toEqual(ok({ verified: ["v1"] })); + }); + }); + it("reports an error, not a mismatch, when the table cannot be read", async () => { const client = { $queryRaw: async () => { @@ -202,22 +430,49 @@ describe("encryption key canary", () => { it("reports an error when no key is configured", async () => { vi.stubEnv("ENCRYPTION_KEY", ""); _resetCryptoCacheForTests(); - const outcome = await checkEncryptionKeyCanaries(fakeClient({}).client); + const outcome = await fakeClient({}).check(); expect(outcome.state).toBe("error"); }); - it("names the key ids and never a key in the log block", () => { + it("names the key ids, the reset statement, and never a key in the log block", () => { const block = keyMismatchLogBlock(["v1"]); expect(block).toContain("'v1'"); expect(block).toContain("encryption.key_mismatch"); + expect(block).toContain( + "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';", + ); expect(block).not.toContain(KEY_A); }); }); +describe("the SQL sampler's column map", () => { + it("resolves every registered column to a table, a column and an order", () => { + // A throwaway client: never connects, only exposes the data model the + // probe reads table and column names from. + const client = new PrismaClient({ + adapter: new PrismaPg({ + connectionString: "postgres://x:y@127.0.0.1:1/x", + }), + }); + for (const column of ENCRYPTED_COLUMNS) { + const loc = resolveColumnLocation(client, column); + expect(loc.table, encryptedColumnKey(column)).toMatch(/^[a-z_0-9]+$/); + expect(loc.column, encryptedColumnKey(column)).toMatch(/^[a-z_0-9]+$/); + if (column.codecField) expect(loc.codec).not.toBeNull(); + } + const backupPieces = ENCRYPTED_COLUMNS.find( + (c) => c.model === "DataBackupChunk", + )!; + expect(resolveColumnLocation(client, backupPieces).ts).toBeNull(); + const user = ENCRYPTED_COLUMNS.find((c) => c.model === "User")!; + expect(resolveColumnLocation(client, user).ts).toBe("updated_at"); + }); +}); + describe("key fingerprint and copy check", () => { beforeEach(() => { vi.unstubAllEnvs(); - useKey(KEY_A); + configureKey(KEY_A); }); it("is the first 12 hex of SHA-256 over the raw key bytes", () => { diff --git a/src/lib/crypto/canary.ts b/src/lib/crypto/canary.ts index e4f77178e..60ee6077f 100644 --- a/src/lib/crypto/canary.ts +++ b/src/lib/crypto/canary.ts @@ -12,44 +12,85 @@ * * - row present: it must decrypt to the expected value, else the key is not * the one the row was written with; - * - row absent: before writing one, sample existing ciphertext under that - * key id from the registered string columns and try it. Without this - * probe, the first boot after an upgrade (or a re-keyed boot over a - * database that predates the canary) would seal the wrong key as the - * right one. + * - row absent: before writing one, probe the ciphertext already stored + * under that key id. Without the probe, the first boot after an upgrade + * (or a re-keyed boot over a database that predates the canary) would + * seal the wrong key as the right one, and from then on refuse the + * right key. * - * The probe samples up to `PROBE_SAMPLE_LIMIT` values, one per column first - * (different columns are different writers, so one bad writer cannot speak - * for the whole database), then further values from columns that had any. - * Its verdict, and what each one does: + * ## The probe * - * - `opens`: at least one value opened. A wrong key opens nothing, so the - * key is right and any value that failed is a damaged or foreign row, not - * a key problem. The canary is written. + * Every registered column is asked, string and Bytes alike (`ENCRYPTED_COLUMNS`), + * for its OLDEST values under the key id: the `encrypt()` string form + * (`.`, or a bare legacy base64 value for the legacy id `v1`), + * the streamed form (`~hlgcm1..`), the same string stored as UTF-8 in a + * Bytes column, and the binary layout (`0x02 `). The answer for a + * column also tells whether it holds ANY value under the id, which is the + * existence check below. + * + * Oldest means the row's `updatedAt`, else its `createdAt`, else (no timestamp + * on the model, e.g. backup pieces) the primary key, ordered after every + * timestamped row. `updatedAt` rather than `createdAt` because a value is + * sealed when its row is written, and a row updated yesterday may carry a + * value sealed yesterday on a row created years ago; `updatedAt` is a ceiling + * on when the value was sealed. The point of the order: a process that served + * under a wrong key without a canary (an inconclusive boot, a version that + * predates this check) wrote rows that open under that wrong key. Those rows + * are by construction newer than the data the database was built with, so + * reading the oldest values first means they are never what decides. + * + * Up to `PROBE_ROWS_PER_COLUMN` values per column are fetched, the merged set + * is ordered by that timestamp, and the oldest `PROBE_SAMPLE_LIMIT` are tried. + * Values larger than `PROBE_VALUE_MAX_BYTES` (stored documents, large backups) + * count as present but are not fetched. The verdict: + * + * - `opens`: the oldest value that opens is preceded by at most ONE older + * value that does not. That one is a damaged or foreign row, not a key + * problem: a wrong key opens nothing. Values newer than the first opening + * one do not count against it; they are what a wrong-key process would + * have left behind. The canary is written. * - `fails`: two or more values were tried and none opened. That is what a * wrong key looks like; the process refuses (mismatch). - * - `inconclusive`: exactly one value was found and it did not open. One - * row cannot tell a wrong key from one damaged row, and the two possible - * mistakes are not equal: declaring a mismatch takes a correctly keyed - * server down with 503 on every request, while serving keeps today's - * behaviour. So the process serves, but writes NO canary either (that - * would seal a key that may be wrong), logs a warning, and probes again - * at the next boot, by which time more data may decide it. - * - `none`: nothing found inside the time budget. The canary is written, - * which is exactly the behaviour before this check existed. + * - `none`: every column answered, and none holds a single value under the + * key id. A genuinely fresh install (or a new id added for rotation); the + * canary is written. + * - `inconclusive`, with a reason: anything not proven. Only one value and + * it did not open (`single-value`); two or more of the oldest values do + * not open but a newer one does (`mixed`); values exist but none could be + * fetched (`unsampled`); the time budget ran out or a column could not be + * read (`incomplete`). The process serves, as before this check existed, + * writes NO canary, logs a warning, and probes again at the next boot. + * Serving rather than refusing because a refusal takes a correctly keyed + * server down with 503 on every request; not sealing because that would + * lock the right key out later. * - * Writes are `ON CONFLICT DO NOTHING`, so two processes racing is fine. + * The first seal can race another process. `ON CONFLICT DO NOTHING` keeps the + * row the other process wrote; when this process's insert affected no row, + * the stored canary is read back and must open under this process's key, or + * the outcome is a mismatch. * * Never logs or returns key material. The canary is written by raw SQL and is - * deliberately not in the rotation registry: it belongs to its key id. + * deliberately not in the rotation registry: it belongs to its key id, and the + * rotation script removes it once no ciphertext remains under that id + * (`retireCanariesWithoutData`). */ import { decrypt, + decryptBytes, + decryptStream, encryptUnderKeyId, extractKeyId, + extractKeyIdFromBytes, + extractStreamKeyId, + getActiveKeyId, getConfiguredKeyIds, + isStreamCiphertext, } from "@/lib/crypto"; -import { ENCRYPTED_COLUMNS } from "@/lib/crypto/encrypted-columns"; +import { + ENCRYPTED_COLUMNS, + encryptedColumnKey, + type EncryptedColumn, +} from "@/lib/crypto/encrypted-columns"; export const CANARY_PREFIX = "healthlog-canary:"; @@ -57,12 +98,21 @@ export function canaryPlaintext(keyId: string): string { return `${CANARY_PREFIX}${keyId}`; } -/** How many existing values the probe tries at most per key id. */ -export const PROBE_SAMPLE_LIMIT = 5; +/** How many of the oldest values the probe tries at most per key id. */ +export const PROBE_SAMPLE_LIMIT = 8; -/** How long the existing-data probe may take before the canary is written anyway. */ +/** How many of its oldest values each column contributes to the pool. */ +export const PROBE_ROWS_PER_COLUMN = 2; + +/** Values larger than this count as present but are not fetched. */ +export const PROBE_VALUE_MAX_BYTES = 2 * 1024 * 1024; + +/** How long the probe may take before it gives up as `inconclusive`. */ export const PROBE_BUDGET_MS = 10_000; +/** The key id the legacy bare-base64 string format belongs to. */ +const LEGACY_KEY_ID = "v1"; + /** The slice of a Prisma client this module needs. */ export interface CanaryClient { $queryRaw( @@ -73,31 +123,53 @@ export interface CanaryClient { query: TemplateStringsArray, ...values: unknown[] ): Promise; + $queryRawUnsafe?( + query: string, + ...values: unknown[] + ): Promise; + $executeRawUnsafe?(query: string, ...values: unknown[]): Promise; } -type ProbeDelegate = { - findMany: (args: { - where: Record; - select: Record; - take: number; - skip?: number; - }) => Promise>>; -}; +/** One stored value under the key id, as the probe sees it. */ +export interface ProbeRow { + /** The stored value; null when it exists but is too large to fetch. */ + value: string | Uint8Array | null; + /** The row's codec column, for a codec-dispatched column. */ + codec?: string | null; + /** The row's write time (see the module comment); null when it has none. */ + ts: Date | null; +} -export type ProbeVerdict = "opens" | "fails" | "inconclusive" | "none"; +/** + * Reads up to `limit` of the oldest values under `keyId` in one column, oldest + * first. Throws when the column cannot be read; an empty array means the + * column holds no value under that id. + */ +export type ColumnSampler = ( + column: EncryptedColumn, + keyId: string, + limit: number, +) => Promise; + +export type InconclusiveReason = + "single-value" | "mixed" | "unsampled" | "incomplete"; + +export type ProbeResult = + | { verdict: "opens" | "fails" | "none" } + | { verdict: "inconclusive"; reason: InconclusiveReason }; export type KeyCheckOutcome = | { state: "ok"; /** Key ids whose canary this run wrote. */ written: string[]; - /** Key ids whose canary existed and opened. */ + /** Key ids whose canary existed (or another process wrote) and opened. */ verified: string[]; /** - * Key ids with no canary whose only existing value did not open: not a - * mismatch, no canary written, probed again at the next boot. + * Key ids with no canary whose stored data proved nothing either way: + * not a mismatch, no canary written, probed again at the next boot. */ - inconclusive: string[]; + inconclusive: Array<{ keyId: string; reason: InconclusiveReason }>; } | { state: "mismatch"; keyIds: string[] } | { state: "error"; message: string }; @@ -112,77 +184,295 @@ function opensAs(ciphertext: string, keyId: string, expected?: string) { } } -/** - * Sample existing values sealed under `keyId` and try them; see the module - * comment for what each verdict means and why. - */ -export async function probeExistingData( - client: unknown, +function isBinaryLayout(column: EncryptedColumn, row: ProbeRow): boolean { + if (column.codec === "binary2") return true; + return column.codecField !== undefined && row.codec === "binary2"; +} + +/** Does one stored value open under `keyId`? Never throws. */ +export function probeValueOpens( + column: EncryptedColumn, + row: ProbeRow, keyId: string, - deadline: number, -): Promise { - const delegates = client as Record; - const columns = ENCRYPTED_COLUMNS.filter( - (c) => c.kind === "string" && !c.codec && !c.codecField, - ); - let tried = 0; - const columnsWithData: typeof columns = []; - - const sample = async ( - column: (typeof columns)[number], - skip: number, - take: number, - ): Promise => { - const delegate = - delegates[column.model.charAt(0).toLowerCase() + column.model.slice(1)]; - if (!delegate || typeof delegate.findMany !== "function") return []; - try { - const rows = await delegate.findMany({ - where: { [column.field]: { startsWith: `${keyId}.` } }, - select: { [column.field]: true }, - take, - ...(skip > 0 ? { skip } : {}), - }); - return rows - .map((row) => row[column.field]) - .filter((v): v is string => typeof v === "string"); - } catch { - return []; +): boolean { + const value = row.value; + if (value === null) return false; + try { + if (typeof value === "string") { + if (isStreamCiphertext(value)) { + if (extractStreamKeyId(value) !== keyId) return false; + decryptStream(value); + return true; + } + const id = extractKeyId(value); + if (id === null) { + // Legacy bare base64: only ever the legacy id's. + if (keyId !== LEGACY_KEY_ID) return false; + decrypt(value); + return true; + } + return opensAs(value, keyId); } - }; - - // Pass 1: one value per column, so different writers are heard first. - for (const column of columns) { - if (tried >= PROBE_SAMPLE_LIMIT || Date.now() > deadline) break; - const values = await sample(column, 0, 1); - if (values.length === 0) continue; - columnsWithData.push(column); - tried += 1; - if (opensAs(values[0], keyId)) return "opens"; - } - // Pass 2: more values from the columns that had any, up to the limit. - for (const column of columnsWithData) { - if (tried >= PROBE_SAMPLE_LIMIT || Date.now() > deadline) break; - const values = await sample(column, 1, PROBE_SAMPLE_LIMIT - tried); - for (const value of values) { - tried += 1; - if (opensAs(value, keyId)) return "opens"; + const buf = Buffer.from(value.buffer, value.byteOffset, value.byteLength); + if (isBinaryLayout(column, row)) { + if (extractKeyIdFromBytes(buf) !== keyId) return false; + decryptBytes(buf, column.aad); + return true; } + return opensAs(buf.toString("utf8"), keyId); + } catch { + return false; } - - if (tried >= 2) return "fails"; - if (tried === 1) return "inconclusive"; - return "none"; } +/** + * Probe the stored values under `keyId`; see the module comment for what each + * verdict means and why. + */ +export async function probeExistingData( + sampler: ColumnSampler, + keyId: string, + deadline: number, + columns: readonly EncryptedColumn[] = ENCRYPTED_COLUMNS, +): Promise { + const pool: Array<{ column: EncryptedColumn; row: ProbeRow; order: number }> = + []; + let present = 0; + for (const column of columns) { + if (Date.now() > deadline) { + return { verdict: "inconclusive", reason: "incomplete" }; + } + let rows: ProbeRow[]; + try { + rows = await sampler(column, keyId, PROBE_ROWS_PER_COLUMN); + } catch { + return { verdict: "inconclusive", reason: "incomplete" }; + } + present += rows.length; + for (const row of rows) { + if (row.value !== null) pool.push({ column, row, order: pool.length }); + } + } + if (Date.now() > deadline) { + return { verdict: "inconclusive", reason: "incomplete" }; + } + if (present === 0) return { verdict: "none" }; + if (pool.length === 0) { + return { verdict: "inconclusive", reason: "unsampled" }; + } + + // Oldest first; rows without a timestamp after every row with one. + pool.sort((a, b) => { + const ta = a.row.ts?.getTime() ?? Number.POSITIVE_INFINITY; + const tb = b.row.ts?.getTime() ?? Number.POSITIVE_INFINITY; + return ta === tb ? a.order - b.order : ta - tb; + }); + const tried = pool.slice(0, PROBE_SAMPLE_LIMIT); + + let failedBeforeFirstOpen = 0; + for (const { column, row } of tried) { + if (probeValueOpens(column, row, keyId)) { + return failedBeforeFirstOpen <= 1 + ? { verdict: "opens" } + : { verdict: "inconclusive", reason: "mixed" }; + } + failedBeforeFirstOpen += 1; + } + return tried.length >= 2 + ? { verdict: "fails" } + : { verdict: "inconclusive", reason: "single-value" }; +} + +// ─── The SQL sampler ──────────────────────────────────────────────────────── + +interface ColumnLocation { + table: string; + column: string; + pk: string; + ts: string | null; + codec: string | null; +} + +type RuntimeDataModel = { + models: Record< + string, + { + dbName: string | null; + fields: Array<{ name: string; dbName?: string | null }>; + } + >; +}; + +const IDENTIFIER = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/; +const KEY_ID = /^[A-Za-z0-9_-]{1,32}$/; + +function quoteIdentifier(name: string): string { + if (!IDENTIFIER.test(name)) { + throw new Error(`Unexpected identifier in the column registry: ${name}`); + } + return `"${name}"`; +} + +/** + * Table and column names for a registry entry, read from the Prisma client's + * runtime data model (the same `@@map` / `@map` names the client queries + * with). `canary.test.ts` pins that every registry column resolves, so a + * Prisma upgrade that moves this fails a test rather than the probe. + */ +export function resolveColumnLocation( + client: unknown, + column: EncryptedColumn, +): ColumnLocation { + const dm = (client as { _runtimeDataModel?: RuntimeDataModel }) + ._runtimeDataModel; + const model = dm?.models?.[column.model]; + if (!model) { + throw new Error(`No data model for ${column.model}`); + } + const dbField = (name: string): string | null => { + const f = model.fields.find((x) => x.name === name); + return f ? (f.dbName ?? f.name) : null; + }; + const columnName = dbField(column.field); + const pk = dbField(column.pkField ?? "id"); + if (!columnName || !pk) { + throw new Error(`No column for ${encryptedColumnKey(column)}`); + } + return { + table: model.dbName ?? column.model, + column: columnName, + pk, + ts: dbField("updatedAt") ?? dbField("createdAt"), + codec: column.codecField ? dbField(column.codecField) : null, + }; +} + +/** + * The SQL that finds values under `keyId` in one column. Identifiers come from + * the Prisma data model and are checked against `IDENTIFIER` before they are + * spliced; the key id is checked against the key-id grammar and travels as a + * bound parameter. The prefix tests use `substr` / `substring`, which read + * only the head of a large out-of-line value instead of the whole of it. + */ +function columnQuery( + loc: ColumnLocation, + column: EncryptedColumn, + keyId: string, + mode: "sample" | "exists", +): { sql: string; params: unknown[] } { + if (!KEY_ID.test(keyId)) throw new Error("Malformed key id"); + const c = quoteIdentifier(loc.column); + const params: unknown[] = []; + const bind = (v: unknown) => { + params.push(v); + return `$${params.length}`; + }; + const tests: string[] = []; + if (column.kind === "string") { + const versioned = `${keyId}.`; + const streamed = `~hlgcm1.${keyId}.`; + tests.push( + `substr(${c}, 1, ${versioned.length}) = ${bind(versioned)}`, + `substr(${c}, 1, ${streamed.length}) = ${bind(streamed)}`, + ); + if (keyId === LEGACY_KEY_ID) { + // A legacy value is bare base64 of at least IV + tag + one block. + tests.push(`substr(${c}, 1, 40) ~ '^[A-Za-z0-9+/]{40}$'`); + } + } else { + const utf8 = Buffer.from(`${keyId}.`, "ascii"); + const id = Buffer.from(keyId, "ascii"); + const binary = Buffer.concat([Buffer.from([0x02, id.length]), id]); + if (column.codec !== "binary2") { + tests.push( + `substring(${c} from 1 for ${utf8.length}) = ${bind(utf8)}::bytea`, + ); + } + if (column.codec === "binary2" || column.codecField) { + tests.push( + `substring(${c} from 1 for ${binary.length}) = ${bind(binary)}::bytea`, + ); + } + } + const table = quoteIdentifier(loc.table); + const where = `${c} IS NOT NULL AND (${tests.join(" OR ")})`; + if (mode === "exists") { + return { + sql: `SELECT 1 AS one FROM ${table} WHERE ${where} LIMIT 1`, + params, + }; + } + const ts = loc.ts ? quoteIdentifier(loc.ts) : null; + const codec = loc.codec ? quoteIdentifier(loc.codec) : "NULL"; + const max = Math.floor(PROBE_VALUE_MAX_BYTES); + const sql = + `SELECT CASE WHEN octet_length(${c}) <= ${max} THEN ${c} END AS value, ` + + `${codec} AS codec, ${ts ?? "NULL::timestamp"} AS ts ` + + `FROM ${table} WHERE ${where} ` + + `ORDER BY ${ts ? `${ts} ASC NULLS LAST, ` : ""}${quoteIdentifier(loc.pk)} ASC ` + + `LIMIT ${bind(0)}`; + return { sql, params }; +} + +function requireUnsafe(client: CanaryClient) { + const query = client.$queryRawUnsafe; + if (typeof query !== "function") { + throw new Error("The database client cannot run the probe queries"); + } + return query.bind(client); +} + +/** The production sampler: one bounded query per column. */ +export function sqlColumnSampler(client: CanaryClient): ColumnSampler { + const query = requireUnsafe(client); + return async (column, keyId, limit) => { + const loc = resolveColumnLocation(client, column); + const { sql, params } = columnQuery(loc, column, keyId, "sample"); + params[params.length - 1] = limit; + const rows = await query< + Array<{ + value: string | Uint8Array | null; + codec: string | null; + ts: Date | null; + }> + >(sql, ...params); + return rows.map((r) => ({ + value: r.value, + codec: r.codec, + ts: r.ts instanceof Date ? r.ts : r.ts ? new Date(r.ts) : null, + })); + }; +} + +/** + * Every registered column that still holds a value under `keyId`, by + * `Model.field`. Exhaustive: one existence query per column, no budget. + */ +export async function columnsHoldingKeyId( + client: CanaryClient, + keyId: string, +): Promise { + const query = requireUnsafe(client); + const holding: string[] = []; + for (const column of ENCRYPTED_COLUMNS) { + const loc = resolveColumnLocation(client, column); + const { sql, params } = columnQuery(loc, column, keyId, "exists"); + const rows = await query(sql, ...params); + if (rows.length > 0) holding.push(encryptedColumnKey(column)); + } + return holding; +} + +// ─── The check ────────────────────────────────────────────────────────────── + /** * Run the check for every configured key id. Never throws: a failure to run - * the check at all is reported as `error`, and only a value that provably does - * not open is reported as `mismatch`. + * the check at all is reported as `error`, and only values that provably do + * not open are reported as `mismatch`. */ export async function checkEncryptionKeyCanaries( client: CanaryClient, - options: { probeBudgetMs?: number } = {}, + options: { probeBudgetMs?: number; sampler?: ColumnSampler } = {}, ): Promise { let keyIds: string[]; try { @@ -217,12 +507,21 @@ export async function checkEncryptionKeyCanaries( const deadline = Date.now() + (options.probeBudgetMs ?? PROBE_BUDGET_MS); const toWrite: string[] = []; - const inconclusive: string[] = []; - for (const keyId of missing) { - const probe = await probeExistingData(client, keyId, deadline); - if (probe === "fails") mismatched.push(keyId); - else if (probe === "inconclusive") inconclusive.push(keyId); - else toWrite.push(keyId); + const inconclusive: Array<{ keyId: string; reason: InconclusiveReason }> = []; + if (missing.length > 0) { + let sampler: ColumnSampler; + try { + sampler = options.sampler ?? sqlColumnSampler(client); + } catch (err) { + return { state: "error", message: (err as Error).message }; + } + for (const keyId of missing) { + const probe = await probeExistingData(sampler, keyId, deadline); + if (probe.verdict === "fails") mismatched.push(keyId); + else if (probe.verdict === "inconclusive") { + inconclusive.push({ keyId, reason: probe.reason }); + } else toWrite.push(keyId); + } } if (mismatched.length > 0) { @@ -231,16 +530,61 @@ export async function checkEncryptionKeyCanaries( return { state: "mismatch", keyIds: mismatched.sort() }; } + const written: string[] = []; try { for (const keyId of toWrite) { const ciphertext = encryptUnderKeyId(canaryPlaintext(keyId), keyId); - await client.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES (${keyId}, ${ciphertext}) ON CONFLICT (key_id) DO NOTHING`; + const inserted = + await client.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES (${keyId}, ${ciphertext}) ON CONFLICT (key_id) DO NOTHING`; + if (inserted > 0) { + written.push(keyId); + continue; + } + // Another process sealed this id first. Its canary has to open under + // this process's key, or one of the two holds the wrong key. + const [winner] = await client.$queryRaw>` + SELECT ciphertext FROM encryption_key_canaries WHERE key_id = ${keyId}`; + if (winner && opensAs(winner.ciphertext, keyId, canaryPlaintext(keyId))) { + verified.push(keyId); + } else { + mismatched.push(keyId); + } } } catch (err) { return { state: "error", message: (err as Error).message }; } + if (mismatched.length > 0) { + return { state: "mismatch", keyIds: mismatched.sort() }; + } - return { state: "ok", written: toWrite, verified, inconclusive }; + return { state: "ok", written, verified, inconclusive }; +} + +/** + * After a rotation: remove the canary of every configured key id other than + * the active one that no longer holds a single value in any registered + * column. The id's key can then leave `ENCRYPTION_KEYS`; and if the id is + * ever reused for a different key, no stale canary refuses it. + */ +export async function retireCanariesWithoutData(client: CanaryClient): Promise<{ + removed: string[]; + remaining: Record; +}> { + const active = getActiveKeyId(); + const removed: string[] = []; + const remaining: Record = {}; + for (const keyId of getConfiguredKeyIds()) { + if (keyId === active) continue; + const holding = await columnsHoldingKeyId(client, keyId); + if (holding.length > 0) { + remaining[keyId] = holding; + continue; + } + const deleted = + await client.$executeRaw`DELETE FROM encryption_key_canaries WHERE key_id = ${keyId}`; + if (deleted > 0) removed.push(keyId); + } + return { removed, remaining }; } /** @@ -266,6 +610,13 @@ export function keyMismatchLogBlock(keyIds: string[]): string { " 3. Start fresh: an empty database with this key. Stored data and", " in-database backups of the old database cannot be read without", " the old key.", + "", + " Only if you are CERTAIN the configured key is the one this data was", + " written with (the check recorded a different key earlier), reset the", + " check and restart; it then probes the stored data again:", + ...keyIds.map( + (id) => ` DELETE FROM encryption_key_canaries WHERE key_id = '${id}';`, + ), "============================================================", ].join("\n"); } diff --git a/src/lib/jobs/encryption-key-rotate.ts b/src/lib/jobs/encryption-key-rotate.ts index cd9c93eb8..cf1c52250 100644 --- a/src/lib/jobs/encryption-key-rotate.ts +++ b/src/lib/jobs/encryption-key-rotate.ts @@ -32,6 +32,10 @@ import { type CorpusClient, } from "@/lib/crypto/encryption-corpus"; import { getWorkerPrisma } from "@/lib/jobs/reminder/shared"; +import { + retireCanariesWithoutData, + type CanaryClient, +} from "@/lib/crypto/canary"; export const ENCRYPTION_KEY_ROTATE_QUEUE = "encryption-key-rotate"; export const ENCRYPTION_KEY_ROTATE_CONCURRENCY = 1; @@ -62,9 +66,20 @@ export async function runEncryptionKeyRotation( totalErrors: number; totalDropped: number; stoppedEarly: boolean; + /** Key ids whose boot key-check record this run removed. */ + retiredKeyIds: string[]; }> { const prisma = getWorkerPrisma(); const out = await rotateCorpus(prisma as unknown as CorpusClient, shouldStop); + // A complete, error-free pass: remove the boot key check's record of every + // other key id that no longer holds a value in any registered column, so + // its key can leave ENCRYPTION_KEYS and the id can never refuse a later, + // different key. The existence check walks every column itself. + const retiredKeyIds = + !out.stoppedEarly && out.totalErrors === 0 + ? (await retireCanariesWithoutData(prisma as unknown as CanaryClient)) + .removed + : []; return { activeKeyId: out.activeKeyId, totalScanned: out.totalScanned, @@ -72,6 +87,7 @@ export async function runEncryptionKeyRotation( totalErrors: out.totalErrors, totalDropped: out.totalDropped, stoppedEarly: out.stoppedEarly, + retiredKeyIds, }; } @@ -98,6 +114,7 @@ export async function handleEncryptionKeyRotate( rotated: result.totalRotated, errors: result.totalErrors, dropped: result.totalDropped, + retiredKeyIds: result.retiredKeyIds, }, }); // Per-row errors are the fail-closed skip of a row under a key the diff --git a/tests/integration/encryption-key-check.test.ts b/tests/integration/encryption-key-check.test.ts index 926221f86..01f2796af 100644 --- a/tests/integration/encryption-key-check.test.ts +++ b/tests/integration/encryption-key-check.test.ts @@ -47,9 +47,17 @@ vi.mock("@/lib/db-compat", () => ({ import { _resetCryptoCacheForTests, encrypt, + encryptUnderKeyId, getKeyFingerprint, } from "@/lib/crypto"; +import { encryptToBytes } from "@/lib/ai/coach/bytes-codec"; import { runBootKeyCheck } from "@/lib/boot/key-check"; +import { + canaryPlaintext, + checkEncryptionKeyCanaries, + retireCanariesWithoutData, + type CanaryClient, +} from "@/lib/crypto/canary"; import { isKeyMismatch, setKeyMismatchState, @@ -223,6 +231,214 @@ describe("boot key check probe over existing data (real Postgres)", () => { }); }); +/** Two users, one oldest, with ciphertext in string AND Bytes columns. */ +async function seedUser( + name: string, + writtenAt: Date, + seal: () => { text: string; bytes: Uint8Array }, +) { + const prisma = getPrismaClient(); + const a = seal(); + const b = seal(); + const user = await prisma.user.create({ + data: { + username: name, + email: `${name}@example.test`, + codexAccessTokenEncrypted: a.text, + codexRefreshTokenEncrypted: b.text, + createdAt: writtenAt, + updatedAt: writtenAt, + }, + }); + await prisma.userHealthProfile.create({ + data: { + userId: user.id, + aboutMeEncrypted: a.bytes, + conditionsEncrypted: b.bytes, + createdAt: writtenAt, + updatedAt: writtenAt, + }, + }); + return user.id; +} + +const sealNow = () => ({ + text: encrypt("a stored value"), + bytes: encryptToBytes("a stored note"), +}); + +describe("boot key check: what decides, and what may seal (real Postgres)", () => { + it("probes Bytes columns: an install whose ciphertext is only Bytes does not seal a wrong key", async () => { + const prisma = getPrismaClient(); + const user = await prisma.user.create({ + data: { username: "bytes-only", email: "bytes-only@example.test" }, + }); + await prisma.userHealthProfile.create({ + data: { + userId: user.id, + aboutMeEncrypted: encryptToBytes("about me"), + conditionsEncrypted: encryptToBytes("conditions"), + allergiesEncrypted: encryptToBytes("allergies"), + }, + }); + + useKey(KEY_B); + const errors = vi.spyOn(console, "error").mockImplementation(() => {}); + expect(await runBootKeyCheck(prisma)).toBe(true); + errors.mockRestore(); + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + + useKey(KEY_A); + expect(await runBootKeyCheck(prisma)).toBe(false); + expect(await prisma.encryptionKeyCanary.count()).toBe(1); + }); + + it("a probe that runs out of time seals nothing and keeps serving", async () => { + const prisma = getPrismaClient(); + await seedUser("slow", new Date("2024-01-01T00:00:00Z"), sealNow); + const outcome = await checkEncryptionKeyCanaries(prisma, { + probeBudgetMs: -1, + }); + expect(outcome.state).toBe("ok"); + if (outcome.state !== "ok") return; + expect(outcome.written).toEqual([]); + expect(outcome.inconclusive.map((i) => i.keyId)).toEqual(["v1"]); + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + }); + + it("a fresh database with no ciphertext under the key id seals it", async () => { + const prisma = getPrismaClient(); + // Ciphertext exists, but under another key id: not evidence about v1. + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_A, v2: KEY_B })); + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v2"); + vi.stubEnv("ENCRYPTION_KEY", ""); + _resetCryptoCacheForTests(); + await seedUser("v2-only", new Date("2024-01-01T00:00:00Z"), sealNow); + expect(await runBootKeyCheck(prisma)).toBe(false); + const ids = (await prisma.encryptionKeyCanary.findMany()) + .map((r) => r.keyId) + .sort(); + expect(ids).toEqual(["v1", "v2"]); + }); + + it("rows a wrong key wrote recently never decide: old rows under A, newer under B", async () => { + const prisma = getPrismaClient(); + // The original data, under A. + await seedUser("original", new Date("2024-01-01T00:00:00Z"), sealNow); + await seedUser("original-2", new Date("2024-02-01T00:00:00Z"), sealNow); + // A process holding B served for a while (no canary yet) and wrote more. + useKey(KEY_B); + for (let i = 0; i < 6; i++) { + await seedUser( + `recent-${i}`, + new Date(Date.UTC(2026, 8, 1 + i)), + sealNow, + ); + } + + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + const errors = vi.spyOn(console, "error").mockImplementation(() => {}); + await runBootKeyCheck(prisma); + warn.mockRestore(); + errors.mockRestore(); + // B is not sealed as the right key, whatever else the verdict is. + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + const outcome = await checkEncryptionKeyCanaries(prisma); + expect( + outcome.state === "mismatch" || + (outcome.state === "ok" && outcome.written.length === 0), + ).toBe(true); + + // The key the oldest data was written with is the right one. + useKey(KEY_A); + setKeyMismatchState(null); + expect(await runBootKeyCheck(prisma)).toBe(false); + expect(isKeyMismatch()).toBe(false); + expect(await prisma.encryptionKeyCanary.count()).toBe(1); + }); + + it("losing the first-seal race to another key is a mismatch, not ok", async () => { + const prisma = getPrismaClient(); + // The other process, holding A, wins the insert between our probe and + // our write. + const sealedUnderA = encryptUnderKeyId(canaryPlaintext("v1"), "v1"); + useKey(KEY_B); + let raced = false; + const racing = new Proxy(prisma, { + get(target, prop, receiver) { + if (prop === "$executeRaw") { + return async (q: TemplateStringsArray, ...values: unknown[]) => { + if (!raced) { + raced = true; + await target.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES ('v1', ${sealedUnderA})`; + } + return target.$executeRaw(q, ...values); + }; + } + const value = Reflect.get(target, prop, receiver); + return typeof value === "function" ? value.bind(target) : value; + }, + }) as unknown as CanaryClient; + const outcome = await checkEncryptionKeyCanaries(racing); + expect(raced).toBe(true); + expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + + // The same race lost to a process holding the SAME key is fine. + await prisma.$executeRaw`DELETE FROM encryption_key_canaries`; + raced = false; + useKey(KEY_A); + const same = await checkEncryptionKeyCanaries(racing); + expect(same.state).toBe("ok"); + }); +}); + +describe("retiring a key id after rotation (real Postgres)", () => { + it("removes the canary of a key id only once no ciphertext remains under it", async () => { + const prisma = getPrismaClient(); + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_A, v2: KEY_B })); + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v1"); + vi.stubEnv("ENCRYPTION_KEY", ""); + _resetCryptoCacheForTests(); + const userId = await seedUser( + "rotating", + new Date("2024-01-01T00:00:00Z"), + sealNow, + ); + expect(await runBootKeyCheck(prisma)).toBe(false); + expect(await prisma.encryptionKeyCanary.count()).toBe(2); + + // Rotation to v2 has not finished: one Bytes value still under v1. + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v2"); + _resetCryptoCacheForTests(); + await prisma.user.update({ + where: { id: userId }, + data: { + codexAccessTokenEncrypted: encrypt("rotated"), + codexRefreshTokenEncrypted: encrypt("rotated"), + }, + }); + await prisma.userHealthProfile.update({ + where: { userId }, + data: { aboutMeEncrypted: encryptToBytes("rotated") }, + }); + const partial = await retireCanariesWithoutData(prisma); + expect(partial.removed).toEqual([]); + expect(partial.remaining).toEqual({ + v1: ["UserHealthProfile.conditionsEncrypted"], + }); + expect(await prisma.encryptionKeyCanary.count()).toBe(2); + + await prisma.userHealthProfile.update({ + where: { userId }, + data: { conditionsEncrypted: encryptToBytes("rotated") }, + }); + const done = await retireCanariesWithoutData(prisma); + expect(done.removed).toEqual(["v1"]); + const left = await prisma.encryptionKeyCanary.findMany(); + expect(left.map((r) => r.keyId)).toEqual(["v2"]); + }); +}); + describe("encryption key backup step (real Postgres)", () => { it("is due, refuses a stale confirmation, records the right one, and is due again after a re-key", async () => { const prisma = getPrismaClient(); From 4783711651c90452267ff61af864cb12feaf65a0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc-Andr=C3=A9=20Bombeck?= Date: Sat, 3 Oct 2026 23:32:07 +0200 Subject: [PATCH 2/3] Refuse MCP and its OAuth endpoints while the encryption key does not match apiHandler answers 503 encryption.key_mismatch before auth, but the remote MCP endpoint and the OAuth authorize, token and register routes wrap in withBackgroundEvent instead and never checked. An MCP write tool could seal new rows under the wrong key, and those rows would open under it at the next boot. The refusal moves into one helper used by apiHandler, /mcp (same envelope) and the OAuth routes (RFC 6749 temporarily_unavailable with the error code as an extension member). A structural test fails when a route outside apiHandler, or a server action, lacks the refusal in every exported handler and is not on a short allowlist of routes that never touch the database. --- .../key-mismatch-refusal-guard.test.ts | 127 ++++++++++++++++++ src/app/api/mcp/oauth/authorize/route.ts | 6 + src/app/api/mcp/oauth/register/route.ts | 5 + src/app/api/mcp/oauth/token/route.ts | 5 + src/app/mcp/route.ts | 7 + src/lib/api-handler.ts | 20 +-- src/lib/boot/key-mismatch-refusal.ts | 67 +++++++++ .../integration/encryption-key-check.test.ts | 64 +++++++++ 8 files changed, 285 insertions(+), 16 deletions(-) create mode 100644 src/__tests__/key-mismatch-refusal-guard.test.ts create mode 100644 src/lib/boot/key-mismatch-refusal.ts diff --git a/src/__tests__/key-mismatch-refusal-guard.test.ts b/src/__tests__/key-mismatch-refusal-guard.test.ts new file mode 100644 index 000000000..9d145e319 --- /dev/null +++ b/src/__tests__/key-mismatch-refusal-guard.test.ts @@ -0,0 +1,127 @@ +/** + * Every request surface outside `apiHandler` refuses while the boot key check + * has found a mismatch. + * + * `apiHandler` answers 503 `encryption.key_mismatch` before auth and before + * the handler, so a process holding the wrong key never writes through it. + * A route that does not wrap in `apiHandler` gets none of that: the remote + * MCP endpoint and its OAuth bridge went without it, and an MCP write tool + * would have sealed new rows under the wrong key, rows the next boot's probe + * could then read as evidence that the wrong key is right. + * + * So: a `route.ts` under `src/app` that does not wrap in `apiHandler` must + * call `refuseOnKeyMismatch` / `refuseOAuthOnKeyMismatch`, or sit in the + * allowlist below with the reason it cannot touch encrypted data. A server + * action (`"use server"`) is held to the same rule; there are none today. + */ +import { describe, expect, it } from "vitest"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +import { walkSourceFiles } from "./helpers/source-files"; + +const SRC = join(process.cwd(), "src"); + +/** + * Routes outside `apiHandler` that need no refusal, each with why. A file + * belongs here only if it reads nothing from the database and writes nothing. + */ +const NO_DATA_ROUTES: Record = { + "app/.well-known/apple-app-site-association/route.ts": + "static JSON built from environment variables; no database access", + "app/.well-known/oauth-authorization-server/route.ts": + "RFC 8414 metadata built from the configured origin; no database access", + "app/.well-known/oauth-protected-resource/route.ts": + "RFC 9728 metadata built from the configured origin; no database access", +}; + +const REFUSAL_CALL = /\brefuse(?:OAuth)?OnKeyMismatch\s*\(/; + +function sourceFiles(): string[] { + return walkSourceFiles(SRC, { floor: 3000 }) + .filter((p) => !p.includes("__tests__")) + .filter((p) => !p.endsWith(".test.ts") && !p.endsWith(".test.tsx")) + .sort(); +} + +/** The text of a top-level function starting at `from`, up to its closing brace. */ +function functionBody(src: string, from: number): string { + const end = src.indexOf("\n}\n", from); + return src.slice(from, end === -1 ? undefined : end); +} + +function read(rel: string): string { + return readFileSync(join(SRC, rel), "utf8"); +} + +describe("key mismatch refusal outside apiHandler", () => { + const routes = sourceFiles().filter( + (p) => p.startsWith("app/") && /\/route\.tsx?$/.test(p), + ); + const outside = routes.filter((p) => !/\bapiHandler\s*\(/.test(read(p))); + + it("finds the route tree and the routes outside apiHandler", () => { + // A matcher that matches nothing would pass every check below. + expect(routes.length).toBeGreaterThan(300); + expect(outside).toEqual( + expect.arrayContaining([ + "app/mcp/route.ts", + "app/api/mcp/oauth/authorize/route.ts", + "app/api/mcp/oauth/register/route.ts", + "app/api/mcp/oauth/token/route.ts", + ]), + ); + }); + + it("every route outside apiHandler refuses, or is allowlisted with a reason", () => { + const missing = outside.filter( + (p) => !(p in NO_DATA_ROUTES) && !REFUSAL_CALL.test(read(p)), + ); + expect(missing).toEqual([]); + }); + + it("an allowlisted route really stays away from the database", () => { + for (const p of Object.keys(NO_DATA_ROUTES)) { + expect(outside, `${p} is no longer outside apiHandler`).toContain(p); + const src = read(p); + expect(src, p).not.toMatch(/@\/lib\/db\b|prisma|\$queryRaw/); + } + }); + + it("every exported handler of a refusing route calls the refusal itself", () => { + // `export async function POST(...) { ... }` must refuse in its own body; + // `export const POST = handleX` must refuse in `handleX`. A refusal in one + // handler of a file proves nothing about its siblings. + let checked = 0; + for (const p of outside.filter((f) => !(f in NO_DATA_ROUTES))) { + const src = read(p); + const bodies: Array<[string, string]> = []; + for (const m of src.matchAll( + /export\s+(?:async\s+)?function\s+(GET|POST|PUT|PATCH|DELETE)\b/g, + )) { + bodies.push([m[1], functionBody(src, m.index)]); + } + for (const m of src.matchAll( + /export\s+const\s+(GET|POST|PUT|PATCH|DELETE)\s*=\s*(\w+)\s*;/g, + )) { + const def = new RegExp(`function\\s+${m[2]}\\s*\\(`).exec(src); + expect(def, `${p}: ${m[1]} → ${m[2]} not found`).not.toBeNull(); + bodies.push([m[1], functionBody(src, def!.index)]); + } + expect(bodies.length, p).toBeGreaterThan(0); + for (const [name, body] of bodies) { + expect(REFUSAL_CALL.test(body), `${p} ${name}`).toBe(true); + checked += 1; + } + } + expect(checked).toBeGreaterThanOrEqual(6); + }); + + it("server actions refuse too", () => { + const actions = sourceFiles().filter((p) => + /^\s*["']use server["']/m.test(read(p)), + ); + const missing = actions.filter((p) => !REFUSAL_CALL.test(read(p))); + expect(missing).toEqual([]); + }); +}); diff --git a/src/app/api/mcp/oauth/authorize/route.ts b/src/app/api/mcp/oauth/authorize/route.ts index b0ad1ef5e..c8a71d9a8 100644 --- a/src/app/api/mcp/oauth/authorize/route.ts +++ b/src/app/api/mcp/oauth/authorize/route.ts @@ -29,6 +29,7 @@ import { NextRequest } from "next/server"; import { z } from "zod/v4"; import { annotate } from "@/lib/logging/context"; +import { refuseOAuthOnKeyMismatch } from "@/lib/boot/key-mismatch-refusal"; import { withBackgroundEvent } from "@/lib/logging/background"; import { auditLog } from "@/lib/auth/audit"; import { getSession } from "@/lib/auth/session"; @@ -344,10 +345,15 @@ function ensureAuthorizeCsp(response: Response): Response { } export async function GET(request: NextRequest): Promise { + // The boot key check refused this process: no grant is issued or stored. + const refusal = refuseOAuthOnKeyMismatch(); + if (refusal) return ensureAuthorizeCsp(refusal); return ensureAuthorizeCsp(await authorizeGet(request)); } export async function POST(request: NextRequest): Promise { + const refusal = refuseOAuthOnKeyMismatch(); + if (refusal) return ensureAuthorizeCsp(refusal); return ensureAuthorizeCsp(await authorizePost(request)); } diff --git a/src/app/api/mcp/oauth/register/route.ts b/src/app/api/mcp/oauth/register/route.ts index aa5d76a83..7e9e1d13c 100644 --- a/src/app/api/mcp/oauth/register/route.ts +++ b/src/app/api/mcp/oauth/register/route.ts @@ -16,6 +16,7 @@ import { NextRequest } from "next/server"; import { z } from "zod/v4"; import { annotate } from "@/lib/logging/context"; +import { refuseOAuthOnKeyMismatch } from "@/lib/boot/key-mismatch-refusal"; import { withBackgroundEvent } from "@/lib/logging/background"; import { checkAuthSurfaceRateLimit, rateLimitHeaders } from "@/lib/rate-limit"; import { isApiGloballyEnabled } from "@/lib/app-settings"; @@ -68,6 +69,10 @@ function oauthError( export async function POST(request: NextRequest): Promise { return withBackgroundEvent("mcp.oauth.register", async () => { + // The boot key check refused this process: mint nothing, write nothing. + const refusal = refuseOAuthOnKeyMismatch(); + if (refusal) return refusal; + const rl = await checkAuthSurfaceRateLimit( request, "mcp:oauth:register", diff --git a/src/app/api/mcp/oauth/token/route.ts b/src/app/api/mcp/oauth/token/route.ts index a82b732a5..fbdaf01af 100644 --- a/src/app/api/mcp/oauth/token/route.ts +++ b/src/app/api/mcp/oauth/token/route.ts @@ -32,6 +32,7 @@ import { NextRequest } from "next/server"; import type { Prisma } from "@/generated/prisma/client"; import { prisma } from "@/lib/db"; import { annotate } from "@/lib/logging/context"; +import { refuseOAuthOnKeyMismatch } from "@/lib/boot/key-mismatch-refusal"; import { withBackgroundEvent } from "@/lib/logging/background"; import { auditLog } from "@/lib/auth/audit"; import { issueApiToken } from "@/lib/auth/issue-token"; @@ -201,6 +202,10 @@ async function finalizeTokenPair( export async function POST(request: NextRequest): Promise { return withBackgroundEvent("mcp.oauth.token", async () => { + // The boot key check refused this process: mint nothing, write nothing. + const refusal = refuseOAuthOnKeyMismatch(); + if (refusal) return refusal; + // M1 — fail closed without a pinned origin; M4 — honour the operator's // global API kill-switch. Either off and the bridge mints no tokens. if (!isMcpOriginConfigured() || !(await isApiGloballyEnabled())) { diff --git a/src/app/mcp/route.ts b/src/app/mcp/route.ts index c71635dc8..30b4b1738 100644 --- a/src/app/mcp/route.ts +++ b/src/app/mcp/route.ts @@ -50,6 +50,7 @@ import { resolveMcpAuthContext, SCOPE_HEALTH_READ, } from "@/lib/mcp"; +import { refuseOnKeyMismatch } from "@/lib/boot/key-mismatch-refusal"; import { withBackgroundEvent } from "@/lib/logging/background"; import { annotate } from "@/lib/logging/context"; import { isModuleEnabled } from "@/lib/modules/gate"; @@ -135,6 +136,12 @@ function readBearer(request: Request): string | null { async function handleMcp(request: Request): Promise { return withBackgroundEvent("mcp.request", async () => { + // 00. KEY CHECK — the boot key check found that the configured encryption + // key does not open this database. The write tools would seal new + // rows under it, so refuse before anything else, as apiHandler does. + const refusal = refuseOnKeyMismatch(); + if (refusal) return refusal; + // 0. ORIGIN — refuse a cross-origin browser request before anything else. if (!originAllowed(request)) { annotate({ action: { name: "mcp.origin.rejected" } }); diff --git a/src/lib/api-handler.ts b/src/lib/api-handler.ts index a47b66ab2..4f00f4b9e 100644 --- a/src/lib/api-handler.ts +++ b/src/lib/api-handler.ts @@ -53,11 +53,7 @@ import { assertRecordSessionFence, attachRecordContextEcho, } from "./sharing/record-session-fence"; -import { - isKeyMismatch, - KEY_MISMATCH_ERROR_CODE, - KEY_MISMATCH_EXEMPT_PATHS, -} from "./boot/key-mismatch-state"; +import { refuseOnKeyMismatch } from "./boot/key-mismatch-refusal"; import { capturedRateLimit, rateLimitResponseHeaders, @@ -266,17 +262,9 @@ export function apiHandler Promise>( // The boot key check found that the configured encryption key // cannot open this database. Refuse before auth, before the // handler: nothing here may read or write encrypted data. - if (isKeyMismatch() && !KEY_MISMATCH_EXEMPT_PATHS.has(url.pathname)) { - annotate({ action: { name: "encryption.key_mismatch.refused" } }); - response = NextResponse.json( - { - data: null, - error: - "The server's encryption key does not match its database. The operator has to restore the original key or point the server at the matching database.", - meta: { errorCode: KEY_MISMATCH_ERROR_CODE }, - }, - { status: 503, headers: { "Retry-After": "300" } }, - ); + const refusal = refuseOnKeyMismatch(url.pathname); + if (refusal) { + response = refusal; } else { response = await handler(...args); } diff --git a/src/lib/boot/key-mismatch-refusal.ts b/src/lib/boot/key-mismatch-refusal.ts new file mode 100644 index 000000000..10533a122 --- /dev/null +++ b/src/lib/boot/key-mismatch-refusal.ts @@ -0,0 +1,67 @@ +/** + * The refusal a request surface answers while the boot key check has found + * that the configured encryption key does not open this database. + * + * `apiHandler` applies it to every API route. A route outside `apiHandler` + * (the remote MCP endpoint, its OAuth bridge) calls one of the two helpers + * below as the first thing it does, before auth and before any read or + * write: a process holding the wrong key must never seal new rows, because + * those rows would open under that key at the next boot and read as evidence + * that it is the right one. `key-mismatch-refusal-guard.test.ts` fails when a + * route outside `apiHandler` does neither. + */ +import { annotate } from "@/lib/logging/context"; +import { + isKeyMismatch, + KEY_MISMATCH_ERROR_CODE, + KEY_MISMATCH_EXEMPT_PATHS, +} from "./key-mismatch-state"; + +export const KEY_MISMATCH_MESSAGE = + "The server's encryption key does not match its database. The operator has to restore the original key or point the server at the matching database."; + +const RETRY_AFTER = { "Retry-After": "300" }; + +/** The 503 in the standard `{ data, error, meta }` envelope. */ +export function keyMismatchResponse(): Response { + annotate({ action: { name: "encryption.key_mismatch.refused" } }); + return Response.json( + { + data: null, + error: KEY_MISMATCH_MESSAGE, + meta: { errorCode: KEY_MISMATCH_ERROR_CODE }, + }, + { status: 503, headers: RETRY_AFTER }, + ); +} + +/** + * The envelope refusal when the process refuses and `pathname` is not one of + * the exempt paths; otherwise null and the caller carries on. + */ +export function refuseOnKeyMismatch(pathname?: string): Response | null { + if (!isKeyMismatch()) return null; + if (pathname !== undefined && KEY_MISMATCH_EXEMPT_PATHS.has(pathname)) { + return null; + } + return keyMismatchResponse(); +} + +/** + * The same refusal in the RFC 6749 error shape the OAuth endpoints speak: + * `temporarily_unavailable` with a description, plus the HealthLog error + * code as an extension member a client may ignore. + */ +export function refuseOAuthOnKeyMismatch(): Response | null { + if (!isKeyMismatch()) return null; + annotate({ action: { name: "encryption.key_mismatch.refused" } }); + return Response.json( + { + error: "temporarily_unavailable", + error_description: + "The server's encryption key does not match its database.", + error_code: KEY_MISMATCH_ERROR_CODE, + }, + { status: 503, headers: { ...RETRY_AFTER, "Cache-Control": "no-store" } }, + ); +} diff --git a/tests/integration/encryption-key-check.test.ts b/tests/integration/encryption-key-check.test.ts index 01f2796af..6e7a99a51 100644 --- a/tests/integration/encryption-key-check.test.ts +++ b/tests/integration/encryption-key-check.test.ts @@ -439,6 +439,70 @@ describe("retiring a key id after rotation (real Postgres)", () => { }); }); +describe("routes outside apiHandler refuse while the key does not match", () => { + beforeEach(() => { + setKeyMismatchState({ keyIds: ["v1"], detectedAt: "2026-10-03T00:00:00Z" }); + }); + + it("/mcp answers 503 encryption.key_mismatch before auth", async () => { + const { POST } = await import("@/app/mcp/route"); + const res = await POST( + new Request("http://localhost/mcp", { + method: "POST", + headers: { + "content-type": "application/json", + authorization: "Bearer hlk_whatever", + }, + body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/list" }), + }), + ); + expect(res.status).toBe(503); + expect((await res.json()).meta).toEqual({ + errorCode: "encryption.key_mismatch", + }); + }); + + it("the MCP OAuth endpoints refuse in their RFC shape", async () => { + const { POST: token } = await import("@/app/api/mcp/oauth/token/route"); + const { POST: register } = + await import("@/app/api/mcp/oauth/register/route"); + const { GET: authorizeGet, POST: authorizePost } = + await import("@/app/api/mcp/oauth/authorize/route"); + const responses = [ + await token( + new NextRequest("http://localhost/api/mcp/oauth/token", { + method: "POST", + headers: { "content-type": "application/x-www-form-urlencoded" }, + body: "grant_type=authorization_code&code=x", + }), + ), + await register( + post("/api/mcp/oauth/register", { + redirect_uris: ["https://client.example/cb"], + }), + ), + await authorizeGet( + new NextRequest( + "http://localhost/api/mcp/oauth/authorize?response_type=code", + ), + ), + await authorizePost( + new NextRequest("http://localhost/api/mcp/oauth/authorize", { + method: "POST", + headers: { "content-type": "application/x-www-form-urlencoded" }, + body: "decision=approve", + }), + ), + ]; + for (const res of responses) { + expect(res.status).toBe(503); + const body = await res.json(); + expect(body.error).toBe("temporarily_unavailable"); + expect(body.error_description).toContain("encryption key"); + } + }); +}); + describe("encryption key backup step (real Postgres)", () => { it("is due, refuses a stale confirmation, records the right one, and is due again after a re-key", async () => { const prisma = getPrismaClient(); From 69bae8bdb475acf77ca95e064926219aec25b5ac Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc-Andr=C3=A9=20Bombeck?= Date: Sat, 3 Oct 2026 23:46:40 +0200 Subject: [PATCH 3/3] Add ENCRYPTION_KEY_CHECK=warn as an escape hatch for the boot key check At a first start there is no recorded key yet, so an operator facing a refusal they are certain is false has no row to delete. With ENCRYPTION_KEY_CHECK=warn (default enforce) the check still runs and logs its full outcome, /api/health names a mismatch as a warning field without failing, requests are served, and no key is recorded unless every key id passes. The variable is on the compose whitelist, in both example files, the env manifest, the Unraid and Portainer templates and the published-env guard. The log block, the mismatch page and the self-hosting pages offer it as the last resort after the three real fixes, and say that it disables a safety check. --- .env.example | 7 ++ .env.production.example | 8 ++ docker-compose.yml | 5 + docs/self-hosting/getting-started.md | 9 ++ docs/self-hosting/portainer/templates.json | 6 ++ docs/self-hosting/truenas.md | 9 ++ docs/self-hosting/unraid/healthlog.xml | 1 + messages/de.json | 1 + messages/en.json | 1 + messages/es.json | 1 + messages/fr.json | 1 + messages/it.json | 1 + messages/ko.json | 1 + messages/pl.json | 1 + scripts/env-manifest.json | 5 + ...ublished-env-reaches-the-container.test.ts | 5 + src/app/api/health/__tests__/route.test.ts | 23 +++++ src/app/api/health/route.ts | 10 +- src/app/key-mismatch-page.tsx | 5 + src/lib/boot/key-check.ts | 33 ++++++- src/lib/boot/key-mismatch-state.ts | 31 +++++++ src/lib/crypto/__tests__/canary.test.ts | 49 ++++++++++ src/lib/crypto/canary.ts | 20 +++- .../integration/encryption-key-check.test.ts | 93 +++++++++++++++++++ 24 files changed, 319 insertions(+), 7 deletions(-) diff --git a/.env.example b/.env.example index efe6a7a8d..8d7802659 100644 --- a/.env.example +++ b/.env.example @@ -67,6 +67,13 @@ NODE_ENV="development" # your encryption key" step shows first; unset means compose. # HEALTHLOG_PLATFORM="compose" +# Start-up encryption key check: enforce (default) refuses to serve when the +# key does not open the stored data; warn only logs it and serves anyway. A +# last resort for a refusal you are certain is false (a first start, with no +# recorded key to reset). It disables a safety check: with a wrong key every +# encrypted value fails and new rows are written under that key. +# ENCRYPTION_KEY_CHECK="enforce" + # ----------------------------------------------------------------------------- # Required Secrets — generate with `openssl rand -hex 32` diff --git a/.env.production.example b/.env.production.example index e913ade7a..ef4445e8b 100644 --- a/.env.production.example +++ b/.env.production.example @@ -266,6 +266,14 @@ API_TOKEN_HMAC_KEY="" # it for you. # HEALTHLOG_PLATFORM="compose" +# Start-up encryption key check: enforce (default) refuses to serve when the +# key does not open the stored data; warn only logs it, reports it as a +# warning on /api/health, and serves anyway. A last resort for a refusal you +# are certain is false (a first start, with no recorded key to reset). It +# disables a safety check: with a wrong key every encrypted value fails and +# new rows are written under that key. Remove it once the key is confirmed. +# ENCRYPTION_KEY_CHECK="enforce" + # ----------------------------------------------------------------------------- # Deploy webhook -- Coolify -> HealthLog auto-deploy feedback diff --git a/docker-compose.yml b/docker-compose.yml index c9216b398..d4022f1a1 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -135,6 +135,11 @@ x-healthlog-env: &healthlog_env # compose | truenas | coolify | unraid | portainer. Only chooses which # instructions the admin's "Back up your encryption key" step shows first. HEALTHLOG_PLATFORM: "${HEALTHLOG_PLATFORM:-}" + # enforce (default) | warn. The start-up encryption key check refuses to + # serve when the key does not open the stored data; warn only logs it and + # serves anyway. A last resort for a refusal you are certain is false: it + # disables a safety check. See docs/self-hosting/getting-started.md. + ENCRYPTION_KEY_CHECK: "${ENCRYPTION_KEY_CHECK:-}" BACKUP_ENCRYPTION_KEY: "${BACKUP_ENCRYPTION_KEY:-}" BACKUP_ENCRYPTION_PREVIOUS_KEYS: "${BACKUP_ENCRYPTION_PREVIOUS_KEYS:-}" BACKUP_S3_ENDPOINT: "${BACKUP_S3_ENDPOINT:-}" diff --git a/docs/self-hosting/getting-started.md b/docs/self-hosting/getting-started.md index cc0499067..88b88f0eb 100644 --- a/docs/self-hosting/getting-started.md +++ b/docs/self-hosting/getting-started.md @@ -295,6 +295,15 @@ one clear refusal. After a key rotation the rotation script removes the record of a retired key id itself, once it reports that no values remain under that id. +At a first start there is no record yet, so there is nothing to reset. If +the check refuses there and you are certain the key is right, the last +resort is `ENCRYPTION_KEY_CHECK=warn` (default `enforce`): the check still +runs and logs its full result, `/api/health` names it as +`warning: "encryption_key_mismatch"` without failing, requests are served, +and no key is recorded while the check does not pass. This disables a +safety check. With a wrong key every encrypted value fails and new rows are +written under that key, so remove the setting once the key is confirmed. + ## Backup and restore Your database is the only stateful piece — back it up, and back up the diff --git a/docs/self-hosting/portainer/templates.json b/docs/self-hosting/portainer/templates.json index 6b2c51f83..bfa70a388 100644 --- a/docs/self-hosting/portainer/templates.json +++ b/docs/self-hosting/portainer/templates.json @@ -36,6 +36,12 @@ "description": "Opens the admin's encryption key backup step on the Portainer instructions.", "default": "portainer" }, + { + "name": "ENCRYPTION_KEY_CHECK", + "label": "Encryption key check", + "description": "enforce refuses to serve when the key does not open the stored data. warn only logs it: a last resort for a refusal you are certain is false; it disables a safety check.", + "default": "enforce" + }, { "name": "SESSION_COOKIE_SECURE", "label": "Session cookie Secure flag", diff --git a/docs/self-hosting/truenas.md b/docs/self-hosting/truenas.md index c39f6e016..0b34fe4eb 100644 --- a/docs/self-hosting/truenas.md +++ b/docs/self-hosting/truenas.md @@ -106,6 +106,15 @@ fails on every encrypted value instead of refusing clearly, so only do this when you are sure. After a key rotation you never need this for the old key id: the rotation script removes its record once no values remain under it. +On a first start there is no record to remove. If the check refuses there and +you are certain the key is right, add `ENCRYPTION_KEY_CHECK` = `warn` under +Apps, HealthLog, Edit, Environment as a last resort. The check still runs and +logs, `/api/health` reports `warning: "encryption_key_mismatch"` without +failing, the app serves, and no key is recorded until the check passes. This +switches off a safety check: with a wrong key every encrypted value fails and +new data is written under that key. Remove the setting once the key is +confirmed. + ## Updates Pick the new tag, take a snapshot of the database dataset first, and update. diff --git a/docs/self-hosting/unraid/healthlog.xml b/docs/self-hosting/unraid/healthlog.xml index 26ee42ee6..c6e92f981 100644 --- a/docs/self-hosting/unraid/healthlog.xml +++ b/docs/self-hosting/unraid/healthlog.xml @@ -54,6 +54,7 @@ false unraid + enforce diff --git a/messages/de.json b/messages/de.json index 26ece40e6..720463289 100644 --- a/messages/de.json +++ b/messages/de.json @@ -7657,6 +7657,7 @@ "fixDatabase": "Oder lass DATABASE_URL auf die Datenbank zeigen, die zu diesem Schlüssel gehört.", "fixFresh": "Oder starte neu mit einer leeren Datenbank. Die alten Daten und ihre Sicherungen lassen sich nur mit dem alten Schlüssel lesen.", "restartNote": "Starte HealthLog nach der Änderung neu.", + "lastResort": "Letzter Ausweg, nur wenn du sicher bist, dass der Schlüssel stimmt, und keiner der drei Wege passt: setze {setting} und starte neu. HealthLog liefert dann trotzdem aus und schreibt den Befund nur ins Log. Damit ist eine Sicherheitsprüfung abgeschaltet: Mit einem falschen Schlüssel scheitert jeder verschlüsselte Wert, und neue Daten werden unter diesem Schlüssel geschrieben.", "codes": "Für den Betrieb: API-Anfragen antworten mit 503 und dem Fehlercode {code}, /api/health meldet den Grund {reason}." } }, diff --git a/messages/en.json b/messages/en.json index e94c08bce..a181746f7 100644 --- a/messages/en.json +++ b/messages/en.json @@ -7657,6 +7657,7 @@ "fixDatabase": "Or point DATABASE_URL at the database that belongs to this key.", "fixFresh": "Or start fresh with an empty database. The old data and its backups can only be read with the old key.", "restartNote": "Restart HealthLog after the change.", + "lastResort": "Last resort, only if you are certain the key is right and none of the three applies: set {setting} and restart. HealthLog then serves anyway and only logs the finding. This disables a safety check: with a wrong key every encrypted value fails and new data is written under that key.", "codes": "For the operator: API requests answer 503 with error code {code}, and /api/health reports reason {reason}." } }, diff --git a/messages/es.json b/messages/es.json index a2fff14e5..11b4babc8 100644 --- a/messages/es.json +++ b/messages/es.json @@ -7657,6 +7657,7 @@ "fixDatabase": "O haz que DATABASE_URL apunte a la base de datos que corresponde a esta clave.", "fixFresh": "O empieza de cero con una base de datos vacía. Los datos antiguos y sus copias solo se pueden leer con la clave antigua.", "restartNote": "Reinicia HealthLog después del cambio.", + "lastResort": "Último recurso, solo si estás seguro de que la clave es correcta y ninguna de las tres opciones sirve: define {setting} y reinicia. HealthLog sirve igualmente y solo registra el hallazgo. Esto desactiva una comprobación de seguridad: con una clave incorrecta falla cada valor cifrado y los datos nuevos se escriben con esa clave.", "codes": "Para el administrador: las peticiones a la API responden 503 con el código de error {code}, y /api/health indica el motivo {reason}." } }, diff --git a/messages/fr.json b/messages/fr.json index 572487db8..b90e7c26f 100644 --- a/messages/fr.json +++ b/messages/fr.json @@ -7657,6 +7657,7 @@ "fixDatabase": "Ou fais pointer DATABASE_URL vers la base de données qui correspond à cette clé.", "fixFresh": "Ou repars de zéro avec une base vide. Les anciennes données et leurs sauvegardes ne se lisent qu'avec l'ancienne clé.", "restartNote": "Redémarre HealthLog après la modification.", + "lastResort": "Dernier recours, seulement si tu es sûr que la clé est la bonne et qu'aucune des trois solutions ne convient : définis {setting} et redémarre. HealthLog sert alors quand même et se contente de journaliser le constat. Cela désactive une vérification de sécurité : avec une mauvaise clé, chaque valeur chiffrée échoue et les nouvelles données sont écrites sous cette clé.", "codes": "Pour l'administrateur : les requêtes API répondent 503 avec le code d'erreur {code}, et /api/health indique la raison {reason}." } }, diff --git a/messages/it.json b/messages/it.json index 58d510fb7..883f22bc2 100644 --- a/messages/it.json +++ b/messages/it.json @@ -7657,6 +7657,7 @@ "fixDatabase": "Oppure fai puntare DATABASE_URL al database che appartiene a questa chiave.", "fixFresh": "Oppure ricomincia con un database vuoto. I vecchi dati e i loro backup si leggono solo con la vecchia chiave.", "restartNote": "Riavvia HealthLog dopo la modifica.", + "lastResort": "Ultima risorsa, solo se sei certo che la chiave sia giusta e nessuna delle tre soluzioni vale: imposta {setting} e riavvia. HealthLog serve comunque e si limita a registrare il problema nel log. Così si disattiva un controllo di sicurezza: con una chiave sbagliata ogni valore cifrato fallisce e i nuovi dati vengono scritti con quella chiave.", "codes": "Per chi gestisce il server: le richieste API rispondono 503 con il codice di errore {code}, e /api/health indica il motivo {reason}." } }, diff --git a/messages/ko.json b/messages/ko.json index 47f95296b..1e388d50f 100644 --- a/messages/ko.json +++ b/messages/ko.json @@ -7657,6 +7657,7 @@ "fixDatabase": "또는 DATABASE_URL이 이 키에 맞는 데이터베이스를 가리키게 하세요.", "fixFresh": "또는 빈 데이터베이스로 새로 시작하세요. 이전 데이터와 백업은 이전 키로만 읽을 수 있습니다.", "restartNote": "변경 후 HealthLog를 다시 시작하세요.", + "lastResort": "최후의 수단: 키가 맞다고 확신하고 위 세 가지가 모두 해당하지 않을 때만 {setting}을 설정하고 다시 시작하세요. HealthLog는 그래도 서비스하며 결과를 로그에만 남깁니다. 이 설정은 안전 검사를 끕니다. 키가 틀리면 모든 암호화된 값이 실패하고 새 데이터가 그 키로 기록됩니다.", "codes": "운영자용: API 요청은 오류 코드 {code}와 함께 503으로 응답하고, /api/health는 원인 {reason}을 알려 줍니다." } }, diff --git a/messages/pl.json b/messages/pl.json index 3cee9f5c9..c45f77f06 100644 --- a/messages/pl.json +++ b/messages/pl.json @@ -7657,6 +7657,7 @@ "fixDatabase": "Albo ustaw DATABASE_URL na bazę danych, która należy do tego klucza.", "fixFresh": "Albo zacznij od nowa z pustą bazą. Stare dane i ich kopie można odczytać tylko starym kluczem.", "restartNote": "Po zmianie uruchom HealthLog ponownie.", + "lastResort": "Ostateczność, tylko jeśli masz pewność, że klucz jest właściwy, a żadna z trzech dróg nie pasuje: ustaw {setting} i uruchom ponownie. HealthLog będzie wtedy działać mimo to i tylko zapisze wynik w logu. To wyłącza kontrolę bezpieczeństwa: przy złym kluczu każda zaszyfrowana wartość zawiedzie, a nowe dane zostaną zapisane tym kluczem.", "codes": "Dla administratora: żądania API odpowiadają 503 z kodem błędu {code}, a /api/health podaje powód {reason}." } }, diff --git a/scripts/env-manifest.json b/scripts/env-manifest.json index 9639d8b0a..d63628946 100644 --- a/scripts/env-manifest.json +++ b/scripts/env-manifest.json @@ -327,6 +327,11 @@ "purpose": "Where this server's environment is configured (compose, truenas, coolify, unraid, portainer). Shown back to the admin: the \"Back up your encryption key\" step opens on that platform's instructions. Unset means compose.", "example": "compose" }, + { + "name": "ENCRYPTION_KEY_CHECK", + "purpose": "The start-up encryption key check. enforce (default) refuses to serve when the configured key does not open the stored data; warn only logs the finding, names it as a warning on /api/health, serves anyway and records no key until the check passes. A last resort for a refusal the operator is certain is false; it disables a safety check.", + "example": "enforce" + }, { "name": "TRUST_PROXY_HOPS", "purpose": "Trusted reverse-proxy hop count; the client IP is read that many entries from the right of X-Forwarded-For.", diff --git a/src/__tests__/published-env-reaches-the-container.test.ts b/src/__tests__/published-env-reaches-the-container.test.ts index 2e1b54b21..11ccefdea 100644 --- a/src/__tests__/published-env-reaches-the-container.test.ts +++ b/src/__tests__/published-env-reaches-the-container.test.ts @@ -76,6 +76,11 @@ const PUBLISHED_TO_USERS = [ shownBy: "the admin's encryption key backup step, which opens on that platform", }, + { + name: "ENCRYPTION_KEY_CHECK", + shownBy: + "the key mismatch page and the start-up log, which offer it as the last resort", + }, { name: "AUDIT_LOG_RETENTION_DAYS", shownBy: "the shared-access activity window and the revoke dialog", diff --git a/src/app/api/health/__tests__/route.test.ts b/src/app/api/health/__tests__/route.test.ts index 691c9c6fe..d3865a248 100644 --- a/src/app/api/health/__tests__/route.test.ts +++ b/src/app/api/health/__tests__/route.test.ts @@ -140,3 +140,26 @@ describe("GET /api/health while the encryption key does not match", () => { expect(body).toEqual({ status: "ok" }); }); }); + +describe("GET /api/health under ENCRYPTION_KEY_CHECK=warn", () => { + it("names the mismatch as a warning and stays ok", async () => { + const { setKeyMismatchWarning } = + await import("@/lib/boot/key-mismatch-state"); + vi.stubEnv("HEALTHLOG_PROCESS_TYPE", "web"); + getGlobalBoss.mockReturnValue({}); + setKeyMismatchWarning({ + keyIds: ["v1"], + detectedAt: "2026-10-03T00:00:00Z", + }); + try { + const { response, body } = await health(); + expect(response.status).toBe(200); + expect(body).toEqual({ + status: "ok", + warning: "encryption_key_mismatch", + }); + } finally { + setKeyMismatchWarning(null); + } + }); +}); diff --git a/src/app/api/health/route.ts b/src/app/api/health/route.ts index 2b27876a6..2e7764036 100644 --- a/src/app/api/health/route.ts +++ b/src/app/api/health/route.ts @@ -7,6 +7,7 @@ import { NextResponse } from "next/server"; import { apiHandler } from "@/lib/api-handler"; import { annotate } from "@/lib/logging/context"; import { + getKeyMismatchWarning, isKeyMismatch, KEY_MISMATCH_HEALTH_REASON, } from "@/lib/boot/key-mismatch-state"; @@ -31,6 +32,12 @@ export const GET = apiHandler(async () => { // it is the one place a catalog install's status page reads. const keyMismatch = isKeyMismatch(); const reason = keyMismatch ? KEY_MISMATCH_HEALTH_REASON : undefined; + // Under ENCRYPTION_KEY_CHECK=warn the same finding does not refuse; it is + // named as a warning and leaves the status alone. + const warning = + !keyMismatch && getKeyMismatchWarning() + ? KEY_MISMATCH_HEALTH_REASON + : undefined; const status = !keyMismatch && dbOk && @@ -51,6 +58,7 @@ export const GET = apiHandler(async () => { { status, ...(reason ? { reason } : {}), + ...(warning ? { warning } : {}), timestamp: new Date().toISOString(), database: dbOk ? "connected" : "disconnected", worker: worker.running ? "running" : "stopped", @@ -63,7 +71,7 @@ export const GET = apiHandler(async () => { } return NextResponse.json( - { status, ...(reason ? { reason } : {}) }, + { status, ...(reason ? { reason } : {}), ...(warning ? { warning } : {}) }, { status: statusCode, headers: cacheHeaders }, ); }); diff --git a/src/app/key-mismatch-page.tsx b/src/app/key-mismatch-page.tsx index 1d95eaf24..5517f1964 100644 --- a/src/app/key-mismatch-page.tsx +++ b/src/app/key-mismatch-page.tsx @@ -31,6 +31,11 @@ export function KeyMismatchPage({ locale }: { locale: Locale }) {

{t("admin.keyMismatch.restartNote")}

+

+ {t("admin.keyMismatch.lastResort", { + setting: "ENCRYPTION_KEY_CHECK=warn", + })} +

{t("admin.keyMismatch.codes", { code: "encryption.key_mismatch", diff --git a/src/lib/boot/key-check.ts b/src/lib/boot/key-check.ts index c4deeb0b7..96b42100c 100644 --- a/src/lib/boot/key-check.ts +++ b/src/lib/boot/key-check.ts @@ -15,7 +15,11 @@ import { type CanaryClient, type InconclusiveReason, } from "@/lib/crypto/canary"; -import { setKeyMismatchState } from "./key-mismatch-state"; +import { + getKeyCheckMode, + setKeyMismatchState, + setKeyMismatchWarning, +} from "./key-mismatch-state"; const INCONCLUSIVE_EXPLANATION: Record = { "single-value": "the only stored value found under that id did not open.", @@ -31,13 +35,28 @@ const INCONCLUSIVE_EXPLANATION: Record = { }; export async function runBootKeyCheck(client: CanaryClient): Promise { - const outcome = await checkEncryptionKeyCanaries(client); + const mode = getKeyCheckMode(); + const outcome = await checkEncryptionKeyCanaries(client, { mode }); + setKeyMismatchWarning(null); if (outcome.state === "mismatch") { - setKeyMismatchState({ + const state = { keyIds: outcome.keyIds, detectedAt: new Date().toISOString(), - }); + }; console.error(keyMismatchLogBlock(outcome.keyIds)); + if (mode === "warn") { + // The operator turned the refusal off. Say so next to the block, keep + // the finding for /api/health, and serve. + setKeyMismatchState(null); + setKeyMismatchWarning(state); + console.error( + "[boot] ENCRYPTION_KEY_CHECK=warn: serving anyway. No key is recorded " + + "while the check does not pass. Remove the setting once the key is " + + "confirmed.", + ); + return false; + } + setKeyMismatchState(state); return true; } setKeyMismatchState(null); @@ -55,6 +74,12 @@ export async function runBootKeyCheck(client: CanaryClient): Promise { "the check runs again at the next start.", ); } + if (mode === "warn" && outcome.inconclusive.length > 0) { + console.warn( + "[boot] ENCRYPTION_KEY_CHECK=warn: no key recorded on this start " + + "because the check did not pass for every key id.", + ); + } if (outcome.written.length > 0) { console.info( `[boot] Encryption key check: recorded key id(s) ${outcome.written.join(", ")}`, diff --git a/src/lib/boot/key-mismatch-state.ts b/src/lib/boot/key-mismatch-state.ts index 4f9864009..8e41c7016 100644 --- a/src/lib/boot/key-mismatch-state.ts +++ b/src/lib/boot/key-mismatch-state.ts @@ -22,11 +22,30 @@ export interface KeyMismatchState { } const STATE_SYMBOL = Symbol.for("healthlog.encryptionKeyMismatch"); +const WARNING_SYMBOL = Symbol.for("healthlog.encryptionKeyMismatchWarning"); type GlobalWithState = typeof globalThis & { [STATE_SYMBOL]?: KeyMismatchState | null; + [WARNING_SYMBOL]?: KeyMismatchState | null; }; +/** + * `ENCRYPTION_KEY_CHECK`: `enforce` (the default, and what any other value + * means) refuses on a mismatch; `warn` runs the check, logs it, reports it as + * a warning on `/api/health`, and serves anyway. `warn` is the operator's way + * out of a refusal they are certain is false at a first boot, where there is + * no recorded key yet whose row they could delete. It disables a safety + * check: a process with the wrong key then serves errors on every encrypted + * value and writes new rows under that key. + */ +export type KeyCheckMode = "enforce" | "warn"; + +export function getKeyCheckMode(): KeyCheckMode { + return process.env.ENCRYPTION_KEY_CHECK?.trim().toLowerCase() === "warn" + ? "warn" + : "enforce"; +} + /** The paths that keep answering while the process refuses everything else. */ export const KEY_MISMATCH_EXEMPT_PATHS: ReadonlySet = new Set([ "/api/health", @@ -50,3 +69,15 @@ export function isKeyMismatch(): boolean { export function setKeyMismatchState(state: KeyMismatchState | null): void { (globalThis as GlobalWithState)[STATE_SYMBOL] = state; } + +/** + * A mismatch the check found while `ENCRYPTION_KEY_CHECK=warn`: recorded so + * `/api/health` can name it as a warning, never read by the refusal. + */ +export function getKeyMismatchWarning(): KeyMismatchState | null { + return (globalThis as GlobalWithState)[WARNING_SYMBOL] ?? null; +} + +export function setKeyMismatchWarning(state: KeyMismatchState | null): void { + (globalThis as GlobalWithState)[WARNING_SYMBOL] = state; +} diff --git a/src/lib/crypto/__tests__/canary.test.ts b/src/lib/crypto/__tests__/canary.test.ts index bb70ee45b..f816bbb4b 100644 --- a/src/lib/crypto/__tests__/canary.test.ts +++ b/src/lib/crypto/__tests__/canary.test.ts @@ -32,6 +32,13 @@ const KEY_A = const KEY_B = "fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210"; +function configureKeys(keys: Record, active: string) { + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify(keys)); + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", active); + vi.stubEnv("ENCRYPTION_KEY", ""); + _resetCryptoCacheForTests(); +} + function configureKey(hex: string) { vi.stubEnv("ENCRYPTION_KEYS", ""); vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", ""); @@ -394,6 +401,48 @@ describe("encryption key canary", () => { }); }); + describe("ENCRYPTION_KEY_CHECK=warn", () => { + it("records nothing for any key id while one is inconclusive", async () => { + configureKeys({ v1: KEY_A, v2: KEY_B }, "v2"); + const lone = encryptUnderKeyId("only value", "v1"); + configureKeys({ v1: KEY_B, v2: KEY_A }, "v2"); + const fake = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [lone] }, + }); + // Enforce records the proven id beside the inconclusive one. + expect(await fake.check()).toEqual( + ok({ + written: ["v2"], + inconclusive: [{ keyId: "v1", reason: "single-value" }], + }), + ); + const warn = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [lone] }, + }); + expect( + await checkEncryptionKeyCanaries(warn.client, { + sampler: warn.sampler, + mode: "warn", + }), + ).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "single-value" }] }), + ); + expect(warn.inserts).toEqual([]); + }); + + it("records a key that passes for every id, as enforce does", async () => { + const warn = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [encrypt("opens")] }, + }); + expect( + await checkEncryptionKeyCanaries(warn.client, { + sampler: warn.sampler, + mode: "warn", + }), + ).toEqual(ok({ written: ["v1"] })); + }); + }); + describe("the first-seal race", () => { it("losing it to a process with another key is a mismatch", async () => { configureKey(KEY_B); diff --git a/src/lib/crypto/canary.ts b/src/lib/crypto/canary.ts index 60ee6077f..9cb3b0ce5 100644 --- a/src/lib/crypto/canary.ts +++ b/src/lib/crypto/canary.ts @@ -472,7 +472,16 @@ export async function columnsHoldingKeyId( */ export async function checkEncryptionKeyCanaries( client: CanaryClient, - options: { probeBudgetMs?: number; sampler?: ColumnSampler } = {}, + options: { + probeBudgetMs?: number; + sampler?: ColumnSampler; + /** + * `warn` (`ENCRYPTION_KEY_CHECK=warn`): record nothing unless every key + * id passed. The process serves whatever the outcome, so a key that is + * not proven for every id must not be recorded beside it. + */ + mode?: "enforce" | "warn"; + } = {}, ): Promise { let keyIds: string[]; try { @@ -531,8 +540,10 @@ export async function checkEncryptionKeyCanaries( } const written: string[] = []; + const sealable = + options.mode === "warn" && inconclusive.length > 0 ? [] : toWrite; try { - for (const keyId of toWrite) { + for (const keyId of sealable) { const ciphertext = encryptUnderKeyId(canaryPlaintext(keyId), keyId); const inserted = await client.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES (${keyId}, ${ciphertext}) ON CONFLICT (key_id) DO NOTHING`; @@ -617,6 +628,11 @@ export function keyMismatchLogBlock(keyIds: string[]): string { ...keyIds.map( (id) => ` DELETE FROM encryption_key_canaries WHERE key_id = '${id}';`, ), + "", + " Last resort, when no record exists yet to reset (a first start) and you", + " are certain the key is right: ENCRYPTION_KEY_CHECK=warn serves anyway", + " and only logs. It disables this safety check; with a wrong key every", + " encrypted value then fails and new rows are written under that key.", "============================================================", ].join("\n"); } diff --git a/tests/integration/encryption-key-check.test.ts b/tests/integration/encryption-key-check.test.ts index 6e7a99a51..6be82f674 100644 --- a/tests/integration/encryption-key-check.test.ts +++ b/tests/integration/encryption-key-check.test.ts @@ -59,8 +59,10 @@ import { type CanaryClient, } from "@/lib/crypto/canary"; import { + getKeyMismatchWarning, isKeyMismatch, setKeyMismatchState, + setKeyMismatchWarning, } from "@/lib/boot/key-mismatch-state"; const KEY_A = @@ -392,6 +394,97 @@ describe("boot key check: what decides, and what may seal (real Postgres)", () = }); }); +describe("ENCRYPTION_KEY_CHECK (real Postgres)", () => { + afterEach(() => setKeyMismatchWarning(null)); + + it("warn: a mismatch is logged and reported, never refused, never recorded", async () => { + const prisma = getPrismaClient(); + await seedUser("warned", new Date("2024-01-01T00:00:00Z"), sealNow); + useKey(KEY_B); + vi.stubEnv("ENCRYPTION_KEY_CHECK", "warn"); + const errors = vi.spyOn(console, "error").mockImplementation(() => {}); + expect(await runBootKeyCheck(prisma)).toBe(false); + const logged = errors.mock.calls.map((c) => String(c[0])).join("\n"); + errors.mockRestore(); + expect(logged).toContain("encryption.key_mismatch"); + expect(logged).toContain("ENCRYPTION_KEY_CHECK=warn"); + expect(isKeyMismatch()).toBe(false); + expect(getKeyMismatchWarning()?.keyIds).toEqual(["v1"]); + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + + // Requests are served, through apiHandler and outside it. + await seedAdmin(); + const { GET: keyBackupGet } = + await import("@/app/api/admin/encryption/key-backup/route"); + const served = await asRoute(keyBackupGet)( + new NextRequest("http://localhost/api/admin/encryption/key-backup"), + ); + expect(served.status).toBe(200); + const { POST: mcp } = await import("@/app/mcp/route"); + const mcpRes = await mcp( + new Request("http://localhost/mcp", { method: "POST", body: "{}" }), + ); + expect(mcpRes.status).not.toBe(503); + + const { GET: healthGet } = await import("@/app/api/health/route"); + const healthBody = await ( + await asRoute(healthGet)(new NextRequest("http://localhost/api/health")) + ).json(); + expect(healthBody.warning).toBe("encryption_key_mismatch"); + expect(healthBody.reason).toBeUndefined(); + }); + + it("warn: nothing is recorded while any key id is inconclusive", async () => { + const prisma = getPrismaClient(); + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_A, v2: KEY_B })); + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v1"); + vi.stubEnv("ENCRYPTION_KEY", ""); + _resetCryptoCacheForTests(); + await prisma.user.create({ + data: { + username: "lone", + email: "lone@example.test", + codexAccessTokenEncrypted: encrypt("the only value"), + }, + }); + // v1 now holds a different key: its one value does not open. v2 has no + // data, which on its own would be recorded. + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_B, v2: KEY_A })); + _resetCryptoCacheForTests(); + vi.stubEnv("ENCRYPTION_KEY_CHECK", "warn"); + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + expect(await runBootKeyCheck(prisma)).toBe(false); + warn.mockRestore(); + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + + // Enforce records the proven id, as before. + vi.stubEnv("ENCRYPTION_KEY_CHECK", "enforce"); + const warn2 = vi.spyOn(console, "warn").mockImplementation(() => {}); + expect(await runBootKeyCheck(prisma)).toBe(false); + warn2.mockRestore(); + const ids = (await prisma.encryptionKeyCanary.findMany()).map( + (r) => r.keyId, + ); + expect(ids).toEqual(["v2"]); + }); + + it("enforce, set explicitly or by any other value, refuses as before", async () => { + const prisma = getPrismaClient(); + await seedUser("enforced", new Date("2024-01-01T00:00:00Z"), sealNow); + useKey(KEY_B); + for (const value of ["enforce", "", "off"]) { + vi.stubEnv("ENCRYPTION_KEY_CHECK", value); + setKeyMismatchState(null); + const errors = vi.spyOn(console, "error").mockImplementation(() => {}); + expect(await runBootKeyCheck(prisma)).toBe(true); + errors.mockRestore(); + expect(isKeyMismatch()).toBe(true); + expect(getKeyMismatchWarning()).toBeNull(); + } + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + }); +}); + describe("retiring a key id after rotation (real Postgres)", () => { it("removes the canary of a key id only once no ciphertext remains under it", async () => { const prisma = getPrismaClient();