diff --git a/.env.example b/.env.example index efe6a7a8d..8d7802659 100644 --- a/.env.example +++ b/.env.example @@ -67,6 +67,13 @@ NODE_ENV="development" # your encryption key" step shows first; unset means compose. # HEALTHLOG_PLATFORM="compose" +# Start-up encryption key check: enforce (default) refuses to serve when the +# key does not open the stored data; warn only logs it and serves anyway. A +# last resort for a refusal you are certain is false (a first start, with no +# recorded key to reset). It disables a safety check: with a wrong key every +# encrypted value fails and new rows are written under that key. +# ENCRYPTION_KEY_CHECK="enforce" + # ----------------------------------------------------------------------------- # Required Secrets — generate with `openssl rand -hex 32` diff --git a/.env.production.example b/.env.production.example index e913ade7a..ef4445e8b 100644 --- a/.env.production.example +++ b/.env.production.example @@ -266,6 +266,14 @@ API_TOKEN_HMAC_KEY="" # it for you. # HEALTHLOG_PLATFORM="compose" +# Start-up encryption key check: enforce (default) refuses to serve when the +# key does not open the stored data; warn only logs it, reports it as a +# warning on /api/health, and serves anyway. A last resort for a refusal you +# are certain is false (a first start, with no recorded key to reset). It +# disables a safety check: with a wrong key every encrypted value fails and +# new rows are written under that key. Remove it once the key is confirmed. +# ENCRYPTION_KEY_CHECK="enforce" + # ----------------------------------------------------------------------------- # Deploy webhook -- Coolify -> HealthLog auto-deploy feedback diff --git a/docker-compose.yml b/docker-compose.yml index c9216b398..d4022f1a1 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -135,6 +135,11 @@ x-healthlog-env: &healthlog_env # compose | truenas | coolify | unraid | portainer. Only chooses which # instructions the admin's "Back up your encryption key" step shows first. HEALTHLOG_PLATFORM: "${HEALTHLOG_PLATFORM:-}" + # enforce (default) | warn. The start-up encryption key check refuses to + # serve when the key does not open the stored data; warn only logs it and + # serves anyway. A last resort for a refusal you are certain is false: it + # disables a safety check. See docs/self-hosting/getting-started.md. + ENCRYPTION_KEY_CHECK: "${ENCRYPTION_KEY_CHECK:-}" BACKUP_ENCRYPTION_KEY: "${BACKUP_ENCRYPTION_KEY:-}" BACKUP_ENCRYPTION_PREVIOUS_KEYS: "${BACKUP_ENCRYPTION_PREVIOUS_KEYS:-}" BACKUP_S3_ENDPOINT: "${BACKUP_S3_ENDPOINT:-}" diff --git a/docs/ops/encryption-key-rotation.md b/docs/ops/encryption-key-rotation.md index 0396e2897..b48c64a72 100644 --- a/docs/ops/encryption-key-rotation.md +++ b/docs/ops/encryption-key-rotation.md @@ -103,6 +103,13 @@ under `v1` (the synthetic id assigned to the existing `ENCRYPTION_KEY`). a full `Columns walked` line. Both read the same registry the rotation walks. + A run that walked every column without an error also removes the boot + key check's record of each previous key id that no longer holds a single + value (`encryption_key_canaries`); the script prints which ids it removed + and, for an id that still holds values, the columns that do. With the record + gone, the id can later carry a different key without the start-up check + refusing it. + A zero only means "nothing left" when the run also says it walked every registered column. A zero from a run that skipped columns means "never looked", and dropping the old key on it makes those rows permanently diff --git a/docs/self-hosting/getting-started.md b/docs/self-hosting/getting-started.md index f67feb66f..88b88f0eb 100644 --- a/docs/self-hosting/getting-started.md +++ b/docs/self-hosting/getting-started.md @@ -267,6 +267,43 @@ every API request answers `503` with `meta.errorCode` Restore the original key, point `DATABASE_URL` at the matching database, or start with an empty database; then restart. +The check remembers the key it first saw for each key id in the table +`encryption_key_canaries`. Before it records a key, it reads the oldest +encrypted values in the database and records the key only if they open, or +if the database holds no encrypted value under that key id at all. When the +stored data proves neither (a single unreadable value, the oldest values +failing while newer ones open, or a probe that ran out of time), the app +keeps serving, records nothing, logs a warning that starts with +`[boot] Encryption key check inconclusive`, and checks again at the next +start. + +If you are certain the configured key is the one your data was written +with, and the check still refuses because it recorded a different key +earlier, reset the check for that key id and restart. The refusal in the log +prints the exact statement; for the default key id it is: + +```bash +docker compose exec -T db psql -U healthlog healthlog \ + -c "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';" +docker compose restart app +``` + +The next start probes the stored data again and records the key only if +that data opens with it. Do this only when you are sure: if the key is in +fact wrong, the app serves errors on every encrypted value instead of the +one clear refusal. After a key rotation the rotation script removes the +record of a retired key id itself, once it reports that no values remain +under that id. + +At a first start there is no record yet, so there is nothing to reset. If +the check refuses there and you are certain the key is right, the last +resort is `ENCRYPTION_KEY_CHECK=warn` (default `enforce`): the check still +runs and logs its full result, `/api/health` names it as +`warning: "encryption_key_mismatch"` without failing, requests are served, +and no key is recorded while the check does not pass. This disables a +safety check. With a wrong key every encrypted value fails and new rows are +written under that key, so remove the setting once the key is confirmed. + ## Backup and restore Your database is the only stateful piece — back it up, and back up the diff --git a/docs/self-hosting/portainer/templates.json b/docs/self-hosting/portainer/templates.json index 6b2c51f83..bfa70a388 100644 --- a/docs/self-hosting/portainer/templates.json +++ b/docs/self-hosting/portainer/templates.json @@ -36,6 +36,12 @@ "description": "Opens the admin's encryption key backup step on the Portainer instructions.", "default": "portainer" }, + { + "name": "ENCRYPTION_KEY_CHECK", + "label": "Encryption key check", + "description": "enforce refuses to serve when the key does not open the stored data. warn only logs it: a last resort for a refusal you are certain is false; it disables a safety check.", + "default": "enforce" + }, { "name": "SESSION_COOKIE_SECURE", "label": "Session cookie Secure flag", diff --git a/docs/self-hosting/truenas.md b/docs/self-hosting/truenas.md index 8bebe7441..0b34fe4eb 100644 --- a/docs/self-hosting/truenas.md +++ b/docs/self-hosting/truenas.md @@ -80,6 +80,41 @@ Fix it one of three ways, then restart the app: 3. Start fresh with an empty database. The old data and its backups need the old key. +Before it records a key as the right one, the check reads the oldest +encrypted values in the database and records the key only if they open (or if +the database holds no encrypted value under that key id yet). When the data +proves neither, the app keeps serving, records nothing, writes a warning +starting with `[boot] Encryption key check inconclusive` to the log, and +checks again at the next start. + +### Resetting the check + +If you are certain the configured key is the one the data was written with, +and the app still refuses because the check recorded a different key earlier, +remove that record and restart. Open a shell on the database container (Apps, +HealthLog, the database container's Shell) and run, with the key id the log +names (`v1` unless you use `ENCRYPTION_KEYS`): + +```bash +psql -U healthlog healthlog \ + -c "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';" +``` + +Then restart the app. It probes the stored data again and records the key +only if that data opens with it. If the key is in fact wrong, the app then +fails on every encrypted value instead of refusing clearly, so only do this +when you are sure. After a key rotation you never need this for the old key +id: the rotation script removes its record once no values remain under it. + +On a first start there is no record to remove. If the check refuses there and +you are certain the key is right, add `ENCRYPTION_KEY_CHECK` = `warn` under +Apps, HealthLog, Edit, Environment as a last resort. The check still runs and +logs, `/api/health` reports `warning: "encryption_key_mismatch"` without +failing, the app serves, and no key is recorded until the check passes. This +switches off a safety check: with a wrong key every encrypted value fails and +new data is written under that key. Remove the setting once the key is +confirmed. + ## Updates Pick the new tag, take a snapshot of the database dataset first, and update. diff --git a/docs/self-hosting/unraid/healthlog.xml b/docs/self-hosting/unraid/healthlog.xml index 26ee42ee6..c6e92f981 100644 --- a/docs/self-hosting/unraid/healthlog.xml +++ b/docs/self-hosting/unraid/healthlog.xml @@ -54,6 +54,7 @@ false unraid + enforce diff --git a/messages/de.json b/messages/de.json index 1653d342d..2e7a699c9 100644 --- a/messages/de.json +++ b/messages/de.json @@ -7657,6 +7657,7 @@ "fixDatabase": "Oder lass DATABASE_URL auf die Datenbank zeigen, die zu diesem Schlüssel gehört.", "fixFresh": "Oder starte neu mit einer leeren Datenbank. Die alten Daten und ihre Sicherungen lassen sich nur mit dem alten Schlüssel lesen.", "restartNote": "Starte HealthLog nach der Änderung neu.", + "lastResort": "Letzter Ausweg, nur wenn du sicher bist, dass der Schlüssel stimmt, und keiner der drei Wege passt: setze {setting} und starte neu. HealthLog liefert dann trotzdem aus und schreibt den Befund nur ins Log. Damit ist eine Sicherheitsprüfung abgeschaltet: Mit einem falschen Schlüssel scheitert jeder verschlüsselte Wert, und neue Daten werden unter diesem Schlüssel geschrieben.", "codes": "Für den Betrieb: API-Anfragen antworten mit 503 und dem Fehlercode {code}, /api/health meldet den Grund {reason}." } }, diff --git a/messages/en.json b/messages/en.json index bfef0d002..4fb883bc5 100644 --- a/messages/en.json +++ b/messages/en.json @@ -7657,6 +7657,7 @@ "fixDatabase": "Or point DATABASE_URL at the database that belongs to this key.", "fixFresh": "Or start fresh with an empty database. The old data and its backups can only be read with the old key.", "restartNote": "Restart HealthLog after the change.", + "lastResort": "Last resort, only if you are certain the key is right and none of the three applies: set {setting} and restart. HealthLog then serves anyway and only logs the finding. This disables a safety check: with a wrong key every encrypted value fails and new data is written under that key.", "codes": "For the operator: API requests answer 503 with error code {code}, and /api/health reports reason {reason}." } }, diff --git a/messages/es.json b/messages/es.json index 473e61765..7127934cf 100644 --- a/messages/es.json +++ b/messages/es.json @@ -7657,6 +7657,7 @@ "fixDatabase": "O haz que DATABASE_URL apunte a la base de datos que corresponde a esta clave.", "fixFresh": "O empieza de cero con una base de datos vacía. Los datos antiguos y sus copias solo se pueden leer con la clave antigua.", "restartNote": "Reinicia HealthLog después del cambio.", + "lastResort": "Último recurso, solo si estás seguro de que la clave es correcta y ninguna de las tres opciones sirve: define {setting} y reinicia. HealthLog sirve igualmente y solo registra el hallazgo. Esto desactiva una comprobación de seguridad: con una clave incorrecta falla cada valor cifrado y los datos nuevos se escriben con esa clave.", "codes": "Para el administrador: las peticiones a la API responden 503 con el código de error {code}, y /api/health indica el motivo {reason}." } }, diff --git a/messages/fr.json b/messages/fr.json index d19c0b067..1a82ab757 100644 --- a/messages/fr.json +++ b/messages/fr.json @@ -7657,6 +7657,7 @@ "fixDatabase": "Ou fais pointer DATABASE_URL vers la base de données qui correspond à cette clé.", "fixFresh": "Ou repars de zéro avec une base vide. Les anciennes données et leurs sauvegardes ne se lisent qu'avec l'ancienne clé.", "restartNote": "Redémarre HealthLog après la modification.", + "lastResort": "Dernier recours, seulement si tu es sûr que la clé est la bonne et qu'aucune des trois solutions ne convient : définis {setting} et redémarre. HealthLog sert alors quand même et se contente de journaliser le constat. Cela désactive une vérification de sécurité : avec une mauvaise clé, chaque valeur chiffrée échoue et les nouvelles données sont écrites sous cette clé.", "codes": "Pour l'administrateur : les requêtes API répondent 503 avec le code d'erreur {code}, et /api/health indique la raison {reason}." } }, diff --git a/messages/it.json b/messages/it.json index da25525b6..09464fbf9 100644 --- a/messages/it.json +++ b/messages/it.json @@ -7657,6 +7657,7 @@ "fixDatabase": "Oppure fai puntare DATABASE_URL al database che appartiene a questa chiave.", "fixFresh": "Oppure ricomincia con un database vuoto. I vecchi dati e i loro backup si leggono solo con la vecchia chiave.", "restartNote": "Riavvia HealthLog dopo la modifica.", + "lastResort": "Ultima risorsa, solo se sei certo che la chiave sia giusta e nessuna delle tre soluzioni vale: imposta {setting} e riavvia. HealthLog serve comunque e si limita a registrare il problema nel log. Così si disattiva un controllo di sicurezza: con una chiave sbagliata ogni valore cifrato fallisce e i nuovi dati vengono scritti con quella chiave.", "codes": "Per chi gestisce il server: le richieste API rispondono 503 con il codice di errore {code}, e /api/health indica il motivo {reason}." } }, diff --git a/messages/ko.json b/messages/ko.json index fd5cc5b55..ab049b462 100644 --- a/messages/ko.json +++ b/messages/ko.json @@ -7657,6 +7657,7 @@ "fixDatabase": "또는 DATABASE_URL이 이 키에 맞는 데이터베이스를 가리키게 하세요.", "fixFresh": "또는 빈 데이터베이스로 새로 시작하세요. 이전 데이터와 백업은 이전 키로만 읽을 수 있습니다.", "restartNote": "변경 후 HealthLog를 다시 시작하세요.", + "lastResort": "최후의 수단: 키가 맞다고 확신하고 위 세 가지가 모두 해당하지 않을 때만 {setting}을 설정하고 다시 시작하세요. HealthLog는 그래도 서비스하며 결과를 로그에만 남깁니다. 이 설정은 안전 검사를 끕니다. 키가 틀리면 모든 암호화된 값이 실패하고 새 데이터가 그 키로 기록됩니다.", "codes": "운영자용: API 요청은 오류 코드 {code}와 함께 503으로 응답하고, /api/health는 원인 {reason}을 알려 줍니다." } }, diff --git a/messages/pl.json b/messages/pl.json index 5600fd0bc..731890d25 100644 --- a/messages/pl.json +++ b/messages/pl.json @@ -7657,6 +7657,7 @@ "fixDatabase": "Albo ustaw DATABASE_URL na bazę danych, która należy do tego klucza.", "fixFresh": "Albo zacznij od nowa z pustą bazą. Stare dane i ich kopie można odczytać tylko starym kluczem.", "restartNote": "Po zmianie uruchom HealthLog ponownie.", + "lastResort": "Ostateczność, tylko jeśli masz pewność, że klucz jest właściwy, a żadna z trzech dróg nie pasuje: ustaw {setting} i uruchom ponownie. HealthLog będzie wtedy działać mimo to i tylko zapisze wynik w logu. To wyłącza kontrolę bezpieczeństwa: przy złym kluczu każda zaszyfrowana wartość zawiedzie, a nowe dane zostaną zapisane tym kluczem.", "codes": "Dla administratora: żądania API odpowiadają 503 z kodem błędu {code}, a /api/health podaje powód {reason}." } }, diff --git a/scripts/env-manifest.json b/scripts/env-manifest.json index 9639d8b0a..d63628946 100644 --- a/scripts/env-manifest.json +++ b/scripts/env-manifest.json @@ -327,6 +327,11 @@ "purpose": "Where this server's environment is configured (compose, truenas, coolify, unraid, portainer). Shown back to the admin: the \"Back up your encryption key\" step opens on that platform's instructions. Unset means compose.", "example": "compose" }, + { + "name": "ENCRYPTION_KEY_CHECK", + "purpose": "The start-up encryption key check. enforce (default) refuses to serve when the configured key does not open the stored data; warn only logs the finding, names it as a warning on /api/health, serves anyway and records no key until the check passes. A last resort for a refusal the operator is certain is false; it disables a safety check.", + "example": "enforce" + }, { "name": "TRUST_PROXY_HOPS", "purpose": "Trusted reverse-proxy hop count; the client IP is read that many entries from the right of X-Forwarded-For.", diff --git a/scripts/rotate-encryption-key.ts b/scripts/rotate-encryption-key.ts index a3cd1742a..6e054ba8b 100644 --- a/scripts/rotate-encryption-key.ts +++ b/scripts/rotate-encryption-key.ts @@ -17,6 +17,10 @@ * non-zero, because an operator reads a zero here as permission to drop the * previous key, and "nothing left" and "never looked" must not print alike. * + * After a complete, error-free run it checks every other configured key id + * for remaining values and removes the boot key check's record (canary) of + * each id that holds none, naming the columns of each id that still does. + * * Usage: * ENCRYPTION_KEYS='{"v1":"","v2":""}' \ * ENCRYPTION_ACTIVE_KEY_ID=v2 \ @@ -27,6 +31,7 @@ import { Buffer } from "node:buffer"; import { PrismaClient } from "@/generated/prisma/client"; import { PrismaPg } from "@prisma/adapter-pg"; import { decrypt, encrypt, extractKeyId, getActiveKeyId } from "@/lib/crypto"; +import { retireCanariesWithoutData } from "@/lib/crypto/canary"; import { rotateColumn, type CorpusClient, @@ -1034,6 +1039,30 @@ async function main() { `drop the previous key:\n ${notWalked.join("\n ")}`, ); } + + // ───── Key-check records of retired key ids ───── + // The boot key check keeps one canary per configured key id. Once no + // registered column holds a single value under an id, that id's key can + // leave ENCRYPTION_KEYS, and its canary goes with it: left behind, it would + // refuse a later, different key configured under the same id. Only after a + // complete, error-free run; the existence check walks every column itself + // rather than trusting the counts above. + if (notWalked.length === 0 && totalErrors === 0) { + const { removed, remaining } = await retireCanariesWithoutData(prisma); + for (const [keyId, columns] of Object.entries(remaining)) { + console.log( + `\nKey id '${keyId}': values remain in ${columns.length} column(s); ` + + `keep its key:\n ${columns.join("\n ")}`, + ); + } + for (const keyId of removed) { + console.log( + `\nKey id '${keyId}': no values remain. Its key-check record was ` + + `removed; its key can now leave ENCRYPTION_KEYS.`, + ); + } + } + await prisma.$disconnect(); if (notWalked.length > 0) process.exit(4); process.exit(totalErrors > 0 ? 3 : 0); diff --git a/src/__tests__/key-mismatch-refusal-guard.test.ts b/src/__tests__/key-mismatch-refusal-guard.test.ts new file mode 100644 index 000000000..9d145e319 --- /dev/null +++ b/src/__tests__/key-mismatch-refusal-guard.test.ts @@ -0,0 +1,127 @@ +/** + * Every request surface outside `apiHandler` refuses while the boot key check + * has found a mismatch. + * + * `apiHandler` answers 503 `encryption.key_mismatch` before auth and before + * the handler, so a process holding the wrong key never writes through it. + * A route that does not wrap in `apiHandler` gets none of that: the remote + * MCP endpoint and its OAuth bridge went without it, and an MCP write tool + * would have sealed new rows under the wrong key, rows the next boot's probe + * could then read as evidence that the wrong key is right. + * + * So: a `route.ts` under `src/app` that does not wrap in `apiHandler` must + * call `refuseOnKeyMismatch` / `refuseOAuthOnKeyMismatch`, or sit in the + * allowlist below with the reason it cannot touch encrypted data. A server + * action (`"use server"`) is held to the same rule; there are none today. + */ +import { describe, expect, it } from "vitest"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; + +import { walkSourceFiles } from "./helpers/source-files"; + +const SRC = join(process.cwd(), "src"); + +/** + * Routes outside `apiHandler` that need no refusal, each with why. A file + * belongs here only if it reads nothing from the database and writes nothing. + */ +const NO_DATA_ROUTES: Record = { + "app/.well-known/apple-app-site-association/route.ts": + "static JSON built from environment variables; no database access", + "app/.well-known/oauth-authorization-server/route.ts": + "RFC 8414 metadata built from the configured origin; no database access", + "app/.well-known/oauth-protected-resource/route.ts": + "RFC 9728 metadata built from the configured origin; no database access", +}; + +const REFUSAL_CALL = /\brefuse(?:OAuth)?OnKeyMismatch\s*\(/; + +function sourceFiles(): string[] { + return walkSourceFiles(SRC, { floor: 3000 }) + .filter((p) => !p.includes("__tests__")) + .filter((p) => !p.endsWith(".test.ts") && !p.endsWith(".test.tsx")) + .sort(); +} + +/** The text of a top-level function starting at `from`, up to its closing brace. */ +function functionBody(src: string, from: number): string { + const end = src.indexOf("\n}\n", from); + return src.slice(from, end === -1 ? undefined : end); +} + +function read(rel: string): string { + return readFileSync(join(SRC, rel), "utf8"); +} + +describe("key mismatch refusal outside apiHandler", () => { + const routes = sourceFiles().filter( + (p) => p.startsWith("app/") && /\/route\.tsx?$/.test(p), + ); + const outside = routes.filter((p) => !/\bapiHandler\s*\(/.test(read(p))); + + it("finds the route tree and the routes outside apiHandler", () => { + // A matcher that matches nothing would pass every check below. + expect(routes.length).toBeGreaterThan(300); + expect(outside).toEqual( + expect.arrayContaining([ + "app/mcp/route.ts", + "app/api/mcp/oauth/authorize/route.ts", + "app/api/mcp/oauth/register/route.ts", + "app/api/mcp/oauth/token/route.ts", + ]), + ); + }); + + it("every route outside apiHandler refuses, or is allowlisted with a reason", () => { + const missing = outside.filter( + (p) => !(p in NO_DATA_ROUTES) && !REFUSAL_CALL.test(read(p)), + ); + expect(missing).toEqual([]); + }); + + it("an allowlisted route really stays away from the database", () => { + for (const p of Object.keys(NO_DATA_ROUTES)) { + expect(outside, `${p} is no longer outside apiHandler`).toContain(p); + const src = read(p); + expect(src, p).not.toMatch(/@\/lib\/db\b|prisma|\$queryRaw/); + } + }); + + it("every exported handler of a refusing route calls the refusal itself", () => { + // `export async function POST(...) { ... }` must refuse in its own body; + // `export const POST = handleX` must refuse in `handleX`. A refusal in one + // handler of a file proves nothing about its siblings. + let checked = 0; + for (const p of outside.filter((f) => !(f in NO_DATA_ROUTES))) { + const src = read(p); + const bodies: Array<[string, string]> = []; + for (const m of src.matchAll( + /export\s+(?:async\s+)?function\s+(GET|POST|PUT|PATCH|DELETE)\b/g, + )) { + bodies.push([m[1], functionBody(src, m.index)]); + } + for (const m of src.matchAll( + /export\s+const\s+(GET|POST|PUT|PATCH|DELETE)\s*=\s*(\w+)\s*;/g, + )) { + const def = new RegExp(`function\\s+${m[2]}\\s*\\(`).exec(src); + expect(def, `${p}: ${m[1]} → ${m[2]} not found`).not.toBeNull(); + bodies.push([m[1], functionBody(src, def!.index)]); + } + expect(bodies.length, p).toBeGreaterThan(0); + for (const [name, body] of bodies) { + expect(REFUSAL_CALL.test(body), `${p} ${name}`).toBe(true); + checked += 1; + } + } + expect(checked).toBeGreaterThanOrEqual(6); + }); + + it("server actions refuse too", () => { + const actions = sourceFiles().filter((p) => + /^\s*["']use server["']/m.test(read(p)), + ); + const missing = actions.filter((p) => !REFUSAL_CALL.test(read(p))); + expect(missing).toEqual([]); + }); +}); diff --git a/src/__tests__/published-env-reaches-the-container.test.ts b/src/__tests__/published-env-reaches-the-container.test.ts index 2e1b54b21..11ccefdea 100644 --- a/src/__tests__/published-env-reaches-the-container.test.ts +++ b/src/__tests__/published-env-reaches-the-container.test.ts @@ -76,6 +76,11 @@ const PUBLISHED_TO_USERS = [ shownBy: "the admin's encryption key backup step, which opens on that platform", }, + { + name: "ENCRYPTION_KEY_CHECK", + shownBy: + "the key mismatch page and the start-up log, which offer it as the last resort", + }, { name: "AUDIT_LOG_RETENTION_DAYS", shownBy: "the shared-access activity window and the revoke dialog", diff --git a/src/app/api/health/__tests__/route.test.ts b/src/app/api/health/__tests__/route.test.ts index 691c9c6fe..d3865a248 100644 --- a/src/app/api/health/__tests__/route.test.ts +++ b/src/app/api/health/__tests__/route.test.ts @@ -140,3 +140,26 @@ describe("GET /api/health while the encryption key does not match", () => { expect(body).toEqual({ status: "ok" }); }); }); + +describe("GET /api/health under ENCRYPTION_KEY_CHECK=warn", () => { + it("names the mismatch as a warning and stays ok", async () => { + const { setKeyMismatchWarning } = + await import("@/lib/boot/key-mismatch-state"); + vi.stubEnv("HEALTHLOG_PROCESS_TYPE", "web"); + getGlobalBoss.mockReturnValue({}); + setKeyMismatchWarning({ + keyIds: ["v1"], + detectedAt: "2026-10-03T00:00:00Z", + }); + try { + const { response, body } = await health(); + expect(response.status).toBe(200); + expect(body).toEqual({ + status: "ok", + warning: "encryption_key_mismatch", + }); + } finally { + setKeyMismatchWarning(null); + } + }); +}); diff --git a/src/app/api/health/route.ts b/src/app/api/health/route.ts index 2b27876a6..2e7764036 100644 --- a/src/app/api/health/route.ts +++ b/src/app/api/health/route.ts @@ -7,6 +7,7 @@ import { NextResponse } from "next/server"; import { apiHandler } from "@/lib/api-handler"; import { annotate } from "@/lib/logging/context"; import { + getKeyMismatchWarning, isKeyMismatch, KEY_MISMATCH_HEALTH_REASON, } from "@/lib/boot/key-mismatch-state"; @@ -31,6 +32,12 @@ export const GET = apiHandler(async () => { // it is the one place a catalog install's status page reads. const keyMismatch = isKeyMismatch(); const reason = keyMismatch ? KEY_MISMATCH_HEALTH_REASON : undefined; + // Under ENCRYPTION_KEY_CHECK=warn the same finding does not refuse; it is + // named as a warning and leaves the status alone. + const warning = + !keyMismatch && getKeyMismatchWarning() + ? KEY_MISMATCH_HEALTH_REASON + : undefined; const status = !keyMismatch && dbOk && @@ -51,6 +58,7 @@ export const GET = apiHandler(async () => { { status, ...(reason ? { reason } : {}), + ...(warning ? { warning } : {}), timestamp: new Date().toISOString(), database: dbOk ? "connected" : "disconnected", worker: worker.running ? "running" : "stopped", @@ -63,7 +71,7 @@ export const GET = apiHandler(async () => { } return NextResponse.json( - { status, ...(reason ? { reason } : {}) }, + { status, ...(reason ? { reason } : {}), ...(warning ? { warning } : {}) }, { status: statusCode, headers: cacheHeaders }, ); }); diff --git a/src/app/api/mcp/oauth/authorize/route.ts b/src/app/api/mcp/oauth/authorize/route.ts index b0ad1ef5e..c8a71d9a8 100644 --- a/src/app/api/mcp/oauth/authorize/route.ts +++ b/src/app/api/mcp/oauth/authorize/route.ts @@ -29,6 +29,7 @@ import { NextRequest } from "next/server"; import { z } from "zod/v4"; import { annotate } from "@/lib/logging/context"; +import { refuseOAuthOnKeyMismatch } from "@/lib/boot/key-mismatch-refusal"; import { withBackgroundEvent } from "@/lib/logging/background"; import { auditLog } from "@/lib/auth/audit"; import { getSession } from "@/lib/auth/session"; @@ -344,10 +345,15 @@ function ensureAuthorizeCsp(response: Response): Response { } export async function GET(request: NextRequest): Promise { + // The boot key check refused this process: no grant is issued or stored. + const refusal = refuseOAuthOnKeyMismatch(); + if (refusal) return ensureAuthorizeCsp(refusal); return ensureAuthorizeCsp(await authorizeGet(request)); } export async function POST(request: NextRequest): Promise { + const refusal = refuseOAuthOnKeyMismatch(); + if (refusal) return ensureAuthorizeCsp(refusal); return ensureAuthorizeCsp(await authorizePost(request)); } diff --git a/src/app/api/mcp/oauth/register/route.ts b/src/app/api/mcp/oauth/register/route.ts index aa5d76a83..7e9e1d13c 100644 --- a/src/app/api/mcp/oauth/register/route.ts +++ b/src/app/api/mcp/oauth/register/route.ts @@ -16,6 +16,7 @@ import { NextRequest } from "next/server"; import { z } from "zod/v4"; import { annotate } from "@/lib/logging/context"; +import { refuseOAuthOnKeyMismatch } from "@/lib/boot/key-mismatch-refusal"; import { withBackgroundEvent } from "@/lib/logging/background"; import { checkAuthSurfaceRateLimit, rateLimitHeaders } from "@/lib/rate-limit"; import { isApiGloballyEnabled } from "@/lib/app-settings"; @@ -68,6 +69,10 @@ function oauthError( export async function POST(request: NextRequest): Promise { return withBackgroundEvent("mcp.oauth.register", async () => { + // The boot key check refused this process: mint nothing, write nothing. + const refusal = refuseOAuthOnKeyMismatch(); + if (refusal) return refusal; + const rl = await checkAuthSurfaceRateLimit( request, "mcp:oauth:register", diff --git a/src/app/api/mcp/oauth/token/route.ts b/src/app/api/mcp/oauth/token/route.ts index a82b732a5..fbdaf01af 100644 --- a/src/app/api/mcp/oauth/token/route.ts +++ b/src/app/api/mcp/oauth/token/route.ts @@ -32,6 +32,7 @@ import { NextRequest } from "next/server"; import type { Prisma } from "@/generated/prisma/client"; import { prisma } from "@/lib/db"; import { annotate } from "@/lib/logging/context"; +import { refuseOAuthOnKeyMismatch } from "@/lib/boot/key-mismatch-refusal"; import { withBackgroundEvent } from "@/lib/logging/background"; import { auditLog } from "@/lib/auth/audit"; import { issueApiToken } from "@/lib/auth/issue-token"; @@ -201,6 +202,10 @@ async function finalizeTokenPair( export async function POST(request: NextRequest): Promise { return withBackgroundEvent("mcp.oauth.token", async () => { + // The boot key check refused this process: mint nothing, write nothing. + const refusal = refuseOAuthOnKeyMismatch(); + if (refusal) return refusal; + // M1 — fail closed without a pinned origin; M4 — honour the operator's // global API kill-switch. Either off and the bridge mints no tokens. if (!isMcpOriginConfigured() || !(await isApiGloballyEnabled())) { diff --git a/src/app/key-mismatch-page.tsx b/src/app/key-mismatch-page.tsx index 1d95eaf24..5517f1964 100644 --- a/src/app/key-mismatch-page.tsx +++ b/src/app/key-mismatch-page.tsx @@ -31,6 +31,11 @@ export function KeyMismatchPage({ locale }: { locale: Locale }) {

{t("admin.keyMismatch.restartNote")}

+

+ {t("admin.keyMismatch.lastResort", { + setting: "ENCRYPTION_KEY_CHECK=warn", + })} +

{t("admin.keyMismatch.codes", { code: "encryption.key_mismatch", diff --git a/src/app/mcp/route.ts b/src/app/mcp/route.ts index c71635dc8..30b4b1738 100644 --- a/src/app/mcp/route.ts +++ b/src/app/mcp/route.ts @@ -50,6 +50,7 @@ import { resolveMcpAuthContext, SCOPE_HEALTH_READ, } from "@/lib/mcp"; +import { refuseOnKeyMismatch } from "@/lib/boot/key-mismatch-refusal"; import { withBackgroundEvent } from "@/lib/logging/background"; import { annotate } from "@/lib/logging/context"; import { isModuleEnabled } from "@/lib/modules/gate"; @@ -135,6 +136,12 @@ function readBearer(request: Request): string | null { async function handleMcp(request: Request): Promise { return withBackgroundEvent("mcp.request", async () => { + // 00. KEY CHECK — the boot key check found that the configured encryption + // key does not open this database. The write tools would seal new + // rows under it, so refuse before anything else, as apiHandler does. + const refusal = refuseOnKeyMismatch(); + if (refusal) return refusal; + // 0. ORIGIN — refuse a cross-origin browser request before anything else. if (!originAllowed(request)) { annotate({ action: { name: "mcp.origin.rejected" } }); diff --git a/src/lib/api-handler.ts b/src/lib/api-handler.ts index a47b66ab2..4f00f4b9e 100644 --- a/src/lib/api-handler.ts +++ b/src/lib/api-handler.ts @@ -53,11 +53,7 @@ import { assertRecordSessionFence, attachRecordContextEcho, } from "./sharing/record-session-fence"; -import { - isKeyMismatch, - KEY_MISMATCH_ERROR_CODE, - KEY_MISMATCH_EXEMPT_PATHS, -} from "./boot/key-mismatch-state"; +import { refuseOnKeyMismatch } from "./boot/key-mismatch-refusal"; import { capturedRateLimit, rateLimitResponseHeaders, @@ -266,17 +262,9 @@ export function apiHandler Promise>( // The boot key check found that the configured encryption key // cannot open this database. Refuse before auth, before the // handler: nothing here may read or write encrypted data. - if (isKeyMismatch() && !KEY_MISMATCH_EXEMPT_PATHS.has(url.pathname)) { - annotate({ action: { name: "encryption.key_mismatch.refused" } }); - response = NextResponse.json( - { - data: null, - error: - "The server's encryption key does not match its database. The operator has to restore the original key or point the server at the matching database.", - meta: { errorCode: KEY_MISMATCH_ERROR_CODE }, - }, - { status: 503, headers: { "Retry-After": "300" } }, - ); + const refusal = refuseOnKeyMismatch(url.pathname); + if (refusal) { + response = refusal; } else { response = await handler(...args); } diff --git a/src/lib/boot/key-check.ts b/src/lib/boot/key-check.ts index 12c0c22bb..96b42100c 100644 --- a/src/lib/boot/key-check.ts +++ b/src/lib/boot/key-check.ts @@ -13,17 +13,50 @@ import { checkEncryptionKeyCanaries, keyMismatchLogBlock, type CanaryClient, + type InconclusiveReason, } from "@/lib/crypto/canary"; -import { setKeyMismatchState } from "./key-mismatch-state"; +import { + getKeyCheckMode, + setKeyMismatchState, + setKeyMismatchWarning, +} from "./key-mismatch-state"; + +const INCONCLUSIVE_EXPLANATION: Record = { + "single-value": "the only stored value found under that id did not open.", + mixed: + "the oldest stored values under that id do not open, newer ones do. " + + "That is what a wrong key that served for a while leaves behind; check " + + "that this is the key the database was first written with.", + unsampled: + "the values stored under that id are too large to probe at start-up.", + incomplete: + "the probe of the stored data did not finish in time or a table could " + + "not be read.", +}; export async function runBootKeyCheck(client: CanaryClient): Promise { - const outcome = await checkEncryptionKeyCanaries(client); + const mode = getKeyCheckMode(); + const outcome = await checkEncryptionKeyCanaries(client, { mode }); + setKeyMismatchWarning(null); if (outcome.state === "mismatch") { - setKeyMismatchState({ + const state = { keyIds: outcome.keyIds, detectedAt: new Date().toISOString(), - }); + }; console.error(keyMismatchLogBlock(outcome.keyIds)); + if (mode === "warn") { + // The operator turned the refusal off. Say so next to the block, keep + // the finding for /api/health, and serve. + setKeyMismatchState(null); + setKeyMismatchWarning(state); + console.error( + "[boot] ENCRYPTION_KEY_CHECK=warn: serving anyway. No key is recorded " + + "while the check does not pass. Remove the setting once the key is " + + "confirmed.", + ); + return false; + } + setKeyMismatchState(state); return true; } setKeyMismatchState(null); @@ -34,11 +67,17 @@ export async function runBootKeyCheck(client: CanaryClient): Promise { `[boot] Encryption key check skipped: ${outcome.message.slice(0, 300)}`, ); } else { - if (outcome.inconclusive.length > 0) { + for (const { keyId, reason } of outcome.inconclusive) { console.warn( - `[boot] Encryption key check inconclusive for key id(s) ${outcome.inconclusive.join(", ")}: ` + - "the only stored value found under that id did not open. Serving; " + - "no canary written; the check runs again at the next start.", + `[boot] Encryption key check inconclusive for key id ${keyId}: ` + + `${INCONCLUSIVE_EXPLANATION[reason]} Serving; no key recorded; ` + + "the check runs again at the next start.", + ); + } + if (mode === "warn" && outcome.inconclusive.length > 0) { + console.warn( + "[boot] ENCRYPTION_KEY_CHECK=warn: no key recorded on this start " + + "because the check did not pass for every key id.", ); } if (outcome.written.length > 0) { diff --git a/src/lib/boot/key-mismatch-refusal.ts b/src/lib/boot/key-mismatch-refusal.ts new file mode 100644 index 000000000..10533a122 --- /dev/null +++ b/src/lib/boot/key-mismatch-refusal.ts @@ -0,0 +1,67 @@ +/** + * The refusal a request surface answers while the boot key check has found + * that the configured encryption key does not open this database. + * + * `apiHandler` applies it to every API route. A route outside `apiHandler` + * (the remote MCP endpoint, its OAuth bridge) calls one of the two helpers + * below as the first thing it does, before auth and before any read or + * write: a process holding the wrong key must never seal new rows, because + * those rows would open under that key at the next boot and read as evidence + * that it is the right one. `key-mismatch-refusal-guard.test.ts` fails when a + * route outside `apiHandler` does neither. + */ +import { annotate } from "@/lib/logging/context"; +import { + isKeyMismatch, + KEY_MISMATCH_ERROR_CODE, + KEY_MISMATCH_EXEMPT_PATHS, +} from "./key-mismatch-state"; + +export const KEY_MISMATCH_MESSAGE = + "The server's encryption key does not match its database. The operator has to restore the original key or point the server at the matching database."; + +const RETRY_AFTER = { "Retry-After": "300" }; + +/** The 503 in the standard `{ data, error, meta }` envelope. */ +export function keyMismatchResponse(): Response { + annotate({ action: { name: "encryption.key_mismatch.refused" } }); + return Response.json( + { + data: null, + error: KEY_MISMATCH_MESSAGE, + meta: { errorCode: KEY_MISMATCH_ERROR_CODE }, + }, + { status: 503, headers: RETRY_AFTER }, + ); +} + +/** + * The envelope refusal when the process refuses and `pathname` is not one of + * the exempt paths; otherwise null and the caller carries on. + */ +export function refuseOnKeyMismatch(pathname?: string): Response | null { + if (!isKeyMismatch()) return null; + if (pathname !== undefined && KEY_MISMATCH_EXEMPT_PATHS.has(pathname)) { + return null; + } + return keyMismatchResponse(); +} + +/** + * The same refusal in the RFC 6749 error shape the OAuth endpoints speak: + * `temporarily_unavailable` with a description, plus the HealthLog error + * code as an extension member a client may ignore. + */ +export function refuseOAuthOnKeyMismatch(): Response | null { + if (!isKeyMismatch()) return null; + annotate({ action: { name: "encryption.key_mismatch.refused" } }); + return Response.json( + { + error: "temporarily_unavailable", + error_description: + "The server's encryption key does not match its database.", + error_code: KEY_MISMATCH_ERROR_CODE, + }, + { status: 503, headers: { ...RETRY_AFTER, "Cache-Control": "no-store" } }, + ); +} diff --git a/src/lib/boot/key-mismatch-state.ts b/src/lib/boot/key-mismatch-state.ts index 4f9864009..8e41c7016 100644 --- a/src/lib/boot/key-mismatch-state.ts +++ b/src/lib/boot/key-mismatch-state.ts @@ -22,11 +22,30 @@ export interface KeyMismatchState { } const STATE_SYMBOL = Symbol.for("healthlog.encryptionKeyMismatch"); +const WARNING_SYMBOL = Symbol.for("healthlog.encryptionKeyMismatchWarning"); type GlobalWithState = typeof globalThis & { [STATE_SYMBOL]?: KeyMismatchState | null; + [WARNING_SYMBOL]?: KeyMismatchState | null; }; +/** + * `ENCRYPTION_KEY_CHECK`: `enforce` (the default, and what any other value + * means) refuses on a mismatch; `warn` runs the check, logs it, reports it as + * a warning on `/api/health`, and serves anyway. `warn` is the operator's way + * out of a refusal they are certain is false at a first boot, where there is + * no recorded key yet whose row they could delete. It disables a safety + * check: a process with the wrong key then serves errors on every encrypted + * value and writes new rows under that key. + */ +export type KeyCheckMode = "enforce" | "warn"; + +export function getKeyCheckMode(): KeyCheckMode { + return process.env.ENCRYPTION_KEY_CHECK?.trim().toLowerCase() === "warn" + ? "warn" + : "enforce"; +} + /** The paths that keep answering while the process refuses everything else. */ export const KEY_MISMATCH_EXEMPT_PATHS: ReadonlySet = new Set([ "/api/health", @@ -50,3 +69,15 @@ export function isKeyMismatch(): boolean { export function setKeyMismatchState(state: KeyMismatchState | null): void { (globalThis as GlobalWithState)[STATE_SYMBOL] = state; } + +/** + * A mismatch the check found while `ENCRYPTION_KEY_CHECK=warn`: recorded so + * `/api/health` can name it as a warning, never read by the refusal. + */ +export function getKeyMismatchWarning(): KeyMismatchState | null { + return (globalThis as GlobalWithState)[WARNING_SYMBOL] ?? null; +} + +export function setKeyMismatchWarning(state: KeyMismatchState | null): void { + (globalThis as GlobalWithState)[WARNING_SYMBOL] = state; +} diff --git a/src/lib/crypto/__tests__/canary.test.ts b/src/lib/crypto/__tests__/canary.test.ts index 170e1e5bf..f816bbb4b 100644 --- a/src/lib/crypto/__tests__/canary.test.ts +++ b/src/lib/crypto/__tests__/canary.test.ts @@ -1,17 +1,30 @@ import { beforeEach, describe, expect, it, vi } from "vitest"; +import { PrismaClient } from "@/generated/prisma/client"; +import { PrismaPg } from "@prisma/adapter-pg"; import { _resetCryptoCacheForTests, candidateMatchesKey, encrypt, + encryptBytes, encryptUnderKeyId, + extractKeyIdFromBytes, fingerprintKeyBytes, getKeyFingerprint, } from "@/lib/crypto"; +import { encryptToBytes } from "@/lib/ai/coach/bytes-codec"; +import { + ENCRYPTED_COLUMNS, + WORKOUT_ROUTE_GEOMETRY_AAD, + encryptedColumnKey, +} from "@/lib/crypto/encrypted-columns"; import { canaryPlaintext, checkEncryptionKeyCanaries, keyMismatchLogBlock, + probeValueOpens, + resolveColumnLocation, type CanaryClient, + type ColumnSampler, } from "../canary"; const KEY_A = @@ -19,23 +32,49 @@ const KEY_A = const KEY_B = "fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210"; -function useKey(hex: string) { +function configureKeys(keys: Record, active: string) { + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify(keys)); + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", active); + vi.stubEnv("ENCRYPTION_KEY", ""); + _resetCryptoCacheForTests(); +} + +function configureKey(hex: string) { vi.stubEnv("ENCRYPTION_KEYS", ""); vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", ""); vi.stubEnv("ENCRYPTION_KEY", hex); _resetCryptoCacheForTests(); } -/** An in-memory stand-in for the two raw statements and the probe delegates. */ +type Seeded = + | string + | Uint8Array + | { value: string | Uint8Array | null; ts?: string; codec?: string }; + +/** + * An in-memory stand-in for the raw statements, plus a sampler over seeded + * values keyed by `Model.field`. A seeded value's `ts` orders it; values + * without one keep their seeded order after every dated value, as the SQL + * sampler orders them. + */ function fakeClient(opts: { canaries?: Record; - /** delegate → field → stored values (in row order). */ - probe?: Record>; + probe?: Record; + /** Make the sampler throw for these columns. */ + unreadable?: string[]; + /** Another process seals this id (with this ciphertext) just before us. */ + race?: { keyId: string; ciphertext: string }; }) { const canaries = new Map(Object.entries(opts.canaries ?? {})); const inserts: string[] = []; + let raced = false; const client = { - async $queryRaw() { + async $queryRaw(_q: TemplateStringsArray, ...values: unknown[]) { + const [arg] = values; + if (typeof arg === "string") { + const c = canaries.get(arg); + return c ? [{ ciphertext: c }] : []; + } return [...canaries.entries()].map(([key_id, ciphertext]) => ({ key_id, ciphertext, @@ -43,104 +82,118 @@ function fakeClient(opts: { }, async $executeRaw(_q: TemplateStringsArray, ...values: unknown[]) { const [keyId, ciphertext] = values as [string, string]; + if (opts.race && !raced && opts.race.keyId === keyId) { + raced = true; + canaries.set(keyId, opts.race.ciphertext); + } inserts.push(keyId); - if (!canaries.has(keyId)) canaries.set(keyId, ciphertext); + if (canaries.has(keyId)) return 0; + canaries.set(keyId, ciphertext); return 1; }, - } as unknown as CanaryClient & Record; - for (const [delegate, fields] of Object.entries(opts.probe ?? {})) { - client[delegate] = { - async findMany(args: { - select: Record; - take: number; - skip?: number; - }) { - const field = Object.keys(args.select)[0]; - const skip = args.skip ?? 0; - return (fields[field] ?? []) - .slice(skip, skip + args.take) - .map((value) => ({ [field]: value })); - }, - }; - } - return { client, canaries, inserts }; + } as unknown as CanaryClient; + const sampler: ColumnSampler = async (column, keyId, limit) => { + const key = `${column.model}.${column.field}`; + if (opts.unreadable?.includes(key)) throw new Error("relation missing"); + const rows = (opts.probe?.[key] ?? []).map((v) => + typeof v === "string" || v instanceof Uint8Array + ? { value: v, ts: null as Date | null, codec: null } + : { + value: v.value, + ts: v.ts ? new Date(v.ts) : null, + codec: v.codec ?? null, + }, + ); + const under = rows.filter((r) => { + if (r.value === null) return true; + if (typeof r.value === "string") return r.value.startsWith(`${keyId}.`); + const b = Buffer.from(r.value); + return ( + b.toString("latin1").startsWith(`${keyId}.`) || + extractKeyIdFromBytes(b) === keyId + ); + }); + return under + .map((r, i) => ({ r, i })) + .sort((a, b) => { + const ta = a.r.ts?.getTime() ?? Infinity; + const tb = b.r.ts?.getTime() ?? Infinity; + return ta === tb ? a.i - b.i : ta - tb; + }) + .slice(0, limit) + .map(({ r }) => r); + }; + const check = (budget?: number) => + checkEncryptionKeyCanaries(client, { sampler, probeBudgetMs: budget }); + return { client, sampler, canaries, inserts, check }; } +const ok = (fields: { + written?: string[]; + verified?: string[]; + inconclusive?: Array<{ keyId: string; reason: string }>; +}) => ({ + state: "ok", + written: fields.written ?? [], + verified: fields.verified ?? [], + inconclusive: fields.inconclusive ?? [], +}); + describe("encryption key canary", () => { beforeEach(() => { vi.unstubAllEnvs(); - useKey(KEY_A); + configureKey(KEY_A); }); - it("writes a canary for a configured key with no row and no data", async () => { - const { client, canaries, inserts } = fakeClient({}); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ - state: "ok", - written: ["v1"], - verified: [], - inconclusive: [], - }); + it("writes a canary for a configured key when no column holds a value under its id", async () => { + const { canaries, inserts, check } = fakeClient({}); + expect(await check()).toEqual(ok({ written: ["v1"] })); expect(inserts).toEqual(["v1"]); expect(canaries.get("v1")?.startsWith("v1.")).toBe(true); }); it("verifies a canary written under the same key", async () => { - const { client, inserts } = fakeClient({ + const { inserts, check } = fakeClient({ canaries: { v1: encryptUnderKeyId(canaryPlaintext("v1"), "v1") }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ - state: "ok", - written: [], - verified: ["v1"], - inconclusive: [], - }); + expect(await check()).toEqual(ok({ verified: ["v1"] })); expect(inserts).toEqual([]); }); it("reports a mismatch when the key changed under the same id", async () => { const sealedUnderA = encryptUnderKeyId(canaryPlaintext("v1"), "v1"); - useKey(KEY_B); - const { client, inserts } = fakeClient({ canaries: { v1: sealedUnderA } }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + configureKey(KEY_B); + const { inserts, check } = fakeClient({ canaries: { v1: sealedUnderA } }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); expect(inserts).toEqual([]); }); it("reports a mismatch for a canary that opens to the wrong value", async () => { // A row from another key id copied over: opens, but says the wrong thing. - const { client } = fakeClient({ + const { check } = fakeClient({ canaries: { v1: encryptUnderKeyId(canaryPlaintext("v2"), "v1") }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome.state).toBe("mismatch"); + expect((await check()).state).toBe("mismatch"); }); it("refuses to seal a wrong key when existing data predates the canary", async () => { const existing = [encrypt("a stored token"), encrypt("another token")]; - useKey(KEY_B); - const { client, inserts } = fakeClient({ + configureKey(KEY_B); + const { inserts, check } = fakeClient({ probe: { - user: { - codexAccessTokenEncrypted: [existing[0]], - codexRefreshTokenEncrypted: [existing[1]], - }, + "User.codexAccessTokenEncrypted": [existing[0]], + "User.codexRefreshTokenEncrypted": [existing[1]], }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); expect(inserts).toEqual([]); }); it("writes the canary when existing data opens under the key", async () => { - const { client, inserts } = fakeClient({ - probe: { - user: { codexAccessTokenEncrypted: [encrypt("a stored token")] }, - }, + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [encrypt("a stored token")] }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome.state).toBe("ok"); + expect((await check()).state).toBe("ok"); expect(inserts).toEqual(["v1"]); }); @@ -148,46 +201,270 @@ describe("encryption key canary", () => { // A row that claims key id v1 but does not open (damaged, or sealed by // something else), found FIRST, then values that open. const damaged = "v1." + Buffer.alloc(40, 7).toString("base64"); - const { client, inserts } = fakeClient({ + const { inserts, check } = fakeClient({ probe: { - user: { - codexAccessTokenEncrypted: [damaged], - codexRefreshTokenEncrypted: [encrypt("a stored token")], - }, + "User.codexAccessTokenEncrypted": [damaged], + "User.codexRefreshTokenEncrypted": [encrypt("a stored token")], }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome.state).toBe("ok"); + expect((await check()).state).toBe("ok"); expect(inserts).toEqual(["v1"]); }); it("several values in one column count, so a wrong key with one busy column still refuses", async () => { const values = [encrypt("one"), encrypt("two"), encrypt("three")]; - useKey(KEY_B); - const { client, inserts } = fakeClient({ - probe: { user: { codexAccessTokenEncrypted: values } }, + configureKey(KEY_B); + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": values }, }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); expect(inserts).toEqual([]); }); it("a single value that does not open is inconclusive: no mismatch, no canary", async () => { const existing = encrypt("the only stored token"); - useKey(KEY_B); - const { client, inserts } = fakeClient({ - probe: { user: { codexAccessTokenEncrypted: [existing] } }, - }); - const outcome = await checkEncryptionKeyCanaries(client); - expect(outcome).toEqual({ - state: "ok", - written: [], - verified: [], - inconclusive: ["v1"], + configureKey(KEY_B); + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [existing] }, }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "single-value" }] }), + ); expect(inserts).toEqual([]); }); + describe("Bytes columns", () => { + it("a wrong key over Bytes-only ciphertext refuses instead of sealing", async () => { + const notes = [encryptToBytes("note one"), encryptToBytes("note two")]; + configureKey(KEY_B); + const { inserts, check } = fakeClient({ + probe: { + "MoodEntry.noteEncrypted": [notes[0]], + "Measurement.notesEncrypted": [notes[1]], + }, + }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); + expect(inserts).toEqual([]); + }); + + it("opens the binary layout with its label, and the UTF-8 layout", async () => { + const route = encryptBytes(Buffer.from("{}"), WORKOUT_ROUTE_GEOMETRY_AAD); + const { inserts, check } = fakeClient({ + probe: { + "WorkoutRoute.geometryEncrypted": [ + { value: route, ts: "2024-01-01" }, + ], + "MoodEntry.noteEncrypted": [ + { value: encryptToBytes("a note"), ts: "2024-01-02" }, + ], + }, + }); + expect(await check()).toEqual(ok({ written: ["v1"] })); + expect(inserts).toEqual(["v1"]); + }); + + it("a binary value under the wrong label does not count as opening", () => { + const sealed = encryptBytes(Buffer.from("x"), "some/other/label"); + const column = ENCRYPTED_COLUMNS.find((c) => c.model === "WorkoutRoute")!; + expect(probeValueOpens(column, { value: sealed, ts: null }, "v1")).toBe( + false, + ); + }); + + it("dispatches a codec column by the row's codec", () => { + const column = ENCRYPTED_COLUMNS.find( + (c) => c.model === "InboundDocument" && c.field === "contentEncrypted", + )!; + const binary = encryptBytes(Buffer.from("%PDF")); + const text = encryptToBytes("JVBERg=="); + expect( + probeValueOpens( + column, + { value: binary, codec: "binary2", ts: null }, + "v1", + ), + ).toBe(true); + expect( + probeValueOpens( + column, + { value: text, codec: "base64v1", ts: null }, + "v1", + ), + ).toBe(true); + }); + }); + + describe("oldest values decide", () => { + function seedOldAThenNewB() { + const old = [ + encrypt("old 1"), + encrypt("old 2"), + encryptToBytes("old note 1"), + encryptToBytes("old note 2"), + ]; + configureKey(KEY_B); + const recent = [ + encrypt("new 1"), + encrypt("new 2"), + encryptToBytes("new note 1"), + encryptToBytes("new note 2"), + ]; + configureKey(KEY_A); + return fakeClient({ + probe: { + "User.codexAccessTokenEncrypted": [ + { value: recent[0], ts: "2026-09-01" }, + { value: old[0], ts: "2024-01-01" }, + ], + "User.codexRefreshTokenEncrypted": [ + { value: old[1], ts: "2024-01-02" }, + { value: recent[1], ts: "2026-09-02" }, + ], + "MoodEntry.noteEncrypted": [ + { value: recent[2], ts: "2026-09-03" }, + { value: old[2], ts: "2024-01-03" }, + ], + "Measurement.notesEncrypted": [ + { value: old[3], ts: "2024-01-04" }, + { value: recent[3], ts: "2026-09-04" }, + ], + }, + }); + } + + it("the key that wrote only the newer rows is never sealed", async () => { + const fake = seedOldAThenNewB(); + configureKey(KEY_B); + const outcome = await fake.check(); + expect(outcome.state).not.toBe("error"); + if (outcome.state === "ok") expect(outcome.written).toEqual([]); + expect(fake.canaries.size).toBe(0); + }); + + it("the key that wrote the oldest rows is sealed, newer unreadable rows notwithstanding", async () => { + const fake = seedOldAThenNewB(); + expect(await fake.check()).toEqual(ok({ written: ["v1"] })); + }); + + it("two oldest values that fail before one that opens are mixed: inconclusive", async () => { + const opens = encrypt("opens"); + configureKey(KEY_B); + const foreign = [encrypt("x"), encrypt("y")]; + configureKey(KEY_A); + const { inserts, check } = fakeClient({ + probe: { + "User.codexAccessTokenEncrypted": [ + { value: foreign[0], ts: "2024-01-01" }, + ], + "User.codexRefreshTokenEncrypted": [ + { value: foreign[1], ts: "2024-01-02" }, + ], + "MoodEntry.noteEncrypted": [{ value: opens, ts: "2025-01-01" }], + }, + }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "mixed" }] }), + ); + expect(inserts).toEqual([]); + }); + }); + + describe("nothing unproven seals", () => { + it("a probe out of time is inconclusive, even over data that opens", async () => { + const { inserts, check } = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [encrypt("opens")] }, + }); + expect(await check(-1)).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "incomplete" }] }), + ); + expect(inserts).toEqual([]); + }); + + it("a column that cannot be read is inconclusive, never a fresh install", async () => { + const { inserts, check } = fakeClient({ + unreadable: ["MoodEntry.noteEncrypted"], + }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "incomplete" }] }), + ); + expect(inserts).toEqual([]); + }); + + it("values too large to fetch are present: inconclusive, not fresh", async () => { + const { inserts, check } = fakeClient({ + probe: { "InboundDocument.contentEncrypted": [{ value: null }] }, + }); + expect(await check()).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "unsampled" }] }), + ); + expect(inserts).toEqual([]); + }); + }); + + describe("ENCRYPTION_KEY_CHECK=warn", () => { + it("records nothing for any key id while one is inconclusive", async () => { + configureKeys({ v1: KEY_A, v2: KEY_B }, "v2"); + const lone = encryptUnderKeyId("only value", "v1"); + configureKeys({ v1: KEY_B, v2: KEY_A }, "v2"); + const fake = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [lone] }, + }); + // Enforce records the proven id beside the inconclusive one. + expect(await fake.check()).toEqual( + ok({ + written: ["v2"], + inconclusive: [{ keyId: "v1", reason: "single-value" }], + }), + ); + const warn = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [lone] }, + }); + expect( + await checkEncryptionKeyCanaries(warn.client, { + sampler: warn.sampler, + mode: "warn", + }), + ).toEqual( + ok({ inconclusive: [{ keyId: "v1", reason: "single-value" }] }), + ); + expect(warn.inserts).toEqual([]); + }); + + it("records a key that passes for every id, as enforce does", async () => { + const warn = fakeClient({ + probe: { "User.codexAccessTokenEncrypted": [encrypt("opens")] }, + }); + expect( + await checkEncryptionKeyCanaries(warn.client, { + sampler: warn.sampler, + mode: "warn", + }), + ).toEqual(ok({ written: ["v1"] })); + }); + }); + + describe("the first-seal race", () => { + it("losing it to a process with another key is a mismatch", async () => { + configureKey(KEY_B); + const otherCanary = encryptUnderKeyId(canaryPlaintext("v1"), "v1"); + configureKey(KEY_A); + const { check } = fakeClient({ + race: { keyId: "v1", ciphertext: otherCanary }, + }); + expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] }); + }); + + it("losing it to a process with the same key verifies", async () => { + const { check } = fakeClient({ + race: { + keyId: "v1", + ciphertext: encryptUnderKeyId(canaryPlaintext("v1"), "v1"), + }, + }); + expect(await check()).toEqual(ok({ verified: ["v1"] })); + }); + }); + it("reports an error, not a mismatch, when the table cannot be read", async () => { const client = { $queryRaw: async () => { @@ -202,22 +479,49 @@ describe("encryption key canary", () => { it("reports an error when no key is configured", async () => { vi.stubEnv("ENCRYPTION_KEY", ""); _resetCryptoCacheForTests(); - const outcome = await checkEncryptionKeyCanaries(fakeClient({}).client); + const outcome = await fakeClient({}).check(); expect(outcome.state).toBe("error"); }); - it("names the key ids and never a key in the log block", () => { + it("names the key ids, the reset statement, and never a key in the log block", () => { const block = keyMismatchLogBlock(["v1"]); expect(block).toContain("'v1'"); expect(block).toContain("encryption.key_mismatch"); + expect(block).toContain( + "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';", + ); expect(block).not.toContain(KEY_A); }); }); +describe("the SQL sampler's column map", () => { + it("resolves every registered column to a table, a column and an order", () => { + // A throwaway client: never connects, only exposes the data model the + // probe reads table and column names from. + const client = new PrismaClient({ + adapter: new PrismaPg({ + connectionString: "postgres://x:y@127.0.0.1:1/x", + }), + }); + for (const column of ENCRYPTED_COLUMNS) { + const loc = resolveColumnLocation(client, column); + expect(loc.table, encryptedColumnKey(column)).toMatch(/^[a-z_0-9]+$/); + expect(loc.column, encryptedColumnKey(column)).toMatch(/^[a-z_0-9]+$/); + if (column.codecField) expect(loc.codec).not.toBeNull(); + } + const backupPieces = ENCRYPTED_COLUMNS.find( + (c) => c.model === "DataBackupChunk", + )!; + expect(resolveColumnLocation(client, backupPieces).ts).toBeNull(); + const user = ENCRYPTED_COLUMNS.find((c) => c.model === "User")!; + expect(resolveColumnLocation(client, user).ts).toBe("updated_at"); + }); +}); + describe("key fingerprint and copy check", () => { beforeEach(() => { vi.unstubAllEnvs(); - useKey(KEY_A); + configureKey(KEY_A); }); it("is the first 12 hex of SHA-256 over the raw key bytes", () => { diff --git a/src/lib/crypto/canary.ts b/src/lib/crypto/canary.ts index e4f77178e..9cb3b0ce5 100644 --- a/src/lib/crypto/canary.ts +++ b/src/lib/crypto/canary.ts @@ -12,44 +12,85 @@ * * - row present: it must decrypt to the expected value, else the key is not * the one the row was written with; - * - row absent: before writing one, sample existing ciphertext under that - * key id from the registered string columns and try it. Without this - * probe, the first boot after an upgrade (or a re-keyed boot over a - * database that predates the canary) would seal the wrong key as the - * right one. + * - row absent: before writing one, probe the ciphertext already stored + * under that key id. Without the probe, the first boot after an upgrade + * (or a re-keyed boot over a database that predates the canary) would + * seal the wrong key as the right one, and from then on refuse the + * right key. * - * The probe samples up to `PROBE_SAMPLE_LIMIT` values, one per column first - * (different columns are different writers, so one bad writer cannot speak - * for the whole database), then further values from columns that had any. - * Its verdict, and what each one does: + * ## The probe * - * - `opens`: at least one value opened. A wrong key opens nothing, so the - * key is right and any value that failed is a damaged or foreign row, not - * a key problem. The canary is written. + * Every registered column is asked, string and Bytes alike (`ENCRYPTED_COLUMNS`), + * for its OLDEST values under the key id: the `encrypt()` string form + * (`.`, or a bare legacy base64 value for the legacy id `v1`), + * the streamed form (`~hlgcm1..`), the same string stored as UTF-8 in a + * Bytes column, and the binary layout (`0x02 `). The answer for a + * column also tells whether it holds ANY value under the id, which is the + * existence check below. + * + * Oldest means the row's `updatedAt`, else its `createdAt`, else (no timestamp + * on the model, e.g. backup pieces) the primary key, ordered after every + * timestamped row. `updatedAt` rather than `createdAt` because a value is + * sealed when its row is written, and a row updated yesterday may carry a + * value sealed yesterday on a row created years ago; `updatedAt` is a ceiling + * on when the value was sealed. The point of the order: a process that served + * under a wrong key without a canary (an inconclusive boot, a version that + * predates this check) wrote rows that open under that wrong key. Those rows + * are by construction newer than the data the database was built with, so + * reading the oldest values first means they are never what decides. + * + * Up to `PROBE_ROWS_PER_COLUMN` values per column are fetched, the merged set + * is ordered by that timestamp, and the oldest `PROBE_SAMPLE_LIMIT` are tried. + * Values larger than `PROBE_VALUE_MAX_BYTES` (stored documents, large backups) + * count as present but are not fetched. The verdict: + * + * - `opens`: the oldest value that opens is preceded by at most ONE older + * value that does not. That one is a damaged or foreign row, not a key + * problem: a wrong key opens nothing. Values newer than the first opening + * one do not count against it; they are what a wrong-key process would + * have left behind. The canary is written. * - `fails`: two or more values were tried and none opened. That is what a * wrong key looks like; the process refuses (mismatch). - * - `inconclusive`: exactly one value was found and it did not open. One - * row cannot tell a wrong key from one damaged row, and the two possible - * mistakes are not equal: declaring a mismatch takes a correctly keyed - * server down with 503 on every request, while serving keeps today's - * behaviour. So the process serves, but writes NO canary either (that - * would seal a key that may be wrong), logs a warning, and probes again - * at the next boot, by which time more data may decide it. - * - `none`: nothing found inside the time budget. The canary is written, - * which is exactly the behaviour before this check existed. + * - `none`: every column answered, and none holds a single value under the + * key id. A genuinely fresh install (or a new id added for rotation); the + * canary is written. + * - `inconclusive`, with a reason: anything not proven. Only one value and + * it did not open (`single-value`); two or more of the oldest values do + * not open but a newer one does (`mixed`); values exist but none could be + * fetched (`unsampled`); the time budget ran out or a column could not be + * read (`incomplete`). The process serves, as before this check existed, + * writes NO canary, logs a warning, and probes again at the next boot. + * Serving rather than refusing because a refusal takes a correctly keyed + * server down with 503 on every request; not sealing because that would + * lock the right key out later. * - * Writes are `ON CONFLICT DO NOTHING`, so two processes racing is fine. + * The first seal can race another process. `ON CONFLICT DO NOTHING` keeps the + * row the other process wrote; when this process's insert affected no row, + * the stored canary is read back and must open under this process's key, or + * the outcome is a mismatch. * * Never logs or returns key material. The canary is written by raw SQL and is - * deliberately not in the rotation registry: it belongs to its key id. + * deliberately not in the rotation registry: it belongs to its key id, and the + * rotation script removes it once no ciphertext remains under that id + * (`retireCanariesWithoutData`). */ import { decrypt, + decryptBytes, + decryptStream, encryptUnderKeyId, extractKeyId, + extractKeyIdFromBytes, + extractStreamKeyId, + getActiveKeyId, getConfiguredKeyIds, + isStreamCiphertext, } from "@/lib/crypto"; -import { ENCRYPTED_COLUMNS } from "@/lib/crypto/encrypted-columns"; +import { + ENCRYPTED_COLUMNS, + encryptedColumnKey, + type EncryptedColumn, +} from "@/lib/crypto/encrypted-columns"; export const CANARY_PREFIX = "healthlog-canary:"; @@ -57,12 +98,21 @@ export function canaryPlaintext(keyId: string): string { return `${CANARY_PREFIX}${keyId}`; } -/** How many existing values the probe tries at most per key id. */ -export const PROBE_SAMPLE_LIMIT = 5; +/** How many of the oldest values the probe tries at most per key id. */ +export const PROBE_SAMPLE_LIMIT = 8; -/** How long the existing-data probe may take before the canary is written anyway. */ +/** How many of its oldest values each column contributes to the pool. */ +export const PROBE_ROWS_PER_COLUMN = 2; + +/** Values larger than this count as present but are not fetched. */ +export const PROBE_VALUE_MAX_BYTES = 2 * 1024 * 1024; + +/** How long the probe may take before it gives up as `inconclusive`. */ export const PROBE_BUDGET_MS = 10_000; +/** The key id the legacy bare-base64 string format belongs to. */ +const LEGACY_KEY_ID = "v1"; + /** The slice of a Prisma client this module needs. */ export interface CanaryClient { $queryRaw( @@ -73,31 +123,53 @@ export interface CanaryClient { query: TemplateStringsArray, ...values: unknown[] ): Promise; + $queryRawUnsafe?( + query: string, + ...values: unknown[] + ): Promise; + $executeRawUnsafe?(query: string, ...values: unknown[]): Promise; } -type ProbeDelegate = { - findMany: (args: { - where: Record; - select: Record; - take: number; - skip?: number; - }) => Promise>>; -}; +/** One stored value under the key id, as the probe sees it. */ +export interface ProbeRow { + /** The stored value; null when it exists but is too large to fetch. */ + value: string | Uint8Array | null; + /** The row's codec column, for a codec-dispatched column. */ + codec?: string | null; + /** The row's write time (see the module comment); null when it has none. */ + ts: Date | null; +} -export type ProbeVerdict = "opens" | "fails" | "inconclusive" | "none"; +/** + * Reads up to `limit` of the oldest values under `keyId` in one column, oldest + * first. Throws when the column cannot be read; an empty array means the + * column holds no value under that id. + */ +export type ColumnSampler = ( + column: EncryptedColumn, + keyId: string, + limit: number, +) => Promise; + +export type InconclusiveReason = + "single-value" | "mixed" | "unsampled" | "incomplete"; + +export type ProbeResult = + | { verdict: "opens" | "fails" | "none" } + | { verdict: "inconclusive"; reason: InconclusiveReason }; export type KeyCheckOutcome = | { state: "ok"; /** Key ids whose canary this run wrote. */ written: string[]; - /** Key ids whose canary existed and opened. */ + /** Key ids whose canary existed (or another process wrote) and opened. */ verified: string[]; /** - * Key ids with no canary whose only existing value did not open: not a - * mismatch, no canary written, probed again at the next boot. + * Key ids with no canary whose stored data proved nothing either way: + * not a mismatch, no canary written, probed again at the next boot. */ - inconclusive: string[]; + inconclusive: Array<{ keyId: string; reason: InconclusiveReason }>; } | { state: "mismatch"; keyIds: string[] } | { state: "error"; message: string }; @@ -112,77 +184,304 @@ function opensAs(ciphertext: string, keyId: string, expected?: string) { } } -/** - * Sample existing values sealed under `keyId` and try them; see the module - * comment for what each verdict means and why. - */ -export async function probeExistingData( - client: unknown, +function isBinaryLayout(column: EncryptedColumn, row: ProbeRow): boolean { + if (column.codec === "binary2") return true; + return column.codecField !== undefined && row.codec === "binary2"; +} + +/** Does one stored value open under `keyId`? Never throws. */ +export function probeValueOpens( + column: EncryptedColumn, + row: ProbeRow, keyId: string, - deadline: number, -): Promise { - const delegates = client as Record; - const columns = ENCRYPTED_COLUMNS.filter( - (c) => c.kind === "string" && !c.codec && !c.codecField, - ); - let tried = 0; - const columnsWithData: typeof columns = []; - - const sample = async ( - column: (typeof columns)[number], - skip: number, - take: number, - ): Promise => { - const delegate = - delegates[column.model.charAt(0).toLowerCase() + column.model.slice(1)]; - if (!delegate || typeof delegate.findMany !== "function") return []; - try { - const rows = await delegate.findMany({ - where: { [column.field]: { startsWith: `${keyId}.` } }, - select: { [column.field]: true }, - take, - ...(skip > 0 ? { skip } : {}), - }); - return rows - .map((row) => row[column.field]) - .filter((v): v is string => typeof v === "string"); - } catch { - return []; +): boolean { + const value = row.value; + if (value === null) return false; + try { + if (typeof value === "string") { + if (isStreamCiphertext(value)) { + if (extractStreamKeyId(value) !== keyId) return false; + decryptStream(value); + return true; + } + const id = extractKeyId(value); + if (id === null) { + // Legacy bare base64: only ever the legacy id's. + if (keyId !== LEGACY_KEY_ID) return false; + decrypt(value); + return true; + } + return opensAs(value, keyId); } - }; - - // Pass 1: one value per column, so different writers are heard first. - for (const column of columns) { - if (tried >= PROBE_SAMPLE_LIMIT || Date.now() > deadline) break; - const values = await sample(column, 0, 1); - if (values.length === 0) continue; - columnsWithData.push(column); - tried += 1; - if (opensAs(values[0], keyId)) return "opens"; - } - // Pass 2: more values from the columns that had any, up to the limit. - for (const column of columnsWithData) { - if (tried >= PROBE_SAMPLE_LIMIT || Date.now() > deadline) break; - const values = await sample(column, 1, PROBE_SAMPLE_LIMIT - tried); - for (const value of values) { - tried += 1; - if (opensAs(value, keyId)) return "opens"; + const buf = Buffer.from(value.buffer, value.byteOffset, value.byteLength); + if (isBinaryLayout(column, row)) { + if (extractKeyIdFromBytes(buf) !== keyId) return false; + decryptBytes(buf, column.aad); + return true; } + return opensAs(buf.toString("utf8"), keyId); + } catch { + return false; } - - if (tried >= 2) return "fails"; - if (tried === 1) return "inconclusive"; - return "none"; } +/** + * Probe the stored values under `keyId`; see the module comment for what each + * verdict means and why. + */ +export async function probeExistingData( + sampler: ColumnSampler, + keyId: string, + deadline: number, + columns: readonly EncryptedColumn[] = ENCRYPTED_COLUMNS, +): Promise { + const pool: Array<{ column: EncryptedColumn; row: ProbeRow; order: number }> = + []; + let present = 0; + for (const column of columns) { + if (Date.now() > deadline) { + return { verdict: "inconclusive", reason: "incomplete" }; + } + let rows: ProbeRow[]; + try { + rows = await sampler(column, keyId, PROBE_ROWS_PER_COLUMN); + } catch { + return { verdict: "inconclusive", reason: "incomplete" }; + } + present += rows.length; + for (const row of rows) { + if (row.value !== null) pool.push({ column, row, order: pool.length }); + } + } + if (Date.now() > deadline) { + return { verdict: "inconclusive", reason: "incomplete" }; + } + if (present === 0) return { verdict: "none" }; + if (pool.length === 0) { + return { verdict: "inconclusive", reason: "unsampled" }; + } + + // Oldest first; rows without a timestamp after every row with one. + pool.sort((a, b) => { + const ta = a.row.ts?.getTime() ?? Number.POSITIVE_INFINITY; + const tb = b.row.ts?.getTime() ?? Number.POSITIVE_INFINITY; + return ta === tb ? a.order - b.order : ta - tb; + }); + const tried = pool.slice(0, PROBE_SAMPLE_LIMIT); + + let failedBeforeFirstOpen = 0; + for (const { column, row } of tried) { + if (probeValueOpens(column, row, keyId)) { + return failedBeforeFirstOpen <= 1 + ? { verdict: "opens" } + : { verdict: "inconclusive", reason: "mixed" }; + } + failedBeforeFirstOpen += 1; + } + return tried.length >= 2 + ? { verdict: "fails" } + : { verdict: "inconclusive", reason: "single-value" }; +} + +// ─── The SQL sampler ──────────────────────────────────────────────────────── + +interface ColumnLocation { + table: string; + column: string; + pk: string; + ts: string | null; + codec: string | null; +} + +type RuntimeDataModel = { + models: Record< + string, + { + dbName: string | null; + fields: Array<{ name: string; dbName?: string | null }>; + } + >; +}; + +const IDENTIFIER = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/; +const KEY_ID = /^[A-Za-z0-9_-]{1,32}$/; + +function quoteIdentifier(name: string): string { + if (!IDENTIFIER.test(name)) { + throw new Error(`Unexpected identifier in the column registry: ${name}`); + } + return `"${name}"`; +} + +/** + * Table and column names for a registry entry, read from the Prisma client's + * runtime data model (the same `@@map` / `@map` names the client queries + * with). `canary.test.ts` pins that every registry column resolves, so a + * Prisma upgrade that moves this fails a test rather than the probe. + */ +export function resolveColumnLocation( + client: unknown, + column: EncryptedColumn, +): ColumnLocation { + const dm = (client as { _runtimeDataModel?: RuntimeDataModel }) + ._runtimeDataModel; + const model = dm?.models?.[column.model]; + if (!model) { + throw new Error(`No data model for ${column.model}`); + } + const dbField = (name: string): string | null => { + const f = model.fields.find((x) => x.name === name); + return f ? (f.dbName ?? f.name) : null; + }; + const columnName = dbField(column.field); + const pk = dbField(column.pkField ?? "id"); + if (!columnName || !pk) { + throw new Error(`No column for ${encryptedColumnKey(column)}`); + } + return { + table: model.dbName ?? column.model, + column: columnName, + pk, + ts: dbField("updatedAt") ?? dbField("createdAt"), + codec: column.codecField ? dbField(column.codecField) : null, + }; +} + +/** + * The SQL that finds values under `keyId` in one column. Identifiers come from + * the Prisma data model and are checked against `IDENTIFIER` before they are + * spliced; the key id is checked against the key-id grammar and travels as a + * bound parameter. The prefix tests use `substr` / `substring`, which read + * only the head of a large out-of-line value instead of the whole of it. + */ +function columnQuery( + loc: ColumnLocation, + column: EncryptedColumn, + keyId: string, + mode: "sample" | "exists", +): { sql: string; params: unknown[] } { + if (!KEY_ID.test(keyId)) throw new Error("Malformed key id"); + const c = quoteIdentifier(loc.column); + const params: unknown[] = []; + const bind = (v: unknown) => { + params.push(v); + return `$${params.length}`; + }; + const tests: string[] = []; + if (column.kind === "string") { + const versioned = `${keyId}.`; + const streamed = `~hlgcm1.${keyId}.`; + tests.push( + `substr(${c}, 1, ${versioned.length}) = ${bind(versioned)}`, + `substr(${c}, 1, ${streamed.length}) = ${bind(streamed)}`, + ); + if (keyId === LEGACY_KEY_ID) { + // A legacy value is bare base64 of at least IV + tag + one block. + tests.push(`substr(${c}, 1, 40) ~ '^[A-Za-z0-9+/]{40}$'`); + } + } else { + const utf8 = Buffer.from(`${keyId}.`, "ascii"); + const id = Buffer.from(keyId, "ascii"); + const binary = Buffer.concat([Buffer.from([0x02, id.length]), id]); + if (column.codec !== "binary2") { + tests.push( + `substring(${c} from 1 for ${utf8.length}) = ${bind(utf8)}::bytea`, + ); + } + if (column.codec === "binary2" || column.codecField) { + tests.push( + `substring(${c} from 1 for ${binary.length}) = ${bind(binary)}::bytea`, + ); + } + } + const table = quoteIdentifier(loc.table); + const where = `${c} IS NOT NULL AND (${tests.join(" OR ")})`; + if (mode === "exists") { + return { + sql: `SELECT 1 AS one FROM ${table} WHERE ${where} LIMIT 1`, + params, + }; + } + const ts = loc.ts ? quoteIdentifier(loc.ts) : null; + const codec = loc.codec ? quoteIdentifier(loc.codec) : "NULL"; + const max = Math.floor(PROBE_VALUE_MAX_BYTES); + const sql = + `SELECT CASE WHEN octet_length(${c}) <= ${max} THEN ${c} END AS value, ` + + `${codec} AS codec, ${ts ?? "NULL::timestamp"} AS ts ` + + `FROM ${table} WHERE ${where} ` + + `ORDER BY ${ts ? `${ts} ASC NULLS LAST, ` : ""}${quoteIdentifier(loc.pk)} ASC ` + + `LIMIT ${bind(0)}`; + return { sql, params }; +} + +function requireUnsafe(client: CanaryClient) { + const query = client.$queryRawUnsafe; + if (typeof query !== "function") { + throw new Error("The database client cannot run the probe queries"); + } + return query.bind(client); +} + +/** The production sampler: one bounded query per column. */ +export function sqlColumnSampler(client: CanaryClient): ColumnSampler { + const query = requireUnsafe(client); + return async (column, keyId, limit) => { + const loc = resolveColumnLocation(client, column); + const { sql, params } = columnQuery(loc, column, keyId, "sample"); + params[params.length - 1] = limit; + const rows = await query< + Array<{ + value: string | Uint8Array | null; + codec: string | null; + ts: Date | null; + }> + >(sql, ...params); + return rows.map((r) => ({ + value: r.value, + codec: r.codec, + ts: r.ts instanceof Date ? r.ts : r.ts ? new Date(r.ts) : null, + })); + }; +} + +/** + * Every registered column that still holds a value under `keyId`, by + * `Model.field`. Exhaustive: one existence query per column, no budget. + */ +export async function columnsHoldingKeyId( + client: CanaryClient, + keyId: string, +): Promise { + const query = requireUnsafe(client); + const holding: string[] = []; + for (const column of ENCRYPTED_COLUMNS) { + const loc = resolveColumnLocation(client, column); + const { sql, params } = columnQuery(loc, column, keyId, "exists"); + const rows = await query(sql, ...params); + if (rows.length > 0) holding.push(encryptedColumnKey(column)); + } + return holding; +} + +// ─── The check ────────────────────────────────────────────────────────────── + /** * Run the check for every configured key id. Never throws: a failure to run - * the check at all is reported as `error`, and only a value that provably does - * not open is reported as `mismatch`. + * the check at all is reported as `error`, and only values that provably do + * not open are reported as `mismatch`. */ export async function checkEncryptionKeyCanaries( client: CanaryClient, - options: { probeBudgetMs?: number } = {}, + options: { + probeBudgetMs?: number; + sampler?: ColumnSampler; + /** + * `warn` (`ENCRYPTION_KEY_CHECK=warn`): record nothing unless every key + * id passed. The process serves whatever the outcome, so a key that is + * not proven for every id must not be recorded beside it. + */ + mode?: "enforce" | "warn"; + } = {}, ): Promise { let keyIds: string[]; try { @@ -217,12 +516,21 @@ export async function checkEncryptionKeyCanaries( const deadline = Date.now() + (options.probeBudgetMs ?? PROBE_BUDGET_MS); const toWrite: string[] = []; - const inconclusive: string[] = []; - for (const keyId of missing) { - const probe = await probeExistingData(client, keyId, deadline); - if (probe === "fails") mismatched.push(keyId); - else if (probe === "inconclusive") inconclusive.push(keyId); - else toWrite.push(keyId); + const inconclusive: Array<{ keyId: string; reason: InconclusiveReason }> = []; + if (missing.length > 0) { + let sampler: ColumnSampler; + try { + sampler = options.sampler ?? sqlColumnSampler(client); + } catch (err) { + return { state: "error", message: (err as Error).message }; + } + for (const keyId of missing) { + const probe = await probeExistingData(sampler, keyId, deadline); + if (probe.verdict === "fails") mismatched.push(keyId); + else if (probe.verdict === "inconclusive") { + inconclusive.push({ keyId, reason: probe.reason }); + } else toWrite.push(keyId); + } } if (mismatched.length > 0) { @@ -231,16 +539,63 @@ export async function checkEncryptionKeyCanaries( return { state: "mismatch", keyIds: mismatched.sort() }; } + const written: string[] = []; + const sealable = + options.mode === "warn" && inconclusive.length > 0 ? [] : toWrite; try { - for (const keyId of toWrite) { + for (const keyId of sealable) { const ciphertext = encryptUnderKeyId(canaryPlaintext(keyId), keyId); - await client.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES (${keyId}, ${ciphertext}) ON CONFLICT (key_id) DO NOTHING`; + const inserted = + await client.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES (${keyId}, ${ciphertext}) ON CONFLICT (key_id) DO NOTHING`; + if (inserted > 0) { + written.push(keyId); + continue; + } + // Another process sealed this id first. Its canary has to open under + // this process's key, or one of the two holds the wrong key. + const [winner] = await client.$queryRaw>` + SELECT ciphertext FROM encryption_key_canaries WHERE key_id = ${keyId}`; + if (winner && opensAs(winner.ciphertext, keyId, canaryPlaintext(keyId))) { + verified.push(keyId); + } else { + mismatched.push(keyId); + } } } catch (err) { return { state: "error", message: (err as Error).message }; } + if (mismatched.length > 0) { + return { state: "mismatch", keyIds: mismatched.sort() }; + } - return { state: "ok", written: toWrite, verified, inconclusive }; + return { state: "ok", written, verified, inconclusive }; +} + +/** + * After a rotation: remove the canary of every configured key id other than + * the active one that no longer holds a single value in any registered + * column. The id's key can then leave `ENCRYPTION_KEYS`; and if the id is + * ever reused for a different key, no stale canary refuses it. + */ +export async function retireCanariesWithoutData(client: CanaryClient): Promise<{ + removed: string[]; + remaining: Record; +}> { + const active = getActiveKeyId(); + const removed: string[] = []; + const remaining: Record = {}; + for (const keyId of getConfiguredKeyIds()) { + if (keyId === active) continue; + const holding = await columnsHoldingKeyId(client, keyId); + if (holding.length > 0) { + remaining[keyId] = holding; + continue; + } + const deleted = + await client.$executeRaw`DELETE FROM encryption_key_canaries WHERE key_id = ${keyId}`; + if (deleted > 0) removed.push(keyId); + } + return { removed, remaining }; } /** @@ -266,6 +621,18 @@ export function keyMismatchLogBlock(keyIds: string[]): string { " 3. Start fresh: an empty database with this key. Stored data and", " in-database backups of the old database cannot be read without", " the old key.", + "", + " Only if you are CERTAIN the configured key is the one this data was", + " written with (the check recorded a different key earlier), reset the", + " check and restart; it then probes the stored data again:", + ...keyIds.map( + (id) => ` DELETE FROM encryption_key_canaries WHERE key_id = '${id}';`, + ), + "", + " Last resort, when no record exists yet to reset (a first start) and you", + " are certain the key is right: ENCRYPTION_KEY_CHECK=warn serves anyway", + " and only logs. It disables this safety check; with a wrong key every", + " encrypted value then fails and new rows are written under that key.", "============================================================", ].join("\n"); } diff --git a/src/lib/jobs/encryption-key-rotate.ts b/src/lib/jobs/encryption-key-rotate.ts index cd9c93eb8..cf1c52250 100644 --- a/src/lib/jobs/encryption-key-rotate.ts +++ b/src/lib/jobs/encryption-key-rotate.ts @@ -32,6 +32,10 @@ import { type CorpusClient, } from "@/lib/crypto/encryption-corpus"; import { getWorkerPrisma } from "@/lib/jobs/reminder/shared"; +import { + retireCanariesWithoutData, + type CanaryClient, +} from "@/lib/crypto/canary"; export const ENCRYPTION_KEY_ROTATE_QUEUE = "encryption-key-rotate"; export const ENCRYPTION_KEY_ROTATE_CONCURRENCY = 1; @@ -62,9 +66,20 @@ export async function runEncryptionKeyRotation( totalErrors: number; totalDropped: number; stoppedEarly: boolean; + /** Key ids whose boot key-check record this run removed. */ + retiredKeyIds: string[]; }> { const prisma = getWorkerPrisma(); const out = await rotateCorpus(prisma as unknown as CorpusClient, shouldStop); + // A complete, error-free pass: remove the boot key check's record of every + // other key id that no longer holds a value in any registered column, so + // its key can leave ENCRYPTION_KEYS and the id can never refuse a later, + // different key. The existence check walks every column itself. + const retiredKeyIds = + !out.stoppedEarly && out.totalErrors === 0 + ? (await retireCanariesWithoutData(prisma as unknown as CanaryClient)) + .removed + : []; return { activeKeyId: out.activeKeyId, totalScanned: out.totalScanned, @@ -72,6 +87,7 @@ export async function runEncryptionKeyRotation( totalErrors: out.totalErrors, totalDropped: out.totalDropped, stoppedEarly: out.stoppedEarly, + retiredKeyIds, }; } @@ -98,6 +114,7 @@ export async function handleEncryptionKeyRotate( rotated: result.totalRotated, errors: result.totalErrors, dropped: result.totalDropped, + retiredKeyIds: result.retiredKeyIds, }, }); // Per-row errors are the fail-closed skip of a row under a key the diff --git a/tests/integration/encryption-key-check.test.ts b/tests/integration/encryption-key-check.test.ts index 926221f86..6be82f674 100644 --- a/tests/integration/encryption-key-check.test.ts +++ b/tests/integration/encryption-key-check.test.ts @@ -47,12 +47,22 @@ vi.mock("@/lib/db-compat", () => ({ import { _resetCryptoCacheForTests, encrypt, + encryptUnderKeyId, getKeyFingerprint, } from "@/lib/crypto"; +import { encryptToBytes } from "@/lib/ai/coach/bytes-codec"; import { runBootKeyCheck } from "@/lib/boot/key-check"; import { + canaryPlaintext, + checkEncryptionKeyCanaries, + retireCanariesWithoutData, + type CanaryClient, +} from "@/lib/crypto/canary"; +import { + getKeyMismatchWarning, isKeyMismatch, setKeyMismatchState, + setKeyMismatchWarning, } from "@/lib/boot/key-mismatch-state"; const KEY_A = @@ -223,6 +233,369 @@ describe("boot key check probe over existing data (real Postgres)", () => { }); }); +/** Two users, one oldest, with ciphertext in string AND Bytes columns. */ +async function seedUser( + name: string, + writtenAt: Date, + seal: () => { text: string; bytes: Uint8Array }, +) { + const prisma = getPrismaClient(); + const a = seal(); + const b = seal(); + const user = await prisma.user.create({ + data: { + username: name, + email: `${name}@example.test`, + codexAccessTokenEncrypted: a.text, + codexRefreshTokenEncrypted: b.text, + createdAt: writtenAt, + updatedAt: writtenAt, + }, + }); + await prisma.userHealthProfile.create({ + data: { + userId: user.id, + aboutMeEncrypted: a.bytes, + conditionsEncrypted: b.bytes, + createdAt: writtenAt, + updatedAt: writtenAt, + }, + }); + return user.id; +} + +const sealNow = () => ({ + text: encrypt("a stored value"), + bytes: encryptToBytes("a stored note"), +}); + +describe("boot key check: what decides, and what may seal (real Postgres)", () => { + it("probes Bytes columns: an install whose ciphertext is only Bytes does not seal a wrong key", async () => { + const prisma = getPrismaClient(); + const user = await prisma.user.create({ + data: { username: "bytes-only", email: "bytes-only@example.test" }, + }); + await prisma.userHealthProfile.create({ + data: { + userId: user.id, + aboutMeEncrypted: encryptToBytes("about me"), + conditionsEncrypted: encryptToBytes("conditions"), + allergiesEncrypted: encryptToBytes("allergies"), + }, + }); + + useKey(KEY_B); + const errors = vi.spyOn(console, "error").mockImplementation(() => {}); + expect(await runBootKeyCheck(prisma)).toBe(true); + errors.mockRestore(); + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + + useKey(KEY_A); + expect(await runBootKeyCheck(prisma)).toBe(false); + expect(await prisma.encryptionKeyCanary.count()).toBe(1); + }); + + it("a probe that runs out of time seals nothing and keeps serving", async () => { + const prisma = getPrismaClient(); + await seedUser("slow", new Date("2024-01-01T00:00:00Z"), sealNow); + const outcome = await checkEncryptionKeyCanaries(prisma, { + probeBudgetMs: -1, + }); + expect(outcome.state).toBe("ok"); + if (outcome.state !== "ok") return; + expect(outcome.written).toEqual([]); + expect(outcome.inconclusive.map((i) => i.keyId)).toEqual(["v1"]); + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + }); + + it("a fresh database with no ciphertext under the key id seals it", async () => { + const prisma = getPrismaClient(); + // Ciphertext exists, but under another key id: not evidence about v1. + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_A, v2: KEY_B })); + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v2"); + vi.stubEnv("ENCRYPTION_KEY", ""); + _resetCryptoCacheForTests(); + await seedUser("v2-only", new Date("2024-01-01T00:00:00Z"), sealNow); + expect(await runBootKeyCheck(prisma)).toBe(false); + const ids = (await prisma.encryptionKeyCanary.findMany()) + .map((r) => r.keyId) + .sort(); + expect(ids).toEqual(["v1", "v2"]); + }); + + it("rows a wrong key wrote recently never decide: old rows under A, newer under B", async () => { + const prisma = getPrismaClient(); + // The original data, under A. + await seedUser("original", new Date("2024-01-01T00:00:00Z"), sealNow); + await seedUser("original-2", new Date("2024-02-01T00:00:00Z"), sealNow); + // A process holding B served for a while (no canary yet) and wrote more. + useKey(KEY_B); + for (let i = 0; i < 6; i++) { + await seedUser( + `recent-${i}`, + new Date(Date.UTC(2026, 8, 1 + i)), + sealNow, + ); + } + + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + const errors = vi.spyOn(console, "error").mockImplementation(() => {}); + await runBootKeyCheck(prisma); + warn.mockRestore(); + errors.mockRestore(); + // B is not sealed as the right key, whatever else the verdict is. + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + const outcome = await checkEncryptionKeyCanaries(prisma); + expect( + outcome.state === "mismatch" || + (outcome.state === "ok" && outcome.written.length === 0), + ).toBe(true); + + // The key the oldest data was written with is the right one. + useKey(KEY_A); + setKeyMismatchState(null); + expect(await runBootKeyCheck(prisma)).toBe(false); + expect(isKeyMismatch()).toBe(false); + expect(await prisma.encryptionKeyCanary.count()).toBe(1); + }); + + it("losing the first-seal race to another key is a mismatch, not ok", async () => { + const prisma = getPrismaClient(); + // The other process, holding A, wins the insert between our probe and + // our write. + const sealedUnderA = encryptUnderKeyId(canaryPlaintext("v1"), "v1"); + useKey(KEY_B); + let raced = false; + const racing = new Proxy(prisma, { + get(target, prop, receiver) { + if (prop === "$executeRaw") { + return async (q: TemplateStringsArray, ...values: unknown[]) => { + if (!raced) { + raced = true; + await target.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES ('v1', ${sealedUnderA})`; + } + return target.$executeRaw(q, ...values); + }; + } + const value = Reflect.get(target, prop, receiver); + return typeof value === "function" ? value.bind(target) : value; + }, + }) as unknown as CanaryClient; + const outcome = await checkEncryptionKeyCanaries(racing); + expect(raced).toBe(true); + expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] }); + + // The same race lost to a process holding the SAME key is fine. + await prisma.$executeRaw`DELETE FROM encryption_key_canaries`; + raced = false; + useKey(KEY_A); + const same = await checkEncryptionKeyCanaries(racing); + expect(same.state).toBe("ok"); + }); +}); + +describe("ENCRYPTION_KEY_CHECK (real Postgres)", () => { + afterEach(() => setKeyMismatchWarning(null)); + + it("warn: a mismatch is logged and reported, never refused, never recorded", async () => { + const prisma = getPrismaClient(); + await seedUser("warned", new Date("2024-01-01T00:00:00Z"), sealNow); + useKey(KEY_B); + vi.stubEnv("ENCRYPTION_KEY_CHECK", "warn"); + const errors = vi.spyOn(console, "error").mockImplementation(() => {}); + expect(await runBootKeyCheck(prisma)).toBe(false); + const logged = errors.mock.calls.map((c) => String(c[0])).join("\n"); + errors.mockRestore(); + expect(logged).toContain("encryption.key_mismatch"); + expect(logged).toContain("ENCRYPTION_KEY_CHECK=warn"); + expect(isKeyMismatch()).toBe(false); + expect(getKeyMismatchWarning()?.keyIds).toEqual(["v1"]); + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + + // Requests are served, through apiHandler and outside it. + await seedAdmin(); + const { GET: keyBackupGet } = + await import("@/app/api/admin/encryption/key-backup/route"); + const served = await asRoute(keyBackupGet)( + new NextRequest("http://localhost/api/admin/encryption/key-backup"), + ); + expect(served.status).toBe(200); + const { POST: mcp } = await import("@/app/mcp/route"); + const mcpRes = await mcp( + new Request("http://localhost/mcp", { method: "POST", body: "{}" }), + ); + expect(mcpRes.status).not.toBe(503); + + const { GET: healthGet } = await import("@/app/api/health/route"); + const healthBody = await ( + await asRoute(healthGet)(new NextRequest("http://localhost/api/health")) + ).json(); + expect(healthBody.warning).toBe("encryption_key_mismatch"); + expect(healthBody.reason).toBeUndefined(); + }); + + it("warn: nothing is recorded while any key id is inconclusive", async () => { + const prisma = getPrismaClient(); + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_A, v2: KEY_B })); + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v1"); + vi.stubEnv("ENCRYPTION_KEY", ""); + _resetCryptoCacheForTests(); + await prisma.user.create({ + data: { + username: "lone", + email: "lone@example.test", + codexAccessTokenEncrypted: encrypt("the only value"), + }, + }); + // v1 now holds a different key: its one value does not open. v2 has no + // data, which on its own would be recorded. + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_B, v2: KEY_A })); + _resetCryptoCacheForTests(); + vi.stubEnv("ENCRYPTION_KEY_CHECK", "warn"); + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + expect(await runBootKeyCheck(prisma)).toBe(false); + warn.mockRestore(); + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + + // Enforce records the proven id, as before. + vi.stubEnv("ENCRYPTION_KEY_CHECK", "enforce"); + const warn2 = vi.spyOn(console, "warn").mockImplementation(() => {}); + expect(await runBootKeyCheck(prisma)).toBe(false); + warn2.mockRestore(); + const ids = (await prisma.encryptionKeyCanary.findMany()).map( + (r) => r.keyId, + ); + expect(ids).toEqual(["v2"]); + }); + + it("enforce, set explicitly or by any other value, refuses as before", async () => { + const prisma = getPrismaClient(); + await seedUser("enforced", new Date("2024-01-01T00:00:00Z"), sealNow); + useKey(KEY_B); + for (const value of ["enforce", "", "off"]) { + vi.stubEnv("ENCRYPTION_KEY_CHECK", value); + setKeyMismatchState(null); + const errors = vi.spyOn(console, "error").mockImplementation(() => {}); + expect(await runBootKeyCheck(prisma)).toBe(true); + errors.mockRestore(); + expect(isKeyMismatch()).toBe(true); + expect(getKeyMismatchWarning()).toBeNull(); + } + expect(await prisma.encryptionKeyCanary.count()).toBe(0); + }); +}); + +describe("retiring a key id after rotation (real Postgres)", () => { + it("removes the canary of a key id only once no ciphertext remains under it", async () => { + const prisma = getPrismaClient(); + vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_A, v2: KEY_B })); + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v1"); + vi.stubEnv("ENCRYPTION_KEY", ""); + _resetCryptoCacheForTests(); + const userId = await seedUser( + "rotating", + new Date("2024-01-01T00:00:00Z"), + sealNow, + ); + expect(await runBootKeyCheck(prisma)).toBe(false); + expect(await prisma.encryptionKeyCanary.count()).toBe(2); + + // Rotation to v2 has not finished: one Bytes value still under v1. + vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v2"); + _resetCryptoCacheForTests(); + await prisma.user.update({ + where: { id: userId }, + data: { + codexAccessTokenEncrypted: encrypt("rotated"), + codexRefreshTokenEncrypted: encrypt("rotated"), + }, + }); + await prisma.userHealthProfile.update({ + where: { userId }, + data: { aboutMeEncrypted: encryptToBytes("rotated") }, + }); + const partial = await retireCanariesWithoutData(prisma); + expect(partial.removed).toEqual([]); + expect(partial.remaining).toEqual({ + v1: ["UserHealthProfile.conditionsEncrypted"], + }); + expect(await prisma.encryptionKeyCanary.count()).toBe(2); + + await prisma.userHealthProfile.update({ + where: { userId }, + data: { conditionsEncrypted: encryptToBytes("rotated") }, + }); + const done = await retireCanariesWithoutData(prisma); + expect(done.removed).toEqual(["v1"]); + const left = await prisma.encryptionKeyCanary.findMany(); + expect(left.map((r) => r.keyId)).toEqual(["v2"]); + }); +}); + +describe("routes outside apiHandler refuse while the key does not match", () => { + beforeEach(() => { + setKeyMismatchState({ keyIds: ["v1"], detectedAt: "2026-10-03T00:00:00Z" }); + }); + + it("/mcp answers 503 encryption.key_mismatch before auth", async () => { + const { POST } = await import("@/app/mcp/route"); + const res = await POST( + new Request("http://localhost/mcp", { + method: "POST", + headers: { + "content-type": "application/json", + authorization: "Bearer hlk_whatever", + }, + body: JSON.stringify({ jsonrpc: "2.0", id: 1, method: "tools/list" }), + }), + ); + expect(res.status).toBe(503); + expect((await res.json()).meta).toEqual({ + errorCode: "encryption.key_mismatch", + }); + }); + + it("the MCP OAuth endpoints refuse in their RFC shape", async () => { + const { POST: token } = await import("@/app/api/mcp/oauth/token/route"); + const { POST: register } = + await import("@/app/api/mcp/oauth/register/route"); + const { GET: authorizeGet, POST: authorizePost } = + await import("@/app/api/mcp/oauth/authorize/route"); + const responses = [ + await token( + new NextRequest("http://localhost/api/mcp/oauth/token", { + method: "POST", + headers: { "content-type": "application/x-www-form-urlencoded" }, + body: "grant_type=authorization_code&code=x", + }), + ), + await register( + post("/api/mcp/oauth/register", { + redirect_uris: ["https://client.example/cb"], + }), + ), + await authorizeGet( + new NextRequest( + "http://localhost/api/mcp/oauth/authorize?response_type=code", + ), + ), + await authorizePost( + new NextRequest("http://localhost/api/mcp/oauth/authorize", { + method: "POST", + headers: { "content-type": "application/x-www-form-urlencoded" }, + body: "decision=approve", + }), + ), + ]; + for (const res of responses) { + expect(res.status).toBe(503); + const body = await res.json(); + expect(body.error).toBe("temporarily_unavailable"); + expect(body.error_description).toContain("encryption key"); + } + }); +}); + describe("encryption key backup step (real Postgres)", () => { it("is due, refuses a stale confirmation, records the right one, and is due again after a re-key", async () => { const prisma = getPrismaClient();