mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 12:32:43 +00:00
Probe the oldest stored values before recording an encryption key
The boot key check recorded whatever key was configured whenever its probe found nothing to try. It only read string columns, so an install whose ciphertext sat in Bytes columns (notes, documents, ECG, symptoms) read as empty; a probe that ran out of time did the same; and rows written by a process that served under a wrong key could open on the next boot and get that key recorded as the right one. The probe now reads every registered column, string and Bytes, in all four stored layouts, and tries the oldest values first, ordered by the row's updatedAt (else createdAt, else primary key). A key is recorded only when the oldest value that opens is preceded by at most one that does not, or when no column holds a single value under the key id. A time-out, an unreadable table, oversized values or mixed results are inconclusive: the process serves, records nothing and checks again at the next start. When the first insert loses a race, the stored record is read back and must open under this process's key, else the outcome is a mismatch. A rotation that walked every column without errors removes the record of each previous key id that no longer holds any value. The refusal log block and the self-hosting pages explain how to reset the check for a key id.
This commit is contained in:
Marc-André Bombeck committed
1 parent
3a99c8cfee
commit
2c295bbea8
9 files changed
+1142
-199
No files matched your search
@@ -103,6 +103,13 @@ under `v1` (the synthetic id assigned to the existing `ENCRYPTION_KEY`).
|
||||
a full `Columns walked` line. Both read the same registry the rotation
|
||||
walks.
|
||||
|
||||
A run that walked every column without an error also removes the boot
|
||||
key check's record of each previous key id that no longer holds a single
|
||||
value (`encryption_key_canaries`); the script prints which ids it removed
|
||||
and, for an id that still holds values, the columns that do. With the record
|
||||
gone, the id can later carry a different key without the start-up check
|
||||
refusing it.
|
||||
|
||||
A zero only means "nothing left" when the run also says it walked every
|
||||
registered column. A zero from a run that skipped columns means "never
|
||||
looked", and dropping the old key on it makes those rows permanently
|
||||
|
||||
@@ -267,6 +267,34 @@ every API request answers `503` with `meta.errorCode`
|
||||
Restore the original key, point `DATABASE_URL` at the matching database, or
|
||||
start with an empty database; then restart.
|
||||
|
||||
The check remembers the key it first saw for each key id in the table
|
||||
`encryption_key_canaries`. Before it records a key, it reads the oldest
|
||||
encrypted values in the database and records the key only if they open, or
|
||||
if the database holds no encrypted value under that key id at all. When the
|
||||
stored data proves neither (a single unreadable value, the oldest values
|
||||
failing while newer ones open, or a probe that ran out of time), the app
|
||||
keeps serving, records nothing, logs a warning that starts with
|
||||
`[boot] Encryption key check inconclusive`, and checks again at the next
|
||||
start.
|
||||
|
||||
If you are certain the configured key is the one your data was written
|
||||
with, and the check still refuses because it recorded a different key
|
||||
earlier, reset the check for that key id and restart. The refusal in the log
|
||||
prints the exact statement; for the default key id it is:
|
||||
|
||||
```bash
|
||||
docker compose exec -T db psql -U healthlog healthlog \
|
||||
-c "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';"
|
||||
docker compose restart app
|
||||
```
|
||||
|
||||
The next start probes the stored data again and records the key only if
|
||||
that data opens with it. Do this only when you are sure: if the key is in
|
||||
fact wrong, the app serves errors on every encrypted value instead of the
|
||||
one clear refusal. After a key rotation the rotation script removes the
|
||||
record of a retired key id itself, once it reports that no values remain
|
||||
under that id.
|
||||
|
||||
## Backup and restore
|
||||
|
||||
Your database is the only stateful piece — back it up, and back up the
|
||||
|
||||
@@ -80,6 +80,32 @@ Fix it one of three ways, then restart the app:
|
||||
3. Start fresh with an empty database. The old data and its backups need the
|
||||
old key.
|
||||
|
||||
Before it records a key as the right one, the check reads the oldest
|
||||
encrypted values in the database and records the key only if they open (or if
|
||||
the database holds no encrypted value under that key id yet). When the data
|
||||
proves neither, the app keeps serving, records nothing, writes a warning
|
||||
starting with `[boot] Encryption key check inconclusive` to the log, and
|
||||
checks again at the next start.
|
||||
|
||||
### Resetting the check
|
||||
|
||||
If you are certain the configured key is the one the data was written with,
|
||||
and the app still refuses because the check recorded a different key earlier,
|
||||
remove that record and restart. Open a shell on the database container (Apps,
|
||||
HealthLog, the database container's Shell) and run, with the key id the log
|
||||
names (`v1` unless you use `ENCRYPTION_KEYS`):
|
||||
|
||||
```bash
|
||||
psql -U healthlog healthlog \
|
||||
-c "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';"
|
||||
```
|
||||
|
||||
Then restart the app. It probes the stored data again and records the key
|
||||
only if that data opens with it. If the key is in fact wrong, the app then
|
||||
fails on every encrypted value instead of refusing clearly, so only do this
|
||||
when you are sure. After a key rotation you never need this for the old key
|
||||
id: the rotation script removes its record once no values remain under it.
|
||||
|
||||
## Updates
|
||||
|
||||
Pick the new tag, take a snapshot of the database dataset first, and update.
|
||||
|
||||
@@ -17,6 +17,10 @@
|
||||
* non-zero, because an operator reads a zero here as permission to drop the
|
||||
* previous key, and "nothing left" and "never looked" must not print alike.
|
||||
*
|
||||
* After a complete, error-free run it checks every other configured key id
|
||||
* for remaining values and removes the boot key check's record (canary) of
|
||||
* each id that holds none, naming the columns of each id that still does.
|
||||
*
|
||||
* Usage:
|
||||
* ENCRYPTION_KEYS='{"v1":"<old>","v2":"<new>"}' \
|
||||
* ENCRYPTION_ACTIVE_KEY_ID=v2 \
|
||||
@@ -27,6 +31,7 @@ import { Buffer } from "node:buffer";
|
||||
import { PrismaClient } from "@/generated/prisma/client";
|
||||
import { PrismaPg } from "@prisma/adapter-pg";
|
||||
import { decrypt, encrypt, extractKeyId, getActiveKeyId } from "@/lib/crypto";
|
||||
import { retireCanariesWithoutData } from "@/lib/crypto/canary";
|
||||
import {
|
||||
rotateColumn,
|
||||
type CorpusClient,
|
||||
@@ -1034,6 +1039,30 @@ async function main() {
|
||||
`drop the previous key:\n ${notWalked.join("\n ")}`,
|
||||
);
|
||||
}
|
||||
|
||||
// ───── Key-check records of retired key ids ─────
|
||||
// The boot key check keeps one canary per configured key id. Once no
|
||||
// registered column holds a single value under an id, that id's key can
|
||||
// leave ENCRYPTION_KEYS, and its canary goes with it: left behind, it would
|
||||
// refuse a later, different key configured under the same id. Only after a
|
||||
// complete, error-free run; the existence check walks every column itself
|
||||
// rather than trusting the counts above.
|
||||
if (notWalked.length === 0 && totalErrors === 0) {
|
||||
const { removed, remaining } = await retireCanariesWithoutData(prisma);
|
||||
for (const [keyId, columns] of Object.entries(remaining)) {
|
||||
console.log(
|
||||
`\nKey id '${keyId}': values remain in ${columns.length} column(s); ` +
|
||||
`keep its key:\n ${columns.join("\n ")}`,
|
||||
);
|
||||
}
|
||||
for (const keyId of removed) {
|
||||
console.log(
|
||||
`\nKey id '${keyId}': no values remain. Its key-check record was ` +
|
||||
`removed; its key can now leave ENCRYPTION_KEYS.`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
await prisma.$disconnect();
|
||||
if (notWalked.length > 0) process.exit(4);
|
||||
process.exit(totalErrors > 0 ? 3 : 0);
|
||||
|
||||
@@ -13,9 +13,23 @@ import {
|
||||
checkEncryptionKeyCanaries,
|
||||
keyMismatchLogBlock,
|
||||
type CanaryClient,
|
||||
type InconclusiveReason,
|
||||
} from "@/lib/crypto/canary";
|
||||
import { setKeyMismatchState } from "./key-mismatch-state";
|
||||
|
||||
const INCONCLUSIVE_EXPLANATION: Record<InconclusiveReason, string> = {
|
||||
"single-value": "the only stored value found under that id did not open.",
|
||||
mixed:
|
||||
"the oldest stored values under that id do not open, newer ones do. " +
|
||||
"That is what a wrong key that served for a while leaves behind; check " +
|
||||
"that this is the key the database was first written with.",
|
||||
unsampled:
|
||||
"the values stored under that id are too large to probe at start-up.",
|
||||
incomplete:
|
||||
"the probe of the stored data did not finish in time or a table could " +
|
||||
"not be read.",
|
||||
};
|
||||
|
||||
export async function runBootKeyCheck(client: CanaryClient): Promise<boolean> {
|
||||
const outcome = await checkEncryptionKeyCanaries(client);
|
||||
if (outcome.state === "mismatch") {
|
||||
@@ -34,11 +48,11 @@ export async function runBootKeyCheck(client: CanaryClient): Promise<boolean> {
|
||||
`[boot] Encryption key check skipped: ${outcome.message.slice(0, 300)}`,
|
||||
);
|
||||
} else {
|
||||
if (outcome.inconclusive.length > 0) {
|
||||
for (const { keyId, reason } of outcome.inconclusive) {
|
||||
console.warn(
|
||||
`[boot] Encryption key check inconclusive for key id(s) ${outcome.inconclusive.join(", ")}: ` +
|
||||
"the only stored value found under that id did not open. Serving; " +
|
||||
"no canary written; the check runs again at the next start.",
|
||||
`[boot] Encryption key check inconclusive for key id ${keyId}: ` +
|
||||
`${INCONCLUSIVE_EXPLANATION[reason]} Serving; no key recorded; ` +
|
||||
"the check runs again at the next start.",
|
||||
);
|
||||
}
|
||||
if (outcome.written.length > 0) {
|
||||
|
||||
@@ -1,17 +1,30 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { PrismaClient } from "@/generated/prisma/client";
|
||||
import { PrismaPg } from "@prisma/adapter-pg";
|
||||
import {
|
||||
_resetCryptoCacheForTests,
|
||||
candidateMatchesKey,
|
||||
encrypt,
|
||||
encryptBytes,
|
||||
encryptUnderKeyId,
|
||||
extractKeyIdFromBytes,
|
||||
fingerprintKeyBytes,
|
||||
getKeyFingerprint,
|
||||
} from "@/lib/crypto";
|
||||
import { encryptToBytes } from "@/lib/ai/coach/bytes-codec";
|
||||
import {
|
||||
ENCRYPTED_COLUMNS,
|
||||
WORKOUT_ROUTE_GEOMETRY_AAD,
|
||||
encryptedColumnKey,
|
||||
} from "@/lib/crypto/encrypted-columns";
|
||||
import {
|
||||
canaryPlaintext,
|
||||
checkEncryptionKeyCanaries,
|
||||
keyMismatchLogBlock,
|
||||
probeValueOpens,
|
||||
resolveColumnLocation,
|
||||
type CanaryClient,
|
||||
type ColumnSampler,
|
||||
} from "../canary";
|
||||
|
||||
const KEY_A =
|
||||
@@ -19,23 +32,42 @@ const KEY_A =
|
||||
const KEY_B =
|
||||
"fedcba9876543210fedcba9876543210fedcba9876543210fedcba9876543210";
|
||||
|
||||
function useKey(hex: string) {
|
||||
function configureKey(hex: string) {
|
||||
vi.stubEnv("ENCRYPTION_KEYS", "");
|
||||
vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "");
|
||||
vi.stubEnv("ENCRYPTION_KEY", hex);
|
||||
_resetCryptoCacheForTests();
|
||||
}
|
||||
|
||||
/** An in-memory stand-in for the two raw statements and the probe delegates. */
|
||||
type Seeded =
|
||||
| string
|
||||
| Uint8Array
|
||||
| { value: string | Uint8Array | null; ts?: string; codec?: string };
|
||||
|
||||
/**
|
||||
* An in-memory stand-in for the raw statements, plus a sampler over seeded
|
||||
* values keyed by `Model.field`. A seeded value's `ts` orders it; values
|
||||
* without one keep their seeded order after every dated value, as the SQL
|
||||
* sampler orders them.
|
||||
*/
|
||||
function fakeClient(opts: {
|
||||
canaries?: Record<string, string>;
|
||||
/** delegate → field → stored values (in row order). */
|
||||
probe?: Record<string, Record<string, string[]>>;
|
||||
probe?: Record<string, Seeded[]>;
|
||||
/** Make the sampler throw for these columns. */
|
||||
unreadable?: string[];
|
||||
/** Another process seals this id (with this ciphertext) just before us. */
|
||||
race?: { keyId: string; ciphertext: string };
|
||||
}) {
|
||||
const canaries = new Map(Object.entries(opts.canaries ?? {}));
|
||||
const inserts: string[] = [];
|
||||
let raced = false;
|
||||
const client = {
|
||||
async $queryRaw() {
|
||||
async $queryRaw(_q: TemplateStringsArray, ...values: unknown[]) {
|
||||
const [arg] = values;
|
||||
if (typeof arg === "string") {
|
||||
const c = canaries.get(arg);
|
||||
return c ? [{ ciphertext: c }] : [];
|
||||
}
|
||||
return [...canaries.entries()].map(([key_id, ciphertext]) => ({
|
||||
key_id,
|
||||
ciphertext,
|
||||
@@ -43,104 +75,118 @@ function fakeClient(opts: {
|
||||
},
|
||||
async $executeRaw(_q: TemplateStringsArray, ...values: unknown[]) {
|
||||
const [keyId, ciphertext] = values as [string, string];
|
||||
if (opts.race && !raced && opts.race.keyId === keyId) {
|
||||
raced = true;
|
||||
canaries.set(keyId, opts.race.ciphertext);
|
||||
}
|
||||
inserts.push(keyId);
|
||||
if (!canaries.has(keyId)) canaries.set(keyId, ciphertext);
|
||||
if (canaries.has(keyId)) return 0;
|
||||
canaries.set(keyId, ciphertext);
|
||||
return 1;
|
||||
},
|
||||
} as unknown as CanaryClient & Record<string, unknown>;
|
||||
for (const [delegate, fields] of Object.entries(opts.probe ?? {})) {
|
||||
client[delegate] = {
|
||||
async findMany(args: {
|
||||
select: Record<string, boolean>;
|
||||
take: number;
|
||||
skip?: number;
|
||||
}) {
|
||||
const field = Object.keys(args.select)[0];
|
||||
const skip = args.skip ?? 0;
|
||||
return (fields[field] ?? [])
|
||||
.slice(skip, skip + args.take)
|
||||
.map((value) => ({ [field]: value }));
|
||||
},
|
||||
};
|
||||
}
|
||||
return { client, canaries, inserts };
|
||||
} as unknown as CanaryClient;
|
||||
const sampler: ColumnSampler = async (column, keyId, limit) => {
|
||||
const key = `${column.model}.${column.field}`;
|
||||
if (opts.unreadable?.includes(key)) throw new Error("relation missing");
|
||||
const rows = (opts.probe?.[key] ?? []).map((v) =>
|
||||
typeof v === "string" || v instanceof Uint8Array
|
||||
? { value: v, ts: null as Date | null, codec: null }
|
||||
: {
|
||||
value: v.value,
|
||||
ts: v.ts ? new Date(v.ts) : null,
|
||||
codec: v.codec ?? null,
|
||||
},
|
||||
);
|
||||
const under = rows.filter((r) => {
|
||||
if (r.value === null) return true;
|
||||
if (typeof r.value === "string") return r.value.startsWith(`${keyId}.`);
|
||||
const b = Buffer.from(r.value);
|
||||
return (
|
||||
b.toString("latin1").startsWith(`${keyId}.`) ||
|
||||
extractKeyIdFromBytes(b) === keyId
|
||||
);
|
||||
});
|
||||
return under
|
||||
.map((r, i) => ({ r, i }))
|
||||
.sort((a, b) => {
|
||||
const ta = a.r.ts?.getTime() ?? Infinity;
|
||||
const tb = b.r.ts?.getTime() ?? Infinity;
|
||||
return ta === tb ? a.i - b.i : ta - tb;
|
||||
})
|
||||
.slice(0, limit)
|
||||
.map(({ r }) => r);
|
||||
};
|
||||
const check = (budget?: number) =>
|
||||
checkEncryptionKeyCanaries(client, { sampler, probeBudgetMs: budget });
|
||||
return { client, sampler, canaries, inserts, check };
|
||||
}
|
||||
|
||||
const ok = (fields: {
|
||||
written?: string[];
|
||||
verified?: string[];
|
||||
inconclusive?: Array<{ keyId: string; reason: string }>;
|
||||
}) => ({
|
||||
state: "ok",
|
||||
written: fields.written ?? [],
|
||||
verified: fields.verified ?? [],
|
||||
inconclusive: fields.inconclusive ?? [],
|
||||
});
|
||||
|
||||
describe("encryption key canary", () => {
|
||||
beforeEach(() => {
|
||||
vi.unstubAllEnvs();
|
||||
useKey(KEY_A);
|
||||
configureKey(KEY_A);
|
||||
});
|
||||
|
||||
it("writes a canary for a configured key with no row and no data", async () => {
|
||||
const { client, canaries, inserts } = fakeClient({});
|
||||
const outcome = await checkEncryptionKeyCanaries(client);
|
||||
expect(outcome).toEqual({
|
||||
state: "ok",
|
||||
written: ["v1"],
|
||||
verified: [],
|
||||
inconclusive: [],
|
||||
});
|
||||
it("writes a canary for a configured key when no column holds a value under its id", async () => {
|
||||
const { canaries, inserts, check } = fakeClient({});
|
||||
expect(await check()).toEqual(ok({ written: ["v1"] }));
|
||||
expect(inserts).toEqual(["v1"]);
|
||||
expect(canaries.get("v1")?.startsWith("v1.")).toBe(true);
|
||||
});
|
||||
|
||||
it("verifies a canary written under the same key", async () => {
|
||||
const { client, inserts } = fakeClient({
|
||||
const { inserts, check } = fakeClient({
|
||||
canaries: { v1: encryptUnderKeyId(canaryPlaintext("v1"), "v1") },
|
||||
});
|
||||
const outcome = await checkEncryptionKeyCanaries(client);
|
||||
expect(outcome).toEqual({
|
||||
state: "ok",
|
||||
written: [],
|
||||
verified: ["v1"],
|
||||
inconclusive: [],
|
||||
});
|
||||
expect(await check()).toEqual(ok({ verified: ["v1"] }));
|
||||
expect(inserts).toEqual([]);
|
||||
});
|
||||
|
||||
it("reports a mismatch when the key changed under the same id", async () => {
|
||||
const sealedUnderA = encryptUnderKeyId(canaryPlaintext("v1"), "v1");
|
||||
useKey(KEY_B);
|
||||
const { client, inserts } = fakeClient({ canaries: { v1: sealedUnderA } });
|
||||
const outcome = await checkEncryptionKeyCanaries(client);
|
||||
expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] });
|
||||
configureKey(KEY_B);
|
||||
const { inserts, check } = fakeClient({ canaries: { v1: sealedUnderA } });
|
||||
expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] });
|
||||
expect(inserts).toEqual([]);
|
||||
});
|
||||
|
||||
it("reports a mismatch for a canary that opens to the wrong value", async () => {
|
||||
// A row from another key id copied over: opens, but says the wrong thing.
|
||||
const { client } = fakeClient({
|
||||
const { check } = fakeClient({
|
||||
canaries: { v1: encryptUnderKeyId(canaryPlaintext("v2"), "v1") },
|
||||
});
|
||||
const outcome = await checkEncryptionKeyCanaries(client);
|
||||
expect(outcome.state).toBe("mismatch");
|
||||
expect((await check()).state).toBe("mismatch");
|
||||
});
|
||||
|
||||
it("refuses to seal a wrong key when existing data predates the canary", async () => {
|
||||
const existing = [encrypt("a stored token"), encrypt("another token")];
|
||||
useKey(KEY_B);
|
||||
const { client, inserts } = fakeClient({
|
||||
configureKey(KEY_B);
|
||||
const { inserts, check } = fakeClient({
|
||||
probe: {
|
||||
user: {
|
||||
codexAccessTokenEncrypted: [existing[0]],
|
||||
codexRefreshTokenEncrypted: [existing[1]],
|
||||
},
|
||||
"User.codexAccessTokenEncrypted": [existing[0]],
|
||||
"User.codexRefreshTokenEncrypted": [existing[1]],
|
||||
},
|
||||
});
|
||||
const outcome = await checkEncryptionKeyCanaries(client);
|
||||
expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] });
|
||||
expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] });
|
||||
expect(inserts).toEqual([]);
|
||||
});
|
||||
|
||||
it("writes the canary when existing data opens under the key", async () => {
|
||||
const { client, inserts } = fakeClient({
|
||||
probe: {
|
||||
user: { codexAccessTokenEncrypted: [encrypt("a stored token")] },
|
||||
},
|
||||
const { inserts, check } = fakeClient({
|
||||
probe: { "User.codexAccessTokenEncrypted": [encrypt("a stored token")] },
|
||||
});
|
||||
const outcome = await checkEncryptionKeyCanaries(client);
|
||||
expect(outcome.state).toBe("ok");
|
||||
expect((await check()).state).toBe("ok");
|
||||
expect(inserts).toEqual(["v1"]);
|
||||
});
|
||||
|
||||
@@ -148,46 +194,228 @@ describe("encryption key canary", () => {
|
||||
// A row that claims key id v1 but does not open (damaged, or sealed by
|
||||
// something else), found FIRST, then values that open.
|
||||
const damaged = "v1." + Buffer.alloc(40, 7).toString("base64");
|
||||
const { client, inserts } = fakeClient({
|
||||
const { inserts, check } = fakeClient({
|
||||
probe: {
|
||||
user: {
|
||||
codexAccessTokenEncrypted: [damaged],
|
||||
codexRefreshTokenEncrypted: [encrypt("a stored token")],
|
||||
},
|
||||
"User.codexAccessTokenEncrypted": [damaged],
|
||||
"User.codexRefreshTokenEncrypted": [encrypt("a stored token")],
|
||||
},
|
||||
});
|
||||
const outcome = await checkEncryptionKeyCanaries(client);
|
||||
expect(outcome.state).toBe("ok");
|
||||
expect((await check()).state).toBe("ok");
|
||||
expect(inserts).toEqual(["v1"]);
|
||||
});
|
||||
|
||||
it("several values in one column count, so a wrong key with one busy column still refuses", async () => {
|
||||
const values = [encrypt("one"), encrypt("two"), encrypt("three")];
|
||||
useKey(KEY_B);
|
||||
const { client, inserts } = fakeClient({
|
||||
probe: { user: { codexAccessTokenEncrypted: values } },
|
||||
configureKey(KEY_B);
|
||||
const { inserts, check } = fakeClient({
|
||||
probe: { "User.codexAccessTokenEncrypted": values },
|
||||
});
|
||||
const outcome = await checkEncryptionKeyCanaries(client);
|
||||
expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] });
|
||||
expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] });
|
||||
expect(inserts).toEqual([]);
|
||||
});
|
||||
|
||||
it("a single value that does not open is inconclusive: no mismatch, no canary", async () => {
|
||||
const existing = encrypt("the only stored token");
|
||||
useKey(KEY_B);
|
||||
const { client, inserts } = fakeClient({
|
||||
probe: { user: { codexAccessTokenEncrypted: [existing] } },
|
||||
});
|
||||
const outcome = await checkEncryptionKeyCanaries(client);
|
||||
expect(outcome).toEqual({
|
||||
state: "ok",
|
||||
written: [],
|
||||
verified: [],
|
||||
inconclusive: ["v1"],
|
||||
configureKey(KEY_B);
|
||||
const { inserts, check } = fakeClient({
|
||||
probe: { "User.codexAccessTokenEncrypted": [existing] },
|
||||
});
|
||||
expect(await check()).toEqual(
|
||||
ok({ inconclusive: [{ keyId: "v1", reason: "single-value" }] }),
|
||||
);
|
||||
expect(inserts).toEqual([]);
|
||||
});
|
||||
|
||||
describe("Bytes columns", () => {
|
||||
it("a wrong key over Bytes-only ciphertext refuses instead of sealing", async () => {
|
||||
const notes = [encryptToBytes("note one"), encryptToBytes("note two")];
|
||||
configureKey(KEY_B);
|
||||
const { inserts, check } = fakeClient({
|
||||
probe: {
|
||||
"MoodEntry.noteEncrypted": [notes[0]],
|
||||
"Measurement.notesEncrypted": [notes[1]],
|
||||
},
|
||||
});
|
||||
expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] });
|
||||
expect(inserts).toEqual([]);
|
||||
});
|
||||
|
||||
it("opens the binary layout with its label, and the UTF-8 layout", async () => {
|
||||
const route = encryptBytes(Buffer.from("{}"), WORKOUT_ROUTE_GEOMETRY_AAD);
|
||||
const { inserts, check } = fakeClient({
|
||||
probe: {
|
||||
"WorkoutRoute.geometryEncrypted": [
|
||||
{ value: route, ts: "2024-01-01" },
|
||||
],
|
||||
"MoodEntry.noteEncrypted": [
|
||||
{ value: encryptToBytes("a note"), ts: "2024-01-02" },
|
||||
],
|
||||
},
|
||||
});
|
||||
expect(await check()).toEqual(ok({ written: ["v1"] }));
|
||||
expect(inserts).toEqual(["v1"]);
|
||||
});
|
||||
|
||||
it("a binary value under the wrong label does not count as opening", () => {
|
||||
const sealed = encryptBytes(Buffer.from("x"), "some/other/label");
|
||||
const column = ENCRYPTED_COLUMNS.find((c) => c.model === "WorkoutRoute")!;
|
||||
expect(probeValueOpens(column, { value: sealed, ts: null }, "v1")).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
|
||||
it("dispatches a codec column by the row's codec", () => {
|
||||
const column = ENCRYPTED_COLUMNS.find(
|
||||
(c) => c.model === "InboundDocument" && c.field === "contentEncrypted",
|
||||
)!;
|
||||
const binary = encryptBytes(Buffer.from("%PDF"));
|
||||
const text = encryptToBytes("JVBERg==");
|
||||
expect(
|
||||
probeValueOpens(
|
||||
column,
|
||||
{ value: binary, codec: "binary2", ts: null },
|
||||
"v1",
|
||||
),
|
||||
).toBe(true);
|
||||
expect(
|
||||
probeValueOpens(
|
||||
column,
|
||||
{ value: text, codec: "base64v1", ts: null },
|
||||
"v1",
|
||||
),
|
||||
).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("oldest values decide", () => {
|
||||
function seedOldAThenNewB() {
|
||||
const old = [
|
||||
encrypt("old 1"),
|
||||
encrypt("old 2"),
|
||||
encryptToBytes("old note 1"),
|
||||
encryptToBytes("old note 2"),
|
||||
];
|
||||
configureKey(KEY_B);
|
||||
const recent = [
|
||||
encrypt("new 1"),
|
||||
encrypt("new 2"),
|
||||
encryptToBytes("new note 1"),
|
||||
encryptToBytes("new note 2"),
|
||||
];
|
||||
configureKey(KEY_A);
|
||||
return fakeClient({
|
||||
probe: {
|
||||
"User.codexAccessTokenEncrypted": [
|
||||
{ value: recent[0], ts: "2026-09-01" },
|
||||
{ value: old[0], ts: "2024-01-01" },
|
||||
],
|
||||
"User.codexRefreshTokenEncrypted": [
|
||||
{ value: old[1], ts: "2024-01-02" },
|
||||
{ value: recent[1], ts: "2026-09-02" },
|
||||
],
|
||||
"MoodEntry.noteEncrypted": [
|
||||
{ value: recent[2], ts: "2026-09-03" },
|
||||
{ value: old[2], ts: "2024-01-03" },
|
||||
],
|
||||
"Measurement.notesEncrypted": [
|
||||
{ value: old[3], ts: "2024-01-04" },
|
||||
{ value: recent[3], ts: "2026-09-04" },
|
||||
],
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
it("the key that wrote only the newer rows is never sealed", async () => {
|
||||
const fake = seedOldAThenNewB();
|
||||
configureKey(KEY_B);
|
||||
const outcome = await fake.check();
|
||||
expect(outcome.state).not.toBe("error");
|
||||
if (outcome.state === "ok") expect(outcome.written).toEqual([]);
|
||||
expect(fake.canaries.size).toBe(0);
|
||||
});
|
||||
|
||||
it("the key that wrote the oldest rows is sealed, newer unreadable rows notwithstanding", async () => {
|
||||
const fake = seedOldAThenNewB();
|
||||
expect(await fake.check()).toEqual(ok({ written: ["v1"] }));
|
||||
});
|
||||
|
||||
it("two oldest values that fail before one that opens are mixed: inconclusive", async () => {
|
||||
const opens = encrypt("opens");
|
||||
configureKey(KEY_B);
|
||||
const foreign = [encrypt("x"), encrypt("y")];
|
||||
configureKey(KEY_A);
|
||||
const { inserts, check } = fakeClient({
|
||||
probe: {
|
||||
"User.codexAccessTokenEncrypted": [
|
||||
{ value: foreign[0], ts: "2024-01-01" },
|
||||
],
|
||||
"User.codexRefreshTokenEncrypted": [
|
||||
{ value: foreign[1], ts: "2024-01-02" },
|
||||
],
|
||||
"MoodEntry.noteEncrypted": [{ value: opens, ts: "2025-01-01" }],
|
||||
},
|
||||
});
|
||||
expect(await check()).toEqual(
|
||||
ok({ inconclusive: [{ keyId: "v1", reason: "mixed" }] }),
|
||||
);
|
||||
expect(inserts).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("nothing unproven seals", () => {
|
||||
it("a probe out of time is inconclusive, even over data that opens", async () => {
|
||||
const { inserts, check } = fakeClient({
|
||||
probe: { "User.codexAccessTokenEncrypted": [encrypt("opens")] },
|
||||
});
|
||||
expect(await check(-1)).toEqual(
|
||||
ok({ inconclusive: [{ keyId: "v1", reason: "incomplete" }] }),
|
||||
);
|
||||
expect(inserts).toEqual([]);
|
||||
});
|
||||
|
||||
it("a column that cannot be read is inconclusive, never a fresh install", async () => {
|
||||
const { inserts, check } = fakeClient({
|
||||
unreadable: ["MoodEntry.noteEncrypted"],
|
||||
});
|
||||
expect(await check()).toEqual(
|
||||
ok({ inconclusive: [{ keyId: "v1", reason: "incomplete" }] }),
|
||||
);
|
||||
expect(inserts).toEqual([]);
|
||||
});
|
||||
|
||||
it("values too large to fetch are present: inconclusive, not fresh", async () => {
|
||||
const { inserts, check } = fakeClient({
|
||||
probe: { "InboundDocument.contentEncrypted": [{ value: null }] },
|
||||
});
|
||||
expect(await check()).toEqual(
|
||||
ok({ inconclusive: [{ keyId: "v1", reason: "unsampled" }] }),
|
||||
);
|
||||
expect(inserts).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the first-seal race", () => {
|
||||
it("losing it to a process with another key is a mismatch", async () => {
|
||||
configureKey(KEY_B);
|
||||
const otherCanary = encryptUnderKeyId(canaryPlaintext("v1"), "v1");
|
||||
configureKey(KEY_A);
|
||||
const { check } = fakeClient({
|
||||
race: { keyId: "v1", ciphertext: otherCanary },
|
||||
});
|
||||
expect(await check()).toEqual({ state: "mismatch", keyIds: ["v1"] });
|
||||
});
|
||||
|
||||
it("losing it to a process with the same key verifies", async () => {
|
||||
const { check } = fakeClient({
|
||||
race: {
|
||||
keyId: "v1",
|
||||
ciphertext: encryptUnderKeyId(canaryPlaintext("v1"), "v1"),
|
||||
},
|
||||
});
|
||||
expect(await check()).toEqual(ok({ verified: ["v1"] }));
|
||||
});
|
||||
});
|
||||
|
||||
it("reports an error, not a mismatch, when the table cannot be read", async () => {
|
||||
const client = {
|
||||
$queryRaw: async () => {
|
||||
@@ -202,22 +430,49 @@ describe("encryption key canary", () => {
|
||||
it("reports an error when no key is configured", async () => {
|
||||
vi.stubEnv("ENCRYPTION_KEY", "");
|
||||
_resetCryptoCacheForTests();
|
||||
const outcome = await checkEncryptionKeyCanaries(fakeClient({}).client);
|
||||
const outcome = await fakeClient({}).check();
|
||||
expect(outcome.state).toBe("error");
|
||||
});
|
||||
|
||||
it("names the key ids and never a key in the log block", () => {
|
||||
it("names the key ids, the reset statement, and never a key in the log block", () => {
|
||||
const block = keyMismatchLogBlock(["v1"]);
|
||||
expect(block).toContain("'v1'");
|
||||
expect(block).toContain("encryption.key_mismatch");
|
||||
expect(block).toContain(
|
||||
"DELETE FROM encryption_key_canaries WHERE key_id = 'v1';",
|
||||
);
|
||||
expect(block).not.toContain(KEY_A);
|
||||
});
|
||||
});
|
||||
|
||||
describe("the SQL sampler's column map", () => {
|
||||
it("resolves every registered column to a table, a column and an order", () => {
|
||||
// A throwaway client: never connects, only exposes the data model the
|
||||
// probe reads table and column names from.
|
||||
const client = new PrismaClient({
|
||||
adapter: new PrismaPg({
|
||||
connectionString: "postgres://x:y@127.0.0.1:1/x",
|
||||
}),
|
||||
});
|
||||
for (const column of ENCRYPTED_COLUMNS) {
|
||||
const loc = resolveColumnLocation(client, column);
|
||||
expect(loc.table, encryptedColumnKey(column)).toMatch(/^[a-z_0-9]+$/);
|
||||
expect(loc.column, encryptedColumnKey(column)).toMatch(/^[a-z_0-9]+$/);
|
||||
if (column.codecField) expect(loc.codec).not.toBeNull();
|
||||
}
|
||||
const backupPieces = ENCRYPTED_COLUMNS.find(
|
||||
(c) => c.model === "DataBackupChunk",
|
||||
)!;
|
||||
expect(resolveColumnLocation(client, backupPieces).ts).toBeNull();
|
||||
const user = ENCRYPTED_COLUMNS.find((c) => c.model === "User")!;
|
||||
expect(resolveColumnLocation(client, user).ts).toBe("updated_at");
|
||||
});
|
||||
});
|
||||
|
||||
describe("key fingerprint and copy check", () => {
|
||||
beforeEach(() => {
|
||||
vi.unstubAllEnvs();
|
||||
useKey(KEY_A);
|
||||
configureKey(KEY_A);
|
||||
});
|
||||
|
||||
it("is the first 12 hex of SHA-256 over the raw key bytes", () => {
|
||||
|
||||
+459
-108
@@ -12,44 +12,85 @@
|
||||
*
|
||||
* - row present: it must decrypt to the expected value, else the key is not
|
||||
* the one the row was written with;
|
||||
* - row absent: before writing one, sample existing ciphertext under that
|
||||
* key id from the registered string columns and try it. Without this
|
||||
* probe, the first boot after an upgrade (or a re-keyed boot over a
|
||||
* database that predates the canary) would seal the wrong key as the
|
||||
* right one.
|
||||
* - row absent: before writing one, probe the ciphertext already stored
|
||||
* under that key id. Without the probe, the first boot after an upgrade
|
||||
* (or a re-keyed boot over a database that predates the canary) would
|
||||
* seal the wrong key as the right one, and from then on refuse the
|
||||
* right key.
|
||||
*
|
||||
* The probe samples up to `PROBE_SAMPLE_LIMIT` values, one per column first
|
||||
* (different columns are different writers, so one bad writer cannot speak
|
||||
* for the whole database), then further values from columns that had any.
|
||||
* Its verdict, and what each one does:
|
||||
* ## The probe
|
||||
*
|
||||
* - `opens`: at least one value opened. A wrong key opens nothing, so the
|
||||
* key is right and any value that failed is a damaged or foreign row, not
|
||||
* a key problem. The canary is written.
|
||||
* Every registered column is asked, string and Bytes alike (`ENCRYPTED_COLUMNS`),
|
||||
* for its OLDEST values under the key id: the `encrypt()` string form
|
||||
* (`<keyId>.<base64>`, or a bare legacy base64 value for the legacy id `v1`),
|
||||
* the streamed form (`~hlgcm1.<keyId>.`), the same string stored as UTF-8 in a
|
||||
* Bytes column, and the binary layout (`0x02 <len> <keyId>`). The answer for a
|
||||
* column also tells whether it holds ANY value under the id, which is the
|
||||
* existence check below.
|
||||
*
|
||||
* Oldest means the row's `updatedAt`, else its `createdAt`, else (no timestamp
|
||||
* on the model, e.g. backup pieces) the primary key, ordered after every
|
||||
* timestamped row. `updatedAt` rather than `createdAt` because a value is
|
||||
* sealed when its row is written, and a row updated yesterday may carry a
|
||||
* value sealed yesterday on a row created years ago; `updatedAt` is a ceiling
|
||||
* on when the value was sealed. The point of the order: a process that served
|
||||
* under a wrong key without a canary (an inconclusive boot, a version that
|
||||
* predates this check) wrote rows that open under that wrong key. Those rows
|
||||
* are by construction newer than the data the database was built with, so
|
||||
* reading the oldest values first means they are never what decides.
|
||||
*
|
||||
* Up to `PROBE_ROWS_PER_COLUMN` values per column are fetched, the merged set
|
||||
* is ordered by that timestamp, and the oldest `PROBE_SAMPLE_LIMIT` are tried.
|
||||
* Values larger than `PROBE_VALUE_MAX_BYTES` (stored documents, large backups)
|
||||
* count as present but are not fetched. The verdict:
|
||||
*
|
||||
* - `opens`: the oldest value that opens is preceded by at most ONE older
|
||||
* value that does not. That one is a damaged or foreign row, not a key
|
||||
* problem: a wrong key opens nothing. Values newer than the first opening
|
||||
* one do not count against it; they are what a wrong-key process would
|
||||
* have left behind. The canary is written.
|
||||
* - `fails`: two or more values were tried and none opened. That is what a
|
||||
* wrong key looks like; the process refuses (mismatch).
|
||||
* - `inconclusive`: exactly one value was found and it did not open. One
|
||||
* row cannot tell a wrong key from one damaged row, and the two possible
|
||||
* mistakes are not equal: declaring a mismatch takes a correctly keyed
|
||||
* server down with 503 on every request, while serving keeps today's
|
||||
* behaviour. So the process serves, but writes NO canary either (that
|
||||
* would seal a key that may be wrong), logs a warning, and probes again
|
||||
* at the next boot, by which time more data may decide it.
|
||||
* - `none`: nothing found inside the time budget. The canary is written,
|
||||
* which is exactly the behaviour before this check existed.
|
||||
* - `none`: every column answered, and none holds a single value under the
|
||||
* key id. A genuinely fresh install (or a new id added for rotation); the
|
||||
* canary is written.
|
||||
* - `inconclusive`, with a reason: anything not proven. Only one value and
|
||||
* it did not open (`single-value`); two or more of the oldest values do
|
||||
* not open but a newer one does (`mixed`); values exist but none could be
|
||||
* fetched (`unsampled`); the time budget ran out or a column could not be
|
||||
* read (`incomplete`). The process serves, as before this check existed,
|
||||
* writes NO canary, logs a warning, and probes again at the next boot.
|
||||
* Serving rather than refusing because a refusal takes a correctly keyed
|
||||
* server down with 503 on every request; not sealing because that would
|
||||
* lock the right key out later.
|
||||
*
|
||||
* Writes are `ON CONFLICT DO NOTHING`, so two processes racing is fine.
|
||||
* The first seal can race another process. `ON CONFLICT DO NOTHING` keeps the
|
||||
* row the other process wrote; when this process's insert affected no row,
|
||||
* the stored canary is read back and must open under this process's key, or
|
||||
* the outcome is a mismatch.
|
||||
*
|
||||
* Never logs or returns key material. The canary is written by raw SQL and is
|
||||
* deliberately not in the rotation registry: it belongs to its key id.
|
||||
* deliberately not in the rotation registry: it belongs to its key id, and the
|
||||
* rotation script removes it once no ciphertext remains under that id
|
||||
* (`retireCanariesWithoutData`).
|
||||
*/
|
||||
import {
|
||||
decrypt,
|
||||
decryptBytes,
|
||||
decryptStream,
|
||||
encryptUnderKeyId,
|
||||
extractKeyId,
|
||||
extractKeyIdFromBytes,
|
||||
extractStreamKeyId,
|
||||
getActiveKeyId,
|
||||
getConfiguredKeyIds,
|
||||
isStreamCiphertext,
|
||||
} from "@/lib/crypto";
|
||||
import { ENCRYPTED_COLUMNS } from "@/lib/crypto/encrypted-columns";
|
||||
import {
|
||||
ENCRYPTED_COLUMNS,
|
||||
encryptedColumnKey,
|
||||
type EncryptedColumn,
|
||||
} from "@/lib/crypto/encrypted-columns";
|
||||
|
||||
export const CANARY_PREFIX = "healthlog-canary:";
|
||||
|
||||
@@ -57,12 +98,21 @@ export function canaryPlaintext(keyId: string): string {
|
||||
return `${CANARY_PREFIX}${keyId}`;
|
||||
}
|
||||
|
||||
/** How many existing values the probe tries at most per key id. */
|
||||
export const PROBE_SAMPLE_LIMIT = 5;
|
||||
/** How many of the oldest values the probe tries at most per key id. */
|
||||
export const PROBE_SAMPLE_LIMIT = 8;
|
||||
|
||||
/** How long the existing-data probe may take before the canary is written anyway. */
|
||||
/** How many of its oldest values each column contributes to the pool. */
|
||||
export const PROBE_ROWS_PER_COLUMN = 2;
|
||||
|
||||
/** Values larger than this count as present but are not fetched. */
|
||||
export const PROBE_VALUE_MAX_BYTES = 2 * 1024 * 1024;
|
||||
|
||||
/** How long the probe may take before it gives up as `inconclusive`. */
|
||||
export const PROBE_BUDGET_MS = 10_000;
|
||||
|
||||
/** The key id the legacy bare-base64 string format belongs to. */
|
||||
const LEGACY_KEY_ID = "v1";
|
||||
|
||||
/** The slice of a Prisma client this module needs. */
|
||||
export interface CanaryClient {
|
||||
$queryRaw<T = unknown>(
|
||||
@@ -73,31 +123,53 @@ export interface CanaryClient {
|
||||
query: TemplateStringsArray,
|
||||
...values: unknown[]
|
||||
): Promise<number>;
|
||||
$queryRawUnsafe?<T = unknown>(
|
||||
query: string,
|
||||
...values: unknown[]
|
||||
): Promise<T>;
|
||||
$executeRawUnsafe?(query: string, ...values: unknown[]): Promise<number>;
|
||||
}
|
||||
|
||||
type ProbeDelegate = {
|
||||
findMany: (args: {
|
||||
where: Record<string, unknown>;
|
||||
select: Record<string, boolean>;
|
||||
take: number;
|
||||
skip?: number;
|
||||
}) => Promise<Array<Record<string, unknown>>>;
|
||||
};
|
||||
/** One stored value under the key id, as the probe sees it. */
|
||||
export interface ProbeRow {
|
||||
/** The stored value; null when it exists but is too large to fetch. */
|
||||
value: string | Uint8Array | null;
|
||||
/** The row's codec column, for a codec-dispatched column. */
|
||||
codec?: string | null;
|
||||
/** The row's write time (see the module comment); null when it has none. */
|
||||
ts: Date | null;
|
||||
}
|
||||
|
||||
export type ProbeVerdict = "opens" | "fails" | "inconclusive" | "none";
|
||||
/**
|
||||
* Reads up to `limit` of the oldest values under `keyId` in one column, oldest
|
||||
* first. Throws when the column cannot be read; an empty array means the
|
||||
* column holds no value under that id.
|
||||
*/
|
||||
export type ColumnSampler = (
|
||||
column: EncryptedColumn,
|
||||
keyId: string,
|
||||
limit: number,
|
||||
) => Promise<ProbeRow[]>;
|
||||
|
||||
export type InconclusiveReason =
|
||||
"single-value" | "mixed" | "unsampled" | "incomplete";
|
||||
|
||||
export type ProbeResult =
|
||||
| { verdict: "opens" | "fails" | "none" }
|
||||
| { verdict: "inconclusive"; reason: InconclusiveReason };
|
||||
|
||||
export type KeyCheckOutcome =
|
||||
| {
|
||||
state: "ok";
|
||||
/** Key ids whose canary this run wrote. */
|
||||
written: string[];
|
||||
/** Key ids whose canary existed and opened. */
|
||||
/** Key ids whose canary existed (or another process wrote) and opened. */
|
||||
verified: string[];
|
||||
/**
|
||||
* Key ids with no canary whose only existing value did not open: not a
|
||||
* mismatch, no canary written, probed again at the next boot.
|
||||
* Key ids with no canary whose stored data proved nothing either way:
|
||||
* not a mismatch, no canary written, probed again at the next boot.
|
||||
*/
|
||||
inconclusive: string[];
|
||||
inconclusive: Array<{ keyId: string; reason: InconclusiveReason }>;
|
||||
}
|
||||
| { state: "mismatch"; keyIds: string[] }
|
||||
| { state: "error"; message: string };
|
||||
@@ -112,77 +184,295 @@ function opensAs(ciphertext: string, keyId: string, expected?: string) {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Sample existing values sealed under `keyId` and try them; see the module
|
||||
* comment for what each verdict means and why.
|
||||
*/
|
||||
export async function probeExistingData(
|
||||
client: unknown,
|
||||
function isBinaryLayout(column: EncryptedColumn, row: ProbeRow): boolean {
|
||||
if (column.codec === "binary2") return true;
|
||||
return column.codecField !== undefined && row.codec === "binary2";
|
||||
}
|
||||
|
||||
/** Does one stored value open under `keyId`? Never throws. */
|
||||
export function probeValueOpens(
|
||||
column: EncryptedColumn,
|
||||
row: ProbeRow,
|
||||
keyId: string,
|
||||
deadline: number,
|
||||
): Promise<ProbeVerdict> {
|
||||
const delegates = client as Record<string, ProbeDelegate | undefined>;
|
||||
const columns = ENCRYPTED_COLUMNS.filter(
|
||||
(c) => c.kind === "string" && !c.codec && !c.codecField,
|
||||
);
|
||||
let tried = 0;
|
||||
const columnsWithData: typeof columns = [];
|
||||
|
||||
const sample = async (
|
||||
column: (typeof columns)[number],
|
||||
skip: number,
|
||||
take: number,
|
||||
): Promise<string[]> => {
|
||||
const delegate =
|
||||
delegates[column.model.charAt(0).toLowerCase() + column.model.slice(1)];
|
||||
if (!delegate || typeof delegate.findMany !== "function") return [];
|
||||
try {
|
||||
const rows = await delegate.findMany({
|
||||
where: { [column.field]: { startsWith: `${keyId}.` } },
|
||||
select: { [column.field]: true },
|
||||
take,
|
||||
...(skip > 0 ? { skip } : {}),
|
||||
});
|
||||
return rows
|
||||
.map((row) => row[column.field])
|
||||
.filter((v): v is string => typeof v === "string");
|
||||
} catch {
|
||||
return [];
|
||||
): boolean {
|
||||
const value = row.value;
|
||||
if (value === null) return false;
|
||||
try {
|
||||
if (typeof value === "string") {
|
||||
if (isStreamCiphertext(value)) {
|
||||
if (extractStreamKeyId(value) !== keyId) return false;
|
||||
decryptStream(value);
|
||||
return true;
|
||||
}
|
||||
const id = extractKeyId(value);
|
||||
if (id === null) {
|
||||
// Legacy bare base64: only ever the legacy id's.
|
||||
if (keyId !== LEGACY_KEY_ID) return false;
|
||||
decrypt(value);
|
||||
return true;
|
||||
}
|
||||
return opensAs(value, keyId);
|
||||
}
|
||||
};
|
||||
|
||||
// Pass 1: one value per column, so different writers are heard first.
|
||||
for (const column of columns) {
|
||||
if (tried >= PROBE_SAMPLE_LIMIT || Date.now() > deadline) break;
|
||||
const values = await sample(column, 0, 1);
|
||||
if (values.length === 0) continue;
|
||||
columnsWithData.push(column);
|
||||
tried += 1;
|
||||
if (opensAs(values[0], keyId)) return "opens";
|
||||
}
|
||||
// Pass 2: more values from the columns that had any, up to the limit.
|
||||
for (const column of columnsWithData) {
|
||||
if (tried >= PROBE_SAMPLE_LIMIT || Date.now() > deadline) break;
|
||||
const values = await sample(column, 1, PROBE_SAMPLE_LIMIT - tried);
|
||||
for (const value of values) {
|
||||
tried += 1;
|
||||
if (opensAs(value, keyId)) return "opens";
|
||||
const buf = Buffer.from(value.buffer, value.byteOffset, value.byteLength);
|
||||
if (isBinaryLayout(column, row)) {
|
||||
if (extractKeyIdFromBytes(buf) !== keyId) return false;
|
||||
decryptBytes(buf, column.aad);
|
||||
return true;
|
||||
}
|
||||
return opensAs(buf.toString("utf8"), keyId);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (tried >= 2) return "fails";
|
||||
if (tried === 1) return "inconclusive";
|
||||
return "none";
|
||||
}
|
||||
|
||||
/**
|
||||
* Probe the stored values under `keyId`; see the module comment for what each
|
||||
* verdict means and why.
|
||||
*/
|
||||
export async function probeExistingData(
|
||||
sampler: ColumnSampler,
|
||||
keyId: string,
|
||||
deadline: number,
|
||||
columns: readonly EncryptedColumn[] = ENCRYPTED_COLUMNS,
|
||||
): Promise<ProbeResult> {
|
||||
const pool: Array<{ column: EncryptedColumn; row: ProbeRow; order: number }> =
|
||||
[];
|
||||
let present = 0;
|
||||
for (const column of columns) {
|
||||
if (Date.now() > deadline) {
|
||||
return { verdict: "inconclusive", reason: "incomplete" };
|
||||
}
|
||||
let rows: ProbeRow[];
|
||||
try {
|
||||
rows = await sampler(column, keyId, PROBE_ROWS_PER_COLUMN);
|
||||
} catch {
|
||||
return { verdict: "inconclusive", reason: "incomplete" };
|
||||
}
|
||||
present += rows.length;
|
||||
for (const row of rows) {
|
||||
if (row.value !== null) pool.push({ column, row, order: pool.length });
|
||||
}
|
||||
}
|
||||
if (Date.now() > deadline) {
|
||||
return { verdict: "inconclusive", reason: "incomplete" };
|
||||
}
|
||||
if (present === 0) return { verdict: "none" };
|
||||
if (pool.length === 0) {
|
||||
return { verdict: "inconclusive", reason: "unsampled" };
|
||||
}
|
||||
|
||||
// Oldest first; rows without a timestamp after every row with one.
|
||||
pool.sort((a, b) => {
|
||||
const ta = a.row.ts?.getTime() ?? Number.POSITIVE_INFINITY;
|
||||
const tb = b.row.ts?.getTime() ?? Number.POSITIVE_INFINITY;
|
||||
return ta === tb ? a.order - b.order : ta - tb;
|
||||
});
|
||||
const tried = pool.slice(0, PROBE_SAMPLE_LIMIT);
|
||||
|
||||
let failedBeforeFirstOpen = 0;
|
||||
for (const { column, row } of tried) {
|
||||
if (probeValueOpens(column, row, keyId)) {
|
||||
return failedBeforeFirstOpen <= 1
|
||||
? { verdict: "opens" }
|
||||
: { verdict: "inconclusive", reason: "mixed" };
|
||||
}
|
||||
failedBeforeFirstOpen += 1;
|
||||
}
|
||||
return tried.length >= 2
|
||||
? { verdict: "fails" }
|
||||
: { verdict: "inconclusive", reason: "single-value" };
|
||||
}
|
||||
|
||||
// ─── The SQL sampler ────────────────────────────────────────────────────────
|
||||
|
||||
interface ColumnLocation {
|
||||
table: string;
|
||||
column: string;
|
||||
pk: string;
|
||||
ts: string | null;
|
||||
codec: string | null;
|
||||
}
|
||||
|
||||
type RuntimeDataModel = {
|
||||
models: Record<
|
||||
string,
|
||||
{
|
||||
dbName: string | null;
|
||||
fields: Array<{ name: string; dbName?: string | null }>;
|
||||
}
|
||||
>;
|
||||
};
|
||||
|
||||
const IDENTIFIER = /^[A-Za-z_][A-Za-z0-9_]{0,62}$/;
|
||||
const KEY_ID = /^[A-Za-z0-9_-]{1,32}$/;
|
||||
|
||||
function quoteIdentifier(name: string): string {
|
||||
if (!IDENTIFIER.test(name)) {
|
||||
throw new Error(`Unexpected identifier in the column registry: ${name}`);
|
||||
}
|
||||
return `"${name}"`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Table and column names for a registry entry, read from the Prisma client's
|
||||
* runtime data model (the same `@@map` / `@map` names the client queries
|
||||
* with). `canary.test.ts` pins that every registry column resolves, so a
|
||||
* Prisma upgrade that moves this fails a test rather than the probe.
|
||||
*/
|
||||
export function resolveColumnLocation(
|
||||
client: unknown,
|
||||
column: EncryptedColumn,
|
||||
): ColumnLocation {
|
||||
const dm = (client as { _runtimeDataModel?: RuntimeDataModel })
|
||||
._runtimeDataModel;
|
||||
const model = dm?.models?.[column.model];
|
||||
if (!model) {
|
||||
throw new Error(`No data model for ${column.model}`);
|
||||
}
|
||||
const dbField = (name: string): string | null => {
|
||||
const f = model.fields.find((x) => x.name === name);
|
||||
return f ? (f.dbName ?? f.name) : null;
|
||||
};
|
||||
const columnName = dbField(column.field);
|
||||
const pk = dbField(column.pkField ?? "id");
|
||||
if (!columnName || !pk) {
|
||||
throw new Error(`No column for ${encryptedColumnKey(column)}`);
|
||||
}
|
||||
return {
|
||||
table: model.dbName ?? column.model,
|
||||
column: columnName,
|
||||
pk,
|
||||
ts: dbField("updatedAt") ?? dbField("createdAt"),
|
||||
codec: column.codecField ? dbField(column.codecField) : null,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The SQL that finds values under `keyId` in one column. Identifiers come from
|
||||
* the Prisma data model and are checked against `IDENTIFIER` before they are
|
||||
* spliced; the key id is checked against the key-id grammar and travels as a
|
||||
* bound parameter. The prefix tests use `substr` / `substring`, which read
|
||||
* only the head of a large out-of-line value instead of the whole of it.
|
||||
*/
|
||||
function columnQuery(
|
||||
loc: ColumnLocation,
|
||||
column: EncryptedColumn,
|
||||
keyId: string,
|
||||
mode: "sample" | "exists",
|
||||
): { sql: string; params: unknown[] } {
|
||||
if (!KEY_ID.test(keyId)) throw new Error("Malformed key id");
|
||||
const c = quoteIdentifier(loc.column);
|
||||
const params: unknown[] = [];
|
||||
const bind = (v: unknown) => {
|
||||
params.push(v);
|
||||
return `$${params.length}`;
|
||||
};
|
||||
const tests: string[] = [];
|
||||
if (column.kind === "string") {
|
||||
const versioned = `${keyId}.`;
|
||||
const streamed = `~hlgcm1.${keyId}.`;
|
||||
tests.push(
|
||||
`substr(${c}, 1, ${versioned.length}) = ${bind(versioned)}`,
|
||||
`substr(${c}, 1, ${streamed.length}) = ${bind(streamed)}`,
|
||||
);
|
||||
if (keyId === LEGACY_KEY_ID) {
|
||||
// A legacy value is bare base64 of at least IV + tag + one block.
|
||||
tests.push(`substr(${c}, 1, 40) ~ '^[A-Za-z0-9+/]{40}$'`);
|
||||
}
|
||||
} else {
|
||||
const utf8 = Buffer.from(`${keyId}.`, "ascii");
|
||||
const id = Buffer.from(keyId, "ascii");
|
||||
const binary = Buffer.concat([Buffer.from([0x02, id.length]), id]);
|
||||
if (column.codec !== "binary2") {
|
||||
tests.push(
|
||||
`substring(${c} from 1 for ${utf8.length}) = ${bind(utf8)}::bytea`,
|
||||
);
|
||||
}
|
||||
if (column.codec === "binary2" || column.codecField) {
|
||||
tests.push(
|
||||
`substring(${c} from 1 for ${binary.length}) = ${bind(binary)}::bytea`,
|
||||
);
|
||||
}
|
||||
}
|
||||
const table = quoteIdentifier(loc.table);
|
||||
const where = `${c} IS NOT NULL AND (${tests.join(" OR ")})`;
|
||||
if (mode === "exists") {
|
||||
return {
|
||||
sql: `SELECT 1 AS one FROM ${table} WHERE ${where} LIMIT 1`,
|
||||
params,
|
||||
};
|
||||
}
|
||||
const ts = loc.ts ? quoteIdentifier(loc.ts) : null;
|
||||
const codec = loc.codec ? quoteIdentifier(loc.codec) : "NULL";
|
||||
const max = Math.floor(PROBE_VALUE_MAX_BYTES);
|
||||
const sql =
|
||||
`SELECT CASE WHEN octet_length(${c}) <= ${max} THEN ${c} END AS value, ` +
|
||||
`${codec} AS codec, ${ts ?? "NULL::timestamp"} AS ts ` +
|
||||
`FROM ${table} WHERE ${where} ` +
|
||||
`ORDER BY ${ts ? `${ts} ASC NULLS LAST, ` : ""}${quoteIdentifier(loc.pk)} ASC ` +
|
||||
`LIMIT ${bind(0)}`;
|
||||
return { sql, params };
|
||||
}
|
||||
|
||||
function requireUnsafe(client: CanaryClient) {
|
||||
const query = client.$queryRawUnsafe;
|
||||
if (typeof query !== "function") {
|
||||
throw new Error("The database client cannot run the probe queries");
|
||||
}
|
||||
return query.bind(client);
|
||||
}
|
||||
|
||||
/** The production sampler: one bounded query per column. */
|
||||
export function sqlColumnSampler(client: CanaryClient): ColumnSampler {
|
||||
const query = requireUnsafe(client);
|
||||
return async (column, keyId, limit) => {
|
||||
const loc = resolveColumnLocation(client, column);
|
||||
const { sql, params } = columnQuery(loc, column, keyId, "sample");
|
||||
params[params.length - 1] = limit;
|
||||
const rows = await query<
|
||||
Array<{
|
||||
value: string | Uint8Array | null;
|
||||
codec: string | null;
|
||||
ts: Date | null;
|
||||
}>
|
||||
>(sql, ...params);
|
||||
return rows.map((r) => ({
|
||||
value: r.value,
|
||||
codec: r.codec,
|
||||
ts: r.ts instanceof Date ? r.ts : r.ts ? new Date(r.ts) : null,
|
||||
}));
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Every registered column that still holds a value under `keyId`, by
|
||||
* `Model.field`. Exhaustive: one existence query per column, no budget.
|
||||
*/
|
||||
export async function columnsHoldingKeyId(
|
||||
client: CanaryClient,
|
||||
keyId: string,
|
||||
): Promise<string[]> {
|
||||
const query = requireUnsafe(client);
|
||||
const holding: string[] = [];
|
||||
for (const column of ENCRYPTED_COLUMNS) {
|
||||
const loc = resolveColumnLocation(client, column);
|
||||
const { sql, params } = columnQuery(loc, column, keyId, "exists");
|
||||
const rows = await query<unknown[]>(sql, ...params);
|
||||
if (rows.length > 0) holding.push(encryptedColumnKey(column));
|
||||
}
|
||||
return holding;
|
||||
}
|
||||
|
||||
// ─── The check ──────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Run the check for every configured key id. Never throws: a failure to run
|
||||
* the check at all is reported as `error`, and only a value that provably does
|
||||
* not open is reported as `mismatch`.
|
||||
* the check at all is reported as `error`, and only values that provably do
|
||||
* not open are reported as `mismatch`.
|
||||
*/
|
||||
export async function checkEncryptionKeyCanaries(
|
||||
client: CanaryClient,
|
||||
options: { probeBudgetMs?: number } = {},
|
||||
options: { probeBudgetMs?: number; sampler?: ColumnSampler } = {},
|
||||
): Promise<KeyCheckOutcome> {
|
||||
let keyIds: string[];
|
||||
try {
|
||||
@@ -217,12 +507,21 @@ export async function checkEncryptionKeyCanaries(
|
||||
|
||||
const deadline = Date.now() + (options.probeBudgetMs ?? PROBE_BUDGET_MS);
|
||||
const toWrite: string[] = [];
|
||||
const inconclusive: string[] = [];
|
||||
for (const keyId of missing) {
|
||||
const probe = await probeExistingData(client, keyId, deadline);
|
||||
if (probe === "fails") mismatched.push(keyId);
|
||||
else if (probe === "inconclusive") inconclusive.push(keyId);
|
||||
else toWrite.push(keyId);
|
||||
const inconclusive: Array<{ keyId: string; reason: InconclusiveReason }> = [];
|
||||
if (missing.length > 0) {
|
||||
let sampler: ColumnSampler;
|
||||
try {
|
||||
sampler = options.sampler ?? sqlColumnSampler(client);
|
||||
} catch (err) {
|
||||
return { state: "error", message: (err as Error).message };
|
||||
}
|
||||
for (const keyId of missing) {
|
||||
const probe = await probeExistingData(sampler, keyId, deadline);
|
||||
if (probe.verdict === "fails") mismatched.push(keyId);
|
||||
else if (probe.verdict === "inconclusive") {
|
||||
inconclusive.push({ keyId, reason: probe.reason });
|
||||
} else toWrite.push(keyId);
|
||||
}
|
||||
}
|
||||
|
||||
if (mismatched.length > 0) {
|
||||
@@ -231,16 +530,61 @@ export async function checkEncryptionKeyCanaries(
|
||||
return { state: "mismatch", keyIds: mismatched.sort() };
|
||||
}
|
||||
|
||||
const written: string[] = [];
|
||||
try {
|
||||
for (const keyId of toWrite) {
|
||||
const ciphertext = encryptUnderKeyId(canaryPlaintext(keyId), keyId);
|
||||
await client.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES (${keyId}, ${ciphertext}) ON CONFLICT (key_id) DO NOTHING`;
|
||||
const inserted =
|
||||
await client.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES (${keyId}, ${ciphertext}) ON CONFLICT (key_id) DO NOTHING`;
|
||||
if (inserted > 0) {
|
||||
written.push(keyId);
|
||||
continue;
|
||||
}
|
||||
// Another process sealed this id first. Its canary has to open under
|
||||
// this process's key, or one of the two holds the wrong key.
|
||||
const [winner] = await client.$queryRaw<Array<{ ciphertext: string }>>`
|
||||
SELECT ciphertext FROM encryption_key_canaries WHERE key_id = ${keyId}`;
|
||||
if (winner && opensAs(winner.ciphertext, keyId, canaryPlaintext(keyId))) {
|
||||
verified.push(keyId);
|
||||
} else {
|
||||
mismatched.push(keyId);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
return { state: "error", message: (err as Error).message };
|
||||
}
|
||||
if (mismatched.length > 0) {
|
||||
return { state: "mismatch", keyIds: mismatched.sort() };
|
||||
}
|
||||
|
||||
return { state: "ok", written: toWrite, verified, inconclusive };
|
||||
return { state: "ok", written, verified, inconclusive };
|
||||
}
|
||||
|
||||
/**
|
||||
* After a rotation: remove the canary of every configured key id other than
|
||||
* the active one that no longer holds a single value in any registered
|
||||
* column. The id's key can then leave `ENCRYPTION_KEYS`; and if the id is
|
||||
* ever reused for a different key, no stale canary refuses it.
|
||||
*/
|
||||
export async function retireCanariesWithoutData(client: CanaryClient): Promise<{
|
||||
removed: string[];
|
||||
remaining: Record<string, string[]>;
|
||||
}> {
|
||||
const active = getActiveKeyId();
|
||||
const removed: string[] = [];
|
||||
const remaining: Record<string, string[]> = {};
|
||||
for (const keyId of getConfiguredKeyIds()) {
|
||||
if (keyId === active) continue;
|
||||
const holding = await columnsHoldingKeyId(client, keyId);
|
||||
if (holding.length > 0) {
|
||||
remaining[keyId] = holding;
|
||||
continue;
|
||||
}
|
||||
const deleted =
|
||||
await client.$executeRaw`DELETE FROM encryption_key_canaries WHERE key_id = ${keyId}`;
|
||||
if (deleted > 0) removed.push(keyId);
|
||||
}
|
||||
return { removed, remaining };
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -266,6 +610,13 @@ export function keyMismatchLogBlock(keyIds: string[]): string {
|
||||
" 3. Start fresh: an empty database with this key. Stored data and",
|
||||
" in-database backups of the old database cannot be read without",
|
||||
" the old key.",
|
||||
"",
|
||||
" Only if you are CERTAIN the configured key is the one this data was",
|
||||
" written with (the check recorded a different key earlier), reset the",
|
||||
" check and restart; it then probes the stored data again:",
|
||||
...keyIds.map(
|
||||
(id) => ` DELETE FROM encryption_key_canaries WHERE key_id = '${id}';`,
|
||||
),
|
||||
"============================================================",
|
||||
].join("\n");
|
||||
}
|
||||
@@ -32,6 +32,10 @@ import {
|
||||
type CorpusClient,
|
||||
} from "@/lib/crypto/encryption-corpus";
|
||||
import { getWorkerPrisma } from "@/lib/jobs/reminder/shared";
|
||||
import {
|
||||
retireCanariesWithoutData,
|
||||
type CanaryClient,
|
||||
} from "@/lib/crypto/canary";
|
||||
|
||||
export const ENCRYPTION_KEY_ROTATE_QUEUE = "encryption-key-rotate";
|
||||
export const ENCRYPTION_KEY_ROTATE_CONCURRENCY = 1;
|
||||
@@ -62,9 +66,20 @@ export async function runEncryptionKeyRotation(
|
||||
totalErrors: number;
|
||||
totalDropped: number;
|
||||
stoppedEarly: boolean;
|
||||
/** Key ids whose boot key-check record this run removed. */
|
||||
retiredKeyIds: string[];
|
||||
}> {
|
||||
const prisma = getWorkerPrisma();
|
||||
const out = await rotateCorpus(prisma as unknown as CorpusClient, shouldStop);
|
||||
// A complete, error-free pass: remove the boot key check's record of every
|
||||
// other key id that no longer holds a value in any registered column, so
|
||||
// its key can leave ENCRYPTION_KEYS and the id can never refuse a later,
|
||||
// different key. The existence check walks every column itself.
|
||||
const retiredKeyIds =
|
||||
!out.stoppedEarly && out.totalErrors === 0
|
||||
? (await retireCanariesWithoutData(prisma as unknown as CanaryClient))
|
||||
.removed
|
||||
: [];
|
||||
return {
|
||||
activeKeyId: out.activeKeyId,
|
||||
totalScanned: out.totalScanned,
|
||||
@@ -72,6 +87,7 @@ export async function runEncryptionKeyRotation(
|
||||
totalErrors: out.totalErrors,
|
||||
totalDropped: out.totalDropped,
|
||||
stoppedEarly: out.stoppedEarly,
|
||||
retiredKeyIds,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -98,6 +114,7 @@ export async function handleEncryptionKeyRotate(
|
||||
rotated: result.totalRotated,
|
||||
errors: result.totalErrors,
|
||||
dropped: result.totalDropped,
|
||||
retiredKeyIds: result.retiredKeyIds,
|
||||
},
|
||||
});
|
||||
// Per-row errors are the fail-closed skip of a row under a key the
|
||||
|
||||
@@ -47,9 +47,17 @@ vi.mock("@/lib/db-compat", () => ({
|
||||
import {
|
||||
_resetCryptoCacheForTests,
|
||||
encrypt,
|
||||
encryptUnderKeyId,
|
||||
getKeyFingerprint,
|
||||
} from "@/lib/crypto";
|
||||
import { encryptToBytes } from "@/lib/ai/coach/bytes-codec";
|
||||
import { runBootKeyCheck } from "@/lib/boot/key-check";
|
||||
import {
|
||||
canaryPlaintext,
|
||||
checkEncryptionKeyCanaries,
|
||||
retireCanariesWithoutData,
|
||||
type CanaryClient,
|
||||
} from "@/lib/crypto/canary";
|
||||
import {
|
||||
isKeyMismatch,
|
||||
setKeyMismatchState,
|
||||
@@ -223,6 +231,214 @@ describe("boot key check probe over existing data (real Postgres)", () => {
|
||||
});
|
||||
});
|
||||
|
||||
/** Two users, one oldest, with ciphertext in string AND Bytes columns. */
|
||||
async function seedUser(
|
||||
name: string,
|
||||
writtenAt: Date,
|
||||
seal: () => { text: string; bytes: Uint8Array<ArrayBuffer> },
|
||||
) {
|
||||
const prisma = getPrismaClient();
|
||||
const a = seal();
|
||||
const b = seal();
|
||||
const user = await prisma.user.create({
|
||||
data: {
|
||||
username: name,
|
||||
email: `${name}@example.test`,
|
||||
codexAccessTokenEncrypted: a.text,
|
||||
codexRefreshTokenEncrypted: b.text,
|
||||
createdAt: writtenAt,
|
||||
updatedAt: writtenAt,
|
||||
},
|
||||
});
|
||||
await prisma.userHealthProfile.create({
|
||||
data: {
|
||||
userId: user.id,
|
||||
aboutMeEncrypted: a.bytes,
|
||||
conditionsEncrypted: b.bytes,
|
||||
createdAt: writtenAt,
|
||||
updatedAt: writtenAt,
|
||||
},
|
||||
});
|
||||
return user.id;
|
||||
}
|
||||
|
||||
const sealNow = () => ({
|
||||
text: encrypt("a stored value"),
|
||||
bytes: encryptToBytes("a stored note"),
|
||||
});
|
||||
|
||||
describe("boot key check: what decides, and what may seal (real Postgres)", () => {
|
||||
it("probes Bytes columns: an install whose ciphertext is only Bytes does not seal a wrong key", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
const user = await prisma.user.create({
|
||||
data: { username: "bytes-only", email: "bytes-only@example.test" },
|
||||
});
|
||||
await prisma.userHealthProfile.create({
|
||||
data: {
|
||||
userId: user.id,
|
||||
aboutMeEncrypted: encryptToBytes("about me"),
|
||||
conditionsEncrypted: encryptToBytes("conditions"),
|
||||
allergiesEncrypted: encryptToBytes("allergies"),
|
||||
},
|
||||
});
|
||||
|
||||
useKey(KEY_B);
|
||||
const errors = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
expect(await runBootKeyCheck(prisma)).toBe(true);
|
||||
errors.mockRestore();
|
||||
expect(await prisma.encryptionKeyCanary.count()).toBe(0);
|
||||
|
||||
useKey(KEY_A);
|
||||
expect(await runBootKeyCheck(prisma)).toBe(false);
|
||||
expect(await prisma.encryptionKeyCanary.count()).toBe(1);
|
||||
});
|
||||
|
||||
it("a probe that runs out of time seals nothing and keeps serving", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
await seedUser("slow", new Date("2024-01-01T00:00:00Z"), sealNow);
|
||||
const outcome = await checkEncryptionKeyCanaries(prisma, {
|
||||
probeBudgetMs: -1,
|
||||
});
|
||||
expect(outcome.state).toBe("ok");
|
||||
if (outcome.state !== "ok") return;
|
||||
expect(outcome.written).toEqual([]);
|
||||
expect(outcome.inconclusive.map((i) => i.keyId)).toEqual(["v1"]);
|
||||
expect(await prisma.encryptionKeyCanary.count()).toBe(0);
|
||||
});
|
||||
|
||||
it("a fresh database with no ciphertext under the key id seals it", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
// Ciphertext exists, but under another key id: not evidence about v1.
|
||||
vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_A, v2: KEY_B }));
|
||||
vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v2");
|
||||
vi.stubEnv("ENCRYPTION_KEY", "");
|
||||
_resetCryptoCacheForTests();
|
||||
await seedUser("v2-only", new Date("2024-01-01T00:00:00Z"), sealNow);
|
||||
expect(await runBootKeyCheck(prisma)).toBe(false);
|
||||
const ids = (await prisma.encryptionKeyCanary.findMany())
|
||||
.map((r) => r.keyId)
|
||||
.sort();
|
||||
expect(ids).toEqual(["v1", "v2"]);
|
||||
});
|
||||
|
||||
it("rows a wrong key wrote recently never decide: old rows under A, newer under B", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
// The original data, under A.
|
||||
await seedUser("original", new Date("2024-01-01T00:00:00Z"), sealNow);
|
||||
await seedUser("original-2", new Date("2024-02-01T00:00:00Z"), sealNow);
|
||||
// A process holding B served for a while (no canary yet) and wrote more.
|
||||
useKey(KEY_B);
|
||||
for (let i = 0; i < 6; i++) {
|
||||
await seedUser(
|
||||
`recent-${i}`,
|
||||
new Date(Date.UTC(2026, 8, 1 + i)),
|
||||
sealNow,
|
||||
);
|
||||
}
|
||||
|
||||
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
|
||||
const errors = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
await runBootKeyCheck(prisma);
|
||||
warn.mockRestore();
|
||||
errors.mockRestore();
|
||||
// B is not sealed as the right key, whatever else the verdict is.
|
||||
expect(await prisma.encryptionKeyCanary.count()).toBe(0);
|
||||
const outcome = await checkEncryptionKeyCanaries(prisma);
|
||||
expect(
|
||||
outcome.state === "mismatch" ||
|
||||
(outcome.state === "ok" && outcome.written.length === 0),
|
||||
).toBe(true);
|
||||
|
||||
// The key the oldest data was written with is the right one.
|
||||
useKey(KEY_A);
|
||||
setKeyMismatchState(null);
|
||||
expect(await runBootKeyCheck(prisma)).toBe(false);
|
||||
expect(isKeyMismatch()).toBe(false);
|
||||
expect(await prisma.encryptionKeyCanary.count()).toBe(1);
|
||||
});
|
||||
|
||||
it("losing the first-seal race to another key is a mismatch, not ok", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
// The other process, holding A, wins the insert between our probe and
|
||||
// our write.
|
||||
const sealedUnderA = encryptUnderKeyId(canaryPlaintext("v1"), "v1");
|
||||
useKey(KEY_B);
|
||||
let raced = false;
|
||||
const racing = new Proxy(prisma, {
|
||||
get(target, prop, receiver) {
|
||||
if (prop === "$executeRaw") {
|
||||
return async (q: TemplateStringsArray, ...values: unknown[]) => {
|
||||
if (!raced) {
|
||||
raced = true;
|
||||
await target.$executeRaw`INSERT INTO encryption_key_canaries (key_id, ciphertext) VALUES ('v1', ${sealedUnderA})`;
|
||||
}
|
||||
return target.$executeRaw(q, ...values);
|
||||
};
|
||||
}
|
||||
const value = Reflect.get(target, prop, receiver);
|
||||
return typeof value === "function" ? value.bind(target) : value;
|
||||
},
|
||||
}) as unknown as CanaryClient;
|
||||
const outcome = await checkEncryptionKeyCanaries(racing);
|
||||
expect(raced).toBe(true);
|
||||
expect(outcome).toEqual({ state: "mismatch", keyIds: ["v1"] });
|
||||
|
||||
// The same race lost to a process holding the SAME key is fine.
|
||||
await prisma.$executeRaw`DELETE FROM encryption_key_canaries`;
|
||||
raced = false;
|
||||
useKey(KEY_A);
|
||||
const same = await checkEncryptionKeyCanaries(racing);
|
||||
expect(same.state).toBe("ok");
|
||||
});
|
||||
});
|
||||
|
||||
describe("retiring a key id after rotation (real Postgres)", () => {
|
||||
it("removes the canary of a key id only once no ciphertext remains under it", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
vi.stubEnv("ENCRYPTION_KEYS", JSON.stringify({ v1: KEY_A, v2: KEY_B }));
|
||||
vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v1");
|
||||
vi.stubEnv("ENCRYPTION_KEY", "");
|
||||
_resetCryptoCacheForTests();
|
||||
const userId = await seedUser(
|
||||
"rotating",
|
||||
new Date("2024-01-01T00:00:00Z"),
|
||||
sealNow,
|
||||
);
|
||||
expect(await runBootKeyCheck(prisma)).toBe(false);
|
||||
expect(await prisma.encryptionKeyCanary.count()).toBe(2);
|
||||
|
||||
// Rotation to v2 has not finished: one Bytes value still under v1.
|
||||
vi.stubEnv("ENCRYPTION_ACTIVE_KEY_ID", "v2");
|
||||
_resetCryptoCacheForTests();
|
||||
await prisma.user.update({
|
||||
where: { id: userId },
|
||||
data: {
|
||||
codexAccessTokenEncrypted: encrypt("rotated"),
|
||||
codexRefreshTokenEncrypted: encrypt("rotated"),
|
||||
},
|
||||
});
|
||||
await prisma.userHealthProfile.update({
|
||||
where: { userId },
|
||||
data: { aboutMeEncrypted: encryptToBytes("rotated") },
|
||||
});
|
||||
const partial = await retireCanariesWithoutData(prisma);
|
||||
expect(partial.removed).toEqual([]);
|
||||
expect(partial.remaining).toEqual({
|
||||
v1: ["UserHealthProfile.conditionsEncrypted"],
|
||||
});
|
||||
expect(await prisma.encryptionKeyCanary.count()).toBe(2);
|
||||
|
||||
await prisma.userHealthProfile.update({
|
||||
where: { userId },
|
||||
data: { conditionsEncrypted: encryptToBytes("rotated") },
|
||||
});
|
||||
const done = await retireCanariesWithoutData(prisma);
|
||||
expect(done.removed).toEqual(["v1"]);
|
||||
const left = await prisma.encryptionKeyCanary.findMany();
|
||||
expect(left.map((r) => r.keyId)).toEqual(["v2"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("encryption key backup step (real Postgres)", () => {
|
||||
it("is due, refuses a stale confirmation, records the right one, and is due again after a re-key", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
|
||||
Reference in new issue
Block a user