feat(admin): replace the key backup step with a recovery kit

The step confirmed one key, the active application key. A restore needs
more: off-host copies are sealed under BACKUP_ENCRYPTION_KEY, a copy
written before a rotation needs the retired application key it was
written under, and every session, token and share link hangs off
API_TOKEN_HMAC_KEY. An operator could do everything the step asked and
still be unable to read a single backup.

The recovery kit lists every key a restore of this server needs, by id
and fingerprint, never key material:

- each id in ENCRYPTION_KEYS, plus any retired id a stored copy
  (innerKeyIds) or an off-host object inside the bucket's retention
  (OffhostBackupKeyUse) still lists, also once the server dropped it;
- BACKUP_ENCRYPTION_KEY and each BACKUP_ENCRYPTION_PREVIOUS_KEYS entry,
  named by the id their objects record;
- API_TOKEN_HMAC_KEY.

Confirming records the set the admin was shown and refuses a stale one
with 409 recovery.kit.stale. The kit is due again as soon as a key it
lists is not in that set (a rotation, a new backup key, a re-keyed
install); a key that merely drops out does not make it due. "Check my
copy" now checks any key in the kit.

Migration 0487 carries a confirmation given under the old step forward
as a kit holding that key, so that part stays confirmed, and drops the
four app_settings columns. It also creates the two tables the readiness
evaluation uses (restore_drill_outcomes, recovery_readiness_notices).

The routes move to /api/admin/recovery-kit; the reader of the keys the
backups still need moves out of the encryption status route into
src/lib/recovery/backup-key-needs.ts so the kit and the status view
share one definition.
This commit is contained in:
Marc-André Bombeck committed 2026-10-25 13:32:29 +01:00
1 parent 953c64f7a4
commit d3c672cc3f
56 files changed
+3042 -1611

No files matched your search

@@ -0,0 +1,101 @@
-- Recovery readiness: what the admins were last told, what the restore drill
-- last found, and the recovery kit confirmation.
--
-- `recovery_readiness_notices` holds, per readiness signal, the state the
-- admins were last notified about and when. It is the whole memory of the
-- hourly evaluation: a state change notifies once, a failing signal is
-- reminded at most once a day, and a recovery is announced once. It also
-- remembers since when a signal has applied, so a dump switched on by an
-- upgrade is not called failing before its first run had a chance.
--
-- `restore_drill_outcomes` keeps the monthly drill's last result for as long
-- as it is the last one. pg-boss forgets a failed run after seven days and
-- the admin failing-jobs card looks back 72 hours, so a failed monthly drill
-- was on screen three days in thirty.
--
-- `recovery_kit_confirmations` replaces the four `encryption_key_backup_*`
-- columns on `app_settings`. Those confirmed one key, the active application
-- key; the kit confirms the set of every key a restore needs, by id and
-- fingerprint. A confirmation given under the old step is carried over as a
-- kit that holds that one key, so an operator who confirmed the active key
-- keeps that part confirmed and is asked only for what the old step never
-- covered.
--
-- A backup file written by a release that still had the four columns carries
-- them in its instance settings; the restore writes an explicit list of
-- settings and ignores them.
--
-- The whole file is one statement, so it applies whole or not at all, and
-- running it again changes nothing. The drop takes an ACCESS EXCLUSIVE lock
-- on `app_settings` for a catalogue update only; the wait is bounded.
--
-- Reversibility: re-adding the four nullable columns restores the old shape;
-- the old step then reads as due until it is confirmed again.
--
-- @retry-on-lock-timeout
DO $migration$
BEGIN
PERFORM set_config('lock_timeout', '5s', true);
CREATE TABLE IF NOT EXISTS "recovery_readiness_notices" (
"signal" TEXT NOT NULL,
"notified_state" TEXT,
"notified_at" TIMESTAMP(3),
"watched_since" TIMESTAMP(3),
CONSTRAINT "recovery_readiness_notices_pkey" PRIMARY KEY ("signal"),
CONSTRAINT "recovery_readiness_notices_state_check"
CHECK ("notified_state" IS NULL OR "notified_state" IN ('ok', 'attention', 'failing'))
);
CREATE TABLE IF NOT EXISTS "restore_drill_outcomes" (
"id" TEXT NOT NULL DEFAULT 'singleton',
"last_run_at" TIMESTAMP(3) NOT NULL,
"last_success_at" TIMESTAMP(3),
"last_failure_at" TIMESTAMP(3),
"last_failure_reason" TEXT,
"accounts_checked" INTEGER NOT NULL DEFAULT 0,
"accounts_failed" INTEGER NOT NULL DEFAULT 0,
CONSTRAINT "restore_drill_outcomes_pkey" PRIMARY KEY ("id")
);
CREATE TABLE IF NOT EXISTS "recovery_kit_confirmations" (
"id" TEXT NOT NULL DEFAULT 'singleton',
"confirmed_at" TIMESTAMP(3) NOT NULL,
"confirmed_by_user_id" TEXT,
"entries" JSONB NOT NULL,
CONSTRAINT "recovery_kit_confirmations_pkey" PRIMARY KEY ("id")
);
IF EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_schema = current_schema()
AND table_name = 'app_settings'
AND column_name = 'encryption_key_backup_confirmed_at'
) THEN
INSERT INTO "recovery_kit_confirmations"
("id", "confirmed_at", "confirmed_by_user_id", "entries")
SELECT
'singleton',
"encryption_key_backup_confirmed_at",
"encryption_key_backup_confirmed_by_user_id",
jsonb_build_array(jsonb_build_object(
'kind', 'application',
'id', "encryption_key_backup_confirmed_key_id",
'fingerprint', "encryption_key_backup_confirmed_fingerprint"
))
FROM "app_settings"
WHERE "id" = 'singleton'
AND "encryption_key_backup_confirmed_at" IS NOT NULL
AND "encryption_key_backup_confirmed_key_id" IS NOT NULL
AND "encryption_key_backup_confirmed_fingerprint" IS NOT NULL
ON CONFLICT ("id") DO NOTHING;
ALTER TABLE "app_settings"
DROP COLUMN IF EXISTS "encryption_key_backup_confirmed_at",
DROP COLUMN IF EXISTS "encryption_key_backup_confirmed_key_id",
DROP COLUMN IF EXISTS "encryption_key_backup_confirmed_fingerprint",
DROP COLUMN IF EXISTS "encryption_key_backup_confirmed_by_user_id";
END IF;
END
$migration$;
+64 -15
View File
@@ -5469,21 +5469,6 @@ model AppSettings {
documentMaxFileBytes Int @default(26214400) @map("document_max_file_bytes")
documentQuotaBytes BigInt @default(1073741824) @map("document_quota_bytes")
// The operator's confirmation that the encryption key is backed up, bound to
// the key it was given for: the active key id and its fingerprint (the first
// twelve hex characters of SHA-256 over the raw key bytes). A rotation or a
// re-keyed install changes either one, and the step is due again. Never the
// key itself. Written by `POST /api/admin/encryption/key-backup/confirm`;
// consumer: `readKeyBackupStatus` (the admin step, the overview tile and the
// dashboard banner for admins).
encryptionKeyBackupConfirmedAt DateTime? @map("encryption_key_backup_confirmed_at")
encryptionKeyBackupConfirmedKeyId String? @map("encryption_key_backup_confirmed_key_id")
encryptionKeyBackupConfirmedFingerprint String? @map("encryption_key_backup_confirmed_fingerprint")
/// Plain id, not a relation: the instance row must not become user-scoped,
/// and a deleted admin leaves the confirmation standing. Consumer: the admin
/// step names who confirmed when the account still exists.
encryptionKeyBackupConfirmedByUserId String? @map("encryption_key_backup_confirmed_by_user_id")
/// When this instance first stored a discovery finding under the seasonally
/// adjusted correlation engine (v1.42). Stamped once, by the first
/// `syncAcceptedPatterns` run of a discovery family, before it writes; null
@@ -5906,6 +5891,70 @@ model OffhostBackupKeyUse {
@@map("offhost_backup_key_use")
}
/// The monthly restore drill's last result, one row for the instance. pg-boss
/// forgets a failed run after seven days and the failing-jobs card looks back
/// 72 hours, so the drill's verdict lives here for as long as it is the last
/// one. Written by `recordRestoreDrillOutcome` at the end of every drill that
/// had something to check; consumer: `readRecoveryReadiness` (the readiness
/// tile and the hourly evaluation).
model RestoreDrillOutcome {
id String @id @default("singleton")
/// When the last drill finished. Consumer: the tile's drill row.
lastRunAt DateTime @map("last_run_at")
/// When a drill last passed every object it checked. A failure stays the
/// verdict until a later success. Consumer: the readiness model.
lastSuccessAt DateTime? @map("last_success_at")
/// When a drill last failed. Consumer: the readiness model.
lastFailureAt DateTime? @map("last_failure_at")
/// One line naming why, already redacted. Consumer: the tile's drill row.
lastFailureReason String? @map("last_failure_reason")
/// Objects the last drill checked and how many of them failed. Consumer:
/// the tile's drill row.
accountsChecked Int @default(0) @map("accounts_checked")
accountsFailed Int @default(0) @map("accounts_failed")
@@map("restore_drill_outcomes")
}
/// Per readiness signal, the state the admins were last notified about. The
/// hourly evaluation compares the current state against it: a change to
/// attention or failing notifies once, a failing signal is reminded at most
/// once a day, and a return to ok is announced once. The update is a
/// compare-and-set on both columns, so two workers never send the same
/// notice. Written and read by `notifyReadinessChanges`.
model RecoveryReadinessNotice {
/// The signal's id (`database_dump`, `weekly_copy`, …).
signal String @id
/// `ok`, `attention` or `failing`; null until anything was announced.
notifiedState String? @map("notified_state")
notifiedAt DateTime? @map("notified_at")
/// Since when the evaluation has seen the signal apply (not `inactive`);
/// cleared when it stops applying. Consumer: the readiness model, which
/// judges a dump that has never been attempted against it.
watchedSince DateTime? @map("watched_since")
@@map("recovery_readiness_notices")
}
/// The recovery kit an admin confirmed: every key a restore needs, by kind,
/// id and fingerprint, never the key itself. The kit is due again as soon as
/// a key it needs is not in this set (a rotation, a new backup key). Written
/// by `POST /api/admin/recovery-kit/confirm`; consumer: `readRecoveryKit`
/// (the kit card, the dashboard banner for admins and the readiness model).
model RecoveryKitConfirmation {
id String @id @default("singleton")
confirmedAt DateTime @map("confirmed_at")
/// Plain id, not a relation: the instance row must not become user-scoped,
/// and a deleted admin leaves the confirmation standing. Consumer: the kit
/// card names who confirmed while the account still exists.
confirmedByUserId String? @map("confirmed_by_user_id")
/// `[{ kind, id, fingerprint }]`, validated on read
/// (`parseKitEntries`). Consumer: `assessRecoveryKit`.
entries Json
@@map("recovery_kit_confirmations")
}
/// An account whose off-host copies have to leave the bucket: it was deleted,
/// or its data was wiped. Written in the same transaction as the deletion, so
/// the request cannot be lost between the two, and deleted once every object