The step confirmed one key, the active application key. A restore needs
more: off-host copies are sealed under BACKUP_ENCRYPTION_KEY, a copy
written before a rotation needs the retired application key it was
written under, and every session, token and share link hangs off
API_TOKEN_HMAC_KEY. An operator could do everything the step asked and
still be unable to read a single backup.
The recovery kit lists every key a restore of this server needs, by id
and fingerprint, never key material:
- each id in ENCRYPTION_KEYS, plus any retired id a stored copy
(innerKeyIds) or an off-host object inside the bucket's retention
(OffhostBackupKeyUse) still lists, also once the server dropped it;
- BACKUP_ENCRYPTION_KEY and each BACKUP_ENCRYPTION_PREVIOUS_KEYS entry,
named by the id their objects record;
- API_TOKEN_HMAC_KEY.
Confirming records the set the admin was shown and refuses a stale one
with 409 recovery.kit.stale. The kit is due again as soon as a key it
lists is not in that set (a rotation, a new backup key, a re-keyed
install); a key that merely drops out does not make it due. "Check my
copy" now checks any key in the kit.
Migration 0487 carries a confirmation given under the old step forward
as a kit holding that key, so that part stays confirmed, and drops the
four app_settings columns. It also creates the two tables the readiness
evaluation uses (restore_drill_outcomes, recovery_readiness_notices).
The routes move to /api/admin/recovery-kit; the reader of the keys the
backups still need moves out of the encryption status route into
src/lib/recovery/backup-key-needs.ts so the kit and the status view
share one definition.
The grant expiry case gave the grant 400 ms and slept 500: on a loaded
runner accepting, signing in and switching took longer than the grant
lived, and the grant expired before it was accepted. The grant now runs
30 seconds and the case moves the clock the resolver reads to the
instant the grant names, so the row still does not change and nothing
sleeps.
The key check's start-up probe is bounded at ten seconds and two per
query. Under load a probe query ran past two seconds and the check
answered inconclusive where the data says mismatch. The verdict tests
now run the check with time to finish; the cases that test the bounds
pass their own budget or sampler and keep them.
The probe decided on the eight oldest values alone. An install
reinstalled under a fresh key long ago holds old rows sealed under a key
that is gone and all current data under the key it runs with; it works
today, and the check would refuse it with 503 on every request.
When every oldest value fails, the newest values of the same columns are
now sampled in reverse order. One that opens makes the verdict
inconclusive (mixed): the process serves, warns and probes again at the
next start. Only old and newest values that all fail are a mismatch. A
newest-only open never seals; sealing stays with the oldest-first rule.
Every probe query now runs in its own transaction under a
transaction-local statement_timeout (2 s), so a large or locked table
ends that column with an error and the probe as incomplete instead of
holding up the boot. The newest order is the exact mirror of the oldest,
so an index on the timestamp serves both directions.
The module comment lists the three conditions that still yield a
mismatch.
At a first start there is no recorded key yet, so an operator facing a
refusal they are certain is false has no row to delete. With
ENCRYPTION_KEY_CHECK=warn (default enforce) the check still runs and logs
its full outcome, /api/health names a mismatch as a warning field without
failing, requests are served, and no key is recorded unless every key id
passes.
The variable is on the compose whitelist, in both example files, the env
manifest, the Unraid and Portainer templates and the published-env guard.
The log block, the mismatch page and the self-hosting pages offer it as the
last resort after the three real fixes, and say that it disables a safety
check.
apiHandler answers 503 encryption.key_mismatch before auth, but the remote
MCP endpoint and the OAuth authorize, token and register routes wrap in
withBackgroundEvent instead and never checked. An MCP write tool could seal
new rows under the wrong key, and those rows would open under it at the
next boot.
The refusal moves into one helper used by apiHandler, /mcp (same envelope)
and the OAuth routes (RFC 6749 temporarily_unavailable with the error code
as an extension member). A structural test fails when a route outside
apiHandler, or a server action, lacks the refusal in every exported handler
and is not on a short allowlist of routes that never touch the database.
The boot key check recorded whatever key was configured whenever its probe
found nothing to try. It only read string columns, so an install whose
ciphertext sat in Bytes columns (notes, documents, ECG, symptoms) read as
empty; a probe that ran out of time did the same; and rows written by a
process that served under a wrong key could open on the next boot and get
that key recorded as the right one.
The probe now reads every registered column, string and Bytes, in all four
stored layouts, and tries the oldest values first, ordered by the row's
updatedAt (else createdAt, else primary key). A key is recorded only when
the oldest value that opens is preceded by at most one that does not, or
when no column holds a single value under the key id. A time-out, an
unreadable table, oversized values or mixed results are inconclusive: the
process serves, records nothing and checks again at the next start.
When the first insert loses a race, the stored record is read back and must
open under this process's key, else the outcome is a mismatch.
A rotation that walked every column without errors removes the record of
each previous key id that no longer holds any value. The refusal log block
and the self-hosting pages explain how to reset the check for a key id.
The boot check decided on the first value it found under a key id, so
one damaged or foreign-sealed row could take a correctly keyed server
down with 503 on every request. The probe now samples up to five
values, one per column first. One value that opens clears the key; a
mismatch needs at least two values and none opening. A single value
that does not open is inconclusive: the server keeps serving, writes no
canary, logs a warning and probes again at the next start.
Integration setup truncates encryption_key_canaries between tests.
Every configured key id now has a canary in encryption_key_canaries,
checked at boot. A key that cannot open its canary, or existing data
written before the canary existed, puts the process in a refusal state:
every API route answers 503 encryption.key_mismatch, /api/health reports
reason encryption_key_mismatch, the pages render one explanation, and
neither the queue producer nor the worker starts. The process stays up
so the reason is visible instead of hiding in a crash loop.
The admin Encryption page gains a "Back up your encryption key" step:
key id and fingerprint (never the key), where the key lives per
platform, a check of a pasted copy that is compared and discarded, and
a confirmation bound to the active key id and fingerprint. The overview
gets a tile and admins get a dashboard banner while it is due.
HEALTHLOG_PLATFORM picks the platform the step opens on. Adds the
TrueNAS self-hosting page and key backup sections in the docs.
Marc-André Bombeck