The recovery kit, "Keys the backups still need" and safeToDropRetiredKeys
read only the stored copies and the off-host account objects. After a
rotation the page called the old key safe to drop while every dump taken
before it, the pre-migration dumps that are the way back from an update
among them, could only be restored with that key: db-restore refuses a dump
whose active key is missing.
A dump now counts for the key its rows were written under, the manifest's
active key: a dump on this server while it is kept (read from the manifests),
an off-host one until the bucket's rule for database/ expires it, or
indefinitely when that rule cannot be read. The nightly run records the key
on its ledger row (migration 0489, database_backup_runs.active_key_id). The
encryption view and the kit list the dumps per key, in every locale, and the
rotation and backup runbooks say so.
The enumeration guard read any query that named userId as scoped, including
an orderBy tiebreak, so the nightly recovery, strain and stress score jobs
grouped every account's measurements, wrote scores into frozen records and
spent cap slots on them. It also looked only at findMany and groupBy under
lib/jobs.
The guard now reads a scope from a query's filter only (orderBy, by,
distinct, select and include are dropped first), covers count, aggregate,
updateMany, deleteMany and executeRaw, and walks the boot discoveries and
sweeps that live beside their domain: the rollup and mood rollup discovery,
the Fitbit jobs, the cumulative record rederivation, the intake slot dedup,
the inventory expiry and the climb backfill. What it then found is filtered
(the score cohorts through excludePendingOwners, which filters before the
cap; Fitbit sync, backfill and sleep repair; coach proposal lapsing and
reminder nagging; expired coach reminders; snooze clearing; inventory
expiry) or says why it needn't be (retention and expiry sweeps over rows,
writes scoped by id).
Jobs queued before a deletion ran anyway, since handlers act on the userId
they carry without asking again. The binding of every queue now completes a
job whose payload names an account pending deletion without loading its
handler (pending-deletion-gate). The silent medication intake push, which
does not go through the dispatcher, refuses a pending account itself.
The weekly and off-host passes skip an account pending deletion on purpose,
and the readiness read counted it anyway: within days of any deletion the
weekly copy read failing and the off-host copies stale, every admin was
pushed a problem and then a recovery after the purge, and a real failure
hid behind a signal already red. The read now filters with ACTIVE_ACCOUNT,
and the enumeration guard covers the file.
A handover link stayed claimable after its profile was deleted: the claim
gave the profile credentials that every sign-in then refused, and the purge
erased the account someone had just taken over. Locking an account for
deletion now revokes the open links for that profile and the links that
account minted, and creating, previewing and claiming a link refuse a profile
pending deletion, the claim under the record lock.
The last-Guardian rule counted Guardians pending deletion and profiles
pending deletion. A Guardian who had deleted their only profile could not
delete their own account; two Guardians sharing a profile could both leave,
and the second one's erasure was then refused on every run. The rule now
counts only Guardians whose account is not pending deletion, ignores profiles
pending deletion, and is decided under the record lock for account deletion,
data wipe, revoking and renouncing a Guardian. An admin's undo of a managed
profile with no such Guardian left is refused (409
account.deletion.no_guardian).
Demoting an admin counted every admin, pending deletion included, so after
one admin deleted their account the other could step down and leave the
instance with no admin who can sign in or undo the deletion. Both the
demotion and an admin's own deletion now count the other admins not pending
deletion, inside the transaction that makes the change, under one roster
lock, so two at the same moment cannot each count the other.
The hourly purge rethrew any error that was not the Guardian refusal, so the
oldest due account, failing on every run, stopped every later erasure and the
sweep of expired pre-wipe copies for good.
Each account is now its own try: a failure is counted, reported through the
worker error path with its first error, and told to the admins once per
request (user.account.erasure_failed), and the run goes on. The pre-wipe
sweep and the off-host kick run in a finally. Every account the run starts is
marked in backup_pass_attempts under the purge's queue name and the mark
closes only on erasure or when the account is no longer due, so an account
that failed or was refused goes behind every other due account next time.
The run stops starting accounts once its job budget is spent, since fifty
erasures of up to two minutes each did not fit the queue's expiry.
A run that kept its dump on the volume and failed the upload is the newest
good dump and the newest failure at the same instant. Compared with ">", the
card showed no failure and readiness said fresh, night after night, while no
dump left the host.
The health read now carries the off-host leg separately: when the newest dump
reached the bucket, and the newest one that did not. One pure reading of it
(databaseDumpProblems) feeds both the backups card and the readiness model.
A configured bucket that missed the newest dump is attention (offhost_failed)
with the upload's error, and failing once the newest dump there is older than
36 hours (offhost_stale) or none ever arrived (offhost_never). The card shows
the refused upload and when the bucket last got a dump. A failure at the
instant of the newest dump for another reason, retention throwing, is now
shown too.
pg_restore reads the table of contents and exits; the read stream piped into
it stayed open on the dump. The long-lived worker checks one dump a night, so
every dump retention deleted stayed allocated until the worker restarted and
the backups volume filled. runTool no longer takes a stdin stream at all.
The resource-limit cases and two older mocks now pass typed values, so the
file carries fewer as-never casts than before; the outcome list's comment
is a line shorter to keep its file at its length. Both ratchet baselines
are written down, the source size one also for offhost-backup.ts, which
had shrunk without its number following.
The flag makes the app believe CF-Connecting-IP from every connection,
before the peer check and whatever TRUST_PROXY_HOPS says, which is safe
only when nothing but Cloudflare can reach the origin. The reverse-proxy
page now says so with the ways to get there, the compose comment and both
example files carry the same condition, and the variable is in the env
manifest. The page's list of peers whose forwarding headers are read now
matches the code: loopback, 0.0.0.0/8, private, CGNAT, link-local,
unique-local and site-local, the reserved blocks and multicast. Those last
two stay trusted, since no internet host can connect from them; a test
pins every listed range and the Cloudflare precedence.
A text read that crossed the isolated reader's memory or time bound came
back as a plain read error, which the index job records as decrypt-error,
so the document said its stored file could not be opened. The local
extractor now answers resource-limit for PdfResourceLimitError, the index
records it under that name, and the detail view says the PDF is too large
or complex to read here, in every locale. Like the thumbnail's refusal of
the same bytes it is final: nothing retries it on its own.
Only the backups page said when the bucket's lifecycle rules left
database/ out, so every dump, and every deleted account inside it, stayed
in the bucket forever without anyone being told. Readiness gains a
dump_retention signal that reads the same cached lifecycle probe: attention
when dumps go off-host and no rule covers database/, ok with the expiry
when one does, quiet when the bucket will not say, inactive without a
bucket or without off-host dumps. The hourly evaluation pushes it to the
admins like every other signal.
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.
A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.
The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
The deletion revokes every session, refresh token and API token, so they
answer like any revoked credential; only new sign-ins get the 403
auth.account.deletion_pending, through the shared login response.
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
When the grace window ends for an account that has become the last
active guardian of a managed profile, the purge job refuses to erase it
and the account stays locked. Until now only the worker error log said
so. Every admin now gets one localized notification naming the account
and the ways out (another guardian, deleting the profile, or undoing
the deletion). An audit row on the account records that it was reported,
so the hourly retries stay quiet; a deletion undone and asked for again
is reported afresh.
The undo confirmation in the admin users list now says that sign-in
works again but API tokens, share links and trusted devices stay revoked
and have to be set up again (for a managed profile: its share links and
tokens).
The backups page stated "Retention: 90 days", from a constant whose
comment said it matched a backup-prune job. There is no such job: nothing
deletes a stored copy by age. The weekly copy is replaced in place, one
per account; the copy taken before "Delete all data" is deleted by the
account-deletion purge when the grace window ends; an uploaded copy stays
until its account is erased.
The list now carries preWipeCopyDays (the grace window, 0 when no such
copy is taken) instead of retentionDays, and the page says exactly the
three facts above, in every language.
Also drops a re-export of the JSON import row limit nothing imports.
The shared backup reader kept documents without their stored bytes but
kept every ECG strip whole, trace included. A long watch history is
hundreds of strips of tens to hundreds of kilobytes each, so an upload,
preview or restore of such a copy held the whole history in memory.
The first read now keeps each strip without its trace (a short stand-in,
after the key it was written under is recorded and its form checked),
and forEachEcgTrace reads the traces again one strip at a time. The ECG
restore writes each strip from that second read, as the document restore
already did, and refuses a file whose two reads disagree on the count.
The upload memory test gains an ECG-heavy file: 150 strips, about 200 MB.
The old reader held 197 MB of it and failed the 48 MB bound; the new one
holds about 1 MB, also under a 524 MB heap.
A switched-off scheduled dump is attention now, sent once and never
reminded, with how to turn it back on (DB_BACKUP_ENABLED). Dumps are on
by default, so off is a choice that leaves no copy able to rebuild the
host. An install that never wrote a dump and has nowhere to write one
stays quiet.
The out-of-date kit notice says what changed: the kit now also lists a
key the last confirmation did not cover, for example API_TOKEN_HMAC_KEY
on an instance upgraded from the one-key step, and that nothing broke.
The French kit and readiness strings use vous, the form the rest of the
French bundle uses.
The browser error relay forwarded anything posted to it into the operator's
GlitchTip without a session, limited only per address, so anyone could
fill the incident list. Separately, a Prisma client error prints the
failed call into its message and stack, with the arguments object (a
reading's value, a note's text) when no source map is at hand, and the
server sent that to GlitchTip and the logs after a credential scrub that
does not look for it.
The relay now needs a signed-in session and is limited per account; errors
on pages before sign-in go unreported. The central redaction cuts a Prisma
call's printed arguments, keeping the call, a reason line that names a
field, and the stack frames, and the GlitchTip sender applies the
redaction to every event at the egress boundary, whichever caller built it.
The deploy-status webhook authenticated with a static secret in a header,
and the timestamp the docs called replay protection was covered by nothing:
whoever captures one request holds the secret and can send a fresh request
with any content. The strict setting, documented as closing the replay
window completely, only made that timestamp mandatory.
A sender can now sign instead: X-Deploy-Webhook-Signature is the
HMAC-SHA256 of the timestamp and the exact body under the shared secret,
with the timestamp within five minutes, and the secret never travels. The
strict setting accepts only signed requests. The secret header still works
by default for senders that cannot sign, and the docs describe its
timestamp as the staleness hint it is.
safeFetch with requirePublicHost vetted the first URL and pinned DNS
answers in the connector, but Node never looks up an IP literal, so a
followed redirect to a literal private or metadata address was dialled
unchecked. No caller follows redirects to a user-chosen host today; the
floor did not hold for the next one that would.
With requirePublicHost, redirects are now followed one hop at a time and
every location goes through isPublicUrl before it is dialled, at most five
hops. isPublicIp also refuses the benchmarking, protocol-assignment,
multicast, reserved and broadcast IPv4 ranges, site-local and multicast
IPv6 and the IPv4-translated spelling of a private address, and isPublicUrl
strips a trailing dot before the hostname denylist.
The quick-answer route charged one per-address bucket before it read the
token. Where the client address cannot be read, every caller shares that
bucket, so two junk requests a minute switched off every person's
notification quick answers.
The token is an HMAC and is now checked first, which costs one hash and no
database read. A genuine token is limited per account; anything else is
limited per address in a separate bucket that genuine answers never draw
from.
The proxy matcher left out every path ending in an image or text extension
so static files skip it. The exclusion was not anchored, so an API path
whose last segment the caller chooses (a custom tag name ending in .png)
skipped the proxy too, and with it the demo-mode write refusal, the
worker-only refusal and the security headers. Authentication was unaffected.
The extension exclusion now applies outside /api/ only.
The MCP server instructions promise that the person's free text comes back
wrapped in USER_TEXT markers, but get_medication_schedule returned
medication names and doses as plain strings, and get_labs returned analyte,
panel and unit names that can come straight out of an uploaded document.
Text written to read like an instruction reached the connecting client's
model unmarked. The marker scrub also removed only the exact uppercase
spelling, so a lower-case or spaced marker survived inside fenced text.
Both tools now fence those fields, and the scrub removes a marker in any
case and with any spacing inside the brackets.
A token scoped to one medication got 404 for a name that matched none of
the owner's medications and 403 for one that matched outside its scope, so
it could confirm which medications exist by name. Its idempotency replay
also ran before the scope check and returned any of the owner's doses
stored under the key.
A token without the wildcard now gets the same 403 for both, the replay is
looked up only after the scope check, and a key already spent on another
medication answers 409 without that dose. The wildcard token keeps its 404.
A share link gated its report on the doctorReport module but listed and
served its attached documents unconditionally, although the comment beside
the gate said the documents kept a gate of their own. An owner who switched
the documents module off still had every attached file decrypted for anyone
holding the link and its passphrase.
The share page lists no document while the module is off, and the document
route answers 404, the same as for an unknown document, before it looks the
membership up.
Dynamic client registration is open by specification and the client names
itself, so anyone could register under a well-known assistant's name with a
callback on a host of their own and send a signed-in person the authorize
link. The consent page showed the borrowed name and "Unverified
application", never where the code would be sent.
Every consent page now says where the access goes: the callback's host, or
an application on this device for a loopback callback, and the Allow button
names the host. For a self-registered client whose callback leaves the
device, a warning that names the host leads the page, above the
self-reported name. Registration stays open: the hosted assistants register
this way with their own https callback.
The session cookie is SameSite=Lax, and SameSite compares the registrable
domain, so a page on a sibling subdomain of the instance could send a
CORS-simple POST with the person's cookie attached. safeJson accepted any
Content-Type containing "application/json", so a text/plain body that
mentions it in a parameter, which needs no preflight, was parsed as the
person's own request: readings added, entries deleted, integrations
disconnected, a delegate invited within the re-proof window.
apiHandler now refuses a POST, PUT, PATCH or DELETE before authentication
when the browser marks it Sec-Fetch-Site same-site or cross-site, or, from
a browser that sends no fetch metadata, when its Origin names a host other
than the one the request went to or APP_URL. A request with an
Authorization header, or with neither header (the phone app, scripts,
webhook senders), is unaffected, as are the webhook and URL-credential
routes that authenticate with their own secret. The JSON media type is
compared exactly, with parameters allowed, in safeJson and in every route
that branched on a substring of it.
The Coach snapshot attached its labs block regardless of the module, a
decision from before Labs could be switched off, while the snapshot's own
prelude says a disabled domain never reaches the model. With Labs off, the
Coach prompt, its get_labs tool, the daily briefing's flagged-marker block,
the biomarker status note, the reaction line to a new panel and the MCP
lab_trend_brief prompt all still read lab values for a model.
Each of those reads now asks the labs module first and reads nothing when it
is off: the Coach tool answers module_disabled, the briefing omits the
block, the status note neither writes nor serves text, the reaction line
gets no value, and the MCP prompt refuses like get_labs and the doctor-visit
prompt already did. The data routes stay as they are. A structural guard
holds every lab-table reader under the model-input paths to the gate or to
a listed reason its rows never reach a model.
The per-account password throttle skipped every attempt that came from an
address one of the account's sessions or device logins had used in the last
thirty days. An address is the easiest thing about a request to share or
claim: in a household or an office the guesser has the owner's address, and
a direct caller could name it in a forwarding header. Either way the
throttle switched off and guessing was bounded only by the hash cost.
A remembered browser (the trusted-device cookie) and a phone presenting the
device id of one of the account's logins still pass; everything else gets
the growing wait.
The client-IP resolver trusted one forwarding hop by default and read the
rightmost X-Forwarded-For entry. Next fills that header with the socket's
address only when the request carries none, so a caller reaching the
published port directly could send its own and choose a fresh address per
request: every per-IP limit (sign-in, second factor, passkey, share unlock,
webhooks) became per-request, and sessions, the audit log and sign-in
notices recorded the chosen address.
The server now writes each connection's own peer address into an internal
header from Node's http.server.request.start diagnostics channel, installed
at boot and overwriting anything a caller sent under that name. The resolver
reads forwarding headers only when that peer is loopback, private,
unique-local or CGNAT, where a reverse proxy sits; a public peer is its own
client address, and a log line says once that its forwarding headers were
ignored. With TRUST_PROXY_HOPS=0 the peer is used instead of no address, so
a directly exposed host no longer shares one bucket across all anonymous
callers. Proxies on the same host, Docker network or tailnet need no change.
The thumbnail's header walk took the byte after any 0xFF as a marker. A
single 0xFF fill byte after the start-of-image marker, which the JPEG
standard allows and every decoder skips, threw it off; it read no size, and
an unread size let the image through to a decode that allocates the full
bitmap. A small JPEG declaring 20000 x 20000 pixels allocated 1.6 GB in the
web process before the post-decode check could see it.
The avatar upload already had a walker that skips fill bytes. Both now use
one reader (lib/image-dimensions) that follows the marker rules: fill bytes,
the length-less markers, and no size when a scan or the end of the image
comes before a frame header. The thumbnail treats an unreadable size as a
refusal (size-unreadable) for every format it decodes, and the backfill
counts that refusal as final.
pdf.js decodes an image at the size the file declares, whatever size the
page is drawn at, and the server opened documents without an image limit.
A small upload declaring a very large image made the web process allocate
gigabytes until the container was killed, for every user at once, and the
thumbnail job ran the same document again after each restart. An image
whose soft mask declares the size does the same past pdf.js' own limit, and
nothing bounded how long a text read could take.
pdf.js is now opened with maxImageSize (20 million pixels, which keeps one
image inside the reader's ceiling and admits a 300 dpi A3 scan), and every
read and render runs in a child process, one at a time: its V8 heap is
capped, its resident memory is sampled and the child killed past 640 MB, on
Linux it volunteers to the out-of-memory killer first, a time limit ends a
read that runs on, and it starts with an almost empty environment. The
loader and the rendering moved into pdf-engine.mjs, which the child runs
with plain Node and which is traced into the standalone image.
A document that crosses a bound answers resource-limit. The thumbnail job
records it as a final failure, so the backfill does not queue it again.
Several surfaces promised more than the per-account copies deliver:
- The full-backup export card said it had the same shape as the weekly
copy. It is a portable export with documents as metadata only; it now
says so, and that it cannot recreate an account on another server.
- The export page offered to "import from a backup". The JSON import adds
measurements and mood entries; it is now named that way, and its help
states the 16 MB and 10,000-entries-per-list limits from the constants
the route enforces.
- The weekly copy is labelled an account rollback on this server, and the
backups page says a copy cannot rebuild a lost server. The upload says
the account must exist here. Two restore refusals pointed to a
"disaster-recovery" copy; they now name the copies that exist.
- The instance-settings restore option was described as being for
rebuilding a host.
docs/ops/backup-restore.md is rewritten short and accurate: what each
copy is, the deletion grace window, that database dumps keep a deleted
account until they age out, how to restore one account, and a link to
the disaster-recovery procedure for a lost server.
The deletion dialogs, the managed-profile delete dialog and the Data &
Privacy page read the window from the privacy summary and say what
happens: the account is locked at once, erased after the stated number
of days, and an admin of the server can undo it until then; a wipe keeps
a copy an admin can restore for the same time. Copies in the server's
backups age out after their retention period, and the text says that
too. With the window set to 0 they keep the immediate wording.
The sign-in page names the erasure date when a sign-in, a passkey or the
single sign-on callback is refused for an account pending deletion. The
admin users list shows a pending account with its date and an "Undo
deletion" action, and the backups list labels the copy taken before a
wipe. Every new string is translated in all nine languages.
Deleting an account, a guardian deleting a managed profile, and "Delete
all data" were final the moment they were confirmed: the account row
cascaded into every table and every stored copy, and the off-host objects
were purged in the same breath. Nothing short of rolling the whole
instance back could return one account.
ACCOUNT_DELETION_GRACE_DAYS (14 by default, at most 30, 0 erases at once
as before) now sets a window:
- Account and managed-profile deletion lock the account (migration 0488:
deletion_requested_at, deletion_purge_at, deletion_requested_by_id).
Every session, refresh token, API token, connector, share link and
step-up elevation is revoked in the same transaction, and the record,
its stored copies and its off-host objects are left as they are.
- Sign-in is refused with 403 auth.account.deletion_pending and the
erasure date, after the credential was proved and before a second
factor is asked for. The session and Bearer resolvers and the acting-
account resolver refuse a pending account too, so a credential minted
in the moment before the lock is useless.
- POST /api/admin/users/{id}/undo-deletion (cookie-only, audit-logged)
clears the lock. The admin users list reports the pending state.
- The hourly account-deletion-purge job erases accounts whose window has
ended, exactly as the immediate deletion did, with the off-host purge
request committed alongside. The Guardian invariant is decided again;
a refused erasure stays pending and is reported.
- A wipe first stores the whole record as a PRE_WIPE_<ms> copy through
the writer the weekly copy uses, keeps it out of the wipe, and holds
its off-host purge back with a new not_before on the request. The
ordinary restore brings the record back; restoring that copy withdraws
the held purge. The purge job deletes the copy when its window ends.
Background work skips accounts pending deletion through one shared
predicate (src/lib/account-deletion/active-account.ts): every reminder,
digest, Coach, sync, backfill, rollup, score and backup enumeration
filters with it, and account-deletion-enumeration-guard.test.ts fails on
a query in background work that neither filters nor says why it need
not. Notification delivery and the AI capability loader refuse a pending
account as chokepoints behind it, and the Withings, WHOOP and Telegram
webhooks and the notification quick actions ignore one.
The privacy summary publishes the window. Two pieces of the reminder tick
and the off-host pass moved into modules of their own to keep the files
inside their size ratchet.
A database dump is the whole database at one moment, so an account deleted
since stays in the dumps taken before until they age out. The database row
on the backups card now says so with the numbers that decide it: how many
nightly dumps this server keeps, and after how many days the bucket's
lifecycle rule removes the off-host copies.
The lifecycle probe used to answer with the shortest expiry of any enabled
rule, whatever its prefix, so a rule scoped to the account copies read as
covering the dumps too. It now also answers for `database/` alone, counting
only rules whose filter takes in a dump's key (prefix, legacy prefix, an
And filter; a tag filter never matches untagged objects), and the card
warns when no rule covers that prefix: dumps there, and the deleted accounts
in them, would never be removed. The reading moved into its own module.
A lost host takes its local dumps with it, and the copy left is the one in
the bucket. `docker compose run --rm db-restore latest --from-bucket` now
lists `database/` in the bucket the BACKUP_S3_* variables name, takes the
newest dump whose manifest is there too (or the one named), downloads both
with the S3 client the image already ships, and continues as for any
off-host copy: opened with BACKUP_ENCRYPTION_KEY, checked against the
manifest, restored into an empty database. `list --from-bucket` shows what
is there. The sealed download is removed once it is opened.
The image build now proves the S3 client resolves from the shared module's
place in the tree, so the restore cannot lose it unnoticed.
Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.
One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:
- database dump: failing once the newest good dump is older than 36 h
while dumps are on, or when it can be kept nowhere; read through a
provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
7 days after setup or as soon as a confirmed kit is out of date.
A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.
The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).
The step confirmed one key, the active application key. A restore needs
more: off-host copies are sealed under BACKUP_ENCRYPTION_KEY, a copy
written before a rotation needs the retired application key it was
written under, and every session, token and share link hangs off
API_TOKEN_HMAC_KEY. An operator could do everything the step asked and
still be unable to read a single backup.
The recovery kit lists every key a restore of this server needs, by id
and fingerprint, never key material:
- each id in ENCRYPTION_KEYS, plus any retired id a stored copy
(innerKeyIds) or an off-host object inside the bucket's retention
(OffhostBackupKeyUse) still lists, also once the server dropped it;
- BACKUP_ENCRYPTION_KEY and each BACKUP_ENCRYPTION_PREVIOUS_KEYS entry,
named by the id their objects record;
- API_TOKEN_HMAC_KEY.
Confirming records the set the admin was shown and refuses a stale one
with 409 recovery.kit.stale. The kit is due again as soon as a key it
lists is not in that set (a rotation, a new backup key, a re-keyed
install); a key that merely drops out does not make it due. "Check my
copy" now checks any key in the kit.
Migration 0487 carries a confirmation given under the old step forward
as a kit holding that key, so that part stays confirmed, and drops the
four app_settings columns. It also creates the two tables the readiness
evaluation uses (restore_drill_outcomes, recovery_readiness_notices).
The routes move to /api/admin/recovery-kit; the reader of the keys the
backups still need moves out of the encryption status route into
src/lib/recovery/backup-key-needs.ts so the kit and the status view
share one definition.
Marc-André Bombeck