The status prompts carry the unattended flag for background calls, the
queue list holds the lab date repair, and an account whose only way to
confirm is its identity provider is now offered that way.
420 000 heart-rate readings, one every five minutes over about four
years, written through the real Google Health backfill with Google's
API faked at the safeFetch seam. The account's DAY rollups lose their
reading extremes, as rows folded before 0417 have them; the boot
discovery finds it, and the windowed full backfill refills every bucket
in the fold window with the raw readings' own extremes. The dense fold
then thins everything older than 90 days to hourly Google Health rows
(420 000 rows to about 59 000) while the DAY tier keeps its figures,
and a second full-history backfill over the same samples inserts
nothing. Without the ingest guard it re-inserts about 394 000 of them.
Each step reports its duration on stderr. On a developer machine:
backfill about 85 s, discovery 0.2 s, full backfill 4 s, fold 40 s,
second backfill 4 s.
Nothing covered the operator's position after an erasure: a backup file,
an admin session, and a server that no longer knows the account. Export
the shared backup seed, delete the account through the settings route
under a 14-day window and run the purge job past its end, then:
- scan every text and JSON column for the account's id; only a
rate-limit bucket that expires within a day may remain;
- upload the file as an admin: refused as owner_not_found, nothing
stored (the documented contract);
- recreate the account row under its id, upload and restore: every
section's count matches the export;
- point the file at a freshly created account, upload and restore: every
count matches and nothing names the erased account afterwards.
Every backup test exports and restores with the same code, so a change
that moves both ends at once passes all of them and still breaks a file
a self-hoster downloaded last month. Commit the disaster-recovery file
the 1.46.1 exporter writes for the shared backup seed, with a manifest of
its counts and SHA-256, and restore each committed file through the admin
upload and restore routes into an empty account, comparing the re-exported
counts section by section.
The folder takes one file per release (GOLDEN_BACKUP_WRITE=1 writes the
current one; the README says when). The seed's own illness-symptom
catalogue key is swapped for a shipped one so every link resolves on a
fresh instance. Ciphertext is sealed under the suite's fixed key. A
measurement tombstone that ages past retention may only lower its count.
A full rollup backfill that failed on every boot showed only on the
admin page's card of failing background jobs. From the account's side
the only symptom was empty dashboard cards, and nothing pointed the
operator at the card (#1298).
A new hourly evaluation reads the newest terminal runs of every queue
whose failure has no account-facing surface (the system audience in
subsystem-surface.ts) from pg-boss's own ledger. A failed run adds its
attempts, retries included, and the first successful run ends the
streak. Three failed attempts in a row send every admin a SYSTEM_ALERT
once, a reminder at most once a day while the queue keeps failing, and
one notice when it runs successfully again: the decision the backup
health notices use, with the same compare-and-set claim, so two workers
never send a notice twice.
The memory is one row per announced queue in system_job_alert_notices
(migration 0490). The notice names the queue, the streak and the last
error, and opens the admin status page.
The full rollup backfill refills each day's figures from the raw
readings, and the dense fold deletes those readings once it has their
hourly means. On a large account the backfill can run longer than the
ten minutes the fold's boot job waits, and a year it reached after the
fold had taken it would compute its days from hourly means: the day
count, minimum and maximum of every folded day would change.
The consolidation walk now asks a per-account deferUser hook before it
reads an account, and both entries of the dense fold (the per-account
boot job and the nightly drain) pass isRollupFullBackfillPending, which
looks for a queued or running full fold of the account in pg-boss. The
account is folded on a later run, once its rollup tier is rebuilt.
There was no way for an admin to remove someone else's account; the
users list only offered the undo for a deletion the account itself or a
Guardian had asked for.
DELETE /api/admin/users/{id} takes the account's own deletion path,
now shared as requestAccountDeletion: with a grace window the account
is locked at once (credentials revoked, record untouched) and erased by
the purge job when the window ends, and the existing undo gives it back
until then; with ACCOUNT_DELETION_GRACE_DAYS=0 it is erased in the
request. The route is cookie-only behind requireAdmin and the
recent-proof gate, takes the typed DELETE_ACCOUNT confirmation, and
refuses the caller's own account, a managed profile (its Guardians
delete it), an account already pending deletion, the last admin who
can sign in and the last Guardian of a managed profile, the last three
decided under their locks inside the transaction. Both the admin and
the account get an audit row.
The users list says who asked for a pending deletion, now including an
admin, and flags managed profiles. Each other account's row carries a
delete button whose confirmation names the purge date, or says the
erasure is immediate when the server keeps no window.
The dense hourly fold only took Apple Health and Health Connect samples,
so heart rate from the provider syncs that store one row per sample
stayed raw forever. One self-hosted account held about 25 million
Google Health pulse rows after a full-history backfill, and every
reader and fold that falls back to the raw table paid for it (#1297).
The fold now walks a map of sources to the dense types each stores
sample by sample: every dense type for Apple Health and Health Connect,
heart rate for Google Health and Oura. Their daily SpO2 values are no
intraday stream and stay as they are; Fitbit does not sync intraday
heart rate, and WHOOP, Withings and Polar write dense types only once
per recovery, night or spot check. Each source folds into its own
hourly stats rows, and the DAY rollup is still taken from the raw
samples before they go.
The fold deletes the samples it averages, and the provider syncs write
first-write-wins on the externalId, so a backfill or full sync would
have put every folded sample back. Google Health would even have
re-keyed an hourly row into a sample that fell on its anchor through
the natural-key rescue. The folded_window rule now has one hour rule
per fold source, and both syncs drop the samples of an hour the fold
already took before they write. Recent samples cost no lookup.
The boot discovery of the fold probes each account and source for the
first raw sample it would take instead of grouping every such row, and
the one-time fold repair stays on the two sources it was written for.
The provider sent PUT /api/auth/me/locale on mount and on every switch,
signed in or not. Signed out the route answers 401, the once-per-tab
marker was never set, and every signed-out page load repeated the
refused write.
The mirror moves out of the provider into LocaleServerSyncBridge, which
watches the account payload without fetching it and writes only once an
account is known. The choice made on the sign-in page therefore reaches
the account right after signing in, and the marker is keyed on the
account as well as the language, so another account signing in on the
same tab is written for too. The browser test stub no longer treats the
PUT as ambient, which is what lets its test see a stray write.
An account created through OIDC holds no password, passkey or second
factor, so the recent-proof gate in front of token minting, MCP
connections, exports and the admin reset had nothing to ask it for and
the web could only say "sign in again" (#1296).
GET /api/auth/oidc/reproof sends the browser back to the configured
provider with prompt=login and max_age=0, its own state, nonce and PKCE
verifier, and the signed-in account and session sealed into the state
cookie. The shared callback branches on that intent: it refuses unless
the browser still carries the bound session (checked before the code is
redeemed), the ID token names the account's linked issuer and subject,
and auth_time lies inside the gate's five-minute window. It then stamps
Session.reproofAt exactly as a password re-proof does, creates no
session, and returns to the page it left with ?reproof=done|failed.
recentProofMethods offers "oidc" only to an account without a second
factor whose identity is linked while a provider is configured, and the
fresh-factor gate never lists it. GET /api/auth/me/mfa publishes the
list as proofMethods, so the passkey card offers what the account can
actually give, and register-options takes { method: "oidc" } on a fresh
provider re-proof, so such an account can add a passkey as a second way
to confirm. Both re-proof dialogs offer the provider and a toast reports
the outcome on return.
The callback's missing-state branch and the passkey card's enrollment
flow move into their own functions to keep both handlers under the size
ratchet.
Systolic, diastolic and the cuff's pulse are three rows with no link
column, and the single DELETE tombstoned only the row it was given, so
deleting a reading's systolic left its diastolic and pulse behind as half
a reading. The bulk delete had the same gap.
A reading is the rows with the same owner, the same source and the
identical measuredAt: that is how the entry form, the MCP write, a
Withings measure group and a HealthKit correlation all store one, and how
the FHIR read pairs a diastolic onto its systolic. The looser five-minute
pairing of the chart reads is right for drawing a point and wrong for
taking rows away.
Deleting a systolic or diastolic now tombstones the rest of its reading in
the same transaction with the same deletedAt, each row with its own
syncVersion bump so paired clients see every tombstone. The restore route
brings back the rows deleted together with a named half (identical
deletedAt), so the undo, which names only the row that was tapped, puts
the whole reading back, from the web list and from any client that calls
it the same way. A pulse is still deleted on its own, and a row deleted at
another moment stays deleted. The response shapes do not change; the
OpenAPI descriptions say what the counts now include.
On an account with tens of millions of readings the boot full backfill
never finished. It ran four aggregates over the whole five-year window,
each held to the connection's 60 s statement limit, so every attempt
was cancelled with 57014, the reading extremes added by 0417 stayed
NULL, and the readers that need them fell back to the raw table and
timed out as well (#1298).
The full backfill now folds the window one UTC calendar year per pass,
newest first, and every statement runs under its own five-minute limit,
the shape the pulse day-mean rebuild already uses. Calendar-year cuts
write exactly the buckets one pass writes: day, month and year buckets
fall inside one window and a week across New Year is folded whole by
both. Each window commits on its own, so a run that reaches its job
budget hands the remaining years to a follow-up job instead of failing
and starting over on the retry.
The boot discovery read every live reading in the window to find
uncovered days, the same scan that kept the fold from finishing. It
now walks the days per account and type off the live index, so its
cost follows the number of days with readings, and runs under the same
limit. The full backfill code moves to its own module.
Before the confirm step dated an undated value by its document, a lab
value confirmed from a report that prints its collection date only in the
header was stored with takenAt set to the moment of confirmation. The fix
covers new confirms; the values already filed that way stay on the wrong
day until something moves them.
A one-shot maintenance job, sent at boot only while there is something to
repair, moves exactly those values to the document's report date (else its
filing date) at noon UTC, the same rule the confirm step now uses. A value
qualifies only when the chain is certain: written by the document confirm
path, named by exactly one approved observation fact, takenAt still within
five seconds of its own createdAt, no calendar date of its own in the
decrypted fact, and a document day that differs from the day the value is
filed under in the owner's zone. A value re-dated since, one really taken
on its confirmation day, or one whose document has no date is left alone.
A moved value no longer sits at its creation moment, so the pass converges
and a second run changes nothing.
The job definition lives in a new registry file beside maintenance.ts so
that file stays under the size ratchet.
A range longer than a year reads its series from the week or month
rollups, which are kept for the last five years. When the rollups held
any rows the read returned them alone, so on a record that goes back
further the All range started where the rollups start: a weight history
from 2015 charted from late 2021. The readings older than the first
rollup day are folded live at the tier's grain and lead the series; the
head ends where the first tier bucket begins, so no month is counted
twice. One indexed lookup when there are none.
Against a real Postgres and the real confirm route: an undated value takes
the report date over the filing date, a value's own date is kept, an
impossible own date counts as none, and a document without a report date
dates the value by its filing date.
The forwarder ran every tool as docker exec -i with its own standard
input piped through, also for pg_dump, which reads none. With -i and an
input already at its end the docker client may close the stream before
all output has arrived; one CI run saw the second of three nightly dumps
fail where every local run and the two runs before passed. Attach stdin
only for a file fed to pg_restore, and put the run's error into the
assertion so a failure says why.
realisedFamily probed for the closest two slots up to range.to only, so
the answer moved with the range. A Saturday + Sunday plan started the
weekend before the autumn DST change, read around its Saturday dose: the
±24 h range ends at 06:00 UTC on the Sunday, whose 08:00 slot is at 07:00
UTC after the change. Cut off, the closest slots were six days apart
(weekly); the wide re-mint saw the 25 h gap (daily). The two passes
disagreed, no band survived, and the dose was filed as unscheduled, so a
second backdated dose within a minute came back as the first one (200).
Probe the same reach on both sides of the range's middle.
The list ended 24 hours after local midnight. On the 25-hour day of an
autumn DST change that left the last hour out, so a dose due then could
not be offered from it; on the 23-hour spring day it reached into the
next day. Take the day's bounds from getUserTodayBounds, as the dashboard
tile does. Anchor the health-status test's history to local noon: now
minus whole days is still today on the long day.
The new backup, dump and deletion files were unknown to the duration
sequencer and gathered in one shard, which ran past its job timeout
while the other three finished in under fifteen minutes.
The list reads ordered schedules by their window and id, but the detail,
create, update, compliance, dose-history, GLP-1 and efficacy reads took
them in whatever order Postgres found them, so two schedules created in
the same millisecond could swap places between two answers. Every read
now uses MEDICATION_SCHEDULE_ORDER, and the guard checks every
schedules include, not only the list reads.
A background AI run's deadline is armed in model time plus a fixed margin.
Time spent waiting in the document reader's queue, and the read itself, came
out of that margin, so a person's lab scan or document read could be timed
out by the reaper because other work held the reader. When the reader grants
the slot it now moves the run's deadline by the wait plus the most the read
may take, and gives back the unused part when the read ends, never past the
job's own expiry and only while the run is running. The interactive queue
wait stays within the margin, which covers the run until the slot is
granted.
The per-account free-text encryption backfill seals what is already stored
and changes nothing the record holds, which is why its module is exempt from
the frozen-account rule; the new pending-deletion gate dropped its per-user
jobs anyway. A queue can now say it acts for pending accounts
(actsForPendingAccounts), and this one does.
Also types the ingest route test's token fixture to the row it now reads,
and drops an unused variable from the deletion grace test.
A cookie named healthlog_session can be set by any host under the same
registrable domain, so a page on a sibling subdomain could plant a session of
its choosing in a visitor's browser. Wherever the session cookie is Secure it
is now issued as __Host-healthlog_session, which a browser accepts only from
this exact host. Plain-HTTP installs, which cannot hold a __Host- cookie,
keep the old name.
Nobody is signed out: the old name is still read after the new one, a
session found under it is re-issued under the new name and the old cookie
expired, and a browser holding the new cookie is never answered from the old
one. Every reader and writer goes through one module, which the proxy, the
native-caller check, the native handoff and the OIDC callback use; the
session-surface guard follows the new reader.
A drill that stopped on its time budget before the first object recorded
ok with zero accounts checked, and the readiness tile said passed. Such a
run now records its time and counts and no verdict, so the last pass or
failure stands, and readiness calls it attention (nothing_checked) in every
locale.
The upsert that keeps a pass and a failure apart was stubbed in the drill's
tests and tested nowhere; it now has an integration test read back through
the readiness rule. The drill test that claimed to keep a failed verdict
when keeping it fails is named for what it checks.
Restoring the copy a data wipe took withdrew every held data_wiped purge of
the account. With two wipes in one window, restoring the first copy also
withdrew the second wipe's purge, and the off-host copies of what the person
entered between the wipes, and wiped, stayed until the bucket's rule removed
them. Only the request written after this copy and before the next one goes
now. The withdrawal had no test; it has one against Postgres.
The recovery kit, "Keys the backups still need" and safeToDropRetiredKeys
read only the stored copies and the off-host account objects. After a
rotation the page called the old key safe to drop while every dump taken
before it, the pre-migration dumps that are the way back from an update
among them, could only be restored with that key: db-restore refuses a dump
whose active key is missing.
A dump now counts for the key its rows were written under, the manifest's
active key: a dump on this server while it is kept (read from the manifests),
an off-host one until the bucket's rule for database/ expires it, or
indefinitely when that rule cannot be read. The nightly run records the key
on its ledger row (migration 0489, database_backup_runs.active_key_id). The
encryption view and the kit list the dumps per key, in every locale, and the
rotation and backup runbooks say so.
The enumeration guard read any query that named userId as scoped, including
an orderBy tiebreak, so the nightly recovery, strain and stress score jobs
grouped every account's measurements, wrote scores into frozen records and
spent cap slots on them. It also looked only at findMany and groupBy under
lib/jobs.
The guard now reads a scope from a query's filter only (orderBy, by,
distinct, select and include are dropped first), covers count, aggregate,
updateMany, deleteMany and executeRaw, and walks the boot discoveries and
sweeps that live beside their domain: the rollup and mood rollup discovery,
the Fitbit jobs, the cumulative record rederivation, the intake slot dedup,
the inventory expiry and the climb backfill. What it then found is filtered
(the score cohorts through excludePendingOwners, which filters before the
cap; Fitbit sync, backfill and sleep repair; coach proposal lapsing and
reminder nagging; expired coach reminders; snooze clearing; inventory
expiry) or says why it needn't be (retention and expiry sweeps over rows,
writes scoped by id).
Jobs queued before a deletion ran anyway, since handlers act on the userId
they carry without asking again. The binding of every queue now completes a
job whose payload names an account pending deletion without loading its
handler (pending-deletion-gate). The silent medication intake push, which
does not go through the dispatcher, refuses a pending account itself.
A handover link stayed claimable after its profile was deleted: the claim
gave the profile credentials that every sign-in then refused, and the purge
erased the account someone had just taken over. Locking an account for
deletion now revokes the open links for that profile and the links that
account minted, and creating, previewing and claiming a link refuse a profile
pending deletion, the claim under the record lock.
The last-Guardian rule counted Guardians pending deletion and profiles
pending deletion. A Guardian who had deleted their only profile could not
delete their own account; two Guardians sharing a profile could both leave,
and the second one's erasure was then refused on every run. The rule now
counts only Guardians whose account is not pending deletion, ignores profiles
pending deletion, and is decided under the record lock for account deletion,
data wipe, revoking and renouncing a Guardian. An admin's undo of a managed
profile with no such Guardian left is refused (409
account.deletion.no_guardian).
A run that kept its dump on the volume and failed the upload is the newest
good dump and the newest failure at the same instant. Compared with ">", the
card showed no failure and readiness said fresh, night after night, while no
dump left the host.
The health read now carries the off-host leg separately: when the newest dump
reached the bucket, and the newest one that did not. One pure reading of it
(databaseDumpProblems) feeds both the backups card and the readiness model.
A configured bucket that missed the newest dump is attention (offhost_failed)
with the upload's error, and failing once the newest dump there is older than
36 hours (offhost_stale) or none ever arrived (offhost_never). The card shows
the refused upload and when the bucket last got a dump. A failure at the
instant of the newest dump for another reason, retention throwing, is now
shown too.
The reproductive-health page promised that deleting all data or the
account erases at once. Both wait for the grace window now, the nightly
dumps keep a record until they age out, and the old pgdata volume is a
full copy until it is removed; the page says so and how to make deletion
final. The README, the getting-started note on majors, the move guide and
the Unraid and Portainer templates catch up with Postgres 18, the backups
volume and the recovery kit. The demo seed's spread bound leaves room for
the seed's own one-in-150 spread.
Only the backups page said when the bucket's lifecycle rules left
database/ out, so every dump, and every deleted account inside it, stayed
in the bucket forever without anyone being told. Readiness gains a
dump_retention signal that reads the same cached lifecycle probe: attention
when dumps go off-host and no rule covers database/, ok with the expiry
when one does, quiet when the bucket will not say, inactive without a
bucket or without off-host dumps. The hourly evaluation pushes it to the
admins like every other signal.
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.
A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.
The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
With the real dump health wired in, an empty test database also reports
a missing dump. The file follows the weekly copy alone; the dump signal
is judged in the readiness unit tests.
When the grace window ends for an account that has become the last
active guardian of a managed profile, the purge job refuses to erase it
and the account stays locked. Until now only the worker error log said
so. Every admin now gets one localized notification naming the account
and the ways out (another guardian, deleting the profile, or undoing
the deletion). An audit row on the account records that it was reported,
so the hourly retries stay quiet; a deletion undone and asked for again
is reported afresh.
The undo confirmation in the admin users list now says that sign-in
works again but API tokens, share links and trusted devices stay revoked
and have to be set up again (for a managed profile: its share links and
tokens).
The shared backup reader kept documents without their stored bytes but
kept every ECG strip whole, trace included. A long watch history is
hundreds of strips of tens to hundreds of kilobytes each, so an upload,
preview or restore of such a copy held the whole history in memory.
The first read now keeps each strip without its trace (a short stand-in,
after the key it was written under is recorded and its form checked),
and forEachEcgTrace reads the traces again one strip at a time. The ECG
restore writes each strip from that second read, as the document restore
already did, and refuses a file whose two reads disagree on the count.
The upload memory test gains an ECG-heavy file: 150 strips, about 200 MB.
The old reader held 197 MB of it and failed the 48 MB bound; the new one
holds about 1 MB, also under a 524 MB heap.
Deleting an account, a guardian deleting a managed profile, and "Delete
all data" were final the moment they were confirmed: the account row
cascaded into every table and every stored copy, and the off-host objects
were purged in the same breath. Nothing short of rolling the whole
instance back could return one account.
ACCOUNT_DELETION_GRACE_DAYS (14 by default, at most 30, 0 erases at once
as before) now sets a window:
- Account and managed-profile deletion lock the account (migration 0488:
deletion_requested_at, deletion_purge_at, deletion_requested_by_id).
Every session, refresh token, API token, connector, share link and
step-up elevation is revoked in the same transaction, and the record,
its stored copies and its off-host objects are left as they are.
- Sign-in is refused with 403 auth.account.deletion_pending and the
erasure date, after the credential was proved and before a second
factor is asked for. The session and Bearer resolvers and the acting-
account resolver refuse a pending account too, so a credential minted
in the moment before the lock is useless.
- POST /api/admin/users/{id}/undo-deletion (cookie-only, audit-logged)
clears the lock. The admin users list reports the pending state.
- The hourly account-deletion-purge job erases accounts whose window has
ended, exactly as the immediate deletion did, with the off-host purge
request committed alongside. The Guardian invariant is decided again;
a refused erasure stays pending and is reported.
- A wipe first stores the whole record as a PRE_WIPE_<ms> copy through
the writer the weekly copy uses, keeps it out of the wipe, and holds
its off-host purge back with a new not_before on the request. The
ordinary restore brings the record back; restoring that copy withdraws
the held purge. The purge job deletes the copy when its window ends.
Background work skips accounts pending deletion through one shared
predicate (src/lib/account-deletion/active-account.ts): every reminder,
digest, Coach, sync, backfill, rollup, score and backup enumeration
filters with it, and account-deletion-enumeration-guard.test.ts fails on
a query in background work that neither filters nor says why it need
not. Notification delivery and the AI capability loader refuse a pending
account as chokepoints behind it, and the Withings, WHOOP and Telegram
webhooks and the notification quick actions ignore one.
The privacy summary publishes the window. Two pieces of the reminder tick
and the off-host pass moved into modules of their own to keep the files
inside their size ratchet.
Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.
One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:
- database dump: failing once the newest good dump is older than 36 h
while dumps are on, or when it can be kept nowhere; read through a
provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
7 days after setup or as soon as a confirmed kit is out of date.
A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.
The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).
The step confirmed one key, the active application key. A restore needs
more: off-host copies are sealed under BACKUP_ENCRYPTION_KEY, a copy
written before a rotation needs the retired application key it was
written under, and every session, token and share link hangs off
API_TOKEN_HMAC_KEY. An operator could do everything the step asked and
still be unable to read a single backup.
The recovery kit lists every key a restore of this server needs, by id
and fingerprint, never key material:
- each id in ENCRYPTION_KEYS, plus any retired id a stored copy
(innerKeyIds) or an off-host object inside the bucket's retention
(OffhostBackupKeyUse) still lists, also once the server dropped it;
- BACKUP_ENCRYPTION_KEY and each BACKUP_ENCRYPTION_PREVIOUS_KEYS entry,
named by the id their objects record;
- API_TOKEN_HMAC_KEY.
Confirming records the set the admin was shown and refuses a stale one
with 409 recovery.kit.stale. The kit is due again as soon as a key it
lists is not in that set (a rotation, a new backup key, a re-keyed
install); a key that merely drops out does not make it due. "Check my
copy" now checks any key in the kit.
Migration 0487 carries a confirmation given under the old step forward
as a kit holding that key, so that part stays confirmed, and drops the
four app_settings columns. It also creates the two tables the readiness
evaluation uses (restore_drill_outcomes, recovery_readiness_notices).
The routes move to /api/admin/recovery-kit; the reader of the keys the
backups still need moves out of the encryption status route into
src/lib/recovery/backup-key-needs.ts so the kit and the status view
share one definition.
Marc-André Bombeck