The stat strip reads the chart's visible window, so with the chart
narrowed to the morning its figures were morning figures under a plain
Systolic label. The column label now says so. Russian texts for the
time-of-day card.
Home monitoring reads morning and evening readings as separate sets, so
the blood pressure page now shows the morning, midday and evening means
side by side over the chart's range, each with its count and pulse, and
the morning-minus-evening difference. A control under them narrows the
chart to one part of the day; the choice lives in `?partOfDay=` so it
survives a reload and a shared link, and the whole day keeps a bare URL.
The split is resolved on the server in the record's own time zone and
published as one response, `GET /api/measurements/blood-pressure/time-of-day`,
so every client renders the same numbers. Morning is 04:00 to 11:59,
midday 12:00 to 17:59, evening 18:00 to 03:59: the guidelines define the
sets by waking and bedtime rather than by the clock, so the windows are
wide, and a reading after midnight is a late evening one. Pairs follow
the blood-pressure series' rule (same source within five minutes); the
pulse is the reading's own. A part shows a mean from three readings on.
`GET /api/measurements` takes `partOfDay` on the blood-pressure and pulse
series read: each local day folds only that part's readings, a day runs
from 04:00, and long ranges fold weekly instead of reading rollups that
cannot tell the hour.
Refs #1331
On a host whose CPU cannot run the page renderer (x64 without AVX2, arm64
musl without LSE/RCpc), every AI read of a PDF for a provider that only
takes images failed with "Scanning PDFs needs a Claude vision provider",
which says nothing about the cause. Most medical PDFs are produced
digitally and carry every word in their text layer, which pdf.js reads
without rendering.
`prepareVisionInput` now answers a text read (`mode: "text"`) when the
renderer is unavailable on this host, the provider cannot take the PDF
whole and the PDF has a usable text layer. Summary, suggest and the
background summary send that text through their existing text mode;
"Read with AI" and the index backfill write it as the local text index it
is, with no provider call; the automatic index leaves it to its local path.
`planVisionReads` does the same for the extraction and the lab scan: text
reads by page group, with anchored provenance, and the pages without a text
layer reported as `extractionIncomplete` on a document and as `unreadPages`
on a lab scan. A supplement label PDF is read from its text layer too.
A scan on such a host now answers `documents.inbound.pdfRendererUnavailable`
and `labs.ocr.pdfRendererUnavailable` (422): this server's processor cannot
run the page renderer, so a scanned PDF can only be read by a provider that
takes PDF files directly, or as photos. The `pdfNeedsAnthropic` codes are no
longer emitted, since with a working renderer every provider reads a PDF;
the recorded index outcome keeps its `pdf-needs-anthropic` value with
corrected copy.
The extraction path no longer folds a render failure of one document into
the capability error: like `prepareVisionInput` since #776 it reports
"The PDF pages couldn't be rendered for scanning." (`extractFailed`), and
the lab scan answers `labs.ocr.extractFailed`.
Hosts that render, and providers that take PDFs whole, are unchanged.
Refs #1311
An injected dose already stores where it went, but the per-medication
history only showed when it was taken and where the entry came from.
The list now renders the site as a chip next to the status, on the
desktop table and the mobile cards, so the rotation can be read back
from the history. Rows without a site are unchanged.
- baselines: take main's entries, keep the two ru-driven line counts (coach-prose-grounding 1091, insight-generator 1115) and regenerate the function baseline.
- messages/ru.json: translate the five unit-less markers keys the release added (labs.unit.none/noneHint, labs.ocr.errorUnitMismatchNoUnit, labs.ocr.unitDiffersBadgeNoUnit, recordImport.unitlessHint) — parity is 11 159 keys again.
Dose bank
- DRUG_UNIT learns мг / мкг / мл / ЕД / МЕ, so "2,4 мг" anchors a dose again.
- The bank matches verb-first Russian word order too ("принимайте 2 таблетки",
"пропустите дозу метформина", "начните с 5 мг"), and the Cyrillic drug names
people actually see on the box; the deferral words ("спросите врача",
"как назначено") stand those shapes down.
Cyrillic-safe regexes
- \w and \b are ASCII in JavaScript: every stem in the ru bank and in
population-norms/ru.ts is [а-яё]*, and the word edges are lookarounds
(?<![а-яёa-z]), so «высокий риск» and «нормальный диапазон» match again.
- coach-prose-grounding: Russian month names in the genitive, a space as the
thousands separator in parseLocaleNumber and numFrag, and Cyrillic-safe
edges around the written-date pattern.
Registered everywhere a locale has to be
- targetLanguageName() returns "Russian", so the coach, the period narrative
and the document descriptions stop saying "write in English".
- i18n-key-tool.mjs, split-description.mjs, LOINC name normalisation (it
stripped Cyrillic, so no Russian lab name resolved), HOST_TLDS (ru/by/kz/ua),
no-key-fallbacks (32 tables), country-label, the AUDIT-C standard drink.
- docs/api/openapi.yaml regenerated.
Questionnaires
- The PHQ-9 / GAD-7 source comments no longer claim no validated Russian
version exists (Zolotareva 2023; Pfizer's official screeners).
- WHO-5 items are gender-neutral: «чувствовал(а)».
- "body fat" and "fat mass" both read «Жировая масса»; body fat is now
«Процент жировой массы», which is what the two metrics measure.
Tests
- ru in the 13 locale tables, plus mood, onboarding, instruments, clinician
view, status summary, describe screen, citation coverage and LOINC.
- i18n-no-dot-separator: the ru bundle is exempt from the dash rule (a spaced
dash is Russian punctuation, not a fragment join); the middle dot still
fails there.
- Two 800-line baselines raised by the 12 and 5 lines this change adds.
pnpm typecheck, pnpm openapi:check and the full suite (36 316 tests) pass.
A ratio (cholesterol/HDL, albumin/globulin) or an index such as INR has
no unit, and every lab write path refuses a numeric value without one.
That refusal stays: an empty unit usually means the extraction dropped
it. Instead the value is stored as the UCUM unity "1", chosen on purpose.
The unit field when a marker is defined, on a scanned report's row and
in the document review edit now carries a "No unit" checkbox that sets
"1". The CSV import takes 1 in the unit column, as the template and the
import sheet say; an empty cell is still refused, and other spellings
such as "ratio" or "-" stay the literal unit they always were.
"1" compares as a unit of its own: it matches only itself, so it is
refused under a marker kept in mg/dL and the other way round.
Refs #1313
Full Russian catalog (11,154 keys, 1:1 with en, identical nesting) and every
locale wiring point the app declares, so ru is a first-class locale instead
of an English fallback:
- messages/ru.json, project.inlang/settings.json, i18n boot script
- i18n config / load-locale, format-locale (ru-RU), date-fns ru
- labs: parse-reference-range RANGE_PROSE for Russian lab sheets
("3,5 - 5,0", "от 3,5 до 5,0", "до 5,6", "не более 5,6")
- AI: citation-coverage normative stems, coach system prompt and insights
reply footer (Russian prompts must override the hard-coded
"Respond in English"), population-norm patterns
- safety: outbound-screen Russian imperative/dose/unit/causality bank
- mental health: PHQ-9, GAD-7, WHO-5 and SCI wording
verify-message-catalogs.mjs passes for all 10 locales.
Restored rows keep the ids they had in the file. Once an erased
account's copy had gone into one account, the same file uploaded into a
second account restored into a primary-key conflict, reported only as
the generic "the database rejected a record" failure.
For a copy put into a chosen account, a primary-key conflict now ends
the restore with backup.restore.rows_exist_elsewhere and a sentence that
says the file was most likely restored into another account already.
The transaction still rolls back, so neither account changes. The code
is in the catalogue, the console names it, and the runbook says a file
goes into one account only.
A restore read the account settings under the operator's AI endpoint
policy whenever the receiving account was an admin. For a copy put into
a chosen account, the settings were written by the erased account's
owner, not by the operator, yet an admin account could be chosen as the
target, and an ordinary target promoted later got the same trust.
Admin accounts are now refused as a target, in the check and the
picker, with backup.target.admin_account. A copy uploaded into a chosen
account is stored as RETARGETED_UPLOAD_<ms>, and its restore never reads
its settings as the operator's, whatever the account's role is by then.
A values list naming more than 24 lab markers answered 422. The list is a
device preference, so once a 25th marker was ticked every later visit
failed: the page rendered its error card, and the values menu that could
untick it only renders on success.
The server now keeps the first 24 markers and leaves the rest out, as it
already does for a marker the record no longer has; a malformed key still
answers 422. The page applies the same cap before it asks, and the values
menu holds further unchosen markers back at 24 with a line saying why,
while a chosen one can always be removed.
With the server naming lab markers as value lines, the page offers them:
the values menu lists the markers the record has numeric readings for,
under the groups the labs list sorts them into, and a choice is stored
and requested like any measurement line. The phone toolbar gains the same
menu as an icon button, since the chronicle shows the chosen lines too.
A lab row sits under the value lines in their rhythm, on its own scale.
Each reading is its own point on the day of the draw, joined in time
order by a thin stroke and never averaged. The reference range is painted
behind the points as the labs page paints it, a faint band with dashed
edges, and the scale holds it so the band is always in view. A reading
outside its range is a triangle pointing the way it left and is worded as
outside wherever it is written out, so colour is never the only sign.
A point reads on its own by pointer, keyboard and tap: the day, the value
with every digit it was stored with, the verdict, the range and the
report it was read from (a link once pinned). The selection bar lists
the bucket's readings with their report, a lab day's row in the phone
chronicle lists that day's readings, a screen-reader table lists them
all, and a marker with readings in a unit that does not convert says how
many were left out. Nothing else on the timeline changes.
To make room without growing the large timeline files, the legend, the
screen-reader tables and the value-line geometry move into modules of
their own, and the chart, the chronicle, the point cards and the page
each hand a part to a helper; the size baselines are written back lower.
Refs #1299
An upload refused with backup.owner.missing no longer ends in an error
toast. The backups console keeps the file and opens a dialog: a new
account by default, which cannot hold anything yet, or an existing one,
which the server checks as soon as it is picked and which the admin
confirms by typing its username. Managed profiles and accounts pending
deletion are not offered. The file is then sent again with that target,
and a refusal stays in the dialog beside the choice it refuses.
The new-account path says the account has no password yet and where to
set its first one. Copy in every app language.
A full rollup backfill that failed on every boot showed only on the
admin page's card of failing background jobs. From the account's side
the only symptom was empty dashboard cards, and nothing pointed the
operator at the card (#1298).
A new hourly evaluation reads the newest terminal runs of every queue
whose failure has no account-facing surface (the system audience in
subsystem-surface.ts) from pg-boss's own ledger. A failed run adds its
attempts, retries included, and the first successful run ends the
streak. Three failed attempts in a row send every admin a SYSTEM_ALERT
once, a reminder at most once a day while the queue keeps failing, and
one notice when it runs successfully again: the decision the backup
health notices use, with the same compare-and-set claim, so two workers
never send a notice twice.
The memory is one row per announced queue in system_job_alert_notices
(migration 0490). The notice names the queue, the streak and the last
error, and opens the admin status page.
The French bundle mixed the familiar and the formal address: about a
third of the strings said "tu", "ton" or "Réessaie", the rest "vous",
sometimes both on one screen and once in one string.
Every string that addressed the reader as "tu" now uses "vous" with the
matching conjugation, possessives and agreement; placeholders, plural
branches and punctuation are unchanged. The Coach prompts a person
sends address the Coach the same way, the casual "Salut" greeting of
the nudges becomes "Bonjour", and "après que" takes the indicative.
A Dutch reader whose screener answer raised the safety path saw only
the international set, because no Dutch set existed.
The Netherlands gets 113 Zelfmoordpreventie (113, the free 0800-0113
and the chat on 113.nl), Belgium the Zelfmoordlijn (1813,
zelfmoord1813.be), both behind 112. The app's one Dutch bundle is read
in both countries and the account keeps no country, so a bare "nl"
lands on a combined set that names each line with its country, the way
the German and Portuguese sets already do; an explicit nl-NL, nl-BE or
region "be" gets its own country's set.
The header always said "welcome back", including to an account the
setup flow had created a few minutes earlier.
/api/auth/me now publishes the account's createdAt, and the greeting
welcomes an account in its first day ("Welcome, name.") and welcomes
back every other one, including an account whose older server does not
send the field. Until the account is known, during the SSR pass and the
hydration render, the line is the time-of-day greeting alone, so
nothing claims "back" before the client knows it is true.
There was no way for an admin to remove someone else's account; the
users list only offered the undo for a deletion the account itself or a
Guardian had asked for.
DELETE /api/admin/users/{id} takes the account's own deletion path,
now shared as requestAccountDeletion: with a grace window the account
is locked at once (credentials revoked, record untouched) and erased by
the purge job when the window ends, and the existing undo gives it back
until then; with ACCOUNT_DELETION_GRACE_DAYS=0 it is erased in the
request. The route is cookie-only behind requireAdmin and the
recent-proof gate, takes the typed DELETE_ACCOUNT confirmation, and
refuses the caller's own account, a managed profile (its Guardians
delete it), an account already pending deletion, the last admin who
can sign in and the last Guardian of a managed profile, the last three
decided under their locks inside the transaction. Both the admin and
the account get an audit row.
The users list says who asked for a pending deletion, now including an
admin, and flags managed profiles. Each other account's row carries a
delete button whose confirmation names the purge date, or says the
erasure is immediate when the server keeps no window.
An account created through OIDC holds no password, passkey or second
factor, so the recent-proof gate in front of token minting, MCP
connections, exports and the admin reset had nothing to ask it for and
the web could only say "sign in again" (#1296).
GET /api/auth/oidc/reproof sends the browser back to the configured
provider with prompt=login and max_age=0, its own state, nonce and PKCE
verifier, and the signed-in account and session sealed into the state
cookie. The shared callback branches on that intent: it refuses unless
the browser still carries the bound session (checked before the code is
redeemed), the ID token names the account's linked issuer and subject,
and auth_time lies inside the gate's five-minute window. It then stamps
Session.reproofAt exactly as a password re-proof does, creates no
session, and returns to the page it left with ?reproof=done|failed.
recentProofMethods offers "oidc" only to an account without a second
factor whose identity is linked while a provider is configured, and the
fresh-factor gate never lists it. GET /api/auth/me/mfa publishes the
list as proofMethods, so the passkey card offers what the account can
actually give, and register-options takes { method: "oidc" } on a fresh
provider re-proof, so such an account can add a passkey as a second way
to confirm. Both re-proof dialogs offer the provider and a toast reports
the outcome on return.
The callback's missing-state branch and the passkey card's enrollment
flow move into their own functions to keep both handlers under the size
ratchet.
With the deletion grace window on, the confirm dialog still asked to
delete the account permanently and its button said so, while the body
promised fourteen days and an admin undo. The title and the button now
match the window, and the success message that names the erase date stays
long enough to read before the sign-in page.
The new account-copy explainer under the Backups heading ran to 99-113
characters in German, Spanish, French, Italian, Polish and Portuguese and
wrapped to a second line at desktop width, so that page's header sat
20 px taller than every other settings and admin page. The German run of
the settings/admin header measurement failed on it. The Korean text was
two sentences in a one-sentence slot.
Each is now one sentence that fits the 1280 px column; English and Dutch
already did and stay as they are.
A read a person started now runs as interactive work of their account, and
the document jobs as background work of theirs, so the reader's queue can
order and cap them. When the reader is busy or stopped before it answered,
a lab scan, label read or document read fails with aiRuns.readerBusy (503,
nothing read or charged) instead of a misleading unreadable-file error, and
the index and summary jobs throw so the queue runs them again rather than
recording an outcome about the document.
A restore of the copy a data wipe took replaces the record with that copy,
so anything added after the wipe is gone, and the wipe ended integration
connections, sharing in both directions, Guardian roles and share links,
none of which are in a copy. Neither the person's confirmation nor the
admin's restore dialog said so. Both do now, in every locale.
A drill that stopped on its time budget before the first object recorded
ok with zero accounts checked, and the readiness tile said passed. Such a
run now records its time and counts and no verdict, so the last pass or
failure stands, and readiness calls it attention (nothing_checked) in every
locale.
The upsert that keeps a pass and a failure apart was stubbed in the drill's
tests and tested nowhere; it now has an integration test read back through
the readiness rule. The drill test that claimed to keep a failed verdict
when keeping it fails is named for what it checks.
Pre-migration dumps were kept by count alone, so an install that updates
rarely kept its last two, and every account deleted since, indefinitely,
while the docs and the backups card stated the nightly retention as the
retention of deleted accounts. A pre-migration dump older than the oldest
nightly dump kept now goes, except the newest one, the way back from the
latest update, which stays until the next update. The card, the disaster
recovery guide and its variable table say so.
The recovery kit, "Keys the backups still need" and safeToDropRetiredKeys
read only the stored copies and the off-host account objects. After a
rotation the page called the old key safe to drop while every dump taken
before it, the pre-migration dumps that are the way back from an update
among them, could only be restored with that key: db-restore refuses a dump
whose active key is missing.
A dump now counts for the key its rows were written under, the manifest's
active key: a dump on this server while it is kept (read from the manifests),
an off-host one until the bucket's rule for database/ expires it, or
indefinitely when that rule cannot be read. The nightly run records the key
on its ledger row (migration 0489, database_backup_runs.active_key_id). The
encryption view and the kit list the dumps per key, in every locale, and the
rotation and backup runbooks say so.
The hourly purge rethrew any error that was not the Guardian refusal, so the
oldest due account, failing on every run, stopped every later erasure and the
sweep of expired pre-wipe copies for good.
Each account is now its own try: a failure is counted, reported through the
worker error path with its first error, and told to the admins once per
request (user.account.erasure_failed), and the run goes on. The pre-wipe
sweep and the off-host kick run in a finally. Every account the run starts is
marked in backup_pass_attempts under the purge's queue name and the mark
closes only on erasure or when the account is no longer due, so an account
that failed or was refused goes behind every other due account next time.
The run stops starting accounts once its job budget is spent, since fifty
erasures of up to two minutes each did not fit the queue's expiry.
A run that kept its dump on the volume and failed the upload is the newest
good dump and the newest failure at the same instant. Compared with ">", the
card showed no failure and readiness said fresh, night after night, while no
dump left the host.
The health read now carries the off-host leg separately: when the newest dump
reached the bucket, and the newest one that did not. One pure reading of it
(databaseDumpProblems) feeds both the backups card and the readiness model.
A configured bucket that missed the newest dump is attention (offhost_failed)
with the upload's error, and failing once the newest dump there is older than
36 hours (offhost_stale) or none ever arrived (offhost_never). The card shows
the refused upload and when the bucket last got a dump. A failure at the
instant of the newest dump for another reason, retention throwing, is now
shown too.
A text read that crossed the isolated reader's memory or time bound came
back as a plain read error, which the index job records as decrypt-error,
so the document said its stored file could not be opened. The local
extractor now answers resource-limit for PdfResourceLimitError, the index
records it under that name, and the detail view says the PDF is too large
or complex to read here, in every locale. Like the thumbnail's refusal of
the same bytes it is final: nothing retries it on its own.
Only the backups page said when the bucket's lifecycle rules left
database/ out, so every dump, and every deleted account inside it, stayed
in the bucket forever without anyone being told. Readiness gains a
dump_retention signal that reads the same cached lifecycle probe: attention
when dumps go off-host and no rule covers database/, ok with the expiry
when one does, quiet when the bucket will not say, inactive without a
bucket or without off-host dumps. The hourly evaluation pushes it to the
admins like every other signal.
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.
A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.
The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
When the grace window ends for an account that has become the last
active guardian of a managed profile, the purge job refuses to erase it
and the account stays locked. Until now only the worker error log said
so. Every admin now gets one localized notification naming the account
and the ways out (another guardian, deleting the profile, or undoing
the deletion). An audit row on the account records that it was reported,
so the hourly retries stay quiet; a deletion undone and asked for again
is reported afresh.
The undo confirmation in the admin users list now says that sign-in
works again but API tokens, share links and trusted devices stay revoked
and have to be set up again (for a managed profile: its share links and
tokens).
The backups page stated "Retention: 90 days", from a constant whose
comment said it matched a backup-prune job. There is no such job: nothing
deletes a stored copy by age. The weekly copy is replaced in place, one
per account; the copy taken before "Delete all data" is deleted by the
account-deletion purge when the grace window ends; an uploaded copy stays
until its account is erased.
The list now carries preWipeCopyDays (the grace window, 0 when no such
copy is taken) instead of retentionDays, and the page says exactly the
three facts above, in every language.
Also drops a re-export of the JSON import row limit nothing imports.
A switched-off scheduled dump is attention now, sent once and never
reminded, with how to turn it back on (DB_BACKUP_ENABLED). Dumps are on
by default, so off is a choice that leaves no copy able to rebuild the
host. An install that never wrote a dump and has nowhere to write one
stays quiet.
The out-of-date kit notice says what changed: the kit now also lists a
key the last confirmation did not cover, for example API_TOKEN_HMAC_KEY
on an instance upgraded from the one-key step, and that nothing broke.
The French kit and readiness strings use vous, the form the rest of the
French bundle uses.
Several surfaces promised more than the per-account copies deliver:
- The full-backup export card said it had the same shape as the weekly
copy. It is a portable export with documents as metadata only; it now
says so, and that it cannot recreate an account on another server.
- The export page offered to "import from a backup". The JSON import adds
measurements and mood entries; it is now named that way, and its help
states the 16 MB and 10,000-entries-per-list limits from the constants
the route enforces.
- The weekly copy is labelled an account rollback on this server, and the
backups page says a copy cannot rebuild a lost server. The upload says
the account must exist here. Two restore refusals pointed to a
"disaster-recovery" copy; they now name the copies that exist.
- The instance-settings restore option was described as being for
rebuilding a host.
docs/ops/backup-restore.md is rewritten short and accurate: what each
copy is, the deletion grace window, that database dumps keep a deleted
account until they age out, how to restore one account, and a link to
the disaster-recovery procedure for a lost server.
The deletion dialogs, the managed-profile delete dialog and the Data &
Privacy page read the window from the privacy summary and say what
happens: the account is locked at once, erased after the stated number
of days, and an admin of the server can undo it until then; a wipe keeps
a copy an admin can restore for the same time. Copies in the server's
backups age out after their retention period, and the text says that
too. With the window set to 0 they keep the immediate wording.
The sign-in page names the erasure date when a sign-in, a passkey or the
single sign-on callback is refused for an account pending deletion. The
admin users list shows a pending account with its date and an "Undo
deletion" action, and the backups list labels the copy taken before a
wipe. Every new string is translated in all nine languages.
A database dump is the whole database at one moment, so an account deleted
since stays in the dumps taken before until they age out. The database row
on the backups card now says so with the numbers that decide it: how many
nightly dumps this server keeps, and after how many days the bucket's
lifecycle rule removes the off-host copies.
The lifecycle probe used to answer with the shortest expiry of any enabled
rule, whatever its prefix, so a rule scoped to the account copies read as
covering the dumps too. It now also answers for `database/` alone, counting
only rules whose filter takes in a dump's key (prefix, legacy prefix, an
And filter; a tag filter never matches untagged objects), and the card
warns when no rule covers that prefix: dumps there, and the deleted accounts
in them, would never be removed. The reading moved into its own module.
Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.
One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:
- database dump: failing once the newest good dump is older than 36 h
while dumps are on, or when it can be kept nowhere; read through a
provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
7 days after setup or as soon as a confirmed kit is out of date.
A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.
The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).
The step confirmed one key, the active application key. A restore needs
more: off-host copies are sealed under BACKUP_ENCRYPTION_KEY, a copy
written before a rotation needs the retired application key it was
written under, and every session, token and share link hangs off
API_TOKEN_HMAC_KEY. An operator could do everything the step asked and
still be unable to read a single backup.
The recovery kit lists every key a restore of this server needs, by id
and fingerprint, never key material:
- each id in ENCRYPTION_KEYS, plus any retired id a stored copy
(innerKeyIds) or an off-host object inside the bucket's retention
(OffhostBackupKeyUse) still lists, also once the server dropped it;
- BACKUP_ENCRYPTION_KEY and each BACKUP_ENCRYPTION_PREVIOUS_KEYS entry,
named by the id their objects record;
- API_TOKEN_HMAC_KEY.
Confirming records the set the admin was shown and refuses a stale one
with 409 recovery.kit.stale. The kit is due again as soon as a key it
lists is not in that set (a rotation, a new backup key, a re-keyed
install); a key that merely drops out does not make it due. "Check my
copy" now checks any key in the kit.
Migration 0487 carries a confirmation given under the old step forward
as a kit holding that key, so that part stays confirmed, and drops the
four app_settings columns. It also creates the two tables the readiness
evaluation uses (restore_drill_outcomes, recovery_readiness_notices).
The routes move to /api/admin/recovery-kit; the reader of the keys the
backups still need moves out of the encryption status route into
src/lib/recovery/backup-key-needs.ts so the kit and the status view
share one definition.
GET /api/admin/backups now carries `database`: the newest good dump (when,
how big, kept on the server, off-host or both), the newest failed run,
whether this server can write its backup directory, and whether off-host
copies are configured. The backups card leads with one row built from it,
with a notice when the last run failed and when dumps can only go
off-host or cannot be taken at all. The route stays an unpublished admin
console route, so the OpenAPI document does not change.
Marc-André Bombeck