A switched-off scheduled dump is attention now, sent once and never
reminded, with how to turn it back on (DB_BACKUP_ENABLED). Dumps are on
by default, so off is a choice that leaves no copy able to rebuild the
host. An install that never wrote a dump and has nowhere to write one
stays quiet.
The out-of-date kit notice says what changed: the kit now also lists a
key the last confirmation did not cover, for example API_TOKEN_HMAC_KEY
on an instance upgraded from the one-key step, and that nothing broke.
The French kit and readiness strings use vous, the form the rest of the
French bundle uses.
The drill now runs an S3-compatible bucket beside the stack (Versity S3
Gateway v1.8.0, pinned by digest; the MinIO images are no longer published
on Docker Hub or quay.io) on a network and a volume of its own, so it
outlives the stack as a real bucket does. The app takes the dump through the
scheduled job and uploads it there; the drill waits for a run that both
kept and uploaded it, destroys every volume of the stack, the local backups
included, and restores on a new stack with only the bucket and the keys:
`docker compose run --rm db-restore latest --from-bucket`. The independent
row count reads the manifest from the bucket with the restore's own reader.
DRILL_SOURCE=local keeps the earlier path (dump files copied off the host,
then `db-restore latest`), and CI runs both. A new mutation,
wrong-backup-key, restores with a different BACKUP_ENCRYPTION_KEY and must
be refused. Ports are free ones picked at start.
The new-server steps lead with `db-restore latest --from-bucket`, the copy
that survives when the host and its local dumps are gone, and keep the
restore from downloaded files beside it. Both backup pages now say that the
database dumps are not purged per account: a deleted account stays in them
until they age out, which is the retention period an operator states to the
people using the instance.
The check failed once with "Can't reach database server at
127.0.0.1:55416": the container reported healthy before its published port
forwarded, and the migrate on the host ran in that gap. It now waits until
the port answers, retries a migrate that could not connect, both within
thirty seconds, and fails with Prisma's own output otherwise. The port is a
free one picked at start unless CHECK_DB_PORT names it, so the check cannot
collide with another stack on the same machine.
A database dump is the whole database at one moment, so an account deleted
since stays in the dumps taken before until they age out. The database row
on the backups card now says so with the numbers that decide it: how many
nightly dumps this server keeps, and after how many days the bucket's
lifecycle rule removes the off-host copies.
The lifecycle probe used to answer with the shortest expiry of any enabled
rule, whatever its prefix, so a rule scoped to the account copies read as
covering the dumps too. It now also answers for `database/` alone, counting
only rules whose filter takes in a dump's key (prefix, legacy prefix, an
And filter; a tag filter never matches untagged objects), and the card
warns when no rule covers that prefix: dumps there, and the deleted accounts
in them, would never be removed. The reading moved into its own module.
A lost host takes its local dumps with it, and the copy left is the one in
the bucket. `docker compose run --rm db-restore latest --from-bucket` now
lists `database/` in the bucket the BACKUP_S3_* variables name, takes the
newest dump whose manifest is there too (or the one named), downloads both
with the S3 client the image already ships, and continues as for any
off-host copy: opened with BACKUP_ENCRYPTION_KEY, checked against the
manifest, restored into an empty database. `list --from-bucket` shows what
is there. The sealed download is removed once it is opened.
The image build now proves the S3 client resolves from the shared module's
place in the tree, so the restore cannot lose it unnoticed.
The key backup sections become "Save the recovery kit": which keys the kit lists and why, the fingerprint of the token key, and the readiness card with its Backup health notification.
Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.
One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:
- database dump: failing once the newest good dump is older than 36 h
while dumps are on, or when it can be kept nowhere; read through a
provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
7 days after setup or as soon as a confirmed kit is out of date.
A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.
The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).
The step confirmed one key, the active application key. A restore needs
more: off-host copies are sealed under BACKUP_ENCRYPTION_KEY, a copy
written before a rotation needs the retired application key it was
written under, and every session, token and share link hangs off
API_TOKEN_HMAC_KEY. An operator could do everything the step asked and
still be unable to read a single backup.
The recovery kit lists every key a restore of this server needs, by id
and fingerprint, never key material:
- each id in ENCRYPTION_KEYS, plus any retired id a stored copy
(innerKeyIds) or an off-host object inside the bucket's retention
(OffhostBackupKeyUse) still lists, also once the server dropped it;
- BACKUP_ENCRYPTION_KEY and each BACKUP_ENCRYPTION_PREVIOUS_KEYS entry,
named by the id their objects record;
- API_TOKEN_HMAC_KEY.
Confirming records the set the admin was shown and refuses a stale one
with 409 recovery.kit.stale. The kit is due again as soon as a key it
lists is not in that set (a rotation, a new backup key, a re-keyed
install); a key that merely drops out does not make it due. "Check my
copy" now checks any key in the kit.
Migration 0487 carries a confirmation given under the old step forward
as a kit holding that key, so that part stays confirmed, and drops the
four app_settings columns. It also creates the two tables the readiness
evaluation uses (restore_drill_outcomes, recovery_readiness_notices).
The routes move to /api/admin/recovery-kit; the reader of the keys the
backups still need moves out of the encryption status route into
src/lib/recovery/backup-key-needs.ts so the kit and the status view
share one definition.
scripts/compose-dr-drill.sh plays docs/ops/disaster-recovery.md against
real containers: it builds the image from the checkout, starts the bundled
compose stack with fixed throwaway keys, seeds the demo account and a
Guardian with a managed profile (through the app's own code), writes a
reading with an encrypted note through the API, and waits for the
scheduled database-backup job to take the dump. It then keeps only the
dump files, destroys every volume, starts a new stack, restores with
`docker compose run --rm db-restore latest`, counts every table against
the manifest apart from the restore, and boots the app: health, the demo
login, the note read back, a document opened, the Guardian still seeing
the profile.
DRILL_MUTATION breaks one step on purpose (wrong-key, skip-table,
non-empty-target), and the drill fails on each. The workflow runs it on
pull requests that touch the compose file, the image, the entrypoint or
the dump and restore code, nightly, and by hand.
docs/ops/disaster-recovery.md says what is dumped and where, that local
dumps are unencrypted and why, which keys a restore needs and what each
one costs when missing, how to restore onto a new server or back to the
state before a migration, and how the procedure is tested. The deploy
runbook and the getting-started guide now describe the dump the
entrypoint takes before migrating, and the off-host account copy page
points at the dump for rebuilding a server.
GET /api/admin/backups now carries `database`: the newest good dump (when,
how big, kept on the server, off-host or both), the newest failed run,
whether this server can write its backup directory, and whether off-host
copies are configured. The backups card leads with one row built from it,
with a notice when the last run failed and when dumps can only go
off-host or cannot be taken at all. The route stays an unpublished admin
console route, so the OpenAPI document does not change.
The image now carries postgresql18-client (pg_dump and pg_restore 18.6)
and a /var/lib/healthlog/backups directory owned by the app user, which
the bundled compose file backs with a new `backups` volume for `app` and
the commented `app-worker`; a fresh named volume inherits that owner.
Before `migrate deploy`, when migrations are pending on a database that
has applied some, the entrypoint writes healthlog-<time>-pre-migration.dump
with its manifest and keeps the last two. A failed dump on a writable
directory stops the boot and names DB_BACKUP_BEFORE_MIGRATE=false; without
a writable directory it warns and continues.
`docker compose run --rm db-restore <file|latest|list>` runs
scripts/db-restore.mjs in the app image with plain node. It opens an
off-host .dump.enc with BACKUP_ENCRYPTION_KEY (or a previous key), checks
the archive against the manifest's SHA-256, compares the keys in the
environment with the manifest's fingerprints and refuses on a different
active encryption key unless --ignore-key-mismatch, refuses a database
that holds any table, restores in one transaction, runs ANALYZE, and
compares every table's row count with the manifest, emptying the database
again on a difference.
DB_BACKUP_ENABLED, DB_BACKUP_DIR, DB_BACKUP_SCHEDULE, DB_BACKUP_KEEP and
DB_BACKUP_BEFORE_MIGRATE join the compose whitelist, both example files
and the env manifest.
The per-account copies roll one account back on the same server; nothing
the app made could rebuild a lost one. A scheduled job now runs pg_dump
over the whole database (every schema, pg-boss included) into
DB_BACKUP_DIR as healthlog-<UTC time>-daily.dump, with a manifest beside
it: app version and build, newest migration, Postgres version, size and
SHA-256, exact row counts of every table, and the fingerprints of the
keys the database needs.
The counts read the same exported snapshot pg_dump reads, so a dump
taken while the app writes still matches its manifest. The archive is
written as .partial, listed back with pg_restore --list, and renamed
after its manifest; a failed run leaves nothing named like a good dump.
Retention removes only files this job wrote, oldest first, and never
the newest good one (DB_BACKUP_KEEP, default 7; two pre-migration dumps).
With the off-host bucket configured, each dump and its manifest go up
under database/ in the envelope the account copies use, streamed. Every
run lands in the new database_backup_runs ledger (migration 0486), and
getDatabaseBackupHealth() answers the last good dump, the last failure,
whether the directory is writable and whether off-host is configured.
The boot readiness summary says where dumps go, or that none can be kept.
The dump, naming, retention and restore rules live in one plain ESM
module so the image's own scripts can share them without tsx.
The tool module was split into one file per tool on this branch, so the
change lands in tools/get-labs.ts, and the advertised tool list snapshot
gains the two declared fields.
Both files had drifted copies of the docs-site guides, which carry the same
sections and more, and nothing in the app, the scripts, the other docs or
the site links to them. Each is now a short pointer to its guide.
Fix two dead anchors (the MCP trust model, the README quick start) and the
statements that no longer match the app: a guardian can connect a managed
profile's services, the environment module is on by default and reaches the
air-quality host too, its readable location columns were dropped in 0409,
settings sections carry their current names, the Withings card lives on
/settings/integrations, the TrueNAS start period matches the image, the
deploy check names HEALTHLOG_PROD_URL, and the docs index describes what
self-hosting/ and integrations/ hold.
The labs block reads the whole record, or up to the lookback limit, not just
recent readings, and a named read also comes back empty when no marker
matches. The header of the labs snapshot and the note at its call site
still described the earlier twelve-month window.
The integration runbooks, the Garmin note, the guardian note on a managed
record's connections, the backups admin link and the README's self-hosting
link all pointed at retired docs paths that only resolve through redirects
(or not at all, on a host without them). They now go to the final guide
pages, and the settings tests assert the new addresses.
The statement already declares `date` and `code` on Observation beside
paging on every search; the route test now asserts that list per resource,
so a parameter the search honours cannot drop out of it unnoticed.
The published contract said the nine health-shaped fields that predate
`health` were removed in v1.44, but every status read still sends them:
older native builds decode them, the current one falls back to them against
a server without `health`, and the web card's copied diagnostics read the
failure streak. They stay deprecated, without a removal release.
The hint shown when a text-only provider needs on-device text recognition
sent people to the AI settings, where no such switch exists. It now names
the switch by its label under Settings → Modules → Labs in every locale and
links straight to that card. Inside somebody else's record the link is
withheld, since Settings there is the caller's own and closed; the sentence
stays.
The turn records up to 64 steps while the read path kept only the first
48, so the tail of a long answer's step list vanished on reload. Both ends
now read one MAX_TURN_STEPS constant, and a unit plus an integration test
round-trip a full 64-step answer through the stored provenance.
The blood-pressure chart scales one axis to fit systolic and diastolic, so
the 12 mmHg the busy stretch added came out as a nearly flat line on a 60 to
140 axis, even with the 7-day trend on.
The stretch now climbs over two days to about 144/91, holds for some ten
days, and comes back down over the following days after ramipril goes from
5 to 10 mg. A dose change and a family-practice visit on the day it turns
carry that step, and the medication now reads 10 mg. The day-to-day walk is
quieter, with a little reading scatter on top, so the 7-day mean draws an
arc rather than jitter.
Two limits keep the rest of the page honest. The lift stops short of
149 mmHg: one more and the padded axis rounds out to 50 to 200 and flattens
the rise again. And the stretch stays well under half of the last 30 days:
the usual range is the median ± 3 MAD of that window, and a longer stretch
moved the median onto it and widened the band to about 87 to 181 mmHg.
The stored texts that describe the last weeks mention the stretch and the
dose increase. The seed-shape test holds the larger swing, the ceiling, the
smaller day-to-day steps, the median of the window and the dose change on
the turn.
The compose file runs Postgres 18. An install from v1.45 or earlier keeps
its Postgres 16 data in `pgdata`, which 18 cannot open, so a one-shot
`db-upgrade` service moves it on the first start: it starts Postgres 16 on
the old volume inside its own container, streams a dump into 18 in one
transaction, compares every table's row count, gathers planner statistics
and marks the old volume as moved. The app waits for it to succeed.
The old volume is never changed apart from the marker, so the way back is
the previous image and volume lines. A move cut short leaves a "moving"
marker and runs again from scratch; an emptied new volume after a finished
move is refused rather than refilled with the old state.
scripts/compose-db-upgrade-check.sh plays all of it against real
containers from the v1.45.0 compose file, and runs in CI on every change
to the compose file. Pull requests run the integration suite and e2e on
Postgres 18; the nightly integration run keeps 16 covered for installs
that bring their own database.
The keys followed each other without waiting for the step to move the focus,
so on a slow run Tab started from the previous answer and the number chose
nothing.
Each step focuses its question, but the first step's question only exists
once the language's wording chunk has arrived; the focus ran while the step
still showed its loading state and was never moved again, so a keyboard or
screen reader user started on the page body.
Scrolling to the end can start a section's load, and its skeleton sat under
the button for a moment on the Insights page in landscape. The check now
scrolls to the end and measures again until the last line is in place;
content that stays under the button still fails it.
A concurrent build that is still running is INVALID with its migration not
yet finished, exactly like a failed one. With the web and worker roles in
two containers, the second boot dropped the index the first was building,
which cancelled the first one's migration and left it failed until someone
resolved it by hand.
The check and the drop now run in one transaction that first takes Prisma's
migration lock without waiting and does nothing when another instance holds
it; an index Postgres reports as being built is never dropped. One
transaction also keeps lock_timeout on the statement it bounds behind a
transaction pooler, and the applied-migrations lookup quotes the schema, so
a mixed-case schema no longer reads as having none applied.
Two sources off every ladder now read at the very same instant on thirty
days: on fifteen the name decides, on the other fifteen the first name's
systolic is under the floor and the second takes the day. A dedicated
case asserts which source each of those days shows, under a UTC and a
Berlin session; the parity cases compare them with the reference as
well. Dropping the name from the window's order turns 26 of 59 cases
red on Postgres 16 and 18.
The guard seeded each day's pair six hours before the clock, with the
second source ten minutes later. Run between 05:50 and 06:00 UTC, the
newest day's second pair landed just past midnight, alone on a UTC day
of its own; the pick rightly chose that source for that day, the old
DISTINCT ON statement chose the same, and the assertion that every
series row is the first source failed for those ten minutes of the day,
on every Postgres major.
The pairs now sit at 10:00 and 10:10 UTC from yesterday back, whatever
the clock. With the clock pinned to 05:55 UTC the old fixture fails the
three series shapes and the new one passes, on 16 and 18 alike.
A later case in the file folds the rollups again, and a fold sums in the
order its plan reads, so the second fold's sums could differ from the
first's in the last bit; the comparison read the table after that fold.
An account counts as done for a run when it succeeded at or after the run's
start, and the follow-up run's start taken with new Date() could share the
millisecond of the retry's last ledger write.
Three readers grouped the same rows twice and joined the second grouping
back to the first: the live buckets joined each hourly-mean day's hour
means to the per-source days, the all-time extremes joined the day
weights the same way, and the doctor report's dense buckets joined each
bucket's latest reading and hour means to the bucket aggregates. On a
measurements table without planner statistics (a new install, or an
import under the autoanalyze threshold) the planner estimated one row on
each side and ran every one of those joins as a nested loop that re-ran
the inner grouping for each outer row.
On ten years of readings over six types (51,000 rows, no statistics) the
dense buckets and the every-type live buckets ran into the 60 s statement
timeout and the all-time extremes took 11 s. With statistics all three
answered in 0.1 to 0.2 s.
Each is now one chain of folds over one scan. The live buckets fold the
readings per local hour and the hours per local day, carrying the day of
hours beside the counts. The all-time extremes take the day weights from
the same windowed rows they count, the shape the pre-fold remainder
already has. The dense buckets carry each bucket's latest reading and each
hour's count and mean on the rows as window values, and one grouping folds
them all. Same fixture without statistics: 0.2 s, 0.1 s and 0.1 s, the same
as with them.
A parity suite runs the joined statements as the reference against the
readers across ladders, zones with whole, half-hour and negative offsets,
grains, overlapping and off-ladder sources, glucose contexts and device
types, tombstones and both DST changes; sums and means agree to
floating-point summation order, everything else exactly. The
unanalysed-table guard now runs the three readers under EXPLAIN ANALYZE:
the joins measured 200 to 3,000 row visits per row stored, the folds about
10.
The dense-bucket loader drops under the 200-line function cap, so its
baseline entry goes.
The blood-pressure series chose each UTC day's source with DISTINCT ON
over the day's floor-passing systolic rows and joined the choices back
to the readings. That is the shape the analytics pick had, with the
same weakness: on a measurements table without planner statistics the
planner estimated one row on each side and ran the join as a nested
loop that rescanned the pick for every reading.
The ten-year series over 6,000 freshly written rows from one source
took 0.57 s for the statement alone, 0.86 s with a second source on
every third day; the plan visited about 4,900 rows per row stored, so
the cost grows with the square of the history.
The pick is now one window over one scan: the day's rows ordered
systolic first, then by ladder rank, then by source name, and the first
row's source taken only when it is a systolic one. The scan starts at
the window's first UTC day, so the pick still sees the whole edge day,
and a day without a floor-passing systolic still shows nothing. Both
fixtures above read in about 10 ms, with or without statistics.
A parity suite runs the old statement as the reference against the new
one across overlapping and off-ladder sources, half readings, sub-floor
rows, tombstones, both DST changes, window starts on and either side of
UTC midnight, and four session zones. The unanalysed-table guard now
covers the series' ten-year, quarter and week reads.
The canonical-source pick grouped the window to (type, day, source),
chose each day's source with DISTINCT ON, and joined the choices back to
the rows. Whether that join ran as a hash join or as a nested loop that
rescanned the whole pick for every row was the planner's call, made from
table statistics. On a measurements table without them (a new install,
or an import still under the autoanalyze threshold) it estimated one row
on each side and took the nested loop.
Binding the request's clock as a parameter instead of reading NOW()
flipped which side of that loop the pick landed on: the analytics read
over 6,000 freshly written blood-pressure rows went from 1.6 s to 8.4 s,
and the demo seed's dashboard headline to 20 s. On a ten-year account
without statistics the 90-day read took 4 s with either clock and the
all-time read ran past two minutes.
The pick is now one window function over one scan: FIRST_VALUE of the
source per (type, UTC day) in ladder order, keeping the rows of that
source. Same rows, same order of preference, so every day keeps its
source; the parity suite against the DISTINCT ON statement holds that.
The analytics read on the fixture above takes about 0.1 s, the headline
0.4 s. On the analysed ten-year account the 90-day and all-time reads
are as fast as the join was, and they no longer depend on statistics.
The rows carry one extra column, canonical_source; every reader selects
its columns by name.
A new integration test runs every reader shape under EXPLAIN ANALYZE on
a table with no statistics and bounds the rows the plan produces or
discards per row seeded. The statement count never moved, so a statement
budget could not see this; the join measured 200 to 8,600 row visits
per row seeded and the window stays at 4.
A status run is one completion, so its ceiling was the completion ceiling
under a second name. The Local client also re-exported a JSON-mode helper
nobody imported from it.
Marc-André Bombeck