docs/ops/disaster-recovery.md says what is dumped and where, that local
dumps are unencrypted and why, which keys a restore needs and what each
one costs when missing, how to restore onto a new server or back to the
state before a migration, and how the procedure is tested. The deploy
runbook and the getting-started guide now describe the dump the
entrypoint takes before migrating, and the off-host account copy page
points at the dump for rebuilding a server.
Fix two dead anchors (the MCP trust model, the README quick start) and the
statements that no longer match the app: a guardian can connect a managed
profile's services, the environment module is on by default and reaches the
air-quality host too, its readable location columns were dropped in 0409,
settings sections carry their current names, the Withings card lives on
/settings/integrations, the TrueNAS start period matches the image, the
deploy check names HEALTHLOG_PROD_URL, and the docs index describes what
self-hosting/ and integrations/ hold.
A concurrent build that is still running is INVALID with its migration not
yet finished, exactly like a failed one. With the web and worker roles in
two containers, the second boot dropped the index the first was building,
which cancelled the first one's migration and left it failed until someone
resolved it by hand.
The check and the drop now run in one transaction that first takes Prisma's
migration lock without waiting and does nothing when another instance holds
it; an index Postgres reports as being built is never dropped. One
transaction also keeps lock_timeout on the statement it bounds behind a
transaction pooler, and the applied-migrations lookup quotes the schema, so
a mixed-case schema no longer reads as having none applied.
A CREATE INDEX CONCURRENTLY that fails half-way leaves an INVALID index,
and the migration re-run after migrate resolve --rolled-back finds it
through IF NOT EXISTS and does nothing: recorded applied, with an index
the planner never uses. 0477 has this shape.
The boot now drops an INVALID index that a not-yet-applied migration
builds concurrently before it runs migrate deploy, with a bounded lock
wait, so the re-run builds it again. An INVALID index whose migration is
recorded applied is named on stderr with the rebuild that mends it. The
deploy runbook says both.
Since handlers load on their queue's first job, a module the image
cannot load no longer fails the worker's boot. The worker comes up
healthy, the queue burns its retries when it first has work, and a
one-shot job (a restore, an import tail) ends failed with nobody told at
deploy time.
After binding every queue the worker now runs `preloadHandlers` in the
background, without awaiting it, so readiness is unchanged. It loads
each handler once through the same memoised loader its queue is bound
to, so a module that loads stays loaded for the first job. Each failure
is reported on the `job.run.failed` line at error, with the queue's name
and `reason: "handler_load_failed"`, plus the worker error report
(stderr and GlitchTip). A summary line on stderr names every queue that
failed. The sweep never throws, and a failing report does not stop it.
docs/ops/deploy.md says what to look for after a deploy.
The TrueNAS page now walks through the catalog install field by field, how
to generate the two secrets, where TrueNAS keeps the key and why that is not
a backup, a dump and restore from the postgres container shell, and updates
with forward-only migrations. The listing is marked as pending review, and
the install-via-YAML values stay for the time until it is live.
The deploy notes list what needs a matching truenas/apps pull request in the
same release cycle: a new or stricter required variable, a port or health
check change, a Postgres major or extension, and any other compose-shape
change.
A web-only process opens its queue connection without migrating and refuses a job-queue schema older than its own, so in a web/worker split the new worker has to start, and upgrade the schema from 45 to 46, before the new web containers. The v1.45.0 notes say so, name the one additive migration, state that a single-container install needs no action, and point a forked compose file at the new stop grace period.
0440 adds its columns with ADD COLUMN IF NOT EXISTS like 0424 and
0432; a structural test holds every migration from 0422 on to that
form. 0430 clears insights_cached_at where no sealed payload is left,
so the nightly pregenerate refills an account whose only copy was the
readable column instead of keeping an empty card fresh.
The deploy runbook notes that 0430 drops a users column: an old process
fails on its next full user read, so stop before starting rather than
running old and new side by side.
A marked migration's DO block holds the locks it takes until it commits,
so an app transaction taking two of its tables in the other order can
deadlock with it. The block then rolled back whole, exactly as on a lock
timeout, so the boot retries 40P01 for marked migrations as it does 55P03.
Corrects 0415's note that the drops hold their lock for an instant.
The entrypoint migrates before the server listens, and a migration that
waits out a held lock retries for up to about eight minutes, while the
health check marked the container unhealthy after two. The start period
is now ten minutes in the compose file and the image; the first answering
probe still ends it, so a normal boot reports healthy as before.
Prisma ignores PGTZ, so a database whose default zone is not UTC ran every
migration in that zone, and a migration mixing timestamp and timestamptz
wrote shifted values. The boot wrapper now appends -c TimeZone=UTC to the
options parameter of DATABASE_URL for its Prisma calls, honouring
DATABASE_SESSION_OPTIONS_DISABLED like the app does.
migrate deploy runs a migration file statement by statement, so the
top-level SET LOCAL lock_timeout in 0412 did nothing: the migration
waited as long as the lock was held, with every write to measurements
queued behind it, and a file that failed half-way stayed half-applied.
0412 now takes its locks inside one DO block with a statement-scoped
timeout, which rolls back whole, after an idempotent CREATE OR REPLACE.
The entrypoint runs migrate deploy through scripts/migrate-deploy.mjs.
A 55P03 failure of a migration marked -- @retry-on-lock-timeout is
marked rolled back and retried after 15/30/60/120/240 s, read from
Prisma's report or, after P3009, from the stored row. Every other
failure stays fatal. A guard holds migrations to the shape that makes
the marker true, and 0411/0413 add their columns with IF NOT EXISTS so
a manual re-run is safe too.
parseCoachPrefs parsed the stored blob as a whole and returned the
defaults on any invalid field. A lookback value this version does not
know therefore dropped the metric exclusions with it. Each field now
falls back on its own, and the exclusion and cluster lists keep their
known entries, so drift narrows what the Coach reads instead of widening
it.
The deploy runbook's rollback section notes the one case this cannot
cover: v1.40.1 and earlier still parse the blob whole, so an account that
saved the new "Last year" lookback loses its exclusions there.
Every configured key id now has a canary in encryption_key_canaries,
checked at boot. A key that cannot open its canary, or existing data
written before the canary existed, puts the process in a refusal state:
every API route answers 503 encryption.key_mismatch, /api/health reports
reason encryption_key_mismatch, the pages render one explanation, and
neither the queue producer nor the worker starts. The process stays up
so the reason is visible instead of hiding in a crash loop.
The admin Encryption page gains a "Back up your encryption key" step:
key id and fingerprint (never the key), where the key lives per
platform, a check of a pasted copy that is compared and discarded, and
a confirmation bound to the active key id and fingerprint. The overview
gets a tile and admins get a dashboard banner while it is due.
HEALTHLOG_PLATFORM picks the platform the step opens on. Adds the
TrueNAS self-hosting page and key backup sections in the docs.
The update runbook ran migrations with a plain docker run and an
.env.production file. No setup guide creates that file, and a plain
docker run starts outside the Compose network, where the database's
service name does not resolve, so the step waited thirty seconds and
gave up with a connection error.
It uses docker compose run now, which starts the one-off container in
the stack with the app's own environment. Refs #961
The badge read the enabled flag plus a DSN that parses. Neither says anything
about the target accepting a report: a wrong public key, a wrong project id and
a host that stopped answering all parse, and every one of them showed green
with a line naming the host. That is the false reassurance an operator finds
out about when they go looking for a crash that was never reported.
Nothing in the tree recorded an outcome, so there was nothing to read. The
sender now stamps one next to the DSN it used — instant on a success, a short
classification on a failure. Never the response body: that is text from a
remote host and has no business in this database or on this page.
The badge is green only while a report has left inside the last fortnight,
amber while a target is set and nothing has come back, and it names the reason
underneath. The clock lives with the ledger, so the verdict is resolved in the
route and the card paints what it is handed. A failure that a later success
superseded is not shown at all. Pressing Test records like any other send, so
the button proves the target on demand.
There is no default error-reporting DSN in this tree and there should not be:
a self-hoster's crash reports must not leave their host because nobody told
them not to. That part was already right, so nothing here adds a default.
What was wrong is what the console said about it. The GlitchTip card showed a
green "Configured" badge as soon as a DSN existed, whether or not the switch
that actually sends anything was on, and it never said where reports go. An
operator reading that badge has been told their errors are being collected
somewhere, which may be true of neither half.
The badge now tracks reports actually leaving, and one line under the switch
states the case: the host they go to, or that they stay here, or that tracking
is on with no target. The host only — the public key in the DSN answers a
different question. The deploy runbook says the same thing for a
maintainer-operated instance.
Post-publish verify proves the published image boots. Nothing proved that
on a running instance a person can sign in, write a reading and read it
back — which is exactly the shape of the defects this line keeps
rediscovering: a value written and never read.
`scripts/synthetic-journey.mjs` walks that journey in plain Node 22 with no
dependencies, so it runs from a workstation, a runner or the deploy host.
It warms `/`, `/api/health`, `/api/version` and the measurements list first
and discards those timings (the first hit on each route after a deploy is a
cold start), then asserts five legs in order: the served version equals the
expected one, the password flow signs in through a cookie jar, a weight
reading is written with an Idempotency-Key, the same row reads back with
its value, unit and source, and the reversible delete leaves the account
clean. One line per leg with the elapsed time; the first failure ends the
run non-zero with the status and a body excerpt that has every token shape
redacted out of it.
The workflow is `workflow_dispatch` only, with `base_url` and
`expected_version` as inputs and the credentials read from repository
secrets. It is deliberately not wired into the deploy path: the target
instance and the secrets are the maintainer's call.
Mutation proof, against a production build served from
`.next/standalone/server.js` on a throwaway Postgres.
Green:
warm 35 ms (discarded)
ok version 2 ms serves 1.38.13
ok sign in 78 ms session cookie held
ok write 64 ms wrote cmtup9lwu0007zhovn0mxajdz
ok read back 8 ms 77.7 kg (MANUAL)
ok delete 27 ms account left clean
synthetic journey green
Read leg broken deliberately — the same server behind a throwaway forwarder
that empties the `GET /api/measurements` list and changes nothing else:
ok version 4 ms serves 1.38.13
ok sign in 38 ms session cookie held
ok write 40 ms wrote cmtupa4ck000gzhovwnij64fz
FAILED read back 8 ms read back: the list came back without the reading
status 200
body {"data":{"measurements":[],"meta":{"total":0}},"error":null}
synthetic journey RED on "read back"
Version leg broken the same way, by asking the instance for the previous
tag:
FAILED version 3 ms version: instance serves "1.38.13", expected "v1.38.12"
synthetic journey RED on "version"
`--self-test` carries that proof in the repository: it runs the journey
against an in-process instance nine times, once with nothing broken and
once per broken leg, and fails if a broken leg still comes back green. The
workflow runs it before it touches the real instance.
Gates: typecheck, `node --check`, lint (the three known warnings, no
errors), prettier, the workflow and CI-determinism guards.
- geolite2: quote the real commented compose mount (./geolite2 to
/opt/geolite2, the GEOLITE2_DIR default, so no env var is needed for
the default path)
- getting-started: healthcheck probes /api/health, not /api/version;
the environment module reaches archive-api and geocoding-api
.open-meteo.com, relevant for egress allowlists
- mcp: the module is opt-in and off by default, but no longer the only
such module
- reverse-proxy: the Coach stream is SSE over plain HTTP, not
WebSockets; note that TRUST_PROXY_HOPS is not on the bundled compose
whitelist and must be added by hand
- scaling: /api/health healthcheck; worker liveness is in-memory and
invisible across a web/worker split; pool sizing now goes through
getConnectionBudget with DB_CONNECTION_LIMIT as the compose knob and
a 2-connection pg-boss share
- sso: the native SSO handoff shipped, OIDC_ONLY works with app builds
that carry it
- unraid: geo databases are baked in only when the build had a licence
key, or mounted; replaceable either way
- attempt-number-semantics: name the real bucket column
(consecutiveFailuresByKind) and drop a drifted line citation
- backup-restore: show the smoke-test response inside the standard
{ data, error } envelope
- deploy: the manual migration step now uses the bundled Prisma CLI
via node (the image strips pnpm); WHOOP vars are already on the
shipped compose whitelist
- encryption-key-rotation: point at the encrypted-column registry
instead of a five-model list several releases behind
- env-check: the CI gate shipped (env-check.yml); the partial-APNs
example now shows the synthetic all-or-none row and the non-zero exit
- password-reset: the admin API reset route exists; only the
self-service route does not
- tls-cert-pin: stop asserting client-repo pin-set details that cannot
be audited from this repository
- drop v141-followup-issues.md: a v1.4.1 punch list whose items are
fixed, removed, or superseded; keeping it under docs/ops invites
re-fixing shipped work
Four small debts from the audit tail. The FHIR bundle's missing
Immunization resource gets the same written-decision paragraph the visits
exclusion carries — deliberate follow-up, not oversight. The service
worker's offline allowlist gains /api/daily/digest (both copies, policy
module and sw.js, plus the test) so the Today rail renders on an offline
reopen. The two Telegram plumbing tables that link to an account by a
bare userId string are frozen as a closed NO_FK_BY_DESIGN set in the wipe
guard — a third such table now fails the test instead of passing rule 1
silently. And deploy.md documents the manual digest-pin bump end to end:
imagetools inspect, the env update, and the three-probe /api/version
verification with the too-fast-deploy warning.
The digest-pinning section explained how to set HEALTHLOG_IMAGE_REF and never
said what happens if it is left behind. It stays until someone changes it, and
every deploy in between reports success while the running instance holds the
previous release.
Adds the warning, the two post-deploy checks that would have caught it (the
version endpoint AND the running container's image, because the first alone can
be served from a cache), the note that a suspiciously fast deploy is a signal,
and the duplicate-key trap: the platform accepts two rows with the same key
without warning and does not define which wins.
The self-context adopt runs read-modify-write inside one transaction,
the admin shell renders nothing until the role is confirmed, and the
json-body helper merges Headers instances correctly. The compose image
reference accepts a pinned digest through an env var, and a promote
script resolves the published digest, verifies its signature and hands
the operator the exact pinning call — the documented path away from a
moving latest tag.
Published manifests are now cosign-signed keyless with a documented
digest-pinning path, a lefthook pre-commit runs gitleaks over staged
changes, responses opt into a credentialless embedder policy, the
deploy webhook can require a fresh timestamp, and a monthly drill
decrypts and parses the newest offhost backup so a broken key or format
surfaces before an emergency does.
Add WHOOP as a synced provider (per-user BYO-keys OAuth2 + HMAC-signed
webhooks), the longitudinal coach (narrative + per-metric trajectory),
and the read-only FHIR R4 REST API plus the scoped, time-limited
clinician share-link record across README, the self-hosting and ops
runbooks, and a new WHOOP integration page. Note the two optional
WHOOP instance env vars, that migrations 0110-0114 are additive, and
the two known limitations (dual-source standard-vital dedup deferred;
coach durable-memory deferred).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/assert-deploy.ts polls /api/version on prod + demo and asserts the
served data.version matches an expected value, printing each target's
buildSha and exiting non-zero on a mismatch or after a bounded retry loop.
This catches the BuildKit layer-cache pitfall where the deploy reports
clean but /api/version still answers the prior release. Run via
`pnpm dlx tsx scripts/assert-deploy.ts <version>` — not bundled into the
standalone image. Document it in the deploy runbook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit a86a9151ef670846c4b33a1c068da40324393308)
- /docs/audit/ moved out of the public docs surface (23 per-release
forensic notes). They were linked from the README but read as
internal traceability artefacts, not user-facing documentation.
Kept local for cross-release lookup; added to .gitignore so new
ones don't re-track.
- /docs/api/openapi-v1422-legacy.yaml removed — superseded by the
current openapi.yaml on the v1.5 line.
- /docs/api/openapi.yaml info.version bumped 1.4.23 → 1.5.0 so the
spec matches package.json (the doc claims the version is mirrored
from the package; previously the mirror had drifted six versions).
- /docs/ops/deploy.md: image name corrected from
`ghcr.io/marc-bombeck/healthlog` → `ghcr.io/mbombeck/healthlog`
(the previous command would have 404'd at GHCR); release-process
framing dropped from a release-pair note.
- /docs/ops/backup-restore.md + /docs/self-hosting/scaling.md:
trailing phase tags (`v1.4 G1`, `v1.4 G3`, `v1.4.40 W-POOL`)
stripped from section headers + body. Replaced an internal-notes
pointer with a concrete `getPoolMax()` reference.
The README pointer to /docs/audit/ is removed in the next commit
along with the rest of the README accuracy pass.
Bundles the deferred v1.4.47 Mediums + Lows with the v1.4.48-deferred items
into a single release. Headline: server-side suppression of MEDICATION_REMINDER
APNs for iOS clients that own their local reminders (closes GH #206), a
push_attempts table backing /api/admin/notifications/diagnostic, and a sweep
of v1.4.48 QA forward-findings.
Performance (the user-visible win):
- /api/analytics cold-fallback path splits the heavy single-statement
aggregate into two parallel queries. The linearly composable columns
(count / min / max / mean) keep the full-partition scan; the windowed
7 / 30 / 90-day avg / slope / r² columns take a 90-day outer cap so
the planner does an index range scan on (user_id, type, measured_at)
instead of a full-partition sequential scan. Output is bit-identical;
cold slim slice drops from multi-second to sub-second on tenants with
large measurement partitions.
Server-side notifications:
- GET + PATCH /api/auth/me/notification-prefs with deep-merge semantics.
Initial category medication.clientManaged: boolean (default false). Cron
medicationReminderJob skips dose-due APNs when the flag is true,
emitting a medication_reminder.suppressed_client_managed wide-event
annotation per skip.
- push_attempts table backs the diagnostic endpoint. Every APNS /
Web-Push / Telegram / NTFY sender writes a fire-and-forget row.
Daily 03:35 Europe/Berlin cleanup prunes rows older than 90 days.
- /api/admin/notifications/diagnostic — admin-only endpoint surfacing
device APNs token presence (masked), channel config presence, and
recent push attempts. Registered in OpenAPI.
- Per-channel APNs "Send test" button now fires with
eventType: "MEDICATION_REMINDER" so the dispatcher's time-sensitive +
priority-10 branch runs. A successful self-test now exercises the
exact path production reminders take; an unsuccessful one points the
user at iOS Settings → Time Sensitive Notifications rather than
appearing to be a server-side bug.
Withings reliability:
- OAuth callback consumes the nonce atomically (delete-first with
P2025 catch for the replay branch). The previous read-then-delete
pattern allowed a race between two concurrent callbacks.
- ?reason=state redirect differentiated into csrf1 / replay / expired /
cross_user so operators can tell mismatch types apart without DB
shell access.
- connect endpoint rate-limited to 10/60s/user; row-create failures
redirect to a consistent error surface.
Observability hardening:
- redactSensitiveFields(body) denylist helper covers password / token /
secret / apiKey / authorization / csrfState / nonce, recursive,
case-insensitive. Caught: /api/devices invalid_format issues were
echoing apnsToken values verbatim into the audit log.
- sanitiseZodIssues(issues, { stripValuesFromMessage: true }) overload
applied to 14 audit-log sites.
Migrations: 0079 (notification_prefs), 0080 (push_attempts).
Tests: 4815 → 5272 unit + integration. Workout-batch integration suite
that had been red on main since v1.4.45 (three release commits shipped
through it) now green again — updated to match v1.4.42 cross-source
dedup semantics.
Resolves PR #207.
Marc-André Bombeck