What an account with tens of millions of readings looks like when it
runs into the per-statement limit (empty cards, job.run.failed with
57014), what the full backfill, the hourly heart-rate fold and the
admin alert now do about it, and when raising
DATABASE_STATEMENT_TIMEOUT_MS still makes sense. The variable was only
named in the compose file and the example env until now (#1298).
The Coach chat, the document chat and the fenced chat declared their
locale field as de/en, so a body naming French, Korean, Brazilian
Portuguese or any other shipped language was refused with 422 before
resolveServerLocale, which carries every locale, ever saw it. The same
pair sat on the profile update, so an account could not store any other
language there either, and the stored language is what the cookie-blind
background work (the Coach nudge, notification texts) writes in.
All four now take z.enum(locales), and the published contract for the
metric, insight and biomarker status queries and the profile update
lists every UI locale instead of two.
The header always said "welcome back", including to an account the
setup flow had created a few minutes earlier.
/api/auth/me now publishes the account's createdAt, and the greeting
welcomes an account in its first day ("Welcome, name.") and welcomes
back every other one, including an account whose older server does not
send the field. Until the account is known, during the SSR pass and the
hydration render, the line is the time-of-day greeting alone, so
nothing claims "back" before the client knows it is true.
There was no way for an admin to remove someone else's account; the
users list only offered the undo for a deletion the account itself or a
Guardian had asked for.
DELETE /api/admin/users/{id} takes the account's own deletion path,
now shared as requestAccountDeletion: with a grace window the account
is locked at once (credentials revoked, record untouched) and erased by
the purge job when the window ends, and the existing undo gives it back
until then; with ACCOUNT_DELETION_GRACE_DAYS=0 it is erased in the
request. The route is cookie-only behind requireAdmin and the
recent-proof gate, takes the typed DELETE_ACCOUNT confirmation, and
refuses the caller's own account, a managed profile (its Guardians
delete it), an account already pending deletion, the last admin who
can sign in and the last Guardian of a managed profile, the last three
decided under their locks inside the transaction. Both the admin and
the account get an audit row.
The users list says who asked for a pending deletion, now including an
admin, and flags managed profiles. Each other account's row carries a
delete button whose confirmation names the purge date, or says the
erasure is immediate when the server keeps no window.
From a HealthLog container with its stored ChatGPT sign-in, gpt-6.1-sol,
gpt-6-sol and gpt-6-luna each answered text and read a lab-report image,
and the account's own model list offered them with image input.
gpt-6.1-sol refuses effort none and names low as its lowest. The spec said
the gpt-6 rungs had never been probed live, and still listed gpt-5.4,
gpt-5.4 mini, gpt-5.3-codex and gpt-5.2 as accepted, all of which OpenAI
has retired for ChatGPT sign-in, with the exact refusal the backend sends.
Also note why a fixed chain goes stale: once GPT-5.5 retired, every model
on the chain shipped at the time was retired with it.
An account created through OIDC holds no password, passkey or second
factor, so the recent-proof gate in front of token minting, MCP
connections, exports and the admin reset had nothing to ask it for and
the web could only say "sign in again" (#1296).
GET /api/auth/oidc/reproof sends the browser back to the configured
provider with prompt=login and max_age=0, its own state, nonce and PKCE
verifier, and the signed-in account and session sealed into the state
cookie. The shared callback branches on that intent: it refuses unless
the browser still carries the bound session (checked before the code is
redeemed), the ID token names the account's linked issuer and subject,
and auth_time lies inside the gate's five-minute window. It then stamps
Session.reproofAt exactly as a password re-proof does, creates no
session, and returns to the page it left with ?reproof=done|failed.
recentProofMethods offers "oidc" only to an account without a second
factor whose identity is linked while a provider is configured, and the
fresh-factor gate never lists it. GET /api/auth/me/mfa publishes the
list as proofMethods, so the passkey card offers what the account can
actually give, and register-options takes { method: "oidc" } on a fresh
provider re-proof, so such an account can add a passkey as a second way
to confirm. Both re-proof dialogs offer the provider and a toast reports
the outcome on return.
The callback's missing-state branch and the passkey card's enrollment
flow move into their own functions to keep both handlers under the size
ratchet.
Systolic, diastolic and the cuff's pulse are three rows with no link
column, and the single DELETE tombstoned only the row it was given, so
deleting a reading's systolic left its diastolic and pulse behind as half
a reading. The bulk delete had the same gap.
A reading is the rows with the same owner, the same source and the
identical measuredAt: that is how the entry form, the MCP write, a
Withings measure group and a HealthKit correlation all store one, and how
the FHIR read pairs a diastolic onto its systolic. The looser five-minute
pairing of the chart reads is right for drawing a point and wrong for
taking rows away.
Deleting a systolic or diastolic now tombstones the rest of its reading in
the same transaction with the same deletedAt, each row with its own
syncVersion bump so paired clients see every tombstone. The restore route
brings back the rows deleted together with a named half (identical
deletedAt), so the undo, which names only the row that was tapped, puts
the whole reading back, from the web list and from any client that calls
it the same way. A pulse is still deleted on its own, and a row deleted at
another moment stays deleted. The response shapes do not change; the
OpenAPI descriptions say what the counts now include.
DEPLOY_WEBHOOK_REQUIRE_TIMESTAMP had been changed to mean signed requests
only, so an operator who set it for a sender that adds a timestamp but cannot
sign would have had every deploy notification refused after the upgrade. It
means the secret header plus a fresh timestamp again. Signature-only, the
setting that stops a replay, is DEPLOY_WEBHOOK_REQUIRE_SIGNATURE, on the
compose whitelist, in the env manifest and examples, and in the deploy
guide.
Forwarding headers were believed from any connection on a private address.
Behind Docker Desktop, OrbStack and docker-proxy every connection to a
published port arrives from the gateway, a private address, so a caller
reaching the port directly could again choose the client address that rate
limits, the audit log and sign-in notices record. TRUSTED_PROXY_CIDRS
(comma-separated CIDRs) now names the proxies: when set, only those peers may
report a client address. The private-range default stays when it is unset,
the boot log states the rule in force in one line, and the reverse-proxy
guide says what to set for a proxy on the same host, on a Docker network, or
on a public address (which, unlisted, puts all clients in one bucket).
An invalid TRUST_PROXY_HOPS threw on every request that resolved the client
address, so every route answered 500, the health probe included. It now
resolves no address, warns once, and is named at boot.
A cookie named healthlog_session can be set by any host under the same
registrable domain, so a page on a sibling subdomain could plant a session of
its choosing in a visitor's browser. Wherever the session cookie is Secure it
is now issued as __Host-healthlog_session, which a browser accepts only from
this exact host. Plain-HTTP installs, which cannot hold a __Host- cookie,
keep the old name.
Nobody is signed out: the old name is still read after the new one, a
session found under it is re-issued under the new name and the old cookie
expired, and a browser holding the new cookie is never answered from the old
one. Every reader and writer goes through one module, which the proxy, the
native-caller check, the native handoff and the OIDC callback use; the
session-surface guard follows the new reader.
A read a person started now runs as interactive work of their account, and
the document jobs as background work of theirs, so the reader's queue can
order and cap them. When the reader is busy or stopped before it answered,
a lab scan, label read or document read fails with aiRuns.readerBusy (503,
nothing read or charged) instead of a misleading unreadable-file error, and
the index and summary jobs throw so the queue runs them again rather than
recording an outcome about the document.
Off-host dumps were named by their time alone. Two instances sharing a bucket
(nothing forbids it, and it was safe for the account objects) overwrote each
other's dump at the same second of the same default schedule, and
latest --from-bucket restored whichever instance's dump was newest. Each
instance now uploads under database/<host of APP_URL>/, the object key (and so
the encryption's associated data) carries it, and the restore takes only this
instance's dumps: it refuses another's and names the instances it found,
unless --instance=<host> or the dump's full key says which one on purpose.
A restore killed outright (out of memory, docker kill) left its .restore- or
.download- directory, a whole database decrypted, in the backups directory,
where nothing removed it. A restore now removes them before it starts and on
SIGINT and SIGTERM, and the nightly retention sweeps any older than twelve
hours.
The notes said the other migrations only add columns and one concurrent
index. 0487 drops four app_settings columns the previous image reads on
nearly every request, and 0488 builds its index on users without
CONCURRENTLY. The rollback step now names the columns to hand back, and the
notes say what v1.45 does with accounts pending deletion and held wipe
purges, and how to settle them before going back.
Pre-migration dumps were kept by count alone, so an install that updates
rarely kept its last two, and every account deleted since, indefinitely,
while the docs and the backups card stated the nightly retention as the
retention of deleted accounts. A pre-migration dump older than the oldest
nightly dump kept now goes, except the newest one, the way back from the
latest update, which stays until the next update. The card, the disaster
recovery guide and its variable table say so.
The recovery kit, "Keys the backups still need" and safeToDropRetiredKeys
read only the stored copies and the off-host account objects. After a
rotation the page called the old key safe to drop while every dump taken
before it, the pre-migration dumps that are the way back from an update
among them, could only be restored with that key: db-restore refuses a dump
whose active key is missing.
A dump now counts for the key its rows were written under, the manifest's
active key: a dump on this server while it is kept (read from the manifests),
an off-host one until the bucket's rule for database/ expires it, or
indefinitely when that rule cannot be read. The nightly run records the key
on its ledger row (migration 0489, database_backup_runs.active_key_id). The
encryption view and the kit list the dumps per key, in every locale, and the
rotation and backup runbooks say so.
The reproductive-health page promised that deleting all data or the
account erases at once. Both wait for the grace window now, the nightly
dumps keep a record until they age out, and the old pgdata volume is a
full copy until it is removed; the page says so and how to make deletion
final. The README, the getting-started note on majors, the move guide and
the Unraid and Portainer templates catch up with Postgres 18, the backups
volume and the recovery kit. The demo seed's spread bound leaves room for
the seed's own one-in-150 spread.
Going back to Postgres 16 after a move that stopped half-way left the
moving marker, and the next start emptied the live 16 database and
started a second server on its data directory. db-upgrade now waits for
the database service and does nothing unless it runs Postgres 18.
Using 16 again after a finished move and then updating started the app
on the move-day copy without a word. The marker now records the old
database's last checkpoint, and a later start refuses when it moved on;
the guide says how to move the current database instead. The disk check
leaves room for the first pre-migration dump, and the move check seeds
the old database with v1.45's own migrations.
The fourth rollback step had been joined onto the paragraph after the
list; it is a list item again, with the way back to the data after it.
The disaster recovery page now says how the server tells admins about
its copies: the recovery readiness card, the recovery kit and the hourly
Backup health notification. The support page put GlitchTip under Admin,
General; the card is under Admin, Integrations, as the deploy runbook
says. The pre-migration dump notes say that without the backups volume
no dump is taken, also where the image's own directory is writable.
The restore refuses a file that carries any document as metadata only
(findIncompleteDocument), whatever the account holds; the page said it
refused when the account held a document.
The flag makes the app believe CF-Connecting-IP from every connection,
before the peer check and whatever TRUST_PROXY_HOPS says, which is safe
only when nothing but Cloudflare can reach the origin. The reverse-proxy
page now says so with the ways to get there, the compose comment and both
example files carry the same condition, and the variable is in the env
manifest. The page's list of peers whose forwarding headers are read now
matches the code: loopback, 0.0.0.0/8, private, CGNAT, link-local,
unique-local and site-local, the reserved blocks and multicast. Those last
two stay trusted, since no internet host can connect from them; a test
pins every listed range and the Cloudflare precedence.
A text read that crossed the isolated reader's memory or time bound came
back as a plain read error, which the index job records as decrypt-error,
so the document said its stored file could not be opened. The local
extractor now answers resource-limit for PdfResourceLimitError, the index
records it under that name, and the detail view says the PDF is too large
or complex to read here, in every locale. Like the thumbnail's refusal of
the same bytes it is final: nothing retries it on its own.
Only the backups page said when the bucket's lifecycle rules left
database/ out, so every dump, and every deleted account inside it, stayed
in the bucket forever without anyone being told. Readiness gains a
dump_retention signal that reads the same cached lifecycle probe: attention
when dumps go off-host and no rule covers database/, ok with the expiry
when one does, quiet when the bucket will not say, inactive without a
bucket or without off-host dumps. The hourly evaluation pushes it to the
admins like every other signal.
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.
A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.
The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
The deletion revokes every session, refresh token and API token, so they
answer like any revoked credential; only new sign-ins get the 403
auth.account.deletion_pending, through the shared login response.
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
The Postgres major-upgrade guide suggested removing the old pgdata volume
"when you are sure". That volume is a full copy of the database as it stood
before the move: account deletions, trash purges and key rotations after it
never reach it. The guide now says so and makes removing it part of the
move.
The deploy-status webhook authenticated with a static secret in a header,
and the timestamp the docs called replay protection was covered by nothing:
whoever captures one request holds the secret and can send a fresh request
with any content. The strict setting, documented as closing the replay
window completely, only made that timestamp mandatory.
A sender can now sign instead: X-Deploy-Webhook-Signature is the
HMAC-SHA256 of the timestamp and the exact body under the shared secret,
with the timestamp within five minutes, and the secret never travels. The
strict setting accepts only signed requests. The secret header still works
by default for senders that cannot sign, and the docs describe its
timestamp as the staleness hint it is.
A token scoped to one medication got 404 for a name that matched none of
the owner's medications and 403 for one that matched outside its scope, so
it could confirm which medications exist by name. Its idempotency replay
also ran before the scope check and returned any of the owner's doses
stored under the key.
A token without the wildcard now gets the same 403 for both, the replay is
looked up only after the scope check, and a key already spent on another
medication answers 409 without that dose. The wildcard token keeps its 404.
The session cookie is SameSite=Lax, and SameSite compares the registrable
domain, so a page on a sibling subdomain of the instance could send a
CORS-simple POST with the person's cookie attached. safeJson accepted any
Content-Type containing "application/json", so a text/plain body that
mentions it in a parameter, which needs no preflight, was parsed as the
person's own request: readings added, entries deleted, integrations
disconnected, a delegate invited within the re-proof window.
apiHandler now refuses a POST, PUT, PATCH or DELETE before authentication
when the browser marks it Sec-Fetch-Site same-site or cross-site, or, from
a browser that sends no fetch metadata, when its Origin names a host other
than the one the request went to or APP_URL. A request with an
Authorization header, or with neither header (the phone app, scripts,
webhook senders), is unaffected, as are the webhook and URL-credential
routes that authenticate with their own secret. The JSON media type is
compared exactly, with parameters allowed, in safeJson and in every route
that branched on a substring of it.
The client-IP resolver trusted one forwarding hop by default and read the
rightmost X-Forwarded-For entry. Next fills that header with the socket's
address only when the request carries none, so a caller reaching the
published port directly could send its own and choose a fresh address per
request: every per-IP limit (sign-in, second factor, passkey, share unlock,
webhooks) became per-request, and sessions, the audit log and sign-in
notices recorded the chosen address.
The server now writes each connection's own peer address into an internal
header from Node's http.server.request.start diagnostics channel, installed
at boot and overwriting anything a caller sent under that name. The resolver
reads forwarding headers only when that peer is loopback, private,
unique-local or CGNAT, where a reverse proxy sits; a public peer is its own
client address, and a log line says once that its forwarding headers were
ignored. With TRUST_PROXY_HOPS=0 the peer is used instead of no address, so
a directly exposed host no longer shares one bucket across all anonymous
callers. Proxies on the same host, Docker network or tailnet need no change.
Several surfaces promised more than the per-account copies deliver:
- The full-backup export card said it had the same shape as the weekly
copy. It is a portable export with documents as metadata only; it now
says so, and that it cannot recreate an account on another server.
- The export page offered to "import from a backup". The JSON import adds
measurements and mood entries; it is now named that way, and its help
states the 16 MB and 10,000-entries-per-list limits from the constants
the route enforces.
- The weekly copy is labelled an account rollback on this server, and the
backups page says a copy cannot rebuild a lost server. The upload says
the account must exist here. Two restore refusals pointed to a
"disaster-recovery" copy; they now name the copies that exist.
- The instance-settings restore option was described as being for
rebuilding a host.
docs/ops/backup-restore.md is rewritten short and accurate: what each
copy is, the deletion grace window, that database dumps keep a deleted
account until they age out, how to restore one account, and a link to
the disaster-recovery procedure for a lost server.
Deleting an account, a guardian deleting a managed profile, and "Delete
all data" were final the moment they were confirmed: the account row
cascaded into every table and every stored copy, and the off-host objects
were purged in the same breath. Nothing short of rolling the whole
instance back could return one account.
ACCOUNT_DELETION_GRACE_DAYS (14 by default, at most 30, 0 erases at once
as before) now sets a window:
- Account and managed-profile deletion lock the account (migration 0488:
deletion_requested_at, deletion_purge_at, deletion_requested_by_id).
Every session, refresh token, API token, connector, share link and
step-up elevation is revoked in the same transaction, and the record,
its stored copies and its off-host objects are left as they are.
- Sign-in is refused with 403 auth.account.deletion_pending and the
erasure date, after the credential was proved and before a second
factor is asked for. The session and Bearer resolvers and the acting-
account resolver refuse a pending account too, so a credential minted
in the moment before the lock is useless.
- POST /api/admin/users/{id}/undo-deletion (cookie-only, audit-logged)
clears the lock. The admin users list reports the pending state.
- The hourly account-deletion-purge job erases accounts whose window has
ended, exactly as the immediate deletion did, with the off-host purge
request committed alongside. The Guardian invariant is decided again;
a refused erasure stays pending and is reported.
- A wipe first stores the whole record as a PRE_WIPE_<ms> copy through
the writer the weekly copy uses, keeps it out of the wipe, and holds
its off-host purge back with a new not_before on the request. The
ordinary restore brings the record back; restoring that copy withdraws
the held purge. The purge job deletes the copy when its window ends.
Background work skips accounts pending deletion through one shared
predicate (src/lib/account-deletion/active-account.ts): every reminder,
digest, Coach, sync, backfill, rollup, score and backup enumeration
filters with it, and account-deletion-enumeration-guard.test.ts fails on
a query in background work that neither filters nor says why it need
not. Notification delivery and the AI capability loader refuse a pending
account as chokepoints behind it, and the Withings, WHOOP and Telegram
webhooks and the notification quick actions ignore one.
The privacy summary publishes the window. Two pieces of the reminder tick
and the off-host pass moved into modules of their own to keep the files
inside their size ratchet.
The new-server steps lead with `db-restore latest --from-bucket`, the copy
that survives when the host and its local dumps are gone, and keep the
restore from downloaded files beside it. Both backup pages now say that the
database dumps are not purged per account: a deleted account stays in them
until they age out, which is the retention period an operator states to the
people using the instance.
The key backup sections become "Save the recovery kit": which keys the kit lists and why, the fingerprint of the token key, and the readiness card with its Backup health notification.
Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.
One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:
- database dump: failing once the newest good dump is older than 36 h
while dumps are on, or when it can be kept nowhere; read through a
provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
7 days after setup or as soon as a confirmed kit is out of date.
A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.
The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).
docs/ops/disaster-recovery.md says what is dumped and where, that local
dumps are unencrypted and why, which keys a restore needs and what each
one costs when missing, how to restore onto a new server or back to the
state before a migration, and how the procedure is tested. The deploy
runbook and the getting-started guide now describe the dump the
entrypoint takes before migrating, and the off-host account copy page
points at the dump for rebuilding a server.
Both files had drifted copies of the docs-site guides, which carry the same
sections and more, and nothing in the app, the scripts, the other docs or
the site links to them. Each is now a short pointer to its guide.
Marc-André Bombeck