DEPLOY_WEBHOOK_REQUIRE_TIMESTAMP had been changed to mean signed requests
only, so an operator who set it for a sender that adds a timestamp but cannot
sign would have had every deploy notification refused after the upgrade. It
means the secret header plus a fresh timestamp again. Signature-only, the
setting that stops a replay, is DEPLOY_WEBHOOK_REQUIRE_SIGNATURE, on the
compose whitelist, in the env manifest and examples, and in the deploy
guide.
Forwarding headers were believed from any connection on a private address.
Behind Docker Desktop, OrbStack and docker-proxy every connection to a
published port arrives from the gateway, a private address, so a caller
reaching the port directly could again choose the client address that rate
limits, the audit log and sign-in notices record. TRUSTED_PROXY_CIDRS
(comma-separated CIDRs) now names the proxies: when set, only those peers may
report a client address. The private-range default stays when it is unset,
the boot log states the rule in force in one line, and the reverse-proxy
guide says what to set for a proxy on the same host, on a Docker network, or
on a public address (which, unlisted, puts all clients in one bucket).
An invalid TRUST_PROXY_HOPS threw on every request that resolved the client
address, so every route answered 500, the health probe included. It now
resolves no address, warns once, and is named at boot.
A cookie named healthlog_session can be set by any host under the same
registrable domain, so a page on a sibling subdomain could plant a session of
its choosing in a visitor's browser. Wherever the session cookie is Secure it
is now issued as __Host-healthlog_session, which a browser accepts only from
this exact host. Plain-HTTP installs, which cannot hold a __Host- cookie,
keep the old name.
Nobody is signed out: the old name is still read after the new one, a
session found under it is re-issued under the new name and the old cookie
expired, and a browser holding the new cookie is never answered from the old
one. Every reader and writer goes through one module, which the proxy, the
native-caller check, the native handoff and the OIDC callback use; the
session-surface guard follows the new reader.
A read a person started now runs as interactive work of their account, and
the document jobs as background work of theirs, so the reader's queue can
order and cap them. When the reader is busy or stopped before it answered,
a lab scan, label read or document read fails with aiRuns.readerBusy (503,
nothing read or charged) instead of a misleading unreadable-file error, and
the index and summary jobs throw so the queue runs them again rather than
recording an outcome about the document.
The reproductive-health page promised that deleting all data or the
account erases at once. Both wait for the grace window now, the nightly
dumps keep a record until they age out, and the old pgdata volume is a
full copy until it is removed; the page says so and how to make deletion
final. The README, the getting-started note on majors, the move guide and
the Unraid and Portainer templates catch up with Postgres 18, the backups
volume and the recovery kit. The demo seed's spread bound leaves room for
the seed's own one-in-150 spread.
Going back to Postgres 16 after a move that stopped half-way left the
moving marker, and the next start emptied the live 16 database and
started a second server on its data directory. db-upgrade now waits for
the database service and does nothing unless it runs Postgres 18.
Using 16 again after a finished move and then updating started the app
on the move-day copy without a word. The marker now records the old
database's last checkpoint, and a later start refuses when it moved on;
the guide says how to move the current database instead. The disk check
leaves room for the first pre-migration dump, and the move check seeds
the old database with v1.45's own migrations.
The fourth rollback step had been joined onto the paragraph after the
list; it is a list item again, with the way back to the data after it.
The disaster recovery page now says how the server tells admins about
its copies: the recovery readiness card, the recovery kit and the hourly
Backup health notification. The support page put GlitchTip under Admin,
General; the card is under Admin, Integrations, as the deploy runbook
says. The pre-migration dump notes say that without the backups volume
no dump is taken, also where the image's own directory is writable.
The restore refuses a file that carries any document as metadata only
(findIncompleteDocument), whatever the account holds; the page said it
refused when the account held a document.
The flag makes the app believe CF-Connecting-IP from every connection,
before the peer check and whatever TRUST_PROXY_HOPS says, which is safe
only when nothing but Cloudflare can reach the origin. The reverse-proxy
page now says so with the ways to get there, the compose comment and both
example files carry the same condition, and the variable is in the env
manifest. The page's list of peers whose forwarding headers are read now
matches the code: loopback, 0.0.0.0/8, private, CGNAT, link-local,
unique-local and site-local, the reserved blocks and multicast. Those last
two stay trusted, since no internet host can connect from them; a test
pins every listed range and the Cloudflare precedence.
A text read that crossed the isolated reader's memory or time bound came
back as a plain read error, which the index job records as decrypt-error,
so the document said its stored file could not be opened. The local
extractor now answers resource-limit for PdfResourceLimitError, the index
records it under that name, and the detail view says the PDF is too large
or complex to read here, in every locale. Like the thumbnail's refusal of
the same bytes it is final: nothing retries it on its own.
Only the backups page said when the bucket's lifecycle rules left
database/ out, so every dump, and every deleted account inside it, stayed
in the bucket forever without anyone being told. Readiness gains a
dump_retention signal that reads the same cached lifecycle probe: attention
when dumps go off-host and no rule covers database/, ok with the expiry
when one does, quiet when the bucket will not say, inactive without a
bucket or without off-host dumps. The hourly evaluation pushes it to the
admins like every other signal.
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.
A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.
The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
The deletion revokes every session, refresh token and API token, so they
answer like any revoked credential; only new sign-ins get the 403
auth.account.deletion_pending, through the shared login response.
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
The Postgres major-upgrade guide suggested removing the old pgdata volume
"when you are sure". That volume is a full copy of the database as it stood
before the move: account deletions, trash purges and key rotations after it
never reach it. The guide now says so and makes removing it part of the
move.
The deploy-status webhook authenticated with a static secret in a header,
and the timestamp the docs called replay protection was covered by nothing:
whoever captures one request holds the secret and can send a fresh request
with any content. The strict setting, documented as closing the replay
window completely, only made that timestamp mandatory.
A sender can now sign instead: X-Deploy-Webhook-Signature is the
HMAC-SHA256 of the timestamp and the exact body under the shared secret,
with the timestamp within five minutes, and the secret never travels. The
strict setting accepts only signed requests. The secret header still works
by default for senders that cannot sign, and the docs describe its
timestamp as the staleness hint it is.
A token scoped to one medication got 404 for a name that matched none of
the owner's medications and 403 for one that matched outside its scope, so
it could confirm which medications exist by name. Its idempotency replay
also ran before the scope check and returned any of the owner's doses
stored under the key.
A token without the wildcard now gets the same 403 for both, the replay is
looked up only after the scope check, and a key already spent on another
medication answers 409 without that dose. The wildcard token keeps its 404.
The session cookie is SameSite=Lax, and SameSite compares the registrable
domain, so a page on a sibling subdomain of the instance could send a
CORS-simple POST with the person's cookie attached. safeJson accepted any
Content-Type containing "application/json", so a text/plain body that
mentions it in a parameter, which needs no preflight, was parsed as the
person's own request: readings added, entries deleted, integrations
disconnected, a delegate invited within the re-proof window.
apiHandler now refuses a POST, PUT, PATCH or DELETE before authentication
when the browser marks it Sec-Fetch-Site same-site or cross-site, or, from
a browser that sends no fetch metadata, when its Origin names a host other
than the one the request went to or APP_URL. A request with an
Authorization header, or with neither header (the phone app, scripts,
webhook senders), is unaffected, as are the webhook and URL-credential
routes that authenticate with their own secret. The JSON media type is
compared exactly, with parameters allowed, in safeJson and in every route
that branched on a substring of it.
The client-IP resolver trusted one forwarding hop by default and read the
rightmost X-Forwarded-For entry. Next fills that header with the socket's
address only when the request carries none, so a caller reaching the
published port directly could send its own and choose a fresh address per
request: every per-IP limit (sign-in, second factor, passkey, share unlock,
webhooks) became per-request, and sessions, the audit log and sign-in
notices recorded the chosen address.
The server now writes each connection's own peer address into an internal
header from Node's http.server.request.start diagnostics channel, installed
at boot and overwriting anything a caller sent under that name. The resolver
reads forwarding headers only when that peer is loopback, private,
unique-local or CGNAT, where a reverse proxy sits; a public peer is its own
client address, and a log line says once that its forwarding headers were
ignored. With TRUST_PROXY_HOPS=0 the peer is used instead of no address, so
a directly exposed host no longer shares one bucket across all anonymous
callers. Proxies on the same host, Docker network or tailnet need no change.
Several surfaces promised more than the per-account copies deliver:
- The full-backup export card said it had the same shape as the weekly
copy. It is a portable export with documents as metadata only; it now
says so, and that it cannot recreate an account on another server.
- The export page offered to "import from a backup". The JSON import adds
measurements and mood entries; it is now named that way, and its help
states the 16 MB and 10,000-entries-per-list limits from the constants
the route enforces.
- The weekly copy is labelled an account rollback on this server, and the
backups page says a copy cannot rebuild a lost server. The upload says
the account must exist here. Two restore refusals pointed to a
"disaster-recovery" copy; they now name the copies that exist.
- The instance-settings restore option was described as being for
rebuilding a host.
docs/ops/backup-restore.md is rewritten short and accurate: what each
copy is, the deletion grace window, that database dumps keep a deleted
account until they age out, how to restore one account, and a link to
the disaster-recovery procedure for a lost server.
Deleting an account, a guardian deleting a managed profile, and "Delete
all data" were final the moment they were confirmed: the account row
cascaded into every table and every stored copy, and the off-host objects
were purged in the same breath. Nothing short of rolling the whole
instance back could return one account.
ACCOUNT_DELETION_GRACE_DAYS (14 by default, at most 30, 0 erases at once
as before) now sets a window:
- Account and managed-profile deletion lock the account (migration 0488:
deletion_requested_at, deletion_purge_at, deletion_requested_by_id).
Every session, refresh token, API token, connector, share link and
step-up elevation is revoked in the same transaction, and the record,
its stored copies and its off-host objects are left as they are.
- Sign-in is refused with 403 auth.account.deletion_pending and the
erasure date, after the credential was proved and before a second
factor is asked for. The session and Bearer resolvers and the acting-
account resolver refuse a pending account too, so a credential minted
in the moment before the lock is useless.
- POST /api/admin/users/{id}/undo-deletion (cookie-only, audit-logged)
clears the lock. The admin users list reports the pending state.
- The hourly account-deletion-purge job erases accounts whose window has
ended, exactly as the immediate deletion did, with the off-host purge
request committed alongside. The Guardian invariant is decided again;
a refused erasure stays pending and is reported.
- A wipe first stores the whole record as a PRE_WIPE_<ms> copy through
the writer the weekly copy uses, keeps it out of the wipe, and holds
its off-host purge back with a new not_before on the request. The
ordinary restore brings the record back; restoring that copy withdraws
the held purge. The purge job deletes the copy when its window ends.
Background work skips accounts pending deletion through one shared
predicate (src/lib/account-deletion/active-account.ts): every reminder,
digest, Coach, sync, backfill, rollup, score and backup enumeration
filters with it, and account-deletion-enumeration-guard.test.ts fails on
a query in background work that neither filters nor says why it need
not. Notification delivery and the AI capability loader refuse a pending
account as chokepoints behind it, and the Withings, WHOOP and Telegram
webhooks and the notification quick actions ignore one.
The privacy summary publishes the window. Two pieces of the reminder tick
and the off-host pass moved into modules of their own to keep the files
inside their size ratchet.
The new-server steps lead with `db-restore latest --from-bucket`, the copy
that survives when the host and its local dumps are gone, and keep the
restore from downloaded files beside it. Both backup pages now say that the
database dumps are not purged per account: a deleted account stays in them
until they age out, which is the retention period an operator states to the
people using the instance.
The key backup sections become "Save the recovery kit": which keys the kit lists and why, the fingerprint of the token key, and the readiness card with its Backup health notification.
Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.
One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:
- database dump: failing once the newest good dump is older than 36 h
while dumps are on, or when it can be kept nowhere; read through a
provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
7 days after setup or as soon as a confirmed kit is out of date.
A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.
The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).
docs/ops/disaster-recovery.md says what is dumped and where, that local
dumps are unencrypted and why, which keys a restore needs and what each
one costs when missing, how to restore onto a new server or back to the
state before a migration, and how the procedure is tested. The deploy
runbook and the getting-started guide now describe the dump the
entrypoint takes before migrating, and the off-host account copy page
points at the dump for rebuilding a server.
Both files had drifted copies of the docs-site guides, which carry the same
sections and more, and nothing in the app, the scripts, the other docs or
the site links to them. Each is now a short pointer to its guide.
Fix two dead anchors (the MCP trust model, the README quick start) and the
statements that no longer match the app: a guardian can connect a managed
profile's services, the environment module is on by default and reaches the
air-quality host too, its readable location columns were dropped in 0409,
settings sections carry their current names, the Withings card lives on
/settings/integrations, the TrueNAS start period matches the image, the
deploy check names HEALTHLOG_PROD_URL, and the docs index describes what
self-hosting/ and integrations/ hold.
The published contract said the nine health-shaped fields that predate
`health` were removed in v1.44, but every status read still sends them:
older native builds decode them, the current one falls back to them against
a server without `health`, and the web card's copied diagnostics read the
failure streak. They stay deprecated, without a removal release.
The compose file runs Postgres 18. An install from v1.45 or earlier keeps
its Postgres 16 data in `pgdata`, which 18 cannot open, so a one-shot
`db-upgrade` service moves it on the first start: it starts Postgres 16 on
the old volume inside its own container, streams a dump into 18 in one
transaction, compares every table's row count, gathers planner statistics
and marks the old volume as moved. The app waits for it to succeed.
The old volume is never changed apart from the marker, so the way back is
the previous image and volume lines. A move cut short leaves a "moving"
marker and runs again from scratch; an emptied new volume after a finished
move is refused rather than refilled with the old state.
scripts/compose-db-upgrade-check.sh plays all of it against real
containers from the v1.45.0 compose file, and runs in CI on every change
to the compose file. Pull requests run the integration suite and e2e on
Postgres 18; the nightly integration run keeps 16 covered for installs
that bring their own database.
A concurrent build that is still running is INVALID with its migration not
yet finished, exactly like a failed one. With the web and worker roles in
two containers, the second boot dropped the index the first was building,
which cancelled the first one's migration and left it failed until someone
resolved it by hand.
The check and the drop now run in one transaction that first takes Prisma's
migration lock without waiting and does nothing when another instance holds
it; an index Postgres reports as being built is never dropped. One
transaction also keeps lock_timeout on the statement it bounds behind a
transaction pooler, and the applied-migrations lookup quotes the schema, so
a mixed-case schema no longer reads as having none applied.
The compose file stays on 16. Changing only the image line stops the stack:
the 18 image refuses a data directory from an older major and expects its
volume one level up. The runbook walks the dump and restore into a fresh
volume, keeps the old one as the way back, and is linked from the update
section and the compose pin.
The index route without Prefer: respond-async, which the iPhone app
calls, continues into the lab staging with no run schedule. The staging
then planned a long report's text as up to twenty reads with a retry
each: forty model calls inside one HTTP request, the same class the
extract route was bounded against.
The staging now plans a missing schedule as one request does
(SYNCHRONOUS_READ_SCHEDULE, at most three reads): a longer report has
its text shared out between them, and what does not fit is left out and
reported as extractionIncomplete on the document. The request bound
moves into ocr-plan beside the schedule it bounds, so the extract run
and the staging share one definition. Background runs and the automatic
index job keep planning against their own deadline.
A CREATE INDEX CONCURRENTLY that fails half-way leaves an INVALID index,
and the migration re-run after migrate resolve --rolled-back finds it
through IF NOT EXISTS and does nothing: recorded applied, with an index
the planner never uses. 0477 has this shape.
The boot now drops an INVALID index that a not-yet-applied migration
builds concurrently before it runs migrate deploy, with a bounded lock
wait, so the re-run builds it again. An INVALID index whose migration is
recorded applied is named on stderr with the rebuild that mends it. The
deploy runbook says both.
The backup carried neither the import ledger nor the tombstones of
deleted documents, so after a restore onto a fresh instance a token
importer stored every imported document the person had deleted on
purpose again.
The document filing section now carries documentImportKeys: the
ledger's rows plus the own and alias keys of the documents waiting in
Recently deleted, which still do not travel as documents. The restore
writes them back, minus any key a restored live document holds, and
counts them in the restore summary and the preview. DocumentImportKey
moves from the excluded models to the two-ended ones, and the round
trip, the column audit and the parity goldens carry a ledger row.
Without Prefer: respond-async the extract route ran the read with no
schedule, so a twenty-page report read a page at a time, with a retry
each, could make forty model calls in one HTTP request. The iPhone app
sends no Prefer header and reads the document detail from the answer,
so the request keeps its 200 and its body.
An extraction without a run deadline now plans at most three reads (six
calls): a longer document has its pages or text shared out between
them, and what does not fit is reported as extractionIncomplete on the
document. The background run keeps reading it a page at a time.
Refs #1257
A long document read in parts can end with a part cut at the provider's
output limit, a part never read, or more facts than one review takes.
That was told once, as a notice when the run finished; a review opened
later looked complete.
InboundDocument.extractionIncomplete (migration 0483) is written with
the facts by both stagings, the extract route or run and lab
auto-staging. The document detail publishes it and the review shows the
same note whenever it is opened. The disaster-recovery backup carries
it; an older file restores it as false.
Refs #1257
The per-account preview switch was kept for the topic overview pages,
which were dropped before they shipped. Its one flag was stored, served
and backed up, and read by nothing; the column-reader audit listed it as
a pending consumer that will not come.
Removed completely:
- `/api/settings/preview-features` (GET, PATCH) is retired: it answers
410 Gone with `meta.errorCode: route.retired` and `removedIn: 1.46.0`,
and the contract publishes it under Retired. The route, its validation
schema, the `PreviewFeatures` OpenAPI schemas and the settings module
are deleted.
- Migration 0485 drops `users.preview_features_json` under a bounded lock
wait, with the retry marker.
- The backup no longer writes the field. A file from v1.44 or v1.45 that
carries it still restores: the schema accepts it and no codec reads it,
so nothing is refused or written for it. The backup-parity goldens
lose the field and apply one setting fewer.
- The data-wipe plan entry and the sharing-surface actor entry go with
the route (frozen count 273 -> 272).
No client build reads the route: neither iOS repository nor any local
checkout references it.
Since handlers load on their queue's first job, a module the image
cannot load no longer fails the worker's boot. The worker comes up
healthy, the queue burns its retries when it first has work, and a
one-shot job (a restore, an import tail) ends failed with nobody told at
deploy time.
After binding every queue the worker now runs `preloadHandlers` in the
background, without awaiting it, so readiness is unchanged. It loads
each handler once through the same memoised loader its queue is bound
to, so a module that loads stays loaded for the first job. Each failure
is reported on the `job.run.failed` line at error, with the queue's name
and `reason: "handler_load_failed"`, plus the worker error report
(stderr and GlitchTip). A summary line on stderr names every queue that
failed. The sweep never throws, and a failing report does not stop it.
docs/ops/deploy.md says what to look for after a deploy.
canStartCourse answered true on the day a course was ended with End
course, while a course starting that day shares its last day and the
start refused it with 422 overlap. The start is now offered from the
day after the last course's end, the first day validation and the
engine accept.
A course ended at an instant is over for every rule that asks about
today, whatever the zone it is read from: after a move west the course
rows read CURRENT while the medication's own status read ENDED. The
course rows, the coach summary, the overlap rule and the previous-end
read take the instant-aware reading.
Refs #1280
Marc-André Bombeck