Commit Graph
9383 Commits
Author SHA1 Message Date
Marc-André Bombeck aebd7ea612 docs(deploy): describe the deploy webhook's signature and strict mode
The compose comment still called DEPLOY_WEBHOOK_REQUIRE_TIMESTAMP a
requirement for a fresh timestamp header, which the route stopped
accepting in strict mode once requests could be signed, and the env
manifest named only the secret header and lacked the strict switch. Both
now say what the route checks: the secret header, which has no replay
protection, or X-Deploy-Webhook-Signature over the timestamp and body,
and that the strict setting takes signed requests only. The development
example says the same and points at the deploy runbook instead of a page
that no longer exists.
2026-10-25 16:00:35 +01:00
Marc-André Bombeck 877db9a5b7 docs(proxy): say TRUST_CF_CONNECTING_IP needs a Cloudflare-only origin
The flag makes the app believe CF-Connecting-IP from every connection,
before the peer check and whatever TRUST_PROXY_HOPS says, which is safe
only when nothing but Cloudflare can reach the origin. The reverse-proxy
page now says so with the ways to get there, the compose comment and both
example files carry the same condition, and the variable is in the env
manifest. The page's list of peers whose forwarding headers are read now
matches the code: loopback, 0.0.0.0/8, private, CGNAT, link-local,
unique-local and site-local, the reserved blocks and multicast. Those last
two stay trusted, since no internet host can connect from them; a test
pins every listed range and the Cloudflare precedence.
2026-10-25 16:00:35 +01:00
Marc-André Bombeck 8471e0edc9 fix(documents): say a PDF past the reader's limits is too large to read
A text read that crossed the isolated reader's memory or time bound came
back as a plain read error, which the index job records as decrypt-error,
so the document said its stored file could not be opened. The local
extractor now answers resource-limit for PdfResourceLimitError, the index
records it under that name, and the detail view says the PDF is too large
or complex to read here, in every locale. Like the thumbnail's refusal of
the same bytes it is final: nothing retries it on its own.
2026-10-25 16:00:35 +01:00
Marc-André Bombeck 66e92f36fe feat(recovery): judge the bucket's rule for database dumps in readiness
Only the backups page said when the bucket's lifecycle rules left
database/ out, so every dump, and every deleted account inside it, stayed
in the bucket forever without anyone being told. Readiness gains a
dump_retention signal that reads the same cached lifecycle probe: attention
when dumps go off-host and no rule covers database/, ok with the expiry
when one does, quiet when the bucket will not say, inactive without a
bucket or without off-host dumps. The hourly evaluation pushes it to the
admins like every other signal.
2026-10-25 16:00:35 +01:00
Marc-André Bombeck 351e7d1fe0 fix(backup): keep a dump locally only where it survives the container
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.

A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.

The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
2026-10-25 16:00:34 +01:00
Marc-André Bombeck fb4ab29aad Merge remote-tracking branch 'origin/release/v1.46.0' into feat/v146-grace 2026-10-25 15:28:23 +01:00
Marc-André Bombeck 1b39d93a8a docs(auth): say that credentials from before a deletion answer 401
The deletion revokes every session, refresh token and API token, so they
answer like any revoked credential; only new sign-ins get the 403
auth.account.deletion_pending, through the shared login response.
2026-10-25 15:21:10 +01:00
Marc-André Bombeck 9db5e755e6 Merge the release into the deletion grace work
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
2026-10-25 15:18:46 +01:00
Marc-André Bombeck 8678a3edc4 test(recovery): keep the weekly-copy push test to its one signal
With the real dump health wired in, an empty test database also reports
a missing dump. The file follows the weekly copy alone; the dump signal
is judged in the readiness unit tests.
2026-10-25 15:16:14 +01:00
Marc-André Bombeck da9d941a24 Merge the release with the security fixes into the backup health work 2026-10-25 15:03:17 +01:00
Marc-André Bombeck 13e62bbb92 docs(coach): keep the labs block note to its old length 2026-10-25 14:50:05 +01:00
Marc-André Bombeck e88dcd2c53 Merge the release into the backup health work and read the real dump health
The readiness model now takes the database dump's state from
getDatabaseBackupHealth, and the dump test reaches the off-host key id
where it moved.
2026-10-25 14:40:31 +01:00
Marc-André Bombeck c25e9fa4e9 refactor: drop an unused test reset for the public-peer warning 2026-10-25 14:28:32 +01:00
Marc-André Bombeck d902125371 feat(account): tell admins about a refused erasure, and what undo leaves revoked
When the grace window ends for an account that has become the last
active guardian of a managed profile, the purge job refuses to erase it
and the account stays locked. Until now only the worker error log said
so. Every admin now gets one localized notification naming the account
and the ways out (another guardian, deleting the profile, or undoing
the deletion). An audit row on the account records that it was reported,
so the hourly retries stay quiet; a deletion undone and asked for again
is reported afresh.

The undo confirmation in the admin users list now says that sign-in
works again but API tokens, share links and trusted devices stay revoked
and have to be set up again (for a managed profile: its share links and
tokens).
2026-10-25 14:28:15 +01:00
Marc-André Bombeck 694befd635 fix(admin): say what actually ages a stored account copy out
The backups page stated "Retention: 90 days", from a constant whose
comment said it matched a backup-prune job. There is no such job: nothing
deletes a stored copy by age. The weekly copy is replaced in place, one
per account; the copy taken before "Delete all data" is deleted by the
account-deletion purge when the grace window ends; an uploaded copy stays
until its account is erased.

The list now carries preWipeCopyDays (the grace window, 0 when no such
copy is taken) instead of retentionDays, and the page says exactly the
three facts above, in every language.

Also drops a re-export of the JSON import row limit nothing imports.
2026-10-25 14:28:07 +01:00
Marc-André Bombeck 459b5baa9d fix(backup): read ECG traces a strip at a time
The shared backup reader kept documents without their stored bytes but
kept every ECG strip whole, trace included. A long watch history is
hundreds of strips of tens to hundreds of kilobytes each, so an upload,
preview or restore of such a copy held the whole history in memory.

The first read now keeps each strip without its trace (a short stand-in,
after the key it was written under is recorded and its form checked),
and forEachEcgTrace reads the traces again one strip at a time. The ECG
restore writes each strip from that second read, as the document restore
already did, and refuses a file whose two reads disagree on the count.

The upload memory test gains an ECG-heavy file: 150 strips, about 200 MB.
The old reader held 197 MB of it and failed the 48 MB bound; the new one
holds about 1 MB, also under a 524 MB heap.
2026-10-25 14:27:56 +01:00
Marc-André Bombeck 0a053d6884 feat(admin): name a switched-off dump and a newly listed kit key plainly
A switched-off scheduled dump is attention now, sent once and never
reminded, with how to turn it back on (DB_BACKUP_ENABLED). Dumps are on
by default, so off is a choice that leaves no copy able to rebuild the
host. An install that never wrote a dump and has nowhere to write one
stays quiet.

The out-of-date kit notice says what changed: the kit now also lists a
key the last confirmation did not cover, for example API_TOKEN_HMAC_KEY
on an instance upgraded from the one-key step, and that nothing broke.

The French kit and readiness strings use vous, the form the rest of the
French bundle uses.
2026-10-25 14:09:41 +01:00
Marc-André Bombeck e79ea08b13 Merge remote-tracking branch 'origin/release/v1.46.0' into fix/v146-security 2026-10-25 14:07:01 +01:00
Marc-André Bombeck 21aa791c3a Merge remote-tracking branch 'origin/release/v1.46.0' into fix/v146-security
# Conflicts:
#	src/lib/ai/coach/labs-snapshot.ts
#	src/lib/ai/coach/snapshot.ts
2026-10-25 14:05:23 +01:00
Marc-André Bombeck 2f19ca1f73 test(ratchet): write back the smaller sizes the fixes left
apiHandler's request wrapper fell under the 200-line mark, and the
reminder-check route and the Coach snapshot shrank by a line or two.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck f7030feba7 docs(ops): say that the pre-upgrade volume keeps data erasure never reaches
The Postgres major-upgrade guide suggested removing the old pgdata volume
"when you are sure". That volume is a full copy of the database as it stood
before the move: account deletions, trash purges and key rotations after it
never reach it. The guide now says so and makes removing it part of the
move.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck a20897b305 fix(monitoring): sign the GlitchTip relay in and strip Prisma call arguments
The browser error relay forwarded anything posted to it into the operator's
GlitchTip without a session, limited only per address, so anyone could
fill the incident list. Separately, a Prisma client error prints the
failed call into its message and stack, with the arguments object (a
reading's value, a note's text) when no source map is at hand, and the
server sent that to GlitchTip and the logs after a credential scrub that
does not look for it.

The relay now needs a signed-in session and is limited per account; errors
on pages before sign-in go unreported. The central redaction cuts a Prisma
call's printed arguments, keeping the call, a reason line that names a
field, and the stack frames, and the GlitchTip sender applies the
redaction to every event at the egress boundary, whichever caller built it.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck f2a9b0fc0e fix(deploy-webhook): accept signed requests and stop calling a bare timestamp replay protection
The deploy-status webhook authenticated with a static secret in a header,
and the timestamp the docs called replay protection was covered by nothing:
whoever captures one request holds the secret and can send a fresh request
with any content. The strict setting, documented as closing the replay
window completely, only made that timestamp mandatory.

A sender can now sign instead: X-Deploy-Webhook-Signature is the
HMAC-SHA256 of the timestamp and the exact body under the shared secret,
with the timestamp within five minutes, and the secret never travels. The
strict setting accepts only signed requests. The secret header still works
by default for senders that cannot sign, and the docs describe its
timestamp as the staleness hint it is.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck 807d483b89 fix(egress): check every redirect hop and close the reserved-range gaps
safeFetch with requirePublicHost vetted the first URL and pinned DNS
answers in the connector, but Node never looks up an IP literal, so a
followed redirect to a literal private or metadata address was dialled
unchecked. No caller follows redirects to a user-chosen host today; the
floor did not hold for the next one that would.

With requirePublicHost, redirects are now followed one hop at a time and
every location goes through isPublicUrl before it is dialled, at most five
hops. isPublicIp also refuses the benchmarking, protocol-assignment,
multicast, reserved and broadcast IPv4 ranges, site-local and multicast
IPv6 and the IPv4-translated spelling of a private address, and isPublicUrl
strips a trailing dot before the hostname denylist.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck e021adde40 fix(notifications): limit quick answers per account, junk in a bucket of its own
The quick-answer route charged one per-address bucket before it read the
token. Where the client address cannot be read, every caller shares that
bucket, so two junk requests a minute switched off every person's
notification quick answers.

The token is an HMAC and is now checked first, which costs one hash and no
database read. A genuine token is limited per account; anything else is
limited per address in a separate bucket that genuine answers never draw
from.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck cd96cb3e60 fix(proxy): keep API paths ending in a file extension behind the proxy
The proxy matcher left out every path ending in an image or text extension
so static files skip it. The exclusion was not anchored, so an API path
whose last segment the caller chooses (a custom tag name ending in .png)
skipped the proxy too, and with it the demo-mode write refusal, the
worker-only refusal and the security headers. Authentication was unaffected.

The extension exclusion now applies outside /api/ only.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck 0039e112dd fix(mcp): fence medication and lab names in tool results
The MCP server instructions promise that the person's free text comes back
wrapped in USER_TEXT markers, but get_medication_schedule returned
medication names and doses as plain strings, and get_labs returned analyte,
panel and unit names that can come straight out of an uploaded document.
Text written to read like an instruction reached the connecting client's
model unmarked. The marker scrub also removed only the exact uppercase
spelling, so a lower-case or spaced marker survived inside fenced text.

Both tools now fence those fields, and the scrub removes a marker in any
case and with any spacing inside the brackets.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck e3598985f5 fix(ingest): answer a scoped token the same for unknown and out-of-scope names
A token scoped to one medication got 404 for a name that matched none of
the owner's medications and 403 for one that matched outside its scope, so
it could confirm which medications exist by name. Its idempotency replay
also ran before the scope check and returned any of the owner's doses
stored under the key.

A token without the wildcard now gets the same 403 for both, the replay is
looked up only after the scope check, and a key already spent on another
medication answers 409 without that dose. The wildcard token keeps its 404.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck 989381b3f6 fix(share): stop serving shared documents once the documents module is off
A share link gated its report on the doctorReport module but listed and
served its attached documents unconditionally, although the comment beside
the gate said the documents kept a gate of their own. An owner who switched
the documents module off still had every attached file decrypted for anyone
holding the link and its passphrase.

The share page lists no document while the module is off, and the document
route answers 404, the same as for an unknown document, before it looks the
membership up.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck d1fcbca63b fix(mcp): name the host an authorization goes to on the consent page
Dynamic client registration is open by specification and the client names
itself, so anyone could register under a well-known assistant's name with a
callback on a host of their own and send a signed-in person the authorize
link. The consent page showed the borrowed name and "Unverified
application", never where the code would be sent.

Every consent page now says where the access goes: the callback's host, or
an application on this device for a loopback callback, and the Allow button
names the host. For a self-registered client whose callback leaves the
device, a warning that names the host leads the page, above the
self-reported name. Registration stays open: the hosted assistants register
this way with their own https callback.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck 77ace106b3 fix(http): refuse state changes a browser sends from another site
The session cookie is SameSite=Lax, and SameSite compares the registrable
domain, so a page on a sibling subdomain of the instance could send a
CORS-simple POST with the person's cookie attached. safeJson accepted any
Content-Type containing "application/json", so a text/plain body that
mentions it in a parameter, which needs no preflight, was parsed as the
person's own request: readings added, entries deleted, integrations
disconnected, a delegate invited within the re-proof window.

apiHandler now refuses a POST, PUT, PATCH or DELETE before authentication
when the browser marks it Sec-Fetch-Site same-site or cross-site, or, from
a browser that sends no fetch metadata, when its Origin names a host other
than the one the request went to or APP_URL. A request with an
Authorization header, or with neither header (the phone app, scripts,
webhook senders), is unaffected, as are the webhook and URL-credential
routes that authenticate with their own secret. The JSON media type is
compared exactly, with parameters allowed, in safeJson and in every route
that branched on a substring of it.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck fa50bcd1cb fix(ai): keep lab values out of every model prompt while Labs is off
The Coach snapshot attached its labs block regardless of the module, a
decision from before Labs could be switched off, while the snapshot's own
prelude says a disabled domain never reaches the model. With Labs off, the
Coach prompt, its get_labs tool, the daily briefing's flagged-marker block,
the biomarker status note, the reaction line to a new panel and the MCP
lab_trend_brief prompt all still read lab values for a model.

Each of those reads now asks the labs module first and reads nothing when it
is off: the Coach tool answers module_disabled, the briefing omits the
block, the status note neither writes nor serves text, the reaction line
gets no value, and the MCP prompt refuses like get_labs and the doctor-visit
prompt already did. The data routes stay as they are. A structural guard
holds every lab-table reader under the model-input paths to the gate or to
a listed reason its rows never reach a model.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck 6fbfe71955 fix(auth): stop exempting a previously used address from the sign-in throttle
The per-account password throttle skipped every attempt that came from an
address one of the account's sessions or device logins had used in the last
thirty days. An address is the easiest thing about a request to share or
claim: in a household or an office the guesser has the owner's address, and
a direct caller could name it in a forwarding header. Either way the
throttle switched off and guessing was bounded only by the hash cost.

A remembered browser (the trusted-device cookie) and a phone presenting the
device id of one of the account's logins still pass; everything else gets
the growing wait.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck d6d0f7e73f fix(http): read forwarding headers only from a connection a proxy could make
The client-IP resolver trusted one forwarding hop by default and read the
rightmost X-Forwarded-For entry. Next fills that header with the socket's
address only when the request carries none, so a caller reaching the
published port directly could send its own and choose a fresh address per
request: every per-IP limit (sign-in, second factor, passkey, share unlock,
webhooks) became per-request, and sessions, the audit log and sign-in
notices recorded the chosen address.

The server now writes each connection's own peer address into an internal
header from Node's http.server.request.start diagnostics channel, installed
at boot and overwriting anything a caller sent under that name. The resolver
reads forwarding headers only when that peer is loopback, private,
unique-local or CGNAT, where a reverse proxy sits; a public peer is its own
client address, and a log line says once that its forwarding headers were
ignored. With TRUST_PROXY_HOPS=0 the peer is used instead of no address, so
a directly exposed host no longer shares one bucket across all anonymous
callers. Proxies on the same host, Docker network or tailnet need no change.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck ba22932101 fix(documents): read a JPEG's size past fill bytes and refuse an unreadable one
The thumbnail's header walk took the byte after any 0xFF as a marker. A
single 0xFF fill byte after the start-of-image marker, which the JPEG
standard allows and every decoder skips, threw it off; it read no size, and
an unread size let the image through to a decode that allocates the full
bitmap. A small JPEG declaring 20000 x 20000 pixels allocated 1.6 GB in the
web process before the post-decode check could see it.

The avatar upload already had a walker that skips fill bytes. Both now use
one reader (lib/image-dimensions) that follows the marker rules: fill bytes,
the length-less markers, and no size when a scan or the end of the image
comes before a frame header. The thumbnail treats an unreadable size as a
refusal (size-unreadable) for every format it decodes, and the backfill
counts that refusal as final.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck 518ce2fd37 fix(documents): read and render PDFs in a bounded child process
pdf.js decodes an image at the size the file declares, whatever size the
page is drawn at, and the server opened documents without an image limit.
A small upload declaring a very large image made the web process allocate
gigabytes until the container was killed, for every user at once, and the
thumbnail job ran the same document again after each restart. An image
whose soft mask declares the size does the same past pdf.js' own limit, and
nothing bounded how long a text read could take.

pdf.js is now opened with maxImageSize (20 million pixels, which keeps one
image inside the reader's ceiling and admits a 300 dpi A3 scan), and every
read and render runs in a child process, one at a time: its V8 heap is
capped, its resident memory is sampled and the child killed past 640 MB, on
Linux it volunteers to the out-of-memory killer first, a time limit ends a
read that runs on, and it starts with an almost empty environment. The
loader and the rendering moved into pdf-engine.mjs, which the child runs
with plain Node and which is traced into the standalone image.

A document that crosses a bound answers resource-limit. The thumbnail job
records it as a final failure, so the backfill does not queue it again.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck 0c5f29303b docs(backup): say what account copies and the JSON import can do
Several surfaces promised more than the per-account copies deliver:

- The full-backup export card said it had the same shape as the weekly
  copy. It is a portable export with documents as metadata only; it now
  says so, and that it cannot recreate an account on another server.
- The export page offered to "import from a backup". The JSON import adds
  measurements and mood entries; it is now named that way, and its help
  states the 16 MB and 10,000-entries-per-list limits from the constants
  the route enforces.
- The weekly copy is labelled an account rollback on this server, and the
  backups page says a copy cannot rebuild a lost server. The upload says
  the account must exist here. Two restore refusals pointed to a
  "disaster-recovery" copy; they now name the copies that exist.
- The instance-settings restore option was described as being for
  rebuilding a host.

docs/ops/backup-restore.md is rewritten short and accurate: what each
copy is, the deletion grace window, that database dumps keep a deleted
account until they age out, how to restore one account, and a link to
the disaster-recovery procedure for a lost server.
2026-10-25 13:51:15 +01:00
Marc-André Bombeck 63574b98a6 feat(account): show the deletion grace window and its undo
The deletion dialogs, the managed-profile delete dialog and the Data &
Privacy page read the window from the privacy summary and say what
happens: the account is locked at once, erased after the stated number
of days, and an admin of the server can undo it until then; a wipe keeps
a copy an admin can restore for the same time. Copies in the server's
backups age out after their retention period, and the text says that
too. With the window set to 0 they keep the immediate wording.

The sign-in page names the erasure date when a sign-in, a passkey or the
single sign-on callback is refused for an account pending deletion. The
admin users list shows a pending account with its date and an "Undo
deletion" action, and the backups list labels the copy taken before a
wipe. Every new string is translated in all nine languages.
2026-10-25 13:51:01 +01:00
Marc-André Bombeck a69666fa16 feat(account): hold account deletion and data wipe for a grace window
Deleting an account, a guardian deleting a managed profile, and "Delete
all data" were final the moment they were confirmed: the account row
cascaded into every table and every stored copy, and the off-host objects
were purged in the same breath. Nothing short of rolling the whole
instance back could return one account.

ACCOUNT_DELETION_GRACE_DAYS (14 by default, at most 30, 0 erases at once
as before) now sets a window:

- Account and managed-profile deletion lock the account (migration 0488:
  deletion_requested_at, deletion_purge_at, deletion_requested_by_id).
  Every session, refresh token, API token, connector, share link and
  step-up elevation is revoked in the same transaction, and the record,
  its stored copies and its off-host objects are left as they are.
- Sign-in is refused with 403 auth.account.deletion_pending and the
  erasure date, after the credential was proved and before a second
  factor is asked for. The session and Bearer resolvers and the acting-
  account resolver refuse a pending account too, so a credential minted
  in the moment before the lock is useless.
- POST /api/admin/users/{id}/undo-deletion (cookie-only, audit-logged)
  clears the lock. The admin users list reports the pending state.
- The hourly account-deletion-purge job erases accounts whose window has
  ended, exactly as the immediate deletion did, with the off-host purge
  request committed alongside. The Guardian invariant is decided again;
  a refused erasure stays pending and is reported.
- A wipe first stores the whole record as a PRE_WIPE_<ms> copy through
  the writer the weekly copy uses, keeps it out of the wipe, and holds
  its off-host purge back with a new not_before on the request. The
  ordinary restore brings the record back; restoring that copy withdraws
  the held purge. The purge job deletes the copy when its window ends.

Background work skips accounts pending deletion through one shared
predicate (src/lib/account-deletion/active-account.ts): every reminder,
digest, Coach, sync, backfill, rollup, score and backup enumeration
filters with it, and account-deletion-enumeration-guard.test.ts fails on
a query in background work that neither filters nor says why it need
not. Notification delivery and the AI capability loader refuse a pending
account as chokepoints behind it, and the Withings, WHOOP and Telegram
webhooks and the notification quick actions ignore one.

The privacy summary publishes the window. Two pieces of the reminder tick
and the off-host pass moved into modules of their own to keep the files
inside their size ratchet.
2026-10-25 13:50:14 +01:00
Marc-André Bombeck adc96fde3e Merge remote-tracking branch 'origin/release/v1.46.0' into feat/v146-dump 2026-10-25 13:49:51 +01:00
Marc-André Bombeck 726a73fcad refactor(admin): drop an unused kit type re-export 2026-10-25 13:49:22 +01:00
Marc-André Bombeck 4ffa3352bd ci: restore the drill from the bucket after the host is lost
The drill now runs an S3-compatible bucket beside the stack (Versity S3
Gateway v1.8.0, pinned by digest; the MinIO images are no longer published
on Docker Hub or quay.io) on a network and a volume of its own, so it
outlives the stack as a real bucket does. The app takes the dump through the
scheduled job and uploads it there; the drill waits for a run that both
kept and uploaded it, destroys every volume of the stack, the local backups
included, and restores on a new stack with only the bucket and the keys:
`docker compose run --rm db-restore latest --from-bucket`. The independent
row count reads the manifest from the bucket with the restore's own reader.

DRILL_SOURCE=local keeps the earlier path (dump files copied off the host,
then `db-restore latest`), and CI runs both. A new mutation,
wrong-backup-key, restores with a different BACKUP_ENCRYPTION_KEY and must
be refused. Ports are free ones picked at start.
2026-10-25 13:43:23 +01:00
Marc-André Bombeck 30a2ba2102 docs(ops): restore from the bucket after a host loss, and state backup retention
The new-server steps lead with `db-restore latest --from-bucket`, the copy
that survives when the host and its local dumps are gone, and keep the
restore from downloaded files beside it. Both backup pages now say that the
database dumps are not purged per account: a deleted account stays in them
until they age out, which is the retention period an operator states to the
people using the instance.
2026-10-25 13:43:23 +01:00
Marc-André Bombeck faf6dfbaef ci: wait for the published port in the database move check
The check failed once with "Can't reach database server at
127.0.0.1:55416": the container reported healthy before its published port
forwarded, and the migrate on the host ran in that gap. It now waits until
the port answers, retries a migrate that could not connect, both within
thirty seconds, and fails with Prisma's own output otherwise. The port is a
free one picked at start unless CHECK_DB_PORT names it, so the check cannot
collide with another stack on the same machine.
2026-10-25 13:43:23 +01:00
Marc-André Bombeck e825a4c4d3 feat(admin): say how long dumps keep deleted accounts, and check the bucket rule for database/
A database dump is the whole database at one moment, so an account deleted
since stays in the dumps taken before until they age out. The database row
on the backups card now says so with the numbers that decide it: how many
nightly dumps this server keeps, and after how many days the bucket's
lifecycle rule removes the off-host copies.

The lifecycle probe used to answer with the shortest expiry of any enabled
rule, whatever its prefix, so a rule scoped to the account copies read as
covering the dumps too. It now also answers for `database/` alone, counting
only rules whose filter takes in a dump's key (prefix, legacy prefix, an
And filter; a tag filter never matches untagged objects), and the card
warns when no rule covers that prefix: dumps there, and the deleted accounts
in them, would never be removed. The reading moved into its own module.
2026-10-25 13:43:23 +01:00
Marc-André Bombeck 82db3de30a feat(backup): restore a dump straight from the off-host bucket
A lost host takes its local dumps with it, and the copy left is the one in
the bucket. `docker compose run --rm db-restore latest --from-bucket` now
lists `database/` in the bucket the BACKUP_S3_* variables name, takes the
newest dump whose manifest is there too (or the one named), downloads both
with the S3 client the image already ships, and continues as for any
off-host copy: opened with BACKUP_ENCRYPTION_KEY, checked against the
manifest, restored into an empty database. `list --from-bucket` shows what
is there. The sealed download is removed once it is opened.

The image build now proves the S3 client resolves from the shared module's
place in the tree, so the restore cannot lose it unnoticed.
2026-10-25 13:43:23 +01:00
Marc-André Bombeck ed51211e82 docs(self-hosting): describe the recovery kit and the readiness push
The key backup sections become "Save the recovery kit": which keys the kit lists and why, the fingerprint of the token key, and the readiness card with its Backup health notification.
2026-10-25 13:32:43 +01:00
Marc-André Bombeck 7dab9cc5e9 feat(admin): judge every backup signal and push changes to admins
Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.

One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:

- database dump: failing once the newest good dump is older than 36 h
  while dumps are on, or when it can be kept nowhere; read through a
  provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
  failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
  window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
  7 days after setup or as soon as a confirmed kit is out of date.

A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.

The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).
2026-10-25 13:32:43 +01:00
Marc-André Bombeck d3c672cc3f feat(admin): replace the key backup step with a recovery kit
The step confirmed one key, the active application key. A restore needs
more: off-host copies are sealed under BACKUP_ENCRYPTION_KEY, a copy
written before a rotation needs the retired application key it was
written under, and every session, token and share link hangs off
API_TOKEN_HMAC_KEY. An operator could do everything the step asked and
still be unable to read a single backup.

The recovery kit lists every key a restore of this server needs, by id
and fingerprint, never key material:

- each id in ENCRYPTION_KEYS, plus any retired id a stored copy
  (innerKeyIds) or an off-host object inside the bucket's retention
  (OffhostBackupKeyUse) still lists, also once the server dropped it;
- BACKUP_ENCRYPTION_KEY and each BACKUP_ENCRYPTION_PREVIOUS_KEYS entry,
  named by the id their objects record;
- API_TOKEN_HMAC_KEY.

Confirming records the set the admin was shown and refuses a stale one
with 409 recovery.kit.stale. The kit is due again as soon as a key it
lists is not in that set (a rotation, a new backup key, a re-keyed
install); a key that merely drops out does not make it due. "Check my
copy" now checks any key in the kit.

Migration 0487 carries a confirmation given under the old step forward
as a kit holding that key, so that part stays confirmed, and drops the
four app_settings columns. It also creates the two tables the readiness
evaluation uses (restore_drill_outcomes, recovery_readiness_notices).

The routes move to /api/admin/recovery-kit; the reader of the keys the
backups still need moves out of the encryption status route into
src/lib/recovery/backup-key-needs.ts so the kit and the status view
share one definition.
2026-10-25 13:32:29 +01:00
Marc-André Bombeck 1a4289545e ci: rebuild a lost server from its dump in a compose drill
scripts/compose-dr-drill.sh plays docs/ops/disaster-recovery.md against
real containers: it builds the image from the checkout, starts the bundled
compose stack with fixed throwaway keys, seeds the demo account and a
Guardian with a managed profile (through the app's own code), writes a
reading with an encrypted note through the API, and waits for the
scheduled database-backup job to take the dump. It then keeps only the
dump files, destroys every volume, starts a new stack, restores with
`docker compose run --rm db-restore latest`, counts every table against
the manifest apart from the restore, and boots the app: health, the demo
login, the note read back, a document opened, the Guardian still seeing
the profile.

DRILL_MUTATION breaks one step on purpose (wrong-key, skip-table,
non-empty-target), and the drill fails on each. The workflow runs it on
pull requests that touch the compose file, the image, the entrypoint or
the dump and restore code, nightly, and by hand.
2026-10-25 13:05:47 +01:00