Commit Graph
675 Commits
Author SHA1 Message Date
Marc-André Bombeck 9eed407ccc feat(measurements): fold Google Health and Oura heart rate like the exports
The dense hourly fold only took Apple Health and Health Connect samples,
so heart rate from the provider syncs that store one row per sample
stayed raw forever. One self-hosted account held about 25 million
Google Health pulse rows after a full-history backfill, and every
reader and fold that falls back to the raw table paid for it (#1297).

The fold now walks a map of sources to the dense types each stores
sample by sample: every dense type for Apple Health and Health Connect,
heart rate for Google Health and Oura. Their daily SpO2 values are no
intraday stream and stay as they are; Fitbit does not sync intraday
heart rate, and WHOOP, Withings and Polar write dense types only once
per recovery, night or spot check. Each source folds into its own
hourly stats rows, and the DAY rollup is still taken from the raw
samples before they go.

The fold deletes the samples it averages, and the provider syncs write
first-write-wins on the externalId, so a backfill or full sync would
have put every folded sample back. Google Health would even have
re-keyed an hourly row into a sample that fell on its anchor through
the natural-key rescue. The folded_window rule now has one hour rule
per fold source, and both syncs drop the samples of an hour the fold
already took before they write. Recent samples cost no lookup.

The boot discovery of the fold probes each account and source for the
first raw sample it would take instead of grouping every such row, and
the one-time fold repair stays on the two sources it was written for.
2026-10-29 09:06:17 +01:00
Marc-André Bombeck 1d8e2e0ee3 fix(rollups): fold the full backfill a year at a time under its own budget
On an account with tens of millions of readings the boot full backfill
never finished. It ran four aggregates over the whole five-year window,
each held to the connection's 60 s statement limit, so every attempt
was cancelled with 57014, the reading extremes added by 0417 stayed
NULL, and the readers that need them fell back to the raw table and
timed out as well (#1298).

The full backfill now folds the window one UTC calendar year per pass,
newest first, and every statement runs under its own five-minute limit,
the shape the pulse day-mean rebuild already uses. Calendar-year cuts
write exactly the buckets one pass writes: day, month and year buckets
fall inside one window and a week across New Year is folded whole by
both. Each window commits on its own, so a run that reaches its job
budget hands the remaining years to a follow-up job instead of failing
and starting over on the retry.

The boot discovery read every live reading in the window to find
uncovered days, the same scan that kept the fold from finishing. It
now walks the days per account and type off the live index, so its
cost follows the number of days with readings, and runs under the same
limit. The full backfill code moves to its own module.
2026-10-29 08:52:36 +01:00
Marc-André Bombeck bcb31c80db Merge the release with the security review fixes into the data-safety review fixes 2026-10-25 18:36:54 +01:00
Marc-André Bombeck d2b8da7839 refactor: keep the size, cast and error-code guards where they were
The fixes above grew six files over 800 lines and three long functions, added
casts in tests and one error code the catalogue did not list. The claimable
profile check moves into its own module, the guardian count is reused by the
revoke path, the lifecycle fallback lives beside the lifecycle rules, the
key-needs list is its own component, the coach sweep wraps its filters with
ownedByActiveAccount, comments are tightened, and the test fixtures are typed
instead of cast. The enumeration guard reads a write keyed by one handed id
(id: runId, id: job.data.jobId) as scoped, so those need no marker.
account.deletion.no_guardian joins the catalogue; two baselines go down.
2026-10-25 18:06:32 +01:00
Marc-André Bombeck f63d5d5819 fix(ai-runs): do not count time in the document reader against a run's deadline
A background AI run's deadline is armed in model time plus a fixed margin.
Time spent waiting in the document reader's queue, and the read itself, came
out of that margin, so a person's lab scan or document read could be timed
out by the reaper because other work held the reader. When the reader grants
the slot it now moves the run's deadline by the wait plus the most the read
may take, and gives back the unused part when the read ends, never past the
job's own expiry and only while the run is running. The interactive queue
wait stays within the margin, which covers the run until the slot is
granted.
2026-10-25 18:02:45 +01:00
Marc-André Bombeck 42940c3a98 fix(jobs): the encryption backfill still runs for an account pending deletion
The per-account free-text encryption backfill seals what is already stored
and changes nothing the record holds, which is why its module is exempt from
the frozen-account rule; the new pending-deletion gate dropped its per-user
jobs anyway. A queue can now say it acts for pending accounts
(actsForPendingAccounts), and this one does.

Also types the ingest route test's token fixture to the row it now reads,
and drops an unused variable from the deletion grace test.
2026-10-25 17:51:13 +01:00
Marc-André Bombeck d3257a1b6b fix(documents): tell people when the document reader is busy, and attribute its work
A read a person started now runs as interactive work of their account, and
the document jobs as background work of theirs, so the reader's queue can
order and cap them. When the reader is busy or stopped before it answered,
a lab scan, label read or document read fails with aiRuns.readerBusy (503,
nothing read or charged) instead of a misleading unreadable-file error, and
the index and summary jobs throw so the queue runs them again rather than
recording an outcome about the document.
2026-10-25 17:38:25 +01:00
Marc-André Bombeck 4d970561fa fix(documents): decode thumbnails in the reader child, share it fairly, judge exits by cause
Three defects in the bounded document reader.

Image uploads were decoded into their thumbnails in the server process. A
header check is all that stood between a decoder and an allocation the file
chooses, and the GIF check showed how that fails. The image is now decoded in
the same child process PDFs run in, under its memory ceiling and time limit;
the header check stays in front of it.

The reader's one slot went to whoever asked first, and a waiting item waited
without end, so a few heavy PDFs from one account held everyone's lab scans
behind them. The slot now goes to a read a person is waiting on before
background work, to accounts in turn, to a few waiting items per account, and
an item that waits past its lane's deadline is answered busy, which callers
treat as retryable.

Any child that ended without an answer counted as a document past its limits,
so a kernel kill caused by the container's overall memory marked a good PDF
as permanently failed. Only a bound the reader measured (the sampled memory,
the time limit, the child's heap cap) is final now; any other end is run once
more and then answered interrupted, which the thumbnail job retries within its
ordinary attempts.
2026-10-25 17:38:25 +01:00
Marc-André Bombeck 2b79c01952 fix(offhost): judge the bucket's rules for database/ against this instance's folder
The dumps now live under database/<host of APP_URL>/, so the rule check reads
a sample key in that folder: a rule scoped to this instance's folder counts,
one scoped to another instance's does not.
2026-10-25 17:30:58 +01:00
Marc-André Bombeck 6051488b65 fix(restore-drill): a drill that read nothing back is no pass
A drill that stopped on its time budget before the first object recorded
ok with zero accounts checked, and the readiness tile said passed. Such a
run now records its time and counts and no verdict, so the last pass or
failure stands, and readiness calls it attention (nothing_checked) in every
locale.

The upsert that keeps a pass and a failure apart was stubbed in the drill's
tests and tested nowhere; it now has an integration test read back through
the readiness rule. The drill test that claimed to keep a failed verdict
when keeping it fails is named for what it checks.
2026-10-25 17:26:40 +01:00
Marc-André Bombeck a857fd0264 fix(backup-restore): a restored pre-wipe copy withdraws only its own wipe's purge
Restoring the copy a data wipe took withdrew every held data_wiped purge of
the account. With two wipes in one window, restoring the first copy also
withdrew the second wipe's purge, and the off-host copies of what the person
entered between the wipes, and wiped, stayed until the bucket's rule removed
them. Only the request written after this copy and before the next one goes
now. The withdrawal had no test; it has one against Postgres.
2026-10-25 17:26:39 +01:00
Marc-André Bombeck 2bb030b656 fix(offhost): keep the bucket's last lifecycle answer through a failed read
A lifecycle read that timed out once was cached as unknown for ten minutes,
which kept every key an off-host object ever needed in the recovery kit: the
kit turned due, the admins were pushed attention, and the next good read
pushed recovered. A failed read now stands on the last answer the bucket gave,
for up to a day.
2026-10-25 17:23:51 +01:00
Marc-André Bombeck 40c310f0ad fix(account-deletion): hold frozen accounts out of the work the guard missed
The enumeration guard read any query that named userId as scoped, including
an orderBy tiebreak, so the nightly recovery, strain and stress score jobs
grouped every account's measurements, wrote scores into frozen records and
spent cap slots on them. It also looked only at findMany and groupBy under
lib/jobs.

The guard now reads a scope from a query's filter only (orderBy, by,
distinct, select and include are dropped first), covers count, aggregate,
updateMany, deleteMany and executeRaw, and walks the boot discoveries and
sweeps that live beside their domain: the rollup and mood rollup discovery,
the Fitbit jobs, the cumulative record rederivation, the intake slot dedup,
the inventory expiry and the climb backfill. What it then found is filtered
(the score cohorts through excludePendingOwners, which filters before the
cap; Fitbit sync, backfill and sleep repair; coach proposal lapsing and
reminder nagging; expired coach reminders; snooze clearing; inventory
expiry) or says why it needn't be (retention and expiry sweeps over rows,
writes scoped by id).

Jobs queued before a deletion ran anyway, since handlers act on the userId
they carry without asking again. The binding of every queue now completes a
job whose payload names an account pending deletion without loading its
handler (pending-deletion-gate). The silent medication intake push, which
does not go through the dispatcher, refuses a pending account itself.
2026-10-25 17:13:09 +01:00
Marc-André Bombeck 4137aa546e fix(account-deletion): one failing erasure no longer holds up the others
The hourly purge rethrew any error that was not the Guardian refusal, so the
oldest due account, failing on every run, stopped every later erasure and the
sweep of expired pre-wipe copies for good.

Each account is now its own try: a failure is counted, reported through the
worker error path with its first error, and told to the admins once per
request (user.account.erasure_failed), and the run goes on. The pre-wipe
sweep and the off-host kick run in a finally. Every account the run starts is
marked in backup_pass_attempts under the purge's queue name and the mark
closes only on erasure or when the account is no longer due, so an account
that failed or was refused goes behind every other due account next time.
The run stops starting accounts once its job budget is spent, since fifty
erasures of up to two minutes each did not fit the queue's expiry.
2026-10-25 16:56:57 +01:00
Marc-André Bombeck 9db5e755e6 Merge the release into the deletion grace work
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
2026-10-25 15:18:46 +01:00
Marc-André Bombeck da9d941a24 Merge the release with the security fixes into the backup health work 2026-10-25 15:03:17 +01:00
Marc-André Bombeck e88dcd2c53 Merge the release into the backup health work and read the real dump health
The readiness model now takes the database dump's state from
getDatabaseBackupHealth, and the dump test reaches the off-host key id
where it moved.
2026-10-25 14:40:31 +01:00
Marc-André Bombeck d902125371 feat(account): tell admins about a refused erasure, and what undo leaves revoked
When the grace window ends for an account that has become the last
active guardian of a managed profile, the purge job refuses to erase it
and the account stays locked. Until now only the worker error log said
so. Every admin now gets one localized notification naming the account
and the ways out (another guardian, deleting the profile, or undoing
the deletion). An audit row on the account records that it was reported,
so the hourly retries stay quiet; a deletion undone and asked for again
is reported afresh.

The undo confirmation in the admin users list now says that sign-in
works again but API tokens, share links and trusted devices stay revoked
and have to be set up again (for a managed profile: its share links and
tokens).
2026-10-25 14:28:15 +01:00
Marc-André Bombeck e79ea08b13 Merge remote-tracking branch 'origin/release/v1.46.0' into fix/v146-security 2026-10-25 14:07:01 +01:00
Marc-André Bombeck fa50bcd1cb fix(ai): keep lab values out of every model prompt while Labs is off
The Coach snapshot attached its labs block regardless of the module, a
decision from before Labs could be switched off, while the snapshot's own
prelude says a disabled domain never reaches the model. With Labs off, the
Coach prompt, its get_labs tool, the daily briefing's flagged-marker block,
the biomarker status note, the reaction line to a new panel and the MCP
lab_trend_brief prompt all still read lab values for a model.

Each of those reads now asks the labs module first and reads nothing when it
is off: the Coach tool answers module_disabled, the briefing omits the
block, the status note neither writes nor serves text, the reaction line
gets no value, and the MCP prompt refuses like get_labs and the doctor-visit
prompt already did. The data routes stay as they are. A structural guard
holds every lab-table reader under the model-input paths to the gate or to
a listed reason its rows never reach a model.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck ba22932101 fix(documents): read a JPEG's size past fill bytes and refuse an unreadable one
The thumbnail's header walk took the byte after any 0xFF as a marker. A
single 0xFF fill byte after the start-of-image marker, which the JPEG
standard allows and every decoder skips, threw it off; it read no size, and
an unread size let the image through to a decode that allocates the full
bitmap. A small JPEG declaring 20000 x 20000 pixels allocated 1.6 GB in the
web process before the post-decode check could see it.

The avatar upload already had a walker that skips fill bytes. Both now use
one reader (lib/image-dimensions) that follows the marker rules: fill bytes,
the length-less markers, and no size when a scan or the end of the image
comes before a frame header. The thumbnail treats an unreadable size as a
refusal (size-unreadable) for every format it decodes, and the backfill
counts that refusal as final.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck 518ce2fd37 fix(documents): read and render PDFs in a bounded child process
pdf.js decodes an image at the size the file declares, whatever size the
page is drawn at, and the server opened documents without an image limit.
A small upload declaring a very large image made the web process allocate
gigabytes until the container was killed, for every user at once, and the
thumbnail job ran the same document again after each restart. An image
whose soft mask declares the size does the same past pdf.js' own limit, and
nothing bounded how long a text read could take.

pdf.js is now opened with maxImageSize (20 million pixels, which keeps one
image inside the reader's ceiling and admits a 300 dpi A3 scan), and every
read and render runs in a child process, one at a time: its V8 heap is
capped, its resident memory is sampled and the child killed past 640 MB, on
Linux it volunteers to the out-of-memory killer first, a time limit ends a
read that runs on, and it starts with an almost empty environment. The
loader and the rendering moved into pdf-engine.mjs, which the child runs
with plain Node and which is traced into the standalone image.

A document that crosses a bound answers resource-limit. The thumbnail job
records it as a final failure, so the backfill does not queue it again.
2026-10-25 13:52:04 +01:00
Marc-André Bombeck a69666fa16 feat(account): hold account deletion and data wipe for a grace window
Deleting an account, a guardian deleting a managed profile, and "Delete
all data" were final the moment they were confirmed: the account row
cascaded into every table and every stored copy, and the off-host objects
were purged in the same breath. Nothing short of rolling the whole
instance back could return one account.

ACCOUNT_DELETION_GRACE_DAYS (14 by default, at most 30, 0 erases at once
as before) now sets a window:

- Account and managed-profile deletion lock the account (migration 0488:
  deletion_requested_at, deletion_purge_at, deletion_requested_by_id).
  Every session, refresh token, API token, connector, share link and
  step-up elevation is revoked in the same transaction, and the record,
  its stored copies and its off-host objects are left as they are.
- Sign-in is refused with 403 auth.account.deletion_pending and the
  erasure date, after the credential was proved and before a second
  factor is asked for. The session and Bearer resolvers and the acting-
  account resolver refuse a pending account too, so a credential minted
  in the moment before the lock is useless.
- POST /api/admin/users/{id}/undo-deletion (cookie-only, audit-logged)
  clears the lock. The admin users list reports the pending state.
- The hourly account-deletion-purge job erases accounts whose window has
  ended, exactly as the immediate deletion did, with the off-host purge
  request committed alongside. The Guardian invariant is decided again;
  a refused erasure stays pending and is reported.
- A wipe first stores the whole record as a PRE_WIPE_<ms> copy through
  the writer the weekly copy uses, keeps it out of the wipe, and holds
  its off-host purge back with a new not_before on the request. The
  ordinary restore brings the record back; restoring that copy withdraws
  the held purge. The purge job deletes the copy when its window ends.

Background work skips accounts pending deletion through one shared
predicate (src/lib/account-deletion/active-account.ts): every reminder,
digest, Coach, sync, backfill, rollup, score and backup enumeration
filters with it, and account-deletion-enumeration-guard.test.ts fails on
a query in background work that neither filters nor says why it need
not. Notification delivery and the AI capability loader refuse a pending
account as chokepoints behind it, and the Withings, WHOOP and Telegram
webhooks and the notification quick actions ignore one.

The privacy summary publishes the window. Two pieces of the reminder tick
and the off-host pass moved into modules of their own to keep the files
inside their size ratchet.
2026-10-25 13:50:14 +01:00
Marc-André Bombeck e825a4c4d3 feat(admin): say how long dumps keep deleted accounts, and check the bucket rule for database/
A database dump is the whole database at one moment, so an account deleted
since stays in the dumps taken before until they age out. The database row
on the backups card now says so with the numbers that decide it: how many
nightly dumps this server keeps, and after how many days the bucket's
lifecycle rule removes the off-host copies.

The lifecycle probe used to answer with the shortest expiry of any enabled
rule, whatever its prefix, so a rule scoped to the account copies read as
covering the dumps too. It now also answers for `database/` alone, counting
only rules whose filter takes in a dump's key (prefix, legacy prefix, an
And filter; a tag filter never matches untagged objects), and the card
warns when no rule covers that prefix: dumps there, and the deleted accounts
in them, would never be removed. The reading moved into its own module.
2026-10-25 13:43:23 +01:00
Marc-André Bombeck 7dab9cc5e9 feat(admin): judge every backup signal and push changes to admins
Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.

One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:

- database dump: failing once the newest good dump is older than 36 h
  while dumps are on, or when it can be kept nowhere; read through a
  provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
  failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
  window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
  7 days after setup or as soon as a confirmed kit is out of date.

A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.

The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).
2026-10-25 13:32:43 +01:00
Marc-André Bombeck d3c672cc3f feat(admin): replace the key backup step with a recovery kit
The step confirmed one key, the active application key. A restore needs
more: off-host copies are sealed under BACKUP_ENCRYPTION_KEY, a copy
written before a rotation needs the retired application key it was
written under, and every session, token and share link hangs off
API_TOKEN_HMAC_KEY. An operator could do everything the step asked and
still be unable to read a single backup.

The recovery kit lists every key a restore of this server needs, by id
and fingerprint, never key material:

- each id in ENCRYPTION_KEYS, plus any retired id a stored copy
  (innerKeyIds) or an off-host object inside the bucket's retention
  (OffhostBackupKeyUse) still lists, also once the server dropped it;
- BACKUP_ENCRYPTION_KEY and each BACKUP_ENCRYPTION_PREVIOUS_KEYS entry,
  named by the id their objects record;
- API_TOKEN_HMAC_KEY.

Confirming records the set the admin was shown and refuses a stale one
with 409 recovery.kit.stale. The kit is due again as soon as a key it
lists is not in that set (a rotation, a new backup key, a re-keyed
install); a key that merely drops out does not make it due. "Check my
copy" now checks any key in the kit.

Migration 0487 carries a confirmation given under the old step forward
as a kit holding that key, so that part stays confirmed, and drops the
four app_settings columns. It also creates the two tables the readiness
evaluation uses (restore_drill_outcomes, recovery_readiness_notices).

The routes move to /api/admin/recovery-kit; the reader of the keys the
backups still need moves out of the encryption status route into
src/lib/recovery/backup-key-needs.ts so the kit and the status view
share one definition.
2026-10-25 13:32:29 +01:00
Marc-André Bombeck 6d3a1582fa feat(backup): take a nightly whole-database dump with a manifest
The per-account copies roll one account back on the same server; nothing
the app made could rebuild a lost one. A scheduled job now runs pg_dump
over the whole database (every schema, pg-boss included) into
DB_BACKUP_DIR as healthlog-<UTC time>-daily.dump, with a manifest beside
it: app version and build, newest migration, Postgres version, size and
SHA-256, exact row counts of every table, and the fingerprints of the
keys the database needs.

The counts read the same exported snapshot pg_dump reads, so a dump
taken while the app writes still matches its manifest. The archive is
written as .partial, listed back with pg_restore --list, and renamed
after its manifest; a failed run leaves nothing named like a good dump.
Retention removes only files this job wrote, oldest first, and never
the newest good one (DB_BACKUP_KEEP, default 7; two pre-migration dumps).

With the off-host bucket configured, each dump and its manifest go up
under database/ in the envelope the account copies use, streamed. Every
run lands in the new database_backup_runs ledger (migration 0486), and
getDatabaseBackupHealth() answers the last good dump, the last failure,
whether the directory is writable and whether off-host is configured.
The boot readiness summary says where dumps go, or that none can be kept.

The dump, naming, retention and restore rules live in one plain ESM
module so the image's own scripts can share them without tsx.
2026-10-25 12:56:33 +01:00
Marc-André Bombeck 41c6eea2a1 fix(backup): stream every large section of an uploaded backup
The admin upload streamed only the measurements while it checked a file,
so a document-heavy account copy was held whole in memory and an upload of
a few hundred megabytes could take the process down.

The sections a reader takes an element at a time are now named once
(BACKUP_STREAMED_SECTIONS in streamed-backup.ts): measurements, intake
events, mood entries, documents and ECG recordings. The upload reads its
file through readStreamedBackup like the restore and the preview, and the
restore drill and the restore script use the same key set.

A behavioural test pushes a 271 MB document-heavy file through the real
upload route and fails when the heap grows with the file. The stream guard
now asserts that every reader goes through the shared reader or key set
and that no file declares its own.
2026-10-25 12:25:15 +01:00
Marc-André Bombeck 240a135b88 refactor(ai): keep one name for one completion's ceiling
A status run is one completion, so its ceiling was the completion ceiling
under a second name. The Local client also re-exported a JSON-mode helper
nobody imported from it.
2026-10-25 04:50:25 +01:00
Marc-André Bombeck d6c2760415 Merge remote-tracking branch 'origin/release/v1.46.0' into fix/review-meds-docs
# Conflicts:
#	src/__tests__/baselines/source-function-lines.json
#	src/__tests__/baselines/version-stamp-comments.json
#	src/__tests__/sharing-surface-guard.test.ts
2026-10-25 02:45:23 +01:00
Marc-André Bombeck 988af10a34 fix(documents): keep module-off accounts from filling the index retry boot
The boot discovery for the documents 0479 marked took the first thousand
by id and skipped those whose account has the documents module off.
Such an account keeps its marks, so with enough of them it held every
place of every boot and the accounts after it never got their retry.

The marks are walked by id past the last row read, an account found
with the module off is left out of the query from then on, and the
boot stops at a thousand documents enqueued rather than read.

Refs #1277
2026-10-25 02:14:17 +01:00
Marc-André Bombeck ea3bdf42b7 fix(documents): settle an index backfill read once and ask the wire before each
The index-all backfill billed the read and wrote the index in one try:
an index write that failed after the read was settled again in the
catch, handing the reservation back twice. The read is settled once,
and a failed write charges nothing back.

The backfill also checked consent and the operator's switches once, at
the pick, then read up to two hundred documents. The wire is asked
again before every read; a refusal hands that document's reservation
back and ends the walk with no-consent or unavailable.
2026-10-25 02:10:17 +01:00
Marc-André Bombeck cc284c4307 fix(jobs): log a failed preload report instead of swallowing it
The sweep keeps going when reporting a load failure itself throws, and
now says so on stderr; the error-level guard refuses a silent catch in
src/lib/jobs. The preload test reads the worker source through
readSource with anchors.
2026-10-25 00:51:32 +02:00
Marc-André Bombeck c6bd27f06f fix(jobs): load every job handler once at worker boot and report failures
Since handlers load on their queue's first job, a module the image
cannot load no longer fails the worker's boot. The worker comes up
healthy, the queue burns its retries when it first has work, and a
one-shot job (a restore, an import tail) ends failed with nobody told at
deploy time.

After binding every queue the worker now runs `preloadHandlers` in the
background, without awaiting it, so readiness is unchanged. It loads
each handler once through the same memoised loader its queue is bound
to, so a module that loads stays loaded for the first job. Each failure
is reported on the `job.run.failed` line at error, with the queue's name
and `reason: "handler_load_failed"`, plus the worker error report
(stderr and GlitchTip). A summary line on stderr names every queue that
failed. The sweep never throws, and a failing report does not stop it.

docs/ops/deploy.md says what to look for after a deploy.
2026-10-25 00:35:36 +02:00
Marc-André Bombeck a35ff5f5ea Merge remote-tracking branch 'origin/release/v1.46.0' into fix/end-medication-contributor-base
# Conflicts:
#	src/__tests__/baselines/source-file-lines.json
#	src/__tests__/baselines/source-function-lines.json
2026-10-24 22:39:30 +02:00
Marc-André Bombeck 23a67f61e6 feat(medications): let a course end at an instant
A course's end date is a calendar day and counts in full: the doses of that
day stay due, reminded and expected until local midnight. That is right for
an end date set in the editor and wrong for "End course", whose dialog
promises that nothing more is due.

A course now carries `endedAt` (migration 0482, additive, nullable), the
instant it was ended on its `endsOn` day, projected onto the medication next
to `startsOn` / `endsOn` by the one course writer. From that instant:

- the schedule engine emits no slot, so a dose later that day is not due,
  not reminded, not projected onto the today list and never expected by
  compliance, while a dose earlier that day stays inside the course;
- the course reads ENDED (`courseStatusAt` / `courseStatusOn`), so the dose
  actions, `remindersActive` and the reminder gate stop at once, follow-ups
  of an earlier dose included;
- every engine context builder, compliance context, dose history, Coach
  snapshot, doctor report and the adherence storyline read it, through one
  select and one picker for the medication window.

Any write that moves a course's `endsOn` clears the instant, so an end date
chosen in the editor keeps meaning the whole day. Backups carry it in both
purposes and a restore projects it again. `courseStatusAt` moves next to the
other course rules.

Refs #1280
2026-10-24 22:20:06 +02:00
Marc-André Bombeck 3f17b40092 Merge remote-tracking branch 'origin/release/v1.46.0' into fix/answerless-retry-bound
# Conflicts:
#	src/__tests__/baselines/source-file-lines.json
#	src/__tests__/baselines/source-function-lines.json
2026-10-24 21:31:24 +02:00
Marc-André Bombeck f6342ebca3 test(jobs): simulate an hour of background work against a model that never answers
Drives the status-card path the way an idle instance runs it: a data sync
every minute dirtying five cards, the production enqueue with its queue
name, singleton key and retry options, the queue's consumer, the card's
single call and fallback, the status chokepoint, the chain walker with the
health ledger's in-memory twin, the gateway client and the note store's
negative windows. Only the wire, pg-boss, the clock and the unrelated
Postgres reads are faked; the wire answers every request with reasoning
and finish_reason "length".

Against the code before this series the hour counted 151 requests, 150 of
them sent with "none", none charged to the day's ledger, and 300 when the
reasoning lesson did not survive from one job to the next (the pattern
behind the reported log). Now it counts six, charges all of them, sends
"none" once, and logs the pause once per window.

Refs #1279
2026-10-24 21:27:58 +02:00
Marc-André Bombeck ed9261264a test(jobs): hold every model-bound queue to bounded, backed-off retries
pg-boss gives a send that names no retry options the queue's defaults:
two retries, at once, without a backoff. A job that reached a model and
failed then ran again immediately, twice, against the provider that had
just failed it. Two model-bound queues still relied on those defaults:
the Coach memory refresh send and the nudge cron. Both now ask for the
same two retries, a minute apart and backed off.

ai-job-retry-bound-guard holds the rule: every send to a queue whose jobs
reach a model names retryLimit (at most 3) and, unless it is 0,
retryBackoff with a retryDelay of at least 30 s; every such cron carries
the same in its schedule options. The queues come from the model-run
sites the expiry guard already keeps complete (now a shared helper), plus
the job files that call a provider directly, which the new guard keeps
complete itself. A planted case runs the checker over sources shaped like
the tree before this change. The queue golden records the nudge cron's
new options.

Refs #1279
2026-10-24 21:22:08 +02:00
Marc-André Bombeck 86c2cb3360 fix(documents): stop the index-all walk when the provider keeps failing
The content-index backfill capped a run at 200 documents, but counted
only the ones it indexed. Against a provider that failed every call (a
model that reasons and never answers) the cap was never reached, so the
walk transcribed every un-indexed document in turn, one provider call
each, until the job's time ran out after more than two hours, and the
refunded failures never reached the day's token ceiling.

The cap now counts provider calls, the walk stops with reason
provider-failing after three failed calls in a row (one document the
model cannot read is still walked past), and a failed read charges what
its reply reported spending. The per-document read moves into its own
function.

Refs #1279
2026-10-24 21:22:01 +02:00
Marc-André Bombeck 4752115d7d feat(documents): add Recently deleted for deleted documents
Deleting a document only stamps a tombstone and the purge removes it
after 30 days, but the only way back was the undo toast, gone a few
seconds after the delete. Recently deleted lists the deleted documents
of the record with type, filing date, where an import came from, when
each was deleted and how long it has left, and offers Restore, Restore
all and Delete now.

GET /api/documents/inbound/deleted lists them (owner-only, capped at 200
with the list-cap meta). POST /api/documents/inbound/{id}/purge removes
one deleted document for good through the same function the nightly
purge runs, so its import keys and aliases still move into the ledger
and an importer re-sending it is still answered "deleted"; a live
document answers 409 documents.inbound.notDeleted. The usage read gains
deletedCount, 0 under a grant. The restore route and bulk restore
share one restore, which now also clears ledger entries for the
document's own keys.

The vault header shows a quiet "Recently deleted (n)" link only while
something is in it, and the delete toasts and confirmations say that
deleted documents stay there for 30 days. The purge's queue facts and
grace window move to document-purge-queue.ts so the worker registry
still loads none of the work.

Refs #1275
2026-10-24 20:51:35 +02:00
Marc-André Bombeck 1ea14e9100 fix(documents): index once more the PDFs a failed local read left out
While two pdf.js versions broke each other, a local text read after a
page render failed and was recorded as decrypt-error, and a scanned PDF
that had to be rasterized for the provider was recorded as raster-failed.
Neither has a content index, and on an instance without an AI provider
nothing offers to index it again.

Migration 0479 clears the attempt time of exactly those PDFs and keeps
their reason, a state no index attempt leaves behind. A new boot
discovery, document-index-retry, enqueues the upload's auto-index job for
each such row, so it runs the same provider-first, local-fallback decision
under the same consent rules; a deferred import stays local because its
row says so. The attempt writes its time back, so every document drops out
after one try, and one that really cannot be read records its reason again
and stays settled. An account with the documents module off keeps its
marks until the module is on.

An integration test seeds both kinds of damage, runs each migration's
statement and only what a boot runs on its own, and sees a preview and an
index appear, while a blank page, a broken ciphertext and a file that is
not a PDF settle after one retry and are not found by the next boot.

Refs #1277
2026-10-24 19:50:20 +02:00
Marc-André Bombeck a1ee2ac003 fix(documents): keep document reads inside their job's expiry
Since the lab staging reads a long report in parts, one auto-index could
make the transcription plus up to twenty reads of two calls each at the
person's response timeout, hours past pg-boss's fifteen-minute default,
which failed the job and retried it beside the reads still running. The
staging now takes the run's deadline as a schedule and plans only the
reads that fit before it. The index job is long, with an expiry above the
calls that run before the plan can say no, a stop on jobDeadline and a
lock per document; the document run hands the same staging its own
deadline, which moves the run's expiry as the reads are planned. The
document summary, one call and its reasoning repeat, gets an expiry above
that worst case.

ocrReadCallMs in the read core is the one per-call timeout both deadlines
divide by. The expiry guard now names every importer of the document and
lab read entry points with the queues that reach it.

The runtime guard's planted cases stripped every file's comments again on
each rescan, which took over a second alone and timed out under a full
parallel run; the stripped text is now memoised.

Refs #1274
2026-10-24 11:28:57 +02:00
Marc-André Bombeck 96eeec6080 Merge origin/release/v1.46.0 into feat/v146-ai 2026-10-24 11:03:44 +02:00
Marc-André Bombeck 02d3d8b7bb fix(insights): let slow-model status notes and AI jobs outlast their calls
A status run (a metric card, the batched assessment, a derived score, a
workout line, a Coach memory pass) is one model call plus the one repeat
the Local and gateway clients send when a thinking model answered with
reasoning alone, each with the person's response timeout. The run's outer
bound was a single call, so the repeat was cut off, and the open card
stopped polling after 48 s, so a note from a model that needs minutes
never reached the page. The bound is now the per-call timeout times
STATUS_MODEL_CALLS, and the card's poll ceiling follows the published
response timeout (fast for the first 48 s, every 15 s after), only while
the route reports the note in flight; the fallback text stays meanwhile.
The status queue is exclusive, so the longer poll cannot queue a second
generation beside a running one.

The morning refresh is one comprehensive generation of up to five calls
in a row, with no outer bound and pg-boss's fifteen-minute expiry, so a
large setting had pg-boss fail and retry it while it was still running.
It now runs behind the same bound as the warms, and the job carries an
expiry derived from that worst case. The per-metric note, the workout line
and the Coach memory refresh get expiries derived the same way.

queue-runtime.ts gains a bounded class for these jobs: model time read off
model-time.ts, an expiry above it, carried on every send. A new guard
names every caller of runStatusCompletion and generateComprehensiveInsight
with the queues that reach it, and holds each such queue to bounded above
its runs or long under a run lock.

Refs #1274
2026-10-24 11:00:32 +02:00
Marc-André Bombeck 5a7bd95c7b fix(documents): read a long document through the lab scan's read core
Extracting lab values from a stored document still sent the whole text
in one call. A multi-page report produced more JSON than one call's
output ceiling, the answer was cut mid-fact, the parse failed twice, and
the read came back as 422 "Couldn't read the document. Try a clearer
copy." The lab scan had already learned to read the same report a page
or a part at a time, but each surface carried its own copy of the read,
so the document path never got it. It also ran under the shared 60 s
timeout, dropped every fact past 120 (or every fact when one did not
match the schema), and charged a flat reservation.

The read now lives once, in lib/documents/ocr-read.ts and ocr-plan.ts:
the per-read limits, the corrective retry, the salvage of a cut
answer's complete rows, the wire re-check and budget reservation before
each read, and the planning of a PDF a page at a time and text in parts
at line boundaries. The lab scan and the document extraction (the
extract route, its background run and the lab auto-staging) both read
through it, and the background run plans its reads on the same deadline
as a lab scan. The per-read budget moves to a shared metered budget.

The document path validates the answer fact by fact, keeps the lab
scan's row cap (400), and says why nothing came back: 422
documents.inbound.outputLimit when the reads stopped at the output
limit, 429 when the budget refused them, extractFailed only for answers
that were not the schema. A run whose reads were cut or not reached
reports `incomplete` on its result.

A guard holds the class: the output ceiling is set only in the core,
both surfaces read through runOcrReads, both runs share the deadline,
and a lab read and a document read go out under the same limits.

Refs #1257
2026-10-24 10:34:31 +02:00
Marc-André Bombeck 552510c654 fix(insights): keep polling for the briefing while the warm can land
The briefing read polls while the warm it enqueued runs, and it stopped
after ten polls (225 s) whatever the response timeout. A briefing that
landed later never reached the open page, which kept the empty card, or
the previous attempt's "timed out" hint, on screen until a reload.

The read's poll ceiling now follows the published response timeout over
the generation's sequential calls, never less per call than the
briefing's 180 s default, plus one interval for the queue to pick the
warm up. A settling regenerate and the overview body keep their ten
polls. A test walks the poll schedule against the warm's own bound for
every setting, and the client-abort guard holds the read to the derived
ceiling.

Refs #1274
2026-10-24 09:23:47 +02:00
Marc-André Bombeck 73e98c6ef5 fix(insights): give the briefing warm every call of its generation
The out-of-band warm the Insights page relies on bounded one
comprehensive generation at the account's response timeout plus 30 s.
A generation is up to five model calls in a row: the generation, the
JSON retry, the grounding retry, the outbound-screen repair, and the one
repeat a thinking model gets when it answered with reasoning alone. Each
call honoured the setting, but a local model needing 300 s per answer
under a 600 s setting met the 630 s bound in the middle of its first
corrective retry, and the briefing was dropped.

The bound is now the per-call timeout times COMPREHENSIVE_MODEL_CALLS
plus the headroom, for the nightly pass and the forced warm alike. A
test drives the real warm, generator, chain runner, Local client,
safeFetch and pinned dispatcher against a loopback server that answers
after 300 s; under the old bound it reports a timeout on the third
answer.

The response-timeout guard now names every outer bound around model
work and holds both briefing generators' stages to the call counts the
outer waits multiply by.

Refs #1274
2026-10-24 09:23:20 +02:00
Marc-André Bombeck d1545cf554 refactor(jobs): keep the lane's hold grace private to the lane 2026-10-23 22:13:00 +02:00
Marc-André Bombeck 111047491e fix(medications): end every medication and intake read on the row id
Postgres returns rows that tie on an ORDER BY in no particular order,
and not the same one twice. The medication reads ordered by columns two
rows share all the time: every read that hands schedules on included
them with no order at all, medications were listed by createdAt alone
(one import writes many in a millisecond), intakes by scheduledFor (a
twice-daily medication puts every morning dose on one instant), and
"the latest intake" lookups by takenAt. A client read the same account
in a different order from one request to the next, and a latest-intake
lookup picked either of two rows.

Every findMany / findFirst on a medication table, nested includes
included, now ends its order on the id in the direction of its last key,
and schedules are read by window, then id (MEDICATION_SCHEDULE_ORDER, the
order the backup already wrote). medication-read-order-guard.test.ts
holds the tree to it. medication-read-order.test.ts writes tying rows in
the opposite order to the one each read promises, through the medication
list and today's intakes, and fails without the tie-breakers.

The parity suite's own state reader had the same gap: it sorts rows by
content with the ids blanked, and rows that tie then kept whatever order
the unordered read returned, so their <id:N> labels swapped under a
loaded run. Its reads are ordered by id, which for a cuid is the order
the rows were written in.

The JSON import's schema left the route, so its entry in the inline
schema strictness register goes; the schema module says why its objects
are not strict.
2026-10-23 21:54:25 +02:00