A full rollup backfill that failed on every boot showed only on the
admin page's card of failing background jobs. From the account's side
the only symptom was empty dashboard cards, and nothing pointed the
operator at the card (#1298).
A new hourly evaluation reads the newest terminal runs of every queue
whose failure has no account-facing surface (the system audience in
subsystem-surface.ts) from pg-boss's own ledger. A failed run adds its
attempts, retries included, and the first successful run ends the
streak. Three failed attempts in a row send every admin a SYSTEM_ALERT
once, a reminder at most once a day while the queue keeps failing, and
one notice when it runs successfully again: the decision the backup
health notices use, with the same compare-and-set claim, so two workers
never send a notice twice.
The memory is one row per announced queue in system_job_alert_notices
(migration 0490). The notice names the queue, the streak and the last
error, and opens the admin status page.
On an account with tens of millions of readings the boot full backfill
never finished. It ran four aggregates over the whole five-year window,
each held to the connection's 60 s statement limit, so every attempt
was cancelled with 57014, the reading extremes added by 0417 stayed
NULL, and the readers that need them fell back to the raw table and
timed out as well (#1298).
The full backfill now folds the window one UTC calendar year per pass,
newest first, and every statement runs under its own five-minute limit,
the shape the pulse day-mean rebuild already uses. Calendar-year cuts
write exactly the buckets one pass writes: day, month and year buckets
fall inside one window and a week across New Year is folded whole by
both. Each window commits on its own, so a run that reaches its job
budget hands the remaining years to a follow-up job instead of failing
and starting over on the retry.
The boot discovery read every live reading in the window to find
uncovered days, the same scan that kept the fold from finishing. It
now walks the days per account and type off the live index, so its
cost follows the number of days with readings, and runs under the same
limit. The full backfill code moves to its own module.
Before the confirm step dated an undated value by its document, a lab
value confirmed from a report that prints its collection date only in the
header was stored with takenAt set to the moment of confirmation. The fix
covers new confirms; the values already filed that way stay on the wrong
day until something moves them.
A one-shot maintenance job, sent at boot only while there is something to
repair, moves exactly those values to the document's report date (else its
filing date) at noon UTC, the same rule the confirm step now uses. A value
qualifies only when the chain is certain: written by the document confirm
path, named by exactly one approved observation fact, takenAt still within
five seconds of its own createdAt, no calendar date of its own in the
decrypted fact, and a document day that differs from the day the value is
filed under in the owner's zone. A value re-dated since, one really taken
on its confirmation day, or one whose document has no date is left alone.
A moved value no longer sits at its creation moment, so the pass converges
and a second run changes nothing.
The job definition lives in a new registry file beside maintenance.ts so
that file stays under the size ratchet.
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
Deleting an account, a guardian deleting a managed profile, and "Delete
all data" were final the moment they were confirmed: the account row
cascaded into every table and every stored copy, and the off-host objects
were purged in the same breath. Nothing short of rolling the whole
instance back could return one account.
ACCOUNT_DELETION_GRACE_DAYS (14 by default, at most 30, 0 erases at once
as before) now sets a window:
- Account and managed-profile deletion lock the account (migration 0488:
deletion_requested_at, deletion_purge_at, deletion_requested_by_id).
Every session, refresh token, API token, connector, share link and
step-up elevation is revoked in the same transaction, and the record,
its stored copies and its off-host objects are left as they are.
- Sign-in is refused with 403 auth.account.deletion_pending and the
erasure date, after the credential was proved and before a second
factor is asked for. The session and Bearer resolvers and the acting-
account resolver refuse a pending account too, so a credential minted
in the moment before the lock is useless.
- POST /api/admin/users/{id}/undo-deletion (cookie-only, audit-logged)
clears the lock. The admin users list reports the pending state.
- The hourly account-deletion-purge job erases accounts whose window has
ended, exactly as the immediate deletion did, with the off-host purge
request committed alongside. The Guardian invariant is decided again;
a refused erasure stays pending and is reported.
- A wipe first stores the whole record as a PRE_WIPE_<ms> copy through
the writer the weekly copy uses, keeps it out of the wipe, and holds
its off-host purge back with a new not_before on the request. The
ordinary restore brings the record back; restoring that copy withdraws
the held purge. The purge job deletes the copy when its window ends.
Background work skips accounts pending deletion through one shared
predicate (src/lib/account-deletion/active-account.ts): every reminder,
digest, Coach, sync, backfill, rollup, score and backup enumeration
filters with it, and account-deletion-enumeration-guard.test.ts fails on
a query in background work that neither filters nor says why it need
not. Notification delivery and the AI capability loader refuse a pending
account as chokepoints behind it, and the Withings, WHOOP and Telegram
webhooks and the notification quick actions ignore one.
The privacy summary publishes the window. Two pieces of the reminder tick
and the off-host pass moved into modules of their own to keep the files
inside their size ratchet.
Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.
One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:
- database dump: failing once the newest good dump is older than 36 h
while dumps are on, or when it can be kept nowhere; read through a
provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
7 days after setup or as soon as a confirmed kit is out of date.
A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.
The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).
The per-account copies roll one account back on the same server; nothing
the app made could rebuild a lost one. A scheduled job now runs pg_dump
over the whole database (every schema, pg-boss included) into
DB_BACKUP_DIR as healthlog-<UTC time>-daily.dump, with a manifest beside
it: app version and build, newest migration, Postgres version, size and
SHA-256, exact row counts of every table, and the fingerprints of the
keys the database needs.
The counts read the same exported snapshot pg_dump reads, so a dump
taken while the app writes still matches its manifest. The archive is
written as .partial, listed back with pg_restore --list, and renamed
after its manifest; a failed run leaves nothing named like a good dump.
Retention removes only files this job wrote, oldest first, and never
the newest good one (DB_BACKUP_KEEP, default 7; two pre-migration dumps).
With the off-host bucket configured, each dump and its manifest go up
under database/ in the envelope the account copies use, streamed. Every
run lands in the new database_backup_runs ledger (migration 0486), and
getDatabaseBackupHealth() answers the last good dump, the last failure,
whether the directory is writable and whether off-host is configured.
The boot readiness summary says where dumps go, or that none can be kept.
The dump, naming, retention and restore rules live in one plain ESM
module so the image's own scripts can share them without tsx.
A status run is one completion, so its ceiling was the completion ceiling
under a second name. The Local client also re-exported a JSON-mode helper
nobody imported from it.
Since the lab staging reads a long report in parts, one auto-index could
make the transcription plus up to twenty reads of two calls each at the
person's response timeout, hours past pg-boss's fifteen-minute default,
which failed the job and retried it beside the reads still running. The
staging now takes the run's deadline as a schedule and plans only the
reads that fit before it. The index job is long, with an expiry above the
calls that run before the plan can say no, a stop on jobDeadline and a
lock per document; the document run hands the same staging its own
deadline, which moves the run's expiry as the reads are planned. The
document summary, one call and its reasoning repeat, gets an expiry above
that worst case.
ocrReadCallMs in the read core is the one per-call timeout both deadlines
divide by. The expiry guard now names every importer of the document and
lab read entry points with the queues that reach it.
The runtime guard's planted cases stripped every file's comments again on
each rescan, which took over a second alone and timed out under a full
parallel run; the stripped text is now memoised.
Refs #1274
A status run (a metric card, the batched assessment, a derived score, a
workout line, a Coach memory pass) is one model call plus the one repeat
the Local and gateway clients send when a thinking model answered with
reasoning alone, each with the person's response timeout. The run's outer
bound was a single call, so the repeat was cut off, and the open card
stopped polling after 48 s, so a note from a model that needs minutes
never reached the page. The bound is now the per-call timeout times
STATUS_MODEL_CALLS, and the card's poll ceiling follows the published
response timeout (fast for the first 48 s, every 15 s after), only while
the route reports the note in flight; the fallback text stays meanwhile.
The status queue is exclusive, so the longer poll cannot queue a second
generation beside a running one.
The morning refresh is one comprehensive generation of up to five calls
in a row, with no outer bound and pg-boss's fifteen-minute expiry, so a
large setting had pg-boss fail and retry it while it was still running.
It now runs behind the same bound as the warms, and the job carries an
expiry derived from that worst case. The per-metric note, the workout line
and the Coach memory refresh get expiries derived the same way.
queue-runtime.ts gains a bounded class for these jobs: model time read off
model-time.ts, an expiry above it, carried on every send. A new guard
names every caller of runStatusCompletion and generateComprehensiveInsight
with the queues that reach it, and holds each such queue to bounded above
its runs or long under a run lock.
Refs #1274
The out-of-band warm the Insights page relies on bounded one
comprehensive generation at the account's response timeout plus 30 s.
A generation is up to five model calls in a row: the generation, the
JSON retry, the grounding retry, the outbound-screen repair, and the one
repeat a thinking model gets when it answered with reasoning alone. Each
call honoured the setting, but a local model needing 300 s per answer
under a 600 s setting met the 630 s bound in the middle of its first
corrective retry, and the briefing was dropped.
The bound is now the per-call timeout times COMPREHENSIVE_MODEL_CALLS
plus the headroom, for the nightly pass and the forced warm alike. A
test drives the real warm, generator, chain runner, Local client,
safeFetch and pinned dispatcher against a loopback server that answers
after 300 s; under the old bound it reports a timeout on the third
answer.
The response-timeout guard now names every outer bound around model
work and holds both briefing generators' stages to the call counts the
outer waits multiply by.
Refs #1274
The five hand-written registrars (about 5,500 lines, one 997-line function)
are replaced by `jobs/registry/`: one `defineJob` entry per queue with its
cron, schedule send options, policy, work options, lock, lane and a handler
loader. `registerJobGroup` creates, schedules, retires and binds each group
in the order the registrars used; the worker boot walks the groups and runs
their boot discovery.
Handlers load on their queue's first job. The facts a declaration needs at
boot (names, crons, concurrency, send options) moved out of the heavy job
modules into light `*-queue.ts` siblings, and their importers point there,
so loading the registry evaluates 288 modules instead of the 1,164 the
registrars pulled in. The formerly inline handlers moved verbatim into
handler modules under `jobs/reminder/`.
Fitbit's queues live in `fitbit/jobs/definitions.ts` and enter the registry
through one spread, so removing the integration later is that folder and
that line.
`tests/integration/queue-registration-parity.test.ts` proves pg-boss holds
exactly what the registrars left (queues, schedules, bindings, byte for
byte) and loads every handler once. The queue tests and the runtime guard
assert on the registry objects instead of reading registrar source.
Every Oura collection, workouts and heart rate included, is now read
back to the start of the account's history by a backfill that runs
through the shared integration lane in five-minute slices. Each
collection walks from the day the backfill began toward the past one
window at a time (ninety days, thirty for sleep, seven for heart rate)
and the progress note on the user row records after every window where
it got to, so a slice cut short by its budget, the lane or a rate limit
resumes there. A 429 ends the slice and the next one waits for the
Retry-After Oura named. Activity and sleep are walked first to learn
where history starts; the rest stop a week before it.
Heart rate streams a page at a time and keeps only the days it touched,
so memory stays flat however long the history is. A collection the
grant does not cover is a scope note, not a failure; the first slice of
the next chain asks again, and a connect clears the completion stamp so
a newly granted scope is read without walking anything twice.
The two columns arrive empty, so every existing connection is offered
the backfill at the next boot without reconnecting. Before anything
else the backfill retires the WALKING_RUNNING_DISTANCE rows earlier
syncs wrote from the equivalent walking distance and refolds their days;
an account that kept such rows after disconnecting is offered too.
The keyed refold's day filter and delete compared the naive UTC
measured_at and bucket_start with a timestamptz[] parameter, which goes
through the session time zone; in any zone but UTC neither matched and
the refold left its buckets stale. Both now bind timestamp[] literals.
Keys are refolded in (type, day) order in chunks of 500, and the
rollup-recompute queue describes the runs of buckets a job now covers.
Sessions, refresh tokens, access tokens, trusted devices and the
second-factor and passkey challenges were never removed after they
expired, so the sign-in tables were mostly dead rows.
auth-ledger-cleanup deletes a session, token or trusted device 30 days
after it expired, which keeps every row the login throttle's 30-day
known-address check and refresh reuse detection can still read; a revoked
token without an expiry stays listed. Challenges go a day after expiry.
Security activity reads the audit log and loses nothing.
Twenty of the twenty-three soft-delete models were never purged: a deleted
lab result, visit, allergy or cycle stayed in the database, in every backup
and in every key rotation for as long as the instance ran.
soft-delete-retention.ts holds one entry per model that has a deletedAt
column, derived from the schema by a guard with a counter-check. The sync
domains and the document vault keep their purges; every other model is
purged by a new nightly soft-delete-purge job after the sync feed's
75-day horizon, per account under the restore lock and in bounded batches.
A cycle start folded into a live cycle stays while that cycle does, and a
removed coach fact stays for the coach message retention, since stored
trails are withheld against it.
medication_compliance_rollups lost its last reader when compliance moved
to the schedule-anchored computation, yet every dose write still upserted
it, a boot backfill refolded it, and the wipe and backup plans carried it.
The per-dose hook keeps the one derived row that still hangs off a dose
day, the supplement nutrients, under dose-day-refresh. The pending-row
mints (today projection, reminder) no longer call it, since an untaken
dose adds no nutrients. The import job finishes without its per-day
recompute phase; its progress keeps touchedDays and rollupProcessed on
the wire, inert and documented as deprecated. The backfill queue is
retired on boot and migration 0414 drops the table under a bounded lock
wait.
The fold was declared short on the strength of its 60-second
statements, but four aggregates and four persist transactions of up
to 120 s come to about thirteen minutes at worst under the defaults,
and nothing bounds them once an operator lifts the statement timeout.
Past pg-boss's fifteen-minute default the retry folded the account a
second time. The send now carries an hour's expiry, the fold stops
between granularities on the job's budget, and the binding holds a
per-account lockedPass outside the lane. The runtime guard now requires
every heavy queue to be declared long.
`updated_at` sits in the sync-feed index, so every update that moved it
wrote a new row version on another page and an entry in every index.
Provider re-syncs rewrite many unchanged rows, which grew the table to
several times its live size and sent each unchanged row down the sync
feed again. Migration 0412 adds a row trigger that keeps `updated_at` and
`sync_version` when an update changes nothing else, so such an update is
heap-only and the feed stays silent; it sets `fillfactor = 85` and a 5 %
autovacuum threshold. The maintenance queue gains a nightly check that
vacuums when dead rows pass a tenth of the live ones and warns when the
heap is more than three times what its live rows need, which only a
`VACUUM FULL` in a window gives back. The admin card keeps showing the
operator's runs only.
The 02:00 batch builds every status card and falls each metric it cannot
cover back to that card's own path, so the 02:05 to 02:30 crons only
re-read cards the batch had just written. Their schedules and queues are
removed on an instance that still has them (`retireQueues`), and the
fresh-switch integration test drives the batch loop instead.
Connect, status, places, settings, test, sync and disconnect replace their
stubs. A nightly job (03:20) and the connect backfill read the timeline,
visits and residency endpoints through safeFetch and store one row of daily
values per local day: distances, places visited, minutes away from the home
place and the sealed country with its travel-day verdict. A private instance
needs its exact origin in DAWARICH_PRIVATE_ORIGINS.
The connection joins the shared ledger and the connection health model,
watched for errors only. The day carries a movement block, the model
projection drops the travel country, the two movement channels join the
correlation matrix under the environment module, travel days become trip
suggestions, and the days travel in the environment backup section.
An hourly and a daily pass flag a continuous source that delivered on
most of the previous 14 days and has gone quiet, and open one operator
incident per connection when a connection stays degraded or needs action
for more than a day. The incident is announced once through the existing
admin alert path, adopted rather than repeated when the sync ledger
already paged, and closed with one recovery note. Scales, cuffs and
workout logs are watched for errors only, never for silence.
A background job fetches air quality, pollen and UV for every earlier
local day with a reading, mood entry, dose, symptom or workout: after the
update (boot and nightly discovery), when the home is set and when air
quality is switched on. A day with a weather row gets its air quality; a
day without one gets both, at the covering location period or else at the
home, also before the home was set. Days the source does not reach (before
2013 in Europe, before August 2022 elsewhere) stay empty and are not
counted.
Pending days are grouped by coarse location and cut into ranges of up to
90 days, joining days at most 14 apart, newest first. A run holds itself to
300 of the account's 400 daily calls and half of each instance window,
stores its progress on the user (migration 0391) and sends a follow-up while
it makes progress; a budget stop waits for the next discovery. The
Location & weather card shows the progress.
The air-quality client now asks pollen and dust from 2021 and skips the
request outside Europe before CAMS global starts.
The daily-mean consolidation and the dense hourly fold cut at
`now - threshold` instead of a local day boundary. A day was folded in two
runs and the second run overwrote the stored mean with the mean of the
later part alone: every mean-type day, and the boundary hour of every
heart-rate, HRV and SpO2 day, plus the derived resting figure and the DAY
rollup of such a day. The Health Connect importer split its raw window at
the same unaligned instant.
- Fold up to the start of the account's local day the threshold ends in
(`foldBoundary`), for every consolidation pass and the Health Connect
importer's raw/hourly split.
- A window that already has a live `stats:` row is refolded from every
sample of it, live rows plus the fold's own tombstones; with none left
the stored mean stays.
- `folded_window` refuses a raw sample only when its local day lies before
the aligned boundary and a live `stats:` row of its source covers it,
also within reach of the sample after a timezone change.
- A deletion inside a day the fold had not finished is class B.
- Add a one-time, resumable fold repair per account that recomputes each
hourly and daily mean from its remaining samples and writes only values
that differ, recorded in `measurement_fold_repairs` (migration 0389).
- The compaction-tombstone purge waits for the repair per account, and the
boot sends no purge run before one account is repaired.
- The restore keys live `stats:` ids by source, and writes compaction
tombstones back from a file older than the account's repair, resetting
the repair.
Land every file the v1.42 work shares before any of it is built, so the
parts built side by side never edit the same file.
Schema and migrations 0379-0386: the HEALTH_CONNECT measurement source,
ImportJob.kind, the air-quality columns and the per-user switch on the
environment tables, sealed environment locations with nullable plaintext,
managed-profile handover links, the per-type HealthKit arrival ledger, lab
unit-conversion provenance, and user-defined vaccines. Every new column is
annotated for the column-reader sweep.
HEALTH_CONNECT reaches every exhaustive list: the source labels, the
read enum (not the write allowlist), the source-priority ladder cap, now
the number of sources rather than a fixed eight, and the write-path smoke
test. RMSSD gets its own priority class and a 300 ms ceiling. The batch
syncTrigger gains `manual`, with its label on the Apple Health card.
Queues for the Health Connect import, the compaction-tombstone purge and
the measurement-table maintenance are registered with stub handlers;
host-metric sampling moves to every five minutes. Environment and work_mem
variables reach the compose whitelist, the production example and the
manifest. The proxy admits the handover link and redirects it at the edge,
and the Health Connect upload bypasses the proxy body limit.
The new routes answer 501 behind their final authentication and are
published in the contract; backup, wipe and key rotation classify the new
models and columns; the seven message bundles carry the base copy. Guards
that list release-pending keys and columns fail once an entry is wired, so
those lists empty themselves before the tag.
Also re-anchor a login-throttle fixture to the wall clock, where its
expiry is compared, and re-base the aggregate bundle cap on a measured
12 KB of new locale strings.
The drill now reads every account's newest off-host object but still ran
on the default fifteen-minute expiry, without a lock and with two
retries, so a large instance could see it killed, run beside itself, and
page every failed account three times. It now carries a four-hour expiry,
stops between accounts on its budget and reports the objects it did not
reach, holds a pass lock at the binding, and is not retried.
A weekly or nightly backup job whose process died (a restart, a
container out of memory) stayed active until its two- or four-hour
expiry, and its retry walked the accounts in the same order: the account
that killed it has the oldest copy, so it led again, and every account
after it went without a copy for as long as that record stayed too
large.
Both passes now send with a five-minute pg-boss heartbeat, so a dead run
fails within minutes and its retry picks it up. Each pass records when it
started and finished every account (backup_pass_attempts, migration
0373); an account whose last attempt never finished goes behind all the
others. The backups page shows when the run in progress started and names
an account a run died under, for the weekly and the off-host card, and a
heartbeat timeout reads as what it means.
"Read with AI" and the lab scan used to run their model calls inside the
request. At the default response time a read of three calls outlasted the
100 s a proxy in front of the server commonly allows, and the person saw a
cut connection for a read the server was still doing.
Both now run as a background run (`DocumentAiRun`, migration 0366):
- `POST /api/documents/inbound/{id}/index` with `Prefer: respond-async`
answers 202 with a run id once every quick refusal has passed. Without
the header it answers exactly as before, which the iPhone app relies on.
A second request while a read of the same document is queued or running
attaches to that run.
- `POST /api/labs/ocr/extract` always answers 202. Only the web client
calls it.
- `GET /api/ai-runs/{id}` serves the run: the same body the synchronous
route answers with, or the failure code and status it would have given.
Scoped to the caller; another account's run is a 404.
The worker (`document-ai-run`, a long queue with a one-hour expiry, one
job per run under `lockedPass`) claims the run, asks the capability again
for the record and re-checks the wire for the provider it picks, sets the
run's deadline from the person's AI response time and refuses a read that
cannot fit the job's expiry. Sync route and worker run one body per kind,
so the two paths cannot drift. Budget is reserved at enqueue and settled
once by whoever ends the run.
Input and result are sealed with labelled AES-256-GCM, registered for key
rotation; the input is dropped when the run ends. A five-minute reaper
fails a run no worker took within fifteen minutes (and hands its
reservation back), fails a run past its deadline, and deletes a finished
run after an hour. Runs are outside backups and inside the data wipe.
The web client polls through the query-key factory and says, calmly, that
the read continues in the background, and when no worker has picked it up.
The insight claim row is now the daily-cap ledger. Its workout_id is
nullable with ON DELETE SET NULL (migration 0367), so deleting a workout
no longer frees the day's slot. The cap counts claims that reached the
provider plus live leases, and drops the separate count of insight rows.
A daily workout-insight-claim-cleanup queue removes claim rows untouched
for seven days.
Every queue bound with createAndWork is now declared in
src/lib/jobs/queue-runtime.ts as short (cannot come near the default
fifteen minutes, with the reason), budgeted (stops itself at three
quarters of the default and resumes) or long (an explicit expiry, a
stop on the job's budget, and a lock or claim per identity; or a
declared one-shot with no retries).
The guard reads every binding, every boss.send and every schedule tuple
in the tree: each names a queue in the table, every entry is bound, a
long queue's sends and schedules all carry its expiry, its handler
stops on jobBudget / jobDeadline where declared and holds lockedPass or
the named marker, and a short or budgeted queue sets no expiry of its
own. Floors pin the counts (112 bindings, 64 sends, 74 schedules), and
planted long queues missing each protection in turn are caught. Run
against the tree before the previous commit it reports 59 findings;
dropping lockedPass and the cron expiry from pr-detection reports both.
Marc-André Bombeck