Three defects in the bounded document reader.
Image uploads were decoded into their thumbnails in the server process. A
header check is all that stood between a decoder and an allocation the file
chooses, and the GIF check showed how that fails. The image is now decoded in
the same child process PDFs run in, under its memory ceiling and time limit;
the header check stays in front of it.
The reader's one slot went to whoever asked first, and a waiting item waited
without end, so a few heavy PDFs from one account held everyone's lab scans
behind them. The slot now goes to a read a person is waiting on before
background work, to accounts in turn, to a few waiting items per account, and
an item that waits past its lane's deadline is answered busy, which callers
treat as retryable.
Any child that ended without an answer counted as a document past its limits,
so a kernel kill caused by the container's overall memory marked a good PDF
as permanently failed. Only a bound the reader measured (the sampled memory,
the time limit, the child's heap cap) is final now; any other end is run once
more and then answered interrupted, which the thumbnail job retries within its
ordinary attempts.
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
When the grace window ends for an account that has become the last
active guardian of a managed profile, the purge job refuses to erase it
and the account stays locked. Until now only the worker error log said
so. Every admin now gets one localized notification naming the account
and the ways out (another guardian, deleting the profile, or undoing
the deletion). An audit row on the account records that it was reported,
so the hourly retries stay quiet; a deletion undone and asked for again
is reported afresh.
The undo confirmation in the admin users list now says that sign-in
works again but API tokens, share links and trusted devices stay revoked
and have to be set up again (for a managed profile: its share links and
tokens).
The Coach snapshot attached its labs block regardless of the module, a
decision from before Labs could be switched off, while the snapshot's own
prelude says a disabled domain never reaches the model. With Labs off, the
Coach prompt, its get_labs tool, the daily briefing's flagged-marker block,
the biomarker status note, the reaction line to a new panel and the MCP
lab_trend_brief prompt all still read lab values for a model.
Each of those reads now asks the labs module first and reads nothing when it
is off: the Coach tool answers module_disabled, the briefing omits the
block, the status note neither writes nor serves text, the reaction line
gets no value, and the MCP prompt refuses like get_labs and the doctor-visit
prompt already did. The data routes stay as they are. A structural guard
holds every lab-table reader under the model-input paths to the gate or to
a listed reason its rows never reach a model.
The thumbnail's header walk took the byte after any 0xFF as a marker. A
single 0xFF fill byte after the start-of-image marker, which the JPEG
standard allows and every decoder skips, threw it off; it read no size, and
an unread size let the image through to a decode that allocates the full
bitmap. A small JPEG declaring 20000 x 20000 pixels allocated 1.6 GB in the
web process before the post-decode check could see it.
The avatar upload already had a walker that skips fill bytes. Both now use
one reader (lib/image-dimensions) that follows the marker rules: fill bytes,
the length-less markers, and no size when a scan or the end of the image
comes before a frame header. The thumbnail treats an unreadable size as a
refusal (size-unreadable) for every format it decodes, and the backfill
counts that refusal as final.
pdf.js decodes an image at the size the file declares, whatever size the
page is drawn at, and the server opened documents without an image limit.
A small upload declaring a very large image made the web process allocate
gigabytes until the container was killed, for every user at once, and the
thumbnail job ran the same document again after each restart. An image
whose soft mask declares the size does the same past pdf.js' own limit, and
nothing bounded how long a text read could take.
pdf.js is now opened with maxImageSize (20 million pixels, which keeps one
image inside the reader's ceiling and admits a 300 dpi A3 scan), and every
read and render runs in a child process, one at a time: its V8 heap is
capped, its resident memory is sampled and the child killed past 640 MB, on
Linux it volunteers to the out-of-memory killer first, a time limit ends a
read that runs on, and it starts with an almost empty environment. The
loader and the rendering moved into pdf-engine.mjs, which the child runs
with plain Node and which is traced into the standalone image.
A document that crosses a bound answers resource-limit. The thumbnail job
records it as a final failure, so the backfill does not queue it again.
Deleting an account, a guardian deleting a managed profile, and "Delete
all data" were final the moment they were confirmed: the account row
cascaded into every table and every stored copy, and the off-host objects
were purged in the same breath. Nothing short of rolling the whole
instance back could return one account.
ACCOUNT_DELETION_GRACE_DAYS (14 by default, at most 30, 0 erases at once
as before) now sets a window:
- Account and managed-profile deletion lock the account (migration 0488:
deletion_requested_at, deletion_purge_at, deletion_requested_by_id).
Every session, refresh token, API token, connector, share link and
step-up elevation is revoked in the same transaction, and the record,
its stored copies and its off-host objects are left as they are.
- Sign-in is refused with 403 auth.account.deletion_pending and the
erasure date, after the credential was proved and before a second
factor is asked for. The session and Bearer resolvers and the acting-
account resolver refuse a pending account too, so a credential minted
in the moment before the lock is useless.
- POST /api/admin/users/{id}/undo-deletion (cookie-only, audit-logged)
clears the lock. The admin users list reports the pending state.
- The hourly account-deletion-purge job erases accounts whose window has
ended, exactly as the immediate deletion did, with the off-host purge
request committed alongside. The Guardian invariant is decided again;
a refused erasure stays pending and is reported.
- A wipe first stores the whole record as a PRE_WIPE_<ms> copy through
the writer the weekly copy uses, keeps it out of the wipe, and holds
its off-host purge back with a new not_before on the request. The
ordinary restore brings the record back; restoring that copy withdraws
the held purge. The purge job deletes the copy when its window ends.
Background work skips accounts pending deletion through one shared
predicate (src/lib/account-deletion/active-account.ts): every reminder,
digest, Coach, sync, backfill, rollup, score and backup enumeration
filters with it, and account-deletion-enumeration-guard.test.ts fails on
a query in background work that neither filters nor says why it need
not. Notification delivery and the AI capability loader refuse a pending
account as chokepoints behind it, and the Withings, WHOOP and Telegram
webhooks and the notification quick actions ignore one.
The privacy summary publishes the window. Two pieces of the reminder tick
and the off-host pass moved into modules of their own to keep the files
inside their size ratchet.
A database dump is the whole database at one moment, so an account deleted
since stays in the dumps taken before until they age out. The database row
on the backups card now says so with the numbers that decide it: how many
nightly dumps this server keeps, and after how many days the bucket's
lifecycle rule removes the off-host copies.
The lifecycle probe used to answer with the shortest expiry of any enabled
rule, whatever its prefix, so a rule scoped to the account copies read as
covering the dumps too. It now also answers for `database/` alone, counting
only rules whose filter takes in a dump's key (prefix, legacy prefix, an
And filter; a tag filter never matches untagged objects), and the card
warns when no rule covers that prefix: dumps there, and the deleted accounts
in them, would never be removed. The reading moved into its own module.
Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.
One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:
- database dump: failing once the newest good dump is older than 36 h
while dumps are on, or when it can be kept nowhere; read through a
provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
7 days after setup or as soon as a confirmed kit is out of date.
A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.
The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).
The step confirmed one key, the active application key. A restore needs
more: off-host copies are sealed under BACKUP_ENCRYPTION_KEY, a copy
written before a rotation needs the retired application key it was
written under, and every session, token and share link hangs off
API_TOKEN_HMAC_KEY. An operator could do everything the step asked and
still be unable to read a single backup.
The recovery kit lists every key a restore of this server needs, by id
and fingerprint, never key material:
- each id in ENCRYPTION_KEYS, plus any retired id a stored copy
(innerKeyIds) or an off-host object inside the bucket's retention
(OffhostBackupKeyUse) still lists, also once the server dropped it;
- BACKUP_ENCRYPTION_KEY and each BACKUP_ENCRYPTION_PREVIOUS_KEYS entry,
named by the id their objects record;
- API_TOKEN_HMAC_KEY.
Confirming records the set the admin was shown and refuses a stale one
with 409 recovery.kit.stale. The kit is due again as soon as a key it
lists is not in that set (a rotation, a new backup key, a re-keyed
install); a key that merely drops out does not make it due. "Check my
copy" now checks any key in the kit.
Migration 0487 carries a confirmation given under the old step forward
as a kit holding that key, so that part stays confirmed, and drops the
four app_settings columns. It also creates the two tables the readiness
evaluation uses (restore_drill_outcomes, recovery_readiness_notices).
The routes move to /api/admin/recovery-kit; the reader of the keys the
backups still need moves out of the encryption status route into
src/lib/recovery/backup-key-needs.ts so the kit and the status view
share one definition.
The per-account copies roll one account back on the same server; nothing
the app made could rebuild a lost one. A scheduled job now runs pg_dump
over the whole database (every schema, pg-boss included) into
DB_BACKUP_DIR as healthlog-<UTC time>-daily.dump, with a manifest beside
it: app version and build, newest migration, Postgres version, size and
SHA-256, exact row counts of every table, and the fingerprints of the
keys the database needs.
The counts read the same exported snapshot pg_dump reads, so a dump
taken while the app writes still matches its manifest. The archive is
written as .partial, listed back with pg_restore --list, and renamed
after its manifest; a failed run leaves nothing named like a good dump.
Retention removes only files this job wrote, oldest first, and never
the newest good one (DB_BACKUP_KEEP, default 7; two pre-migration dumps).
With the off-host bucket configured, each dump and its manifest go up
under database/ in the envelope the account copies use, streamed. Every
run lands in the new database_backup_runs ledger (migration 0486), and
getDatabaseBackupHealth() answers the last good dump, the last failure,
whether the directory is writable and whether off-host is configured.
The boot readiness summary says where dumps go, or that none can be kept.
The dump, naming, retention and restore rules live in one plain ESM
module so the image's own scripts can share them without tsx.
The admin upload streamed only the measurements while it checked a file,
so a document-heavy account copy was held whole in memory and an upload of
a few hundred megabytes could take the process down.
The sections a reader takes an element at a time are now named once
(BACKUP_STREAMED_SECTIONS in streamed-backup.ts): measurements, intake
events, mood entries, documents and ECG recordings. The upload reads its
file through readStreamedBackup like the restore and the preview, and the
restore drill and the restore script use the same key set.
A behavioural test pushes a 271 MB document-heavy file through the real
upload route and fails when the heap grows with the file. The stream guard
now asserts that every reader goes through the shared reader or key set
and that no file declares its own.
A status run is one completion, so its ceiling was the completion ceiling
under a second name. The Local client also re-exported a JSON-mode helper
nobody imported from it.
The boot discovery for the documents 0479 marked took the first thousand
by id and skipped those whose account has the documents module off.
Such an account keeps its marks, so with enough of them it held every
place of every boot and the accounts after it never got their retry.
The marks are walked by id past the last row read, an account found
with the module off is left out of the query from then on, and the
boot stops at a thousand documents enqueued rather than read.
Refs #1277
The index-all backfill billed the read and wrote the index in one try:
an index write that failed after the read was settled again in the
catch, handing the reservation back twice. The read is settled once,
and a failed write charges nothing back.
The backfill also checked consent and the operator's switches once, at
the pick, then read up to two hundred documents. The wire is asked
again before every read; a refusal hands that document's reservation
back and ends the walk with no-consent or unavailable.
The sweep keeps going when reporting a load failure itself throws, and
now says so on stderr; the error-level guard refuses a silent catch in
src/lib/jobs. The preload test reads the worker source through
readSource with anchors.
Since handlers load on their queue's first job, a module the image
cannot load no longer fails the worker's boot. The worker comes up
healthy, the queue burns its retries when it first has work, and a
one-shot job (a restore, an import tail) ends failed with nobody told at
deploy time.
After binding every queue the worker now runs `preloadHandlers` in the
background, without awaiting it, so readiness is unchanged. It loads
each handler once through the same memoised loader its queue is bound
to, so a module that loads stays loaded for the first job. Each failure
is reported on the `job.run.failed` line at error, with the queue's name
and `reason: "handler_load_failed"`, plus the worker error report
(stderr and GlitchTip). A summary line on stderr names every queue that
failed. The sweep never throws, and a failing report does not stop it.
docs/ops/deploy.md says what to look for after a deploy.
A course's end date is a calendar day and counts in full: the doses of that
day stay due, reminded and expected until local midnight. That is right for
an end date set in the editor and wrong for "End course", whose dialog
promises that nothing more is due.
A course now carries `endedAt` (migration 0482, additive, nullable), the
instant it was ended on its `endsOn` day, projected onto the medication next
to `startsOn` / `endsOn` by the one course writer. From that instant:
- the schedule engine emits no slot, so a dose later that day is not due,
not reminded, not projected onto the today list and never expected by
compliance, while a dose earlier that day stays inside the course;
- the course reads ENDED (`courseStatusAt` / `courseStatusOn`), so the dose
actions, `remindersActive` and the reminder gate stop at once, follow-ups
of an earlier dose included;
- every engine context builder, compliance context, dose history, Coach
snapshot, doctor report and the adherence storyline read it, through one
select and one picker for the medication window.
Any write that moves a course's `endsOn` clears the instant, so an end date
chosen in the editor keeps meaning the whole day. Backups carry it in both
purposes and a restore projects it again. `courseStatusAt` moves next to the
other course rules.
Refs #1280
Drives the status-card path the way an idle instance runs it: a data sync
every minute dirtying five cards, the production enqueue with its queue
name, singleton key and retry options, the queue's consumer, the card's
single call and fallback, the status chokepoint, the chain walker with the
health ledger's in-memory twin, the gateway client and the note store's
negative windows. Only the wire, pg-boss, the clock and the unrelated
Postgres reads are faked; the wire answers every request with reasoning
and finish_reason "length".
Against the code before this series the hour counted 151 requests, 150 of
them sent with "none", none charged to the day's ledger, and 300 when the
reasoning lesson did not survive from one job to the next (the pattern
behind the reported log). Now it counts six, charges all of them, sends
"none" once, and logs the pause once per window.
Refs #1279
pg-boss gives a send that names no retry options the queue's defaults:
two retries, at once, without a backoff. A job that reached a model and
failed then ran again immediately, twice, against the provider that had
just failed it. Two model-bound queues still relied on those defaults:
the Coach memory refresh send and the nudge cron. Both now ask for the
same two retries, a minute apart and backed off.
ai-job-retry-bound-guard holds the rule: every send to a queue whose jobs
reach a model names retryLimit (at most 3) and, unless it is 0,
retryBackoff with a retryDelay of at least 30 s; every such cron carries
the same in its schedule options. The queues come from the model-run
sites the expiry guard already keeps complete (now a shared helper), plus
the job files that call a provider directly, which the new guard keeps
complete itself. A planted case runs the checker over sources shaped like
the tree before this change. The queue golden records the nudge cron's
new options.
Refs #1279
The content-index backfill capped a run at 200 documents, but counted
only the ones it indexed. Against a provider that failed every call (a
model that reasons and never answers) the cap was never reached, so the
walk transcribed every un-indexed document in turn, one provider call
each, until the job's time ran out after more than two hours, and the
refunded failures never reached the day's token ceiling.
The cap now counts provider calls, the walk stops with reason
provider-failing after three failed calls in a row (one document the
model cannot read is still walked past), and a failed read charges what
its reply reported spending. The per-document read moves into its own
function.
Refs #1279
Deleting a document only stamps a tombstone and the purge removes it
after 30 days, but the only way back was the undo toast, gone a few
seconds after the delete. Recently deleted lists the deleted documents
of the record with type, filing date, where an import came from, when
each was deleted and how long it has left, and offers Restore, Restore
all and Delete now.
GET /api/documents/inbound/deleted lists them (owner-only, capped at 200
with the list-cap meta). POST /api/documents/inbound/{id}/purge removes
one deleted document for good through the same function the nightly
purge runs, so its import keys and aliases still move into the ledger
and an importer re-sending it is still answered "deleted"; a live
document answers 409 documents.inbound.notDeleted. The usage read gains
deletedCount, 0 under a grant. The restore route and bulk restore
share one restore, which now also clears ledger entries for the
document's own keys.
The vault header shows a quiet "Recently deleted (n)" link only while
something is in it, and the delete toasts and confirmations say that
deleted documents stay there for 30 days. The purge's queue facts and
grace window move to document-purge-queue.ts so the worker registry
still loads none of the work.
Refs #1275
While two pdf.js versions broke each other, a local text read after a
page render failed and was recorded as decrypt-error, and a scanned PDF
that had to be rasterized for the provider was recorded as raster-failed.
Neither has a content index, and on an instance without an AI provider
nothing offers to index it again.
Migration 0479 clears the attempt time of exactly those PDFs and keeps
their reason, a state no index attempt leaves behind. A new boot
discovery, document-index-retry, enqueues the upload's auto-index job for
each such row, so it runs the same provider-first, local-fallback decision
under the same consent rules; a deferred import stays local because its
row says so. The attempt writes its time back, so every document drops out
after one try, and one that really cannot be read records its reason again
and stays settled. An account with the documents module off keeps its
marks until the module is on.
An integration test seeds both kinds of damage, runs each migration's
statement and only what a boot runs on its own, and sees a preview and an
index appear, while a blank page, a broken ciphertext and a file that is
not a PDF settle after one retry and are not found by the next boot.
Refs #1277
Since the lab staging reads a long report in parts, one auto-index could
make the transcription plus up to twenty reads of two calls each at the
person's response timeout, hours past pg-boss's fifteen-minute default,
which failed the job and retried it beside the reads still running. The
staging now takes the run's deadline as a schedule and plans only the
reads that fit before it. The index job is long, with an expiry above the
calls that run before the plan can say no, a stop on jobDeadline and a
lock per document; the document run hands the same staging its own
deadline, which moves the run's expiry as the reads are planned. The
document summary, one call and its reasoning repeat, gets an expiry above
that worst case.
ocrReadCallMs in the read core is the one per-call timeout both deadlines
divide by. The expiry guard now names every importer of the document and
lab read entry points with the queues that reach it.
The runtime guard's planted cases stripped every file's comments again on
each rescan, which took over a second alone and timed out under a full
parallel run; the stripped text is now memoised.
Refs #1274
A status run (a metric card, the batched assessment, a derived score, a
workout line, a Coach memory pass) is one model call plus the one repeat
the Local and gateway clients send when a thinking model answered with
reasoning alone, each with the person's response timeout. The run's outer
bound was a single call, so the repeat was cut off, and the open card
stopped polling after 48 s, so a note from a model that needs minutes
never reached the page. The bound is now the per-call timeout times
STATUS_MODEL_CALLS, and the card's poll ceiling follows the published
response timeout (fast for the first 48 s, every 15 s after), only while
the route reports the note in flight; the fallback text stays meanwhile.
The status queue is exclusive, so the longer poll cannot queue a second
generation beside a running one.
The morning refresh is one comprehensive generation of up to five calls
in a row, with no outer bound and pg-boss's fifteen-minute expiry, so a
large setting had pg-boss fail and retry it while it was still running.
It now runs behind the same bound as the warms, and the job carries an
expiry derived from that worst case. The per-metric note, the workout line
and the Coach memory refresh get expiries derived the same way.
queue-runtime.ts gains a bounded class for these jobs: model time read off
model-time.ts, an expiry above it, carried on every send. A new guard
names every caller of runStatusCompletion and generateComprehensiveInsight
with the queues that reach it, and holds each such queue to bounded above
its runs or long under a run lock.
Refs #1274
Extracting lab values from a stored document still sent the whole text
in one call. A multi-page report produced more JSON than one call's
output ceiling, the answer was cut mid-fact, the parse failed twice, and
the read came back as 422 "Couldn't read the document. Try a clearer
copy." The lab scan had already learned to read the same report a page
or a part at a time, but each surface carried its own copy of the read,
so the document path never got it. It also ran under the shared 60 s
timeout, dropped every fact past 120 (or every fact when one did not
match the schema), and charged a flat reservation.
The read now lives once, in lib/documents/ocr-read.ts and ocr-plan.ts:
the per-read limits, the corrective retry, the salvage of a cut
answer's complete rows, the wire re-check and budget reservation before
each read, and the planning of a PDF a page at a time and text in parts
at line boundaries. The lab scan and the document extraction (the
extract route, its background run and the lab auto-staging) both read
through it, and the background run plans its reads on the same deadline
as a lab scan. The per-read budget moves to a shared metered budget.
The document path validates the answer fact by fact, keeps the lab
scan's row cap (400), and says why nothing came back: 422
documents.inbound.outputLimit when the reads stopped at the output
limit, 429 when the budget refused them, extractFailed only for answers
that were not the schema. A run whose reads were cut or not reached
reports `incomplete` on its result.
A guard holds the class: the output ceiling is set only in the core,
both surfaces read through runOcrReads, both runs share the deadline,
and a lab read and a document read go out under the same limits.
Refs #1257
The briefing read polls while the warm it enqueued runs, and it stopped
after ten polls (225 s) whatever the response timeout. A briefing that
landed later never reached the open page, which kept the empty card, or
the previous attempt's "timed out" hint, on screen until a reload.
The read's poll ceiling now follows the published response timeout over
the generation's sequential calls, never less per call than the
briefing's 180 s default, plus one interval for the queue to pick the
warm up. A settling regenerate and the overview body keep their ten
polls. A test walks the poll schedule against the warm's own bound for
every setting, and the client-abort guard holds the read to the derived
ceiling.
Refs #1274
The out-of-band warm the Insights page relies on bounded one
comprehensive generation at the account's response timeout plus 30 s.
A generation is up to five model calls in a row: the generation, the
JSON retry, the grounding retry, the outbound-screen repair, and the one
repeat a thinking model gets when it answered with reasoning alone. Each
call honoured the setting, but a local model needing 300 s per answer
under a 600 s setting met the 630 s bound in the middle of its first
corrective retry, and the briefing was dropped.
The bound is now the per-call timeout times COMPREHENSIVE_MODEL_CALLS
plus the headroom, for the nightly pass and the forced warm alike. A
test drives the real warm, generator, chain runner, Local client,
safeFetch and pinned dispatcher against a loopback server that answers
after 300 s; under the old bound it reports a timeout on the third
answer.
The response-timeout guard now names every outer bound around model
work and holds both briefing generators' stages to the call counts the
outer waits multiply by.
Refs #1274
Postgres returns rows that tie on an ORDER BY in no particular order,
and not the same one twice. The medication reads ordered by columns two
rows share all the time: every read that hands schedules on included
them with no order at all, medications were listed by createdAt alone
(one import writes many in a millisecond), intakes by scheduledFor (a
twice-daily medication puts every morning dose on one instant), and
"the latest intake" lookups by takenAt. A client read the same account
in a different order from one request to the next, and a latest-intake
lookup picked either of two rows.
Every findMany / findFirst on a medication table, nested includes
included, now ends its order on the id in the direction of its last key,
and schedules are read by window, then id (MEDICATION_SCHEDULE_ORDER, the
order the backup already wrote). medication-read-order-guard.test.ts
holds the tree to it. medication-read-order.test.ts writes tying rows in
the opposite order to the one each read promises, through the medication
list and today's intakes, and fails without the tie-breakers.
The parity suite's own state reader had the same gap: it sorts rows by
content with the ids blanked, and rows that tie then kept whatever order
the unordered read returned, so their <id:N> labels swapped under a
loaded run. Its reads are ordered by id, which for a cuid is the order
the rows were written in.
The JSON import's schema left the route, so its entry in the inline
schema strictness register goes; the schema module says why its objects
are not strict.
A restore never runs beside a backup pass or another large job: the
heavy lane puts its delivery back in the queue for a minute at a time.
The job's own row says queued for the whole wait, and the console said
the restore starts "as soon as the background worker picks it up",
which read as a stuck worker while a backup ran.
The status routes now ask the queue. A delivery the lane put back sits
in created with a start_after ahead of now, and the active row of a
heavy queue is what holds the lane. The job view carries
waiting: "backup" | "busy" | null, and the console says it waits for
the running backup, or for another large job, and since when it has
been queued, in all nine locales. A queue that cannot be asked answers
no wait and is reported at warn.
The job view and its reads move to backup-restore-view.ts, which keeps
the worker module under its size baseline.
A hold ended only when its handler settled. pg-boss fails a job at its
expiry and fires its abort signal but does not stop the handler, so a
backup that hung held the lane for the life of the process: every heavy
job, a restore an operator started among them, went back to the queue
every minute from then on, and the only trace was one warn line after
two hours.
A hold now carries the longest expiry of the jobs it runs. A job that
finds the lane held past that expiry and one deferral of grace reports
job.heavy_lane.holder_expired at error and takes the lane; the old
handler, should it ever settle, releases only a hold that is still its
own.
Forty-two guard files carried their own comment stripper, block comments
first in one non-greedy pass, so a /* inside a line comment opened a
block that hid the code after it, and the stricter copies hid code
behind a /* inside a string, which every e2e route glob has. They all
read through the shared stripComments now, and it scans the way the
parser does: strings, template literals and regular expressions keep
every character, a comment inside a template expression goes, and line
numbers are kept. A parity test strips every source file both ways,
with the scanner and with the TypeScript parser, and fails on the first
that differs.
No guard reports anything new. Each one fails when the stripper returns
nothing; three of them passed on an empty file and assert that the code
they ban is read at all now.
Work a request starts and does not wait for runs inside the request's
context, so an annotate({ action }) from it landed on the request's event
whenever it ran before the event was emitted. The measurement tail's
status re-warm logged a hand-entered reading as
insights.status.invalidate.skipped on some requests and as
measurement.create on others.
runDetached starts such work so that its action is dropped for the event
it was detached from; meta, warnings and the auth it reads stay as they
were, and an event the work opens for itself keeps its own action.
fireAndForget now takes a function and starts it detached, and every
trackBackgroundTask handle on a request path is built with runDetached,
which a structural guard holds.
The backfill, the sleep repair, the OAuth-state cleanup and the sync
handlers move from lib/jobs into lib/fitbit/jobs, beside the queue facts
and definitions the registry already spreads. Their tests move with them.
Removing the integration is now the folder, its routes, the registry
spread and the drain's two cases.
The updatedAt backfill guard also walks provider jobs folders, so the
moved backfill stays covered.
Oura, Polar, WHOOP and Strava each keep the transport and endpoints in
`client.ts` and gain `wire-types.ts` (the record shapes) and `map.ts` (the
pure record-to-row mappers, no I/O). Google Health's 1,600-line client
becomes `budget.ts` (slice budget and outcome tally), `request.ts` (error
detail, retry, rate limits), `data-points.ts` (the paginated walker) and a
small `client.ts` for the profile and roll-up reads; its `export * from
"./mappers"` facade is gone and its importers name the module that defines
what they use.
The code moved verbatim; the mapper goldens written before the move and
the provider integration tests pass unchanged.
`integrations/sync-kit/after-write.ts` is the tail every provider write
ends in: the arrivals go out, the touched days are refolded, the read
caches are marked stale back to the earliest touched reading, and the
touched status cards are re-warmed (through `afterMeasurementMutation`,
shared with the interactive routes); Withings also evicts the dashboard
snapshot. A full-history backfill refolds its whole touched range once
through `afterProviderBackfill`.
Withings, WHOOP, Polar, Nightscout, Google Health, Fitbit and Oura drop
their hand-written tails, and no provider module imports the comprehensive
insight generator any more; a guard holds the provider folders to that.
Differences from the copies: the status cards are re-warmed even when the
fold failed, and the caches are marked stale back to the earliest touched
reading instead of wholesale, so older closed blocks survive a sync.
Several provider unit tests mocked an export the old tails never reached
(their failures were caught and only logged); the mocks now name what the
code calls.
`oauth-kit/client.ts` builds a provider's consent URL, code exchange,
refresh and deauthorization from a declaration of what differs: endpoints
and scope, client authentication (form or HTTP Basic, with or without the
client id beside it), the fields a grant adds, how an answer is classified
and the refusal named, the error type, and where the token sits in the
body. The seven hand-written copies of the redirect URI, authorization URL,
token post, Basic header and PKCE pair are gone; each provider keeps an
`oauth.ts` with its declaration.
Shared behaviour is now uniform: a 2xx without a token object (an empty or
non-JSON body) is a transient `empty_token_body` failure for every
provider. WHOOP, Strava and Google Health used to return null, and
Withings threw an unclassified SyntaxError; the OAuth wire golden records
exactly these lines changing and nothing else. Oura's live/legacy endpoint
fallback, Google's invalid_grant lift, Strava's Fault summary and Fitbit's
errorType are declared per provider.
The five hand-written registrars (about 5,500 lines, one 997-line function)
are replaced by `jobs/registry/`: one `defineJob` entry per queue with its
cron, schedule send options, policy, work options, lock, lane and a handler
loader. `registerJobGroup` creates, schedules, retires and binds each group
in the order the registrars used; the worker boot walks the groups and runs
their boot discovery.
Handlers load on their queue's first job. The facts a declaration needs at
boot (names, crons, concurrency, send options) moved out of the heavy job
modules into light `*-queue.ts` siblings, and their importers point there,
so loading the registry evaluates 288 modules instead of the 1,164 the
registrars pulled in. The formerly inline handlers moved verbatim into
handler modules under `jobs/reminder/`.
Fitbit's queues live in `fitbit/jobs/definitions.ts` and enter the registry
through one spread, so removing the integration later is that folder and
that line.
`tests/integration/queue-registration-parity.test.ts` proves pg-boss holds
exactly what the registrars left (queues, schedules, bindings, byte for
byte) and loads every handler once. The queue tests and the runtime guard
assert on the registry objects instead of reading registrar source.
The module gate and the cycle gate imported each other for one pure
predicate. isCycleEnabled moves to lib/cycle/enabled.ts and both gates
import it from there.
The band minter read DOSE_WINDOW_DEFAULTS through the analytics/compliance
re-export barrel, which loads the compliance display and with it the
scheduling modules that import the minter. It now imports the constant from
the module that defines it. Runtime import cycles fall from 6 to 4.
The nightly measurements check reported its bloat factor, a ratio such as
1.3, as a job fact. A fact is a non-negative whole number, so the outcome
validation refused it on every run, after the vacuum and the reindex had
already done their work, and the run failed. It now reports bloat_percent
(130), and the duration goes through the same rounding.
A malformed fact no longer fails a run whose work is done: runJob drops it
from the result and emits job.outcome.fact_invalid at error level with the
queue and the key, where failing would have made pg-boss retry finished
work. A failed outcome is unchanged. wholeFact and percentFact are the
intended way to report a measured value or a ratio, and a guard reads
every outcome producer and refuses a fact that is a division or names a
ratio unless it is rounded on purpose. The other producers were checked:
their numeric facts are counts and integer millisecond durations.
Refs #1271