A full rollup backfill that failed on every boot showed only on the
admin page's card of failing background jobs. From the account's side
the only symptom was empty dashboard cards, and nothing pointed the
operator at the card (#1298).
A new hourly evaluation reads the newest terminal runs of every queue
whose failure has no account-facing surface (the system audience in
subsystem-surface.ts) from pg-boss's own ledger. A failed run adds its
attempts, retries included, and the first successful run ends the
streak. Three failed attempts in a row send every admin a SYSTEM_ALERT
once, a reminder at most once a day while the queue keeps failing, and
one notice when it runs successfully again: the decision the backup
health notices use, with the same compare-and-set claim, so two workers
never send a notice twice.
The memory is one row per announced queue in system_job_alert_notices
(migration 0490). The notice names the queue, the streak and the last
error, and opens the admin status page.
Deleting an account, a guardian deleting a managed profile, and "Delete
all data" were final the moment they were confirmed: the account row
cascaded into every table and every stored copy, and the off-host objects
were purged in the same breath. Nothing short of rolling the whole
instance back could return one account.
ACCOUNT_DELETION_GRACE_DAYS (14 by default, at most 30, 0 erases at once
as before) now sets a window:
- Account and managed-profile deletion lock the account (migration 0488:
deletion_requested_at, deletion_purge_at, deletion_requested_by_id).
Every session, refresh token, API token, connector, share link and
step-up elevation is revoked in the same transaction, and the record,
its stored copies and its off-host objects are left as they are.
- Sign-in is refused with 403 auth.account.deletion_pending and the
erasure date, after the credential was proved and before a second
factor is asked for. The session and Bearer resolvers and the acting-
account resolver refuse a pending account too, so a credential minted
in the moment before the lock is useless.
- POST /api/admin/users/{id}/undo-deletion (cookie-only, audit-logged)
clears the lock. The admin users list reports the pending state.
- The hourly account-deletion-purge job erases accounts whose window has
ended, exactly as the immediate deletion did, with the off-host purge
request committed alongside. The Guardian invariant is decided again;
a refused erasure stays pending and is reported.
- A wipe first stores the whole record as a PRE_WIPE_<ms> copy through
the writer the weekly copy uses, keeps it out of the wipe, and holds
its off-host purge back with a new not_before on the request. The
ordinary restore brings the record back; restoring that copy withdraws
the held purge. The purge job deletes the copy when its window ends.
Background work skips accounts pending deletion through one shared
predicate (src/lib/account-deletion/active-account.ts): every reminder,
digest, Coach, sync, backfill, rollup, score and backup enumeration
filters with it, and account-deletion-enumeration-guard.test.ts fails on
a query in background work that neither filters nor says why it need
not. Notification delivery and the AI capability loader refuse a pending
account as chokepoints behind it, and the Withings, WHOOP and Telegram
webhooks and the notification quick actions ignore one.
The privacy summary publishes the window. Two pieces of the reminder tick
and the off-host pass moved into modules of their own to keep the files
inside their size ratchet.
walkSourceFiles now lists each indexed folder (src, e2e, prisma) once
per worker and filters it by prefix, readSource and readSourceFolder read
through the same memoised file cache, and readSourceText is the cached
read for guards that read files themselves. The guards project shares a
module graph per worker, so a file is read once however many guards ask
for it. sourcesContaining adds the substring pre-filter for sweeps that
look for a few markers. Every guard over 1.5 s now reads through it;
anything outside the indexed folders is read from disk as before.
- Load each settings section on its own: the section route imported all
twenty-one statically. A preload starts the chunk beside the record
gate's read, and the loading state paints the settings card shell.
/settings/[section] goes from 611 to 403 KB gz.
- Take Zod out of the shell every route shares: the account-access parse
run on each boot is plain TypeScript now, held to the published schema
by an equivalence test over good and broken blocks. The canonical unit
table moves to its own module, so the display transform no longer
pulls Zod in either. Layout entry 199 to 136 KB gz; /insights 369 to
304, /medications 428 to 365, /labs 400 to 337.
- Open a saved Coach conversation at its last answer: an instant jump
instead of the smooth scroll that lost the end, then follow late
layout (charts, tables) for a few seconds until the reader scrolls.
- Show the document archive as a month grid of preview tiles, two
across on a phone and three or four on a desktop, with the month
headings kept.
- Add the admin card for the measurement table maintenance: a status
read on the maintenance route, one calm confirmation, and the run's
state while it is queued, running, finished or stopped.
- Re-base the bundle budget: /insights at 304, /settings/[section]
watched at 403, aggregate cap 4335 with the reason written down.
A catch that drops what it caught is where the silent nightly failures
started. Every `catch` and `.catch()` under src/lib/jobs now rethrows,
fails the job, writes to a ledger an operator reads, counts into the run's
failure facts, hands the error back to its caller, or logs at warn under a
stable action. The enqueue helpers that swallowed a failed send (briefing,
status, narrative, morning digest, document index / summary / thumbnail,
workout insight, boot-time sweeps), the Coach nudge's provider fallback,
the workout and reaction-line cleanups, the advisory-lock release, the
backup attempt bookkeeping and the cycle reminder's dedupe reads now leave
a warn line instead of nothing or an info annotation.
`error-level-discipline-guard` holds the rule. The catches that stay
silent are listed with their reason (the pgboss readers' documented null on
a web-only deployment, the key-ring trial, the restore drill's per-account
result, an already-deleted Telegram message); a new silent catch fails, an
entry that no longer matches fails, and each entry is shown to be
load-bearing by removing it.
A Coach turn cut short by its token, time or round budget writes
`coach.turn.budget_stop` at warn.
Marc-André Bombeck