feat(jobs): tell admins when a system job keeps failing

A full rollup backfill that failed on every boot showed only on the
admin page's card of failing background jobs. From the account's side
the only symptom was empty dashboard cards, and nothing pointed the
operator at the card (#1298).

A new hourly evaluation reads the newest terminal runs of every queue
whose failure has no account-facing surface (the system audience in
subsystem-surface.ts) from pg-boss's own ledger. A failed run adds its
attempts, retries included, and the first successful run ends the
streak. Three failed attempts in a row send every admin a SYSTEM_ALERT
once, a reminder at most once a day while the queue keeps failing, and
one notice when it runs successfully again: the decision the backup
health notices use, with the same compare-and-set claim, so two workers
never send a notice twice.

The memory is one row per announced queue in system_job_alert_notices
(migration 0490). The notice names the queue, the streak and the last
error, and opens the admin status page.
This commit is contained in:
Marc-André Bombeck committed 2026-10-29 09:54:43 +01:00
1 parent a571dcc47a
commit f99f87eca6
23 files changed
+756 -3

No files matched your search

@@ -163,6 +163,11 @@ const ALLOWED: Readonly<Record<string, { count: number; reason: string }>> = {
count: 1,
reason: "No /proc/diskstats outside Linux; see captureHostMetric.",
},
"lib/jobs/system-job-alert.ts::readTerminalRuns": {
count: 1,
reason:
"No pgboss schema on a web-only deployment; null means no queue to ask, and the evaluation then judges nothing.",
},
"lib/jobs/job-failures.ts::readActiveJobsStartedBefore": {
count: 1,
reason: