feat(jobs): tell admins when a system job keeps failing

A full rollup backfill that failed on every boot showed only on the
admin page's card of failing background jobs. From the account's side
the only symptom was empty dashboard cards, and nothing pointed the
operator at the card (#1298).

A new hourly evaluation reads the newest terminal runs of every queue
whose failure has no account-facing surface (the system audience in
subsystem-surface.ts) from pg-boss's own ledger. A failed run adds its
attempts, retries included, and the first successful run ends the
streak. Three failed attempts in a row send every admin a SYSTEM_ALERT
once, a reminder at most once a day while the queue keeps failing, and
one notice when it runs successfully again: the decision the backup
health notices use, with the same compare-and-set claim, so two workers
never send a notice twice.

The memory is one row per announced queue in system_job_alert_notices
(migration 0490). The notice names the queue, the streak and the last
error, and opens the admin status page.
This commit is contained in:
Marc-André Bombeck committed 2026-10-29 09:54:43 +01:00
1 parent a571dcc47a
commit f99f87eca6
23 files changed
+756 -3

No files matched your search

@@ -0,0 +1,24 @@
-- System job alert: what the admins were last told about a background job of
-- the instance that keeps failing.
--
-- `system_job_alert_notices` holds, per pg-boss queue, the state the admins
-- were last notified about and when. It is the whole memory of the hourly
-- evaluation (`src/lib/jobs/system-job-alert.ts`): a queue that has failed
-- three runs in a row notifies once, is reminded at most once a day while it
-- keeps failing, and its first successful run after that is announced once.
-- A row exists only for a queue something was announced about.
--
-- One new table, nothing else touched, so the migration takes no lock on an
-- existing table and running it again changes nothing.
--
-- Reversibility: dropping the table loses only the memory of what was
-- announced; the next evaluation announces a still-failing queue once more.
CREATE TABLE IF NOT EXISTS "system_job_alert_notices" (
"queue" TEXT NOT NULL,
"notified_state" TEXT,
"notified_at" TIMESTAMP(3),
CONSTRAINT "system_job_alert_notices_pkey" PRIMARY KEY ("queue"),
CONSTRAINT "system_job_alert_notices_state_check"
CHECK ("notified_state" IS NULL OR "notified_state" IN ('ok', 'failing'))
);
+17
View File
@@ -5958,6 +5958,23 @@ model RecoveryReadinessNotice {
@@map("recovery_readiness_notices")
}
/// Per background queue of the instance, the state the admins were last
/// notified about. The hourly system job evaluation compares a queue's run
/// history against it: three failed runs in a row notify once, a queue still
/// failing is reminded at most once a day, and its first successful run after
/// that is announced once. The update is a compare-and-set on both columns, so
/// two workers never send the same notice. Written and read by
/// `notifySystemJobFailures`.
model SystemJobAlertNotice {
/// The pg-boss queue name (`rollup-full-backfill`, …).
queue String @id
/// `ok` or `failing`; null once a queue's history has aged out.
notifiedState String? @map("notified_state")
notifiedAt DateTime? @map("notified_at")
@@map("system_job_alert_notices")
}
/// The recovery kit an admin confirmed: every key a restore needs, by kind,
/// id and fingerprint, never the key itself. The kit is due again as soon as
/// a key it needs is not in this set (a rotation, a new backup key). Written