Marc-André Bombeck 7dab9cc5e9 feat(admin): judge every backup signal and push changes to admins ...

Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.

One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:

- database dump: failing once the newest good dump is older than 36 h
  while dumps are on, or when it can be kept nowhere; read through a
  provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
  failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
  window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
  7 days after setup or as soon as a confirmed kit is out of date.

A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.

The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).

2026-10-25 13:32:43 +01:00

3.3 MiB

Raw Blame History