fix(backup): notice a backup run the process died under, and take its account last

A weekly or nightly backup job whose process died (a restart, a
container out of memory) stayed active until its two- or four-hour
expiry, and its retry walked the accounts in the same order: the account
that killed it has the oldest copy, so it led again, and every account
after it went without a copy for as long as that record stayed too
large.

Both passes now send with a five-minute pg-boss heartbeat, so a dead run
fails within minutes and its retry picks it up. Each pass records when it
started and finished every account (backup_pass_attempts, migration
0373); an account whose last attempt never finished goes behind all the
others. The backups page shows when the run in progress started and names
an account a run died under, for the weekly and the off-host card, and a
heartbeat timeout reads as what it means.
This commit is contained in:
Marc-André Bombeck committed 2026-10-04 15:56:56 +02:00
1 parent d55e329c48
commit f18e6ed34b
31 files changed
+734 -19

No files matched your search

+16
View File
@@ -252,6 +252,22 @@ A run where SOME account got a copy still succeeds, with the rest counted in
`offhost_backup_failed`. Failing the whole queue over one account's object
would re-upload everybody's on every retry.
### A run the process died under
Both backup jobs carry a pg-boss heartbeat of five minutes. While a run is
alive the worker refreshes it; a run whose process died (a restart, or a
container killed for memory) stops refreshing, and pg-boss fails the job with
`job heartbeat timeout` within minutes instead of leaving it `active` until its
two- or four-hour expiry. The retries the queue already allows then pick it up.
Each pass also notes when it started and when it got past every account
(`backup_pass_attempts`). An account whose last attempt started and never
finished is the one the process died under, and the next run takes it last, so
one record too large for the container cannot keep every account after it from
a copy. **Admin console → Backups** shows when the run in progress started and
names such an account under both the weekly and the off-host card. Seeing the
same account there twice means it needs more memory than the container has.
### Which account has no copy
The counts above say how many accounts were uploaded, never which. An account