mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-12 06:06:10 +00:00
fix(backup): notice a backup run the process died under, and take its account last
A weekly or nightly backup job whose process died (a restart, a container out of memory) stayed active until its two- or four-hour expiry, and its retry walked the accounts in the same order: the account that killed it has the oldest copy, so it led again, and every account after it went without a copy for as long as that record stayed too large. Both passes now send with a five-minute pg-boss heartbeat, so a dead run fails within minutes and its retry picks it up. Each pass records when it started and finished every account (backup_pass_attempts, migration 0373); an account whose last attempt never finished goes behind all the others. The backups page shows when the run in progress started and names an account a run died under, for the weekly and the off-host card, and a heartbeat timeout reads as what it means.
This commit is contained in:
1 parent
d55e329c48
commit
f18e6ed34b
31 files changed
+734
-19
No files matched your search
@@ -252,6 +252,22 @@ A run where SOME account got a copy still succeeds, with the rest counted in
|
||||
`offhost_backup_failed`. Failing the whole queue over one account's object
|
||||
would re-upload everybody's on every retry.
|
||||
|
||||
### A run the process died under
|
||||
|
||||
Both backup jobs carry a pg-boss heartbeat of five minutes. While a run is
|
||||
alive the worker refreshes it; a run whose process died (a restart, or a
|
||||
container killed for memory) stops refreshing, and pg-boss fails the job with
|
||||
`job heartbeat timeout` within minutes instead of leaving it `active` until its
|
||||
two- or four-hour expiry. The retries the queue already allows then pick it up.
|
||||
|
||||
Each pass also notes when it started and when it got past every account
|
||||
(`backup_pass_attempts`). An account whose last attempt started and never
|
||||
finished is the one the process died under, and the next run takes it last, so
|
||||
one record too large for the container cannot keep every account after it from
|
||||
a copy. **Admin console → Backups** shows when the run in progress started and
|
||||
names such an account under both the weekly and the off-host card. Seeing the
|
||||
same account there twice means it needs more memory than the container has.
|
||||
|
||||
### Which account has no copy
|
||||
|
||||
The counts above say how many accounts were uploaded, never which. An account
|
||||
|
||||
Reference in new issue
Block a user