fix(backup): notice a backup run the process died under, and take its account last

A weekly or nightly backup job whose process died (a restart, a
container out of memory) stayed active until its two- or four-hour
expiry, and its retry walked the accounts in the same order: the account
that killed it has the oldest copy, so it led again, and every account
after it went without a copy for as long as that record stayed too
large.

Both passes now send with a five-minute pg-boss heartbeat, so a dead run
fails within minutes and its retry picks it up. Each pass records when it
started and finished every account (backup_pass_attempts, migration
0373); an account whose last attempt never finished goes behind all the
others. The backups page shows when the run in progress started and names
an account a run died under, for the weekly and the off-host card, and a
heartbeat timeout reads as what it means.
This commit is contained in:
Marc-André Bombeck committed 2026-10-04 15:56:56 +02:00
1 parent d55e329c48
commit f18e6ed34b
31 files changed
+734 -19

No files matched your search

+2
View File
@@ -706,6 +706,8 @@ export const NOT_IN_BACKUP_MODELS: Readonly<Record<string, string>> = {
"The record of a restore this host ran from one of its own stored copies. It names a backup row that exists only in this database, so carrying it would describe a restore the receiving host never performed.",
OffhostBackupState:
"When this host last put this account's copy in this operator's bucket, and how big it was. It describes one deployment's relationship with one bucket, so restoring it elsewhere would assert an off-host copy that host has never written.",
BackupPassAttempt:
"When this host's backup passes last started and finished this account, kept so a pass the process died under is taken last next time and named on the backups page. It describes this deployment's own runs, so restoring it elsewhere would report an interrupted run that host never had, or hide one it did.",
DocumentImportKey:
"The source keys of imported documents the person deleted and the purge then removed, kept so a re-run of the importer does not store them again. The backup carries only live documents, so the tombstones that hold the same memory before the purge do not travel either; carrying the ledger alone would restore half of one decision. After a restore, a deliberate re-import can bring such a document back, and the person deletes it again.",
DocumentAiRun: