fix(backup): notice a backup run the process died under, and take its account last

A weekly or nightly backup job whose process died (a restart, a
container out of memory) stayed active until its two- or four-hour
expiry, and its retry walked the accounts in the same order: the account
that killed it has the oldest copy, so it led again, and every account
after it went without a copy for as long as that record stayed too
large.

Both passes now send with a five-minute pg-boss heartbeat, so a dead run
fails within minutes and its retry picks it up. Each pass records when it
started and finished every account (backup_pass_attempts, migration
0373); an account whose last attempt never finished goes behind all the
others. The backups page shows when the run in progress started and names
an account a run died under, for the weekly and the off-host card, and a
heartbeat timeout reads as what it means.
This commit is contained in:
Marc-André Bombeck committed 2026-10-04 15:56:56 +02:00
1 parent d55e329c48
commit f18e6ed34b
31 files changed
+734 -19

No files matched your search

@@ -0,0 +1,19 @@
-- One row per account and backup pass, written when the pass starts the
-- account and again when it gets past it. A start newer than the finish with
-- no run active is an attempt the process died under; the pass takes that
-- account last, and the admin backups page names it.
CREATE TABLE "backup_pass_attempts" (
"user_id" TEXT NOT NULL,
-- The pass's queue name: data-backup or data-backup-offhost.
"pass" TEXT NOT NULL,
"started_at" TIMESTAMP(3) NOT NULL,
-- NULL until the pass has once got past this account.
"finished_at" TIMESTAMP(3),
CONSTRAINT "backup_pass_attempts_pkey" PRIMARY KEY ("user_id", "pass")
);
ALTER TABLE "backup_pass_attempts"
ADD CONSTRAINT "backup_pass_attempts_user_id_fkey"
FOREIGN KEY ("user_id") REFERENCES "users"("id")
ON DELETE CASCADE ON UPDATE CASCADE;
+28 -1
View File
@@ -765,6 +765,7 @@ model User {
dataBackups DataBackup[]
backupRestoreJobs BackupRestoreJob[]
offhostBackupState OffhostBackupState?
backupPassAttempts BackupPassAttempt[]
achievements UserAchievement[]
recommendationFeedback RecommendationFeedback[]
idempotencyKeys IdempotencyKey[]
@@ -5160,7 +5161,7 @@ model DataBackup {
/// What the restore preview shows for this copy, worked out while the copy
/// was written (`src/lib/export/backup-preview.ts`): the section counts, the
/// schema version, and the inner key ids with the sections they sit in and
/// one sample value each, so the key verdict is taken again against this
/// the shortest value of each section, so the key verdict is taken again against this
/// server's keys every time it is read. It names the copy it describes and
/// is ignored once the copy changes. Plain JSON on purpose: counts and dates
/// only, and the samples are ciphertext already, which a key rotation must
@@ -5335,6 +5336,32 @@ model OffhostBackupState {
@@map("offhost_backup_state")
}
/// Where one account stands in a backup pass that walks every account (the
/// weekly copy, `data-backup`, and the nightly off-host upload,
/// `data-backup-offhost`): when the pass last started the account and when it
/// last got past it, stored or failed in a way the pass caught. A start newer
/// than the finish, with no run of that pass active, is an attempt the
/// process died under: a restart, or a container killed for memory by a
/// record too large for it. The pass takes such an account last, so one
/// account cannot keep every account after it from a copy, and the admin
/// backups page names it.
/// Consumers: `orderInterruptedLast()` in both passes, and
/// `readBackupPassActivity()` behind `GET /api/admin/backups`.
model BackupPassAttempt {
userId String @map("user_id")
/// The pass's queue name: `data-backup` or `data-backup-offhost`.
pass String
/// When the pass last started on this account.
startedAt DateTime @map("started_at")
/// When the pass last got past it. NULL until it has once.
finishedAt DateTime? @map("finished_at")
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
@@id([userId, pass])
@@map("backup_pass_attempts")
}
/// Per application key id, when the nightly off-host run last put an object
/// in the bucket whose inner ciphertext needed that key. The bucket's
/// lifecycle rule keeps each object for its retention, so a retired key is