mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-19 08:12:52 +00:00
fix(backup): notice a backup run the process died under, and take its account last
A weekly or nightly backup job whose process died (a restart, a container out of memory) stayed active until its two- or four-hour expiry, and its retry walked the accounts in the same order: the account that killed it has the oldest copy, so it led again, and every account after it went without a copy for as long as that record stayed too large. Both passes now send with a five-minute pg-boss heartbeat, so a dead run fails within minutes and its retry picks it up. Each pass records when it started and finished every account (backup_pass_attempts, migration 0373); an account whose last attempt never finished goes behind all the others. The backups page shows when the run in progress started and names an account a run died under, for the weekly and the off-host card, and a heartbeat timeout reads as what it means.
This commit is contained in:
1 parent
d55e329c48
commit
f18e6ed34b
31 files changed
+734
-19
No files matched your search
@@ -0,0 +1,19 @@
|
||||
-- One row per account and backup pass, written when the pass starts the
|
||||
-- account and again when it gets past it. A start newer than the finish with
|
||||
-- no run active is an attempt the process died under; the pass takes that
|
||||
-- account last, and the admin backups page names it.
|
||||
CREATE TABLE "backup_pass_attempts" (
|
||||
"user_id" TEXT NOT NULL,
|
||||
-- The pass's queue name: data-backup or data-backup-offhost.
|
||||
"pass" TEXT NOT NULL,
|
||||
"started_at" TIMESTAMP(3) NOT NULL,
|
||||
-- NULL until the pass has once got past this account.
|
||||
"finished_at" TIMESTAMP(3),
|
||||
|
||||
CONSTRAINT "backup_pass_attempts_pkey" PRIMARY KEY ("user_id", "pass")
|
||||
);
|
||||
|
||||
ALTER TABLE "backup_pass_attempts"
|
||||
ADD CONSTRAINT "backup_pass_attempts_user_id_fkey"
|
||||
FOREIGN KEY ("user_id") REFERENCES "users"("id")
|
||||
ON DELETE CASCADE ON UPDATE CASCADE;
|
||||
+28
-1
@@ -765,6 +765,7 @@ model User {
|
||||
dataBackups DataBackup[]
|
||||
backupRestoreJobs BackupRestoreJob[]
|
||||
offhostBackupState OffhostBackupState?
|
||||
backupPassAttempts BackupPassAttempt[]
|
||||
achievements UserAchievement[]
|
||||
recommendationFeedback RecommendationFeedback[]
|
||||
idempotencyKeys IdempotencyKey[]
|
||||
@@ -5160,7 +5161,7 @@ model DataBackup {
|
||||
/// What the restore preview shows for this copy, worked out while the copy
|
||||
/// was written (`src/lib/export/backup-preview.ts`): the section counts, the
|
||||
/// schema version, and the inner key ids with the sections they sit in and
|
||||
/// one sample value each, so the key verdict is taken again against this
|
||||
/// the shortest value of each section, so the key verdict is taken again against this
|
||||
/// server's keys every time it is read. It names the copy it describes and
|
||||
/// is ignored once the copy changes. Plain JSON on purpose: counts and dates
|
||||
/// only, and the samples are ciphertext already, which a key rotation must
|
||||
@@ -5335,6 +5336,32 @@ model OffhostBackupState {
|
||||
@@map("offhost_backup_state")
|
||||
}
|
||||
|
||||
/// Where one account stands in a backup pass that walks every account (the
|
||||
/// weekly copy, `data-backup`, and the nightly off-host upload,
|
||||
/// `data-backup-offhost`): when the pass last started the account and when it
|
||||
/// last got past it, stored or failed in a way the pass caught. A start newer
|
||||
/// than the finish, with no run of that pass active, is an attempt the
|
||||
/// process died under: a restart, or a container killed for memory by a
|
||||
/// record too large for it. The pass takes such an account last, so one
|
||||
/// account cannot keep every account after it from a copy, and the admin
|
||||
/// backups page names it.
|
||||
/// Consumers: `orderInterruptedLast()` in both passes, and
|
||||
/// `readBackupPassActivity()` behind `GET /api/admin/backups`.
|
||||
model BackupPassAttempt {
|
||||
userId String @map("user_id")
|
||||
/// The pass's queue name: `data-backup` or `data-backup-offhost`.
|
||||
pass String
|
||||
/// When the pass last started on this account.
|
||||
startedAt DateTime @map("started_at")
|
||||
/// When the pass last got past it. NULL until it has once.
|
||||
finishedAt DateTime? @map("finished_at")
|
||||
|
||||
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
|
||||
|
||||
@@id([userId, pass])
|
||||
@@map("backup_pass_attempts")
|
||||
}
|
||||
|
||||
/// Per application key id, when the nightly off-host run last put an object
|
||||
/// in the bucket whose inner ciphertext needed that key. The bucket's
|
||||
/// lifecycle rule keeps each object for its retention, so a retired key is
|
||||
|
||||
Reference in new issue
Block a user