A restore never runs beside a backup pass or another large job: the
heavy lane puts its delivery back in the queue for a minute at a time.
The job's own row says queued for the whole wait, and the console said
the restore starts "as soon as the background worker picks it up",
which read as a stuck worker while a backup ran.
The status routes now ask the queue. A delivery the lane put back sits
in created with a start_after ahead of now, and the active row of a
heavy queue is what holds the lane. The job view carries
waiting: "backup" | "busy" | null, and the console says it waits for
the running backup, or for another large job, and since when it has
been queued, in all nine locales. A queue that cannot be asked answers
no wait and is reported at warn.
The job view and its reads move to backup-restore-view.ts, which keeps
the worker module under its size baseline.
Global setup migrates one template database and reads what a fresh install
holds: its schemas, its tables and the rows the migrations seed. Each Vitest
worker clones the template on its first file with CREATE DATABASE ... TEMPLATE
and keeps the clone for the rest of the run, so files run in parallel without
seeing each other's rows, advisory locks or queue schema. Schemas created at
runtime (pg-boss, migration probes) are dropped when the next file starts.
The truncate list is the catalogue now, every table in public except the
migration bookkeeping, instead of a hand-kept list of 42 of 166 tables. The
seeded catalogues are put back after each truncate in foreign-key order, so a
test starts from what a fresh install has. The settings singleton is the one
seeded table a test starts without, declared with its reason.
next/headers and the boot compatibility probe are mocked once in the setup
file. Two lock-wait probes now look at their own database only, since other
workers' backends share pg_stat_activity.
A restore on a freshly booted instance failed its clearing step with a
deadlock (40P01) while the boot-time step consolidation was folding the
same account. A consolidation day updates or mints the day's total and
then soft-deletes the day's per-sample rows; the restore deletes every
reading of the account in one statement. When the restore reached a
day's samples before its total, each side held what the other needed
next.
The restore already took an advisory lock per account before its first
delete. Every day transaction of the passes built on runConsolidation
(step and mean consolidation, the per-sample cumulative drain, the dense
intraday retention) now takes the same lock in shared mode as its first
statement. A restore that starts mid-day waits for that day to commit;
a day that would start while a restore runs is refused at once, and the
pass leaves that account for its next run instead of folding rows the
restore is about to replace. The passes still share the lock with each
other.
A new integration test builds the lock order that deadlocked, with a
row trigger holding the consolidation on the day's total while the
restore starts; before this change the consolidation was cancelled with
a deadlock. A guard test requires every transaction in a pass on
runConsolidation to open with the shared lock.
Refs #1031
Marc-André Bombeck