mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-12 14:16:12 +00:00
Marc-André Bombeck
add6d1fadc
A restore on a freshly booted instance failed its clearing step with a deadlock (40P01) while the boot-time step consolidation was folding the same account. A consolidation day updates or mints the day's total and then soft-deletes the day's per-sample rows; the restore deletes every reading of the account in one statement. When the restore reached a day's samples before its total, each side held what the other needed next. The restore already took an advisory lock per account before its first delete. Every day transaction of the passes built on runConsolidation (step and mean consolidation, the per-sample cumulative drain, the dense intraday retention) now takes the same lock in shared mode as its first statement. A restore that starts mid-day waits for that day to commit; a day that would start while a restore runs is refused at once, and the pass leaves that account for its next run instead of folding rows the restore is about to replace. The passes still share the lock with each other. A new integration test builds the lock order that deadlocked, with a row trigger holding the consolidation on the day's total while the restore starts; before this change the consolidation was cancelled with a deadlock. A guard test requires every transaction in a pass on runConsolidation to open with the shared lock. Refs #1031
81 lines
3.5 KiB
TypeScript
81 lines
3.5 KiB
TypeScript
/**
|
|
* The account lock a restore holds, and the shared side of it for the passes
|
|
* that rewrite an account's readings in the background.
|
|
*
|
|
* A restore deletes every reading of the account in one statement, inside
|
|
* one long transaction. The consolidation passes (step and mean
|
|
* consolidation, the per-sample cumulative drain, the dense intraday
|
|
* retention) fold a day at a time: each day's transaction updates or mints
|
|
* the day's total and then soft-deletes or deletes the day's per-sample
|
|
* rows. Both take row locks on the same readings, each in its own order, and
|
|
* when the restore reached a day's samples before its total while a pass was
|
|
* folding that day, each held what the other needed next. Postgres broke the
|
|
* cycle by cancelling one of them (`40P01`, #1031), and on a freshly booted
|
|
* instance, where the step consolidation runs over every account with legacy
|
|
* rows, that was the restore.
|
|
*
|
|
* So the two are ordered by one advisory lock before either touches a row.
|
|
* The restore takes it exclusively as the first act of its transaction
|
|
* ({@link takeRestoreLock}); every day transaction of those passes takes it in
|
|
* shared mode as its first act ({@link holdAccountAgainstRestore}). Passes
|
|
* share the lock with each other, so nothing about how they run together
|
|
* changes. A restore that starts while a day is being folded waits for that
|
|
* one day to commit, milliseconds. A day that would start while a restore is
|
|
* running does not wait for it, which could take many minutes: it is refused
|
|
* with {@link AccountRestoreInProgressError}, and the pass leaves the account
|
|
* for its next run, which reads the restored rows rather than the ones it had
|
|
* scanned before the restore replaced them.
|
|
*/
|
|
import type { Prisma } from "@/generated/prisma/client";
|
|
|
|
type LockClient = Pick<Prisma.TransactionClient, "$queryRaw">;
|
|
|
|
/** The lock's key. One per account; the restore has always used this one. */
|
|
function restoreLockKey(userId: string): string {
|
|
return `backup-restore:${userId}`;
|
|
}
|
|
|
|
/**
|
|
* Take the account's restore lock exclusively, for the rest of the calling
|
|
* transaction. Waits for any day transaction of a background pass to commit,
|
|
* and for another restore of the same account to finish.
|
|
*/
|
|
export async function takeRestoreLock(
|
|
tx: LockClient,
|
|
userId: string,
|
|
): Promise<void> {
|
|
// `pg_advisory_xact_lock` returns void, which the client cannot read as a
|
|
// column; selecting from it yields a plain row.
|
|
await tx.$queryRaw`
|
|
SELECT 1 AS locked
|
|
FROM pg_advisory_xact_lock(hashtext(${restoreLockKey(userId)}))
|
|
`;
|
|
}
|
|
|
|
/** A restore of the account is running; the pass should leave it for now. */
|
|
export class AccountRestoreInProgressError extends Error {
|
|
readonly userId: string;
|
|
constructor(userId: string) {
|
|
super("A restore of this account is running");
|
|
this.name = "AccountRestoreInProgressError";
|
|
this.userId = userId;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Hold the account against a restore for the rest of the calling
|
|
* transaction: the shared side of {@link takeRestoreLock}. Call it before the
|
|
* transaction's first read or write of the account's readings. Throws
|
|
* {@link AccountRestoreInProgressError} at once when a restore holds the
|
|
* lock, rather than waiting for it.
|
|
*/
|
|
export async function holdAccountAgainstRestore(
|
|
tx: LockClient,
|
|
userId: string,
|
|
): Promise<void> {
|
|
const [row] = await tx.$queryRaw<Array<{ held: boolean }>>`
|
|
SELECT pg_try_advisory_xact_lock_shared(hashtext(${restoreLockKey(userId)})) AS held
|
|
`;
|
|
if (!row?.held) throw new AccountRestoreInProgressError(userId);
|
|
}
|