fix(migrations): bound 0412's lock wait where it takes effect and retry a lock timeout on boot

migrate deploy runs a migration file statement by statement, so the
top-level SET LOCAL lock_timeout in 0412 did nothing: the migration
waited as long as the lock was held, with every write to measurements
queued behind it, and a file that failed half-way stayed half-applied.
0412 now takes its locks inside one DO block with a statement-scoped
timeout, which rolls back whole, after an idempotent CREATE OR REPLACE.

The entrypoint runs migrate deploy through scripts/migrate-deploy.mjs.
A 55P03 failure of a migration marked -- @retry-on-lock-timeout is
marked rolled back and retried after 15/30/60/120/240 s, read from
Prisma's report or, after P3009, from the stored row. Every other
failure stays fatal. A guard holds migrations to the shape that makes
the marker true, and 0411/0413 add their columns with IF NOT EXISTS so
a manual re-run is safe too.
This commit is contained in:
Marc-André Bombeck committed 2026-10-20 10:45:14 +02:00
1 parent 47f20b773d
commit 9c89503019
11 files changed
+1004 -69

No files matched your search

+6 -3
View File
@@ -212,11 +212,13 @@ COPY --from=builder /app/src/generated ./src/generated
# application graph (`@/…`, the generated Prisma client) run from a source
# checkout instead. `src/__tests__/container-scripts-ship-guard.test.ts` holds
# the runbooks and the entrypoint against this list. `pending-migrations.mjs`
# is the entrypoint's own: it logs the pending migration count before
# `migrate deploy`. Root-owned on purpose: the app user can run them but not
# and `migrate-deploy.mjs` are the entrypoint's own: the first logs the
# pending migration count, the second runs `migrate deploy` with its
# lock-timeout retry. Root-owned on purpose: the app user can run them but not
# rewrite them.
COPY --from=builder /app/scripts/reset-password.mjs ./scripts/reset-password.mjs
COPY --from=builder /app/scripts/pending-migrations.mjs ./scripts/pending-migrations.mjs
COPY --from=builder /app/scripts/migrate-deploy.mjs ./scripts/migrate-deploy.mjs
COPY --from=builder /app/src/lib/auth/argon2-params.mjs ./src/lib/auth/argon2-params.mjs
COPY --from=builder /app/scripts/repair-intake-anomalies.ts ./scripts/repair-intake-anomalies.ts
# Prove the reset CLI's imports resolve in this tree: with every import found
@@ -224,7 +226,8 @@ COPY --from=builder /app/scripts/repair-intake-anomalies.ts ./scripts/repair-int
RUN node scripts/reset-password.mjs 2>&1 | grep -q "reset-password: usage:"
# The boot notice parses, and the `pg` it loads on a real boot resolves here.
RUN env -u DATABASE_URL node scripts/pending-migrations.mjs && \
node --input-type=module -e "await import('pg')"
node --input-type=module -e "await import('pg')" && \
node --input-type=module -e "await import('./scripts/migrate-deploy.mjs')"
# Install the migration CLI, its config dependency, and a pinned launcher for
# the maintenance scripts already shipped in the standalone tree. The Prisma
+4 -1
View File
@@ -88,9 +88,12 @@ echo "HealthLog: Database is reachable."
# Say how many migrations this image is about to apply, with a reminder to
# back up first. Reporting only: it never stops the boot.
node scripts/pending-migrations.mjs || true
# `migrate-deploy.mjs` runs `prisma migrate deploy` and retries only a
# migration that timed out on a lock and is marked safe to re-run; any other
# failure stops the boot with Prisma's exit code (docs/ops/deploy.md).
echo "HealthLog: Running database migrations..."
NODE_PATH=/opt/prisma-cli/node_modules \
node /opt/prisma-cli/node_modules/prisma/build/index.js migrate deploy
node scripts/migrate-deploy.mjs
echo "HealthLog: Migrations complete."
# ── Start application ────────────────────────────────────────
+49
View File
@@ -40,6 +40,55 @@ not reachable". The entrypoint also runs the migrations on every boot, so
step 2 is optional; it is there for when you want to see the migration
result before the new container takes traffic.
### A migration that timed out on a lock
Some migrations bound how long they wait for a table lock, so that a
long-running holder (most often an autovacuum that is preventing
transaction-ID wraparound, which does not step aside) cannot queue every
write to that table behind the migration. Migration `0412` waits at most ten
seconds for `measurements`. When the wait runs out, the boot log shows
Prisma's report followed by a line from the entrypoint:
```
Error: P3018
...
Migration name: 0412_measurement_hot_updates
Database error code: 55P03
...
HealthLog: migration 0412_measurement_hot_updates timed out waiting for a lock (55P03) and changed nothing; marking it rolled back and retrying in 15s (retry 1 of 5).
```
What happens on its own: such a migration is written so that a timed-out
attempt changes nothing, and it says so in its file
(`-- @retry-on-lock-timeout`). The entrypoint then marks that one migration
rolled back (`prisma migrate resolve --rolled-back`) and runs
`migrate deploy` again after 15, 30, 60, 120 and 240 seconds. The app starts
as soon as an attempt gets the lock. A later boot that finds the attempt still
on record (Prisma's `P3009`) reads the cause from `_prisma_migrations` and
does the same. Nothing else is retried: any other failure, and a lock timeout
in a migration that does not carry the marker, stops the boot exactly as
before.
If all five retries time out, the log ends with
`HealthLog: ERROR — migration … timed out on a lock 6 times in a row; giving
up.` and the container exits. Something has held the table for about eight
minutes. Find it:
```sh
docker compose exec db psql -U healthlog -c "
SELECT pid, backend_type, state, now() - xact_start AS running, query
FROM pg_stat_activity
WHERE pid IN (SELECT pid FROM pg_locks
WHERE relation = 'measurements'::regclass)"
```
An `autovacuum: VACUUM … (to prevent wraparound)` row finishes on its own;
wait for it to disappear and start the container again (a restart policy does
that for you, and each boot retries). Anything else is a session you can
judge: a stuck client may be ended with `SELECT pg_terminate_backend(<pid>)`.
You never need to run `migrate resolve` by hand for this case.
### v1.11.0 notes
Migrations `0110`–`0114` are additive (new tables / columns for the
@@ -16,7 +16,7 @@
--
-- Additive only: no table rewrite, no data change.
ALTER TABLE "users" ADD COLUMN "insights_cached_text_encrypted" BYTEA;
ALTER TABLE "users" ADD COLUMN IF NOT EXISTS "insights_cached_text_encrypted" BYTEA;
-- Partial index on exactly the backfill's discovery predicate, so the boot
-- scan is index-only and shrinks to nothing as the backfill converges (the
@@ -28,13 +28,22 @@
-- `ALTER TABLE ... SET` SHARE UPDATE EXCLUSIVE; both wait for an autovacuum
-- that is preventing wraparound, which does not step aside, and while they
-- wait every write to `measurements` queues behind them, including the ones
-- the running release still serves during a deploy. `migrate deploy` runs the
-- whole file as one transaction, so `SET LOCAL` bounds the wait for this file
-- alone: past ten seconds the migration fails and rolls back whole, the
-- writes go on, and the next deploy tries again once `prisma migrate resolve
-- --rolled-back 0412_measurement_hot_updates` has cleared the failed attempt.
SET LOCAL lock_timeout = '10s';
-- the running release still serves during a deploy. The wait is bounded at
-- ten seconds; past that the attempt fails with SQLSTATE 55P03 and the queued
-- writes go on.
--
-- `migrate deploy` does not run a migration file inside one transaction: each
-- statement commits on its own, so a top-level `SET LOCAL` does nothing and a
-- file that fails half-way stays half-applied. The two statements that take a
-- lock therefore run inside one `DO` block, which is a single statement and
-- so commits or rolls back whole, and the timeout is set inside it with
-- `set_config(..., true)`, scoped to that statement. The function before it
-- is `CREATE OR REPLACE` and harmless to run again. A lock timeout leaves the
-- file safe to apply once more, which is what the marker below declares:
-- `scripts/migrate-deploy.mjs` marks the attempt rolled back and retries it
-- with a backoff (docs/ops/deploy.md, "A migration that timed out on a lock").
--
-- @retry-on-lock-timeout
CREATE OR REPLACE FUNCTION "measurements_keep_stamps_on_noop"() RETURNS trigger
LANGUAGE plpgsql AS $$
@@ -48,13 +57,19 @@ BEGIN
END;
$$;
CREATE TRIGGER "measurements_noop_update_keeps_stamps"
BEFORE UPDATE ON "measurements"
FOR EACH ROW
EXECUTE FUNCTION "measurements_keep_stamps_on_noop"();
DO $migration$
BEGIN
PERFORM set_config('lock_timeout', '10s', true);
ALTER TABLE "measurements" SET (
fillfactor = 85,
autovacuum_vacuum_scale_factor = 0.05,
autovacuum_analyze_scale_factor = 0.05
);
CREATE TRIGGER "measurements_noop_update_keeps_stamps"
BEFORE UPDATE ON "measurements"
FOR EACH ROW
EXECUTE FUNCTION "measurements_keep_stamps_on_noop"();
ALTER TABLE "measurements" SET (
fillfactor = 85,
autovacuum_vacuum_scale_factor = 0.05,
autovacuum_analyze_scale_factor = 0.05
);
END
$migration$;
@@ -9,4 +9,4 @@
-- Nullable and unset for every existing row: an unfinished backfill starts
-- once more from the newest page and records its place from then on.
ALTER TABLE "users" ADD COLUMN "strava_backfill_resume_before" TIMESTAMP(3);
ALTER TABLE "users" ADD COLUMN IF NOT EXISTS "strava_backfill_resume_before" TIMESTAMP(3);
+283
View File
@@ -0,0 +1,283 @@
/**
* The boot's `migrate deploy` wrapper (`scripts/migrate-deploy.mjs`): which
* failures it retries, and the loop it runs around Prisma.
*
* The report texts below are what Prisma 7's `migrate deploy` printed against
* a real Postgres 16 for a migration that hit its `lock_timeout` (P3018), and
* for the boot after that (P3009). The loop is driven end to end through a
* stand-in CLI that plays those reports back, so no database is needed here;
* `tests/integration/migrate-deploy-lock-retry.test.ts` runs the real CLI.
*/
import { spawnSync } from "node:child_process";
import {
existsSync,
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import {
decideLockRetry,
lockTimeoutInReport,
retryDelays,
} from "../migrate-deploy.mjs";
const P3018_LOCK = `Applying migration \`0412_measurement_hot_updates\`
Error: P3018
A migration failed to apply. New migrations cannot be applied before the error is recovered from. Read more about how to resolve migration issues in a production database: https://pris.ly/d/migrate-resolve
Migration name: 0412_measurement_hot_updates
Database error code: 55P03
Database error:
ERROR: canceling statement due to lock timeout
`;
const P3018_UNIQUE = P3018_LOCK.replace("55P03", "23505").replace(
"canceling statement due to lock timeout",
'duplicate key value violates unique constraint "x"',
);
const P3009 = `Error: P3009
migrate found failed migrations in the target database, new migrations will not be applied. Read more about how to resolve migration issues in a production database: https://pris.ly/d/migrate-resolve
The \`0412_measurement_hot_updates\` migration started at 2026-10-20 08:38:22.114155 UTC failed
`;
const MARKED = "-- @retry-on-lock-timeout\nDO $$ BEGIN END $$;\n";
const UNMARKED = "ALTER TABLE x ADD COLUMN y int;\n";
const reader =
(sql: string | null) =>
(_name: string): string | null =>
sql;
describe("which failure is retried", () => {
it("reads the failed migration and its SQLSTATE from Prisma's report", () => {
expect(lockTimeoutInReport(P3018_LOCK)).toBe(
"0412_measurement_hot_updates",
);
expect(lockTimeoutInReport(P3018_UNIQUE)).toBeNull();
expect(lockTimeoutInReport(P3009)).toBeNull();
expect(lockTimeoutInReport(null)).toBeNull();
});
it("retries a lock timeout of a migration marked safe to re-run", () => {
expect(
decideLockRetry({
output: P3018_LOCK,
failedRows: null,
readMigration: reader(MARKED),
}),
).toEqual({ retry: true, migration: "0412_measurement_hot_updates" });
});
it("does not retry a lock timeout of an unmarked migration", () => {
const decision = decideLockRetry({
output: P3018_LOCK,
failedRows: null,
readMigration: reader(UNMARKED),
});
expect(decision.retry).toBe(false);
expect(decision).toMatchObject({
reason: expect.stringContaining("not marked safe to re-run"),
});
});
it("does not retry any other database error", () => {
expect(
decideLockRetry({
output: P3018_UNIQUE,
failedRows: null,
readMigration: reader(MARKED),
}),
).toEqual({ retry: false, reason: "not a lock timeout" });
});
it("does not retry a migration the image does not ship", () => {
expect(
decideLockRetry({
output: P3018_LOCK,
failedRows: null,
readMigration: reader(null),
}).retry,
).toBe(false);
});
it("on the boot after a failure, reads the cause from the stored row", () => {
const failedRows = [
{ migration_name: "0412_measurement_hot_updates", logs: P3018_LOCK },
];
expect(
decideLockRetry({
output: P3009,
failedRows,
readMigration: reader(MARKED),
}),
).toEqual({ retry: true, migration: "0412_measurement_hot_updates" });
// A stored failure of another kind stays fatal.
expect(
decideLockRetry({
output: P3009,
failedRows: [{ ...failedRows[0], logs: P3018_UNIQUE }],
readMigration: reader(MARKED),
}).retry,
).toBe(false);
// Two failed rows, or rows that could not be read: not a state to guess at.
expect(
decideLockRetry({
output: P3009,
failedRows: [failedRows[0], failedRows[0]],
readMigration: reader(MARKED),
}).retry,
).toBe(false);
expect(
decideLockRetry({
output: P3009,
failedRows: null,
readMigration: reader(MARKED),
}).retry,
).toBe(false);
});
it("refuses a migration name that is not a directory name", () => {
expect(
decideLockRetry({
output: P3018_LOCK.replace(
"Migration name: 0412_measurement_hot_updates",
"Migration name: ../../etc",
),
failedRows: null,
readMigration: reader(MARKED),
}).retry,
).toBe(false);
});
it("reads the backoff from the environment and falls back on nonsense", () => {
expect(retryDelays(undefined)).toEqual([15, 30, 60, 120, 240]);
expect(retryDelays("0, 1")).toEqual([0, 1]);
expect(retryDelays("a,1")).toEqual([15, 30, 60, 120, 240]);
});
});
describe("the retry loop", () => {
const dirs: string[] = [];
afterEach(() => {
for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true });
});
/**
* A stand-in Prisma CLI: each `migrate deploy` prints the next scripted
* report and exits with its code; every call is appended to `calls.log`.
*/
function setup(reports: Array<[number, string]>, sql: string) {
const dir = mkdtempSync(join(tmpdir(), "hl-migrate-deploy-"));
dirs.push(dir);
const migrations = join(dir, "migrations");
mkdirSync(join(migrations, "0412_measurement_hot_updates"), {
recursive: true,
});
writeFileSync(
join(migrations, "0412_measurement_hot_updates", "migration.sql"),
sql,
);
writeFileSync(join(dir, "reports.json"), JSON.stringify(reports));
const cli = join(dir, "fake-prisma.mjs");
writeFileSync(
cli,
`import { appendFileSync, readFileSync, writeFileSync, existsSync } from "node:fs";
const dir = ${JSON.stringify(dir)};
const args = process.argv.slice(2);
appendFileSync(dir + "/calls.log", args.join(" ") + "\\n");
if (args[0] === "migrate" && args[1] === "deploy") {
const n = existsSync(dir + "/n") ? Number(readFileSync(dir + "/n", "utf8")) : 0;
writeFileSync(dir + "/n", String(n + 1));
const [code, text] = JSON.parse(readFileSync(dir + "/reports.json", "utf8"))[n];
process.stderr.write(text);
process.exit(code);
}
process.exit(0);
`,
);
const run = () =>
spawnSync(
process.execPath,
[join(process.cwd(), "scripts/migrate-deploy.mjs"), "--schema", "x"],
{
encoding: "utf8",
env: {
...process.env,
DATABASE_URL: "",
HEALTHLOG_PRISMA_CLI: cli,
HEALTHLOG_MIGRATIONS_DIR: migrations,
HEALTHLOG_MIGRATE_LOCK_RETRY_DELAYS: "0,0",
},
},
);
const calls = () =>
existsSync(join(dir, "calls.log"))
? readFileSync(join(dir, "calls.log"), "utf8").trim().split("\n")
: [];
return { run, calls };
}
it("marks the timed-out migration rolled back and deploys again", () => {
const { run, calls } = setup(
[
[1, P3018_LOCK],
[0, "All migrations have been successfully applied."],
],
MARKED,
);
const result = run();
expect(result.status).toBe(0);
expect(calls()).toEqual([
"migrate deploy --schema x",
"migrate resolve --rolled-back 0412_measurement_hot_updates --schema x",
"migrate deploy --schema x",
]);
expect(result.stdout).toContain("timed out waiting for a lock (55P03)");
});
it("gives up after the last delay with Prisma's exit code", () => {
const { run, calls } = setup(
[
[1, P3018_LOCK],
[1, P3018_LOCK],
[1, P3018_LOCK],
],
MARKED,
);
const result = run();
expect(result.status).toBe(1);
expect(calls().filter((c) => c.startsWith("migrate deploy"))).toHaveLength(
3,
);
expect(calls().filter((c) => c.startsWith("migrate resolve"))).toHaveLength(
2,
);
expect(result.stderr).toContain("giving up");
});
it("stops at the first failure that is not a lock timeout", () => {
const { run, calls } = setup([[1, P3018_UNIQUE]], MARKED);
const result = run();
expect(result.status).toBe(1);
expect(calls()).toEqual(["migrate deploy --schema x"]);
});
it("stops at a lock timeout of a migration that is not marked", () => {
const { run, calls } = setup([[1, P3018_LOCK]], UNMARKED);
const result = run();
expect(result.status).toBe(1);
expect(calls()).toEqual(["migrate deploy --schema x"]);
expect(result.stderr).toContain("not marked safe to re-run");
});
});
+239
View File
@@ -0,0 +1,239 @@
/**
* `prisma migrate deploy` for the container boot, with one narrow retry: a
* migration that gave up waiting for a lock.
*
* Run by `docker-entrypoint.sh` as `node scripts/migrate-deploy.mjs`.
*
* A migration that bounds its lock wait (`lock_timeout`) fails with SQLSTATE
* 55P03 when something holds the table longer, typically an autovacuum that
* is preventing wraparound and does not step aside. Prisma then records the
* attempt as failed, and every later `migrate deploy` stops at P3009 until an
* operator runs `prisma migrate resolve --rolled-back`: under a restart
* policy that is a container that never comes up.
*
* `migrate deploy` does not wrap a migration file in a transaction, so a
* failed file is not necessarily rolled back: statements before the one that
* failed stay applied. Marking such an attempt rolled back and running it
* again is only safe when the file was written for it. A migration declares
* that with a `-- @retry-on-lock-timeout` line, which its author adds only
* when every statement before the lock-taking one is harmless to repeat and
* the lock-taking statements form one statement (a `DO` block) that rolls
* back whole. `src/__tests__/migration-lock-timeout-guard.test.ts` holds the
* migrations to that.
*
* So: on a 55P03 failure of a migration carrying the marker, mark that one
* migration rolled back and run `migrate deploy` again after a backoff (15,
* 30, 60, 120, 240 seconds). Any other failure, a lock timeout of a migration
* without the marker, or a fifth lock timeout in a row, ends the boot with
* Prisma's exit code exactly as before. The failure is read from Prisma's
* own report (`Migration name:` and `Database error code:`), or, on a boot
* after a failed one (P3009), from the `logs` Prisma stored on the failed
* row in `_prisma_migrations`.
*
* Environment:
* HEALTHLOG_PRISMA_CLI Prisma's CLI entry (default: the image's copy)
* HEALTHLOG_MIGRATIONS_DIR where the migration files are (default:
* prisma/migrations under the working directory)
* HEALTHLOG_MIGRATE_LOCK_RETRY_DELAYS
* comma-separated backoff in seconds
* Arguments are passed through to every Prisma call (`--schema …`).
*/
import { spawn } from "node:child_process";
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { pathToFileURL } from "node:url";
const DEFAULT_RETRY_DELAYS_S = [15, 30, 60, 120, 240];
/** The line a migration carries when a lock-timeout retry is safe for it. */
export const RETRY_MARKER = /^--\s*@retry-on-lock-timeout\s*$/m;
const LOCK_TIMEOUT = "55P03";
const MIGRATION_NAME = /^[A-Za-z0-9_]+$/;
/**
* The migration a P3018 report says failed on a lock timeout, or null.
* Prisma prints `Migration name: <name>` and `Database error code: <state>`
* for a migration that failed to apply.
*/
export function lockTimeoutInReport(text) {
if (typeof text !== "string") return null;
const name = /^Migration name: (\S+)\s*$/m.exec(text)?.[1];
const code = /^Database error code: (\S+)\s*$/m.exec(text)?.[1];
return name && code === LOCK_TIMEOUT ? name : null;
}
/** Whether Prisma refused to start because a failed migration is on record. */
function isFailedMigrationOnRecord(output) {
return /\bP3009\b/.test(output);
}
/**
* Decide whether a failed `migrate deploy` is the one failure retried here.
*
* @param {object} input
* @param {string} input.output everything Prisma printed
* @param {Array<{migration_name: string, logs: string | null}> | null} input.failedRows
* the unfinished, not-rolled-back rows of `_prisma_migrations`, or null
* when they were not read
* @param {(name: string) => string | null} input.readMigration
* the migration's SQL, or null when the image does not ship it
* @returns {{ retry: true, migration: string } | { retry: false, reason: string }}
*/
export function decideLockRetry({ output, failedRows, readMigration }) {
let migration = lockTimeoutInReport(output);
if (!migration && isFailedMigrationOnRecord(output) && failedRows) {
// More than one failed row is not a state this boot can reason about.
if (failedRows.length === 1) {
migration = lockTimeoutInReport(failedRows[0].logs);
}
}
if (!migration) {
return { retry: false, reason: "not a lock timeout" };
}
if (!MIGRATION_NAME.test(migration)) {
return { retry: false, reason: `unexpected migration name ${migration}` };
}
const sql = readMigration(migration);
if (sql === null) {
return { retry: false, reason: `${migration} is not in this image` };
}
if (!RETRY_MARKER.test(sql)) {
return {
retry: false,
reason: `${migration} timed out on a lock but is not marked safe to re-run`,
};
}
return { retry: true, migration };
}
/** The backoff schedule, from the environment or the default. */
export function retryDelays(raw) {
if (!raw) return DEFAULT_RETRY_DELAYS_S;
const delays = raw.split(",").map((part) => Number(part.trim()));
return delays.every((d) => Number.isFinite(d) && d >= 0)
? delays
: DEFAULT_RETRY_DELAYS_S;
}
function runPrisma(cli, args) {
return new Promise((resolve) => {
const child = spawn(process.execPath, [cli, ...args], {
stdio: ["ignore", "pipe", "pipe"],
});
let output = "";
child.stdout.on("data", (chunk) => {
output += chunk;
process.stdout.write(chunk);
});
child.stderr.on("data", (chunk) => {
output += chunk;
process.stderr.write(chunk);
});
child.on("error", (error) => {
output += String(error);
resolve({ code: 1, output });
});
child.on("close", (code) => resolve({ code: code ?? 1, output }));
});
}
/** The failed rows of `_prisma_migrations`, or null when they cannot be read. */
async function readFailedRows(connectionString) {
if (!connectionString) return null;
try {
const { default: pg } = await import("pg");
const url = new URL(connectionString);
const schema = url.searchParams.get("schema") || "public";
url.searchParams.delete("schema");
const client = new pg.Client({
connectionString: url.toString(),
connectionTimeoutMillis: 5_000,
statement_timeout: 5_000,
});
await client.connect();
try {
const { rows } = await client.query(
`SELECT migration_name, logs FROM ${client.escapeIdentifier(schema)}._prisma_migrations
WHERE finished_at IS NULL AND rolled_back_at IS NULL`,
);
return rows;
} finally {
await client.end().catch(() => {});
}
} catch {
return null;
}
}
function migrationReader(dir) {
return (name) => {
try {
return readFileSync(join(dir, name, "migration.sql"), "utf8");
} catch {
return null;
}
};
}
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
async function main({
cli = process.env.HEALTHLOG_PRISMA_CLI ||
"/opt/prisma-cli/node_modules/prisma/build/index.js",
migrationsDir = process.env.HEALTHLOG_MIGRATIONS_DIR ||
join(process.cwd(), "prisma", "migrations"),
delays = retryDelays(process.env.HEALTHLOG_MIGRATE_LOCK_RETRY_DELAYS),
extraArgs = process.argv.slice(2),
} = {}) {
const readMigration = migrationReader(migrationsDir);
for (let attempt = 0; ; attempt++) {
const deploy = await runPrisma(cli, ["migrate", "deploy", ...extraArgs]);
if (deploy.code === 0) return 0;
const failedRows = isFailedMigrationOnRecord(deploy.output)
? await readFailedRows(process.env.DATABASE_URL)
: null;
const decision = decideLockRetry({
output: deploy.output,
failedRows,
readMigration,
});
if (!decision.retry) {
if (decision.reason !== "not a lock timeout") {
console.error(
`HealthLog: not retrying the migration: ${decision.reason}.`,
);
}
return deploy.code;
}
if (attempt >= delays.length) {
console.error(
`HealthLog: ERROR — migration ${decision.migration} timed out on a lock ` +
`${attempt + 1} times in a row; giving up. Something holds the table ` +
"for minutes: see docs/ops/deploy.md, " +
'"A migration that timed out on a lock".',
);
return deploy.code;
}
const delay = delays[attempt];
console.log(
`HealthLog: migration ${decision.migration} timed out waiting for a lock ` +
`(55P03) and changed nothing; marking it rolled back and retrying in ` +
`${delay}s (retry ${attempt + 1} of ${delays.length}).`,
);
const resolve = await runPrisma(cli, [
"migrate",
"resolve",
"--rolled-back",
decision.migration,
...extraArgs,
]);
if (resolve.code !== 0) return resolve.code;
await sleep(delay * 1000);
}
}
if (import.meta.url === pathToFileURL(process.argv[1] ?? "").href) {
process.exitCode = await main();
}
@@ -0,0 +1,128 @@
/**
* Structural guard: how a migration bounds its lock wait, and when the boot
* may retry it.
*
* `prisma migrate deploy` runs a migration file statement by statement, each
* committing on its own; it does not wrap the file in a transaction. Two
* consequences were written into this repository the other way round:
*
* - A top-level `SET LOCAL lock_timeout` does nothing (there is no
* transaction for it to be local to), so the migration waits as long as
* the lock is held, and every write to the table queues behind it.
* - A file that fails half-way stays half-applied, so marking it rolled back
* and running it again can fail on what already exists, or apply a change
* twice.
*
* So a lock wait is bounded inside a `DO` block with
* `set_config('lock_timeout', …, true)`, which is one statement and rolls back
* whole; and the `-- @retry-on-lock-timeout` marker, which lets
* `scripts/migrate-deploy.mjs` retry a 55P03 failure on boot, is allowed only
* on a file whose every other statement is harmless to run again.
*/
import { readdirSync, readFileSync, existsSync } from "node:fs";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { RETRY_MARKER } from "../../scripts/migrate-deploy.mjs";
const MIGRATIONS_DIR = join(process.cwd(), "prisma", "migrations");
function migrations(): Array<{ name: string; sql: string }> {
return readdirSync(MIGRATIONS_DIR, { withFileTypes: true })
.filter((entry) => entry.isDirectory())
.map((entry) => join(MIGRATIONS_DIR, entry.name, "migration.sql"))
.filter((file) => existsSync(file))
.map((file) => ({
name: file.split("/").at(-2) ?? file,
sql: readFileSync(file, "utf8"),
}));
}
/** The statements of a file, comments removed, dollar-quoted bodies blanked. */
function topLevelStatements(sql: string): string[] {
return sql
.replace(/--[^\n]*/g, "")
.replace(/\$([A-Za-z_]*)\$[\s\S]*?\$\1\$/g, "$BODY$")
.split(";")
.map((statement) => statement.trim().replace(/\s+/g, " "))
.filter((statement) => statement.length > 0);
}
/** Statements that leave the database as they found it when run twice. */
const REPEATABLE = [
/^DO \$BODY\$$/i,
/^CREATE OR REPLACE (FUNCTION|VIEW|PROCEDURE) /i,
/^CREATE (UNIQUE )?INDEX IF NOT EXISTS /i,
/^CREATE TABLE IF NOT EXISTS /i,
/^ALTER TABLE "?\w+"? ADD COLUMN IF NOT EXISTS /i,
/^DROP (INDEX|TRIGGER|FUNCTION|TABLE) IF EXISTS /i,
];
/** A session setting changed outside a transaction-scoped `DO` body. */
const TOP_LEVEL_TIMEOUT =
/^SET (LOCAL |SESSION )?(lock_timeout|statement_timeout)\b|^SET LOCAL\b/i;
function violations(name: string, sql: string): string[] {
const found: string[] = [];
const statements = topLevelStatements(sql);
for (const statement of statements) {
if (TOP_LEVEL_TIMEOUT.test(statement)) {
found.push(`${name}: top-level "${statement}" has no effect`);
}
}
const boundsLockWait = /set_config\(\s*'lock_timeout'/i.test(sql);
const marked = RETRY_MARKER.test(sql);
if (boundsLockWait && !marked) {
found.push(
`${name}: bounds its lock wait but lacks -- @retry-on-lock-timeout`,
);
}
if (marked) {
for (const statement of statements) {
if (!REPEATABLE.some((pattern) => pattern.test(statement))) {
found.push(
`${name}: marked retry-safe, but "${statement.slice(0, 60)}" is not repeatable`,
);
}
}
}
return found;
}
describe("migration lock waits", () => {
it("bounds lock waits where they take effect and marks only repeatable files", () => {
const found = migrations().flatMap(({ name, sql }) =>
violations(name, sql),
);
expect(found, found.join("\n")).toEqual([]);
});
it("sees the one migration that bounds its wait today", () => {
const marked = migrations()
.filter(({ sql }) => RETRY_MARKER.test(sql))
.map(({ name }) => name);
expect(marked).toContain("0412_measurement_hot_updates");
});
it("catches each shape it exists to refuse", () => {
expect(violations("a", "SET LOCAL lock_timeout = '10s';")).toHaveLength(1);
expect(
violations(
"b",
"DO $x$ BEGIN PERFORM set_config('lock_timeout', '1s', true); END $x$;",
),
).toHaveLength(1);
expect(
violations(
"c",
"-- @retry-on-lock-timeout\nCREATE TRIGGER t BEFORE UPDATE ON m FOR EACH ROW EXECUTE FUNCTION f();\nDO $x$ BEGIN PERFORM set_config('lock_timeout', '1s', true); END $x$;",
),
).toEqual([expect.stringContaining("is not repeatable")]);
expect(
violations(
"d",
"-- @retry-on-lock-timeout\nCREATE OR REPLACE FUNCTION f() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN RETURN NEW; END; $$;\nDO $x$ BEGIN PERFORM set_config('lock_timeout', '1s', true); END $x$;",
),
).toEqual([]);
});
});
@@ -8,11 +8,10 @@
* here against Postgres: the no-op update keeps `updated_at` and
* `sync_version` and is counted as HOT in the transaction's own statistics,
* a real change still moves both, the nightly check reads the table's
* statistics and leaves the vacuuming to autovacuum, and the migration gives
* up on a lock it cannot get instead of queueing every writer behind it.
* statistics and leaves the vacuuming to autovacuum. The migration's lock
* wait is proven through the real `migrate deploy` in
* `migrate-deploy-lock-retry.test.ts`.
*/
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { Client } from "pg";
import { beforeEach, describe, expect, it } from "vitest";
@@ -121,46 +120,3 @@ describe("the nightly check", () => {
}
});
});
describe("the migration's lock wait", () => {
it("fails within its lock timeout while another session holds the table", async () => {
// The migration's own text under fresh names, run the way `migrate
// deploy` runs a file: one multi-statement query, one transaction.
const sql = readFileSync(
join(
process.cwd(),
"prisma/migrations/0412_measurement_hot_updates/migration.sql",
),
"utf8",
)
.replaceAll("measurements_keep_stamps_on_noop", "lock_probe_fn")
.replaceAll("measurements_noop_update_keeps_stamps", "lock_probe_trg");
const holder = new Client({ connectionString: process.env.DATABASE_URL });
const migrator = new Client({ connectionString: process.env.DATABASE_URL });
await holder.connect();
await migrator.connect();
try {
// What an anti-wraparound autovacuum holds, and does not give up.
await holder.query("BEGIN");
await holder.query(
"LOCK TABLE measurements IN SHARE UPDATE EXCLUSIVE MODE",
);
const startedAt = Date.now();
const error = await migrator.query(sql).then(
() => null,
(err: { code?: string }) => err,
);
expect(error?.code).toBe("55P03");
expect(Date.now() - startedAt).toBeLessThan(30_000);
// Rolled back whole: nothing of the probe is left behind.
const { rows } = await migrator.query(
"SELECT to_regproc('lock_probe_fn') AS fn",
);
expect(rows[0].fn).toBeNull();
} finally {
await holder.query("ROLLBACK").catch(() => {});
await holder.end();
await migrator.end();
}
}, 60_000);
});
@@ -0,0 +1,259 @@
/**
* The boot's `migrate deploy` (`scripts/migrate-deploy.mjs`) against the real
* Prisma CLI and Postgres: migration 0412's lock wait, and the retry after it.
*
* Runs 0412's own text (its ten-second timeout cut to one) in a schema of its
* own, on a minimal `measurements` table, while another session holds the
* lock an anti-wraparound autovacuum would hold. Prisma runs a migration file
* statement by statement, so this is the only place the timeout and the
* all-or-nothing `DO` block are proven the way they will run on a host; a
* multi-statement `pg` query is one implicit transaction and would prove both
* even for a file that cannot deliver them.
*/
import { spawn } from "node:child_process";
import {
cpSync,
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { Client } from "pg";
import { afterAll, beforeEach, describe, expect, it } from "vitest";
const SCHEMA = "lock_retry_probe";
const NAME = "0412_measurement_hot_updates";
const ROOT = process.cwd();
const dirs: string[] = [];
afterAll(async () => {
for (const dir of dirs.splice(0)) rmSync(dir, { recursive: true });
const db = await connect();
await db.query(`DROP SCHEMA IF EXISTS ${SCHEMA} CASCADE`);
await db.end();
});
function baseUrl(): string {
const url = process.env.DATABASE_URL;
if (!url) throw new Error("DATABASE_URL is not set for the integration run");
return url;
}
function probeUrl(): string {
const url = new URL(baseUrl());
url.searchParams.set("schema", SCHEMA);
return url.toString();
}
async function connect(): Promise<Client> {
const url = new URL(baseUrl());
url.searchParams.delete("schema");
const client = new Client({ connectionString: url.toString() });
await client.connect();
return client;
}
/**
* A migrations directory whose first migration creates the probe table
* (Prisma will not migrate a schema it did not build) and is applied here.
*/
async function migrationsDir(): Promise<string> {
const dir = mkdtempSync(join(tmpdir(), "hl-lock-retry-"));
dirs.push(dir);
writeFileSync(
join(dir, "schema.prisma"),
'datasource db {\n provider = "postgresql"\n}\n',
);
mkdirSync(join(dir, "migrations", "0001_probe_table"), { recursive: true });
cpSync(
join(ROOT, "prisma", "migrations", "migration_lock.toml"),
join(dir, "migrations", "migration_lock.toml"),
);
writeFileSync(
join(dir, "migrations", "0001_probe_table", "migration.sql"),
`CREATE TABLE "measurements" (
"id" integer PRIMARY KEY, "value" integer,
"updated_at" timestamp(3), "sync_version" integer);`,
);
const setup = await migrateDeploy(dir);
expect(setup.code, setup.stderr).toBe(0);
return dir;
}
/** Add 0412's own text, its timeout cut to one second, optionally unmarked. */
function add0412(dir: string, { marked }: { marked: boolean }): void {
const original = readFileSync(
join(ROOT, "prisma", "migrations", NAME, "migration.sql"),
"utf8",
);
let sql = original.replace("'lock_timeout', '10s'", "'lock_timeout', '1s'");
expect(
sql,
"0412 still sets its timeout the way this test shortens it",
).not.toBe(original);
if (!marked) sql = sql.replace(/^-- @retry-on-lock-timeout\n/m, "");
mkdirSync(join(dir, "migrations", NAME), { recursive: true });
writeFileSync(join(dir, "migrations", NAME, "migration.sql"), sql);
}
function migrateDeploy(
dir: string,
): Promise<{ code: number; stdout: string; stderr: string }> {
return new Promise((resolve) => {
const child = spawn(
process.execPath,
[
join(ROOT, "scripts", "migrate-deploy.mjs"),
"--schema",
join(dir, "schema.prisma"),
],
{
cwd: ROOT,
env: {
...process.env,
DATABASE_URL: probeUrl(),
HEALTHLOG_PRISMA_CLI: join(
ROOT,
"node_modules",
"prisma",
"build",
"index.js",
),
HEALTHLOG_MIGRATIONS_DIR: join(dir, "migrations"),
HEALTHLOG_MIGRATE_LOCK_RETRY_DELAYS: "2,2,2",
},
},
);
let stdout = "";
let stderr = "";
child.stdout.on("data", (chunk) => (stdout += chunk));
child.stderr.on("data", (chunk) => (stderr += chunk));
child.on("close", (code) => resolve({ code: code ?? 1, stdout, stderr }));
});
}
/** Hold what an anti-wraparound autovacuum holds on the probe table. */
async function holdLock(): Promise<Client> {
const holder = await connect();
await holder.query("BEGIN");
await holder.query(
`LOCK TABLE ${SCHEMA}.measurements IN SHARE UPDATE EXCLUSIVE MODE`,
);
return holder;
}
async function state(db: Client) {
const rows = await db.query(
`SELECT migration_name, finished_at IS NOT NULL AS finished,
rolled_back_at IS NOT NULL AS rolled_back, logs
FROM ${SCHEMA}._prisma_migrations ORDER BY started_at`,
);
const trigger = await db.query(
`SELECT 1 FROM pg_trigger t JOIN pg_class c ON c.oid = t.tgrelid
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = $1 AND t.tgname = 'measurements_noop_update_keeps_stamps'`,
[SCHEMA],
);
const options = await db.query(
`SELECT c.reloptions FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = $1 AND c.relname = 'measurements'`,
[SCHEMA],
);
return {
migrations: rows.rows,
trigger: trigger.rowCount === 1,
reloptions: (options.rows[0]?.reloptions ?? []) as string[],
};
}
beforeEach(async () => {
const db = await connect();
await db.query(`DROP SCHEMA IF EXISTS ${SCHEMA} CASCADE`);
await db.end();
});
describe("migration 0412 under a held lock", () => {
it("gives up within its timeout, changes nothing, and the boot retries it", async () => {
const dir = await migrationsDir();
add0412(dir, { marked: true });
const holder = await holdLock();
// The first attempt meets the lock; the holder lets go during the backoff.
const release = setTimeout(() => {
void holder.query("ROLLBACK");
}, 2_500);
const startedAt = Date.now();
const result = await migrateDeploy(dir);
clearTimeout(release);
await holder.query("ROLLBACK").catch(() => {});
await holder.end();
expect(result.code, result.stderr).toBe(0);
expect(result.stderr).toContain("Database error code: 55P03");
expect(result.stdout).toContain(
`migration ${NAME} timed out waiting for a lock (55P03)`,
);
expect(Date.now() - startedAt).toBeLessThan(45_000);
const db = await connect();
const after = await state(db);
await db.end();
expect(
after.migrations.map((m) => [
m.migration_name,
m.finished,
m.rolled_back,
]),
).toEqual([
["0001_probe_table", true, false],
[NAME, false, true],
[NAME, true, false],
]);
expect(after.trigger).toBe(true);
expect(after.reloptions).toContain("fillfactor=85");
}, 90_000);
it("leaves nothing half-applied, stays fatal unmarked, and a later boot recovers", async () => {
const dir = await migrationsDir();
add0412(dir, { marked: false });
const holder = await holdLock();
let result;
try {
result = await migrateDeploy(dir);
} finally {
await holder.query("ROLLBACK").catch(() => {});
await holder.end();
}
expect(result.code).not.toBe(0);
expect(result.stderr).toContain("not marked safe to re-run");
const db = await connect();
const failed = await state(db);
// The lock-taking statements rolled back as one: no trigger, no options.
expect(failed.trigger).toBe(false);
expect(failed.reloptions).toEqual([]);
expect(failed.migrations).toHaveLength(2);
expect(failed.migrations[1]).toMatchObject({
migration_name: NAME,
finished: false,
rolled_back: false,
});
expect(failed.migrations[1].logs).toContain("Database error code: 55P03");
// The next image ships the marked file; its boot meets P3009 and reads
// the cause from the stored row.
add0412(dir, { marked: true });
const recovered = await migrateDeploy(dir);
expect(recovered.code, recovered.stderr).toBe(0);
expect(recovered.stderr).toContain("P3009");
const after = await state(db);
await db.end();
expect(after.trigger).toBe(true);
expect(after.migrations.at(-1)).toMatchObject({
finished: true,
rolled_back: false,
});
}, 90_000);
});