From a17f85c0503237b8cf8217e530f126bc9dbf3a10 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc-Andr=C3=A9=20Bombeck?= Date: Fri, 16 Oct 2026 03:57:46 +0200 Subject: [PATCH] docs(ops): add the recovery steps for an upgrade stopped at migration 0409 --- docs/ops/migration-0409-plaintext-location.md | 98 +++++++++++++++++++ docs/self-hosting/getting-started.md | 4 + .../migration.sql | 5 +- .../drop-plaintext-location-migration.test.ts | 15 ++- 4 files changed, 119 insertions(+), 3 deletions(-) create mode 100644 docs/ops/migration-0409-plaintext-location.md diff --git a/docs/ops/migration-0409-plaintext-location.md b/docs/ops/migration-0409-plaintext-location.md new file mode 100644 index 000000000..93f5231e6 --- /dev/null +++ b/docs/ops/migration-0409-plaintext-location.md @@ -0,0 +1,98 @@ +# Upgrade stopped at migration 0409 + +v1.43 removes the readable location columns that v1.42 replaced with sealed +copies (`users.home_lat/home_lon/home_label`, +`environment_contexts.lat/lon/location_label`, +`environment_travel_locations.lat/lon/label`). Before it drops anything, +migration `0409_drop_plaintext_location` checks that the v1.42 encryption +backfill has sealed every one of those values. If any row still holds a +readable location, the migration stops and the app container does not start. + +## What you see + +The container exits during the migration step with: + +```text +A migration failed to apply. New migrations cannot be applied before the error is recovered from. ... +Migration name: 0409_drop_plaintext_location +Database error code: P0001 +Database error: +ERROR: HealthLog migration 0409 stopped: 3 home location(s), 0 location period(s) and 12 daily location(s) are still stored in plaintext. ... +``` + +On every later start, Prisma refuses with `P3009` ("migrate found failed +migrations in the target database") until the failed attempt is cleared. + +## What it means + +Nothing was lost and nothing was dropped. The check runs before any other +statement in the migration, so the location columns and their values are all +still there. The migrations before 0409 in the same image did apply; they only +add tables, columns and enum values, so the previous release runs on top of +them unchanged. + +The cause is a v1.42 instance that was not running long enough for its +encryption backfill to reach every account, for example a host that went from +v1.41 straight to v1.43, or one that was stopped soon after updating to v1.42. + +## How to recover + +The order matters: clear the failed attempt first, because the previous +release runs `prisma migrate deploy` on start as well and refuses to boot +while a failed migration is on record. + +```bash +# 1. Mark the failed 0409 attempt as rolled back. The entrypoint is replaced +# so the container does not try to migrate first. +docker compose run --rm --entrypoint sh app -c \ + 'NODE_PATH=/opt/prisma-cli/node_modules node /opt/prisma-cli/node_modules/prisma/build/index.js migrate resolve --rolled-back 0409_drop_plaintext_location' + +# 2. Start the previous release. Set this in the compose `.env` (or your +# platform's environment) and recreate the container. +# HEALTHLOG_IMAGE_REF=:v1.42.4 +docker compose up -d app +``` + +The previous release starts the encryption backfill about three minutes after +boot and again once a day. It works through the accounts one at a time. Watch +it with the three counts the migration checks; all three must reach 0: + +```bash +docker compose exec -T db psql -U healthlog healthlog -c " + SELECT + (SELECT count(*) FROM users WHERE home_lat IS NOT NULL) AS home, + (SELECT count(*) FROM environment_travel_locations WHERE lat IS NOT NULL) AS periods, + (SELECT count(*) FROM environment_contexts WHERE lat IS NOT NULL) AS days;" +``` + +The app log shows each pass as +`[free-text-encryption-backfill] user=... ` lines. If the counts stop going +down, look for `[free-text-encryption-backfill] user=... failed` in the log; +the error on the line after it names the cause. + +Once all three counts are 0: + +```bash +# 3. Dump the database. This is the copy you go back to if anything else +# goes wrong; keep it with your encryption key. +docker compose exec -T db pg_dump -U healthlog healthlog \ + | gzip > healthlog-before-0409-$(date +%F).sql.gz + +# 4. Upgrade again: remove the pin (or set it to the new version) and +# recreate the container. Migration 0409 now runs through. +docker compose pull app +docker compose up -d app +curl -s http://localhost:3000/api/version +``` + +## What not to do + +- Do not clear the readable columns by hand to get past the check. The + check exists because those values have no sealed copy yet; setting them to + `NULL` deletes the location for good. +- Do not run `migrate resolve --applied 0409_drop_plaintext_location`. That + records the migration as done while the columns are still there, and the + v1.43 schema no longer matches the database. + +Background: [`deploy.md`](deploy.md) for the general update and rollback +order, [`backup-restore.md`](backup-restore.md) for restoring a dump. diff --git a/docs/self-hosting/getting-started.md b/docs/self-hosting/getting-started.md index aa6ed6ba4..58e7f5856 100644 --- a/docs/self-hosting/getting-started.md +++ b/docs/self-hosting/getting-started.md @@ -403,5 +403,9 @@ The line is a notice and does not pause the start: the migrations run right after it. So the dump belongs before step 2, not after you see the line; by then the database is already being migrated. +If an update to v1.43 stops at migration `0409_drop_plaintext_location`, +nothing was changed; the recovery is in +[`docs/ops/migration-0409-plaintext-location.md`](../ops/migration-0409-plaintext-location.md). + For the ordering on a host where you want the migration result before the new container takes traffic, see [`docs/ops/deploy.md`](../ops/deploy.md). diff --git a/prisma/migrations/0409_drop_plaintext_location/migration.sql b/prisma/migrations/0409_drop_plaintext_location/migration.sql index af7f5f4fd..cbce4e700 100644 --- a/prisma/migrations/0409_drop_plaintext_location/migration.sql +++ b/prisma/migrations/0409_drop_plaintext_location/migration.sql @@ -10,7 +10,8 @@ -- It refuses to run while any row still holds a plaintext location: such a -- row has not been sealed yet, and dropping the column would lose it. Start -- the previous release, let the backfill finish (the three counts below must --- all be 0), then upgrade again. +-- all be 0), then upgrade again. The operator steps, including clearing the +-- failed attempt, are in docs/ops/migration-0409-plaintext-location.md. DO $$ DECLARE home_rows BIGINT; @@ -21,7 +22,7 @@ BEGIN SELECT count(*) INTO travel_rows FROM "environment_travel_locations" WHERE "lat" IS NOT NULL; SELECT count(*) INTO day_rows FROM "environment_contexts" WHERE "lat" IS NOT NULL; IF home_rows > 0 OR travel_rows > 0 OR day_rows > 0 THEN - RAISE EXCEPTION 'HealthLog migration 0409 stopped: % home location(s), % location period(s) and % daily location(s) are still stored in plaintext. Run the previous release until its free-text encryption backfill reports zero remaining rows, then upgrade again. Nothing was changed.', home_rows, travel_rows, day_rows; + RAISE EXCEPTION 'HealthLog migration 0409 stopped: % home location(s), % location period(s) and % daily location(s) are still stored in plaintext. Run the previous release until its free-text encryption backfill reports zero remaining rows, then upgrade again. Nothing was changed. Recovery steps: docs/ops/migration-0409-plaintext-location.md', home_rows, travel_rows, day_rows; END IF; END $$; diff --git a/tests/integration/drop-plaintext-location-migration.test.ts b/tests/integration/drop-plaintext-location-migration.test.ts index e20d48ea4..f20ce0785 100644 --- a/tests/integration/drop-plaintext-location-migration.test.ts +++ b/tests/integration/drop-plaintext-location-migration.test.ts @@ -13,7 +13,7 @@ * - Without readable values it drops the nine columns and the three * indexes, and the sealed copies are untouched. */ -import { readFileSync } from "node:fs"; +import { existsSync, readFileSync } from "node:fs"; import { resolve } from "node:path"; import { afterAll, beforeAll, describe, expect, it } from "vitest"; @@ -177,6 +177,19 @@ describe("migration 0409 (drop the readable location columns)", () => { expect(error?.message).toMatch( /1 home location\(s\), 0 location period\(s\) and 0 daily location\(s\)/, ); + // The operator reads this line in the container log; it points at the + // recovery steps, which exist. + expect(error?.message).toMatch( + /docs\/ops\/migration-0409-plaintext-location\.md/, + ); + expect( + existsSync( + resolve( + process.cwd(), + "docs/ops/migration-0409-plaintext-location.md", + ), + ), + ).toBe(true); // Nothing was dropped and the value is where it was. expect(await columnCount(tx)).toBe(9);