diff --git a/docs/ops/deploy.md b/docs/ops/deploy.md index c7484a315..65593cd4b 100644 --- a/docs/ops/deploy.md +++ b/docs/ops/deploy.md @@ -36,12 +36,13 @@ docker compose up -d app ``` Step 2 takes the pre-migration dump too, in the one-off container, before it -migrates. When the backups directory is writable and that dump fails (a full -disk, most often), the entrypoint stops before migrating and says so; free -the space and run the step again, or set `DB_BACKUP_BEFORE_MIGRATE=false` for -one start without the dump. Without a writable directory (an install that -runs the image without the `backups` volume) the log carries a warning, no -dump is taken, and the migrations run. +migrates. When the backups directory is on its volume and that dump fails (a +full disk, most often), the entrypoint stops before migrating and says so; +free the space and run the step again, or set `DB_BACKUP_BEFORE_MIGRATE=false` +for one start without the dump. Without the `backups` volume the directory +is not writable, or it is the image's own and sits inside the container that +the update replaces; either way the log carries a warning, no dump is taken, +and the migrations run. Use `docker compose run` rather than a plain `docker run` for step 2. A plain `docker run` starts outside the stack's network, where `db` does not @@ -236,19 +237,20 @@ roll back: app version reads forward-compatible-ish shapes for nearly every release pair shipped on the v1.4.x line. 3. If a column-drop migration ships in the rollback target, hand-roll - the column back via a fresh forward migration (`pnpm prisma migrate -dev --create-only --name vX_Y_Z_rollback_`), edit the SQL, - commit, and re-deploy. Never `migrate resolve --rolled-back` against + the column back via a fresh forward migration + (`pnpm prisma migrate dev --create-only --name vX_Y_Z_rollback_`), + edit the SQL, commit, and re-deploy. Never `migrate resolve --rolled-back` against production unless you've taken a fresh `pg_dump` first. +4. Rolling back from v1.40.2 or later to v1.40.1 or earlier: an account + that saved the Coach lookback "Last year" reads its whole Coach + preference blob as defaults on the older version, metric exclusions + included, until the setting is saved again. Ask affected accounts to + pick another lookback before the rollback, or re-save their exclusions + after it. When the data itself has to go back to before the update, restore the pre-migration dump with the previous image; the steps are in -[`disaster-recovery.md`](disaster-recovery.md#going-back-to-before-a-migration). 4. Rolling back from v1.40.2 or later to v1.40.1 or earlier: an account -that saved the Coach lookback "Last year" reads its whole Coach -preference blob as defaults on the older version, metric exclusions -included, until the setting is saved again. Ask affected accounts to -pick another lookback before the rollback, or re-save their exclusions -after it. +[`disaster-recovery.md`](disaster-recovery.md#going-back-to-before-a-migration). ## Catalog templates diff --git a/docs/ops/disaster-recovery.md b/docs/ops/disaster-recovery.md index 3d9f971b5..9114e368b 100644 --- a/docs/ops/disaster-recovery.md +++ b/docs/ops/disaster-recovery.md @@ -157,10 +157,32 @@ docker compose up -d ``` The dump before a migration is taken while the new image starts. If it fails -on a writable directory (a full disk, most often), the start stops before +on a directory that keeps it (a full disk, most often), the start stops before anything is migrated and the log names the switch: `DB_BACKUP_BEFORE_MIGRATE=false` starts without it, once. +## Knowing the copies are there + +Nobody opens the backups page until the day it matters, so the server tells +the admins instead: + +- The **Recovery readiness** card, on the admin overview and on Admin, + Backups, judges every copy in one verdict each: the database dump (fresh, + failed, or kept nowhere that survives the container), the weekly account + copies, the off-host copies, the last restore drill, whether the bucket is + set up, whether its lifecycle rule ever removes the dumps under + `database/`, and the recovery kit. +- The **recovery kit** under Admin, Encryption lists every key a restore + needs (the table above), by id and fingerprint, and lets an admin check a + saved copy and confirm it. It is due again after any key change. +- An hourly check sends each admin a **Backup health** notification + (`BACKUP_HEALTH`) when a signal on the card turns to needs attention or at + risk, a reminder once a day while one stays at risk, and one more when it + recovers. Each admin chooses the channels for it under Notifications. + +[Getting started](../self-hosting/getting-started.md#save-the-recovery-kit) +walks through saving the kit. + ## How this is tested `scripts/compose-dr-drill.sh` plays this page against real containers: it diff --git a/docs/self-hosting/getting-started.md b/docs/self-hosting/getting-started.md index 0cd793b9a..51ead775a 100644 --- a/docs/self-hosting/getting-started.md +++ b/docs/self-hosting/getting-started.md @@ -295,7 +295,8 @@ key, a re-keyed install. The kit also links the The **Recovery readiness** card on the admin overview and the Backups page judges every backup signal in one place: the database dump, the weekly account copies, the off-host copies, the last restore drill, whether off-host copies -are set up, and the recovery kit. An hourly check sends each admin a +are set up, whether the bucket ever removes old database dumps, and the +recovery kit. An hourly check sends each admin a **Backup health** notification when a signal turns to "needs attention" or "at risk", once a day while it stays at risk, and once when it recovers. Route or switch it off per channel under Notifications. diff --git a/docs/self-hosting/support.md b/docs/self-hosting/support.md index d9e8af841..7f1a34b24 100644 --- a/docs/self-hosting/support.md +++ b/docs/self-hosting/support.md @@ -101,5 +101,6 @@ and host names before it is shown. `docker compose logs app` prints every wide event as one JSON line; `LOG_LEVEL=debug` adds debug events. With `LOKI_ENDPOINT` set the same events -go to Loki, and GlitchTip (Admin, General) receives unexpected server errors. +go to Loki, and GlitchTip (Admin, Integrations, once a DSN is set there) +receives unexpected server errors. Each line carries `request_id`, so an error id finds its line there too.