docs(ops): rollback list, how admins hear about backups, GlitchTip's place

The fourth rollback step had been joined onto the paragraph after the
list; it is a list item again, with the way back to the data after it.
The disaster recovery page now says how the server tells admins about
its copies: the recovery readiness card, the recovery kit and the hourly
Backup health notification. The support page put GlitchTip under Admin,
General; the card is under Admin, Integrations, as the deploy runbook
says. The pre-migration dump notes say that without the backups volume
no dump is taken, also where the image's own directory is writable.
This commit is contained in:
Marc-André Bombeck committed 2026-10-25 16:01:46 +01:00
1 parent c7040ccdaf
commit 430468932c
4 files changed
+44 -18

No files matched your search

+17 -15
View File
@@ -36,12 +36,13 @@ docker compose up -d app
```
Step 2 takes the pre-migration dump too, in the one-off container, before it
migrates. When the backups directory is writable and that dump fails (a full
disk, most often), the entrypoint stops before migrating and says so; free
the space and run the step again, or set `DB_BACKUP_BEFORE_MIGRATE=false` for
one start without the dump. Without a writable directory (an install that
runs the image without the `backups` volume) the log carries a warning, no
dump is taken, and the migrations run.
migrates. When the backups directory is on its volume and that dump fails (a
full disk, most often), the entrypoint stops before migrating and says so;
free the space and run the step again, or set `DB_BACKUP_BEFORE_MIGRATE=false`
for one start without the dump. Without the `backups` volume the directory
is not writable, or it is the image's own and sits inside the container that
the update replaces; either way the log carries a warning, no dump is taken,
and the migrations run.
Use `docker compose run` rather than a plain `docker run` for step 2. A
plain `docker run` starts outside the stack's network, where `db` does not
@@ -236,19 +237,20 @@ roll back:
app version reads forward-compatible-ish shapes for nearly every
release pair shipped on the v1.4.x line.
3. If a column-drop migration ships in the rollback target, hand-roll
the column back via a fresh forward migration (`pnpm prisma migrate
dev --create-only --name vX_Y_Z_rollback_<col>`), edit the SQL,
commit, and re-deploy. Never `migrate resolve --rolled-back` against
the column back via a fresh forward migration
(`pnpm prisma migrate dev --create-only --name vX_Y_Z_rollback_<col>`),
edit the SQL, commit, and re-deploy. Never `migrate resolve --rolled-back` against
production unless you've taken a fresh `pg_dump` first.
4. Rolling back from v1.40.2 or later to v1.40.1 or earlier: an account
that saved the Coach lookback "Last year" reads its whole Coach
preference blob as defaults on the older version, metric exclusions
included, until the setting is saved again. Ask affected accounts to
pick another lookback before the rollback, or re-save their exclusions
after it.
When the data itself has to go back to before the update, restore the
pre-migration dump with the previous image; the steps are in
[`disaster-recovery.md`](disaster-recovery.md#going-back-to-before-a-migration). 4. Rolling back from v1.40.2 or later to v1.40.1 or earlier: an account
that saved the Coach lookback "Last year" reads its whole Coach
preference blob as defaults on the older version, metric exclusions
included, until the setting is saved again. Ask affected accounts to
pick another lookback before the rollback, or re-save their exclusions
after it.
[`disaster-recovery.md`](disaster-recovery.md#going-back-to-before-a-migration).
## Catalog templates
+23 -1
View File
@@ -157,10 +157,32 @@ docker compose up -d
```
The dump before a migration is taken while the new image starts. If it fails
on a writable directory (a full disk, most often), the start stops before
on a directory that keeps it (a full disk, most often), the start stops before
anything is migrated and the log names the switch: `DB_BACKUP_BEFORE_MIGRATE=false`
starts without it, once.
## Knowing the copies are there
Nobody opens the backups page until the day it matters, so the server tells
the admins instead:
- The **Recovery readiness** card, on the admin overview and on Admin,
Backups, judges every copy in one verdict each: the database dump (fresh,
failed, or kept nowhere that survives the container), the weekly account
copies, the off-host copies, the last restore drill, whether the bucket is
set up, whether its lifecycle rule ever removes the dumps under
`database/`, and the recovery kit.
- The **recovery kit** under Admin, Encryption lists every key a restore
needs (the table above), by id and fingerprint, and lets an admin check a
saved copy and confirm it. It is due again after any key change.
- An hourly check sends each admin a **Backup health** notification
(`BACKUP_HEALTH`) when a signal on the card turns to needs attention or at
risk, a reminder once a day while one stays at risk, and one more when it
recovers. Each admin chooses the channels for it under Notifications.
[Getting started](../self-hosting/getting-started.md#save-the-recovery-kit)
walks through saving the kit.
## How this is tested
`scripts/compose-dr-drill.sh` plays this page against real containers: it
+2 -1
View File
@@ -295,7 +295,8 @@ key, a re-keyed install. The kit also links the
The **Recovery readiness** card on the admin overview and the Backups page
judges every backup signal in one place: the database dump, the weekly account
copies, the off-host copies, the last restore drill, whether off-host copies
are set up, and the recovery kit. An hourly check sends each admin a
are set up, whether the bucket ever removes old database dumps, and the
recovery kit. An hourly check sends each admin a
**Backup health** notification when a signal turns to "needs attention" or "at
risk", once a day while it stays at risk, and once when it recovers. Route or
switch it off per channel under Notifications.
+2 -1
View File
@@ -101,5 +101,6 @@ and host names before it is shown.
`docker compose logs app` prints every wide event as one JSON line;
`LOG_LEVEL=debug` adds debug events. With `LOKI_ENDPOINT` set the same events
go to Loki, and GlitchTip (Admin, General) receives unexpected server errors.
go to Loki, and GlitchTip (Admin, Integrations, once a DSN is set there)
receives unexpected server errors.
Each line carries `request_id`, so an error id finds its line there too.