mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 20:42:43 +00:00
docs(ops): rollback list, how admins hear about backups, GlitchTip's place
The fourth rollback step had been joined onto the paragraph after the list; it is a list item again, with the way back to the data after it. The disaster recovery page now says how the server tells admins about its copies: the recovery readiness card, the recovery kit and the hourly Backup health notification. The support page put GlitchTip under Admin, General; the card is under Admin, Integrations, as the deploy runbook says. The pre-migration dump notes say that without the backups volume no dump is taken, also where the image's own directory is writable.
This commit is contained in:
1 parent
c7040ccdaf
commit
430468932c
4 files changed
+44
-18
No files matched your search
+17
-15
@@ -36,12 +36,13 @@ docker compose up -d app
|
||||
```
|
||||
|
||||
Step 2 takes the pre-migration dump too, in the one-off container, before it
|
||||
migrates. When the backups directory is writable and that dump fails (a full
|
||||
disk, most often), the entrypoint stops before migrating and says so; free
|
||||
the space and run the step again, or set `DB_BACKUP_BEFORE_MIGRATE=false` for
|
||||
one start without the dump. Without a writable directory (an install that
|
||||
runs the image without the `backups` volume) the log carries a warning, no
|
||||
dump is taken, and the migrations run.
|
||||
migrates. When the backups directory is on its volume and that dump fails (a
|
||||
full disk, most often), the entrypoint stops before migrating and says so;
|
||||
free the space and run the step again, or set `DB_BACKUP_BEFORE_MIGRATE=false`
|
||||
for one start without the dump. Without the `backups` volume the directory
|
||||
is not writable, or it is the image's own and sits inside the container that
|
||||
the update replaces; either way the log carries a warning, no dump is taken,
|
||||
and the migrations run.
|
||||
|
||||
Use `docker compose run` rather than a plain `docker run` for step 2. A
|
||||
plain `docker run` starts outside the stack's network, where `db` does not
|
||||
@@ -236,19 +237,20 @@ roll back:
|
||||
app version reads forward-compatible-ish shapes for nearly every
|
||||
release pair shipped on the v1.4.x line.
|
||||
3. If a column-drop migration ships in the rollback target, hand-roll
|
||||
the column back via a fresh forward migration (`pnpm prisma migrate
|
||||
dev --create-only --name vX_Y_Z_rollback_<col>`), edit the SQL,
|
||||
commit, and re-deploy. Never `migrate resolve --rolled-back` against
|
||||
the column back via a fresh forward migration
|
||||
(`pnpm prisma migrate dev --create-only --name vX_Y_Z_rollback_<col>`),
|
||||
edit the SQL, commit, and re-deploy. Never `migrate resolve --rolled-back` against
|
||||
production unless you've taken a fresh `pg_dump` first.
|
||||
4. Rolling back from v1.40.2 or later to v1.40.1 or earlier: an account
|
||||
that saved the Coach lookback "Last year" reads its whole Coach
|
||||
preference blob as defaults on the older version, metric exclusions
|
||||
included, until the setting is saved again. Ask affected accounts to
|
||||
pick another lookback before the rollback, or re-save their exclusions
|
||||
after it.
|
||||
|
||||
When the data itself has to go back to before the update, restore the
|
||||
pre-migration dump with the previous image; the steps are in
|
||||
[`disaster-recovery.md`](disaster-recovery.md#going-back-to-before-a-migration). 4. Rolling back from v1.40.2 or later to v1.40.1 or earlier: an account
|
||||
that saved the Coach lookback "Last year" reads its whole Coach
|
||||
preference blob as defaults on the older version, metric exclusions
|
||||
included, until the setting is saved again. Ask affected accounts to
|
||||
pick another lookback before the rollback, or re-save their exclusions
|
||||
after it.
|
||||
[`disaster-recovery.md`](disaster-recovery.md#going-back-to-before-a-migration).
|
||||
|
||||
## Catalog templates
|
||||
|
||||
|
||||
@@ -157,10 +157,32 @@ docker compose up -d
|
||||
```
|
||||
|
||||
The dump before a migration is taken while the new image starts. If it fails
|
||||
on a writable directory (a full disk, most often), the start stops before
|
||||
on a directory that keeps it (a full disk, most often), the start stops before
|
||||
anything is migrated and the log names the switch: `DB_BACKUP_BEFORE_MIGRATE=false`
|
||||
starts without it, once.
|
||||
|
||||
## Knowing the copies are there
|
||||
|
||||
Nobody opens the backups page until the day it matters, so the server tells
|
||||
the admins instead:
|
||||
|
||||
- The **Recovery readiness** card, on the admin overview and on Admin,
|
||||
Backups, judges every copy in one verdict each: the database dump (fresh,
|
||||
failed, or kept nowhere that survives the container), the weekly account
|
||||
copies, the off-host copies, the last restore drill, whether the bucket is
|
||||
set up, whether its lifecycle rule ever removes the dumps under
|
||||
`database/`, and the recovery kit.
|
||||
- The **recovery kit** under Admin, Encryption lists every key a restore
|
||||
needs (the table above), by id and fingerprint, and lets an admin check a
|
||||
saved copy and confirm it. It is due again after any key change.
|
||||
- An hourly check sends each admin a **Backup health** notification
|
||||
(`BACKUP_HEALTH`) when a signal on the card turns to needs attention or at
|
||||
risk, a reminder once a day while one stays at risk, and one more when it
|
||||
recovers. Each admin chooses the channels for it under Notifications.
|
||||
|
||||
[Getting started](../self-hosting/getting-started.md#save-the-recovery-kit)
|
||||
walks through saving the kit.
|
||||
|
||||
## How this is tested
|
||||
|
||||
`scripts/compose-dr-drill.sh` plays this page against real containers: it
|
||||
|
||||
@@ -295,7 +295,8 @@ key, a re-keyed install. The kit also links the
|
||||
The **Recovery readiness** card on the admin overview and the Backups page
|
||||
judges every backup signal in one place: the database dump, the weekly account
|
||||
copies, the off-host copies, the last restore drill, whether off-host copies
|
||||
are set up, and the recovery kit. An hourly check sends each admin a
|
||||
are set up, whether the bucket ever removes old database dumps, and the
|
||||
recovery kit. An hourly check sends each admin a
|
||||
**Backup health** notification when a signal turns to "needs attention" or "at
|
||||
risk", once a day while it stays at risk, and once when it recovers. Route or
|
||||
switch it off per channel under Notifications.
|
||||
|
||||
@@ -101,5 +101,6 @@ and host names before it is shown.
|
||||
|
||||
`docker compose logs app` prints every wide event as one JSON line;
|
||||
`LOG_LEVEL=debug` adds debug events. With `LOKI_ENDPOINT` set the same events
|
||||
go to Loki, and GlitchTip (Admin, General) receives unexpected server errors.
|
||||
go to Loki, and GlitchTip (Admin, Integrations, once a DSN is set there)
|
||||
receives unexpected server errors.
|
||||
Each line carries `request_id`, so an error id finds its line there too.
|
||||
Reference in new issue
Block a user