tom/HealthLog
1
0 Fork 0
mirror of https://github.com/MBombeck/HealthLog.git synced 2026-11-10 13:22:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(ops): describe disaster recovery from the database dump

Browse Source 
docs/ops/disaster-recovery.md says what is dumped and where, that local
dumps are unencrypted and why, which keys a restore needs and what each
one costs when missing, how to restore onto a new server or back to the
state before a migration, and how the procedure is tested. The deploy
runbook and the getting-started guide now describe the dump the
entrypoint takes before migrating, and the off-host account copy page
points at the dump for rebuilding a server.
This commit is contained in:
Marc-André Bombeck committed 2026-10-25 12:56:33 +01:00
1 parent 3646502a86
commit ca644647a0
4 files changed
+215 -17
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/ops/backup-restore.md
+5
View File
@@ -5,6 +5,11 @@ user's JSON dump, encrypted with AES-256-GCM under a SEPARATE key
(`BACKUP_ENCRYPTION_KEY`), to any S3-compatible bucket — Cloudflare R2,
AWS S3, Backblaze B2, MinIO, etc.
These per-account copies roll one account back on the same server. To
rebuild a server, use the nightly database dump, which the same job
configuration also uploads (`database/healthlog-<time>-daily.dump.enc`, in the
envelope below): [`disaster-recovery.md`](disaster-recovery.md).
The backup runs at **02:30 Europe/Berlin** every day from the worker
container (queue `data-backup-offhost`). Object key layout:
docs/ops/deploy.md
+24 -11
View File
@@ -13,11 +13,13 @@ Migrations land **before** the new app container takes traffic. The
canonical command sequence on a single-instance host:
```sh
# 0. Dump the database first. A migration is not undone by going back to
# the previous image; the dump is the way back. See "Updates" in
# docs/self-hosting/getting-started.md.
docker compose exec -T db pg_dump -U healthlog healthlog \
| gzip > healthlog-before-update-$(date +%F).sql.gz
# 0. Nothing to do by hand for the way back: when the new image brings
# migrations, its entrypoint first writes
# healthlog-<time>-pre-migration.dump into DB_BACKUP_DIR (the `backups`
# volume) and keeps the last two. A migration is not undone by going back
# to the previous image; that dump is the way back
# (docs/ops/disaster-recovery.md, "Going back to before a migration").
# Copy it off the host if the host itself is what you are worried about.
# 1. Pull the new image. If you pin a version with HEALTHLOG_IMAGE_REF,
# set it to the new one first.
@@ -33,6 +35,14 @@ docker compose run --rm app sh -c 'exit 0'
docker compose up -d app
```
Step 2 takes the pre-migration dump too, in the one-off container, before it
migrates. When the backups directory is writable and that dump fails (a full
disk, most often), the entrypoint stops before migrating and says so; free
the space and run the step again, or set `DB_BACKUP_BEFORE_MIGRATE=false` for
one start without the dump. Without a writable directory (an install that
runs the image without the `backups` volume) the log carries a warning, no
dump is taken, and the migrations run.
Use `docker compose run` rather than a plain `docker run` for step 2. A
plain `docker run` starts outside the stack's network, where `db` does not
resolve, and the entrypoint gives up after thirty seconds with "Database
@@ -230,12 +240,15 @@ roll back:
dev --create-only --name vX_Y_Z_rollback_<col>`), edit the SQL,
commit, and re-deploy. Never `migrate resolve --rolled-back` against
production unless you've taken a fresh `pg_dump` first.
4. Rolling back from v1.40.2 or later to v1.40.1 or earlier: an account
that saved the Coach lookback "Last year" reads its whole Coach
preference blob as defaults on the older version, metric exclusions
included, until the setting is saved again. Ask affected accounts to
pick another lookback before the rollback, or re-save their exclusions
after it.
When the data itself has to go back to before the update, restore the
pre-migration dump with the previous image; the steps are in
[`disaster-recovery.md`](disaster-recovery.md#going-back-to-before-a-migration). 4. Rolling back from v1.40.2 or later to v1.40.1 or earlier: an account
that saved the Coach lookback "Last year" reads its whole Coach
preference blob as defaults on the older version, metric exclusions
included, until the setting is saved again. Ask affected accounts to
pick another lookback before the rollback, or re-save their exclusions
after it.
## Catalog templates
docs/ops/disaster-recovery.md
+164
View File
@@ -0,0 +1,164 @@
# Disaster recovery
The copy that rebuilds a lost server is the database dump. This page says
what is dumped and where it goes, which keys a restore needs, how to restore
onto a new server, and how the procedure is tested.
## What is backed up, and where
| Copy | When | Where | Kept |
| ----------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------- | ------------------------------- |
| Nightly dump | 01:30 Europe/Berlin, the zone every schedule runs in (`DB_BACKUP_SCHEDULE`) | `DB_BACKUP_DIR`, the `backups` volume of the bundled compose file | the newest 7 (`DB_BACKUP_KEEP`) |
| Pre-migration dump | on start, when the new image brings migrations, before them | the same directory | the newest 2 |
| Off-host copy of a dump | right after each nightly dump, if the bucket is configured | your bucket, under `database/`, sealed with `BACKUP_ENCRYPTION_KEY` | your bucket's lifecycle rule |
A dump is `pg_dump --format=custom` of the whole database: every account,
sign-ins, passkeys, sharing grants and managed profiles, documents, the job
queue and the instance settings. It is two files:
- `healthlog-<UTC time>-<daily|pre-migration>.dump`, the archive;
- `healthlog-<UTC time>-<daily|pre-migration>.manifest.json` beside it: the
app version and build, the newest applied migration, the Postgres version,
the archive's size and SHA-256, the row count of every table, and the
fingerprints (never the values) of the keys the database needs.
A dump counts as good only when both files exist under their final names. The
archive is written as `.partial`, read back with `pg_restore --list`, and
renamed after its manifest; a run that fails leaves nothing that looks good.
The admin backups page shows the newest good dump, where it is, and the
newest failed run. The start log says on every boot where dumps go, or that
none can be kept.
Local dumps are not encrypted. They sit beside the database volume, in the
same trust zone: whoever can read one volume can read the other. The columns
HealthLog encrypts stay encrypted inside the dump, under `ENCRYPTION_KEYS`.
Keep a copy of the dumps off the host; the off-host bucket does that for you,
and it seals each copy with `BACKUP_ENCRYPTION_KEY` in the envelope the
per-account copies use.
A dump holds every account that existed when it was taken, including one
deleted since. Deleted accounts leave the dumps as they age out: after
`DB_BACKUP_KEEP` nights on the server, and after your lifecycle rule in the
bucket. Give that rule a prefix that covers `database/` too.
Without a writable `DB_BACKUP_DIR` (an install that runs the image without the
`backups` volume), the nightly dump is uploaded off-host and nothing is kept
on the server, and no dump is taken before migrations; with no bucket either,
no dump is taken at all. The start log and the admin page say so.
The per-account copies (the weekly in-database copy, the nightly per-account
objects) roll one account back on the same server. They cannot rebuild a
server; see [`backup-restore.md`](backup-restore.md).
## The keys a restore needs
| Key | Without it |
| ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `ENCRYPTION_KEYS` + `ENCRYPTION_ACTIVE_KEY_ID`, or `ENCRYPTION_KEY` | Nothing encrypted can be read. The restore refuses when the active key is missing or different. Keep every retired id still in the map. |
| `API_TOKEN_HMAC_KEY` | No data is lost, but every session, app token and share link stops working; people sign in again and re-create tokens. |
| `BACKUP_ENCRYPTION_KEY` (and `BACKUP_ENCRYPTION_PREVIOUS_KEYS`) | Off-host copies do not open. |
| `POSTGRES_PASSWORD` | Not needed: a new server can use a new one. |
The restore compares the keys in its environment with the fingerprints in the
manifest before it writes anything, and names each key that is missing or
different.
## Restore onto a new server
1. Install Docker and take the `docker-compose.yml` of the version that wrote
the dump or a newer one. The manifest's `appVersion` names it; to run that
exact version set `HEALTHLOG_IMAGE_REF=:v<version>` in `.env`.
2. Write `.env` with the keys from your records (see above) and a new
`POSTGRES_PASSWORD`.
3. Do not run `docker compose up` yet. The app would create an empty schema,
and the restore only writes into a database with no tables. Start only the
database:
```bash
docker compose up -d db
```
4. Put the dump and its manifest in a directory, for example `./restore`, and
make them readable for the container (`chmod a+r restore/*`). From the
bucket, download both `database/<name>.dump.enc` and
`database/<name>.manifest.json.enc`; the restore opens them with
`BACKUP_ENCRYPTION_KEY`.
```bash
docker compose run --rm -v "$PWD/restore:/restore:ro" db-restore list /restore
docker compose run --rm -v "$PWD/restore:/restore:ro" db-restore \
/restore/healthlog-20261024T233000Z-daily.dump
```
It checks the archive against the manifest's SHA-256, the keys against
the manifest's fingerprints, and that the database is empty. Then it
restores in one transaction, gathers planner statistics and counts every
table against the manifest. A count that differs empties the database
again and fails. Wait for the line that starts with `Restored`.
5. Start the stack and check it:
```bash
docker compose up -d
curl -s http://localhost:3000/api/health
```
Sign in. The app's own key check runs at start and refuses to serve on a
key that does not open the data.
When the dumps are still in the server's `backups` volume (the database
volume was lost, the backups volume was not), step 4 is
`docker compose run --rm db-restore latest`.
`--ignore-key-mismatch` restores although the active encryption key differs.
Use it only when that key is lost for good: everything encrypted stays
unreadable.
## Going back to before a migration
The entrypoint dumps the database before it applies migrations. To return to
that state (the previous image, the data as it was before the update):
```bash
docker compose stop app
# Keep the current state too, in case it is wanted later.
docker compose exec -T db pg_dump -U healthlog -Fc healthlog > healthlog-current.dump
docker compose exec db dropdb -U healthlog healthlog
docker compose exec db createdb -U healthlog healthlog
docker compose run --rm db-restore list
docker compose run --rm db-restore healthlog-<time>-pre-migration.dump
# Set HEALTHLOG_IMAGE_REF in .env to the version you ran before the update.
docker compose up -d
```
The dump before a migration is taken while the new image starts. If it fails
on a writable directory (a full disk, most often), the start stops before
anything is migrated and the log names the switch: `DB_BACKUP_BEFORE_MIGRATE=false`
starts without it, once.
## How this is tested
`scripts/compose-dr-drill.sh` plays this page against real containers: it
builds the image, seeds several accounts (the demo account with documents
and encrypted notes, a Guardian with a managed profile), lets the scheduled
job take the dump, destroys every volume, restores with
`docker compose run --rm db-restore latest` on a new stack given only the dump
and the keys, counts every table against the manifest and boots the app:
health, sign-in, an encrypted note read back, a document opened, the Guardian
still seeing the profile. It runs in CI nightly and on every change to the
compose file, the image or the dump and restore code
(`.github/workflows/compose-dr-drill.yml`), and it is checked to fail on a
wrong key, a dump that lost a table's rows and a target that is not empty.
Test your own copies too: once, and after any change to how the server is
run, restore last night's dump on a spare machine with the steps above.
## Settings
| Variable | Default | What it does |
| -------------------------- | ---------------------------- | -------------------------------------------------------------------------------- |
| `DB_BACKUP_ENABLED` | on | `false` stops the nightly dump. |
| `DB_BACKUP_DIR` | `/var/lib/healthlog/backups` | Where dumps are written and read. Change it only together with the volume mount. |
| `DB_BACKUP_SCHEDULE` | `30 1 * * *` | Five-field cron, Europe/Berlin time like every HealthLog schedule. |
| `DB_BACKUP_KEEP` | `7` | Nightly dumps kept (at least 1). Two pre-migration dumps are kept besides. |
| `DB_BACKUP_BEFORE_MIGRATE` | on | `false` starts an update without the dump before its migrations. |
docs/self-hosting/getting-started.md
+22 -6
View File
@@ -346,6 +346,15 @@ written under that key, so remove the setting once the key is confirmed.
Your database is the only stateful piece — back it up, and back up the
key that decrypts it.
**Database dumps the app takes itself.** Every night, and before an update
applies migrations, the app writes a `pg_dump` of the whole database with a
manifest into the `backups` volume and keeps the newest few. With the
off-host bucket configured, each nightly dump is also uploaded, encrypted.
`docker compose run --rm db-restore latest` restores one into an empty
database. What is kept where, the keys a restore needs, and restoring onto a
new server: [`docs/ops/disaster-recovery.md`](../ops/disaster-recovery.md).
Keep a copy of those dumps off this host.
**Local snapshot (the common single-box case).** Dump the `db` service
with `pg_dump`:
@@ -379,8 +388,11 @@ encryption key safely (keep the old key until zero rows remain on it) see
A new image can bring database migrations, and `docker-entrypoint.sh`
applies them on the first start of that image. Most are additive, but some
rewrite data, and going back to the previous image does not undo them. The
only way back from a migration that went wrong is the dump you took before
it. So take one before every pull:
way back from a migration that went wrong is a dump taken before it. The
entrypoint takes one itself (`healthlog-<time>-pre-migration.dump` in the
`backups` volume, see
[`docs/ops/disaster-recovery.md`](../ops/disaster-recovery.md)); a dump of
your own before every pull, kept off the host, is the second copy:
```bash
# 1. Dump the database (same command as under "Backup and restore").
@@ -405,14 +417,18 @@ the `image:` line, a new major (16 to 18) is a dump and restore, described in
The start log says what is about to happen before the migrations run:
```text
HealthLog: 2 database migrations pending. Back up the database before updating (pg_dump, see the Updates section of docs/self-hosting/getting-started.md); a migration is not undone by going back to the previous image.
HealthLog: 2 database migrations pending. Unless DB_BACKUP_BEFORE_MIGRATE=false, a pg_dump of the database is written to DB_BACKUP_DIR before they run (docs/ops/disaster-recovery.md); a migration is not undone by going back to the previous image.
HealthLog: taking a database dump before 2 pending migration(s) into /var/lib/healthlog/backups...
HealthLog: pre-migration dump written: healthlog-20261025T081502Z-pre-migration.dump (212.4 MB, 231 tables, 38 s).
HealthLog: Running database migrations...
```
or `HealthLog: Database schema is up to date.` when the image brings none.
The line is a notice and does not pause the start: the migrations run right
after it. So the dump belongs before step 2, not after you see the line; by
then the database is already being migrated.
The migrations run right after the dump. A dump that fails on a writable
directory stops the start before anything is migrated, and the log names the
switch that skips it once (`DB_BACKUP_BEFORE_MIGRATE=false`). Your own dump
belongs before step 2, not after you see these lines; by then the database is
already being migrated.
If an update to v1.43 stops at migration `0409_drop_plaintext_location`,
nothing was changed; the recovery is in