From ca644647a0153a1fa8d783960b25b9db537b2442 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Marc-Andr=C3=A9=20Bombeck?= <mbombeck@gmail.com>
Date: Sun, 25 Oct 2026 12:52:56 +0100
Subject: [PATCH] docs(ops): describe disaster recovery from the database dump

docs/ops/disaster-recovery.md says what is dumped and where, that local
dumps are unencrypted and why, which keys a restore needs and what each
one costs when missing, how to restore onto a new server or back to the
state before a migration, and how the procedure is tested. The deploy
runbook and the getting-started guide now describe the dump the
entrypoint takes before migrating, and the off-host account copy page
points at the dump for rebuilding a server.
---
 docs/ops/backup-restore.md           |   5 +
 docs/ops/deploy.md                   |  35 ++++--
 docs/ops/disaster-recovery.md        | 164 +++++++++++++++++++++++++++
 docs/self-hosting/getting-started.md |  28 ++++-
 4 files changed, 215 insertions(+), 17 deletions(-)
 create mode 100644 docs/ops/disaster-recovery.md

diff --git a/docs/ops/backup-restore.md b/docs/ops/backup-restore.md
index 09524b2c8..f3d4c0571 100644
--- a/docs/ops/backup-restore.md
+++ b/docs/ops/backup-restore.md
@@ -5,6 +5,11 @@ user's JSON dump, encrypted with AES-256-GCM under a SEPARATE key
 (`BACKUP_ENCRYPTION_KEY`), to any S3-compatible bucket — Cloudflare R2,
 AWS S3, Backblaze B2, MinIO, etc.
 
+These per-account copies roll one account back on the same server. To
+rebuild a server, use the nightly database dump, which the same job
+configuration also uploads (`database/healthlog-<time>-daily.dump.enc`, in the
+envelope below): [`disaster-recovery.md`](disaster-recovery.md).
+
 The backup runs at **02:30 Europe/Berlin** every day from the worker
 container (queue `data-backup-offhost`). Object key layout:
 
diff --git a/docs/ops/deploy.md b/docs/ops/deploy.md
index 0a1f8ff7c..301278cc1 100644
--- a/docs/ops/deploy.md
+++ b/docs/ops/deploy.md
@@ -13,11 +13,13 @@ Migrations land **before** the new app container takes traffic. The
 canonical command sequence on a single-instance host:
 
 ```sh
-# 0. Dump the database first. A migration is not undone by going back to
-#    the previous image; the dump is the way back. See "Updates" in
-#    docs/self-hosting/getting-started.md.
-docker compose exec -T db pg_dump -U healthlog healthlog \
-  | gzip > healthlog-before-update-$(date +%F).sql.gz
+# 0. Nothing to do by hand for the way back: when the new image brings
+#    migrations, its entrypoint first writes
+#    healthlog-<time>-pre-migration.dump into DB_BACKUP_DIR (the `backups`
+#    volume) and keeps the last two. A migration is not undone by going back
+#    to the previous image; that dump is the way back
+#    (docs/ops/disaster-recovery.md, "Going back to before a migration").
+#    Copy it off the host if the host itself is what you are worried about.
 
 # 1. Pull the new image. If you pin a version with HEALTHLOG_IMAGE_REF,
 #    set it to the new one first.
@@ -33,6 +35,14 @@ docker compose run --rm app sh -c 'exit 0'
 docker compose up -d app
 ```
 
+Step 2 takes the pre-migration dump too, in the one-off container, before it
+migrates. When the backups directory is writable and that dump fails (a full
+disk, most often), the entrypoint stops before migrating and says so; free
+the space and run the step again, or set `DB_BACKUP_BEFORE_MIGRATE=false` for
+one start without the dump. Without a writable directory (an install that
+runs the image without the `backups` volume) the log carries a warning, no
+dump is taken, and the migrations run.
+
 Use `docker compose run` rather than a plain `docker run` for step 2. A
 plain `docker run` starts outside the stack's network, where `db` does not
 resolve, and the entrypoint gives up after thirty seconds with "Database
@@ -230,12 +240,15 @@ roll back:
 dev --create-only --name vX_Y_Z_rollback_<col>`), edit the SQL,
    commit, and re-deploy. Never `migrate resolve --rolled-back` against
    production unless you've taken a fresh `pg_dump` first.
-4. Rolling back from v1.40.2 or later to v1.40.1 or earlier: an account
-   that saved the Coach lookback "Last year" reads its whole Coach
-   preference blob as defaults on the older version, metric exclusions
-   included, until the setting is saved again. Ask affected accounts to
-   pick another lookback before the rollback, or re-save their exclusions
-   after it.
+
+When the data itself has to go back to before the update, restore the
+pre-migration dump with the previous image; the steps are in
+[`disaster-recovery.md`](disaster-recovery.md#going-back-to-before-a-migration). 4. Rolling back from v1.40.2 or later to v1.40.1 or earlier: an account
+that saved the Coach lookback "Last year" reads its whole Coach
+preference blob as defaults on the older version, metric exclusions
+included, until the setting is saved again. Ask affected accounts to
+pick another lookback before the rollback, or re-save their exclusions
+after it.
 
 ## Catalog templates
 
diff --git a/docs/ops/disaster-recovery.md b/docs/ops/disaster-recovery.md
new file mode 100644
index 000000000..b4cf5ab89
--- /dev/null
+++ b/docs/ops/disaster-recovery.md
@@ -0,0 +1,164 @@
+# Disaster recovery
+
+The copy that rebuilds a lost server is the database dump. This page says
+what is dumped and where it goes, which keys a restore needs, how to restore
+onto a new server, and how the procedure is tested.
+
+## What is backed up, and where
+
+| Copy                    | When                                                                        | Where                                                               | Kept                            |
+| ----------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------- | ------------------------------- |
+| Nightly dump            | 01:30 Europe/Berlin, the zone every schedule runs in (`DB_BACKUP_SCHEDULE`) | `DB_BACKUP_DIR`, the `backups` volume of the bundled compose file   | the newest 7 (`DB_BACKUP_KEEP`) |
+| Pre-migration dump      | on start, when the new image brings migrations, before them                 | the same directory                                                  | the newest 2                    |
+| Off-host copy of a dump | right after each nightly dump, if the bucket is configured                  | your bucket, under `database/`, sealed with `BACKUP_ENCRYPTION_KEY` | your bucket's lifecycle rule    |
+
+A dump is `pg_dump --format=custom` of the whole database: every account,
+sign-ins, passkeys, sharing grants and managed profiles, documents, the job
+queue and the instance settings. It is two files:
+
+- `healthlog-<UTC time>-<daily|pre-migration>.dump`, the archive;
+- `healthlog-<UTC time>-<daily|pre-migration>.manifest.json` beside it: the
+  app version and build, the newest applied migration, the Postgres version,
+  the archive's size and SHA-256, the row count of every table, and the
+  fingerprints (never the values) of the keys the database needs.
+
+A dump counts as good only when both files exist under their final names. The
+archive is written as `.partial`, read back with `pg_restore --list`, and
+renamed after its manifest; a run that fails leaves nothing that looks good.
+The admin backups page shows the newest good dump, where it is, and the
+newest failed run. The start log says on every boot where dumps go, or that
+none can be kept.
+
+Local dumps are not encrypted. They sit beside the database volume, in the
+same trust zone: whoever can read one volume can read the other. The columns
+HealthLog encrypts stay encrypted inside the dump, under `ENCRYPTION_KEYS`.
+Keep a copy of the dumps off the host; the off-host bucket does that for you,
+and it seals each copy with `BACKUP_ENCRYPTION_KEY` in the envelope the
+per-account copies use.
+
+A dump holds every account that existed when it was taken, including one
+deleted since. Deleted accounts leave the dumps as they age out: after
+`DB_BACKUP_KEEP` nights on the server, and after your lifecycle rule in the
+bucket. Give that rule a prefix that covers `database/` too.
+
+Without a writable `DB_BACKUP_DIR` (an install that runs the image without the
+`backups` volume), the nightly dump is uploaded off-host and nothing is kept
+on the server, and no dump is taken before migrations; with no bucket either,
+no dump is taken at all. The start log and the admin page say so.
+
+The per-account copies (the weekly in-database copy, the nightly per-account
+objects) roll one account back on the same server. They cannot rebuild a
+server; see [`backup-restore.md`](backup-restore.md).
+
+## The keys a restore needs
+
+| Key                                                                 | Without it                                                                                                                              |
+| ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
+| `ENCRYPTION_KEYS` + `ENCRYPTION_ACTIVE_KEY_ID`, or `ENCRYPTION_KEY` | Nothing encrypted can be read. The restore refuses when the active key is missing or different. Keep every retired id still in the map. |
+| `API_TOKEN_HMAC_KEY`                                                | No data is lost, but every session, app token and share link stops working; people sign in again and re-create tokens.                  |
+| `BACKUP_ENCRYPTION_KEY` (and `BACKUP_ENCRYPTION_PREVIOUS_KEYS`)     | Off-host copies do not open.                                                                                                            |
+| `POSTGRES_PASSWORD`                                                 | Not needed: a new server can use a new one.                                                                                             |
+
+The restore compares the keys in its environment with the fingerprints in the
+manifest before it writes anything, and names each key that is missing or
+different.
+
+## Restore onto a new server
+
+1. Install Docker and take the `docker-compose.yml` of the version that wrote
+   the dump or a newer one. The manifest's `appVersion` names it; to run that
+   exact version set `HEALTHLOG_IMAGE_REF=:v<version>` in `.env`.
+2. Write `.env` with the keys from your records (see above) and a new
+   `POSTGRES_PASSWORD`.
+3. Do not run `docker compose up` yet. The app would create an empty schema,
+   and the restore only writes into a database with no tables. Start only the
+   database:
+
+   ```bash
+   docker compose up -d db
+   ```
+
+4. Put the dump and its manifest in a directory, for example `./restore`, and
+   make them readable for the container (`chmod a+r restore/*`). From the
+   bucket, download both `database/<name>.dump.enc` and
+   `database/<name>.manifest.json.enc`; the restore opens them with
+   `BACKUP_ENCRYPTION_KEY`.
+
+   ```bash
+   docker compose run --rm -v "$PWD/restore:/restore:ro" db-restore list /restore
+   docker compose run --rm -v "$PWD/restore:/restore:ro" db-restore \
+     /restore/healthlog-20261024T233000Z-daily.dump
+   ```
+
+   It checks the archive against the manifest's SHA-256, the keys against
+   the manifest's fingerprints, and that the database is empty. Then it
+   restores in one transaction, gathers planner statistics and counts every
+   table against the manifest. A count that differs empties the database
+   again and fails. Wait for the line that starts with `Restored`.
+
+5. Start the stack and check it:
+
+   ```bash
+   docker compose up -d
+   curl -s http://localhost:3000/api/health
+   ```
+
+   Sign in. The app's own key check runs at start and refuses to serve on a
+   key that does not open the data.
+
+When the dumps are still in the server's `backups` volume (the database
+volume was lost, the backups volume was not), step 4 is
+`docker compose run --rm db-restore latest`.
+
+`--ignore-key-mismatch` restores although the active encryption key differs.
+Use it only when that key is lost for good: everything encrypted stays
+unreadable.
+
+## Going back to before a migration
+
+The entrypoint dumps the database before it applies migrations. To return to
+that state (the previous image, the data as it was before the update):
+
+```bash
+docker compose stop app
+# Keep the current state too, in case it is wanted later.
+docker compose exec -T db pg_dump -U healthlog -Fc healthlog > healthlog-current.dump
+docker compose exec db dropdb -U healthlog healthlog
+docker compose exec db createdb -U healthlog healthlog
+docker compose run --rm db-restore list
+docker compose run --rm db-restore healthlog-<time>-pre-migration.dump
+# Set HEALTHLOG_IMAGE_REF in .env to the version you ran before the update.
+docker compose up -d
+```
+
+The dump before a migration is taken while the new image starts. If it fails
+on a writable directory (a full disk, most often), the start stops before
+anything is migrated and the log names the switch: `DB_BACKUP_BEFORE_MIGRATE=false`
+starts without it, once.
+
+## How this is tested
+
+`scripts/compose-dr-drill.sh` plays this page against real containers: it
+builds the image, seeds several accounts (the demo account with documents
+and encrypted notes, a Guardian with a managed profile), lets the scheduled
+job take the dump, destroys every volume, restores with
+`docker compose run --rm db-restore latest` on a new stack given only the dump
+and the keys, counts every table against the manifest and boots the app:
+health, sign-in, an encrypted note read back, a document opened, the Guardian
+still seeing the profile. It runs in CI nightly and on every change to the
+compose file, the image or the dump and restore code
+(`.github/workflows/compose-dr-drill.yml`), and it is checked to fail on a
+wrong key, a dump that lost a table's rows and a target that is not empty.
+
+Test your own copies too: once, and after any change to how the server is
+run, restore last night's dump on a spare machine with the steps above.
+
+## Settings
+
+| Variable                   | Default                      | What it does                                                                     |
+| -------------------------- | ---------------------------- | -------------------------------------------------------------------------------- |
+| `DB_BACKUP_ENABLED`        | on                           | `false` stops the nightly dump.                                                  |
+| `DB_BACKUP_DIR`            | `/var/lib/healthlog/backups` | Where dumps are written and read. Change it only together with the volume mount. |
+| `DB_BACKUP_SCHEDULE`       | `30 1 * * *`                 | Five-field cron, Europe/Berlin time like every HealthLog schedule.               |
+| `DB_BACKUP_KEEP`           | `7`                          | Nightly dumps kept (at least 1). Two pre-migration dumps are kept besides.       |
+| `DB_BACKUP_BEFORE_MIGRATE` | on                           | `false` starts an update without the dump before its migrations.                 |
diff --git a/docs/self-hosting/getting-started.md b/docs/self-hosting/getting-started.md
index f4fb5d208..b80e4df31 100644
--- a/docs/self-hosting/getting-started.md
+++ b/docs/self-hosting/getting-started.md
@@ -346,6 +346,15 @@ written under that key, so remove the setting once the key is confirmed.
 Your database is the only stateful piece — back it up, and back up the
 key that decrypts it.
 
+**Database dumps the app takes itself.** Every night, and before an update
+applies migrations, the app writes a `pg_dump` of the whole database with a
+manifest into the `backups` volume and keeps the newest few. With the
+off-host bucket configured, each nightly dump is also uploaded, encrypted.
+`docker compose run --rm db-restore latest` restores one into an empty
+database. What is kept where, the keys a restore needs, and restoring onto a
+new server: [`docs/ops/disaster-recovery.md`](../ops/disaster-recovery.md).
+Keep a copy of those dumps off this host.
+
 **Local snapshot (the common single-box case).** Dump the `db` service
 with `pg_dump`:
 
@@ -379,8 +388,11 @@ encryption key safely (keep the old key until zero rows remain on it) see
 A new image can bring database migrations, and `docker-entrypoint.sh`
 applies them on the first start of that image. Most are additive, but some
 rewrite data, and going back to the previous image does not undo them. The
-only way back from a migration that went wrong is the dump you took before
-it. So take one before every pull:
+way back from a migration that went wrong is a dump taken before it. The
+entrypoint takes one itself (`healthlog-<time>-pre-migration.dump` in the
+`backups` volume, see
+[`docs/ops/disaster-recovery.md`](../ops/disaster-recovery.md)); a dump of
+your own before every pull, kept off the host, is the second copy:
 
 ```bash
 # 1. Dump the database (same command as under "Backup and restore").
@@ -405,14 +417,18 @@ the `image:` line, a new major (16 to 18) is a dump and restore, described in
 The start log says what is about to happen before the migrations run:
 
 ```text
-HealthLog: 2 database migrations pending. Back up the database before updating (pg_dump, see the Updates section of docs/self-hosting/getting-started.md); a migration is not undone by going back to the previous image.
+HealthLog: 2 database migrations pending. Unless DB_BACKUP_BEFORE_MIGRATE=false, a pg_dump of the database is written to DB_BACKUP_DIR before they run (docs/ops/disaster-recovery.md); a migration is not undone by going back to the previous image.
+HealthLog: taking a database dump before 2 pending migration(s) into /var/lib/healthlog/backups...
+HealthLog: pre-migration dump written: healthlog-20261025T081502Z-pre-migration.dump (212.4 MB, 231 tables, 38 s).
 HealthLog: Running database migrations...
 ```
 
 or `HealthLog: Database schema is up to date.` when the image brings none.
-The line is a notice and does not pause the start: the migrations run right
-after it. So the dump belongs before step 2, not after you see the line; by
-then the database is already being migrated.
+The migrations run right after the dump. A dump that fails on a writable
+directory stops the start before anything is migrated, and the log names the
+switch that skips it once (`DB_BACKUP_BEFORE_MIGRATE=false`). Your own dump
+belongs before step 2, not after you see these lines; by then the database is
+already being migrated.
 
 If an update to v1.43 stops at migration `0409_drop_plaintext_location`,
 nothing was changed; the recovery is in