Files
HealthLog/docs/ops/disaster-recovery.md
T
Marc-André Bombeck ca644647a0 docs(ops): describe disaster recovery from the database dump
docs/ops/disaster-recovery.md says what is dumped and where, that local
dumps are unencrypted and why, which keys a restore needs and what each
one costs when missing, how to restore onto a new server or back to the
state before a migration, and how the procedure is tested. The deploy
runbook and the getting-started guide now describe the dump the
entrypoint takes before migrating, and the off-host account copy page
points at the dump for rebuilding a server.
2026-10-25 12:56:33 +01:00

9.6 KiB

Disaster recovery

The copy that rebuilds a lost server is the database dump. This page says what is dumped and where it goes, which keys a restore needs, how to restore onto a new server, and how the procedure is tested.

What is backed up, and where

Copy When Where Kept
Nightly dump 01:30 Europe/Berlin, the zone every schedule runs in (DB_BACKUP_SCHEDULE) DB_BACKUP_DIR, the backups volume of the bundled compose file the newest 7 (DB_BACKUP_KEEP)
Pre-migration dump on start, when the new image brings migrations, before them the same directory the newest 2
Off-host copy of a dump right after each nightly dump, if the bucket is configured your bucket, under database/, sealed with BACKUP_ENCRYPTION_KEY your bucket's lifecycle rule

A dump is pg_dump --format=custom of the whole database: every account, sign-ins, passkeys, sharing grants and managed profiles, documents, the job queue and the instance settings. It is two files:

  • healthlog-<UTC time>-<daily|pre-migration>.dump, the archive;
  • healthlog-<UTC time>-<daily|pre-migration>.manifest.json beside it: the app version and build, the newest applied migration, the Postgres version, the archive's size and SHA-256, the row count of every table, and the fingerprints (never the values) of the keys the database needs.

A dump counts as good only when both files exist under their final names. The archive is written as .partial, read back with pg_restore --list, and renamed after its manifest; a run that fails leaves nothing that looks good. The admin backups page shows the newest good dump, where it is, and the newest failed run. The start log says on every boot where dumps go, or that none can be kept.

Local dumps are not encrypted. They sit beside the database volume, in the same trust zone: whoever can read one volume can read the other. The columns HealthLog encrypts stay encrypted inside the dump, under ENCRYPTION_KEYS. Keep a copy of the dumps off the host; the off-host bucket does that for you, and it seals each copy with BACKUP_ENCRYPTION_KEY in the envelope the per-account copies use.

A dump holds every account that existed when it was taken, including one deleted since. Deleted accounts leave the dumps as they age out: after DB_BACKUP_KEEP nights on the server, and after your lifecycle rule in the bucket. Give that rule a prefix that covers database/ too.

Without a writable DB_BACKUP_DIR (an install that runs the image without the backups volume), the nightly dump is uploaded off-host and nothing is kept on the server, and no dump is taken before migrations; with no bucket either, no dump is taken at all. The start log and the admin page say so.

The per-account copies (the weekly in-database copy, the nightly per-account objects) roll one account back on the same server. They cannot rebuild a server; see backup-restore.md.

The keys a restore needs

Key Without it
ENCRYPTION_KEYS + ENCRYPTION_ACTIVE_KEY_ID, or ENCRYPTION_KEY Nothing encrypted can be read. The restore refuses when the active key is missing or different. Keep every retired id still in the map.
API_TOKEN_HMAC_KEY No data is lost, but every session, app token and share link stops working; people sign in again and re-create tokens.
BACKUP_ENCRYPTION_KEY (and BACKUP_ENCRYPTION_PREVIOUS_KEYS) Off-host copies do not open.
POSTGRES_PASSWORD Not needed: a new server can use a new one.

The restore compares the keys in its environment with the fingerprints in the manifest before it writes anything, and names each key that is missing or different.

Restore onto a new server

  1. Install Docker and take the docker-compose.yml of the version that wrote the dump or a newer one. The manifest's appVersion names it; to run that exact version set HEALTHLOG_IMAGE_REF=:v<version> in .env.

  2. Write .env with the keys from your records (see above) and a new POSTGRES_PASSWORD.

  3. Do not run docker compose up yet. The app would create an empty schema, and the restore only writes into a database with no tables. Start only the database:

    docker compose up -d db
    
  4. Put the dump and its manifest in a directory, for example ./restore, and make them readable for the container (chmod a+r restore/*). From the bucket, download both database/<name>.dump.enc and database/<name>.manifest.json.enc; the restore opens them with BACKUP_ENCRYPTION_KEY.

    docker compose run --rm -v "$PWD/restore:/restore:ro" db-restore list /restore
    docker compose run --rm -v "$PWD/restore:/restore:ro" db-restore \
      /restore/healthlog-20261024T233000Z-daily.dump
    

    It checks the archive against the manifest's SHA-256, the keys against the manifest's fingerprints, and that the database is empty. Then it restores in one transaction, gathers planner statistics and counts every table against the manifest. A count that differs empties the database again and fails. Wait for the line that starts with Restored.

  5. Start the stack and check it:

    docker compose up -d
    curl -s http://localhost:3000/api/health
    

    Sign in. The app's own key check runs at start and refuses to serve on a key that does not open the data.

When the dumps are still in the server's backups volume (the database volume was lost, the backups volume was not), step 4 is docker compose run --rm db-restore latest.

--ignore-key-mismatch restores although the active encryption key differs. Use it only when that key is lost for good: everything encrypted stays unreadable.

Going back to before a migration

The entrypoint dumps the database before it applies migrations. To return to that state (the previous image, the data as it was before the update):

docker compose stop app
# Keep the current state too, in case it is wanted later.
docker compose exec -T db pg_dump -U healthlog -Fc healthlog > healthlog-current.dump
docker compose exec db dropdb -U healthlog healthlog
docker compose exec db createdb -U healthlog healthlog
docker compose run --rm db-restore list
docker compose run --rm db-restore healthlog-<time>-pre-migration.dump
# Set HEALTHLOG_IMAGE_REF in .env to the version you ran before the update.
docker compose up -d

The dump before a migration is taken while the new image starts. If it fails on a writable directory (a full disk, most often), the start stops before anything is migrated and the log names the switch: DB_BACKUP_BEFORE_MIGRATE=false starts without it, once.

How this is tested

scripts/compose-dr-drill.sh plays this page against real containers: it builds the image, seeds several accounts (the demo account with documents and encrypted notes, a Guardian with a managed profile), lets the scheduled job take the dump, destroys every volume, restores with docker compose run --rm db-restore latest on a new stack given only the dump and the keys, counts every table against the manifest and boots the app: health, sign-in, an encrypted note read back, a document opened, the Guardian still seeing the profile. It runs in CI nightly and on every change to the compose file, the image or the dump and restore code (.github/workflows/compose-dr-drill.yml), and it is checked to fail on a wrong key, a dump that lost a table's rows and a target that is not empty.

Test your own copies too: once, and after any change to how the server is run, restore last night's dump on a spare machine with the steps above.

Settings

Variable Default What it does
DB_BACKUP_ENABLED on false stops the nightly dump.
DB_BACKUP_DIR /var/lib/healthlog/backups Where dumps are written and read. Change it only together with the volume mount.
DB_BACKUP_SCHEDULE 30 1 * * * Five-field cron, Europe/Berlin time like every HealthLog schedule.
DB_BACKUP_KEEP 7 Nightly dumps kept (at least 1). Two pre-migration dumps are kept besides.
DB_BACKUP_BEFORE_MIGRATE on false starts an update without the dump before its migrations.