fix(db-backup): keep instances apart in a shared bucket, and no decrypted dump behind

Off-host dumps were named by their time alone. Two instances sharing a bucket
(nothing forbids it, and it was safe for the account objects) overwrote each
other's dump at the same second of the same default schedule, and
latest --from-bucket restored whichever instance's dump was newest. Each
instance now uploads under database/<host of APP_URL>/, the object key (and so
the encryption's associated data) carries it, and the restore takes only this
instance's dumps: it refuses another's and names the instances it found,
unless --instance=<host> or the dump's full key says which one on purpose.

A restore killed outright (out of memory, docker kill) left its .restore- or
.download- directory, a whole database decrypted, in the backups directory,
where nothing removed it. A restore now removes them before it starts and on
SIGINT and SIGTERM, and the nightly retention sweeps any older than twelve
hours.
This commit is contained in:
Marc-André Bombeck committed 2026-10-25 17:30:29 +01:00
1 parent 6051488b65
commit a7691223fd
8 files changed
+493 -92

No files matched your search

+3 -2
View File
@@ -113,8 +113,9 @@ gzipped and sealed with AES-256-GCM under `BACKUP_ENCRYPTION_KEY`, with the
object's own name as associated data and the key's id in the header.
The nightly database dump goes into the same bucket, as
`database/healthlog-<time>-daily.dump.enc` with its manifest beside it, sealed
the same way. It is the copy that rebuilds a lost server
`database/<host of APP_URL>/healthlog-<time>-daily.dump.enc` with its manifest
beside it, sealed the same way. The folder keeps the dumps of instances that
share a bucket apart. It is the copy that rebuilds a lost server
([disaster-recovery.md](disaster-recovery.md)).
**Permissions.** The worker needs `PutObject`, `GetObject`, `ListBucket`,
+18 -6
View File
@@ -6,11 +6,11 @@ onto a new server, and how the procedure is tested.
## What is backed up, and where
| Copy | When | Where | Kept |
| ----------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------- | ------------------------------- |
| Nightly dump | 01:30 Europe/Berlin, the zone every schedule runs in (`DB_BACKUP_SCHEDULE`) | `DB_BACKUP_DIR`, the `backups` volume of the bundled compose file | the newest 7 (`DB_BACKUP_KEEP`) |
| Pre-migration dump | on start, when the new image brings migrations, before them | the same directory | the newest 2 |
| Off-host copy of a dump | right after each nightly dump, if the bucket is configured | your bucket, under `database/`, sealed with `BACKUP_ENCRYPTION_KEY` | your bucket's lifecycle rule |
| Copy | When | Where | Kept |
| ----------------------- | --------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | ------------------------------- |
| Nightly dump | 01:30 Europe/Berlin, the zone every schedule runs in (`DB_BACKUP_SCHEDULE`) | `DB_BACKUP_DIR`, the `backups` volume of the bundled compose file | the newest 7 (`DB_BACKUP_KEEP`) |
| Pre-migration dump | on start, when the new image brings migrations, before them | the same directory | the newest 2 |
| Off-host copy of a dump | right after each nightly dump, if the bucket is configured | your bucket, under `database/<host of APP_URL>/`, sealed with `BACKUP_ENCRYPTION_KEY` | your bucket's lifecycle rule |
A dump is `pg_dump --format=custom` of the whole database: every account,
sign-ins, passkeys, sharing grants and managed profiles, documents, the job
@@ -103,9 +103,21 @@ different.
docker compose run --rm db-restore latest --from-bucket
```
`latest --from-bucket` downloads the newest `database/…dump.enc` whose
`latest --from-bucket` downloads the newest dump of this instance whose
manifest is in the bucket too, opens both with `BACKUP_ENCRYPTION_KEY`
and restores. Name a dump instead of `latest` to take an older one.
Each instance uploads under its own folder, `database/<host of APP_URL>/`,
so set `APP_URL` as on the old server. A bucket shared with another
instance holds its dumps too: `latest` refuses rather than take one of
those, and `list --from-bucket` marks them. To restore one on purpose
(after moving to a new domain, the old one), add
`--instance=<old host>`, or name the dump by its full key,
`database/<host>/healthlog-….dump.enc`.
The opened copy is a whole database, decrypted, in a scratch directory
beside the backups. The restore removes it when it ends, on Ctrl-C and
`docker compose stop` too. One left by a restore that was killed outright
is removed when the next restore starts, and by the nightly retention.
With the files in hand instead (copied off the host, or downloaded from
the bucket with your own tools: both `<name>.dump.enc` and