Off-host dumps were named by their time alone. Two instances sharing a bucket
(nothing forbids it, and it was safe for the account objects) overwrote each
other's dump at the same second of the same default schedule, and
latest --from-bucket restored whichever instance's dump was newest. Each
instance now uploads under database/<host of APP_URL>/, the object key (and so
the encryption's associated data) carries it, and the restore takes only this
instance's dumps: it refuses another's and names the instances it found,
unless --instance=<host> or the dump's full key says which one on purpose.
A restore killed outright (out of memory, docker kill) left its .restore- or
.download- directory, a whole database decrypted, in the backups directory,
where nothing removed it. A restore now removes them before it starts and on
SIGINT and SIGTERM, and the nightly retention sweeps any older than twelve
hours.
grep -q stops at the first match, and with pipefail the SIGPIPE that then
ends docker logs read as a miss, so the drill failed on a line that was
there. The Dockerfile notes that the dump directory keeps nothing without
a mount.
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.
A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.
The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
The drill now runs an S3-compatible bucket beside the stack (Versity S3
Gateway v1.8.0, pinned by digest; the MinIO images are no longer published
on Docker Hub or quay.io) on a network and a volume of its own, so it
outlives the stack as a real bucket does. The app takes the dump through the
scheduled job and uploads it there; the drill waits for a run that both
kept and uploaded it, destroys every volume of the stack, the local backups
included, and restores on a new stack with only the bucket and the keys:
`docker compose run --rm db-restore latest --from-bucket`. The independent
row count reads the manifest from the bucket with the restore's own reader.
DRILL_SOURCE=local keeps the earlier path (dump files copied off the host,
then `db-restore latest`), and CI runs both. A new mutation,
wrong-backup-key, restores with a different BACKUP_ENCRYPTION_KEY and must
be refused. Ports are free ones picked at start.
scripts/compose-dr-drill.sh plays docs/ops/disaster-recovery.md against
real containers: it builds the image from the checkout, starts the bundled
compose stack with fixed throwaway keys, seeds the demo account and a
Guardian with a managed profile (through the app's own code), writes a
reading with an encrypted note through the API, and waits for the
scheduled database-backup job to take the dump. It then keeps only the
dump files, destroys every volume, starts a new stack, restores with
`docker compose run --rm db-restore latest`, counts every table against
the manifest apart from the restore, and boots the app: health, the demo
login, the note read back, a document opened, the Guardian still seeing
the profile.
DRILL_MUTATION breaks one step on purpose (wrong-key, skip-table,
non-empty-target), and the drill fails on each. The workflow runs it on
pull requests that touch the compose file, the image, the entrypoint or
the dump and restore code, nightly, and by hand.
Marc-André Bombeck