Files
Marc-André Bombeck 4ffa3352bd ci: restore the drill from the bucket after the host is lost
The drill now runs an S3-compatible bucket beside the stack (Versity S3
Gateway v1.8.0, pinned by digest; the MinIO images are no longer published
on Docker Hub or quay.io) on a network and a volume of its own, so it
outlives the stack as a real bucket does. The app takes the dump through the
scheduled job and uploads it there; the drill waits for a run that both
kept and uploaded it, destroys every volume of the stack, the local backups
included, and restores on a new stack with only the bucket and the keys:
`docker compose run --rm db-restore latest --from-bucket`. The independent
row count reads the manifest from the bucket with the restore's own reader.

DRILL_SOURCE=local keeps the earlier path (dump files copied off the host,
then `db-restore latest`), and CI runs both. A new mutation,
wrong-backup-key, restores with a different BACKUP_ENCRYPTION_KEY and must
be refused. Ports are free ones picked at start.
2026-10-25 13:43:23 +01:00

71 lines
2.3 KiB
YAML

name: Disaster recovery drill
# Rebuilds a lost server from the nightly database dump and the keys alone,
# against real containers: the app image built from this checkout, the
# bundled compose file with an S3-compatible bucket outside it, the demo seed
# plus a Guardian with a managed profile, the dump taken and uploaded by the
# scheduled job, every volume of the stack destroyed, a new stack, the
# restore, and the app booted on the result (health, login, an encrypted
# note, a document, the Guardian's profile, every table's row count against
# the manifest). Two sources: from the bucket (`db-restore latest
# --from-bucket`, the disaster path) and from dump files copied off the host
# (`db-restore latest`).
# `scripts/compose-dr-drill.sh` is the same drill, runnable locally; its
# header lists what it checks and the mutations that must turn it red.
on:
pull_request:
branches: [main]
paths:
- docker-compose.yml
- Dockerfile
- docker-entrypoint.sh
- scripts/compose-dr-drill.sh
- scripts/dr-drill-seed.ts
- scripts/db-dump.mjs
- scripts/db-restore.mjs
- scripts/pending-migrations.mjs
- src/lib/database-backup/**
- src/lib/jobs/database-backup-queue.ts
- src/lib/jobs/reminder/backup-handlers.ts
- .github/workflows/compose-dr-drill.yml
schedule:
# Nightly, 02:41 UTC: off the hour, after the evening merges.
- cron: "41 2 * * *"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: compose-dr-drill-${{ github.ref }}
cancel-in-progress: true
jobs:
drill:
name: Restore a lost server from its dump (${{ matrix.source }})
runs-on: ubuntu-24.04
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
source: [bucket, local]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with:
node-version: 22
cache: pnpm
# The seed scripts run from the checkout against the drill's database.
- run: pnpm install --frozen-lockfile
- run: pnpm db:generate
- name: Drill
env:
DRILL_SOURCE: ${{ matrix.source }}
run: bash scripts/compose-dr-drill.sh