chore(ops): signed images, a secret gate before commit, and a monthly restore drill

Published manifests are now cosign-signed keyless with a documented
digest-pinning path, a lefthook pre-commit runs gitleaks over staged
changes, responses opt into a credentialless embedder policy, the
deploy webhook can require a fresh timestamp, and a monthly drill
decrypts and parses the newest offhost backup so a broken key or format
surfaces before an emergency does.
This commit is contained in:
Marc-André Bombeck committed 2026-06-11 00:10:01 +02:00
1 parent 410c62c292
commit ce3b4c70c8
11 files changed
+433 -4

No files matched your search

+27
View File
@@ -85,3 +85,30 @@ pnpm tsx scripts/restore-backup.ts \
The script downloads the object, decrypts it, and writes the JSON dump
to disk. Importing the JSON back into a HealthLog instance is left to
the operator (use `prisma db seed` or a custom script).
## Monthly restore drill (automatic)
Since v1.16.4 a pg-boss job (`data-restore-drill`, cron `11 4 1 * *` —
04:11 on the 1st of each month) exercises the read path end-to-end:
fetch the most recent backup object from the bucket, decrypt it under
the current `BACKUP_ENCRYPTION_KEY`, JSON-parse it, and sanity-check
the payload shape. It performs **no database restore** — it validates
the artefact, not the import path.
Outcomes:
- **Success** — record counts, object age, and sizes land in the
wide-event meta (`job.restore_drill`).
- **Stale chain** — the newest object is older than 3 days: the nightly
uploader has stalled (or the lifecycle rule is too aggressive). The
drill pages via the worker error reporter (stderr + GlitchTip).
- **Failure** — empty bucket, fetch error, decryption failure (wrong or
rotated key), malformed JSON: pages the same way. A decryption
failure right after a `BACKUP_ENCRYPTION_KEY` change means the new
key cannot read the existing objects — re-encrypt or accept that
pre-rotation backups are only readable with the retired key.
- **Not configured** — deployments without the `BACKUP_S3_*` vars skip
silently (wide-event warning only).
The drill needs no IAM grant beyond the uploader's existing
`GetObject` + `ListBucket`.
+74 -1
View File
@@ -70,7 +70,7 @@ roll back:
release pair shipped on the v1.4.x line.
3. If a column-drop migration ships in the rollback target, hand-roll
the column back via a fresh forward migration (`pnpm prisma migrate
dev --create-only --name vX_Y_Z_rollback_<col>`), edit the SQL,
dev --create-only --name vX_Y_Z_rollback_<col>`), edit the SQL,
commit, and re-deploy. Never `migrate resolve --rolled-back` against
production unless you've taken a fresh `pg_dump` first.
@@ -90,3 +90,76 @@ pnpm dlx tsx scripts/assert-deploy.ts 1.9.0
Pass `--only=prod` / `--only=demo` to check a single target, or
`--attempts=N --interval=ms` to widen the retry window while a slow
pull settles.
## Verify the image signature
Every release image (the multi-arch manifest **index**, covering both
`amd64` and `arm64`) is signed keyless via Sigstore in the
`docker-publish.yml` merge job: GitHub's OIDC token is exchanged for a
short-lived Fulcio certificate bound to the workflow identity, and the
signature lands in the Rekor transparency log. There is no long-lived
signing key to leak or rotate.
Verify before (or after) a deploy with [cosign](https://docs.sigstore.dev/cosign/system_config/installation/):
```sh
cosign verify \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity-regexp 'https://github.com/MBombeck/HealthLog/\.github/workflows/docker-publish\.yml@refs/tags/v.*' \
ghcr.io/mbombeck/healthlog:1.16.4
```
A passing verification proves the image was built by this repository's
release workflow on a `v*` tag — not by anyone who merely obtained push
access to the GHCR package.
### Prefer digest-pinned deploys
Tags (`:latest`, `:1.16.4`) are mutable aliases: a registry-side
compromise or an accidental re-push can repoint them. The digest is the
image's only immutable identity, and it is what the signature actually
covers. For any deploy where supply-chain integrity matters more than
convenience:
1. Resolve and verify the digest once:
```sh
cosign verify ... ghcr.io/mbombeck/healthlog:1.16.4 \
| jq -r '.[0].critical.image."docker-manifest-digest"'
```
2. Pin the compose `image:` (or the Coolify image reference) to
`ghcr.io/mbombeck/healthlog@sha256:<digest>` instead of the tag.
The day-to-day `:latest` + `pull_policy: always` flow stays the
documented default for self-hosters; digest pinning is the recommended
hardening for operators with a threat model that includes the registry.
## Deploy-status webhook: replay protection
`POST /api/internal/deploy-webhook` authenticates via the static
`X-Deploy-Webhook-Secret` header. Since v1.16.4 it additionally
enforces a freshness window: a request carrying an
`X-Deploy-Webhook-Timestamp` header (unix seconds, unix milliseconds,
or ISO-8601) is rejected with `401 {"status":"stale"}` when the
timestamp lies more than **5 minutes** from server time — so a captured
request cannot be replayed later to re-page admins or forge deploy
outcomes in the audit log.
Two modes:
- **Default (tolerant).** Requests _without_ the header pass — Coolify's
stock notification sender does not attach one. A header that is
present but stale or unparseable is always rejected.
- **Strict.** Set `DEPLOY_WEBHOOK_REQUIRE_TIMESTAMP=true` in the app
environment to make the header mandatory, closing the replay window
completely. Use this when the sender (or a reverse proxy in front of
the app) can attach the timestamp, e.g.:
```sh
curl -X POST https://<host>/api/internal/deploy-webhook \
-H "X-Deploy-Webhook-Secret: $SECRET" \
-H "X-Deploy-Webhook-Timestamp: $(date +%s)" \
-H "Content-Type: application/json" \
-d '{"status":"success"}'
```