Merge the per-account off-host backup freshness and the honest error-report badge

This commit is contained in:
Marc-André Bombeck committed 2026-09-10 10:37:00 +02:00
commit e8a2cbe82e
36 files changed
+2227 -27

No files matched your search

+36
View File
@@ -215,6 +215,42 @@ A run where SOME account got a copy still succeeds, with the rest counted in
`offhost_backup_failed`. Failing the whole queue over one account's object
would re-upload everybody's on every retry.
### Which account has no copy
The counts above say how many accounts were uploaded, never which. An account
whose object has failed every night for a month reads as `99 uploaded, 1
failed` each time, which looks like weather.
**Admin console → Backups → Off-host copies** answers it per account: when this
host last put that account's encrypted copy in the bucket, how big it was, and
how that compares with the nightly schedule. **Fresh** is inside one run,
**due** means one run produced nothing for that account, **stale** means two
did not, **never** means a run walked the account and no object has ever
landed for it, and **no record yet** means no run has recorded the account at
all. On a host without `BACKUP_S3_*` the card says off-host backup is not
configured and stops there.
The ledger starts empty on the release that adds it, so on a host that has
been uploading nightly for months every account reads **no record yet** until
the first run after the upgrade. That is the ledger having no history, not the
bucket being empty — which is why it is a separate word from **never**, and
why the row says so rather than claiming nothing has ever reached the bucket.
Each verdict carries six hours of slack on top of the schedule's own 24, and
that slack is load-bearing on two nights. The cron runs at 02:30 Europe/Berlin,
so on the DST fall-back Sunday two consecutive runs are 25 hours apart; and the
timestamp is when _that account's_ object landed, so on a cohort walked one
account at a time an account reached later tonight than last night drifts by
hours. Without the slack a healthy host would paint its whole cohort **due**
once a year.
The verdicts come from a ledger the worker writes when the object lands
(`offhost_backup_state`), not from a listing of the bucket. Keep the worker's
grant as documented — PutObject, GetObject, AbortMultipartUpload — and the card
still tells the truth. A row that goes stale while the run reports success is
that account's object being refused, and the reason is in the run's
`offhost_backup_failures` meta.
## The weekly in-database backup (`data-backup`)
Separate from the off-host job above, and easy to confuse with it. A second
+21
View File
@@ -143,6 +143,27 @@ itself still works: it walks the journey against an in-process instance
once per leg with that leg broken, and fails if any of them comes back
green.
## Error reporting
HealthLog ships no default error-reporting target and never will: an
operator's crash reports must not leave their host because nobody told them
not to. Reporting starts only when someone types a DSN into **Admin →
Integrations → GlitchTip**, and that card states plainly whether reports are
going anywhere and which host they go to.
For a maintainer-operated instance, set the DSN there after the first deploy —
it lives in the database, not in the environment, so it survives an image
upgrade and does not need a compose change.
The badge on that card tracks delivery, not intent. A DSN that is set and
parses is not the same fact as reports arriving: a wrong public key, a wrong
project id and a host that stopped answering all parse. So the card is green
only while a report has actually left this host inside the last fortnight,
amber while a target is set and nothing has come back from it, and it names the
reason underneath when the last attempt was refused. Press **Test** to prove
the target on demand — that send is recorded like any other, so the badge
answers immediately.
## Verify the image signature
Every release image (the multi-arch manifest **index**, covering both