Off-host dumps were named by their time alone. Two instances sharing a bucket
(nothing forbids it, and it was safe for the account objects) overwrote each
other's dump at the same second of the same default schedule, and
latest --from-bucket restored whichever instance's dump was newest. Each
instance now uploads under database/<host of APP_URL>/, the object key (and so
the encryption's associated data) carries it, and the restore takes only this
instance's dumps: it refuses another's and names the instances it found,
unless --instance=<host> or the dump's full key says which one on purpose.
A restore killed outright (out of memory, docker kill) left its .restore- or
.download- directory, a whole database decrypted, in the backups directory,
where nothing removed it. A restore now removes them before it starts and on
SIGINT and SIGTERM, and the nightly retention sweeps any older than twelve
hours.
Pre-migration dumps were kept by count alone, so an install that updates
rarely kept its last two, and every account deleted since, indefinitely,
while the docs and the backups card stated the nightly retention as the
retention of deleted accounts. A pre-migration dump older than the oldest
nightly dump kept now goes, except the newest one, the way back from the
latest update, which stays until the next update. The card, the disaster
recovery guide and its variable table say so.
The fourth rollback step had been joined onto the paragraph after the
list; it is a list item again, with the way back to the data after it.
The disaster recovery page now says how the server tells admins about
its copies: the recovery readiness card, the recovery kit and the hourly
Backup health notification. The support page put GlitchTip under Admin,
General; the card is under Admin, Integrations, as the deploy runbook
says. The pre-migration dump notes say that without the backups volume
no dump is taken, also where the image's own directory is writable.
Only the backups page said when the bucket's lifecycle rules left
database/ out, so every dump, and every deleted account inside it, stayed
in the bucket forever without anyone being told. Readiness gains a
dump_retention signal that reads the same cached lifecycle probe: attention
when dumps go off-host and no rule covers database/, ok with the expiry
when one does, quiet when the bucket will not say, inactive without a
bucket or without off-host dumps. The hourly evaluation pushes it to the
admins like every other signal.
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.
A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.
The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
The new-server steps lead with `db-restore latest --from-bucket`, the copy
that survives when the host and its local dumps are gone, and keep the
restore from downloaded files beside it. Both backup pages now say that the
database dumps are not purged per account: a deleted account stays in them
until they age out, which is the retention period an operator states to the
people using the instance.
docs/ops/disaster-recovery.md says what is dumped and where, that local
dumps are unencrypted and why, which keys a restore needs and what each
one costs when missing, how to restore onto a new server or back to the
state before a migration, and how the procedure is tested. The deploy
runbook and the getting-started guide now describe the dump the
entrypoint takes before migrating, and the off-host account copy page
points at the dump for rebuilding a server.
Marc-André Bombeck