Commit Graph
6 Commits
Author SHA1 Message Date
Marc-André Bombeck 1c7785c23b fix(backup): take off-host copies out with the account, and say which keys backups still need
Deleting an account, deleting a managed profile, "Delete all data" and the
admin wipe now write an off-host deletion request in the same transaction
as the deletion. A new job (offhost-backup-purge, kicked straight away and
run nightly as a backstop) lists the bucket once, deletes every
<date>/user-<id>.json.enc object of each pending account and removes the
request only when nothing of the account is left. A refused delete keeps
the request with the bucket's answer and is retried; the off-host card
shows how many deletions are outstanding and why. A copy the nightly run
was writing while the account was deleted is taken back after the upload.

The off-host card also reads the bucket's lifecycle rule and says plainly
when there is none (every nightly copy kept forever) or when the
credential may not read it.

Every stored backup now records the key ids its content was written
under, and every off-host upload records the same per key. The admin
encryption view lists, per key, the stored copies that need it with the
oldest date, and when the last off-host copy needing it expires. The
"safe to drop" badge now requires that no backup still needs a retired
key, instead of only that every row was rotated.

Migration 0358 adds data_backups.inner_key_ids, the key-use table and the
deletion requests.
2026-09-26 19:54:40 +02:00
Marc-André Bombeck a7faae8ded refactor(admin): read the off-host card from evidence, sort the worst row first
Four smaller things around the same card.

The configured flag came from the environment of whichever process serves the
request. A hand-rolled split that gives BACKUP_S3_* to the worker and not to
the web process would have shown an operator "nothing leaves this host" over a
ledger full of fresh rows. The ledger is queried either way now, and a row in
it outranks an environment read: a row is proof an upload happened.

Rows sorted by username put the one account that needs attention wherever the
alphabet leaves it, which on a hundred-account cohort is the crowd problem the
card exists to end, moved from a count into a list. Worst verdict first,
alphabetical within a verdict, decided server-side so the tests and the
accessibility fixture see the order the operator does.

The card's docblock claimed the worker holds PutObject and GetObject. The
runbook is right and the multipart path needs AbortMultipartUpload too, so the
prose no longer documents a grant that would break the uploader.

And the byte formatter moved to lib. The off-host card was importing it out of
a thousand-line client component, which costs nothing while both cards render
on one route and becomes a cost the moment either moves.
2026-09-10 10:32:33 +02:00
Marc-André Bombeck 25f0310596 feat(backups): tell an empty ledger apart from an empty bucket
On the release that adds the ledger the table is empty, so on a host that has
been uploading nightly for months every account read Never, under a sentence
saying no copy had ever reached the bucket. That sentence is untrue on exactly
the morning an operator opens the new card for the first time, and it is a
claim the host cannot make: what is empty is the record, not the bucket.

The run now records every account it walks, not only the ones an object came
of. A walk that produced nothing moves the attempt instant and leaves the
success columns alone, so the two cases separate: no row at all is no record
yet, a row without a success is never. Never now means what the word says —
a run looked at this account and has never put anything there.

The card names the last check on the never row, so the state is a fact with a
timestamp rather than an assertion. Runbook and all seven bundles carry the
same two sentences. The integration file proves both arms against real
Postgres, including that a failed run leaves the previous good copy on the
row.
2026-09-10 10:15:31 +02:00
Marc-André Bombeck cfc19a211c fix(backups): give the off-host verdicts slack for a Berlin night that is 25 hours long
The threshold was the schedule period with no budget on top of it. Two things
spend that budget on a host where nothing is wrong. The cron runs 02:30
Europe/Berlin, so on the DST fall-back Sunday the gap between two runs is 25
hours and every account on every host would cross one period on the same
morning. And the ledger instant is when that account's object landed, by
design, so on a cohort walked one account at a time an account reached an hour
later tonight than last night drifts past 24 with two clean runs behind it.

The schedule period and the verdict threshold are now separate numbers. The
period is still what the card quotes; the threshold is the period plus six
hours, applied to both steps, so due still means one run produced nothing and
stale still means two. The fall-back night is pinned in the unit file by its
real dates rather than by an arithmetic 25, and a genuinely missed night at 31
hours still reads due. The runbook says why the slack is there.
2026-09-10 10:09:59 +02:00
Marc-André Bombeck e64435f6a5 fix(backups): size the off-host ledger column for the object the uploader allows
The column was INTEGER. A single object may occupy 8 MiB across 10 000
multipart parts, which is the 80 GB the uploader explicitly permits, and
Postgres INTEGER stops at 2.1 GB. An account between the two uploads fine and
then cannot be recorded, so the biggest account on the host is the one the
console would have to report on and could not.

BIGINT in the table, BigInt in the model, Number() where the route builds the
wire row so the payload type stays a plain number. The migration is edited in
place rather than followed by a second one: the table has not shipped, and a
column widened before its first release is one column, not a history.

The comparison with the in-database backup does not carry over, for the
record: that size is octet_length over a bytea Postgres caps at 1 GB, while
this object never touches the database at all.
2026-09-10 10:07:28 +02:00
Marc-André Bombeck 1c10bbbde8 feat(admin): say per account how fresh the off-host copy is
The nightly off-host job reports counts — uploaded, failed, total — and a
count cannot name the account it is failing for. A cohort where one object
has been refused every night for a month reads as "99 uploaded, 1 failed"
each time, which looks like weather rather than like an account with no
disaster-recovery copy at all.

The worker now writes a ledger row when an object lands (account, instant,
size), and Admin → Backups gains a second card that rates every account
against the nightly schedule: fresh inside one run, due when one run
produced nothing, stale when two did not, never when no object has ever
landed. On a host without BACKUP_S3_* the card says off-host backup is not
configured and stops there.

The verdicts are computed from the ledger, never from a bucket listing: the
worker's grant is deliberately PutObject + GetObject, and an admin page must
not put the operator's credentials on its render path to state a number the
database already holds.
2026-09-10 09:27:58 +02:00