Commit Graph
12 Commits
Author SHA1 Message Date
Marc-André Bombeck 78de978dcd fix(recovery): check a pasted token key against its slow fingerprint
The kit's check compared keyed SHA-256 digests of the pasted value and the
configured key, which code scanning still reads as a fast hash of a
password. It now compares the scrypt fingerprints in constant time.
2026-10-25 19:58:40 +01:00
Marc-André Bombeck ae0b8d604a fix(recovery): fingerprint a chosen secret with scrypt, not a fast hash
API_TOKEN_HMAC_KEY can be a phrase, and its fingerprint is shown on the
admin page and written into dump manifests in the bucket; a fast hash
there lets a guessed phrase be checked offline. Text secrets now use a
scrypt fingerprint; the random encryption keys keep theirs. The kit's
check of a pasted token key compares keyed digests that never leave the
process.
2026-10-25 19:09:27 +01:00
Marc-André Bombeck 6051488b65 fix(restore-drill): a drill that read nothing back is no pass
A drill that stopped on its time budget before the first object recorded
ok with zero accounts checked, and the readiness tile said passed. Such a
run now records its time and counts and no verdict, so the last pass or
failure stands, and readiness calls it attention (nothing_checked) in every
locale.

The upsert that keeps a pass and a failure apart was stubbed in the drill's
tests and tested nowhere; it now has an integration test read back through
the readiness rule. The drill test that claimed to keep a failed verdict
when keeping it fails is named for what it checks.
2026-10-25 17:26:40 +01:00
Marc-André Bombeck 26a9efdf52 fix(recovery): the database dumps count in the keys the backups still need
The recovery kit, "Keys the backups still need" and safeToDropRetiredKeys
read only the stored copies and the off-host account objects. After a
rotation the page called the old key safe to drop while every dump taken
before it, the pre-migration dumps that are the way back from an update
among them, could only be restored with that key: db-restore refuses a dump
whose active key is missing.

A dump now counts for the key its rows were written under, the manifest's
active key: a dump on this server while it is kept (read from the manifests),
an off-host one until the bucket's rule for database/ expires it, or
indefinitely when that rule cannot be read. The nightly run records the key
on its ledger row (migration 0489, database_backup_runs.active_key_id). The
encryption view and the kit list the dumps per key, in every locale, and the
rotation and backup runbooks say so.
2026-10-25 17:20:16 +01:00
Marc-André Bombeck 763cb48273 fix(recovery): readiness leaves accounts pending deletion out
The weekly and off-host passes skip an account pending deletion on purpose,
and the readiness read counted it anyway: within days of any deletion the
weekly copy read failing and the off-host copies stale, every admin was
pushed a problem and then a recovery after the purge, and a real failure
hid behind a signal already red. The read now filters with ACTIVE_ACCOUNT,
and the enumeration guard covers the file.
2026-10-25 17:03:10 +01:00
Marc-André Bombeck 664b46d2c6 fix(db-backup): read the off-host leg of the dump on its own
A run that kept its dump on the volume and failed the upload is the newest
good dump and the newest failure at the same instant. Compared with ">", the
card showed no failure and readiness said fresh, night after night, while no
dump left the host.

The health read now carries the off-host leg separately: when the newest dump
reached the bucket, and the newest one that did not. One pure reading of it
(databaseDumpProblems) feeds both the backups card and the readiness model.
A configured bucket that missed the newest dump is attention (offhost_failed)
with the upload's error, and failing once the newest dump there is older than
36 hours (offhost_stale) or none ever arrived (offhost_never). The card shows
the refused upload and when the bucket last got a dump. A failure at the
instant of the newest dump for another reason, retention throwing, is now
shown too.
2026-10-25 16:53:24 +01:00
Marc-André Bombeck 66e92f36fe feat(recovery): judge the bucket's rule for database dumps in readiness
Only the backups page said when the bucket's lifecycle rules left
database/ out, so every dump, and every deleted account inside it, stayed
in the bucket forever without anyone being told. Readiness gains a
dump_retention signal that reads the same cached lifecycle probe: attention
when dumps go off-host and no rule covers database/, ok with the expiry
when one does, quiet when the bucket will not say, inactive without a
bucket or without off-host dumps. The hourly evaluation pushes it to the
admins like every other signal.
2026-10-25 16:00:35 +01:00
Marc-André Bombeck 351e7d1fe0 fix(backup): keep a dump locally only where it survives the container
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.

A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.

The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
2026-10-25 16:00:34 +01:00
Marc-André Bombeck e88dcd2c53 Merge the release into the backup health work and read the real dump health
The readiness model now takes the database dump's state from
getDatabaseBackupHealth, and the dump test reaches the off-host key id
where it moved.
2026-10-25 14:40:31 +01:00
Marc-André Bombeck 0a053d6884 feat(admin): name a switched-off dump and a newly listed kit key plainly
A switched-off scheduled dump is attention now, sent once and never
reminded, with how to turn it back on (DB_BACKUP_ENABLED). Dumps are on
by default, so off is a choice that leaves no copy able to rebuild the
host. An install that never wrote a dump and has nowhere to write one
stays quiet.

The out-of-date kit notice says what changed: the kit now also lists a
key the last confirmation did not cover, for example API_TOKEN_HMAC_KEY
on an instance upgraded from the one-key step, and that nothing broke.

The French kit and readiness strings use vous, the form the rest of the
French bundle uses.
2026-10-25 14:09:41 +01:00
Marc-André Bombeck 7dab9cc5e9 feat(admin): judge every backup signal and push changes to admins
Every backup failure was a pull, on a page nobody opens until the day it
matters: the weekly copy stopped for 46 days and showed an ordinary row,
a failed monthly drill was on the failing-jobs card three days in thirty
(job-failures.ts reads 72 hours), and an off-host bucket that was never
set up showed only on the backups page.

One readiness model (src/lib/recovery/readiness-model.ts) judges six
signals as ok, attention or failing, with a reason code, the time it is
about and where to fix it:

- database dump: failing once the newest good dump is older than 36 h
  while dumps are on, or when it can be kept nowhere; read through a
  provider that reports "not available" until the dump lands;
- weekly account copies: worst account decides, attention past 9 days,
  failing past 14; an account without a copy ages from its creation;
- off-host copies: attention when any account is two nights behind;
- restore drill: failing until a later drill passes, with no time
  window; the drill now keeps its verdict in restore_drill_outcomes;
- off-host setup: attention 7 days after the first admin exists;
- recovery kit: attention when a needed key is not confirmed, by id,
  7 days after setup or as soon as a confirmed kit is out of date.

A "Recovery readiness" card shows them on the admin overview and the
backups page. An hourly job (recovery-readiness, :33) sends each change
to every admin through the localised admin path, under a new
admin-only BACKUP_HEALTH event that the channel preferences route: once
when a signal turns to attention or failing, once a day while it stays
failing, once when it recovers. Each notice is claimed with a
compare-and-set before it is sent, so two workers never send the same
one, and nothing is claimed while no admin exists.

The backups page calls the weekly copy stale at the same instant as the
model (past nine days rather than past ten).
2026-10-25 13:32:43 +01:00
Marc-André Bombeck d3c672cc3f feat(admin): replace the key backup step with a recovery kit
The step confirmed one key, the active application key. A restore needs
more: off-host copies are sealed under BACKUP_ENCRYPTION_KEY, a copy
written before a rotation needs the retired application key it was
written under, and every session, token and share link hangs off
API_TOKEN_HMAC_KEY. An operator could do everything the step asked and
still be unable to read a single backup.

The recovery kit lists every key a restore of this server needs, by id
and fingerprint, never key material:

- each id in ENCRYPTION_KEYS, plus any retired id a stored copy
  (innerKeyIds) or an off-host object inside the bucket's retention
  (OffhostBackupKeyUse) still lists, also once the server dropped it;
- BACKUP_ENCRYPTION_KEY and each BACKUP_ENCRYPTION_PREVIOUS_KEYS entry,
  named by the id their objects record;
- API_TOKEN_HMAC_KEY.

Confirming records the set the admin was shown and refuses a stale one
with 409 recovery.kit.stale. The kit is due again as soon as a key it
lists is not in that set (a rotation, a new backup key, a re-keyed
install); a key that merely drops out does not make it due. "Check my
copy" now checks any key in the kit.

Migration 0487 carries a confirmation given under the old step forward
as a kit holding that key, so that part stays confirmed, and drops the
four app_settings columns. It also creates the two tables the readiness
evaluation uses (restore_drill_outcomes, recovery_readiness_notices).

The routes move to /api/admin/recovery-kit; the reader of the keys the
backups still need moves out of the encryption status route into
src/lib/recovery/backup-key-needs.ts so the kit and the status view
share one definition.
2026-10-25 13:32:29 +01:00