fix(backup): keep a dump locally only where it survives the container

The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.

A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.

The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
This commit is contained in:
Marc-André Bombeck committed 2026-10-25 16:00:34 +01:00
1 parent fb4ab29aad
commit 351e7d1fe0
32 files changed
+882 -126

No files matched your search

+3 -1
View File
@@ -292,7 +292,9 @@ API_TOKEN_HMAC_KEY="REPLACE-WITH-OUTPUT-OF-openssl-rand-hex-32"
# Set "false" to stop the nightly dump.
# DB_BACKUP_ENABLED="true"
# Where dumps are written. The bundled compose file mounts the `backups`
# volume here; change it only together with that mount.
# volume here; change it only together with that mount. In a container a
# directory that is not on a mount of its own sits in the container's layer
# and is lost at the next update, so no dump is kept there.
# DB_BACKUP_DIR="/var/lib/healthlog/backups"
# Five-field cron, in Europe/Berlin time like every scheduled job.
# DB_BACKUP_SCHEDULE="30 1 * * *"
+3 -1
View File
@@ -429,7 +429,9 @@ API_TOKEN_HMAC_KEY="<openssl rand -hex 32>"
# Set "false" to stop the nightly dump.
# DB_BACKUP_ENABLED="true"
# Where dumps are written. The bundled compose file mounts the `backups`
# volume here; change it only together with that mount.
# volume here; change it only together with that mount. In a container a
# directory that is not on a mount of its own sits in the container's layer
# and is lost at the next update, so no dump is kept there.
# DB_BACKUP_DIR="/var/lib/healthlog/backups"
# Five-field cron, in Europe/Berlin time like every scheduled job.
# DB_BACKUP_SCHEDULE="30 1 * * *"
+4 -2
View File
@@ -90,8 +90,10 @@ echo "HealthLog: Database is reachable."
node scripts/pending-migrations.mjs || true
# When migrations are pending, dump the database into DB_BACKUP_DIR first, so
# the state before them can be restored (docs/ops/disaster-recovery.md). No
# writable directory: a warning, and the boot goes on. A writable directory
# and a failed dump: the boot stops here, before anything is migrated.
# directory that survives the container (none writable, or the image's own
# without the backups volume): a warning, and the boot goes on. A kept
# directory and a failed dump: the boot stops here, before anything is
# migrated.
if ! node scripts/db-dump.mjs pre-migration; then
echo "HealthLog: ERROR — not migrating without the dump. Free disk space or fix the error above and start again." >&2
echo "HealthLog: to migrate without a dump this once, set DB_BACKUP_BEFORE_MIGRATE=false." >&2
+11 -4
View File
@@ -44,10 +44,17 @@ backups page reads the bucket's rules and says when none does. This is the
retention period to state where you tell people that deleting their account
deletes their data.
Without a writable `DB_BACKUP_DIR` (an install that runs the image without the
`backups` volume), the nightly dump is uploaded off-host and nothing is kept
on the server, and no dump is taken before migrations; with no bucket either,
no dump is taken at all. The start log and the admin page say so.
A dump is kept on the server only when `DB_BACKUP_DIR` survives the container
being replaced: a mounted volume or bind mount, or any writable directory when
HealthLog does not run in a container. The image creates the default
directory, so an install that runs it without the `backups` volume can write
there, but the directory sits in the container's own layer and the next update
throws it away. HealthLog reads the container's mount table to tell the two
apart and treats such a directory, or one on a tmpfs, like one it cannot
write: the nightly dump is uploaded off-host and nothing is kept on the
server, and no dump is taken before migrations; with no bucket either, no
dump is taken at all. The start log, the admin backups page and the recovery
readiness card say so.
The per-account copies (the weekly in-database copy, the nightly per-account
objects) roll one account back on the same server. They cannot rebuild a
+14 -1
View File
@@ -54,6 +54,15 @@ example `UPDATE_CHECK_DISABLED=1`. The full list is in
To use your own GeoLite2 databases, add the directory under Additional Storage
and point `GEOLITE2_DIR` at its mount path ([geolite2.md](geolite2.md)).
HealthLog writes a whole-database dump every night, and one before each update
that brings migrations, into `/var/lib/healthlog/backups` inside the app
container. Give that path a dataset of its own (a host path or an ixVolume,
writable by uid `1001`), under Additional Storage if the form has no field for
it. Without one the directory sits inside the container, which an update
replaces, so HealthLog keeps no dump there: the start log, the admin backups
page and the recovery readiness card say so, and with an off-host bucket the
nightly dump goes to the bucket only.
## Plain HTTP on the LAN
Most NAS installs are reached over `http://`. Leave **Secure Cookies** off for
@@ -113,6 +122,10 @@ back.
The file lands in the Postgres Data Storage dataset, next to the database
files, so copy it off the NAS as well.
- **The nightly dump.** With a dataset on `/var/lib/healthlog/backups` (see
above), the newest seven nightly dumps and the last two taken before a
migration stay there; [disaster-recovery.md](../ops/disaster-recovery.md)
restores one onto a new server.
- **Off-host.** HealthLog can push encrypted nightly backups to any
S3-compatible bucket; see [backup-restore.md](../ops/backup-restore.md).
@@ -216,7 +229,7 @@ the menu next to Custom App, Install via YAML, using these values:
| Health check | `GET /api/health`, `200` when ready |
| Start period | 10 min, like the image: migrations run at every boot and wait out a held lock |
| Database | PostgreSQL 18 (16 works too), no extensions, in its own container with its own dataset |
| Volumes | none for the app; the database gets its own dataset |
| Volumes | `/var/lib/healthlog/backups` for the database dumps (uid `1001`); the database its own |
| Variable | Required | Notes |
| ----------------------- | -------- | ---------------------------------------------------------------------------------------- |
+19 -8
View File
@@ -6,10 +6,13 @@ self-contained install (app + database as one unit) and is recommended. The
Unraid UI, but because CA templates describe a single container you install
PostgreSQL separately and wire the two together.
HealthLog stores all persistent state in PostgreSQL — the app container itself is
stateless (the offline geo-IP databases are either baked in at build time or
mounted read-only; replaceable either way, not state). A backup of
the database is a backup of everything.
HealthLog stores all persistent state in PostgreSQL. The app container keeps one
more thing, its database dumps: every night, and before each update that brings
migrations, it writes a whole-database dump to `/var/lib/healthlog/backups`
(the offline geo-IP databases are either baked in at build time or mounted
read-only; replaceable either way, not state). A backup of the database is a
backup of everything, and those dumps are that backup, so that path needs a
share of its own.
## What you need
@@ -103,12 +106,20 @@ Fill in the fields:
| `SESSION_COOKIE_SECURE` | `false` for plain-HTTP LAN access |
| `NEXT_PUBLIC_APP_URL` | `http://tower:3000` (or your domain) |
| `APP_URL` | same as `NEXT_PUBLIC_APP_URL` |
| Database dumps | `/mnt/user/appdata/healthlog/backups` (the default) |
Replace `<password>` and `<postgres-host>` with the values from Step 1. Apply,
then open `http://tower:3000` — the first registered user becomes the admin.
Replace `<password>` and `<postgres-host>` with the values from Step 1. The
dumps folder must be writable by the app's user, uid `1001`; create it once
from the terminal with
`mkdir -p /mnt/user/appdata/healthlog/backups && chown 1001:1001 /mnt/user/appdata/healthlog/backups`.
Apply, then open `http://tower:3000` — the first registered user becomes the
admin.
The HealthLog container has no persistent volume of its own; all state is in the
PostgreSQL container you installed in Step 1. Back that one up.
All account data is in the PostgreSQL container you installed in Step 1, and
the nightly dumps of it land in the dumps folder. Without that path the dumps
would sit inside the HealthLog container, which every update replaces, so
HealthLog keeps none: the start log, the admin backups page and the recovery
readiness card say so. Back up both.
---
+2
View File
@@ -63,4 +63,6 @@
<Config Name="ENCRYPTION_KEYS" Target="ENCRYPTION_KEYS" Default="" Mode="" Description="Optional. Key-rotation map, e.g. {&quot;v1&quot;:&quot;&lt;hex&gt;&quot;,&quot;v2&quot;:&quot;&lt;hex&gt;&quot;}. Leave empty unless rotating; use ENCRYPTION_KEY instead." Type="Variable" Display="advanced" Required="false" Mask="true"/>
<Config Name="ENCRYPTION_ACTIVE_KEY_ID" Target="ENCRYPTION_ACTIVE_KEY_ID" Default="" Mode="" Description="Optional. Active key id for the ENCRYPTION_KEYS map (e.g. v2). Leave empty when using ENCRYPTION_KEY." Type="Variable" Display="advanced" Required="false" Mask="false"/>
<Config Name="Database dumps" Target="/var/lib/healthlog/backups" Default="/mnt/user/appdata/healthlog/backups" Mode="rw" Description="Where HealthLog keeps its nightly database dump and the one it takes before each update that brings migrations. Without this path the dumps would sit inside the container, which an update replaces, so none is kept. Must be writable by uid 1001 (chown 1001:1001 the folder once)." Type="Path" Display="always" Required="true" Mask="false">/mnt/user/appdata/healthlog/backups</Config>
</Container>
+6 -2
View File
@@ -8346,7 +8346,9 @@
"noDirectory": "Dieser Server kann nicht nach {dir} schreiben und es ist kein externer Speicher eingerichtet. Deshalb entsteht kein Datenbank-Dump. Binde das Backup-Volume ein.",
"retentionLocal": "Gelöschte Konten bleiben in den Dumps, bis diese aussortiert werden: Auf diesem Server bleiben die neuesten {nights}.",
"retentionOffhost": "Externe Kopien entfernt die Lebenszyklus-Regel des Buckets nach {days} Tagen.",
"retentionOffhostMissing": "Keine Lebenszyklus-Regel im Bucket gilt für database/. Dumps dort und die gelöschten Konten darin werden deshalb nie entfernt. Lege eine Regel für dieses Präfix an."
"retentionOffhostMissing": "Keine Lebenszyklus-Regel im Bucket gilt für database/. Dumps dort und die gelöschten Konten darin werden deshalb nie entfernt. Lege eine Regel für dieses Präfix an.",
"notPersistentOffhost": "{dir} liegt nicht auf dauerhaftem Speicher (im Container statt auf einem eingebundenen Volume, oder im Arbeitsspeicher). Ein Dump dort wäre beim nächsten Update verloren. Dumps gehen deshalb nur extern: Hier wird keiner aufbewahrt, und vor einer Migration wird keiner erstellt. Binde das Backup-Volume ein.",
"notPersistent": "{dir} liegt nicht auf dauerhaftem Speicher (im Container statt auf einem eingebundenen Volume, oder im Arbeitsspeicher), und es ist kein externer Bucket eingerichtet. Damit wird nirgends ein Datenbank-Dump aufbewahrt. Binde das Backup-Volume ein."
}
},
"danger-zone": {
@@ -8689,7 +8691,9 @@
"stale": "Der neueste Datenbank-Dump ist {hours} Stunden alt.",
"last_failed": "Der letzte Datenbank-Dump ist fehlgeschlagen. Der davor ist noch aktuell.",
"fresh": "Der neueste Datenbank-Dump ist aktuell.",
"nowhere_to_write": "Das Dump-Verzeichnis {directory} ist nicht beschreibbar und es ist kein externer Bucket eingerichtet, daher kann kein Dump aufbewahrt werden."
"nowhere_to_write": "Das Dump-Verzeichnis {directory} ist nicht beschreibbar und es ist kein externer Bucket eingerichtet, daher kann kein Dump aufbewahrt werden.",
"not_persistent": "Das Dump-Verzeichnis {directory} liegt nicht auf dauerhaftem Speicher, und es ist kein externer Bucket eingerichtet. Jeder Dump wäre beim nächsten Update verloren. Binde das Backup-Volume ein.",
"fresh_offhost_only": "Der neueste Datenbank-Dump ist aktuell und liegt im externen Bucket. Auf diesem Server wird keiner aufbewahrt, weil {directory} nicht auf dauerhaftem Speicher liegt, und vor einer Migration wird keiner erstellt."
},
"weekly_copy": {
"no_accounts": "Es gibt noch keine Konten zum Kopieren.",
+6 -2
View File
@@ -8346,7 +8346,9 @@
"noDirectory": "This server cannot write to {dir} and no off-host bucket is set, so no database dump can be taken. Mount the backups volume.",
"retentionLocal": "Deleted accounts stay in the dumps until those age out: the newest {nights} are kept on this server.",
"retentionOffhost": "Off-host copies are removed after {days} days by the bucket's lifecycle rule.",
"retentionOffhostMissing": "No lifecycle rule in the bucket covers database/, so dumps there, and the deleted accounts inside them, are never removed. Add a rule for that prefix."
"retentionOffhostMissing": "No lifecycle rule in the bucket covers database/, so dumps there, and the deleted accounts inside them, are never removed. Add a rule for that prefix.",
"notPersistentOffhost": "{dir} is not on persistent storage (it sits inside the container rather than on a mounted volume, or in memory), so a dump written there would be lost at the next update. Dumps go off-host only: none is kept here and none is taken before a migration. Mount the backups volume.",
"notPersistent": "{dir} is not on persistent storage (it sits inside the container rather than on a mounted volume, or in memory) and no off-host bucket is set, so no database dump is kept anywhere. Mount the backups volume."
}
},
"danger-zone": {
@@ -8689,7 +8691,9 @@
"stale": "The newest database dump is {hours} hours old.",
"last_failed": "The last database dump failed. The one before it is still recent.",
"fresh": "The newest database dump is recent.",
"nowhere_to_write": "The dump directory {directory} cannot be written to and no off-host bucket is set up, so no dump can be kept."
"nowhere_to_write": "The dump directory {directory} cannot be written to and no off-host bucket is set up, so no dump can be kept.",
"not_persistent": "The dump directory {directory} is not on persistent storage and no off-host bucket is set up, so every dump would be lost at the next update. Mount the backups volume.",
"fresh_offhost_only": "The newest database dump is recent and in the off-host bucket. None is kept on this server, because {directory} is not on persistent storage, and none is taken before a migration."
},
"weekly_copy": {
"no_accounts": "There are no accounts to copy yet.",
+6 -2
View File
@@ -8346,7 +8346,9 @@
"noDirectory": "Este servidor no puede escribir en {dir} y no hay almacenamiento externo configurado, así que no se puede hacer ningún volcado. Monta el volumen de copias.",
"retentionLocal": "Las cuentas eliminadas siguen en los volcados hasta que estos caducan: en este servidor se guardan los {nights} más recientes.",
"retentionOffhost": "La regla de ciclo de vida del bucket elimina las copias externas tras {days} días.",
"retentionOffhostMissing": "Ninguna regla de ciclo de vida del bucket cubre database/, así que los volcados de ahí, y las cuentas eliminadas que contienen, nunca se borran. Añade una regla para ese prefijo."
"retentionOffhostMissing": "Ninguna regla de ciclo de vida del bucket cubre database/, así que los volcados de ahí, y las cuentas eliminadas que contienen, nunca se borran. Añade una regla para ese prefijo.",
"notPersistentOffhost": "{dir} no está en almacenamiento persistente (está dentro del contenedor y no en un volumen montado, o en memoria), así que un volcado guardado ahí se perdería en la próxima actualización. Los volcados solo se guardan fuera: aquí no queda ninguno y no se hace ninguno antes de una migración. Monta el volumen de copias.",
"notPersistent": "{dir} no está en almacenamiento persistente (está dentro del contenedor y no en un volumen montado, o en memoria) y no hay un bucket externo configurado, así que no se conserva ningún volcado de la base de datos en ningún sitio. Monta el volumen de copias."
}
},
"danger-zone": {
@@ -8689,7 +8691,9 @@
"stale": "El volcado de la base de datos más reciente tiene {hours} horas.",
"last_failed": "El último volcado de la base de datos falló. El anterior sigue siendo reciente.",
"fresh": "El último volcado de la base de datos es reciente.",
"nowhere_to_write": "No se puede escribir en el directorio de volcados {directory} y no hay ningún bucket externo configurado, así que no se puede guardar ningún volcado."
"nowhere_to_write": "No se puede escribir en el directorio de volcados {directory} y no hay ningún bucket externo configurado, así que no se puede guardar ningún volcado.",
"not_persistent": "El directorio de volcados {directory} no está en almacenamiento persistente y no hay un bucket externo configurado, así que cada volcado se perdería en la próxima actualización. Monta el volumen de copias.",
"fresh_offhost_only": "El volcado más reciente de la base de datos es actual y está en el bucket externo. En este servidor no se conserva ninguno, porque {directory} no está en almacenamiento persistente, y no se hace ninguno antes de una migración."
},
"weekly_copy": {
"no_accounts": "Todavía no hay cuentas que copiar.",
+6 -2
View File
@@ -8346,7 +8346,9 @@
"noDirectory": "Ce serveur ne peut pas écrire dans {dir} et aucun stockage hors site n'est configuré : aucun dump ne peut être fait. Montez le volume de sauvegarde.",
"retentionLocal": "Les comptes supprimés restent dans les dumps jusqu'à leur expiration : ce serveur conserve les {nights} plus récents.",
"retentionOffhost": "La règle de cycle de vie du bucket supprime les copies hors site après {days} jours.",
"retentionOffhostMissing": "Aucune règle de cycle de vie du bucket ne couvre database/ : les dumps qui s'y trouvent, et les comptes supprimés qu'ils contiennent, ne sont jamais effacés. Ajoutez une règle pour ce préfixe."
"retentionOffhostMissing": "Aucune règle de cycle de vie du bucket ne couvre database/ : les dumps qui s'y trouvent, et les comptes supprimés qu'ils contiennent, ne sont jamais effacés. Ajoutez une règle pour ce préfixe.",
"notPersistentOffhost": "{dir} n'est pas sur un stockage persistant (il se trouve dans le conteneur et non sur un volume monté, ou en mémoire) : un dump écrit là serait perdu à la prochaine mise à jour. Les dumps partent uniquement hors site : aucun n'est conservé ici et aucun n'est pris avant une migration. Montez le volume de sauvegarde.",
"notPersistent": "{dir} n'est pas sur un stockage persistant (il se trouve dans le conteneur et non sur un volume monté, ou en mémoire) et aucun bucket hors site n'est configuré : aucun dump de la base n'est conservé nulle part. Montez le volume de sauvegarde."
}
},
"danger-zone": {
@@ -8689,7 +8691,9 @@
"stale": "Le dump de la base de données le plus récent date de {hours} heures.",
"last_failed": "Le dernier dump de la base de données a échoué. Le précédent est encore récent.",
"fresh": "Le dernier dump de la base de données est récent.",
"nowhere_to_write": "Le dossier des dumps {directory} n'est pas accessible en écriture et aucun bucket externe n'est configuré, donc aucun dump ne peut être conservé."
"nowhere_to_write": "Le dossier des dumps {directory} n'est pas accessible en écriture et aucun bucket externe n'est configuré, donc aucun dump ne peut être conservé.",
"not_persistent": "Le répertoire des dumps {directory} n'est pas sur un stockage persistant et aucun bucket hors site n'est configuré : chaque dump serait perdu à la prochaine mise à jour. Montez le volume de sauvegarde.",
"fresh_offhost_only": "Le dump de base le plus récent est à jour et se trouve dans le bucket hors site. Aucun n'est conservé sur ce serveur, car {directory} n'est pas sur un stockage persistant, et aucun n'est pris avant une migration."
},
"weekly_copy": {
"no_accounts": "Il n'y a pas encore de compte à copier.",
+6 -2
View File
@@ -8346,7 +8346,9 @@
"noDirectory": "Questo server non può scrivere in {dir} e non è configurato alcuno storage esterno, quindi non si può fare alcun dump. Monta il volume dei backup.",
"retentionLocal": "Gli account eliminati restano nei dump finché questi non scadono: su questo server restano i {nights} più recenti.",
"retentionOffhost": "La regola del ciclo di vita del bucket elimina le copie esterne dopo {days} giorni.",
"retentionOffhostMissing": "Nessuna regola del ciclo di vita del bucket copre database/, quindi i dump lì, e gli account eliminati che contengono, non vengono mai rimossi. Aggiungi una regola per quel prefisso."
"retentionOffhostMissing": "Nessuna regola del ciclo di vita del bucket copre database/, quindi i dump lì, e gli account eliminati che contengono, non vengono mai rimossi. Aggiungi una regola per quel prefisso.",
"notPersistentOffhost": "{dir} non è su uno spazio persistente (si trova dentro il container e non su un volume montato, oppure in memoria), quindi un dump scritto lì andrebbe perso al prossimo aggiornamento. I dump vanno solo fuori sede: qui non ne resta nessuno e prima di una migrazione non ne viene fatto nessuno. Monta il volume dei backup.",
"notPersistent": "{dir} non è su uno spazio persistente (si trova dentro il container e non su un volume montato, oppure in memoria) e non è configurato alcun bucket esterno, quindi nessun dump del database viene conservato da nessuna parte. Monta il volume dei backup."
}
},
"danger-zone": {
@@ -8689,7 +8691,9 @@
"stale": "Il dump del database più recente ha {hours} ore.",
"last_failed": "L'ultimo dump del database non è riuscito. Quello precedente è ancora recente.",
"fresh": "L'ultimo dump del database è recente.",
"nowhere_to_write": "La cartella dei dump {directory} non è scrivibile e non è configurato nessun bucket esterno, quindi non si può conservare alcun dump."
"nowhere_to_write": "La cartella dei dump {directory} non è scrivibile e non è configurato nessun bucket esterno, quindi non si può conservare alcun dump.",
"not_persistent": "La cartella dei dump {directory} non è su uno spazio persistente e non è configurato alcun bucket esterno, quindi ogni dump andrebbe perso al prossimo aggiornamento. Monta il volume dei backup.",
"fresh_offhost_only": "Il dump del database più recente è aggiornato e si trova nel bucket esterno. Su questo server non ne viene conservato nessuno, perché {directory} non è su uno spazio persistente, e prima di una migrazione non ne viene fatto nessuno."
},
"weekly_copy": {
"no_accounts": "Non ci sono ancora account da copiare.",
+6 -2
View File
@@ -8321,7 +8321,9 @@
"noDirectory": "이 서버는 {dir}에 쓸 수 없고 외부 저장소도 설정되지 않아 데이터베이스 덤프를 만들 수 없습니다. 백업 볼륨을 마운트하세요.",
"retentionLocal": "삭제된 계정은 덤프가 만료될 때까지 덤프에 남습니다. 이 서버에는 최신 덤프 {nights}개가 보관됩니다.",
"retentionOffhost": "버킷의 수명 주기 규칙이 외부 사본을 {days}일 후에 삭제합니다.",
"retentionOffhostMissing": "버킷의 어떤 수명 주기 규칙도 database/를 다루지 않아 그곳의 덤프와 그 안의 삭제된 계정이 절대 삭제되지 않습니다. 이 접두사에 대한 규칙을 추가하세요."
"retentionOffhostMissing": "버킷의 어떤 수명 주기 규칙도 database/를 다루지 않아 그곳의 덤프와 그 안의 삭제된 계정이 절대 삭제되지 않습니다. 이 접두사에 대한 규칙을 추가하세요.",
"notPersistentOffhost": "{dir}은(는) 영구 저장소에 있지 않습니다(마운트된 볼륨이 아니라 컨테이너 안이나 메모리에 있음). 그래서 여기에 쓴 덤프는 다음 업데이트 때 사라집니다. 덤프는 외부 저장소로만 가며, 여기에는 보관되지 않고 마이그레이션 전에도 만들어지지 않습니다. 백업 볼륨을 마운트하세요.",
"notPersistent": "{dir}은(는) 영구 저장소에 있지 않고(마운트된 볼륨이 아니라 컨테이너 안이나 메모리에 있음) 외부 버킷도 설정되지 않아, 데이터베이스 덤프가 어디에도 보관되지 않습니다. 백업 볼륨을 마운트하세요."
}
},
"danger-zone": {
@@ -8689,7 +8691,9 @@
"stale": "최신 데이터베이스 덤프가 {hours}시간 전 것입니다.",
"last_failed": "마지막 데이터베이스 덤프가 실패했습니다. 그 전 덤프는 아직 최근 것입니다.",
"fresh": "최신 데이터베이스 덤프가 최근 것입니다.",
"nowhere_to_write": "덤프 디렉터리 {directory}에 쓸 수 없고 외부 버킷도 설정되어 있지 않아 덤프를 보관할 수 없습니다."
"nowhere_to_write": "덤프 디렉터리 {directory}에 쓸 수 없고 외부 버킷도 설정되어 있지 않아 덤프를 보관할 수 없습니다.",
"not_persistent": "덤프 디렉터리 {directory}이(가) 영구 저장소에 있지 않고 외부 버킷도 설정되지 않아, 모든 덤프가 다음 업데이트 때 사라집니다. 백업 볼륨을 마운트하세요.",
"fresh_offhost_only": "최신 데이터베이스 덤프는 최근 것이며 외부 버킷에 있습니다. {directory}이(가) 영구 저장소에 있지 않아 이 서버에는 보관되지 않으며, 마이그레이션 전에도 만들어지지 않습니다."
},
"weekly_copy": {
"no_accounts": "아직 복사할 계정이 없습니다.",
+6 -2
View File
@@ -8346,7 +8346,9 @@
"noDirectory": "Deze server kan niet schrijven naar {dir} en er is geen externe opslag ingesteld, dus er kan geen databasedump worden gemaakt. Koppel het back-upvolume.",
"retentionLocal": "Verwijderde accounts blijven in de dumps tot die verlopen: op deze server blijven de nieuwste {nights} bewaard.",
"retentionOffhost": "De levenscyclusregel van de bucket verwijdert externe kopieën na {days} dagen.",
"retentionOffhostMissing": "Geen levenscyclusregel in de bucket geldt voor database/, dus dumps daar, en de verwijderde accounts erin, worden nooit verwijderd. Voeg een regel voor dat voorvoegsel toe."
"retentionOffhostMissing": "Geen levenscyclusregel in de bucket geldt voor database/, dus dumps daar, en de verwijderde accounts erin, worden nooit verwijderd. Voeg een regel voor dat voorvoegsel toe.",
"notPersistentOffhost": "{dir} staat niet op blijvende opslag (in de container in plaats van op een gekoppeld volume, of in het geheugen), dus een dump die daar wordt geschreven gaat bij de volgende update verloren. Dumps gaan alleen extern: hier wordt er geen bewaard en vóór een migratie wordt er geen gemaakt. Koppel het back-upvolume.",
"notPersistent": "{dir} staat niet op blijvende opslag (in de container in plaats van op een gekoppeld volume, of in het geheugen) en er is geen externe bucket ingesteld, dus nergens wordt een databasedump bewaard. Koppel het back-upvolume."
}
},
"danger-zone": {
@@ -8689,7 +8691,9 @@
"stale": "De nieuwste databasedump is {hours} uur oud.",
"last_failed": "De laatste databasedump is mislukt. De dump daarvoor is nog recent.",
"fresh": "De nieuwste databasedump is recent.",
"nowhere_to_write": "De dumpmap {directory} is niet beschrijfbaar en er is geen externe bucket ingesteld, dus er kan geen dump worden bewaard."
"nowhere_to_write": "De dumpmap {directory} is niet beschrijfbaar en er is geen externe bucket ingesteld, dus er kan geen dump worden bewaard.",
"not_persistent": "De dumpmap {directory} staat niet op blijvende opslag en er is geen externe bucket ingesteld, dus elke dump gaat bij de volgende update verloren. Koppel het back-upvolume.",
"fresh_offhost_only": "De nieuwste databasedump is recent en staat in de externe bucket. Op deze server wordt er geen bewaard, omdat {directory} niet op blijvende opslag staat, en vóór een migratie wordt er geen gemaakt."
},
"weekly_copy": {
"no_accounts": "Er zijn nog geen accounts om te kopiëren.",
+6 -2
View File
@@ -8346,7 +8346,9 @@
"noDirectory": "Ten serwer nie może zapisywać w {dir} i nie ustawiono magazynu zewnętrznego, więc nie da się wykonać zrzutu bazy. Podłącz wolumin kopii.",
"retentionLocal": "Usunięte konta pozostają w zrzutach, dopóki te nie wygasną: na tym serwerze zostaje {nights} najnowszych.",
"retentionOffhost": "Reguła cyklu życia bucketu usuwa kopie zewnętrzne po {days} dniach.",
"retentionOffhostMissing": "Żadna reguła cyklu życia bucketu nie obejmuje database/, więc zrzuty tam, wraz z usuniętymi kontami w nich, nigdy nie są usuwane. Dodaj regułę dla tego prefiksu."
"retentionOffhostMissing": "Żadna reguła cyklu życia bucketu nie obejmuje database/, więc zrzuty tam, wraz z usuniętymi kontami w nich, nigdy nie są usuwane. Dodaj regułę dla tego prefiksu.",
"notPersistentOffhost": "{dir} nie leży na trwałym nośniku (jest wewnątrz kontenera zamiast na podłączonym woluminie albo w pamięci), więc zapisany tam zrzut przepadnie przy następnej aktualizacji. Zrzuty trafiają tylko poza serwer: żaden nie zostaje tutaj i żaden nie powstaje przed migracją. Podłącz wolumin kopii.",
"notPersistent": "{dir} nie leży na trwałym nośniku (jest wewnątrz kontenera zamiast na podłączonym woluminie albo w pamięci), a zewnętrzny bucket nie jest ustawiony, więc żaden zrzut bazy nie jest nigdzie przechowywany. Podłącz wolumin kopii."
}
},
"danger-zone": {
@@ -8689,7 +8691,9 @@
"stale": "Najnowszy zrzut bazy danych ma {hours} h.",
"last_failed": "Ostatni zrzut bazy danych nie powiódł się. Poprzedni jest jeszcze aktualny.",
"fresh": "Najnowszy zrzut bazy danych jest aktualny.",
"nowhere_to_write": "Katalog zrzutów {directory} nie jest zapisywalny i nie skonfigurowano zewnętrznego bucketu, więc żaden zrzut nie może zostać zachowany."
"nowhere_to_write": "Katalog zrzutów {directory} nie jest zapisywalny i nie skonfigurowano zewnętrznego bucketu, więc żaden zrzut nie może zostać zachowany.",
"not_persistent": "Katalog zrzutów {directory} nie leży na trwałym nośniku, a zewnętrzny bucket nie jest ustawiony, więc każdy zrzut przepadnie przy następnej aktualizacji. Podłącz wolumin kopii.",
"fresh_offhost_only": "Najnowszy zrzut bazy jest aktualny i leży w zewnętrznym buckecie. Na tym serwerze żaden nie zostaje, bo {directory} nie leży na trwałym nośniku, i żaden nie powstaje przed migracją."
},
"weekly_copy": {
"no_accounts": "Nie ma jeszcze kont do skopiowania.",
+6 -2
View File
@@ -8346,7 +8346,9 @@
"noDirectory": "Este servidor não consegue gravar em {dir} e não há armazenamento externo configurado, então nenhum dump pode ser feito. Monte o volume de backups.",
"retentionLocal": "Contas excluídas continuam nos dumps até que eles expirem: este servidor guarda os {nights} mais recentes.",
"retentionOffhost": "A regra de ciclo de vida do bucket remove as cópias externas após {days} dias.",
"retentionOffhostMissing": "Nenhuma regra de ciclo de vida do bucket cobre database/, então os dumps ali, e as contas excluídas dentro deles, nunca são removidos. Adicione uma regra para esse prefixo."
"retentionOffhostMissing": "Nenhuma regra de ciclo de vida do bucket cobre database/, então os dumps ali, e as contas excluídas dentro deles, nunca são removidos. Adicione uma regra para esse prefixo.",
"notPersistentOffhost": "{dir} não está em armazenamento persistente (fica dentro do contêiner e não num volume montado, ou na memória), então um dump gravado ali se perde na próxima atualização. Os dumps vão só para fora: nenhum fica aqui e nenhum é feito antes de uma migração. Monte o volume de backups.",
"notPersistent": "{dir} não está em armazenamento persistente (fica dentro do contêiner e não num volume montado, ou na memória) e nenhum bucket externo está configurado, então nenhum dump do banco é guardado em lugar nenhum. Monte o volume de backups."
}
},
"danger-zone": {
@@ -8689,7 +8691,9 @@
"stale": "O dump do banco de dados mais recente tem {hours} horas.",
"last_failed": "O último dump do banco de dados falhou. O anterior ainda é recente.",
"fresh": "O dump do banco de dados mais recente está em dia.",
"nowhere_to_write": "Não é possível gravar no diretório de dumps {directory} e nenhum bucket externo está configurado, então nenhum dump pode ser guardado."
"nowhere_to_write": "Não é possível gravar no diretório de dumps {directory} e nenhum bucket externo está configurado, então nenhum dump pode ser guardado.",
"not_persistent": "A pasta de dumps {directory} não está em armazenamento persistente e nenhum bucket externo está configurado, então cada dump se perde na próxima atualização. Monte o volume de backups.",
"fresh_offhost_only": "O dump mais recente do banco é atual e está no bucket externo. Nenhum fica neste servidor, porque {directory} não está em armazenamento persistente, e nenhum é feito antes de uma migração."
},
"weekly_copy": {
"no_accounts": "Ainda não há contas para copiar.",
+3 -1
View File
@@ -6022,7 +6022,9 @@ model DatabaseBackupRun {
/// Size of the dump in bytes.
dumpBytes BigInt? @map("dump_bytes")
/// `written` (kept in DB_BACKUP_DIR), `no_directory` (no writable
/// directory, the dump went off-host only), or `failed`.
/// directory, the dump went off-host only), `not_persistent` (writable but
/// in the container's layer or in memory, so nothing was kept there),
/// `skipped`, or `failed`.
localResult String @map("local_result")
/// `uploaded`, `not_configured` or `failed`.
offhostResult String @map("offhost_result")
+31
View File
@@ -6,6 +6,10 @@
# 1. Build the app image from this checkout and start the bundled compose
# stack with fixed throwaway keys, and an S3-compatible bucket outside
# it (Versity S3 Gateway, pinned) as the off-host target.
# First without the `backups` volume: the run must keep nothing in the
# container's layer, say so in its ledger row and the start log, and
# still upload the dump. Then the stack is thrown away and started again
# as shipped.
# 2. Seed real data: the demo seed (documents, encrypted notes, years of
# readings) and a Guardian with a managed profile, written by the app's
# own code; then, through the API, a reading with an encrypted note.
@@ -184,6 +188,33 @@ docker run -d --name "$BUCKET" --network "$BUCKET" -p "127.0.0.1:$BUCKET_PORT:70
"mkdir -p /data/$BACKUP_S3_BUCKET && exec versitygw --access $BACKUP_S3_ACCESS_KEY --secret $BACKUP_S3_SECRET_KEY posix /data" \
>/dev/null || fail "the bucket did not start"
step "a server without the backups volume keeps no dump on itself"
# The image creates the dump directory, so without the volume it is writable,
# and it sits in the container's layer that the next update throws away. No
# dump may be kept, or counted as kept, there; the bucket still gets one.
cat >"$WORK/no-volume.yml" <<YAML
services:
app:
volumes: !reset []
YAML
stack -f "$WORK/no-volume.yml" up -d app >"$WORK/up.log" 2>&1 ||
{ tail -20 "$WORK/up.log" >&2; fail "the stack without the volume did not start"; }
wait_healthy
i=0
until RUN=$(sql "SELECT status || ' ' || local_result || ' ' || offhost_result FROM database_backup_runs WHERE status <> 'running' ORDER BY started_at DESC LIMIT 1") && [ -n "$RUN" ]; do
i=$((i + 1))
[ "$i" -lt 120 ] || fail "no database-backup run within four minutes without the volume"
sleep 2
done
[ "$RUN" = "succeeded not_persistent uploaded" ] ||
fail "without the volume the run read '$RUN', not 'succeeded not_persistent uploaded'"
LEFT=$(docker exec "$PROJECT-app" sh -c 'ls -A /var/lib/healthlog/backups')
[ -z "$LEFT" ] || fail "a dump was left in the container's layer: $LEFT"
docker logs "$PROJECT-app" 2>&1 | grep -q "is inside the container, not on a mounted volume" ||
fail "the start log does not say the dump directory is inside the container"
echo " $RUN, nothing in the container's layer"
stack down -v --remove-orphans >/dev/null 2>&1
step "a running server with real data"
stack up -d app >"$WORK/up.log" 2>&1 || { tail -20 "$WORK/up.log" >&2; fail "the stack did not start"; }
wait_healthy
+19 -7
View File
@@ -13,9 +13,11 @@
*
* Exit 0: the dump was written, or there was nothing to do (no pending
* migration, a new database, `DB_BACKUP_BEFORE_MIGRATE=false`), or there is no
* writable directory to put it in, which is said as a warning and is not a
* reason to keep the app down. Exit 1: the directory is writable and the dump
* failed. The entrypoint then stops the boot and names the switch that skips
* directory to put it in that survives the container (none writable, or only
* the image's own directory without the volume, which the update that brings
* these migrations is about to throw away), which is said as a warning and is
* not a reason to keep the app down. Exit 1: the directory is kept and the
* dump failed. The entrypoint then stops the boot and names the switch that skips
* the dump, because migrating without the way back the operator was promised
* is the one thing this step exists to prevent.
*
@@ -28,9 +30,9 @@ import { pathToFileURL } from "node:url";
import {
applyRetention,
backupDirectoryStatus,
connect,
createDatabaseDump,
isWritableDirectory,
latestMigration,
listUserTables,
connectionFromUrl,
@@ -60,7 +62,11 @@ function mb(bytes) {
return `${(bytes / 1024 / 1024).toFixed(1)} MB`;
}
export async function preMigrationDump(env = process.env) {
/**
* @param {Record<string, string | undefined>} [env]
* @param {{ directoryStatus?: typeof backupDirectoryStatus }} [options]
*/
export async function preMigrationDump(env = process.env, options = {}) {
if (!preMigrationDumpEnabled(env)) {
log("DB_BACKUP_BEFORE_MIGRATE=false, no dump before migrating.");
return 0;
@@ -101,9 +107,15 @@ export async function preMigrationDump(env = process.env) {
}
const dir = resolveBackupDir(env);
if (!isWritableDirectory(dir)) {
const status = (options.directoryStatus ?? backupDirectoryStatus)(dir);
if (!status.persistent) {
const why = !status.writable
? "is not writable"
: status.reason === "memory"
? "is in memory (tmpfs) and would not keep a dump past a restart"
: "is inside the container, not on a mounted volume, and would not keep a dump past this update";
warn(
`${dir} is not writable, so no dump is taken before the ${pending.length} pending migration(s). ` +
`${dir} ${why}, so no dump is taken before the ${pending.length} pending migration(s). ` +
"Mount the backups volume (docker-compose.yml) to keep a way back; continuing.",
);
return 0;
+1 -1
View File
@@ -372,7 +372,7 @@
},
{
"name": "DB_BACKUP_DIR",
"purpose": "Directory the dumps are written to and restored from. Default /var/lib/healthlog/backups, the `backups` volume. Without a writable directory the nightly dump only goes off-host and no dump is taken before migrations.",
"purpose": "Directory the dumps are written to and restored from. Default /var/lib/healthlog/backups, the `backups` volume. In a container it must be a mount of its own: a directory in the container's layer is lost at the next update and counts as none. Without one the nightly dump only goes off-host and no dump is taken before migrations.",
"example": "/var/lib/healthlog/backups"
},
{
@@ -63,6 +63,8 @@ const databaseHealth = {
directory: "/var/lib/healthlog/backups",
keepNightly: 7,
localDirWritable: true,
localPersistent: true,
localReason: "volume",
offhostConfigured: true,
lastGood: {
at: "2026-09-21T23:31:40.000Z",
@@ -17,6 +17,8 @@ const HEALTHY: DatabaseBackupHealth = {
directory: "/var/lib/healthlog/backups",
keepNightly: 7,
localDirWritable: true,
localPersistent: true,
localReason: "volume",
offhostConfigured: true,
lastGood: {
at: "2026-10-24T23:31:00.000Z",
@@ -77,17 +79,36 @@ describe("<DatabaseDumpRow>", () => {
});
it("says when dumps go off-host only, or cannot be taken at all", () => {
expect(render({ ...HEALTHY, localDirWritable: false }, null)).toContain(
"dumps go off-host only",
const unwritable = {
...HEALTHY,
localDirWritable: false,
localPersistent: false,
localReason: "not_writable" as const,
};
expect(render(unwritable, null)).toContain("dumps go off-host only");
expect(render({ ...unwritable, offhostConfigured: false }, null)).toContain(
"no database dump can be taken",
);
expect(
render(
{ ...HEALTHY, localDirWritable: false, offhostConfigured: false },
null,
),
).toContain("no database dump can be taken");
expect(render({ ...HEALTHY, lastGood: null }, null)).toContain(
"No database dump yet",
);
});
it("does not promise dumps on this server from a directory inside the container", () => {
const layer = {
...HEALTHY,
localPersistent: false,
localReason: "container_layer" as const,
lastGood: { ...HEALTHY.lastGood!, local: false },
};
const html = render(layer);
expect(html).toContain('data-slot="database-dump-setup"');
expect(html).toContain("lost at the next update");
expect(html).toContain("Dumps go off-host only");
expect(html).not.toContain("kept on this server");
expect(html).toContain("removed after 30 days");
const nowhere = render({ ...layer, offhostConfigured: false }, null);
expect(nowhere).toContain("no database dump is kept anywhere");
expect(nowhere).not.toContain('data-slot="database-dump-retention"');
});
});
@@ -6,7 +6,10 @@
* The account copies below it restore one account on this server; the dump is
* the copy that rebuilds the server. So the row leads the card and says three
* things: when the newest good dump was taken and where it is, whether the
* newest run failed, and whether this server can keep a dump at all.
* newest run failed, and whether this server can keep a dump at all. A
* directory inside the container (the image's own, without the volume) is
* writable and still keeps nothing past the next update, so "kept on this
* server" is only said of a directory that survives it.
*/
import { AlertTriangle, HardDrive } from "lucide-react";
@@ -45,7 +48,15 @@ export function DatabaseDumpRow({
let setupProblem: string | null = null;
if (!health.enabled) {
setupProblem = t("admin.section.backups.database.disabled");
} else if (!health.localDirWritable) {
} else if (health.localDirWritable && !health.localPersistent) {
setupProblem = health.offhostConfigured
? t("admin.section.backups.database.notPersistentOffhost", {
dir: health.directory,
})
: t("admin.section.backups.database.notPersistent", {
dir: health.directory,
});
} else if (!health.localPersistent) {
setupProblem = health.offhostConfigured
? t("admin.section.backups.database.noDirectoryOffhost", {
dir: health.directory,
@@ -55,6 +66,19 @@ export function DatabaseDumpRow({
});
}
const retention = [
health.localPersistent
? t("admin.section.backups.database.retentionLocal", {
nights: health.keepNightly,
})
: null,
bucketLifecycle?.state === "configured"
? t("admin.section.backups.database.retentionOffhost", {
days: bucketLifecycle.expirationDays ?? 0,
})
: null,
].filter((line): line is string => line !== null);
return (
<div className="space-y-2" data-slot="database-dump">
<div className="flex items-start gap-2">
@@ -80,19 +104,14 @@ export function DatabaseDumpRow({
</p>
{/* How long a deleted account stays in the dumps: the operator's
side of the deletion promise (GDPR retention of backups). */}
<p
className="text-muted-foreground text-xs"
data-slot="database-dump-retention"
>
{t("admin.section.backups.database.retentionLocal", {
nights: health.keepNightly,
})}
{bucketLifecycle?.state === "configured"
? ` ${t("admin.section.backups.database.retentionOffhost", {
days: bucketLifecycle.expirationDays ?? 0,
})}`
: null}
</p>
{retention.length > 0 ? (
<p
className="text-muted-foreground text-xs"
data-slot="database-dump-retention"
>
{retention.join(" ")}
</p>
) : null}
</div>
</div>
@@ -158,43 +158,67 @@ describe("collectReadiness — the database dump", () => {
BACKUP_S3_SECRET_KEY: "s",
BACKUP_ENCRYPTION_KEY: HEX32,
};
const STATUS = {
volume: { writable: true, persistent: true, reason: "volume" },
layer: { writable: true, persistent: false, reason: "container_layer" },
memory: { writable: true, persistent: false, reason: "memory" },
none: { writable: false, persistent: false, reason: "not_writable" },
} as const;
const dumpLine = (
env: Record<string, string | undefined>,
writable: boolean,
status: keyof typeof STATUS,
) =>
collectReadiness(env, () => writable).lines.find(
collectReadiness(env, () => STATUS[status]).lines.find(
(l) => l.label === "Database dump",
);
it("names the schedule, the directory and the copies kept", () => {
const line = dumpLine(baseEnv(), true);
const line = dumpLine(baseEnv(), "volume");
expect(line?.status).toBe("ok");
expect(line?.detail).toBe(
"nightly (30 1 * * *) into /var/lib/healthlog/backups, keeping 7",
);
expect(dumpLine({ ...baseEnv(), ...offhost }, true)?.detail).toContain(
expect(dumpLine({ ...baseEnv(), ...offhost }, "volume")?.detail).toContain(
"copied off-host",
);
});
it("warns when nothing can be kept on the server", () => {
expect(dumpLine({ ...baseEnv(), ...offhost }, false)).toMatchObject({
expect(dumpLine({ ...baseEnv(), ...offhost }, "none")).toMatchObject({
status: "warn",
detail: expect.stringContaining("goes off-host only"),
});
expect(dumpLine(baseEnv(), false)).toMatchObject({
expect(dumpLine(baseEnv(), "none")).toMatchObject({
status: "warn",
detail: expect.stringContaining("no database dump can be taken"),
detail: expect.stringContaining("no database dump is kept anywhere"),
});
});
it("does not call a directory inside the container kept: it is writable and lost at the next update", () => {
const offhostOnly = dumpLine({ ...baseEnv(), ...offhost }, "layer");
expect(offhostOnly?.status).toBe("warn");
expect(offhostOnly?.detail).toContain(
"/var/lib/healthlog/backups is inside the container, not on a mounted volume",
);
expect(offhostOnly?.detail).toContain("none is kept on this server");
expect(offhostOnly?.detail).toContain("none is taken before a migration");
expect(offhostOnly?.detail).not.toContain("keeping 7");
expect(dumpLine(baseEnv(), "layer")).toMatchObject({
status: "warn",
detail: expect.stringContaining("no database dump is kept anywhere"),
});
expect(dumpLine(baseEnv(), "memory")?.detail).toContain(
"lost at the next restart",
);
});
it("warns when it is switched off, and names a schedule it ignored", () => {
expect(
dumpLine({ ...baseEnv(), DB_BACKUP_ENABLED: "false" }, true),
dumpLine({ ...baseEnv(), DB_BACKUP_ENABLED: "false" }, "volume"),
).toMatchObject({ status: "warn" });
const report = collectReadiness(
{ ...baseEnv(), DB_BACKUP_SCHEDULE: "nightly" },
() => true,
() => STATUS.volume,
);
expect(
report.lines.find((l) => l.label === "DB_BACKUP_SCHEDULE")?.detail,
+28 -18
View File
@@ -20,11 +20,12 @@ import {
legacyAnyHostConfigured,
} from "@/lib/ai/local-host-allowlist";
import {
backupDirectoryStatus,
databaseBackupEnabled,
isWritableDirectory,
resolveBackupDir,
resolveBackupSchedule,
resolveDailyKeep,
type BackupDirectoryStatus,
} from "@/lib/database-backup/dump-core.mjs";
export interface ReadinessLine {
@@ -74,7 +75,7 @@ function anySet(env: Env, keys: string[]): boolean {
*/
export function databaseBackupReadiness(
env: Env,
writable: (dir: string) => boolean,
directoryStatus: (dir: string) => BackupDirectoryStatus,
): ReadinessLine[] {
const label = "Database dump";
if (!databaseBackupEnabled(env)) {
@@ -104,35 +105,44 @@ export function databaseBackupReadiness(
"BACKUP_S3_SECRET_KEY",
"BACKUP_ENCRYPTION_KEY",
]);
if (writable(dir)) {
const status = directoryStatus(dir);
if (status.persistent) {
lines.push({
label,
status: "ok",
detail: `nightly (${schedule.cron}) into ${dir}, keeping ${resolveDailyKeep(env)}${offhost ? ", copied off-host" : ""}`,
});
} else if (offhost) {
lines.push({
label,
status: "warn",
detail: `${dir} is not writable: the nightly dump goes off-host only and none is kept on this server. Mount the backups volume (see docker-compose.yml) or set DB_BACKUP_DIR`,
});
} else {
lines.push({
label,
status: "warn",
detail: `${dir} is not writable and no off-host bucket is configured: no database dump can be taken. Mount the backups volume (see docker-compose.yml) or set DB_BACKUP_DIR`,
});
return lines;
}
// Writable but thrown away with the container (the image's own directory
// without the volume) or at a restart (a tmpfs): no dump is kept there,
// because one that is would be counted as kept and then lost.
const where = !status.writable
? `${dir} is not writable`
: status.reason === "memory"
? `${dir} is in memory (tmpfs), so a dump there is lost at the next restart`
: `${dir} is inside the container, not on a mounted volume, so a dump there is lost at the next update`;
const fix =
"Mount the backups volume (see docker-compose.yml) or set DB_BACKUP_DIR to a mounted directory";
lines.push({
label,
status: "warn",
detail: offhost
? `${where}: the nightly dump goes off-host only, none is kept on this server and none is taken before a migration. ${fix}`
: `${where} and no off-host bucket is configured: no database dump is kept anywhere. ${fix}`,
});
return lines;
}
/**
* Inspect the environment and build the readiness report. Pure apart from
* `writable`, the probe of the dump directory, which tests replace.
* `directoryStatus`, the probe of the dump directory, which tests replace.
*/
export function collectReadiness(
env: Env = process.env,
writable: (dir: string) => boolean = isWritableDirectory,
directoryStatus: (
dir: string,
) => BackupDirectoryStatus = backupDirectoryStatus,
): ReadinessReport {
const lines: ReadinessLine[] = [];
@@ -211,7 +221,7 @@ export function collectReadiness(
});
}
lines.push(...databaseBackupReadiness(env, writable));
lines.push(...databaseBackupReadiness(env, directoryStatus));
// --- Optional subsystems (informational) --------------------------------
const optional: Array<[string, boolean]> = [
@@ -25,6 +25,7 @@ import { afterEach, describe, expect, it, vi } from "vitest";
import {
applyRetention,
backupDirectoryStatus,
bucketConfigFromEnv,
pickBucketDump,
buildManifest,
@@ -37,13 +38,17 @@ import {
formatDumpTimestamp,
keyFingerprints,
listDumpSets,
mountContaining,
offhostObjectKey,
parseMountInfo,
parseDumpFileName,
parseManifest,
resolveBackupSchedule,
resolveDailyKeep,
selectForDeletion,
type DirectoryProbe,
type DumpSet,
type Env,
} from "../dump-core.mjs";
import type { S3Like } from "@/lib/jobs/offhost-backup";
@@ -687,3 +692,168 @@ describe("finding a dump in the bucket", () => {
).toBeNull();
});
});
describe("whether the dump directory survives the container", () => {
// /proc/self/mountinfo of the app container under the bundled compose
// file, trimmed to the lines that matter, with and without the volume.
const ROOT =
"1470 1351 0:330 / / rw,relatime master:396 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/A:/var/lib/docker/overlay2/l/B,upperdir=/var/lib/docker/overlay2/x/diff,workdir=/var/lib/docker/overlay2/x/work";
const COMMON = [
"1471 1470 0:333 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw",
"1472 1470 0:334 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755",
"1481 1470 8:1 /var/lib/docker/containers/c/resolv.conf /etc/resolv.conf rw,relatime - ext4 /dev/sda1 rw",
"1483 1470 8:1 /var/lib/docker/containers/c/hosts /etc/hosts rw,relatime - ext4 /dev/sda1 rw",
];
const VOLUME =
"1480 1470 8:1 /var/lib/docker/volumes/healthlog_backups/_data /var/lib/healthlog/backups rw,relatime master:1 - ext4 /dev/sda1 rw,discard";
const DIR = "/var/lib/healthlog/backups";
function probe(
over: Partial<DirectoryProbe> & { files?: Record<string, string> } = {},
): DirectoryProbe {
const { files = {}, ...rest } = over;
return {
writable: () => true,
realpath: (p) => p,
readText: (p) => files[p] ?? null,
exists: (p) => p in files,
device: () => null,
env: {},
...rest,
};
}
const inDocker = (...lines: string[]) =>
probe({
files: {
"/.dockerenv": "",
"/proc/self/mountinfo": [ROOT, ...COMMON, ...lines].join("\n"),
},
});
it("reads mount points and types, escapes undone, odd lines skipped", () => {
const mounts = parseMountInfo(
[
ROOT,
"90 1470 8:1 / /mnt/my\\040disk rw - ext4 /dev/sdb1 rw",
"not a mount line",
"",
].join("\n"),
);
expect(mounts).toEqual([
{ mountPoint: "/", fsType: "overlay" },
{ mountPoint: "/mnt/my disk", fsType: "ext4" },
]);
});
it("finds the longest mount that holds a path, by whole segments", () => {
const mounts = [
{ mountPoint: "/", fsType: "overlay" },
{ mountPoint: "/var/lib/healthlog/back", fsType: "ext4" },
{ mountPoint: "/var/lib/healthlog", fsType: "xfs" },
];
expect(mountContaining(DIR, mounts)?.mountPoint).toBe("/var/lib/healthlog");
expect(mountContaining("/var/lib/other", mounts)?.mountPoint).toBe("/");
expect(mountContaining("/var/lib/healthlog", mounts)?.fsType).toBe("xfs");
});
it("counts the backups volume as kept", () => {
expect(backupDirectoryStatus(DIR, inDocker(VOLUME))).toEqual({
writable: true,
persistent: true,
reason: "volume",
});
});
it("does not count the image's own directory without the volume, though it is writable", () => {
expect(backupDirectoryStatus(DIR, inDocker())).toEqual({
writable: true,
persistent: false,
reason: "container_layer",
});
});
it("counts a volume on a parent directory, and follows a symlink to where the files land", () => {
const parent =
"1480 1470 8:1 /srv/healthlog /var/lib/healthlog rw - ext4 /dev/sda1 rw";
expect(backupDirectoryStatus(DIR, inDocker(parent)).persistent).toBe(true);
const linked = {
...inDocker(VOLUME),
realpath: (p: string) => (p === "/backups" ? DIR : p),
};
expect(backupDirectoryStatus("/backups", linked).reason).toBe("volume");
expect(backupDirectoryStatus("/elsewhere", inDocker(VOLUME)).reason).toBe(
"container_layer",
);
});
it("does not count a tmpfs as kept", () => {
const tmpfs =
"1480 1470 0:400 / /var/lib/healthlog/backups rw - tmpfs tmpfs rw,size=1g";
expect(backupDirectoryStatus(DIR, inDocker(tmpfs))).toEqual({
writable: true,
persistent: false,
reason: "memory",
});
});
it("recognises a container by Podman's marker, Kubernetes, the cgroup or an overlay root", () => {
const table = [ROOT, ...COMMON].join("\n");
const cases: Array<{ files?: Record<string, string>; env?: Env }> = [
{ files: { "/run/.containerenv": "" } },
{ env: { KUBERNETES_SERVICE_HOST: "10.0.0.1" } },
{ files: { "/proc/self/cgroup": "12:pids:/docker/0123abcd\n" } },
{},
];
for (const extra of cases) {
const p = probe({
env: extra.env ?? {},
files: { ...extra.files, "/proc/self/mountinfo": table },
});
expect(backupDirectoryStatus(DIR, p).reason).toBe("container_layer");
}
});
it("treats a writable directory outside a container as kept: a host, a VM", () => {
const host = probe({
files: {
"/proc/self/cgroup": "0::/user.slice/user-1000.slice\n",
"/proc/self/mountinfo":
"22 1 259:2 / / rw,relatime shared:1 - ext4 /dev/nvme0n1p2 rw",
},
});
expect(backupDirectoryStatus(DIR, host)).toEqual({
writable: true,
persistent: true,
reason: "host",
});
// No /proc at all (macOS, a BSD): a host too.
expect(backupDirectoryStatus(DIR, probe()).reason).toBe("host");
});
it("falls back to the device numbers in a container without a mount table", () => {
const same = probe({ files: { "/.dockerenv": "" }, device: () => 42 });
expect(backupDirectoryStatus(DIR, same).reason).toBe("container_layer");
const apart = probe({
files: { "/.dockerenv": "" },
device: (p) => (p === "/" ? 42 : 7),
});
expect(backupDirectoryStatus(DIR, apart).reason).toBe("volume");
});
it("says when nothing can be written at all, before anything else", () => {
expect(
backupDirectoryStatus(DIR, {
...inDocker(VOLUME),
writable: () => false,
}),
).toEqual({ writable: false, persistent: false, reason: "not_writable" });
});
it("answers for a real directory on this machine without throwing", () => {
const status = backupDirectoryStatus(tempDir());
expect(status.writable).toBe(true);
expect(["host", "volume", "container_layer", "memory"]).toContain(
status.reason,
);
});
});
+190
View File
@@ -33,10 +33,13 @@ import {
constants as fsConstants,
createReadStream,
createWriteStream,
existsSync,
mkdirSync,
openSync,
readFileSync,
readSync,
readdirSync,
realpathSync,
statSync,
} from "node:fs";
import { readFile, rename, rm, stat, writeFile } from "node:fs/promises";
@@ -222,6 +225,193 @@ export function isWritableDirectory(dir) {
}
}
// ── Whether a dump kept there survives ─────────────────────────────────────
/**
* File systems that live in memory: a dump on one is gone at the next restart.
*/
const EPHEMERAL_FS_TYPES = new Set(["tmpfs", "ramfs"]);
/**
* cgroup paths that name a container runtime (cgroup v1; under v2 a container
* sees `0::/`, and the marker files or the root mount answer instead).
*/
const CONTAINER_CGROUP = /\/(docker|kubepods|containerd|libpod|podman)[/-]/;
/**
* @typedef {{ mountPoint: string; fsType: string }} MountEntry
*/
/** Undo the octal escapes the kernel writes for space, tab, newline, `\`. */
function unescapeMountPath(raw) {
return raw.replace(/\\([0-7]{3})/g, (_, oct) =>
String.fromCharCode(Number.parseInt(oct, 8)),
);
}
/**
* The mount points and file system types in `/proc/self/mountinfo` text.
* Lines that do not have the documented shape are skipped.
*
* @param {string} text
* @returns {MountEntry[]}
*/
export function parseMountInfo(text) {
/** @type {MountEntry[]} */
const mounts = [];
for (const line of text.split("\n")) {
const sep = line.indexOf(" - ");
if (sep < 0) continue;
const left = line.slice(0, sep).split(" ");
const right = line.slice(sep + 3).split(" ");
if (left.length < 6 || !right[0]) continue;
mounts.push({ mountPoint: unescapeMountPath(left[4]), fsType: right[0] });
}
return mounts;
}
/**
* The mount `path` lives on: the longest mount point that is `path` or one of
* its parents. A later entry for the same point wins, as it covers the earlier
* one. Null when nothing matches (no `/` in the list).
*
* @param {string} path an absolute path, symlinks resolved
* @param {MountEntry[]} mounts
* @returns {MountEntry | null}
*/
export function mountContaining(path, mounts) {
/** @type {MountEntry | null} */
let best = null;
for (const mount of mounts) {
const point = mount.mountPoint;
const covers =
point === "/" ||
path === point ||
path.startsWith(point.endsWith("/") ? point : `${point}/`);
if (covers && (best === null || point.length >= best.mountPoint.length)) {
best = mount;
}
}
return best;
}
/**
* @typedef {{
* writable: (dir: string) => boolean;
* realpath: (path: string) => string;
* readText: (path: string) => string | null;
* exists: (path: string) => boolean;
* device: (path: string) => number | null;
* env: Env;
* }} DirectoryProbe
*/
/** @type {DirectoryProbe} */
const SYSTEM_PROBE = {
writable: isWritableDirectory,
realpath: (path) => {
try {
return realpathSync(path);
} catch {
return path;
}
},
readText: (path) => {
try {
return readFileSync(path, "utf8");
} catch {
return null;
}
},
exists: (path) => existsSync(path),
device: (path) => {
try {
return statSync(path).dev;
} catch {
return null;
}
},
env: process.env,
};
/**
* Whether this process runs in a container whose root file system is thrown
* away when the container is replaced: Docker (`/.dockerenv`), Podman
* (`/run/.containerenv`), Kubernetes, a container cgroup, or a root file
* system that is an overlay. A host, a VM or a system container with a disk
* of its own answers false.
*
* @param {DirectoryProbe} probe
* @param {MountEntry[] | null} mounts
*/
export function runsInContainer(probe, mounts) {
if (probe.exists("/.dockerenv") || probe.exists("/run/.containerenv")) {
return true;
}
if ((probe.env.KUBERNETES_SERVICE_HOST ?? "").trim()) return true;
if (CONTAINER_CGROUP.test(probe.readText("/proc/self/cgroup") ?? "")) {
return true;
}
const root = mounts ? mountContaining("/", mounts) : null;
return root?.mountPoint === "/" && root.fsType === "overlay";
}
/**
* @typedef {{
* writable: boolean;
* persistent: boolean;
* reason: "volume" | "host" | "container_layer" | "memory" | "not_writable";
* }} BackupDirectoryStatus
*/
/**
* Whether a dump written to `dir` is still there after the container is
* replaced, which is what "kept on this server" promises.
*
* volume — in a container, on a mount of its own (the `backups`
* volume, a bind mount): it survives.
* host — not in a container: a writable directory is a disk.
* container_layer — in a container, on its root file system: an update
* replaces the container and the dump with it. The image
* creates the default directory, so without the volume it
* is writable and looks fine; this is the case to catch.
* memory — on a tmpfs: gone at the next restart.
* not_writable — nothing can be written there at all.
*
* Never throws. `probe` is the file system; tests hand in their own.
*
* @param {string} dir
* @param {DirectoryProbe} [probe]
* @returns {BackupDirectoryStatus}
*/
export function backupDirectoryStatus(dir, probe = SYSTEM_PROBE) {
if (!probe.writable(dir)) {
return { writable: false, persistent: false, reason: "not_writable" };
}
const text = probe.readText("/proc/self/mountinfo");
const mounts = text === null ? null : parseMountInfo(text);
if (!runsInContainer(probe, mounts)) {
return { writable: true, persistent: true, reason: "host" };
}
const real = probe.realpath(dir);
const mount = mounts ? mountContaining(real, mounts) : null;
if (mount) {
if (EPHEMERAL_FS_TYPES.has(mount.fsType)) {
return { writable: true, persistent: false, reason: "memory" };
}
return mount.mountPoint === "/"
? { writable: true, persistent: false, reason: "container_layer" }
: { writable: true, persistent: true, reason: "volume" };
}
// No mount table to read: a directory on the root's own device is in the
// container's layer.
const own = probe.device(real);
const root = probe.device("/");
return own !== null && own === root
? { writable: true, persistent: false, reason: "container_layer" }
: { writable: true, persistent: true, reason: "volume" };
}
// ── Keys ───────────────────────────────────────────────────────────────────
/**
+32 -10
View File
@@ -3,8 +3,8 @@
* alert can act on.
*
* Read from the ledger the job writes (`DatabaseBackupRun`), plus two facts of
* this process: whether `DB_BACKUP_DIR` is writable here, and whether the
* off-host bucket is configured. A ledger row that says a copy went off-host
* this process: whether `DB_BACKUP_DIR` is writable here and survives the
* container being replaced, and whether the off-host bucket is configured. A ledger row that says a copy went off-host
* outranks this process's environment, for the reason the off-host card gives
* (a split install can hand the bucket variables to the worker only).
*/
@@ -13,10 +13,11 @@ import { DATABASE_BACKUP_EXPIRE_SECONDS } from "@/lib/jobs/database-backup-queue
import { offhostBackupConfigured } from "@/lib/jobs/offhost-backup";
import {
backupDirectoryStatus,
databaseBackupEnabled,
isWritableDirectory,
resolveBackupDir,
resolveDailyKeep,
type BackupDirectoryStatus,
} from "./dump-core.mjs";
export interface DatabaseBackupHealth {
@@ -30,8 +31,20 @@ export interface DatabaseBackupHealth {
* page says.
*/
keepNightly: number;
/** This process can write there. False means dumps only go off-host. */
/** This process can write there. */
localDirWritable: boolean;
/**
* A dump written there survives an update: the directory is on a volume or
* a bind mount, or the app does not run in a container. False means no dump
* is kept on this server, whatever an older ledger row says, and dumps only
* go off-host. `localReason` says why.
*/
localPersistent: boolean;
/**
* `volume`, `host`, `container_layer` (the image's own directory, without
* the volume mounted), `memory` (a tmpfs) or `not_writable`.
*/
localReason: BackupDirectoryStatus["reason"];
/** The off-host bucket is configured, or a dump has been uploaded to it. */
offhostConfigured: boolean;
/** The newest dump that was kept on the server or reached the bucket. */
@@ -55,15 +68,22 @@ const ABANDONED_AFTER_MS = DATABASE_BACKUP_EXPIRE_SECONDS * 1000;
export async function getDatabaseBackupHealth(
now: Date = new Date(),
env: NodeJS.ProcessEnv = process.env,
directoryStatus: (
dir: string,
) => BackupDirectoryStatus = backupDirectoryStatus,
): Promise<DatabaseBackupHealth> {
const directory = resolveBackupDir(env);
const local = directoryStatus(directory);
// A dump recorded as written into a directory that does not survive the
// container was lost with the container that wrote it: only a copy in the
// bucket still counts.
const keptWhere = local.persistent
? [{ localResult: "written" }, { offhostResult: "uploaded" }]
: [{ offhostResult: "uploaded" }];
const [lastGood, lastFailed, lastAbandoned, everUploaded] = await Promise.all(
[
prisma.databaseBackupRun.findFirst({
where: {
fileName: { not: null },
OR: [{ localResult: "written" }, { offhostResult: "uploaded" }],
},
where: { fileName: { not: null }, OR: keptWhere },
orderBy: { startedAt: "desc" },
select: {
startedAt: true,
@@ -108,7 +128,9 @@ export async function getDatabaseBackupHealth(
enabled: databaseBackupEnabled(env),
directory,
keepNightly: resolveDailyKeep(env),
localDirWritable: isWritableDirectory(directory),
localDirWritable: local.writable,
localPersistent: local.persistent,
localReason: local.reason,
offhostConfigured: offhostBackupConfigured() || everUploaded > 0,
lastGood: lastGood
? {
@@ -116,7 +138,7 @@ export async function getDatabaseBackupHealth(
fileName: lastGood.fileName ?? "",
sizeBytes:
lastGood.dumpBytes === null ? null : Number(lastGood.dumpBytes),
local: lastGood.localResult === "written",
local: local.persistent && lastGood.localResult === "written",
offhost: lastGood.offhostResult === "uploaded",
}
: null,
+28 -14
View File
@@ -9,11 +9,14 @@
* under `database/` in the same envelope the per-account copies use, sealed
* with `BACKUP_ENCRYPTION_KEY`.
*
* Without a writable directory (an install that runs the image without the
* volume) the dump is written to a temporary directory, uploaded and removed:
* the off-host copy still happens, nothing is kept on the server, and the
* ledger and the boot summary say so. With neither, the run is skipped and
* recorded as such.
* Without a directory that survives the container (an install that runs the
* image without the volume: the image's own directory is writable but sits in
* the container's layer, which the next update throws away) the dump is
* written to a temporary directory, uploaded and removed: the off-host copy
* still happens, nothing is kept on the server, and the ledger and the boot
* summary say so. A dump left in the container's layer would be counted as
* kept and lost with the container, so none is. With neither, the run is
* skipped and recorded as such.
*
* Every run leaves one `DatabaseBackupRun` row: started, then finished with
* what happened on each leg. `getDatabaseBackupHealth` reads them.
@@ -35,19 +38,20 @@ import packageJson from "../../../package.json";
import {
applyRetention,
backupDirectoryStatus,
createDatabaseDump,
databaseBackupEnabled,
isWritableDirectory,
offhostObjectKey,
resolveBackupDir,
resolveDailyKeep,
type BackupDirectoryStatus,
} from "./dump-core.mjs";
/** How long ledger rows are kept. A run a day; a year and a bit of history. */
const LEDGER_RETENTION_DAYS = 400;
export type DatabaseBackupLocalResult =
"written" | "no_directory" | "failed" | "skipped";
"written" | "no_directory" | "not_persistent" | "failed" | "skipped";
export type DatabaseBackupOffhostResult =
"uploaded" | "not_configured" | "failed" | "skipped";
@@ -69,6 +73,8 @@ export interface RunDatabaseBackupOptions {
env?: NodeJS.ProcessEnv;
/** A bucket double for tests; the configured bucket otherwise. */
s3?: S3Like;
/** Where `DB_BACKUP_DIR` stands; the file system's answer otherwise. */
directoryStatus?: (dir: string) => BackupDirectoryStatus;
}
/** The running build's version, as `/api/version` answers it. */
@@ -111,7 +117,9 @@ export async function runDatabaseBackup(
const env = options.env ?? process.env;
const now = options.now ?? new Date();
const dir = resolveBackupDir(env);
const localWritable = isWritableDirectory(dir);
const directory = (options.directoryStatus ?? backupDirectoryStatus)(dir);
// Kept here only where it survives the container being replaced.
const keepLocal = directory.persistent;
const offhostConfig = (() => {
try {
return loadOffhostConfig();
@@ -128,7 +136,11 @@ export async function runDatabaseBackup(
status: "failed",
fileName: null,
dumpBytes: null,
local: localWritable ? "failed" : "no_directory",
local: keepLocal
? "failed"
: directory.writable
? "not_persistent"
: "no_directory",
offhost: offhostConfig ? "failed" : "not_configured",
removed: 0,
error: null,
@@ -150,11 +162,13 @@ export async function runDatabaseBackup(
report.local = "skipped";
report.offhost = "skipped";
report.error = "DB_BACKUP_ENABLED is false";
} else if (!localWritable && !offhostConfig) {
} else if (!keepLocal && !offhostConfig) {
report.status = "skipped";
report.error = `no writable backup directory (${dir}) and no off-host bucket configured`;
report.error = directory.writable
? `the backup directory (${dir}) is not on persistent storage, so a dump there would be lost with the container, and no off-host bucket is configured`
: `no writable backup directory (${dir}) and no off-host bucket configured`;
} else {
const target = localWritable
const target = keepLocal
? dir
: (scratch = await mkdtemp(join(tmpdir(), "healthlog-dump-")));
const dump = await createDatabaseDump({
@@ -167,7 +181,7 @@ export async function runDatabaseBackup(
buildSha: env.NEXT_PUBLIC_APP_BUILD_SHA?.trim() || null,
deadline: options.deadline,
});
if (localWritable) report.local = "written";
if (keepLocal) report.local = "written";
report.fileName = basename(dump.dumpPath);
report.dumpBytes = dump.manifest.dump.bytes;
@@ -188,7 +202,7 @@ export async function runDatabaseBackup(
}
}
if (localWritable) {
if (keepLocal) {
const removed = await applyRetention(dir, {
dailyKeep: resolveDailyKeep(env),
now: now.getTime(),
@@ -45,11 +45,15 @@ function dump(
lastFailureReason,
...rest
} = over;
// A directory that cannot be written keeps nothing either.
const writable = rest.localDirWritable ?? true;
return {
enabled: true,
directory: "/data/db-backups",
keepNightly: 7,
localDirWritable: true,
localPersistent: writable,
localReason: writable ? "volume" : "not_writable",
offhostConfigured: false,
lastGood: lastGoodAt
? {
@@ -172,6 +176,37 @@ describe("database dump", () => {
).toBe("ok");
});
it("does not count a writable directory inside the container as a place to keep dumps", () => {
const layer = {
localDirWritable: true,
localPersistent: false,
localReason: "container_layer" as const,
};
// No bucket: the dumps written there are lost at the next update.
expect(evalDump(dump(layer))).toMatchObject({
state: "failing",
reason: "not_persistent",
params: { directory: "/data/db-backups" },
});
// A bucket: fine, and the tile says the bucket holds the only copy.
expect(evalDump(dump({ ...layer, offhostConfigured: true }))).toMatchObject(
{
state: "ok",
reason: "fresh_offhost_only",
params: { directory: "/data/db-backups" },
},
);
// The same directory on a volume reads as before.
expect(evalDump(dump({ offhostConfigured: true }))).toMatchObject({
state: "ok",
reason: "fresh",
});
// Switched off with nowhere lasting to write: not news.
expect(
evalDump(dump({ ...layer, enabled: false, lastGoodAt: null })),
).toMatchObject({ state: "inactive", reason: "disabled" });
});
it("gives a newly switched-on dump 36 hours before calling it failing", () => {
const fresh = dump({ lastGoodAt: null });
expect(evalDump(fresh, ago(HOUR))).toMatchObject({
+19 -8
View File
@@ -185,19 +185,24 @@ export function evaluateDatabaseDump(
// directory, no bucket) switched off what it could not do, which is not
// news.
const neverUsed = dump.lastGood === null && dump.lastFailure === null;
const cannotWrite = !dump.localDirWritable && !dump.offhostConfigured;
const cannotWrite = !dump.localPersistent && !dump.offhostConfigured;
return neverUsed && cannotWrite
? signal(id, "inactive", "disabled", null)
: signal(id, "attention", "disabled", lastGoodAt);
}
const lastFailureAt = dump.lastFailure ? new Date(dump.lastFailure.at) : null;
const failureReason = dump.lastFailure?.error ?? null;
// A directory this process cannot write is a valid setup when the dumps go
// to the bucket; with no bucket either, no dump can be kept anywhere.
if (!dump.localDirWritable && !dump.offhostConfigured) {
return signal(id, "failing", "nowhere_to_write", lastGoodAt, {
directory: dump.directory,
});
// A directory that keeps nothing (not writable, or writable inside the
// container and lost at the next update) is a valid setup when the dumps
// go to the bucket; with no bucket either, no dump is kept anywhere.
if (!dump.localPersistent && !dump.offhostConfigured) {
return signal(
id,
"failing",
dump.localDirWritable ? "not_persistent" : "nowhere_to_write",
lastGoodAt,
{ directory: dump.directory },
);
}
const limitMs = DUMP_FAILING_AFTER_HOURS * HOUR_MS;
const failedSinceGood =
@@ -242,7 +247,13 @@ export function evaluateDatabaseDump(
failureReason,
);
}
return signal(id, "ok", "fresh", lastGoodAt);
// Fine, and worth saying: the bucket holds the only copy, and no dump is
// taken before a migration.
return dump.localPersistent
? signal(id, "ok", "fresh", lastGoodAt)
: signal(id, "ok", "fresh_offhost_only", lastGoodAt, {
directory: dump.directory,
});
}
/**
+118
View File
@@ -303,6 +303,94 @@ describe("the nightly database dump", () => {
});
});
it("keeps nothing in a directory inside the container, and stops counting what was written there", async () => {
const { mkdirSync } = await import("node:fs");
const layerDir = join(work, "container-layer");
mkdirSync(layerDir);
// The image's own directory without the volume: writable, and thrown
// away with the container at the next update.
const layer = () =>
({
writable: true,
persistent: false,
reason: "container_layer",
}) as const;
const env = backupEnv({ DB_BACKUP_DIR: layerDir });
const skipped = await runDatabaseBackup(prisma, {
now: new Date("2026-10-24T05:00:00Z"),
env,
directoryStatus: layer,
});
expect(skipped).toMatchObject({
status: "skipped",
local: "not_persistent",
offhost: "not_configured",
});
expect(skipped.error).toMatch(/not on persistent storage/);
expect(readdirSync(layerDir)).toEqual([]);
// Read as keeping nothing, the newest dump that still exists is the one
// that reached the bucket on the 20th, not the local-only one of the
// 23rd; read as a volume, it is the 23rd's.
const asLayer = await getDatabaseBackupHealth(
new Date("2026-10-24T05:30:00Z"),
env,
layer,
);
expect(asLayer).toMatchObject({
localDirWritable: true,
localPersistent: false,
localReason: "container_layer",
lastGood: {
fileName: "healthlog-20261020T013000Z-daily.dump",
local: false,
offhost: true,
},
});
const asVolume = await getDatabaseBackupHealth(
new Date("2026-10-24T05:30:00Z"),
backupEnv(),
);
expect(asVolume.localPersistent).toBe(true);
expect(asVolume.lastGood).toMatchObject({
fileName: "healthlog-20261023T013000Z-daily.dump",
local: true,
});
vi.stubEnv("BACKUP_S3_ENDPOINT", "https://bucket.invalid");
vi.stubEnv("BACKUP_S3_BUCKET", "b");
vi.stubEnv("BACKUP_S3_ACCESS_KEY", "a");
vi.stubEnv("BACKUP_S3_SECRET_KEY", "s");
vi.stubEnv("BACKUP_ENCRYPTION_KEY", OFFHOST_KEY);
const bucket = memoryBucket();
const uploaded = await runDatabaseBackup(prisma, {
now: new Date("2026-10-24T06:00:00Z"),
env: backupEnv({ DB_BACKUP_DIR: layerDir }),
s3: bucket as never,
directoryStatus: layer,
});
vi.unstubAllEnvs();
expect(uploaded).toMatchObject({
status: "succeeded",
local: "not_persistent",
offhost: "uploaded",
removed: 0,
});
expect(readdirSync(layerDir)).toEqual([]);
expect([...bucket.objects.keys()].sort()).toEqual([
"database/healthlog-20261024T060000Z-daily.dump.enc",
"database/healthlog-20261024T060000Z-daily.manifest.json.enc",
]);
const row = await prisma.databaseBackupRun.findFirst({
where: { startedAt: new Date("2026-10-24T06:00:00Z") },
});
expect(row).toMatchObject({
localResult: "not_persistent",
offhostResult: "uploaded",
});
});
describe("restoring it", () => {
const target = `restore_${process.pid}_${Date.now()}`;
let targetUrl = "";
@@ -468,6 +556,36 @@ describe("the dump before migrating", () => {
);
});
it("takes no dump into a directory inside the container, says why, and lets the boot go on", async () => {
const { preMigrationDump } = await import("../../scripts/db-dump.mjs");
const before = readdirSync(preDir).sort();
const errors: string[] = [];
const spyErr = vi
.spyOn(console, "error")
.mockImplementation((l: string) => void errors.push(String(l)));
const spyLog = vi.spyOn(console, "log").mockImplementation(() => {});
try {
const code = await preMigrationDump(
{ ...process.env, DB_BACKUP_DIR: preDir },
{
directoryStatus: () => ({
writable: true,
persistent: false,
reason: "container_layer",
}),
},
);
expect(code).toBe(0);
} finally {
spyErr.mockRestore();
spyLog.mockRestore();
}
expect(errors.join("\n")).toMatch(
/is inside the container, not on a mounted volume, and would not keep a dump past this update, so no dump is taken before the 1 pending migration/,
);
expect(readdirSync(preDir).sort()).toEqual(before);
});
it("stops the boot when the dump fails on a writable directory, and leaves no dump behind", async () => {
// A second of its own, so the name cannot meet the last dump's.
await new Promise((r) => setTimeout(r, 1100));