mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 12:32:43 +00:00
Merge the per-account off-host backup freshness and the honest error-report badge
This commit is contained in:
Marc-André Bombeck committed
commit
e8a2cbe82e
36 files changed
+2227
-27
No files matched your search
@@ -215,6 +215,42 @@ A run where SOME account got a copy still succeeds, with the rest counted in
|
||||
`offhost_backup_failed`. Failing the whole queue over one account's object
|
||||
would re-upload everybody's on every retry.
|
||||
|
||||
### Which account has no copy
|
||||
|
||||
The counts above say how many accounts were uploaded, never which. An account
|
||||
whose object has failed every night for a month reads as `99 uploaded, 1
|
||||
failed` each time, which looks like weather.
|
||||
|
||||
**Admin console → Backups → Off-host copies** answers it per account: when this
|
||||
host last put that account's encrypted copy in the bucket, how big it was, and
|
||||
how that compares with the nightly schedule. **Fresh** is inside one run,
|
||||
**due** means one run produced nothing for that account, **stale** means two
|
||||
did not, **never** means a run walked the account and no object has ever
|
||||
landed for it, and **no record yet** means no run has recorded the account at
|
||||
all. On a host without `BACKUP_S3_*` the card says off-host backup is not
|
||||
configured and stops there.
|
||||
|
||||
The ledger starts empty on the release that adds it, so on a host that has
|
||||
been uploading nightly for months every account reads **no record yet** until
|
||||
the first run after the upgrade. That is the ledger having no history, not the
|
||||
bucket being empty — which is why it is a separate word from **never**, and
|
||||
why the row says so rather than claiming nothing has ever reached the bucket.
|
||||
|
||||
Each verdict carries six hours of slack on top of the schedule's own 24, and
|
||||
that slack is load-bearing on two nights. The cron runs at 02:30 Europe/Berlin,
|
||||
so on the DST fall-back Sunday two consecutive runs are 25 hours apart; and the
|
||||
timestamp is when _that account's_ object landed, so on a cohort walked one
|
||||
account at a time an account reached later tonight than last night drifts by
|
||||
hours. Without the slack a healthy host would paint its whole cohort **due**
|
||||
once a year.
|
||||
|
||||
The verdicts come from a ledger the worker writes when the object lands
|
||||
(`offhost_backup_state`), not from a listing of the bucket. Keep the worker's
|
||||
grant as documented — PutObject, GetObject, AbortMultipartUpload — and the card
|
||||
still tells the truth. A row that goes stale while the run reports success is
|
||||
that account's object being refused, and the reason is in the run's
|
||||
`offhost_backup_failures` meta.
|
||||
|
||||
## The weekly in-database backup (`data-backup`)
|
||||
|
||||
Separate from the off-host job above, and easy to confuse with it. A second
|
||||
|
||||
@@ -143,6 +143,27 @@ itself still works: it walks the journey against an in-process instance
|
||||
once per leg with that leg broken, and fails if any of them comes back
|
||||
green.
|
||||
|
||||
## Error reporting
|
||||
|
||||
HealthLog ships no default error-reporting target and never will: an
|
||||
operator's crash reports must not leave their host because nobody told them
|
||||
not to. Reporting starts only when someone types a DSN into **Admin →
|
||||
Integrations → GlitchTip**, and that card states plainly whether reports are
|
||||
going anywhere and which host they go to.
|
||||
|
||||
For a maintainer-operated instance, set the DSN there after the first deploy —
|
||||
it lives in the database, not in the environment, so it survives an image
|
||||
upgrade and does not need a compose change.
|
||||
|
||||
The badge on that card tracks delivery, not intent. A DSN that is set and
|
||||
parses is not the same fact as reports arriving: a wrong public key, a wrong
|
||||
project id and a host that stopped answering all parse. So the card is green
|
||||
only while a report has actually left this host inside the last fortnight,
|
||||
amber while a target is set and nothing has come back from it, and it names the
|
||||
reason underneath when the last attempt was refused. Press **Test** to prove
|
||||
the target on demand — that send is recorded like any other, so the badge
|
||||
answers immediately.
|
||||
|
||||
## Verify the image signature
|
||||
|
||||
Every release image (the multi-arch manifest **index**, covering both
|
||||
|
||||
@@ -707,6 +707,51 @@ async function installA11yMocks(page: Page) {
|
||||
},
|
||||
],
|
||||
retentionDays: 30,
|
||||
// The off-host card, with one row in each state — the scan has to reach
|
||||
// the status washes, and a card that paints only its header proves
|
||||
// nothing about the rows an operator actually reads.
|
||||
offhost: {
|
||||
configured: true,
|
||||
periodHours: 24,
|
||||
rows: [
|
||||
{
|
||||
userId: "a11y-user",
|
||||
username: "a11y-backup-user",
|
||||
lastAttemptAt: "2026-07-20T03:00:00.000Z",
|
||||
lastSuccessAt: "2026-07-20T03:00:00.000Z",
|
||||
sizeBytes: 4096,
|
||||
ageHours: 3,
|
||||
freshness: "fresh",
|
||||
},
|
||||
{
|
||||
userId: "a11y-user-2",
|
||||
username: "a11y-backup-user-2",
|
||||
lastAttemptAt: "2026-07-14T03:00:00.000Z",
|
||||
lastSuccessAt: "2026-07-14T03:00:00.000Z",
|
||||
sizeBytes: 2048,
|
||||
ageHours: 150,
|
||||
freshness: "stale",
|
||||
},
|
||||
{
|
||||
userId: "a11y-user-3",
|
||||
username: "a11y-backup-user-3",
|
||||
lastAttemptAt: "2026-07-20T03:00:00.000Z",
|
||||
lastSuccessAt: null,
|
||||
sizeBytes: null,
|
||||
ageHours: null,
|
||||
freshness: "never",
|
||||
},
|
||||
{
|
||||
userId: "a11y-user-4",
|
||||
username: "a11y-backup-user-4",
|
||||
lastAttemptAt: null,
|
||||
lastSuccessAt: null,
|
||||
sizeBytes: null,
|
||||
ageHours: null,
|
||||
freshness: "unknown",
|
||||
},
|
||||
],
|
||||
},
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
+24
-1
@@ -6597,6 +6597,13 @@
|
||||
"glitchtipDsnPlaceholder": "https://<public-key>@glitchtip.example.com/<project-id>",
|
||||
"glitchtipEnvironment": "GlitchTip Environment",
|
||||
"glitchtipEnvironmentPlaceholder": "production",
|
||||
"glitchtipTargetOn": "Fehlerberichte gehen an {host}.",
|
||||
"glitchtipTargetOff": "Fehlerberichte bleiben auf diesem Host. Es wird nichts verschickt.",
|
||||
"glitchtipTargetNoTarget": "Fehler-Tracking ist an, aber kein Ziel gesetzt, also wird nichts verschickt.",
|
||||
"glitchtipUnproven": "Unbestätigt",
|
||||
"glitchtipDeliveryUnproven": "Ein Ziel ist gesetzt, aber noch kein Bericht hat diesen Host verlassen. Drücke Test, um zu sehen, ob es einen annimmt.",
|
||||
"glitchtipDeliveryStale": "In den letzten {hours} h hat kein Bericht diesen Host verlassen. Drücke Test, um zu prüfen, ob das Ziel noch annimmt.",
|
||||
"glitchtipDeliveryFailed": "Der letzte Bericht kam nicht durch ({reason}).",
|
||||
"webPushVapidDescription": "Konfiguriere VAPID-Schlüssel und Subject direkt hier, damit Browser-Push ohne ENV-Variablen funktioniert.",
|
||||
"webPushVapidPublicKey": "VAPID Public Key",
|
||||
"webPushVapidPublicKeyPlaceholder": "BElx... (Base64URL)",
|
||||
@@ -7012,7 +7019,23 @@
|
||||
"restoreMissingDocuments": "Dokumente",
|
||||
"restoreMissingWorkouts": "Workouts",
|
||||
"restoreMissingMentalHealth": "Psychische Fragebögen",
|
||||
"restoreMissingConsent": "Einwilligungsnachweise"
|
||||
"restoreMissingConsent": "Einwilligungsnachweise",
|
||||
"offhost": {
|
||||
"title": "Kopien außer Haus",
|
||||
"description": "Wann dieser Host zuletzt die verschlüsselte Kopie jedes Kontos in deinen Bucket gelegt hat.",
|
||||
"loadError": "Der Status außer Haus konnte nicht geladen werden.",
|
||||
"notConfiguredTitle": "Sicherung außer Haus ist nicht eingerichtet",
|
||||
"notConfiguredDescription": "Es sind weder Bucket noch Schlüssel gesetzt, also verlässt nichts diesen Host. Setze BACKUP_S3_ENDPOINT, BACKUP_S3_BUCKET, BACKUP_S3_ACCESS_KEY, BACKUP_S3_SECRET_KEY und BACKUP_ENCRYPTION_KEY, um den nächtlichen Upload einzuschalten.",
|
||||
"stateFresh": "Frisch",
|
||||
"stateDue": "Fällig",
|
||||
"stateStale": "Veraltet",
|
||||
"stateNever": "Nie",
|
||||
"stateUnknown": "Noch kein Eintrag",
|
||||
"neverDetail": "Für dieses Konto ist nichts angekommen. Zuletzt geprüft {when}.",
|
||||
"unknownDetail": "Noch kein nächtlicher Lauf hat dieses Konto erfasst.",
|
||||
"age": "vor {hours} h",
|
||||
"scheduleNote": "Der Upload läuft alle {hours} h. Frisch heißt innerhalb eines Laufs, fällig heißt ein Lauf blieb ohne Ergebnis, veraltet heißt zwei."
|
||||
}
|
||||
},
|
||||
"danger-zone": {
|
||||
"title": "Gefahrenzone",
|
||||
|
||||
+24
-1
@@ -6597,6 +6597,13 @@
|
||||
"glitchtipDsnPlaceholder": "https://<public-key>@glitchtip.example.com/<project-id>",
|
||||
"glitchtipEnvironment": "GlitchTip environment",
|
||||
"glitchtipEnvironmentPlaceholder": "production",
|
||||
"glitchtipTargetOn": "Error reports go to {host}.",
|
||||
"glitchtipTargetOff": "Error reports stay on this host. Nothing is sent anywhere.",
|
||||
"glitchtipTargetNoTarget": "Error tracking is on, but no target is set, so nothing is sent.",
|
||||
"glitchtipUnproven": "Unproven",
|
||||
"glitchtipDeliveryUnproven": "A target is set, but no report has left this host yet. Press Test to find out whether it accepts one.",
|
||||
"glitchtipDeliveryStale": "No report has left this host in the last {hours} h. Press Test to check the target still accepts one.",
|
||||
"glitchtipDeliveryFailed": "The last report did not get through ({reason}).",
|
||||
"webPushVapidDescription": "Configure VAPID keys and subject here so browser push works without environment variables.",
|
||||
"webPushVapidPublicKey": "VAPID public key",
|
||||
"webPushVapidPublicKeyPlaceholder": "BElx... (Base64URL)",
|
||||
@@ -7012,7 +7019,23 @@
|
||||
"restoreMissingDocuments": "Documents",
|
||||
"restoreMissingWorkouts": "Workouts",
|
||||
"restoreMissingMentalHealth": "Mental-health screeners",
|
||||
"restoreMissingConsent": "Consent receipts"
|
||||
"restoreMissingConsent": "Consent receipts",
|
||||
"offhost": {
|
||||
"title": "Off-host copies",
|
||||
"description": "When this host last put each account's encrypted copy in your bucket.",
|
||||
"loadError": "Could not load the off-host status.",
|
||||
"notConfiguredTitle": "Off-host backup is not configured",
|
||||
"notConfiguredDescription": "No bucket and no key are set, so nothing leaves this host. Set BACKUP_S3_ENDPOINT, BACKUP_S3_BUCKET, BACKUP_S3_ACCESS_KEY, BACKUP_S3_SECRET_KEY and BACKUP_ENCRYPTION_KEY to turn the nightly upload on.",
|
||||
"stateFresh": "Fresh",
|
||||
"stateDue": "Due",
|
||||
"stateStale": "Stale",
|
||||
"stateNever": "Never",
|
||||
"stateUnknown": "No record yet",
|
||||
"neverDetail": "Nothing has landed for this account. Last checked {when}.",
|
||||
"unknownDetail": "No nightly run has recorded this account yet.",
|
||||
"age": "{hours} h ago",
|
||||
"scheduleNote": "The upload runs every {hours} h. Fresh means inside one run, due means one run produced nothing, stale means two did not."
|
||||
}
|
||||
},
|
||||
"danger-zone": {
|
||||
"title": "Danger Zone",
|
||||
|
||||
+24
-1
@@ -6597,6 +6597,13 @@
|
||||
"glitchtipDsnPlaceholder": "https://<public-key>@glitchtip.example.com/<project-id>",
|
||||
"glitchtipEnvironment": "Entorno de GlitchTip",
|
||||
"glitchtipEnvironmentPlaceholder": "production",
|
||||
"glitchtipTargetOn": "Los informes de errores van a {host}.",
|
||||
"glitchtipTargetOff": "Los informes de errores se quedan en este host. No se envía nada.",
|
||||
"glitchtipTargetNoTarget": "El seguimiento de errores está activado, pero no hay destino, así que no se envía nada.",
|
||||
"glitchtipUnproven": "Sin confirmar",
|
||||
"glitchtipDeliveryUnproven": "Hay un destino definido, pero ningún informe ha salido aún de este host. Pulsa Probar para ver si lo acepta.",
|
||||
"glitchtipDeliveryStale": "Ningún informe ha salido de este host en las últimas {hours} h. Pulsa Probar para comprobar que el destino sigue aceptándolos.",
|
||||
"glitchtipDeliveryFailed": "El último informe no llegó ({reason}).",
|
||||
"webPushVapidDescription": "Configura aquí las claves VAPID y el subject para que el push del navegador funcione sin variables de entorno.",
|
||||
"webPushVapidPublicKey": "Clave pública VAPID",
|
||||
"webPushVapidPublicKeyPlaceholder": "BElx... (Base64URL)",
|
||||
@@ -7012,7 +7019,23 @@
|
||||
"restoreMissingDocuments": "Documentos",
|
||||
"restoreMissingWorkouts": "Entrenamientos",
|
||||
"restoreMissingMentalHealth": "Cuestionarios de salud mental",
|
||||
"restoreMissingConsent": "Comprobantes de consentimiento"
|
||||
"restoreMissingConsent": "Comprobantes de consentimiento",
|
||||
"offhost": {
|
||||
"title": "Copias fuera del host",
|
||||
"description": "Cuándo puso este host por última vez la copia cifrada de cada cuenta en tu bucket.",
|
||||
"loadError": "No se pudo cargar el estado fuera del host.",
|
||||
"notConfiguredTitle": "La copia fuera del host no está configurada",
|
||||
"notConfiguredDescription": "No hay bucket ni clave definidos, así que nada sale de este host. Define BACKUP_S3_ENDPOINT, BACKUP_S3_BUCKET, BACKUP_S3_ACCESS_KEY, BACKUP_S3_SECRET_KEY y BACKUP_ENCRYPTION_KEY para activar la subida nocturna.",
|
||||
"stateFresh": "Reciente",
|
||||
"stateDue": "Pendiente",
|
||||
"stateStale": "Obsoleta",
|
||||
"stateNever": "Nunca",
|
||||
"stateUnknown": "Sin registro aún",
|
||||
"neverDetail": "No ha llegado nada para esta cuenta. Última comprobación {when}.",
|
||||
"unknownDetail": "Ninguna ejecución nocturna ha registrado esta cuenta todavía.",
|
||||
"age": "hace {hours} h",
|
||||
"scheduleNote": "La subida se ejecuta cada {hours} h. Reciente significa dentro de una ejecución, pendiente que una no produjo nada, obsoleta que fueron dos."
|
||||
}
|
||||
},
|
||||
"danger-zone": {
|
||||
"title": "Zona de peligro",
|
||||
|
||||
+24
-1
@@ -6597,6 +6597,13 @@
|
||||
"glitchtipDsnPlaceholder": "https://<public-key>@glitchtip.example.com/<project-id>",
|
||||
"glitchtipEnvironment": "Environnement GlitchTip",
|
||||
"glitchtipEnvironmentPlaceholder": "production",
|
||||
"glitchtipTargetOn": "Les rapports d'erreur vont vers {host}.",
|
||||
"glitchtipTargetOff": "Les rapports d'erreur restent sur cet hôte. Rien n'est envoyé.",
|
||||
"glitchtipTargetNoTarget": "Le suivi des erreurs est activé, mais aucune cible n'est définie, donc rien n'est envoyé.",
|
||||
"glitchtipUnproven": "Non confirmé",
|
||||
"glitchtipDeliveryUnproven": "Une cible est définie, mais aucun rapport n'a encore quitté cet hôte. Appuyez sur Tester pour voir si elle en accepte un.",
|
||||
"glitchtipDeliveryStale": "Aucun rapport n'a quitté cet hôte depuis {hours} h. Appuyez sur Tester pour vérifier que la cible en accepte toujours.",
|
||||
"glitchtipDeliveryFailed": "Le dernier rapport n'est pas passé ({reason}).",
|
||||
"webPushVapidDescription": "Configurez ici les clés VAPID et le subject pour que le push du navigateur fonctionne sans variables d'environnement.",
|
||||
"webPushVapidPublicKey": "Clé publique VAPID",
|
||||
"webPushVapidPublicKeyPlaceholder": "BElx... (Base64URL)",
|
||||
@@ -7012,7 +7019,23 @@
|
||||
"restoreMissingDocuments": "Documents",
|
||||
"restoreMissingWorkouts": "Séances",
|
||||
"restoreMissingMentalHealth": "Questionnaires de santé mentale",
|
||||
"restoreMissingConsent": "Preuves de consentement"
|
||||
"restoreMissingConsent": "Preuves de consentement",
|
||||
"offhost": {
|
||||
"title": "Copies hors hôte",
|
||||
"description": "Quand cet hôte a déposé pour la dernière fois la copie chiffrée de chaque compte dans votre bucket.",
|
||||
"loadError": "Impossible de charger l'état hors hôte.",
|
||||
"notConfiguredTitle": "La sauvegarde hors hôte n'est pas configurée",
|
||||
"notConfiguredDescription": "Aucun bucket ni clé ne sont définis, donc rien ne quitte cet hôte. Définissez BACKUP_S3_ENDPOINT, BACKUP_S3_BUCKET, BACKUP_S3_ACCESS_KEY, BACKUP_S3_SECRET_KEY et BACKUP_ENCRYPTION_KEY pour activer l'envoi nocturne.",
|
||||
"stateFresh": "Récente",
|
||||
"stateDue": "Attendue",
|
||||
"stateStale": "Périmée",
|
||||
"stateNever": "Jamais",
|
||||
"stateUnknown": "Pas encore d'enregistrement",
|
||||
"neverDetail": "Rien n'est arrivé pour ce compte. Dernière vérification {when}.",
|
||||
"unknownDetail": "Aucun cycle nocturne n'a encore enregistré ce compte.",
|
||||
"age": "il y a {hours} h",
|
||||
"scheduleNote": "L'envoi tourne toutes les {hours} h. Récente signifie dans un cycle, attendue qu'un cycle n'a rien produit, périmée que deux n'ont rien produit."
|
||||
}
|
||||
},
|
||||
"danger-zone": {
|
||||
"title": "Zone de danger",
|
||||
|
||||
+24
-1
@@ -6597,6 +6597,13 @@
|
||||
"glitchtipDsnPlaceholder": "https://<public-key>@glitchtip.example.com/<project-id>",
|
||||
"glitchtipEnvironment": "Ambiente GlitchTip",
|
||||
"glitchtipEnvironmentPlaceholder": "production",
|
||||
"glitchtipTargetOn": "Le segnalazioni di errore vanno a {host}.",
|
||||
"glitchtipTargetOff": "Le segnalazioni di errore restano su questo host. Non viene inviato nulla.",
|
||||
"glitchtipTargetNoTarget": "Il tracciamento degli errori è attivo, ma non c'è una destinazione, quindi non viene inviato nulla.",
|
||||
"glitchtipUnproven": "Non confermato",
|
||||
"glitchtipDeliveryUnproven": "Una destinazione è impostata, ma nessuna segnalazione ha ancora lasciato questo host. Premi Prova per vedere se ne accetta una.",
|
||||
"glitchtipDeliveryStale": "Nessuna segnalazione ha lasciato questo host nelle ultime {hours} h. Premi Prova per verificare che la destinazione le accetti ancora.",
|
||||
"glitchtipDeliveryFailed": "L'ultima segnalazione non è arrivata ({reason}).",
|
||||
"webPushVapidDescription": "Configura qui le chiavi VAPID e il subject affinché il push del browser funzioni senza variabili d'ambiente.",
|
||||
"webPushVapidPublicKey": "Chiave pubblica VAPID",
|
||||
"webPushVapidPublicKeyPlaceholder": "BElx... (Base64URL)",
|
||||
@@ -7012,7 +7019,23 @@
|
||||
"restoreMissingDocuments": "Documenti",
|
||||
"restoreMissingWorkouts": "Allenamenti",
|
||||
"restoreMissingMentalHealth": "Questionari sulla salute mentale",
|
||||
"restoreMissingConsent": "Ricevute di consenso"
|
||||
"restoreMissingConsent": "Ricevute di consenso",
|
||||
"offhost": {
|
||||
"title": "Copie fuori host",
|
||||
"description": "Quando questo host ha messo per l'ultima volta la copia cifrata di ogni account nel tuo bucket.",
|
||||
"loadError": "Impossibile caricare lo stato fuori host.",
|
||||
"notConfiguredTitle": "Il backup fuori host non è configurato",
|
||||
"notConfiguredDescription": "Non sono impostati né bucket né chiave, quindi nulla lascia questo host. Imposta BACKUP_S3_ENDPOINT, BACKUP_S3_BUCKET, BACKUP_S3_ACCESS_KEY, BACKUP_S3_SECRET_KEY e BACKUP_ENCRYPTION_KEY per attivare il caricamento notturno.",
|
||||
"stateFresh": "Recente",
|
||||
"stateDue": "Attesa",
|
||||
"stateStale": "Obsoleta",
|
||||
"stateNever": "Mai",
|
||||
"stateUnknown": "Nessuna registrazione",
|
||||
"neverDetail": "Per questo account non è arrivato nulla. Ultimo controllo {when}.",
|
||||
"unknownDetail": "Nessun giro notturno ha ancora registrato questo account.",
|
||||
"age": "{hours} h fa",
|
||||
"scheduleNote": "Il caricamento gira ogni {hours} h. Recente significa entro un giro, attesa che un giro non ha prodotto nulla, obsoleta che non lo hanno fatto due."
|
||||
}
|
||||
},
|
||||
"danger-zone": {
|
||||
"title": "Zona pericolosa",
|
||||
|
||||
+24
-1
@@ -6577,6 +6577,13 @@
|
||||
"glitchtipDsnPlaceholder": "https://<public-key>@glitchtip.example.com/<project-id>",
|
||||
"glitchtipEnvironment": "GlitchTip 환경",
|
||||
"glitchtipEnvironmentPlaceholder": "production",
|
||||
"glitchtipTargetOn": "오류 보고서는 {host}(으)로 전송됩니다.",
|
||||
"glitchtipTargetOff": "오류 보고서는 이 호스트에 남습니다. 아무것도 전송되지 않습니다.",
|
||||
"glitchtipTargetNoTarget": "오류 추적은 켜져 있지만 대상이 없어 아무것도 전송되지 않습니다.",
|
||||
"glitchtipUnproven": "미확인",
|
||||
"glitchtipDeliveryUnproven": "대상이 설정되어 있지만 아직 이 호스트를 떠난 보고서가 없습니다. 테스트를 눌러 수락하는지 확인하세요.",
|
||||
"glitchtipDeliveryStale": "지난 {hours}시간 동안 이 호스트를 떠난 보고서가 없습니다. 테스트를 눌러 대상이 여전히 수락하는지 확인하세요.",
|
||||
"glitchtipDeliveryFailed": "마지막 보고서가 전달되지 않았습니다 ({reason}).",
|
||||
"webPushVapidDescription": "환경 변수 없이 브라우저 푸시가 동작하도록 VAPID 키와 subject를 여기서 설정하세요.",
|
||||
"webPushVapidPublicKey": "VAPID 공개 키",
|
||||
"webPushVapidPublicKeyPlaceholder": "BElx... (Base64URL)",
|
||||
@@ -6992,7 +6999,23 @@
|
||||
"restoreMissingDocuments": "문서",
|
||||
"restoreMissingWorkouts": "운동",
|
||||
"restoreMissingMentalHealth": "정신 건강 선별 검사",
|
||||
"restoreMissingConsent": "동의 기록"
|
||||
"restoreMissingConsent": "동의 기록",
|
||||
"offhost": {
|
||||
"title": "외부 보관 사본",
|
||||
"description": "이 호스트가 각 계정의 암호화 사본을 버킷에 마지막으로 올린 시점입니다.",
|
||||
"loadError": "외부 보관 상태를 불러오지 못했습니다.",
|
||||
"notConfiguredTitle": "외부 보관 백업이 설정되지 않았습니다",
|
||||
"notConfiguredDescription": "버킷과 키가 모두 설정되지 않아 이 호스트를 벗어나는 것이 없습니다. 야간 업로드를 켜려면 BACKUP_S3_ENDPOINT, BACKUP_S3_BUCKET, BACKUP_S3_ACCESS_KEY, BACKUP_S3_SECRET_KEY, BACKUP_ENCRYPTION_KEY를 설정하세요.",
|
||||
"stateFresh": "최신",
|
||||
"stateDue": "지연",
|
||||
"stateStale": "오래됨",
|
||||
"stateNever": "없음",
|
||||
"stateUnknown": "기록 없음",
|
||||
"neverDetail": "이 계정으로 도착한 것이 없습니다. 마지막 확인 {when}.",
|
||||
"unknownDetail": "아직 어떤 야간 실행도 이 계정을 기록하지 않았습니다.",
|
||||
"age": "{hours}시간 전",
|
||||
"scheduleNote": "업로드는 {hours}시간마다 실행됩니다. 최신은 한 번의 실행 이내, 지연은 한 번이 아무것도 만들지 못한 상태, 오래됨은 두 번이 그런 상태입니다."
|
||||
}
|
||||
},
|
||||
"danger-zone": {
|
||||
"title": "위험 구역",
|
||||
|
||||
+24
-1
@@ -6597,6 +6597,13 @@
|
||||
"glitchtipDsnPlaceholder": "https://<public-key>@glitchtip.example.com/<project-id>",
|
||||
"glitchtipEnvironment": "Środowisko GlitchTip",
|
||||
"glitchtipEnvironmentPlaceholder": "production",
|
||||
"glitchtipTargetOn": "Zgłoszenia błędów trafiają do {host}.",
|
||||
"glitchtipTargetOff": "Zgłoszenia błędów zostają na tym hoście. Nic nie jest wysyłane.",
|
||||
"glitchtipTargetNoTarget": "Śledzenie błędów jest włączone, ale nie ustawiono celu, więc nic nie jest wysyłane.",
|
||||
"glitchtipUnproven": "Niepotwierdzone",
|
||||
"glitchtipDeliveryUnproven": "Cel jest ustawiony, ale żaden raport nie opuścił jeszcze tego hosta. Naciśnij Testuj, aby sprawdzić, czy zostanie przyjęty.",
|
||||
"glitchtipDeliveryStale": "Przez ostatnie {hours} h żaden raport nie opuścił tego hosta. Naciśnij Testuj, aby sprawdzić, czy cel nadal je przyjmuje.",
|
||||
"glitchtipDeliveryFailed": "Ostatni raport nie dotarł ({reason}).",
|
||||
"webPushVapidDescription": "Skonfiguruj tutaj klucze VAPID i subject, aby push przeglądarki działał bez zmiennych środowiskowych.",
|
||||
"webPushVapidPublicKey": "Klucz publiczny VAPID",
|
||||
"webPushVapidPublicKeyPlaceholder": "BElx... (Base64URL)",
|
||||
@@ -7012,7 +7019,23 @@
|
||||
"restoreMissingDocuments": "Dokumenty",
|
||||
"restoreMissingWorkouts": "Treningi",
|
||||
"restoreMissingMentalHealth": "Kwestionariusze zdrowia psychicznego",
|
||||
"restoreMissingConsent": "Potwierdzenia zgody"
|
||||
"restoreMissingConsent": "Potwierdzenia zgody",
|
||||
"offhost": {
|
||||
"title": "Kopie poza hostem",
|
||||
"description": "Kiedy ten host ostatnio umieścił zaszyfrowaną kopię każdego konta w Twoim buckecie.",
|
||||
"loadError": "Nie udało się wczytać stanu poza hostem.",
|
||||
"notConfiguredTitle": "Kopia poza hostem nie jest skonfigurowana",
|
||||
"notConfiguredDescription": "Nie ustawiono ani bucketu, ani klucza, więc nic nie opuszcza tego hosta. Ustaw BACKUP_S3_ENDPOINT, BACKUP_S3_BUCKET, BACKUP_S3_ACCESS_KEY, BACKUP_S3_SECRET_KEY i BACKUP_ENCRYPTION_KEY, aby włączyć nocne wysyłanie.",
|
||||
"stateFresh": "Świeża",
|
||||
"stateDue": "Zaległa",
|
||||
"stateStale": "Przeterminowana",
|
||||
"stateNever": "Nigdy",
|
||||
"stateUnknown": "Brak wpisu",
|
||||
"neverDetail": "Dla tego konta nic nie dotarło. Ostatnie sprawdzenie {when}.",
|
||||
"unknownDetail": "Żaden nocny przebieg nie zarejestrował jeszcze tego konta.",
|
||||
"age": "{hours} h temu",
|
||||
"scheduleNote": "Wysyłka działa co {hours} h. Świeża znaczy w obrębie jednego przebiegu, zaległa że jeden nic nie dał, przeterminowana że dwa."
|
||||
}
|
||||
},
|
||||
"danger-zone": {
|
||||
"title": "Strefa zagrożenia",
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
-- One row per account, written by the nightly off-host backup job when an
|
||||
-- object lands in the bucket. The admin console reads it to say, per account,
|
||||
-- whether the newest off-host copy is as fresh as the schedule promises,
|
||||
-- without the page ever talking to the bucket.
|
||||
CREATE TABLE "offhost_backup_state" (
|
||||
"user_id" TEXT NOT NULL,
|
||||
-- Written on every account the run walks, success or not. A row's mere
|
||||
-- existence is what lets the console tell "no run has reached this
|
||||
-- account yet" from "a run reached it and nothing landed".
|
||||
"last_attempt_at" TIMESTAMP(3) NOT NULL,
|
||||
-- NULL until an object lands for this account.
|
||||
"last_success_at" TIMESTAMP(3),
|
||||
-- BIGINT, not INTEGER: a single object may occupy 8 MiB x 10 000 multipart
|
||||
-- parts, so the uploader permits 80 GB and a 32-bit column would refuse
|
||||
-- every account past 2.1 GB.
|
||||
"size_bytes" BIGINT,
|
||||
|
||||
CONSTRAINT "offhost_backup_state_pkey" PRIMARY KEY ("user_id")
|
||||
);
|
||||
|
||||
ALTER TABLE "offhost_backup_state"
|
||||
ADD CONSTRAINT "offhost_backup_state_user_id_fkey"
|
||||
FOREIGN KEY ("user_id") REFERENCES "users"("id")
|
||||
ON DELETE CASCADE ON UPDATE CASCADE;
|
||||
@@ -0,0 +1,8 @@
|
||||
-- Delivery outcome of the last error report this host tried to send.
|
||||
-- "A DSN is set and it parses" and "reports are arriving" are different
|
||||
-- facts, and only the second one is worth a green badge: a wrong public key,
|
||||
-- a wrong project id and an unreachable host all parse.
|
||||
ALTER TABLE "app_settings"
|
||||
ADD COLUMN "glitchtip_last_ok_at" TIMESTAMP(3),
|
||||
ADD COLUMN "glitchtip_last_failure_at" TIMESTAMP(3),
|
||||
ADD COLUMN "glitchtip_last_failure_reason" TEXT;
|
||||
@@ -770,6 +770,7 @@ model User {
|
||||
notificationChannels NotificationChannel[]
|
||||
pushSubscriptions PushSubscription[]
|
||||
dataBackups DataBackup[]
|
||||
offhostBackupState OffhostBackupState?
|
||||
achievements UserAchievement[]
|
||||
recommendationFeedback RecommendationFeedback[]
|
||||
idempotencyKeys IdempotencyKey[]
|
||||
@@ -4700,6 +4701,21 @@ model AppSettings {
|
||||
glitchtipDsn String? @map("glitchtip_dsn")
|
||||
glitchtipEnvironment String? @map("glitchtip_environment")
|
||||
|
||||
// Delivery outcome of the last error report this host tried to send.
|
||||
// Without it the admin card can only say a DSN is set and parses, which is
|
||||
// not the same fact as reports arriving: a wrong public key, a wrong project
|
||||
// id or an unreachable host all parse perfectly and deliver nothing.
|
||||
/// Instant the last report left successfully. Consumer: the GlitchTip card's
|
||||
/// badge, which is only green while this is inside the recent window.
|
||||
glitchtipLastOkAt DateTime? @map("glitchtip_last_ok_at")
|
||||
/// Instant the last attempt failed. Consumer: the GlitchTip card's status
|
||||
/// line, which names the failure rather than showing a reassuring badge.
|
||||
glitchtipLastFailureAt DateTime? @map("glitchtip_last_failure_at")
|
||||
/// Short classification of that failure (`invalid_dsn`, `http_403`,
|
||||
/// `network`) — never a response body, which is a remote host's text.
|
||||
/// Consumer: the same status line.
|
||||
glitchtipLastFailureReason String? @map("glitchtip_last_failure_reason")
|
||||
|
||||
// Medication reminder thresholds (minutes after windowEnd)
|
||||
reminderLateMinutes Int @default(120) @map("reminder_late_minutes")
|
||||
reminderMissedMinutes Int @default(240) @map("reminder_missed_minutes")
|
||||
@@ -4937,6 +4953,49 @@ model DataBackup {
|
||||
@@map("data_backups")
|
||||
}
|
||||
|
||||
// ─── Off-host backup ledger ──────────────────────────────────
|
||||
//
|
||||
// One row per account, written by the nightly off-host job the moment an
|
||||
// object lands in the bucket. It exists because "the job ran" and "this
|
||||
// account has a recent copy off-host" are different facts: the run reports
|
||||
// counts, and a cohort of a hundred accounts where one has silently failed
|
||||
// every night for a month reads exactly like a healthy one.
|
||||
//
|
||||
// The admin console's off-host card is the reader, and it is the only reader
|
||||
// there will be: the page computes staleness from this timestamp rather than
|
||||
// listing the bucket, so an operator who has revoked the worker's read grant
|
||||
// still sees the truth.
|
||||
model OffhostBackupState {
|
||||
userId String @id @map("user_id")
|
||||
|
||||
/// When the last nightly run walked this account, whether or not an object
|
||||
/// came of it. It is what separates "no run has reached this account yet"
|
||||
/// from "a run reached it and put nothing there" — two facts that look
|
||||
/// identical in a ledger keyed only on success, and the first of which is
|
||||
/// what every account reads on the day this table ships. Consumer:
|
||||
/// `classifyOffhostBackup()` behind `GET /api/admin/backups`, and the
|
||||
/// card's "last checked" line.
|
||||
lastAttemptAt DateTime @map("last_attempt_at")
|
||||
|
||||
/// When the newest object for this account finished uploading. NULL means
|
||||
/// a run has looked at this account and nothing has ever landed for it.
|
||||
/// Consumer: `classifyOffhostBackup()` behind `GET /api/admin/backups`.
|
||||
lastSuccessAt DateTime? @map("last_success_at")
|
||||
|
||||
/// Size of that object in bytes — the same number the run reports as its
|
||||
/// largest, per account. `BigInt` because the uploader permits an object of
|
||||
/// up to 8 MiB × 10 000 parts = 80 GB and a 32-bit column stops at 2.1 GB:
|
||||
/// the biggest account on the host, the one an operator cares most about,
|
||||
/// is exactly the one whose row would refuse to be written.
|
||||
/// NULL alongside a NULL `lastSuccessAt`.
|
||||
/// Consumer: the off-host card's size column.
|
||||
sizeBytes BigInt? @map("size_bytes")
|
||||
|
||||
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
|
||||
|
||||
@@map("offhost_backup_state")
|
||||
}
|
||||
|
||||
// ─── Apple Health export.zip import (v1.4.34) ───────────────
|
||||
//
|
||||
// Mirrors a `pg-boss` `apple-health-import` job's lifecycle so the
|
||||
|
||||
@@ -109,6 +109,10 @@ const PINNED_AFFORDANCES: Record<
|
||||
"src/components/admin/module-availability-section.tsx": {
|
||||
"toast.success": 1,
|
||||
},
|
||||
// Reads a ledger, writes nothing: the wash marks how fresh one account's
|
||||
// off-host copy is, and the state it paints green came out of the database
|
||||
// rather than out of a request this card just made.
|
||||
"src/components/admin/offhost-backups-section.tsx": { "text-success": 1 },
|
||||
"src/components/admin/recent-audit-preview.tsx": {
|
||||
CheckCircle2: 2,
|
||||
"text-success": 1,
|
||||
|
||||
@@ -25,6 +25,7 @@ import { CoachFeedbackSection } from "@/components/admin/coach-feedback-section"
|
||||
import { ApiTokenOverviewSection } from "@/components/admin/api-token-overview-section";
|
||||
import { AppLogPreviewSection } from "@/components/admin/app-log-preview-section";
|
||||
import { BackupsSection } from "@/components/admin/backups-section";
|
||||
import { OffhostBackupsSection } from "@/components/admin/offhost-backups-section";
|
||||
import { DangerZoneSection } from "@/components/admin/danger-zone-section";
|
||||
import { EncryptionSection } from "@/components/admin/encryption-section";
|
||||
import { GeneralSettingsSection } from "@/components/admin/general-settings-section";
|
||||
@@ -144,9 +145,12 @@ export function AdminSectionRenderer({
|
||||
</SectionFrame>
|
||||
);
|
||||
case "backups":
|
||||
// Two cards, two independent failure modes: the copies held in this
|
||||
// database, then the copies that reached the operator's bucket.
|
||||
return (
|
||||
<SectionFrame>
|
||||
<BackupsSection />
|
||||
<OffhostBackupsSection />
|
||||
</SectionFrame>
|
||||
);
|
||||
case "encryption":
|
||||
|
||||
@@ -0,0 +1,199 @@
|
||||
/**
|
||||
* `GET /api/admin/backups` — the off-host leg.
|
||||
*
|
||||
* The rows the weekly pass writes into this database were already covered by
|
||||
* the page that lists them. What was not covered is the question this response
|
||||
* now answers: per account, when did an encrypted copy last reach the
|
||||
* operator's bucket, and is that inside what the nightly schedule promises.
|
||||
*
|
||||
* The bucket is never touched here, and that is the assertion that matters —
|
||||
* the verdicts come from the ledger the worker writes, so an operator whose
|
||||
* worker holds no listing grant on the bucket still gets a true answer.
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
vi.mock("@/lib/api-handler", () => ({
|
||||
apiHandler: <T extends (...args: unknown[]) => unknown>(fn: T) => fn,
|
||||
requireAdmin: vi.fn(async () => ({ user: { id: "admin-1" } })),
|
||||
}));
|
||||
|
||||
vi.mock("@/lib/logging/context", () => ({ annotate: vi.fn() }));
|
||||
|
||||
vi.mock("@/lib/jobs/job-failures", () => ({
|
||||
readLastQueueRun: vi.fn(async () => null),
|
||||
}));
|
||||
|
||||
const queryRawMock = vi.fn();
|
||||
const findManyMock = vi.fn();
|
||||
vi.mock("@/lib/db", () => ({
|
||||
prisma: {
|
||||
$queryRaw: (...a: unknown[]) => queryRawMock(...a),
|
||||
user: { findMany: (...a: unknown[]) => findManyMock(...a) },
|
||||
},
|
||||
}));
|
||||
|
||||
const configuredMock = vi.fn(() => true);
|
||||
vi.mock("@/lib/jobs/offhost-backup", () => ({
|
||||
offhostBackupConfigured: () => configuredMock(),
|
||||
}));
|
||||
|
||||
import { GET } from "../route";
|
||||
import type { BackupsList } from "@/types/backups";
|
||||
|
||||
const HOUR = 3_600_000;
|
||||
|
||||
async function read(): Promise<BackupsList> {
|
||||
const res = await (GET as unknown as () => Promise<Response>)();
|
||||
expect(res.status).toBe(200);
|
||||
return (await res.json()).data as BackupsList;
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
configuredMock.mockReturnValue(true);
|
||||
queryRawMock.mockResolvedValue([]);
|
||||
findManyMock.mockResolvedValue([]);
|
||||
});
|
||||
|
||||
describe("GET /api/admin/backups — off-host freshness", () => {
|
||||
it("rates every account against the nightly schedule without listing the bucket", async () => {
|
||||
const now = Date.now();
|
||||
findManyMock.mockResolvedValue([
|
||||
{
|
||||
id: "u1",
|
||||
username: "account-one",
|
||||
offhostBackupState: {
|
||||
lastAttemptAt: new Date(now - 3 * HOUR),
|
||||
lastSuccessAt: new Date(now - 3 * HOUR),
|
||||
// The column is BigInt — an object may be 80 GB — and the wire type
|
||||
// is a plain number.
|
||||
sizeBytes: BigInt(4096),
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "u2",
|
||||
username: "account-two",
|
||||
offhostBackupState: {
|
||||
lastAttemptAt: new Date(now - 31 * HOUR),
|
||||
lastSuccessAt: new Date(now - 31 * HOUR),
|
||||
sizeBytes: BigInt(8192),
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "u3",
|
||||
username: "account-three",
|
||||
offhostBackupState: {
|
||||
lastAttemptAt: new Date(now - 100 * HOUR),
|
||||
lastSuccessAt: new Date(now - 100 * HOUR),
|
||||
sizeBytes: BigInt(128),
|
||||
},
|
||||
},
|
||||
{
|
||||
id: "u4",
|
||||
username: "account-four",
|
||||
// Walked by a run, nothing landed.
|
||||
offhostBackupState: {
|
||||
lastAttemptAt: new Date(now - 2 * HOUR),
|
||||
lastSuccessAt: null,
|
||||
sizeBytes: null,
|
||||
},
|
||||
},
|
||||
// No run has recorded this one at all — the state of every account on
|
||||
// the morning the ledger ships.
|
||||
{ id: "u5", username: "account-five", offhostBackupState: null },
|
||||
]);
|
||||
|
||||
const body = await read();
|
||||
|
||||
expect(body.offhost.configured).toBe(true);
|
||||
expect(body.offhost.periodHours).toBe(24);
|
||||
// Worst first, alphabetical within a verdict. The database hands them
|
||||
// back by username; the one account that needs attention must not be
|
||||
// wherever the alphabet leaves it on a hundred-account cohort.
|
||||
expect(
|
||||
body.offhost.rows.map((row) => [row.username, row.freshness]),
|
||||
).toEqual([
|
||||
["account-three", "stale"],
|
||||
["account-two", "due"],
|
||||
["account-four", "never"],
|
||||
["account-five", "unknown"],
|
||||
["account-one", "fresh"],
|
||||
]);
|
||||
const byName = new Map(body.offhost.rows.map((row) => [row.username, row]));
|
||||
expect(byName.get("account-one")?.sizeBytes).toBe(4096);
|
||||
expect(typeof byName.get("account-one")?.sizeBytes).toBe("number");
|
||||
expect(byName.get("account-one")?.ageHours).toBe(3);
|
||||
expect(byName.get("account-four")?.lastSuccessAt).toBeNull();
|
||||
expect(byName.get("account-four")?.sizeBytes).toBeNull();
|
||||
expect(byName.get("account-four")?.lastAttemptAt).not.toBeNull();
|
||||
// `unknown` is the honest one: the ledger has no history for this
|
||||
// account, which is not the same claim as "the bucket is empty".
|
||||
expect(byName.get("account-five")?.lastAttemptAt).toBeNull();
|
||||
});
|
||||
|
||||
it("says so when neither the environment nor the ledger knows of any upload", async () => {
|
||||
configuredMock.mockReturnValue(false);
|
||||
findManyMock.mockResolvedValue([
|
||||
{ id: "u1", username: "account-one", offhostBackupState: null },
|
||||
]);
|
||||
|
||||
const body = await read();
|
||||
|
||||
expect(body.offhost).toEqual({
|
||||
configured: false,
|
||||
periodHours: 24,
|
||||
rows: [],
|
||||
});
|
||||
});
|
||||
|
||||
it("lets a ledger row outrank this process's own environment", async () => {
|
||||
// A hand-rolled split deployment can give BACKUP_S3_* to the worker and
|
||||
// not to the web process. The rows are proof uploads are happening;
|
||||
// saying "nothing leaves this host" over them would be this process
|
||||
// reporting on a job it cannot see.
|
||||
configuredMock.mockReturnValue(false);
|
||||
findManyMock.mockResolvedValue([
|
||||
{
|
||||
id: "u1",
|
||||
username: "account-one",
|
||||
offhostBackupState: {
|
||||
lastAttemptAt: new Date(),
|
||||
lastSuccessAt: new Date(),
|
||||
sizeBytes: BigInt(4096),
|
||||
},
|
||||
},
|
||||
]);
|
||||
|
||||
const body = await read();
|
||||
|
||||
expect(body.offhost.configured).toBe(true);
|
||||
expect(body.offhost.rows.map((row) => row.freshness)).toEqual(["fresh"]);
|
||||
});
|
||||
|
||||
it("never ships a credential or an object key with the verdicts", async () => {
|
||||
findManyMock.mockResolvedValue([
|
||||
{
|
||||
id: "u1",
|
||||
username: "account-one",
|
||||
offhostBackupState: {
|
||||
lastAttemptAt: new Date(),
|
||||
lastSuccessAt: new Date(),
|
||||
sizeBytes: BigInt(1),
|
||||
},
|
||||
},
|
||||
]);
|
||||
|
||||
const body = await read();
|
||||
|
||||
expect(Object.keys(body.offhost.rows[0] ?? {}).sort()).toEqual([
|
||||
"ageHours",
|
||||
"freshness",
|
||||
"lastAttemptAt",
|
||||
"lastSuccessAt",
|
||||
"sizeBytes",
|
||||
"userId",
|
||||
"username",
|
||||
]);
|
||||
expect(JSON.stringify(body.offhost)).not.toMatch(/bucket|endpoint|secret/i);
|
||||
});
|
||||
});
|
||||
@@ -13,6 +13,12 @@
|
||||
* Without those two the page cannot tell a working backup from one that
|
||||
* stopped six weeks ago — every row it lists has a perfectly ordinary
|
||||
* timestamp either way.
|
||||
*
|
||||
* And alongside both, the off-host leg: one row per account saying when this
|
||||
* host last put that account's encrypted copy in the operator's bucket and how
|
||||
* stale that is against the nightly schedule. It comes from the ledger the
|
||||
* worker writes, never from a bucket listing — the answer has to hold on a host
|
||||
* whose worker holds no listing grant on the bucket.
|
||||
*/
|
||||
import { prisma } from "@/lib/db";
|
||||
import { apiHandler, requireAdmin } from "@/lib/api-handler";
|
||||
@@ -21,10 +27,20 @@ import { annotate } from "@/lib/logging/context";
|
||||
import { DATA_BACKUP_QUEUE } from "@/lib/jobs/data-backup-policy";
|
||||
import { readLastQueueRun } from "@/lib/jobs/job-failures";
|
||||
import { summariseBackupSchedule } from "@/lib/jobs/backup-schedule-status";
|
||||
import { offhostBackupConfigured } from "@/lib/jobs/offhost-backup";
|
||||
import {
|
||||
classifyOffhostBackup,
|
||||
OFFHOST_BACKUP_PERIOD_HOURS,
|
||||
type OffhostBackupFreshness,
|
||||
} from "@/lib/jobs/offhost-backup-freshness";
|
||||
// v1.4.41 W-ORG — `BackupRow` / `BackupsList` moved to `src/types/backups.ts`
|
||||
// so callers (in particular `components/admin/backups-section.tsx`) don't
|
||||
// have to reach across the component → route-handler layer boundary.
|
||||
import type { BackupRow, BackupsList } from "@/types/backups";
|
||||
import type {
|
||||
BackupRow,
|
||||
BackupsList,
|
||||
OffhostAccountRow,
|
||||
} from "@/types/backups";
|
||||
|
||||
export const dynamic = "force-dynamic";
|
||||
|
||||
@@ -67,6 +83,79 @@ export const GET = apiHandler(async () => {
|
||||
createdAt: row.created_at.toISOString(),
|
||||
}));
|
||||
|
||||
// The off-host leg, per account, read from the ledger the nightly worker
|
||||
// writes rather than from the bucket. The page must be able to answer "which
|
||||
// account has no recent copy off-host" on a host whose worker holds the
|
||||
// three grants the runbook documents and no listing grant at all, and a
|
||||
// listing call from a page render would put the operator's credentials on
|
||||
// the request path of every page view.
|
||||
const now = new Date();
|
||||
const offhostAccounts = await prisma.user.findMany({
|
||||
select: {
|
||||
id: true,
|
||||
username: true,
|
||||
offhostBackupState: {
|
||||
select: {
|
||||
lastAttemptAt: true,
|
||||
lastSuccessAt: true,
|
||||
sizeBytes: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
orderBy: { username: "asc" },
|
||||
});
|
||||
|
||||
// Evidence beats this process's environment. `offhostBackupConfigured()`
|
||||
// reads the variables of whichever process serves this request, and a
|
||||
// hand-rolled split that gives `BACKUP_S3_*` only to the worker would make
|
||||
// the web process say "nothing leaves this host" over a ledger full of
|
||||
// fresh rows. A row in that table is proof an upload happened; no
|
||||
// environment read can outrank it.
|
||||
const configured =
|
||||
offhostBackupConfigured() ||
|
||||
offhostAccounts.some((account) => account.offhostBackupState !== null);
|
||||
|
||||
const offhostRows: OffhostAccountRow[] = offhostAccounts.map((account) => {
|
||||
const state = account.offhostBackupState;
|
||||
const verdict = classifyOffhostBackup({
|
||||
lastAttemptAt: state?.lastAttemptAt ?? null,
|
||||
lastSuccessAt: state?.lastSuccessAt ?? null,
|
||||
now,
|
||||
});
|
||||
return {
|
||||
userId: account.id,
|
||||
username: account.username,
|
||||
lastAttemptAt: state?.lastAttemptAt.toISOString() ?? null,
|
||||
lastSuccessAt: state?.lastSuccessAt?.toISOString() ?? null,
|
||||
// BigInt column -> number on the wire. The value is bytes of one
|
||||
// object, capped by the uploader at 80 GB, so it is nowhere near
|
||||
// Number.MAX_SAFE_INTEGER and JSON has no BigInt.
|
||||
sizeBytes:
|
||||
state?.sizeBytes === null || state?.sizeBytes === undefined
|
||||
? null
|
||||
: Number(state.sizeBytes),
|
||||
ageHours: verdict.ageHours,
|
||||
freshness: verdict.freshness,
|
||||
};
|
||||
});
|
||||
|
||||
// Worst first. Sorting by username puts the one account that needs
|
||||
// attention wherever the alphabet happens to leave it, which on a
|
||||
// hundred-account cohort is the same "one bad row hides in the crowd"
|
||||
// problem this card was built to end, moved out of a count and into a list.
|
||||
const FRESHNESS_ORDER: Record<OffhostBackupFreshness, number> = {
|
||||
stale: 0,
|
||||
due: 1,
|
||||
never: 2,
|
||||
unknown: 3,
|
||||
fresh: 4,
|
||||
};
|
||||
offhostRows.sort(
|
||||
(a, b) =>
|
||||
FRESHNESS_ORDER[a.freshness] - FRESHNESS_ORDER[b.freshness] ||
|
||||
a.username.localeCompare(b.username),
|
||||
);
|
||||
|
||||
const payload: BackupsList = {
|
||||
rows: list,
|
||||
// Matches the retention window the backup-prune job enforces so the
|
||||
@@ -77,8 +166,15 @@ export const GET = apiHandler(async () => {
|
||||
.filter((row) => row.type === SCHEDULED_BACKUP_TYPE)
|
||||
.map((row) => row.created_at),
|
||||
lastRun: await readLastQueueRun(DATA_BACKUP_QUEUE),
|
||||
now: new Date(),
|
||||
now,
|
||||
}),
|
||||
offhost: {
|
||||
configured,
|
||||
periodHours: OFFHOST_BACKUP_PERIOD_HOURS,
|
||||
// An unconfigured host has nothing to list, and the card says so rather
|
||||
// than painting an empty table that reads as "no problems".
|
||||
rows: configured ? offhostRows : [],
|
||||
},
|
||||
};
|
||||
|
||||
return apiSuccess(payload);
|
||||
|
||||
@@ -16,9 +16,32 @@ import {
|
||||
invalidateServerDefaultTimezone,
|
||||
isValidTimezone,
|
||||
} from "@/lib/tz/resolver";
|
||||
import { summariseGlitchtipDelivery } from "@/lib/monitoring/glitchtip-delivery";
|
||||
|
||||
export const dynamic = "force-dynamic";
|
||||
|
||||
/** The delivery verdict, flattened onto the settings payload. */
|
||||
function summariseGlitchtipDeliveryFields(state: {
|
||||
lastOkAt?: Date | null;
|
||||
lastFailureAt?: Date | null;
|
||||
lastFailureReason?: string | null;
|
||||
}) {
|
||||
const summary = summariseGlitchtipDelivery(
|
||||
{
|
||||
lastOkAt: state.lastOkAt ?? null,
|
||||
lastFailureAt: state.lastFailureAt ?? null,
|
||||
lastFailureReason: state.lastFailureReason ?? null,
|
||||
},
|
||||
new Date(),
|
||||
);
|
||||
return {
|
||||
glitchtipReportsDelivering: summary.reportsDelivering,
|
||||
glitchtipEverDelivered: summary.everDelivered,
|
||||
glitchtipLastFailureReason: summary.lastFailureReason,
|
||||
glitchtipDeliveryWindowHours: summary.windowHours,
|
||||
};
|
||||
}
|
||||
|
||||
export const GET = apiHandler(async () => {
|
||||
await requireAdmin();
|
||||
annotate({ action: { name: "admin.settings.get" } });
|
||||
@@ -48,6 +71,15 @@ export const GET = apiHandler(async () => {
|
||||
glitchtipEnabled: settings?.glitchtipEnabled ?? false,
|
||||
glitchtipDsn: settings?.glitchtipDsn ?? null,
|
||||
glitchtipEnvironment: settings?.glitchtipEnvironment ?? "production",
|
||||
// The delivery outcome, not the intent, and resolved here rather than on
|
||||
// the client: a DSN that is set and parses says a target was typed, only
|
||||
// this says something left the host, and whether it left recently enough
|
||||
// is a question about a clock that belongs with the ledger.
|
||||
...summariseGlitchtipDeliveryFields({
|
||||
lastOkAt: settings?.glitchtipLastOkAt ?? null,
|
||||
lastFailureAt: settings?.glitchtipLastFailureAt ?? null,
|
||||
lastFailureReason: settings?.glitchtipLastFailureReason ?? null,
|
||||
}),
|
||||
reminderLateMinutes: settings?.reminderLateMinutes ?? 120,
|
||||
reminderMissedMinutes: settings?.reminderMissedMinutes ?? 240,
|
||||
// Document vault limits (per-file cap + per-user quota default). BigInt
|
||||
@@ -274,6 +306,11 @@ export const PUT = apiHandler(async (request: NextRequest) => {
|
||||
glitchtipEnabled: settings.glitchtipEnabled,
|
||||
glitchtipDsn: settings.glitchtipDsn,
|
||||
glitchtipEnvironment: settings.glitchtipEnvironment ?? "production",
|
||||
...summariseGlitchtipDeliveryFields({
|
||||
lastOkAt: settings.glitchtipLastOkAt,
|
||||
lastFailureAt: settings.glitchtipLastFailureAt,
|
||||
lastFailureReason: settings.glitchtipLastFailureReason,
|
||||
}),
|
||||
reminderLateMinutes: settings.reminderLateMinutes,
|
||||
reminderMissedMinutes: settings.reminderMissedMinutes,
|
||||
documentMaxFileBytes: settings.documentMaxFileBytes,
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
/**
|
||||
* The GlitchTip badge answers "are reports arriving", not "did somebody type
|
||||
* a DSN".
|
||||
*
|
||||
* The two questions used to have one answer: enabled plus a URL that parses.
|
||||
* A wrong public key, a wrong project id and a host that stopped answering all
|
||||
* parse perfectly, so a green badge over any of them is the false reassurance
|
||||
* an operator only discovers when they go looking for a crash that was never
|
||||
* reported. The card now reads the delivery outcome this host recorded.
|
||||
*/
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { renderToStaticMarkup } from "react-dom/server";
|
||||
import type { AdminSettings } from "../_shared";
|
||||
|
||||
const settingsResult = vi.hoisted(() => ({
|
||||
current: null as Partial<AdminSettings> | null,
|
||||
}));
|
||||
|
||||
vi.mock("@tanstack/react-query", () => ({
|
||||
useQuery: () => ({ data: settingsResult.current }),
|
||||
useMutation: () => ({ mutate: vi.fn(), isPending: false }),
|
||||
useQueryClient: () => ({ invalidateQueries: vi.fn() }),
|
||||
}));
|
||||
|
||||
vi.mock("../_shared", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("../_shared")>();
|
||||
return {
|
||||
...actual,
|
||||
useAdminSettings: () => ({ data: settingsResult.current }),
|
||||
useUpdateSettings: () => ({ mutate: vi.fn(), isPending: false }),
|
||||
};
|
||||
});
|
||||
|
||||
import { I18nProvider } from "@/lib/i18n/context";
|
||||
import { GlitchtipSection } from "../glitchtip-section";
|
||||
|
||||
const BASE = {
|
||||
glitchtipEnabled: true,
|
||||
glitchtipDsn: "https://public-key@glitchtip.example.com/7",
|
||||
glitchtipEnvironment: "production",
|
||||
glitchtipReportsDelivering: false,
|
||||
glitchtipEverDelivered: false,
|
||||
glitchtipLastFailureReason: null,
|
||||
glitchtipDeliveryWindowHours: 336,
|
||||
} satisfies Partial<AdminSettings>;
|
||||
|
||||
function render(settings: Partial<AdminSettings> | null): string {
|
||||
settingsResult.current = settings;
|
||||
return renderToStaticMarkup(
|
||||
<I18nProvider initialLocale="en">
|
||||
<GlitchtipSection />
|
||||
</I18nProvider>,
|
||||
);
|
||||
}
|
||||
|
||||
describe("<GlitchtipSection> delivery badge", () => {
|
||||
it("does not call a parseable DSN configured until something has left the host", () => {
|
||||
const html = render(BASE);
|
||||
expect(html).toContain("Unproven");
|
||||
expect(html).not.toContain(">Configured<");
|
||||
expect(html).toContain("no report has left this host yet");
|
||||
});
|
||||
|
||||
it("names the failure when the target refused the last report", () => {
|
||||
const html = render({ ...BASE, glitchtipLastFailureReason: "http_403" });
|
||||
expect(html).toContain("Unproven");
|
||||
expect(html).toContain("did not get through");
|
||||
expect(html).toContain("http_403");
|
||||
});
|
||||
|
||||
it("goes green only for a report that left inside the window", () => {
|
||||
const html = render({
|
||||
...BASE,
|
||||
glitchtipReportsDelivering: true,
|
||||
glitchtipEverDelivered: true,
|
||||
});
|
||||
expect(html).toContain(">Configured<");
|
||||
expect(html).toContain("glitchtip.example.com");
|
||||
expect(html).not.toContain('data-slot="glitchtip-delivery"');
|
||||
});
|
||||
|
||||
it("stops claiming delivery once the last success ages out of the window", () => {
|
||||
const html = render({
|
||||
...BASE,
|
||||
glitchtipDeliveryWindowHours: 24,
|
||||
glitchtipEverDelivered: true,
|
||||
});
|
||||
expect(html).toContain("Unproven");
|
||||
expect(html).toContain("No report has left this host in the last");
|
||||
});
|
||||
|
||||
it("says nothing is sent when the switch is off", () => {
|
||||
const html = render({ ...BASE, glitchtipEnabled: false });
|
||||
expect(html).toContain("Error reports stay on this host");
|
||||
expect(html).not.toContain("Unproven");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,112 @@
|
||||
/**
|
||||
* The off-host card paints what the response says, including the case an
|
||||
* operator most needs stated out loud: this host is not sending anything
|
||||
* anywhere. A card that renders an empty list on an unconfigured host reads
|
||||
* as "no problems".
|
||||
*/
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import { renderToStaticMarkup } from "react-dom/server";
|
||||
import type { BackupsList } from "@/types/backups";
|
||||
|
||||
const queryResult = vi.hoisted(() => ({
|
||||
current: {
|
||||
data: null as BackupsList | null,
|
||||
isLoading: false,
|
||||
isError: false,
|
||||
},
|
||||
}));
|
||||
|
||||
vi.mock("@tanstack/react-query", () => ({
|
||||
useQuery: () => ({ ...queryResult.current, refetch: vi.fn() }),
|
||||
}));
|
||||
|
||||
import { I18nProvider } from "@/lib/i18n/context";
|
||||
import { OffhostBackupsSection } from "../offhost-backups-section";
|
||||
|
||||
function render(data: BackupsList["offhost"] | null): string {
|
||||
queryResult.current = {
|
||||
data:
|
||||
data === null
|
||||
? null
|
||||
: ({
|
||||
rows: [],
|
||||
retentionDays: 90,
|
||||
schedule: null,
|
||||
offhost: data,
|
||||
} as unknown as BackupsList),
|
||||
isLoading: false,
|
||||
isError: false,
|
||||
};
|
||||
return renderToStaticMarkup(
|
||||
<I18nProvider initialLocale="en">
|
||||
<OffhostBackupsSection />
|
||||
</I18nProvider>,
|
||||
);
|
||||
}
|
||||
|
||||
describe("<OffhostBackupsSection>", () => {
|
||||
it("says off-host backup is not configured rather than showing an empty list", () => {
|
||||
const html = render({ configured: false, periodHours: 24, rows: [] });
|
||||
expect(html).toContain("Off-host backup is not configured");
|
||||
expect(html).not.toContain('data-slot="offhost-backup-rows"');
|
||||
});
|
||||
|
||||
it("paints one row per account with its verdict", () => {
|
||||
const html = render({
|
||||
configured: true,
|
||||
periodHours: 24,
|
||||
rows: [
|
||||
{
|
||||
userId: "u1",
|
||||
username: "account-one",
|
||||
lastAttemptAt: "2026-09-10T06:00:00.000Z",
|
||||
lastSuccessAt: "2026-09-10T06:00:00.000Z",
|
||||
sizeBytes: 4096,
|
||||
ageHours: 3,
|
||||
freshness: "fresh",
|
||||
},
|
||||
{
|
||||
userId: "u2",
|
||||
username: "account-two",
|
||||
lastAttemptAt: "2026-09-06T06:00:00.000Z",
|
||||
lastSuccessAt: "2026-09-06T06:00:00.000Z",
|
||||
sizeBytes: 2048,
|
||||
ageHours: 99,
|
||||
freshness: "stale",
|
||||
},
|
||||
{
|
||||
userId: "u3",
|
||||
username: "account-three",
|
||||
lastAttemptAt: "2026-09-10T02:30:00.000Z",
|
||||
lastSuccessAt: null,
|
||||
sizeBytes: null,
|
||||
ageHours: null,
|
||||
freshness: "never",
|
||||
},
|
||||
{
|
||||
userId: "u4",
|
||||
username: "account-four",
|
||||
lastAttemptAt: null,
|
||||
lastSuccessAt: null,
|
||||
sizeBytes: null,
|
||||
ageHours: null,
|
||||
freshness: "unknown",
|
||||
},
|
||||
],
|
||||
});
|
||||
|
||||
expect(html).toContain('data-offhost-freshness="fresh"');
|
||||
expect(html).toContain('data-offhost-freshness="stale"');
|
||||
expect(html).toContain('data-offhost-freshness="never"');
|
||||
expect(html).toContain('data-offhost-freshness="unknown"');
|
||||
expect(html).toContain("account-two");
|
||||
// The account a run walked and found nothing for says that, with the
|
||||
// instant it was checked.
|
||||
expect(html).toContain("Nothing has landed for this account");
|
||||
// The account no run has recorded says THAT — not "never", which would
|
||||
// claim the bucket is empty on the strength of an empty ledger.
|
||||
expect(html).toContain("No nightly run has recorded this account yet");
|
||||
// Never the bucket's own coordinates.
|
||||
expect(html).not.toMatch(/secret|access[- ]key/i);
|
||||
});
|
||||
});
|
||||
@@ -101,6 +101,17 @@ export interface AdminSettings {
|
||||
glitchtipEnabled: boolean;
|
||||
glitchtipDsn: string | null;
|
||||
glitchtipEnvironment: string | null;
|
||||
/** A report left this host successfully inside the delivery window. */
|
||||
glitchtipReportsDelivering: boolean;
|
||||
/** A report has left this host successfully at some point. */
|
||||
glitchtipEverDelivered: boolean;
|
||||
/**
|
||||
* Why the last attempt failed, when that failure is newer than the last
|
||||
* success. A short classification, never a remote response body.
|
||||
*/
|
||||
glitchtipLastFailureReason: string | null;
|
||||
/** How long a successful send stands for before the badge stops claiming it. */
|
||||
glitchtipDeliveryWindowHours: number;
|
||||
reminderLateMinutes: number;
|
||||
reminderMissedMinutes: number;
|
||||
// v1.4.25 W7 — null means "fall back to Europe/Berlin in the resolver".
|
||||
|
||||
@@ -47,6 +47,7 @@ import { SettingsCard } from "@/components/settings/settings-card";
|
||||
import { SettingsCardHeader } from "@/components/settings/_card-header";
|
||||
import { ListRow } from "@/components/ui/list-row";
|
||||
import { useFormatters, useTranslations } from "@/lib/i18n/context";
|
||||
import { formatBytes } from "@/lib/format/bytes";
|
||||
import { queryKeys } from "@/lib/query-keys";
|
||||
import { randomId } from "@/lib/random-id";
|
||||
import type { BackupRow, BackupsList } from "@/types/backups";
|
||||
@@ -322,14 +323,6 @@ function RestoreRowDialog({
|
||||
);
|
||||
}
|
||||
|
||||
function formatBytes(bytes: number, fmt: ReturnType<typeof useFormatters>) {
|
||||
if (bytes < 1024) return `${bytes} B`;
|
||||
if (bytes < 1024 * 1024) {
|
||||
return `${fmt.integer(Math.round(bytes / 1024))} KB`;
|
||||
}
|
||||
return `${fmt.number(bytes / 1024 / 1024, 2)} MB`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Render the `DataBackup.type` enum (`WEEKLY_AUTO` / `MANUAL`) as a
|
||||
* human label. Unknown values fall through to the raw enum so a future
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
"use client";
|
||||
|
||||
import { useState } from "react";
|
||||
import { useMutation } from "@tanstack/react-query";
|
||||
import { useMutation, useQueryClient } from "@tanstack/react-query";
|
||||
import { AlertTriangle, Loader2 } from "lucide-react";
|
||||
import { toast } from "sonner";
|
||||
import { SettingsCard } from "@/components/settings/settings-card";
|
||||
import { SettingsCardActions } from "@/components/settings/_card-actions";
|
||||
import { SettingsCardHeader } from "@/components/settings/_card-header";
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import { Input } from "@/components/ui/input";
|
||||
import { Label } from "@/components/ui/label";
|
||||
@@ -18,9 +19,11 @@ import {
|
||||
ConfiguredBadge,
|
||||
} from "./_shared";
|
||||
import { apiPost } from "@/lib/api/api-fetch";
|
||||
import { queryKeys } from "@/lib/query-keys";
|
||||
|
||||
export function GlitchtipSection() {
|
||||
const { t } = useTranslations();
|
||||
const queryClient = useQueryClient();
|
||||
const { data: settings } = useAdminSettings();
|
||||
const updateSettings = useUpdateSettings();
|
||||
const [glitchtipDsnDraft, setGlitchtipDsnDraft] = useState<string | null>(
|
||||
@@ -34,7 +37,63 @@ export function GlitchtipSection() {
|
||||
const glitchtipEnvironmentValue =
|
||||
glitchtipEnvironmentDraft ?? settings?.glitchtipEnvironment ?? "production";
|
||||
|
||||
const configured = Boolean(settings?.glitchtipDsn);
|
||||
/**
|
||||
* The host the DSN addresses, and nothing else from it. An operator asking
|
||||
* "where do my errors go" is asking for a hostname; the public key in the
|
||||
* DSN answers a different question and does not belong in a status line.
|
||||
*
|
||||
* There is deliberately no default here and none anywhere else in the tree:
|
||||
* a self-hoster's crash reports must not leave their host because nobody
|
||||
* told them not to. Reporting starts when this operator types a target.
|
||||
*/
|
||||
const targetHost = ((): string | null => {
|
||||
const dsn = settings?.glitchtipDsn;
|
||||
if (!dsn) return null;
|
||||
try {
|
||||
return new URL(dsn).host || null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
})();
|
||||
|
||||
/**
|
||||
* Three states, because "a target was typed" and "reports are arriving" are
|
||||
* different facts and only the second earns a green badge.
|
||||
*
|
||||
* off — the switch is off, or no target parses. Nothing is sent.
|
||||
* unproven — a target is set and nothing has come back from it inside the
|
||||
* window. A wrong public key, a wrong project id and a host
|
||||
* that stopped answering all look exactly like this, which is
|
||||
* why it is not green.
|
||||
* delivering — a report left successfully inside the window. The test
|
||||
* button is the cheap way to prove it on demand.
|
||||
*/
|
||||
const deliveryState = ((): "off" | "unproven" | "delivering" => {
|
||||
if (!settings?.glitchtipEnabled || !targetHost) return "off";
|
||||
// Whether the last success is recent enough is decided where the ledger
|
||||
// lives, not here: a render is not the place to read a clock.
|
||||
return settings.glitchtipReportsDelivering ? "delivering" : "unproven";
|
||||
})();
|
||||
|
||||
/**
|
||||
* Why the badge is not green, in words. A reason the host recorded beats a
|
||||
* generic "unproven": an operator who typed a DSN with the wrong project id
|
||||
* gets told the target refused it rather than left to guess.
|
||||
*/
|
||||
const unprovenReason = ((): string | null => {
|
||||
if (deliveryState !== "unproven") return null;
|
||||
if (settings?.glitchtipLastFailureReason) {
|
||||
return t("admin.glitchtipDeliveryFailed", {
|
||||
reason: settings.glitchtipLastFailureReason,
|
||||
});
|
||||
}
|
||||
if (settings?.glitchtipEverDelivered) {
|
||||
return t("admin.glitchtipDeliveryStale", {
|
||||
hours: settings.glitchtipDeliveryWindowHours,
|
||||
});
|
||||
}
|
||||
return t("admin.glitchtipDeliveryUnproven");
|
||||
})();
|
||||
|
||||
const testGlitchtip = useMutation({
|
||||
mutationFn: async () => {
|
||||
@@ -45,6 +104,11 @@ export function GlitchtipSection() {
|
||||
},
|
||||
onSuccess: (message) => {
|
||||
toast.success(message);
|
||||
// The send just recorded an outcome. Re-read it, so the badge answers
|
||||
// the question the operator pressed the button to ask.
|
||||
void queryClient.invalidateQueries({
|
||||
queryKey: queryKeys.adminSettings(),
|
||||
});
|
||||
},
|
||||
onError: (error) => {
|
||||
toast.error(
|
||||
@@ -52,6 +116,9 @@ export function GlitchtipSection() {
|
||||
? error.message
|
||||
: t("admin.monitoringTestFailed"),
|
||||
);
|
||||
void queryClient.invalidateQueries({
|
||||
queryKey: queryKeys.adminSettings(),
|
||||
});
|
||||
},
|
||||
});
|
||||
|
||||
@@ -76,7 +143,15 @@ export function GlitchtipSection() {
|
||||
icon={AlertTriangle}
|
||||
title={t("admin.glitchtipTitle")}
|
||||
description={t("admin.glitchtipDescription")}
|
||||
status={configured ? <ConfiguredBadge /> : null}
|
||||
status={
|
||||
deliveryState === "delivering" ? (
|
||||
<ConfiguredBadge />
|
||||
) : deliveryState === "unproven" ? (
|
||||
<Badge className="border-warning/30 bg-warning/15 text-warning">
|
||||
{t("admin.glitchtipUnproven")}
|
||||
</Badge>
|
||||
) : null
|
||||
}
|
||||
/>
|
||||
|
||||
<div className="space-y-3">
|
||||
@@ -89,6 +164,28 @@ export function GlitchtipSection() {
|
||||
}
|
||||
disabled={updateSettings.isPending}
|
||||
/>
|
||||
{/* Only once the settings are in hand: a line that says "nothing is
|
||||
sent" while the answer is still loading is a claim, not a
|
||||
placeholder. */}
|
||||
{settings ? (
|
||||
<div className="space-y-1">
|
||||
<p className="text-sm" data-slot="glitchtip-target">
|
||||
{deliveryState !== "off" && targetHost
|
||||
? t("admin.glitchtipTargetOn", { host: targetHost })
|
||||
: settings.glitchtipEnabled
|
||||
? t("admin.glitchtipTargetNoTarget")
|
||||
: t("admin.glitchtipTargetOff")}
|
||||
</p>
|
||||
{unprovenReason ? (
|
||||
<p
|
||||
className="text-muted-foreground text-xs"
|
||||
data-slot="glitchtip-delivery"
|
||||
>
|
||||
{unprovenReason}
|
||||
</p>
|
||||
) : null}
|
||||
</div>
|
||||
) : null}
|
||||
<div className="grid gap-3 sm:grid-cols-2">
|
||||
<div className="space-y-1.5 sm:col-span-2">
|
||||
<Label htmlFor="admin-glitchtip-dsn" className="text-xs">
|
||||
|
||||
@@ -0,0 +1,176 @@
|
||||
"use client";
|
||||
|
||||
/**
|
||||
* `<OffhostBackupsSection>` — per-account freshness of the off-host copies.
|
||||
*
|
||||
* The card above it lists the copies that live in THIS database. This one
|
||||
* answers the different question: for each account, how long ago did this host
|
||||
* put an encrypted copy in the operator's bucket, and is that inside what the
|
||||
* nightly schedule promises. The two fail independently — a database whose
|
||||
* weekly rows are current tells an operator nothing about a bucket that has
|
||||
* been refusing the worker's signature since March.
|
||||
*
|
||||
* Every verdict is computed server-side from the ledger the worker writes. The
|
||||
* page never lists the bucket: the worker's grant is deliberately PutObject,
|
||||
* GetObject and AbortMultipartUpload — the three the runbook documents, the
|
||||
* abort being what the multipart path needs — and putting a credentialed
|
||||
* listing call on the render path of an admin page would be a poor trade for
|
||||
* a number the database already holds.
|
||||
*/
|
||||
|
||||
import { useQuery } from "@tanstack/react-query";
|
||||
import { CloudUpload } from "lucide-react";
|
||||
import { SettingsCard } from "@/components/settings/settings-card";
|
||||
import { SettingsCardHeader } from "@/components/settings/_card-header";
|
||||
import { Badge } from "@/components/ui/badge";
|
||||
import { EmptyState } from "@/components/ui/empty-state";
|
||||
import { ListRow } from "@/components/ui/list-row";
|
||||
import { QueryErrorRow } from "@/components/ui/query-error-row";
|
||||
import { Skeleton } from "@/components/ui/skeleton";
|
||||
import { apiGet } from "@/lib/api/api-fetch";
|
||||
import { cn } from "@/lib/utils";
|
||||
import { useFormatters, useTranslations } from "@/lib/i18n/context";
|
||||
import { queryKeys } from "@/lib/query-keys";
|
||||
import type { OffhostBackupFreshness } from "@/lib/jobs/offhost-backup-freshness";
|
||||
import type { BackupsList, OffhostAccountRow } from "@/types/backups";
|
||||
import { formatBytes } from "@/lib/format/bytes";
|
||||
|
||||
/**
|
||||
* Status wash per verdict. `never` and `unknown` are deliberately neutral
|
||||
* rather than alarming: a host that turned the off-host job on this afternoon
|
||||
* has not failed at anything yet, and painting every row red on day one is how
|
||||
* an operator learns to stop reading the colour.
|
||||
*/
|
||||
const FRESHNESS_STYLE: Record<OffhostBackupFreshness, string> = {
|
||||
fresh: "border-success/30 bg-success/10 text-success",
|
||||
due: "border-warning/30 bg-warning/10 text-warning",
|
||||
stale: "border-destructive/30 bg-destructive/10 text-destructive",
|
||||
never: "border-border bg-muted text-muted-foreground",
|
||||
unknown: "border-border bg-muted text-muted-foreground",
|
||||
};
|
||||
|
||||
function freshnessLabel(
|
||||
freshness: OffhostBackupFreshness,
|
||||
t: ReturnType<typeof useTranslations>["t"],
|
||||
): string {
|
||||
// Literal keys, one per arm: a template-built key resolves at runtime and is
|
||||
// invisible to the call-site coverage guard.
|
||||
switch (freshness) {
|
||||
case "fresh":
|
||||
return t("admin.section.backups.offhost.stateFresh");
|
||||
case "due":
|
||||
return t("admin.section.backups.offhost.stateDue");
|
||||
case "stale":
|
||||
return t("admin.section.backups.offhost.stateStale");
|
||||
case "never":
|
||||
return t("admin.section.backups.offhost.stateNever");
|
||||
case "unknown":
|
||||
return t("admin.section.backups.offhost.stateUnknown");
|
||||
}
|
||||
}
|
||||
|
||||
function AccountRow({ row }: { row: OffhostAccountRow }) {
|
||||
const { t } = useTranslations();
|
||||
const fmt = useFormatters();
|
||||
|
||||
return (
|
||||
<ListRow asChild className="bg-muted/30 border-border">
|
||||
<li
|
||||
data-slot="offhost-backup-row"
|
||||
data-offhost-username={row.username}
|
||||
data-offhost-freshness={row.freshness}
|
||||
>
|
||||
<div className="flex items-start justify-between gap-3">
|
||||
<span className="min-w-0 flex-1 truncate font-medium">
|
||||
{row.username}
|
||||
</span>
|
||||
<Badge
|
||||
variant="outline"
|
||||
className={cn("shrink-0", FRESHNESS_STYLE[row.freshness])}
|
||||
>
|
||||
{freshnessLabel(row.freshness, t)}
|
||||
</Badge>
|
||||
</div>
|
||||
<p className="text-muted-foreground text-xs">
|
||||
{row.lastAttemptAt === null
|
||||
? // No run has recorded this account. Saying "never" here would
|
||||
// claim the bucket is empty when what is empty is the ledger,
|
||||
// which is exactly what every account looks like between the
|
||||
// upgrade that adds it and the first nightly run after that.
|
||||
t("admin.section.backups.offhost.unknownDetail")
|
||||
: row.lastSuccessAt === null || row.sizeBytes === null
|
||||
? t("admin.section.backups.offhost.neverDetail", {
|
||||
when: fmt.dateTime(row.lastAttemptAt),
|
||||
})
|
||||
: `${fmt.dateTime(row.lastSuccessAt)} · ${formatBytes(
|
||||
row.sizeBytes,
|
||||
fmt,
|
||||
)} · ${t("admin.section.backups.offhost.age", {
|
||||
hours: row.ageHours ?? 0,
|
||||
})}`}
|
||||
</p>
|
||||
</li>
|
||||
</ListRow>
|
||||
);
|
||||
}
|
||||
|
||||
export function OffhostBackupsSection() {
|
||||
const { t } = useTranslations();
|
||||
|
||||
const { data, isLoading, isError, refetch } = useQuery({
|
||||
queryKey: queryKeys.adminBackups(),
|
||||
queryFn: async () => apiGet<BackupsList>("/api/admin/backups"),
|
||||
});
|
||||
|
||||
const offhost = data?.offhost;
|
||||
|
||||
return (
|
||||
<SettingsCard>
|
||||
<SettingsCardHeader
|
||||
icon={CloudUpload}
|
||||
title={t("admin.section.backups.offhost.title")}
|
||||
description={t("admin.section.backups.offhost.description")}
|
||||
status={
|
||||
offhost && offhost.configured ? (
|
||||
<Badge variant="secondary" className="text-xs">
|
||||
{offhost.rows.length}
|
||||
</Badge>
|
||||
) : null
|
||||
}
|
||||
/>
|
||||
|
||||
{isLoading ? (
|
||||
<div className="space-y-2" data-slot="offhost-backup-loading">
|
||||
<Skeleton className="h-14 w-full" />
|
||||
<Skeleton className="h-14 w-full" />
|
||||
</div>
|
||||
) : isError ? (
|
||||
<QueryErrorRow
|
||||
message={t("admin.section.backups.offhost.loadError")}
|
||||
onRetry={() => void refetch()}
|
||||
/>
|
||||
) : offhost && !offhost.configured ? (
|
||||
<EmptyState
|
||||
icon={<CloudUpload className="size-6" />}
|
||||
title={t("admin.section.backups.offhost.notConfiguredTitle")}
|
||||
description={t(
|
||||
"admin.section.backups.offhost.notConfiguredDescription",
|
||||
)}
|
||||
/>
|
||||
) : offhost ? (
|
||||
<div className="space-y-2">
|
||||
<ul className="space-y-2" data-slot="offhost-backup-rows">
|
||||
{offhost.rows.map((row) => (
|
||||
<AccountRow key={row.userId} row={row} />
|
||||
))}
|
||||
</ul>
|
||||
<p className="text-muted-foreground text-xs">
|
||||
{t("admin.section.backups.offhost.scheduleNote", {
|
||||
hours: offhost.periodHours,
|
||||
})}
|
||||
</p>
|
||||
</div>
|
||||
) : null}
|
||||
</SettingsCard>
|
||||
);
|
||||
}
|
||||
@@ -210,6 +210,11 @@ export const WIPE_MODELS = [
|
||||
// just returned.
|
||||
"IdempotencyKey",
|
||||
"DataBackup",
|
||||
// What this host last put in the operator's bucket for this account. The
|
||||
// objects themselves stay — the worker holds no DeleteObject grant and the
|
||||
// bucket's lifecycle rule owns their retirement — but the row saying when
|
||||
// they were written is about this account and goes with it.
|
||||
"OffhostBackupState",
|
||||
"AuditLog",
|
||||
"ConsentReceipt",
|
||||
"UserKnownDevice",
|
||||
|
||||
@@ -642,6 +642,8 @@ export const NOT_IN_BACKUP_MODELS: Readonly<Record<string, string>> = {
|
||||
"Replay protection for requests this host already answered. Restoring it would suppress a legitimate new request.",
|
||||
DataBackup:
|
||||
"The backup catalogue itself. A backup that contains the list of backups is a recursion with no reader.",
|
||||
OffhostBackupState:
|
||||
"When this host last put this account's copy in this operator's bucket, and how big it was. It describes one deployment's relationship with one bucket, so restoring it elsewhere would assert an off-host copy that host has never written.",
|
||||
ImportJob:
|
||||
"A job record pointing at an uploaded file that the backup does not carry, so restoring it would resurrect a task with nothing to work on.",
|
||||
InviteToken:
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
import type { useFormatters } from "@/lib/i18n/context";
|
||||
|
||||
/**
|
||||
* One spelling of "1.25 MB" for every surface that shows a byte count from
|
||||
* the admin backups response.
|
||||
*
|
||||
* It lives here rather than beside the card that happened to need it first:
|
||||
* the second reader was importing a formatter out of a thousand-line client
|
||||
* component, which is a dependency on a page's worth of code for four lines of
|
||||
* arithmetic. Both cards render on the same route today, so it costs nothing
|
||||
* today — and it becomes a cost the moment either one moves.
|
||||
*/
|
||||
export function formatBytes(
|
||||
bytes: number,
|
||||
fmt: ReturnType<typeof useFormatters>,
|
||||
): string {
|
||||
if (bytes < 1024) return `${bytes} B`;
|
||||
if (bytes < 1024 * 1024) {
|
||||
return `${fmt.integer(Math.round(bytes / 1024))} KB`;
|
||||
}
|
||||
return `${fmt.number(bytes / 1024 / 1024, 2)} MB`;
|
||||
}
|
||||
@@ -0,0 +1,161 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
classifyOffhostBackup,
|
||||
OFFHOST_BACKUP_GRACE_HOURS,
|
||||
OFFHOST_BACKUP_PERIOD_HOURS,
|
||||
} from "@/lib/jobs/offhost-backup-freshness";
|
||||
|
||||
const NOW = new Date("2026-09-10T09:00:00.000Z");
|
||||
const HOUR = 3_600_000;
|
||||
const PERIOD = OFFHOST_BACKUP_PERIOD_HOURS * HOUR;
|
||||
const GRACE = OFFHOST_BACKUP_GRACE_HOURS * HOUR;
|
||||
|
||||
/** `age` hours before NOW. */
|
||||
function ago(ms: number): Date {
|
||||
return new Date(NOW.getTime() - ms);
|
||||
}
|
||||
|
||||
describe("classifyOffhostBackup", () => {
|
||||
it("says unknown, not never, when no run has recorded this account", () => {
|
||||
// The ledger starts empty. On the first morning after the upgrade every
|
||||
// account on a perfectly healthy host looks like this, and `never` would
|
||||
// assert something about the bucket the host has not looked at.
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: null,
|
||||
lastSuccessAt: null,
|
||||
now: NOW,
|
||||
}),
|
||||
).toEqual({ freshness: "unknown", ageHours: null });
|
||||
});
|
||||
|
||||
it("says never once a run has walked the account and put nothing there", () => {
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(HOUR),
|
||||
lastSuccessAt: null,
|
||||
now: NOW,
|
||||
}),
|
||||
).toEqual({ freshness: "never", ageHours: null });
|
||||
});
|
||||
|
||||
it("is fresh right up to one period plus the grace and due one millisecond past it", () => {
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(0),
|
||||
lastSuccessAt: ago(PERIOD + GRACE),
|
||||
now: NOW,
|
||||
}).freshness,
|
||||
).toBe("fresh");
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(0),
|
||||
lastSuccessAt: ago(PERIOD + GRACE + 1),
|
||||
now: NOW,
|
||||
}).freshness,
|
||||
).toBe("due");
|
||||
});
|
||||
|
||||
it("is due right up to two periods plus the grace and stale one millisecond past it", () => {
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(0),
|
||||
lastSuccessAt: ago(PERIOD * 2 + GRACE),
|
||||
now: NOW,
|
||||
}).freshness,
|
||||
).toBe("due");
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(0),
|
||||
lastSuccessAt: ago(PERIOD * 2 + GRACE + 1),
|
||||
now: NOW,
|
||||
}).freshness,
|
||||
).toBe("stale");
|
||||
});
|
||||
|
||||
it("keeps the whole cohort fresh across the Berlin DST fall-back night", () => {
|
||||
// 2026-10-25 is the fall-back Sunday in Europe/Berlin: the clocks go back
|
||||
// at 03:00 CEST, so 02:30 local happens 25 hours after the previous
|
||||
// 02:30 local. Both instants are pinned in UTC because that is what the
|
||||
// ledger stores, and the test must not depend on the runner's own zone.
|
||||
const beforeTheChange = new Date("2026-10-24T00:30:00.000Z"); // 02:30 CEST
|
||||
const afterTheChange = new Date("2026-10-25T01:30:00.000Z"); // 02:30 CET
|
||||
const gapHours =
|
||||
(afterTheChange.getTime() - beforeTheChange.getTime()) / HOUR;
|
||||
expect(gapHours).toBe(25);
|
||||
|
||||
// Without slack the whole cohort would read `due` on that one morning,
|
||||
// on every host, with nothing wrong anywhere.
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(0),
|
||||
lastSuccessAt: beforeTheChange,
|
||||
now: afterTheChange,
|
||||
}).freshness,
|
||||
).toBe("fresh");
|
||||
});
|
||||
|
||||
it("still calls a genuinely missed night due", () => {
|
||||
// 31 hours: past one period even with six hours of slack, so a run that
|
||||
// produced nothing for this account still shows.
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(0),
|
||||
lastSuccessAt: ago(31 * HOUR),
|
||||
now: NOW,
|
||||
}).freshness,
|
||||
).toBe("due");
|
||||
});
|
||||
|
||||
it("reports whole hours of age", () => {
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(0),
|
||||
lastSuccessAt: ago(90 * 60_000),
|
||||
now: NOW,
|
||||
}),
|
||||
).toEqual({ freshness: "fresh", ageHours: 1 });
|
||||
});
|
||||
|
||||
it("honours a caller's own period rather than the nightly default", () => {
|
||||
// A six-hour schedule: the same 12-hour-old object that is fresh under the
|
||||
// nightly cron is two periods behind under this one. The grace is passed
|
||||
// as zero so the thresholds under test are the caller's period alone.
|
||||
const twelveHours = ago(12 * HOUR);
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(0),
|
||||
lastSuccessAt: twelveHours,
|
||||
now: NOW,
|
||||
}).freshness,
|
||||
).toBe("fresh");
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(0),
|
||||
lastSuccessAt: twelveHours,
|
||||
now: NOW,
|
||||
periodHours: 6,
|
||||
graceHours: 0,
|
||||
}).freshness,
|
||||
).toBe("due");
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(0),
|
||||
lastSuccessAt: twelveHours,
|
||||
now: NOW,
|
||||
periodHours: 5,
|
||||
graceHours: 0,
|
||||
}).freshness,
|
||||
).toBe("stale");
|
||||
});
|
||||
|
||||
it("does not call a copy stale because the host clock ran backwards", () => {
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: ago(0),
|
||||
lastSuccessAt: new Date(NOW.getTime() + 2 * HOUR),
|
||||
now: NOW,
|
||||
}),
|
||||
).toEqual({ freshness: "fresh", ageHours: 0 });
|
||||
});
|
||||
});
|
||||
@@ -184,6 +184,7 @@ describe("runOffhostBackup", () => {
|
||||
s3.store.set("2020-01-01/user-old.json.enc", Buffer.from([0]));
|
||||
|
||||
const prisma = {
|
||||
offhostBackupState: { upsert: vi.fn() },
|
||||
user: {
|
||||
findMany: vi.fn().mockResolvedValue([{ id: "u1" }, { id: "u2" }]),
|
||||
},
|
||||
@@ -229,11 +230,28 @@ describe("runOffhostBackup", () => {
|
||||
// Nothing went through the whole-buffer arm.
|
||||
expect(s3.putObject).not.toHaveBeenCalled();
|
||||
expect(s3.putStream).toHaveBeenCalledTimes(2);
|
||||
|
||||
// Every account that got an object gets a ledger row, because the run's
|
||||
// own counts cannot say WHICH account has a copy off-host and which one
|
||||
// has been failing quietly since March.
|
||||
expect(prisma.offhostBackupState.upsert).toHaveBeenCalledTimes(2);
|
||||
const ledgered = prisma.offhostBackupState.upsert.mock.calls.map(
|
||||
(call) => (call[0] as { where: { userId: string } }).where.userId,
|
||||
);
|
||||
expect(ledgered).toEqual(["u1", "u2"]);
|
||||
const first = prisma.offhostBackupState.upsert.mock.calls[0]?.[0] as {
|
||||
create: { sizeBytes: bigint; lastSuccessAt: Date };
|
||||
};
|
||||
// BigInt, because the uploader permits an 80 GB object and a 32-bit
|
||||
// column would refuse the biggest account on the host.
|
||||
expect(first.create.sizeBytes).toBe(BigInt(ct.byteLength));
|
||||
expect(first.create.lastSuccessAt).toBeInstanceOf(Date);
|
||||
});
|
||||
|
||||
it("uploads the canonical builder output without reshaping it", async () => {
|
||||
const s3 = makeS3Mock();
|
||||
const prisma = {
|
||||
offhostBackupState: { upsert: vi.fn() },
|
||||
user: { findMany: vi.fn().mockResolvedValue([{ id: "u1" }]) },
|
||||
};
|
||||
const canonicalPayload = {
|
||||
@@ -290,7 +308,9 @@ describe("runOffhostBackup", () => {
|
||||
|
||||
it("counts per-user failures without aborting the whole run", async () => {
|
||||
const s3 = makeS3Mock();
|
||||
const upsert = vi.fn();
|
||||
const prisma = {
|
||||
offhostBackupState: { upsert },
|
||||
user: {
|
||||
findMany: vi.fn().mockResolvedValue([{ id: "u1" }, { id: "u2" }]),
|
||||
},
|
||||
@@ -314,11 +334,55 @@ describe("runOffhostBackup", () => {
|
||||
|
||||
expect(report.uploaded).toBe(1);
|
||||
expect(report.failed).toBe(1);
|
||||
// Both accounts are recorded as walked — that is what separates "a run
|
||||
// looked and found nothing" from "no run has looked at this account" —
|
||||
// but only the one whose object landed carries a success.
|
||||
type Upsert = {
|
||||
where: { userId: string };
|
||||
update: Record<string, unknown>;
|
||||
create: Record<string, unknown>;
|
||||
};
|
||||
const calls = upsert.mock.calls.map((call) => call[0] as Upsert);
|
||||
expect(calls.map((call) => call.where.userId)).toEqual(["u1", "u2"]);
|
||||
expect(calls[0]?.update).toHaveProperty("lastSuccessAt");
|
||||
// The guarantee an operator leans on: an account whose upload failed keeps
|
||||
// whatever `lastSuccessAt` it already had. The failed arm names only the
|
||||
// walk instant, so yesterday's good copy is still what the console
|
||||
// reports — the run does not overwrite it with today's failure.
|
||||
expect(calls[1]?.update).not.toHaveProperty("lastSuccessAt");
|
||||
expect(calls[1]?.update).not.toHaveProperty("sizeBytes");
|
||||
expect(calls[1]?.create).not.toHaveProperty("lastSuccessAt");
|
||||
});
|
||||
|
||||
it("still reports a landed object as uploaded when the ledger write fails", async () => {
|
||||
const s3 = makeS3Mock();
|
||||
const prisma = {
|
||||
offhostBackupState: {
|
||||
upsert: vi.fn().mockRejectedValue(new Error("pool timeout")),
|
||||
},
|
||||
user: { findMany: vi.fn().mockResolvedValue([{ id: "u1" }]) },
|
||||
};
|
||||
|
||||
const report = await runOffhostBackup(
|
||||
prisma as never,
|
||||
s3,
|
||||
new Date("2026-05-08T00:00:00Z"),
|
||||
);
|
||||
|
||||
// The object is in the bucket. Only the record of it is missing, and that
|
||||
// is a different fact: counting it as a failure would tell the operator
|
||||
// their disaster-recovery copy does not exist when it does.
|
||||
expect(report.uploaded).toBe(1);
|
||||
expect(report.failed).toBe(0);
|
||||
expect(report.failures).toEqual([]);
|
||||
expect(s3.store.has("2026-05-08/user-u1.json.enc")).toBe(true);
|
||||
expect(prisma.offhostBackupState.upsert).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it("refuses an account whose object outgrows one multipart upload", async () => {
|
||||
const s3 = makeS3Mock();
|
||||
const prisma = {
|
||||
offhostBackupState: { upsert: vi.fn() },
|
||||
user: { findMany: vi.fn().mockResolvedValue([{ id: "u1" }]) },
|
||||
};
|
||||
mocks.buildFullBackupPayload.mockResolvedValue({
|
||||
@@ -355,6 +419,7 @@ describe("runOffhostBackup", () => {
|
||||
it("reports the largest object it wrote", async () => {
|
||||
const s3 = makeS3Mock();
|
||||
const prisma = {
|
||||
offhostBackupState: { upsert: vi.fn() },
|
||||
user: { findMany: vi.fn().mockResolvedValue([{ id: "u1" }]) },
|
||||
};
|
||||
const report = await runOffhostBackup(
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
/**
|
||||
* How fresh one account's newest off-host copy is.
|
||||
*
|
||||
* The nightly off-host job reports counts — uploaded, failed, total — and a
|
||||
* count is exactly the shape that hides a single account. A cohort where one
|
||||
* account's object has failed every night for a month reports "99 uploaded, 1
|
||||
* failed" every night, which reads like weather rather than like an account
|
||||
* that has no disaster-recovery copy at all. The per-account ledger the worker
|
||||
* writes (`OffhostBackupState`) turns that into a fact the console can state
|
||||
* per row, and this function is the only place that fact is judged.
|
||||
*
|
||||
* Deliberately pure and deliberately not talking to the bucket: the admin page
|
||||
* computes staleness from the timestamp the worker wrote, so the answer is the
|
||||
* same whether or not the app process holds a read grant on the bucket, and a
|
||||
* listing of somebody's backups never crosses the network to show a page.
|
||||
*
|
||||
* Five states, because the run is nightly and one missed night is not the same
|
||||
* event as four — and because "we have not looked yet" is not a verdict:
|
||||
*
|
||||
* unknown — no nightly run has recorded this account. The ledger starts
|
||||
* empty, so this is what every account on a perfectly healthy host
|
||||
* reads between the upgrade that adds the table and the first run
|
||||
* after it. Saying "never" there would assert something the host
|
||||
* cannot know.
|
||||
* never — a run walked this account and put no object there, and no earlier
|
||||
* run ever did either.
|
||||
* fresh — inside one schedule period plus the grace below. This is what
|
||||
* every account looks like on a host where the cron is doing its
|
||||
* job.
|
||||
* due — past one period, inside two. One night produced nothing: a retry, a
|
||||
* long run, a restart during the window. Worth showing, not worth
|
||||
* alarming about.
|
||||
* stale — past two periods. Two consecutive nights produced nothing for this
|
||||
* account, which no ordinary hiccup explains.
|
||||
*/
|
||||
|
||||
/**
|
||||
* Hours between two scheduled off-host runs. Tracks `OFFHOST_BACKUP_CRON`
|
||||
* (`30 2 * * *`) in `src/lib/jobs/reminder/register-maintenance.ts`, which
|
||||
* every schedule runs in `Europe/Berlin` (`tz` in `registrar-shared.ts`).
|
||||
* This is the number the card quotes — the schedule's own period, not the
|
||||
* threshold a verdict is measured against.
|
||||
*/
|
||||
export const OFFHOST_BACKUP_PERIOD_HOURS = 24;
|
||||
|
||||
/**
|
||||
* Slack on top of one period, before a verdict moves.
|
||||
*
|
||||
* Two things make a healthy host miss a bare 24 hours. The schedule is
|
||||
* Berlin-local, so on the DST fall-back night the gap between two 02:30 runs
|
||||
* is 25 hours and every account on every host would cross one period on the
|
||||
* same morning. And the ledger instant is when THAT account's object landed,
|
||||
* deliberately: on a cohort walked one account at a time, an account reached
|
||||
* at 05:30 one night and 06:30 the next is 25 hours old with two entirely
|
||||
* successful runs behind it.
|
||||
*
|
||||
* Six hours is the budget for both. A card whose argument is that a count
|
||||
* trains an operator to stop reading it cannot afford an annual cohort-wide
|
||||
* false `due`.
|
||||
*/
|
||||
export const OFFHOST_BACKUP_GRACE_HOURS = 6;
|
||||
|
||||
export type OffhostBackupFreshness =
|
||||
"unknown" | "never" | "fresh" | "due" | "stale";
|
||||
|
||||
export interface OffhostBackupVerdict {
|
||||
freshness: OffhostBackupFreshness;
|
||||
/** Whole hours since the newest object landed; null when there is none. */
|
||||
ageHours: number | null;
|
||||
}
|
||||
|
||||
export function classifyOffhostBackup(args: {
|
||||
/**
|
||||
* When a run last walked this account, or null when none has. Null is the
|
||||
* only thing that separates `unknown` from `never`.
|
||||
*/
|
||||
lastAttemptAt: Date | null;
|
||||
lastSuccessAt: Date | null;
|
||||
now: Date;
|
||||
/** Defaults to the nightly schedule. Tests pass their own. */
|
||||
periodHours?: number;
|
||||
/** Defaults to `OFFHOST_BACKUP_GRACE_HOURS`. Tests pass their own. */
|
||||
graceHours?: number;
|
||||
}): OffhostBackupVerdict {
|
||||
const { lastAttemptAt, lastSuccessAt, now } = args;
|
||||
if (lastAttemptAt === null) return { freshness: "unknown", ageHours: null };
|
||||
if (lastSuccessAt === null) return { freshness: "never", ageHours: null };
|
||||
|
||||
const periodMs =
|
||||
(args.periodHours ?? OFFHOST_BACKUP_PERIOD_HOURS) * 3_600_000;
|
||||
const graceMs = (args.graceHours ?? OFFHOST_BACKUP_GRACE_HOURS) * 3_600_000;
|
||||
const ageMs = now.getTime() - lastSuccessAt.getTime();
|
||||
// A clock that ran backwards between the upload and this read reports a
|
||||
// negative age. That is a host-clock problem, not a backup problem, and
|
||||
// calling the copy stale over it would point the operator at the wrong
|
||||
// thing.
|
||||
const ageHours = Math.max(0, Math.floor(ageMs / 3_600_000));
|
||||
|
||||
// The grace rides on each threshold rather than on the period, so `due`
|
||||
// still means "one scheduled run produced nothing" and `stale` still means
|
||||
// two — the slack only keeps a long night from being read as a missed one.
|
||||
if (ageMs <= periodMs + graceMs) return { freshness: "fresh", ageHours };
|
||||
if (ageMs <= periodMs * 2 + graceMs) return { freshness: "due", ageHours };
|
||||
return { freshness: "stale", ageHours };
|
||||
}
|
||||
@@ -28,7 +28,7 @@ import { createGzip, gunzipSync, gzipSync } from "node:zlib";
|
||||
import type { PrismaClient } from "@/generated/prisma/client";
|
||||
import { createRawStreamEncryptor, decryptRawStream } from "@/lib/crypto";
|
||||
import { streamFullBackupJson } from "@/lib/export/full-backup-stream";
|
||||
import { getEvent } from "@/lib/logging/context";
|
||||
import { annotate, getEvent } from "@/lib/logging/context";
|
||||
|
||||
const ALGORITHM = "aes-256-gcm";
|
||||
const IV_LENGTH = 12;
|
||||
@@ -61,6 +61,27 @@ function decodeBackupKey(raw: string): Buffer {
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether this host has the five variables the nightly job needs.
|
||||
*
|
||||
* Separate from `loadOffhostConfig()` because a surface that only wants to say
|
||||
* "off-host backup is not set up here" must not decode the encryption key to
|
||||
* find out — `decodeBackupKey` throws on a malformed one, and an operator
|
||||
* whose key has a typo should see the admin page, not a 500.
|
||||
*/
|
||||
export function offhostBackupConfigured(): boolean {
|
||||
return OFFHOST_ENV_VARS.every((name) => Boolean(process.env[name]));
|
||||
}
|
||||
|
||||
/** The five variables the nightly job needs, in one place. */
|
||||
const OFFHOST_ENV_VARS = [
|
||||
"BACKUP_S3_ENDPOINT",
|
||||
"BACKUP_S3_BUCKET",
|
||||
"BACKUP_S3_ACCESS_KEY",
|
||||
"BACKUP_S3_SECRET_KEY",
|
||||
"BACKUP_ENCRYPTION_KEY",
|
||||
] as const;
|
||||
|
||||
export function loadOffhostConfig(): OffhostBackupConfig | null {
|
||||
const endpoint = process.env.BACKUP_S3_ENDPOINT;
|
||||
const bucket = process.env.BACKUP_S3_BUCKET;
|
||||
@@ -555,10 +576,12 @@ export async function runOffhostBackup(
|
||||
let oversized = 0;
|
||||
let largestObjectBytes = 0;
|
||||
const failures: Array<{ userId: string; message: string }> = [];
|
||||
let ledgerWriteFailures = 0;
|
||||
const evt = getEvent();
|
||||
for (const user of users) {
|
||||
let objectBytes: number | null = null;
|
||||
try {
|
||||
const objectBytes = await uploadEncryptedBackup(
|
||||
objectBytes = await uploadEncryptedBackup(
|
||||
s3,
|
||||
`${dateKey}/user-${user.id}.json.enc`,
|
||||
cfg.encryptionKey,
|
||||
@@ -586,6 +609,62 @@ export async function runOffhostBackup(
|
||||
`offhost-backup user ${user.id} failed: ${message.slice(0, 200)}`,
|
||||
);
|
||||
}
|
||||
|
||||
// The per-account ledger, and deliberately NOT inside the upload's `try`.
|
||||
// `uploadEncryptedBackup` resolves only after the multipart completes, so
|
||||
// by here the object is durably in the bucket and this account's backup
|
||||
// has succeeded whatever the database does next. A pool timeout on the
|
||||
// row would otherwise un-count a copy that exists — the run would report
|
||||
// one failure too many and the console would read stale for an account
|
||||
// whose disaster-recovery object is fine, which is exactly the inversion
|
||||
// the ledger exists to remove.
|
||||
//
|
||||
// The row is written whether or not an object came of the walk, because a
|
||||
// ledger keyed only on success cannot tell "no run has reached this
|
||||
// account" from "a run reached it and nothing landed" — and the first of
|
||||
// those is what every account on a running host looks like the day this
|
||||
// table ships. A failed walk touches `lastAttemptAt` and leaves
|
||||
// `lastSuccessAt` exactly as it was, so yesterday's good copy still reads
|
||||
// as the copy this account has.
|
||||
//
|
||||
// `new Date()` rather than the run's `now`, because on a large cohort the
|
||||
// two are hours apart and the row is meant to say when this account was
|
||||
// reached.
|
||||
const at = new Date();
|
||||
try {
|
||||
await prisma.offhostBackupState.upsert({
|
||||
where: { userId: user.id },
|
||||
update:
|
||||
objectBytes === null
|
||||
? { lastAttemptAt: at }
|
||||
: {
|
||||
lastAttemptAt: at,
|
||||
lastSuccessAt: at,
|
||||
sizeBytes: BigInt(objectBytes),
|
||||
},
|
||||
create:
|
||||
objectBytes === null
|
||||
? { userId: user.id, lastAttemptAt: at }
|
||||
: {
|
||||
userId: user.id,
|
||||
lastAttemptAt: at,
|
||||
lastSuccessAt: at,
|
||||
sizeBytes: BigInt(objectBytes),
|
||||
},
|
||||
});
|
||||
} catch (err) {
|
||||
ledgerWriteFailures++;
|
||||
const message = (err as Error).message ?? "unknown";
|
||||
evt?.addWarning(
|
||||
`offhost-backup ledger write failed for ${user.id}: ${message.slice(0, 200)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (ledgerWriteFailures > 0) {
|
||||
// A distinct fact from a failed upload, and one an operator has to be able
|
||||
// to tell apart: the copies are in the bucket, the console's per-account
|
||||
// view of them is behind.
|
||||
annotate({ meta: { offhost_ledger_write_failures: ledgerWriteFailures } });
|
||||
}
|
||||
|
||||
return {
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
/**
|
||||
* The write end of the delivery ledger.
|
||||
*
|
||||
* The badge is only as honest as what gets recorded, and two things have to
|
||||
* hold: a failure must never carry a remote host's response body into this
|
||||
* database, and a send that could not be recorded must not become a second
|
||||
* error on a path that is already handling one.
|
||||
*/
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
|
||||
const updateMock = vi.fn();
|
||||
vi.mock("@/lib/db", () => ({
|
||||
prisma: { appSettings: { update: (...a: unknown[]) => updateMock(...a) } },
|
||||
}));
|
||||
|
||||
import {
|
||||
classifyGlitchtipFailure,
|
||||
GLITCHTIP_DELIVERY_WINDOW_HOURS,
|
||||
recordGlitchtipDelivery,
|
||||
summariseGlitchtipDelivery,
|
||||
} from "../glitchtip-delivery";
|
||||
|
||||
const NOW = new Date("2026-09-10T09:00:00.000Z");
|
||||
const HOUR = 3_600_000;
|
||||
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
updateMock.mockResolvedValue({});
|
||||
});
|
||||
|
||||
describe("classifyGlitchtipFailure", () => {
|
||||
it("keeps the classification short and free of the remote body", () => {
|
||||
expect(
|
||||
classifyGlitchtipFailure({
|
||||
ok: false,
|
||||
status: 403,
|
||||
details: "<html>forbidden, and here is our whole error page</html>",
|
||||
}),
|
||||
).toBe("http_403");
|
||||
expect(
|
||||
classifyGlitchtipFailure({ ok: false, details: "invalid_dsn" }),
|
||||
).toBe("invalid_dsn");
|
||||
expect(classifyGlitchtipFailure({ ok: false })).toBe("network");
|
||||
});
|
||||
});
|
||||
|
||||
describe("recordGlitchtipDelivery", () => {
|
||||
it("stamps a success and leaves the failure fields alone", async () => {
|
||||
await recordGlitchtipDelivery({ ok: true, status: 200 });
|
||||
|
||||
const data = updateMock.mock.calls[0]?.[0]?.data as Record<string, unknown>;
|
||||
expect(data.glitchtipLastOkAt).toBeInstanceOf(Date);
|
||||
expect(data).not.toHaveProperty("glitchtipLastFailureAt");
|
||||
});
|
||||
|
||||
it("records a failure by classification, never by response body", async () => {
|
||||
await recordGlitchtipDelivery({
|
||||
ok: false,
|
||||
status: 401,
|
||||
details: "invalid public key: abcdef0123456789",
|
||||
});
|
||||
|
||||
const data = updateMock.mock.calls[0]?.[0]?.data as Record<string, unknown>;
|
||||
expect(data.glitchtipLastFailureReason).toBe("http_401");
|
||||
expect(JSON.stringify(data)).not.toContain("abcdef0123456789");
|
||||
expect(data).not.toHaveProperty("glitchtipLastOkAt");
|
||||
});
|
||||
|
||||
it("swallows a write failure rather than replacing the original error", async () => {
|
||||
updateMock.mockRejectedValue(new Error("no singleton row yet"));
|
||||
await expect(
|
||||
recordGlitchtipDelivery({ ok: false, status: 500 }),
|
||||
).resolves.toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("summariseGlitchtipDelivery", () => {
|
||||
it("claims delivery only for a success inside the window", () => {
|
||||
const inside = new Date(
|
||||
NOW.getTime() - (GLITCHTIP_DELIVERY_WINDOW_HOURS - 1) * HOUR,
|
||||
);
|
||||
const outside = new Date(
|
||||
NOW.getTime() - (GLITCHTIP_DELIVERY_WINDOW_HOURS + 1) * HOUR,
|
||||
);
|
||||
|
||||
expect(
|
||||
summariseGlitchtipDelivery(
|
||||
{ lastOkAt: inside, lastFailureAt: null, lastFailureReason: null },
|
||||
NOW,
|
||||
).reportsDelivering,
|
||||
).toBe(true);
|
||||
|
||||
const aged = summariseGlitchtipDelivery(
|
||||
{ lastOkAt: outside, lastFailureAt: null, lastFailureReason: null },
|
||||
NOW,
|
||||
);
|
||||
// Still true that something once got through, no longer true that the
|
||||
// target is working — two different claims, and only one is a badge.
|
||||
expect(aged.reportsDelivering).toBe(false);
|
||||
expect(aged.everDelivered).toBe(true);
|
||||
});
|
||||
|
||||
it("says nothing has been proven when the ledger is empty", () => {
|
||||
expect(
|
||||
summariseGlitchtipDelivery(
|
||||
{ lastOkAt: null, lastFailureAt: null, lastFailureReason: null },
|
||||
NOW,
|
||||
),
|
||||
).toEqual({
|
||||
reportsDelivering: false,
|
||||
everDelivered: false,
|
||||
lastFailureReason: null,
|
||||
windowHours: GLITCHTIP_DELIVERY_WINDOW_HOURS,
|
||||
});
|
||||
});
|
||||
|
||||
it("drops a failure that a later success has superseded", () => {
|
||||
const summary = summariseGlitchtipDelivery(
|
||||
{
|
||||
lastOkAt: new Date(NOW.getTime() - HOUR),
|
||||
lastFailureAt: new Date(NOW.getTime() - 5 * HOUR),
|
||||
lastFailureReason: "http_403",
|
||||
},
|
||||
NOW,
|
||||
);
|
||||
// The target refused a report and then accepted one. Showing the refusal
|
||||
// would report a problem that has since been fixed.
|
||||
expect(summary.lastFailureReason).toBeNull();
|
||||
expect(summary.reportsDelivering).toBe(true);
|
||||
});
|
||||
|
||||
it("keeps a failure that is newer than the last success", () => {
|
||||
expect(
|
||||
summariseGlitchtipDelivery(
|
||||
{
|
||||
lastOkAt: new Date(NOW.getTime() - 5 * HOUR),
|
||||
lastFailureAt: new Date(NOW.getTime() - HOUR),
|
||||
lastFailureReason: "http_403",
|
||||
},
|
||||
NOW,
|
||||
).lastFailureReason,
|
||||
).toBe("http_403");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,132 @@
|
||||
/**
|
||||
* What actually happened to the last error report this host tried to send.
|
||||
*
|
||||
* The admin card used to derive "configured" from the enabled flag plus a DSN
|
||||
* that parses. Those two say a target was typed; they say nothing about the
|
||||
* target accepting anything. A wrong public key, a wrong project id, a host
|
||||
* that stopped answering — every one of them parses, and every one of them
|
||||
* would have shown a green badge and a line naming the host, which is the
|
||||
* false reassurance an operator only discovers when they go looking for a
|
||||
* crash that was never reported.
|
||||
*
|
||||
* So the outcome is recorded where the DSN already lives, next to the flag it
|
||||
* qualifies, and the badge reads it. `sendGlitchtipEvent` is the one writer —
|
||||
* every caller in the tree goes through it, so no send path can skip the
|
||||
* ledger by construction.
|
||||
*/
|
||||
import { prisma } from "@/lib/db";
|
||||
import type { GlitchtipDeliveryResult } from "@/lib/monitoring/glitchtip";
|
||||
|
||||
/**
|
||||
* How long a successful send stands for. A self-hoster's instance can go days
|
||||
* without an unhandled error, and a badge that expired overnight would train
|
||||
* the operator to ignore it; a fortnight is long enough that only a target
|
||||
* that has genuinely stopped working falls out of it, and the test button
|
||||
* refreshes it on demand.
|
||||
*/
|
||||
export const GLITCHTIP_DELIVERY_WINDOW_HOURS = 14 * 24;
|
||||
|
||||
export interface GlitchtipDeliveryState {
|
||||
lastOkAt: Date | null;
|
||||
lastFailureAt: Date | null;
|
||||
lastFailureReason: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* A short, safe classification of a failure. Never the response body: that is
|
||||
* text from a remote host, and an incident target's error page has no business
|
||||
* being rendered on the admin page or stored in this database.
|
||||
*/
|
||||
export function classifyGlitchtipFailure(
|
||||
result: GlitchtipDeliveryResult,
|
||||
): string {
|
||||
if (result.details === "invalid_dsn") return "invalid_dsn";
|
||||
if (typeof result.status === "number") return `http_${result.status}`;
|
||||
return "network";
|
||||
}
|
||||
|
||||
/**
|
||||
* Record the outcome of one send. Best-effort by design — a report that could
|
||||
* not be filed must not become a second error, and the singleton row may not
|
||||
* exist yet on a host that has never opened the admin page.
|
||||
*/
|
||||
export async function recordGlitchtipDelivery(
|
||||
result: GlitchtipDeliveryResult,
|
||||
): Promise<void> {
|
||||
const now = new Date();
|
||||
try {
|
||||
await prisma.appSettings.update({
|
||||
where: { id: "singleton" },
|
||||
data: result.ok
|
||||
? { glitchtipLastOkAt: now }
|
||||
: {
|
||||
glitchtipLastFailureAt: now,
|
||||
glitchtipLastFailureReason: classifyGlitchtipFailure(result),
|
||||
},
|
||||
});
|
||||
} catch {
|
||||
// Nothing to do and nothing to report to: the caller is already on an
|
||||
// error path, and throwing here would replace the original failure with
|
||||
// this one.
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The judgement, resolved server-side.
|
||||
*
|
||||
* The client gets the verdict, not the timestamps: whether a report is inside
|
||||
* the window is a question about a clock, and the process that holds the
|
||||
* ledger is the one that should answer it. A failure older than the last
|
||||
* success is not surfaced at all — it has been superseded by a report that
|
||||
* got through, and showing it would be reporting a fixed problem.
|
||||
*/
|
||||
export interface GlitchtipDeliverySummary {
|
||||
/** A report left this host successfully inside the window. */
|
||||
reportsDelivering: boolean;
|
||||
/** A report has left this host successfully at some point. */
|
||||
everDelivered: boolean;
|
||||
/** Why the last attempt failed, when that failure is the newest fact. */
|
||||
lastFailureReason: string | null;
|
||||
/** How long a success stands for, so the card can say it. */
|
||||
windowHours: number;
|
||||
}
|
||||
|
||||
export function summariseGlitchtipDelivery(
|
||||
state: GlitchtipDeliveryState,
|
||||
now: Date,
|
||||
): GlitchtipDeliverySummary {
|
||||
const windowMs = GLITCHTIP_DELIVERY_WINDOW_HOURS * 3_600_000;
|
||||
const okAge =
|
||||
state.lastOkAt === null ? null : now.getTime() - state.lastOkAt.getTime();
|
||||
const failureIsNewest =
|
||||
state.lastFailureAt !== null &&
|
||||
(state.lastOkAt === null ||
|
||||
state.lastFailureAt.getTime() > state.lastOkAt.getTime());
|
||||
|
||||
return {
|
||||
reportsDelivering: okAge !== null && okAge >= 0 && okAge <= windowMs,
|
||||
everDelivered: state.lastOkAt !== null,
|
||||
lastFailureReason: failureIsNewest ? state.lastFailureReason : null,
|
||||
windowHours: GLITCHTIP_DELIVERY_WINDOW_HOURS,
|
||||
};
|
||||
}
|
||||
|
||||
export async function getGlitchtipDeliveryState(): Promise<GlitchtipDeliveryState> {
|
||||
try {
|
||||
const settings = await prisma.appSettings.findUnique({
|
||||
where: { id: "singleton" },
|
||||
select: {
|
||||
glitchtipLastOkAt: true,
|
||||
glitchtipLastFailureAt: true,
|
||||
glitchtipLastFailureReason: true,
|
||||
},
|
||||
});
|
||||
return {
|
||||
lastOkAt: settings?.glitchtipLastOkAt ?? null,
|
||||
lastFailureAt: settings?.glitchtipLastFailureAt ?? null,
|
||||
lastFailureReason: settings?.glitchtipLastFailureReason ?? null,
|
||||
};
|
||||
} catch {
|
||||
return { lastOkAt: null, lastFailureAt: null, lastFailureReason: null };
|
||||
}
|
||||
}
|
||||
@@ -214,13 +214,34 @@ async function tryStoreHeader(
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Record what became of one send, in the one place every caller passes
|
||||
* through. Dynamically imported so this module stays free of a database
|
||||
* dependency at import time, and awaited so a test can observe it; the
|
||||
* recorder itself never throws.
|
||||
*/
|
||||
async function record(result: GlitchtipDeliveryResult): Promise<void> {
|
||||
try {
|
||||
const { recordGlitchtipDelivery } =
|
||||
await import("@/lib/monitoring/glitchtip-delivery");
|
||||
await recordGlitchtipDelivery(result);
|
||||
} catch {
|
||||
// The report is the point; its bookkeeping is not worth an exception.
|
||||
}
|
||||
}
|
||||
|
||||
export async function sendGlitchtipEvent(params: {
|
||||
dsn: string;
|
||||
input: GlitchtipEventPayloadInput;
|
||||
}): Promise<GlitchtipDeliveryResult> {
|
||||
const config = parseGlitchtipDsn(params.dsn);
|
||||
if (!config) {
|
||||
return { ok: false, details: "invalid_dsn" };
|
||||
const invalid: GlitchtipDeliveryResult = {
|
||||
ok: false,
|
||||
details: "invalid_dsn",
|
||||
};
|
||||
await record(invalid);
|
||||
return invalid;
|
||||
}
|
||||
|
||||
const eventId = createEventId();
|
||||
@@ -228,15 +249,24 @@ export async function sendGlitchtipEvent(params: {
|
||||
const envelopeBody = buildEnvelopeBody(config, eventId, payload);
|
||||
|
||||
const envelopeResult = await tryEnvelope(config, envelopeBody);
|
||||
if (envelopeResult.ok) return envelopeResult;
|
||||
if (envelopeResult.ok) {
|
||||
await record(envelopeResult);
|
||||
return envelopeResult;
|
||||
}
|
||||
|
||||
const storeQueryResult = await tryStoreQuery(config, payload);
|
||||
if (storeQueryResult.ok) return storeQueryResult;
|
||||
if (storeQueryResult.ok) {
|
||||
await record(storeQueryResult);
|
||||
return storeQueryResult;
|
||||
}
|
||||
|
||||
const storeHeaderResult = await tryStoreHeader(config, payload);
|
||||
if (storeHeaderResult.ok) return storeHeaderResult;
|
||||
if (storeHeaderResult.ok) {
|
||||
await record(storeHeaderResult);
|
||||
return storeHeaderResult;
|
||||
}
|
||||
|
||||
return {
|
||||
const failure: GlitchtipDeliveryResult = {
|
||||
ok: false,
|
||||
method: storeHeaderResult.method,
|
||||
status:
|
||||
@@ -248,4 +278,6 @@ export async function sendGlitchtipEvent(params: {
|
||||
storeQueryResult.details ??
|
||||
envelopeResult.details,
|
||||
};
|
||||
await record(failure);
|
||||
return failure;
|
||||
}
|
||||
@@ -10,6 +10,7 @@
|
||||
* keeps owning the HTTP contract, the component keeps owning the UI.
|
||||
*/
|
||||
import type { BackupScheduleStatus } from "@/lib/jobs/backup-schedule-status";
|
||||
import type { OffhostBackupFreshness } from "@/lib/jobs/offhost-backup-freshness";
|
||||
|
||||
export interface BackupRow {
|
||||
id: string;
|
||||
@@ -24,6 +25,41 @@ export interface BackupRow {
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
/** One account's newest object in the operator's off-host bucket. */
|
||||
export interface OffhostAccountRow {
|
||||
userId: string;
|
||||
username: string;
|
||||
/**
|
||||
* ISO instant a run last walked this account, or null when none has. Null
|
||||
* is what the whole cohort reads until the first run after the upgrade that
|
||||
* adds the ledger, and the card says so rather than claiming `never`.
|
||||
*/
|
||||
lastAttemptAt: string | null;
|
||||
/** ISO instant the newest object landed, or null when there is none. */
|
||||
lastSuccessAt: string | null;
|
||||
/** Size of that object in bytes, or null when there is none. */
|
||||
sizeBytes: number | null;
|
||||
/** Whole hours since it landed, or null when there is none. */
|
||||
ageHours: number | null;
|
||||
freshness: OffhostBackupFreshness;
|
||||
}
|
||||
|
||||
/**
|
||||
* The off-host picture, per account.
|
||||
*
|
||||
* Computed from the ledger the nightly worker writes, never from the bucket:
|
||||
* the page must be able to answer "does this account have a recent copy
|
||||
* off-host" on a host whose worker holds no listing grant on the bucket.
|
||||
*/
|
||||
export interface OffhostBackupOverview {
|
||||
/** Whether this host has the variables the nightly job needs. */
|
||||
configured: boolean;
|
||||
/** Hours between two scheduled runs — the number behind the verdicts. */
|
||||
periodHours: number;
|
||||
/** One row per account. Empty when off-host backup is not configured. */
|
||||
rows: OffhostAccountRow[];
|
||||
}
|
||||
|
||||
export interface BackupsList {
|
||||
rows: BackupRow[];
|
||||
/**
|
||||
@@ -32,6 +68,12 @@ export interface BackupsList {
|
||||
* made six weeks ago and one made on Sunday look identical in a table.
|
||||
*/
|
||||
schedule: BackupScheduleStatus;
|
||||
/**
|
||||
* The off-host leg. A weekly row in this table says the copy that lives in
|
||||
* this database is current; it says nothing about whether anything reached
|
||||
* the operator's bucket, and those two fail independently.
|
||||
*/
|
||||
offhost: OffhostBackupOverview;
|
||||
/**
|
||||
* Soft retention hint — the worker is configured for weekly backups
|
||||
* (see `DATA_BACKUP_CRON` in `src/lib/jobs/reminder-worker.ts`), and
|
||||
|
||||
@@ -0,0 +1,231 @@
|
||||
/**
|
||||
* The off-host ledger, end to end against real Postgres.
|
||||
*
|
||||
* Both ends and the pipe: the nightly job is what writes the row, the admin
|
||||
* console is what reads it, and a unit test that mocks Prisma proves neither
|
||||
* the migration nor the shape they agree on. Here the real
|
||||
* `runOffhostBackup()` uploads into an in-memory bucket double, the row lands
|
||||
* in the real table, and the verdict the console shows is computed from what
|
||||
* came back out.
|
||||
*/
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import { Buffer } from "node:buffer";
|
||||
import type { Readable } from "node:stream";
|
||||
|
||||
import {
|
||||
classifyOffhostBackup,
|
||||
OFFHOST_BACKUP_GRACE_HOURS,
|
||||
OFFHOST_BACKUP_PERIOD_HOURS,
|
||||
} from "@/lib/jobs/offhost-backup-freshness";
|
||||
import {
|
||||
offhostBackupConfigured,
|
||||
runOffhostBackup,
|
||||
type S3Like,
|
||||
} from "@/lib/jobs/offhost-backup";
|
||||
import { getPrismaClient, truncateAllTables } from "./setup";
|
||||
|
||||
const HOUR = 3_600_000;
|
||||
|
||||
/** The five variables the job insists on, none of which reach a network here. */
|
||||
const ENV: Record<string, string> = {
|
||||
BACKUP_S3_ENDPOINT: "http://bucket.invalid",
|
||||
BACKUP_S3_BUCKET: "healthlog-test",
|
||||
BACKUP_S3_ACCESS_KEY: "test-access-key",
|
||||
BACKUP_S3_SECRET_KEY: "test-secret-key",
|
||||
BACKUP_ENCRYPTION_KEY:
|
||||
"1111111111111111111111111111111111111111111111111111111111111111",
|
||||
};
|
||||
|
||||
function makeBucket(): S3Like & { store: Map<string, Buffer> } {
|
||||
const store = new Map<string, Buffer>();
|
||||
return {
|
||||
store,
|
||||
putObject: async (key, body) => {
|
||||
store.set(key, Buffer.from(body));
|
||||
},
|
||||
putStream: async (key: string, body: Readable) => {
|
||||
const chunks: Buffer[] = [];
|
||||
for await (const chunk of body) chunks.push(Buffer.from(chunk));
|
||||
store.set(key, Buffer.concat(chunks));
|
||||
},
|
||||
getObject: async (key) => store.get(key) ?? Buffer.alloc(0),
|
||||
headObject: async (key) => store.has(key),
|
||||
listObjects: async () => [],
|
||||
deleteObject: async (key) => {
|
||||
store.delete(key);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function seedUser(id: string): Promise<void> {
|
||||
await getPrismaClient().user.create({
|
||||
data: {
|
||||
id,
|
||||
username: id,
|
||||
email: `${id}@example.test`,
|
||||
timezone: "Europe/Berlin",
|
||||
},
|
||||
});
|
||||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
await truncateAllTables(getPrismaClient());
|
||||
for (const [name, value] of Object.entries(ENV)) process.env[name] = value;
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
for (const name of Object.keys(ENV)) delete process.env[name];
|
||||
});
|
||||
|
||||
describe("off-host backup ledger (real Postgres)", () => {
|
||||
it("writes one row per uploaded account and reads back as fresh", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
await seedUser("account-one");
|
||||
await seedUser("account-two");
|
||||
|
||||
expect(offhostBackupConfigured()).toBe(true);
|
||||
|
||||
const bucket = makeBucket();
|
||||
const report = await runOffhostBackup(prisma, bucket, new Date());
|
||||
expect(report.uploaded).toBe(2);
|
||||
expect(report.failed).toBe(0);
|
||||
|
||||
const rows = await prisma.offhostBackupState.findMany({
|
||||
orderBy: { userId: "asc" },
|
||||
});
|
||||
expect(rows.map((row) => row.userId)).toEqual([
|
||||
"account-one",
|
||||
"account-two",
|
||||
]);
|
||||
|
||||
for (const row of rows) {
|
||||
// The size the ledger claims is the size of the object that landed, not
|
||||
// an estimate: an operator sizing a bucket reads this column.
|
||||
const object = bucket.store.get(
|
||||
`${new Date().toISOString().slice(0, 10)}/user-${row.userId}.json.enc`,
|
||||
);
|
||||
expect(object).toBeDefined();
|
||||
expect(Number(row.sizeBytes)).toBe(object?.byteLength);
|
||||
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: row.lastAttemptAt,
|
||||
lastSuccessAt: row.lastSuccessAt,
|
||||
now: new Date(),
|
||||
}).freshness,
|
||||
).toBe("fresh");
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps one row per account across runs and ages into stale", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
await seedUser("account-one");
|
||||
|
||||
const bucket = makeBucket();
|
||||
await runOffhostBackup(prisma, bucket, new Date());
|
||||
await runOffhostBackup(prisma, bucket, new Date());
|
||||
|
||||
expect(await prisma.offhostBackupState.count()).toBe(1);
|
||||
|
||||
// Age the row past two nightly runs — the state the console must not miss,
|
||||
// because the run that skipped this account still reported a success for
|
||||
// everybody else.
|
||||
const aged = new Date(
|
||||
Date.now() -
|
||||
(OFFHOST_BACKUP_PERIOD_HOURS * 2 + OFFHOST_BACKUP_GRACE_HOURS + 1) *
|
||||
HOUR,
|
||||
);
|
||||
await prisma.offhostBackupState.update({
|
||||
where: { userId: "account-one" },
|
||||
data: { lastSuccessAt: aged },
|
||||
});
|
||||
|
||||
const row = await prisma.offhostBackupState.findUniqueOrThrow({
|
||||
where: { userId: "account-one" },
|
||||
});
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: row.lastAttemptAt,
|
||||
lastSuccessAt: row.lastSuccessAt,
|
||||
now: new Date(),
|
||||
}).freshness,
|
||||
).toBe("stale");
|
||||
});
|
||||
|
||||
it("separates an account no run has recorded from one a run found nothing for", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
await seedUser("account-one");
|
||||
|
||||
// Before any run: the ledger is empty, which is what every account on a
|
||||
// host that has been uploading for months looks like the moment this
|
||||
// table ships. It must not read as "nothing has ever reached the bucket".
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: null,
|
||||
lastSuccessAt: null,
|
||||
now: new Date(),
|
||||
}).freshness,
|
||||
).toBe("unknown");
|
||||
|
||||
// A run that reaches the account and produces no object records the walk
|
||||
// and leaves the success columns alone.
|
||||
const bucket = makeBucket();
|
||||
bucket.putStream = async () => {
|
||||
throw new Error("bucket refused the signature");
|
||||
};
|
||||
const report = await runOffhostBackup(prisma, bucket, new Date());
|
||||
expect(report.uploaded).toBe(0);
|
||||
expect(report.failed).toBe(1);
|
||||
|
||||
const row = await prisma.offhostBackupState.findUniqueOrThrow({
|
||||
where: { userId: "account-one" },
|
||||
});
|
||||
expect(row.lastSuccessAt).toBeNull();
|
||||
expect(row.sizeBytes).toBeNull();
|
||||
expect(
|
||||
classifyOffhostBackup({
|
||||
lastAttemptAt: row.lastAttemptAt,
|
||||
lastSuccessAt: row.lastSuccessAt,
|
||||
now: new Date(),
|
||||
}).freshness,
|
||||
).toBe("never");
|
||||
});
|
||||
|
||||
it("keeps the last good copy on the row when a later run fails", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
await seedUser("account-one");
|
||||
|
||||
const bucket = makeBucket();
|
||||
await runOffhostBackup(prisma, bucket, new Date());
|
||||
const good = await prisma.offhostBackupState.findUniqueOrThrow({
|
||||
where: { userId: "account-one" },
|
||||
});
|
||||
expect(good.lastSuccessAt).not.toBeNull();
|
||||
|
||||
bucket.putStream = async () => {
|
||||
throw new Error("bucket refused the signature");
|
||||
};
|
||||
await runOffhostBackup(prisma, bucket, new Date());
|
||||
|
||||
const after = await prisma.offhostBackupState.findUniqueOrThrow({
|
||||
where: { userId: "account-one" },
|
||||
});
|
||||
// The failed run moved the walk instant and nothing else: yesterday's
|
||||
// copy is still the copy this account has off-host.
|
||||
expect(after.lastSuccessAt?.getTime()).toBe(good.lastSuccessAt?.getTime());
|
||||
expect(after.sizeBytes).toBe(good.sizeBytes);
|
||||
expect(after.lastAttemptAt.getTime()).toBeGreaterThanOrEqual(
|
||||
good.lastAttemptAt.getTime(),
|
||||
);
|
||||
});
|
||||
|
||||
it("goes with the account it describes", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
await seedUser("account-one");
|
||||
await runOffhostBackup(prisma, makeBucket(), new Date());
|
||||
expect(await prisma.offhostBackupState.count()).toBe(1);
|
||||
|
||||
await prisma.user.delete({ where: { id: "account-one" } });
|
||||
expect(await prisma.offhostBackupState.count()).toBe(0);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user