fix(notifications): auto-disable channels on persistent hard rejects (410, etc.)

v1.4.15 Phase B3, criteria 1 + 2 (atomic — they share migration + retry-
policy types). Until this release the dispatcher fired channels best-
effort and kept retrying every 60s on a 410 Gone — that meant a stale
Web-Push subscription never recovered, a Telegram bot whose chat was
deleted got the same treatment, and the user had no visibility into
WHY their phone had stopped buzzing.

Reliability rules now:
- HARD reject (HTTP 410 web-push / ntfy, Telegram "chat not found" /
  "blocked by the user") → channel auto-disabled, disabled_reason
  captured, audit log notification.channel.auto_disabled written.
  Sender NEVER retried until the user clicks "Re-enable" in Settings.
- SOFT reject (5xx, 429, network timeout) → exponential backoff:
  [30s, 5min, 30min, 2h]. Counter consecutive_failures increments
  per failure; after the 5th in-a-row the channel is auto-disabled
  with reason give_up_after_5_failures.
- COOLDOWN window (next_retry_at > now) → dispatcher skips the
  channel without calling the sender — defeats the API-quota-burn
  loop a flapping upstream would otherwise create.

Implementation:
- Migration 0028 adds 6 columns to notification_channels (all
  nullable / zero-default → no-op for existing rows).
- src/lib/notifications/retry-policy.ts: classifiers + frozen
  BACKOFF_SCHEDULE_MS + nextRetryAt() helpers.
- src/lib/notifications/channel-state.ts: only writer of the new
  columns (recordChannelSuccess, recordChannelHardReject,
  recordChannelTransientFailure, reEnableChannel,
  isChannelInCooldown). Each transient/hard path writes audit.
- Senders return a structured SendOutcome instead of bare booleans.
- Dispatcher: cooldown-skip → success → hard-reject → transient/
  give-up. Sender-throws absorbed as transient failures.

Tests: +20 unit tests (retry-policy.test.ts, channel-state.test.ts,
dispatcher.test.ts). 883/883 unit tests pass; typecheck clean; lint
clean (no new warnings).

Pre-existing bug fix: src/app/api/settings/telegram/test/route.ts
had if (!ok) against a SendMessageResult object — always truthy,
so the route never reported test-send failures. Now reads result.ok.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Marc-André BombeckandClaude Opus 4.7 committed 2026-05-09 21:01:45 +02:00
1 parent 080545248e
commit 87a40fd0a4
15 files changed
+1187 -35

No files matched your search

@@ -0,0 +1,34 @@
-- v1.4.15 Phase B3: notification reliability metadata.
--
-- Until this release the dispatcher fired channels best-effort and the
-- only feedback signal we kept was a Wide-Event warning per failure.
-- That meant a stale Web-Push subscription kept burning bandwidth on
-- every dispatch (a 410 from FCM is a permanent reject — re-trying it
-- never recovers), a Telegram bot whose chat was deleted got the same
-- treatment, and the user never saw the channel's actual health.
--
-- These columns let the dispatcher:
-- 1. Auto-disable a channel on a hard reject (HTTP 410, Telegram
-- "chat not found" or "blocked by the user", ntfy 410), with the
-- reason captured for the Settings UI.
-- 2. Apply exponential backoff [30s, 5min, 30min, 2h] to soft errors
-- (5xx, 429, network timeout) instead of hammering a flapping
-- upstream — and after 5 consecutive transient failures, treat
-- the channel as effectively dead and auto-disable it too.
-- 3. Surface "Active / Auto-disabled / Sending paused" + last
-- success / failure timestamps in Settings → Notifications, with
-- a "Re-enable" button that clears `disabled_reason` +
-- `consecutive_failures` and a "Send test" button.
--
-- All columns are nullable / zero-default so the migration is a no-op
-- for existing rows: a healthy channel reports
-- consecutive_failures=0, last_success_at=null, disabled_reason=null,
-- which renders as "Active (no recent send recorded)" in the UI.
ALTER TABLE "notification_channels"
ADD COLUMN "disabled_reason" TEXT,
ADD COLUMN "consecutive_failures" INTEGER NOT NULL DEFAULT 0,
ADD COLUMN "last_success_at" TIMESTAMP(3),
ADD COLUMN "last_failure_at" TIMESTAMP(3),
ADD COLUMN "last_failure_reason" TEXT,
ADD COLUMN "next_retry_at" TIMESTAMP(3);
+58
View File
@@ -106,6 +106,7 @@ model User {
feedback Feedback[]
idempotencyKeys IdempotencyKey[]
devices Device[]
integrationStatuses IntegrationStatus[]
@@map("users")
}
@@ -444,6 +445,46 @@ model MoodEntry {
@@map("mood_entries")
}
// ─── Integration Status (v1.4.15) ─────────────────────────
//
// Per-(user, integration) row that captures the most recent sync attempt
// outcome so the Settings → Integrations UI can show:
// - state ("connected", "error_reauth", "error_transient", "disconnected")
// - last successful sync timestamp
// - last attempted sync timestamp
// - last error message (encrypted at rest if non-null — see crypto.ts)
// - consecutive failure counter (resets on success)
//
// `recordSyncSuccess` / `recordSyncFailure` helpers in
// `src/lib/integrations/status.ts` are the only writers. The
// `consecutiveFailures` counter feeds the admin-Telegram alerting
// threshold (default 3). State `error_reauth` parks scheduled syncs
// until the user reconnects (the sync entry-points consult this row
// before doing any external work).
//
// `integration` values are bare strings ("withings", "moodlog") rather
// than an enum so future integrations don't require a migration.
model IntegrationStatus {
id String @id @default(cuid())
userId String @map("user_id")
integration String // "withings" | "moodlog"
state String @default("connected") // "connected" | "error_reauth" | "error_transient" | "disconnected"
lastSuccessAt DateTime? @map("last_success_at")
lastAttemptAt DateTime? @map("last_attempt_at")
lastError String? @map("last_error") // AES-256-GCM encrypted (may contain token-fragment metadata)
consecutiveFailures Int @default(0) @map("consecutive_failures")
alertedAt DateTime? @map("alerted_at") // last time admin Telegram alert fired (idempotency guard)
createdAt DateTime @default(now()) @map("created_at")
updatedAt DateTime @updatedAt @map("updated_at")
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
@@unique([userId, integration])
@@index([userId])
@@index([state])
@@map("integration_statuses")
}
// ─── App Settings ─────────────────────────────────────────
model AppSettings {
@@ -559,6 +600,23 @@ model NotificationChannel {
createdAt DateTime @default(now()) @map("created_at")
updatedAt DateTime @updatedAt @map("updated_at")
// ── v1.4.15 Phase B3: reliability fields ─────────────────────
// Reason the channel was auto-disabled (e.g. "hard_reject_410",
// "telegram_blocked_by_user", "telegram_chat_not_found",
// "give_up_after_5_failures"). NULL when the channel is healthy
// OR was disabled manually (in which case `enabled=false` alone
// is enough — no automated process should treat that as
// re-enable-able).
disabledReason String? @map("disabled_reason")
consecutiveFailures Int @default(0) @map("consecutive_failures")
lastSuccessAt DateTime? @map("last_success_at")
lastFailureAt DateTime? @map("last_failure_at")
lastFailureReason String? @map("last_failure_reason")
// The next time the dispatcher is allowed to attempt a send for
// this channel after a transient failure. NULL = no cooldown.
// Hard rejects flip `enabled=false` instead of writing this.
nextRetryAt DateTime? @map("next_retry_at")
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
preferences NotificationPreference[]