The backup schedule case assumed the shared database had no queue schema,
but files that drive the queue create it and one leaves a finished backup
run behind, so the case failed whenever it ran after them. It now clears
that queue's history first. The delegate refusal case scanned the raw
response for a digit run that the random request id can contain; the id
is taken out before the scan.
Restoring a large account was one request: 94 s for 1.25 million readings.
A reverse proxy that cuts a request at 60 s (nginx's default) showed the
operator an error for a restore that went on to finish on the server.
The restore request now checks the typed confirmation, that the backup
exists and that its stored copy opens, admits one job for the account and
answers 202 with its id. The `backup-restore` queue runs the restore and
records its phase, counts and outcome on a `backup_restore_jobs` row
(migration 0347), which `GET /api/admin/backups/restores` and
`GET /api/admin/backups/restores/{id}` read back.
What a restore checks and writes is unchanged. The body moved from the
route into `restoreBackup` (src/lib/export/restore-backup.ts): every
refusal still happens before the first delete and the replacement is still
one transaction, so a failure part-way leaves the account as it was. While
it runs the account keeps showing its current data, and other writes are
neither refused nor held, as before.
- One restore per account: a partial unique index admits one queued or
running job, and a second request gets 409 `backup.restore.active` naming
the running job. A per-account advisory lock inside the transaction holds
even if a worker thought gone is still writing.
- Time budget: the job's expiry is two hours and a file whose transaction
limit would run past three quarters of it is refused before anything is
deleted (`jobDeadline`, beside `jobBudget`).
- Resuming: the job writes a heartbeat every two seconds. A boot sweep
queues a job whose worker stopped again from the start, at most twice,
then fails it as interrupted. The admission check releases an abandoned
job so it never holds the account's slot.
- The staged file is the stored `DataBackup` row, already encrypted at
rest. It stays whatever the outcome, so a retry needs no new upload. The
job records a digest of the copy and refuses one that was replaced while
it waited.
- Failures carry a stable code and a sentence that says what to do.
- The console follows the job: phase and counts while it runs, the outcome
after, and after a reload it finds the job again from the server. A poll
that fails on the network is never shown as a failed restore. A web
process separate from the worker evicts the account's caches when it
first reports the restore finished. The success line goes through the
outcome module, so an empty file is not drawn as a tick.
The model is registered in the backup plan (not carried) and the wipe plan
(wiped with the account); the queue in the maintenance registrar, the
subsystem surface and the job-fact allowlist; the new code in the error
catalogue; the routes in the OpenAPI contract. The tests that pin restore
semantics now drive the real route and the real job through one helper, and
`backup-restore-job.test.ts` pins the asynchronous contract itself.
The weekly pass can stop and leave no trace where anyone looks. The
handler's own error never runs when the job is killed rather than thrown;
`recordError()` increments an unnamed counter; and the admin status page's
failing-queue panel looks back 72 hours, which for a weekly queue means a
permanent failure is on screen three days in seven. The backups page
itself showed an ordinary row with an ordinary timestamp, because a copy
made six weeks ago reads exactly like Sunday's until somebody subtracts
the dates.
The listing now carries two facts that fail at different times: the age of
the newest scheduled copy, which never leaves any retention window, and
the terminal state of the last scheduled run, which is the only place the
reason is written down. `readLastQueueRun` takes the newest run rather
than a window, so a weekly queue's outcome stays readable until the next
one replaces it, and it unwraps the value pg-boss stores a thrown error
under, so the page says "job timed out" instead of raw JSON. The
failing-queue reader gets the same unwrap.
Manual uploads are excluded from the cadence judgement by the shape of the
input, so an ad-hoc upload cannot make a dead cron look alive, and a fresh
instance is not called overdue for having no copy yet.
The listing also stops reading every blob into the web process to measure
it; `octet_length` answers that in the database, which on an instance with
large records was its own way of running out of memory.
v1.4.15 Phase B3, criteria 1 + 2 (atomic — they share migration + retry-
policy types). Until this release the dispatcher fired channels best-
effort and kept retrying every 60s on a 410 Gone — that meant a stale
Web-Push subscription never recovered, a Telegram bot whose chat was
deleted got the same treatment, and the user had no visibility into
WHY their phone had stopped buzzing.
Reliability rules now:
- HARD reject (HTTP 410 web-push / ntfy, Telegram "chat not found" /
"blocked by the user") → channel auto-disabled, `disabled_reason`
captured, audit log `notification.channel.auto_disabled` written.
Sender NEVER retried until the user clicks "Re-enable" in Settings.
- SOFT reject (5xx, 429, network timeout) → exponential backoff:
[30s, 5min, 30min, 2h]. Counter `consecutive_failures` increments
per failure; after the 5th in-a-row the channel is auto-disabled
with reason `give_up_after_5_failures` (criterion 2's "treat as
hard reject after N failures" rule).
- COOLDOWN window (`next_retry_at > now`) → dispatcher skips the
channel without calling the sender — defeats the API-quota-burn
loop a flapping upstream would otherwise create.
Implementation:
- Migration 0028 adds 6 columns to `notification_channels`:
`disabled_reason`, `consecutive_failures`, `last_success_at`,
`last_failure_at`, `last_failure_reason`, `next_retry_at`. All
nullable / zero-default → no-op for existing rows.
- `src/lib/notifications/retry-policy.ts` exports the classifier
(`classifyTelegramError`, `classifyHttpStatus`) plus
`BACKOFF_SCHEDULE_MS` (frozen) and `nextRetryAt(count, now)` /
`shouldAutoDisableAfterTransient(count)` helpers.
- `src/lib/notifications/channel-state.ts` is the only writer of
the new columns: `recordChannelSuccess`, `recordChannelHardReject`,
`recordChannelTransientFailure`, `reEnableChannel`,
`isChannelInCooldown`. Each transient/hard path writes an audit
entry so the user can trace WHY the channel went silent.
- Senders now return a structured `SendOutcome` (`{ ok, hardReject?,
statusCode?, reason?, message? }`) instead of bare booleans. The
Telegram sender additionally exposes `errorDescription` from the
Bot API so "chat not found" / "bot was blocked" can be classified
without parsing free-form text in the dispatcher.
- Dispatcher: cooldown-skip → success → hard-reject branch →
transient/give-up branch. Sender-throws are absorbed as transient
failures (defence-in-depth).
Tests: +20 unit tests (`retry-policy.test.ts`, `channel-state.test.ts`,
`dispatcher.test.ts`). 883/883 pass. Hard-reject scenarios covered:
web-push 410, Telegram chat-not-found, ntfy 410. Soft-reject
scenarios: 429 schedules backoff without disabling; cooldown skip
prevents repeat call; 5th in-a-row triggers give-up + audit.
Pre-existing bug fix: `src/app/api/settings/telegram/test/route.ts`
had `if (!ok)` against a `SendMessageResult` object — always truthy,
so the route never reported test-send failures. Now reads
`result.ok`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Marc-André Bombeck