fix(integrations): log a failed sync as a signal, not an audit row

The audit log is the trail of what a person or an admin did to an account.
Every failed provider sync wrote a row there, burying that trail, and kept
the provider's error message in plain text while the status row seals it.
A failed sync now emits an integrations.sync.failed warning with the same
counts and no message.
This commit is contained in:
Marc-André Bombeck committed 2026-10-20 10:03:30 +02:00
1 parent 537a73d3b2
commit 689796db33
6 files changed
+125 -90

No files matched your search

+18 -12
View File
@@ -1,18 +1,24 @@
# `attemptNumber` audit-row vs alert-signal semantics
# `attempt_number` signal vs alert semantics
Two numbers describe a failing integration sync, and they can diverge.
On-call operators reading an audit-log row alongside an admin alert
should expect the alert number to be smaller than (or equal to) the
audit number, never the other way around.
Operators reading an `integrations.sync.failed` log line alongside an admin
alert should expect the alert number to be smaller than (or equal to) the
logged number, never the other way around.
Since v1.43.7 a failed sync is a `warn` signal in the log store, not an
audit-log row: the audit log records what a person or an admin did to an
account, and a provider answering 503 is neither. The provider's message is
not on the signal; it stays encrypted on the integration's status row and
shows in the integration's settings.
## The two numbers
`integrations.sync.failed` audit rows write
`attemptNumber = transient + reauth_required + persistent` — the **sum**
`integrations.sync.failed` signals carry
`attempt_number = transient + reauth_required + persistent` — the **sum**
of all three failure buckets. This is the historical "consecutive
failures" running total carried by the legacy
`IntegrationStatus.consecutiveFailures` column (dropped in migration
`0077`, v1.4.47). Auditors looking at a single row see one integer
`0077`, v1.4.47). An operator looking at a single line sees one integer
that captures how many failures the bucket ledger has accumulated since
the last success.
@@ -27,14 +33,14 @@ single-bucket threshold even if its sum looks alarming.
## When the two diverge
A typed-failure-streak edge case: integration fails 2× transient, then
2× reauth_required, then 1× persistent. Audit row shows
`attemptNumber: 5`. Alert signal is `max(2, 2, 1) = 2`, which is below
the default threshold of 3, so no page fires. From the audit log this
2× reauth_required, then 1× persistent. The last signal shows
`attempt_number: 5`. The alert number is `max(2, 2, 1) = 2`, which is below
the default threshold of 3, so no page fires. From the log this
looks like a sustained 5-failure outage; from the alert dispatcher it
looks like three small bumps that never crossed a single-bucket
ceiling.
Both readings are correct for their purpose. The audit log records what
Both readings are correct for their purpose. The signal records what
happened end-to-end; the alert dispatcher only fires when a single
failure mode persists, because mixed-bucket streaks usually indicate a
transient network blip layered on top of an unrelated reauth prompt —
@@ -43,7 +49,7 @@ we want to wake someone up for.
## Runbook implication
If you see an audit row with `attemptNumber >= threshold` but no alert
If you see a signal with `attempt_number >= threshold` but no alert
was dispatched, check the per-bucket counts on
`IntegrationStatus.consecutiveFailuresByKind` (DB column
`consecutive_failures_by_kind`) for that user/integration pair. The