Merge branch 'fix/v1437-r2-jobs' into release/v1.43.7

This commit is contained in:
Marc-André Bombeck committed 2026-10-20 10:17:01 +02:00
commit e5b7eebe64
62 files changed
+1745 -774

No files matched your search

+18 -12
View File
@@ -1,18 +1,24 @@
# `attemptNumber` audit-row vs alert-signal semantics
# `attempt_number` signal vs alert semantics
Two numbers describe a failing integration sync, and they can diverge.
On-call operators reading an audit-log row alongside an admin alert
should expect the alert number to be smaller than (or equal to) the
audit number, never the other way around.
Operators reading an `integrations.sync.failed` log line alongside an admin
alert should expect the alert number to be smaller than (or equal to) the
logged number, never the other way around.
Since v1.43.7 a failed sync is a `warn` signal in the log store, not an
audit-log row: the audit log records what a person or an admin did to an
account, and a provider answering 503 is neither. The provider's message is
not on the signal; it stays encrypted on the integration's status row and
shows in the integration's settings.
## The two numbers
`integrations.sync.failed` audit rows write
`attemptNumber = transient + reauth_required + persistent` — the **sum**
`integrations.sync.failed` signals carry
`attempt_number = transient + reauth_required + persistent` — the **sum**
of all three failure buckets. This is the historical "consecutive
failures" running total carried by the legacy
`IntegrationStatus.consecutiveFailures` column (dropped in migration
`0077`, v1.4.47). Auditors looking at a single row see one integer
`0077`, v1.4.47). An operator looking at a single line sees one integer
that captures how many failures the bucket ledger has accumulated since
the last success.
@@ -27,14 +33,14 @@ single-bucket threshold even if its sum looks alarming.
## When the two diverge
A typed-failure-streak edge case: integration fails 2× transient, then
2× reauth_required, then 1× persistent. Audit row shows
`attemptNumber: 5`. Alert signal is `max(2, 2, 1) = 2`, which is below
the default threshold of 3, so no page fires. From the audit log this
2× reauth_required, then 1× persistent. The last signal shows
`attempt_number: 5`. The alert number is `max(2, 2, 1) = 2`, which is below
the default threshold of 3, so no page fires. From the log this
looks like a sustained 5-failure outage; from the alert dispatcher it
looks like three small bumps that never crossed a single-bucket
ceiling.
Both readings are correct for their purpose. The audit log records what
Both readings are correct for their purpose. The signal records what
happened end-to-end; the alert dispatcher only fires when a single
failure mode persists, because mixed-bucket streaks usually indicate a
transient network blip layered on top of an unrelated reauth prompt —
@@ -43,7 +49,7 @@ we want to wake someone up for.
## Runbook implication
If you see an audit row with `attemptNumber >= threshold` but no alert
If you see a signal with `attempt_number >= threshold` but no alert
was dispatched, check the per-bucket counts on
`IntegrationStatus.consecutiveFailuresByKind` (DB column
`consecutive_failures_by_kind`) for that user/integration pair. The
+15 -10
View File
@@ -137,8 +137,8 @@ it.
## Reading the result
The run's wide event `job.measurement_maintenance` carries
`maintenance_outcome` (`completed`, `nothing_due`, `refused_purge_running`,
`failed`), the live and dead rows the run started from, the
`maintenance_outcome` (`completed`, `nothing_due` for the nightly check,
`refused_purge_running`, `failed`), the live and dead rows the run started from, the
table and index sizes before and after, and `maintenance_steps`, one entry
per step with its target, size before and after, and duration. The same
sizes from `psql`:
@@ -151,17 +151,22 @@ SELECT pg_size_pretty(pg_total_relation_size('measurements')) AS total,
## The nightly check
Since v1.43.7 the same queue also runs by itself at 03:40 (Europe/Berlin).
That run names no steps: it reads the table's live and dead rows and runs a
plain `VACUUM (ANALYZE)` only when there are at least 10,000 dead rows and
more than one for every ten live ones. It never rebuilds an index, and a
plain vacuum stops neither reads nor writes. It shares the singleton key with
the run you start, so the two never overlap. The admin card keeps showing
only the runs you started.
That run names no steps and changes nothing: vacuuming the table is
autovacuum's job, which migration 0412 starts at 5 % dead rows instead of
Postgres's 20 %. The check reads the table's live and dead rows and warns
(`maintenance_autovacuum_behind` on the wide event) when there are at least
10,000 dead rows and more than one for every five live ones, four times the
point at which autovacuum should already have run. That means autovacuum is
not keeping up: look for a transaction left open for hours
(`pg_stat_activity`, `state = 'idle in transaction'`) or for autovacuum
workers busy elsewhere. The check never rebuilds an index. It shares the
singleton key with the run you start, so the two never overlap. The admin
card keeps showing only the runs you started.
The check also measures how far the table file has grown past what its live
rows need (`maintenance_bloat_factor` on the wide event, `bloat_factor` in the
job result). Past 3 it adds a warning to the event: a plain vacuum makes the
space reusable but does not shrink the file, and only `VACUUM FULL` does (see
job result). Past 3 it adds a warning to the event: a vacuum makes the space
reusable but does not shrink the file, and only `VACUUM FULL` does (see
below).
Why the table grew in the first place: until v1.43.7 an update that changed