mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-09 21:10:59 +00:00
Merge branch 'fix/v1437-r2-jobs' into release/v1.43.7
This commit is contained in:
commit
e5b7eebe64
62 files changed
+1745
-774
No files matched your search
@@ -1,18 +1,24 @@
|
||||
# `attemptNumber` audit-row vs alert-signal semantics
|
||||
# `attempt_number` signal vs alert semantics
|
||||
|
||||
Two numbers describe a failing integration sync, and they can diverge.
|
||||
On-call operators reading an audit-log row alongside an admin alert
|
||||
should expect the alert number to be smaller than (or equal to) the
|
||||
audit number, never the other way around.
|
||||
Operators reading an `integrations.sync.failed` log line alongside an admin
|
||||
alert should expect the alert number to be smaller than (or equal to) the
|
||||
logged number, never the other way around.
|
||||
|
||||
Since v1.43.7 a failed sync is a `warn` signal in the log store, not an
|
||||
audit-log row: the audit log records what a person or an admin did to an
|
||||
account, and a provider answering 503 is neither. The provider's message is
|
||||
not on the signal; it stays encrypted on the integration's status row and
|
||||
shows in the integration's settings.
|
||||
|
||||
## The two numbers
|
||||
|
||||
`integrations.sync.failed` audit rows write
|
||||
`attemptNumber = transient + reauth_required + persistent` — the **sum**
|
||||
`integrations.sync.failed` signals carry
|
||||
`attempt_number = transient + reauth_required + persistent` — the **sum**
|
||||
of all three failure buckets. This is the historical "consecutive
|
||||
failures" running total carried by the legacy
|
||||
`IntegrationStatus.consecutiveFailures` column (dropped in migration
|
||||
`0077`, v1.4.47). Auditors looking at a single row see one integer
|
||||
`0077`, v1.4.47). An operator looking at a single line sees one integer
|
||||
that captures how many failures the bucket ledger has accumulated since
|
||||
the last success.
|
||||
|
||||
@@ -27,14 +33,14 @@ single-bucket threshold even if its sum looks alarming.
|
||||
## When the two diverge
|
||||
|
||||
A typed-failure-streak edge case: integration fails 2× transient, then
|
||||
2× reauth_required, then 1× persistent. Audit row shows
|
||||
`attemptNumber: 5`. Alert signal is `max(2, 2, 1) = 2`, which is below
|
||||
the default threshold of 3, so no page fires. From the audit log this
|
||||
2× reauth_required, then 1× persistent. The last signal shows
|
||||
`attempt_number: 5`. The alert number is `max(2, 2, 1) = 2`, which is below
|
||||
the default threshold of 3, so no page fires. From the log this
|
||||
looks like a sustained 5-failure outage; from the alert dispatcher it
|
||||
looks like three small bumps that never crossed a single-bucket
|
||||
ceiling.
|
||||
|
||||
Both readings are correct for their purpose. The audit log records what
|
||||
Both readings are correct for their purpose. The signal records what
|
||||
happened end-to-end; the alert dispatcher only fires when a single
|
||||
failure mode persists, because mixed-bucket streaks usually indicate a
|
||||
transient network blip layered on top of an unrelated reauth prompt —
|
||||
@@ -43,7 +49,7 @@ we want to wake someone up for.
|
||||
|
||||
## Runbook implication
|
||||
|
||||
If you see an audit row with `attemptNumber >= threshold` but no alert
|
||||
If you see a signal with `attempt_number >= threshold` but no alert
|
||||
was dispatched, check the per-bucket counts on
|
||||
`IntegrationStatus.consecutiveFailuresByKind` (DB column
|
||||
`consecutive_failures_by_kind`) for that user/integration pair. The
|
||||
|
||||
@@ -137,8 +137,8 @@ it.
|
||||
## Reading the result
|
||||
|
||||
The run's wide event `job.measurement_maintenance` carries
|
||||
`maintenance_outcome` (`completed`, `nothing_due`, `refused_purge_running`,
|
||||
`failed`), the live and dead rows the run started from, the
|
||||
`maintenance_outcome` (`completed`, `nothing_due` for the nightly check,
|
||||
`refused_purge_running`, `failed`), the live and dead rows the run started from, the
|
||||
table and index sizes before and after, and `maintenance_steps`, one entry
|
||||
per step with its target, size before and after, and duration. The same
|
||||
sizes from `psql`:
|
||||
@@ -151,17 +151,22 @@ SELECT pg_size_pretty(pg_total_relation_size('measurements')) AS total,
|
||||
## The nightly check
|
||||
|
||||
Since v1.43.7 the same queue also runs by itself at 03:40 (Europe/Berlin).
|
||||
That run names no steps: it reads the table's live and dead rows and runs a
|
||||
plain `VACUUM (ANALYZE)` only when there are at least 10,000 dead rows and
|
||||
more than one for every ten live ones. It never rebuilds an index, and a
|
||||
plain vacuum stops neither reads nor writes. It shares the singleton key with
|
||||
the run you start, so the two never overlap. The admin card keeps showing
|
||||
only the runs you started.
|
||||
That run names no steps and changes nothing: vacuuming the table is
|
||||
autovacuum's job, which migration 0412 starts at 5 % dead rows instead of
|
||||
Postgres's 20 %. The check reads the table's live and dead rows and warns
|
||||
(`maintenance_autovacuum_behind` on the wide event) when there are at least
|
||||
10,000 dead rows and more than one for every five live ones, four times the
|
||||
point at which autovacuum should already have run. That means autovacuum is
|
||||
not keeping up: look for a transaction left open for hours
|
||||
(`pg_stat_activity`, `state = 'idle in transaction'`) or for autovacuum
|
||||
workers busy elsewhere. The check never rebuilds an index. It shares the
|
||||
singleton key with the run you start, so the two never overlap. The admin
|
||||
card keeps showing only the runs you started.
|
||||
|
||||
The check also measures how far the table file has grown past what its live
|
||||
rows need (`maintenance_bloat_factor` on the wide event, `bloat_factor` in the
|
||||
job result). Past 3 it adds a warning to the event: a plain vacuum makes the
|
||||
space reusable but does not shrink the file, and only `VACUUM FULL` does (see
|
||||
job result). Past 3 it adds a warning to the event: a vacuum makes the space
|
||||
reusable but does not shrink the file, and only `VACUUM FULL` does (see
|
||||
below).
|
||||
|
||||
Why the table grew in the first place: until v1.43.7 an update that changed
|
||||
|
||||
Reference in new issue
Block a user