The ledger stamped last_failure_at, last_ok_at, next_retry_at and
updated_at with the database's NOW(), while every reader compares them
with the app's clock: the runner's skip hints and the background bench,
the admin provider-health card, the server-health tri-state and the
support report. A backoff or pause therefore sat wherever the skew
between the two clocks put it. With the app clock pinned, a five-minute
backoff lasted until the database's clock plus five minutes, months away,
or was over before it began.
Both writes now bind the instant they took as a parameter
(`$N::timestamp`, the naive UTC columns), the same one-clock rule the
summary reads follow. The escalating pause arithmetic is unchanged.
The pause integration test pins the app clock years behind and ahead of
the database's and checks the stored instants, a backoff that lifts after
five minutes, a pause that benches background work for its full length,
and a success the server health reads as recent.
Refs #1279
The nightly briefing warm completed every night and read "all-failed",
because one run event held every account's annotations and the last
account won. Each candidate now runs in its own wide event and the run
reports every outcome through `reportJobRun`, so ok, dead sign-in,
withheld and failed accounts show side by side.
A dead Codex refresh token threw a bare error with no status, so the chain
read it as a network failure: the health ledger recorded hard_failed, no
"reconnect" hint appeared, and every night tried the same token. The
refresh now throws `CodexRefreshError` with a 401 for a dead grant
(invalid_grant / invalid_refresh_token on a 400 included), as do a missing
or undecryptable stored credential. The ledger records auth_failed, the
chain marks the primary credential expired, and the nightly pass reports
the account as auth_failed without queuing a retry.
A briefing the outbound screen withheld (a causal claim) failed outright,
and the 45-minute provider retry then asked the same question eight times
a night. The generator now makes one bounded repair that names the broken
rule and keeps the repaired payload only if it passes the same parse,
grounding and screen; a withheld outcome queues no retry.
A provider that keeps failing is paused: from five consecutive hard
failures the skip window starts at 30 minutes and doubles per further
failure, capped at a day. A paused provider is walked last and tried again
when the window lifts; one success resumes it. The operator card shows how
many accounts it is paused for and until when. After a 5xx from a
non-canonical OpenAI-compatible endpoint the client asks the free model
listing, and a configured model the endpoint no longer offers is named in
the error, on the hop line and on the card.
Also:
- The tiered context's live day band read `measured_at AT TIME ZONE tz`,
which treats the stored UTC as local time; a far-from-UTC reader's
evening readings landed on the next day. It converts from UTC first now.
- Google Health intraday heart rate re-reads two hours per sync instead of
a day; daily summaries and sleep keep the day (#1023).
- The first filtered daily read that answers 400 and triggers the
snake_case retry is recorded as a note, not an error (#1023).
- The single derived-score read goes through the derived cache with
stale-while-revalidate, like the batch beside it.
Marc-André Bombeck