fix(ai): name the real cause of failed briefings and stop retrying what cannot heal

The nightly briefing warm completed every night and read "all-failed",
because one run event held every account's annotations and the last
account won. Each candidate now runs in its own wide event and the run
reports every outcome through `reportJobRun`, so ok, dead sign-in,
withheld and failed accounts show side by side.

A dead Codex refresh token threw a bare error with no status, so the chain
read it as a network failure: the health ledger recorded hard_failed, no
"reconnect" hint appeared, and every night tried the same token. The
refresh now throws `CodexRefreshError` with a 401 for a dead grant
(invalid_grant / invalid_refresh_token on a 400 included), as do a missing
or undecryptable stored credential. The ledger records auth_failed, the
chain marks the primary credential expired, and the nightly pass reports
the account as auth_failed without queuing a retry.

A briefing the outbound screen withheld (a causal claim) failed outright,
and the 45-minute provider retry then asked the same question eight times
a night. The generator now makes one bounded repair that names the broken
rule and keeps the repaired payload only if it passes the same parse,
grounding and screen; a withheld outcome queues no retry.

A provider that keeps failing is paused: from five consecutive hard
failures the skip window starts at 30 minutes and doubles per further
failure, capped at a day. A paused provider is walked last and tried again
when the window lifts; one success resumes it. The operator card shows how
many accounts it is paused for and until when. After a 5xx from a
non-canonical OpenAI-compatible endpoint the client asks the free model
listing, and a configured model the endpoint no longer offers is named in
the error, on the hop line and on the card.

Also:
- The tiered context's live day band read `measured_at AT TIME ZONE tz`,
  which treats the stored UTC as local time; a far-from-UTC reader's
  evening readings landed on the next day. It converts from UTC first now.
- Google Health intraday heart rate re-reads two hours per sync instead of
  a day; daily summaries and sleep keep the day (#1023).
- The first filtered daily read that answers 400 and triggers the
  snake_case retry is recorded as a note, not an error (#1023).
- The single derived-score read goes through the derived cache with
  stale-while-revalidate, like the batch beside it.
This commit is contained in:
Marc-André Bombeck committed 2026-10-08 09:06:09 +02:00
1 parent 2210f823d6
commit 27342ee734
34 files changed
+2309 -261

No files matched your search

@@ -48,6 +48,14 @@ interface ProviderHealthRow {
lastOkAt: string | null;
lastFailureAt: string | null;
lastFailureStatus: number | null;
/** v1.42 — users this provider is paused for, and when the last lifts. */
pausedUsers?: number;
pausedUntil?: string | null;
/** v1.42 — the operator key's model check, while it is failing. */
modelListing?: {
model: string;
listing: "listed" | "not_listed" | "unknown";
} | null;
}
export function ProviderHealthSection() {
@@ -151,11 +159,33 @@ export function ProviderHealthSection() {
})}`
: ""}
</span>
{p.pausedUsers && p.pausedUntil ? (
<span
className="text-muted-foreground text-xs"
data-slot="provider-paused"
>
{t("admin.providerHealth.pausedHint", {
count: p.pausedUsers,
at: formatDateTime(p.pausedUntil),
})}
</span>
) : null}
{p.modelListing?.listing === "not_listed" ? (
<span className="text-sm" data-slot="provider-model-missing">
{t("admin.providerHealth.modelNotServed", {
model: p.modelListing.model,
})}
</span>
) : null}
</div>
{p.failing === 0 ? (
<Badge variant="secondary">
{t("admin.providerHealth.statusHealthy")}
</Badge>
) : p.pausedUsers ? (
<Badge variant="destructive">
{t("admin.providerHealth.statusPaused")}
</Badge>
) : (
<Badge variant="destructive">
{t("admin.providerHealth.statusFailing", {