The nightly briefing warm completed every night and read "all-failed",
because one run event held every account's annotations and the last
account won. Each candidate now runs in its own wide event and the run
reports every outcome through `reportJobRun`, so ok, dead sign-in,
withheld and failed accounts show side by side.
A dead Codex refresh token threw a bare error with no status, so the chain
read it as a network failure: the health ledger recorded hard_failed, no
"reconnect" hint appeared, and every night tried the same token. The
refresh now throws `CodexRefreshError` with a 401 for a dead grant
(invalid_grant / invalid_refresh_token on a 400 included), as do a missing
or undecryptable stored credential. The ledger records auth_failed, the
chain marks the primary credential expired, and the nightly pass reports
the account as auth_failed without queuing a retry.
A briefing the outbound screen withheld (a causal claim) failed outright,
and the 45-minute provider retry then asked the same question eight times
a night. The generator now makes one bounded repair that names the broken
rule and keeps the repaired payload only if it passes the same parse,
grounding and screen; a withheld outcome queues no retry.
A provider that keeps failing is paused: from five consecutive hard
failures the skip window starts at 30 minutes and doubles per further
failure, capped at a day. A paused provider is walked last and tried again
when the window lifts; one success resumes it. The operator card shows how
many accounts it is paused for and until when. After a 5xx from a
non-canonical OpenAI-compatible endpoint the client asks the free model
listing, and a configured model the endpoint no longer offers is named in
the error, on the hop line and on the card.
Also:
- The tiered context's live day band read `measured_at AT TIME ZONE tz`,
which treats the stored UTC as local time; a far-from-UTC reader's
evening readings landed on the next day. It converts from UTC first now.
- Google Health intraday heart rate re-reads two hours per sync instead of
a day; daily summaries and sleep keep the day (#1023).
- The first filtered daily read that answers 400 and triggers the
snake_case retry is recorded as a note, not an error (#1023).
- The single derived-score read goes through the derived cache with
stale-while-revalidate, like the batch beside it.
The list of provider tags that send data out through a credential the
operator holds (admin-openai, admin-codex) existed four times: in the
consent gate, in the capability resolver, in the shared-provider health
read and in the admin provider-health ordering. A tag added to one and not
the others would be consented for in one place and not the other. The list
and the local-provider test now live in src/lib/ai/provider-egress.ts and
every reader imports them.
The guard meant to keep one copy matched a variable name, so the copy in
the resolver, under another name, never tripped it. It now matches the
list itself in either order and names the one file allowed to hold it.
CLAUDE.md forbids planning shorthand in committed artefacts, and this branch
introduced 74 lines of it across source, tests, a schema field and two
migrations. Someone reading the migration SQL in a year has no way to resolve a
wave letter or a round number, and a comment that points at something outside
the repository names no reason at all.
Three passes. A version followed by a parenthetical keeps the version and loses
the parenthetical, because the version says since when something holds. A
sentence that leaned on the vocabulary to carry its meaning is rewritten to
state the reason directly: the arrival reservation now says it was refused over
money the operator never spent, rather than naming the defect a wave existed to
remove. Three test titles carried the marker in the name itself and are renamed
to what they pin.
Comments only. No behaviour, no renames beyond those three titles.
The provider-health endpoint showed delivery health only. When a surface
refuses on budget the operator's next question is which ceiling he hit,
and one mixed token figure cannot answer it: a day at 1.24 M tokens looks
alarming until you see that 151 200 of them ran on the instance's own key
and the rest on the users' plans.
`spendToday` carries the day's key and both counters, aggregated across
accounts. The operator ceiling is enforced against `operatorTokens`, so
that is the figure the readout puts beside the total.
The provider-health ledger has recorded the HTTP status of every failure
since v1.11.0 and nothing has ever read it. The operator card could say
three of five chains were failing without saying whose problem that was,
which is the one thing the operator needs: a 401 is a credential only they
can replace, a 429 or a 503 is the provider's afternoon and waiting is the
right move.
The fold carries the status of the failure whose instant the row already
shows, not an aggregate over the column. A MAX of two HTTP statuses is
arithmetic on a label, and a status describing a different failure than
the one on screen would send the operator to the wrong place. The column
joins the grouping key instead, and the fold picks the group belonging to
the newest failure.
Blank where there was nothing to record: a connection that never completed
has no status, and a chain that has since recovered had its status cleared
along with the rest of its failure state.
The per-user retry ledger has recorded every AI delivery outcome for a
long time, but no surface ever read it for the operator: a central
provider could fail on every call for weeks and the admin console said
nothing. The AI settings section now carries a health card, backed by a
new admin-only endpoint that folds the ledger into one row per provider
type: users tracked, users currently failing, the worst uninterrupted
failure run, and the last success and failure instants. Operator-managed
types sort first because their failure takes the shared fallback down
for everyone whose chain relies on it. Counts and timestamps only; the
response never names a user.