fix(insights): do not count a status run the bench sent nothing for

A status run whose whole chain was benched for background work sent no
request, yet the caller opened the note's negative window as for any
failure and raised its failure streak. The streak doubles the window up to
two hours, so a note kept waiting long after its provider recovered.

AllProvidersFailedError now says when no hop sent anything (sentNothing:
every provider benched, or refused by the operator cost cap). The status
chokepoint then holds the note for the first window without touching the
streak (holdStatusNoteWhileBenched), and the caller's own failure write
lands inside that window and neither moves nor counts it. A run that
reached a provider counts as before.

Refs #1279
This commit is contained in:
Marc-André Bombeck committed 2026-10-25 00:40:23 +02:00
1 parent b0ea3b1654
commit f1163d5efb
8 files changed
+225 -53

No files matched your search

@@ -111,6 +111,19 @@ describe("#1279: background work and a benched provider", () => {
]);
}
expect(gateway.calls).toBe(HARD_FAILURE_SKIP_THRESHOLD);
// The bench sent nothing; a hop that reached the provider did.
const benched = await walk(
"job",
"u-bench",
"openai-compatible",
gateway,
ledger,
).then(
() => null,
(e: unknown) => e,
);
expect(benched).toBeInstanceOf(AllProvidersFailedError);
expect((benched as AllProvidersFailedError).sentNothing).toBe(true);
// One line for the window, not one per job.
expect(signals("ai.chain.background_paused")).toHaveLength(1);
expect(signals("ai.chain.background_paused")[0].meta).toMatchObject({
@@ -139,6 +152,7 @@ describe("#1279: background work and a benched provider", () => {
expect((err as AllProvidersFailedError).attempts[0].failureKind).toBe(
"reasoning_only",
);
expect((err as AllProvidersFailedError).sentNothing).toBe(false);
});
it("walks the benched provider for a job-rationed call a person waits on", async () => {
+38 -38
View File
@@ -38,50 +38,36 @@ import {
const NEVER_SKIP: ReadonlySet<ProviderChainType> = new Set(["local"]);
/**
* v1.4.16 — multi-provider redundancy runner.
* The multi-provider redundancy runner: a provider completion with a
* hard-failure walk across an ordered list of providers, so one provider's
* outage (a 401 from an expired Codex sign-in, a 503 brown-out, a network
* reset) cascades to the next configured provider instead of failing the
* request.
*
* The chain runner wraps a provider completion with a hard-failure
* walkthrough across an ordered list of providers. The motivation is
* the maintainer's "ein Provider-Ausfall darf die Insights nicht zerschiessen"
* mandate: a 401 from Codex (OAuth expired), a 503 from OpenAI
* (upstream brown-out), or a network reset MUST cascade to the next
* configured provider rather than 422-ing the whole request.
* Hard failures (`isHardProviderFailure`) cascade. A malformed-JSON reply is
* NOT a hard failure — it is a prompt-following issue, so it bubbles to the
* caller's own validator rather than masking itself as provider
* unavailability and walking the chain. Every provider receives the same
* enriched system prompt; `outcome.workingProvider.providerType` attributes
* feedback per provider.
*
* Hard failures (`isHardProviderFailure`) cascade. A malformed-JSON
* reply is NOT a hard failure — it is a prompt-following issue, so it
* bubbles to the caller's own validator rather than masking itself as
* provider unavailability and walking the chain.
*
* Cross-feature coupling:
* - B5a's `MEDICAL_REFERENCES` is shared via the system prompt; each
* provider in the chain receives the same enriched context.
* - B5e (planned v1.4.17) reads `outcome.workingProvider.providerType`
* to attribute feedback per provider; that's why we re-export it
* out of the runner instead of the wrapper.
*
* The cache: a provider that worked on the last call gets first crack
* on the next, so a user whose Codex OAuth lapsed doesn't re-burn the
* 401 round-trip on every request for the next hour. Cache is
* in-process (per worker), keyed by `userId`, TTL 1h. No coordination
* across workers — when scaling out we'd move this to Redis, but for
* v1.4.16 the per-worker cache is enough to absorb the typical
* "polling reload" pattern.
*
* v1.18.7 — DECISION: the volatile per-worker hint is left
* in-process and NOT folded into the durable Postgres health ledger. The
* ledger already shares the load-bearing signal across workers (the
* skip-hint reorder in `resolveChainOrder` pushes a dead-credential /
* backoff provider to the chain tail for every worker), so the only thing
* The cache: a provider that worked on the last call gets first crack on the
* next, so a user whose Codex sign-in lapsed doesn't re-burn the 401
* round-trip on every request for the next hour. In-process (per worker),
* keyed by `userId`, TTL 1h. It stays out of the durable Postgres health
* ledger on purpose: the ledger already shares the load-bearing signal across
* workers (the skip-hint reorder in `resolveChainOrder`), so the only thing
* uncoordinated is a soft "tried this one last, put it first" reorder whose
* worst case is a single redundant first-hop per worker per hour — exactly
* the bound the v1.18.7 AI audit deemed acceptable at single-maintainer
* scale. Promoting the hint to a per-request ledger write would add write
* amplification on the hot path for a negligible saving; revisit only if a
* multi-worker deploy shows measurable redundant-hop cost.
* worst case is one redundant first hop per worker per hour. Promoting it to
* a per-request ledger write would add write amplification on the hot path;
* revisit only if a multi-worker deploy shows measurable redundant-hop cost.
*/
const LAST_WORKING_TTL_MS = 60 * 60 * 1000; // 1h
/** The `failureReason` of a hop the operator cost cap refused unsent. */
const OPERATOR_CAP_REFUSAL = "operator-cost-cap-exhausted";
interface CacheEntry {
providerType: ProviderChainType;
expiresAt: number;
@@ -223,6 +209,13 @@ export class AllProvidersFailedError extends Error {
* a model that reasons without answering is paid for every time.
*/
readonly tokensSpent: number;
/**
* #1279 — true when no hop sent a request: every provider was benched for
* background work or refused by the operator cost cap. Such a failure says
* nothing about what was asked, so a caller that backs off the record it
* failed on does not count it.
*/
readonly sentNothing: boolean;
constructor(attempts: FallbackHop[]) {
super(
@@ -236,6 +229,13 @@ export class AllProvidersFailedError extends Error {
(sum, a) => sum + (a.tokensSpent ?? 0),
0,
);
this.sentNothing =
attempts.length > 0 &&
attempts.every(
(a) =>
a.failureKind === "paused" ||
a.failureReason === OPERATOR_CAP_REFUSAL,
);
const first = attempts[0];
this.primaryCredentialExpired =
first !== undefined &&
@@ -631,7 +631,7 @@ async function runRawChain(
const hop: FallbackHop = {
providerType: candidate.providerType,
attempt: i + 1,
failureReason: "operator-cost-cap-exhausted",
failureReason: OPERATOR_CAP_REFUSAL,
httpStatus: null,
};
hops.push(hop);