mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 20:42:43 +00:00
Marc-André Bombeck
27342ee734
The nightly briefing warm completed every night and read "all-failed", because one run event held every account's annotations and the last account won. Each candidate now runs in its own wide event and the run reports every outcome through `reportJobRun`, so ok, dead sign-in, withheld and failed accounts show side by side. A dead Codex refresh token threw a bare error with no status, so the chain read it as a network failure: the health ledger recorded hard_failed, no "reconnect" hint appeared, and every night tried the same token. The refresh now throws `CodexRefreshError` with a 401 for a dead grant (invalid_grant / invalid_refresh_token on a 400 included), as do a missing or undecryptable stored credential. The ledger records auth_failed, the chain marks the primary credential expired, and the nightly pass reports the account as auth_failed without queuing a retry. A briefing the outbound screen withheld (a causal claim) failed outright, and the 45-minute provider retry then asked the same question eight times a night. The generator now makes one bounded repair that names the broken rule and keeps the repaired payload only if it passes the same parse, grounding and screen; a withheld outcome queues no retry. A provider that keeps failing is paused: from five consecutive hard failures the skip window starts at 30 minutes and doubles per further failure, capped at a day. A paused provider is walked last and tried again when the window lifts; one success resumes it. The operator card shows how many accounts it is paused for and until when. After a 5xx from a non-canonical OpenAI-compatible endpoint the client asks the free model listing, and a configured model the endpoint no longer offers is named in the error, on the hop line and on the card. Also: - The tiered context's live day band read `measured_at AT TIME ZONE tz`, which treats the stored UTC as local time; a far-from-UTC reader's evening readings landed on the next day. It converts from UTC first now. - Google Health intraday heart rate re-reads two hours per sync instead of a day; daily summaries and sleep keep the day (#1023). - The first filtered daily read that answers 400 and triggers the snake_case retry is recorded as a note, not an error (#1023). - The single derived-score read goes through the derived cache with stale-while-revalidate, like the batch beside it.
81 lines
2.7 KiB
TypeScript
81 lines
2.7 KiB
TypeScript
/**
|
|
* v1.42 — the escalating provider pause, against the real upsert.
|
|
*
|
|
* The pause is computed inside the one `INSERT … ON CONFLICT DO UPDATE` that
|
|
* records a failure, from the count that same statement writes, so two
|
|
* workers failing at once still arm the pause their combined count deserves.
|
|
* Only Postgres can show the CASE arithmetic is right; the unit test can only
|
|
* show the statement carries it.
|
|
*/
|
|
import { beforeEach, describe, expect, it, vi } from "vitest";
|
|
|
|
import { getPrismaClient, truncateAllTables } from "./setup";
|
|
|
|
vi.mock("@/lib/db-compat", () => ({
|
|
ensureDbCompatibility: vi.fn().mockResolvedValue(undefined),
|
|
}));
|
|
|
|
import {
|
|
HARD_FAILURE_COOLDOWN_MS,
|
|
PROVIDER_PAUSE_BASE_MS,
|
|
PROVIDER_PAUSE_THRESHOLD,
|
|
postgresProviderHealthLedger as ledger,
|
|
} from "@/lib/ai/provider-health-ledger";
|
|
|
|
const prisma = getPrismaClient();
|
|
|
|
async function windowMs(userId: string): Promise<number> {
|
|
const row = await prisma.providerHealth.findFirstOrThrow({
|
|
where: { userId, providerType: "admin-openai" },
|
|
select: { nextRetryAt: true, lastFailureAt: true },
|
|
});
|
|
return row.nextRetryAt!.getTime() - row.lastFailureAt!.getTime();
|
|
}
|
|
|
|
describe("provider pause in the Postgres ledger", () => {
|
|
let userId: string;
|
|
|
|
beforeEach(async () => {
|
|
await truncateAllTables(prisma);
|
|
const user = await prisma.user.create({
|
|
data: {
|
|
email: "pause@example.test",
|
|
username: "pause",
|
|
passwordHash: "x",
|
|
},
|
|
});
|
|
userId = user.id;
|
|
});
|
|
|
|
it("backs off briefly, then pauses from the threshold, doubling, and a success resumes", async () => {
|
|
for (let i = 1; i < PROVIDER_PAUSE_THRESHOLD; i += 1) {
|
|
await ledger.recordFailure(userId, "admin-openai", 500);
|
|
}
|
|
expect(await windowMs(userId)).toBe(HARD_FAILURE_COOLDOWN_MS);
|
|
expect(
|
|
(await ledger.getSkipHints(userId)).get("admin-openai")?.reason,
|
|
).toBe("backoff");
|
|
|
|
await ledger.recordFailure(userId, "admin-openai", 500);
|
|
expect(await windowMs(userId)).toBe(PROVIDER_PAUSE_BASE_MS);
|
|
expect(
|
|
(await ledger.getSkipHints(userId)).get("admin-openai")?.reason,
|
|
).toBe("paused");
|
|
|
|
await ledger.recordFailure(userId, "admin-openai", 500);
|
|
expect(await windowMs(userId)).toBe(2 * PROVIDER_PAUSE_BASE_MS);
|
|
|
|
await ledger.recordSuccess(userId, "admin-openai");
|
|
expect((await ledger.getSkipHints(userId)).has("admin-openai")).toBe(false);
|
|
});
|
|
|
|
it("keeps the fixed cooldown for a dead credential, however long the run", async () => {
|
|
for (let i = 0; i < PROVIDER_PAUSE_THRESHOLD + 2; i += 1) {
|
|
await ledger.recordFailure(userId, "admin-openai", 401);
|
|
}
|
|
expect(
|
|
(await ledger.getSkipHints(userId)).get("admin-openai")?.reason,
|
|
).toBe("credential_expired");
|
|
});
|
|
});
|