mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-10 05:12:14 +00:00
Marc-André Bombeck
27342ee734
The nightly briefing warm completed every night and read "all-failed", because one run event held every account's annotations and the last account won. Each candidate now runs in its own wide event and the run reports every outcome through `reportJobRun`, so ok, dead sign-in, withheld and failed accounts show side by side. A dead Codex refresh token threw a bare error with no status, so the chain read it as a network failure: the health ledger recorded hard_failed, no "reconnect" hint appeared, and every night tried the same token. The refresh now throws `CodexRefreshError` with a 401 for a dead grant (invalid_grant / invalid_refresh_token on a 400 included), as do a missing or undecryptable stored credential. The ledger records auth_failed, the chain marks the primary credential expired, and the nightly pass reports the account as auth_failed without queuing a retry. A briefing the outbound screen withheld (a causal claim) failed outright, and the 45-minute provider retry then asked the same question eight times a night. The generator now makes one bounded repair that names the broken rule and keeps the repaired payload only if it passes the same parse, grounding and screen; a withheld outcome queues no retry. A provider that keeps failing is paused: from five consecutive hard failures the skip window starts at 30 minutes and doubles per further failure, capped at a day. A paused provider is walked last and tried again when the window lifts; one success resumes it. The operator card shows how many accounts it is paused for and until when. After a 5xx from a non-canonical OpenAI-compatible endpoint the client asks the free model listing, and a configured model the endpoint no longer offers is named in the error, on the hop line and on the card. Also: - The tiered context's live day band read `measured_at AT TIME ZONE tz`, which treats the stored UTC as local time; a far-from-UTC reader's evening readings landed on the next day. It converts from UTC first now. - Google Health intraday heart rate re-reads two hours per sync instead of a day; daily summaries and sleep keep the day (#1023). - The first filtered daily read that answers 400 and triggers the snake_case retry is recorded as a note, not an error (#1023). - The single derived-score read goes through the derived cache with stale-while-revalidate, like the batch beside it.
92 lines
3.1 KiB
TypeScript
92 lines
3.1 KiB
TypeScript
import { safeFetch } from "@/lib/safe-fetch";
|
|
import { aiEgressPolicyFor } from "./local-host-allowlist";
|
|
|
|
/**
|
|
* Does an OpenAI-compatible endpoint still offer the configured model?
|
|
*
|
|
* The operator's provider pointed at an OAuth proxy that installs
|
|
* `openai-oauth@latest` on every restart. One restart later the proxy listed
|
|
* only newer model names, the configured one was gone, and every call
|
|
* answered HTTP 500 with a body that named nothing. From the outside that
|
|
* reads as "the provider is down", and the fix (pick a model the endpoint
|
|
* offers) is invisible.
|
|
*
|
|
* `GET {baseUrl}/models` answers the question for free: it is the standard
|
|
* listing every OpenAI-compatible server implements, it bills nothing, and
|
|
* it needs the same bearer as a completion. The answer is cached for a few
|
|
* minutes per endpoint and model, so a burst of failures probes once.
|
|
*
|
|
* Tri-state on purpose. `unknown` (no listing route, a non-JSON answer, the
|
|
* probe itself failing) never claims the model is missing: only an endpoint
|
|
* that answered with a list that does not contain the name earns
|
|
* `not_listed`.
|
|
*/
|
|
export type ModelListing = "listed" | "not_listed" | "unknown";
|
|
|
|
const PROBE_TIMEOUT_MS = 5_000;
|
|
const CACHE_TTL_MS = 10 * 60 * 1000;
|
|
|
|
const cache = new Map<string, { at: number; value: ModelListing }>();
|
|
|
|
/** Test-only: forget every cached answer. */
|
|
export function clearModelListingCache(): void {
|
|
cache.clear();
|
|
}
|
|
|
|
export async function probeModelListing(input: {
|
|
baseUrl: string;
|
|
apiKey: string;
|
|
model: string;
|
|
/** See `OpenAIClient`'s `operatorTrusted`. */
|
|
operatorTrusted?: boolean;
|
|
now?: number;
|
|
}): Promise<ModelListing> {
|
|
const now = input.now ?? Date.now();
|
|
const key = `${input.baseUrl}\u0000${input.model}`;
|
|
const hit = cache.get(key);
|
|
if (hit && now - hit.at < CACHE_TTL_MS) return hit.value;
|
|
|
|
const value = await probeOnce(input);
|
|
cache.set(key, { at: now, value });
|
|
return value;
|
|
}
|
|
|
|
async function probeOnce(input: {
|
|
baseUrl: string;
|
|
apiKey: string;
|
|
model: string;
|
|
operatorTrusted?: boolean;
|
|
}): Promise<ModelListing> {
|
|
const url = `${input.baseUrl.replace(/\/$/, "")}/models`;
|
|
const headers: Record<string, string> = { Accept: "application/json" };
|
|
const key = input.apiKey.trim();
|
|
if (key) headers.Authorization = `Bearer ${key}`;
|
|
try {
|
|
const res = await safeFetch(
|
|
url,
|
|
{ method: "GET", headers },
|
|
{
|
|
timeoutMs: PROBE_TIMEOUT_MS,
|
|
...aiEgressPolicyFor(url, { operatorTrusted: input.operatorTrusted }),
|
|
},
|
|
);
|
|
if (!res.ok) return "unknown";
|
|
const json = (await res.json().catch(() => null)) as {
|
|
data?: unknown;
|
|
} | null;
|
|
if (!json || !Array.isArray(json.data)) return "unknown";
|
|
const ids = json.data
|
|
.map((entry) =>
|
|
entry && typeof entry === "object"
|
|
? (entry as { id?: unknown }).id
|
|
: undefined,
|
|
)
|
|
.filter((id): id is string => typeof id === "string");
|
|
if (ids.length === 0) return "unknown";
|
|
return ids.includes(input.model) ? "listed" : "not_listed";
|
|
} catch {
|
|
// The probe is a diagnosis, never a gate: a failed probe says nothing.
|
|
return "unknown";
|
|
}
|
|
}
|