mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-09 04:50:59 +00:00
fix(coach): say so when the model answered with reasoning alone
When every provider in the chain answered a Coach round with reasoning and no reply, even after being asked once more without reasoning, the turn ended with "could not reach a provider, please try again in a moment". The provider was reached, and trying again in a moment changes nothing for a model that cannot stop reasoning. Such a chain now ends with its own code, coach.provider.reasoning_only, and the drawer says the model only reasoned and never wrote an answer, and suggests another model. The choice between credential_expired, rate_limited, reasoning_only and unavailable moved into one function beside the other turn error rules, where it is tested on its own. An un-wrapped reasoning-only error maps the same way. Clients that do not know the code keep showing the generic provider message. Refs #1276
This commit is contained in:
1 parent
c9673883b1
commit
31e107b96a
14 files changed
+144
-28
No files matched your search
@@ -3020,6 +3020,7 @@
|
||||
"refusalInjection": "In deiner Nachricht stehen Anweisungen, die meine Vorgaben überschreiben sollen. Ich bin der HealthLog-Coach und fasse ausschließlich deine eigenen Gesundheitsdaten zusammen. Formuliere deine Frage bitte in diesem Rahmen neu.",
|
||||
"errorBudget": "Du hast das heutige Coach-Limit erreicht. Das Budget setzt sich um Mitternacht in deiner Zeitzone zurück.",
|
||||
"errorProvider": "Der Coach konnte gerade keinen Auswertungs-Anbieter erreichen. Bitte versuche es in einem Moment erneut.",
|
||||
"errorReasoningOnly": "Das Modell hat nur nachgedacht und keine Antwort geschrieben. Versuche es noch einmal oder wähle in den KI-Einstellungen ein anderes Modell.",
|
||||
"errorDemo": "In der Demo antwortet der Coach nicht.",
|
||||
"errorNoProvider": "Der Coach liest deine erfassten Gesundheitsdaten und beantwortet Fragen dazu (keine medizinische Beratung). Dafür braucht HealthLog einen KI-Anbieter. Verbinde deinen eigenen Schlüssel oder ein lokales Modell.",
|
||||
"errorNoProviderAction": "KI-Anbieter verbinden",
|
||||
|
||||
@@ -3020,6 +3020,7 @@
|
||||
"refusalInjection": "I noticed wording that tries to override my instructions. I am the HealthLog Coach and only summarise your own health data. Please rephrase your question in those terms.",
|
||||
"errorBudget": "You've reached today's Coach usage limit. The budget refreshes at midnight in your time zone.",
|
||||
"errorProvider": "The Coach could not reach an Insights provider just now. Please try again in a moment.",
|
||||
"errorReasoningOnly": "The model only reasoned and never wrote an answer. Try again, or choose a different model in your AI settings.",
|
||||
"errorDemo": "The Coach doesn't answer in the demo.",
|
||||
"errorNoProvider": "The Coach reads your logged health data and answers questions about it (not medical advice). HealthLog needs an AI provider for that. Connect your own key or a local model.",
|
||||
"errorNoProviderAction": "Connect an AI provider",
|
||||
|
||||
@@ -3020,6 +3020,7 @@
|
||||
"refusalInjection": "Tu mensaje contiene instrucciones que pretenden sobrescribir mis indicaciones. Soy el coach de HealthLog y solo resumo tus propios datos de salud. Reformula tu pregunta dentro de ese marco.",
|
||||
"errorBudget": "Has alcanzado el límite diario del coach. El cupo se reinicia a medianoche en tu zona horaria.",
|
||||
"errorProvider": "El coach no ha podido contactar con un proveedor de análisis ahora mismo. Inténtalo en un momento.",
|
||||
"errorReasoningOnly": "El modelo solo razonó y no llegó a escribir una respuesta. Inténtalo de nuevo o elige otro modelo en los ajustes de IA.",
|
||||
"errorDemo": "En la demo, el Coach no responde.",
|
||||
"errorNoProvider": "El coach lee tus datos de salud registrados y responde preguntas sobre ellos (no es consejo médico). HealthLog necesita un proveedor de IA para ello: conecta tu propia clave o un modelo local.",
|
||||
"errorNoProviderAction": "Conectar un proveedor de IA",
|
||||
|
||||
@@ -3020,6 +3020,7 @@
|
||||
"refusalInjection": "Ton message contient des instructions qui cherchent à remplacer mes consignes. Je suis le coach HealthLog et je ne résume que tes propres données de santé. Reformule ta question dans ce cadre.",
|
||||
"errorBudget": "Tu as atteint la limite quotidienne du coach. Le quota se réinitialise à minuit dans ton fuseau horaire.",
|
||||
"errorProvider": "Le coach n’a pas pu joindre un fournisseur d’analyse maintenant. Réessaie dans un instant.",
|
||||
"errorReasoningOnly": "Le modèle a seulement réfléchi, sans écrire de réponse. Réessaie ou choisis un autre modèle dans les réglages IA.",
|
||||
"errorDemo": "Dans la démo, le Coach ne répond pas.",
|
||||
"errorNoProvider": "Le coach lit tes données de santé enregistrées et répond aux questions à leur sujet (pas un avis médical). HealthLog a besoin d'un fournisseur d'IA pour cela : connecte ta propre clé ou un modèle local.",
|
||||
"errorNoProviderAction": "Connecter un fournisseur d'IA",
|
||||
|
||||
@@ -3020,6 +3020,7 @@
|
||||
"refusalInjection": "Nel tuo messaggio ci sono istruzioni che cercano di sovrascrivere le mie indicazioni. Sono il coach di HealthLog e riassumo solo i tuoi dati di salute: riformula la domanda in questo perimetro.",
|
||||
"errorBudget": "Hai raggiunto il limite giornaliero del coach. Il budget si reimposta a mezzanotte nel tuo fuso orario.",
|
||||
"errorProvider": "Il coach non è riuscito a contattare un provider di analisi ora. Riprova fra poco.",
|
||||
"errorReasoningOnly": "Il modello ha solo ragionato senza scrivere una risposta. Riprova oppure scegli un altro modello nelle impostazioni IA.",
|
||||
"errorDemo": "Nella demo il Coach non risponde.",
|
||||
"errorNoProvider": "Il coach legge i tuoi dati di salute registrati e risponde a domande su di essi (non è consulenza medica). HealthLog ha bisogno di un provider IA per questo: collega la tua chiave o un modello locale.",
|
||||
"errorNoProviderAction": "Collega un provider IA",
|
||||
|
||||
@@ -3020,6 +3020,7 @@
|
||||
"refusalInjection": "제 지시를 덮어쓰려는 표현이 보였어요. 저는 HealthLog 코치이고 회원님의 건강 데이터만 정리해 드려요. 그 범위로 질문을 다시 적어 주세요.",
|
||||
"errorBudget": "오늘 코치 사용 한도에 도달했어요. 사용량은 내 시간대의 자정에 초기화돼요.",
|
||||
"errorProvider": "지금은 코치가 인사이트 제공자에 연결하지 못했어요. 잠시 후 다시 시도해 주세요.",
|
||||
"errorReasoningOnly": "모델이 생각만 하고 답변을 쓰지 않았어요. 다시 시도하거나 AI 설정에서 다른 모델을 선택해 주세요.",
|
||||
"errorDemo": "데모에서는 코치가 답하지 않아요.",
|
||||
"errorNoProvider": "코치는 기록된 건강 데이터를 읽고 그에 대한 질문에 답해요(의학적 조언은 아니에요). 그러려면 HealthLog에 AI 제공자가 필요해요. 직접 발급한 키나 로컬 모델을 연결해 주세요.",
|
||||
"errorNoProviderAction": "AI 제공자 연결",
|
||||
|
||||
@@ -3020,6 +3020,7 @@
|
||||
"refusalInjection": "Ik zie formuleringen die proberen mijn instructies te omzeilen. Ik ben de HealthLog Coach en vat alleen je eigen gezondheidsgegevens samen. Formuleer je vraag opnieuw binnen dat kader.",
|
||||
"errorBudget": "Je hebt de gebruikslimiet van de Coach voor vandaag bereikt. Het budget wordt om middernacht in jouw tijdzone vernieuwd.",
|
||||
"errorProvider": "De Coach kan op dit moment geen provider voor Inzichten bereiken. Probeer het zo opnieuw.",
|
||||
"errorReasoningOnly": "Het model heeft alleen nagedacht en geen antwoord geschreven. Probeer het opnieuw of kies een ander model in de AI-instellingen.",
|
||||
"errorDemo": "De Coach geeft in de demo geen antwoord.",
|
||||
"errorNoProvider": "De Coach leest je vastgelegde gezondheidsgegevens en beantwoordt vragen daarover (geen medisch advies). Daarvoor heeft HealthLog een AI-provider nodig. Koppel je eigen sleutel of een lokaal model.",
|
||||
"errorNoProviderAction": "AI-provider koppelen",
|
||||
|
||||
@@ -3020,6 +3020,7 @@
|
||||
"refusalInjection": "Twoja wiadomość zawiera instrukcje, które próbują nadpisać moje wytyczne. Jestem coachem HealthLog i podsumowuję wyłącznie twoje własne dane zdrowotne. Przeformułuj pytanie w tych ramach.",
|
||||
"errorBudget": "Osiągnięto dzienny limit coacha. Limit resetuje się o północy w Twojej strefie czasowej.",
|
||||
"errorProvider": "Coach nie mógł teraz skontaktować się z dostawcą analizy. Spróbuj za chwilę.",
|
||||
"errorReasoningOnly": "Model tylko rozumował i nie napisał odpowiedzi. Spróbuj ponownie albo wybierz inny model w ustawieniach AI.",
|
||||
"errorDemo": "W wersji demo Coach nie odpowiada.",
|
||||
"errorNoProvider": "Coach czyta Twoje zapisane dane zdrowotne i odpowiada na pytania o nie (to nie porada medyczna). HealthLog potrzebuje do tego dostawcy AI: połącz własny klucz lub model lokalny.",
|
||||
"errorNoProviderAction": "Połącz dostawcę AI",
|
||||
|
||||
@@ -3020,6 +3020,7 @@
|
||||
"refusalInjection": "Percebi uma formulação que tenta substituir minhas instruções. Sou o Coach do HealthLog e só resumo seus próprios dados de saúde. Reformule sua pergunta nesses termos.",
|
||||
"errorBudget": "Você atingiu o limite de uso do Coach hoje. O orçamento é renovado à meia-noite no seu fuso horário.",
|
||||
"errorProvider": "O Coach não conseguiu alcançar um provedor de Insights agora. Tente novamente em instantes.",
|
||||
"errorReasoningOnly": "O modelo apenas raciocinou e não escreveu uma resposta. Tente novamente ou escolha outro modelo nas configurações de IA.",
|
||||
"errorDemo": "O Coach não responde na demonstração.",
|
||||
"errorNoProvider": "O Coach lê seus dados de saúde registrados e responde perguntas sobre eles (não é aconselhamento médico). O HealthLog precisa de um provedor de IA para isso. Conecte sua própria chave ou um modelo local.",
|
||||
"errorNoProviderAction": "Conectar um provedor de IA",
|
||||
|
||||
@@ -817,6 +817,14 @@ describe("errorCodeToI18nKey", () => {
|
||||
);
|
||||
});
|
||||
|
||||
// #1276 — a model that only reasoned was reached; "try again in a
|
||||
// moment" is the wrong advice for it.
|
||||
it("maps coach.provider.reasoning_only to its own copy", () => {
|
||||
expect(errorCodeToI18nKey("coach.provider.reasoning_only")).toBe(
|
||||
"insights.coach.errorReasoningOnly",
|
||||
);
|
||||
});
|
||||
|
||||
// v1.39 — a refused turn names its reason. The capability refusals carry
|
||||
// their code in `meta.errorCode`; each maps to copy that says what is off
|
||||
// and who can change it, never the generic "provider unavailable".
|
||||
|
||||
@@ -87,6 +87,11 @@ export function errorCodeToI18nKey(code: string): string {
|
||||
// the user to guided setup with a one-line explainer + a Settings
|
||||
// link rendered alongside the bubble copy.
|
||||
return "insights.coach.errorNoProvider";
|
||||
case "coach.provider.reasoning_only":
|
||||
// #1276 — the model was reached and answered with its reasoning
|
||||
// only, even after being asked once more. "Try again in a moment"
|
||||
// would be the wrong advice; the copy names what happened.
|
||||
return "insights.coach.errorReasoningOnly";
|
||||
case "coach.provider.unavailable":
|
||||
case "coach.provider.empty":
|
||||
case "coach.stream":
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
/**
|
||||
* The `coach.provider.*` code a failed provider chain surfaces. #1276 — a
|
||||
* chain whose every hop answered with reasoning and no reply gets its own
|
||||
* code: "could not reach a provider, try again in a moment" is wrong for it,
|
||||
* because the provider was reached and answered, just never with an answer.
|
||||
*/
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import type { FallbackHop } from "@/lib/ai/provider-runner";
|
||||
|
||||
import { chainFailureCode, classifyBubblingProviderError } from "../errors";
|
||||
|
||||
function hop(over: Partial<FallbackHop>): FallbackHop {
|
||||
return {
|
||||
providerType: "openai-compatible",
|
||||
attempt: 1,
|
||||
failureReason: "OpenAI-compatible gateway returned empty content",
|
||||
httpStatus: 0,
|
||||
...over,
|
||||
};
|
||||
}
|
||||
|
||||
describe("chainFailureCode", () => {
|
||||
it("names a chain that only ever reasoned", () => {
|
||||
expect(
|
||||
chainFailureCode({
|
||||
attempts: [hop({ failureKind: "reasoning_only" })],
|
||||
primaryCredentialExpired: false,
|
||||
}),
|
||||
).toBe("coach.provider.reasoning_only");
|
||||
});
|
||||
|
||||
it("stays generic when another hop failed for another reason", () => {
|
||||
expect(
|
||||
chainFailureCode({
|
||||
attempts: [
|
||||
hop({ failureKind: "reasoning_only" }),
|
||||
hop({ attempt: 2, httpStatus: 503, failureKind: "http" }),
|
||||
],
|
||||
primaryCredentialExpired: false,
|
||||
}),
|
||||
).toBe("coach.provider.unavailable");
|
||||
});
|
||||
|
||||
it("keeps the existing codes", () => {
|
||||
expect(
|
||||
chainFailureCode({
|
||||
attempts: [hop({ httpStatus: 401 })],
|
||||
primaryCredentialExpired: true,
|
||||
}),
|
||||
).toBe("coach.provider.credential_expired");
|
||||
expect(
|
||||
chainFailureCode({
|
||||
attempts: [hop({ httpStatus: 429 }), hop({ httpStatus: 429 })],
|
||||
primaryCredentialExpired: false,
|
||||
}),
|
||||
).toBe("coach.provider.rate_limited");
|
||||
expect(
|
||||
chainFailureCode({ attempts: [], primaryCredentialExpired: false }),
|
||||
).toBe("coach.provider.unavailable");
|
||||
});
|
||||
});
|
||||
|
||||
describe("classifyBubblingProviderError", () => {
|
||||
it("names an un-wrapped reasoning-only reply", () => {
|
||||
const err = Object.assign(new Error("returned empty content"), {
|
||||
httpStatus: 0,
|
||||
kind: "reasoning_only",
|
||||
upstream: "openai-compatible",
|
||||
});
|
||||
expect(classifyBubblingProviderError(err)).toEqual({
|
||||
code: "reasoning_only",
|
||||
httpStatus: 0,
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -8,9 +8,41 @@
|
||||
*/
|
||||
import { annotate, getEvent } from "@/lib/logging/context";
|
||||
import { redactOptional, redactSecrets } from "@/lib/logging/redact";
|
||||
import type { AllProvidersFailedError } from "@/lib/ai/provider-runner";
|
||||
|
||||
import type { TurnEmitter } from "./types";
|
||||
|
||||
/**
|
||||
* The `coach.provider.*` code for a provider chain where every hop failed.
|
||||
*
|
||||
* - v1.11.0 — the person's PRIMARY provider failed auth-class (401/403):
|
||||
* the credential is dead, not the service, so the drawer deep-links to
|
||||
* reconnect (`credential_expired`).
|
||||
* - v1.4.25 — every hop landed on 429: transient, on the provider's side
|
||||
* (`rate_limited`).
|
||||
* - #1276 — every hop answered with reasoning and no reply, after the
|
||||
* client already asked once more (`reasoning_only`). The provider was
|
||||
* reached; "try again in a moment" is the wrong advice, and the thought
|
||||
* itself is never shown as the answer.
|
||||
* - Anything else: `unavailable`.
|
||||
*/
|
||||
export function chainFailureCode(
|
||||
err: Pick<AllProvidersFailedError, "attempts" | "primaryCredentialExpired">,
|
||||
): string {
|
||||
if (err.primaryCredentialExpired) return "coach.provider.credential_expired";
|
||||
const { attempts } = err;
|
||||
if (attempts.length > 0 && attempts.every((a) => a.httpStatus === 429)) {
|
||||
return "coach.provider.rate_limited";
|
||||
}
|
||||
if (
|
||||
attempts.length > 0 &&
|
||||
attempts.every((a) => a.failureKind === "reasoning_only")
|
||||
) {
|
||||
return "coach.provider.reasoning_only";
|
||||
}
|
||||
return "coach.provider.unavailable";
|
||||
}
|
||||
|
||||
/**
|
||||
* v1.21.3 — classify a provider error that bubbled out of the chain runner
|
||||
* un-wrapped (i.e. not an `AllProvidersFailedError`). The provider clients tag
|
||||
@@ -18,15 +50,17 @@ import type { TurnEmitter } from "./types";
|
||||
* provider failure that must surface a graceful `coach.provider.*` frame rather
|
||||
* than rethrow into an HTTP 500. Returns `null` for anything that is NOT a
|
||||
* recognisable provider error (a real server bug), so those keep the 500 +
|
||||
* GlitchTip path. Status mapping mirrors `AllProvidersFailedError`: 401/403 →
|
||||
* credential_expired, 429 → rate_limited, everything else → unavailable.
|
||||
* GlitchTip path. Status mapping mirrors `chainFailureCode`: 401/403 →
|
||||
* credential_expired, 429 → rate_limited, a reasoning-only reply →
|
||||
* reasoning_only, everything else → unavailable.
|
||||
*/
|
||||
export function classifyBubblingProviderError(err: unknown): {
|
||||
code: "credential_expired" | "rate_limited" | "unavailable";
|
||||
code:
|
||||
"credential_expired" | "rate_limited" | "reasoning_only" | "unavailable";
|
||||
httpStatus: number | null;
|
||||
} | null {
|
||||
if (err === null || typeof err !== "object") return null;
|
||||
const e = err as { upstream?: unknown; httpStatus?: unknown };
|
||||
const e = err as { upstream?: unknown; httpStatus?: unknown; kind?: unknown };
|
||||
const hasUpstreamTag = typeof e.upstream === "string";
|
||||
const status = typeof e.httpStatus === "number" ? e.httpStatus : null;
|
||||
// Require the wire tag — a bare `{ httpStatus }` from unrelated code must not
|
||||
@@ -36,6 +70,10 @@ export function classifyBubblingProviderError(err: unknown): {
|
||||
return { code: "credential_expired", httpStatus: status };
|
||||
}
|
||||
if (status === 429) return { code: "rate_limited", httpStatus: status };
|
||||
// #1276 — the empty-reply sentinel of a reply that was reasoning only.
|
||||
if (e.kind === "reasoning_only") {
|
||||
return { code: "reasoning_only", httpStatus: status };
|
||||
}
|
||||
return { code: "unavailable", httpStatus: status };
|
||||
}
|
||||
|
||||
|
||||
@@ -85,7 +85,7 @@ import type { TurnLedger } from "./budget";
|
||||
import type { TurnReasoning } from "./reasoning";
|
||||
import type { TurnChain } from "./chain";
|
||||
import type { TurnContext } from "./context";
|
||||
import { classifyBubblingProviderError } from "./errors";
|
||||
import { chainFailureCode, classifyBubblingProviderError } from "./errors";
|
||||
import { snapshotStep, toStep } from "./steps";
|
||||
import type { TurnEmitter } from "./types";
|
||||
|
||||
@@ -769,29 +769,9 @@ export async function runTurnModel(args: {
|
||||
credentialExpired: err.primaryCredentialExpired,
|
||||
},
|
||||
});
|
||||
// v1.11.0 — when the user's PRIMARY provider failed with an
|
||||
// auth-class status (401/403), the credential is dead, not the
|
||||
// service. Surface a distinct `credential_expired` frame so the
|
||||
// drawer can deep-link the user to reconnect rather than telling
|
||||
// them to "try again later" — the gap that let an expired codex
|
||||
// token silently kill all generation.
|
||||
if (err.primaryCredentialExpired) {
|
||||
return { ok: false, code: "coach.provider.credential_expired" };
|
||||
}
|
||||
// v1.4.25 — distinguish provider rate-limit (every attempt
|
||||
// landed on 429) from generic unavailability. The drawer's
|
||||
// error-decoder surfaces the rate-limit copy with a warning
|
||||
// toast instead of the generic provider-down message, so the
|
||||
// user understands the limit is transient.
|
||||
const allRateLimited =
|
||||
err.attempts.length > 0 &&
|
||||
err.attempts.every((a) => a.httpStatus === 429);
|
||||
return {
|
||||
ok: false,
|
||||
code: allRateLimited
|
||||
? "coach.provider.rate_limited"
|
||||
: "coach.provider.unavailable",
|
||||
};
|
||||
// Credential expired, rate limited, reasoning without an answer
|
||||
// (#1276), or unavailable: `chainFailureCode` says which and why.
|
||||
return { ok: false, code: chainFailureCode(err) };
|
||||
}
|
||||
// v1.21.3 — defence in depth. The chain runner wraps every hard provider
|
||||
// failure in `AllProvidersFailedError`, but a provider client can still
|
||||
|
||||
Reference in new issue
Block a user