The live evaluation now captures the tool calls a generation makes, with
their arguments, and hands them to the judge. runScenarioLive puts a
scenario to the model in the chat turn's own context (system prompt,
tool-mode and dialog addenda, DATA INVENTORY, EARLIER TABLES, transcript)
over the real tool catalogue, answers each call with a stand-in result,
and reads the reply through the turn's clarification and chip parsers.
runScenarioJudge grades each scenario a few times with the same
evaluateScenario the deterministic suite uses and reports a pass rate per
scenario; the server-only scenarios and checks are left out.
Like the existing judge it runs only with COACH_EVAL_API_KEY, from the
nightly workflow's non-blocking step, and never throws.
A deterministic golden-set + red-team floor that gates every Coach change
offline, plus an opt-in live-judge layer that scores warmth and groundedness
with a pinned model only when COACH_EVAL_API_KEY is set.
- 42 golden cases across grounding, cross-metric, data-honesty, provider-parity
and own-baseline taxonomies, graded by behaviour-anchored binary criteria.
- Structured-claim grounding helpers extend the numeric verifier from numbers
to threshold / own-baseline / confident-verdict claims; the numeric API is
unchanged so the existing grounding tests stay green.
- Red-team battery asserts the existing inbound refusal, outbound dose screen,
acute red-flag escalation and learn-link guards fire.
- Nightly + manual workflow runs the deterministic floor always; the live judge
step runs only when the secret is present and is non-blocking.
Marc-André Bombeck