mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-09 13:01:00 +00:00
test(coach): put the dialog scenarios to a real model nightly
The live evaluation now captures the tool calls a generation makes, with their arguments, and hands them to the judge. runScenarioLive puts a scenario to the model in the chat turn's own context (system prompt, tool-mode and dialog addenda, DATA INVENTORY, EARLIER TABLES, transcript) over the real tool catalogue, answers each call with a stand-in result, and reads the reply through the turn's clarification and chip parsers. runScenarioJudge grades each scenario a few times with the same evaluateScenario the deterministic suite uses and reports a pass rate per scenario; the server-only scenarios and checks are left out. Like the existing judge it runs only with COACH_EVAL_API_KEY, from the nightly workflow's non-blocking step, and never throws.
This commit is contained in:
1 parent
ec281c02e2
commit
c2ab9cda11
6 files changed
+651
-8
No files matched your search
@@ -3,12 +3,14 @@ name: Coach eval
|
||||
# B0 (v1.21.3) — the Coach evaluation harness, two layers.
|
||||
#
|
||||
# 1. The DETERMINISTIC suite (golden set + red-team battery + the grounding
|
||||
# grader) runs ALWAYS. It is free, offline, and the per-change floor — the
|
||||
# grader + the dialog scenarios) runs ALWAYS. It is free, offline, and the per-change floor — the
|
||||
# same suite `pnpm test` runs, pinned here as a standing nightly check.
|
||||
# 2. The LIVE JUDGE step runs ONLY when the `COACH_EVAL_API_KEY` repository
|
||||
# secret is present, and is NON-BLOCKING (`continue-on-error: true`). With
|
||||
# the secret absent it no-ops with a clear log line — the workflow stays
|
||||
# green. The operator flips the secret on later; no rebuild needed.
|
||||
# green. The operator flips the secret on later; no rebuild needed. The
|
||||
# same step puts the dialog scenarios to the model and logs a pass rate
|
||||
# per scenario, with the tool calls and their arguments.
|
||||
#
|
||||
# Schedule + manual only. This is NOT a required check.
|
||||
|
||||
@@ -51,7 +53,7 @@ jobs:
|
||||
# Layer 1 — the deterministic floor. Free, offline, ALWAYS runs. A
|
||||
# failure here reds the workflow: a regression in the golden set or the
|
||||
# red-team battery is a real find.
|
||||
- name: Deterministic floor (golden set + red team)
|
||||
- name: Deterministic floor (golden set + red team + dialog scenarios)
|
||||
run: pnpm exec vitest run src/lib/ai/coach/eval/__tests__/
|
||||
|
||||
# Layer 2 — the live judge. Runs ONLY when the secret is present, and is
|
||||
|
||||
Reference in new issue
Block a user