diff --git a/.planning/STATE.md b/.planning/STATE.md index ffdccf632..a1851dc43 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -544,14 +544,95 @@ B1 / B2 / B3: v1.4.16 should adopt ### C1 — AI/Codex hardening (infrastructure for hallucination-resistance) -- [ ] Provider abstraction review — multi-provider readiness -- [ ] Schema enforcement — output structured (JSON-schema), no free-form unbounded text -- [ ] Citation requirement — every claim must reference user-data point -- [ ] System-prompt scope hardening — medical-context-only, refuse out-of-scope -- [ ] Slug-drift defense — model fallback chain, dynamic discovery -- [ ] Document v1.4.16 backlog: medical-reference grounding (AHA guidelines etc.), multi-provider redundancy +- [x] Provider abstraction review — `MockAIProvider` + 7-test contract suite (`27310e4`) +- [x] Schema enforcement — `aiInsightResponseSchema` (zod) + `generateInsight()` retry-once wrapper + 422-on-fail (swept into sibling commit `5510ed5`; diff correct on main) +- [x] Citation requirement — every recommendation must cite a snapshot data point; cross-check enforced (`d657f79`) +- [x] System-prompt scope hardening — versioned prompt with refusal pattern, doctor-consult CTA, ESH/ESC 2024 generic guidance (`4e85c38`) +- [x] Slug-drift defense — fallback chain + 1h positive cache + structured 503 on all-failed; spec §7b extended (`4bba951`) +- [x] Document v1.4.16 backlog — `docs/audit/v1416-ai-roadmap.md` (`fa11f10`) - Detailed report: `.planning/phase-C1-report.md` +#### C1 status block — 2026-05-09T22:00+02:00 — done + +6 commits on `origin/main`. Tests +67 unit (965 → 1048; some +sibling-agent tests merged too), 31 / 31 integration pass, typecheck +clean for C1 files, lint 0 errors. + +Schema decisions: + +- Strict response shape `{summary, recommendations[], citations[], + warnings[]}` with `recommendations[].metricSource` mandatory (zero- + hallucination guard); cross-check rejects responses whose + recommendations cite metricSources not in `citations[]`. +- Wrapper `generateInsight()` calls provider once, parses+validates, + retries ONCE with corrective system message embedding the violated + zod issues, throws `InsightSchemaError(httpStatus 422)` on second + failure. Provider-level errors (5xx, 401-after-refresh, 429) bubble + through untouched — wrapper retries only on schema violation. +- `.passthrough()` on the strict schema lets legacy fields ride along + so cached v1.4.14 payloads still hydrate the dashboard. v1.4.16 + retires the passthrough. + +Slug-drift defence rationale: + +- v1.4.7..v1.4.13 lost a week to upstream allow-list rotation + (`gpt-5-codex` → `gpt-5.3-codex`). Marc: "Slug drift darf halt + immer überhaupt nicht sein." Hard mandate. +- Fix: `CodexClient` walks `[cached?, ...DEFAULT_SLUG_FALLBACK_CHAIN]` + on every fresh request. Walk triggers on 400 + canonical rejection + body / `model_not_found` / `does not exist` / 404. Auth (401) goes + through refresh + same-slug retry — does NOT walk. 5xx / 429 / SSE + `invalid_prompt` propagate immediately. All-failed: structured 503 + with `attempted[]` list. +- Cache: process-local 1h TTL (`src/lib/ai/codex-slug-cache.ts`), + single slot, auto-evict on read. Cache hit short-circuits chain + walk. Cache invalidated on first slug-rejection walk. +- `CODEX_MODEL_FALLBACK_CHAIN` env override; `CODEX_MODEL` folds into + position 0 of the chain when set. +- Spec source-of-truth FIRST (per Marc's v1.4.6 hard rule): + `docs/codex-protocol-spec.md` §7b extended before implementation. + +Out-of-scope refusal: prompt instructs the model to return a fixed +JSON payload (`OUT_OF_SCOPE_REFUSAL_EN/DE`) when the snapshot is +empty or off-topic. Both variants validate against the strict schema +so the wrapper passes them through cleanly. + +NOT done in v1.4.15 (deliberately deferred to v1.4.16 per scope): + +- Route-side wiring of `generateInsight()` into + `/api/insights/generate` — the existing UI consumes the rich legacy + shape (`{summary, classification, findings, correlations, + dataQuality, ...}`); switching the wrapper without migrating the UI + would break the dashboard. v1.4.16 ships both together. +- Reading `client.getLastDiagnostics()` into Wide-Event annotation — + one-line follow-up; left out to keep the v1.4.15 diff scoped to + provider / wrapper / schema / prompt / cache. + +Roadmap items captured in `docs/audit/v1416-ai-roadmap.md`: + +1. v1.4.16 — Medical-reference grounding (curated bundle of ~40 + excerpts: ESH/ESC 2024, AHA/ACC 2017, WHO, AASM, Saint-Maurice + 2020, DGE/DEGAM; pre-flight relevance selection; schema gains + `referenceId`). +2. v1.4.16 — Multi-provider redundancy (`MultiProviderCascade` + wrapping `AIProvider`; admin-tier ordering; per-request 60s budget). +3. v1.4.16 — Per-recommendation explainability + (`rationale.{dataWindow, comparedTo, deviation}` + UI "Why?" tooltip). +4. v1.4.16 — Route + UI migration to the strict schema. +5. v1.4.17 — User-feedback loop (`InsightFeedback` table; admin + `/admin/ai-quality` dashboard with helpful-rate by recommendation + type / locale / provider / `PROMPT_VERSION`). +6. v1.4.17 — Calibrated per-recommendation confidence score. +7. v1.5 — Optional "deep mode" with Codex reasoning summaries + (separate schema field, never folded into `summary`). + +Cross-agent: same shared-cwd race documented across A2 / A4 / B-mobile +/ B1-B6 / C2 / C3 / C4 struck commit 2 — my schema.ts + generate- +insight.ts + 11-test file got swept into a sibling agent's empty-states +commit `5510ed5`. Code on `main` correct (verified via +`git ls-files src/lib/ai/`); message scope misleading. v1.4.16 should +adopt `superpowers:using-git-worktrees` per agent. + ### C2 — Auto-deployment - [x] Coolify webhook on GHCR push (`docker-publish.yml` → diff --git a/.planning/phase-C1-report.md b/.planning/phase-C1-report.md new file mode 100644 index 000000000..89f011c3d --- /dev/null +++ b/.planning/phase-C1-report.md @@ -0,0 +1,210 @@ +# Phase C1 — AI/Codex hardening (v1.4.15 infrastructure) + +Status: done +Last update: 2026-05-09T22:00:00+02:00 +Owner: Phase C1 agent (Claude Opus 4.7) + +## Marc's mandate (verbatim) + +> "Die Integration des Slug Drift Risiko darf halt immer überhaupt +> nicht sein. Das soll nicht funktional sein, sondern das soll stark +> sein. ... Es darf null Halluzinationen haben und es muss sich halt +> irgendwie stützen auf medizinische Dinge. Ich möchte da eine gewisse +> Dynamik drin haben." + +Translation: slug-drift risk must vanish; AI must move FUNCTIONAL → +STRONG; zero hallucinations; ground on medical facts; some dynamism. +Multi-iteration acceptable. + +v1.4.15 ships **infrastructure**. Medical-reference grounding (AHA / +ESH excerpts as system context) and full multi-provider redundancy +land in v1.4.16+ — see `docs/audit/v1416-ai-roadmap.md`. + +## Commits (in order) + +| Commit | Subject | Role | +| ----------------------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ | +| `27310e4` | refactor(ai): consolidate providers behind AIProvider interface | MockAIProvider + 7 contract tests (every provider satisfies CompletionResult shape) | +| `5510ed5` (sibling-merged, scope is mine) | docs(audit): v1.4.15 empty-states audit + i18n keys | swept in my schema.ts + generate-insight.ts + generate-insight.test.ts (parallel-agent race) | +| `d657f79` | feat(ai): enforce citation-from-data on every recommendation | 11-test invariant suite — schema rejects missing/empty metricSource; cross-check enforced | +| `4e85c38` | feat(ai): scope-hardened system prompt with refusal pattern | Versioned prompt at `src/lib/ai/prompts/insight-generator.ts` (`PROMPT_VERSION = "4.15.0"`) + 18 tests | +| `4bba951` | feat(ai): fallback-chain slug discovery with 1h positive cache | `CodexClient` walks fallback chain + cache; spec §7b extended; 20 tests | +| `fa11f10` | docs(audit): v1.4.16 AI hardening roadmap | Iteration plan Marc explicitly invited | + +Cross-agent observation: same shared-cwd race documented across A2 / +A4 / B-mobile / B1-B6 / C2 / C3 / C4 / B5 of the v1.4.15 marathon +struck commit 2 — my schema.ts + generate-insight.ts + the 11-test +file got swept into a sibling agent's `docs(audit): v1.4.15 empty- +states audit + i18n keys` commit `5510ed5`. The diff is correct on +`origin/main` (verified via `git ls-files src/lib/ai/`), only the +commit-message scope is misleading. v1.4.16 should adopt +`superpowers:using-git-worktrees` per parallel agent — recommendation +already echoed across the marathon's STATE log. + +## Schema decisions + +### Strict response schema (`src/lib/ai/schema.ts`) + +```ts +{ + summary: string, // user-facing English/German + recommendations: [{ + id: string, // stable per-response slug + text: string, + severity: "info" | "suggestion" | "important" | "urgent", + metricSource: { type, timeRange, summary, n? }, + }], + citations: [{ type, timeRange, summary }], + warnings: [{ topic, message, severity? }], +} +``` + +Three-pronged guard: + +1. **Required-field guard** — empty `metricSource.summary` rejected at + parse time. The model cannot fabricate a recommendation that + points at "nothing". +2. **Cross-check guard** — `findUncitedRecommendations()` rejects + responses where any recommendation cites a `(type, timeRange)` not + in `citations[]`. Two-step verification: parse → cross-check. +3. **Retry-once guard** — `generateInsight()` wraps the provider call + and on schema failure injects the corrective context into the user + prompt and re-tries ONCE. On second failure: throws + `InsightSchemaError` with `httpStatus: 422` for the route. + +The strict schema uses `.passthrough()` so legacy fields +(`classification`, `findings`, `correlations`, `dataQuality`, +`disclaimer`) ride along — cached payloads from v1.4.14 still hydrate +the dashboard. v1.4.16 retires `.passthrough()` once the UI migrates +to the strict shape. + +### Versioned system prompt + +`PROMPT_VERSION = "4.15.0"` is embedded in both locale variants and +exported for Wide-Event annotation. Future iterations bump the +revision so logs can attribute response quality to a specific prompt +version. Versioning policy documented in the prompt file's header. + +### Slug fallback chain default + +```ts +DEFAULT_SLUG_FALLBACK_CHAIN = [ + "gpt-5.3-codex", // verified 2026-05-09 on Plus/Pro + "gpt-5-codex", // historical default — kept as second-chance retry + "gpt-5", // bare slug — currently rejected on ChatGPT-auth + "gpt-4o", // last-ditch capability fallback +] +``` + +Override via `CODEX_MODEL_FALLBACK_CHAIN` (comma-separated). When +`CODEX_MODEL` is also set, it folds into chain position 0; defaults +survive behind it. Stable de-duplication. + +## Fallback chain rationale + +Per `docs/codex-protocol-spec.md` §7b (extended in this phase): + +| Status | Action | +| ----------------------------------------------- | ------------------------------------- | +| `400` + "not supported when using Codex with a ChatGPT account" | walk | +| `400` + `model_not_found` | walk | +| `400` + `does not exist` AND mentions a model | walk | +| `404` | walk | +| `401` (first time) | refresh and retry SAME slug; DON'T walk | +| `401` after refresh, `403`, `429`, `5xx` | propagate (don't walk) | +| `200` + SSE `response.failed.error.code === "invalid_prompt"` | propagate (request shape is wrong) | + +Positive cache: process-local Map at `src/lib/ai/codex-slug-cache.ts`, +single slot, 1 h TTL. Cache hit makes the working slug come first on +the next call. Cache invalidated on first slug-rejection walk. + +All-failed: structured 503 with `attempted: string[]`, +`httpStatus: 503`, message `"AI provider unreachable — all configured +Codex slugs were rejected"`. The route layer surfaces "AI provider +unreachable — check Settings". + +The auth path (401 → refresh + retry SAME slug) is preserved verbatim +from v1.4.7..v1.4.13 — auth-state failures don't get fixed by walking +the slug chain. Subtle but load-bearing. + +Diagnostics: `client.getLastDiagnostics()` returns +`{ attempted, cacheState, workingSlug }` for Wide-Event annotation. +The route at `/api/insights/generate` does NOT yet read this — adding +the annotation is a v1.4.15.1 follow-up (one-line `annotate()` call). + +## Out-of-scope refusal pattern + +The scope-hardened prompt instructs the model to return a fixed +refusal payload when the snapshot is empty or off-topic: + +```json +{ + "summary": "I can only summarise the health metrics in your log. The submitted data did not contain measurements I can analyse.", + "recommendations": [], + "citations": [], + "warnings": [] +} +``` + +Both EN and DE variants exported as `OUT_OF_SCOPE_REFUSAL_EN/DE`. +Tests pin that the payload validates against the strict schema and +the wrapper passes it through cleanly. + +## Tests added + +| File | Tests | Focus | +| ------------------------------------------ | ----- | ------------------------------------------------------------ | +| `provider-contract.test.ts` | 7 | Every provider returns CompletionResult shape | +| `mock-client.ts` | - | Test infrastructure, no tests of its own (covered above) | +| `schema.ts` (no test file) | - | Tested via the next two files | +| `generate-insight.test.ts` | 11 | Schema parse, retry-once with correction, 422 on second-fail | +| `citation-enforcement.test.ts` | 11 | Schema-level + cross-check + wrapper end-to-end | +| `insight-generator-prompt.test.ts` | 18 | Both locales, scope hardening, refusal payload, ESH/ESC etc. | +| `codex-slug-cache.test.ts` | 7 | TTL boundaries, invalidation, replacement | +| `codex-slug-fallback.test.ts` | 13 | Walk triggers, no-walk on 5xx/429/401, all-failed, cache | + +Total: +67 unit tests across 6 new files. 965 → 1048 unit pass +(+83 net from baseline; some sibling-agent tests merged in too). +Integration: 31 / 31 still pass. Typecheck: no new errors. Lint: 0 +errors, 11 pre-existing warnings. + +## Link to v1.4.16 roadmap + +`docs/audit/v1416-ai-roadmap.md` enumerates the deferred items: + +1. Medical-reference grounding (hand-curated bundle of ESH/ESC, AHA, + WHO, AASM, DGE/DEGAM excerpts; pre-flight relevance selection). +2. Multi-provider redundancy (`MultiProviderCascade` wrapping + `AIProvider`; admin-tier ordering). +3. Per-recommendation explainability + (`rationale.{dataWindow, comparedTo, deviation}` + UI tooltip). +4. Route-side wiring: `/api/insights/generate` adopts + `generateInsight()` + the strict prompt; UI migrated to the new + schema shape. +5. v1.4.17: user-feedback loop (`InsightFeedback` table, admin + `/admin/ai-quality` dashboard, prompt-revision workflow). +6. v1.4.17: per-recommendation confidence score (calibrated from the + feedback loop's ground-truth). +7. v1.5: optional "deep mode" with Codex reasoning summaries. + +## Open / deferred + +- Route-side wiring of `generateInsight()` — deferred because the + existing UI consumes a different schema shape; would have broken + the dashboard. v1.4.16 ships the consumer-side migration alongside + it. +- Wide-Event annotation reading `client.getLastDiagnostics()` — one- + line follow-up; left out of v1.4.15 to keep the AI-touching diff + scoped to provider/wrapper/schema/prompt/cache. + +## Build status + +``` +pnpm test → 1048 / 1048 pass +pnpm test:integration → 31 / 31 pass +pnpm typecheck → only pre-existing dashboard-layout.test.ts errors (A4-owned) +pnpm lint → 0 errors, 11 pre-existing warnings +``` + +All 6 commits on `origin/main`, each pushed cleanly after rebase-with- +autostash. No `--no-verify`, no `--no-gpg-sign`, no force pushes.