HealthLog/docs/audit/v1416-ai-roadmap.md

# v1.4.16+ AI hardening roadmap

Authored: 2026-05-09 during the v1.4.15 Stage C1 cycle.

Maintainer brief, verbatim 2026-05-09:

> "Es darf null Halluzinationen haben und es muss sich halt irgendwie
> stützen auf medizinische Dinge. Ich möchte da eine gewisse Dynamik
> drin haben."
>
> ("Zero hallucinations. Must ground on medical facts. I want some
> dynamism.")

v1.4.15 Stage C1 ships **infrastructure** — the schema, the wrapper,
the citation enforcement, the scope-hardened prompt, the slug-drift
defence. The medical-grounding work, the multi-provider redundancy,
the explainability layer and the feedback loop land in v1.4.16+.

This document is the iteration roadmap the maintainer asked for.

## What landed in v1.4.15 (recap)

| Deliverable                                        | Commit subject                                                                  |
| -------------------------------------------------- | ------------------------------------------------------------------------------- |
| `AIProvider` interface + `MockAIProvider` + tests  | `refactor(ai): consolidate providers behind AIProvider interface`               |
| Strict response schema + retry-once wrapper        | swept into `docs(audit): v1.4.15 empty-states audit + i18n keys` (sibling race) |
| Citation-from-data invariant (11 tests)            | `feat(ai): enforce citation-from-data on every recommendation`                  |
| Scope-hardened versioned prompt (`PROMPT_VERSION`) | `feat(ai): scope-hardened system prompt with refusal pattern`                   |
| Slug-drift defence (chain + cache)                 | `feat(ai): fallback-chain slug discovery with 1h positive cache`                |
| Spec `docs/codex-protocol-spec.md` §7b             | (rolled into the slug-drift commit)                                             |

NOT done in v1.4.15 (deliberately deferred):

- The `/api/insights/generate` route does NOT yet adopt
  `generateInsight()` — the existing UI consumes the rich legacy
  shape (`{summary, classification, findings, correlations,
dataQuality, ...}`). Migrating the consumer is v1.4.16 work because
  it requires UI changes to `<InsightAdvisorCard />` and
  `<InsightsCard />` to render the new strict shape.
- The scope-hardened prompt at `src/lib/ai/prompts/insight-generator.ts`
  is exported but not yet wired in. Same reason — the UI needs to
  consume `recommendations[]`-as-objects + `citations[]` + `warnings[]`
  before the prompt can be flipped.

## v1.4.16 — Medical-reference grounding

Goal: the model no longer cites "ESH/ESC 2024 generic guidance" verbatim
from the prompt. Instead it cites a small **embedded** quote from the
guideline document, with a rendered footnote in the UI.

### Subtasks

1. **Curate the reference set.** A versioned bundle in
   `src/lib/ai/references/` containing ~40 short excerpts:
   - ESH/ESC 2024 BP guidelines — adult target, hypertension stages,
     non-pharmacological measures.
   - AHA/ACC 2017 BP guidelines — alternative thresholds (USA-context).
   - WHO 2021 sleep / activity guidance.
   - Saint-Maurice et al. 2020 step-count dose-response.
   - DGE / DEGAM (German GP) lifestyle leaflets — already-translated
     plain language for the German UI.
   - DGE healthy-weight (BMI bands).
   - AASM 2015 sleep duration consensus.
     Each excerpt: `{ id, source, citation, excerpt, locale, validity }`.
     Curated by hand (NOT scraped) — no quote longer than 200 characters
     to stay inside fair-use.

2. **Inject relevant excerpts as system context per request.** A
   pre-flight step `selectRelevantReferences(features)` looks at the
   user's snapshot and selects 3-5 excerpts most relevant (BP elevated
   → ESH/ESC excerpts; weight gain → DGE BMI excerpts). Injected into
   the system prompt as a `## Relevant guidelines` block with each
   excerpt verbatim.

3. **Schema extension.** Each `recommendation[].metricSource` and each
   `citations[]` entry gains an optional `referenceId` field pointing
   into the bundle — the model uses an excerpt's `id` to back its
   citation. The UI then renders a footnote `[1] ESH/ESC 2024,
p. 17 — "..."` next to the recommendation.

4. **Hallucination guard for references.** Schema rejects any
   `referenceId` not in the bundle. The model cannot fabricate a
   citation pointing at a guideline that doesn't exist.

5. **Locale support.** German excerpts shipped first — the live
   tenant's primary locale. English fallback on missing.

Test pivot: a fixture user with elevated BP must surface ESH/ESC
excerpt #4 via the model's recommendation; a low-data user must NOT
surface anything (the prompt instructs an empty `recommendations[]`).

## v1.4.16 — Multi-provider redundancy

Goal: a single provider's outage no longer breaks insights generation.

### Subtasks

1. **`MultiProviderCascade` wrapper.** Implements `AIProvider`. Holds
   an ordered list of underlying providers (Codex, OpenAI, Anthropic,
   Local). Tries each in order; on transient error (5xx, 429,
   `slug_rejected_503` from the new chain), falls through to the
   next. On terminal-but-permission error (401, 403), surfaces
   verbatim — no fall-through (the user has to fix the credential).

2. **Selection policy in admin settings.** Admin can pick the cascade
   order in `/admin/integrations`. Default: user's chosen primary,
   then the admin OpenAI key, then any other connected per-user
   provider.

3. **Cost / latency budget.** Cascade only kicks in on hard error,
   never on schema-mismatch (the strict-schema retry already handles
   that). Per-request total wall-clock budget: 60 s.

4. **Telemetry.** Wide-Event `ai.cascade.attempts: ["codex","admin-key"]`
   - `ai.cascade.outcome: "success" | "all-failed"`.

## v1.4.16 — Per-recommendation explainability

Goal: every recommendation can answer "why did you tell me this?" in
one click.

Schema gains `recommendation[].rationale: { dataWindow, comparedTo,
deviation }`:

```ts
rationale: {
  dataWindow: "last7days",            // window the recommendation analysed
  comparedTo: "your last90days avg",  // baseline used for the call
  deviation: "+5 mmHg",               // magnitude of the divergence
}
```

UI: a small "Why?" chevron under each recommendation expands a one-
line tooltip: `"Your avg7 (78 bpm) is 5 bpm above your 90-day median
(73 bpm). That's clinically meaningful per ±5 bpm threshold (ESH/ESC
2024)."` — assembled from `rationale` + the linked reference excerpt.

## v1.4.17 — User-feedback loop

Goal: the system prompt ratchets over time based on user signals.

1. **Per-recommendation thumb-up / thumb-down** in the UI. Persisted
   in a new `InsightFeedback` table:

   ```
   id            uuid
   userId        uuid (FK)
   recommendationId   text  (the schema's stable id)
   recommendationText text  (snapshot — id is per-response, may be reused)
   helpful       bool
   createdAt     timestamp
   ```

2. **Aggregate dashboard for admin** under `/admin/ai-quality`. Shows
   helpful-rate per recommendation type (severity, metricSource.type),
   per locale, per provider type, per `PROMPT_VERSION`.

3. **Prompt revision workflow.** When a recommendation type drops
   below 60% helpful-rate over 100 ratings, an admin alert proposes
   a prompt revision — `PROMPT_VERSION` bumped, A/B-tested for one
   week, kept or reverted.

## v1.4.17 — Confidence score per recommendation

Goal: every recommendation gets a calibrated confidence score the UI
renders as a coloured pill (green ≥ 0.85, yellow 0.6-0.85, red < 0.6).

Schema: `recommendation[].confidence: number ∈ [0, 1]`. Threshold
calibration done from the v1.4.17 feedback loop's ground-truth ratings.

This is NOT the same as the legacy `dataQuality.confidence` enum (which
captures snapshot density, not per-recommendation confidence).

## v1.5 — Optional "deep mode" with reasoning

The Codex protocol supports `reasoning` summaries (§1a of the spec).
We currently emit `reasoning: null` to keep responses fast and cheap.
v1.5 introduces an opt-in "Deep analysis" toggle in Settings that
switches `reasoning: { effort: "medium", summary: "concise" }` on,
fetches the reasoning channel separately, and renders a collapsed
"Show reasoning" section under each insight. Reasoning content goes
into a separate `reasoning` schema field — never folded into
`summary`.

This is gated behind a per-user setting because reasoning costs ~5x
more tokens.

## Discoverability sanity-check

Before each iteration: re-fetch
`gh api repos/openai/codex/contents/codex-rs/models-manager/models.json`
and update `DEFAULT_SLUG_FALLBACK_CHAIN`. The chain was correct for
the 2026-05-09 allow-list snapshot; future drift is automatically
absorbed by the cache + walk, but the **default order** should still
match upstream's "default plan" preference so a freshly-deployed
instance never has to walk on its first request.

## What's NOT on the roadmap

- **Self-hosted fine-tuning.** Out of scope — HealthLog's user base
  doesn't justify training cost.
- **Image input.** Snapshots are numeric; image processing belongs to
  a separate "doctor-report scan" feature.
- **Conversational memory.** Insights are stateless per call. The
  multi-turn UX is owned by a future `/coach` route, not the
  insights generator.

---

Owner: maintainer + future Stage-C work. Cross-link from
`.planning/phase-C1-report.md` and the v1.4.15 release notes.