mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-07 12:22:03 +00:00
docs(audit): v1.4.16 AI hardening roadmap (medical grounding, multi-provider, explainability)
Phase C1 (v1.4.15) — closes the last acceptance criterion.
Marc, verbatim 2026-05-09: "können wir auch gerne in vielen
verschiedenen Iterationen weiter das ausarbeiten" ("happy to keep
iterating in many further passes"). This document is the iteration
roadmap that explicit invitation called for.
Captures everything intentionally deferred from v1.4.15:
v1.4.16
- Medical-reference grounding: hand-curated bundle of ~40 short
excerpts (ESH/ESC 2024, AHA, WHO, AASM, DGE/DEGAM); pre-flight
`selectRelevantReferences(features)` injects 3-5 into the
system prompt; schema extension `referenceId` rejects non-bundle
citations.
- Multi-provider redundancy: `MultiProviderCascade` wrapping the
`AIProvider` interface (already in place from this phase's
refactor commit). Per-tenant order in admin settings.
- Per-recommendation explainability: schema gains
`rationale.{dataWindow, comparedTo, deviation}`; UI renders a
"Why?" tooltip wired through to the linked reference excerpt.
- Route-side wiring: `/api/insights/generate` adopts
`generateInsight()` + the strict prompt; UI components migrated
to the new shape.
v1.4.17
- User-feedback loop: per-recommendation helpful / not-helpful;
`InsightFeedback` table; admin dashboard `/admin/ai-quality`
with helpful-rate by recommendation type, locale, provider,
`PROMPT_VERSION`. Ratchets the prompt revision workflow.
- Confidence score per recommendation: calibrated from the
feedback loop's ground-truth.
v1.5
- Optional "deep mode" with Codex reasoning summaries —
per-user toggle, separate `reasoning` schema field, never
folded into `summary` (5x token cost).
Also documents the operational rule: re-fetch
`models-manager/models.json` before each iteration to keep
`DEFAULT_SLUG_FALLBACK_CHAIN` aligned with the upstream allow-list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Marc-André BombeckandClaude Opus 4.7 committed
1 parent
4bba951121
commit
fa11f10bee
1 file changed
+211
@@ -0,0 +1,211 @@
|
||||
# v1.4.16+ AI hardening roadmap
|
||||
|
||||
Authored: 2026-05-09 during the v1.4.15 Phase C1 marathon.
|
||||
|
||||
Marc, verbatim 2026-05-09:
|
||||
|
||||
> "Es darf null Halluzinationen haben und es muss sich halt irgendwie
|
||||
> stützen auf medizinische Dinge. Ich möchte da eine gewisse Dynamik
|
||||
> drin haben."
|
||||
>
|
||||
> ("Zero hallucinations. Must ground on medical facts. I want some
|
||||
> dynamism.")
|
||||
|
||||
v1.4.15 Phase C1 ships **infrastructure** — the schema, the wrapper,
|
||||
the citation enforcement, the scope-hardened prompt, the slug-drift
|
||||
defence. The medical-grounding work, the multi-provider redundancy,
|
||||
the explainability layer and the feedback loop land in v1.4.16+.
|
||||
|
||||
This document is the iteration roadmap Marc explicitly invited.
|
||||
|
||||
## What landed in v1.4.15 (recap)
|
||||
|
||||
| Deliverable | Commit subject |
|
||||
| -------------------------------------------------- | ------------------------------------------------------------------------------- |
|
||||
| `AIProvider` interface + `MockAIProvider` + tests | `refactor(ai): consolidate providers behind AIProvider interface` |
|
||||
| Strict response schema + retry-once wrapper | swept into `docs(audit): v1.4.15 empty-states audit + i18n keys` (sibling race) |
|
||||
| Citation-from-data invariant (11 tests) | `feat(ai): enforce citation-from-data on every recommendation` |
|
||||
| Scope-hardened versioned prompt (`PROMPT_VERSION`) | `feat(ai): scope-hardened system prompt with refusal pattern` |
|
||||
| Slug-drift defence (chain + cache) | `feat(ai): fallback-chain slug discovery with 1h positive cache` |
|
||||
| Spec `docs/codex-protocol-spec.md` §7b | (rolled into the slug-drift commit) |
|
||||
|
||||
NOT done in v1.4.15 (deliberately deferred):
|
||||
|
||||
- The `/api/insights/generate` route does NOT yet adopt
|
||||
`generateInsight()` — the existing UI consumes the rich legacy
|
||||
shape (`{summary, classification, findings, correlations,
|
||||
dataQuality, ...}`). Migrating the consumer is v1.4.16 work because
|
||||
it requires UI changes to `<InsightAdvisorCard />` and
|
||||
`<InsightsCard />` to render the new strict shape.
|
||||
- The scope-hardened prompt at `src/lib/ai/prompts/insight-generator.ts`
|
||||
is exported but not yet wired in. Same reason — the UI needs to
|
||||
consume `recommendations[]`-as-objects + `citations[]` + `warnings[]`
|
||||
before the prompt can be flipped.
|
||||
|
||||
## v1.4.16 — Medical-reference grounding
|
||||
|
||||
Goal: the model no longer cites "ESH/ESC 2024 generic guidance" verbatim
|
||||
from the prompt. Instead it cites a small **embedded** quote from the
|
||||
guideline document, with a rendered footnote in the UI.
|
||||
|
||||
### Subtasks
|
||||
|
||||
1. **Curate the reference set.** A versioned bundle in
|
||||
`src/lib/ai/references/` containing ~40 short excerpts:
|
||||
- ESH/ESC 2024 BP guidelines — adult target, hypertension stages,
|
||||
non-pharmacological measures.
|
||||
- AHA/ACC 2017 BP guidelines — alternative thresholds (USA-context).
|
||||
- WHO 2021 sleep / activity guidance.
|
||||
- Saint-Maurice et al. 2020 step-count dose-response.
|
||||
- DGE / DEGAM (German GP) lifestyle leaflets — already-translated
|
||||
plain language for the German UI.
|
||||
- DGE healthy-weight (BMI bands).
|
||||
- AASM 2015 sleep duration consensus.
|
||||
Each excerpt: `{ id, source, citation, excerpt, locale, validity }`.
|
||||
Curated by hand (NOT scraped) — no quote longer than 200 characters
|
||||
to stay inside fair-use.
|
||||
|
||||
2. **Inject relevant excerpts as system context per request.** A
|
||||
pre-flight step `selectRelevantReferences(features)` looks at the
|
||||
user's snapshot and selects 3-5 excerpts most relevant (BP elevated
|
||||
→ ESH/ESC excerpts; weight gain → DGE BMI excerpts). Injected into
|
||||
the system prompt as a `## Relevant guidelines` block with each
|
||||
excerpt verbatim.
|
||||
|
||||
3. **Schema extension.** Each `recommendation[].metricSource` and each
|
||||
`citations[]` entry gains an optional `referenceId` field pointing
|
||||
into the bundle — the model uses an excerpt's `id` to back its
|
||||
citation. The UI then renders a footnote `[1] ESH/ESC 2024,
|
||||
p. 17 — "..."` next to the recommendation.
|
||||
|
||||
4. **Hallucination guard for references.** Schema rejects any
|
||||
`referenceId` not in the bundle. The model cannot fabricate a
|
||||
citation pointing at a guideline that doesn't exist.
|
||||
|
||||
5. **Locale support.** German excerpts shipped first — Marc's primary
|
||||
locale. English fallback on missing.
|
||||
|
||||
Test pivot: a fixture user with elevated BP must surface ESH/ESC
|
||||
excerpt #4 via the model's recommendation; a low-data user must NOT
|
||||
surface anything (the prompt instructs an empty `recommendations[]`).
|
||||
|
||||
## v1.4.16 — Multi-provider redundancy
|
||||
|
||||
Goal: a single provider's outage no longer breaks insights generation.
|
||||
|
||||
### Subtasks
|
||||
|
||||
1. **`MultiProviderCascade` wrapper.** Implements `AIProvider`. Holds
|
||||
an ordered list of underlying providers (Codex, OpenAI, Anthropic,
|
||||
Local). Tries each in order; on transient error (5xx, 429,
|
||||
`slug_rejected_503` from the new chain), falls through to the
|
||||
next. On terminal-but-permission error (401, 403), surfaces
|
||||
verbatim — no fall-through (the user has to fix the credential).
|
||||
|
||||
2. **Selection policy in admin settings.** Admin can pick the cascade
|
||||
order in `/admin/integrations`. Default: user's chosen primary,
|
||||
then the admin OpenAI key, then any other connected per-user
|
||||
provider.
|
||||
|
||||
3. **Cost / latency budget.** Cascade only kicks in on hard error,
|
||||
never on schema-mismatch (the strict-schema retry already handles
|
||||
that). Per-request total wall-clock budget: 60 s.
|
||||
|
||||
4. **Telemetry.** Wide-Event `ai.cascade.attempts: ["codex","admin-key"]`
|
||||
+ `ai.cascade.outcome: "success" | "all-failed"`.
|
||||
|
||||
## v1.4.16 — Per-recommendation explainability
|
||||
|
||||
Goal: every recommendation can answer "why did you tell me this?" in
|
||||
one click.
|
||||
|
||||
Schema gains `recommendation[].rationale: { dataWindow, comparedTo,
|
||||
deviation }`:
|
||||
|
||||
```ts
|
||||
rationale: {
|
||||
dataWindow: "last7days", // window the recommendation analysed
|
||||
comparedTo: "your last90days avg", // baseline used for the call
|
||||
deviation: "+5 mmHg", // magnitude of the divergence
|
||||
}
|
||||
```
|
||||
|
||||
UI: a small "Why?" chevron under each recommendation expands a one-
|
||||
line tooltip: `"Your avg7 (78 bpm) is 5 bpm above your 90-day median
|
||||
(73 bpm). That's clinically meaningful per ±5 bpm threshold (ESH/ESC
|
||||
2024)."` — assembled from `rationale` + the linked reference excerpt.
|
||||
|
||||
## v1.4.17 — User-feedback loop
|
||||
|
||||
Goal: the system prompt ratchets over time based on user signals.
|
||||
|
||||
1. **Per-recommendation thumb-up / thumb-down** in the UI. Persisted
|
||||
in a new `InsightFeedback` table:
|
||||
|
||||
```
|
||||
id uuid
|
||||
userId uuid (FK)
|
||||
recommendationId text (the schema's stable id)
|
||||
recommendationText text (snapshot — id is per-response, may be reused)
|
||||
helpful bool
|
||||
createdAt timestamp
|
||||
```
|
||||
|
||||
2. **Aggregate dashboard for admin** under `/admin/ai-quality`. Shows
|
||||
helpful-rate per recommendation type (severity, metricSource.type),
|
||||
per locale, per provider type, per `PROMPT_VERSION`.
|
||||
|
||||
3. **Prompt revision workflow.** When a recommendation type drops
|
||||
below 60% helpful-rate over 100 ratings, an admin alert proposes
|
||||
a prompt revision — `PROMPT_VERSION` bumped, A/B-tested for one
|
||||
week, kept or reverted.
|
||||
|
||||
## v1.4.17 — Confidence score per recommendation
|
||||
|
||||
Goal: every recommendation gets a calibrated confidence score the UI
|
||||
renders as a coloured pill (green ≥ 0.85, yellow 0.6-0.85, red < 0.6).
|
||||
|
||||
Schema: `recommendation[].confidence: number ∈ [0, 1]`. Threshold
|
||||
calibration done from the v1.4.17 feedback loop's ground-truth ratings.
|
||||
|
||||
This is NOT the same as the legacy `dataQuality.confidence` enum (which
|
||||
captures snapshot density, not per-recommendation confidence).
|
||||
|
||||
## v1.5 — Optional "deep mode" with reasoning
|
||||
|
||||
The Codex protocol supports `reasoning` summaries (§1a of the spec).
|
||||
We currently emit `reasoning: null` to keep responses fast and cheap.
|
||||
v1.5 introduces an opt-in "Deep analysis" toggle in Settings that
|
||||
switches `reasoning: { effort: "medium", summary: "concise" }` on,
|
||||
fetches the reasoning channel separately, and renders a collapsed
|
||||
"Show reasoning" section under each insight. Reasoning content goes
|
||||
into a separate `reasoning` schema field — never folded into
|
||||
`summary`.
|
||||
|
||||
This is gated behind a per-user setting because reasoning costs ~5x
|
||||
more tokens.
|
||||
|
||||
## Discoverability sanity-check
|
||||
|
||||
Before each iteration: re-fetch
|
||||
`gh api repos/openai/codex/contents/codex-rs/models-manager/models.json`
|
||||
and update `DEFAULT_SLUG_FALLBACK_CHAIN`. The chain was correct for
|
||||
the 2026-05-09 allow-list snapshot; future drift is automatically
|
||||
absorbed by the cache + walk, but the **default order** should still
|
||||
match upstream's "default plan" preference so a freshly-deployed
|
||||
instance never has to walk on its first request.
|
||||
|
||||
## What's NOT on the roadmap
|
||||
|
||||
- **Self-hosted fine-tuning.** Out of scope — HealthLog's user base
|
||||
doesn't justify training cost.
|
||||
- **Image input.** Snapshots are numeric; image processing belongs to
|
||||
a separate "doctor-report scan" feature.
|
||||
- **Conversational memory.** Insights are stateless per call. The
|
||||
multi-turn UX is owned by a future `/coach` route, not the
|
||||
insights generator.
|
||||
|
||||
---
|
||||
|
||||
Owner: Marc + future-Phase-C agent. Cross-link from
|
||||
`.planning/phase-C1-report.md` and the v1.4.15 release notes.
|
||||
Reference in new issue
Block a user