fix(insights): stripChartTokens covers capitalised Metric + orphan enum patterns + Apple Health additives

Marc reported a raw "Metric Pressure_Sys" token leak on /insights
2026-05-14. The v1.4.22 stripper only matched the canonical
`metric:<TYPE>` colon form; the model emitted a capitalised "Metric"
prefix followed by a Pascal/snake-cased identifier and the substring
sailed through to the rendered prose unchanged.

Extend `stripChartTokens()` with two additional matchers:

- `Metric <Word>` capitalised two-word phrase (catches the exact
  pattern Marc saw, plus PascalCase variants and the v1.4.23
  Apple-Health additions like `HEART_RATE_VARIABILITY`).
- Fixed-list orphan upper-snake-case enum identifiers
  (`BLOOD_PRESSURE_SYS`, `PULSE_BPM`, `MOOD_SCORE`,
  `MEDICATION_COMPLIANCE_PCT`, plus the v1.4.23 Apple Health enum
  surface: `HEART_RATE_VARIABILITY`, `RESTING_HEART_RATE`,
  `ACTIVE_ENERGY_BURNED`, `FLIGHTS_CLIMBED`,
  `WALKING_RUNNING_DISTANCE`, `VO2_MAX`, `BODY_TEMPERATURE`,
  `SLEEP_DURATION`).

The orphan-enum list is fixed (not a generic `[A-Z_]+`) so single
capitalised words like `WEIGHT` or `MOOD` stay legible in normal
prose — only the suffixed/database-style enum names get cleaved.
Lowercase "metric" inside legitimate English prose ("Each metric
tells part of the story") is also left untouched.

Tests: 14 new cases covering every documented leak pattern plus two
defence-in-depth assertions (combined leaks; German "Metrik" safe).
This commit is contained in:
Marc-André Bombeck committed 2026-05-14 12:44:47 +02:00
1 parent 5d3a4c0b13
commit 5deed3f92c
2 files changed
+176

No files matched your search

@@ -70,6 +70,124 @@ describe("stripChartTokens", () => {
});
});
// v1.4.25 W5b — Marc reported the raw "Metric Pressure_Sys" leak on
// /insights 2026-05-14. The v1.4.22 W1a regex only caught the canonical
// `metric:<TYPE>` colon form; the model emitted a capitalised "Metric"
// word followed by a Pascal/enum-cased identifier. The fix extends the
// stripper with two additional matchers: (1) `Metric <Word>` capitalised
// phrase, (2) bare upper-snake-case enum identifiers from the
// MeasurementType list (including the v1.4.23 Apple-Health additions).
describe("stripChartTokens — v1.4.25 W5b capitalised + orphan-enum leaks", () => {
it("strips the capitalised `Metric Pressure_Sys` phrase Marc reported", () => {
expect(
stripChartTokens("Your Metric Pressure_Sys is sitting above 140."),
).toBe("Your is sitting above 140.");
});
it("strips the capitalised form with snake_case enum body", () => {
expect(
stripChartTokens("trend points up for Metric BLOOD_PRESSURE_SYS today"),
).toBe("trend points up for today");
});
it("strips orphan enum mentions in prose — BLOOD_PRESSURE_SYS", () => {
expect(
stripChartTokens("Your BLOOD_PRESSURE_SYS reading is elevated."),
).toBe("Your reading is elevated.");
});
it("strips orphan enum mentions — WEIGHT bare upper-snake-case is left alone", () => {
// The bare word `WEIGHT` is shared with the user-facing metric
// label, so the orphan-enum stripper deliberately allowlists it
// against the user-facing label. Only suffixed enums (e.g.
// BLOOD_PRESSURE_SYS) get cleaved.
expect(stripChartTokens("Your WEIGHT trend is stable.")).toBe(
"Your WEIGHT trend is stable.",
);
});
it("strips orphan enum mentions — PULSE_BPM", () => {
expect(stripChartTokens("PULSE_BPM stays within target.")).toBe(
"stays within target.",
);
});
it("strips orphan enum mentions — MOOD_SCORE", () => {
expect(stripChartTokens("MOOD_SCORE drifted down this week.")).toBe(
"drifted down this week.",
);
});
it("strips orphan enum mentions — MEDICATION_COMPLIANCE_PCT", () => {
expect(
stripChartTokens("MEDICATION_COMPLIANCE_PCT held above 90."),
).toBe("held above 90.");
});
it("strips v1.4.23 Apple Health additions — HEART_RATE_VARIABILITY", () => {
expect(
stripChartTokens("Last week HEART_RATE_VARIABILITY trended up."),
).toBe("Last week trended up.");
});
it("strips v1.4.23 Apple Health additions — RESTING_HEART_RATE", () => {
expect(
stripChartTokens("Your RESTING_HEART_RATE is stable around 58 bpm."),
).toBe("Your is stable around 58 bpm.");
});
it("strips v1.4.23 Apple Health additions — ACTIVE_ENERGY_BURNED", () => {
expect(
stripChartTokens("ACTIVE_ENERGY_BURNED has been below target."),
).toBe("has been below target.");
});
it("strips v1.4.23 Apple Health additions — FLIGHTS_CLIMBED, WALKING_RUNNING_DISTANCE, VO2_MAX, BODY_TEMPERATURE, SLEEP_DURATION", () => {
expect(stripChartTokens("FLIGHTS_CLIMBED at 5/day.")).toBe("at 5/day.");
expect(
stripChartTokens("WALKING_RUNNING_DISTANCE averaged 4 km."),
).toBe("averaged 4 km.");
expect(stripChartTokens("VO2_MAX has crept up.")).toBe("has crept up.");
expect(
stripChartTokens("BODY_TEMPERATURE sat at 36.8 most days."),
).toBe("sat at 36.8 most days.");
expect(stripChartTokens("SLEEP_DURATION averaged 7.4h.")).toBe(
"averaged 7.4h.",
);
});
it("strips the Metric phrase + the enum together (defence in depth)", () => {
// The model can emit both forms in the same sentence — the
// capitalised phrase AND the bare enum identifier. Both pattern
// matchers run independently so each cleaves its own substring.
expect(
stripChartTokens(
"Metric BLOOD_PRESSURE_SYS shows your BLOOD_PRESSURE_SYS is elevated.",
),
).toBe("shows your is elevated.");
});
it("does not strip ordinary prose that incidentally contains 'metric' as a word", () => {
// Lower-case word "metric" without the colon form must not be
// touched — it's a perfectly legitimate English word inside copy
// like "Each metric in the snapshot ...". Marc's directive
// explicitly targets the leak patterns, not legitimate prose.
expect(
stripChartTokens("Each metric tells part of the story."),
).toBe("Each metric tells part of the story.");
});
it("does not strip the canonical Metrik token in German prose", () => {
// German prose uses "Metrik" (capitalised noun) freely; the
// English-targeted `Metric <Word>` matcher must not bleed into
// German copy. The regex is anchored to the literal English
// word "Metric" + whitespace + identifier so "Metrik" is safe.
expect(stripChartTokens("Jede Metrik liefert ein Stück.")).toBe(
"Jede Metrik liefert ein Stück.",
);
});
});
describe("parseChartTokens — defensive (v1.4.17)", () => {
it("returns empty array for undefined input", () => {
expect(parseChartTokens(undefined)).toEqual([]);
+58
View File
@@ -64,6 +64,61 @@ const ALLOWED_SET = new Set<string>(ALLOWED_CHART_TOKENS);
const STRIP_TOKEN_REGEX = /metric:[A-Za-z0-9_]+/g;
const PARSE_TOKEN_REGEX = /metric:[A-Z_]+/g;
// v1.4.25 W5b — capitalised-Metric form. Marc reported the leak
// pattern `Metric Pressure_Sys` on /insights 2026-05-14: the model
// emitted the metric reference as a two-word phrase ("Metric" + space
// + enum-cased identifier) instead of the canonical `metric:<TYPE>`
// colon form. The colon-form regex above never matched this so the
// raw enum surfaced in the rendered prose.
//
// Match the capitalised "Metric" prefix followed by a space and one
// of: an upper-snake-case enum (`BLOOD_PRESSURE_SYS`) or a
// PascalCase-with-underscore form (`Pressure_Sys`). The character
// class allows mixed-case + digits + underscores so v1.4.23
// Apple-Health additions (`HEART_RATE_VARIABILITY`,
// `WALKING_RUNNING_DISTANCE`, etc.) get cleaved cleanly too. Word
// boundary `\b` on both ends prevents accidental matches inside
// ordinary prose ("Metricated", "submetric_x").
const METRIC_WORD_REGEX = /\bMetric\s+[A-Za-z][A-Za-z0-9_]*\b/g;
// v1.4.25 W5b — orphan enum-identifier leaks. The model sometimes
// drops the bare MeasurementType enum name straight into prose
// ("Your BLOOD_PRESSURE_SYS is elevated"). This was harmless when
// the only enums were short capitalised words but leaks badly with
// the v1.4.23 Apple-Health additions because the resulting prose
// reads as a database field name. Match the full enum list as a
// fixed alternation so we never accidentally cleave a real word
// (e.g. "MOOD") out of legitimate prose unless the model wrote it
// as the raw enum in upper-snake form.
const ORPHAN_ENUMS = [
"BLOOD_PRESSURE_SYS",
"BLOOD_PRESSURE_DIA",
"PULSE_BPM",
"MOOD_SCORE",
"MEDICATION_COMPLIANCE_PCT",
// v1.4.23 Apple Health additions — see ALLOWED_CHART_TOKENS above.
"HEART_RATE_VARIABILITY",
"RESTING_HEART_RATE",
"ACTIVE_ENERGY_BURNED",
"FLIGHTS_CLIMBED",
"WALKING_RUNNING_DISTANCE",
"VO2_MAX",
"BODY_TEMPERATURE",
"SLEEP_DURATION",
] as const;
// `\b` boundaries keep ordinary English prose untouched — "weight"
// (lowercase) or "BMI" (no underscore) never match. The fixed list
// also means we never accidentally strip a single capitalised word
// like "MOOD" or "WEIGHT" that has legitimate use as an in-prose
// metric label in display copy. Marc's directive (2026-05-14) is to
// strip ONLY upper-snake / suffixed enum identifiers, not the
// canonical user-facing labels.
const ORPHAN_ENUM_REGEX = new RegExp(
`\\b(?:${ORPHAN_ENUMS.join("|")})\\b`,
"g",
);
/**
* Find every well-formed chart token in `text` and return only those that
* are in the allowlist. Hallucinated tokens (e.g. `metric:NUKE`) are
@@ -98,6 +153,9 @@ export function stripChartTokens(text: string | null | undefined): string {
if (typeof text !== "string") return "";
return text
.replace(STRIP_TOKEN_REGEX, "")
.replace(METRIC_WORD_REGEX, "")
.replace(ORPHAN_ENUM_REGEX, "")
.replace(/\s+([.,;:!?])/g, "$1")
.replace(/\s{2,}/g, " ")
.trim();
}