Files
HealthLog/src/lib/validations/external-id.ts
T
Marc-André Bombeck 15b5f6f92f fix(ingest): hold the stability floor for identities no request schema sees
The dedup key on the medication intake surfaces is not only `externalId`.
`MedicationIntakeEvent.idempotency_key` is a `@unique` column persisted
with the dose row forever, with no TTL and no sweep, matched by plain
equality on the per-medication POST, the bulk drain and the external
ingest bridge. That makes it an identity, not a replay token, and it
carries the same failure mode: a key that rotates per client launch never
matches its own earlier post, so every retry logs the dose again. The
blank case is sharper still — the column is unique table-wide, so an
empty key resolves to whichever unrelated row stored it first. Refuse
both shapes at the schema on the single-entry paths and per entry on the
bulk drain.

The MCP write path hashes a host-supplied key into the `externalId` half
of its dedup key, so the shape is laundered before anything downstream
could see it and the check has to run on the input. That closes a live
one: every tool handler falls back to `""` when the host omits the
argument, and `""` digests to a constant, so each keyless write after the
first reported `already_logged` and was silently dropped.

The HTTP `Idempotency-Key` header stays out, and now says why in the
guard: `IdempotencyKey.expiresAt` bounds it and the cleanup job sweeps
it, so it is a windowed replay token and rotation is its design.

Extend the structural guard with a provenance rule for identities that
never pass a Zod request schema — a spreadsheet column, an MCP tool
argument. Inside the import and MCP trees, a module handling an identity
must apply the floor itself or import a module that does. That is a real
chain rather than a list of known offenders: a new importer that mints an
`externalId` from file input and imports nothing checked fails it, with
no allowlist to escape into.
2026-07-25 18:07:13 +02:00

242 lines
10 KiB
TypeScript

/**
* Client-supplied external identifiers — stability floor.
*
* Every ingest surface that accepts an `externalId` uses it as one half of
* a dedup key: `(userId, source, externalId)`, `(userId, externalSource,
* externalId)`, `(userId, type, source, externalId)`. Re-posting the same
* triple is idempotent and returns the existing row. That idempotency is
* only as good as the STABILITY of the identifier — an id that changes
* between two runs of the same client never matches its own earlier row,
* so every sync sweep mints a fresh record and the user's history fills
* with phantom rows that carry no doses, no schedule, no follow-up data.
*
* That is not hypothetical. A native client serialised an opaque platform
* object with the language's default object description and sent the
* result as `externalId`. The default description of an object with no
* description contract is `<ClassName: 0xADDRESS>` — a per-allocation
* MEMORY ADDRESS. A live instance accumulated 23 duplicate medications in
* a single day, one per sync sweep, each with a different address in its
* id, and the dose events that should have attached to them resolved
* through the same rotating string and were never stored at all.
*
* This module is the one place that recognises an identifier which CANNOT
* be a stable identity. It is deliberately narrow: the cost of a false
* negative is a duplicate row, the cost of a false positive is a
* legitimate client losing the ability to sync at all. Only shapes that
* are provably not identities are rejected:
*
* 1. `blank` — empty or whitespace-only after trimming.
* Carries no identity by construction.
* 2. `pointer_address` — a bare `0x…` hex value and nothing else.
* A raw address, valid only inside the
* process that printed it.
* 3. `object_description` — an angle-bracketed description containing
* an `0x…` hex token, e.g. `<Foo: 0x12568db80>`
* or `<Foo: 0x1; bar=2>`. This is the general
* form of a default object description, not a
* match on one platform class: no identity
* format wraps itself in angle brackets around
* a hex pointer.
*
* Everything else passes. In particular the identifier shapes this system
* actually receives all survive:
*
* - UUIDs, upper or lower case, hyphenated or not
* (`8AD2A9CB-...`, HealthKit sample UUIDs, SwiftData row ids)
* - structured prefix forms
* (`stats:HKQuantityTypeIdentifierStepCount:2026-07-25`,
* `stats:HKQuantityTypeIdentifierHeartRate:2026-07-25T08:10:00Z`,
* `assessment:<id>`, `whoop:<id>`, `oura:<id>`)
* - opaque vendor identifiers (numeric ids, base64-ish blobs, hashes —
* including a 64-char SHA-256 hex digest, which has no `0x` prefix)
* - platform type identifiers (`HKQuantityTypeIdentifierStepCount`)
*
* There is deliberately NO rule on length, on the presence of spaces, on
* character class, or on "looks random enough". Those would reject real
* vendor identifiers, and a rule that cannot be justified against a shape
* this system actually receives does not belong here.
*/
import { z } from "zod/v4";
/** The closed set of shapes that cannot be a stable identity. */
export type UnstableExternalIdShape =
"blank" | "pointer_address" | "object_description";
/**
* A bare hex pointer and nothing else — `0x12568db80`. Anchored on both
* ends so an opaque vendor id that merely CONTAINS "0x" is untouched.
*/
const BARE_POINTER = /^0x[0-9a-f]+$/i;
/**
* An angle-bracketed description carrying a hex pointer, searched
* anywhere in the value so a client that prefixes its own namespace
* (`apple:<Foo: 0x1234>`) is caught too. `[^<>]*` keeps the bracket pair
* adjacent — the token has to be one description, not two unrelated
* angle brackets spanning half the string.
*/
const POINTER_DESCRIPTION = /<[^<>]*0x[0-9a-f]+[^<>]*>/i;
/**
* Classify an external identifier. Returns `null` when the value is
* usable as an identity, or the shape that disqualifies it.
*/
export function classifyExternalId(
value: string,
): UnstableExternalIdShape | null {
const trimmed = value.trim();
if (trimmed.length === 0) return "blank";
if (BARE_POINTER.test(trimmed)) return "pointer_address";
if (POINTER_DESCRIPTION.test(trimmed)) return "object_description";
return null;
}
/** Convenience predicate over {@link classifyExternalId}. */
export function isStableExternalId(value: string): boolean {
return classifyExternalId(value) === null;
}
/**
* How the offending field is named back to the caller. The rule is about
* the VALUE, so it applies to every client-supplied string this system
* later matches on as an identity — not only `externalId`.
*/
export const EXTERNAL_ID_LABEL = "external identifier";
export const IDEMPOTENCY_KEY_LABEL = "idempotency key";
/** The 422 message a client developer reads. Names the shape received. */
export function unstableExternalIdMessage(
shape: UnstableExternalIdShape,
label: string = EXTERNAL_ID_LABEL,
): string {
switch (shape) {
case "blank":
return `${label} must not be empty`;
case "pointer_address":
return `${label} must be stable across app restarts; received a memory address (0x…), which is a new value on every launch and mints a duplicate record on every sync`;
case "object_description":
return `${label} must be stable across app restarts; received an object description (<Class: 0x…>), which embeds a memory address and is a new value on every launch — serialise the identity itself, not the object's description`;
}
}
function assertStableIdentityField(
raw: unknown,
ctx: z.RefinementCtx,
field: string,
label: string,
): void {
if (typeof raw !== "string") return;
const shape = classifyExternalId(raw);
if (shape === null) return;
ctx.addIssue({
code: "custom",
path: [field],
message: unstableExternalIdMessage(shape, label),
});
}
/**
* Object-level Zod check for a single-entry request schema that carries a
* top-level `externalId`. Used as `schema.superRefine(assertStableExternalId)`
* so the refusal rides the standard 422 multi-issue envelope
* (`returnAllZodIssues` / `sanitiseZodIssues`) with `path: ["externalId"]`
* instead of a hand-rolled error.
*
* Batch / bulk surfaces deliberately do NOT use this: a schema-level
* refusal fails the whole payload, and one bad row must not stop the other
* 499 from landing. Those routes call {@link classifyExternalId} per entry
* and report the entry through their own `skipped` contract.
*/
export function assertStableExternalId(
value: { externalId?: string | null },
ctx: z.RefinementCtx,
): void {
assertStableIdentityField(
value.externalId,
ctx,
"externalId",
EXTERNAL_ID_LABEL,
);
}
/**
* The same floor for a client-supplied `idempotencyKey`.
*
* The HTTP `Idempotency-Key` header is a replay token inside a TTL and is
* deliberately out of scope — `IdempotencyKey.expiresAt` bounds it and a
* cron sweeps it. `MedicationIntakeEvent.idempotency_key` is NOT that: it
* is a `@unique` column persisted with the dose row forever, with no TTL
* and no sweep, matched by plain equality on the per-medication intake
* POST, the bulk intake ingest and the external ingest surface. That makes
* it an identity, and it carries the same failure mode — a value that
* rotates per client launch never matches its own earlier row, so every
* re-sync logs the dose again.
*
* The blank arm matters more here than anywhere else: the column is
* `@unique` across the table, so a second row carrying "" cannot exist,
* and an empty key would otherwise resolve to whichever unrelated row got
* there first.
*/
export function assertStableIdempotencyKey(
value: { idempotencyKey?: string | null },
ctx: z.RefinementCtx,
): void {
assertStableIdentityField(
value.idempotencyKey,
ctx,
"idempotencyKey",
IDEMPOTENCY_KEY_LABEL,
);
}
/**
* Read a top-level `externalId` off an unparsed request body and classify
* it. Single-entry routes use this on the 422 path to annotate the wide
* event with the SHAPE that was refused — never the id itself, which is
* client free-text and can carry anything.
*/
export function unstableExternalIdShape(
body: unknown,
field: "externalId" | "idempotencyKey" = "externalId",
): UnstableExternalIdShape | null {
if (typeof body !== "object" || body === null) return null;
const raw = (body as Record<string, unknown>)[field];
if (typeof raw !== "string") return null;
return classifyExternalId(raw);
}
/**
* Pinned wide-event meta for a refusal. Keys are fixed so dashboards stay
* stable; the payload carries the count, the classification, and the
* surface — never the identifier itself, which is client free-text.
*
* `external_id_shapes` is the sorted, de-duplicated shape set as a single
* comma-joined string — a scalar rather than an array so it groups
* cleanly in the event store.
*
* The surface rides the META rather than only the action name because a
* wide event carries exactly one action, and most of these refusals
* happen on a path that already names its own (`…​.validation-failed`,
* `measurement.batch.ingest`). Overwriting that action would cost more
* than it buys, so `external_id_rejected > 0` is the one query that finds
* every occurrence on every surface; routes whose failure path claims no
* action of its own additionally set `<surface>.external_id.rejected`.
*/
export function unstableExternalIdMeta(
surface: string,
shapes: UnstableExternalIdShape[],
): {
external_id_rejected: number;
external_id_shapes: string;
external_id_surface: string;
} {
return {
external_id_rejected: shapes.length,
external_id_shapes: [...new Set(shapes)].sort().join(","),
external_id_surface: surface,
};
}
/** Per-entry `reason` code every batch / bulk surface reports. */
export const UNSTABLE_EXTERNAL_ID_REASON = "unstable_external_id";