fix(ingest): refuse an external id that cannot be a stable identity

An external id is half of every ingest dedup key — (userId, source,
externalId), (userId, externalSource, externalId), (userId, type,
source, externalId). Re-posting the same triple is idempotent and
returns the existing row, which holds only while the id is stable
across client launches. A client that serialised an opaque platform
object with its default object description sent a per-allocation
memory address instead, so no sweep ever matched its own earlier row
and each one minted a fresh record: a live instance grew 23 phantom
medications in a day, none of which could collect a dose, because the
dose side resolved through the same rotating string.

Add one shared classifier for identifiers that provably cannot be an
identity — an angle-bracketed object description carrying a hex
pointer, a bare 0x address, and a blank value — and wire it at every
surface that accepts a client-supplied externalId.

Single-entry creates refuse through the schema, so the client gets the
standard 422 multi-issue envelope with a message naming the shape it
sent and what to send instead. Batch and bulk surfaces classify per
entry and report the offending row as skipped with reason
unstable_external_id, leaving the rest of the payload to land: one bad
row must not stop a sync. Every refusal carries pinned wide-event meta
(count, shape, surface) and never the id itself.

A structural guard freezes both routes into the floor: a request-side
externalId declaration either applies the field check or carries a
marker naming the file that checks it per entry, and the guard verifies
that file really does. The backup-restore payload is the one exemption,
with its reason recorded — it replays ids the server itself wrote.
This commit is contained in:
Marc-André Bombeck committed 2026-07-25 17:21:00 +02:00
1 parent ef9891fec5
commit 2ab66ab2e3
26 files changed
+1232 -64

No files matched your search

+15 -3
View File
@@ -12091,6 +12091,10 @@ components:
{00,10,20,30,40,50}, trailing `Z` (e.g. `2026-06-21T14:10:00.000Z`). A re-post OVERWRITES the row and
returns `updated`, so iOS uploads ~144 rows/day instead of one per raw HR sample. A row that targets this
prefix with a malformed bucket-start suffix is `skipped` with reason `malformed_hr_bucket_id`.
The id MUST be stable across app restarts. An object description or memory address (`<Class: 0x…>`,
`0x12ab34cd`) is a new value on every launch, so it can never match its own earlier row; such an entry is
`skipped` with reason `unstable_external_id` while the rest of the batch still lands.
externalSourceVersion:
type: string
minLength: 1
@@ -12294,7 +12298,10 @@ components:
type: string
const: APPLE_HEALTH
externalId:
description: The HealthKit dose-event UUID. Drives the idempotent re-sync dedup (first-write-wins per Apple dose).
description: "The HealthKit dose-event UUID. Drives the idempotent re-sync dedup (first-write-wins per Apple dose). Must
be stable across app restarts — an object description or memory address (`<Class: 0x…>`, `0x12ab34cd`) is a
new value on every launch; such an entry is `skipped` with reason `unstable_external_id` while the rest of
the batch still lands."
type: string
minLength: 1
maxLength: 128
@@ -12411,7 +12418,9 @@ components:
externalId:
description: "Stable external identifier of the mirrored medication (the HealthKit `medicationConceptIdentifier`, max
128 chars). Must be supplied together with `externalSource`. Idempotent: re-posting the same pair returns
the existing medication with status 200 instead of minting a duplicate."
the existing medication with status 200 instead of minting a duplicate. The value MUST be identical across
app restarts: an object description or memory address (`<Class: 0x…>`, `0x12ab34cd`) is a new value on every
launch, never matches its own earlier row, and is refused with 422 rather than minting a duplicate mirror."
type: string
minLength: 1
maxLength: 128
@@ -13667,7 +13676,10 @@ components:
- TELEGRAM
- DAYLIO
externalId:
description: iOS-side source-stable id for idempotent dedup on the `(userId, source, externalId)` key.
description: "iOS-side source-stable id for idempotent dedup on the `(userId, source, externalId)` key. Must be stable
across app restarts — an object description or memory address (`<Class: 0x…>`, `0x12ab34cd`) is a new value
on every launch; such an entry is `skipped` with reason `unstable_external_id` while the rest of the batch
still lands."
type: string
minLength: 1
maxLength: 120
@@ -0,0 +1,174 @@
/**
* Structural guard on the external-id stability floor.
*
* A client-supplied `externalId` is half of every ingest dedup key. An id
* that rotates between client launches never matches its own earlier row,
* so each sync sweep mints a fresh record — the failure mode that filled a
* live instance with phantom medications, one per sweep, none of which
* could ever collect a dose.
*
* The fix is only durable if it cannot be bypassed by the NEXT route.
* These guards freeze the two legal ways to accept the field:
*
* 1. the request schema routes it through `assertStableExternalId`, so
* the refusal rides the standard 422 multi-issue envelope; or
* 2. the field is deliberately permissive because a schema-level refusal
* would fail a whole batch on one bad row — in which case the
* declaration carries an `@external-id-checked-per-entry: <path>`
* marker naming the file that runs `classifyExternalId` per entry,
* and this guard verifies that file really does.
*
* They are tripwires, not proofs. A reviewer who waves through a bad
* addition defeats them, and no test substitutes for that review.
*/
import { describe, it, expect } from "vitest";
import { readFileSync } from "node:fs";
import { join, sep } from "node:path";
import { globSync } from "node:fs";
const SRC = join(process.cwd(), "src");
const VALIDATOR_MODULE = "@/lib/validations/external-id";
function sourceFiles(): string[] {
return globSync("**/*.{ts,tsx}", { cwd: SRC })
.filter(
(p) => !p.startsWith(`generated${sep}`) && !p.startsWith("generated/"),
)
.filter((p) => !p.includes("__tests__"))
.filter((p) => !p.endsWith(".test.ts") && !p.endsWith(".test.tsx"))
.map((p) => p.split(sep).join("/"))
.sort();
}
function read(rel: string): string {
return readFileSync(join(SRC, rel), "utf8");
}
/**
* A REQUEST-side declaration of a client-supplied external identifier:
* `externalId: z.…` / `externalSourceId: z.…` in a Zod object shape.
*
* Response DTOs under `lib/openapi/` are excluded below — those echo an
* id the server already stored and validating them would be validating
* our own output.
*/
const DECLARATION = /^\s*(externalId|externalSourceId)\s*:\s*z\b/gm;
/** `// @external-id-checked-per-entry: <path under repo root>` */
const MARKER = /@external-id-checked-per-entry:\s*(\S+)/;
/**
* Files that declare the field but are deliberately outside the floor.
* Each entry states WHY; an addition here is a decision, not a shortcut.
*/
const EXEMPT: Record<string, string> = {
// Backup restore reads a payload the SERVER itself wrote and encrypted.
// It has to reproduce the stored rows byte-for-byte — including ids that
// predate this floor — or a self-hoster with an already-poisoned export
// could never restore it. Nothing client-authored reaches this schema.
"lib/validations/backup.ts":
"restore payload is a server-written backup; refusing a stored id would block a restore",
};
/** Lines of `rel` (1-based) on which a request-side declaration appears. */
function declarationLines(rel: string): number[] {
const source = read(rel);
const lines: number[] = [];
DECLARATION.lastIndex = 0;
let m: RegExpExecArray | null;
while ((m = DECLARATION.exec(source)) !== null) {
lines.push(source.slice(0, m.index).split("\n").length);
}
return lines;
}
function declaringFiles(): string[] {
return sourceFiles()
.filter((rel) => !rel.startsWith("lib/openapi/"))
.filter((rel) => declarationLines(rel).length > 0);
}
describe("every client-supplied external id routes through the stability floor", () => {
it("finds the declaring files at all (the matcher still matches)", () => {
// A guard whose matcher silently stops matching passes forever. Pin the
// known set so a rename of the field or of the Zod idiom trips here.
const files = declaringFiles();
expect(files.length).toBeGreaterThanOrEqual(8);
expect(files).toContain("app/api/measurements/batch/route.ts");
expect(files).toContain("lib/validations/medication/create-update.ts");
});
it.each(declaringFiles())(
"%s either applies the field check or names the per-entry checker",
(rel) => {
if (rel in EXEMPT) {
expect(EXEMPT[rel].length).toBeGreaterThan(20);
return;
}
const source = read(rel);
const usesFieldCheck =
source.includes(VALIDATOR_MODULE) &&
source.includes("assertStableExternalId");
const marksPerEntry = MARKER.test(source);
expect(
usesFieldCheck || marksPerEntry,
`${rel} declares a client-supplied external id but neither applies ` +
"`assertStableExternalId` to its request schema nor marks the declaration " +
"`@external-id-checked-per-entry: <path>` naming the route that checks it per entry.",
).toBe(true);
},
);
it.each(declaringFiles())(
"%s — every per-entry marker points at a file that really checks per entry",
(rel) => {
if (rel in EXEMPT) return;
const source = read(rel);
MARKER.lastIndex = 0;
const targets = [...source.matchAll(new RegExp(MARKER, "g"))].map(
(m) => m[1],
);
for (const target of targets) {
expect(
target.startsWith("src/"),
`${rel}: marker target "${target}" must be a repo-root-relative src/ path`,
).toBe(true);
const checker = read(target.replace(/^src\//, ""));
expect(
checker.includes("classifyExternalId"),
`${rel} points its per-entry marker at ${target}, but that file never calls classifyExternalId.`,
).toBe(true);
}
},
);
});
describe("the batch surfaces refuse per entry, never whole-batch", () => {
// Each of these drains a client sync queue. A schema-level refusal here
// would stop a whole batch of good rows because of one bad one, which is
// worse than the duplicate it prevents.
const BATCH_ROUTES = [
"app/api/measurements/batch/route.ts",
"app/api/mood-entries/bulk/route.ts",
"app/api/medications/intake/bulk/route.ts",
"app/api/workouts/batch/route.ts",
"app/api/cycle/day-logs/bulk/route.ts",
"app/api/import/route.ts",
];
it.each(BATCH_ROUTES)("%s classifies each entry itself", (rel) => {
const source = read(rel);
expect(source).toContain("classifyExternalId");
expect(source).toContain("unstableExternalIdMeta");
});
it.each(BATCH_ROUTES.filter((r) => r !== "app/api/import/route.ts"))(
"%s reports the shared per-entry reason code",
(rel) => {
// The JSON importer has no per-entry status envelope — it reports a
// skipped COUNT — so it is the one batch surface without the code.
expect(read(rel)).toContain("UNSTABLE_EXTERNAL_ID_REASON");
},
);
});
+33
View File
@@ -32,6 +32,12 @@ import {
cycleBulkSchema,
MAX_CYCLE_BULK_ENTRIES,
} from "@/lib/validations/cycle";
import {
classifyExternalId,
unstableExternalIdMeta,
UNSTABLE_EXTERNAL_ID_REASON,
type UnstableExternalIdShape,
} from "@/lib/validations/external-id";
import { upsertCycleDayLog } from "@/lib/cycle/day-log-write";
import { DEFAULT_TIMEZONE } from "@/lib/mood/date-key";
@@ -121,9 +127,33 @@ async function postBulk(request: NextRequest): Promise<Response> {
let updated = 0;
let duplicates = 0;
let skipped = 0;
const unstableShapes: UnstableExternalIdShape[] = [];
for (let i = 0; i < entries.length; i++) {
const entry = entries[i];
// The `(userId, source, externalId)` upsert key is only idempotent
// while the id is STABLE across client launches. An id that rotates
// per process (an object description carrying a memory address)
// never matches its own earlier row, so every re-post mints another
// day log. Refused per entry — one bad row must not stop the rest of
// the batch landing.
const unstable =
entry.externalId === undefined
? null
: classifyExternalId(entry.externalId);
if (unstable) {
unstableShapes.push(unstable);
skipped += 1;
results.push({
index: i,
status: "skipped",
reason: UNSTABLE_EXTERNAL_ID_REASON,
externalId: entry.externalId,
});
continue;
}
try {
const cycleId = owningCycleId(entry.date);
const r = await upsertCycleDayLog(
@@ -206,6 +236,9 @@ async function postBulk(request: NextRequest): Promise<Response> {
updated,
duplicates,
skipped,
...(unstableShapes.length > 0
? unstableExternalIdMeta("cycle.day_log.bulk", unstableShapes)
: {}),
},
});
+13 -3
View File
@@ -27,9 +27,13 @@ import {
import { withIdempotency } from "@/lib/idempotency";
import { requireCycleEnabled } from "@/lib/cycle/gate";
import {
cycleDayLogInputSchema,
cycleDayLogCreateSchema,
cycleDayLogQuerySchema,
} from "@/lib/validations/cycle";
import {
unstableExternalIdMeta,
unstableExternalIdShape,
} from "@/lib/validations/external-id";
import { upsertCycleDayLog } from "@/lib/cycle/day-log-write";
import { findOwningCycleId } from "@/lib/cycle/cycle-attribution";
import { toCycleDayLogDTO, dayLogSymptomInclude } from "@/lib/cycle/dto";
@@ -82,11 +86,17 @@ async function postDayLog(request: NextRequest): Promise<Response> {
});
if (jsonError) return jsonError;
const parsed = cycleDayLogInputSchema.safeParse(rawBody);
const parsed = cycleDayLogCreateSchema.safeParse(rawBody);
if (!parsed.success) {
const shape = unstableExternalIdShape(rawBody);
annotate({
action: { name: "cycle.day-log.validation-failed" },
meta: { issue_count: parsed.error.issues.length },
meta: {
issue_count: parsed.error.issues.length,
...(shape
? unstableExternalIdMeta("cycle.day_log.create", [shape])
: {}),
},
});
return returnAllZodIssues(parsed.error, 422, {
errorCode: "cycle.day-log.invalid",
+9 -1
View File
@@ -24,6 +24,10 @@ import {
import { requireCycleEnabled } from "@/lib/cycle/gate";
import { withIdempotency } from "@/lib/idempotency";
import { cyclePeriodSchema } from "@/lib/validations/cycle";
import {
unstableExternalIdMeta,
unstableExternalIdShape,
} from "@/lib/validations/external-id";
import { upsertCycleDayLog } from "@/lib/cycle/day-log-write";
import type { FlowLevel } from "@/lib/cycle/types";
import {
@@ -49,9 +53,13 @@ async function postPeriod(request: NextRequest): Promise<Response> {
const parsed = cyclePeriodSchema.safeParse(rawBody);
if (!parsed.success) {
const shape = unstableExternalIdShape(rawBody);
annotate({
action: { name: "cycle.period.validation-failed" },
meta: { issue_count: parsed.error.issues.length },
meta: {
issue_count: parsed.error.issues.length,
...(shape ? unstableExternalIdMeta("cycle.period", [shape]) : {}),
},
});
return returnAllZodIssues(parsed.error, 422, {
errorCode: "cycle.period.invalid",
+42 -1
View File
@@ -23,6 +23,11 @@ import {
isPlausibleEntryInstant,
validateEntryInstant,
} from "@/lib/validations/entry-instant";
import {
classifyExternalId,
unstableExternalIdMeta,
type UnstableExternalIdShape,
} from "@/lib/validations/external-id";
import { encryptNote } from "@/lib/crypto/note-cipher";
import { recomputeUserMoodRollups } from "@/lib/rollups/mood-rollups";
import {
@@ -60,6 +65,10 @@ const measurementSchema = z
// export is idempotent rather than minting a duplicate set — mirrors the
// mood path. Absent → first-write-wins create with a NULL externalId
// (distinct in the unique key), which DOES duplicate on re-import.
// Stability is enforced per entry in the import loop, not at the
// field: a schema-level refusal would fail the whole file on one bad
// row.
// @external-id-checked-per-entry: src/app/api/import/route.ts
externalId: z.string().min(1).max(120).optional(),
})
.refine(
@@ -92,6 +101,10 @@ const moodEntrySchema = z
// present, the import upserts on `(userId, source, externalId)` so a
// re-import of the same export is idempotent rather than minting a
// duplicate; absent → the legacy first-write-wins create path.
// Stability is enforced per entry in the import loop, not at the
// field: a schema-level refusal would fail the whole file on one bad
// row.
// @external-id-checked-per-entry: src/app/api/import/route.ts
externalId: z.string().min(1).max(120).optional(),
})
.superRefine((entry, ctx) => {
@@ -170,6 +183,7 @@ export const POST = apiHandler(async (request: NextRequest) => {
const userId = user.id;
const data = parsed.data;
const stats = { measurements: 0, moodEntries: 0, skipped: 0 };
const unstableShapes: UnstableExternalIdShape[] = [];
let writeFailed = false;
// Import measurements
@@ -189,6 +203,17 @@ export const POST = apiHandler(async (request: NextRequest) => {
}> = [];
if (data.measurements?.length) {
for (const m of data.measurements) {
// The `(userId, type, source, externalId)` upsert key is only
// idempotent while the id is STABLE; an id that rotates between
// exports re-imports as a fresh row every time. Skipped per entry,
// like every other rejected row in this importer.
const unstable =
m.externalId === undefined ? null : classifyExternalId(m.externalId);
if (unstable) {
unstableShapes.push(unstable);
stats.skipped++;
continue;
}
try {
// `measuredAt` is already a `Date` (validateEntryInstant transform).
const measuredAt = m.measuredAt;
@@ -299,6 +324,15 @@ export const POST = apiHandler(async (request: NextRequest) => {
const fallbackOccurrences = new Map<string, number>();
const reservedFallbackInstants = new Set<number>();
for (const e of data.moodEntries) {
// Same stability floor as the measurement loop above — the mood
// upsert keys on `(userId, source, externalId)`.
const unstableMood =
e.externalId === undefined ? null : classifyExternalId(e.externalId);
if (unstableMood) {
unstableShapes.push(unstableMood);
stats.skipped++;
continue;
}
const identity = e.loggedAt ? null : fallbackMoodIdentity(e);
const occurrence =
identity === null ? 0 : (fallbackOccurrences.get(identity) ?? 0);
@@ -455,7 +489,14 @@ export const POST = apiHandler(async (request: NextRequest) => {
}
}
annotate({ meta: { import_stats: stats } });
annotate({
meta: {
import_stats: stats,
...(unstableShapes.length > 0
? unstableExternalIdMeta("import.json", unstableShapes)
: {}),
},
});
try {
await auditLog("import.upload", {
@@ -0,0 +1,239 @@
/**
* `POST /api/measurements/batch` — external-id stability floor.
*
* `(user_id, type, source, external_id)` is idempotent only while the id
* is stable across client launches. An id that rotates per process — an
* object description carrying a memory address — never matches its own
* earlier row, so every sync sweep mints a fresh measurement.
*
* The refusal is PER ENTRY, matching the range / timestamp guards on this
* route: a client draining a HealthKit queue must not lose 499 good rows
* because one carried a bad id.
*/
import { describe, expect, it, vi, beforeEach } from "vitest";
import { NextRequest } from "next/server";
vi.mock("@/lib/db", () => ({
prisma: {
user: { update: vi.fn() },
measurement: {
findMany: vi.fn(),
createManyAndReturn: vi.fn(),
updateMany: vi.fn(),
},
$transaction: vi.fn(async (fn: unknown) => {
if (typeof fn === "function") {
return (fn as (tx: unknown) => unknown)(prisma);
}
}),
},
}));
vi.mock("@/lib/measurements/reconcile-external-measurement", () => ({
reconcileExternalMeasurement: vi.fn(
async (_tx: unknown, input: Record<string, unknown>) => ({
status: "inserted",
row: { id: "row-1", ...input },
}),
),
}));
vi.mock("@/lib/auth/session", () => ({ getSession: vi.fn() }));
vi.mock("@/lib/auth/audit", () => ({
auditLog: vi.fn().mockResolvedValue(undefined),
}));
vi.mock("@/lib/logging/transports", () => ({ emitIfSampled: vi.fn() }));
vi.mock("@/lib/db-compat", () => ({
ensureDbCompatibility: vi.fn().mockResolvedValue(undefined),
}));
vi.mock("@/lib/rate-limit", () => ({
checkRateLimit: vi.fn(),
}));
vi.mock("@/lib/jobs/pr-detection", () => ({
enqueuePrDetection: vi.fn().mockResolvedValue(undefined),
}));
vi.mock("@/lib/jobs/reminder-satisfy", () => ({
enqueueReminderSatisfy: vi.fn().mockResolvedValue(undefined),
}));
vi.mock("@/lib/cache/invalidate", () => ({
invalidateUserMeasurements: vi.fn(),
}));
vi.mock("@/lib/insights/comprehensive-generate", () => ({
invalidateStatusInsightsForTypes: vi.fn().mockResolvedValue(undefined),
}));
vi.mock("@/lib/rollups/measurement-rollups", () => ({
recomputeBucketsForMeasurement: vi.fn().mockResolvedValue(undefined),
collapseToTypeDayKeys: vi.fn(() => []),
}));
vi.mock("@/lib/daily/morning-refresh-trigger", () => ({
maybeEnqueueMorningRefresh: vi.fn().mockResolvedValue(undefined),
}));
vi.mock("@/lib/arrivals/emit-shared", () => ({
emitDataArrival: vi.fn().mockResolvedValue(undefined),
}));
vi.mock("@/lib/logging/context", async (importOriginal) => {
const actual = await importOriginal<Record<string, unknown>>();
return { ...actual, annotate: vi.fn() };
});
vi.mock("next/headers", () => ({
headers: vi.fn(async () => ({ get: () => null })),
cookies: vi.fn(async () => ({
get: () => undefined,
set: () => {},
delete: () => {},
})),
}));
import { POST } from "../route";
import { prisma } from "@/lib/db";
import { getSession } from "@/lib/auth/session";
import { checkRateLimit } from "@/lib/rate-limit";
import { annotate } from "@/lib/logging/context";
import { reconcileExternalMeasurement } from "@/lib/measurements/reconcile-external-measurement";
const SESSION_OK = {
session: { id: "sess-1", expiresAt: new Date(Date.now() + 3_600_000) },
user: { id: "user-1", username: "testuser", role: "USER" as const },
};
const DAY_MS = 24 * 60 * 60 * 1000;
function makeRequest(body: unknown): NextRequest {
return new NextRequest("http://localhost/api/measurements/batch", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify(body),
});
}
/** A plain step entry anchored a day back so it never date-bombs. */
function stepEntry(externalId: string) {
const at = new Date(Date.now() - DAY_MS).toISOString();
return {
hkIdentifier: "HKQuantityTypeIdentifierStepCount",
value: 1200,
unit: "count",
startDate: at,
endDate: at,
externalId,
};
}
async function statusesFor(entries: unknown[]) {
const res = await POST(makeRequest({ entries }));
expect(res.status).toBe(200);
const body = (await res.json()) as {
data: {
entries: Array<{ index: number; status: string; reason?: string }>;
};
};
return body.data.entries;
}
beforeEach(() => {
vi.clearAllMocks();
vi.mocked(getSession).mockResolvedValue(SESSION_OK as never);
vi.mocked(checkRateLimit).mockResolvedValue({
allowed: true,
remaining: 60,
resetAt: Date.now() + 60_000,
});
vi.mocked(prisma.measurement.findMany).mockResolvedValue([]);
vi.mocked(prisma.measurement.createManyAndReturn).mockResolvedValue([
{ id: "row-1" },
] as never);
vi.mocked(prisma.measurement.updateMany).mockResolvedValue({ count: 0 });
});
describe("POST /api/measurements/batch — unstable external ids", () => {
it("skips an object-description id instead of minting a row", async () => {
const results = await statusesFor([
stepEntry("<HKHealthConceptIdentifier: 0x12568db80>"),
]);
expect(results[0]).toMatchObject({
index: 0,
status: "skipped",
reason: "unstable_external_id",
});
expect(reconcileExternalMeasurement).not.toHaveBeenCalled();
});
it("skips a bare memory-address id", async () => {
const results = await statusesFor([stepEntry("0x126b25160")]);
expect(results[0]).toMatchObject({
status: "skipped",
reason: "unstable_external_id",
});
});
it("skips ONLY the offending entries — the rest of the batch still lands", async () => {
const results = await statusesFor([
stepEntry("hk-uuid-1"),
stepEntry("<HKHealthConceptIdentifier: 0x12568db80>"),
stepEntry("stats:HKQuantityTypeIdentifierStepCount:2026-07-25"),
stepEntry("0xdeadbeef"),
]);
expect(results[0]!.status).not.toBe("skipped");
expect(results[1]).toMatchObject({ reason: "unstable_external_id" });
expect(results[2]!.status).not.toBe("skipped");
expect(results[3]).toMatchObject({ reason: "unstable_external_id" });
// Two good rows still reached the writer.
expect(reconcileExternalMeasurement).toHaveBeenCalledTimes(2);
});
it("never turns one bad row into a whole-batch 422", async () => {
const res = await POST(
makeRequest({
entries: [stepEntry("hk-uuid-1"), stepEntry("0xdeadbeef")],
}),
);
expect(res.status).toBe(200);
});
it("accepts the real id shapes this route receives", async () => {
const results = await statusesFor([
stepEntry("8AD2A9CB-3F0C-4E4D-9C1E-4B7E2A1D6F30"),
stepEntry("stats:HKQuantityTypeIdentifierStepCount:2026-07-25"),
stepEntry("hk-uuid-41"),
]);
for (const r of results) expect(r.status).not.toBe("skipped");
});
it("annotates the refusal with the shape, never the id", async () => {
await statusesFor([
stepEntry("<HKHealthConceptIdentifier: 0x12568db80>"),
stepEntry("0xdeadbeef"),
]);
const calls = vi.mocked(annotate).mock.calls.map(([c]) => c);
const rejection = calls.find(
(c) => c?.meta && "external_id_rejected" in c.meta,
);
expect(rejection?.meta).toMatchObject({
external_id_rejected: 2,
external_id_shapes: "object_description,pointer_address",
external_id_surface: "measurement.batch",
});
expect(JSON.stringify(rejection)).not.toContain("0x12568db80");
});
it("emits no rejection annotation on a clean batch", async () => {
await statusesFor([stepEntry("hk-uuid-1")]);
const calls = vi.mocked(annotate).mock.calls.map(([c]) => c);
expect(calls.some((c) => c?.meta && "external_id_rejected" in c.meta)).toBe(
false,
);
});
});
+35
View File
@@ -62,6 +62,12 @@ import {
} from "@/lib/measurements/cross-source-merge";
import { reconcileExternalMeasurement } from "@/lib/measurements/reconcile-external-measurement";
import { isPlausibleEntryInstant } from "@/lib/validations/entry-instant";
import {
classifyExternalId,
unstableExternalIdMeta,
UNSTABLE_EXTERNAL_ID_REASON,
type UnstableExternalIdShape,
} from "@/lib/validations/external-id";
import { validateMeasurementRange } from "@/lib/validations/measurement";
import { deviceTypeEnum } from "@/lib/validations/source-priority";
import { checkRateLimit } from "@/lib/rate-limit";
@@ -139,6 +145,9 @@ const batchEntrySchema = z.object({
// iOS build omits them and the bucket keeps the avg-only contract.
valueMin: z.number().finite().optional(),
valueMax: z.number().finite().optional(),
// Stability is enforced per entry in the loop below, not at the field:
// a schema-level refusal would fail the whole batch on one bad row.
// @external-id-checked-per-entry: src/app/api/measurements/batch/route.ts
externalId: z.string().min(1).max(120),
externalSourceVersion: z.string().min(1).max(120).optional(),
// v1.8.6 W6 — optional per-entry source tag. Defaults to
@@ -268,9 +277,29 @@ async function postBatch(request: NextRequest): Promise<Response> {
};
const prepared: Prepared[] = [];
const results: EntryResult[] = new Array(entries.length);
const unstableShapes: UnstableExternalIdShape[] = [];
for (let index = 0; index < entries.length; index++) {
const entry = entries[index];
// The `(user_id, type, source, external_id)` dedup key above is only
// idempotent while the id is STABLE across client launches. An id
// that rotates per process (an object description carrying a memory
// address) never matches its own earlier row, so every sync sweep
// mints a fresh measurement. Refused per entry, like the range and
// timestamp guards below — one malformed row must not stop the other
// 499 from landing.
const unstable = classifyExternalId(entry.externalId);
if (unstable) {
unstableShapes.push(unstable);
results[index] = {
index,
status: "skipped",
reason: UNSTABLE_EXTERNAL_ID_REASON,
};
continue;
}
const mapped = mapAppleHealthEntry({
hkIdentifier: entry.hkIdentifier,
value: entry.value,
@@ -396,6 +425,12 @@ async function postBatch(request: NextRequest): Promise<Response> {
});
}
if (unstableShapes.length > 0) {
annotate({
meta: unstableExternalIdMeta("measurement.batch", unstableShapes),
});
}
let insertedCount = 0;
let updatedCount = 0;
let duplicateCount = 0;
@@ -622,6 +622,65 @@ describe("POST /api/medications — Apple Health mirror (v1.28)", () => {
expect(prisma.medication.create).not.toHaveBeenCalled();
});
/**
* The mirror's idempotency is only as good as the stability of the
* concept id. A client that serialised an opaque platform object with
* its default description sent `<HKHealthConceptIdentifier: 0x…>` — a
* memory address that changes on every launch — and a live instance
* grew 23 phantom medications in a day, one per sync sweep, none of
* which could ever collect a dose. The create refuses that shape now.
*/
it("422s on an object-description externalId instead of minting a phantom mirror", async () => {
vi.mocked(prisma.medication.findFirst).mockResolvedValue(null as never);
const res = await POST(
postReq({
...MIRROR_BODY,
externalId: "<HKHealthConceptIdentifier: 0x12568db80>",
}),
);
expect(res.status).toBe(422);
const body = (await res.json()) as {
details: { issues: Array<{ path: string; message: string }> };
};
const issue = body.details.issues.find((i) => i.path === "externalId");
expect(issue).toBeDefined();
// The client developer has to be able to act on this without asking.
expect(issue!.message).toContain("stable across app restarts");
expect(issue!.message).toContain("object description");
expect(prisma.medication.create).not.toHaveBeenCalled();
});
it("422s on a bare memory-address externalId", async () => {
vi.mocked(prisma.medication.findFirst).mockResolvedValue(null as never);
const res = await POST(
postReq({ ...MIRROR_BODY, externalId: "0xdeadbeef" }),
);
expect(res.status).toBe(422);
const body = (await res.json()) as {
details: { issues: Array<{ path: string; message: string }> };
};
expect(
body.details.issues.some(
(i) => i.path === "externalId" && i.message.includes("memory address"),
),
).toBe(true);
expect(prisma.medication.create).not.toHaveBeenCalled();
});
it("still accepts the stable concept-id shapes the mirror really sends", async () => {
for (const externalId of [
"hk-concept-1",
"8AD2A9CB-3F0C-4E4D-9C1E-4B7E2A1D6F30",
"a3f5c9e1b7d2486fa0c3e5b7d9f1a2c4e6b8d0f2a4c6e8b0d2f4a6c8e0b2d4f6",
]) {
vi.mocked(prisma.medication.create).mockClear();
vi.mocked(prisma.medication.findFirst).mockResolvedValue(null as never);
const res = await POST(postReq({ ...MIRROR_BODY, externalId }));
expect(res.status).toBe(201);
expect(lastCreateData().externalId).toBe(externalId);
}
});
it("a plain create keeps today's behavior — no external columns forwarded", async () => {
const res = await POST(
postReq({
@@ -75,6 +75,12 @@ import {
injectionSiteEnum,
type InjectionSiteValue,
} from "@/lib/validations/medication";
import {
classifyExternalId,
unstableExternalIdMeta,
UNSTABLE_EXTERNAL_ID_REASON,
type UnstableExternalIdShape,
} from "@/lib/validations/external-id";
import type { InjectionSiteKey } from "@/lib/medications/injection-sites";
import { queueMedicationIntakeSync } from "@/lib/notifications/medication-intake-sync";
import { dispatchMedicationIntakeWebClearBulk } from "@/lib/notifications/web-push-clear";
@@ -128,6 +134,10 @@ const bulkEntrySchema = z
// v1.28 — the HealthKit `HKMedicationDoseEvent` UUID. Drives the
// idempotent re-sync: an entry whose `(userId, externalId)` already
// exists live reports `duplicate` (first-write-wins per Apple dose).
// Stability is enforced per entry in the loop below, not at the
// field: a schema-level refusal would fail the whole batch on one
// bad row.
// @external-id-checked-per-entry: src/app/api/medications/intake/bulk/route.ts
externalId: z.string().trim().min(1).max(128).optional(),
})
.refine(
@@ -318,6 +328,7 @@ async function postBulk(request: NextRequest): Promise<Response> {
let updated = 0;
let duplicates = 0;
const skipped: Array<{ index: number; reason: string }> = [];
const unstableShapes: UnstableExternalIdShape[] = [];
// v1.4.39 W-MED — collect distinct `(medicationId, dayKey)` pairs
// touched by the batch so one rollup recompute fires per pair after
// all inserts complete. Per-row recompute would balloon a 500-entry
@@ -345,6 +356,27 @@ async function postBulk(request: NextRequest): Promise<Response> {
continue;
}
// The `(userId, externalId)` dose-event dedup below is only
// idempotent while the id is STABLE across client launches. An id
// that rotates per process (an object description carrying a memory
// address) never matches its own earlier row, so every re-sync mints
// another intake event. Refused per entry — one bad row must not
// stop the rest of the batch landing.
const unstable =
entry.externalId === undefined
? null
: classifyExternalId(entry.externalId);
if (unstable) {
unstableShapes.push(unstable);
skipped.push({ index: i, reason: UNSTABLE_EXTERNAL_ID_REASON });
results.push({
index: i,
status: "skipped",
reason: UNSTABLE_EXTERNAL_ID_REASON,
});
continue;
}
// v1.28 — Apple dose-event replay: the batch pre-check found a live
// row already carrying this external UUID (or an earlier entry in
// THIS batch inserted it). First-write-wins per Apple dose — report
@@ -803,6 +835,9 @@ async function postBulk(request: NextRequest): Promise<Response> {
updated,
duplicates,
skipped: skipped.length,
...(unstableShapes.length > 0
? unstableExternalIdMeta("medication.intake.bulk", unstableShapes)
: {}),
},
});
+14
View File
@@ -10,6 +10,10 @@ import {
safeJson,
} from "@/lib/api-response";
import { createMedicationSchema } from "@/lib/validations/medication";
import {
unstableExternalIdMeta,
unstableExternalIdShape,
} from "@/lib/validations/external-id";
import {
getMedicationCategories,
setMedicationCategory,
@@ -98,6 +102,16 @@ export const POST = apiHandler(async (request: NextRequest) => {
if (jsonError) return jsonError;
const parsed = createMedicationSchema.safeParse(body);
if (!parsed.success) {
// Surface an unstable concept id as its own wide event: the refusal
// is the visible half of the phantom-mirror class, and a silent 422
// in the generic bucket is how it stayed invisible the first time.
const shape = unstableExternalIdShape(body);
if (shape) {
annotate({
action: { name: "medication.external_id.rejected" },
meta: unstableExternalIdMeta("medication.create", [shape]),
});
}
// v1.4.43 W6 — multi-issue 422.
return returnAllZodIssues(parsed.error, 422);
}
+11 -1
View File
@@ -36,6 +36,10 @@ import {
createAssessmentSchema,
listAssessmentsSchema,
} from "@/lib/validations/mental-health";
import {
unstableExternalIdMeta,
unstableExternalIdShape,
} from "@/lib/validations/external-id";
import {
INSTRUMENTS,
INSTRUMENT_MEASUREMENT_TYPE,
@@ -183,9 +187,15 @@ async function postAssessment(request: NextRequest): Promise<Response> {
const parsed = createAssessmentSchema.safeParse(body);
if (!parsed.success) {
const shape = unstableExternalIdShape(body);
annotate({
action: { name: "mental-health.create.validation-failed" },
meta: { issue_count: sanitiseZodIssues(parsed.error.issues).length },
meta: {
issue_count: sanitiseZodIssues(parsed.error.issues).length,
...(shape
? unstableExternalIdMeta("mental_health.assessment.create", [shape])
: {}),
},
});
return returnAllZodIssues(parsed.error, 422);
}
+35
View File
@@ -43,6 +43,12 @@ import {
moodLevelEnum,
moodSourceEnum,
} from "@/lib/validations/mood";
import {
classifyExternalId,
unstableExternalIdMeta,
UNSTABLE_EXTERNAL_ID_REASON,
type UnstableExternalIdShape,
} from "@/lib/validations/external-id";
import { moodDateKey, DEFAULT_TIMEZONE } from "@/lib/mood/date-key";
import { encryptNote } from "@/lib/crypto/note-cipher";
import { invalidateUserMood } from "@/lib/cache/invalidate";
@@ -95,6 +101,10 @@ const bulkEntrySchema = z.object({
* the measurements batch endpoint. NULL = no dedup hint; the
* existing `(userId, date, moodLoggedAt)` unique index still
* protects against straight-up duplicates.
*
* Stability is enforced per entry in the loop below, not at the field:
* a schema-level refusal would fail the whole batch on one bad row.
* @external-id-checked-per-entry: src/app/api/mood-entries/bulk/route.ts
*/
externalId: z.string().min(1).max(120).optional(),
});
@@ -187,6 +197,7 @@ async function postBulk(request: NextRequest): Promise<Response> {
let inserted = 0;
let duplicates = 0;
const skipped: Array<{ index: number; reason: string }> = [];
const unstableShapes: UnstableExternalIdShape[] = [];
// Perf — a single indexed existence read replaces the per-entry
// `findUnique` probe (up to 500 sequential round trips pre-fix). Every
@@ -241,6 +252,27 @@ async function postBulk(request: NextRequest): Promise<Response> {
for (let i = 0; i < entries.length; i++) {
const entry = entries[i];
// The `(userId, source, externalId)` dedup key only holds while the
// id is STABLE across client launches; a per-process value (an object
// description carrying a memory address) mints a fresh entry on every
// re-post. Per-entry refusal, never a whole-batch 422.
const unstable =
entry.externalId === undefined
? null
: classifyExternalId(entry.externalId);
if (unstable) {
unstableShapes.push(unstable);
skipped.push({ index: i, reason: UNSTABLE_EXTERNAL_ID_REASON });
results.push({
index: i,
status: "skipped",
reason: UNSTABLE_EXTERNAL_ID_REASON,
externalId: entry.externalId,
});
continue;
}
const date = moodDateKey(entry.moodLoggedAt, tz);
const score = getScoreForMood(entry.mood);
@@ -415,6 +447,9 @@ async function postBulk(request: NextRequest): Promise<Response> {
inserted,
duplicates,
skipped: skipped.length,
...(unstableShapes.length > 0
? unstableExternalIdMeta("mood.bulk", unstableShapes)
: {}),
},
});
+9 -1
View File
@@ -13,6 +13,10 @@ import {
listMoodEntriesSchema,
getScoreForMood,
} from "@/lib/validations/mood";
import {
unstableExternalIdMeta,
unstableExternalIdShape,
} from "@/lib/validations/external-id";
import { NextRequest } from "next/server";
import { Prisma } from "@/generated/prisma/client";
import { apiHandler, requireAuth } from "@/lib/api-handler";
@@ -160,9 +164,13 @@ async function postMoodEntry(request: NextRequest) {
// v1.4.43 W6 — iOS mood log hot path; multi-issue 422 + audit
// breadcrumb keyed `mood-entries.create.validation-failed`.
const issues = sanitiseZodIssues(parsed.error.issues);
const shape = unstableExternalIdShape(body);
annotate({
action: { name: "mood-entries.create.validation-failed" },
meta: { issue_count: issues.length },
meta: {
issue_count: issues.length,
...(shape ? unstableExternalIdMeta("mood.create", [shape]) : {}),
},
});
// v1.4.49 — strip `message` from the audit-ledger row; the
// mood-create schema's free-text `note` + `tags` could land in a
+43 -10
View File
@@ -70,6 +70,12 @@ import {
createBatchWorkoutSchema,
MAX_WORKOUTS_PER_BATCH,
} from "@/lib/validations/workout";
import {
classifyExternalId,
unstableExternalIdMeta,
UNSTABLE_EXTERNAL_ID_REASON,
type UnstableExternalIdShape,
} from "@/lib/validations/external-id";
import { dedupeWorkoutBatch } from "@/lib/workouts/canonical-rows";
import { Prisma, type MeasurementSource } from "@/generated/prisma/client";
@@ -281,7 +287,26 @@ async function postBatch(request: NextRequest): Promise<Response> {
};
const results: EntryResult[] = new Array(workouts.length);
const prepared: Prepared[] = workouts.map((w, index) => {
const unstableShapes: UnstableExternalIdShape[] = [];
const prepared: Prepared[] = workouts.flatMap((w, index) => {
// The `(userId, source, externalId)` dedup key is only idempotent
// while the id is STABLE across client launches. An id that rotates
// per process (an object description carrying a memory address)
// never matches its own earlier row, so every re-sync mints another
// workout. Refused per entry — one bad row must not stop the rest of
// the batch landing.
const unstable =
w.externalId === undefined ? null : classifyExternalId(w.externalId);
if (unstable) {
unstableShapes.push(unstable);
results[index] = {
index,
status: "skipped",
reason: UNSTABLE_EXTERNAL_ID_REASON,
};
return [];
}
const durationSec = Math.max(
0,
Math.round((w.endedAt.getTime() - w.startedAt.getTime()) / 1000),
@@ -319,17 +344,25 @@ async function postBatch(request: NextRequest): Promise<Response> {
? (w.metadata as Prisma.InputJsonValue)
: Prisma.JsonNull,
};
return {
index,
row,
route,
samples,
dedupKey: w.externalId
? { source: w.source, externalId: w.externalId }
: null,
};
return [
{
index,
row,
route,
samples,
dedupKey: w.externalId
? { source: w.source, externalId: w.externalId }
: null,
},
];
});
if (unstableShapes.length > 0) {
annotate({
meta: unstableExternalIdMeta("workout.batch", unstableShapes),
});
}
// v1.4.42 W5 — write-time cross-source dedup. The v1.5 iOS app
// drains a HealthKit observer queue that frequently carries the
// SAME logical workout from two paired sensors (Apple Watch +
+10
View File
@@ -40,6 +40,7 @@ import {
MEASUREMENT_NOTES_MAX_LENGTH,
} from "@/lib/validations/measurement";
import { isPlausibleEntryInstant } from "@/lib/validations/entry-instant";
import { isStableExternalId } from "@/lib/validations/external-id";
import { resolveToCanonicalUnit } from "@/lib/measurements/unit-aliases";
/** A normalised, write-ready row in the `/api/import` measurement shape. */
@@ -68,6 +69,7 @@ export type CsvRowReason =
| "invalid_glucose_context"
| "notes_too_long"
| "external_id_too_long"
| "unstable_external_id"
| "wrong_column_count";
export type CsvRowStatus = "ok" | "skipped";
@@ -304,6 +306,14 @@ export function parseCsvMeasurements(
continue;
}
// The import upserts on `(userId, type, source, externalId)`, so an
// id that cannot be stable across exports re-imports as a fresh row
// every time instead of converging on the one it already wrote.
if (rawExternalId.length > 0 && !isStableExternalId(rawExternalId)) {
skip("unstable_external_id");
continue;
}
rows.push({
line,
status: "ok",
+2 -1
View File
@@ -81,7 +81,8 @@ const batchEntrySchema = z
"Dedup key for the `(userId, type, source, externalId)` composite index. Three shapes:\n" +
"• `HKSample.uuid` — a per-sample reading. First-write-wins: a re-post returns `duplicate` and the stored row is immutable.\n" +
"• `stats:<HKQuantityTypeIdentifier>:<YYYY-MM-DD>` — a per-day cumulative total (Steps, Active Energy, Sleep Duration, Walking/Running Distance, Flights Climbed). A re-post OVERWRITES the row and returns `updated`.\n" +
"• `stats:HKQuantityTypeIdentifierHeartRate:<bucket-start>` — v1.19.0 (iOS #34), refined to 10-minute granularity in v1.30.7, heart-rate bucket carrying the bucket's AVERAGE bpm as one PULSE row. `<bucket-start>` is the ISO-8601 UTC 10-minute boundary — zeroed seconds/millis, minutes ∈ {00,10,20,30,40,50}, trailing `Z` (e.g. `2026-06-21T14:10:00.000Z`). A re-post OVERWRITES the row and returns `updated`, so iOS uploads ~144 rows/day instead of one per raw HR sample. A row that targets this prefix with a malformed bucket-start suffix is `skipped` with reason `malformed_hr_bucket_id`.",
"• `stats:HKQuantityTypeIdentifierHeartRate:<bucket-start>` — v1.19.0 (iOS #34), refined to 10-minute granularity in v1.30.7, heart-rate bucket carrying the bucket's AVERAGE bpm as one PULSE row. `<bucket-start>` is the ISO-8601 UTC 10-minute boundary — zeroed seconds/millis, minutes ∈ {00,10,20,30,40,50}, trailing `Z` (e.g. `2026-06-21T14:10:00.000Z`). A re-post OVERWRITES the row and returns `updated`, so iOS uploads ~144 rows/day instead of one per raw HR sample. A row that targets this prefix with a malformed bucket-start suffix is `skipped` with reason `malformed_hr_bucket_id`.\n" +
"The id MUST be stable across app restarts. An object description or memory address (`<Class: 0x…>`, `0x12ab34cd`) is a new value on every launch, so it can never match its own earlier row; such an entry is `skipped` with reason `unstable_external_id` while the rest of the batch still lands.",
),
externalSourceVersion: z.string().min(1).max(120).optional(),
// v1.8.6 W6 — optional per-entry source tag. Defaults to
+1 -1
View File
@@ -104,7 +104,7 @@ const bulkIntakeEntry = z
.max(128)
.optional()
.describe(
"The HealthKit dose-event UUID. Drives the idempotent re-sync dedup (first-write-wins per Apple dose).",
"The HealthKit dose-event UUID. Drives the idempotent re-sync dedup (first-write-wins per Apple dose). Must be stable across app restarts — an object description or memory address (`<Class: 0x…>`, `0x12ab34cd`) is a new value on every launch; such an entry is `skipped` with reason `unstable_external_id` while the rest of the batch still lands.",
),
})
.meta({
+1 -1
View File
@@ -221,7 +221,7 @@ const bulkMoodEntry = z
.max(120)
.optional()
.describe(
"iOS-side source-stable id for idempotent dedup on the `(userId, source, externalId)` key.",
"iOS-side source-stable id for idempotent dedup on the `(userId, source, externalId)` key. Must be stable across app restarts — an object description or memory address (`<Class: 0x…>`, `0x12ab34cd`) is a new value on every launch; such an entry is `skipped` with reason `unstable_external_id` while the rest of the batch still lands.",
),
})
.meta({
@@ -0,0 +1,182 @@
/**
* External-id stability floor.
*
* The rejected cases are the shapes that provably cannot be an identity.
* The accepted cases are every external-id shape this repository actually
* receives — grepped out of the ingest routes, the batch contracts, the
* OpenAPI examples and the existing fixtures. A false positive here costs
* a working client its sync, so the accepted list is the real guard.
*/
import { describe, it, expect } from "vitest";
import {
assertStableExternalId,
classifyExternalId,
isStableExternalId,
unstableExternalIdMessage,
unstableExternalIdMeta,
unstableExternalIdShape,
UNSTABLE_EXTERNAL_ID_REASON,
} from "@/lib/validations/external-id";
import { z } from "zod/v4";
describe("classifyExternalId — refused shapes", () => {
it("refuses the object description that caused the live incident", () => {
expect(classifyExternalId("<HKHealthConceptIdentifier: 0x12568db80>")).toBe(
"object_description",
);
});
it("refuses any <Class: 0xHEX> description, not just the one class", () => {
for (const value of [
"<NSObject: 0x600000c1a2b0>",
"<SomeVendorType: 0X7FFEE3B0>",
"<Foo: 0x1>",
"<HKObject: 0x126b25160; type=bar>",
"apple:<HKHealthConceptIdentifier: 0x11f47e3e0>",
]) {
expect(classifyExternalId(value)).toBe("object_description");
}
});
it("refuses a bare pointer value", () => {
expect(classifyExternalId("0x12568db80")).toBe("pointer_address");
expect(classifyExternalId("0XDEADBEEF")).toBe("pointer_address");
});
it("refuses an empty or whitespace-only id", () => {
expect(classifyExternalId("")).toBe("blank");
expect(classifyExternalId(" ")).toBe("blank");
expect(classifyExternalId("\t\n ")).toBe("blank");
});
it("classifies a padded description after trimming", () => {
expect(classifyExternalId(" <HKFoo: 0xabc123> ")).toBe(
"object_description",
);
});
});
describe("classifyExternalId — the id shapes this system actually receives", () => {
// Every entry below is a shape that exists in this repository today:
// the iOS batch contract, the Apple Health mirror, the Withings /
// WHOOP / Oura / Fitbit / Polar sync paths, the CSV importer, and the
// web client's own UUID mints.
const accepted = [
// HealthKit sample UUIDs (upper + lower case, hyphenated)
"8AD2A9CB-3F0C-4E4D-9C1E-4B7E2A1D6F30",
"3f0c4e4d-9c1e-4b7e-2a1d-6f30ad2a9cb1",
// Un-hyphenated / compact ids
"8AD2A9CB3F0C4E4D9C1E4B7E2A1D6F30",
// The `stats:` aggregate contract (per-day + hourly bucket)
"stats:HKQuantityTypeIdentifierStepCount:2026-07-25",
"stats:HKQuantityTypeIdentifierActiveEnergyBurned:2026-07-25",
"stats:HKQuantityTypeIdentifierHeartRate:2026-07-25T08:10:00Z",
// Other structured prefix forms minted in this tree today
"assessment:clx1a2b3c4d5e6f7g8h9",
"withings:activity:clx1a2b3c4d5e6f7g8h9:2026-07-25:steps",
"withings:sleep:clx1a2b3c4d5e6f7g8h9:998877:hr_average",
"cycle:123456:strain",
"sleep:0f3a8c2e-2b1d-4f5a-9c8e-1d2f3a4b5c6d:hrv_rmssd",
"spo2:2026-07-25T03:12:00Z:unknown",
"hkcycle:2026-07-25",
"stats:steps:2026-07-25",
"retired:clx1a2b3c4d5e6f7g8h9:hk-uuid-41",
"hk-uuid-41",
// Opaque vendor identifiers
"1234567890",
"clx1a2b3c4d5e6f7g8h9",
"a3f5c9e1b7d2486fa0c3e5b7d9f1a2c4e6b8d0f2a4c6e8b0d2f4a6c8e0b2d4f6",
// Base64-ish blob
"aGVhbHRobG9nOnNhbXBleQ==",
// Platform type identifiers
"HKQuantityTypeIdentifierStepCount",
// Values that merely CONTAIN hex-looking text but are not pointers
"0xygen-sample-1",
"sample-0x12ab-tail",
"not<angle>brackets",
"<no-pointer-here>",
];
it.each(accepted)("accepts %s", (value) => {
expect(classifyExternalId(value)).toBeNull();
expect(isStableExternalId(value)).toBe(true);
});
it("does not reject on length, spaces, or unusual characters alone", () => {
expect(isStableExternalId("My Glucose Meter Export Row 41")).toBe(true);
expect(isStableExternalId("a")).toBe(true);
expect(isStableExternalId("ärztliche-messung-2026-07-25")).toBe(true);
});
});
describe("messages", () => {
it("names what is wrong and what to send instead", () => {
expect(unstableExternalIdMessage("object_description")).toContain(
"stable across app restarts",
);
expect(unstableExternalIdMessage("object_description")).toContain(
"object description",
);
expect(unstableExternalIdMessage("pointer_address")).toContain(
"memory address",
);
expect(unstableExternalIdMessage("blank")).toContain("must not be empty");
});
});
describe("assertStableExternalId — object-level Zod check", () => {
const schema = z
.object({ externalId: z.string().max(128).optional() })
.superRefine(assertStableExternalId);
it("rejects an object description with the actionable message on the field", () => {
const parsed = schema.safeParse({
externalId: "<HKHealthConceptIdentifier: 0x12568db80>",
});
expect(parsed.success).toBe(false);
if (parsed.success) return;
expect(parsed.error.issues).toHaveLength(1);
expect(parsed.error.issues[0].path).toEqual(["externalId"]);
expect(parsed.error.issues[0].message).toBe(
unstableExternalIdMessage("object_description"),
);
});
it("accepts a stable id and an absent id alike", () => {
expect(
schema.safeParse({ externalId: "stats:HKFoo:2026-07-25" }).success,
).toBe(true);
expect(schema.safeParse({}).success).toBe(true);
});
});
describe("wide-event helpers", () => {
it("reads a top-level externalId off an unparsed body", () => {
expect(unstableExternalIdShape({ externalId: "0xdeadbeef" })).toBe(
"pointer_address",
);
expect(unstableExternalIdShape({ externalId: "uuid-1" })).toBeNull();
expect(unstableExternalIdShape({ externalId: 42 })).toBeNull();
expect(unstableExternalIdShape(null)).toBeNull();
expect(unstableExternalIdShape("not-an-object")).toBeNull();
});
it("pins the meta shape and never carries the id itself", () => {
expect(
unstableExternalIdMeta("measurement.batch", [
"object_description",
"object_description",
"pointer_address",
]),
).toEqual({
external_id_rejected: 3,
external_id_shapes: "object_description,pointer_address",
external_id_surface: "measurement.batch",
});
});
it("exposes one per-entry reason code for every batch surface", () => {
expect(UNSTABLE_EXTERNAL_ID_REASON).toBe("unstable_external_id");
});
});
+26 -6
View File
@@ -11,6 +11,7 @@
*/
import { z } from "zod/v4";
import { isPlausibleEntryInstant } from "@/lib/validations/entry-instant";
import { assertStableExternalId } from "@/lib/validations/external-id";
/**
* v1.17 W1b — `loggedAt` instant on the cycle wire stays a string (the iOS
@@ -138,11 +139,28 @@ export const cycleDayLogInputSchema = z.object({
note: z.string().max(500).optional(),
loggedAt: boundedLoggedAt,
source: cycleSourceEnum.optional().default("MANUAL"),
// Shared with the bulk ingest, which enforces external-id stability per
// entry rather than at the field (a schema-level refusal would fail the
// whole batch on one bad row). The single-entry POST parses through
// `cycleDayLogCreateSchema` below, which adds the field check.
// @external-id-checked-per-entry: src/app/api/cycle/day-logs/bulk/route.ts
externalId: z.string().min(1).max(120).optional(),
});
export type CycleDayLogInput = z.infer<typeof cycleDayLogInputSchema>;
/**
* Single-entry POST body. Same shape as the bulk entry plus the
* external-id stability floor: a rotating id defeats the
* `(userId, source, externalId)` upsert and mints a fresh day log on every
* re-post. The bulk path deliberately keeps the permissive entry schema —
* it reports the offending entry as `skipped` rather than failing the
* whole batch — and applies the same rule per entry in the route.
*/
export const cycleDayLogCreateSchema = cycleDayLogInputSchema.superRefine(
assertStableExternalId,
);
/** PATCH body — every field optional; `date`/`source`/`externalId` are immutable on update. */
export const cycleDayLogPatchSchema = z.object({
flow: flowLevelEnum.nullable().optional(),
@@ -180,12 +198,14 @@ export const cycleBulkSchema = z.object({
/* ── period-boundary shortcut ────────────────────────────────────── */
export const cyclePeriodSchema = z.object({
action: z.enum(["start", "end"]),
date: dateString,
externalId: z.string().min(1).max(120).optional(),
loggedAt: boundedLoggedAt,
});
export const cyclePeriodSchema = z
.object({
action: z.enum(["start", "end"]),
date: dateString,
externalId: z.string().min(1).max(120).optional(),
loggedAt: boundedLoggedAt,
})
.superRefine(assertStableExternalId);
/* ── calendar + history queries ──────────────────────────────────── */
+189
View File
@@ -0,0 +1,189 @@
/**
* Client-supplied external identifiers — stability floor.
*
* Every ingest surface that accepts an `externalId` uses it as one half of
* a dedup key: `(userId, source, externalId)`, `(userId, externalSource,
* externalId)`, `(userId, type, source, externalId)`. Re-posting the same
* triple is idempotent and returns the existing row. That idempotency is
* only as good as the STABILITY of the identifier — an id that changes
* between two runs of the same client never matches its own earlier row,
* so every sync sweep mints a fresh record and the user's history fills
* with phantom rows that carry no doses, no schedule, no follow-up data.
*
* That is not hypothetical. A native client serialised an opaque platform
* object with the language's default object description and sent the
* result as `externalId`. The default description of an object with no
* description contract is `<ClassName: 0xADDRESS>` — a per-allocation
* MEMORY ADDRESS. A live instance accumulated 23 duplicate medications in
* a single day, one per sync sweep, each with a different address in its
* id, and the dose events that should have attached to them resolved
* through the same rotating string and were never stored at all.
*
* This module is the one place that recognises an identifier which CANNOT
* be a stable identity. It is deliberately narrow: the cost of a false
* negative is a duplicate row, the cost of a false positive is a
* legitimate client losing the ability to sync at all. Only shapes that
* are provably not identities are rejected:
*
* 1. `blank` — empty or whitespace-only after trimming.
* Carries no identity by construction.
* 2. `pointer_address` — a bare `0x…` hex value and nothing else.
* A raw address, valid only inside the
* process that printed it.
* 3. `object_description` — an angle-bracketed description containing
* an `0x…` hex token, e.g. `<Foo: 0x12568db80>`
* or `<Foo: 0x1; bar=2>`. This is the general
* form of a default object description, not a
* match on one platform class: no identity
* format wraps itself in angle brackets around
* a hex pointer.
*
* Everything else passes. In particular the identifier shapes this system
* actually receives all survive:
*
* - UUIDs, upper or lower case, hyphenated or not
* (`8AD2A9CB-...`, HealthKit sample UUIDs, SwiftData row ids)
* - structured prefix forms
* (`stats:HKQuantityTypeIdentifierStepCount:2026-07-25`,
* `stats:HKQuantityTypeIdentifierHeartRate:2026-07-25T08:10:00Z`,
* `assessment:<id>`, `whoop:<id>`, `oura:<id>`)
* - opaque vendor identifiers (numeric ids, base64-ish blobs, hashes —
* including a 64-char SHA-256 hex digest, which has no `0x` prefix)
* - platform type identifiers (`HKQuantityTypeIdentifierStepCount`)
*
* There is deliberately NO rule on length, on the presence of spaces, on
* character class, or on "looks random enough". Those would reject real
* vendor identifiers, and a rule that cannot be justified against a shape
* this system actually receives does not belong here.
*/
import { z } from "zod/v4";
/** The closed set of shapes that cannot be a stable identity. */
export type UnstableExternalIdShape =
"blank" | "pointer_address" | "object_description";
/**
* A bare hex pointer and nothing else — `0x12568db80`. Anchored on both
* ends so an opaque vendor id that merely CONTAINS "0x" is untouched.
*/
const BARE_POINTER = /^0x[0-9a-f]+$/i;
/**
* An angle-bracketed description carrying a hex pointer, searched
* anywhere in the value so a client that prefixes its own namespace
* (`apple:<Foo: 0x1234>`) is caught too. `[^<>]*` keeps the bracket pair
* adjacent — the token has to be one description, not two unrelated
* angle brackets spanning half the string.
*/
const POINTER_DESCRIPTION = /<[^<>]*0x[0-9a-f]+[^<>]*>/i;
/**
* Classify an external identifier. Returns `null` when the value is
* usable as an identity, or the shape that disqualifies it.
*/
export function classifyExternalId(
value: string,
): UnstableExternalIdShape | null {
const trimmed = value.trim();
if (trimmed.length === 0) return "blank";
if (BARE_POINTER.test(trimmed)) return "pointer_address";
if (POINTER_DESCRIPTION.test(trimmed)) return "object_description";
return null;
}
/** Convenience predicate over {@link classifyExternalId}. */
export function isStableExternalId(value: string): boolean {
return classifyExternalId(value) === null;
}
const SHAPE_MESSAGES: Record<UnstableExternalIdShape, string> = {
blank: "external identifier must not be empty",
pointer_address:
"external identifier must be stable across app restarts; received a memory address (0x…), which is a new value on every launch and mints a duplicate record on every sync",
object_description:
"external identifier must be stable across app restarts; received an object description (<Class: 0x…>), which embeds a memory address and is a new value on every launch — serialise the identity itself, not the object's description",
};
/** The 422 message a client developer reads. Names the shape received. */
export function unstableExternalIdMessage(
shape: UnstableExternalIdShape,
): string {
return SHAPE_MESSAGES[shape];
}
/**
* Object-level Zod check for a single-entry request schema that carries a
* top-level `externalId`. Used as `schema.superRefine(assertStableExternalId)`
* so the refusal rides the standard 422 multi-issue envelope
* (`returnAllZodIssues` / `sanitiseZodIssues`) with `path: ["externalId"]`
* instead of a hand-rolled error.
*
* Batch / bulk surfaces deliberately do NOT use this: a schema-level
* refusal fails the whole payload, and one bad row must not stop the other
* 499 from landing. Those routes call {@link classifyExternalId} per entry
* and report the entry through their own `skipped` contract.
*/
export function assertStableExternalId(
value: { externalId?: string | null },
ctx: z.RefinementCtx,
): void {
const raw = value.externalId;
if (typeof raw !== "string") return;
const shape = classifyExternalId(raw);
if (shape === null) return;
ctx.addIssue({
code: "custom",
path: ["externalId"],
message: unstableExternalIdMessage(shape),
});
}
/**
* Read a top-level `externalId` off an unparsed request body and classify
* it. Single-entry routes use this on the 422 path to annotate the wide
* event with the SHAPE that was refused — never the id itself, which is
* client free-text and can carry anything.
*/
export function unstableExternalIdShape(
body: unknown,
): UnstableExternalIdShape | null {
if (typeof body !== "object" || body === null) return null;
const raw = (body as { externalId?: unknown }).externalId;
if (typeof raw !== "string") return null;
return classifyExternalId(raw);
}
/**
* Pinned wide-event meta for a refusal. Keys are fixed so dashboards stay
* stable; the payload carries the count, the classification, and the
* surface — never the identifier itself, which is client free-text.
*
* `external_id_shapes` is the sorted, de-duplicated shape set as a single
* comma-joined string — a scalar rather than an array so it groups
* cleanly in the event store.
*
* The surface rides the META rather than only the action name because a
* wide event carries exactly one action, and most of these refusals
* happen on a path that already names its own (`…​.validation-failed`,
* `measurement.batch.ingest`). Overwriting that action would cost more
* than it buys, so `external_id_rejected > 0` is the one query that finds
* every occurrence on every surface; routes whose failure path claims no
* action of its own additionally set `<surface>.external_id.rejected`.
*/
export function unstableExternalIdMeta(
surface: string,
shapes: UnstableExternalIdShape[],
): {
external_id_rejected: number;
external_id_shapes: string;
external_id_surface: string;
} {
return {
external_id_rejected: shapes.length,
external_id_shapes: [...new Set(shapes)].sort().join(","),
external_id_surface: surface,
};
}
/** Per-entry `reason` code every batch / bulk surface reports. */
export const UNSTABLE_EXTERNAL_ID_REASON = "unstable_external_id";
@@ -12,6 +12,7 @@ import {
rxNormCodeField,
} from "./base";
import { scheduleSchema } from "./schedule";
import { assertStableExternalId } from "@/lib/validations/external-id";
/**
* v1.5 — medication-level course window + one-shot flag. The fields
@@ -187,7 +188,7 @@ export const createMedicationSchema = z
.max(128)
.optional()
.describe(
"Stable external identifier of the mirrored medication (the HealthKit `medicationConceptIdentifier`, max 128 chars). Must be supplied together with `externalSource`. Idempotent: re-posting the same pair returns the existing medication with status 200 instead of minting a duplicate.",
"Stable external identifier of the mirrored medication (the HealthKit `medicationConceptIdentifier`, max 128 chars). Must be supplied together with `externalSource`. Idempotent: re-posting the same pair returns the existing medication with status 200 instead of minting a duplicate. The value MUST be identical across app restarts: an object description or memory address (`<Class: 0x…>`, `0x12ab34cd`) is a new value on every launch, never matches its own earlier row, and is refused with 422 rather than minting a duplicate mirror.",
),
...courseWindowFields,
/**
@@ -206,6 +207,10 @@ export const createMedicationSchema = z
path: ["externalId"],
},
)
// The concept id has to be STABLE across client launches or the
// idempotent-mirror pre-query above never matches its own earlier row
// and every sync sweep mints another phantom medication.
.superRefine(assertStableExternalId)
.refine((b) => b.oneShot !== true || !!b.startsOn, {
message: "startsOn is required when oneShot is true",
path: ["startsOn"],
+5 -1
View File
@@ -11,6 +11,7 @@ import {
INSTRUMENTS,
type InstrumentId,
} from "@/lib/mental-health/instruments";
import { assertStableExternalId } from "@/lib/validations/external-id";
export const assessmentInstrumentEnum = z.enum(["PHQ9", "GAD7", "WHO5", "SCI"]);
@@ -69,7 +70,10 @@ export const createAssessmentSchema = z
});
}
});
});
})
// The replay anchor only protects against a duplicate administration
// while the id is stable across client launches.
.superRefine(assertStableExternalId);
export const listAssessmentsSchema = z.object({
instrument: assessmentInstrumentEnum.optional(),
+40 -33
View File
@@ -8,6 +8,7 @@
*/
import { z } from "zod/v4";
import { validateEntryInstant } from "@/lib/validations/entry-instant";
import { assertStableExternalId } from "@/lib/validations/external-id";
// --- CRUD schemas for mood entries ---
@@ -68,39 +69,45 @@ const ratedFactor = z.object({
});
const ratedFactors = z.array(ratedFactor).max(30);
export const createMoodEntrySchema = z.object({
mood: moodLevelEnum,
tags: z.array(z.string().max(50)).max(20).optional(),
// v1.8.5 — structured-tag keys from the taxonomy. Server resolves each
// key to a `MoodTag` row and writes the `MoodEntryTagLink` join;
// unknown keys are dropped silently (the catalog is the source of
// truth, a stale client can't mint a tag).
tagKeys: structuredTagKeys.optional(),
// v1.12.0 — rated factors scored 1..5 (or the factor's own scale).
// Parallel to the binary `tagKeys`; persisted on
// `MoodEntryTagLink.rating`. Out-of-scale or non-RATED keys are
// rejected (422) / dropped server-side per the catalog.
ratedFactors: ratedFactors.optional(),
// v1.4.30 H-5 — first-class free-text note. Replaces the
// `tags: ["note:<text>"]` workaround. Capped at 500 chars so the
// Coach evidence shelf renders cleanly without truncating chips.
note: z.string().max(500).optional(),
// v1.17 W1b — plausibility bound (shared `validateEntryInstant`): no
// future instants beyond a 5-min clock-skew tolerance, no instant before
// 1900. Mirrors the measurement + medication-intake bound.
moodLoggedAt: validateEntryInstant(
z.iso.datetime({ offset: true }).transform((s) => new Date(s)),
),
source: moodSourceEnum.optional().default("MANUAL"),
// v1.12.1 — optional source-stable id (e.g. an iOS SwiftData row UUID).
// When present, the create upserts on `(userId, source, externalId)`
// so a re-post with the same id updates the existing row in place
// instead of either 409-ing or minting a duplicate — the idempotent
// re-import iOS drives over Bearer. NULL keeps the legacy
// `(userId, date, moodLoggedAt)` behaviour. Bound matches the bulk
// `externalId` so one path can't accept an id the other rejects.
externalId: z.string().min(1).max(120).optional(),
});
export const createMoodEntrySchema = z
.object({
mood: moodLevelEnum,
tags: z.array(z.string().max(50)).max(20).optional(),
// v1.8.5 — structured-tag keys from the taxonomy. Server resolves each
// key to a `MoodTag` row and writes the `MoodEntryTagLink` join;
// unknown keys are dropped silently (the catalog is the source of
// truth, a stale client can't mint a tag).
tagKeys: structuredTagKeys.optional(),
// v1.12.0 — rated factors scored 1..5 (or the factor's own scale).
// Parallel to the binary `tagKeys`; persisted on
// `MoodEntryTagLink.rating`. Out-of-scale or non-RATED keys are
// rejected (422) / dropped server-side per the catalog.
ratedFactors: ratedFactors.optional(),
// v1.4.30 H-5 — first-class free-text note. Replaces the
// `tags: ["note:<text>"]` workaround. Capped at 500 chars so the
// Coach evidence shelf renders cleanly without truncating chips.
note: z.string().max(500).optional(),
// v1.17 W1b — plausibility bound (shared `validateEntryInstant`): no
// future instants beyond a 5-min clock-skew tolerance, no instant before
// 1900. Mirrors the measurement + medication-intake bound.
moodLoggedAt: validateEntryInstant(
z.iso.datetime({ offset: true }).transform((s) => new Date(s)),
),
source: moodSourceEnum.optional().default("MANUAL"),
// v1.12.1 — optional source-stable id (e.g. an iOS SwiftData row UUID).
// When present, the create upserts on `(userId, source, externalId)`
// so a re-post with the same id updates the existing row in place
// instead of either 409-ing or minting a duplicate — the idempotent
// re-import iOS drives over Bearer. NULL keeps the legacy
// `(userId, date, moodLoggedAt)` behaviour. Bound matches the bulk
// `externalId` so one path can't accept an id the other rejects.
externalId: z.string().min(1).max(120).optional(),
})
// The upsert key is only idempotent while the id is stable; an id that
// rotates per client launch mints a fresh entry on every re-post. The
// bulk twin carries the same rule per entry (it cannot fail the whole
// batch on one bad row).
.superRefine(assertStableExternalId);
export const updateMoodEntrySchema = z.object({
mood: moodLevelEnum.optional(),
+4
View File
@@ -173,6 +173,10 @@ export const createWorkoutSchema = z
// schema, so a client cannot forge a row attributed to an integration it
// does not own.
source: writableMeasurementSourceEnum.optional().default("MANUAL"),
// The only writer of this schema is the batch ingest, which enforces
// external-id stability per entry rather than at the field: a
// schema-level refusal would fail the whole batch on one bad row.
// @external-id-checked-per-entry: src/app/api/workouts/batch/route.ts
externalId: z.string().max(128).optional(),
externalSourceVersion: z.string().max(64).optional(),
metadata: z.record(z.string(), z.unknown()).optional(),