Merge branch 'feat/v1462-data' into release/v1.46.2

This commit is contained in:
Marc-André Bombeck committed 2026-10-29 09:45:37 +01:00
commit 4a99658632
33 files changed
+1724 -91

No files matched your search

+10 -3
View File
@@ -4825,7 +4825,10 @@ paths:
tags:
- Measurements
summary: Delete a measurement
description: Soft-deletes the row (tombstone) so paired clients can reconcile the deletion.
description: "Soft-deletes the row (tombstone) so paired clients can reconcile the deletion. A blood-pressure systolic
or diastolic takes the rest of its reading with it: the other half and the pulse with the same source and the
identical `measuredAt`, tombstoned in the same transaction with the same `deletedAt`, each with its own
`syncVersion` bump. A pulse is deleted on its own."
parameters:
- in: path
name: id
@@ -4936,7 +4939,8 @@ paths:
description: Soft-deletes (tombstones) up to 200 of the caller's measurement rows in one call, mirroring the
single-DELETE contract. Idempotent via the `Idempotency-Key` header; rate-limited
`measurements:bulk-delete:<userId>`. Forged / foreign / already-deleted ids are silently skipped — `deleted` is
the count of rows actually tombstoned. The deletions surface in `/api/sync/changes` as tombstones.
the count of rows actually tombstoned, which includes the rest of any blood-pressure reading a selected systolic
or diastolic belongs to (as on the single DELETE). The deletions surface in `/api/sync/changes` as tombstones.
parameters:
- *a6
requestBody:
@@ -4965,7 +4969,10 @@ paths:
description: Un-tombstones up to 200 of the caller's soft-deleted measurement rows in one call — the delete-Undo
counterpart to `/api/measurements/bulk-delete`. Idempotent via the `Idempotency-Key` header; rate-limited
`measurements:restore:<userId>`. Forged / foreign / not-deleted ids are silently skipped — `restored` is the
count of rows actually un-tombstoned. The restored rows re-surface in `/api/sync/changes` as upserts.
count of rows actually un-tombstoned. A blood-pressure systolic or diastolic also restores the rows of its
reading that were deleted with it (same source, identical `measuredAt` and `deletedAt`), so naming one half
undoes the whole reading; a row deleted at another moment stays deleted. The restored rows re-surface in
`/api/sync/changes` as upserts.
parameters:
- *a6
requestBody:
+6 -4
View File
@@ -631,10 +631,10 @@ Confirmed-rejected slugs (despite the source-tree references in the table above)
Confirmed-accepted slugs on a ChatGPT Plus subscription (per `available_in_plans` in `models.json`):
- `gpt-5.6-terra` — HealthLog's default (first chain entry) as of 2026-10-24; the backend rejected `gpt-5.5` and everything below it that day. `gpt-5.6-luna` and `gpt-5.6-sol` are its siblings on the same line
- `gpt-6-sol`, `gpt-6.1-sol`, `gpt-6-luna`, `gpt-6-astra` — the gpt-6 line, listed in `models.json` since 2026-09-22 (`gpt-6.1-sol` the catalog default since 2026-09-29). The catalog names `gpt-6-sol` as the upgrade of `gpt-5.6-terra` and `gpt-5.6-sol`, and `gpt-6-luna` as that of `gpt-5.6-luna`. Every gpt-6 slug lists `input_modalities: ["text", "image"]` and the same reasoning presets as the gpt-5.6 line (`low` to `max`, sol and astra also `ultra`). The presets omit `none`, as they do for `gpt-5.5`, which a live probe on 2026-10-06 showed accepts it; HealthLog sends `none` for "off" on the gpt-6 line too and learns a refusal per slug. The gpt-6 slugs were not probed against a live backend when they joined the chain
- `gpt-5.5` — the default from 2026-06 until 2026-10-24
- `gpt-5.4` / `gpt-5.4-mini` / `gpt-5.2` — accepted per plan
- `gpt-5.3-codex` — was the default (verified 2026-05-09); the backend rotated off the `-codex` line on 2026-06-02 and this slug is now only accepted intermittently, so it sits on a late chain rung
- `gpt-6-sol`, `gpt-6.1-sol`, `gpt-6-luna`, `gpt-6-astra` — the gpt-6 line, listed in `models.json` since 2026-09-22 (`gpt-6.1-sol` the catalog default since 2026-09-29). The catalog names `gpt-6-sol` as the upgrade of `gpt-5.6-terra` and `gpt-5.6-sol`, and `gpt-6-luna` as that of `gpt-5.6-luna`. Every gpt-6 slug lists `input_modalities: ["text", "image"]` and the same reasoning presets as the gpt-5.6 line (`low` to `max`, sol and astra also `ultra`). The presets omit `none`, as they do for `gpt-5.5`, which a live probe on 2026-10-06 showed accepts it; HealthLog sends `none` for "off" on the gpt-6 line too and learns a refusal per slug. A live check on 2026-10-24 found the account's own list (`GET /backend-api/codex/models?client_version=…`) offering `gpt-6.1-sol`, `gpt-6-astra`, `gpt-6-sol`, `gpt-6-luna` and the `gpt-5.6` line, all with `input_modalities: ["text", "image"]`. From a HealthLog container with its stored sign-in, `gpt-6.1-sol`, `gpt-6-sol` and `gpt-6-luna` each answered text and read a lab-report image correctly. `gpt-6.1-sol` refuses `reasoning.effort: "none"` and names `low` as its lowest effort, which the client learns and steps up to on the same slug; the other two accept `none`
- `gpt-5.5` — the default from 2026-06 until 2026-10-24, kept as the last chain rung while plans differ
Retired for ChatGPT sign-in (OpenAI's Codex models page): `gpt-5.2` and `gpt-5.3-codex` on 2026-05-26, `gpt-5.4` and `gpt-5.4-mini` on 2026-08-31, `gpt-5.5` on 2026-10-14 at 19:00 UTC. The backend answers each of them with 400 `model_not_found`, "The '<slug>' model is not supported when using Codex with a ChatGPT account.", which the client reads as a reason to try the next slug.
The `codex-rs/tui/src/model_migration.rs` references to `gpt-5-codex` etc. are _migration prompts_ shown to existing users — they are NOT the slugs sent on the wire after migration. The wire slugs come from `models.json`. Always cross-check `models.json` (`gh api repos/openai/codex/contents/codex-rs/models-manager/models.json`) before picking a default.
@@ -662,6 +662,8 @@ Every default rung must be a model family the reasoning, sampling and vision gat
The shipped constant in `src/lib/ai/codex-slug-chain.ts` is authoritative; when the upstream rotates again, that file moves first and this listing follows.
A fixed chain goes stale. From 2026-10-14, when GPT-5.5 retired, every model on the chain HealthLog shipped then was retired, and every Codex request answered 503 "all configured Codex slugs were rejected" until the chain was updated or `CODEX_MODEL` was set by hand. Reading the account's live list (`GET /backend-api/codex/models`, as the Codex CLI does) would avoid this.
The chain is overridable via the env var `CODEX_MODEL_FALLBACK_CHAIN` (comma-separated). The first env-var-named slug supersedes `CODEX_MODEL`; if both are set, `CODEX_MODEL` is folded into position 0 and de-duplicated.
**Trigger condition** (when to walk to the next slug):
@@ -65,6 +65,8 @@ const NOT_MODEL_INPUT: Record<string, string> = {
"links stored rows to their biomarker; no model is involved",
"src/lib/jobs/lab-biomarker-backfill.ts#enqueueBootTimeLabBiomarkerBackfill":
"finds the accounts whose rows need linking; no model is involved",
"src/lib/jobs/lab-document-date-repair.ts#candidateQuery":
"finds document lab values filed at their confirmation moment to re-date them; no model is involved",
"src/lib/jobs/soft-delete-retention.ts#SOFT_DELETE_RETENTION":
"purges rows past the trash window; no model is involved",
"src/lib/insights/correlation-channel-series.ts#fetchLabDraws":
@@ -43,7 +43,8 @@ async function confirmDocument(
const document = await prisma.inboundDocument.findFirst({
where: { id, userId: user.id, deletedAt: null },
select: { id: true },
// The two dates date a value that states none of its own.
select: { id: true, reportDate: true, documentDate: true },
});
if (!document) {
return apiError("Document not found", 404, {
@@ -94,10 +95,15 @@ async function confirmDocument(
try {
// v1.43 — an OBSERVATION may name the marker the person chose
// on the review screen; the label is then remembered as its alias.
const ref = await commitApprovedFact(user.id, fact, {
biomarkerId: decision.biomarkerId,
rememberAlias: decision.rememberAlias,
});
const ref = await commitApprovedFact(
user.id,
fact,
{
biomarkerId: decision.biomarkerId,
rememberAlias: decision.rememberAlias,
},
document,
);
await prisma.extractedFact.update({
where: { id: fact.id },
data: {
+39 -8
View File
@@ -23,6 +23,7 @@ import {
} from "@/lib/measurements/apple-health-mirror";
import { invalidateUserMeasurements } from "@/lib/cache/invalidate";
import { afterMeasurementMutation } from "@/lib/rollups/after-measurement-mutation";
import { bloodPressureSiblings } from "@/lib/measurements/blood-pressure-reading";
import { Prisma, type MeasurementType } from "@/generated/prisma/client";
import { NextRequest } from "next/server";
import { z } from "zod";
@@ -237,18 +238,45 @@ export const DELETE = apiHandler(
// (see `measurements/route.ts:100`), so the row is invisible to
// normal reads from this point on. A row that is already tombstoned
// re-bumps `syncVersion` harmlessly (idempotent re-delete).
await prisma.measurement.update({
where: { id },
data: {
deletedAt: new Date(),
syncVersion: { increment: 1 },
},
//
// A systolic or diastolic takes the rest of its reading with it (the
// other half and the cuff's pulse, `bloodPressureSiblings`), in the same
// transaction and with the same `deletedAt`, which is how the restore
// route knows to bring them back together. Each sibling bumps its own
// `syncVersion`, so a paired client sees every row's tombstone.
const deletedAt = new Date();
const siblings = await prisma.$transaction(async (tx) => {
const found = await bloodPressureSiblings(
tx,
user.id,
[existing],
"live",
);
await tx.measurement.update({
where: { id },
data: { deletedAt, syncVersion: { increment: 1 } },
});
if (found.length > 0) {
await tx.measurement.updateMany({
where: {
id: { in: found.map((s) => s.id) },
userId: user.id,
deletedAt: null,
},
data: { deletedAt, syncVersion: { increment: 1 } },
});
}
return found;
});
await auditLog("measurement.delete", {
userId: user.id,
ipAddress: getClientIp(request),
details: { measurementId: id, type: existing.type },
details: {
measurementId: id,
type: existing.type,
readingRows: siblings.length,
},
});
annotate({
@@ -269,7 +297,10 @@ export const DELETE = apiHandler(
// the rollup row when the day's measurement count goes to zero).
await afterMeasurementMutation(
user.id,
[{ type: existing.type, measuredAt: existing.measuredAt }],
[
{ type: existing.type, measuredAt: existing.measuredAt },
...siblings.map((s) => ({ type: s.type, measuredAt: s.measuredAt })),
],
{ source: "manual" },
);
@@ -10,15 +10,19 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { NextRequest } from "next/server";
vi.mock("@/lib/db", () => ({
prisma: {
vi.mock("@/lib/db", () => {
const prisma = {
measurement: {
findMany: vi.fn(),
updateMany: vi.fn(),
},
auditLog: { create: vi.fn() },
},
}));
// The read and the write run in one transaction; the stub hands the
// callback the same mocked client.
$transaction: vi.fn(async (fn: (tx: unknown) => unknown) => fn(prisma)),
};
return { prisma };
});
vi.mock("@/lib/auth/session", () => ({ getSession: vi.fn() }));
vi.mock("@/lib/auth/audit", () => ({
@@ -113,8 +117,16 @@ describe("POST /api/measurements/bulk-delete", () => {
it("soft-deletes only owned, non-tombstoned rows and returns that count", async () => {
// Caller asked for 3 ids but only 2 are owned + live.
vi.mocked(prisma.measurement.findMany).mockResolvedValue([
{ type: "WEIGHT", measuredAt: new Date("2026-01-01T08:00:00Z") },
{ type: "WEIGHT", measuredAt: new Date("2026-01-01T20:00:00Z") },
{
id: "m1",
type: "WEIGHT",
measuredAt: new Date("2026-01-01T08:00:00Z"),
},
{
id: "m2",
type: "WEIGHT",
measuredAt: new Date("2026-01-01T20:00:00Z"),
},
] as never);
vi.mocked(prisma.measurement.updateMany).mockResolvedValue({
count: 2,
@@ -129,7 +141,9 @@ describe("POST /api/measurements/bulk-delete", () => {
// never 404s, it's just excluded from the count.
const call = vi.mocked(prisma.measurement.updateMany).mock.calls[0][0];
expect(call.where).toMatchObject({
id: { in: ["m1", "m2", "foreign"] },
// The write names the rows the scoped read found, never the
// forged id the caller sent.
id: { in: ["m1", "m2"] },
userId: "user-1",
deletedAt: null,
});
+23 -10
View File
@@ -7,6 +7,8 @@
*
* Soft-deletes (tombstones) every owned, not-already-deleted row in one
* `updateMany` — matching the single-DELETE iOS tombstone contract. A
* selected blood-pressure half takes the rest of its reading with it, as the
* single DELETE does, and `deleted` counts every row tombstoned. A
* forged / foreign id is a silent no-op (never a 404 existence leak): the
* mutation `where` pins `userId` so it only ever touches the caller's rows.
*
@@ -26,6 +28,7 @@ import { withIdempotency } from "@/lib/idempotency";
import { checkRateLimit } from "@/lib/rate-limit";
import { invalidateUserMeasurements } from "@/lib/cache/invalidate";
import { afterMeasurementMutation } from "@/lib/rollups/after-measurement-mutation";
import { bloodPressureSiblings } from "@/lib/measurements/blood-pressure-reading";
import type { MeasurementType } from "@/generated/prisma/client";
const MAX_IDS_PER_BATCH = 200;
@@ -73,17 +76,27 @@ async function postBulkDelete(request: NextRequest): Promise<Response> {
// a status-insight cache drop. Ownership-scoped + `deletedAt: null` so
// a forged / foreign / already-tombstoned id is excluded here exactly
// as it is from the mutation below — no existence leak.
const affected = await prisma.measurement.findMany({
where: { id: { in: ids }, userId: user.id, deletedAt: null },
select: { type: true, measuredAt: true },
});
//
// A selected systolic or diastolic brings the rest of its reading (the
// other half and the cuff's pulse, `bloodPressureSiblings`) into the same
// statement, so the shared `deletedAt` lets the restore route bring the
// reading back whole.
const { affected, count } = await prisma.$transaction(async (tx) => {
const selected = await tx.measurement.findMany({
where: { id: { in: ids }, userId: user.id, deletedAt: null },
select: { id: true, type: true, source: true, measuredAt: true },
});
const siblings = await bloodPressureSiblings(tx, user.id, selected, "live");
const rowIds = [...selected, ...siblings].map((row) => row.id);
// Soft-delete (tombstone) in one statement. The `where` pins `userId`
// so a forged id for another user is silently a no-op; `deletedAt: null`
// keeps the count honest (an already-tombstoned id doesn't re-count).
const { count } = await prisma.measurement.updateMany({
where: { id: { in: ids }, userId: user.id, deletedAt: null },
data: { deletedAt: now, syncVersion: { increment: 1 } },
// Soft-delete (tombstone) in one statement. The `where` pins `userId`
// so a forged id for another user is silently a no-op; `deletedAt: null`
// keeps the count honest (an already-tombstoned id doesn't re-count).
const { count } = await tx.measurement.updateMany({
where: { id: { in: rowIds }, userId: user.id, deletedAt: null },
data: { deletedAt: now, syncVersion: { increment: 1 } },
});
return { affected: [...selected, ...siblings], count };
});
await auditLog("measurement.delete.bulk", {
@@ -10,15 +10,19 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import { NextRequest } from "next/server";
vi.mock("@/lib/db", () => ({
prisma: {
vi.mock("@/lib/db", () => {
const prisma = {
measurement: {
findMany: vi.fn(),
updateMany: vi.fn(),
},
auditLog: { create: vi.fn() },
},
}));
// The read and the write run in one transaction; the stub hands the
// callback the same mocked client.
$transaction: vi.fn(async (fn: (tx: unknown) => unknown) => fn(prisma)),
};
return { prisma };
});
vi.mock("@/lib/auth/session", () => ({ getSession: vi.fn() }));
vi.mock("@/lib/auth/audit", () => ({
@@ -110,8 +114,16 @@ describe("POST /api/measurements/restore", () => {
it("un-tombstones only owned, tombstoned rows and returns that count", async () => {
// Caller asked for 3 ids but only 2 are owned + tombstoned.
vi.mocked(prisma.measurement.findMany).mockResolvedValue([
{ type: "WEIGHT", measuredAt: new Date("2026-01-01T08:00:00Z") },
{ type: "WEIGHT", measuredAt: new Date("2026-01-01T20:00:00Z") },
{
id: "m1",
type: "WEIGHT",
measuredAt: new Date("2026-01-01T08:00:00Z"),
},
{
id: "m2",
type: "WEIGHT",
measuredAt: new Date("2026-01-01T20:00:00Z"),
},
] as never);
vi.mocked(prisma.measurement.updateMany).mockResolvedValue({
count: 2,
@@ -126,7 +138,9 @@ describe("POST /api/measurements/restore", () => {
// forged / live id never 404s, it's just excluded from the count.
const call = vi.mocked(prisma.measurement.updateMany).mock.calls[0][0];
expect(call.where).toMatchObject({
id: { in: ["m1", "m2", "foreign"] },
// The write names the rows the scoped read found, never the
// forged id the caller sent.
id: { in: ["m1", "m2"] },
userId: "user-1",
deletedAt: { not: null },
});
+33 -10
View File
@@ -28,6 +28,7 @@ import { withIdempotency } from "@/lib/idempotency";
import { checkRateLimit } from "@/lib/rate-limit";
import { invalidateUserMeasurements } from "@/lib/cache/invalidate";
import { afterMeasurementMutation } from "@/lib/rollups/after-measurement-mutation";
import { bloodPressureSiblings } from "@/lib/measurements/blood-pressure-reading";
import type { MeasurementType } from "@/generated/prisma/client";
const MAX_IDS_PER_BATCH = 200;
@@ -73,17 +74,39 @@ async function postRestore(request: NextRequest): Promise<Response> {
// a status-insight cache drop. Ownership-scoped + `deletedAt != null`
// so a forged / foreign / never-deleted id is excluded here exactly as
// it is from the mutation below — no existence leak.
const affected = await prisma.measurement.findMany({
where: { id: { in: ids }, userId: user.id, deletedAt: { not: null } },
select: { type: true, measuredAt: true },
});
//
// A blood-pressure half brings back the rows of its reading that were
// tombstoned in the same statement (identical `deletedAt`,
// `bloodPressureSiblings`), so an undo that names only the row the person
// tapped restores the reading the delete took. A row deleted on its own,
// at another moment, stays deleted.
const { affected, count } = await prisma.$transaction(async (tx) => {
const selected = await tx.measurement.findMany({
where: { id: { in: ids }, userId: user.id, deletedAt: { not: null } },
select: {
id: true,
type: true,
source: true,
measuredAt: true,
deletedAt: true,
},
});
const siblings = await bloodPressureSiblings(
tx,
user.id,
selected,
"deletedWith",
);
const rowIds = [...selected, ...siblings].map((row) => row.id);
// Un-tombstone in one statement. The `where` pins `userId` so a forged
// id for another user is silently a no-op; `deletedAt: { not: null }`
// keeps the count honest (a live row doesn't re-count).
const { count } = await prisma.measurement.updateMany({
where: { id: { in: ids }, userId: user.id, deletedAt: { not: null } },
data: { deletedAt: null, syncVersion: { increment: 1 } },
// Un-tombstone in one statement. The `where` pins `userId` so a forged
// id for another user is silently a no-op; `deletedAt: { not: null }`
// keeps the count honest (a live row doesn't re-count).
const { count } = await tx.measurement.updateMany({
where: { id: { in: rowIds }, userId: user.id, deletedAt: { not: null } },
data: { deletedAt: null, syncVersion: { increment: 1 } },
});
return { affected: [...selected, ...siblings], count };
});
await auditLog("measurement.restore", {
@@ -137,6 +137,49 @@ describe("Codex reasoning request", () => {
expect(body.include).toEqual([]);
expect(result.reasoning?.downgradedTo).toBeUndefined();
});
it("on a slug that refuses none, steps up to its lowest effort and stays on the slug", async () => {
// gpt-6.1-sol, a rung on the default list, refuses "none" (seen 2026-10-24).
// That refusal is about the effort, not the model: no walk to the next slug.
const refusal = JSON.stringify({
error: {
message:
"Unsupported value: 'none' is not supported with the 'gpt-6.1-sol' model. Supported values are: 'low', 'medium', 'high', 'xhigh', and 'max'.",
type: "invalid_request_error",
param: "reasoning.effort",
code: "unsupported_value",
},
});
const mock = vi
.fn()
.mockResolvedValueOnce(errorResponse(400, refusal))
.mockImplementation(async () =>
sseResponse(fixture("codex-after-tool.sse")),
);
vi.stubGlobal("fetch", mock);
const c = new CodexClient({
accessToken: "test-token",
accountId: "acct-test",
onTokenRefresh: vi
.fn()
.mockResolvedValue({ accessToken: "x", accountId: "acct-test" }),
slugChain: ["gpt-6.1-sol", "gpt-6-sol"],
});
await c.generateCompletion(
params({ reasoning: { effort: "off", summaries: false } }),
);
expect(sentBody(mock, 0).reasoning).toEqual({ effort: "none" });
expect(sentBody(mock, 1).model).toBe("gpt-6.1-sol");
expect(sentBody(mock, 1).reasoning).toEqual({ effort: "low" });
await c.generateCompletion(
params({ reasoning: { effort: "off", summaries: false } }),
);
expect(mock).toHaveBeenCalledTimes(3);
expect(sentBody(mock, 2).model).toBe("gpt-6.1-sol");
expect(sentBody(mock, 2).reasoning).toEqual({ effort: "low" });
});
});
describe("Codex reasoning stream", () => {
@@ -91,6 +91,48 @@ describe("Codex slug fallback chain", () => {
clearCodexSlugCache();
});
it("an account without the head of the default list walks on to the next rung", async () => {
const origModel = process.env.CODEX_MODEL;
const origChain = process.env.CODEX_MODEL_FALLBACK_CHAIN;
delete process.env.CODEX_MODEL;
delete process.env.CODEX_MODEL_FALLBACK_CHAIN;
try {
// The exact refusal a ChatGPT sign-in gets for a retired model.
const mockFetch = vi
.fn()
.mockResolvedValueOnce(
rejectionResponse(
400,
'{"detail":"The \'gpt-5.6-terra\' model is not supported when using Codex with a ChatGPT account.","error":{"type":"invalid_request_error","code":"model_not_found"}}',
),
)
.mockResolvedValueOnce(sseSuccessResponse());
vi.stubGlobal("fetch", mockFetch);
const client = new CodexClient({
accessToken: "test",
accountId: "acct",
onTokenRefresh,
});
await client.generateCompletion(
singleUserTurn({ system: "s", user: "u" }),
);
expect(client.getLastDiagnostics()?.attempted).toEqual([
"gpt-5.6-terra",
"gpt-6-sol",
]);
expect(client.getLastDiagnostics()?.workingSlug).toBe("gpt-6-sol");
expect(getCachedCodexSlug()).toBe("gpt-6-sol");
} finally {
if (origModel === undefined) delete process.env.CODEX_MODEL;
else process.env.CODEX_MODEL = origModel;
if (origChain === undefined)
delete process.env.CODEX_MODEL_FALLBACK_CHAIN;
else process.env.CODEX_MODEL_FALLBACK_CHAIN = origChain;
}
});
it("first slug succeeds — uses head-of-chain and caches it", async () => {
const mockFetch = vi.fn().mockResolvedValue(sseSuccessResponse());
vi.stubGlobal("fetch", mockFetch);
@@ -340,6 +382,14 @@ describe("Codex slug fallback chain", () => {
).toBe(true);
expect(helper(400, '{"error":{"code":"model_not_found"}}')).toBe(true);
expect(helper(400, "model gpt-5 does not exist")).toBe(true);
// The refusal a ChatGPT sign-in got for gpt-5.5 after its retirement
// (seen 2026-10-24).
expect(
helper(
400,
'{"detail":"The \'gpt-5.5\' model is not supported when using Codex with a ChatGPT account.","error":{"type":"invalid_request_error","code":"model_not_found"}}',
),
).toBe(true);
expect(helper(404, "anything")).toBe(true);
expect(helper(500, "internal error")).toBe(false);
expect(helper(429, "rate limited")).toBe(false);
@@ -360,6 +410,17 @@ describe("Codex slug fallback chain", () => {
"gpt-5.6-sol",
"gpt-5.5",
]);
// OpenAI retired these from Codex for ChatGPT sign-in: 5.2 and 5.3-Codex
// on 2026-05-26, 5.4 and 5.4 mini on 2026-08-31. A retired rung costs
// every account that reaches it a refused request.
for (const retired of [
"gpt-5.4",
"gpt-5.4-mini",
"gpt-5.3-codex",
"gpt-5.2",
]) {
expect(__test.DEFAULT_SLUG_FALLBACK_CHAIN).not.toContain(retired);
}
});
it("DEFAULT_SLUG_FALLBACK_CHAIN holds only slugs every family gate recognises", () => {
@@ -45,16 +45,22 @@ interface SeenRequest {
* `always`: the thought fills the ceiling whatever the effort (gpt-oss, GLM).
* `answers-unreasoned`: the same, except asked with `none` it answers
* (Gemma 4 on Ollama).
* `answers-with-room`: ignores the effort and answers once the ceiling leaves
* it the `high` thinking room (GLM 5.3 behind a gateway, #1276).
*/
let mode: "always" | "answers-unreasoned" = "always";
let mode: "always" | "answers-unreasoned" | "answers-with-room" = "always";
let server: http.Server;
let origin: string;
let seen: SeenRequest[] = [];
const ANSWER = JSON.stringify({ summary: "Steady." });
function reply(effort: unknown) {
const answers = mode === "answers-unreasoned" && effort === "none";
function reply(effort: unknown, maxTokens: unknown) {
const answers =
(mode === "answers-unreasoned" && effort === "none") ||
(mode === "answers-with-room" &&
typeof maxTokens === "number" &&
maxTokens >= 12_000);
return {
id: "chatcmpl-1279",
object: "chat.completion",
@@ -87,7 +93,14 @@ beforeAll(async () => {
maxTokens: body.max_tokens ?? body.max_completion_tokens,
});
res.writeHead(200, { "Content-Type": "application/json" });
res.end(JSON.stringify(reply(body.reasoning_effort)));
res.end(
JSON.stringify(
reply(
body.reasoning_effort,
body.max_tokens ?? body.max_completion_tokens,
),
),
);
});
});
await new Promise<void>((r) => server.listen(0, "127.0.0.1", r));
@@ -210,3 +223,77 @@ describe.each([
});
},
);
/** A daily briefing call as the chain runner hands it to a client. */
function briefingParams(unattended: boolean): CompletionParams {
return {
...singleUserTurn({
system: "You write the daily briefing.",
user: "Snapshot: …",
maxTokens: 300,
timeoutMs: 30_000,
responseFormat: "json",
}),
reasoning: { effort: "medium", summaries: false },
...(unattended ? { unattended: true } : {}),
};
}
describe.each([
["the OpenAI-compatible gateway", gateway],
["the Local client", local],
] as const)(
"#1276 — %s, background work for a model that cannot be switched down",
(_, make) => {
it("gives the #1267 repeat the High room, and answers on the first call", async () => {
mode = "answers-with-room";
const result = await make().generateCompletion(briefingParams(true));
expect(result.content).toBe(ANSWER);
// Two requests, the call's budget: medium at its own room, then the
// repeat with `none` and the High room the reply showed it needs.
expect(seen).toEqual([
{ reasoningEffort: "medium", maxTokens: 4396 },
{ reasoningEffort: "none", maxTokens: 12_300 },
]);
// Every later background call starts with the High room.
seen = [];
await make().generateCompletion(briefingParams(true));
expect(seen).toEqual([{ reasoningEffort: "none", maxTokens: 12_300 }]);
});
it("gives a status card, which asks no level, the same room", async () => {
mode = "answers-with-room";
const result = await make().generateCompletion({
...statusParams(),
unattended: true,
});
expect(result.content).toBe(ANSWER);
expect(seen.map((s) => s.maxTokens)).toEqual([300, 12_300]);
});
it("keeps the High room once even `none` is learned useless, and stays at one request", async () => {
const err = (await make()
.generateCompletion(briefingParams(true))
.catch((e) => e)) as Record<string, unknown>;
expect(err.kind).toBe("reasoning_only");
expect(seen.map((s) => s.maxTokens)).toEqual([4396, 12_300]);
seen = [];
await make()
.generateCompletion(briefingParams(true))
.catch(() => {});
expect(seen).toEqual([{ reasoningEffort: "medium", maxTokens: 12_300 }]);
});
it("leaves a call a person is reading at the room of its level", async () => {
await make()
.generateCompletion(briefingParams(false))
.catch(() => {});
seen = [];
await make()
.generateCompletion(briefingParams(false))
.catch(() => {});
expect(seen).toEqual([{ reasoningEffort: "medium", maxTokens: 4396 }]);
});
},
);
@@ -272,3 +272,29 @@ describe("#1279: background work and a benched provider", () => {
expect((err as AllProvidersFailedError).tokensSpent).toBe(5400);
});
});
describe("#1276 — the runner marks background calls", () => {
class Recorder implements AIProvider {
readonly type = "openai-compatible" as const;
seen: CompletionParams[] = [];
async generateCompletion(p: CompletionParams): Promise<CompletionResult> {
this.seen.push(p);
return { content: "{}", tokensUsed: 1 } as CompletionResult;
}
}
it.each([
["job", true],
["coach", undefined],
] as const)("a %s walk carries unattended: %s", async (surface, expected) => {
const instance = new Recorder();
await walk(
surface,
`u-unattended-${surface}`,
"openai-compatible",
instance,
createInMemoryProviderHealthLedger(),
);
expect(instance.seen[0]?.unattended).toBe(expected);
});
});
+5 -7
View File
@@ -27,11 +27,10 @@ import {
noteReasoningSeen,
refusesReasoningField,
} from "./reasoning/dialect-cache";
import { uncontrolledThinkingRoom } from "./reasoning/levels";
import {
askAgainUnreasoned,
knownReasoner,
reasonsUnasked,
uncontrolledCallThinkingRoom,
repeatUnreasoned,
} from "./reasoning/answerless";
import {
@@ -149,11 +148,10 @@ export class LocalOpenAICompatibleClient implements AIProvider {
// `none` sent for Off or after the #1267 lesson must not leave it the
// bare answer budget to think in. A call without a level gets the `low`
// room once the model has shown it reasons unasked.
const thinkingTokens = params.reasoning
? uncontrolledThinkingRoom(params.reasoning.effort)
: reasonsUnasked(this.reasoningDialectKey())
? uncontrolledThinkingRoom("off")
: 0;
const thinkingTokens = uncontrolledCallThinkingRoom(
this.reasoningDialectKey(),
params,
);
const body = JSON.stringify({
model: this.config.model,
messages,
+5 -12
View File
@@ -21,15 +21,12 @@ import {
import {
askAgainUnreasoned,
knownReasoner,
reasonsUnasked,
uncontrolledCallThinkingRoom,
repeatUnreasoned,
} from "./reasoning/answerless";
import { RequestHops } from "./request-hops";
import { learnFromRefusal } from "./wire-refusal";
import {
REASONING_THINKING_BUDGET,
uncontrolledThinkingRoom,
} from "./reasoning/levels";
import { REASONING_THINKING_BUDGET } from "./reasoning/levels";
import {
answerOf,
isOpenRouterEndpoint,
@@ -573,19 +570,15 @@ export class OpenAIClient implements AIProvider {
* included, and a `none` sent for the person's Off or after the #1267
* lesson must not leave it the bare answer budget to think in. Without a
* level of its own, a call gets the `low` room once the model has shown it
* reasons unasked.
* reasons unasked, and background work the `high` room once the model has
* answered with reasoning alone (`unattendedThinkingRoom`).
*/
private thinkingRoom(
params: CompletionParams,
wire: OpenAIReasoningWire,
): number {
if (this.isGateway) {
if (params.reasoning) {
return uncontrolledThinkingRoom(params.reasoning.effort);
}
return reasonsUnasked(this.reasoningDialectKey())
? uncontrolledThinkingRoom("off")
: 0;
return uncontrolledCallThinkingRoom(this.reasoningDialectKey(), params);
}
if (wire.kind === "none" || wire.effort === "none") return 0;
const level = levelOfWireEffort(wire.effort);
+8 -3
View File
@@ -743,14 +743,17 @@ export async function runStreamingRawCompletionWithFallback(
onDelta: (delta: string) => void;
},
): Promise<RunRawWithFallbackResult> {
// #1276 — a job walk is background work; stamped here, not per call site.
const params: CompletionParams =
args.surface === "job" ? { ...args.params, unattended: true } : args.params;
return runRawChain(
args,
// Prefer the provider's streaming wire; fall back to the buffered call for
// providers (cloud clients today) that do not implement streaming.
(candidate) =>
candidate.instance.generateCompletionStream
? candidate.instance.generateCompletionStream(args.params, args.onDelta)
: candidate.instance.generateCompletion(args.params),
? candidate.instance.generateCompletionStream(params, args.onDelta)
: candidate.instance.generateCompletion(params),
(candidate) => ({
ai_chain_streamed: Boolean(candidate.instance.generateCompletionStream),
}),
@@ -789,7 +792,9 @@ export interface RunRawWithFallbackResult {
export async function runRawCompletionWithFallback(
args: ChainWalkArgs,
): Promise<RunRawWithFallbackResult> {
const params: CompletionParams =
args.surface === "job" ? { ...args.params, unattended: true } : args.params;
return runRawChain(args, (candidate) =>
candidate.instance.generateCompletion(args.params),
candidate.instance.generateCompletion(params),
);
}
+48
View File
@@ -36,6 +36,11 @@ import {
rememberReasoningDialect,
type ReasoningDialectKey,
} from "./dialect-cache";
import {
REASONING_THINKING_BUDGET,
uncontrolledThinkingRoom,
type ReasoningLevel,
} from "./levels";
/**
* What the client does next with an answerless reply.
@@ -137,6 +142,49 @@ export function reasonsUnasked(key: ReasoningDialectKey): boolean {
);
}
/**
* #1276 — the thinking room of an unattended call (`CompletionParams.unattended`)
* to a model that has answered with reasoning alone (`reasonsUnasked`): at
* least the `high` room the Coach gets on High. Background work asks for
* `medium` at most, and a model that cannot be switched down (GLM behind
* vLLM) spends all of it thinking and answers nothing, the daily briefing's
* `reasoning_only`. The room applies from the #1267 repeat with `none` on:
* one call may send two requests (`request-hops.ts`), and a once-a-day job
* meets an expired lesson every morning, so the room has to ride on the
* repeat or the first job of the day never gets it. A ceiling, not a charge:
* a model that does stop at `none` never touches it, one that does not
* thinks as long as it does either way and reaches the answer. Every call a
* person is reading keeps `room`.
*/
export function unattendedThinkingRoom(
key: ReasoningDialectKey,
room: number,
unattended: boolean | undefined,
): number {
return unattended && reasonsUnasked(key)
? Math.max(room, REASONING_THINKING_BUDGET.high)
: room;
}
/**
* #1276 — the thinking room of one call on a wire whose reasoning switch the
* client cannot verify (a gateway, a local server): by the call's level, not
* by what the wire says, the `low` room for a call without a level once the
* model has shown it reasons unasked, and the `high` room for background work
* once it has answered with reasoning alone (`unattendedThinkingRoom`).
*/
export function uncontrolledCallThinkingRoom(
key: ReasoningDialectKey,
params: { reasoning?: { effort: ReasoningLevel }; unattended?: boolean },
): number {
const room = params.reasoning
? uncontrolledThinkingRoom(params.reasoning.effort)
: reasonsUnasked(key)
? uncontrolledThinkingRoom("off")
: 0;
return unattendedThinkingRoom(key, room, params.unattended);
}
/**
* #1276 — true when the model has shown it reasons, for a request that did
* (`asked`) or did not ask it to: a lesson that says it reasons unasked, or a
+8
View File
@@ -383,6 +383,14 @@ export interface CompletionParams {
* client sends exactly what it sent before the field existed.
*/
reasoning?: { effort: ReasoningLevel; summaries: boolean };
/**
* #1276 — no person reads this reply as it streams: the call runs as
* background work. Set by the chain runner for every `"job"` walk, never by
* a call site. The gateway and local clients give such a call the `high`
* thinking room once its model has answered with reasoning alone
* (`unattendedThinkingRoom`); nothing else reads it.
*/
unattended?: boolean;
/**
* v1.41.2 — a stable key for calls that resend the same prompt prefix (the
* rounds of one Coach turn, the turns of one conversation). Providers that
+188 -1
View File
@@ -12,6 +12,7 @@ import { beforeEach, describe, expect, it, vi } from "vitest";
vi.mock("@/lib/db", () => ({
prisma: {
labResult: { create: vi.fn() },
illnessEpisode: { create: vi.fn() },
},
}));
@@ -27,8 +28,16 @@ import { commitApprovedFact, FactCommitError } from "@/lib/documents/commit";
import { prisma } from "@/lib/db";
import { resolveOrMintBiomarker } from "@/lib/labs/biomarker-store";
import { decryptFactData } from "@/lib/documents/store";
import type { ExtractedFact } from "@/generated/prisma/client";
import type {
ExtractedFact,
IllnessEpisode,
LabResult,
} from "@/generated/prisma/client";
import { userDayKey } from "@/lib/tz/format";
import type {
ConditionFactData,
ObservationFactData,
} from "@/lib/validations/inbound-documents";
function observationFact(): ExtractedFact {
return {
@@ -213,3 +222,181 @@ describe("commitApprovedFact — OBSERVATION unit integrity", () => {
).resolves.toEqual({ recordType: "labResult", recordId: "lr-2" });
});
});
describe("commitApprovedFact — OBSERVATION date when the value states none", () => {
// A lab report that prints the collection date once in its header carries
// no date beside each value. Mutation check: drop the document day from
// `takenAt` → the report-date, filing-date and impossible-date cases file
// the reading under today.
function undated(effectiveDate: string | null = null) {
const data: ObservationFactData = {
label: "Ferritin",
code: null,
codeSystem: null,
value: 80,
valueText: null,
unit: "ng/mL",
referenceLow: null,
referenceHigh: null,
effectiveDate,
};
vi.mocked(decryptFactData).mockReturnValue(data);
vi.mocked(resolveOrMintBiomarker).mockResolvedValue({
id: "bm-3",
name: "Ferritin",
unit: "ng/mL",
lowerBound: null,
upperBound: null,
panel: null,
analyteKey: null,
});
vi.mocked(prisma.labResult.create).mockResolvedValue({
id: "lr-3",
} as LabResult);
}
/** The dates as the confirm route reads them off the stored document. */
function dates(reportDate: string | null, documentDate: string | null) {
return {
// An older report date sits at UTC midnight, newer ones at noon UTC.
reportDate: reportDate ? new Date(`${reportDate}T00:00:00.000Z`) : null,
documentDate: documentDate
? new Date(`${documentDate}T12:00:00.000Z`)
: null,
};
}
function storedTakenAt(): Date {
return vi.mocked(prisma.labResult.create).mock.calls[0]![0].data
.takenAt as Date;
}
it("keeps the value's own date", async () => {
undated("2021-03-04");
await commitApprovedFact(
"user-1",
observationFact(),
{},
dates("2021-03-01", "2026-10-20"),
);
expect(storedTakenAt().toISOString()).toBe("2021-03-04T12:00:00.000Z");
});
it("takes the report date the document states over its filing date", async () => {
undated();
await commitApprovedFact(
"user-1",
observationFact(),
{},
dates("2021-03-01", "2026-10-20"),
);
expect(storedTakenAt().toISOString()).toBe("2021-03-01T12:00:00.000Z");
});
it("takes the filing date when the document states no report date", async () => {
undated();
await commitApprovedFact(
"user-1",
observationFact(),
{},
dates(null, "2021-02-15"),
);
expect(storedTakenAt().toISOString()).toBe("2021-02-15T12:00:00.000Z");
});
it.each([
["not a date", "March 2021"],
["a day the month does not have", "2021-02-30"],
["a month that does not exist", "2021-13-01"],
])(
"reads a value whose own date is %s as undated",
async (_label, effectiveDate) => {
undated(effectiveDate);
await commitApprovedFact(
"user-1",
observationFact(),
{},
dates("2021-03-01", null),
);
// Not 2 March for 30 February, and no Invalid Date for month 13.
expect(storedTakenAt().toISOString()).toBe("2021-03-01T12:00:00.000Z");
},
);
it("files the reading under now when neither the value nor the document has a date", async () => {
undated();
const before = Date.now();
await commitApprovedFact(
"user-1",
observationFact(),
{},
dates(null, null),
);
const after = Date.now();
const takenAt = storedTakenAt().getTime();
expect(takenAt).toBeGreaterThanOrEqual(before);
expect(takenAt).toBeLessThanOrEqual(after);
});
});
describe("commitApprovedFact — CONDITION onset", () => {
// The onset goes through the same stated-day check as a lab value's date.
function condition(onsetDate: string | null) {
const data: ConditionFactData = {
label: "Iron deficiency",
code: null,
codeSystem: null,
clinicalStatus: null,
verificationStatus: null,
onsetDate,
};
vi.mocked(decryptFactData).mockReturnValue(data);
vi.mocked(prisma.illnessEpisode.create).mockResolvedValue({
id: "ep-1",
} as IllnessEpisode);
}
function conditionFact(): ExtractedFact {
return {
id: "fact-2",
factType: "CONDITION",
dataEncrypted: new Uint8Array(),
} as unknown as ExtractedFact;
}
function storedOnset(): Date {
return vi.mocked(prisma.illnessEpisode.create).mock.calls[0]![0].data
.onsetAt as Date;
}
it("keeps a stated onset day", async () => {
condition("2021-03-04");
await commitApprovedFact("user-1", conditionFact());
expect(storedOnset().toISOString()).toBe("2021-03-04T12:00:00.000Z");
});
it("files an onset that is not a real day under now, not an invalid date", async () => {
// 2021-13-01 used to become an invalid date that failed the whole confirm.
condition("2021-13-01");
const before = Date.now();
await commitApprovedFact("user-1", conditionFact());
const after = Date.now();
const onset = storedOnset().getTime();
expect(onset).toBeGreaterThanOrEqual(before);
expect(onset).toBeLessThanOrEqual(after);
});
});
+45 -7
View File
@@ -44,7 +44,11 @@ import type {
MedicationStatementFactData,
ObservationFactData,
} from "@/lib/validations/inbound-documents";
import { dateOnlyAtNoonUtc, isDateOnlyKey } from "@/lib/tz/date-only";
import {
dateOnlyAtNoonUtc,
dateOnlyKey,
isCalendarDateKey,
} from "@/lib/tz/date-only";
/** A per-fact commit failure the caller maps to a per-fact 422 entry. */
export class FactCommitError extends Error {
@@ -61,14 +65,41 @@ export interface CommittedRecordRef {
recordId: string;
}
/** A stated day, or null when it is not a real calendar date. */
export function statedDay(date: string | null): string | null {
return date && isCalendarDateKey(date) ? date : null;
}
/**
* A stated YYYY-MM-DD as the instant a date-only value is stored at (noon
* UTC), or now when the document states none. UTC midnight read back as the
* UTC), else the fallback day, else now. UTC midnight read back as the
* previous day anywhere west of UTC.
*/
function statedDateOrNow(date: string | null): Date {
if (date && isDateOnlyKey(date)) return dateOnlyAtNoonUtc(date);
return new Date();
function statedDateOrNow(
date: string | null,
fallback: string | null = null,
): Date {
const day = statedDay(date) ?? fallback;
return day ? dateOnlyAtNoonUtc(day) : new Date();
}
/** The two dates a stored document carries, as the confirm route reads them. */
export interface FactDocumentDates {
reportDate: Date | null;
documentDate: Date | null;
}
/**
* The day a value from this document took place when it states none of its
* own: the report date the model read off the document, else the document's
* filing date, the same order the day view and the timeline place a document
* by. A lab report that prints the collection date once in its header carries
* no date beside each value, and every reading used to be filed under the day
* it was confirmed.
*/
export function documentDay(dates: FactDocumentDates | null): string | null {
const day = dates?.reportDate ?? dates?.documentDate ?? null;
return day ? dateOnlyKey(day) : null;
}
/**
@@ -84,6 +115,7 @@ async function commitObservation(
userId: string,
data: ObservationFactData,
choice: ObservationChoice = {},
fallbackDay: string | null = null,
): Promise<CommittedRecordRef> {
const isQualitative = typeof data.value !== "number";
// A numeric reading needs a unit so the catalog has something to mint with —
@@ -228,7 +260,7 @@ async function commitObservation(
sourceReferenceLow: storedSourceLow,
sourceReferenceHigh: storedSourceHigh,
sourceReferenceText: isQualitative ? null : sourceText,
takenAt: statedDateOrNow(data.effectiveDate),
takenAt: statedDateOrNow(data.effectiveDate, fallbackDay),
source: "DOCUMENT",
noteEncrypted: null,
},
@@ -317,12 +349,18 @@ export async function commitApprovedFact(
userId: string,
fact: ExtractedFact,
choice: ObservationChoice = {},
documentDates: FactDocumentDates | null = null,
): Promise<CommittedRecordRef> {
// Decrypt the staged payload at confirm time to write it into the normal
// structured store. Fail-closed: a bad key id throws and aborts the commit.
const data = decryptFactData(fact.dataEncrypted);
if (fact.factType === "OBSERVATION") {
return commitObservation(userId, data as ObservationFactData, choice);
return commitObservation(
userId,
data as ObservationFactData,
choice,
documentDay(documentDates),
);
}
if (fact.factType === "CONDITION") {
return commitCondition(userId, data as ConditionFactData);
@@ -112,6 +112,15 @@ describe("maintenance group queue policies", () => {
expectPolicy("maintenance", queue, policy);
});
// One-shot repairs on a fixed singleton key: a boot during a run must be
// able to queue the next pass, so only queued duplicates collapse.
it.each([["lab-document-date-repair", "short"]])(
"%s is %s",
(queue, policy) => {
expectPolicy("maintenance", queue, policy);
},
);
it("environment-fetch is deliberately left on standard", () => {
// Its explicit-range backfill sends with no key at all, on purpose, so any
// policy would collapse two different requested date ranges onto the shared
@@ -0,0 +1,10 @@
/**
* The queue facts of `lab-document-date-repair.ts`: its name and its one
* singleton key. They live apart from the job so the worker registry and the
* boot discovery can read them without loading it.
*/
export const LAB_DOCUMENT_DATE_REPAIR_QUEUE = "lab-document-date-repair";
/** One pass covers every account, so there is only ever one job. */
export const LAB_DOCUMENT_DATE_REPAIR_SINGLETON_KEY =
"lab-document-date-repair";
+234
View File
@@ -0,0 +1,234 @@
/**
* One-shot repair: lab values confirmed from a document before the confirm
* step learned to date an undated value by its document.
*
* A lab report usually prints the collection date once, in its header. The
* extraction stores that date on the document (`reportDate`) and leaves each
* value's own date empty, and the confirm step used to file such a value
* under the moment it was confirmed. The confirm step now takes the
* document's day (`documentDay` in `src/lib/documents/commit.ts`); this pass
* moves the values it filed wrongly before, by the same rule.
*
* A value is moved only when every link in the chain is certain:
* - the lab result was written by the document confirm path
* (`source = 'DOCUMENT'`, which no other writer sets) and is live;
* - exactly one APPROVED observation fact names it as its committed record,
* so the document it came from is known;
* - its `takenAt` still sits within a few seconds of its own `createdAt`,
* which is what a value filed under its confirmation moment looks like
* (both were stamped in the same create). A value the person re-dated
* since, or one that carried its own date, sits elsewhere;
* - the staged fact states no calendar date of its own (decrypted here;
* SQL cannot see it);
* - the document has a report date, else a filing date, and that day is
* not the day the value is filed under in the owner's time zone. A value
* really taken on the day it was confirmed stays where it is.
*
* The moved value lands at noon UTC on the document's day, the anchor every
* date-only lab value uses. Idempotent: a moved value no longer sits at its
* creation moment, so a second run finds nothing, and a boot with nothing to
* repair costs one query.
*/
import type { Job } from "pg-boss";
import { Prisma, type PrismaClient } from "@/generated/prisma/client";
import { activeAccountSql } from "@/lib/account-deletion/active-account";
import { invalidateUserHealthScore } from "@/lib/cache/invalidate";
import { prisma as defaultPrisma } from "@/lib/db";
import { documentDay, statedDay } from "@/lib/documents/commit";
import { decryptFactData } from "@/lib/documents/store";
import { getGlobalBoss } from "@/lib/jobs/boss-instance";
import { jobDone, jobFailed, type JobOutcome } from "@/lib/jobs/job-outcome";
import { workerLog } from "@/lib/jobs/reminder/shared";
import { annotate } from "@/lib/logging/context";
import { withBackgroundEvent } from "@/lib/logging/background";
import { dateOnlyAtNoonUtc } from "@/lib/tz/date-only";
import { userDayKey } from "@/lib/tz/format";
import { resolveUserTimezone } from "@/lib/tz/resolver";
import type { ObservationFactData } from "@/lib/validations/inbound-documents";
import {
LAB_DOCUMENT_DATE_REPAIR_QUEUE,
LAB_DOCUMENT_DATE_REPAIR_SINGLETON_KEY,
} from "./lab-document-date-repair-queue";
/**
* How far `takenAt` may sit from `createdAt` and still count as the moment of
* confirmation. The two are stamped a few milliseconds apart in one create; a
* few seconds leaves room for a clock step and nothing a person would type.
*/
const CONFIRM_MOMENT_TOLERANCE_SECONDS = 5;
interface CandidateRow {
id: string;
userId: string;
takenAt: Date;
dataEncrypted: Uint8Array;
reportDate: Date | null;
documentDate: Date | null;
}
/**
* The SQL half of the rule. `limit` turns it into the boot's existence
* check. `.planning/v1.46.2-fragments/lab-repair-count.sql` is the same
* predicate as a read-only count.
*/
function candidateQuery(limit: number | null): Prisma.Sql {
// The day comparison runs here as well as in the repair, so a value that
// really was taken on its confirmation day drops out of the boot check and
// the pass converges to an empty set. A zone Postgres does not know is read
// as UTC rather than failing the query; the repair re-checks every row in
// the zone the app resolves.
return Prisma.sql`
WITH "known_zones" AS (SELECT "name" FROM pg_timezone_names)
SELECT r."id",
r."user_id" AS "userId",
r."taken_at" AS "takenAt",
f."data_encrypted" AS "dataEncrypted",
d."report_date" AS "reportDate",
d."document_date" AS "documentDate"
FROM "lab_results" r
JOIN "extracted_facts" f
ON f."committed_record_id" = r."id"
AND f."committed_record_type" = 'labResult'
AND f."status" = 'APPROVED'
AND f."fact_type" = 'OBSERVATION'
AND f."user_id" = r."user_id"
JOIN "inbound_documents" d
ON d."id" = f."document_id"
AND d."user_id" = r."user_id"
JOIN "users" u ON u."id" = r."user_id"
WHERE r."source" = 'DOCUMENT'
AND r."deleted_at" IS NULL
AND abs(extract(epoch FROM (r."taken_at" - r."created_at"))) <= ${CONFIRM_MOMENT_TOLERANCE_SECONDS}
AND COALESCE(d."report_date", d."document_date") IS NOT NULL
AND to_char(COALESCE(d."report_date", d."document_date"), 'YYYY-MM-DD')
<> to_char(
(r."taken_at" AT TIME ZONE 'UTC') AT TIME ZONE (
CASE WHEN u."timezone" IN (SELECT "name" FROM "known_zones")
THEN u."timezone" ELSE 'UTC' END
),
'YYYY-MM-DD'
)
AND (
SELECT count(*) FROM "extracted_facts" other
WHERE other."committed_record_id" = r."id"
AND other."committed_record_type" = 'labResult'
) = 1
AND ${activeAccountSql('r."user_id"')}
ORDER BY r."id"
${limit === null ? Prisma.empty : Prisma.sql`LIMIT ${limit}`}
`;
}
export interface LabDocumentDateRepairResult {
/** Rows the SQL predicate returned. */
scanned: number;
/** Rows moved to their document's day. */
updated: number;
/** Rows left alone: an own date, or already on the document's day. */
unchanged: number;
}
/** The repair itself, across every account. */
export async function repairUndatedDocumentLabValues(
prisma: PrismaClient = defaultPrisma,
): Promise<LabDocumentDateRepairResult> {
const rows = await prisma.$queryRaw<CandidateRow[]>(candidateQuery(null));
const touchedUsers = new Set<string>();
let updated = 0;
let unchanged = 0;
for (const row of rows) {
const data = decryptFactData(row.dataEncrypted) as ObservationFactData;
// A value with a date of its own is never touched, whatever its row says.
if (statedDay(data.effectiveDate ?? null) !== null) {
unchanged += 1;
continue;
}
const day = documentDay({
reportDate: row.reportDate,
documentDate: row.documentDate,
});
const tz = await resolveUserTimezone(row.userId);
if (!day || userDayKey(row.takenAt, tz) === day) {
unchanged += 1;
continue;
}
// Conditional on the value read: a re-date that landed since the read
// wins over the repair.
const moved = await prisma.labResult.updateMany({
where: {
id: row.id,
userId: row.userId,
takenAt: row.takenAt,
deletedAt: null,
},
data: { takenAt: dateOnlyAtNoonUtc(day) },
});
if (moved.count === 0) {
unchanged += 1;
continue;
}
updated += 1;
touchedUsers.add(row.userId);
}
for (const userId of touchedUsers) invalidateUserHealthScore(userId);
return { scanned: rows.length, updated, unchanged };
}
/** The queue handler: one run over every account. */
export async function handleLabDocumentDateRepair(
_jobs: Job<Record<string, never>>[],
prisma: PrismaClient = defaultPrisma,
): Promise<JobOutcome> {
return withBackgroundEvent("job.lab_document_date_repair", async () => {
let result: LabDocumentDateRepairResult;
try {
result = await repairUndatedDocumentLabValues(prisma);
} catch (err) {
workerLog("error", "[lab-document-date-repair] failed", err);
return jobFailed("lab document date repair failed", err);
}
workerLog(
"info",
`[lab-document-date-repair] scanned=${result.scanned} updated=${result.updated} unchanged=${result.unchanged}`,
);
annotate({
action: { name: "labs.documentDate.repair" },
meta: { ...result },
});
return jobDone({
candidates_scanned: result.scanned,
updated: result.updated,
unchanged: result.unchanged,
});
});
}
/**
* Boot discovery: one existence query, and the one job only when there is
* something to repair.
*/
export async function enqueueBootTimeLabDocumentDateRepair(
prisma: PrismaClient = defaultPrisma,
): Promise<{ enqueued: boolean; error: string | null }> {
const boss = getGlobalBoss();
if (!boss) return { enqueued: false, error: null };
try {
const any = await prisma.$queryRaw<CandidateRow[]>(candidateQuery(1));
if (any.length === 0) return { enqueued: false, error: null };
const id = await boss.send(
LAB_DOCUMENT_DATE_REPAIR_QUEUE,
{},
{ singletonKey: LAB_DOCUMENT_DATE_REPAIR_SINGLETON_KEY, retryLimit: 2 },
);
return { enqueued: id !== null, error: null };
} catch (err) {
return {
enqueued: false,
error: err instanceof Error ? err.message : String(err),
};
}
}
+3
View File
@@ -485,6 +485,9 @@ export const QUEUE_RUNTIME: Readonly<Record<string, QueueRuntime>> = {
},
why: "One account's folded means, a day at a time; stops at three quarters of the default expiry and sends a follow-up that resumes at the next day.",
},
"lab-document-date-repair": short(
"Reads the document lab values still filed at their confirmation moment, a handful per account, and moves each with one update.",
),
"measurement-maintenance": {
runtime: "long",
expireInSeconds: MEASUREMENT_MAINTENANCE_SEND_OPTIONS.expireInSeconds,
@@ -77,6 +77,15 @@ export const MAINTENANCE_DISCOVERY: readonly BootDiscovery[] = [
await import("@/lib/jobs/measurement-fold-repair")
).enqueueBootTimeMeasurementFoldRepair(),
},
// Converges to nothing once the misdated values are moved; a boot with
// none costs one query.
{
name: "lab-document-date-repair",
run: async () =>
(
await import("@/lib/jobs/lab-document-date-repair")
).enqueueBootTimeLabDocumentDateRepair(),
},
// Self-limiting and idempotent; a boot with no backlog costs one query.
{
name: "compaction-tombstone-purge",
@@ -0,0 +1,25 @@
/**
* One-shot data repairs of the maintenance group, kept beside
* `maintenance.ts` so that file stays under the size ratchet. Each is sent
* by its boot discovery in `maintenance-discovery.ts` only while it has
* something to repair.
*/
import { LAB_DOCUMENT_DATE_REPAIR_QUEUE } from "@/lib/jobs/lab-document-date-repair-queue";
import { defineJob, type JobDefinition } from "./define-job";
export const MAINTENANCE_REPAIR_JOBS: readonly JobDefinition[] = [
// Document lab values filed under their confirmation moment.
defineJob({
queue: LAB_DOCUMENT_DATE_REPAIR_QUEUE,
policy: {
policy: "short",
reason:
"Fixed singleton key; collapse queued duplicates only, so a boot during a run can queue the next one.",
},
work: { localConcurrency: 1 },
load: async () =>
(await import("@/lib/jobs/lab-document-date-repair"))
.handleLabDocumentDateRepair,
}),
];
+2
View File
@@ -202,6 +202,7 @@ import {
} from "@/lib/medications/intake-slot-dedup";
import { MAINTENANCE_DISCOVERY } from "./maintenance-discovery";
import { MAINTENANCE_REPAIR_JOBS } from "./maintenance-repairs";
import {
defineJob,
type JobDefinition,
@@ -524,6 +525,7 @@ const jobs: readonly JobDefinition[] = [
(await import("@/lib/jobs/measurement-fold-repair"))
.handleMeasurementFoldRepair,
}),
...MAINTENANCE_REPAIR_JOBS,
// The nightly check vacuums when dead rows pile up; the same singleton key
// as the operator's run, so the two collapse.
defineJob({
+1
View File
@@ -120,6 +120,7 @@ const subsystemSurface = {
"health-connect-import": { audience: "account" },
"compaction-tombstone-purge": { audience: "system" },
"measurement-fold-repair": { audience: "system" },
"lab-document-date-repair": { audience: "system" },
"measurement-maintenance": { audience: "system" },
"pulse-day-mean": { audience: "system" },
"medication-intake-import": { audience: "system" },
@@ -0,0 +1,82 @@
/**
* One blood-pressure reading is up to three rows: systolic, diastolic and
* the pulse the cuff measured with them. No column links them. What makes
* them one reading is how every writer stores it: the entry form, the MCP
* write, a Withings measure group and a HealthKit correlation all write the
* halves with the same source at the same instant, and the FHIR read pairs a
* diastolic onto its systolic by exactly that instant.
*
* So the rule here is the strict one: same owner, same source, identical
* `measuredAt`. The series and in-target reads pair more loosely (nearest
* diastolic within five minutes) because they only draw a point; a delete
* takes rows away, so it takes only the rows that can be nothing else.
*
* Deleting a systolic or a diastolic takes the whole reading with it.
* Deleting a pulse takes only the pulse: a pulse is also a value of its own,
* and removing one is no reason to remove a blood pressure.
*/
import type { MeasurementType, Prisma } from "@/generated/prisma/client";
/** The types that anchor a reading: deleting one deletes the reading. */
const BP_HALVES: ReadonlySet<string> = new Set([
"BLOOD_PRESSURE_SYS",
"BLOOD_PRESSURE_DIA",
]);
/** Every type a reading's rows can carry. */
const BP_READING_TYPES: MeasurementType[] = [
"BLOOD_PRESSURE_SYS",
"BLOOD_PRESSURE_DIA",
"PULSE",
];
export interface ReadingAnchor {
id: string;
type: MeasurementType | string;
source: string;
measuredAt: Date;
deletedAt?: Date | null;
}
type MeasurementReader = {
measurement: Pick<Prisma.TransactionClient["measurement"], "findMany">;
};
export interface ReadingSibling {
id: string;
type: MeasurementType;
measuredAt: Date;
}
/**
* The other rows of the readings `rows` belong to, not counting `rows`
* themselves. `state: "live"` finds the rows a delete takes with it;
* `state: "deletedWith"` finds the rows that were tombstoned in the same
* statement as their anchor (the identical `deletedAt`), which is what a
* restore brings back with it, and never a row deleted on its own.
*/
export async function bloodPressureSiblings(
db: MeasurementReader,
userId: string,
rows: readonly ReadingAnchor[],
state: "live" | "deletedWith",
): Promise<ReadingSibling[]> {
const anchors = rows.filter(
(r) => BP_HALVES.has(r.type) && (state === "live" || r.deletedAt),
);
if (anchors.length === 0) return [];
const found = await db.measurement.findMany({
where: {
userId,
id: { notIn: rows.map((r) => r.id) },
type: { in: BP_READING_TYPES },
OR: anchors.map((a) => ({
source: a.source as Prisma.MeasurementWhereInput["source"],
measuredAt: a.measuredAt,
deletedAt: state === "live" ? null : a.deletedAt,
})),
},
select: { id: true, type: true, measuredAt: true },
});
return found;
}
+3 -3
View File
@@ -1055,7 +1055,7 @@ export const measurementPaths: NonNullable<ZodOpenApiObject["paths"]> = {
tags: ["Measurements"],
summary: "Delete a measurement",
description:
"Soft-deletes the row (tombstone) so paired clients can reconcile the deletion.",
"Soft-deletes the row (tombstone) so paired clients can reconcile the deletion. A blood-pressure systolic or diastolic takes the rest of its reading with it: the other half and the pulse with the same source and the identical `measuredAt`, tombstoned in the same transaction with the same `deletedAt`, each with its own `syncVersion` bump. A pulse is deleted on its own.",
requestParams: { path: z.object({ id: z.string() }) },
responses: {
...recordRefusal(),
@@ -1145,7 +1145,7 @@ export const measurementPaths: NonNullable<ZodOpenApiObject["paths"]> = {
tags: ["Measurements"],
summary: "Bulk soft-delete measurements",
description:
"Soft-deletes (tombstones) up to 200 of the caller's measurement rows in one call, mirroring the single-DELETE contract. Idempotent via the `Idempotency-Key` header; rate-limited `measurements:bulk-delete:<userId>`. Forged / foreign / already-deleted ids are silently skipped — `deleted` is the count of rows actually tombstoned. The deletions surface in `/api/sync/changes` as tombstones.",
"Soft-deletes (tombstones) up to 200 of the caller's measurement rows in one call, mirroring the single-DELETE contract. Idempotent via the `Idempotency-Key` header; rate-limited `measurements:bulk-delete:<userId>`. Forged / foreign / already-deleted ids are silently skipped — `deleted` is the count of rows actually tombstoned, which includes the rest of any blood-pressure reading a selected systolic or diastolic belongs to (as on the single DELETE). The deletions surface in `/api/sync/changes` as tombstones.",
requestBody: {
required: true,
content: {
@@ -1176,7 +1176,7 @@ export const measurementPaths: NonNullable<ZodOpenApiObject["paths"]> = {
tags: ["Measurements"],
summary: "Restore soft-deleted measurements",
description:
"Un-tombstones up to 200 of the caller's soft-deleted measurement rows in one call — the delete-Undo counterpart to `/api/measurements/bulk-delete`. Idempotent via the `Idempotency-Key` header; rate-limited `measurements:restore:<userId>`. Forged / foreign / not-deleted ids are silently skipped — `restored` is the count of rows actually un-tombstoned. The restored rows re-surface in `/api/sync/changes` as upserts.",
"Un-tombstones up to 200 of the caller's soft-deleted measurement rows in one call — the delete-Undo counterpart to `/api/measurements/bulk-delete`. Idempotent via the `Idempotency-Key` header; rate-limited `measurements:restore:<userId>`. Forged / foreign / not-deleted ids are silently skipped — `restored` is the count of rows actually un-tombstoned. A blood-pressure systolic or diastolic also restores the rows of its reading that were deleted with it (same source, identical `measuredAt` and `deletedAt`), so naming one half undoes the whole reading; a row deleted at another moment stays deleted. The restored rows re-surface in `/api/sync/changes` as upserts.",
requestBody: {
required: true,
content: {
@@ -0,0 +1,198 @@
/**
* Deleting one row of a blood-pressure reading deletes the reading, through
* the real routes against a real Postgres.
*
* Systolic, diastolic and the cuff's pulse are three rows with no link
* column; they are one reading because every writer stores them with the
* same source at the same instant. Deleting the systolic used to tombstone
* only the systolic and leave a diastolic and a pulse behind. Now the
* single DELETE and the bulk delete take the whole reading, every row bumps
* its own syncVersion (so the sync feed carries each tombstone), and the
* undo, which names only the row the person tapped, brings the whole
* reading back. A reading of another source at the same instant, a reading
* a minute later, and a pulse deleted on its own are left alone.
*/
import { NextRequest } from "next/server";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { cookieJar, headerJar } from "./mock-next-headers";
import { getPrismaClient, truncateAllTables } from "./setup";
vi.mock("next/headers", async () => {
const { cookieJar, headerJar } = await import("./mock-next-headers");
return {
headers: vi.fn(async () => ({
get: (name: string) => headerJar.get(name.toLowerCase()) ?? null,
})),
cookies: vi.fn(async () => ({
get: (name: string) => {
const value = cookieJar.get(name);
return value ? { name, value } : undefined;
},
set: (name: string, value: string) => {
cookieJar.set(name, value);
},
delete: (name: string) => {
cookieJar.delete(name);
},
})),
};
});
vi.mock("@/lib/db-compat", () => ({
ensureDbCompatibility: vi.fn().mockResolvedValue(undefined),
}));
const USER = "user-bp-reading-delete";
const AT = new Date("2026-10-20T07:30:00.000Z");
const MINUTE_LATER = new Date(AT.getTime() + 60_000);
beforeEach(async () => {
const prisma = getPrismaClient();
await truncateAllTables(prisma);
cookieJar.clear();
headerJar.clear();
await prisma.user.create({
data: {
id: USER,
username: "bp-reading-delete",
email: "bp-reading-delete@example.test",
},
});
const session = await prisma.session.create({
data: { userId: USER, expiresAt: new Date(Date.now() + 3_600_000) },
});
cookieJar.set("healthlog_session", session.id);
const row = (
id: string,
type: "BLOOD_PRESSURE_SYS" | "BLOOD_PRESSURE_DIA" | "PULSE",
value: number,
source: "MANUAL" | "WITHINGS",
measuredAt: Date,
) => ({
id,
userId: USER,
type,
value,
unit: type === "PULSE" ? "bpm" : "mmHg",
source,
measuredAt,
});
await prisma.measurement.createMany({
data: [
// The reading under test.
row("sys", "BLOOD_PRESSURE_SYS", 128, "MANUAL", AT),
row("dia", "BLOOD_PRESSURE_DIA", 82, "MANUAL", AT),
row("pulse", "PULSE", 64, "MANUAL", AT),
// The same instant from another source: another reading.
row("w-sys", "BLOOD_PRESSURE_SYS", 131, "WITHINGS", AT),
row("w-dia", "BLOOD_PRESSURE_DIA", 84, "WITHINGS", AT),
// A minute later from the same source: another reading.
row("next-sys", "BLOOD_PRESSURE_SYS", 125, "MANUAL", MINUTE_LATER),
row("next-dia", "BLOOD_PRESSURE_DIA", 80, "MANUAL", MINUTE_LATER),
row("next-pulse", "PULSE", 62, "MANUAL", MINUTE_LATER),
],
});
});
async function rows(): Promise<
Record<string, { deleted: boolean; syncVersion: number }>
> {
const all = await getPrismaClient().measurement.findMany({
where: { userId: USER },
select: { id: true, deletedAt: true, syncVersion: true },
});
return Object.fromEntries(
all.map((r) => [
r.id,
{ deleted: r.deletedAt !== null, syncVersion: r.syncVersion },
]),
);
}
const deletedIds = async () =>
Object.entries(await rows())
.filter(([, r]) => r.deleted)
.map(([id]) => id)
.sort();
async function deleteOne(id: string) {
const { DELETE } = await import("@/app/api/measurements/[id]/route");
const res = await DELETE(
new NextRequest(`http://localhost/api/measurements/${id}`, {
method: "DELETE",
}),
{ params: Promise.resolve({ id }) },
);
expect(res.status).toBe(200);
}
async function post(path: "bulk-delete" | "restore", ids: string[]) {
const route =
path === "bulk-delete"
? await import("@/app/api/measurements/bulk-delete/route")
: await import("@/app/api/measurements/restore/route");
const res = await route.POST(
new NextRequest(`http://localhost/api/measurements/${path}`, {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ ids }),
}),
);
expect(res.status).toBe(200);
return (await res.json()).data as { deleted?: number; restored?: number };
}
describe("deleting a blood-pressure reading", () => {
it("takes the other half and the pulse with the systolic, and the undo brings all three back", async () => {
await deleteOne("sys");
expect(await deletedIds()).toEqual(["dia", "pulse", "sys"]);
const after = await rows();
for (const id of ["sys", "dia", "pulse"]) {
expect(after[id]!.syncVersion).toBe(2);
}
for (const id of ["w-sys", "w-dia", "next-sys", "next-dia"]) {
expect(after[id]!.syncVersion).toBe(1);
}
// The undo names only the row that was tapped.
const { restored } = await post("restore", ["sys"]);
expect(restored).toBe(3);
expect(await deletedIds()).toEqual([]);
});
it("takes the systolic and the pulse with the diastolic", async () => {
await deleteOne("dia");
expect(await deletedIds()).toEqual(["dia", "pulse", "sys"]);
});
it("deletes a pulse on its own, and its undo leaves the reading as it was", async () => {
await deleteOne("pulse");
expect(await deletedIds()).toEqual(["pulse"]);
await post("restore", ["pulse"]);
expect(await deletedIds()).toEqual([]);
});
it("does the same for the bulk delete, and restores only what went in the same delete", async () => {
// The pulse of the second reading was deleted on its own earlier.
await deleteOne("next-pulse");
const { deleted } = await post("bulk-delete", ["sys", "next-dia"]);
expect(deleted).toBe(5);
expect(await deletedIds()).toEqual([
"dia",
"next-dia",
"next-pulse",
"next-sys",
"pulse",
"sys",
]);
const { restored } = await post("restore", ["sys", "next-dia"]);
expect(restored).toBe(5);
// The pulse deleted at another moment stays deleted.
expect(await deletedIds()).toEqual(["next-pulse"]);
});
});
@@ -0,0 +1,222 @@
/**
* A lab value without its own date, confirmed from a stored document, against
* a real Postgres and through the real confirm route.
*
* A lab report usually prints the collection date once, in its header. The
* extraction puts that date on the document (`reportDate`) and leaves each
* value's own date empty, so the confirm step has to date the value by its
* document. It used to file every such reading under the moment it was
* confirmed, so a report from years ago showed up as today's results.
*
* The route reads the document's two dates once and hands them to the commit
* step: the report date first, else the filing date (`documentDate`), the order
* the day view places a document by. A value's own date still wins, but only
* when it names a real calendar day.
*/
import { NextRequest } from "next/server";
import { beforeEach, describe, expect, it, vi } from "vitest";
import { cookieJar } from "./mock-next-headers";
import { getPrismaClient, truncateAllTables } from "./setup";
const USER_ID = "user-undated-lab-value";
vi.mock("next/headers", async () => {
const { cookieJar, headerJar } = await import("./mock-next-headers");
return {
headers: vi.fn(async () => ({
get: (name: string) => headerJar.get(name.toLowerCase()) ?? null,
})),
cookies: vi.fn(async () => ({
get: (name: string) => {
const value = cookieJar.get(name);
return value ? { name, value } : undefined;
},
set: (name: string, value: string) => {
cookieJar.set(name, value);
},
delete: (name: string) => {
cookieJar.delete(name);
},
})),
};
});
vi.mock("@/lib/db-compat", () => ({
ensureDbCompatibility: vi.fn().mockResolvedValue(undefined),
}));
beforeEach(async () => {
await truncateAllTables(getPrismaClient());
cookieJar.clear();
const prisma = getPrismaClient();
await prisma.user.create({
data: {
id: USER_ID,
username: "undated-lab-value",
email: "undated-lab-value@example.test",
modulePreferencesJson: { inboundDocuments: true, labs: true },
},
});
const session = await prisma.session.create({
data: { userId: USER_ID, expiresAt: new Date(Date.now() + 3_600_000) },
});
cookieJar.set("healthlog_session", session.id);
for (const [name, unit] of [
["Ferritin", "ng/mL"],
["Glucose", "mmol/L"],
["Haemoglobin", "g/L"],
]) {
await prisma.biomarker.create({ data: { userId: USER_ID, name, unit } });
}
});
const noon = (day: string) => new Date(`${day}T12:00:00.000Z`);
async function storedDocument(dates: {
reportDate: string | null;
documentDate: string | null;
}) {
return getPrismaClient().inboundDocument.create({
data: {
userId: USER_ID,
filename: "befund.pdf",
mimeType: "application/pdf",
byteSize: 1024,
contentEncrypted: Buffer.from("stand-in for the stored bytes"),
status: "EXTRACTED",
reportDate: dates.reportDate ? noon(dates.reportDate) : null,
documentDate: dates.documentDate ? noon(dates.documentDate) : null,
},
});
}
async function stagedValue(
documentId: string,
label: string,
value: number,
unit: string,
effectiveDate: string | null,
) {
const { encryptFactData, encryptFactProvenance } =
await import("@/lib/documents/store");
return getPrismaClient().extractedFact.create({
data: {
documentId,
userId: USER_ID,
factType: "OBSERVATION",
status: "PENDING",
confidence: 0.95,
needsReview: false,
dataEncrypted: encryptFactData({
label,
code: null,
codeSystem: null,
value,
valueText: null,
unit,
referenceLow: null,
referenceHigh: null,
referenceText: null,
effectiveDate,
}),
provenanceEncrypted: encryptFactProvenance({
sourceText: `${label} ${value} ${unit}`,
anchored: true,
sourceOffset: 0,
page: 0,
confidence: 0.95,
}),
},
});
}
async function confirm(documentId: string, factIds: string[]) {
const { POST } =
await import("@/app/api/documents/inbound/[id]/confirm/route");
const res = await POST(
new NextRequest(
`http://localhost/api/documents/inbound/${documentId}/confirm`,
{
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({
decisions: factIds.map((factId) => ({ factId, action: "approve" })),
}),
},
),
{ params: Promise.resolve({ id: documentId }) },
);
expect(res.status).toBe(200);
const body = await res.json();
expect(body.data.failed).toEqual([]);
expect(body.data.approved).toHaveLength(factIds.length);
return body.data.approved as { factId: string; recordId: string }[];
}
async function takenAtOf(recordId: string) {
const row = await getPrismaClient().labResult.findUniqueOrThrow({
where: { id: recordId },
});
return row.takenAt.toISOString();
}
describe("an undated lab value is dated by its document", () => {
it("takes the report date over the filing date, keeps a value's own date, and ignores an impossible one", async () => {
const today = new Date().toISOString().slice(0, 10);
// Collection date printed once in the header; filed today (the upload-day
// default for documentDate).
const doc = await storedDocument({
reportDate: "2021-03-01",
documentDate: today,
});
const undated = await stagedValue(doc.id, "Ferritin", 80, "ng/mL", null);
const ownDate = await stagedValue(
doc.id,
"Glucose",
5.1,
"mmol/L",
"2021-02-20",
);
const impossible = await stagedValue(
doc.id,
"Haemoglobin",
142,
"g/L",
"2021-02-30",
);
const approved = await confirm(doc.id, [
undated.id,
ownDate.id,
impossible.id,
]);
const recordOf = (factId: string) =>
approved.find((a) => a.factId === factId)!.recordId;
expect(await takenAtOf(recordOf(undated.id))).toBe(
"2021-03-01T12:00:00.000Z",
);
expect(await takenAtOf(recordOf(ownDate.id))).toBe(
"2021-02-20T12:00:00.000Z",
);
// Not 2 March: a day the month does not have is no date at all.
expect(await takenAtOf(recordOf(impossible.id))).toBe(
"2021-03-01T12:00:00.000Z",
);
});
it("takes the filing date when the document has no report date", async () => {
const doc = await storedDocument({
reportDate: null,
documentDate: "2021-02-15",
});
const undated = await stagedValue(doc.id, "Ferritin", 80, "ng/mL", null);
const [approved] = await confirm(doc.id, [undated.id]);
expect(await takenAtOf(approved!.recordId)).toBe(
"2021-02-15T12:00:00.000Z",
);
});
});
@@ -0,0 +1,232 @@
/**
* The one-shot repair of lab values a document confirm filed under the
* moment of confirmation (real Postgres).
*
* Before the confirm step dated an undated value by its document, such a
* value got `takenAt = now`, the same instant as its `createdAt`. The repair
* moves exactly those to the document's report date (else its filing date)
* at noon UTC, and leaves alone a value with a date of its own, a value that
* really was taken on its confirmation day in the owner's zone, one the
* person re-dated since, and one whose document has no date. A second run
* changes nothing.
*/
import { beforeEach, describe, expect, it } from "vitest";
import { encryptFactData, encryptFactProvenance } from "@/lib/documents/store";
import { repairUndatedDocumentLabValues } from "@/lib/jobs/lab-document-date-repair";
import { getPrismaClient, truncateAllTables } from "./setup";
const USER = "user-lab-document-date-repair";
const noon = (day: string) => new Date(`${day}T12:00:00.000Z`);
beforeEach(async () => {
const prisma = getPrismaClient();
await truncateAllTables(prisma);
await prisma.user.create({
data: {
id: USER,
username: "lab-date-repair",
email: "lab-date-repair@example.test",
timezone: "Europe/Berlin",
},
});
});
/**
* A document, one confirmed value from it and the fact that links the two,
* as the confirm route left them.
*/
async function confirmedValue(opts: {
id: string;
reportDate: string | null;
documentDate: string | null;
effectiveDate: string | null;
createdAt: Date;
takenAt: Date;
}) {
const prisma = getPrismaClient();
const doc = await prisma.inboundDocument.create({
data: {
userId: USER,
filename: `${opts.id}.pdf`,
mimeType: "application/pdf",
byteSize: 1024,
contentEncrypted: Buffer.from("stand-in for the stored bytes"),
status: "CONFIRMED",
reportDate: opts.reportDate ? noon(opts.reportDate) : null,
documentDate: opts.documentDate ? noon(opts.documentDate) : null,
},
});
await prisma.labResult.create({
data: {
id: opts.id,
userId: USER,
analyte: "Ferritin",
value: 80,
unit: "ng/mL",
source: "DOCUMENT",
takenAt: opts.takenAt,
createdAt: opts.createdAt,
},
});
await prisma.extractedFact.create({
data: {
documentId: doc.id,
userId: USER,
factType: "OBSERVATION",
status: "APPROVED",
confidence: 0.95,
needsReview: false,
dataEncrypted: encryptFactData({
label: "Ferritin",
code: null,
codeSystem: null,
value: 80,
valueText: null,
unit: "ng/mL",
referenceLow: null,
referenceHigh: null,
referenceText: null,
effectiveDate: opts.effectiveDate,
}),
provenanceEncrypted: encryptFactProvenance({
sourceText: "Ferritin 80 ng/mL",
anchored: true,
sourceOffset: 0,
page: 0,
confidence: 0.95,
}),
committedRecordId: opts.id,
committedRecordType: "labResult",
},
});
}
async function takenAt(): Promise<Record<string, string>> {
const rows = await getPrismaClient().labResult.findMany({
where: { userId: USER },
select: { id: true, takenAt: true },
});
return Object.fromEntries(rows.map((r) => [r.id, r.takenAt.toISOString()]));
}
describe("lab document date repair", () => {
it("moves only values filed under their confirmation moment, and converges", async () => {
const confirmed = new Date("2026-10-20T08:00:00.123Z");
// Collection date in the header, value undated, filed at confirmation.
await confirmedValue({
id: "misdated",
reportDate: "2021-03-01",
documentDate: "2026-10-18",
effectiveDate: null,
createdAt: confirmed,
takenAt: confirmed,
});
// No report date: the filing date is the document's day.
await confirmedValue({
id: "misdated-filing-date",
reportDate: null,
documentDate: "2021-02-15",
effectiveDate: null,
createdAt: confirmed,
takenAt: new Date(confirmed.getTime() + 2),
});
// Its own date, even where the row looks like a confirmation moment.
await confirmedValue({
id: "own-date",
reportDate: "2021-03-01",
documentDate: null,
effectiveDate: "2021-02-20",
createdAt: confirmed,
takenAt: confirmed,
});
// Confirmed at 00:30 in Berlin on the report day itself: 22:30 UTC the
// evening before, which a UTC comparison would have moved.
const justAfterMidnight = new Date("2026-10-19T22:30:00.000Z");
await confirmedValue({
id: "taken-on-confirm-day",
reportDate: "2026-10-20",
documentDate: "2026-10-20",
effectiveDate: null,
createdAt: justAfterMidnight,
takenAt: justAfterMidnight,
});
// Re-dated by the person after the confirm.
await confirmedValue({
id: "re-dated",
reportDate: "2021-03-01",
documentDate: null,
effectiveDate: null,
createdAt: confirmed,
takenAt: new Date("2026-10-01T07:45:00.000Z"),
});
// A document with no date at all leaves nothing to move to.
await confirmedValue({
id: "undated-document",
reportDate: null,
documentDate: null,
effectiveDate: null,
createdAt: confirmed,
takenAt: confirmed,
});
const before = await takenAt();
const first = await repairUndatedDocumentLabValues(getPrismaClient());
expect(first.updated).toBe(2);
expect(await takenAt()).toEqual({
...before,
misdated: "2021-03-01T12:00:00.000Z",
"misdated-filing-date": "2021-02-15T12:00:00.000Z",
});
const afterFirst = await takenAt();
const second = await repairUndatedDocumentLabValues(getPrismaClient());
expect(second.updated).toBe(0);
expect(await takenAt()).toEqual(afterFirst);
});
it("leaves a lab value alone when a second fact also names it", async () => {
const confirmed = new Date("2026-10-20T08:00:00.000Z");
await confirmedValue({
id: "linked-twice",
reportDate: "2021-03-01",
documentDate: null,
effectiveDate: null,
createdAt: confirmed,
takenAt: confirmed,
});
const prisma = getPrismaClient();
const second = await prisma.inboundDocument.create({
data: {
userId: USER,
filename: "other.pdf",
mimeType: "application/pdf",
byteSize: 1,
contentEncrypted: Buffer.from("x"),
reportDate: noon("2020-01-01"),
},
});
const fact = await prisma.extractedFact.findFirstOrThrow({
where: { committedRecordId: "linked-twice" },
});
await prisma.extractedFact.create({
data: {
documentId: second.id,
userId: USER,
factType: "OBSERVATION",
status: "APPROVED",
dataEncrypted: fact.dataEncrypted,
provenanceEncrypted: fact.provenanceEncrypted,
committedRecordId: "linked-twice",
committedRecordType: "labResult",
},
});
const result = await repairUndatedDocumentLabValues(prisma);
expect(result.updated).toBe(0);
expect((await takenAt())["linked-twice"]).toBe(confirmed.toISOString());
});
});