From 4122c79d66744d1cae579c1cfb883215ad389694 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc-Andr=C3=A9=20Bombeck?= Date: Thu, 21 May 2026 09:03:33 +0200 Subject: [PATCH] =?UTF-8?q?chore(release):=20v1.4.39.1=20=E2=80=94=20rollu?= =?UTF-8?q?p=20tier=20catches=20Withings=20+=20import=20+=20admin-restore?= =?UTF-8?q?=20write=20paths?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Hotfix on top of v1.4.39. Three measurement write paths (Withings sync, `/api/import`, admin backup restore) were bypassing the persistent rollup tier's write hook, so the dashboard chart's `source=rollup` fast-path painted "Noch nicht genug Daten" on the 30 / 90-point range for accounts whose recent data only came through those paths. The 7-day window kept rendering because the chart's raw fetch never consults the rollup tier. Closes the read-coverage gap by wiring `recomputeBucketsForMeasurement` into every remaining ingest path and re-anchors the boot-time backfill discovery to per-day coverage so already-deployed instances re-converge on the next worker boot. The legacy `sum_value IS NULL` branch stays on the same UNION so the v1.4.39 column-backfill continues working. Also re-anchors one date-bound test in the medications-intake route suite so the green CI run no longer depends on the wall clock. Tests 4647 → 4648 passing; one long-standing skip unchanged. --- .planning/round-audit-marathon-closure.md | 5 + .planning/round-v1439-perf-verify.md | 68 ++++++++++ CHANGELOG.md | 47 +++++++ package.json | 2 +- .../api/admin/backups/[id]/restore/route.ts | 32 +++++ src/app/api/import/__tests__/route.test.ts | 119 ++++++++++++++++++ src/app/api/import/route.ts | 43 ++++++- .../intake/__tests__/route.test.ts | 14 ++- .../measurements/__tests__/rollups.test.ts | 28 ++++- src/lib/measurements/rollups.ts | 39 +++--- .../withings/__tests__/sync-activity.test.ts | 78 ++++++++++++ src/lib/withings/__tests__/sync-sleep.test.ts | 59 +++++++++ src/lib/withings/sync-activity.ts | 24 ++++ src/lib/withings/sync-sleep.ts | 23 ++++ src/lib/withings/sync.ts | 33 +++++ 15 files changed, 594 insertions(+), 20 deletions(-) create mode 100644 .planning/round-v1439-perf-verify.md diff --git a/.planning/round-audit-marathon-closure.md b/.planning/round-audit-marathon-closure.md index 9c90fef6a..7515266e3 100644 --- a/.planning/round-audit-marathon-closure.md +++ b/.planning/round-audit-marathon-closure.md @@ -16,6 +16,11 @@ Shipped: 2026-05-17 → 2026-05-18 (thirteen consecutive releases — capped by | v1.4.38.2 | Mood-reminder hotfix bundle + Settings toggle — six-axis QA on PR #190 surfaced enough real bugs that the feature could not have been used safely (locale resolver demoted 4 of 6 locales to English; ledger committed BEFORE delivery, so a transient APNs blip silently nuked the day's nudge AND blocked retries; DST arithmetic drifted the local-22:00 baseline by an hour twice a year; APNs payload forwarded Telegram `replyMarkup` to Apple + lockscreen; **no Settings UI to opt in** = dead opt-in). v1.4.38.2 closes all of the above plus adds `MoodReminderCard` Switch, `dispatchNotification` outcome bubble, `localHmAsUtc` DST-safe helper, APNs metadata allowlist, per-user `try` wrapper, 90-day retention sweep, dead-code drop, CHANGELOG `EVENT_DEFAULT_ENABLED` identifier-leak scrub. Unit suite 4565 → 4551 (mood-reminder.test.ts rewrite for new contract, not coverage loss). | [v1.4.38.2](https://github.com/MBombeck/HealthLog/releases/tag/v1.4.38.2) | | v1.4.38.3 | CI green-up + chunk-load auto-recover — three pre-existing main reds closed (TODO marker on correlations `degraded` sentinel, rollup integration test racing the v1.4.37.1 fire-and-forget rollup populator, e2e drift after the v1.4.37 doctor-report hero refactor + the dashboard touch-target gate + `measurement-flow` mock missing `unit`/`source`). Plus new `AppError` branch detects the chunk-load error family and auto-reloads the SPA once per session — closes the stale-shell paper-cut after every deploy. Squash skipped: six atomic Marc-Voice commits landed directly on main, then back-merged to develop. | [v1.4.38.3](https://github.com/MBombeck/HealthLog/releases/tag/v1.4.38.3) | | v1.4.38.4 | Pro-active stale-shell self-heal — Marc reported `healthlog.bombeck.io` "nicht erreichbar" on the v1.4.38.3 PWA; live curl was 100 % healthy (`/api/version` + `/api/health` both 200), so the symptom traced to a cached pre-v1.4.38.3 shell that never tripped the new `AppError` reload path. v1.4.38.4 layers three fixes so future deploys self-heal without user action: `` client component mounted in `` polls `/api/version` every 60 s and on mismatch unregisters every SW, wipes CacheStorage, and `window.location.reload()`s (sessionStorage-gated to once per session); `next.config.ts` injects `NEXT_PUBLIC_APP_VERSION` from `package.json` at build time so the running shell knows which release it was built against; service worker `CACHE_VERSION` now keys to the release tag so the precached chunk graph is evicted on the next SW install. Tag re-pushed once after the editor-race dropped the `package.json` bump from the first release commit; the docker-publish workflow was cancelled mid-build and re-run from the correct commit. | [v1.4.38.4](https://github.com/MBombeck/HealthLog/releases/tag/v1.4.38.4) | +| v1.4.38.5 | Analytics fast-path restored on long-running accounts — boot-time backfill discovery widened to find every user that lacks DAY-bucket coverage for a measurement type they actually have, not only the empty-rollup case. The post-v1.4.38 backfill had only discovered users with zero rollup rows total, leaving every long-running account stranded on the live-SQL path. New discovery query is per-(user, type) so any single uncovered type triggers a full fold. | [v1.4.38.5](https://github.com/MBombeck/HealthLog/releases/tag/v1.4.38.5) | +| v1.4.38.6 | Boot-backfill discovery SQL fix — the widened v1.4.38.5 LEFT JOIN keyed on `bucket_start IS NULL` as the "no row" sentinel because `measurement_rollups` has no `id` column (composite PK). Without this fix the LEFT JOIN matched on the wrong key and the boot fold ran zero work. Single-line correctness fix; perf-verify continued into v1.4.38.7. | [v1.4.38.6](https://github.com/MBombeck/HealthLog/releases/tag/v1.4.38.6) | +| v1.4.38.7 | Rollup recompute observability + admin trigger — `POST /api/admin/rollups/recompute` lets the operator synchronously fold one user or re-trigger boot-discovery without restarting the container. Adds `fallback_reason` + `missing_types` annotates on the slim-summaries live-fallback path so a slow `/api/analytics` cold mount maps to the responsible type in the wide-event. Boot-discovery now logs every result (including the silent "no users to backfill" case). | [v1.4.38.7](https://github.com/MBombeck/HealthLog/releases/tag/v1.4.38.7) | +| v1.4.38.8 | Analytics fast-path gates per-type only — `isFullyCovered(coverage) && specificTypes` was poisoning every fast-path: a single uncovered orphan type (e.g. an iOS-pushed brand-new `ACTIVITY_FLIGHTS`) flipped the AND false and stranded the entire helper on live SQL across the full measurement table. `correlations-fast-path` now gates on `BLOOD_PRESSURE_SYS + PULSE + WEIGHT`; `bp-in-target-fast-path` on `BLOOD_PRESSURE_SYS + BLOOD_PRESSURE_DIA`; `health-score-fast-path` on `WEIGHT`. Cold-mount `/api/analytics` on power-user accounts drops from 30–75 s to ~3–5 s. Coverage-probe semantics unchanged. | [v1.4.38.8](https://github.com/MBombeck/HealthLog/releases/tag/v1.4.38.8) | +| v1.4.39 | Mood + medication-compliance + cumulative-sum rollup tiers — extends the v1.4.38 fast-path posture to the two remaining cold-mount hot spots. New `mood_entry_rollups` (per-(user, granularity, bucket) mood stats) eliminates the unbounded `MoodEntry.findMany` walk on `/api/mood/analytics` (12.7 s → ~200 ms expected). New `medication_compliance_rollups` (per-(user, medication, day) scheduled / taken / skipped ledger anchored to user-tz `YYYY-MM-DD`) eliminates the unbounded intake-event walk on `/api/medications/intake?scope=compliance` (3.2 s → ~200 ms expected). New nullable `measurement_rollups.sum_value` column lets cumulative metrics (`ACTIVITY_STEPS / FLIGHTS_CLIMBED / WALKING_RUNNING_DISTANCE / TIME_IN_DAYLIGHT / ACTIVE_ENERGY`) skip per-type JS aggregation in the dashboard sparkline + `groupBy=day` paths. Defense-in-depth `since` cap on the `/api/analytics` live-fallback path (~347 k rows → ~5 k rows; 425 d window preserves `summary.avg30LastYear`). New WEEK / MONTH / YEAR reader helpers ready for the v1.5 multi-year trend card. Five wave agents in parallel + six QA reviewers + nine reconcile fixes (race-safe atomic upsert, partial-coverage probe, fire-and-forget worker enqueue, multi-entry-day summarize parity, DST-boundary test pin, user-scoped backfill enqueue, …). Tests 4 524 → 4 640 (+116). Squash to main clean. | [v1.4.39](https://github.com/MBombeck/HealthLog/releases/tag/v1.4.39) | ## Audit waves diff --git a/.planning/round-v1439-perf-verify.md b/.planning/round-v1439-perf-verify.md new file mode 100644 index 000000000..d03cde1ab --- /dev/null +++ b/.planning/round-v1439-perf-verify.md @@ -0,0 +1,68 @@ +# v1.4.39 — Post-deploy perf verification + +Live deploy: 2026-05-21T05:46:56Z — Coolify deploy uuid +`ujgsxpa46esyzybjldgxjeox`, commit hash +`d84922cdc3bfd118b10b74e6b8664a784837fe58`. + +Migrations applied at boot: + +``` +Applying migration `0070_v1439_mood_rollups` +Applying migration `0071_v1439_medication_compliance_rollups` +Applying migration `0072_v1439_rollup_sum_value` +All migrations have been successfully applied. +``` + +## Unauthenticated probes (from this orchestrator) + +| Endpoint | n | Median | Max | +| --- | --- | --- | --- | +| `GET /api/version` | 10 | **86 ms** | 104 ms | +| `GET /api/health` (Uptime-Kuma) | per-log | 18–78 ms | — | + +The 60 s response cache wrap is not yet warm for the authenticated +heavy endpoints — perf-verify on `/api/mood/analytics`, +`/api/medications/intake?scope=compliance`, +`/api/dashboard/summary`, `/api/analytics` waits for Marc's next +browser session against the live deploy. The current open-internet +probes confirm the public-surface contract is fast. + +## Coolify production log excerpt (boot) + +- App container boots in ~1.5 s after migrations. +- Reminder worker registered. +- `host_metric_sample` background job running every 30 s without + pressure. +- No 4xx / 5xx in the first ~2 min of logs. + +## What gets verified by Marc's first authenticated session + +For each of the four read-path swaps, the wide-event annotates +`meta..path = "rollup"` on the rollup-fast-path or +`path = "live"` on the fall-through. Marc loading the dashboard / +analytics page populates the 60 s LRU; a second mount within 60 s +hits warm cache (< 50 ms). The audit-predicted cold targets: + +| Endpoint | Pre-v1.4.39 cold | Expected v1.4.39 cold | +| --- | --- | --- | +| `/api/mood/analytics` | 12.7 s (live) | ~200 ms (rollup) | +| `/api/medications/intake?scope=compliance` | 3.2 s (live) | ~200 ms (rollup) | +| `/api/dashboard/summary` cumulative sparkline | ~500 ms | ~300 ms | +| `/api/analytics` full slice live-fallback row cap | 347 k rows | ~5 k rows (425 d) | + +## Self-heal verification + +`/api/version` returns `1.4.39` end-to-end (curl + 10× burst). The +`VersionPoller` from v1.4.38.4 will pick up the new version-string at +the next 60 s tick on Marc's browser shell; the cached pre-v1.4.39 +shell evicts CacheStorage + unregisters every SW + reloads +exactly once per session. + +## Next perf-verify checkpoint + +After Marc's first authenticated session, capture the wide-event +`path` annotates on each of the four endpoints from Coolify logs +and update this file with the live cold-mount numbers. v1.4.40 +backlog already covers the carry-over consumers +(`health-score-fast-path` swap, `/api/analytics` A2 cumulative skip, +slope90 monthly bucket). diff --git a/CHANGELOG.md b/CHANGELOG.md index 09ab1f416..ba6ed8672 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,52 @@ # Changelog +## [1.4.39.1] — 2026-05-21 — Rollup tier catches Withings + import + admin-restore write paths + +The v1.4.39 dashboard chart "Noch nicht genug Daten" empty-state on +the 30-point range traced to three measurement write paths bypassing +the rollup table's per-write hook: Withings sync (BP / weight / pulse / +body-fat / activity / sleep), the JSON / CSV `/api/import` endpoint, +and the admin backup restore. Once the chart's `source=rollup` fast- +path landed in v1.4.36, those days never reached the chart at 30 / 90 +days — the live raw fetch at 7 days kept showing data, the rollup +fast-path at 30 / 90 days under-counted distinct days, and the chart +tripped its `< 3 daily points` empty-state. v1.4.39.1 wires the rollup +write hook into every remaining ingest path and tightens the boot-time +backfill discovery so any stranded accounts re-converge on the next +worker boot. + +### Fixed + +- **Withings sync** (`sync.ts` / `sync-activity.ts` / `sync-sleep.ts`): + every (type, day) pair the sync touched is now handed to + `recomputeBucketsForMeasurement` after the row writes. Collapsed + via `collapseToTypeDayKeys` so a 30-day catch-up costs at most ~N + (type, day) recomputes rather than one per row. Best-effort: a + populator hiccup never fails the user's sync. +- **`/api/import` endpoint**: measurement creates now track touched + `(type, day)` pairs and fold the rollup tier at the end of the + batch. Mirrors the v1.4.39 mood-rollup hook on the same route. +- **Admin backup restore**: the transaction now wipes the user's + `measurement_rollups` partition alongside `mood_entry_rollups`, and + the post-transaction step kicks a full `recomputeUserRollups` so + the restored dataset paints fresh tiles instead of carrying the + previous owner's daily means forward. +- **Boot-time backfill discovery** (`enqueueBootTimeRollupBackfill`): + the missing-coverage join moved from per-type to per-day, so an + account whose Withings sync wrote 27 days of BP without ever firing + the rollup hook now re-surfaces on the next worker boot even when + one prior rollup row already existed for the same type. The legacy + `sum_value IS NULL` branch is preserved on the same UNION. + +### Notes + +- No schema migration. Additive write-hook plumbing only. +- `pnpm test --run` green at 4648 / 4649 (1 long-standing skip). +- The dashboard chart's read-side `source=rollup` short-circuit stays + put — once every write path folds the rollup tier on touch, the + short-circuit is always correct. A defensive coverage-mismatch + fallback is on the v1.4.40 backlog. + ## [1.4.39] — 2026-05-21 — Mood, medication-compliance, and cumulative-sum rollup tiers The v1.4.38 chain settled the measurement-rollup fast-path. v1.4.39 diff --git a/package.json b/package.json index 772f824ff..f6bd0b9b1 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "healthlog", - "version": "1.4.39", + "version": "1.4.39.1", "description": "Self-hosted personal-health-tracking PWA with Withings integration, AI insights, and doctor-report PDF export.", "license": "AGPL-3.0-only", "homepage": "https://healthlog.dev", diff --git a/src/app/api/admin/backups/[id]/restore/route.ts b/src/app/api/admin/backups/[id]/restore/route.ts index e3c4e69f4..7126123b1 100644 --- a/src/app/api/admin/backups/[id]/restore/route.ts +++ b/src/app/api/admin/backups/[id]/restore/route.ts @@ -48,6 +48,7 @@ import { recomputeUserMedicationCompliance, MEDICATION_COMPLIANCE_BACKFILL_DAYS, } from "@/lib/medications/compliance-rollups"; +import { recomputeUserRollups } from "@/lib/measurements/rollups"; export const dynamic = "force-dynamic"; @@ -291,6 +292,14 @@ const handler = apiHandler( // pre-restore daily means. The fold below mints fresh rows // from the restored mood entries. await tx.moodEntryRollup.deleteMany({ where: { userId: ownerId } }); + // v1.4.39.1 — wipe the persisted measurement rollup partition + // for the same reason. Pre-fix the partition kept the previous + // owner's daily means even after the underlying measurements + // were replaced — the chart's `source=rollup` fast-path could + // surface a stale 30-day mean built from rows that no longer + // existed. The fold below mints fresh rows from the restored + // measurement set. + await tx.measurementRollup.deleteMany({ where: { userId: ownerId } }); const channels = await tx.notificationChannel.deleteMany({ where: { userId: ownerId }, }); @@ -420,6 +429,29 @@ const handler = apiHandler( return apiError("Restore failed", 500); } + // v1.4.39.1 — re-fold the persistent measurement rollup tier from + // the just-restored measurements. Pre-fix the restore left the + // rollup table empty for the owner, so the dashboard chart's + // `source=rollup` fast-path silently returned zero buckets until + // the next worker boot ran the backfill discovery — multi-hour + // window of empty charts for the operator-restored account. Runs + // outside the transaction (the 5-year fold would otherwise hold a + // long write lock) and is best-effort so a populator hiccup never + // undoes the restore. The boot-time backfill is the safety net. + if (payload.measurements.length > 0) { + try { + await recomputeUserRollups(ownerId); + } catch (err) { + annotate({ + meta: { + measurement_rollup_restore_failed: true, + measurement_rollup_restore_error: + err instanceof Error ? err.message : String(err), + }, + }); + } + } + // v1.4.39 W-MOOD — re-fold the mood rollup tier from the just- // restored entries. Runs outside the transaction so the (5-year) // fold can't hold a long write lock; best-effort so a populator diff --git a/src/app/api/import/__tests__/route.test.ts b/src/app/api/import/__tests__/route.test.ts index 346ad5933..272a8a77d 100644 --- a/src/app/api/import/__tests__/route.test.ts +++ b/src/app/api/import/__tests__/route.test.ts @@ -27,9 +27,29 @@ vi.mock("@/lib/logging/context", () => ({ annotate: vi.fn(), })); +// v1.4.39.1 — surface the rollup hook so the new regression test can +// assert that the import path now folds the persistent rollup tier +// for each touched (type, day). Mood rollup mock stays no-op because +// the existing v1.4.39 W-MOOD hook already covers the moodlog branch. +vi.mock("@/lib/measurements/rollups", async () => { + const actual = await vi.importActual< + typeof import("@/lib/measurements/rollups") + >("@/lib/measurements/rollups"); + return { + ...actual, + recomputeBucketsForMeasurement: vi.fn().mockResolvedValue(undefined), + }; +}); + +vi.mock("@/lib/mood/rollups", () => ({ + recomputeUserMoodRollups: vi.fn().mockResolvedValue(undefined), +})); + import { NextRequest } from "next/server"; import { POST } from "../route"; import { checkRateLimit } from "@/lib/rate-limit"; +import { prisma } from "@/lib/db"; +import { recomputeBucketsForMeasurement } from "@/lib/measurements/rollups"; beforeEach(() => { vi.resetAllMocks(); @@ -107,3 +127,102 @@ describe("POST /api/import — rate-limit guard", () => { expect(body.error).toBeTruthy(); }); }); + +// v1.4.39.1 — the import path used to write measurements without +// firing the rollup hook, so the dashboard chart's `source=rollup` +// fast-path silently under-counted any imported days until the next +// worker boot ran the backfill discovery. Cover the wiring with a +// regression test so a future refactor can't silently drop it again. +describe("POST /api/import — measurement rollup hook (v1.4.39.1)", () => { + it("folds the persistent rollup table for each (type, day) the import touched", async () => { + vi.mocked(checkRateLimit).mockResolvedValue({ + allowed: true, + remaining: 4, + resetAt: new Date(Date.now() + 1000), + } as never); + vi.mocked(prisma.measurement.create).mockResolvedValue({} as never); + + const response = await POST( + new NextRequest("http://localhost/api/import", { + method: "POST", + body: JSON.stringify({ + measurements: [ + { + type: "BLOOD_PRESSURE_SYS", + value: 124, + unit: "mmHg", + measuredAt: "2026-05-12T07:30:00.000Z", + }, + { + type: "BLOOD_PRESSURE_SYS", + value: 121, + unit: "mmHg", + measuredAt: "2026-05-12T19:30:00.000Z", + }, + { + type: "BLOOD_PRESSURE_SYS", + value: 119, + unit: "mmHg", + measuredAt: "2026-05-14T08:00:00.000Z", + }, + { + type: "WEIGHT", + value: 82.4, + unit: "kg", + measuredAt: "2026-05-14T07:00:00.000Z", + }, + ], + }), + headers: { "content-type": "application/json" }, + }), + ); + + expect(response.status).toBeLessThan(400); + // Two BP rows on 2026-05-12 collapse to one rollup call; one BP + // row on 2026-05-14 is its own call; one WEIGHT row on 2026-05-14 + // is a separate (type, day) pair. Total = 3 distinct rollup folds. + expect(recomputeBucketsForMeasurement).toHaveBeenCalledTimes(3); + const calls = vi + .mocked(recomputeBucketsForMeasurement) + .mock.calls.map((c) => `${c[1]}|${(c[2] as Date).toISOString()}`); + expect(new Set(calls)).toEqual( + new Set([ + "BLOOD_PRESSURE_SYS|2026-05-12T00:00:00.000Z", + "BLOOD_PRESSURE_SYS|2026-05-14T00:00:00.000Z", + "WEIGHT|2026-05-14T00:00:00.000Z", + ]), + ); + }); + + it("does not call the rollup hook when no measurements were written", async () => { + vi.mocked(checkRateLimit).mockResolvedValue({ + allowed: true, + remaining: 4, + resetAt: new Date(Date.now() + 1000), + } as never); + vi.mocked(prisma.measurement.create).mockRejectedValue( + new Error("P2002 duplicate"), + ); + + const response = await POST( + new NextRequest("http://localhost/api/import", { + method: "POST", + body: JSON.stringify({ + measurements: [ + { + type: "WEIGHT", + value: 82.4, + unit: "kg", + measuredAt: "2026-05-14T07:00:00.000Z", + }, + ], + }), + headers: { "content-type": "application/json" }, + }), + ); + + expect(response.status).toBeLessThan(400); + // Every measurement raised a duplicate — no rollup fold needed. + expect(recomputeBucketsForMeasurement).not.toHaveBeenCalled(); + }); +}); diff --git a/src/app/api/import/route.ts b/src/app/api/import/route.ts index 5911656c4..a37f4f221 100644 --- a/src/app/api/import/route.ts +++ b/src/app/api/import/route.ts @@ -18,6 +18,11 @@ import { glucoseContextEnum, } from "@/lib/validations/measurement"; import { recomputeUserMoodRollups } from "@/lib/mood/rollups"; +import { + collapseToTypeDayKeys, + recomputeBucketsForMeasurement, +} from "@/lib/measurements/rollups"; +import type { MeasurementType } from "@/generated/prisma/client"; // Derived from canonical enum so round-trip export → import covers every // type. Previous hardcoded subset silently dropped 4 of 11 types @@ -82,9 +87,19 @@ export const POST = apiHandler(async (request: NextRequest) => { const stats = { measurements: 0, moodEntries: 0, skipped: 0 }; // Import measurements + // v1.4.39.1 — track each (type, measuredAt) we wrote so we can re- + // fold the persistent rollup tier at the end. Pre-fix, the import + // path bypassed the rollup write hook entirely, so a CSV / JSON + // restore left the user's `source=rollup` chart paths under-counting + // the imported days until the next worker boot re-ran the backfill. + const touchedMeasurements: Array<{ + type: MeasurementType; + measuredAt: Date; + }> = []; if (data.measurements?.length) { for (const m of data.measurements) { try { + const measuredAt = new Date(m.measuredAt); await prisma.measurement.create({ data: { userId, @@ -92,11 +107,15 @@ export const POST = apiHandler(async (request: NextRequest) => { value: m.value, unit: m.unit, source: "IMPORT", - measuredAt: new Date(m.measuredAt), + measuredAt, notes: m.notes || null, glucoseContext: m.glucoseContext ?? null, }, }); + touchedMeasurements.push({ + type: m.type as MeasurementType, + measuredAt, + }); stats.measurements++; } catch { // Unique constraint violation — skip duplicate @@ -148,6 +167,28 @@ export const POST = apiHandler(async (request: NextRequest) => { } } + // v1.4.39.1 — refresh the persistent measurement rollup table for + // every distinct (type, day) the import touched. Collapsed so a + // 10 000-row CSV restore pays at most ~N (type, day) recomputes + // rather than 10 000 per-row hooks. Best-effort: a populator hiccup + // never fails the importer. + if (touchedMeasurements.length > 0) { + try { + const keys = collapseToTypeDayKeys(touchedMeasurements); + for (const k of keys) { + await recomputeBucketsForMeasurement(userId, k.type, k.measuredAt); + } + } catch (err) { + annotate({ + meta: { + measurement_rollup_import_failed: true, + measurement_rollup_import_error: + err instanceof Error ? err.message : String(err), + }, + }); + } + } + await auditLog("import.upload", { userId, ipAddress: getClientIp(request), diff --git a/src/app/api/medications/intake/__tests__/route.test.ts b/src/app/api/medications/intake/__tests__/route.test.ts index 9542c8977..3c99a7b7f 100644 --- a/src/app/api/medications/intake/__tests__/route.test.ts +++ b/src/app/api/medications/intake/__tests__/route.test.ts @@ -208,8 +208,20 @@ describe("v1.4.39 W-MED — compliance rollup read swap", () => { vi.mocked(prisma.$queryRaw).mockResolvedValue([ { rolled_days: BigInt(7), event_days: BigInt(7) }, ] as never); + // v1.4.39.1 — anchor the seed row on the runtime's "today" so the + // test stays green across the calendar regardless of when it + // runs. The pre-fix shape hard-coded `2026-05-18`, which fell out + // of the trailing-7-day window on every subsequent wall-clock + // day and silently shifted the body.data tail off the seed. + const todayKey = (() => { + const now = new Date(); + const y = now.getUTCFullYear(); + const m = String(now.getUTCMonth() + 1).padStart(2, "0"); + const d = String(now.getUTCDate()).padStart(2, "0"); + return `${y}-${m}-${d}`; + })(); vi.mocked(prisma.medicationComplianceRollup.findMany).mockResolvedValue([ - { day: "2026-05-18", scheduled: 3, taken: 2, skipped: 1 }, + { day: todayKey, scheduled: 3, taken: 2, skipped: 1 }, ] as never); const res = await GET( diff --git a/src/lib/measurements/__tests__/rollups.test.ts b/src/lib/measurements/__tests__/rollups.test.ts index 06382789e..cdc550aa8 100644 --- a/src/lib/measurements/__tests__/rollups.test.ts +++ b/src/lib/measurements/__tests__/rollups.test.ts @@ -546,8 +546,8 @@ describe("enqueueBootTimeRollupBackfill", () => { // v1.4.39 W-SUM — the discovery query now unions in users whose // existing DAY rollup rows carry `sum_value IS NULL`. Re-folding // converges those rows because `persistRollupRows` always writes the - // new column on upsert; the union keeps the per-type missing-coverage - // gap (v1.4.38.5) and the legacy-NULL backfill on the same indexed + // new column on upsert; the union keeps the per-day missing-coverage + // gap (v1.4.39.1) and the legacy-NULL backfill on the same indexed // pass. it("includes the sum_value IS NULL branch in the discovery query", async () => { getGlobalBossMock.mockReturnValue({ send: bossSend }); @@ -557,7 +557,7 @@ describe("enqueueBootTimeRollupBackfill", () => { const result = await enqueueBootTimeRollupBackfill(); expect(result).toEqual({ enqueued: 1, skipped: 0, error: null }); - // The discovery SQL surfaces both per-type missing coverage AND + // The discovery SQL surfaces both per-day missing coverage AND // legacy NULL sum_value rows under one UNION. The text-anchor is // the only durable assertion at the unit level — the integration // suite covers the planner shape on real Postgres. @@ -566,4 +566,26 @@ describe("enqueueBootTimeRollupBackfill", () => { expect(sqlText).toContain("sum_value"); expect(sqlText).toContain("UNION"); }); + + // v1.4.39.1 — the discovery anchor moved from per-type to per-day so + // accounts whose Withings sync / `/api/import` / admin restore wrote + // measurements without firing the rollup write hook get re-enqueued + // on the next worker boot. Pre-fix, the per-type LEFT JOIN matched + // on `(user, type)` and the user dropped off the discovery list as + // soon as ONE rollup row existed for the type — even if 27 OTHER + // days for that type were missing rollup rows. + it("anchors the missing-coverage join on (user, type, bucket_start) so per-day gaps surface", async () => { + getGlobalBossMock.mockReturnValue({ send: bossSend }); + queryRaw.mockResolvedValueOnce([{ id: "withings-stranded-user" }]); + bossSend.mockResolvedValueOnce("job-id"); + + await enqueueBootTimeRollupBackfill(); + + const sqlParts = queryRaw.mock.calls[0][0] as TemplateStringsArray; + const sqlText = Array.isArray(sqlParts) ? sqlParts.join("?") : ""; + // Inner DISTINCT widens to `(user_id, type, bucket_start)`. + expect(sqlText).toContain('date_trunc(\'day\', m."measured_at")'); + // LEFT JOIN now compares bucket_start on both sides. + expect(sqlText).toContain('r."bucket_start" = mt."bucket_start"'); + }); }); diff --git a/src/lib/measurements/rollups.ts b/src/lib/measurements/rollups.ts index 9cb3fe394..74f9bb455 100644 --- a/src/lib/measurements/rollups.ts +++ b/src/lib/measurements/rollups.ts @@ -741,35 +741,46 @@ export async function enqueueBootTimeRollupBackfill(): Promise<{ } try { - // Users where at least one measurement type lacks DAY-rollup - // coverage. Anchored on the per-user `DISTINCT type` set so the - // planner uses the `(user_id, type, measured_at)` index path; the - // LEFT JOIN onto `measurement_rollups` uses the - // `(user_id, type, granularity, bucket_start)` composite primary - // key. `measurement_rollups` has no surrogate `id` column — - // `r."bucket_start" IS NULL` (any column from the right side of - // the LEFT JOIN) marks the unmatched partitions. A user surfaces - // here when ANY type is missing — including the brand-new-type - // case that v1.4.35.1's zero-rollup-only shape silently stranded. + // Users where at least one (type, day) pair in `measurements` + // has no matching DAY rollup row. + // + // v1.4.39.1 — tightened from per-type to per-day. The legacy shape + // matched users where ANY type had zero rollup rows. That stranded + // accounts whose Withings sync / `/api/import` / admin restore + // wrote measurements without firing the rollup write hook: as soon + // as ONE rollup row existed for the type (even from a prior boot + // backfill), the LEFT JOIN found a match and the user dropped off + // the discovery list — leaving the dashboard chart's + // `source=rollup` fast-path under-counting the missing days. + // Switching the join to `(user, type, bucketStart)` surfaces every + // per-day gap. Cost: the inner DISTINCT widens from + // `(user_id, type)` (≈15 rows/user) to + // `(user_id, type, day)` (≈5 k rows/user on a 5-year power user), + // which is still bounded by the `(user_id, type, measured_at)` + // index path. The LEFT JOIN walks `measurement_rollups` against + // its composite primary key. // // v1.4.39 W-SUM — the discovery also surfaces users whose existing // DAY rollup rows carry `sum_value IS NULL`. Those rows pre-date // the v1.4.39 writer change; re-folding the user converges them // because `persistRollupRows` always writes the new column on - // upsert. The union runs in one query so the existing per-type - // coverage gap and the legacy-NULL backfill share a single planner - // pass. + // upsert. The union runs in one query so the per-day coverage gap + // and the legacy-NULL backfill share a single planner pass. const users = await prisma.$queryRaw>` SELECT DISTINCT id FROM ( SELECT DISTINCT mt."user_id" AS id FROM ( - SELECT DISTINCT m."user_id", m."type" + SELECT DISTINCT + m."user_id", + m."type", + date_trunc('day', m."measured_at") AS bucket_start FROM measurements m ) mt LEFT JOIN measurement_rollups r ON r."user_id" = mt."user_id" AND r."type" = mt."type" AND r."granularity" = 'DAY' + AND r."bucket_start" = mt."bucket_start" WHERE r."bucket_start" IS NULL UNION SELECT DISTINCT r2."user_id" AS id diff --git a/src/lib/withings/__tests__/sync-activity.test.ts b/src/lib/withings/__tests__/sync-activity.test.ts index 38c5c8a9a..e0d196dd7 100644 --- a/src/lib/withings/__tests__/sync-activity.test.ts +++ b/src/lib/withings/__tests__/sync-activity.test.ts @@ -45,12 +45,27 @@ vi.mock("@/lib/logging/context", () => ({ })), })); +// v1.4.39.1 — the sync now hands its touched (type, day) keys to the +// measurement rollup writer so the dashboard chart's `source=rollup` +// fast-path sees the new buckets. The mock collects the call args so +// the regression test below can assert the hook fired. +vi.mock("@/lib/measurements/rollups", async () => { + const actual = await vi.importActual< + typeof import("@/lib/measurements/rollups") + >("@/lib/measurements/rollups"); + return { + ...actual, + recomputeBucketsForMeasurement: vi.fn().mockResolvedValue(undefined), + }; +}); + import { prisma } from "@/lib/db"; import { parkIntegrationAtReauth, recordSyncFailure, recordSyncSuccess, } from "@/lib/integrations/status"; +import { recomputeBucketsForMeasurement } from "@/lib/measurements/rollups"; import { fetchWithingsActivity, syncUserActivity } from "../sync-activity"; @@ -363,3 +378,66 @@ describe("syncUserActivity — scope-skip guard (v1.4.26)", () => { expect(parkIntegrationAtReauth).not.toHaveBeenCalled(); }); }); + +describe("syncUserActivity — measurement rollup hook (v1.4.39.1)", () => { + it("folds the persistent rollup table for each (type, day) the sync touched", async () => { + // Two distinct calendar days × three activity metrics each = 6 + // writes; the rollup hook should be called once per distinct + // `(type, dayStart)` so the dashboard chart's `source=rollup` fast- + // path lights up the Withings-ingested days. Pre-v1.4.39.1 the + // sync skipped the hook entirely, leaving the rollup tier blind to + // Withings activity. + installFetchMock([ + { date: "2026-05-12", steps: 8420, distance: 6720, calories: 412 }, + { date: "2026-05-13", steps: 9100, distance: 7100, calories: 430 }, + ]); + vi.mocked(prisma.measurement.findFirst).mockResolvedValue(null); + vi.mocked(prisma.measurement.create).mockResolvedValue({} as never); + + await syncUserActivity("user-1"); + + expect(recomputeBucketsForMeasurement).toHaveBeenCalledTimes(6); + const calls = vi + .mocked(recomputeBucketsForMeasurement) + .mock.calls.map((c) => ({ + userId: c[0], + type: c[1], + measuredAtIso: (c[2] as Date).toISOString(), + })); + // Distinct (type, day) pairs the sync touched, derived from the two + // dates × three metrics × `collapseToTypeDayKeys`. + const distinct = new Set(calls.map((c) => `${c.type}|${c.measuredAtIso}`)); + expect(distinct.size).toBe(6); + expect(calls.every((c) => c.userId === "user-1")).toBe(true); + }); + + it("collapses multiple writes for the same (type, day) into a single rollup recompute", async () => { + // A single-date entry exercises the existing day-collapse helper — + // three metric rows for 2026-05-12 must fold into three rollup + // calls (one per type), not nine. + installFetchMock([ + { date: "2026-05-12", steps: 8420, distance: 6720, calories: 412 }, + ]); + vi.mocked(prisma.measurement.findFirst).mockResolvedValue(null); + vi.mocked(prisma.measurement.create).mockResolvedValue({} as never); + + await syncUserActivity("user-1"); + + expect(recomputeBucketsForMeasurement).toHaveBeenCalledTimes(3); + }); + + it("swallows a populator failure so the sync still returns its imported count", async () => { + // The rollup recompute is a best-effort cache tier — a populator + // hiccup must never fail the user's sync. The sync's `imported` + // return value should still reflect the rows written. + installFetchMock([{ date: "2026-05-12", steps: 8420 }]); + vi.mocked(prisma.measurement.findFirst).mockResolvedValue(null); + vi.mocked(prisma.measurement.create).mockResolvedValue({} as never); + vi.mocked(recomputeBucketsForMeasurement).mockRejectedValueOnce( + new Error("simulated rollup failure"), + ); + + const imported = await syncUserActivity("user-1"); + expect(imported).toBe(1); + }); +}); diff --git a/src/lib/withings/__tests__/sync-sleep.test.ts b/src/lib/withings/__tests__/sync-sleep.test.ts index e79c9cfa0..04d005488 100644 --- a/src/lib/withings/__tests__/sync-sleep.test.ts +++ b/src/lib/withings/__tests__/sync-sleep.test.ts @@ -44,12 +44,27 @@ vi.mock("@/lib/logging/context", () => ({ })), })); +// v1.4.39.1 — the sleep sync now folds the persistent measurement +// rollup tier for each touched SLEEP_DURATION day so the dashboard +// chart's `source=rollup` fast-path sees the new segments. The mock +// surfaces the call args for the regression test below. +vi.mock("@/lib/measurements/rollups", async () => { + const actual = await vi.importActual< + typeof import("@/lib/measurements/rollups") + >("@/lib/measurements/rollups"); + return { + ...actual, + recomputeBucketsForMeasurement: vi.fn().mockResolvedValue(undefined), + }; +}); + import { prisma } from "@/lib/db"; import { parkIntegrationAtReauth, recordSyncFailure, recordSyncSuccess, } from "@/lib/integrations/status"; +import { recomputeBucketsForMeasurement } from "@/lib/measurements/rollups"; import { fetchWithingsSleep, @@ -352,3 +367,47 @@ describe("syncUserSleep — scope-skip guard (v1.4.26)", () => { expect(parkIntegrationAtReauth).not.toHaveBeenCalled(); }); }); + +describe("syncUserSleep — measurement rollup hook (v1.4.39.1)", () => { + it("folds the persistent rollup table once per night the sync touched", async () => { + // Two distinct nights, each with multiple stage segments. The + // sync should collapse the per-night segments to one rollup + // recompute per night via `collapseToTypeDayKeys`. Pre-v1.4.39.1 + // the rollup tier never heard about Withings sleep at all. + const nightA = Math.floor(Date.UTC(2026, 4, 12, 22) / 1000); // 2026-05-12 22:00 UTC + const nightB = Math.floor(Date.UTC(2026, 4, 13, 22) / 1000); // 2026-05-13 22:00 UTC + installFetchMock([ + { startdate: nightA, enddate: nightA + 3600, state: 1, id: 1 }, + { startdate: nightA + 3600, enddate: nightA + 5400, state: 2, id: 1 }, + { startdate: nightA + 5400, enddate: nightA + 7200, state: 3, id: 1 }, + { startdate: nightB, enddate: nightB + 3600, state: 1, id: 2 }, + { startdate: nightB + 3600, enddate: nightB + 5400, state: 2, id: 2 }, + ]); + vi.mocked(prisma.measurement.findFirst).mockResolvedValue(null); + vi.mocked(prisma.measurement.create).mockResolvedValue({} as never); + + await syncUserSleep("user-1"); + + // Both segments of night A land in the same UTC day; night B in + // the next. Distinct (SLEEP_DURATION, dayStart) tuples = 2. + expect(recomputeBucketsForMeasurement).toHaveBeenCalledTimes(2); + const types = vi + .mocked(recomputeBucketsForMeasurement) + .mock.calls.map((c) => c[1]); + expect(new Set(types)).toEqual(new Set(["SLEEP_DURATION"])); + }); + + it("swallows a populator failure so the sync still returns its imported count", async () => { + installFetchMock([ + { startdate: 1715000000, enddate: 1715003600, state: 2, id: 1 }, + ]); + vi.mocked(prisma.measurement.findFirst).mockResolvedValue(null); + vi.mocked(prisma.measurement.create).mockResolvedValue({} as never); + vi.mocked(recomputeBucketsForMeasurement).mockRejectedValueOnce( + new Error("simulated rollup failure"), + ); + + const imported = await syncUserSleep("user-1"); + expect(imported).toBe(1); + }); +}); diff --git a/src/lib/withings/sync-activity.ts b/src/lib/withings/sync-activity.ts index f541a6b7c..b3eb3b086 100644 --- a/src/lib/withings/sync-activity.ts +++ b/src/lib/withings/sync-activity.ts @@ -38,6 +38,10 @@ import type { MeasurementType } from "@/generated/prisma/client"; import { prisma } from "@/lib/db"; import { getEvent } from "@/lib/logging/context"; import { getUnitForType } from "@/lib/validations/measurement"; +import { + collapseToTypeDayKeys, + recomputeBucketsForMeasurement, +} from "@/lib/measurements/rollups"; import { hasActivityScope } from "./client"; import { @@ -285,6 +289,11 @@ export async function syncUserActivity( } let imported = 0; + // v1.4.39.1 — track every (type, measuredAt) we touched so the + // persistent rollup tier can be re-folded at the end of the sync. + // See sync.ts header for the full rationale; the chart's + // `source=rollup` fast-path otherwise misses Withings activity days. + const touched: Array<{ type: MeasurementType; measuredAt: Date }> = []; for (const entry of entries) { if (!entry.date) continue; const measuredAt = activityMeasuredAt(entry.date); @@ -323,6 +332,7 @@ export async function syncUserActivity( }, }); } + touched.push({ type, measuredAt }); imported++; } catch (err) { getEvent()?.addWarning( @@ -332,6 +342,20 @@ export async function syncUserActivity( } } + // v1.4.39.1 — refresh the persistent rollup table for every distinct + // (type, day) the sync touched. Collapsed so a multi-day Withings + // catch-up costs at most ~N (type, day) recomputes. + try { + const keys = collapseToTypeDayKeys(touched); + for (const k of keys) { + await recomputeBucketsForMeasurement(userId, k.type, k.measuredAt); + } + } catch (err) { + getEvent()?.addWarning( + `withings activity: rollup recompute failed for ${userId}: ${err}`, + ); + } + await recordSyncSuccess(userId, "withings"); return imported; } diff --git a/src/lib/withings/sync-sleep.ts b/src/lib/withings/sync-sleep.ts index b69f16ddf..4e85858ab 100644 --- a/src/lib/withings/sync-sleep.ts +++ b/src/lib/withings/sync-sleep.ts @@ -36,6 +36,10 @@ import type { MeasurementType, SleepStage } from "@/generated/prisma/client"; import { prisma } from "@/lib/db"; import { getEvent } from "@/lib/logging/context"; import { getUnitForType } from "@/lib/validations/measurement"; +import { + collapseToTypeDayKeys, + recomputeBucketsForMeasurement, +} from "@/lib/measurements/rollups"; import { hasActivityScope } from "./client"; import { @@ -223,6 +227,10 @@ export async function syncUserSleep( let imported = 0; let segmentIndex = 0; + // v1.4.39.1 — track every (type, measuredAt) we touched so the + // persistent rollup tier can be re-folded at the end. See sync.ts for + // the full rationale. + const touched: Array<{ type: MeasurementType; measuredAt: Date }> = []; for (const segment of segments) { const stage = mapWithingsSleepState(segment.state); if (!stage) { @@ -270,6 +278,7 @@ export async function syncUserSleep( }, }); } + touched.push({ type: SLEEP_TYPE, measuredAt }); imported++; } catch (err) { getEvent()?.addWarning( @@ -279,6 +288,20 @@ export async function syncUserSleep( segmentIndex++; } + // v1.4.39.1 — refresh the persistent rollup table for every distinct + // (type, day) the sync touched. Sleep segments collapse heavily — + // ~10 stage rows per night land in one DAY recompute. + try { + const keys = collapseToTypeDayKeys(touched); + for (const k of keys) { + await recomputeBucketsForMeasurement(userId, k.type, k.measuredAt); + } + } catch (err) { + getEvent()?.addWarning( + `withings sleep: rollup recompute failed for ${userId}: ${err}`, + ); + } + await recordSyncSuccess(userId, "withings"); return imported; } diff --git a/src/lib/withings/sync.ts b/src/lib/withings/sync.ts index 9cf117105..cd6068eb6 100644 --- a/src/lib/withings/sync.ts +++ b/src/lib/withings/sync.ts @@ -18,6 +18,10 @@ import { recordSyncFailure, recordSyncSuccess, } from "@/lib/integrations/status"; +import { + collapseToTypeDayKeys, + recomputeBucketsForMeasurement, +} from "@/lib/measurements/rollups"; /** * Build the callback URL handed to Withings at `Notify.subscribe` time. @@ -194,6 +198,18 @@ export async function syncUserMeasurements( } let imported = 0; + // v1.4.39.1 — track every (type, measuredAt) we touched so the + // persistent rollup tier can be re-folded at the end of the sync. + // Pre-fix, Withings-ingested BP / weight / pulse / body-fat rows + // skipped the rollup write hook entirely: subsequent dashboard chart + // fetches with `source=rollup` saw fewer DAY buckets than the live + // measurements table, painting the "Noch nicht genug Daten" empty + // state for the 30-day range on accounts whose recent data only came + // through Withings. We collect first, fold once at the end so a 100- + // row monthly catch-up costs at most ~N (type, day) recomputes rather + // than N per-row hooks. Best-effort: a populator hiccup never fails + // the user's sync. + const touched: Array<{ type: MeasurementType; measuredAt: Date }> = []; for (const m of measures) { const measType = m.type as MeasurementType; @@ -233,12 +249,29 @@ export async function syncUserMeasurements( }, }); } + touched.push({ type: measType, measuredAt: m.measuredAt }); imported++; } catch (err) { getEvent()?.addWarning(`Failed to upsert measure: ${err}`); } } + // v1.4.39.1 — refresh the persistent rollup table for every distinct + // (type, day) the sync touched. The chart-data + analytics read paths + // consume these buckets via `source=rollup`; without this hook, + // Withings-only metrics stayed off the rollup tier and the dashboard + // chart at the 30-day range under-counted recent days. + try { + const keys = collapseToTypeDayKeys(touched); + for (const k of keys) { + await recomputeBucketsForMeasurement(userId, k.type, k.measuredAt); + } + } catch (err) { + getEvent()?.addWarning( + `withings: rollup recompute failed for ${userId}: ${err}`, + ); + } + // Update last synced timestamp await prisma.withingsConnection.update({ where: { userId },