v1.4.35 — persistent measurement rollups + partial read-swap

Foundation release for the persistent measurement_rollups cache tier.
Adds the additive table + enum, populates it from every measurement
write hook, lands the Apple-Health-import end-of-job backfill, and
ships the parity-checked read-swap on two reader surfaces.

The comprehensive insights aggregator and the slim analytics
summaries slice now source count / min / max / mean per type from the
DAY rollup buckets via aggregateBuckets, plus the comprehensive
dailyByType correlation feed straight from the bucket means. Slope /
R² / standard deviation / anomaly counts stay on live SQL — they do
not compose across DAY buckets. A defensive parity check against the
live aggregate's COUNT(*) falls back to live SQL when divergence is
detected.

Includes the new scripts/backfill-rollups.ts CLI for the one-shot
fold of historical measurements into all four granularities.

Tests: 4280 unit + 228 integration. tsc + lint clean. Migration
0067_v1434_measurement_rollups is idempotent under re-run.
This commit is contained in:
Marc-André Bombeck committed 2026-05-17 09:55:33 +02:00
1 parent 9715831f82
commit 8dc9edb65e
26 files changed
+2650 -65

No files matched your search

@@ -0,0 +1,54 @@
-- v1.5.0 — persistent measurement rollup table.
--
-- Second cache tier behind the 60-s LRU. Carries the
-- count / min / max / mean / sd / slope / r2 stats per
-- (userId, type, granularity, bucketStart). The DAY granularity is
-- recomputed inline with every measurement write so the next read is
-- correct; WEEK / MONTH / YEAR granularities are recomputed via the
-- pg-boss `rollup-recompute` queue so the write path stays sub-ms.
--
-- Additive only — no ALTER on existing tables, no DROP. Adds one
-- enum type, one table, and one composite-descending index. The
-- primary key is `(user_id, type, granularity, bucket_start)`, which
-- guarantees idempotent upserts under re-run.
-- Granularity enum. `DO $$ … EXCEPTION WHEN duplicate_object` is the
-- canonical guard for `CREATE TYPE` because Postgres does not accept
-- `CREATE TYPE IF NOT EXISTS`.
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_type
WHERE typname = 'measurement_rollup_granularity'
) THEN
CREATE TYPE "measurement_rollup_granularity"
AS ENUM ('DAY', 'WEEK', 'MONTH', 'YEAR');
END IF;
END
$$;
CREATE TABLE IF NOT EXISTS "measurement_rollups" (
"user_id" TEXT NOT NULL,
"type" "measurement_type" NOT NULL,
"granularity" "measurement_rollup_granularity" NOT NULL,
"bucket_start" TIMESTAMPTZ(3) NOT NULL,
"count" INTEGER NOT NULL,
"mean" DOUBLE PRECISION NOT NULL,
"min_value" DOUBLE PRECISION NOT NULL,
"max_value" DOUBLE PRECISION NOT NULL,
"sd" DOUBLE PRECISION,
"slope" DOUBLE PRECISION,
"r2" DOUBLE PRECISION,
"computed_at" TIMESTAMPTZ(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT "measurement_rollups_pkey"
PRIMARY KEY ("user_id", "type", "granularity", "bucket_start")
);
CREATE INDEX IF NOT EXISTS "measurement_rollups_user_type_granularity_bucket_desc_idx"
ON "measurement_rollups" ("user_id", "type", "granularity", "bucket_start" DESC);
ALTER TABLE "measurement_rollups"
ADD CONSTRAINT "measurement_rollups_user_id_fkey"
FOREIGN KEY ("user_id") REFERENCES "users"("id")
ON DELETE CASCADE ON UPDATE CASCADE;
+54
View File
@@ -253,6 +253,14 @@ model User {
// pg-boss job state so the polling endpoint can read live per-
// phase progress without round-tripping through pg-boss internals.
importJobs ImportJob[]
// v1.5.0 — persistent measurement rollup table. Carries the
// count/min/max/mean/sd/slope/r2 stats per (type, granularity,
// bucketStart) so the analytics + comprehensive read paths bypass
// the trailing-window row walk and survive process restarts +
// multi-instance fan-out. Populated synchronously on the DAY
// granularity from the measurement write hooks; WEEK/MONTH/YEAR
// are enqueued through pg-boss for the worker to fold.
measurementRollups MeasurementRollup[]
@@map("users")
}
@@ -454,6 +462,52 @@ model Measurement {
@@map("measurements")
}
// ─── Measurement rollups (v1.5.0) ──────────────────────────
//
// Persistent rollup table — second cache tier behind the 60-s LRU.
// One row per `(userId, type, granularity, bucketStart)`. The DAY
// granularity is recomputed inline with every measurement write so
// the next read is correct; WEEK / MONTH / YEAR granularities are
// recomputed via the pg-boss `rollup-recompute` queue so the write
// path stays sub-ms. The Apple Health import worker skips the per-
// row hook and calls `recomputeUserRollups` once at the end of the
// import to avoid quadratic write costs on 100k-row backfills.
//
// Stats semantics mirror the v1.4.34.1 comprehensive aggregator:
// - `sd` = `STDDEV_POP(value)` (population SD, /N)
// - `slope` = `REGR_SLOPE(value, EXTRACT(EPOCH FROM measured_at) / 86400.0)`
// - `r2` = `REGR_R2(value, EXTRACT(EPOCH FROM measured_at) / 86400.0)`
// so cached + live aggregates agree byte-for-byte.
enum RollupGranularity {
DAY
WEEK
MONTH
YEAR
@@map("measurement_rollup_granularity")
}
model MeasurementRollup {
userId String @map("user_id")
type MeasurementType
granularity RollupGranularity
bucketStart DateTime @map("bucket_start") @db.Timestamptz(3)
count Int
mean Float
minValue Float @map("min_value")
maxValue Float @map("max_value")
sd Float?
slope Float?
r2 Float?
computedAt DateTime @default(now()) @map("computed_at") @db.Timestamptz(3)
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
@@id([userId, type, granularity, bucketStart])
@@index([userId, type, granularity, bucketStart(sort: Desc)])
@@map("measurement_rollups")
}
// ─── Workouts (v1.4.25 W8d) ────────────────────────────────
//
// HKWorkout-aligned typed columns for the analytics hot path + an