tom/HealthLog
1
0 Fork 0
mirror of https://github.com/MBombeck/HealthLog.git synced 2026-11-08 20:42:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

perf(dashboard): drive the per-type summary reads from the type list

Browse Source 
Two reads behind the metric tiles scanned far more than the rows they
return, because PostgreSQL 16 has no index skip scan: a query that wants
one row per measurement type but never names a type cannot be served by
any index keyed on (user_id, type, ...).

The latest-reading read used DISTINCT ON (type), so it produced every
live row of the account in (type, measured_at DESC) order before keeping
one row per group. On a multi-year account that is a sequential scan plus
a sort that spills to disk once it outgrows work_mem. It is now a LATERAL
join over the type list, one lookup per type against the live-row partial
index from migration 0183, each stopping at the first row. CROSS JOIN
LATERAL is an inner join, so a type with no live reading drops out as
DISTINCT ON dropped it, and ORDER BY reproduces the same enum ordering.

The sparkline read filtered measurement_rollups on user_id and
granularity only, skipping the second column of the table's key, so on a
single-account host neither the primary key nor the user/type/
granularity/bucket index could drive it and every stored granularity got
scanned. It now names the full canonical enum, which filters nothing out
and restores the equality the key needs; the existing index serves it.

Measured on a fixture matching a real account (454k measurements, 79%
tombstoned, 75.5k rollup rows): the first read goes from a seq scan plus
a 3.2 MB external-merge sort at 112 ms to a nested loop at 0.7 ms, and
the second from a seq scan discarding 72k rows at 6.9 ms to a bitmap
index scan at 1.6 ms. Same rows, same order, same types.

Both reads move out of the route into src/lib/dashboard/summary-reads.ts
so the integration suite can exercise the SQL itself rather than a copy
of it, the same seam buildMedsTodayBlock already uses. No schema change:
adding an index on (user_id, granularity, bucket_start) would have landed
in the same range while leaving the existing index unused and adding a
fourth write target to a table the rollup workers upsert into constantly.
This commit is contained in:
Marc-André Bombeck committed 2026-09-03 01:47:41 +02:00
1 parent 214b268b41
commit f2e98e3885
2 files changed
+187 -90
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/app/api/dashboard/summary/route.ts
+26 -90
View File
@@ -22,14 +22,16 @@
* - the set of YYYY-MM-DD day-keys with any activity in 365 days
*
* v1.4.38 swaps the two unbounded reads for SQL aggregates:
* - `DISTINCT ON (type)` → one row per type carrying the latest
* value + measuredAt (≤ N_metrics rows). REG-11 (v1.4.44) dropped
* the trailing-7-day window from the WHERE clause; the tile now
* surfaces the all-time-latest reading per type so a 60-day-old BP
* measurement still feeds the tile (paired with the stale-caption
* hint built off `lastSeenAt`).
* - `measurement_rollups` DAY buckets keyed `(user_id, granularity,
* bucketStart)` → at most 7 buckets per metric × N_metrics rows.
* - one row per type carrying the latest value + measuredAt
* (≤ N_metrics rows). REG-11 (v1.4.44) dropped the trailing-7-day
* window from the WHERE clause; the tile now surfaces the all-time-
* latest reading per type so a 60-day-old BP measurement still feeds
* the tile (paired with the stale-caption hint built off
* `lastSeenAt`). v1.38.5 replaced the `DISTINCT ON (type)` that
* produced those rows with a LATERAL join over the type list; both
* per-type reads now live in `@/lib/dashboard/summary-reads`.
* - `measurement_rollups` DAY buckets keyed `(user_id, type,
* granularity, bucketStart)` → at most 7 buckets per metric × N_metrics rows.
* Sparkline points become the bucket means rather than individual
* raw samples, which is a *smoother* trend signal for high-volume
* metrics like ACTIVITY_STEPS and bounded for every other metric.
@@ -63,6 +65,10 @@ import { defaultLocale, locales, type Locale } from "@/lib/i18n/config";
import { userDayKey, DEFAULT_TIMEZONE } from "@/lib/tz/resolver";
import { cachedSwr, caches, type ServerCache } from "@/lib/cache/server-cache";
import { buildMedsTodayBlock } from "@/lib/dashboard/meds-today";
import {
readLatestEver,
readSparkBuckets,
} from "@/lib/dashboard/summary-reads";
import { resolveModuleMap } from "@/lib/modules/gate";
import { gateMetricCardsByModules } from "@/lib/dashboard/widget-modules";
import {
@@ -295,48 +301,6 @@ function computeStreak(activityDays: Set<string>, userTz: string): StreakInfo {
return { currentDays, longest };
}
/** v1.4.38 W-F — `DISTINCT ON (type)` row for the single most recent
* reading per measurement type, irrespective of age. One row per
* metric the user has ever touched; replaces the legacy unbounded
* `prisma.measurement.findMany`.
*
* REG-11 (v1.4.44): dropped the trailing-7-day window from the WHERE
* clause. The old shape returned `latestValue: null` + an empty
* sparkline for accounts whose last BP / pulse reading was older than
* 7 days, leaving the iOS tile blank even though the historical row
* was still in the database. The all-time aggregate (`groupBy` on
* `measurements`) already proves the row exists; the row count is
* still bounded by `|measurementTypes|` via `DISTINCT ON`. */
interface LatestEverRow {
type: MeasurementType;
value: number;
measured_at: Date;
}
/** v1.4.38 W-F — per-day measurement_rollup bucket feeding the dashboard
* sparkline. At most 7 buckets per metric × N metrics — bounded by
* `SPARK_DAYS * |measurementTypes|` rather than the raw row count.
*
* v1.4.39 W-SUM — `sum_value` rides along so the cumulative tile
* (ACTIVITY_STEPS) renders the daily SUM rather than the per-bucket
* MEAN. Spot metrics ignore the column; cumulative tiles fall back to
* `mean * count` when the legacy NULL hits (boot-backfill convergence
* window).
*
* REG-11 (v1.4.44): switched from a calendar-window filter
* (`bucket_start >= sevenDaysAgo`) to a `ROW_NUMBER() OVER (PARTITION
* BY type ORDER BY bucket_start DESC)` window so the sparkline takes
* the last `SPARK_DAYS` daily buckets per type regardless of age. An
* account whose last BP reading is 60 days old now still gets the
* trailing 7 days of historical buckets feeding the tile chart. */
interface SparklineRow {
type: MeasurementType;
bucket_start: Date;
mean: number;
count: number;
sum_value: number | null;
}
/** v1.4.38 W-F — distinct activity day-keys from the streak window.
* The route only needs YYYY-MM-DD strings, so the aggregate runs
* `date_trunc('day', m.measured_at AT TIME ZONE $userTz)` and returns
@@ -478,9 +442,9 @@ async function buildDashboardSummary(
// v1.4.38 W-F — six bounded sub-queries replace the legacy 4
// unbounded ones. Row counts are now:
// - latestEver: ≤ N metric types (one row per type via DISTINCT ON,
// REG-11: dropped the 7-day window so a stale-but-
// valid historical reading still feeds the tile)
// - latestEver: ≤ N metric types (one indexed lookup per type via
// LATERAL, REG-11: dropped the 7-day window so a
// stale-but-valid historical reading still feeds the tile)
// - sparkBuckets: ≤ SPARK_DAYS × N metric types (typically <80,
// REG-11: taken via ROW_NUMBER window, no calendar
// filter, so a 60-day-old metric still paints a chart)
@@ -498,43 +462,15 @@ async function buildDashboardSummary(
measurementStreakDays,
sleepStageRows,
] = await Promise.all([
time(
"latestEver",
() =>
prisma.$queryRaw<LatestEverRow[]>`
SELECT DISTINCT ON (m."type")
m."type" AS type,
m."value"::double precision AS value,
m."measured_at" AS measured_at
FROM measurements m
WHERE m."user_id" = ${userId}
AND m."deleted_at" IS NULL
ORDER BY m."type", m."measured_at" DESC
`,
),
time(
"sparkline",
() =>
prisma.$queryRaw<SparklineRow[]>`
SELECT type, bucket_start, mean, count, sum_value
FROM (
SELECT
r."type" AS type,
r."bucket_start" AS bucket_start,
r."mean"::double precision AS mean,
r."count"::int AS count,
r."sum_value"::double precision AS sum_value,
ROW_NUMBER() OVER (
PARTITION BY r."type"
ORDER BY r."bucket_start" DESC
) AS rn
FROM measurement_rollups r
WHERE r."user_id" = ${userId}
AND r."granularity" = 'DAY'
) sub
WHERE rn <= ${SPARK_DAYS}
ORDER BY type, bucket_start ASC
`,
// Both reads live in `@/lib/dashboard/summary-reads` — the SQL is a
// contract the integration suite exercises directly against Postgres,
// and each carries the reason it names the type list at its
// definition. Short version: PostgreSQL 16 has no index skip scan, so
// "one row per type" has to be driven FROM the type list or it
// degrades to a full pass over the account.
time("latestEver", () => readLatestEver(prisma, userId, measurementTypes)),
time("sparkline", () =>
readSparkBuckets(prisma, userId, measurementTypes, SPARK_DAYS),
),
time("allTime", () =>
prisma.measurement.groupBy({
src/lib/dashboard/summary-reads.ts
+161
View File
@@ -0,0 +1,161 @@
/**
* The two per-type reads behind the dashboard summary's metric tiles.
*
* Both used to live inline in `src/app/api/dashboard/summary/route.ts`.
* They are lifted out for the same reason `buildMedsTodayBlock` is: the
* SQL is the contract, and a contract that only exists inside a route
* handler cannot be exercised against a real Postgres. The integration
* suite runs THESE functions, not a transcription of them.
*
* Both reads are shaped by one property of the schema: PostgreSQL 16 has
* no index skip scan. An index on `(user_id, type, …)` cannot hand back
* "one row per type" on its own — something has to supply the type list.
* So both queries name it. `measurementTypes` is the complete canonical
* enum, so naming it filters nothing out; it exists to give the planner
* an equality on the leading type column, which turns a full pass over
* the account into one bounded lookup per type.
*/
import type { PrismaClient } from "@/generated/prisma/client";
import type { MeasurementType } from "@/generated/prisma/client";
/** v1.4.38 W-F — the single most recent reading per measurement
* type, irrespective of age. One row per metric the user has ever
* touched; replaces the legacy unbounded `prisma.measurement.findMany`.
*
* REG-11 (v1.4.44): dropped the trailing-7-day window from the WHERE
* clause. The old shape returned `latestValue: null` + an empty
* sparkline for accounts whose last BP / pulse reading was older than
* 7 days, leaving the iOS tile blank even though the historical row
* was still in the database. The all-time aggregate (`groupBy` on
* `measurements`) already proves the row exists; the row count is
* still bounded by `|measurementTypes|`, now because the query is
* driven FROM that list (v1.38.5) rather than grouped down to it. */
export interface LatestEverRow {
type: MeasurementType;
value: number;
measured_at: Date;
}
/** v1.4.38 W-F — per-day measurement_rollup bucket feeding the dashboard
* sparkline. At most `sparkDays` buckets per metric × N metrics —
* bounded by `sparkDays * |measurementTypes|` rather than the raw row
* count.
*
* v1.4.39 W-SUM — `sum_value` rides along so the cumulative tile
* (ACTIVITY_STEPS) renders the daily SUM rather than the per-bucket
* MEAN. Spot metrics ignore the column; cumulative tiles fall back to
* `mean * count` when the legacy NULL hits (boot-backfill convergence
* window).
*
* REG-11 (v1.4.44): switched from a calendar-window filter
* (`bucket_start >= sevenDaysAgo`) to a `ROW_NUMBER() OVER (PARTITION
* BY type ORDER BY bucket_start DESC)` window so the sparkline takes
* the last `sparkDays` daily buckets per type regardless of age. An
* account whose last BP reading is 60 days old now still gets the
* trailing 7 days of historical buckets feeding the tile chart. */
export interface SparklineRow {
type: MeasurementType;
bucket_start: Date;
mean: number;
count: number;
sum_value: number | null;
}
/**
* Latest live reading per measurement type.
*
* v1.38.5 — a LATERAL join over the type list, replacing the
* `DISTINCT ON (m."type")` this read used to be.
*
* The DISTINCT-ON shape has to produce every live row of the account in
* `(type, measured_at DESC)` order before it can throw all but the first
* of each group away. With no skip scan there is no index that hands it
* one row per type, so on a multi-year account it degrades to a
* sequential scan plus an in-memory sort that spills to disk once it
* outgrows `work_mem`.
*
* Driving the read from the type list instead makes it one index lookup
* per type against `measurements_live_covering_idx` (the partial
* `(user_id, type, measured_at DESC) WHERE deleted_at IS NULL` index from
* migration 0183), each stopping at the first row. `CROSS JOIN LATERAL`
* is an inner join, so a type with no live reading drops out exactly as
* DISTINCT ON dropped it, and `ORDER BY mt."type"` reproduces the enum
* ordering the old `ORDER BY m."type"` emitted. Same rows, same order.
*
* The driving relation is aliased `mt`, not `t`: a one-letter `t` alias
* makes `t("type")` in the SQL text look exactly like a `t()` translation
* call to the i18n call-site scanner, which then reports a missing key.
*/
export function readLatestEver(
db: PrismaClient,
userId: string,
measurementTypes: readonly MeasurementType[],
): Promise<LatestEverRow[]> {
return db.$queryRaw<LatestEverRow[]>`
SELECT
mt."type" AS type,
l."value" AS value,
l."measured_at" AS measured_at
FROM unnest(${[...measurementTypes]}::"measurement_type"[]) AS mt("type")
CROSS JOIN LATERAL (
SELECT
m."value"::double precision AS value,
m."measured_at" AS measured_at
FROM measurements m
WHERE m."user_id" = ${userId}
AND m."type" = mt."type"
AND m."deleted_at" IS NULL
ORDER BY m."measured_at" DESC
LIMIT 1
) l
ORDER BY mt."type"
`;
}
/**
* Trailing `sparkDays` DAY rollup buckets per measurement type.
*
* v1.38.5 — the `type` predicate is what makes this read indexable.
*
* `measurement_rollups` is keyed `(user_id, type, granularity,
* bucket_start, source)`. Filtering on `user_id` and `granularity` alone
* skips the second key column, so on a single-account host — where
* `user_id` selects nearly the whole table — neither the primary key nor
* `measurement_rollups_user_type_granularity_bucket_desc_idx` can drive
* the read, and Postgres falls back to a sequential scan over every
* granularity the account has ever rolled up.
*
* Naming the full enum restores the equality on `type` and the read
* becomes a bitmap index scan over the DAY slice. Deliberately NOT
* narrowed to the types the metric cards happen to render: the full enum
* keeps the returned row set provably identical to the unfiltered read,
* and a new measurement type stays included by construction.
*/
export function readSparkBuckets(
db: PrismaClient,
userId: string,
measurementTypes: readonly MeasurementType[],
sparkDays: number,
): Promise<SparklineRow[]> {
return db.$queryRaw<SparklineRow[]>`
SELECT type, bucket_start, mean, count, sum_value
FROM (
SELECT
r."type" AS type,
r."bucket_start" AS bucket_start,
r."mean"::double precision AS mean,
r."count"::int AS count,
r."sum_value"::double precision AS sum_value,
ROW_NUMBER() OVER (
PARTITION BY r."type"
ORDER BY r."bucket_start" DESC
) AS rn
FROM measurement_rollups r
WHERE r."user_id" = ${userId}
AND r."type" = ANY(${[...measurementTypes]}::"measurement_type"[])
AND r."granularity" = 'DAY'
) sub
WHERE rn <= ${sparkDays}
ORDER BY type, bucket_start ASC
`;
}