mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-12 06:06:10 +00:00
perf(rollups): boot-backfill discovery widens to per-type missing coverage
Marc's account hit the slow-path live SQL aggregator on every read
after every deploy: `/api/analytics` cold-mounted in ~75 s,
`/api/mood/analytics` in ~13 s, `slim_summaries` annotated
`path:"live", row_count:347114`. Root cause traced to
`enqueueBootTimeRollupBackfill`'s discovery query:
WHERE NOT EXISTS (
SELECT 1 FROM measurement_rollups r WHERE r."user_id" = u."id"
)
The clause only matched users with ZERO rollup rows. Any account
that had ever been folded (every long-running user) silently
stranded once a brand-new measurement type landed: the existing
WEIGHT bucket kept the user invisible to discovery, but the new
type's empty DAY partition kept `isFullyCovered(coverage)` at
`false` forever. Every analytics / dashboard read then fanned out
to the live aggregator across the full measurement table.
Discovery now joins `(user, distinct type)` against
`measurement_rollups` on the `(user, type, granularity=DAY)`
partition and surfaces users where any partition is unmatched. The
v1.4.35.1 zero-rollup case is still covered (the LEFT JOIN of an
empty rollup table emits a row for every type the user has logged).
Integration test pins the partial-coverage regression: a user with
WEIGHT rolled up + a brand-new PULSE row now lands on the queue.
This commit is contained in:
1 parent
e1528952bc
commit
93d639c746
2 files changed
+88
-15
No files matched your search
@@ -703,9 +703,16 @@ async function ensureUserRollupsFreshImpl(
|
||||
* persistent rollup tier without operator intervention.
|
||||
*
|
||||
* Idempotent across reboots:
|
||||
* - the discovery query only matches accounts with no rollup rows at
|
||||
* all, so once the fold completes the user drops off the list and
|
||||
* subsequent boots skip them
|
||||
* - the discovery query matches accounts that either have zero
|
||||
* rollup coverage at all OR have at least one measurement type
|
||||
* whose DAY-rollup partition is empty. Once every type the user
|
||||
* has logged has at least one DAY bucket, the user drops off the
|
||||
* list and subsequent boots skip them. The v1.4.35.1 shape (zero-
|
||||
* rollups only) silently stranded users who logged a brand-new
|
||||
* type after the initial fold — their `isFullyCovered` probe stays
|
||||
* `false`, every read fans out to the live SQL aggregator, and
|
||||
* `/api/analytics` cold-mounts in tens of seconds instead of
|
||||
* 1.5-3 s. v1.4.38.5 widens discovery to per-type missing coverage.
|
||||
* - pg-boss `singletonKey` coalesces duplicate sends within the
|
||||
* queue, so a fast restart while a backfill is queued doesn't
|
||||
* double up
|
||||
@@ -724,19 +731,26 @@ export async function enqueueBootTimeRollupBackfill(): Promise<{
|
||||
}
|
||||
|
||||
try {
|
||||
// Users with at least one measurement but no rollup row yet. The
|
||||
// `LIMIT 1` inside each `EXISTS` keeps the planner from materialising
|
||||
// the full child relation just to answer existence — important on
|
||||
// power-user accounts with 100k+ measurements.
|
||||
// Users where at least one measurement type lacks DAY-rollup
|
||||
// coverage. Anchored on the per-user `DISTINCT type` set so the
|
||||
// planner uses the `(user_id, type, measured_at)` index path; the
|
||||
// LEFT JOIN onto `measurement_rollups` uses the
|
||||
// `(user_id, type, granularity, bucket_start)` composite primary
|
||||
// key. `r."id" IS NULL` filters the unmatched partitions. A user
|
||||
// surfaces here when ANY type is missing — including the brand-
|
||||
// new-type case that v1.4.35.1's zero-rollup-only shape silently
|
||||
// stranded.
|
||||
const users = await prisma.$queryRaw<Array<{ id: string }>>`
|
||||
SELECT u."id"
|
||||
FROM users u
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM measurements m WHERE m."user_id" = u."id" LIMIT 1
|
||||
)
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM measurement_rollups r WHERE r."user_id" = u."id" LIMIT 1
|
||||
)
|
||||
SELECT DISTINCT mt."user_id" AS id
|
||||
FROM (
|
||||
SELECT DISTINCT m."user_id", m."type"
|
||||
FROM measurements m
|
||||
) mt
|
||||
LEFT JOIN measurement_rollups r
|
||||
ON r."user_id" = mt."user_id"
|
||||
AND r."type" = mt."type"
|
||||
AND r."granularity" = 'DAY'
|
||||
WHERE r."id" IS NULL
|
||||
`;
|
||||
|
||||
if (users.length === 0) {
|
||||
|
||||
@@ -638,6 +638,65 @@ describe("measurement rollups — integration", () => {
|
||||
vi.mocked(getGlobalBoss).mockReturnValue(null as never);
|
||||
});
|
||||
|
||||
// ─── v1.4.38.5 regression — partial coverage discovery ──────
|
||||
//
|
||||
// The v1.4.35.1 shape only matched users with ZERO rollup rows
|
||||
// anywhere. That silently stranded users who logged a brand-new
|
||||
// type after the initial fold: their existing WEIGHT bucket made
|
||||
// them invisible to discovery, but the brand-new PULSE type still
|
||||
// had no DAY bucket → `isFullyCovered` stayed `false` → every read
|
||||
// fanned out to the live aggregator (10s of seconds on power-user
|
||||
// accounts). v1.4.38.5 widens discovery to per-type-missing.
|
||||
|
||||
it("v1.4.38.5 — enqueues a user with rollups for one type but a brand-new uncovered type", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
const user = await prisma.user.create({
|
||||
data: {
|
||||
username: "boot-backfill-partial",
|
||||
email: "boot-backfill-partial@example.test",
|
||||
role: "USER",
|
||||
},
|
||||
});
|
||||
await prisma.measurement.create({
|
||||
data: {
|
||||
userId: user.id,
|
||||
type: "WEIGHT",
|
||||
value: 80,
|
||||
unit: "kg",
|
||||
source: "MANUAL",
|
||||
measuredAt: new Date("2026-04-01T08:00:00.000Z"),
|
||||
},
|
||||
});
|
||||
// Fold WEIGHT — this is the state a long-running account is in
|
||||
// after the initial v1.4.35.1 boot-time backfill landed.
|
||||
await recomputeUserRollups(user.id, { granularities: ["DAY"] });
|
||||
|
||||
// Now log a brand-new type. The v1.4.35.1 discovery saw "user
|
||||
// has at least one rollup row" and skipped them; v1.4.38.5
|
||||
// discovery checks per-type coverage and surfaces them.
|
||||
await prisma.measurement.create({
|
||||
data: {
|
||||
userId: user.id,
|
||||
type: "PULSE",
|
||||
value: 65,
|
||||
unit: "bpm",
|
||||
source: "MANUAL",
|
||||
measuredAt: new Date("2026-04-02T08:00:00.000Z"),
|
||||
},
|
||||
});
|
||||
|
||||
bossSend.mockResolvedValue("job-id");
|
||||
vi.mocked(getGlobalBoss).mockReturnValue({ send: bossSend } as never);
|
||||
|
||||
const result = await enqueueBootTimeRollupBackfill();
|
||||
|
||||
expect(result.enqueued).toBe(1);
|
||||
expect(bossSend).toHaveBeenCalledTimes(1);
|
||||
expect(bossSend.mock.calls[0][1]).toMatchObject({ userId: user.id });
|
||||
|
||||
vi.mocked(getGlobalBoss).mockReturnValue(null as never);
|
||||
});
|
||||
|
||||
// ─── v1.4.36 rollup-fresh skip-live contract ──────────────────
|
||||
//
|
||||
// The v1.4.35 read-swap kept the heavy live aggregate running in
|
||||
|
||||
Reference in new issue
Block a user