fix(rollups): rebase regression x-axis to a fixed origin for exact accumulators

The per-bucket OLS accumulators stored x on the raw epoch-day axis (x ~ 20540),
so sum_xx accumulated past ~1e10 and the square shed ~10 of a double's
significant digits before the value was ever stored. The sub-day x detail was
lost at accumulation time, which no read-side identity can recover; an
ill-conditioned window's composed slope/r2 then drifted from the live REGR_*
probe past the parity gauge.

Rebase x to a fixed origin (epoch-days of 2020-01-01) at write time so the
squared terms stay O(1e7) and exact. Slope, r2, and population sd are invariant
under the affine x-shift, so the live raw-epoch probe still parity-matches the
cross-bucket compose; composeRegression needs no formula change. The per-bucket
slope/r2 columns stay on the raw axis (shift-invariant).

Migration 0193 recomputes sum_x/sum_xy/sum_xx from measurements on the rebased
basis, one set-based update per granularity. Recompute-from-source is idempotent
and rerun-safe; it touches only rows that already carry the accumulators and
leaves pre-migration NULLs for the boot re-fold.

Un-quarantine the DST-boundary parity test: it now matches live REGR_* within
the existing 1e-9 relative bound, with the WEIGHT case still bit-identical.
This commit is contained in:
Marc-André Bombeck committed 2026-06-26 13:24:30 +02:00
1 parent 8471384d01
commit 8265abedff
5 files changed
+260 -59

No files matched your search

+40 -8
View File
@@ -53,6 +53,34 @@ export interface DailyMeanRow {
maxValue: number;
}
/**
* v1.21.0 — regression x-origin (epoch-days of 2020-01-01 UTC).
*
* The regression accumulators (`sum_x / sum_xy / sum_xx`) store x RELATIVE to
* this fixed origin: `x = EXTRACT(EPOCH FROM measured_at)/86400.0 − ORIGIN`.
* Raw epoch-days sit at x ≈ 20 540, so `x²` lands near 4.2e8 and `Σxx`
* accumulates past ~1e10 — squaring shaves ~10 of double's ~15-16 significant
* decimal digits BEFORE the value is ever stored, so the sub-day fractional x
* (the time-of-day) is already lossy at write time. No read-side identity can
* recover bits lost at accumulation. Rebasing to a recent origin keeps x in the
* low thousands (x² ≤ ~1e7 for any realistic window), so the squared terms stay
* comfortably exact.
*
* Slope / r² / population-sd are INVARIANT under an affine x-shift (subtracting
* a constant from every x leaves Sxx, Sxy, Syy unchanged), so the composed
* result is identical to the un-shifted basis — only the unreported intercept
* would move. The writer SQL and migration 0193 share this exact literal; the
* live REGR_* probe stays on raw epoch-days and matches because slope / r² / sd
* do not depend on the origin.
*
* 2020-01-01 = 18 262 days after the Unix epoch. It MUST stay a stable
* compile-time constant — changing it would orphan every stored accumulator
* until a full re-fold. Pick a value comfortably before any plausible reading.
*/
export const REGRESSION_X_ORIGIN_DAYS = Math.floor(
Date.UTC(2020, 0, 1) / 86_400_000,
); // 18262
/**
* v1.20.0 F6 — per-bucket OLS regression accumulators (epoch-day x-axis).
* Mirrors the four columns migration 0190 added to `measurement_rollups`
@@ -112,14 +140,18 @@ export interface ComposedRegression {
*
* This is algebraically identical to the determinant form (multiply numerator
* and denominator by n) and to Postgres' REGR_* / STDDEV_POP, but numerically
* stable: on the epoch-day x-axis (x ≈ 20 540) the determinant form computes
* `n·Σxx − Σx²` as a difference of two ~1e10 magnitudes and sheds ~10
* significant digits to catastrophic cancellation before the divide. The
* centered form subtracts `Σx²/n` from `Σxx` at the same scale per term, so
* the cancellation is bounded by the true x-variance, not the absolute x
* magnitude. Σy is read from the exact stored `sumValue` accumulator when
* present (falling back to `mean·count`), removing the AVG→multiply ULP
* residual on the y side too.
* stable: the centered form subtracts `Σx²/n` from `Σxx` at the same scale per
* term, so the cancellation is bounded by the true x-variance, not the absolute
* x magnitude.
*
* v1.21.0 — the accumulators are stored REBASED to `REGRESSION_X_ORIGIN_DAYS`
* (x = epoch_days − origin), so `Σxx` stays O(1e7) instead of O(1e10) and the
* squared terms never shed precision at write time. Slope / r² / sd are
* invariant under the x-shift, so this composes the SAME regression the live
* (un-rebased) REGR_* probe folds — the rebase only restores the bits the raw
* epoch-day square would have lost. Σy is read from the exact stored `sumValue`
* accumulator when present (falling back to `mean·count`), removing the
* AVG→multiply ULP residual on the y side too.
*
* The caller MUST collapse overlapping sources to the canonical source
* BEFORE handing rows here — the accumulators are per-source, so summing a
+49 -26
View File
@@ -34,6 +34,7 @@ import { isP2002 } from "@/lib/prisma-errors";
import {
collapseRollupRowsBySource,
loadUserSourcePriority,
REGRESSION_X_ORIGIN_DAYS,
} from "@/lib/rollups/measurement-read";
import { startOfUtcDay } from "@/lib/tz/start-of-utc-day";
import type {
@@ -59,6 +60,30 @@ export const ROLLUP_RECOMPUTE_QUEUE = "rollup-recompute";
/** Worker concurrency cap — recomputes are read-heavy on Postgres. */
export const ROLLUP_RECOMPUTE_CONCURRENCY = 2;
/**
* v1.21.0 — rebased epoch-day x for the regression accumulators.
*
* `sum_x / sum_xy / sum_xx` store x RELATIVE to `REGRESSION_X_ORIGIN_DAYS`
* (epoch-days of 2020-01-01) so the squared terms stay O(1e7) instead of the
* O(1e10) the raw epoch-day (x ≈ 20 540) basis produced — squaring a ~20 540 x
* sheds ~10 of double's significant decimal digits BEFORE the sum is stored,
* losing the sub-day x detail at write time. Slope / r² / sd are invariant under
* the x-shift, so the live REGR_* probe (which stays on raw epoch-days) still
* matches the cross-bucket compose. The `slope` / `r2` per-bucket columns also
* stay on raw epoch-days (shift-invariant; their stored values are unchanged).
*
* `REGRESSION_X_ORIGIN_DAYS` is a trusted compile-time integer; assert it stays
* one at SQL-build time before splicing it in (no user-input surface). Built
* lazily inside `runRollupAggregate` rather than at module init so a partial
* test mock of `measurement-read` can't trip the assertion before the SQL runs.
*/
function rebasedX(): string {
if (!Number.isInteger(REGRESSION_X_ORIGIN_DAYS)) {
throw new Error("REGRESSION_X_ORIGIN_DAYS must be an integer");
}
return `(EXTRACT(EPOCH FROM m."measured_at") / 86400.0 - ${REGRESSION_X_ORIGIN_DAYS})`;
}
/**
* Payload `boss.send` carries onto the `rollup-recompute` queue. The
* worker hands the values straight to `recomputeUserRollups`.
@@ -368,6 +393,10 @@ async function runRollupAggregate(input: {
}
const dateTrunc = `date_trunc('${truncUnit}', m."measured_at")`;
// v1.21.0 — rebased epoch-day x for the regression accumulators (see the
// `rebasedX` doc). Built per-call so the assertion fires with the real module.
const REBASED_X = rebasedX();
if (input.types && input.types.length > 0) {
// Build the type-list literal carefully. The enum values come
// from `MeasurementType` (Prisma-generated TS enum); we whitelist
@@ -401,20 +430,18 @@ async function runRollupAggregate(input: {
m."value",
EXTRACT(EPOCH FROM m."measured_at") / 86400.0
)::double precision AS r2,
-- v1.20.0 F6 — regression accumulators (epoch-day x-axis). Summed
-- per (type, source, bucket) here so the read tier composes the
-- windowed slope / r2 / sd across DAY buckets without a live scan.
-- value is NOT NULL on measurements, so every row contributes; the
-- sums are over the same rows REGR_/STDDEV_POP fold above.
SUM(EXTRACT(EPOCH FROM m."measured_at") / 86400.0)::double precision
AS sum_x,
SUM(
(EXTRACT(EPOCH FROM m."measured_at") / 86400.0) * m."value"
)::double precision AS sum_xy,
SUM(
(EXTRACT(EPOCH FROM m."measured_at") / 86400.0)
* (EXTRACT(EPOCH FROM m."measured_at") / 86400.0)
)::double precision AS sum_xx,
-- v1.20.0 F6 — regression accumulators. Summed per (type, source,
-- bucket) here so the read tier composes the windowed slope / r2 / sd
-- across DAY buckets without a live scan. value is NOT NULL on
-- measurements, so every row contributes; the sums are over the same
-- rows REGR_/STDDEV_POP fold above.
-- v1.21.0 — x is REBASED to REGRESSION_X_ORIGIN_DAYS so sum_xx stays
-- O(1e7) and never sheds precision squaring a ~20 540 epoch-day. Slope
-- / r² / sd are shift-invariant, so the live (raw-epoch) REGR_* probe
-- still parity-matches the cross-bucket compose.
SUM(${REBASED_X})::double precision AS sum_x,
SUM(${REBASED_X} * m."value")::double precision AS sum_xy,
SUM(${REBASED_X} * ${REBASED_X})::double precision AS sum_xx,
SUM(m."value" * m."value")::double precision AS sum_yy
FROM measurements m
WHERE m."user_id" = $1
@@ -451,18 +478,14 @@ async function runRollupAggregate(input: {
m."value",
EXTRACT(EPOCH FROM m."measured_at") / 86400.0
)::double precision AS r2,
-- v1.20.0 F6 — regression accumulators (epoch-day x-axis). See the
-- typed-list query above for the rationale; the read tier composes
-- windowed slope / r² / sd from these summed across DAY buckets.
SUM(EXTRACT(EPOCH FROM m."measured_at") / 86400.0)::double precision
AS sum_x,
SUM(
(EXTRACT(EPOCH FROM m."measured_at") / 86400.0) * m."value"
)::double precision AS sum_xy,
SUM(
(EXTRACT(EPOCH FROM m."measured_at") / 86400.0)
* (EXTRACT(EPOCH FROM m."measured_at") / 86400.0)
)::double precision AS sum_xx,
-- v1.20.0 F6 — regression accumulators. See the typed-list query above
-- for the rationale; the read tier composes windowed slope / r² / sd from
-- these summed across DAY buckets.
-- v1.21.0 — x is REBASED to REGRESSION_X_ORIGIN_DAYS (see above) so the
-- squared terms stay exact; slope / r² / sd are shift-invariant.
SUM(${REBASED_X})::double precision AS sum_x,
SUM(${REBASED_X} * m."value")::double precision AS sum_xy,
SUM(${REBASED_X} * ${REBASED_X})::double precision AS sum_xx,
SUM(m."value" * m."value")::double precision AS sum_yy
FROM measurements m
WHERE m."user_id" = $1