mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-09 04:50:59 +00:00
fix(rollups): rebase regression x-axis to a fixed origin for exact accumulators
The per-bucket OLS accumulators stored x on the raw epoch-day axis (x ~ 20540), so sum_xx accumulated past ~1e10 and the square shed ~10 of a double's significant digits before the value was ever stored. The sub-day x detail was lost at accumulation time, which no read-side identity can recover; an ill-conditioned window's composed slope/r2 then drifted from the live REGR_* probe past the parity gauge. Rebase x to a fixed origin (epoch-days of 2020-01-01) at write time so the squared terms stay O(1e7) and exact. Slope, r2, and population sd are invariant under the affine x-shift, so the live raw-epoch probe still parity-matches the cross-bucket compose; composeRegression needs no formula change. The per-bucket slope/r2 columns stay on the raw axis (shift-invariant). Migration 0193 recomputes sum_x/sum_xy/sum_xx from measurements on the rebased basis, one set-based update per granularity. Recompute-from-source is idempotent and rerun-safe; it touches only rows that already carry the accumulators and leaves pre-migration NULLs for the boot re-fold. Un-quarantine the DST-boundary parity test: it now matches live REGR_* within the existing 1e-9 relative bound, with the WEIGHT case still bit-identical.
This commit is contained in:
1 parent
8471384d01
commit
8265abedff
5 files changed
+260
-59
No files matched your search
@@ -53,6 +53,34 @@ export interface DailyMeanRow {
|
||||
maxValue: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* v1.21.0 — regression x-origin (epoch-days of 2020-01-01 UTC).
|
||||
*
|
||||
* The regression accumulators (`sum_x / sum_xy / sum_xx`) store x RELATIVE to
|
||||
* this fixed origin: `x = EXTRACT(EPOCH FROM measured_at)/86400.0 − ORIGIN`.
|
||||
* Raw epoch-days sit at x ≈ 20 540, so `x²` lands near 4.2e8 and `Σxx`
|
||||
* accumulates past ~1e10 — squaring shaves ~10 of double's ~15-16 significant
|
||||
* decimal digits BEFORE the value is ever stored, so the sub-day fractional x
|
||||
* (the time-of-day) is already lossy at write time. No read-side identity can
|
||||
* recover bits lost at accumulation. Rebasing to a recent origin keeps x in the
|
||||
* low thousands (x² ≤ ~1e7 for any realistic window), so the squared terms stay
|
||||
* comfortably exact.
|
||||
*
|
||||
* Slope / r² / population-sd are INVARIANT under an affine x-shift (subtracting
|
||||
* a constant from every x leaves Sxx, Sxy, Syy unchanged), so the composed
|
||||
* result is identical to the un-shifted basis — only the unreported intercept
|
||||
* would move. The writer SQL and migration 0193 share this exact literal; the
|
||||
* live REGR_* probe stays on raw epoch-days and matches because slope / r² / sd
|
||||
* do not depend on the origin.
|
||||
*
|
||||
* 2020-01-01 = 18 262 days after the Unix epoch. It MUST stay a stable
|
||||
* compile-time constant — changing it would orphan every stored accumulator
|
||||
* until a full re-fold. Pick a value comfortably before any plausible reading.
|
||||
*/
|
||||
export const REGRESSION_X_ORIGIN_DAYS = Math.floor(
|
||||
Date.UTC(2020, 0, 1) / 86_400_000,
|
||||
); // 18262
|
||||
|
||||
/**
|
||||
* v1.20.0 F6 — per-bucket OLS regression accumulators (epoch-day x-axis).
|
||||
* Mirrors the four columns migration 0190 added to `measurement_rollups`
|
||||
@@ -112,14 +140,18 @@ export interface ComposedRegression {
|
||||
*
|
||||
* This is algebraically identical to the determinant form (multiply numerator
|
||||
* and denominator by n) and to Postgres' REGR_* / STDDEV_POP, but numerically
|
||||
* stable: on the epoch-day x-axis (x ≈ 20 540) the determinant form computes
|
||||
* `n·Σxx − Σx²` as a difference of two ~1e10 magnitudes and sheds ~10
|
||||
* significant digits to catastrophic cancellation before the divide. The
|
||||
* centered form subtracts `Σx²/n` from `Σxx` at the same scale per term, so
|
||||
* the cancellation is bounded by the true x-variance, not the absolute x
|
||||
* magnitude. Σy is read from the exact stored `sumValue` accumulator when
|
||||
* present (falling back to `mean·count`), removing the AVG→multiply ULP
|
||||
* residual on the y side too.
|
||||
* stable: the centered form subtracts `Σx²/n` from `Σxx` at the same scale per
|
||||
* term, so the cancellation is bounded by the true x-variance, not the absolute
|
||||
* x magnitude.
|
||||
*
|
||||
* v1.21.0 — the accumulators are stored REBASED to `REGRESSION_X_ORIGIN_DAYS`
|
||||
* (x = epoch_days − origin), so `Σxx` stays O(1e7) instead of O(1e10) and the
|
||||
* squared terms never shed precision at write time. Slope / r² / sd are
|
||||
* invariant under the x-shift, so this composes the SAME regression the live
|
||||
* (un-rebased) REGR_* probe folds — the rebase only restores the bits the raw
|
||||
* epoch-day square would have lost. Σy is read from the exact stored `sumValue`
|
||||
* accumulator when present (falling back to `mean·count`), removing the
|
||||
* AVG→multiply ULP residual on the y side too.
|
||||
*
|
||||
* The caller MUST collapse overlapping sources to the canonical source
|
||||
* BEFORE handing rows here — the accumulators are per-source, so summing a
|
||||
|
||||
@@ -34,6 +34,7 @@ import { isP2002 } from "@/lib/prisma-errors";
|
||||
import {
|
||||
collapseRollupRowsBySource,
|
||||
loadUserSourcePriority,
|
||||
REGRESSION_X_ORIGIN_DAYS,
|
||||
} from "@/lib/rollups/measurement-read";
|
||||
import { startOfUtcDay } from "@/lib/tz/start-of-utc-day";
|
||||
import type {
|
||||
@@ -59,6 +60,30 @@ export const ROLLUP_RECOMPUTE_QUEUE = "rollup-recompute";
|
||||
/** Worker concurrency cap — recomputes are read-heavy on Postgres. */
|
||||
export const ROLLUP_RECOMPUTE_CONCURRENCY = 2;
|
||||
|
||||
/**
|
||||
* v1.21.0 — rebased epoch-day x for the regression accumulators.
|
||||
*
|
||||
* `sum_x / sum_xy / sum_xx` store x RELATIVE to `REGRESSION_X_ORIGIN_DAYS`
|
||||
* (epoch-days of 2020-01-01) so the squared terms stay O(1e7) instead of the
|
||||
* O(1e10) the raw epoch-day (x ≈ 20 540) basis produced — squaring a ~20 540 x
|
||||
* sheds ~10 of double's significant decimal digits BEFORE the sum is stored,
|
||||
* losing the sub-day x detail at write time. Slope / r² / sd are invariant under
|
||||
* the x-shift, so the live REGR_* probe (which stays on raw epoch-days) still
|
||||
* matches the cross-bucket compose. The `slope` / `r2` per-bucket columns also
|
||||
* stay on raw epoch-days (shift-invariant; their stored values are unchanged).
|
||||
*
|
||||
* `REGRESSION_X_ORIGIN_DAYS` is a trusted compile-time integer; assert it stays
|
||||
* one at SQL-build time before splicing it in (no user-input surface). Built
|
||||
* lazily inside `runRollupAggregate` rather than at module init so a partial
|
||||
* test mock of `measurement-read` can't trip the assertion before the SQL runs.
|
||||
*/
|
||||
function rebasedX(): string {
|
||||
if (!Number.isInteger(REGRESSION_X_ORIGIN_DAYS)) {
|
||||
throw new Error("REGRESSION_X_ORIGIN_DAYS must be an integer");
|
||||
}
|
||||
return `(EXTRACT(EPOCH FROM m."measured_at") / 86400.0 - ${REGRESSION_X_ORIGIN_DAYS})`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Payload `boss.send` carries onto the `rollup-recompute` queue. The
|
||||
* worker hands the values straight to `recomputeUserRollups`.
|
||||
@@ -368,6 +393,10 @@ async function runRollupAggregate(input: {
|
||||
}
|
||||
const dateTrunc = `date_trunc('${truncUnit}', m."measured_at")`;
|
||||
|
||||
// v1.21.0 — rebased epoch-day x for the regression accumulators (see the
|
||||
// `rebasedX` doc). Built per-call so the assertion fires with the real module.
|
||||
const REBASED_X = rebasedX();
|
||||
|
||||
if (input.types && input.types.length > 0) {
|
||||
// Build the type-list literal carefully. The enum values come
|
||||
// from `MeasurementType` (Prisma-generated TS enum); we whitelist
|
||||
@@ -401,20 +430,18 @@ async function runRollupAggregate(input: {
|
||||
m."value",
|
||||
EXTRACT(EPOCH FROM m."measured_at") / 86400.0
|
||||
)::double precision AS r2,
|
||||
-- v1.20.0 F6 — regression accumulators (epoch-day x-axis). Summed
|
||||
-- per (type, source, bucket) here so the read tier composes the
|
||||
-- windowed slope / r2 / sd across DAY buckets without a live scan.
|
||||
-- value is NOT NULL on measurements, so every row contributes; the
|
||||
-- sums are over the same rows REGR_/STDDEV_POP fold above.
|
||||
SUM(EXTRACT(EPOCH FROM m."measured_at") / 86400.0)::double precision
|
||||
AS sum_x,
|
||||
SUM(
|
||||
(EXTRACT(EPOCH FROM m."measured_at") / 86400.0) * m."value"
|
||||
)::double precision AS sum_xy,
|
||||
SUM(
|
||||
(EXTRACT(EPOCH FROM m."measured_at") / 86400.0)
|
||||
* (EXTRACT(EPOCH FROM m."measured_at") / 86400.0)
|
||||
)::double precision AS sum_xx,
|
||||
-- v1.20.0 F6 — regression accumulators. Summed per (type, source,
|
||||
-- bucket) here so the read tier composes the windowed slope / r2 / sd
|
||||
-- across DAY buckets without a live scan. value is NOT NULL on
|
||||
-- measurements, so every row contributes; the sums are over the same
|
||||
-- rows REGR_/STDDEV_POP fold above.
|
||||
-- v1.21.0 — x is REBASED to REGRESSION_X_ORIGIN_DAYS so sum_xx stays
|
||||
-- O(1e7) and never sheds precision squaring a ~20 540 epoch-day. Slope
|
||||
-- / r² / sd are shift-invariant, so the live (raw-epoch) REGR_* probe
|
||||
-- still parity-matches the cross-bucket compose.
|
||||
SUM(${REBASED_X})::double precision AS sum_x,
|
||||
SUM(${REBASED_X} * m."value")::double precision AS sum_xy,
|
||||
SUM(${REBASED_X} * ${REBASED_X})::double precision AS sum_xx,
|
||||
SUM(m."value" * m."value")::double precision AS sum_yy
|
||||
FROM measurements m
|
||||
WHERE m."user_id" = $1
|
||||
@@ -451,18 +478,14 @@ async function runRollupAggregate(input: {
|
||||
m."value",
|
||||
EXTRACT(EPOCH FROM m."measured_at") / 86400.0
|
||||
)::double precision AS r2,
|
||||
-- v1.20.0 F6 — regression accumulators (epoch-day x-axis). See the
|
||||
-- typed-list query above for the rationale; the read tier composes
|
||||
-- windowed slope / r² / sd from these summed across DAY buckets.
|
||||
SUM(EXTRACT(EPOCH FROM m."measured_at") / 86400.0)::double precision
|
||||
AS sum_x,
|
||||
SUM(
|
||||
(EXTRACT(EPOCH FROM m."measured_at") / 86400.0) * m."value"
|
||||
)::double precision AS sum_xy,
|
||||
SUM(
|
||||
(EXTRACT(EPOCH FROM m."measured_at") / 86400.0)
|
||||
* (EXTRACT(EPOCH FROM m."measured_at") / 86400.0)
|
||||
)::double precision AS sum_xx,
|
||||
-- v1.20.0 F6 — regression accumulators. See the typed-list query above
|
||||
-- for the rationale; the read tier composes windowed slope / r² / sd from
|
||||
-- these summed across DAY buckets.
|
||||
-- v1.21.0 — x is REBASED to REGRESSION_X_ORIGIN_DAYS (see above) so the
|
||||
-- squared terms stay exact; slope / r² / sd are shift-invariant.
|
||||
SUM(${REBASED_X})::double precision AS sum_x,
|
||||
SUM(${REBASED_X} * m."value")::double precision AS sum_xy,
|
||||
SUM(${REBASED_X} * ${REBASED_X})::double precision AS sum_xx,
|
||||
SUM(m."value" * m."value")::double precision AS sum_yy
|
||||
FROM measurements m
|
||||
WHERE m."user_id" = $1
|
||||
|
||||
Reference in new issue
Block a user