The per-bucket OLS accumulators stored x on the raw epoch-day axis (x ~ 20540),
so sum_xx accumulated past ~1e10 and the square shed ~10 of a double's
significant digits before the value was ever stored. The sub-day x detail was
lost at accumulation time, which no read-side identity can recover; an
ill-conditioned window's composed slope/r2 then drifted from the live REGR_*
probe past the parity gauge.
Rebase x to a fixed origin (epoch-days of 2020-01-01) at write time so the
squared terms stay O(1e7) and exact. Slope, r2, and population sd are invariant
under the affine x-shift, so the live raw-epoch probe still parity-matches the
cross-bucket compose; composeRegression needs no formula change. The per-bucket
slope/r2 columns stay on the raw axis (shift-invariant).
Migration 0193 recomputes sum_x/sum_xy/sum_xx from measurements on the rebased
basis, one set-based update per granularity. Recompute-from-source is idempotent
and rerun-safe; it touches only rows that already carry the accumulators and
leaves pre-migration NULLs for the boot re-fold.
Un-quarantine the DST-boundary parity test: it now matches live REGR_* within
the existing 1e-9 relative bound, with the WEIGHT case still bit-identical.
Reformulate composeRegression from the determinant form (n*Sxx - Sx^2)
to the mean-centered corrected-sum identities (Sxx = Sxx - Sx^2/n, ...).
Algebraically identical to the prior form and to Postgres REGR_* /
STDDEV_POP, but it subtracts the x mean at matched scale and avoids the
catastrophic cancellation the determinant form suffers on the ~1e10
epoch-day x-axis. Also read the exact stored sum_value accumulator for
the Sy term instead of reconstructing it as mean*count, dropping the
AVG-to-multiply float residual; fall back to mean*count when the column
is null.
Composition-only: no migration, no schema change. The WEIGHT live-parity
case stays bit-identical at the 1e-9 relative bound. The near-flat DST
parity seed improves by ~30x but still exceeds 1e-9, so it remains
quarantined with an updated note pointing at a window-local x-rescale as
the real fix.
The windowed slope / r2 / sd readers compose from the per-bucket
regression accumulators (migration 0190). Between a deploy and the
boot-backfill completing, a tenant's pre-0190 DAY buckets still carry
NULL accumulators, so composeWindowedRegression returns a full miss and
the slim / comprehensive readers emit a null slope. The null is the
honest answer until the backfill refills the accumulators, but the miss
was passing as a plain rollup read with no signal, against the no-silent-
cap rule.
Add hasPendingAccumulatorBackfill to the read tier and key a new
regression_source annotation off it (unavailable_pending_backfill vs
rollup) on both warm paths. No live-SQL fallback is added; the design
intentionally avoids the heavy scan.
Two suites pin that the windowed slope / r2 / population-sd composed from
the rollup accumulators equals the live Postgres REGR_SLOPE / REGR_R2 /
STDDEV_POP over the same raw rows.
compose-regression.test.ts (container-free) builds per-DAY-bucket
accumulators from a raw fixture exactly as the populator's SUM terms do,
then asserts composeRegression over those buckets equals a direct OLS over
the flat raw rows - the same closed form Postgres folds - on an uneven
multi-bucket fixture, plus the coverage contract (NULL accumulator,
< 2 readings, and zero-x-variance all return a full miss so the reader
falls back to live SQL).
rollup-regression-parity.integration.test.ts (real Postgres testcontainer)
seeds raw measurements across many uneven DAY buckets, folds the rollups,
and compares the accumulator compose to live REGR_*/STDDEV_POP to <= 1e-9:
a single-source WEIGHT series, a dual-source RHR series proving the
source-collapse runs BEFORE accumulator summation (composed == live WHOOP
-only, != the cross-source blend), and a DST-boundary series confirming
the UTC epoch-day x-axis parity-matches across the transition.
The residual is pure floating-point reassociation (per-bucket-then-window
vs flat summation), orders of magnitude below the read tier's rounding.
Migration 0190 left existing DAY rollup rows with NULL regression
accumulators until a re-fold refills them. Add a second discovery arm to
enqueueBootTimeRollupBackfill, UNION-ed with the per-day missing-coverage
arm: a user whose DAY buckets carry sum_xy IS NULL inside the fold window
needs a one-time re-fold so the windowed slope/r2/sd readers compose from
the accumulators instead of falling back to the live REGR_* scan.
New writes self-populate the accumulators (they ride the same aggregate
as mean/sum), so the arm drains to empty once the boot re-fold catches up
- the same convergence the v1.4.39 sum_value rollout reached through this
queue. Bounded to ROLLUP_FOLD_WINDOW_MS so days outside the window never
re-flag a user on every boot.
Windowed slope, r2 and population sd still ran a live REGR_*/STDDEV_POP
query over the raw measurements partition on every warm-rollup request,
because the per-bucket mean cannot reconstruct an OLS regression over the
raw points in a window. The per-bucket sd/slope/r2 columns are a
within-bucket fold and do not compose linearly across DAY buckets.
Add the four regression-accumulator sums Postgres folds for REGR_* /
STDDEV_POP - sum_x, sum_xy, sum_xx, sum_yy - per (user, type,
granularity, bucketStart, source) over the same canonical-source rows
the live query reads, with the x-axis in epoch days. With n = count and
sum_y = mean*count already stored, all six terms are additive across
buckets, so a 7/30/90-day slope/r2/sd composes from the summed
accumulators with a bit-identical result to the live closed form.
Migration 0190 adds the four nullable columns; nullable keeps it
additive so existing rows read NULL until the boot re-fold refills them
(mirrors the v1.4.39 sum_value rollout). The populator emits the four
SUM terms inside the existing aggregate (no extra round-trip) and
persistRollupRows writes them. composeRegression / composeWindowedRegression
evaluate the closed form from already-collapsed buckets and return a full
miss when any in-window bucket lacks accumulators, so the reader falls
back to live SQL rather than composing a partial window.
A multi-year "All" range read its series through the DAY-only daily
reader, which caps the result at BUCKET_CAP.daily (365) buckets. The
older history was silently dropped — the range collapsed to roughly the
most recent year instead of the whole span.
Add readTieredRollupSeries, which reads the bucket tier matching the
charts own display downsampler (WEEK for 1-2 years, MONTH beyond) and
returns whole-history coverage at that tier. readDailySeries now routes
any window wider than the DAY cap through it, falling finer on a coverage
miss and logging the miss rather than truncating silently. Short and
normal ranges never enter the branch, so the common path is unchanged.
The 0–14d raw band folded same-day readings on a UTC calendar key, so a
late-evening reading slipped onto the wrong day for non-UTC users and
across DST transitions. Key it on userDayKey(tz) instead and make the
timezone a required option so a caller cannot silently fall back to UTC.
The coarser DAY/WEEK/MONTH/YEAR bands stay UTC-anchored on purpose: the
rollup writer and the live aggregator both bucket on the UTC calendar
day and are parity-locked for the read-swap fallback. Document the split
and thread the resolved user timezone from the coach snapshot caller.
One-shot prettier pass so the new format:check gate starts from a clean
baseline. Formatting only — no behaviour change. Generator-owned and
vendored artifacts stay untouched via .prettierignore.
Add a coarseOnly option to buildTieredSeries that skips the raw 0–14d
read and the DAY (14–30d) / WEEK (30–90d) band reads, returning only the
MONTH / YEAR bands plus the anomaly envelope. The Coach snapshot's
coarse timeline tail consumes only month/year + anomalies, so the
near-term bands were read and discarded every turn; it now passes
coarseOnly. The briefing path is unchanged. Look the learn-guide prompt
example up by slug rather than by array index so reordering the catalog
cannot silently change the example link.
Wire the Coach snapshot's core-metric history onto the shared tiered
series so blood pressure, weight, and pulse carry a coarse tail the
in-window weekly fold could not produce — 90d-1y monthly and >1y yearly
buckets plus the preserved peak/trough envelope, so a spike from months
ago survives a coarse bucket. The tiered read skips the raw 0-14d band
the snapshot already holds, so no duplicate round-trip; the coarse tail
is bounded per band and sheds first under the prompt-budget degrader, so
token size holds.
Surface the active illness episodes and Rest Mode state to the Coach
prompt: the snapshot already carries the illness block, and the system
prompt now instructs the Coach to read an off-band vital as plausibly
illness-explained and avoid pushing a heavier cadence while the user is
in Rest Mode.
Add a static catalog of the public /learn guides and inject a compact
list into the system prompt. The Coach may reference at most one guide
by topic and URL when genuinely helpful, and only a slug the catalog
lists — a fabricated /learn URL is impossible by construction.
Introduce buildTieredSeries(): a five-band progressive data-context helper —
0-14d raw daily, 14-30d DAY, 30-90d WEEK, 90d-1y MONTH, >1y YEAR — plus a
per-metric min/max-envelope anomaly list so a peak from months ago survives a
coarse bucket. Reuses probeRollupCoverage / ensureUserRollupsFresh with the
existing replace-fall-back-on-miss contract; readers never throw, so a
coverage gap simply yields an empty band. Caps each metric near 600 tokens to
stay within the existing context budget. Shared across the Coach snapshot and
the daily briefing.
The snapshot's thick phase gated on every type the account had ever
logged having rollup coverage — one fresh reading from a new
integration nulled the health score and the blood-pressure extras
while the insights page showed a score for the same account. The gate
now checks only the three types the phase reads, per the established
per-type pattern. Underneath sat the destabiliser: the large rollup
fold deleted the window's buckets and re-inserted without a
transaction or duplicate tolerance, so a racing sync write aborted the
insert and left the delete committed — whole types lost their buckets
until the next clean fold, and accounts with data older than the
five-year fold window re-ran the full backfill on every boot because
discovery scanned unbounded history. Delete and insert now commit
atomically with duplicate tolerance, discovery shares the fold's
bound, the provisional ring says the score is being computed when the
inputs exist and reserves the no-data wording for accounts that have
none, and the greeting leads the band typographically.
A typed apiFetch wrapper owns the envelope unwrap, the ok-check and the
error read for every client call — two hundred call sites across the
app migrate onto it and a lint rule keeps raw api fetches out for good.
The legacy timezone helpers fold into the tz module and test fixtures
drop personal names.
A take whose slot the legacy resolver rejected fell back to a standalone
insert that kept the client's scheduledFor, parking a second live row on
an anchor the reminder projector had already minted — the partial unique
index permits that because it spans source. Both intake write routes now
probe the slot source-agnostically and converge onto the existing row
before inserting, and a standalone insert anchors on the take time per
the ad-hoc contract. The compliance rollup counts distinct slots instead
of rows, which also corrects historical duplicates on the next recompute,
and the slot dedup pass runs nightly instead of only at worker boot.
The medication compliance percentage and the history view drew their
per-dose attribution from four different matchers — the ±12h pairDoses
proximity used by calculateCompliance, the deriveDoseStatus bands (open
dose only), the 3h+120min classifyIntakeTiming heatmap heuristic, and a
raw takenAt-IS-NOT-NULL per-day rollup. They could and did disagree: a
dose could read "taken late" in the rate while the history called it an
off-schedule ad-hoc take.
Route the rate through the shared cadence-aware band minter +
reconstructDoseHistory so the percentage is a tally over the SAME ledger
the history view renders. The same intake can no longer be taken in the %
and ad-hoc in the ledger.
- calculateCompliance: when a medication context is supplied, the
count fields (taken / skipped / missed / rate) come from
tallyComplianceFromLedger. Numerator = taken_on_time + taken_late (a
late dose still counts as taken — the taking-vs-timing split is kept
available); denominator = taken + missed; skipped, ad-hoc, upcoming and
whole PRN groups are excluded; rate capped at 100. The streak still
walks the cadence timeline. Context-less pure-math callers keep the
legacy timeline tally byte-stable.
- complianceChips + missedDoses: adherence rate + missed count come from
the same ledger when an engine context is threaded.
- /api/medications/[id]/compliance heatmap: the per-day timing split and
missed marks are bucketed from the one ledger instead of the per-day
classifyIntakeTiming + expectedSlotCountForDay walk, so the cells agree
with the headline rate.
- medication_compliance_rollups stays a raw cross-medication daily
coverage count, documented as distinct from the band-attributed % (the
band minter is per-cadence and cannot run inside a single SQL tally).
Downstream readers (health score, insight compliance status) keep calling
calculateCompliance; their numbers shift to the honest band-attributed
values. Some currently-counted off-time takes (e.g. an 11:29 take of a
07:00 dose) now read ad-hoc with the slot missed rather than taken-late —
the configurable per-dose window is the lever to widen the on-time band.
Tests: the irregular twice-daily, single-time-late heatmap, and legacy
parity fixtures are updated with rationale for the intended number
changes; new tally tests pin the taking/timing split, the skip / ad-hoc /
upcoming / PRN exclusions, auto-missed=missed, the 100% cap, rolling
retrospective families, and the cross-consistency invariant.
The Prisma unique-constraint (P2002) narrowing predicate had been
hand-copied into five call sites. Fold them into a single isP2002 in
src/lib/prisma-errors.ts and import it everywhere (the two step-drain
sites keep their local isUniqueConstraintViolation name via an import
alias). Behaviour-identical; one place to chase a future P2002-semantics
change instead of five.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The synchronous DAY rollup hook carries no pg-boss singleton, so two
measurement writes landing on the same (user, type, day) can interleave
their delete-then-insert and the losing createMany trips the per-source
unique index. The peer rewrote the identical source-collapsed partition
— the aggregate is a pure function of the same underlying rows — so the
rollup is already correct. Catch the P2002 in the single-chunk
transactional path, annotate it, and return rather than failing the
recompute. Any other Prisma error still bubbles.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Drop the unused `now` field from the coach `ExtractOpts` type; fact
extraction never read it (conversation-summary keeps its own `now`).
- Correct the no-ladder tiebreak docstring in measurement-read: the
collapse picks the alphabetically smallest source name, matching the
live-SQL `ORDER BY … source` tiebreak, not max count.
- Replace the hand-inlined canonical-source subquery in the summaries
rollup-path narrows query with `canonicalMeasurementsFrom(rank, "90 days")`
for live/rollup parity with its siblings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- conversation-summary: fault-isolate the per-message decrypt in the
fold loop so one undecryptable historic turn (e.g. mid key-rotation)
no longer permanently stalls a conversation's summary refresh.
- source collapse: unify the no-ladder tiebreak on alphabetically
smallest source across the JS reader and the all-time rollup SQL, so a
ranked vital whose day carries only off-ladder sources resolves to the
same canonical source on the warm and the live-SQL read paths.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Complete the source-aware rollup read path: the slim summaries slice
(rollup + live), the comprehensive aggregator (both paths, including the
dailyByType correlation feed), and the /api/measurements route (rollup
read + the live date_trunc fallback) now resolve overlapping sources to
the source-priority ladder's canonical reading per (type, day) before
aggregating. A dual-source vital (e.g. WHOOP + Apple Watch resting heart
rate) surfaces one canonical series instead of a blend or a doubled
count; cumulative metrics keep summing per canonical source.
The collapse runs in SQL where the aggregate must stay a single
round-trip (buildSourceRankCase emits a per-user ladder-rank CASE;
canonicalMeasurementsFrom restricts a live read to canonical-source rows)
and in application code (collapseRollupRowsBySource) for the row readers,
both driven by the same resolved ladders so live and rollup agree.
Also widens the enum-literal SQL whitelist to accept digits so VO2_MAX
no longer throws on the typed write-hook recompute or the rank CASE.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
readRollupBuckets and the WEEK/MONTH/YEAR readers now select the source
column and run collapseRollupRowsBySource before composing, so every
rollup consumer (dashboard sparkline, fast-paths, derived metrics, range
deltas) sees one canonical row per bucket. loadUserSourcePriority loads
the ladder blob once; readBestGranularityRollups threads it through each
granularity probe so the per-floor walk never re-queries the user.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
collapseRollupRowsBySource resolves the per-source rollup rows minted by
the source-aware writer down to one row per bucket, using the user's
source-priority ladder (the same ordering the raw-row picker uses). A
metricKeyForType superset maps every overlapping non-cumulative vital
(RHR, HRV-SDNN, SpO2, body/skin temperature, respiratory rate, weight,
body fat, blood pressure, pulse, VO2max, recovery) plus the cumulative
keys onto a SourcePriorityMetricKey. Unlisted sources fall back to the
highest-count row so a bucket never doubles or goes dark.
Wired into the rollup readers in the following commits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rollup rows were grouped by (type, day) only, so two sources reporting
the same standard vital for one day (e.g. WHOOP + Apple Watch resting
heart rate) were AVG-blended into a single row instead of resolving to
the source-priority ladder's canonical reading.
Re-grain the rollup table to (type, day, source): the writer now groups
by source and delete-then-inserts each affected partition (so a source
that stops reporting a bucket cannot strand a stale row), and the read
path will collapse overlapping sources to the canonical reading in a
follow-up commit. Cumulative metrics (steps, energy) keep summing per
source. Migration 0115 purges the legacy source-blind aggregates; the
boot backfill + read-time self-heal re-mint per-source rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Six touch-disjoint waves landed on develop before this release:
- W1-KNIP — exports/types tiers flipped to enforcing-mode on main;
35 unused exports + 52 unused types triaged to 0/0; shadcn surface
+ zod-infer contract types scoped via knip.json ignore-block.
- W2-ZOD-MULTI-ISSUE — `returnAllZodIssues` shared helper +
`/api/dashboard/widgets` 422 now returns every Zod issue + worker
audit-ledger breadcrumb. 41 sibling routes documented for v1.4.43.
- W3-QUERYKEY-LONGTAIL — 40 settings/medications/admin/hooks files
migrated to the queryKeys factory; 9 new factory entries; ESLint
rule + test-guard scope extended in lockstep.
- W4-ARCH-HYGIENE — BERLIN_DAY_FORMATTER consolidated across nine
sites (seven status helpers + bucket-series + bp-in-target);
Suspense double-comment consolidated; doctor-report sanitiser regex
rewritten to escape-form so diffs are reviewable; pr-detection-
worker soft-delete filter; offhost-backup DR-intent comment.
- W5-IOS-WORKOUTS — `dedupeWorkoutBatch` write-time cross-source
dedup helper for the v1.5 iOS HealthKit observer ingest; dashboard
tile-strip placeholder parity with the live TrendCard chrome.
- W6-WITHINGS-OFF + ENV-CHECK — Withings off-response classifier
(transient/reauth/persistent) + WithingsApiError typed throw + the
`pnpm check-env` CLI. APNs `allOrNone: true` closes the AP-2
silent-disable gap the wave was conceived to close.
No migration, no env-var change, no API contract break for iOS v0.5.4.
The new `/api/dashboard/widgets` 422 envelope is additively extended.
pnpm test --run green at 4815 passing / 1 skipped.
pnpm typecheck, pnpm lint, pnpm knip (no --include flag) green.
Eight touch-disjoint waves landed on develop before this release:
- W-INSIGHTS-HOT + W-RECONCILE-INSIGHTS — extract the timeout-stub
persist helper and wire all seven *-status routes through it, closing
the recurring ~14 s warm response the iOS client sees on dashboard
open once the AI provider has stalled once that day.
- W-DELETED-2 — close the remaining three soft-delete reader-tier gaps
(export bundle, gamification achievements, doctor-report aggregator)
with a five-route integration-test pin.
- W-ORG — types consolidation into src/types/analytics.ts and
src/types/backups.ts; prompt directory unification.
- W-FRONTEND-FACTORY — queryKey factory expansion + ESLint rule lift.
- W-IOS-COORD — POST /api/auth/check-user four-branch onboarding
discovery, identifier matched verbatim, per-IP rate-limited.
- W-SIMPLIFIER — today-intake projection helper extract; unused tx?
params dropped; 13 knip-flagged dead exports trimmed.
- W-PERF-OPS — v1.4.39 W-SUM legacy-NULL UNION discovery arm retired.
- W-PROCESS-DOCS — pg.Pool sizing operator doc; ESLint rule
formalisation; knip CI gate staged for v1.4.42 flip.
No migration, no env-var change, no API contract break for iOS v0.5.4.
The new check-user endpoint is an additive surface. AP-2 .p8 key is
installed; SB-5 time-sensitive APNs is live as of this deploy.
pnpm test --run green at 4732 passing / 1 skipped.
pnpm typecheck, pnpm lint, pnpm knip green.
Eleven touch-disjoint waves landed on develop plus a six-axis QA
reconcile round before tagging. See CHANGELOG.md for the user-facing
prose; this commit message documents the wave + report inventory for
release-management traceability.
Waves shipped (each with a phase report under .planning/):
- phase-W-POOL-v1440-report.md — p-limit(4) wrapper around the
analytics 15-way fan-out + pg.Pool max raised 10 -> 20. Closes the
v1.4.39 empirical-trace finding #1 (Prisma pool starvation).
- phase-W-DELETED-v1440-report.md — 11 reader-tier helpers + 3
integration contracts close audit Critical Finding #3 (soft-delete
invisibility) across dashboard, analytics, summaries, rollup tiers,
and the Coach snapshot path.
- phase-W-INSIGHTS-v1440-report.md — mood-rollup swap on the three
/api/insights/{features,targets,cards} routes plus the
compliance-rollup hook gap closure for bulk-projection mints; the
post-QA reconcile patch closes the six remaining measurement.findMany
sites in the cluster that still leaked tombstoned rows.
- phase-W-INFRA-v1440-report.md — rollup helper umbrella consolidation
under src/lib/rollups/* with zero orphan re-exports; lint, typecheck,
knip, and 93/93 rollup tests green post-restructure.
- phase-W-RSC-v1440-report.md — per-tile Suspense boundaries plus the
queryKey factory-bypass walker test (extended to comparison/ in the
reconcile pass).
- phase-W-PRIVACY-v1440-report.md — bilingual /privacy page with 17
paired-section tests pinning the 9 SB-3 disclosure requirements.
- phase-W-AASA-v1440-report.md — /.well-known/apple-app-site-association
endpoint pinned to S8WDX4W5KX.dev.healthlog.app with app-ID parity
assertions guarding against split rotation.
- phase-W-APNS-NOTIFY-v1440-report.md — MEDICATION_REMINDER
time-sensitive APNs payload (priority 10 + interruption-level)
scoped via a parameterised test that pins all 6 other event-types
do not bypass Focus. AP-2 .p8 install gates real delivery.
- phase-W-CONSENT-v1440-report.md — AI consent receipts CRUD with
append-only invariant pinned in test; revoke-then-regrant leaves
both rows; idempotent DELETE returns 200 + receipt: null.
- phase-W-WMY-WIRE-v1440-report.md — three previously-dead WMY readers
(week/month/year) wired through summaries-slice and
health-score-fast-path with a linear-composability parity test
pinning the cross-consumer routing contract.
- phase-W-GHOSTS-v1440-report.md — ~1 177 lines of dead code removed
across 9 atomic commits; startOfUtcDay + wallClockInTz consolidation
eliminates a sibling-drift risk; knip whitelist trimmed to three
files pending follow-up.
QA reconcile patches applied per the v1.4.40 QA round (six reports
under .planning/round-v1440-QA-*-findings.md):
- senior C1 — dashboard-suspense-boundaries regex aligned with the
post-W-INFRA Thread 2 userTimezone-local lift.
- senior H1 — soft-delete filter added to the six remaining
insights measurement.findMany sites.
- specialist H1 — Promise.allSettled around bulk-projection
compliance recompute on both /api/medications/intake and
/api/dashboard/summary.
- ux H1 — compare-toggle migrated to queryKeys.dashboardWidgets
factory; walker test extended to src/components/comparison/.
- security M1 — consent artefact 64 KB cap enforced via
Buffer.byteLength (UTF-8 bytes, not UTF-16 code units).
Quality gates green at tag: pnpm typecheck, pnpm lint, pnpm knip
--include files,dependencies,binaries,unlisted, pnpm test --run
(4726 passing / 1 skipped). Tests 4525 -> 4726 unit + 1 skipped over
the v1.4.39 -> v1.4.40 span.
User-facing prose: see CHANGELOG.md.
