The nightly briefing warm completed every night and read "all-failed",
because one run event held every account's annotations and the last
account won. Each candidate now runs in its own wide event and the run
reports every outcome through `reportJobRun`, so ok, dead sign-in,
withheld and failed accounts show side by side.
A dead Codex refresh token threw a bare error with no status, so the chain
read it as a network failure: the health ledger recorded hard_failed, no
"reconnect" hint appeared, and every night tried the same token. The
refresh now throws `CodexRefreshError` with a 401 for a dead grant
(invalid_grant / invalid_refresh_token on a 400 included), as do a missing
or undecryptable stored credential. The ledger records auth_failed, the
chain marks the primary credential expired, and the nightly pass reports
the account as auth_failed without queuing a retry.
A briefing the outbound screen withheld (a causal claim) failed outright,
and the 45-minute provider retry then asked the same question eight times
a night. The generator now makes one bounded repair that names the broken
rule and keeps the repaired payload only if it passes the same parse,
grounding and screen; a withheld outcome queues no retry.
A provider that keeps failing is paused: from five consecutive hard
failures the skip window starts at 30 minutes and doubles per further
failure, capped at a day. A paused provider is walked last and tried again
when the window lifts; one success resumes it. The operator card shows how
many accounts it is paused for and until when. After a 5xx from a
non-canonical OpenAI-compatible endpoint the client asks the free model
listing, and a configured model the endpoint no longer offers is named in
the error, on the hop line and on the card.
Also:
- The tiered context's live day band read `measured_at AT TIME ZONE tz`,
which treats the stored UTC as local time; a far-from-UTC reader's
evening readings landed on the next day. It converts from UTC first now.
- Google Health intraday heart rate re-reads two hours per sync instead of
a day; daily summaries and sleep keep the day (#1023).
- The first filtered daily read that answers 400 and triggers the
snake_case retry is recorded as a note, not an error (#1023).
- The single derived-score read goes through the derived cache with
stale-while-revalidate, like the batch beside it.
Getting started names the APP_MEMORY_LIMIT default and how to lower it
on a 2 GB host; the scaling guide says that web and worker share one
pool in the bundled single-container setup. The rollup comment names
migration 0372, which carries the hourly-mean statistic.
A Coach snapshot, a derived-metric batch or the Insights page fans out
into a dozen readers that each probe the same account's rollup coverage
at once. In production one MCP get_metrics call held 59 of these probes
open on the database together, each taking about 320 ms under that
contention, and the same burst repeated every 15 minutes.
Concurrent probes for one account now join the query already running.
Nothing is kept after it settles, each caller gets its own map, and a
measurement write drops the in-flight entry so a probe that starts after
a write always reads afresh. The registry lives on globalThis so every
server bundle shares it.
probeRollupCoverage runs ahead of the dashboard snapshot, the analytics
slices and the comprehensive insights. It read every live row of the
account for SELECT DISTINCT type and counted every DAY bucket per type,
which the planner served with a sequential scan of measurement_rollups.
Walk the distinct types with a recursive CTE along the live index and
test coverage with EXISTS. On a copy of a long-lived account the probe
drops from 24-50 ms to under 2 ms and returns the same map; an
integration test keeps the old query as the oracle.
The doctor report's pulse average and day points, the correlation channels,
the illness correlation, the personal baseline and the Coach's tiered context
each folded pulse readings into a day by plain mean. They now take the day as
the mean of its local hours' means and a window as the mean of its days, in
the zone each reader buckets by. The day aggregates carry the weighted sums for
an hourly-mean type, so every caller reads one day value. Every other type is
unchanged.
Rows written before this change keep the reading mean: the boot backfill and
the freshness check fill only missing days, so nothing rewrites them. Migration
0373 recomputes the mean of every stored PULSE row from the readings, DAY as
the mean of its hours' means and WEEK / MONTH / YEAR as the mean of their days,
and leaves count, min, max and the sums alone. The writer stores the coarse
rows the same way, so a refold and the migration agree.
A watch samples pulse about every 5 to 10 minutes at rest and about every
5 seconds in a workout. The daily value was the plain mean over a day's
readings, so a workout hour of several hundred readings outweighed the other
23 hours: a day with one hard hour and 23 resting hours (60 bpm) read about
128 bpm, not 64.
A day of pulse is now the mean of its per-hour means, in the rollup DAY
bucket, the live per-day read the chart serves, and the series route's day
grain. Windows over several days weigh each day once instead of by its
reading count, so the bias does not return one level up. Reading counts,
min/max, and the regression sums still run over every reading. The chart
tooltip and the data table name the statistic, in all seven locales.
The statistic is limited to pulse; HRV, SpO2 and glucose keep the plain mean
(see src/lib/measurements/day-statistic.ts). Stored DAY rows are rewritten by
the next rollup recompute (admin recompute route), not on read.
Fixes#1107
`instant.toISOString().slice(0, 10)` and parsing a key back as UTC
midnight are the idioms behind every day-boundary fix this year: the
evening west of UTC and the small hours east of it land on the
neighbouring day. A new rule, healthlog/no-utc-day-key, refuses them in
src/ outside src/lib/tz and tests, and refuses a disable directive for
itself that is not next-line or gives no reason.
The time zone module now names every legitimate form: userDayKey for a
person's day, dateOnlyKey / dateOnlyAtNoonUtc / dayKeyAsUtcMidnight for
date-only values and day labels, daysBetweenDateKeys / weekdayOfDateKey /
isCalendarDateKey / statedDateKey for key arithmetic and parsing.
Of the 133 sites the rule found, 84 are gone. Most moved onto a named
helper with no change in behaviour; these changed behaviour:
- cycle calendar, insights, Coach snapshot and prediction refresh start
their lookbacks at the first instant of the user's day
- a model-transcribed date like "June 10, 2026" no longer lands a day
early in a process east of UTC (documents and lab scans)
- a custom report range covers exactly the picked days in the profile
zone instead of starting the evening before west of UTC
- date-of-birth pickers cap at the user's today, not the UTC one
- an allergy onset date is stored at noon UTC like other date-only values
- the report's cycle summary, the Coach's usual BP range and the intake
projector's rollup recompute use the user's day
- vaccination dates render as dates instead of the previous evening
49 remain, each with its reason on the line above: 39 UTC by design (15
file name stamps, rollup and object keys, the UTC-day score engines) and
10 marked baseline. A guard test requires a classified reason on every
directive and holds the baseline to a ceiling that only goes down.
The All range is read from the WEEK or MONTH rollup tier once a history is
long enough, from DAY rows for a short one, and from the live table on a
coverage miss. The chart folds every path into the average day of each
week or month, but a tier row carried a cumulative metric's whole total
and a level's reading-weighted mean. A month of steps therefore read about
thirty times higher for a long history than for a short one, under a
caption that said average.
A tier row now carries the average day of its bucket on every tier: a
cumulative metric's total over the days that hold a reading, a level's
mean of its daily means. The rollup fold counts the bucket's days and adds
up their means for it. Only the web chart's series reads (the chart
endpoint and the dashboard batch) use these rows.
A restore now leaves out measurement tombstones deleted longer ago than
the tombstone retention. The nightly purge removes exactly those rows,
the delta feed answers cursorExpired for every cursor that could still
be served one, and the consolidation repairs look only inside the same
horizon. Younger tombstones keep their readers (a lagging paired client,
the undo of a recent delete, the repairs, the duplicate guard on
re-uploaded samples) and are written as before. The backup file is
unchanged. On a seeded account of 1.89 million readings whose 1.68
million tombstones were past retention, the restore on a slowed database
took 5.4 min instead of 15.
The rollup fold after the commit rebuilt five years of buckets under the
60 s request statement limit and could be dropped on a slow disk, leaving
the charts empty until the next boot. `recomputeUserRollups` takes a
`statementTimeoutMs` that scopes each aggregate and persist transaction
with SET LOCAL, and the restore passes its after-commit allowance.
Fixture tombstones in two round-trip tests were dated in July 2026 and
had aged past the retention; they are now relative to the run.
The granularity router read only the coarsest tier the window cleared, and
a coarse bucket that starts before the window is left out. Data that sat
only in that leading partial bucket, such as three months at the start of
a 731-day window, read as no data at all. The router now steps one tier
finer until a tier has buckets, down to DAY.
Building the coarse buckets from the DAY tier read every day of the window
into the process: a five-year yearly read moved thousands of rows per type,
and the Coach's history bands read several such windows per request. The
per-day source pick and the fold now run in Postgres (DISTINCT ON the day,
ordered by the ladder rank and then the source name, the same order the
in-process pick uses), so a coarse read returns one row per bucket. The
long-range chart picks its tier from the span of the window's days and reads
that tier once. The Coach snapshot hands its already-loaded source order to
the history bands instead of each band reading it again.
Local day bounds had three implementations. The day-key one read the zone
offset at UTC midnight, which on a transition day in zones far from UTC
(Pacific/Auckland, Australia/Sydney) is the offset after the change, so the
day started an hour off. The instant one converged on local 00:00, which in
zones that skip midnight (America/Santiago, America/Havana) does not exist,
and returned 23:00 of the previous day.
startOfLocalDayKey and localDayWindow in lib/tz/local-day are now the only
implementation: the first instant that reads the day, tried under every
offset in force around it. A skipped wall clock resolves to the moment the
clocks jump, the day-keyed copies in the measurement drains, the compliance
rollups and the dashboard batch window are gone, and the noon and mid-hour
anchors use the same converge.
The rollup writer keeps one row per bucket and source. Collapsing the stored
WEEK / MONTH / YEAR rows picked one source for the whole bucket, so a month
of manual blood-pressure readings plus one synced reading reported the synced
day alone, and a step month that switched devices lost every day the losing
device carried. The live weekly and monthly chart fold made the same pick.
Every coarse bucket is now folded from the canonical DAY buckets: the source
ladder picks one source per day, then the week, month or year counts every
one of its days. Spread, slope and r² are composed from the summed day
accumulators. The live fold picks per local day before folding into weeks
and months. Ties between sources without a ladder rank break alphabetically
on the SQL paths too, matching the rollup reader.
The snapshot's second phase (blood pressure in target, the per-context
glucose tiles, the Health Score) waits until weight, both blood-pressure
halves and steps have DAY buckets. The fold writes buckets only inside
the five-year window, so a type whose every reading is older than that
never gets one and the phase stayed null for good. Since the Apple Health
import now folds only inside the window too, an account that imported
years-old blood pressure or steps and nothing since lost those tiles and
its score with no explanation.
An uncovered gate type now holds the phase back only when it has a
reading inside the window, which is the case the backfill will still
reach. The extra query runs on the miss path alone and asks one indexed
EXISTS per uncovered type.
The runtime accepts "+05:30" as a zone id, so isValidTimezone let it through,
but Postgres reads a bare offset in AT TIME ZONE with the POSIX sign (east
of UTC is negative), and the day-grain SQL aggregates would cut every day on
the wrong side of UTC.
isValidTimezone now accepts named zones only: whatever the runtime resolves
to a bare offset is refused, while named zones, their aliases, UTC and the
Etc/GMT family stay accepted. Every write that already validated through it
refuses offsets now, and the screener and mood linked-context request schemas
validate their zone too. A value stored before this change can still sit in
the database, so the SQL call sites that read the stored zone directly (the
dashboard summary, achievements, the doctor report, the tiered AI context)
fall back rather than splice it, as the resolver and the day aggregates
already did.
A single account syncing heart rate from a watch took the worker down
within half an hour. The status cards' graded series read the last 91
days of raw rows, bounded in time but not in rows, so a stream sampled
every minute or faster materialised six or seven figures of objects on
every refresh and the read ran into the statement timeout.
Add readDayAggregates, which folds one type into per-day sum, count, min
and max in Postgres (optionally split into age segments and filtered to
the metric's plausibility domain), and move the readers that folded raw
rows into days in memory onto it: the graded series (recent, weekly and
the cold-tier fallback, for every type), the vitals day-mean fallback
behind the personal band, the tiered context's recent days and the plan
review. Recent, weekly and monthly buckets are identical to the raw
fold; the cold-tier yearly slope now runs over day means.
On 91 days of per-minute pulse the graded series now allocates 1.6 MB
where it allocated 62 MB, while the raw read alone holds 22 MB.
A structural guard freezes the remaining untaken measurement reads in
the worker and insight directories: each must name a sparse literal
type or carry a listed reason.
Refs #1023
The three score engines' unit tests hand `persist*Score` a fake Prisma, but
`upsertScoreRow` routes the written row through `afterMeasurementMutation`,
which reads the `@/lib/db` singleton instead of the injected client. Both legs
of that tail therefore opened a socket to a Postgres the unit job does not run:
on CI the attempt failed with ECONNREFUSED a few hundred milliseconds later and
logged on the way out, after the test that started it had ended. A console write
that lands while the worker is closing its RPC channel fails the whole run with
`EnvironmentTeardownError: Closing rpc while "onUserConsoleLog" was pending` —
four of nine unit-suite failures in thirty days, every one of them with all
tests green, blamed on whichever score file the worker happened to be tearing
down. Run 31961548185 carries the full stack: recomputeBucketsForMeasurement <-
afterMeasurementMutation <- upsertScoreRow <- persistStressScore <-
stress-score.test.ts:105.
It stayed invisible on a maintainer's machine because a local Postgres answers
on 5432: the same call succeeded quietly, and the suite was writing rollup rows
for its fixture users into a real database.
The four files that drive `upsertScoreRow` stub the tail. That removes the
doomed I/O rather than the log line, and the call they were making by accident
is now asserted in score-row.test.ts instead: the v1.37.19 wiring is pinned by
an expectation rather than by a stack trace. Verified by mutation — dropping the
`afterMeasurementMutation` call from score-row.ts fails the new assertion.
The settle guard's sweep is widened while here. Its eight-line handler window
and five-line registration preamble were both measured against the sites that
existed when it was written and both cut through real code: the console write in
mood-rollups.ts sits nine lines into its handler, so the one genuinely
unregistered detached promise in the tree was never seen, and the registration in
fire-and-forget.ts sits below its handler, so a correct site read as a violation.
Registration is now looked for wherever it wraps the statement, by paren depth
rather than by line count, and mood-rollups.ts registers its WEEK/MONTH/YEAR
enqueue. Verified by mutation: unwrapping that call fails the guard.
Gate: the four files, twenty consecutive runs, 20/20 green (53 tests each).
Test time across the three engine files falls from 708 ms to 20 ms — the
round-trips that are no longer made.
The unit suite reported all 1937 files green and then failed the job with
`EnvironmentTeardownError: Closing rpc while "onUserConsoleLog" was
pending`, naming a different scoring test each time.
Vitest replaces globalThis.console with a sink that forwards every write
to the main thread as an onUserConsoleLog RPC call. When a worker
finishes a test file it awaits the calls in flight and then rejects
whatever was started after that point, so a console write from work that
outlived its test lands in the gap and fails the run. The blame goes to
the file being torn down rather than to the test that started the work,
which is why the named file never logged anything itself.
Three paths detach a promise and log on the handle: fireAndForget, the
stale-while-revalidate cache refresh, and the offline query persister.
They now register the handled handle in a set that both suite setups
drain in afterEach, so no test leaves work in flight. The registry is
inert outside test: nothing there awaits it, and holding a reference to
an unsettled promise would retain it for the life of a worker. No
production behaviour changes, so there is no changelog entry.
Evidence. A focused harness that starts failing background tasks and
returns hit the teardown error 10 runs out of 10 before the change and
0 out of 10 after. Instrumenting the real suite showed two detached
console writes per run escaping their test, both from the rollup tail;
after the change, none, and thirteen writes per run now complete inside
the test that started them.
The guard asserts the behaviour, not a count: a registered task must
keep settleBackgroundTasks from resolving ahead of it, and the set must
drain back to empty. It goes red on unregistered work, on a disabled
registry, on an unwrapped call site, on a sweep that matches nothing,
and on a setup that drops the hook.
The tier for a chart series keyed off the requested window width. The
All range always requests ~3650 days, so every account got MONTH
buckets regardless of how much history exists — a record spanning four
months came back as four monthly means, and the chart captioned them
from the real span as a weekly average. The reader now re-reads at the
tier the returned rows' span calls for; a coverage miss on the finer
tier keeps the coarser rows.
The ladder comment claimed to mirror the client's pickBucket
thresholds and misquoted them. The two ladders are deliberately not
identical; a test now pins the direction that keeps the caption honest
(server never coarser than the client captions) for every span up to
ten years.
The knip blind spot (ignoreExportsUsedInFile) left eight symbols carrying
an export nothing outside their file consumes — the modifier goes, the
symbols stay. The score series module goes whole: its only importer was
its own guard test, whose composite- and config-guarding suites survive
in place.
DocumentContentIndex.tokenizerVersion gains the consumer that was the
point of storing it: the content-index job now re-tokenises rows whose
version trails CONTENT_TOKENIZER_VERSION, from the decrypted index text
along the same path the rotation script uses, keyset-paginated and
fault-isolated per row — a future tokenizer bump migrates the index
instead of silently splitting it. ImportJob.uploadSha256 turned out to
already have its dedupe consumer (both kick-off routes short-circuit on
same-bytes non-failed jobs); its comment now names that instead of
pretending to be write-only. The remaining write-only columns carry
their honest annotations in the shape the audit sweep parses.
- The tiered AI context's band reader now falls back to a live grouped
aggregate when a rollup band is empty (the module doc promised
replace-with-fallback from day one; only the replace half existed), so
a coverage-miss or never-folded YEAR partition no longer hands the
model empty history. Far-from-UTC users get the day-grain band keyed
on their own calendar (the same ±3h guard the correlation readers
carry).
- The slim analytics slice splices measurements older than the 5-year
fold window into its all-time count/min/max/mean: the rollup tier
only covers the window, so "all time" silently truncated for long
histories while the live fallback told the truth. A type whose only
rows are pre-fold no longer reads as never logged.
A soft-deleted measurement never reached the rollup tier: the WHOOP
delete webhook tombstoned rows without recomputing the dirtied buckets,
and the freshness watermark probe filtered tombstones out, so the
deletion's updatedAt bump was invisible and the ghost aggregate read as
fresh forever.
Extract the post-write tail every measurement surface repeats into
afterMeasurementMutation (rollup recompute + status-insight
invalidation, both best-effort), call it from the WHOOP delete webhook
with the (type, measuredAt) pre-image collected before the updateMany,
and drop the deletedAt filter from the watermark probe so a tombstone
counts as staleness.
The persistent mood-rollup writer bucketed on the UTC truncation of
`mood_logged_at` and ignored the canonical per-row `MoodEntry.date`
label. Every coverage-miss fallback keys on `entry.date`, so the rollup
tier disagreed with the fallback and with the day the user logged the
mood: a late-evening or after-midnight local mood landed on the wrong
calendar day, and the label shifted the moment the boot backfill flipped
a user from the live fallback onto the rollup tier.
Key every bucket off the stored `date` label instead — DAY, WEEK, MONTH
and YEAR — with `bucket_start` set to that label's UTC-midnight instant.
The rollup is now byte-identical to the live `entry.date` fallback and
timezone-independent by construction: DST correctness stays where it
already lives, in `moodDateKey` at write time. No consumer changes, no
schema change; every rollup reader already formats `bucket_start` as a
UTC day label.
Migration 0269 empties `mood_entry_rollups` so the boot-time
full-backfill worker re-mints every user's buckets under the corrected
keying; the worker now deletes-then-refolds so it is self-cleaning
against any row left by an older instance. While the table is empty,
reads fall back to the live `date`-keyed walk, which was already
correct, so there is no window where a wrong day is shown.
persistRollupRows scoped its delete to the [min, max] bucket range of the rows
the aggregate RETURNED. That made the repair self-limiting in exactly the case
that needs it: a recompute could only overwrite a bucket the live aggregate
still produces, so a bucket whose source rows had all gone was never in the
delete range and kept its stale aggregate.
Three shapes of the same hole. An empty aggregate skipped the write entirely
(the caller short-circuited on rows.length === 0). A returned range narrower
than the requested window left the edge buckets untouched. A type that lost
every reading in the window dropped out of the returned rows, so the per-type
delete filter could never reach it.
The DAY write hook masked it — it deletes its own bucket by hand when the day
empties — but nothing covered the WEEK / MONTH / YEAR worker, which recomputes
one bucket span at a time: deleting the last reading of a type in a week
dropped the DAY bucket and left the coarser ones stale forever. The coverage
probe still reported hasBuckets = true, so the read-swap served them.
A recompute now prunes the window it was asked for and writes its rows in the
same transaction. The single-bucket write hook keeps its own scope, which is
correct there because it already deletes its bucket when the aggregate empties.
Still one tier: the rollup replaces the legacy read and falls back to live SQL
only on a coverage miss.
Each case fails on the pre-fix code: the rollup tests on an unaligned
recompute window, the batch tests on the older stats: snapshot winning,
and the resting test on a series that stops at the fold horizon.
recomputeUserRollups filtered measurements by a caller-supplied from/to
but grouped them by date_trunc, and persistRollupRows then deletes every
rollup row in the bucket range it is about to write. A from that landed
mid-bucket therefore replaced a complete bucket with an aggregate
computed only from the readings after that instant.
Every caller passes a moving instant — now minus ninety days on the
read-path refresh, an arbitrary instant on the coverage fallback, now
minus five years on the boot backfill and the restore path — so a
different oldest bucket was corrupted on each invocation and nothing
repaired it. Snap both edges out to whole buckets via bucketSpan, whose
own contract note already required exactly this of every recompute.
Two layers. The integration suite runs a real pg-boss against a real
Postgres and proves the semantics the policies rest on: that a bare
singletonKey under the default policy de-duplicates nothing, that
exclusive suppresses a duplicate while the first job is queued AND while
it is active, that short suppresses only while queued and admits a send
once the job is active, and that a distinct key always enqueues.
It also pins the two facts that make the fix reach a running instance:
createQueue does not change the policy of a queue that already exists,
and updateQueue refuses a policy change, so the reconcile is required.
The unit guard couples each enqueue site to its registrar entry. The key
and the policy live in different files with nothing in the type system
holding them together, so deleting an entry would leave the call site
looking correct while it silently stopped working.
Refine the go-forward aggregated HR wire contract (iOS #34) from the never-
adopted hourly bucket to a 10-minute UTC bucket, so one contract serves both
the measurements-list de-explosion and the intraday-pulse 10-min curve.
- Batch route accepts stats:HKQuantityTypeIdentifierHeartRate:<10-min ISO-Z>
with avg + min/max, overwrite semantics, malformed-suffix rejection; the
validator is identifier-generic (allowlist) so respiratoryRate/SpO2 can join
later without a grammar change.
- The intraday reader classifies a day's PULSE rows three ways by externalId
shape (raw / uploaded 10-min bucket / server-folded hourly), builds the
10-min curve directly from uploaded buckets with the min/max envelope and
tension detection enabled, and overlays buckets over raw on the one
cutover-straddling day.
- DAY rollup min/max read COALESCE(value_min, value) so a bucket-native day's
band reflects the true intra-bucket extremes.
No schema change (value_min/value_max exist since v1.19.2). Reconciliation is
go-forward cutover; the nightly retention fold handles the raw backlog.
Rollup writes were delete-then-createMany inside a transaction; under READ
COMMITTED a concurrent recompute racing the import's full fold on the same
(user, type, granularity, bucket, source) partition tripped the composite
PK — and the P2002 swallow dropped the loser's freshly computed aggregate,
leaving a stale bucket. persistRollupRows now writes chunked
INSERT … ON CONFLICT DO UPDATE with deterministic (type, source, bucket)
lock ordering inside one transaction: a race can neither fail the import
nor lose an aggregate, and previously-staled buckets self-heal on their
next recompute. The vanished-source delete and in-transaction guarantees
are preserved; the now-unreachable P2002 swallow is removed.
The masked ENOENT is gone with it: long imports outlived the queue's
15-minute expiry, pg-boss redelivered the still-running job, and the
redelivery re-opened the consumed staging file — overwriting the real
terminal state with a raw ENOENT. Import sends now carry retryLimit 0 +
6h expiry (a retry after the staging file is consumed can only mask),
deliveries for terminal ImportJob rows are no-ops, a genuinely missing
staging file reports an honest re-upload message, and the extracted XML
is cleaned up on failure too.
Long-term retention for the dense intra-day tier (PULSE, HRV, SpO2) folds
to per-local-hour means instead of a single daily mean: the last 14 days
keep every raw sample, older days keep up to 24 points per metric — the
day's shape survives permanently at ~9k rows/metric/year. Hourly rows key
as stats:<HK>:<YYYY-MM-DD>T<HH> anchored at local HH:30, adopt-in-place
against both unique indexes with the P2002 retry outside the transaction.
The pre-fold DAY-rollup recompute extends to HRV (mean fidelity, not only
min/max) and the post-fold recompute is gone for the whole tier. Derived
resting rows for proxy users still mint from the day's raw samples.
A one-shot rebuild reconstructs the hourly shape for every already-folded
day from the soft-deleted raw samples and retires the daily row in the
same transaction — at no instant are both grains live, so no
AVG-over-live-rows reader can double-count. Boot discovery pairs live
daily-grain rows with surviving tombstones (parameter-bound SQL, pinned
against real Postgres) and converges to zero once converted; days whose
tombstones were pruned keep their daily mean. PULSE/SpO2 DAY buckets keep
the true pre-fold stats; HRV DAY buckets recompute from the hourly rows
(strictly better than the collapsed fold-time bucket); WEEK/MONTH/YEAR
respans enqueue whole-bucket recomputes.
The week/month/year rollup read anchored on now and always returned the
trailing window, so a historic from/to range (API and iOS callers) came
back with buckets outside the requested span. Thread the resolved from/to
bounds through so the read is bounded on both ends, matching the day-tier
and live-SQL fallback contract; tier selection is unchanged.
Cap and page the unbounded reads that let a multi-year tenant pull its
whole measurement history into memory on hot paths.
- extractFeatures: read newest-first with a 6000-row cap (mirrors the
Coach snapshot) and reverse to ascending, so a 365-400 day window no
longer materialises hundreds of thousands of PULSE / glucose rows per
Coach turn and nightly briefing. Full-history extremes still come from
readAllTimeExtremes, so the cap never relabels an extreme.
- Add a (userId, measuredAt) index on measurements (migration 0210) for
the type-less range scans that extractFeatures and the consolidation
passes drive; the existing (userId, type, measuredAt) cannot serve them
as a prefix.
- runConsolidation: keyset-paginate the per-(user, type) source scan on
(measuredAt, id) instead of one unbounded findMany, removing the giant
single result-set allocation that drove the boot storm. Output is
identical; covered by a paging unit test.
- Stagger the boot-time backfill enqueues with increasing startAfter
offsets so the heavy full-history loads no longer all drain in parallel
onto one tenant at startup. Non-boot callers keep their immediate send.
- Backups: replace load-all-then-stringify with paged narrow-select reads
for the weekly and off-host writers, bounding transient heap. Off-host
still includes soft-deleted rows. Covered by a serialize/restore
round-trip test.
- Drop the per-request auth.bearer.success audit insert; the wide event
already records auth_method and user_id, so the polling client no longer
drives a write + pool checkout on every request. Failure-path audit
stays.
- Coach conversation detail: cap the message include to the newest 200 so
a long thread no longer decrypts every message on each open; response
shape unchanged.
- Admin audit-log: floor the scan at the retention horizon so the listing
and its count stop sweeping open-ended over the substring filters.
The per-bucket OLS accumulators stored x on the raw epoch-day axis (x ~ 20540),
so sum_xx accumulated past ~1e10 and the square shed ~10 of a double's
significant digits before the value was ever stored. The sub-day x detail was
lost at accumulation time, which no read-side identity can recover; an
ill-conditioned window's composed slope/r2 then drifted from the live REGR_*
probe past the parity gauge.
Rebase x to a fixed origin (epoch-days of 2020-01-01) at write time so the
squared terms stay O(1e7) and exact. Slope, r2, and population sd are invariant
under the affine x-shift, so the live raw-epoch probe still parity-matches the
cross-bucket compose; composeRegression needs no formula change. The per-bucket
slope/r2 columns stay on the raw axis (shift-invariant).
Migration 0193 recomputes sum_x/sum_xy/sum_xx from measurements on the rebased
basis, one set-based update per granularity. Recompute-from-source is idempotent
and rerun-safe; it touches only rows that already carry the accumulators and
leaves pre-migration NULLs for the boot re-fold.
Un-quarantine the DST-boundary parity test: it now matches live REGR_* within
the existing 1e-9 relative bound, with the WEIGHT case still bit-identical.
Reformulate composeRegression from the determinant form (n*Sxx - Sx^2)
to the mean-centered corrected-sum identities (Sxx = Sxx - Sx^2/n, ...).
Algebraically identical to the prior form and to Postgres REGR_* /
STDDEV_POP, but it subtracts the x mean at matched scale and avoids the
catastrophic cancellation the determinant form suffers on the ~1e10
epoch-day x-axis. Also read the exact stored sum_value accumulator for
the Sy term instead of reconstructing it as mean*count, dropping the
AVG-to-multiply float residual; fall back to mean*count when the column
is null.
Composition-only: no migration, no schema change. The WEIGHT live-parity
case stays bit-identical at the 1e-9 relative bound. The near-flat DST
parity seed improves by ~30x but still exceeds 1e-9, so it remains
quarantined with an updated note pointing at a window-local x-rescale as
the real fix.
The windowed slope / r2 / sd readers compose from the per-bucket
regression accumulators (migration 0190). Between a deploy and the
boot-backfill completing, a tenant's pre-0190 DAY buckets still carry
NULL accumulators, so composeWindowedRegression returns a full miss and
the slim / comprehensive readers emit a null slope. The null is the
honest answer until the backfill refills the accumulators, but the miss
was passing as a plain rollup read with no signal, against the no-silent-
cap rule.
Add hasPendingAccumulatorBackfill to the read tier and key a new
regression_source annotation off it (unavailable_pending_backfill vs
rollup) on both warm paths. No live-SQL fallback is added; the design
intentionally avoids the heavy scan.
Two suites pin that the windowed slope / r2 / population-sd composed from
the rollup accumulators equals the live Postgres REGR_SLOPE / REGR_R2 /
STDDEV_POP over the same raw rows.
compose-regression.test.ts (container-free) builds per-DAY-bucket
accumulators from a raw fixture exactly as the populator's SUM terms do,
then asserts composeRegression over those buckets equals a direct OLS over
the flat raw rows - the same closed form Postgres folds - on an uneven
multi-bucket fixture, plus the coverage contract (NULL accumulator,
< 2 readings, and zero-x-variance all return a full miss so the reader
falls back to live SQL).
rollup-regression-parity.integration.test.ts (real Postgres testcontainer)
seeds raw measurements across many uneven DAY buckets, folds the rollups,
and compares the accumulator compose to live REGR_*/STDDEV_POP to <= 1e-9:
a single-source WEIGHT series, a dual-source RHR series proving the
source-collapse runs BEFORE accumulator summation (composed == live WHOOP
-only, != the cross-source blend), and a DST-boundary series confirming
the UTC epoch-day x-axis parity-matches across the transition.
The residual is pure floating-point reassociation (per-bucket-then-window
vs flat summation), orders of magnitude below the read tier's rounding.
Migration 0190 left existing DAY rollup rows with NULL regression
accumulators until a re-fold refills them. Add a second discovery arm to
enqueueBootTimeRollupBackfill, UNION-ed with the per-day missing-coverage
arm: a user whose DAY buckets carry sum_xy IS NULL inside the fold window
needs a one-time re-fold so the windowed slope/r2/sd readers compose from
the accumulators instead of falling back to the live REGR_* scan.
New writes self-populate the accumulators (they ride the same aggregate
as mean/sum), so the arm drains to empty once the boot re-fold catches up
- the same convergence the v1.4.39 sum_value rollout reached through this
queue. Bounded to ROLLUP_FOLD_WINDOW_MS so days outside the window never
re-flag a user on every boot.
Windowed slope, r2 and population sd still ran a live REGR_*/STDDEV_POP
query over the raw measurements partition on every warm-rollup request,
because the per-bucket mean cannot reconstruct an OLS regression over the
raw points in a window. The per-bucket sd/slope/r2 columns are a
within-bucket fold and do not compose linearly across DAY buckets.
Add the four regression-accumulator sums Postgres folds for REGR_* /
STDDEV_POP - sum_x, sum_xy, sum_xx, sum_yy - per (user, type,
granularity, bucketStart, source) over the same canonical-source rows
the live query reads, with the x-axis in epoch days. With n = count and
sum_y = mean*count already stored, all six terms are additive across
buckets, so a 7/30/90-day slope/r2/sd composes from the summed
accumulators with a bit-identical result to the live closed form.
Migration 0190 adds the four nullable columns; nullable keeps it
additive so existing rows read NULL until the boot re-fold refills them
(mirrors the v1.4.39 sum_value rollout). The populator emits the four
SUM terms inside the existing aggregate (no extra round-trip) and
persistRollupRows writes them. composeRegression / composeWindowedRegression
evaluate the closed form from already-collapsed buckets and return a full
miss when any in-window bucket lacks accumulators, so the reader falls
back to live SQL rather than composing a partial window.
A multi-year "All" range read its series through the DAY-only daily
reader, which caps the result at BUCKET_CAP.daily (365) buckets. The
older history was silently dropped — the range collapsed to roughly the
most recent year instead of the whole span.
Add readTieredRollupSeries, which reads the bucket tier matching the
charts own display downsampler (WEEK for 1-2 years, MONTH beyond) and
returns whole-history coverage at that tier. readDailySeries now routes
any window wider than the DAY cap through it, falling finer on a coverage
miss and logging the miss rather than truncating silently. Short and
normal ranges never enter the branch, so the common path is unchanged.
The 0–14d raw band folded same-day readings on a UTC calendar key, so a
late-evening reading slipped onto the wrong day for non-UTC users and
across DST transitions. Key it on userDayKey(tz) instead and make the
timezone a required option so a caller cannot silently fall back to UTC.
The coarser DAY/WEEK/MONTH/YEAR bands stay UTC-anchored on purpose: the
rollup writer and the live aggregator both bucket on the UTC calendar
day and are parity-locked for the read-swap fallback. Document the split
and thread the resolved user timezone from the coach snapshot caller.
One-shot prettier pass so the new format:check gate starts from a clean
baseline. Formatting only — no behaviour change. Generator-owned and
vendored artifacts stay untouched via .prettierignore.
Add a coarseOnly option to buildTieredSeries that skips the raw 0–14d
read and the DAY (14–30d) / WEEK (30–90d) band reads, returning only the
MONTH / YEAR bands plus the anomaly envelope. The Coach snapshot's
coarse timeline tail consumes only month/year + anomalies, so the
near-term bands were read and discarded every turn; it now passes
coarseOnly. The briefing path is unchanged. Look the learn-guide prompt
example up by slug rather than by array index so reordering the catalog
cannot silently change the example link.
Wire the Coach snapshot's core-metric history onto the shared tiered
series so blood pressure, weight, and pulse carry a coarse tail the
in-window weekly fold could not produce — 90d-1y monthly and >1y yearly
buckets plus the preserved peak/trough envelope, so a spike from months
ago survives a coarse bucket. The tiered read skips the raw 0-14d band
the snapshot already holds, so no duplicate round-trip; the coarse tail
is bounded per band and sheds first under the prompt-budget degrader, so
token size holds.
Surface the active illness episodes and Rest Mode state to the Coach
prompt: the snapshot already carries the illness block, and the system
prompt now instructs the Coach to read an off-band vital as plausibly
illness-explained and avoid pushing a heavier cadence while the user is
in Rest Mode.
Add a static catalog of the public /learn guides and inject a compact
list into the system prompt. The Coach may reference at most one guide
by topic and URL when genuinely helpful, and only a slug the catalog
lists — a fabricated /learn URL is impossible by construction.
Introduce buildTieredSeries(): a five-band progressive data-context helper —
0-14d raw daily, 14-30d DAY, 30-90d WEEK, 90d-1y MONTH, >1y YEAR — plus a
per-metric min/max-envelope anomaly list so a peak from months ago survives a
coarse bucket. Reuses probeRollupCoverage / ensureUserRollupsFresh with the
existing replace-fall-back-on-miss contract; readers never throw, so a
coverage gap simply yields an empty band. Caps each metric near 600 tokens to
stay within the existing context budget. Shared across the Coach snapshot and
the daily briefing.
Marc-André Bombeck