Every ingest path in the product can now emit a typed, salience-classified
event into pg-boss, consumed by one worker that claims a durable per-day
reaction marker and fans out to the reaction surfaces.
The recency classifier runs at the seam, before anything is enqueued, so a
mass backfill — a multi-year Apple Health export, a provider re-sync, a
30-day catch-up — emits nothing at all. Non-salient writes are annotated and
dropped.
The worker makes no provider call by construction: it imports no provider
module, writes only markers, and enqueues nothing that is not budget-gated
downstream. Business refusals return a status rather than throwing, so a
refusal is never retried against a ceiling that will not move until the local
day rolls over.
Seams: sleep (one hook, inherited by all seven transports), workouts (the
batch route, the four provider syncs, the export importer), weight and blood
pressure (both measurement routes), labs (manual, OCR commit, document fact
commit).
The provider workout syncs could not previously tell a created row from an
updated one, so a re-sync of a recent session would have re-fired on every
poll; they now recover the distinction from the row the upsert already
returns.
Reaction lines are AES-256-GCM at rest, following the InsightNarrative
precedent, and the column is registered for key rotation. Markers are pruned
after 14 days by a daily maintenance job.
The webhook-driven per-resource sync advanced the resource cursor but never
touched IntegrationStatus, so lastSuccessAt stayed frozen at the last full
poll — the Settings 'last synced' pill and the integration-status read
reported stale for up to an hour after a webhook had already landed last
night's data. Stamp recordSyncSuccess per resource, reusing the full sync's
exact success guard (a genuine import clears the error state; a 403 soft-skip
leaves the looks-healthy window closed).
WHOOP was the only workout integration that never normalised its sport,
unlike the Fitbit and Google Health mappers — it wrote WHOOP's raw
sport_name or a literal whoop_sport_<id> placeholder straight into
Workout.sportType. Confirmed in prod: a user's WHOOP-tracked cycling
workouts never rendered as cycling on /insights/workouts because nothing
in the UI could map the non-canonical string to an icon or label.
sport-map.ts adds a sport_id/sport_name table sourced from WHOOP's public
Workout-object reference and mirrors the Fitbit/Google mapper shape:
sport_id is preferred (retired past 2025-09-01 but still exact when
present), sport_name is normalised before the fallback lookup, and
anything unrecognised resolves to "other" — never a placeholder. The raw
WHOOP fields stay in Workout.metadata for traceability.
Adds a shared table-driven test that checks Fitbit, Google Health, and
WHOOP sport mappers together so the three integrations can't silently
drift out of parity on common sports.
The nightly insight pre-pass runs ~04:30, before last night's sleep has
synced, so the morning digest and health score reference the day without
the current sleep. Close the gap the event-driven way instead of an
on-mount generation.
A day's digest is now provisional until the sleep-dependent output has
been regenerated with last night's sleep, and final after. The lifecycle
is driven by a real marker (User.morningDigestRefreshedOn, the user-local
date the refresh completed), not a heuristic: it flips the instant the
refresh runs, ahead of the snapshot cache that feeds the sleep last-seen
fallback. A day with no sleep device stays honestly provisional and keeps
the sleep-pending note rather than showing a stale final score.
When a completed sleep segment for last night lands from any of the three
sleep transports — Withings sync-sleep, the WHOOP sync, the Apple-Health
measurement batch — a debounced morning-digest-refresh job is enqueued.
"Last night" is judged in the user's profile timezone, so a full-history
re-sync never re-triggers for an old night. The queue singletonKey plus
the marker collapse the many samples of one night, across all sources,
into one refresh per user per local morning; the job re-runs the same
comprehensive pipeline the nightly cron uses (forced, so the content-hash
gate turns a no-change regen into a cheap no-op) and stamps the marker. A
hard generation failure leaves the day provisional and hands recovery to
the existing 45-minute briefing-retry machinery.
Platform-wide hardening: the failure classes found live this week hunted down
across every integration.
Fitbit carried the same tombstone wedge fixed for Google — probe now matches
soft-deleted rows and the update resurrects them; the comment describing a
partial unique index that never existed is corrected. Sleep segment ids are
stable across re-scoring for Withings (session+segment-start), WHOOP/Polar
(one segment per stage, no positional index) and Oura (run-start keyed), each
with a session-scoped sweep that soft-deletes rows a re-score orphaned — so
legacy and affected nights heal as they are re-read.
One platform rule for deleted rows: source-owned readings resurrect on
re-import (Withings, WHOOP, Oura, Polar, Nightscout, mood webhook, CSV
re-imports, Apple stats day-totals); Apple sample-grain deletions stick per
the paired-client contract, and the mood bulk route's tombstone match is now
a true no-op reported as duplicate. Import counts reconcile against actual
createMany counts.
Google lifecycle: a dead token registers on the cycle's hard-fail ledger, so
an expired connection parks instead of stamping success hourly; backfill and
sleep-repair markers stamp only after a clean pass (pg-boss retries are now
real); failed measurement writes hold the watermark; no-op updates are
skipped; mapping.md tells the current truth.
Scale: doctor-report reads only consumed columns and enabled sections;
exports and backups keyset-paginate; achievements tally via SQL buckets;
series charts day-bucket dense kinds beyond 90 days; BP pairing is a linear
two-pointer; distinct-type scans run server-side; the graded-series cold
fallback is bounded; the fitbit defer tracker loops instead of spreading.
Fitbit, WHOOP, and Oura all issue one-time-use refresh tokens. The token
persistence did an unguarded read → refresh → write, so two overlapping
syncs for one user both read the same stored refresh token and both POST
it. The provider honours only the first; the loser's refresh comes back 401,
gets classified as reauth, and the connection is parked at `error_reauth`
spuriously even though the peer rotated the token successfully.
Add a shared `persistRotatedToken` helper that writes the rotated pair with
a compare-and-swap on the stored ciphertext (matched verbatim, since AES-GCM
re-encryption of the same plaintext differs byte-for-byte). When the
conditional update touches zero rows a peer already won the race; re-read the
row and reuse the peer's freshly rotated access token instead of erroring.
Wire it into all three providers — Fitbit and WHOOP `getValidToken`
(proactive expiry refresh, keyed by connection id) and Oura `storeOuraTokens`
(reactive 401 refresh, keyed by user id). Both racing callers now end with a
usable token and neither drops the connection.
One-shot prettier pass so the new format:check gate starts from a clean
baseline. Formatting only — no behaviour change. Generator-owned and
vendored artifacts stay untouched via .prettierignore.
Polar exposes only per-stage duration totals, so its mapper already
reconstructs a contiguous CORE-DEEP-REM order exactly like WHOOP and flags
each segment reconstructed. The night reader's honesty allow-list only held
WHOOP, so a Polar-won night was served with reconstructed:false — presenting
a synthetic stage order to web and iOS as if it were the device's measured
timing. Add POLAR to the allow-list so the layout is labelled approximate.
Polar's total_interruption_duration carried the night's awake time but was
dropped, so every Polar night read as fully consolidated (awakeMinutes:null,
awakenings:0) with overstated asleep-vs-in-bed efficiency. Lay it as a
leading AWAKE segment so the asleep stages partition the remaining window and
the reader surfaces real awake time.
WHOOP and Polar independently implemented the same contiguous-timeline walk.
Extract a shared reconstructContiguousSleepTimeline helper under src/lib/sleep
and call it from both; each mapper keeps only its field normalisation. Oura's
measured 5-min hypnogram is a different algorithm and is left untouched.
WHOOP's v2 API returns only per-stage sleep duration totals — no onset
timestamps, no hypnogram endpoint. The mapper stamped all five stage
totals on the one sleep-END instant, so the hypnogram reconstructed
every stage as a span touching the night's right edge: bars stacked
right-aligned with no clock times whenever WHOOP was the sole source.
Lay the asleep stages back-to-back from sleep onset in a fixed
physiological order (AWAKE lead, then CORE → DEEP → REM), emitting one
timed row per segment with measuredAt at that segment's end and a
distinct indexed externalId. IN_BED stays a single envelope row over the
whole window. The existing reader and renderer are untouched — they now
see distinct per-segment instants and place each block at its own time.
The order is synthesised, not measured, so every reconstructed row is
flagged and the night DTO carries a per-session `reconstructed` boolean
(GET /api/sleep/night, regenerated OpenAPI). The client renders the
hypnogram but labels it an approximate layout and never recomputes;
real-series sources (Apple Health, Withings, Fitbit) stay false.
The four WHOOP resources are independent pg-boss queues that run concurrently
for the same user, so the read-modify-write that merged one cursor key into the
map and wrote it back could let a sibling's write clobber a just-advanced
cursor. Replace it with a single jsonb_set UPDATE that merges in place under the
row lock, and GREATEST so the shared last-synced instant only moves forward.
The resource key is parameter-bound and asserted against the closed resource set
before it reaches the JSON path.
Replace the single shared last-synced cursor with a per-resource cursor map
on the WHOOP connection. Recovery, sleep, workout, and cycle each advance only
their own cursor on a successful fetch+upsert, so a slow or 403-ing collection
never drags the incremental window forward for its siblings and its own
late-arriving records stay inside the overlap. Legacy connections fall back to
the shared last-synced instant, so no historical state is lost.
Add single-record fetch-by-id client routes and per-resource by-id syncs. The
update webhook now carries the resource id through the queue payload, so the
worker resolves and upserts that one record immediately rather than re-walking
the whole collection on the next overlap tick.
Anchor a full sync at a fixed deep-history point instead of returning no lower
bound, so a backfill covers the multi-year span an incremental tick would skip
and a stale cursor cannot silently narrow it.
The header comment still described WHOOP_RECOVERY_SLEEP_OVERLAP_MS as 24 h
and claimed workout ingest used the narrow default. The constant has been
7 days, and workout ingest now shares it so a late phone-sync to the WHOOP
cloud no longer drops the workout outside the window. Comment only.
The WHOOP workout sync used the default 1 h incremental overlap while
recovery and sleep use the wider re-score overlap. The activity collection
filters on the workout's own time range, not on when it reached the cloud,
so a workout whose phone sync lands more than an hour after its window fell
permanently before the cursor and was never ingested — operator-visible
data loss. The workout sync now reuses the recovery/sleep overlap; the
externalId upsert keeps the re-fetch idempotent.
Ladder reconciliation ranks every enum source, so the unranked
fall-through needs out-of-enum fixtures now; a stored single-source
ladder resolving one canonical row is pinned as the new contract. The
body-sync test sheds the import its retired weight path used.
WHOOP's body endpoint carries a self-reported profile weight, not a
reading — the overwrite-in-place row took a fresh measuredAt on every
sync, so a stale manual entry kept resurfacing as the newest weight no
matter what the scale said. The sync no longer ingests it and clears
the legacy row off existing accounts on its next pass.
A trailing space from the portal's copy button reached WHOOP verbatim
and answered as an unknown client; the credential schema trims both
fields. The redirect override fell through nullish coalescing, so the
empty string the compose whitelist materialises for an unset var was
sent as the literal redirect target — empty now derives the callback
from the app URL.
Sleep nights collapse per writer instead of per source: several
HealthKit apps land behind the same source value, so a phone's coarse
awake and in-bed blocks could blend into or fully constitute a night
whose stages another writer had recorded. The winning writer per
session is the one with the most distinct granular stages, then the
user's source ladder, and its rows are kept whole. The device type is
threaded through every sleep read including the stage stacked bar, and
the sync overlap for late-arriving nights widens from one day to seven
so a night scored after the cursor passed is fetched on the next tick —
assembly is read-time, so affected nights heal without manual repair.
Mood notes expand to their full text in the list on desktop and touch,
replacing the hover-only tooltip, and the mood CSV export carries a
note column.
mapCycle now emits AVERAGE_HEART_RATE and MAX_HEART_RATE from the cycle
score's average_heart_rate / max_heart_rate — both were already on the typed
WhoopCycle interface but dropped on the floor. mapSleep now reads
stage_summary.disturbance_count (added to the WhoopSleep interface) and emits
SLEEP_DISTURBANCE_COUNT. All three guard nulls / unscored records and carry
distinct field tags (avg_hr / max_hr / disturbances) so they don't collide
with the existing strain / energy / stage rows under the shared resource
externalId. Update mapping.md to document the three and note the cycle HR
pair were previously fetched-but-dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two connect-in-app enhancements for the native client.
return_scheme: GET /api/whoop/connect accepts an optional custom scheme,
threaded through the server-side OAuth-state row so it survives the round
trip without ever riding in the URL or cookie. When a valid scheme is
present, the callback's final redirect targets <scheme>://whoop?whoop=
connected|error&reason=... so an in-app web session auto-completes on its
custom-scheme match; absent or invalid, the web settings redirect is
unchanged. Schemes validate against a strict allowlist (custom app scheme
only, ^[a-z][a-z0-9.+-]*$, http/https/javascript/data/file/vbscript
rejected, pinned to dev.healthlog.app). A rejected scheme falls back to
the web redirect and never reflects an arbitrary value.
Bearer connect ticket: a purely Bearer-authenticated client holds no web
session cookie, so it cannot start the handshake through the cookie-gated
connect route. POST /api/whoop/connect/ticket (Bearer) mints a one-time,
~60s, opaque ticket bound to the user; only its HMAC-SHA256 hash is stored
(the raw value never persists). GET /api/whoop/connect?ticket=<opaque>
accepts the ticket in lieu of a cookie, consumes it atomically (single-use,
reuse rejected), sets the nonce cookie, and 302s to WHOOP exactly as the
cookie path. Expired, consumed, or invalid tickets return a typed 401. The
two combine. The daily WHOOP OAuth-state cleanup also sweeps expired ticket
rows.
The ticket query param is added to the egress redaction denylist so the
opaque value never reaches logs or error reports.
Migration 0124 adds whoop_connect_tickets (id, user_id FK cascade,
token_hash unique, expires_at, consumed_at, created_at) and a nullable
return_scheme column on whoop_oauth_states.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 221b362011d52dda314139e0be7cc195c47d016e)
After the graceful collection-403 soft-skip, a genuine grant revoke 403s every
data class: each resource soft-skips (returns 0, records no failure), so the
orchestrator sees no error and stamps recordSyncSuccess — leaving the dead
connection looking healthy until the token-refresh path catches the 401 up to
~1 h later. Track the soft-skip count across the resource loop via an
AsyncLocalStorage tracker scoped to the cycle; when every resource soft-skipped
and nothing imported, leave the status as-is instead of stamping success. A
partial cycle (any import, or any resource that did not soft-skip) stamps as
normal.
A revoked refresh token returns HTTP 400 with an OAuth body of
`error: invalid_grant` from the token endpoint. The status-only classifier
buckets every 400 as persistent, so the connection never prompts a reconnect
even though the grant is gone. Store the upstream OAuth error on WhoopApiError
and lift a 400 invalid_grant to reauth_required in classifyWhoopError; every
other 400 (a malformed request, invalid_client) stays persistent.
WHOOP body fields are self-reported profile data; guard mapBody against
absent, NaN/Infinity, or non-positive values so a garbage reading never
seeds a WEIGHT measurement or a profile height of 0.
Extract handleCollectionFetchError so the five resource syncs share one
soft-skip-vs-rethrow path instead of five copies. Behaviour is unchanged:
a per-resource 403 returns 0 and keeps the connection connected, anything
else records a classified failure and rethrows.
The body-measurement endpoint was fetched by a function nobody called, so WHOOP
weight, max heart rate, and height never reached the database despite being
documented in the mapping table.
Add sync-body.ts and wire it into the per-user sync loop (and, through it, the
backfill):
- weight_kilogram → a WEIGHT Measurement (source = WHOOP) on the stable
externalId whoop:body:weight with overwrite semantics. A body measurement is
a single self-reported profile value, not a time series, so a re-sync updates
the same row in place rather than accumulating one row per poll. measuredAt is
the fetch time; the source-priority picker ranks a real scale above WHOOP.
- max_heart_rate → WhoopConnection.maxHeartRate (a profile constant).
- height_meter → User.heightCm (m→cm), written ONLY when the user has no height
yet — a user-set height is never overwritten and height is never minted as a
Measurement.
Every write is field-by-field and idempotent across reruns. Document the body
mapping and the watch-only blood-pressure stance in mapping.md.
A 403 on a single WHOOP collection endpoint was classified reauth_required,
which parked the whole connection at error_reauth and short-circuited every
future sync. A lower-tier user missing one gated data class would brick the
entire integration.
Treat a per-resource collection 403 as a soft skip of that data class: log,
return 0, and continue so sibling resources still sync. Reserve connection-wide
reauth for a 401 (token rejected) and for a 403 on the token-refresh / profile
path, which run outside the per-resource catch blocks and remain unchanged.
Add isCollectionForbidden as the shared gate and cover both branches in tests.
Connect mints the state nonce into the WhoopOAuthState ledger and redirects
to the WHOOP authorize page; callback verifies the nonce (atomic
delete-first consume, four distinguished reason tags), exchanges the code,
fetches the WHOOP profile for the user id, persists the encrypted
WhoopConnection, and enqueues the self-converging history backfill.
The webhook handler rate-limits per source first, verifies the
WHOOP_WEBHOOK_SECRET path segment, then verifies the HMAC-SHA256 body
signature over the raw bytes (timestamp + body, timingSafeEqual, stale
timestamp reject) before any work. Valid *.updated enqueues the matching
per-resource sync; *.deleted soft-deletes (workouts hard-delete, no
soft-delete column). Unknown users return 200 to stop retries.
CSP gates api.prod.whoop.com to the WHOOP settings surface + /api/whoop/*;
the webhook path joins the public-path allowlist and the path-secret redact
registry. Env wiring adds WHOOP_WEBHOOK_SECRET + WHOOP_REDIRECT_URI to the
compose whitelist and .env.production.example.
Build the WHOOP sync layer on the W2 client, mirroring the Withings
queues. A per-resource sync (recovery/sleep/cycle/workout) pulls since
the last cursor with a 24h overlap on recovery/sleep to absorb WHOOP's
after-the-fact re-scoring, maps via the W2 mappers, and upserts
Measurements with source=WHOOP keyed on (userId,type,source,externalId)
so a re-post overwrites in place. Workouts upsert into Workout with
per-workout strain + HR zones in metadata and kJ->kcal energy.
Token refresh persists BOTH rotated tokens (WHOOP invalidates the prior
access and refresh token on every refresh). A self-converging boot
backfill enqueues one full-history sync per un-backfilled connection and
stamps backfillCompletedAt; the discovery query drops completed accounts,
idempotent across reboots.
Every new queue is registered in the worker's allQueues + cron schedule
(recovery/sleep/workout poll catch-all, cycle poll-only, backfill,
oauth-state sweep). IntegrationStatus gains the whoop key + label.
Tests: rotating-refresh persistence, idempotent measurement upsert,
incremental-window overlap, backfill self-convergence, and a
queue-registration inventory guard.
Add the WHOOP integration data layer on copy-of-Withings rails:
- client.ts: hand-rolled fetch over safeFetch (no SDK) — OAuth authorize
URL / code exchange / rotating-refresh, the four v2 collection fetchers
(recovery, sleep, cycle, workout) with next_token pagination, single
body-measurement + profile reads, and the field->Measurement mappers
(RECOVERY_SCORE/HRV_RMSSD/RESTING_HEART_RATE/OXYGEN_SATURATION/
SKIN_TEMPERATURE, per-stage SLEEP_DURATION ms->min, SLEEP_NEED/SLEEP_*,
DAY_STRAIN/ENERGY_EXPENDITURE_KJ, source=WHOOP). Day strain maps to
DAY_STRAIN and RMSSD to HRV_RMSSD so the device-native values never
share a bucket with the COMPUTED STRAIN_SCORE / SDNN proxies.
- response-classifier.ts: HTTP-status-driven success/transient/
reauth_required/persistent verdicts + WhoopApiError, mirroring the
Withings reason-tag classifier.
- credentials.ts: per-user BYO-key resolution, decrypting the client
id/secret from the User columns.
- oauth-state.ts: opaque-nonce mint helper + cookie/TTL constants.
- mapping.md: field->Measurement reference, kept in sync with the mappers.
Unit tests cover the classifier, pagination, every fetcher, the mappers
(ms->min, kJ->kcal factor, stage mapping, unscored-record handling), and
credential resolution against mocked WHOOP JSON. No routes / OAuth flow /
sync workers / migration yet.
Marc-André Bombeck