The webhook-driven per-resource sync advanced the resource cursor but never
touched IntegrationStatus, so lastSuccessAt stayed frozen at the last full
poll — the Settings 'last synced' pill and the integration-status read
reported stale for up to an hour after a webhook had already landed last
night's data. Stamp recordSyncSuccess per resource, reusing the full sync's
exact success guard (a genuine import clears the error state; a 403 soft-skip
leaves the looks-healthy window closed).
Platform-wide hardening: the failure classes found live this week hunted down
across every integration.
Fitbit carried the same tombstone wedge fixed for Google — probe now matches
soft-deleted rows and the update resurrects them; the comment describing a
partial unique index that never existed is corrected. Sleep segment ids are
stable across re-scoring for Withings (session+segment-start), WHOOP/Polar
(one segment per stage, no positional index) and Oura (run-start keyed), each
with a session-scoped sweep that soft-deletes rows a re-score orphaned — so
legacy and affected nights heal as they are re-read.
One platform rule for deleted rows: source-owned readings resurrect on
re-import (Withings, WHOOP, Oura, Polar, Nightscout, mood webhook, CSV
re-imports, Apple stats day-totals); Apple sample-grain deletions stick per
the paired-client contract, and the mood bulk route's tombstone match is now
a true no-op reported as duplicate. Import counts reconcile against actual
createMany counts.
Google lifecycle: a dead token registers on the cycle's hard-fail ledger, so
an expired connection parks instead of stamping success hourly; backfill and
sleep-repair markers stamp only after a clean pass (pg-boss retries are now
real); failed measurement writes hold the watermark; no-op updates are
skipped; mapping.md tells the current truth.
Scale: doctor-report reads only consumed columns and enabled sections;
exports and backups keyset-paginate; achievements tally via SQL buckets;
series charts day-bucket dense kinds beyond 90 days; BP pairing is a linear
two-pointer; distinct-type scans run server-side; the graded-series cold
fallback is bounded; the fitbit defer tracker loops instead of spreading.
Fitbit, WHOOP, and Oura all issue one-time-use refresh tokens. The token
persistence did an unguarded read → refresh → write, so two overlapping
syncs for one user both read the same stored refresh token and both POST
it. The provider honours only the first; the loser's refresh comes back 401,
gets classified as reauth, and the connection is parked at `error_reauth`
spuriously even though the peer rotated the token successfully.
Add a shared `persistRotatedToken` helper that writes the rotated pair with
a compare-and-swap on the stored ciphertext (matched verbatim, since AES-GCM
re-encryption of the same plaintext differs byte-for-byte). When the
conditional update touches zero rows a peer already won the race; re-read the
row and reuse the peer's freshly rotated access token instead of erroring.
Wire it into all three providers — Fitbit and WHOOP `getValidToken`
(proactive expiry refresh, keyed by connection id) and Oura `storeOuraTokens`
(reactive 401 refresh, keyed by user id). Both racing callers now end with a
usable token and neither drops the connection.
One-shot prettier pass so the new format:check gate starts from a clean
baseline. Formatting only — no behaviour change. Generator-owned and
vendored artifacts stay untouched via .prettierignore.
The four WHOOP resources are independent pg-boss queues that run concurrently
for the same user, so the read-modify-write that merged one cursor key into the
map and wrote it back could let a sibling's write clobber a just-advanced
cursor. Replace it with a single jsonb_set UPDATE that merges in place under the
row lock, and GREATEST so the shared last-synced instant only moves forward.
The resource key is parameter-bound and asserted against the closed resource set
before it reaches the JSON path.
Replace the single shared last-synced cursor with a per-resource cursor map
on the WHOOP connection. Recovery, sleep, workout, and cycle each advance only
their own cursor on a successful fetch+upsert, so a slow or 403-ing collection
never drags the incremental window forward for its siblings and its own
late-arriving records stay inside the overlap. Legacy connections fall back to
the shared last-synced instant, so no historical state is lost.
Add single-record fetch-by-id client routes and per-resource by-id syncs. The
update webhook now carries the resource id through the queue payload, so the
worker resolves and upserts that one record immediately rather than re-walking
the whole collection on the next overlap tick.
Anchor a full sync at a fixed deep-history point instead of returning no lower
bound, so a backfill covers the multi-year span an incremental tick would skip
and a stale cursor cannot silently narrow it.
The header comment still described WHOOP_RECOVERY_SLEEP_OVERLAP_MS as 24 h
and claimed workout ingest used the narrow default. The constant has been
7 days, and workout ingest now shares it so a late phone-sync to the WHOOP
cloud no longer drops the workout outside the window. Comment only.
Sleep nights collapse per writer instead of per source: several
HealthKit apps land behind the same source value, so a phone's coarse
awake and in-bed blocks could blend into or fully constitute a night
whose stages another writer had recorded. The winning writer per
session is the one with the most distinct granular stages, then the
user's source ladder, and its rows are kept whole. The device type is
threaded through every sleep read including the stage stacked bar, and
the sync overlap for late-arriving nights widens from one day to seven
so a night scored after the cursor passed is fetched on the next tick —
assembly is read-time, so affected nights heal without manual repair.
Mood notes expand to their full text in the list on desktop and touch,
replacing the hover-only tooltip, and the mood CSV export carries a
note column.
After the graceful collection-403 soft-skip, a genuine grant revoke 403s every
data class: each resource soft-skips (returns 0, records no failure), so the
orchestrator sees no error and stamps recordSyncSuccess — leaving the dead
connection looking healthy until the token-refresh path catches the 401 up to
~1 h later. Track the soft-skip count across the resource loop via an
AsyncLocalStorage tracker scoped to the cycle; when every resource soft-skipped
and nothing imported, leave the status as-is instead of stamping success. A
partial cycle (any import, or any resource that did not soft-skip) stamps as
normal.
Extract handleCollectionFetchError so the five resource syncs share one
soft-skip-vs-rethrow path instead of five copies. Behaviour is unchanged:
a per-resource 403 returns 0 and keeps the connection connected, anything
else records a classified failure and rethrows.
The body-measurement endpoint was fetched by a function nobody called, so WHOOP
weight, max heart rate, and height never reached the database despite being
documented in the mapping table.
Add sync-body.ts and wire it into the per-user sync loop (and, through it, the
backfill):
- weight_kilogram → a WEIGHT Measurement (source = WHOOP) on the stable
externalId whoop:body:weight with overwrite semantics. A body measurement is
a single self-reported profile value, not a time series, so a re-sync updates
the same row in place rather than accumulating one row per poll. measuredAt is
the fetch time; the source-priority picker ranks a real scale above WHOOP.
- max_heart_rate → WhoopConnection.maxHeartRate (a profile constant).
- height_meter → User.heightCm (m→cm), written ONLY when the user has no height
yet — a user-set height is never overwritten and height is never minted as a
Measurement.
Every write is field-by-field and idempotent across reruns. Document the body
mapping and the watch-only blood-pressure stance in mapping.md.
A 403 on a single WHOOP collection endpoint was classified reauth_required,
which parked the whole connection at error_reauth and short-circuited every
future sync. A lower-tier user missing one gated data class would brick the
entire integration.
Treat a per-resource collection 403 as a soft skip of that data class: log,
return 0, and continue so sibling resources still sync. Reserve connection-wide
reauth for a 401 (token rejected) and for a 403 on the token-refresh / profile
path, which run outside the per-resource catch blocks and remain unchanged.
Add isCollectionForbidden as the shared gate and cover both branches in tests.
Build the WHOOP sync layer on the W2 client, mirroring the Withings
queues. A per-resource sync (recovery/sleep/cycle/workout) pulls since
the last cursor with a 24h overlap on recovery/sleep to absorb WHOOP's
after-the-fact re-scoring, maps via the W2 mappers, and upserts
Measurements with source=WHOOP keyed on (userId,type,source,externalId)
so a re-post overwrites in place. Workouts upsert into Workout with
per-workout strain + HR zones in metadata and kJ->kcal energy.
Token refresh persists BOTH rotated tokens (WHOOP invalidates the prior
access and refresh token on every refresh). A self-converging boot
backfill enqueues one full-history sync per un-backfilled connection and
stamps backfillCompletedAt; the discovery query drops completed accounts,
idempotent across reboots.
Every new queue is registered in the worker's allQueues + cron schedule
(recovery/sleep/workout poll catch-all, cycle poll-only, backfill,
oauth-state sweep). IntegrationStatus gains the whoop key + label.
Tests: rotating-refresh persistence, idempotent measurement upsert,
incremental-window overlap, backfill self-convergence, and a
queue-registration inventory guard.
Marc-André Bombeck