Every ingest path in the product can now emit a typed, salience-classified
event into pg-boss, consumed by one worker that claims a durable per-day
reaction marker and fans out to the reaction surfaces.
The recency classifier runs at the seam, before anything is enqueued, so a
mass backfill — a multi-year Apple Health export, a provider re-sync, a
30-day catch-up — emits nothing at all. Non-salient writes are annotated and
dropped.
The worker makes no provider call by construction: it imports no provider
module, writes only markers, and enqueues nothing that is not budget-gated
downstream. Business refusals return a status rather than throwing, so a
refusal is never retried against a ceiling that will not move until the local
day rolls over.
Seams: sleep (one hook, inherited by all seven transports), workouts (the
batch route, the four provider syncs, the export importer), weight and blood
pressure (both measurement routes), labs (manual, OCR commit, document fact
commit).
The provider workout syncs could not previously tell a created row from an
updated one, so a re-sync of a recent session would have re-fired on every
poll; they now recover the distinction from the row the upsert already
returns.
Reaction lines are AES-256-GCM at rest, following the InsightNarrative
precedent, and the column is registered for key rotation. Markers are pruned
after 14 days by a daily maintenance job.
The Today digest went stale after logging a dose: nothing invalidated the
`["daily","digest"]` query, and it reads meds from the same server snapshot
the intake routes already hard-evict — so the "dose due" rail item lingered
until a hard reload. Add the digest key to the medication-dependent bundle and
force the inactive digest (unmounted while the user is on the medication card)
to refetch alongside the dashboard snapshot; the server snapshot eviction seam
is reused, so the refetch returns post-write dose state immediately.
Remove the score-ring cluster from the Today hero — it read as uneven and
wasted tile space. The health-score ring is the hero's only ring now; the
score-ring selection contract stays server-side for the native client. Tighten
the hero's vertical rhythm so the worth-a-look rail sits close under the lead
instead of floating below a half-empty column.
Wire the remaining sleep transports into the morning-refresh trigger: Google
Health, Fitbit, Oura, and Polar now enqueue the debounced morning refresh on a
last-night segment landing, matching the Withings / WHOOP / Apple seams — those
sources had no trigger at all. Make the trigger observable: each early-return
gate now annotates a distinct `daily.morning_refresh.skipped` reason (the
not-last-night gate also carries the samples' resolved local dates, today's key
and the tz) so a real landing shows exactly which gate fired.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018V8V9qapgHi7fhgtMJ8SkM
Single-fix release mirroring the Google Health natural-key rescue from
v1.28.28 into the Fitbit sync. The measurements natural-key unique index
(user, type, measured_at, source, sleep_stage — NULLS NOT DISTINCT,
migration 0055) covers tombstoned rows, so a re-import carrying NEW
externalIds for previously-removed rows collided on the natural key and
skipDuplicates dropped the insert silently. Every planned create now gets
a second chunked probe by natural key with no deletedAt filter; a match —
live or tombstoned — is updated in place with the fresh value, the new
externalId, and deletedAt: null. Probe failure notes the hard-failure
ledger so the watermark holds and the next tick retries. No other code
path touched.
Platform-wide hardening: the failure classes found live this week hunted down
across every integration.
Fitbit carried the same tombstone wedge fixed for Google — probe now matches
soft-deleted rows and the update resurrects them; the comment describing a
partial unique index that never existed is corrected. Sleep segment ids are
stable across re-scoring for Withings (session+segment-start), WHOOP/Polar
(one segment per stage, no positional index) and Oura (run-start keyed), each
with a session-scoped sweep that soft-deletes rows a re-score orphaned — so
legacy and affected nights heal as they are re-read.
One platform rule for deleted rows: source-owned readings resurrect on
re-import (Withings, WHOOP, Oura, Polar, Nightscout, mood webhook, CSV
re-imports, Apple stats day-totals); Apple sample-grain deletions stick per
the paired-client contract, and the mood bulk route's tombstone match is now
a true no-op reported as duplicate. Import counts reconcile against actual
createMany counts.
Google lifecycle: a dead token registers on the cycle's hard-fail ledger, so
an expired connection parks instead of stamping success hourly; backfill and
sleep-repair markers stamp only after a clean pass (pg-boss retries are now
real); failed measurement writes hold the watermark; no-op updates are
skipped; mapping.md tells the current truth.
Scale: doctor-report reads only consumed columns and enabled sections;
exports and backups keyset-paginate; achievements tally via SQL buckets;
series charts day-bucket dense kinds beyond 90 days; BP pairing is a linear
two-pointer; distinct-type scans run server-side; the graded-series cold
fallback is bounded; the fitbit defer tracker loops instead of spreading.
Fitbit's handleCollectionFetchError soft-skipped a 403 but RETHREW any
other hard failure, aborting every endpoint ordered after the bad one in
the same resource and stalling the watermark so the whole integration
stayed dead. Record hard failures on a per-cycle AsyncLocalStorage ledger
and return 0 instead of rethrowing — the sibling endpoints keep fetching,
and the ledger still fails the cycle so success is not stamped and the
next tick refetches. Also wrap the per-metric map fan-out in the same
per-collection catch so one malformed point can't skip its siblings.
Reconciles the v1.26.1 Fitbit setup-guide fix with the Google Health
release: google-health.md stays the full runbook (with a pointer to the
classic fitbit.md guide), the doc-parity guard now asserts the Google
Health page against the google-health client's endpoints and scopes,
and the changelog carries both sections.
The Fitbit integration connects through the classic Fitbit Web API, so a
self-hoster registers a Fitbit developer app at dev.fitbit.com — but the
setup guide walked users through creating a Google Cloud OAuth client,
whose client id the Fitbit sign-in rejects (unauthorized_client). Rewrite
the guide for the dev.fitbit.com flow (app type, HTTPS callback, scopes,
PKCE), leave the old page as a pointer, and add a test that keeps the
documented scopes and endpoints in sync with the client so the guide
cannot drift from the code again. Flags the September 2026 sunset of the
classic Fitbit Web API.
Drop the unreachable text-only-model reason from the OCR capability
enum and its DTO type. Replace the literal NUL delimiter in the Fitbit
dedup map keys with a named constant so text tooling stops treating the
file as binary (behaviour unchanged). Correct the Coach tool-executor
docblock, which overstated cross-tool snapshot-cache sharing.
Fitbit, WHOOP, and Oura all issue one-time-use refresh tokens. The token
persistence did an unguarded read → refresh → write, so two overlapping
syncs for one user both read the same stored refresh token and both POST
it. The provider honours only the first; the loser's refresh comes back 401,
gets classified as reauth, and the connection is parked at `error_reauth`
spuriously even though the peer rotated the token successfully.
Add a shared `persistRotatedToken` helper that writes the rotated pair with
a compare-and-swap on the stored ciphertext (matched verbatim, since AES-GCM
re-encryption of the same plaintext differs byte-for-byte). When the
conditional update touches zero rows a peer already won the race; re-read the
row and reuse the peer's freshly rotated access token instead of erroring.
Wire it into all three providers — Fitbit and WHOOP `getValidToken`
(proactive expiry refresh, keyed by connection id) and Oura `storeOuraTokens`
(reactive 401 refresh, keyed by user id). Both racing callers now end with a
usable token and neither drops the connection.
The classic 1.2 sleep log and the activities list emit local wall-clock
ISO strings without an offset. `parseLocalInstant` parsed them with a bare
`new Date(iso)`, which reads an offset-less string in the process zone (UTC
in production). A non-UTC user's sleep-segment `measuredAt` (the segment
end) was therefore shifted by their UTC offset, and a near-midnight segment
could flip onto the wrong wake-day in the night reconstruction.
Resolve offset-less timestamps against the user's stored timezone via a new
`zonedWallClockToUtc` helper (the inverse of `wallClockInTz`, DST-safe via
the same two-pass converge as `startOfLocalDayInTz`). Thread the resolved
zone through the sleep and workout sync paths. Strings that already carry an
offset or `Z` stay absolute and parse as-is; an absent zone keeps the prior
host-local fallback.
The shipped Fitbit integration spoke the Google Health API, whose Restricted
scopes need brand verification plus an annual CASA assessment before leaving
testing mode — a wall most self-hosters cannot clear. Retarget the transport to
the classic api.fitbit.com Web API: a self-hoster registers an app at
dev.fitbit.com in minutes, no CASA.
Forks only the transport; the upsert/dedup/rollup tail, source-priority,
backfill, poll worker, and status ledger are reused unchanged.
- OAuth: www.fitbit.com/oauth2/authorize + api.fitbit.com/oauth2/token, PKCE
(S256), classic self-serve scopes.
- Refresh tokens rotate (one-time use): persist the new token every refresh,
with a defensive keep-existing guard on a malformed reply.
- Per-endpoint date-range fetchers chunked to the API range caps; the backfill
horizon and poll concurrency are tuned for the 150 req/h-per-user budget.
- Verified classic-shape extractors for weight, body fat, SpO2, HRV (dailyRmssd),
resting HR, respiratory rate, steps, distance, active calories, floors, VO2 max
(range midpoint), per-segment sleep stages, and workouts.
- Skips intraday HR (needs a per-app grant; daily resting HR covers it) and skin
temperature (a baseline delta with no honest absolute slot).
One-shot prettier pass so the new format:check gate starts from a clean
baseline. Formatting only — no behaviour change. Generator-owned and
vendored artifacts stay untouched via .prettierignore.
Add a reminder-satisfy pg-boss queue (mirrors pr-detection) that every
measurement and lab ingest path enqueues — manual create, batch, and the
Withings / Polar / Oura / Nightscout / Fitbit / WHOOP syncs. A reminder
now auto-resolves the moment a matching reading lands instead of waiting
up to 15 minutes. The 15-minute cron stays as the idempotent safety net;
the forward-only guard lets the eventful hook and the cron converge
without double-stamping.
The worker honours the module toggle (a reminder for a disabled module
produces no engine activity) and runs the same satisfaction primitives
the cron uses.
The Google sleep payload carries a real per-stage segment series, but the
mapper summed each stage's segments into one row stamped on the stage's
last end. The hypnogram then reconstructed all of a stage's time as a
single block ending at the night's right edge — the same stacked-bar
artefact as WHOOP, despite the source carrying true timing.
Emit one row per segment with measuredAt at that segment's end and an
indexed externalId so the several segments of one stage stay distinct
under the dedup key. The timeline is measured (real onsets), so these
rows are not flagged reconstructed.
Close the launch-blocking gap where the ConsentReceipt infrastructure was
built but never enforced as a precondition: a direct API caller with a valid
token could forward a health snapshot to a server-managed external LLM with no
receipt on file. Consent was enforced client-side only.
Add a consent guard (src/lib/ai/consent-guard.ts) and call it before the first
external-provider call on the server-managed path. The gate fires only when the
resolved chain could egress via the operator's global key (admin-openai); a
user's own BYOK key (openai/anthropic), their own ChatGPT OAuth account, and
the self-hosted local provider stay ungated — that egress is the user's own act
and the settings flow is its consent. The check fails closed: a BYOK-primary
chain with an admin-openai fallback still requires a receipt, since the runner
may cascade to the operator key. The surface maps to the consent kind (coach ->
ai_coach, insights -> ai_insights_only; master ai_full satisfies either).
- Interactive routes (insights/chat, insights/generate POST) throw
ConsentRequiredError; api-handler renders a 403 with
meta.errorCode = "consent.ai.required", mirroring assistant.disabled.* so
the client can render an inline grant-consent notice.
- Off-request pipelines fail closed quietly: comprehensive generation returns a
typed skipped:no-consent outcome, and the per-metric status family
(runStatusCompletion, gating every status card + the period-narrative warm +
the off-budget Coach memory workers) surfaces the no-key fallback instead of
egressing.
Also rate-limit POST /api/fitbit/sync (the one Fitbit route without a limiter):
a 5/60s baseline plus a tighter 1/hour bucket on the expensive fullSync walk,
keyed by user id, matching the sibling Fitbit routes.
Harden the Fitbit OAuth redirect_uri: assert it is an absolute https origin (or
http on localhost), lands on /api/fitbit/callback, and stays same-origin with
NEXT_PUBLIC_APP_URL — defence-in-depth behind Google's registered-URI check so
a Host-coerced or misconfigured app URL cannot redirect the authorization code
off-origin.
(cherry picked from commit 46b5fc2cea1b6e80bee0445bfdacf5790442c67d)
Three hardening + performance fixes to the Fitbit/Google Health sync.
Stop soft-delete resurrection. The measurement write probed the existing
row by the `(userId, type, source, externalId)` unique key without excluding
soft-deleted rows, so a reading the user deleted came back on the next hourly
sync (the upsert matched the tombstone and took the update branch — silently
undoing the delete). The write now probes only live rows (`deletedAt: null`)
and treats a tombstone as absent, minting a fresh insert instead. The fresh
insert relies on the partial unique index over `deleted_at IS NULL` to keep
live-row uniqueness; the tombstone sits outside it, and `skipDuplicates` guards
a live-row race.
Batch the backfill writes + collapse the rollup recompute. The write loop
was one upsert per reading (N+1) followed by a per-(type,day) DAY-rollup
recompute, so a multi-year backfill paid thousands of serial round-trips on a
concurrency-1 worker. The existence probe is now a single `findMany` over the
batch; fresh rows go through chunked `createMany`; only already-live rows take a
per-row update for their differing values. On a `fullSync` the per-write rollup
hook is deferred and the touched type-days collapse into one
`recomputeUserRollups(from, to)` pass at the end of the cycle. The incremental
path keeps the inline per-day hook. The `stats:` overwrite contract, the
single-watermark `markSynced`, and the all-403 soft-skip guard are unchanged.
Bound the hourly poll concurrency. The poll synced every connection in one
serial loop, so a single slow Google response stalled the whole cohort. The
cohort now fans out through a bounded `p-limit` pool with per-user error
isolation, extracted into `runFitbitPollCohort` so the contract is unit testable
without exporting worker internals. The accumulators are read after the await to
avoid a compound-assignment lost-update across overlapping pool tasks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 8369af2471fb8da04a938b5a784433318d81a248)
Use FITBIT_ACTIVITY_PAGE_SIZE for the sleep and exercise reads instead of a
magic 25, remove the unused FitbitDataTypeKey type, and declare the crosstab
day floors independently so they aren't a duplicate re-export of the influence
floors.
Snapshot the incremental watermark once per sync cycle and thread the same
lower bound to every resource (metrics, activity, sleep, workout), stamping
lastSyncedAt once at the end. Previously each resource read lastSyncedAt then
stamped now() independently, so the first resource advanced the watermark and
later resources only fetched the last overlap window — silently dropping the
gap after an outage longer than the overlap. The full/backfill path is
unaffected (it passes no lower bound).
Reject an OAuth callback whose token response carries no refresh_token instead
of persisting an empty string. An empty refresh token bricks every future
token refresh; the callback now errors out cleanly so the user re-consents
rather than landing a dead connection.
Model the INTERVAL data types (steps/distance/calories/floors daily totals,
sleep and exercise sessions) on Google Health's interval.start_time /
interval.civil_start_time anchor instead of a sample_time / bare date. The
incremental filter for sleep/exercise previously targeted
sample_time.physical_time, which 400s/empties for interval types and stalls
incremental sync. Daily-summary metrics and spot readings keep their existing
date / sample anchors. Shape mismatches still fail safe (return []/null).
Mark the Google Health (Fitbit & Pixel) settings card as experimental with a
badge and a short note that data coverage is still being verified, in all six
locales.
(cherry picked from commit 3f116802149eebe9144729e2779f7f4405f248f2)
Extend the Fitbit / Google Health provider with the remaining sync
resources, driven by the existing syncUserFitbit orchestrator.
Activity (daily cumulative): steps -> ACTIVITY_STEPS, distance ->
WALKING_RUNNING_DISTANCE, the active-energy portion -> ACTIVE_ENERGY_BURNED,
floors -> FLIGHTS_CLIMBED, and VO2 max -> VO2_MAX. Each daily total mints a
stats:<tag>:<YYYY-MM-DD> externalId so a re-fetched day overwrites in place
rather than duplicating, matching the Apple-Health daily-total contract. The
running totals preserve a legitimate zero (a rest day is real data); VO2 max
stays strictly positive (daily latest-wins).
Sleep: map a session's per-stage segments to per-stage SLEEP_DURATION rows
(minutes; measuredAt = the stage end), harmonised onto the shared SleepStage
enum (light -> CORE, deep -> DEEP, rem -> REM, awake/restless -> AWAKE,
in-bed -> IN_BED, classic asleep -> ASLEEP) so the night-total and hypnogram
readers consume the same shape they already read for WHOOP and Apple.
Workouts: map exercise sessions to Workout rows keyed (userId, FITBIT,
externalId), with sport-type, start/end, duration, energy, distance, and HR.
Cross-source twins stay distinct at ingest; the read-time canonical picker
collapses them via the source ladder (Fitbit already ranked below WHOOP).
All three join the orchestrator's resource list and ride the per-resource
403 soft-skip so an independently-granted Restricted bundle that the user
withheld skips only its own resource. Rollups recompute and status-insight
caches invalidate through the shared upsert. No new MeasurementType, no
migration, no new dependency.
Extend the Fitbit/Google Health provider with the read path: the
dataPoints.list client half, the metrics-bundle sync, the poll + backfill
jobs, and the connection-management routes. Builds on the existing OAuth +
credentials + token-refresh layer.
- client: paginated `fetchDataPoints` walker over `safeFetch` keyed on
`nextPageToken`, encoding each data type kebab-case in the path and
snake_case in the filter predicate; defensive per-type mappers (finite +
positive guards) for weight, body fat, daily SpO2, daily HRV, daily resting
heart rate, daily respiratory rate, intraday heart rate, sleep-temperature
derivations, and a height profile-seed extractor. `FITBIT_FIELD_MAP` +
`mapping.md` are the source of truth.
- mapping (no new MeasurementType): weight→WEIGHT, body-fat→BODY_FAT,
SpO2→OXYGEN_SATURATION, HRV→HEART_RATE_VARIABILITY (SDNN slot, not
HRV_RMSSD), resting-HR→RESTING_HEART_RATE, respiratory-rate→RESPIRATORY_RATE,
heart-rate→PULSE, sleep-temp→WRIST_TEMPERATURE, height→User.heightCm when
null. Steps/distance/energy/floors/VO2 + sleep + workouts stay for a later
wave.
- sync: `syncUserFitbit` orchestrator with the reauth short-circuit, the
per-data-class 403 soft-skip (the six Restricted bundles grant
independently), the all-soft-skipped success guard, and incremental-vs-full
windows; `upsertFitbitMeasurements` keyed (userId,type,FITBIT,externalId)
with a 24 h overlap for after-the-fact daily re-rolls, recomputing rollups
and invalidating status caches. Google refresh tokens do not rotate, so the
stored refresh token is kept unless a new one is returned.
- jobs: hourly `fitbit-sync` poll (poll-only; Pub/Sub deferred), the
self-converging boot backfill, and the daily OAuth-state sweep — every queue
registered in both `allQueues` and a `boss.work` binding, staggered off the
WHOOP/Withings crons.
- routes: status / sync / disconnect mirroring the WHOOP shape.
Tests cover the mappers, the casing invariant, `fetchDataPoints`, the
orchestrator soft-skip guard, the idempotent upsert, the backfill discovery,
and the queue registration; an integration test pins the upsert idempotency,
the DAY rollup fold, and the Fitbit-vs-Withings per-source no-collapse against
the Postgres testcontainer.
Land the OAuth + credentials half of the Fitbit / Google Health provider,
cloning the WHOOP provider structure and adapting it to the Google identity
flow.
- Client OAuth half (src/lib/fitbit/client.ts): Google authorize-URL builder
(access_type=offline + prompt=consent), authorization-code exchange, token
refresh, and a profile fetch for the connection's external user id. Client
credentials ride in an HTTP Basic-auth header on the token endpoint; the
four launch Restricted read scopes are pinned in FITBIT_OAUTH_SCOPE. All
egress goes through safeFetch.
- Non-rotating refresh tokens: getValidToken refreshes at expiry-minus-five-
minutes, persists the new access token + expiry, and overwrites the stored
refresh token ONLY when the response carries a fresh one — Google keeps the
refresh token stable, unlike WHOOP.
- BYO credentials route (PUT/GET/DELETE /api/fitbit/credentials): per-user
client id/secret stored encrypted on the User columns, built field-by-field
under requireAuth.
- OAuth state + connect + callback routes: a random base64url nonce backed by
a 10-minute FitbitOAuthState ledger row carries the (nonce -> userId)
mapping; the callback validates the cookie/URL state with timingSafeEqual,
consumes the row atomically (replay/expired/cross-user reason tags),
exchanges the code, upserts the encrypted FitbitConnection, clears prior
reauth state, and enqueues the self-converging history backfill.
- Response classifier mirrors WHOOP's HTTP-status buckets; Google signals a
revoked grant with a 401, so there is no invalid_grant body special-case.
Tests cover the authorize URL, Basic-auth token exchange, the no-rotation
refresh branch (token preserved when omitted, overwritten when present), the
classifier status table, and the OAuth-state nonce mint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Marc-André Bombeck