`integrations/sync-kit/after-write.ts` is the tail every provider write
ends in: the arrivals go out, the touched days are refolded, the read
caches are marked stale back to the earliest touched reading, and the
touched status cards are re-warmed (through `afterMeasurementMutation`,
shared with the interactive routes); Withings also evicts the dashboard
snapshot. A full-history backfill refolds its whole touched range once
through `afterProviderBackfill`.
Withings, WHOOP, Polar, Nightscout, Google Health, Fitbit and Oura drop
their hand-written tails, and no provider module imports the comprehensive
insight generator any more; a guard holds the provider folders to that.
Differences from the copies: the status cards are re-warmed even when the
fold failed, and the caches are marked stale back to the earliest touched
reading instead of wholesale, so older closed blocks survive a sync.
Several provider unit tests mocked an export the old tails never reached
(their failures were caught and only logged); the mocks now name what the
code calls.
`oauth-kit/client.ts` builds a provider's consent URL, code exchange,
refresh and deauthorization from a declaration of what differs: endpoints
and scope, client authentication (form or HTTP Basic, with or without the
client id beside it), the fields a grant adds, how an answer is classified
and the refusal named, the error type, and where the token sits in the
body. The seven hand-written copies of the redirect URI, authorization URL,
token post, Basic header and PKCE pair are gone; each provider keeps an
`oauth.ts` with its declaration.
Shared behaviour is now uniform: a 2xx without a token object (an empty or
non-JSON body) is a transient `empty_token_body` failure for every
provider. WHOOP, Strava and Google Health used to return null, and
Withings threw an unclassified SyntaxError; the OAuth wire golden records
exactly these lines changing and nothing else. Oura's live/legacy endpoint
fallback, Google's invalid_grant lift, Strava's Fault summary and Fitbit's
errorType are declared per provider.
lib/sharing/grants.ts held both the grant transitions and the predicates
every reader asks (state, level, scope, the live grant for a pair). The
transitions reach into the managed-profile lifecycle and the OAuth connect
target, and both of those read a live grant, so grants, lifecycle and
connect-target formed an import cycle.
The predicates, findActiveGrant and touchGrantUsage move verbatim to
lib/sharing/grant-access.ts, which imports only the database client and
the scope vocabulary. Importers and the mocks that replaced those
functions point at the new module. Runtime import cycles fall from 7 to 6.
The batch route counted arrivals inline and grew past its size baseline
with the new-sample rule. tallyHealthKitArrivals now owns the fold, the
route hands it each landed entry and hour with whether it brought a new
value, and the size baselines move down to what the route is now.
Refs #1259
A batch that carried a type with duplicates only moved its
lastReceivedAt, lastTrigger and lastBackgroundAt, so a type whose live
delivery had stalled kept reading as freshly arrived for as long as a
replay, a retry or an older client re-sent its old samples.
The arrival ledger now moves those stamps only for a type the batch
brought at least one new or changed row for. An unchanged stats:
re-post and an hour posted again with its stored value count as
duplicates here; the wire keeps answering updated for them. A
duplicates-only type stamps the new last_seen_duplicate_at column on
its existing row and nothing else. The intake bulk route's dose entry
follows the same rule through the shared recorder. lastSyncedAt and
healthKitLastSyncedAt are unchanged.
GET /api/integrations/healthkit serves the stamp additively as
metricFreshness[].lastSeenDuplicateAt, and the Apple Health card names a
re-send that came after the last arrival.
Refs #1259
A guardian finishing a connect for a managed profile could attach their
own provider account to the profile. Every OAuth callback (Withings,
WHOOP, Fitbit, Google Health, Strava, Polar, Oura) now refuses a
provider account another record already holds with reason
already_linked, shown as its own message on the integrations page.
WHOOP's owner_conflict tag becomes already_linked as well.
Oura keeps the account id of a grant (personal_info, migration 0441) so
the check has something to compare; existing connections fill it at
their next connect.
A disconnect leaves Withings webhook subscriptions and the Strava app
authorisation in place while another record holds the same account,
and the Withings webhook routes an account held twice to the record
that linked it first instead of whichever row a lookup returned.
The provider app copied from a guardian onto a managed profile is now
marked (migration 0442) and removed again on disconnect or when that
guardian no longer holds a live MANAGE grant: revoke, renounce,
removal, handover, account deletion and data wipe. Each removal is
audited as integration.client_app.removed.
A guardian acting on a managed profile can now connect and disconnect its
Withings, WHOOP, Oura, Polar, Strava and Google Health from the profile's
integration cards, and each card says whose record it connects for. The
connect starts with a fenced fetch that mints a state naming the profile;
the provider's callback lands tokens and data there only after checking
that the browser belongs to a guardian with a live MANAGE grant on it.
Without a provider app of its own, the profile borrows the guardian's,
with an audit row. Withings and Google Health callbacks read the session
directly, since a provider redirect cannot carry the record-session header.
Minting the state, building the consent URL and dropping a grant each had a
copy per provider route. The routes keep their auth, rate limits and
redirects and hand the rest to one implementation, which the managed-profile
routes reuse.
A provider describes where its grant lives and how its token endpoint is
called; the kit reads the grant, refreshes it under the per-user advisory
lock, re-reads under the lock so a peer's refresh or a reconnect wins, and
writes back only over the refresh token it spent. Retry-After is read in
one place, both forms, with a cap.
A 429 in the middle of the activity walk threw out of it before the
upsert, so everything the run had read was discarded and the next run
started over. The walk now ends where the limit hit: what was read
lands, the cursor moves only as far as an oldest-first walk proves,
the failure is booked transient, and the run reports the wait Strava's
X-RateLimit / X-ReadRateLimit headers name (next quarter hour, or
midnight UTC once the daily count is spent). The backfill waits that
long before its next run. `walkPages` learns an `interrupted` page for
this, reported truncated like a walk cut off at its ceiling.
An update mapped from the summary alone (detail budget spent, or the
detail read failed) wrote a null over the calories an earlier detail
read had stored, and over its heart rate and description too. Those
detail-only fields are now kept unless this run read the detail.
Google turns the legacy Fitbit Web API off on 30 October 2026 (support
ended on 30 September). From that day the hourly poll, the backfill,
the sleep repair and the token refresh make no call, the connect, sync
and test routes refuse with `provider_retired`, and the connection
reads as not connected to the health read and the watchdog, so the
shutdown pages nobody and asks nobody to reconnect. The connection row
and everything it imported stay.
The status envelope publishes `retirement` on the Fitbit entry (the
day, whether it has come, the successor), and the Fitbit card shows
when the connection stops, how to move to Google Health (same devices,
one more consent, tokens cannot be carried over) and that the data
stays; sync and connect grey out after the day. Both integrations can
run side by side meanwhile: every type Fitbit writes is laddered with
Fitbit just above Google Health, so a day both deliver counts once.
The test setups pin the turndown off, so the Fitbit suites do not turn
red on the day; the turndown's own tests flip it.
Two callers refreshing one Codex token raced: the loser heard
refresh_token_reused, read it as a dead grant and parked the connection
before the winner wrote its rotated tokens, leaving fresh tokens beside
an expired status. Refreshes now share one flight per process and queue
on the per-credential advisory lock across processes, re-read the row
inside it, use a token a peer already rotated, write the rotated tokens
compare-and-set on the spent one together with connected, and re-read
after a refusal before parking. refresh_token_reused no longer counts as
a dead grant, and neither does a bare 403 without an OAuth error code:
an edge challenge answers a valid grant so, and it keeps the ledger's
timed cooldown instead of parking until the next sign-in.
Resilience sits behind Oura's `stress` scope, not `heart_health`, so a
connection that granted every scope HealthLog asked for still answered 401
on `daily_resilience`. `src/lib/oura/scope.ts` now maps every collection the
sync reads to the scope the live API enforces and is the one source for the
authorize request, the per-collection permission detection, the reconnect
hint and the settings card. The request asks for `stress` as well.
Oura names a missing scope in its 401 body ("Token is not authorized access
stress scope."), while a dead token reads differently. A collection refusal
now carries that scope: it spends no token refresh, never counts toward the
reconnect threshold and marks only the collections it names. A 403 (a lapsed
membership) is a persistent failure rather than a reconnect. The callback
also records what the token response says was granted, so an unticked
permission shows on the card before the first sync.
A browser can deliver an OAuth callback twice with the same code. Both
requests exchanged it, and Oura answers a reused code by revoking the token
the first exchange issued. Every connect callback (Oura, Polar, Strava,
WHOOP, Withings, Fitbit, Google Health) now claims its state in
`oauth_callback_claims` before the exchange: one request exchanges, the
duplicate waits for it and redirects to the same page without touching any
token. The ledger flows claim inside the transaction that consumes their
state row. A failed exchange settles with its error page, and a new connect
mints a new state.
Oura token refreshes are serialised per user with an advisory lock, so
overlapping syncs no longer spend the same single-use refresh token, and the
write stays conditional so a reconnect's new grant is never overwritten by a
refresh of the old one.
A reading entered here goes to Apple Health as well, and the quiet
verdict already counted it, but the row still showed the newest reading
that arrived from the phone, so a type entered two minutes ago read one
day old. The server now resolves lastSeenAt to the newer of the two for
the types the app writes back, and the row and the verdict agree.
Refs #1247
The audit log is the trail of what a person or an admin did to an account.
Every failed provider sync wrote a row there, burying that trail, and kept
the provider's error message in plain text while the status row seals it.
A failed sync now emits an integrations.sync.failed warning with the same
counts and no message.
The Health Connect import stored its unit factors' binary noise as is
(grams to kilograms, millis to minutes, the day sums) and rounded the
pulse means and nutrients privately. Every measurement value now takes
importValue at the write boundary, and importValue takes a precision for
the two places that need another one. The rounding guard now covers the
import directory and the scaled Math.round form.
WHOOP and Oura answer the last page with an empty next_token as well as
with null. The walk followed the empty string as a cursor, the client sent
no token, read the first page a second time and the walk ended as a repeat,
so the records came in twice and the backfill stayed incomplete for good.
An empty cursor now ends the walk, for every provider that walks through it.
Output of `node scripts/strip-planning-tags.mjs --write`: 3,157 comment
lines in 1,308 files lose the work-item identifier at their head. Release
versions and issue references stay; nothing outside comment text
changes, which the script checks per file against a comment-free
printout. healthlog/no-planning-tags is enabled at error level with this
commit, so the tree it lands on is clean.
The page-walk guard flagged the backfill progress codec's page > 0 check, a
validity test rather than a ceiling, and missed the list walk's own loop
(pageCount - skipped < maxPages). The matcher now reads a counter offset and
ignores a literal bound of 0 or 1. Strava and the rollup chunks walk through
walkPages, so their pending entries go. The dataPoints.list walk stays outside
as a named exception with a match allowance: it is bounded by progress rather
than by a cursor ceiling, streams pages, stops on a slice budget and keeps its
place, and its backfill decides with a sliced verdict.
The WHOOP, Oura and Withings list walks and the Fitbit exercise walk each
kept a page loop that, at its ceiling or on a cursor that did not advance,
returned what it had read and said nothing, and the backfill around it
stamped the history complete. `walkPages` keeps a ceiling but reports a
walk that stops with a page pending, or on a cursor it already followed,
as truncated. A backfill reads that through one verdict: a failure
retries, a truncated collection leaves the marker unset without a retry,
a clean run stamps it. The Fitbit exercise walk may read a hundred pages
during a backfill instead of twelve. A guard keeps page-counting loops out
of the provider directories, with Google Health and Strava listed until
their next change.
Weight, blood pressure and blood glucose are written back to the phone
while their entry is bidirectional. A reading entered here goes to Apple
Health and never arrives back, so a person who keeps those readings here
saw the type go quiet after two weeks. The Apple Health status read now
adds `lastEnteredHereAt` to each written-back type, the newest reading
from any source other than Apple Health, and `stale` counts it; the card
shows it as "Last entered here". The value is resolved on the server so a
client renders it as sent.
Fitbit, WHOOP, Oura, Polar and Withings each kept a private
`parseFloat(n.toFixed(2))`, which mis-rounds halves on the double below
the decimal. `importValue` keeps the same two places, rounds half away
from zero on the printed decimal and returns the nearest double, so a
value prints cleanly. A guard keeps `toFixed` and private `round2`
helpers out of the provider directories; the Google Health mapper is the
one listed exception until its next change.
Connect, status, places, settings, test, sync and disconnect replace their
stubs. A nightly job (03:20) and the connect backfill read the timeline,
visits and residency endpoints through safeFetch and store one row of daily
values per local day: distances, places visited, minutes away from the home
place and the sealed country with its travel-day verdict. A private instance
needs its exact origin in DAWARICH_PRIVATE_ORIGINS.
The connection joins the shared ledger and the connection health model,
watched for errors only. The day carries a movement block, the model
projection drops the travel country, the two movement channels join the
correlation matrix under the environment module, travel days become trip
suggestions, and the days travel in the environment backup section.
The watchdog's first pass after a deploy records standing incidents as
announced without paging. Their end then sent a recovery note for
something the operator was never told about. Such incidents now carry
the epoch as their announcement time, which no real announcement can
have, and close without a note.
The watchdog sent one page per account, so a provider outage or an
expired app credential paged once for every connected account. A pass
now collects its messages and sends them per provider at its end: more
than three accounts of one provider opening (or closing) an incident in
the same pass become one message naming the count, the reasons with
their earliest start and the accounts, and one recovery note. Every
incident in it counts as announced.
A failure streak reaching the alert threshold paged the operator even
when the watchdog had already announced an open incident for the same
connection. The ledger now checks for that announced incident first,
stamps the streak as announced and sends nothing; the watchdog's
recovery note still ends it.
The watchdog opened an incident for any degraded state that lasted a
day, so an Apple Health account whose phone app stopped running read as
no_new_data for good and paged. Incidents now open for action_required,
for provider errors and for the watchdog's own unexpected silence; stale
or stalled data alone never pages. The first pass after a deploy, with
no watch row yet, records what stands as already announced instead of
paging every long-broken connection at once.
Migration 0401 turns healthkit_type_syncs.type into text so the ledger can
hold MEDICATION_DOSE, and adds last_background_at (seeded from rows whose
latest arrival was a background one). POST /api/medications/intake/bulk
takes syncTrigger and records APPLE_HEALTH dose arrivals; the Apple Health
status read serves a MEDICATION_DOSE freshness entry and lastBackgroundAt
on every entry with arrival facts.
Their rings and straps upload only when the phone app is opened, so a day
without opening it is ordinary and must not page the operator. Wearables
whose cloud receives data on its own keep 24 h, Apple Health keeps 48 h,
and scales, cuffs and Strava never alert on silence.
An hourly and a daily pass flag a continuous source that delivered on
most of the previous 14 days and has gone quiet, and open one operator
incident per connection when a connection stays degraded or needs action
for more than a day. The incident is announced once through the existing
admin alert path, adopted rather than repeated when the sync ledger
already paged, and closed with one recovery note. Scales, cuffs and
workout logs are watched for errors only, never for silence.
Fold the ledger state, liveness verdict, plan and scope gaps and the
provider reauth flag into ok / limited / degraded / action_required with a
reason code, the one action, the unavailable types and redacted
diagnostics. Keep the last ten sync attempts per connection for those
diagnostics, publish the state on the integrations envelope, every
per-provider status route and the Apple Health config read, and page the
operator once per failure streak instead of once a day.
The in-use probe matched tombstoned readings, so a source stayed marked
on the priority ladder after every reading from it was deleted. Both
the read and the save response of the source-priority route go through
this one probe.
The read and save of /api/auth/me/source-priority now carry inUse: the
ranked sources the account has connected or holds at least one
measurement from, one index probe per source. The settings page lists
only those, hides a metric nothing delivers, and opens one metric group
at a time, the first by default, with the winning source named on each
collapsed line. The ladders stay whole: moving a source swaps it with
its nearest listed neighbour, so a hidden source keeps its slot.
The status verdict looked only at the user's last-batch column, which
moves on a live batch alone. Rows that reached the account another way,
such as the export.zip import, left it null, and the card said 'waiting
for first data' beside a four-digit record count. The verdict now takes
the newest of that stamp, the per-type arrival ledger and the newest
stored sample.
With NEXT_PUBLIC_APP_URL unset, some providers spliced the missing value
into their redirect template and the integration cards told the reader to
register 'undefined/api/polar/callback'. A callback that is not an
absolute http(s) URL now counts as not configured, so the card shows the
fixed path it shows for every other unconfigured provider.
The Apple Health card listed each type's newest sample. A type the phone
only hands over on "Sync all" can carry a reading from this morning and
never have arrived in the background, so the sample time could not answer
whether a type flows on its own.
The batch route's per-type ledger now records, in one upsert per batch,
when each Apple Health type last arrived, the trigger that carried it
(including the new `manual`), and when a batch last brought a new or
updated sample of it. Duplicate-only re-sends move the arrival but keep the
last new value; scoped credentials and MANUAL rows write nothing, the same
rule as the sync checkpoint. A ledger failure leaves a warn line and never
fails the ingest.
`GET /api/integrations/healthkit` lays `lastReceivedAt`, `lastTrigger` and
`lastNewSampleAt` over each `metricFreshness` entry (null for a type no live
sync has carried, absent on the workouts entry and when the ledger read
failed), and the card shows them as a second line per type.
The per-type newest-sample read walks live types through
`listLiveMeasurementTypes` and probes each type's newest row instead of
grouping every live row of the source.
Land every file the v1.42 work shares before any of it is built, so the
parts built side by side never edit the same file.
Schema and migrations 0379-0386: the HEALTH_CONNECT measurement source,
ImportJob.kind, the air-quality columns and the per-user switch on the
environment tables, sealed environment locations with nullable plaintext,
managed-profile handover links, the per-type HealthKit arrival ledger, lab
unit-conversion provenance, and user-defined vaccines. Every new column is
annotated for the column-reader sweep.
HEALTH_CONNECT reaches every exhaustive list: the source labels, the
read enum (not the write allowlist), the source-priority ladder cap, now
the number of sources rather than a fixed eight, and the write-path smoke
test. RMSSD gets its own priority class and a 300 ms ceiling. The batch
syncTrigger gains `manual`, with its label on the Apple Health card.
Queues for the Health Connect import, the compaction-tombstone purge and
the measurement-table maintenance are registered with stub handlers;
host-metric sampling moves to every five minutes. Environment and work_mem
variables reach the compose whitelist, the production example and the
manifest. The proxy admits the handover link and redirects it at the edge,
and the Health Connect upload bypasses the proxy body limit.
The new routes answer 501 behind their final authentication and are
published in the contract; backup, wipe and key rotation classify the new
models and columns; the seven message bundles carry the base copy. Guards
that list release-pending keys and columns fail once an entry is wired, so
those lists empty themselves before the tag.
Also re-anchor a login-throttle fixture to the wall clock, where its
expiry is compared, and re-base the aggregate bundle cap on a measured
12 KB of new locale strings.
The compose file passes every whitelisted variable through, so an unset
redirect URI reaches the process as an empty string. Withings then sent
redirect_uri= to the provider, and Fitbit and Google Health refused to start
the connection as not configured although NEXT_PUBLIC_APP_URL was set. Every
integration now treats an empty or blank value as unset and derives the
callback from the app URL.
Thirty-one files carried internal build vocabulary and anchors pointing at
notes that are not in the repository: a reader of `checklist.ts` in a year
would find its rationale keyed to a research id they cannot open, inside a
comment that also names an internal build stage. Two test titles carried the
same anchors.
The version tags and the whole of the reasoning stay; only the vocabulary and
the dangling references go, with the few sentences that were built around an
anchor rewritten to stand on their own.
The ledger's synthetic "no row" snapshot returned `state: "connected"`, so
the field's default meant the opposite of its name: every account that had
never connected a provider published `connected` on the status envelope.
Settings survived it by never reading `state`; the setup flow's connect step
did read it, and told brand-new accounts their wearable was connected.
The snapshot now reads `unknown`. `syncHealth.verdict` is unchanged and stays
the only liveness truth; the managed-record projection maps `unknown` onto
`disconnected` the way it already maps a missing connection.
The v1.38.0 notice that warns when the configured OAuth callback origin
differs from the origin the browser is on never rendered in the published
image, and the copyable callback address in the redirect guide showed a
bare path. `integrationCallbackUrl()` read `process.env.NEXT_PUBLIC_APP_URL`
inside a "use client" module. Next.js inlines NEXT_PUBLIC_* into client
bundles at build time, and the GHCR image is built without that variable
(only the version, build-sha and built-at args reach the build), so the
base compiled to an empty string, `new URL("/api/<provider>/callback")`
threw, and the notice returned null on every card for every self-hoster.
The value now comes from the server at request time. A new map
(`src/lib/integrations/callback-urls.ts`) calls each provider's own
redirect-URI function (the four private `getRedirectUri()` helpers are
exported under provider-specific names; Polar, Oura and Strava already
exported theirs), so the page shows exactly what the handshake registers,
explicit `*_REDIRECT_URI` override included. The settings route resolves
the map and passes it as a prop through `IntegrationsSection` and
`ConnectionsPanel` to every card. Fitbit and Google Health validate their
URI and throw on a bad configuration; that surfaces as `null`, the notice
skips the comparison and the guide falls back to the fixed path.
Tests: a map test proving the explicit override wins and the derived form
is used otherwise, a component test rendering the notice from a passed
value (mismatch renders, match does not), and a structural guard that
fails when any "use client" file under src/components/settings/integrations
reads NEXT_PUBLIC_APP_URL.
Each of these made the server report something that was not so, and each now
has a test that fails without the fix.
The Withings credential DELETE wrote no audit row and left the ledger at
whatever state it last held. Fitbit, Google Health and WHOOP all park theirs,
and three of them say in their own comments that they do it for parity with a
list that never included Withings. A client reading `state` after the teardown
was told `connected` about a connection that no longer existed.
Oura's and Strava's credential DELETEs audited only when an access token
happened to be present, so removing a stored pair that had never completed
OAuth left no trail at all. Polar's twin has always done both unconditionally;
they match it now. The old Oura test asserted the skip, which is why the gap
survived a rewrite — that expectation is inverted here, with the reason next to
it.
The Withings and WHOOP manual syncs had no rate limiter, while `fullSync` walks
full history against a per-user provider budget. The Fitbit route's own comment
records that it was the outlier until it got one; these two were never brought
along. Both now carry the same 5/60s baseline and 1/hour full-sync buckets as
their siblings.
Every sync route parsed its body inside a catch commented "no body provided",
which was true for one of the three ways to reach it. An absent body still
means an incremental run — that is the documented shape and the web cards send
it. A body that is present and unparseable is now 400, and one that fails the
schema is 422: `{"fullSync": "true"}` used to be read as false and answered 200,
so a client that asked for full history and mistyped the value was told the run
it never requested had succeeded. Unknown keys are still ignored. The four
copies collapse into one helper.
The consolidated integration status swallowed a failed freshness query into an
empty array, which is exactly what a record with nothing in it also produces —
so the signal built to say "this metric has gone quiet" answered "nothing has
gone quiet" when it had failed to look. `metricFreshnessDegraded` separates the
two, as one flag rather than one per entry, because it is a single query for all
eight providers.
The contract follows the behaviour in the same commit.
Refs #778. Two honesty gaps from the first-run report: the web onboarding
presented Apple Health like a web connector although the connection only
completes in the iPhone app, and the initial backfill gave no sign of
progress, so a working first sync looked broken.
The onboarding source step now says plainly that Apple Health is set up in
the iPhone app after onboarding, without dead-ending the flow. The
integration card gains a server-authoritative progress summary — rows
accepted, oldest reading reached, and an honest flowing/waiting line
derived from the last accepted batch. Nothing the server does not know is
shown: no percentage, no ETA, no invented throttle state; a quiet stretch
reads as waiting for the phone, not as an error.
Marc-André Bombeck