Behavioural cases first: the Google mapper keeps the raw `exerciseType`
when the sport falls back to "other", reports a missing one as null
rather than an empty string, and the ingest writes it under
`metadata.googleExerciseType` on both the insert and the update arm — the
update is what a full re-sync goes through, so a row written by an older
build repairs in place. The key is omitted rather than nulled when a
session carries no type, so a re-sync cannot wipe what an earlier one
recorded.
Then the structural guard, because the behavioural ones only cover the
provider that was reported. It freezes the set of sport-label mappers and
the set of files creating Workout rows, and asserts every mapping ingest
carries its raw label into metadata. Fitbit has the same gap and is
excluded on purpose — it ships on its own, small and isolated, per the
standing rule for that provider. The exception is an assertion, not an
omission: when Fitbit gains the write, the guard fails and forces the
entry to move. An exception nobody has to retire is how this gap survived
in the first place.
An unmapped Google `exerciseType` resolves to the literal "other", and
until now that was the end of the label: nothing else carried it, so a
workout Google typed as a sport the map does not know became
indistinguishable from a genuine `OTHER_WORKOUT`. Carry the trimmed
upstream value through `GoogleHealthMappedWorkout.sportTypeRaw` and write
it to `Workout.metadata.googleExerciseType`, the shape Polar, Strava and
WHOOP already use. Insert and update share one payload, so a full sync
repairs rows written by an older build.
The comment above the type map claimed an unmapped type "still persists"
because the column is free-text. It does not, and it did not when the
comment was written. Prose that asserts what the code does not do is the
same defect as the missing write, so correct it here rather than later.
Every ingest path in the product can now emit a typed, salience-classified
event into pg-boss, consumed by one worker that claims a durable per-day
reaction marker and fans out to the reaction surfaces.
The recency classifier runs at the seam, before anything is enqueued, so a
mass backfill — a multi-year Apple Health export, a provider re-sync, a
30-day catch-up — emits nothing at all. Non-salient writes are annotated and
dropped.
The worker makes no provider call by construction: it imports no provider
module, writes only markers, and enqueues nothing that is not budget-gated
downstream. Business refusals return a status rather than throwing, so a
refusal is never retried against a ceiling that will not move until the local
day rolls over.
Seams: sleep (one hook, inherited by all seven transports), workouts (the
batch route, the four provider syncs, the export importer), weight and blood
pressure (both measurement routes), labs (manual, OCR commit, document fact
commit).
The provider workout syncs could not previously tell a created row from an
updated one, so a re-sync of a recent session would have re-fired on every
poll; they now recover the distinction from the row the upsert already
returns.
Reaction lines are AES-256-GCM at rest, following the InsightNarrative
precedent, and the column is registered for key rotation. Markers are pruned
after 14 days by a daily maintenance job.
The Today digest went stale after logging a dose: nothing invalidated the
`["daily","digest"]` query, and it reads meds from the same server snapshot
the intake routes already hard-evict — so the "dose due" rail item lingered
until a hard reload. Add the digest key to the medication-dependent bundle and
force the inactive digest (unmounted while the user is on the medication card)
to refetch alongside the dashboard snapshot; the server snapshot eviction seam
is reused, so the refetch returns post-write dose state immediately.
Remove the score-ring cluster from the Today hero — it read as uneven and
wasted tile space. The health-score ring is the hero's only ring now; the
score-ring selection contract stays server-side for the native client. Tighten
the hero's vertical rhythm so the worth-a-look rail sits close under the lead
instead of floating below a half-empty column.
Wire the remaining sleep transports into the morning-refresh trigger: Google
Health, Fitbit, Oura, and Polar now enqueue the debounced morning refresh on a
last-night segment landing, matching the Withings / WHOOP / Apple seams — those
sources had no trigger at all. Make the trigger observable: each early-return
gate now annotates a distinct `daily.morning_refresh.skipped` reason (the
not-last-night gate also carries the samples' resolved local dates, today's key
and the tz) so a real landing shows exactly which gate fired.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018V8V9qapgHi7fhgtMJ8SkM
The Local provider speaks the standard JSON wire: response_format by default,
one self-healing fallback per endpoint when a server rejects it (dialect
cached per base URL) — LiteLLM, OpenRouter, vLLM, LM Studio and plain Ollama
all work from the same form, and the form + provider docs now say so. A
gateway that shims a Claude-family model behind a synthesized tool call is
parsed from the tool-call arguments instead of yielding silently empty
insights.
The insight token budget is configurable (INSIGHTS_MAX_TOKENS, default 2500,
clamped 500-8000) and unified across all six call sites; a reply cut off
mid-JSON returns a distinct truncation error instead of a generic parse
failure. A briefing withheld by the number-grounding check now says so on the
card instead of pretending nothing was generated. The provider test button
answers 4xx with a bad_request category instead of claiming the endpoint was
unreachable.
Sleep repair follow-up: measurements carries a second full unique index — the
natural key (type, measured_at, source, sleep_stage) — so a key-format
migration could tombstone the old rows while their re-keyed replacements
collided with that index and were silently dropped (live incident: a
full-history re-sync erased a user's sleep). Planned creates now run a
natural-key rescue probe: a match, live or tombstoned, is updated in place —
fresh value, the new externalId, deletedAt cleared. A rescue-probe failure
holds the watermark so nothing is lost past the overlap.
Platform-wide hardening: the failure classes found live this week hunted down
across every integration.
Fitbit carried the same tombstone wedge fixed for Google — probe now matches
soft-deleted rows and the update resurrects them; the comment describing a
partial unique index that never existed is corrected. Sleep segment ids are
stable across re-scoring for Withings (session+segment-start), WHOOP/Polar
(one segment per stage, no positional index) and Oura (run-start keyed), each
with a session-scoped sweep that soft-deletes rows a re-score orphaned — so
legacy and affected nights heal as they are re-read.
One platform rule for deleted rows: source-owned readings resurrect on
re-import (Withings, WHOOP, Oura, Polar, Nightscout, mood webhook, CSV
re-imports, Apple stats day-totals); Apple sample-grain deletions stick per
the paired-client contract, and the mood bulk route's tombstone match is now
a true no-op reported as duplicate. Import counts reconcile against actual
createMany counts.
Google lifecycle: a dead token registers on the cycle's hard-fail ledger, so
an expired connection parks instead of stamping success hourly; backfill and
sleep-repair markers stamp only after a clean pass (pg-boss retries are now
real); failed measurement writes hold the watermark; no-op updates are
skipped; mapping.md tells the current truth.
Scale: doctor-report reads only consumed columns and enabled sections;
exports and backups keyset-paginate; achievements tally via SQL buckets;
series charts day-bucket dense kinds beyond 90 days; BP pairing is a linear
two-pointer; distinct-type scans run server-side; the graded-series cold
fallback is bounded; the fitbit defer tracker loops instead of spreading.
The measurements unique index on (userId, type, source, externalId) is FULL —
it covers soft-deleted rows — so a tombstoned Google row permanently owned its
key: the live-only probe treated it as absent, planned a fresh insert, and
createMany's skipDuplicates dropped that insert silently against the
tombstone's key. Every sync, forever. Live incident: a self-hoster's step days
were stuck exactly this way (five tombstoned dates, confirmed in their data).
The probe now matches tombstoned rows too and the update branch carries
deletedAt: null unconditionally — a no-op on a live row, a deliberate
resurrection on a tombstoned one. Google remains the source of truth for its
own rows; removing Google data permanently means disconnecting the
integration. The stale 'partial unique index' comments are corrected.
Google Health: the fullSync rollup-defer tracker collected its touched keys via
a single spread push — one argument PER READING, so a multi-year heart-rate
history (six figures of samples) overflowed the call stack and aborted the
metrics collection mid-cycle, exactly on the accounts with the most data. A
plain loop replaces the spread. Reported by a self-hoster; the v1.28.21 sync
diagnostics confirmed every other read on that account healthy in one log line.
Documents: the vault card's read-by-AI tag no longer takes its own line — a
small glyph rides inline on the date-and-size row. The detail view drops the
status bar under the preview entirely; with automatic reading on and nothing to
review the whole block collapses, so the content starts straight with the
document's fields. The provenance contract moves with it: the e2e specs now pin
the card marker instead of the sheet pill, and the unmounted status-pill
component is removed rather than left dead.
A one-time background pass re-reads each connected account's Google Health
sleep history so nights stored twice before the sleep fix collapse to their
true total (new sleep_repaired_at marker, migration 0238; watermark-safe,
enqueued once at boot per un-repaired connection).
The sleep night read reports when two writers disagree about the same night —
the served total is unchanged, the disagreement rides along as an annotation
(sourceDiscrepancy on the night DTO) so clients can mark the number instead of
silently picking a side.
A GLP-1 medication whose weight has plateaued on the current dose says so next
to the estimated drug level (efficacy tab + insights) — a factual note from the
existing detector, association only, verb-guard covered.
Google Health diagnostics: the sync log now separates a daily-totals read that
returned nothing from one the importer could not parse
(googleHealth.rollup.emptyResponse vs droppedAll), and the connection test's
structure probe reports the raw response envelope key names when a rollup type
parses to zero — a per-account naming drift becomes visible from one report.
Documents vault card: date + size and an AI-read marker now sit directly under
the title instead of the meta line running off under the thumbnail.
Document detail view: a larger close control replaces the small X and the
redundant top-right shortcut; the inline read-with-AI row under the preview is
gone (the suggest / summarise / show-text actions stay); a button to ask about
the document sits next to share and download.
Mental-wellbeing cards no longer wash the whole tile on hover — only the title
is the open target, matching the medication cards.
Medication API settings render inline instead of behind an expander that never
had anything beside it.
Google re-scores a night after the fact, and the sync re-reads recent nights to
pick that up. The per-segment sleep externalId keyed on a volatile anchor (the
session end instant plus a positional index), so a re-read minted fresh ids and
landed as a SECOND parallel copy of the night instead of overwriting the first —
the night-total then summed both (a real 7h35 night could read as 10h+).
Each segment now keys on a stable identity: the session's own resource id (as the
workout mapper already does) plus the segment's own start, dropping the shifting
end anchor and the positional index. A mere re-classification of a block now
updates in place. And before writing a re-read night, the sync soft-deletes any
superseded rows left in that night's window (bounded to the session's own span,
never a fresh row), so the night collapses to one canonical set.
Nights already stored with duplicates heal as they are re-read within the sync
overlap; scripts/repair-google-health-sleep.ts re-syncs the full history at once
for older nights (dry-run by default, --run to execute).
Fitbit's handleCollectionFetchError soft-skipped a 403 but RETHREW any
other hard failure, aborting every endpoint ordered after the bad one in
the same resource and stalling the watermark so the whole integration
stayed dead. Record hard failures on a per-cycle AsyncLocalStorage ledger
and return 0 instead of rethrowing — the sibling endpoints keep fetching,
and the ledger still fails the cycle so success is not stamped and the
next tick refetches. Also wrap the per-metric map fan-out in the same
per-collection catch so one malformed point can't skip its siblings.
Full-catalogue coverage pass against the v4 data-types index. Three types
map onto existing MeasurementTypes and are now fetched in the metrics
bundle:
- blood-glucose -> BLOOD_GLUCOSE. `bloodGlucoseMilligramsPerDeciliter` is
already the canonical mg/dL storage unit; meal/timing/specimen enums are
not stored.
- core-body-temperature -> BODY_TEMPERATURE. `temperatureCelsius`, the core
slot — distinct from the surface-temperature buckets.
- daily-sleep-temperature-derivations -> WRIST_TEMPERATURE. The documented
`nightlyTemperatureCelsius` is an ABSOLUTE reading (the mean of the
night's skin-temperature samples), which refutes the earlier deviation-
only premise behind the launch omission. The baseline/stddev derivations
are not stored — the user's own series carries the baseline.
Every other catalogue type is now an explicit documented skip in
mapping.md with its reason (per-sample twins of the chosen daily grains,
zone-minutes types without a HealthLog concept, total-calories folding in
BMR, hydration/nutrition without a module, ECG/IRN behind their scope
policy). The structure probe iterates the type table, so the three new
types are probed automatically.
Daily HRV and resting heart rate returned HTTP 200 with zero imports on a
live Fitbit-band account. Full reconciliation against the v4 reference
(data-types index, DataPoint schema, list/reconcile/dailyRollUp pages)
surfaces three defects:
- The daily-summary `.date` filter prefix is the docs' own contradiction:
the data-types index's filter-parameter column says snake_case
(`daily_heart_rate_variability`), while the list reference's only worked
daily example is `dailyHeartRateVariability.date < "2024-08-15"` — the
camelCase payload key. Live, the snake form is accepted (200) but matches
nothing. Send the worked-example camelCase form first and re-walk once
with snake_case if the very first page 400s; annotate the accepted style
(`googleHealth.dateFilter.style`) and surface it through the structure
probe as `requestShape` so a live account settles the grammar.
- `daily-respiratory-rate` reads `breathsPerMinute` (a plain number); the
previously assumed `dailyRespiratoryRateBpm` leaf does not exist, so
RESPIRATORY_RATE never parsed a single row.
- `height` carries `heightMillimeters` (int64 string), not `heightMeters`;
the heightCm profile seed never fired. Convert mm -> cm.
Also cover the documented Exercise.ExerciseType catalogue in the sport map
(MOUNTAIN_BIKE, TRAIL_RUN, TREADMILL_WALK, STATIONARY_BIKE, SWIMMING_POOL,
FREE_WEIGHTS, CROSSFIT, TABATA_WORKOUT, ...) — the enum's multi-word values
previously fell through to "other".
Tests pin the documented field names per fixture, the camel-first/snake-
fallback walk (including no-fallback for sample shapes and unfiltered
backfills), and the extended sport map. mapping.md records the filter
contradiction and the corrected leaves.
The live 400 on fetchSteps suppressed fetchDistance / fetchFloors /
fetchActiveEnergy entirely: handleCollectionFetchError rethrew anything
that was not a per-class 403, and the rethrow escaped the rollup loop in
sync-activity, emptying the whole cumulative group after the first bad
collection.
- handleCollectionFetchError records the classified failure, warns with
the per-collection detail, and returns instead of rethrowing; every
call site already sits in a per-collection catch, so the sibling
collections keep fetching.
- A per-cycle hard-fail ledger (AsyncLocalStorage, mirroring the 403
soft-skip tracker) still fails the cycle: the run counts as a partial
error, the watermark is not stamped, and the next tick refetches the
window - a broken collection cannot silently advance past its gap.
- The orchestrator's per-resource warning names the resource explicitly;
fn.name minifies to a single letter in the production bundle (the live
log read "google-health c failed").
- Tests pin the no-rethrow contract per classification and the activity
loop reaching all four rollup types plus VO2 max after a steps failure.
The first live sync on v1.27.4 imported every list-based type but died on
the first cumulative chunk: fetchSteps(chunk=0,page=0) returned
HTTP 400 INVALID_ARGUMENT "Invalid argument in request.".
Re-verified the request contract against the v4 reference:
- Request body (users.dataTypes.dataPoints/dailyRollUp): range
{ start/end: CivilDateTime }, windowSizeDays, pageSize, pageToken,
dataSourceFamily - every field the client sends is legal under that
exact name.
- CivilDateTime (inlined on the DataPoint page): nested
{date:{year,month,day}, time:{hours,minutes,seconds,nanos}} objects.
- range: "Required. Closed-open range of data points that will be rolled
up. The start time must be aligned with the aggregation window. The
maximum range for calories-in-heart-rate-zone, heart-rate,
active-minutes and total-calories is 14 days. The maximum range for
all other data types is 90 days." - steps / distance / active-energy /
floors sit in the 90-day class, so the 90-day chunk constant stands
(the in-file "14-day-max" note applies to heart-rate only).
- The worked dailyRollUp example (developers.google.com/health/endpoints)
rolls up ONE civil day as start = {date: day, time: 00:00:00} and
end = {date: SAME day, time: 23:59:59} - the end bound names the last
covered day, not the next day's midnight.
The previous body sent end = {date: <next day>} with no time object, so
a maximal 90-day chunk touched 91 civil dates - the one divergence from
the documented example, and the prime suspect for the 400 (the docs do
not spell out how the cap counts days).
- buildDailyRollUpBody() pins the documented shape: explicit date+time
on both bounds, end = last covered day at 23:59:59, windowSizeDays: 1,
no pageSize (the documented default of 1440 covers the <=90 windows a
chunk can produce), pageToken threaded on follow-up pages.
- fetchDailyRollUp() keeps 90-day chunking; if the very FIRST request of
a walk is rejected with a 400 it re-walks the range once at the most
conservative documented span (14 days, the tightest cap the docs state
for any type) and annotates googleHealth.rollup.shape with the shape
that succeeded, so the next live sync reports which constraint binds.
- extractGoogleApiErrorDetail() surfaces AIP-193 error.details[]
BadRequest fieldViolations as "field: description" fragments (field
paths and descriptions only, redacted and capped at three), so the
next 400 names the offending request field in the logs instead of the
bare "Invalid argument in request.".
- mapping.md documents the verified range semantics; unit tests pin the
exact body JSON, the 90/91-day boundary, the fallback walk and the
violation extraction.
The previous notes carried two wrong premises: that every launch type
supports dataPoints.list, and that the response payload is snake_case.
Document the three per-type encodings (kebab path, snake_case filter,
camelCase payload union), the int64-as-string fields, the per-shape
filter grammar, the dailyRollUp read for the cumulative activity types,
the corrected daily-* data-type ids, the sleep end-time filter, the
exercise metricsSummary payload, the AIP-193 error surfacing and the
structure probe.
- On a non-2xx data-point response, extract the AIP-193 error envelope
into a redacted STATUS: message detail (bearer tokens stripped, URL
query strings dropped, 200-char cap) and thread it into the thrown
GoogleHealthApiError plus the external-call annotation, so a
field-grammar 400 is diagnosable from operator logs. The OAuth token
endpoint keeps its flat invalid_grant handling.
- POST /api/integrations/google-health/test with {"probe":"structure"}
now fetches one recent page/window per data type and returns only the
payload STRUCTURE (field names + typeof leaves, never values) plus the
per-type error verdict — paste-safe diagnostics for payload-shape
drift. Same auth and rate limit as the plain connection check.
The v4 client read none of its documented response shapes: the DataPoint
value is a camelCase union keyed by the camelCase type name with camelCase
nested times, proto3 int64 fields arrive as JSON strings, and four fetchers
targeted wrong data types or illegal filter fields. Every HTTP-200 type
parsed zero measurements and four fetchers returned HTTP 400.
- Read every value/time path in camelCase; keep snake_case only inside the
filter parameter. Pin path/filter/payload-key triples per data type.
- Coerce int64 JSON strings in every numeric extractor.
- weight.weightGrams -> kg, height.heightMeters -> cm,
distance millimetres -> m, bodyFat.percentage, heartRate.beatsPerMinute.
- Move SpO2, HRV, respiratory rate and VO2 max to their daily-* data types
(the bare sample types reject a .date filter).
- Move steps, distance, active energy and floors to POST :dailyRollUp with
windowSizeDays 1 (list returns minute-grain buckets, not daily totals;
floors has no list method), chunked at 90 civil days, day-keyed on
civilStartTime.date at the UTC-midday anchor.
- Filter sleep on interval.end_time (the only legal sleep time field) and
anchor sessions on interval.endTime; filter exercise on
interval.civil_start_time with an offset-less civil bound in the user's
zone; read exercise metrics from metricsSummary and take the workout id
from the DataPoint name.
- Seed height from max(sampleTime) explicitly - list order is descending,
so last-row-wins picked the oldest sample.
- Rewrite the mapper fixtures to the documented shapes.
Tighten the transport so device data lands on the right day and in the
right slot.
- Cumulative daily totals (steps, distance, energy, floors) now key their
measuredAt and stats:<tag>:<day> external id off the interval's civil
start date anchored at UTC-midday, not the physical instant. Keying off
the physical instant put a positive-UTC-offset user's day one calendar
day early and split the same local day across the Google, Apple, and
Fitbit sources so neither won the per-day source pick.
- Thread the user's timezone through the sleep, workout, and interval
parses so an offset-less wall-clock timestamp resolves in the user's
zone rather than the process zone — the same fix the classic Fitbit
sleep path already carries.
- Drop skin temperature from the launch set. Google reports a signed
nightly deviation from baseline, not an absolute reading, so mapping it
into the absolute wrist-temperature slot stored a delta as a
temperature and dropped every cold-night negative. It returns once a
signed-delta model exists.
- Remove the unused daily-rollup read path and field-map, and the
experimental ECG and irregular-rhythm scopes that had no reader — the
connect flow now requests exactly the four core scopes it uses.
Add the transport that reads a user's device data from the Google Health
API (health.googleapis.com/v4): Google OAuth with PKCE and offline
access, the paginated dataPoints reader with a daily-rollup fallback for
the types that lack a list method, the metric/activity/sleep/workout
mappers onto the shared measurement shape, and a response classifier
that separates retryable errors, hard rejects, and the re-consent case.
Access tokens live an hour and refresh tokens are reused, not rotated. A
refresh that comes back invalid_grant raises a distinct reauth-required
signal so the app can prompt a fresh consent rather than fail silently —
the periodic re-consent a Testing-mode Google client needs. The four
core readonly scopes ship by default; the Pixel Watch ECG and
irregular-rhythm scopes stay behind an opt-in flag.
Only the transport is new: the mapped-measurement output, the
upsert/dedup/rollup tail, and the source ladders are the shared ones.
Marc-André Bombeck