Withings, Fitbit, Google Health, Polar and Oura wrote each daily total and
daily summary at noon UTC of its date. From UTC+12 on (New Zealand all year,
Fiji, Tonga, Samoa) noon UTC is already the next local day, so every synced
step count, SpO2 or resting-pulse day showed up a day late. The rows now sit
at local noon of their date in the user's zone, and the fetch windows run
through the user's own today.
Fitbit weigh-ins carry a local wall clock without an offset and were read
as UTC; they are read in the user's zone now.
ExternalIds stay on the calendar date, so the next sync finds each row and
moves it in place instead of adding a second one, and the day a row leaves
is refolded. Rows inside each integration's re-read window heal on the next
sync; Fitbit and Google Health re-read the whole history on reconnect.
Both hourly sync tails left the per-user analytics, correlations,
targets, derived and achievements caches serving pre-sync bodies until
their TTL lapsed — the correlations route documents the exact invariant
these tails were breaking. Mirror the Fitbit tail: mark the cells stale
whenever rows landed, on partial failures too.
Every interactive path, every import, MCP and Telegram checked an incoming
measurement against VALUE_RANGES before storing it. The provider sync writers
did not, so a decode slip or a glitched sensor landed a row the application
itself calls impossible, and every mean, median and personal band derived from
it inherited the impossibility. Filtering the read side hid the row. It kept
landing.
The gate is one module reading the domain the app already declares, applied at
two kinds of seam. reconcileExternalMeasurement is the shared identity writer
for Polar, Oura, WHOOP, the Withings sleep leg, the Apple export importer and
the batch endpoint, so gating there means a provider added later inherits the
refusal rather than having to remember a check it never knew about; the new
rejected_range verdict carries no row, which is what makes the compiler force
every caller to handle it. The five writers that hold Prisma themselves gate at
their entry, before the existence probe decides insert versus overwrite, so an
impossible reading can no longer replace a good stored value.
An out-of-band value is dropped. Not clamped, because a clamp invents a reading
nobody took. Not stored behind a flag, because a quarantined row is still a row
and every consumer would have to learn the flag. Not fatal to the batch, because
one bad sample must not strand the run's other readings behind a held watermark.
Absence reads as absence.
A silent drop, though, is indistinguishable from a provider that sent nothing,
so every drop is tallied on the wide event under measurement.range_rejected with
the metric, the provider and the edge it broke. Counts only: the reading itself
is health data and never enters the log line, and the direction plus the
metric's own declared bound is what a diagnosis actually needs. The tally
accumulates rather than replacing, because a provider gates once per resource
leg and the reconciler once per row. One warning per provider lifts the event to
warn so a drop does not hide inside a green line.
A structural guard enumerates all twenty measurement writers with the reason
each is allowed to write, so a new one has to answer the question before it
ships. scripts/diagnose-implausible-measurements.ts reports the rows already
stored, aggregated per metric and source with no value beyond the bucket's own
extremes. It repairs nothing: whether a stored impossible row is a decode slip
or a band that is too narrow is not a call a script gets to make.
A round trip through the real handlers: post a recording, read it back
through the list and the detail routes, and check what a client actually
receives rather than a fixture rebuilt at each end. Post it twice and
the second call says updated, not inserted, and the table does not grow.
Feed the same strip to the export-archive importer afterwards and it
says skipped; feed it to the importer first and the route says
duplicate. Both directions leave one row.
The migration test drops the index, manufactures the collisions the
resolution step exists for, and runs the migration file through the same
executor the deploy uses. It checks which row survived each tiebreak,
that the salvaged event link followed the keeper, that rows differing
only by user, source or sampling rate were left alone, and that the
constraint rejects a new twin afterwards. Applying the migration to the
empty testcontainer proves none of that.
The Withings ECG tests now hand the transaction the same delegate the
shared writer uses, so they still watch the real write instead of a stub
standing in for it.
Six provider syncs returned a bare count, which narrowed their failures away:
a Nightscout window in which every SGV row was refused by a constraint
returned the same 0 as a quiet night, and an Oura or Polar run whose
collections all failed returned 0 too. The routes then answered 200 with a
count and the settings card drew a green tick over it.
Every sync now reports what it imported AND whether any part of it did not
land, and the route resolves the outcome server-side so the web card and the
native client cannot disagree. Google Health keeps its 502 for a run that
failed and wrote nothing, but a run that failed after writing some resources
is now a partial rather than a discarded result. The labs OCR commit carries
the same resolved outcome.
A sync failure the classifier bucketed as `transient` had no exit. The park
test read `kind === "persistent"` and the persistent streak anchor was only
ever stamped by a persistent failure, so a streak could run for a thousand
attempts with the persistent counter at zero, the state frozen at
`error_transient`, and no ladder it could climb. `transient` is a hypothesis
about retryability, and the classifier defaults every timeout, 429, 5xx and
unrecognised response to it precisely because the wire cannot tell a reboot
from a permanent break. That default is right for one attempt and wrong for a
thousand: an unbroken streak retried hourly for a full day has refuted it by
experiment. Record it as `persistent` from there, which feeds the park ladder
that already exists — 48 h of unbroken failure end to end before an
integration stops attempting.
The counters could not accumulate anyway. `IntegrationStatus` is keyed
(user, integration) while Withings runs four legs on four crons and WHOOP four
resources on four more, and a success cleared the error, every bucket, the
streak anchor and the state whatever leg had recorded them. The ECG tick at
:41 wiped what the sleep tick recorded at :15 — including a reauth park — so
the strike ladder was reset several times an hour and the card painted a green
"connected · 12 minutes ago" over a data class that had stopped arriving. A
success now clears only what it is entitled to clear.
Two columns carry it: `failing_since_at` dates the streak, `failing_leg` names
its owner. The migration backfills the anchor for rows already in an error
state so a connection broken for weeks escalates on its next attempt rather
than restarting its clock.
Add an hourly ECG catch-net cron so a watch-only account with no scale pulls its
recordings without a webhook: the queue was registered and bound but never
scheduled, and the handler now grows a cohort arm that walks every connection on
a payload-less tick, mirroring the activity and sleep handlers.
Make the activity and sleep write paths honest. A swallowed activity batch-write
fell through to the success stamp — a green pill over lost data, green forever on
a persistent write bug; it now records a transient failure and refuses the
stamp. A reconcile throw out of either sleep transaction propagated past the
stamp and was swallowed by both callers, leaving neither success nor failure
recorded; the source now records a transient failure and rethrows so the cron
and webhook paths stay honest without changing their warn-and-continue
behaviour.
Four verified HIGH data-integrity fixes, each in-place per-provider (no
shared helper), from the campaign's data-integrity audit + adversarial
verification:
F1 — drain-per-sample-cumulative buildScanWhere was the only one of four
sibling drains missing deletedAt: null; the nightly cumulative fold
re-summed soft-deleted Apple samples and hard-deleted the legacy-steps
audit tombstones. Added the filter (mirrors consolidate-daily-mean,
dense-intraday-retention, consolidate-legacy-steps).
F2 — the additive late-fold merge had no single-flight guard, so two
overlapping runs (deploy overlap, or the admin endpoint racing the cron)
produced a permanent 2x daily total. Added a pg_advisory_xact_lock per
(user, externalId) plus an in-transaction re-read of the live
contributors so the serialised-second fold contributes zero (a bare lock
alone would still re-add its stale scanned sum). Mirrors the documents
quota advisory-lock pattern.
F3 — Withings sleep externalId keyed on segment START while measuredAt +
the natural key use END; a re-scored segment missed the externalId probe,
the sweep tombstoned the old row, and the re-keyed insert collided on the
natural key and wedged. Added the natural-key rescue (re-key in place),
mirroring the same-file measure path.
F4 — Withings measure sync stamped lastSyncedAt + recordSyncSuccess
unconditionally after a warn-only per-row loop; a transient write failure
stranded the reading behind the 10-min overlap forever. Now holds the
watermark and records failure on a hard row error (mirrors
google-health/fitbit), while a benign P2002 still advances.
Regression test per finding (reverting the three source files fails
exactly four tests). Integration suite green against real Postgres.
Platform-wide hardening: the failure classes found live this week hunted down
across every integration.
Fitbit carried the same tombstone wedge fixed for Google — probe now matches
soft-deleted rows and the update resurrects them; the comment describing a
partial unique index that never existed is corrected. Sleep segment ids are
stable across re-scoring for Withings (session+segment-start), WHOOP/Polar
(one segment per stage, no positional index) and Oura (run-start keyed), each
with a session-scoped sweep that soft-deletes rows a re-score orphaned — so
legacy and affected nights heal as they are re-read.
One platform rule for deleted rows: source-owned readings resurrect on
re-import (Withings, WHOOP, Oura, Polar, Nightscout, mood webhook, CSV
re-imports, Apple stats day-totals); Apple sample-grain deletions stick per
the paired-client contract, and the mood bulk route's tombstone match is now
a true no-op reported as duplicate. Import counts reconcile against actual
createMany counts.
Google lifecycle: a dead token registers on the cycle's hard-fail ledger, so
an expired connection parks instead of stamping success hourly; backfill and
sleep-repair markers stamp only after a clean pass (pg-boss retries are now
real); failed measurement writes hold the watermark; no-op updates are
skipped; mapping.md tells the current truth.
Scale: doctor-report reads only consumed columns and enabled sections;
exports and backups keyset-paginate; achievements tally via SQL buckets;
series charts day-bucket dense kinds beyond 90 days; BP pairing is a linear
two-pointer; distinct-type scans run server-side; the graded-series cold
fallback is bounded; the fitbit defer tracker loops instead of spreading.
Withings backdates some readings and lastSyncedAt advances on every
success (including a healthy 200-with-0 cycle), so a reading landing more
than 60s before the next cycle's now() could be missed. Widen the poll
catch-up overlap to 10 minutes — the upserts are idempotent, so re-touching
a handful of rows is harmless.
The activity sync ran a per-(entry, field) findFirst followed by an
update or create, roughly entries x 3 x 2 round-trips — about 2,190 on
a 365-day backfill. Flatten the entry/field grid into the concrete rows
the sync wants to land, resolve every overwrite-vs-insert decision in
one findMany keyed on the same four dedup columns, then run a single
createMany for fresh slots and one transaction for the changed-value
overwrites. A backfill is now a bounded handful of queries regardless of
the day count, and an unchanged re-sync is a no-op on the write path.
The overwrite contract is unchanged: an existing WITHINGS row for a
(type, measuredAt) slot keeps its row and only rewrites value; a fresh
slot inserts with the stable externalId. createMany uses skipDuplicates
so a concurrent sibling sync landing the same slot is tolerated by the
four-column unique.
The Heart sync recorded only the device's atrial-fibrillation verdict per
on-device ECG. Fetch each recording's full waveform via the Heart get
endpoint and store the raw micro-volt samples so a real trace can later be
rendered.
The sample array is sensitive health data: it is held AES-256-GCM encrypted
at rest in the new EcgRecording.waveformEncrypted Bytes column (fail-closed
crypto), with the sampling frequency, sample count, duration, lead, average
heart rate, and AFib verdict kept as plaintext descriptors. A re-sync upserts
on (userId, source, externalRecordingId) so a recording is never duplicated;
a waveform fetch failure never fails the verdict ingest. Rendering UI is a
follow-up.
The Heart List endpoint returns a per-recording AFib verdict from the
ScanWatch family that no sync fetched. Capture each recording as an
IRREGULAR_RHYTHM_NOTIFICATION event row carrying the device's own verdict
in rhythm_classification, mirroring the Apple Health categorical-event
treatment. Reuses the existing enum, column, and constraint, so no
migration. Fires on the measure webhook path, which shares the metrics
scope the Heart List endpoint needs.
The per-segment sleep sync wrote only the stage timeline; the per-night
summary's vitals — average HR, respiratory rate, SDNN-based HRV, average
SpO2 and the sleep score — were dropped at the fetch. Add a getsummary
fetch that maps each vital to its existing MeasurementType (RESTING_HEART_RATE,
RESPIRATORY_RATE, HEART_RATE_VARIABILITY, OXYGEN_SATURATION, SLEEP_SCORE), so
no migration and no client-contract change: the rows flow to the sync delta
under the WITHINGS source like every other server-side fetch.
Each vital upserts on the stable (userId, type, source, externalId) composite
keyed by the night's session id, so a re-sync overwrites rather than
duplicates. A summary fetch failure is isolated so the stage segments already
written survive. Fractional SpO2 is normalised to percent.
One-shot prettier pass so the new format:check gate starts from a clean
baseline. Formatting only — no behaviour change. Generator-owned and
vendored artifacts stay untouched via .prettierignore.
With measuredAt now stamped at the segment END, a night that Withings
re-aggregates between syncs shifts its enddate, so the measuredAt-keyed
lookup missed the prior row and the create then collided with the unique
externalId — leaving the night stuck at its stale value. Key the lookup on
the stable externalId and refresh value, measuredAt and stage on update.
Caught by the withings-sleep-sync integration test against real Postgres.
Withings sleep sync stamped each segment's measuredAt with its startdate,
but every reader treats measuredAt as the segment END and resolves the
span as start = end − duration. Each night was therefore rendered one
segment-length earlier than reality. Stamp the enddate so the span lands
at its true clock time.
iOS-operator-blocking hotfix for REG-11 — Home dashboard tile renders
neither chart nor latest value for BP / Puls / Körperfett when the
most recent reading is older than 7 days. Root cause was in
`/api/dashboard/summary` SQL gates; `latestIn7d` + `sparkBuckets` both
required `measured_at >= sevenDaysAgo`, which returned empty for
sparse accounts, leaving the iOS tile with `latestValue: null` +
`sparkline: []` to render from. Detail-screen
(`/api/measurements/series?kind=…`) escaped because of its broader
window.
Server-side changes:
- `latestIn7d` → `latestEver` (drops 7-day filter; DISTINCT ON returns
latest reading EVER per type, bounded at N_metrics rows).
- `sparkBuckets` rewritten via ROW_NUMBER() OVER (PARTITION BY type
ORDER BY bucket_start DESC) so the trailing N daily buckets per
type render regardless of age, bounded at SPARK_DAYS × N_metrics.
- BP + Pulse metrics.push gated on `latest || allTimeCount > 0` like
BodyFat already is — accounts with zero readings ever no longer get
empty placeholder tiles.
- 4 new regression tests in summary/__tests__/route.test.ts cover
sparse accounts, zero-data accounts, within-7d guards.
Other develop work bundled in this release:
- W4-QoL — copy nits.
- W5-WORKOUTS — user source priority threaded through write-time
picker + batch route.
- W7-WITHINGS-RELIABILITY — response classifier + sync hardening.
- W8-OPS — ops phase docs.
- B11 — NEXT_PUBLIC_APP_VERSION build-arg routing so /api/version
cannot drift from CI tag.
- env-check workflow — manifest / example lockstep enforcement.
Resolves PR #194 (squash-merge produced via local -X theirs strategy
because GitHub's UI couldn't auto-resolve develop's post-v1.4.42
contiguous work against the v1.4.42 squash on main; resulting tree is
byte-identical to develop's HEAD).
Six touch-disjoint waves landed on develop before this release:
- W1-KNIP — exports/types tiers flipped to enforcing-mode on main;
35 unused exports + 52 unused types triaged to 0/0; shadcn surface
+ zod-infer contract types scoped via knip.json ignore-block.
- W2-ZOD-MULTI-ISSUE — `returnAllZodIssues` shared helper +
`/api/dashboard/widgets` 422 now returns every Zod issue + worker
audit-ledger breadcrumb. 41 sibling routes documented for v1.4.43.
- W3-QUERYKEY-LONGTAIL — 40 settings/medications/admin/hooks files
migrated to the queryKeys factory; 9 new factory entries; ESLint
rule + test-guard scope extended in lockstep.
- W4-ARCH-HYGIENE — BERLIN_DAY_FORMATTER consolidated across nine
sites (seven status helpers + bucket-series + bp-in-target);
Suspense double-comment consolidated; doctor-report sanitiser regex
rewritten to escape-form so diffs are reviewable; pr-detection-
worker soft-delete filter; offhost-backup DR-intent comment.
- W5-IOS-WORKOUTS — `dedupeWorkoutBatch` write-time cross-source
dedup helper for the v1.5 iOS HealthKit observer ingest; dashboard
tile-strip placeholder parity with the live TrendCard chrome.
- W6-WITHINGS-OFF + ENV-CHECK — Withings off-response classifier
(transient/reauth/persistent) + WithingsApiError typed throw + the
`pnpm check-env` CLI. APNs `allOrNone: true` closes the AP-2
silent-disable gap the wave was conceived to close.
No migration, no env-var change, no API contract break for iOS v0.5.4.
The new `/api/dashboard/widgets` 422 envelope is additively extended.
pnpm test --run green at 4815 passing / 1 skipped.
pnpm typecheck, pnpm lint, pnpm knip (no --include flag) green.
Eleven touch-disjoint waves landed on develop plus a six-axis QA
reconcile round before tagging. See CHANGELOG.md for the user-facing
prose; this commit message documents the wave + report inventory for
release-management traceability.
Waves shipped (each with a phase report under .planning/):
- phase-W-POOL-v1440-report.md — p-limit(4) wrapper around the
analytics 15-way fan-out + pg.Pool max raised 10 -> 20. Closes the
v1.4.39 empirical-trace finding #1 (Prisma pool starvation).
- phase-W-DELETED-v1440-report.md — 11 reader-tier helpers + 3
integration contracts close audit Critical Finding #3 (soft-delete
invisibility) across dashboard, analytics, summaries, rollup tiers,
and the Coach snapshot path.
- phase-W-INSIGHTS-v1440-report.md — mood-rollup swap on the three
/api/insights/{features,targets,cards} routes plus the
compliance-rollup hook gap closure for bulk-projection mints; the
post-QA reconcile patch closes the six remaining measurement.findMany
sites in the cluster that still leaked tombstoned rows.
- phase-W-INFRA-v1440-report.md — rollup helper umbrella consolidation
under src/lib/rollups/* with zero orphan re-exports; lint, typecheck,
knip, and 93/93 rollup tests green post-restructure.
- phase-W-RSC-v1440-report.md — per-tile Suspense boundaries plus the
queryKey factory-bypass walker test (extended to comparison/ in the
reconcile pass).
- phase-W-PRIVACY-v1440-report.md — bilingual /privacy page with 17
paired-section tests pinning the 9 SB-3 disclosure requirements.
- phase-W-AASA-v1440-report.md — /.well-known/apple-app-site-association
endpoint pinned to S8WDX4W5KX.dev.healthlog.app with app-ID parity
assertions guarding against split rotation.
- phase-W-APNS-NOTIFY-v1440-report.md — MEDICATION_REMINDER
time-sensitive APNs payload (priority 10 + interruption-level)
scoped via a parameterised test that pins all 6 other event-types
do not bypass Focus. AP-2 .p8 install gates real delivery.
- phase-W-CONSENT-v1440-report.md — AI consent receipts CRUD with
append-only invariant pinned in test; revoke-then-regrant leaves
both rows; idempotent DELETE returns 200 + receipt: null.
- phase-W-WMY-WIRE-v1440-report.md — three previously-dead WMY readers
(week/month/year) wired through summaries-slice and
health-score-fast-path with a linear-composability parity test
pinning the cross-consumer routing contract.
- phase-W-GHOSTS-v1440-report.md — ~1 177 lines of dead code removed
across 9 atomic commits; startOfUtcDay + wallClockInTz consolidation
eliminates a sibling-drift risk; knip whitelist trimmed to three
files pending follow-up.
QA reconcile patches applied per the v1.4.40 QA round (six reports
under .planning/round-v1440-QA-*-findings.md):
- senior C1 — dashboard-suspense-boundaries regex aligned with the
post-W-INFRA Thread 2 userTimezone-local lift.
- senior H1 — soft-delete filter added to the six remaining
insights measurement.findMany sites.
- specialist H1 — Promise.allSettled around bulk-projection
compliance recompute on both /api/medications/intake and
/api/dashboard/summary.
- ux H1 — compare-toggle migrated to queryKeys.dashboardWidgets
factory; walker test extended to src/components/comparison/.
- security M1 — consent artefact 64 KB cap enforced via
Buffer.byteLength (UTF-8 bytes, not UTF-16 code units).
Quality gates green at tag: pnpm typecheck, pnpm lint, pnpm knip
--include files,dependencies,binaries,unlisted, pnpm test --run
(4726 passing / 1 skipped). Tests 4525 -> 4726 unit + 1 skipped over
the v1.4.39 -> v1.4.40 span.
User-facing prose: see CHANGELOG.md.
Hotfix on top of v1.4.39. Three measurement write paths (Withings
sync, `/api/import`, admin backup restore) were bypassing the
persistent rollup tier's write hook, so the dashboard chart's
`source=rollup` fast-path painted "Noch nicht genug Daten" on the
30 / 90-point range for accounts whose recent data only came through
those paths. The 7-day window kept rendering because the chart's raw
fetch never consults the rollup tier.
Closes the read-coverage gap by wiring `recomputeBucketsForMeasurement`
into every remaining ingest path and re-anchors the boot-time backfill
discovery to per-day coverage so already-deployed instances re-converge
on the next worker boot. The legacy `sum_value IS NULL` branch stays
on the same UNION so the v1.4.39 column-backfill continues working.
Also re-anchors one date-bound test in the medications-intake route
suite so the green CI run no longer depends on the wall clock.
Tests 4647 → 4648 passing; one long-standing skip unchanged.
Largest feature delta in the v1.4.x line. Insights expands into seven dedicated metric sub-pages; GLP-1 medication tracking lands end-to-end (drug-level chart with Research Mode acknowledgment, EMA titration ladder, cadence + compliance, side-effect taxonomy, pen/vial inventory); cross-source priority becomes a two-axis resolver; per-user timezone threads through ten analytics surfaces; Withings activity + sleep v2 sync; onboarding rebuilt as a nested-route wizard; Personal Records ship end-to-end; Health Score provenance accordion; Coach runs native first-party prompts across six locales with a 1800-assertion refusal-probe matrix; OpenAPI drift gate hard-fails; multi-arch GHCR image. Migrations 0043 through 0060 are additive and forward-only. PROMPT_VERSION 4.24.0 → 4.25.0 with GROUND RULE 9 (refuses dose recommendations) and GROUND RULE 15 (refuses drug-level estimates with MDR + MDCG 2021-24 cites).
See CHANGELOG.md for the full per-section delta.
Marc-André Bombeck