Both hourly sync tails left the per-user analytics, correlations,
targets, derived and achievements caches serving pre-sync bodies until
their TTL lapsed — the correlations route documents the exact invariant
these tails were breaking. Mirror the Fitbit tail: mark the cells stale
whenever rows landed, on partial failures too.
Every interactive path, every import, MCP and Telegram checked an incoming
measurement against VALUE_RANGES before storing it. The provider sync writers
did not, so a decode slip or a glitched sensor landed a row the application
itself calls impossible, and every mean, median and personal band derived from
it inherited the impossibility. Filtering the read side hid the row. It kept
landing.
The gate is one module reading the domain the app already declares, applied at
two kinds of seam. reconcileExternalMeasurement is the shared identity writer
for Polar, Oura, WHOOP, the Withings sleep leg, the Apple export importer and
the batch endpoint, so gating there means a provider added later inherits the
refusal rather than having to remember a check it never knew about; the new
rejected_range verdict carries no row, which is what makes the compiler force
every caller to handle it. The five writers that hold Prisma themselves gate at
their entry, before the existence probe decides insert versus overwrite, so an
impossible reading can no longer replace a good stored value.
An out-of-band value is dropped. Not clamped, because a clamp invents a reading
nobody took. Not stored behind a flag, because a quarantined row is still a row
and every consumer would have to learn the flag. Not fatal to the batch, because
one bad sample must not strand the run's other readings behind a held watermark.
Absence reads as absence.
A silent drop, though, is indistinguishable from a provider that sent nothing,
so every drop is tallied on the wide event under measurement.range_rejected with
the metric, the provider and the edge it broke. Counts only: the reading itself
is health data and never enters the log line, and the direction plus the
metric's own declared bound is what a diagnosis actually needs. The tally
accumulates rather than replacing, because a provider gates once per resource
leg and the reconciler once per row. One warning per provider lifts the event to
warn so a drop does not hide inside a green line.
A structural guard enumerates all twenty measurement writers with the reason
each is allowed to write, so a new one has to answer the question before it
ships. scripts/diagnose-implausible-measurements.ts reports the rows already
stored, aggregated per metric and source with no value beyond the bucket's own
extremes. It repairs nothing: whether a stored impossible row is a decode slip
or a band that is too narrow is not a call a script gets to make.
Six provider syncs returned a bare count, which narrowed their failures away:
a Nightscout window in which every SGV row was refused by a constraint
returned the same 0 as a quiet night, and an Oura or Polar run whose
collections all failed returned 0 too. The routes then answered 200 with a
count and the settings card drew a green tick over it.
Every sync now reports what it imported AND whether any part of it did not
land, and the route resolves the outcome server-side so the web card and the
native client cannot disagree. Google Health keeps its 502 for a run that
failed and wrote nothing, but a run that failed after writing some resources
is now a partial rather than a discarded result. The labs OCR commit carries
the same resolved outcome.
A sync failure the classifier bucketed as `transient` had no exit. The park
test read `kind === "persistent"` and the persistent streak anchor was only
ever stamped by a persistent failure, so a streak could run for a thousand
attempts with the persistent counter at zero, the state frozen at
`error_transient`, and no ladder it could climb. `transient` is a hypothesis
about retryability, and the classifier defaults every timeout, 429, 5xx and
unrecognised response to it precisely because the wire cannot tell a reboot
from a permanent break. That default is right for one attempt and wrong for a
thousand: an unbroken streak retried hourly for a full day has refuted it by
experiment. Record it as `persistent` from there, which feeds the park ladder
that already exists — 48 h of unbroken failure end to end before an
integration stops attempting.
The counters could not accumulate anyway. `IntegrationStatus` is keyed
(user, integration) while Withings runs four legs on four crons and WHOOP four
resources on four more, and a success cleared the error, every bucket, the
streak anchor and the state whatever leg had recorded them. The ECG tick at
:41 wiped what the sleep tick recorded at :15 — including a reauth park — so
the strike ladder was reset several times an hour and the card painted a green
"connected · 12 minutes ago" over a data class that had stopped arriving. A
success now clears only what it is entitled to clear.
Two columns carry it: `failing_since_at` dates the streak, `failing_leg` names
its owner. The migration backfills the anchor for rows already in an error
state so a connection broken for weeks escalates on its next attempt rather
than restarting its clock.
Four verified HIGH data-integrity fixes, each in-place per-provider (no
shared helper), from the campaign's data-integrity audit + adversarial
verification:
F1 — drain-per-sample-cumulative buildScanWhere was the only one of four
sibling drains missing deletedAt: null; the nightly cumulative fold
re-summed soft-deleted Apple samples and hard-deleted the legacy-steps
audit tombstones. Added the filter (mirrors consolidate-daily-mean,
dense-intraday-retention, consolidate-legacy-steps).
F2 — the additive late-fold merge had no single-flight guard, so two
overlapping runs (deploy overlap, or the admin endpoint racing the cron)
produced a permanent 2x daily total. Added a pg_advisory_xact_lock per
(user, externalId) plus an in-transaction re-read of the live
contributors so the serialised-second fold contributes zero (a bare lock
alone would still re-add its stale scanned sum). Mirrors the documents
quota advisory-lock pattern.
F3 — Withings sleep externalId keyed on segment START while measuredAt +
the natural key use END; a re-scored segment missed the externalId probe,
the sweep tombstoned the old row, and the re-keyed insert collided on the
natural key and wedged. Added the natural-key rescue (re-key in place),
mirroring the same-file measure path.
F4 — Withings measure sync stamped lastSyncedAt + recordSyncSuccess
unconditionally after a warn-only per-row loop; a transient write failure
stranded the reading behind the 10-min overlap forever. Now holds the
watermark and records failure on a hard row error (mirrors
google-health/fitbit), while a benign P2002 still advances.
Regression test per finding (reverting the three source files fails
exactly four tests). Integration suite green against real Postgres.
Platform-wide hardening: the failure classes found live this week hunted down
across every integration.
Fitbit carried the same tombstone wedge fixed for Google — probe now matches
soft-deleted rows and the update resurrects them; the comment describing a
partial unique index that never existed is corrected. Sleep segment ids are
stable across re-scoring for Withings (session+segment-start), WHOOP/Polar
(one segment per stage, no positional index) and Oura (run-start keyed), each
with a session-scoped sweep that soft-deletes rows a re-score orphaned — so
legacy and affected nights heal as they are re-read.
One platform rule for deleted rows: source-owned readings resurrect on
re-import (Withings, WHOOP, Oura, Polar, Nightscout, mood webhook, CSV
re-imports, Apple stats day-totals); Apple sample-grain deletions stick per
the paired-client contract, and the mood bulk route's tombstone match is now
a true no-op reported as duplicate. Import counts reconcile against actual
createMany counts.
Google lifecycle: a dead token registers on the cycle's hard-fail ledger, so
an expired connection parks instead of stamping success hourly; backfill and
sleep-repair markers stamp only after a clean pass (pg-boss retries are now
real); failed measurement writes hold the watermark; no-op updates are
skipped; mapping.md tells the current truth.
Scale: doctor-report reads only consumed columns and enabled sections;
exports and backups keyset-paginate; achievements tally via SQL buckets;
series charts day-bucket dense kinds beyond 90 days; BP pairing is a linear
two-pointer; distinct-type scans run server-side; the graded-series cold
fallback is bounded; the fitbit defer tracker loops instead of spreading.
Withings backdates some readings and lastSyncedAt advances on every
success (including a healthy 200-with-0 cycle), so a reading landing more
than 60s before the next cycle's now() could be missed. Widen the poll
catch-up overlap to 10 minutes — the upserts are idempotent, so re-touching
a handful of rows is harmless.
The in-app measurement routes evict the snapshot cache, but the iOS
batch ingest and the server-side Withings sync only marked it stale (or
never touched it), so a focus-refetch could serve pre-sync data for up
to the snapshot TTL. Route both background-sync entry points through a
hard evict so returning to the dashboard surfaces freshly-synced rows.
Hang `invalidateStatusInsightsForTypes` off every measurement ingest
path so a new reading of a type drops the cached assessment rows that
reading dirties — the next mount or nightly warm pass then regenerates
them against the new data instead of serving the pre-measurement text
for the rest of the day.
Wired into both `/api/measurements` modes (single + multi-entry batch),
`/api/measurements/batch`, and the three Withings sync paths
(measures + activity + sleep), reusing the `collapseToTypeDayKeys`
result each already computes for its rollup recompute. The call is
fire-and-forget after the user's write commits: idempotent, never a
blocker on the request path, and a failure only warns.
iOS-operator-blocking hotfix for REG-11 — Home dashboard tile renders
neither chart nor latest value for BP / Puls / Körperfett when the
most recent reading is older than 7 days. Root cause was in
`/api/dashboard/summary` SQL gates; `latestIn7d` + `sparkBuckets` both
required `measured_at >= sevenDaysAgo`, which returned empty for
sparse accounts, leaving the iOS tile with `latestValue: null` +
`sparkline: []` to render from. Detail-screen
(`/api/measurements/series?kind=…`) escaped because of its broader
window.
Server-side changes:
- `latestIn7d` → `latestEver` (drops 7-day filter; DISTINCT ON returns
latest reading EVER per type, bounded at N_metrics rows).
- `sparkBuckets` rewritten via ROW_NUMBER() OVER (PARTITION BY type
ORDER BY bucket_start DESC) so the trailing N daily buckets per
type render regardless of age, bounded at SPARK_DAYS × N_metrics.
- BP + Pulse metrics.push gated on `latest || allTimeCount > 0` like
BodyFat already is — accounts with zero readings ever no longer get
empty placeholder tiles.
- 4 new regression tests in summary/__tests__/route.test.ts cover
sparse accounts, zero-data accounts, within-7d guards.
Other develop work bundled in this release:
- W4-QoL — copy nits.
- W5-WORKOUTS — user source priority threaded through write-time
picker + batch route.
- W7-WITHINGS-RELIABILITY — response classifier + sync hardening.
- W8-OPS — ops phase docs.
- B11 — NEXT_PUBLIC_APP_VERSION build-arg routing so /api/version
cannot drift from CI tag.
- env-check workflow — manifest / example lockstep enforcement.
Resolves PR #194 (squash-merge produced via local -X theirs strategy
because GitHub's UI couldn't auto-resolve develop's post-v1.4.42
contiguous work against the v1.4.42 squash on main; resulting tree is
byte-identical to develop's HEAD).
Six touch-disjoint waves landed on develop before this release:
- W1-KNIP — exports/types tiers flipped to enforcing-mode on main;
35 unused exports + 52 unused types triaged to 0/0; shadcn surface
+ zod-infer contract types scoped via knip.json ignore-block.
- W2-ZOD-MULTI-ISSUE — `returnAllZodIssues` shared helper +
`/api/dashboard/widgets` 422 now returns every Zod issue + worker
audit-ledger breadcrumb. 41 sibling routes documented for v1.4.43.
- W3-QUERYKEY-LONGTAIL — 40 settings/medications/admin/hooks files
migrated to the queryKeys factory; 9 new factory entries; ESLint
rule + test-guard scope extended in lockstep.
- W4-ARCH-HYGIENE — BERLIN_DAY_FORMATTER consolidated across nine
sites (seven status helpers + bucket-series + bp-in-target);
Suspense double-comment consolidated; doctor-report sanitiser regex
rewritten to escape-form so diffs are reviewable; pr-detection-
worker soft-delete filter; offhost-backup DR-intent comment.
- W5-IOS-WORKOUTS — `dedupeWorkoutBatch` write-time cross-source
dedup helper for the v1.5 iOS HealthKit observer ingest; dashboard
tile-strip placeholder parity with the live TrendCard chrome.
- W6-WITHINGS-OFF + ENV-CHECK — Withings off-response classifier
(transient/reauth/persistent) + WithingsApiError typed throw + the
`pnpm check-env` CLI. APNs `allOrNone: true` closes the AP-2
silent-disable gap the wave was conceived to close.
No migration, no env-var change, no API contract break for iOS v0.5.4.
The new `/api/dashboard/widgets` 422 envelope is additively extended.
pnpm test --run green at 4815 passing / 1 skipped.
pnpm typecheck, pnpm lint, pnpm knip (no --include flag) green.
Eleven touch-disjoint waves landed on develop plus a six-axis QA
reconcile round before tagging. See CHANGELOG.md for the user-facing
prose; this commit message documents the wave + report inventory for
release-management traceability.
Waves shipped (each with a phase report under .planning/):
- phase-W-POOL-v1440-report.md — p-limit(4) wrapper around the
analytics 15-way fan-out + pg.Pool max raised 10 -> 20. Closes the
v1.4.39 empirical-trace finding #1 (Prisma pool starvation).
- phase-W-DELETED-v1440-report.md — 11 reader-tier helpers + 3
integration contracts close audit Critical Finding #3 (soft-delete
invisibility) across dashboard, analytics, summaries, rollup tiers,
and the Coach snapshot path.
- phase-W-INSIGHTS-v1440-report.md — mood-rollup swap on the three
/api/insights/{features,targets,cards} routes plus the
compliance-rollup hook gap closure for bulk-projection mints; the
post-QA reconcile patch closes the six remaining measurement.findMany
sites in the cluster that still leaked tombstoned rows.
- phase-W-INFRA-v1440-report.md — rollup helper umbrella consolidation
under src/lib/rollups/* with zero orphan re-exports; lint, typecheck,
knip, and 93/93 rollup tests green post-restructure.
- phase-W-RSC-v1440-report.md — per-tile Suspense boundaries plus the
queryKey factory-bypass walker test (extended to comparison/ in the
reconcile pass).
- phase-W-PRIVACY-v1440-report.md — bilingual /privacy page with 17
paired-section tests pinning the 9 SB-3 disclosure requirements.
- phase-W-AASA-v1440-report.md — /.well-known/apple-app-site-association
endpoint pinned to S8WDX4W5KX.dev.healthlog.app with app-ID parity
assertions guarding against split rotation.
- phase-W-APNS-NOTIFY-v1440-report.md — MEDICATION_REMINDER
time-sensitive APNs payload (priority 10 + interruption-level)
scoped via a parameterised test that pins all 6 other event-types
do not bypass Focus. AP-2 .p8 install gates real delivery.
- phase-W-CONSENT-v1440-report.md — AI consent receipts CRUD with
append-only invariant pinned in test; revoke-then-regrant leaves
both rows; idempotent DELETE returns 200 + receipt: null.
- phase-W-WMY-WIRE-v1440-report.md — three previously-dead WMY readers
(week/month/year) wired through summaries-slice and
health-score-fast-path with a linear-composability parity test
pinning the cross-consumer routing contract.
- phase-W-GHOSTS-v1440-report.md — ~1 177 lines of dead code removed
across 9 atomic commits; startOfUtcDay + wallClockInTz consolidation
eliminates a sibling-drift risk; knip whitelist trimmed to three
files pending follow-up.
QA reconcile patches applied per the v1.4.40 QA round (six reports
under .planning/round-v1440-QA-*-findings.md):
- senior C1 — dashboard-suspense-boundaries regex aligned with the
post-W-INFRA Thread 2 userTimezone-local lift.
- senior H1 — soft-delete filter added to the six remaining
insights measurement.findMany sites.
- specialist H1 — Promise.allSettled around bulk-projection
compliance recompute on both /api/medications/intake and
/api/dashboard/summary.
- ux H1 — compare-toggle migrated to queryKeys.dashboardWidgets
factory; walker test extended to src/components/comparison/.
- security M1 — consent artefact 64 KB cap enforced via
Buffer.byteLength (UTF-8 bytes, not UTF-16 code units).
Quality gates green at tag: pnpm typecheck, pnpm lint, pnpm knip
--include files,dependencies,binaries,unlisted, pnpm test --run
(4726 passing / 1 skipped). Tests 4525 -> 4726 unit + 1 skipped over
the v1.4.39 -> v1.4.40 span.
User-facing prose: see CHANGELOG.md.
Hotfix on top of v1.4.39. Three measurement write paths (Withings
sync, `/api/import`, admin backup restore) were bypassing the
persistent rollup tier's write hook, so the dashboard chart's
`source=rollup` fast-path painted "Noch nicht genug Daten" on the
30 / 90-point range for accounts whose recent data only came through
those paths. The 7-day window kept rendering because the chart's raw
fetch never consults the rollup tier.
Closes the read-coverage gap by wiring `recomputeBucketsForMeasurement`
into every remaining ingest path and re-anchors the boot-time backfill
discovery to per-day coverage so already-deployed instances re-converge
on the next worker boot. The legacy `sum_value IS NULL` branch stays
on the same UNION so the v1.4.39 column-backfill continues working.
Also re-anchors one date-bound test in the medications-intake route
suite so the green CI run no longer depends on the wall clock.
Tests 4647 → 4648 passing; one long-standing skip unchanged.
Largest feature delta in the v1.4.x line. Insights expands into seven dedicated metric sub-pages; GLP-1 medication tracking lands end-to-end (drug-level chart with Research Mode acknowledgment, EMA titration ladder, cadence + compliance, side-effect taxonomy, pen/vial inventory); cross-source priority becomes a two-axis resolver; per-user timezone threads through ten analytics surfaces; Withings activity + sleep v2 sync; onboarding rebuilt as a nested-route wizard; Personal Records ship end-to-end; Health Score provenance accordion; Coach runs native first-party prompts across six locales with a 1800-assertion refusal-probe matrix; OpenAPI drift gate hard-fails; multi-arch GHCR image. Migrations 0043 through 0060 are additive and forward-only. PROMPT_VERSION 4.24.0 → 4.25.0 with GROUND RULE 9 (refuses dose recommendations) and GROUND RULE 15 (refuses drug-level estimates with MDR + MDCG 2021-24 cites).
See CHANGELOG.md for the full per-section delta.
v1.4.15 Phase B2, criterion 2. The Phase B2 status helper (commit
4db72a8) defined a markReauthRequired / recordSyncFailure API but
nothing called it. A Withings refresh token that had revoked got the
same "transient error" treatment as a 503 from the upstream API,
scheduled syncs kept hammering the OAuth endpoint, and the Settings
UI showed "Connected" until the user manually triggered a sync and
read the inline error.
This commit wires every sync / OAuth path:
- getValidToken() (Withings): refresh failures classified by
Withings status code. 100/101/102 (auth failed / invalid token /
user gone) and 200..299 (OAuth grant errors) -> error_reauth.
Anything else (5xx, network) -> error_transient. Same logic
inlined in /api/withings/status so the proactive refresh path
parks the row too.
- syncUserMeasurements() and syncMoodLogEntries() short-circuit
immediately when isReauthRequired() returns true. Scheduled
pg-boss jobs no longer burn API quota on a parked connection.
- moodLog 401/403 from the upstream -> error_reauth. 5xx / 3xx
redirect / network / invalid-JSON / bad-format -> error_transient.
SSRF refusal also writes a status row so it surfaces in the UI.
- /api/withings/callback calls markReconnected() after writing the
new tokens, the next sync runs.
- /api/withings/disconnect and DELETE /api/settings/moodlog call
markDisconnected() to clear any prior error state. moodLog
PUT /api/settings/moodlog (re-save credentials) markReconnected()s
for the same reason.
The helper writes AuditLog action=integrations.sync.failed on every
failure (criterion 4) and dispatches a SYSTEM_ALERT notification to
admins after consecutiveFailures >= 3 with a 24h re-alert window
(criterion 3). Both paths are covered by the unit + integration test
suites that landed in 4db72a8 — this commit makes them live.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(api): test endpoints for Withings, moodLog, Web Push, Glitchtip, Umami
Five POST /test endpoints modelled on /api/ai/test (apiHandler +
requireAuth/requireAdmin + checkRateLimit 5/60s + annotate() +
sanitised error mapping):
- /api/integrations/withings/test — pings wbsapi.withings.net /measure
with the decrypted user access token
- /api/integrations/moodlog/test — HEAD against the user's decrypted
moodLog URL (5 s timeout, isPublicUrl
SSRF guard)
- /api/notifications/web-push/test — sends "Push test" via web-push to
every PushSubscription, returns only
endpoint host (never full URL)
- /api/monitoring/glitchtip/test — admin-only; sends a stub event via
sendGlitchtipEvent() using the saved
AppSettings.glitchtipDsn
- /api/monitoring/umami/test — admin-only; HTTPS-only GET against
AppSettings.umamiScriptUrl, checks
body for "umami" marker
Each route has a vitest suite covering: happy path, rate-limit denial,
missing config (422), and upstream-401 sanitisation (no sk-/Bearer/full
URL leak).
Co-authored-by: Marc-André Bombeck <noreply@anthropic.com>
* fix(integration-tests): address review findings — singleton key, redirect SSRF, token refresh, scoped push
Multi-agent review caught a regression set on the five integration test
endpoints. Fixes here address every CRITICAL + HIGH finding:
- AppSettings is keyed on `id: "singleton"` everywhere else; the Glitchtip
and Umami test routes were querying `id: "default"` and would silently
return 422 in production. Tests now assert the where argument so the
typo can't sneak back.
- moodLog and Umami fetch with `redirect: "manual"` to neutralise the
redirect-follow SSRF (a public host could 302 to 169.254.169.254 or
RFC1918 ranges); 3xx is now treated as a configuration error.
- Withings test goes through the existing `getValidToken()` helper, so an
expired access token refreshes the same way the real sync does instead
of reporting "credentials rejected" until the next scheduled sync.
- Glitchtip DSN is validated as HTTPS + public host before any outbound
call, and the dispatch is wrapped in a 5 s `Promise.race` deadline.
- Withings dispatch picks up an `AbortController` 5 s deadline; the
Withings API-level `status` field (HTTP 200 + `status !== 0`) is now
categorised separately from the HTTP status (e.g. `status: 401` →
credentials_rejected, `status: 601` → rate_limited).
- moodLog test does an authenticated GET against the actual sync
endpoint instead of a bare HEAD on the configured URL — a HEAD on the
bare URL would return 200 from a generic landing page even when the
API key was invalid.
- Web Push test sends to the most recent subscription only (was: every
registered subscription, so a user with three devices got three buzzes
per click); tests assert findFirst+orderBy so this can't regress.
- Web Push response shape is the same shape on success and upstream
failure (errors land in the meta envelope, not in the visible
response body) — the prior dead-code in the 401 sanitisation test is
gone.
- Every error response now carries an `errorCode` in `meta` (e.g.
`credentials_rejected`, `rate_limited`, `dsn_host_not_public`) so the
upcoming UI can localise via `t("settings.testConnection.errors." + code)`
instead of rendering the server's English fallback.
- Outbound error annotations strip `Bearer <token>` and full URLs
before they reach Wide Events / Loki — push providers in particular
embed the subscription endpoint (which is the routing secret) in
410/404 error messages.
- Umami response read is capped at 256 KB so a hostile public host
cannot stream gigabytes through us; the marker check is tightened to
tracker shapes (`window.umami`, `umami.track`, `"umami"`) so a 404
page that says "Powered by Umami" no longer false-positives.
- `apiError` now optionally carries a `meta` object so error responses
can include the `errorCode` without breaking the existing
`{ data: null, error: <string> }` envelope.
477 tests green.
---------
Co-authored-by: Marc-André Bombeck <noreply@anthropic.com>
HealthLog — personal health-tracking PWA with weight, blood pressure,
pulse, mood tracking, medication compliance, Withings integration,
doctor report PDF export, and Dracula-themed UI.
