Commit Graph
178 Commits
Author SHA1 Message Date
Marc-André Bombeck 92f17c8ca4 Merge pull request #1140 from MBombeck/ready/pr-1131
fix(import): leave out Apple Health samples HealthLog wrote itself
2026-10-04 17:33:47 +02:00
Marc-André Bombeck 28046f05f7 test(measurements): group the hourly-mean fake once instead of per day
The in-memory day-aggregate fake re-filtered every row for every day,
converting each timestamp to the profile timezone again, so a 1000-day
series took about five seconds and the graded pulse-status test hit its
timeout. It now groups by day and segment in one pass (4.8 s to 85 ms).
2026-10-04 16:14:10 +02:00
Marc-André Bombeck bc0d9f915f test(import): answer the MANUAL-type lookup in the import mocks 2026-10-04 14:49:07 +02:00
Marc-André Bombeck 4dabc83513 fix(import): match HealthLog's own samples by row id and ask for mirrors per type
A sample the app wrote before the origin marker existed still carries the
HealthLog row id as HKExternalUUID. A hit on one of the account's rows of
the same kind now leaves it out exactly, for Withings and import mirrors as
well as manual ones; either id of a blood pressure pair covers both halves,
since the app stamps both samples of a reading with one id.

The manual-mirror lookup read one OR clause per row on every batch, even
for an account without a single MANUAL row. The types that hold MANUAL rows
are now read once per import; a batch asks nothing for any other type, and
one time window per type for those.

Cycle samples carry the marker too and are now left out like the rest, and
a sample the export repeats (a correlation's records also stand on their
own) is counted once.
2026-10-04 14:34:38 +02:00
Marc-André Bombeck db524ded98 Average pulse by day in the doctor report, correlations and baselines
The doctor report's pulse average and day points, the correlation channels,
the illness correlation, the personal baseline and the Coach's tiered context
each folded pulse readings into a day by plain mean. They now take the day as
the mean of its local hours' means and a window as the mean of its days, in
the zone each reader buckets by. The day aggregates carry the weighted sums for
an hourly-mean type, so every caller reads one day value. Every other type is
unchanged.
2026-10-04 11:33:42 +02:00
BernhardBuckel 1763ac85d9 fix(import): honour the origin marker on a blood pressure correlation
A blood pressure reading is a <Correlation> wrapping its systolic and diastolic
records, and the marker can sit on the correlation instead of on each record.
The import only read it from a record's own metadata, so a marked correlation
would have come back in full.

A correlation's records are now held until it closes, then all kept or all left
out together: the marker on the correlation, before or after its records, or on
a record, applies. Tests cover each, the marker not spilling onto the next
record, and a self-closing record.
2026-10-04 11:28:08 +02:00
Marc-André Bombeck c755a42d93 Add one day-mean helper for the hourly-mean types
A day of pulse is the mean of its hours' means and a window the mean of its
days, but only the rollup tier and the chart knew it. The helper gives every
other reader the same statistic in both shapes it needs: in TypeScript over
readings it holds, and in SQL as a per-reading day weight whose sums give the
mean of the day values over any window. Every other type keeps its plain mean.
2026-10-04 11:05:35 +02:00
BernhardBuckel a5e98c9e71 fix(import): leave out Apple Health samples HealthLog wrote itself
The iOS app writes manual entries into Apple Health, and mirrors Withings and
import rows into it, each stamped with dev.healthlog.app.origin = healthlog.
The server already holds those readings under their own source, so the same
sample arriving through an Apple Health export is a second copy: the series
shows it twice and the rollups count it twice.

The export import now leaves out a record carrying that marker. The marker is a
child <MetadataEntry>, read after the record's open tag, so a record is
committed at its close tag. As a fallback for manual entries mirrored before the
app set the marker, a record is also left out when a MANUAL row of the same
type and value sits within 2 s (same rule and tolerance as the sync path).
Withings and import rows are not matched by value and time: they reach Apple
Health only through the mirror, which always sets the marker.

The import summary says how many records were left out for this reason.

Fixes #1109
2026-10-04 10:53:23 +02:00
BernhardBuckel 497da7ca8b fix(pulse): make a day of pulse the mean of its hours' means
A watch samples pulse about every 5 to 10 minutes at rest and about every
5 seconds in a workout. The daily value was the plain mean over a day's
readings, so a workout hour of several hundred readings outweighed the other
23 hours: a day with one hard hour and 23 resting hours (60 bpm) read about
128 bpm, not 64.

A day of pulse is now the mean of its per-hour means, in the rollup DAY
bucket, the live per-day read the chart serves, and the series route's day
grain. Windows over several days weigh each day once instead of by its
reading count, so the bias does not return one level up. Reading counts,
min/max, and the regression sums still run over every reading. The chart
tooltip and the data table name the statistic, in all seven locales.

The statistic is limited to pulse; HRV, SpO2 and glucose keep the plain mean
(see src/lib/measurements/day-statistic.ts). Stored DAY rows are rewritten by
the next rollup recompute (admin recompute route), not on read.

Fixes #1107
2026-10-04 10:41:08 +02:00
Marc-André Bombeck 57a54061be Merge pull request #1112 from BernhardBuckel/fix/series-source-ladder
fix(series): follow the source priority ladder in the dense and raw series reads
2026-10-03 21:41:33 +02:00
BernhardBuckel bb1acfab64 fix(series): follow the source priority ladder in the dense and raw series reads
The rollup tier and sleep resolve every (type, day) to the source the person's
ladder picks before they aggregate. The dense and raw reads of
/api/measurements/series filtered on user and type only, so with a second
provider a day's value blended the two, a reading present in both showed
twice, and the configured ladder had no effect on those charts.

Both reads now take the ladder-winning source per day through the shared
canonicalMeasurementsCte helper, as the other live readers do:

- the day and hour bucket read of the dense kinds (pulse, glucose, HRV, SpO2),
  and the stats strip over its rows;
- the generic raw read (weight, body fat, steps, resting heart rate, ...).

The collapse is per day for the hour grain too, as everywhere else. The helper
gains an optional single-type scope so a series scans one metric, not every row
the person has in the window.

Only a type that has a ladder is collapsed. For a type without one (glucose,
body water, bone mass) the helper breaks the tie by source name, which would
silently hide one of two real sources, so those keep every source as before.
A single-source account reads exactly what is stored.

Tests: a real-database test covering the ladder in both directions, the
per-day fallback, a reading mirrored into two sources, the day-bucket, hour-
bucket and raw reads, a type with no ladder and a single-source account. The
route's unit test file reads its fixtures through a small adapter for the new
raw statements, and the dense-read guard drops the exception for the removed
findMany. The OpenAPI description states the rule.

Refs #1108
2026-10-03 20:18:50 +02:00
Marc-André Bombeck 313d8491de feat(measurements): read a sleep duration written as 7:30 or 7h 30m
The sleep field asked for decimal hours only. A shared parser
(parse-duration.ts) now reads a bare number with point or comma,
H:MM, and hours and minutes with unit words (7h, 7h30, 7 h 30 min,
45m), answers in the field's unit, and refuses a negative, unknown or
longer-than-a-day entry. The form shows how the entry was read under
the field and puts a refusal on the field itself. The stored value and
the API are unchanged.

Refs #1101
2026-10-03 18:28:39 +02:00
Marc-André Bombeck 27ac4c1d63 Merge origin/main into fix/v1397-tz 2026-10-01 10:39:26 +02:00
Marc-André Bombeck 7c5ff89def feat(lint): refuse UTC day keys outside the time zone module
`instant.toISOString().slice(0, 10)` and parsing a key back as UTC
midnight are the idioms behind every day-boundary fix this year: the
evening west of UTC and the small hours east of it land on the
neighbouring day. A new rule, healthlog/no-utc-day-key, refuses them in
src/ outside src/lib/tz and tests, and refuses a disable directive for
itself that is not next-line or gives no reason.

The time zone module now names every legitimate form: userDayKey for a
person's day, dateOnlyKey / dateOnlyAtNoonUtc / dayKeyAsUtcMidnight for
date-only values and day labels, daysBetweenDateKeys / weekdayOfDateKey /
isCalendarDateKey / statedDateKey for key arithmetic and parsing.

Of the 133 sites the rule found, 84 are gone. Most moved onto a named
helper with no change in behaviour; these changed behaviour:

- cycle calendar, insights, Coach snapshot and prediction refresh start
  their lookbacks at the first instant of the user's day
- a model-transcribed date like "June 10, 2026" no longer lands a day
  early in a process east of UTC (documents and lab scans)
- a custom report range covers exactly the picked days in the profile
  zone instead of starting the evening before west of UTC
- date-of-birth pickers cap at the user's today, not the UTC one
- an allergy onset date is stored at noon UTC like other date-only values
- the report's cycle summary, the Coach's usual BP range and the intake
  projector's rollup recompute use the user's day
- vaccination dates render as dates instead of the previous evening

49 remain, each with its reason on the line above: 39 UTC by design (15
file name stamps, rollup and object keys, the UTC-day score engines) and
10 marked baseline. A guard test requires a classified reason on every
directive and holds the baseline to a ceiling that only goes down.
2026-10-01 09:59:57 +02:00
Marc-André Bombeck a1fa84dcdf chore(units): drop the preference loader nothing calls yet 2026-10-01 09:53:45 +02:00
Marc-André Bombeck d6f79c8bef fix(workouts): show distance, pace and elevation in the reader's units
The workout list, the detail header, the stat tiles, the sport-average
line and the elevation chart printed kilometres, minutes per kilometre
and metres for every reader. They now read distance, pace and elevation
through the unit registry (km or mi, /km or /mi, m or ft). The splits
table keeps its kilometre rows, which are cut server-side, and says so.
A pace that rounded to sixty seconds now carries into the minute.

Refs #1067
2026-10-01 09:03:51 +02:00
Marc-André Bombeck 70e6e07a89 refactor(units): resolve both unit preferences through one registry
Metric/imperial and the glucose unit were two separate lookups, so every
server-side text producer had to remember both and most remembered
neither. display-transform.ts now carries a glucose branch keyed by the
glucose preference (getReadingTransform), the metre-valued workout
quantities (distance, elevation, pace), and a resolver for the two
stored columns together. loadUnitPreferences reads them in one query and
useUnitDisplay hands the glucose unit out beside the metric/imperial
branch.

Refs #1067
2026-10-01 08:48:39 +02:00
Marc-André Bombeck add6d1fadc fix(restore): order the restore and the consolidation passes by one lock
A restore on a freshly booted instance failed its clearing step with a
deadlock (40P01) while the boot-time step consolidation was folding the
same account. A consolidation day updates or mints the day's total and
then soft-deletes the day's per-sample rows; the restore deletes every
reading of the account in one statement. When the restore reached a
day's samples before its total, each side held what the other needed
next.

The restore already took an advisory lock per account before its first
delete. Every day transaction of the passes built on runConsolidation
(step and mean consolidation, the per-sample cumulative drain, the dense
intraday retention) now takes the same lock in shared mode as its first
statement. A restore that starts mid-day waits for that day to commit;
a day that would start while a restore runs is refused at once, and the
pass leaves that account for its next run instead of folding rows the
restore is about to replace. The passes still share the lock with each
other.

A new integration test builds the lock order that deadlocked, with a
row trigger holding the consolidation on the day's total while the
restore starts; before this change the consolidation was cancelled with
a deadlock. A guard test requires every transaction in a pass on
runConsolidation to open with the shared lock.

Refs #1031
2026-09-28 00:34:11 +02:00
Marc-André Bombeck 9d2fd1db14 fix(tz): one default zone, and the user's own calendar where Berlin was hard-coded
The status assessments folded their recent days, weeks, months and years on
Berlin days whatever the user's zone, the mood and cycle crosstabs keyed
their metric days in Berlin, the resting-pulse proxy defaulted to Berlin
days, the briefing's opener rotated at Berlin midnight, and every
assessment prompt labelled today's date as Berlin's. They now use the
user's zone. Rollup months and years are labelled by their UTC start,
which is how they are cut; read in a zone west of UTC the label named the
month before.

"Europe/Berlin" was written out in some eighty places. DEFAULT_TIMEZONE in
lib/tz/format is now the one definition: the second copy in mood/date-key,
DISPLAY_TIMEZONE, the Berlin-only day key helpers and the per-route copies
are gone. An in-repo lint rule rejects the literal outside lib/tz and
tests; four areas another change owns are listed as pending.
2026-09-26 16:23:14 +02:00
Marc-André Bombeck c88754a0b5 fix(measurements): re-warm status assessments from every measurement write
The MCP writes, the Telegram reply, the CSV and backup importers and the
Apple export import refreshed the rollups by hand and never re-warmed the
cached status assessments, so a reading logged there kept the day's old
assessment until the nightly pass. They now run the shared post-write tail,
and the export import re-warms every type it wrote. The WHOOP body sync
removed its legacy weight row without refolding that day; it does now.

A structural test lists every file that writes measurements and how it keeps
both readers current, so a new write site fails until it says.
2026-09-26 16:07:51 +02:00
Marc-André Bombeck b2bb81cac3 fix(apple-health): keep one source per device across an OS update in the export import
The export importer keeps the largest source subtotal per day for step-like
totals, and it told sources apart by name, version and device. On the day a
phone or watch updates, its records carry two versions, so the one device
became two sources and the day kept only the larger half. The identity is
now the source and its device without any version.
2026-09-26 15:59:07 +02:00
Marc-André Bombeck 34f847f12f fix(integrations): anchor synced daily totals at local noon in the user's zone
Withings, Fitbit, Google Health, Polar and Oura wrote each daily total and
daily summary at noon UTC of its date. From UTC+12 on (New Zealand all year,
Fiji, Tonga, Samoa) noon UTC is already the next local day, so every synced
step count, SpO2 or resting-pulse day showed up a day late. The rows now sit
at local noon of their date in the user's zone, and the fetch windows run
through the user's own today.

Fitbit weigh-ins carry a local wall clock without an offset and were read
as UTC; they are read in the user's zone now.

ExternalIds stay on the calendar date, so the next sync finds each row and
moves it in place instead of adding a second one, and the day a row leaves
is refolded. Rows inside each integration's re-read window heal on the next
sync; Fitbit and Google Health re-read the whole history on reconnect.
2026-09-26 15:58:02 +02:00
Marc-André Bombeck 73e0653b2c fix(tz): one start-of-day for every zone, including skipped midnights
Local day bounds had three implementations. The day-key one read the zone
offset at UTC midnight, which on a transition day in zones far from UTC
(Pacific/Auckland, Australia/Sydney) is the offset after the change, so the
day started an hour off. The instant one converged on local 00:00, which in
zones that skip midnight (America/Santiago, America/Havana) does not exist,
and returned 23:00 of the previous day.

startOfLocalDayKey and localDayWindow in lib/tz/local-day are now the only
implementation: the first instant that reads the day, tried under every
offset in force around it. A skipped wall clock resolves to the moment the
clocks jump, the day-keyed copies in the measurement drains, the compliance
rollups and the dashboard batch window are gone, and the noon and mid-hour
anchors use the same converge.
2026-09-26 15:41:16 +02:00
Marc-André Bombeck d51ba9f134 fix(rollups): choose the canonical source per day inside weekly, monthly and yearly buckets
The rollup writer keeps one row per bucket and source. Collapsing the stored
WEEK / MONTH / YEAR rows picked one source for the whole bucket, so a month
of manual blood-pressure readings plus one synced reading reported the synced
day alone, and a step month that switched devices lost every day the losing
device carried. The live weekly and monthly chart fold made the same pick.

Every coarse bucket is now folded from the canonical DAY buckets: the source
ladder picks one source per day, then the week, month or year counts every
one of its days. Spread, slope and r² are composed from the summed day
accumulators. The live fold picks per local day before folding into weeks
and months. Ties between sources without a ladder rank break alphabetically
on the SQL paths too, matching the rollup reader.
2026-09-26 15:37:10 +02:00
Marc-André Bombeck d0e321c54a Merge branch 'fix/v1391-contract-and-ingest' into release/v1.39.1
# Conflicts:
#	docs/api/openapi.yaml
#	src/lib/openapi/routes/measurements.ts
2026-09-25 08:04:30 +02:00
Marc-André Bombeck 74dedee645 fix(import): write Apple Health spot rows without Prisma's per-call heap
Each `createManyAndReturn` flush of the export import left about 5 MB of
heap behind until some two dozen calls had piled up. A 300 000-record
synthetic import under a 524 MB heap settled at 53 MB retained; a run of
back-to-back 600-row calls with nothing between them reached 212 MB.

Spot rows now go through the column-array `INSERT … SELECT FROM unnest`
the restore already uses, with `ON CONFLICT DO NOTHING RETURNING`, which
is what `createMany({ skipDuplicates })` emits. The same import holds
11.5 MB and finishes in 35 s instead of 55 s. Ids are minted in the
cuid() shape, defaults are written as the client would write them, and
integration tests compare the returned rows and the resulting table with
`createManyAndReturn` for fresh rows, conflicts on either unique
identity, and duplicates inside one call.
2026-09-25 07:39:07 +02:00
Marc-André Bombeck 702a13873d Merge branch 'main' into release/v1.39.1 2026-09-25 07:21:34 +02:00
Antiheld86 5ec361abec feat(measurements): mirror EXTERNAL in Zod, keep those rows correctable
Adds the `EXTERNAL_SOURCE` leaf beside `scopes.ts` and mirrors the value in
`measurementSourceEnum` so the read shapes and the sync decoder see it. It
stays out of `WRITABLE_MEASUREMENT_SOURCES`: a source a client could assert
would prove nothing about where a reading came from, and that constant is
published to clients as `ingest.writeAllowlist`.

The value-edit gate in `PUT /api/measurements/[id]` needs the other question
answered, though — whether the number is the user's own to fix — so it now
reads a separate `USER_CORRECTABLE_MEASUREMENT_SOURCES`. The 409 that
protects a Withings row exists because the value is the provider's reading;
the hardware behind an ingest token is the user's own scale, and locking
those rows would contradict what the settings card and the Home Assistant
guide promise. That is the only site reading the wider set.
2026-09-25 06:55:17 +02:00
Marc-André Bombeck c757981638 fix(jobs): give the hourly rebuild the same time budget
The one-shot hourly rebuild of already-folded heart-rate, HRV and SpO2
days walks one account's whole folded history in one job with the
default fifteen-minute expiry, about fifty statements per day. Like the
drains it now stops between days once 75 % of the expiry has passed or
pg-boss aborts the job, and fails so pg-boss retries it (the boot send
allows three retries); a rebuilt day leaves the candidate set, so the
retry starts at the first day not yet rebuilt.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck 4242b454ce fix(import): fold rollups over the span the import wrote
After an Apple Health import the worker folded the rollup table from the
account's first measurement to its last, whatever the import carried, and
so also folded history older than the five-year window that the boot
backfill, its discovery and every read path keep to.

The import now records the oldest and newest instant it wrote
(`measuredSpan` on the job result), and the fold covers that span,
clamped to the fold window. On the seeded account of 1.25 million rows the
full fold took 12 s, the slowest statement 2.7 s against the 60 s
statement limit, so this is about the window rather than time.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck 77bfa2f4e1 perf(records): find the best cumulative day a day at a time
The personal-record pass, every half hour for every account, read every
live row of each cumulative type into memory to find the best day. With
784 000 active-energy rows on one account (the per-sample rows pile up
when the nightly drain cannot finish, #1031) it ran a 524 MB heap out of
memory; the old code needed 1.1 GB of heap for that account.

It now reads the type in bounded keyset pages and handles one local day
at a time. The canonical-source pick it applies is per day, so the day
totals and the winner are the same; a tie still keeps the earliest day.
Measured on the same account: 189 MB peak heap, the same record. The
warm-up gate counts only up to its threshold instead of counting a whole
history.

`iterateSourcePages` and `iterateDayBuckets` in the consolidation base
take the row type as a parameter, and the day grouping can keep every row.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck f4b51cf134 perf(consolidation): fold a day at a time and batch the hourly writes
The consolidation drains (cumulative, daily mean, dense intra-day
retention, legacy steps) gathered a type's whole history into one array
before folding the first day. On an account of 1.25 million rows the
heart-rate array alone held about 230 MB, and the dense retention peaked
at 460 MB of heap against a 524 MB limit; in a 1 GB container running the
web server and the worker together the process died of heap exhaustion
within a minute and the job was later marked "job timed out" (#1031).

- Pages are now handed on one at a time and each local day is folded as
  soon as its last row has been read, so a pass holds one page and one
  day.
- The keyset cursor gains a redundant `measuredAt >= cursor` bound.
  Without it Postgres could not use the cursor as an index condition and
  re-read every earlier row on each page (47 ms and 248 000 filtered rows
  for one page halfway through the heart-rate history; 5 ms now).
- The dense fold resolves a day's hourly rows with one lookup and one
  insert instead of two lookups and a write per hour. On the seeded
  account the pass went from about 290 000 statements to 54 000.
- `shouldStop` lets a caller end a pass between days; the result says
  `stoppedEarly`. Days commit one by one and a folded day leaves the
  scan, so the next run continues at the first day not reached.
2026-09-25 00:31:40 +02:00
Marc-André Bombeck 70dba821be perf(measurements): fold each source before picking the canonical one
The live daily fold ranked every row by source before folding, which sorted
the whole window twice. A year of per-minute heart rate (525,600 rows) took
about 0.55 s per series on Postgres 16, with both sorts spilling to disk.

The fold now aggregates per source and bucket in one hashed pass and then
keeps each bucket's ladder-canonical source. Folding the canonical source's
rows is the same as folding that source's group, so the numbers do not
change; the same year reads in about 0.29 s. A new integration case pins the
canonical pick, mean, count and spread for a day with three sources.
2026-09-24 12:37:51 +02:00
Marc-André Bombeck c41df33dd3 fix(cycle): read the phase crosstab as per-source day aggregates
The coach's cycle snapshot read a year of raw readings for the phase
crosstab, including steps, heart-rate variability and CGM glucose. On a
CGM account that is over 100 000 objects per snapshot.

The crosstab only uses each day's sum and count after picking one
source and device per day, and the pick depends only on which sources
and devices are present. readSourceDayAggregates returns one row per
type, day, source and device, and the day map now carries each row's
reading count, so the day values are the ones the raw readings give.

The 30-day clinical glucose reads stay raw: time in range, SD, J-index
and the risk indices are functions of individual readings, and a
sensor's rate bounds the window below 45 000 rows.

Refs #1023
2026-09-24 08:06:47 +02:00
Marc-André Bombeck 5a0b6ee153 fix(insights): fold dense measurement reads per day in SQL
A single account syncing heart rate from a watch took the worker down
within half an hour. The status cards' graded series read the last 91
days of raw rows, bounded in time but not in rows, so a stream sampled
every minute or faster materialised six or seven figures of objects on
every refresh and the read ran into the statement timeout.

Add readDayAggregates, which folds one type into per-day sum, count, min
and max in Postgres (optionally split into age segments and filtered to
the metric's plausibility domain), and move the readers that folded raw
rows into days in memory onto it: the graded series (recent, weekly and
the cold-tier fallback, for every type), the vitals day-mean fallback
behind the personal band, the tiered context's recent days and the plan
review. Recent, weekly and monthly buckets are identical to the raw
fold; the cold-tier yearly slope now runs over day means.

On 91 days of per-minute pulse the graded series now allocates 1.6 MB
where it allocated 62 MB, while the raw read alone holds 22 MB.

A structural guard freezes the remaining untaken measurement reads in
the worker and insight directories: each must name a sparse literal
type or carry a listed reason.

Refs #1023
2026-09-24 07:56:39 +02:00
Marc-André Bombeck 88cea09c7b fix(measurements): cut chart day buckets at the user's midnight
Switching a chart from 7 to 30 days moved steps, active energy,
flights and distance totals onto neighbouring days. The 7-day range
buckets raw rows by the user's calendar day; the 30-day range read the
DAY rollup tier, whose buckets are UTC days, and the chart labelled each
bucket's UTC-midnight start in the user's zone. East of UTC a sample
shortly after local midnight counted on the previous day; west of UTC
every bucket read one day early.

The daily series (the chart read and the batched dashboard read) and the
aggregate=daily|weekly|monthly list now fold live in the user's zone and
return the instant of local midnight, so every range and every summed
metric shares one day boundary. The fold runs in Postgres and returns
one row per day; about 0.1 s for 90 days of per-minute samples. Windows
longer than the DAY cap still step up to the coarser rollup tiers.

Refs #1026
2026-09-24 07:56:33 +02:00
Marc-André Bombeck 9d9e8570f7 fix(apple-health): take a percent from a fraction or an already-scaled client
Shipped iOS builds multiply oxygen saturation and body fat by 100 before
upload. The importer scaled them again, so 0.97 was stored as 9700, the
plausibility guard rejected the row as out of range inside an HTTP 200,
and the client treats that skip as final and moves its HealthKit anchor
past the sample. Every reading of those two types was lost, silently, on
every install that syncs them.

The contract says values travel raw and the server scales, and that is
what the OpenAPI description publishes. One comment in the schema said
the opposite, that the client must multiply by 100 first, and that is
the line the app was built against. It is corrected here.

The five percent-shaped HealthKit identifiers now take either spelling:
a value above the fraction ceiling is already a percent and passes
through, anything at or below it is scaled. The domains do not overlap,
and the per-type range guard still rejects what is impossible either
way. Current app builds stop losing readings without an app release.

The tests pin both spellings, the range after conversion, and that the
tolerant converter covers the percent class and nothing else: waist
circumference uses the same factor to turn metres into centimetres, and
a blanket change made a 1.02 m waist into 1.02 cm.

Refs MBombeck/healthlog-iOS#7
2026-09-18 16:40:11 +02:00
Marc-André Bombeck fd337a4e7a fix(import): match the molar-mass annotation by the shape Apple writes
The catch-all angle-bracket pattern read as an HTML sanitiser to static analysis. The annotation Apple writes is a number, so the pattern now matches exactly that, and a unit with anything else between the brackets stays unknown instead of being silently trimmed into a lookup.
2026-09-10 03:25:16 +02:00
Marc-André Bombeck c3b5585fd6 fix(ops): stop the distance dry run promising work it will refuse
Low, review 2. An account with one out-of-range candidate is refused
whole, rows and audit row included, but the dry run counted its rows into
"N row(s) would be repaired" — so the number an operator read before
--apply was larger than the number --apply could ever produce. The refused
accounts are now counted and reported separately, and the per-account line
says its rows are not in the total.

The refusal itself told the operator to check the Health app "before
forcing anything". There is no --force and never was. It now names the
only route that exists: re-run with the matching --unit if the archive
really was in another one.
2026-09-10 01:30:26 +02:00
Marc-André Bombeck 610083aa4d fix(ops): stop the distance repair losing its rollup tail in silence
The rollups and the cached status insights are recomputed after the
transaction, so a throw there left the rows repaired and every chart on
the pre-repair numbers — and no later run would notice, because the
audit row makes the account skip from then on.

The tail is wrapped now. The audit row is written with
`rollupsRefreshed: false` and only flipped once the recompute returned,
so a re-run can tell "repaired, rollups pending" from "done" and says so.
A run that could not finish the tail names the account, prints the
backfill to re-run — `scripts/backfill-rollups.ts --user <userId>` — and
exits non-zero.

Review finding M2.
2026-09-10 00:37:15 +02:00
Marc-André Bombeck 361e577864 fix(ops): refuse an account whose rows are not the thousandfold class
The provenance criterion proves where a row came from, not which build
imported it. An account re-imported on the fixed build carries the same
`EXPORT_XML_SOURCE_MAX` stamp and the same `stats:` external id with the
right numbers and no audit row, so it is selected exactly like a broken
one — and its rest days, under 200 m because the phone stayed at home,
survive the plausibility check after another thousandfold multiply.

So one candidate row that would leave the range now refuses the whole
account: nothing written, no audit row, the offending rows and a printed
reason instead. The idempotency guard only ever held against the
script's own prior run; this holds against the case it could not see.

The runbook says the same in words — a re-imported account must not be
repaired, the re-import already did it — and says what the dry run is
for: it is where the decision is made, not a formality in front of one
already taken.

Integration coverage for the re-imported account: correct rows, an
`--apply` run, nothing changes and no audit row is left behind.

Review finding H1.
2026-09-10 00:27:43 +02:00
Marc-André Bombeck 65e9ecbde6 fix(ingest): keep the batch sample's unit captured, not authoritative
The record-unit conversion is now opt-in per caller and only the
`export.xml` archive path opts in. There the attribute is authoritative:
Apple stamps every quantity `<Record>` with the account's own display
unit, so `km` means kilometres and the reading is meaningless without it.

`POST /api/measurements/batch` keeps its documented contract — `unit` is
captured for audit and never read. Nothing in the tree pins the native
client's unit strings for the seventeen convertible identifiers, and a
client that reads a sample in metres while stamping the person's display
unit into the field would have had every walking distance multiplied by a
thousand on ingest. Reading the field there is a wire change and needs
the client's strings pinned first.

The mapping-table audit and the shared conversion module stay. The batch
path's identity behaviour is written down where it is decided and pinned
twice: every convertible identifier round-trips a sibling unit unchanged
by default, and a walking-distance entry sent as `km` is stored as it
arrived. The OpenAPI description says what the server actually does.

Review finding H2.
2026-09-10 00:24:59 +02:00
Marc-André Bombeck e031e2211c fix(import): honour the energy unit an archived workout carries
`totalEnergyBurned` rides the same account-display-unit contract as the
distance beside it: a Health app set to kilojoules writes `kJ`, and the
column is kilocalories. Convert it through the shared table.

Refs #944
2026-09-09 22:47:05 +02:00
Marc-André Bombeck bc7eff8322 feat(ops): repair walking distances stored a thousand times too small
The unit fix only helps the next import. An account that imported an
archive before it carries a decade of days understated 1000x, and the
authoritative repair — re-importing the archive on the fixed build —
needs an archive the account may no longer have.

`scripts/repair-apple-health-distance.ts` is dry-run by default and
selects rows only by the stamp the archive fold itself writes: the
EXPORT_XML_SOURCE_MAX provenance plus the `stats:` external id minted
for that fold. Never by "the value looks small", so a row a native sync
has since overwritten is left alone. `--apply` multiplies one account
per transaction and writes the audit row in the same transaction, which
is what makes a second run a no-op. The archive's own unit is named by
the operator (`--unit=mi`) because the discarded attribute is the defect
itself, and a row whose repaired value would leave the plausible range
is reported rather than written.

Refs #944
2026-09-09 22:45:13 +02:00
Marc-André Bombeck 5c7c7afd59 fix(import): honour the unit an Apple Health record carries
Apple stamps every quantity `<Record>` of an export archive with the
account's own display unit, not the unit the mapping table assumes: a
metric archive reports walking distance in km, an imperial one in mi,
body mass in lb, energy in kJ, temperature in degF. The mapper accepted
`input.unit` and never read it, so a 2.484 km day was folded into the
day's total as 2.484 m and a decade of imported history read as a couple
of metres walked.

`hk-units.ts` is now the one place a HealthKit unit is converted, shared
by the record path and the workout path (which carried its own km/mi
branch — the duplicate is what let the record path stay wrong). It
converts only inside a family and only for units Apple can emit, and
returns the reading untouched for anything it cannot place, so an
archive that already speaks the table's unit is unchanged.

Every mapping-table entry whose unit cannot be converted — a
dimensionless count, a pinned event, a logarithmic dB level, a compound
rate, the percent units whose value rides as a 0..1 fraction — now
carries a written reason, and a structural guard fails on an entry that
offers neither a convertible unit nor a reason.

Refs #944
2026-09-09 22:31:31 +02:00
Marc-André Bombeck 704ef11eab test: pin the km-scale walking distance the export import stores as metres
The Apple export writes DistanceWalkingRunning in the account's own
length unit; the record mapping assumes metres and never reads the
attribute, so a 2.484 km day lands as 2.484 m.

Refs #944
2026-09-09 22:21:05 +02:00
Marc-André Bombeck 841593a5b2 fix: converge the two factors a glucose reading was converted by
A value imported in mmol/L was multiplied by 18.016; one displayed in
mmol/L was divided by 18.0182. The two sat in different files, both
correct-looking, and a reading that went in as 5.3 did not come back out
as 5.3. Latent while nobody could select mmol/L, live the moment they can.

One exported constant now, on the guideline figure the display path
already used, and the glucose insight page takes it from there rather than
keeping a third copy. The test names the constant instead of repeating the
digits, which is how the two came apart in the first place, and it asserts
the round trip a reader on mmol/L actually experiences.
2026-09-03 06:05:58 +02:00
Marc-André Bombeck 2ea08c6966 fix: record the export stamp an Apple Health import already promised
The import status endpoint returns the instant the Health app stamped on
the archive, the published contract requires the field, and the schema
comment said the parser filled it during unpack. It never did: ExportDate
sat on the parser's ignore list, so the column answered null on every job
ever run, for a year.

The parser reads the element now and hands the stamp to the worker on the
next chunk boundary, so the value lands while the run is still going and a
job that later fails still records which export it was working from. An
archive that omits the element, or writes a value the runtime cannot read,
leaves the column null rather than guessing at an instant.

No contract change: the endpoint and the OpenAPI description already
described this behaviour.
2026-09-03 06:05:58 +02:00
Antiheld86 cba7a0fce5 Add a self-service measurements:write API token scope
Since v1.30.17 there has been no self-service path to a token that can
write a measurement. That release made the Bearer scope default
fail-closed, which closed a real privilege-escalation hole, and removed
the generic mint at POST /api/tokens along with it. The generic mint had
been the only way to get a token that could reach /api/measurements, so
a local sensor pushed through Home Assistant or Node-RED lost its door
while the OAuth integrations kept theirs.

Add a narrow scope accepted by exactly two routes, both writes, both on
the holder's own record.

The scope constant is a leaf module beside the surface that consumes it,
matching FHIR_READ_SCOPE and the MCP scopes. requireRecordAuth grows an
options object rather than a sibling function, because the sharing
surface guard keys every leg on that one identifier and a second name
would mean two copies of the delegable declaration. AuthContext carries
the admitting scope as a single nullable string rather than the token's
permission array, so nothing downstream can re-derive an authorisation
decision from it.

A scoped credential is refused a record selector before any grant is
read. Placing the refusal above the sharing check rather than beside it
means no future widening of what a grant permits can reach one of these
tokens.

Rows are attributed MANUAL. An entry that explicitly names APPLE_HEALTH
is refused 422 rather than relabelled: that source is half the
(userId, type, source, externalId) dedup key the iOS client writes into,
it participates in the cross-source merge, and it gates the HealthKit
sync fields. For the same reason a scoped write no longer advances
User.lastSyncedAt, which the phone reads to size its undelivered window.

Also closes a narrower gap found while building this. The idempotency
wrapper resolved a replayed delegated response against a live grant
without considering which credential was presenting it, so a cached
response about a shared record could be returned to a token the handler
itself would refuse. No write was reachable that way; the response body
was.

Guards: T3 gains the mint site, T4's vocabulary gains the scope and
learns to read the requireRecordAuth form (without which this change
would open a way to widen a route that the closed vocabulary could not
see), and a new T5 freezes the set of delegable routes naming a scope.
The sharing surface guard's AST matcher reads only the first two
arguments, so it needed prose but no code.
2026-08-31 08:46:16 +00:00
Marc-André Bombeck 1d733ef192 Let a German verdict reach the classifier the merge gave it
The two halves of the ECG fix were built apart. One taught the parser
Apple's full English verdict set; the other taught it the metadata keys and
verdict wording a German watch writes. Each was correct alone, and each
left "Sinusrhythmus" unresolved for a different reason.

Together they meet: the alias table translates the wording onto Apple's
own English, and the classifier decides what it means. One place still
owns the meaning, which is what stops a per-language verdict table from
drifting the first time Apple adds a case.

The test that expected null was pinning the gap between the branches
rather than a decision about German. It now expects NOT_DETECTED, and a
French verdict takes over the null case, because four of the six
languages genuinely have no observed key map yet.
2026-08-23 19:42:45 +02:00