A restore on a freshly booted instance failed its clearing step with a
deadlock (40P01) while the boot-time step consolidation was folding the
same account. A consolidation day updates or mints the day's total and
then soft-deletes the day's per-sample rows; the restore deletes every
reading of the account in one statement. When the restore reached a
day's samples before its total, each side held what the other needed
next.
The restore already took an advisory lock per account before its first
delete. Every day transaction of the passes built on runConsolidation
(step and mean consolidation, the per-sample cumulative drain, the dense
intraday retention) now takes the same lock in shared mode as its first
statement. A restore that starts mid-day waits for that day to commit;
a day that would start while a restore runs is refused at once, and the
pass leaves that account for its next run instead of folding rows the
restore is about to replace. The passes still share the lock with each
other.
A new integration test builds the lock order that deadlocked, with a
row trigger holding the consolidation on the day's total while the
restore starts; before this change the consolidation was cancelled with
a deadlock. A guard test requires every transaction in a pass on
runConsolidation to open with the shared lock.
Refs #1031
The status assessments folded their recent days, weeks, months and years on
Berlin days whatever the user's zone, the mood and cycle crosstabs keyed
their metric days in Berlin, the resting-pulse proxy defaulted to Berlin
days, the briefing's opener rotated at Berlin midnight, and every
assessment prompt labelled today's date as Berlin's. They now use the
user's zone. Rollup months and years are labelled by their UTC start,
which is how they are cut; read in a zone west of UTC the label named the
month before.
"Europe/Berlin" was written out in some eighty places. DEFAULT_TIMEZONE in
lib/tz/format is now the one definition: the second copy in mood/date-key,
DISPLAY_TIMEZONE, the Berlin-only day key helpers and the per-route copies
are gone. An in-repo lint rule rejects the literal outside lib/tz and
tests; four areas another change owns are listed as pending.
The MCP writes, the Telegram reply, the CSV and backup importers and the
Apple export import refreshed the rollups by hand and never re-warmed the
cached status assessments, so a reading logged there kept the day's old
assessment until the nightly pass. They now run the shared post-write tail,
and the export import re-warms every type it wrote. The WHOOP body sync
removed its legacy weight row without refolding that day; it does now.
A structural test lists every file that writes measurements and how it keeps
both readers current, so a new write site fails until it says.
The export importer keeps the largest source subtotal per day for step-like
totals, and it told sources apart by name, version and device. On the day a
phone or watch updates, its records carry two versions, so the one device
became two sources and the day kept only the larger half. The identity is
now the source and its device without any version.
Withings, Fitbit, Google Health, Polar and Oura wrote each daily total and
daily summary at noon UTC of its date. From UTC+12 on (New Zealand all year,
Fiji, Tonga, Samoa) noon UTC is already the next local day, so every synced
step count, SpO2 or resting-pulse day showed up a day late. The rows now sit
at local noon of their date in the user's zone, and the fetch windows run
through the user's own today.
Fitbit weigh-ins carry a local wall clock without an offset and were read
as UTC; they are read in the user's zone now.
ExternalIds stay on the calendar date, so the next sync finds each row and
moves it in place instead of adding a second one, and the day a row leaves
is refolded. Rows inside each integration's re-read window heal on the next
sync; Fitbit and Google Health re-read the whole history on reconnect.
Local day bounds had three implementations. The day-key one read the zone
offset at UTC midnight, which on a transition day in zones far from UTC
(Pacific/Auckland, Australia/Sydney) is the offset after the change, so the
day started an hour off. The instant one converged on local 00:00, which in
zones that skip midnight (America/Santiago, America/Havana) does not exist,
and returned 23:00 of the previous day.
startOfLocalDayKey and localDayWindow in lib/tz/local-day are now the only
implementation: the first instant that reads the day, tried under every
offset in force around it. A skipped wall clock resolves to the moment the
clocks jump, the day-keyed copies in the measurement drains, the compliance
rollups and the dashboard batch window are gone, and the noon and mid-hour
anchors use the same converge.
The rollup writer keeps one row per bucket and source. Collapsing the stored
WEEK / MONTH / YEAR rows picked one source for the whole bucket, so a month
of manual blood-pressure readings plus one synced reading reported the synced
day alone, and a step month that switched devices lost every day the losing
device carried. The live weekly and monthly chart fold made the same pick.
Every coarse bucket is now folded from the canonical DAY buckets: the source
ladder picks one source per day, then the week, month or year counts every
one of its days. Spread, slope and r² are composed from the summed day
accumulators. The live fold picks per local day before folding into weeks
and months. Ties between sources without a ladder rank break alphabetically
on the SQL paths too, matching the rollup reader.
Each `createManyAndReturn` flush of the export import left about 5 MB of
heap behind until some two dozen calls had piled up. A 300 000-record
synthetic import under a 524 MB heap settled at 53 MB retained; a run of
back-to-back 600-row calls with nothing between them reached 212 MB.
Spot rows now go through the column-array `INSERT … SELECT FROM unnest`
the restore already uses, with `ON CONFLICT DO NOTHING RETURNING`, which
is what `createMany({ skipDuplicates })` emits. The same import holds
11.5 MB and finishes in 35 s instead of 55 s. Ids are minted in the
cuid() shape, defaults are written as the client would write them, and
integration tests compare the returned rows and the resulting table with
`createManyAndReturn` for fresh rows, conflicts on either unique
identity, and duplicates inside one call.
Adds the `EXTERNAL_SOURCE` leaf beside `scopes.ts` and mirrors the value in
`measurementSourceEnum` so the read shapes and the sync decoder see it. It
stays out of `WRITABLE_MEASUREMENT_SOURCES`: a source a client could assert
would prove nothing about where a reading came from, and that constant is
published to clients as `ingest.writeAllowlist`.
The value-edit gate in `PUT /api/measurements/[id]` needs the other question
answered, though — whether the number is the user's own to fix — so it now
reads a separate `USER_CORRECTABLE_MEASUREMENT_SOURCES`. The 409 that
protects a Withings row exists because the value is the provider's reading;
the hardware behind an ingest token is the user's own scale, and locking
those rows would contradict what the settings card and the Home Assistant
guide promise. That is the only site reading the wider set.
The one-shot hourly rebuild of already-folded heart-rate, HRV and SpO2
days walks one account's whole folded history in one job with the
default fifteen-minute expiry, about fifty statements per day. Like the
drains it now stops between days once 75 % of the expiry has passed or
pg-boss aborts the job, and fails so pg-boss retries it (the boot send
allows three retries); a rebuilt day leaves the candidate set, so the
retry starts at the first day not yet rebuilt.
After an Apple Health import the worker folded the rollup table from the
account's first measurement to its last, whatever the import carried, and
so also folded history older than the five-year window that the boot
backfill, its discovery and every read path keep to.
The import now records the oldest and newest instant it wrote
(`measuredSpan` on the job result), and the fold covers that span,
clamped to the fold window. On the seeded account of 1.25 million rows the
full fold took 12 s, the slowest statement 2.7 s against the 60 s
statement limit, so this is about the window rather than time.
The personal-record pass, every half hour for every account, read every
live row of each cumulative type into memory to find the best day. With
784 000 active-energy rows on one account (the per-sample rows pile up
when the nightly drain cannot finish, #1031) it ran a 524 MB heap out of
memory; the old code needed 1.1 GB of heap for that account.
It now reads the type in bounded keyset pages and handles one local day
at a time. The canonical-source pick it applies is per day, so the day
totals and the winner are the same; a tie still keeps the earliest day.
Measured on the same account: 189 MB peak heap, the same record. The
warm-up gate counts only up to its threshold instead of counting a whole
history.
`iterateSourcePages` and `iterateDayBuckets` in the consolidation base
take the row type as a parameter, and the day grouping can keep every row.
The consolidation drains (cumulative, daily mean, dense intra-day
retention, legacy steps) gathered a type's whole history into one array
before folding the first day. On an account of 1.25 million rows the
heart-rate array alone held about 230 MB, and the dense retention peaked
at 460 MB of heap against a 524 MB limit; in a 1 GB container running the
web server and the worker together the process died of heap exhaustion
within a minute and the job was later marked "job timed out" (#1031).
- Pages are now handed on one at a time and each local day is folded as
soon as its last row has been read, so a pass holds one page and one
day.
- The keyset cursor gains a redundant `measuredAt >= cursor` bound.
Without it Postgres could not use the cursor as an index condition and
re-read every earlier row on each page (47 ms and 248 000 filtered rows
for one page halfway through the heart-rate history; 5 ms now).
- The dense fold resolves a day's hourly rows with one lookup and one
insert instead of two lookups and a write per hour. On the seeded
account the pass went from about 290 000 statements to 54 000.
- `shouldStop` lets a caller end a pass between days; the result says
`stoppedEarly`. Days commit one by one and a folded day leaves the
scan, so the next run continues at the first day not reached.
The live daily fold ranked every row by source before folding, which sorted
the whole window twice. A year of per-minute heart rate (525,600 rows) took
about 0.55 s per series on Postgres 16, with both sorts spilling to disk.
The fold now aggregates per source and bucket in one hashed pass and then
keeps each bucket's ladder-canonical source. Folding the canonical source's
rows is the same as folding that source's group, so the numbers do not
change; the same year reads in about 0.29 s. A new integration case pins the
canonical pick, mean, count and spread for a day with three sources.
The coach's cycle snapshot read a year of raw readings for the phase
crosstab, including steps, heart-rate variability and CGM glucose. On a
CGM account that is over 100 000 objects per snapshot.
The crosstab only uses each day's sum and count after picking one
source and device per day, and the pick depends only on which sources
and devices are present. readSourceDayAggregates returns one row per
type, day, source and device, and the day map now carries each row's
reading count, so the day values are the ones the raw readings give.
The 30-day clinical glucose reads stay raw: time in range, SD, J-index
and the risk indices are functions of individual readings, and a
sensor's rate bounds the window below 45 000 rows.
Refs #1023
A single account syncing heart rate from a watch took the worker down
within half an hour. The status cards' graded series read the last 91
days of raw rows, bounded in time but not in rows, so a stream sampled
every minute or faster materialised six or seven figures of objects on
every refresh and the read ran into the statement timeout.
Add readDayAggregates, which folds one type into per-day sum, count, min
and max in Postgres (optionally split into age segments and filtered to
the metric's plausibility domain), and move the readers that folded raw
rows into days in memory onto it: the graded series (recent, weekly and
the cold-tier fallback, for every type), the vitals day-mean fallback
behind the personal band, the tiered context's recent days and the plan
review. Recent, weekly and monthly buckets are identical to the raw
fold; the cold-tier yearly slope now runs over day means.
On 91 days of per-minute pulse the graded series now allocates 1.6 MB
where it allocated 62 MB, while the raw read alone holds 22 MB.
A structural guard freezes the remaining untaken measurement reads in
the worker and insight directories: each must name a sparse literal
type or carry a listed reason.
Refs #1023
Switching a chart from 7 to 30 days moved steps, active energy,
flights and distance totals onto neighbouring days. The 7-day range
buckets raw rows by the user's calendar day; the 30-day range read the
DAY rollup tier, whose buckets are UTC days, and the chart labelled each
bucket's UTC-midnight start in the user's zone. East of UTC a sample
shortly after local midnight counted on the previous day; west of UTC
every bucket read one day early.
The daily series (the chart read and the batched dashboard read) and the
aggregate=daily|weekly|monthly list now fold live in the user's zone and
return the instant of local midnight, so every range and every summed
metric shares one day boundary. The fold runs in Postgres and returns
one row per day; about 0.1 s for 90 days of per-minute samples. Windows
longer than the DAY cap still step up to the coarser rollup tiers.
Refs #1026
Shipped iOS builds multiply oxygen saturation and body fat by 100 before
upload. The importer scaled them again, so 0.97 was stored as 9700, the
plausibility guard rejected the row as out of range inside an HTTP 200,
and the client treats that skip as final and moves its HealthKit anchor
past the sample. Every reading of those two types was lost, silently, on
every install that syncs them.
The contract says values travel raw and the server scales, and that is
what the OpenAPI description publishes. One comment in the schema said
the opposite, that the client must multiply by 100 first, and that is
the line the app was built against. It is corrected here.
The five percent-shaped HealthKit identifiers now take either spelling:
a value above the fraction ceiling is already a percent and passes
through, anything at or below it is scaled. The domains do not overlap,
and the per-type range guard still rejects what is impossible either
way. Current app builds stop losing readings without an app release.
The tests pin both spellings, the range after conversion, and that the
tolerant converter covers the percent class and nothing else: waist
circumference uses the same factor to turn metres into centimetres, and
a blanket change made a 1.02 m waist into 1.02 cm.
Refs MBombeck/healthlog-iOS#7
The catch-all angle-bracket pattern read as an HTML sanitiser to static analysis. The annotation Apple writes is a number, so the pattern now matches exactly that, and a unit with anything else between the brackets stays unknown instead of being silently trimmed into a lookup.
Low, review 2. An account with one out-of-range candidate is refused
whole, rows and audit row included, but the dry run counted its rows into
"N row(s) would be repaired" — so the number an operator read before
--apply was larger than the number --apply could ever produce. The refused
accounts are now counted and reported separately, and the per-account line
says its rows are not in the total.
The refusal itself told the operator to check the Health app "before
forcing anything". There is no --force and never was. It now names the
only route that exists: re-run with the matching --unit if the archive
really was in another one.
The rollups and the cached status insights are recomputed after the
transaction, so a throw there left the rows repaired and every chart on
the pre-repair numbers — and no later run would notice, because the
audit row makes the account skip from then on.
The tail is wrapped now. The audit row is written with
`rollupsRefreshed: false` and only flipped once the recompute returned,
so a re-run can tell "repaired, rollups pending" from "done" and says so.
A run that could not finish the tail names the account, prints the
backfill to re-run — `scripts/backfill-rollups.ts --user <userId>` — and
exits non-zero.
Review finding M2.
The provenance criterion proves where a row came from, not which build
imported it. An account re-imported on the fixed build carries the same
`EXPORT_XML_SOURCE_MAX` stamp and the same `stats:` external id with the
right numbers and no audit row, so it is selected exactly like a broken
one — and its rest days, under 200 m because the phone stayed at home,
survive the plausibility check after another thousandfold multiply.
So one candidate row that would leave the range now refuses the whole
account: nothing written, no audit row, the offending rows and a printed
reason instead. The idempotency guard only ever held against the
script's own prior run; this holds against the case it could not see.
The runbook says the same in words — a re-imported account must not be
repaired, the re-import already did it — and says what the dry run is
for: it is where the decision is made, not a formality in front of one
already taken.
Integration coverage for the re-imported account: correct rows, an
`--apply` run, nothing changes and no audit row is left behind.
Review finding H1.
The record-unit conversion is now opt-in per caller and only the
`export.xml` archive path opts in. There the attribute is authoritative:
Apple stamps every quantity `<Record>` with the account's own display
unit, so `km` means kilometres and the reading is meaningless without it.
`POST /api/measurements/batch` keeps its documented contract — `unit` is
captured for audit and never read. Nothing in the tree pins the native
client's unit strings for the seventeen convertible identifiers, and a
client that reads a sample in metres while stamping the person's display
unit into the field would have had every walking distance multiplied by a
thousand on ingest. Reading the field there is a wire change and needs
the client's strings pinned first.
The mapping-table audit and the shared conversion module stay. The batch
path's identity behaviour is written down where it is decided and pinned
twice: every convertible identifier round-trips a sibling unit unchanged
by default, and a walking-distance entry sent as `km` is stored as it
arrived. The OpenAPI description says what the server actually does.
Review finding H2.
`totalEnergyBurned` rides the same account-display-unit contract as the
distance beside it: a Health app set to kilojoules writes `kJ`, and the
column is kilocalories. Convert it through the shared table.
Refs #944
The unit fix only helps the next import. An account that imported an
archive before it carries a decade of days understated 1000x, and the
authoritative repair — re-importing the archive on the fixed build —
needs an archive the account may no longer have.
`scripts/repair-apple-health-distance.ts` is dry-run by default and
selects rows only by the stamp the archive fold itself writes: the
EXPORT_XML_SOURCE_MAX provenance plus the `stats:` external id minted
for that fold. Never by "the value looks small", so a row a native sync
has since overwritten is left alone. `--apply` multiplies one account
per transaction and writes the audit row in the same transaction, which
is what makes a second run a no-op. The archive's own unit is named by
the operator (`--unit=mi`) because the discarded attribute is the defect
itself, and a row whose repaired value would leave the plausible range
is reported rather than written.
Refs #944
Apple stamps every quantity `<Record>` of an export archive with the
account's own display unit, not the unit the mapping table assumes: a
metric archive reports walking distance in km, an imperial one in mi,
body mass in lb, energy in kJ, temperature in degF. The mapper accepted
`input.unit` and never read it, so a 2.484 km day was folded into the
day's total as 2.484 m and a decade of imported history read as a couple
of metres walked.
`hk-units.ts` is now the one place a HealthKit unit is converted, shared
by the record path and the workout path (which carried its own km/mi
branch — the duplicate is what let the record path stay wrong). It
converts only inside a family and only for units Apple can emit, and
returns the reading untouched for anything it cannot place, so an
archive that already speaks the table's unit is unchanged.
Every mapping-table entry whose unit cannot be converted — a
dimensionless count, a pinned event, a logarithmic dB level, a compound
rate, the percent units whose value rides as a 0..1 fraction — now
carries a written reason, and a structural guard fails on an entry that
offers neither a convertible unit nor a reason.
Refs #944
The Apple export writes DistanceWalkingRunning in the account's own
length unit; the record mapping assumes metres and never reads the
attribute, so a 2.484 km day lands as 2.484 m.
Refs #944
A value imported in mmol/L was multiplied by 18.016; one displayed in
mmol/L was divided by 18.0182. The two sat in different files, both
correct-looking, and a reading that went in as 5.3 did not come back out
as 5.3. Latent while nobody could select mmol/L, live the moment they can.
One exported constant now, on the guideline figure the display path
already used, and the glucose insight page takes it from there rather than
keeping a third copy. The test names the constant instead of repeating the
digits, which is how the two came apart in the first place, and it asserts
the round trip a reader on mmol/L actually experiences.
The import status endpoint returns the instant the Health app stamped on
the archive, the published contract requires the field, and the schema
comment said the parser filled it during unpack. It never did: ExportDate
sat on the parser's ignore list, so the column answered null on every job
ever run, for a year.
The parser reads the element now and hands the stamp to the worker on the
next chunk boundary, so the value lands while the run is still going and a
job that later fails still records which export it was working from. An
archive that omits the element, or writes a value the runtime cannot read,
leaves the column null rather than guessing at an instant.
No contract change: the endpoint and the OpenAPI description already
described this behaviour.
Since v1.30.17 there has been no self-service path to a token that can
write a measurement. That release made the Bearer scope default
fail-closed, which closed a real privilege-escalation hole, and removed
the generic mint at POST /api/tokens along with it. The generic mint had
been the only way to get a token that could reach /api/measurements, so
a local sensor pushed through Home Assistant or Node-RED lost its door
while the OAuth integrations kept theirs.
Add a narrow scope accepted by exactly two routes, both writes, both on
the holder's own record.
The scope constant is a leaf module beside the surface that consumes it,
matching FHIR_READ_SCOPE and the MCP scopes. requireRecordAuth grows an
options object rather than a sibling function, because the sharing
surface guard keys every leg on that one identifier and a second name
would mean two copies of the delegable declaration. AuthContext carries
the admitting scope as a single nullable string rather than the token's
permission array, so nothing downstream can re-derive an authorisation
decision from it.
A scoped credential is refused a record selector before any grant is
read. Placing the refusal above the sharing check rather than beside it
means no future widening of what a grant permits can reach one of these
tokens.
Rows are attributed MANUAL. An entry that explicitly names APPLE_HEALTH
is refused 422 rather than relabelled: that source is half the
(userId, type, source, externalId) dedup key the iOS client writes into,
it participates in the cross-source merge, and it gates the HealthKit
sync fields. For the same reason a scoped write no longer advances
User.lastSyncedAt, which the phone reads to size its undelivered window.
Also closes a narrower gap found while building this. The idempotency
wrapper resolved a replayed delegated response against a live grant
without considering which credential was presenting it, so a cached
response about a shared record could be returned to a token the handler
itself would refuse. No write was reachable that way; the response body
was.
Guards: T3 gains the mint site, T4's vocabulary gains the scope and
learns to read the requireRecordAuth form (without which this change
would open a way to widen a route that the closed vocabulary could not
see), and a new T5 freezes the set of delegable routes naming a scope.
The sharing surface guard's AST matcher reads only the first two
arguments, so it needed prose but no code.
The two halves of the ECG fix were built apart. One taught the parser
Apple's full English verdict set; the other taught it the metadata keys and
verdict wording a German watch writes. Each was correct alone, and each
left "Sinusrhythmus" unresolved for a different reason.
Together they meet: the alias table translates the wording onto Apple's
own English, and the classifier decides what it means. One place still
owns the meaning, which is what stops a per-language verdict table from
drifting the first time Apple adds a case.
The test that expected null was pinning the gap between the branches
rather than a decision about German. It now expects NOT_DETECTED, and a
French verdict takes over the null case, because four of the six
languages genuinely have no observed key map yet.
The CSV is written in the device's language, and the keys are translated
along with the values: a German watch writes `Aufzeichnungsdatum` and
`Klassifizierung`, not `Recorded Date` and `Classification`. Indexing on
the English keys found none of them, so the parser never located the
waveform and rejected the recording outright — a lost import, not a
missing field.
Two alias maps close it, both taken from real exports rather than from a
dictionary: nine metadata keys and two verdicts, consistent across every
German file observed. Adding a language is a data addition to those maps
and nothing else.
The verdict map is an indirection onto the English wording rather than a
second verdict-to-enum table, so there stays exactly one place that
decides what a verdict means.
The other four shipped languages are absent on purpose. A wrong key
silently mis-files a value and a wrong clinical verdict is worse than
none, so an export this parser cannot place is refused — which an operator
can see — rather than read into the wrong column, which they cannot. Two
gaps are written down at the map: no observed file carries an
average-heart-rate row in any language, and the paired layout's column
header has only ever been seen in English.
The rate matcher demanded "Hz". Observed exports spell the unit out and
lower-case it — "511,422 hertz" — and the German ones separate it from the
number with a no-break space. The fixtures had only ever used "512 Hz", so
the mismatch never showed.
Rate and heart rate now split into a magnitude and a unit word, with the
magnitude read through the same decimal helper the waveform uses. "512 Hz",
"512 hertz" and "511,562 Hertz" all resolve; a missing unit, the wrong
unit, a grouped number and a rate past the bound are all still refused.
The unit word is matched loosely and the magnitude strictly. That is the
right way round: the unit is presentation and varies by region, while the
number is the measurement.
A watch set to a comma-decimal region writes its samples as "-180,596".
The single-column reader rejected any row containing a comma outright, so
every one of those files failed on its first sample row.
The two waveform layouts disagree about what a comma means, which is why
this cannot be decided per line. The parser now names the section it is in
— header, single-column, paired — and both crossings are commented at the
point they happen: the paired layout announces itself with a `Lead,Voltage`
column header, while the single-column one begins at the first blank line
AFTER Lead and Unit have been declared, because real exports also carry
blank lines mid-header.
Scalars in both waveforms now read either dialect through one helper, so
"-180.596" and "-180,596" give the same microvolts and neither is
privileged. A number carrying thousands grouping is refused rather than
guessed at, since resolving it needs the file's region and the file never
states it. The one genuinely ambiguous shape is a separator followed by
exactly three digits; it is read as a decimal, because observed waveforms
put two- and three-digit fractions in the same file and grouping is always
three, and because a machine writing one bare value per row has no reason
to group at all.
Every real export begins with a bare `Name` when the name field is blank —
Apple omits the trailing comma rather than writing an empty value. The
parser required a comma on every header row, so it rejected the file on
line 1, before it had read anything.
A valueless header row is now skipped. It contributes no metadata entry
rather than an empty-string one, because `Lead` and `Unit` presence is what
switches the parser into the single-column waveform mode and a key with no
value must not be able to trip that switch.
The relaxation is confined to the header. Inside the single-column
waveform a comma-less row is a sample and is read as one; inside the
paired waveform every row is lead and voltage, so a missing comma there is
still malformed.
The classification map held three values. Apple's set is larger, so a real
verdict was being thrown away on English devices: "Heart Rate Over 120"
and "Heart Rate Under 50" both became null, and the column cannot tell
that apart from a recording that carries no classification at all.
The wordings are observed rather than remembered. The CSV writes a display
string and it does not match the framework's case names — the case
documented as InconclusiveHighHeartRate reaches the file as "Heart Rate
Over 120". Across a corpus of 128 real exports the field held five
distinct values; three of them mapped and two were dropped. The threshold
belongs to the watch generation, so the two heart-rate arms match the
number instead of pinning it, and the rest of the map takes its wording
from Apple's ECG instructions for use.
Classification is now split from the column write so the parser can tell a
verdict it has never heard of from one it knows it cannot represent.
Unrecognized is the second kind: the device did not fail to classify the
waveform, it received a verdict it does not know, so recording it as
"could not classify" would assert something Apple never said. Both still
store null, because that is the column's whole vocabulary, and no column
is added for a distinction nothing reads yet.
The old table pinned one row for "Localized or unknown verdict". That
assertion is still true, but it fused three different facts — a localised
verdict, an unknown one, and the two documented verdicts being silently
dropped beside them — and it read as "everything else is unknown" when two
of those were verdicts Apple documents. It is split into the four outcomes.
The language half stays open and is now named at the mapping. A German
export writes the KEY in German too, so a non-English verdict never
reaches this function; it fails the parse earlier.
Apple Health exports write ECG CSVs in two shapes. The importer only
understood the paired Lead,Voltage form; the single-column form (Lead
and Unit as metadata rows, a blank separator, then one bare reading per
line) failed every row as malformed and the whole recording was skipped.
The parser now treats a blank line after Lead and Unit metadata as the
start of the waveform, reads one bounded numeric sample per row, and
converts the declared unit to integer microvolts before the existing
amplitude and safe-integer bounds. The paired layout parses exactly as
before, and an unknown unit fails with a fixed message.
Refs #794
Three silent gaps in the import accounting, one commit:
The reproductive samples of an export land in recordsRead but were
discarded without a counter when cycle tracking is off — the result now
names them (samplesSkippedModuleDisabled). A cycle flush that failed
partway zeroed its stats AFTER day-by-day writes had landed; the flush
is now fault-isolated per day and reports daysFailed plus the first
failure reason instead of claiming nothing happened. And the import
card hardcoded skipped: 0 while the worker has computed, persisted and
served per-class skip counters since v1.15 — it now renders them,
splitting deliberate exclusions (unsupported and deferred types, a
disabled cycle module) from genuine refusals (unparseable, out of
range, failed writes), with a per-type breakdown. Only refusals tip
the outcome tone.
Generate a 700k-record export.xml (~200 MB decompressed, 500k
cumulative step records each carrying a fresh HKDevice runtime address,
200k interleaved spot rows) streamed into a temp stored-method ZIP —
never checked in, never held in memory whole — and push it through the
real extractExportXml + streamParseExportXml pipeline against a
discarding Prisma stand-in.
Live heap is measured after forced collections at every progress tick;
the asserted growth bound (48 MiB) is a quarter of the decompressed
volume. Both historical failure modes blow past it: reverting the
device-address normalization retains ~61 MiB of fragmented source
buckets, and unbounding the batch buffers retains ~76 MiB and starves
the progress ticks. The fixed pipeline measures under 1 MiB of live
growth. The suite also pins semantics alongside memory: every record
read, every spot row flushed, and exactly one cumulative estimate row
per day rather than one per device address. Refs #775.
The export.zip extractor buffered the whole archive through
readFileSync — hundreds of MB resident for the entire unpack phase on
a real multi-year export. The central directory is now located through
a bounded tail read on a file handle, read as one size-capped slice,
and the export.xml member streams through a byte-range read stream
into the inflate pipeline, so peak extraction memory no longer scales
with archive size.
The cumulative fold hashed the raw device attribute into its per-day
source buckets. Apple serialises that attribute from the live HKDevice
object including its runtime address (<<HKDevice: 0x283a08640>, ...),
which is not stable across the export enumeration — the map fragmented
toward one bucket per record and an 11-million-record export grew it
past the default 1 GB Node heap, killing the worker mid-parse (#775).
Stripping the volatile address before hashing restores the documented
invariant: parser state bounded by actual source cardinality.
Also carry split multi-byte UTF-8 sequences across read-chunk
boundaries with a StringDecoder — a bare per-chunk toString decayed a
split umlaut in a source name to U+FFFD.
Every interactive path, every import, MCP and Telegram checked an incoming
measurement against VALUE_RANGES before storing it. The provider sync writers
did not, so a decode slip or a glitched sensor landed a row the application
itself calls impossible, and every mean, median and personal band derived from
it inherited the impossibility. Filtering the read side hid the row. It kept
landing.
The gate is one module reading the domain the app already declares, applied at
two kinds of seam. reconcileExternalMeasurement is the shared identity writer
for Polar, Oura, WHOOP, the Withings sleep leg, the Apple export importer and
the batch endpoint, so gating there means a provider added later inherits the
refusal rather than having to remember a check it never knew about; the new
rejected_range verdict carries no row, which is what makes the compiler force
every caller to handle it. The five writers that hold Prisma themselves gate at
their entry, before the existence probe decides insert versus overwrite, so an
impossible reading can no longer replace a good stored value.
An out-of-band value is dropped. Not clamped, because a clamp invents a reading
nobody took. Not stored behind a flag, because a quarantined row is still a row
and every consumer would have to learn the flag. Not fatal to the batch, because
one bad sample must not strand the run's other readings behind a held watermark.
Absence reads as absence.
A silent drop, though, is indistinguishable from a provider that sent nothing,
so every drop is tallied on the wide event under measurement.range_rejected with
the metric, the provider and the edge it broke. Counts only: the reading itself
is health data and never enters the log line, and the direction plus the
metric's own declared bound is what a diagnosis actually needs. The tally
accumulates rather than replacing, because a provider gates once per resource
leg and the reconciler once per row. One warning per provider lifts the event to
warn so a drop does not hide inside a green line.
A structural guard enumerates all twenty measurement writers with the reason
each is allowed to write, so a new one has to answer the question before it
ships. scripts/diagnose-implausible-measurements.ts reports the rows already
stored, aggregated per metric and source with no value beyond the bucket's own
extremes. It repairs nothing: whether a stored impossible row is a decode slip
or a band that is too narrow is not a call a script gets to make.
`hasEntryUnitConversion` and `canonicalValueToEntry` shipped beside the
hours-to-minutes conversion for an editing surface that does not exist and has
no ticket. Neither has a caller, so both were only two more things to keep
true, and the dead-code gate is right to say so.
The module comment claimed the inverse was there so a future surface could not
land on a different answer. It says what is actually true now: one direction,
one place. The git history keeps the rest if that surface ever arrives.
The manual sleep field is labelled in hours. The column stores minutes, because
that is what every sync source delivers and because a single stage segment is
only a few of them. The form sent the typed number straight through, so a night
entered as 7.5 was filed as seven and a half minutes and the list showed eight
minutes of sleep.
Nothing rejected it. Seven and a half minutes is inside the column's
plausibility band — a stage really can be that short — so the entry looked
accepted and the loss was silent.
Both ends were already right: the field says hours, the list formats minutes as
hours and minutes. The assembly between them was where the factor went missing,
which is why the round-trip test runs the whole of it — the form's conversion,
the shipped POST, the stored row, and the formatter the list paints with.
The storage unit stays minutes. The conversion sits at the entry boundary in
one place with its inverse beside it, so a future editing surface cannot land
on a different answer, and it rounds to whole minutes rather than storing
437.99999999999994 for 7.3 hours.
The value field also lost "7,5" outright. `type="number"` sanitises a decimal
comma away on the engines that do not localise the field, so the entry arrived
as an empty string with no explanation, and `parseFloat` would have read it as
7 on the engines where it survived. The field now takes the same shape the labs
and custom-metric fields use — a numeric keypad and a submit path that reads
the comma — so every decimal metric on the form gains it, not only sleep.
`scripts/diagnose-manual-sleep-hours.ts` lists the rows already stored this
way. It reports and does not repair: an eight-minute stage and a night typed as
eight hours are the same number in the same column, and only the person who
entered it knows which it was.
The baseline-drift card reported a pulse in the hundreds of millions against a
usual range in the tens of thousands. Neither figure is a heart rate, and both
came from the same place.
The application declares a plausibility domain per metric, and every
interactive, import, MCP and Telegram write validates against it. The
derivation layer did not ask. So a single stored value outside that domain — a
provider unit-decode slip, or a row from a writer that never had the gate — was
folded into the day mean, carried by the median into the band, and then held
against itself as "currently". The two numbers in the sentence were wrong by
different amounts, which is why the line read as nonsense rather than as an
exaggeration.
Both readers now refuse a value the application has already declared
impossible: the per-day means behind the band, and the latest-day mean the
deviation reads. A day left with no plausible reading contributes nothing
rather than a fabricated point, and the vital drops out of the comparison — the
coverage floor already knows how to report an absent day. The rollup branch
applies the same rule to a bucket mean, since the rollup writer averages
whatever is stored. Everything downstream of `readDayMeanSeries` inherits it,
which is every consumer of the usual range: the band strip, the drift card, the
changepoint detector, readiness, trajectory, the coach read.
The card also printed raw doubles. Each figure now renders at the metric's own
decimal precision through the shared formatter, so a mean reads as a number
rather than as a float and follows the reader's number convention.
Left open, and recorded for a separate pass: the provider sync writers do not
enforce the domain on the way in, so the impossible rows keep landing.
The canonical picker keyed a session on sport, start, end and duration. Only the
workout list selected the end timestamp; the Coach narration, the per-sport
baseline and the deep-link cluster passed rows without it, so those rows carried
no session identity at all and the duplicate survived. One record counted once in
the list and two or three times everywhere else, including in the averages a
person is compared against.
Drop the end timestamp from the identity. On a workout with pauses it measures
wall-clock time while the duration counts only moving time, so two re-sends of
one session can differ in that field alone and the stricter key rejected a
genuine match. It also bought nothing in exchange: nobody starts a second
workout of the same sport in the same instant with the same duration.
The deep-link cluster now selects the duration it needs. A structural guard
freezes the call-site set and records which file owns each caller's select, so a
new surface cannot join the set without fetching the identity, and the identity
cannot grow a field only some callers fetch.
Marc-André Bombeck