Merge the Apple Health record-unit fix and the distance repair (#944)

This commit is contained in:
Marc-André Bombeck committed 2026-09-10 01:06:58 +02:00
commit 071b110e8b
14 files changed
+1821 -21

No files matched your search

+3
View File
@@ -21350,6 +21350,9 @@ components:
type: string
minLength: 1
maxLength: 60
description: "Apple's `HKUnit` string for `value`, captured for audit. The server does NOT convert from it: `value` is
read in the unit HealthLog stores the identifier in (`m` for walking distance, `kg` for body mass, `kcal`
for energy, `degC` for temperature). Send the sample in that unit."
startDate:
type: string
format: date-time
+156
View File
@@ -0,0 +1,156 @@
# Apple Health walking-distance repair
`scripts/repair-apple-health-distance.ts` heals walking distances that the
`export.zip` import stored a thousand times too small.
## The defect
Apple stamps every quantity `<Record>` in an `export.xml` with the account's
own display unit: a metric archive reports `HKQuantityTypeIdentifierDistance-
WalkingRunning` in `km`, an imperial one in `mi`. Until this release the record
mapping accepted that attribute and never read it, so the day's total was
folded as if the number were already metres. A 2.484 km day landed as
`2.484 m`, and an imported decade reads as a couple of metres walked. The
workout path was never affected — it converted its own distance unit.
The code fix converts the record's unit before storage, so every import from
this release on is correct. This script is for the rows already in the
database.
## Prefer a re-import
If the account still has its `export.zip`, **re-import it**. The archive fold
UPSERTs the same `stats:<identifier>:<day>` rows, so a re-import on the fixed
build rewrites every affected day with the right number, needs no flags, and
also repairs rows this script deliberately leaves alone (see the criterion
below). The script exists for accounts that no longer have the archive.
**An account that has been re-imported must not then be repaired.** The
re-import already fixed it. Its rows keep the same `EXPORT_XML_SOURCE_MAX`
stamp and the same `stats:` external id, and they carry no audit row, so the
criterion selects them exactly as it selects a broken account — the stamp
proves where a row came from, not which build imported it. Multiplying them
again turns a 2.484 km day into 2484 km. The script refuses such an account on
its own (any day whose value would leave the plausible range after the
multiply stops the whole account), but that guard fires on the big days: keep
the two apart yourself before you type `--apply`.
## Which rows it touches
Only rows carrying the stamp the archive importer itself writes:
| Column | Value |
| ------------------------ | ------------------------------------------------------------ |
| `type` | `WALKING_RUNNING_DISTANCE` |
| `source` | `APPLE_HEALTH` |
| `aggregation_provenance` | `EXPORT_XML_SOURCE_MAX` |
| `external_id` | `stats:HKQuantityTypeIdentifierDistanceWalkingRunning:<day>` |
| `deleted_at` | `NULL` |
`EXPORT_XML_SOURCE_MAX` is written in exactly one place in the tree — the
archive fold — and the `stats:` external id is minted for that same fold, so
the pair is a proof of origin rather than a guess. Nothing is ever selected
because a value "looks too small".
Deliberately out of scope:
- Rows a native iOS sync has since overwritten (`HEALTHKIT_STATISTICS`): their
value came from HealthKit's own statistics query and is correct.
- Rows whose provenance is `LEGACY_UNKNOWN` or NULL: an archive origin cannot
be proven for them. Re-import the archive to repair those.
## Run a dry-run first
Always, and read it. The dry run is where the decision is made, not a
formality in front of a decision already taken: it is the only place the
script shows you real numbers from the account before it changes them. If the
worked example does not match what that person's Health app shows for that day,
the run is wrong and `--apply` will make it permanent.
The default mode reports and never writes:
```bash
# from a source checkout, with DATABASE_URL pointing at the instance
pnpm dlx tsx scripts/repair-apple-health-distance.ts
```
It prints one line per account: how many rows would be repaired, the date span
they cover, one worked example (`2.484 m -> 2484 m`), and any row whose
repaired value would leave the plausibility range.
This script imports the application dependency graph, so it runs from a source
checkout, not through the production image's `healthlog-tsx` launcher.
## Name the archive's unit
The stored row keeps no record of the unit its archive was written in — that
is the defect — so the operator names it. `--unit=km` is the default and is
what the Health app writes in every metric locale; pass `--unit=mi` for an
imperial archive:
```bash
pnpm dlx tsx scripts/repair-apple-health-distance.ts --unit=mi
```
Check the dry-run's worked example against what the account's Health app shows
for that day before applying. If the two units are mixed across accounts, run
the script once per unit with `--apply` and confirm each account's example
first.
## Apply
Take a backup first — `docs/ops/backup-restore.md`. The multiply is in place
and has no reverse operation: once an account is written wrong, the database
copy you took beforehand is the only way back.
```bash
pnpm dlx tsx scripts/repair-apple-health-distance.ts --apply
```
Per account, in one transaction: every selected row multiplied by the unit's
metre factor, `sync_version` incremented and `updated_at` bumped so paired iOS
clients pick the rows up on their next delta sync, and an audit row
(`measurement.apple_health_distance.repaired`) recording the unit, the factor
and the row count. The DAY/WEEK/MONTH/YEAR rollups and the cached status
insights for the touched days are recomputed afterwards, outside that
transaction.
If that recompute fails, the rows are right and every chart is not, and no
later run of this script would notice — it skips a repaired account. So the
audit row is written with `rollupsRefreshed: false` and only flipped to `true`
once the recompute has returned. A run that could not finish it prints the
account and the command to re-run, exits non-zero, and a later dry run repeats
the warning until the backfill has been run:
```bash
pnpm dlx tsx scripts/backfill-rollups.ts --user <userId>
```
A row whose repaired value would leave the plausible range (0–200 000 m)
refuses the **whole account**: nothing is written, the offending rows and the
reason are printed, and the run moves on to the next account. One such row is
evidence the account is not the 1000x class — a re-imported account looks
exactly like this — and repairing the rest of it would multiply days that were
already right.
## It runs once per account
The audit row is written inside the same transaction as the update, and an
account that carries it is skipped. A second run reports the account as already
repaired and changes nothing, so the multiplier can never be applied twice.
## Verify
```sql
SELECT date_trunc('year', measured_at) AS year,
round(avg(value)) AS avg_metres_per_day
FROM measurements
WHERE user_id = '<userId>'
AND type = 'WALKING_RUNNING_DISTANCE'
AND aggregation_provenance = 'EXPORT_XML_SOURCE_MAX'
GROUP BY 1 ORDER BY 1;
```
A plausible daily average is thousands of metres. Single digits mean the rows
were not repaired; millions mean a factor was applied twice, which the audit
row is there to prevent.
+1
View File
@@ -24,6 +24,7 @@ typecheck.
- **`assert-deploy.ts`** — post-deploy `/api/version` assertion.
- **`rotate-encryption-key.ts`** — AES-256-GCM key rotation; see `docs/ops/encryption-key-rotation.md`.
- **`restore-backup.ts`** — restore an off-host AES-GCM backup from an S3-compatible bucket.
- **`repair-apple-health-distance.ts`** — repair walking-distance rows the `export.zip` import stored 1000x too small; dry-run by default, see `docs/ops/apple-health-distance-repair.md`.
- **`repair-intake-anomalies.ts`** — repair historic medication-ledger defects (duplicate dose-slot rows, implausible `taken_at`); dry-run by default, see `docs/ops/intake-repair.md`.
- **`backfill-rollups.ts`**, **`drain-per-sample-cumulative.ts`**, **`backfill-mood-note-column.ts`** — historical data backfills (note: the boot-time `rollup-full-backfill` queue handles new accounts automatically; the CLI is the manual fallback).
- **`seed-demo.ts`** + **`seed-demo-v15.sql`** — populate a demo tenant.
+190
View File
@@ -0,0 +1,190 @@
/**
* scripts/repair-apple-health-distance.ts — operator repair for walking
* distances the `export.zip` import stored 1000x too small (issue #944).
*
* THE DEFECT (fixed in code from this release): the `<Record>` mapping read
* the value but never the record's own `unit` attribute, so a walking-distance
* record Apple writes in the account's display unit (`km` on a metric archive,
* `mi` on an imperial one) was folded into the day's total as if the number
* were already metres. Ten years of history read as a couple of metres a day.
*
* WHICH ROWS: only the ones carrying the stamp the archive importer writes —
* `WALKING_RUNNING_DISTANCE` + `APPLE_HEALTH` + provenance
* `EXPORT_XML_SOURCE_MAX` + a `stats:HKQuantityTypeIdentifierDistanceWalking-
* Running:<day>` externalId. Never a guess from "the value looks small". Rows
* a native iOS sync has since overwritten carry `HEALTHKIT_STATISTICS` and are
* correct; rows whose provenance cannot prove an archive origin are left alone.
*
* If the account still has its `export.zip`, RE-IMPORTING it on the fixed build
* is the authoritative repair — the fold UPSERTs the same `stats:` rows with
* the right numbers. This script is for the accounts that no longer have it.
*
* SAFETY:
* - Dry-run by DEFAULT: lists the candidate rows per account and exits.
* - `--apply` writes, one transaction per account, and records an audit row
* in that same transaction. An account carrying that row is skipped, so a
* second run finds nothing.
* - A row whose repaired value would leave the plausibility range refuses
* the WHOLE account: one such row is evidence the rows are not the 1000x
* class (an account re-imported on the fixed build looks exactly like
* this), and its rest days would otherwise be multiplied again.
* - The rollup/status tail runs after the transaction. If it throws, the
* rows are repaired and the derived tier is not; the script names the
* backfill to re-run, the audit row keeps `rollupsRefreshed: false` so a
* later run can still see it, and the exit code is non-zero.
* - `--unit=mi` for an imperial archive (default `km`). The stored row keeps
* no record of the archive's unit — that is the defect — so the operator
* names it; a mismatched unit is why the flag exists rather than a guess.
*
* RUN (never `pnpm tsx` — the standalone image strips tsx):
* pnpm dlx tsx scripts/repair-apple-health-distance.ts # dry run
* pnpm dlx tsx scripts/repair-apple-health-distance.ts --apply
* pnpm dlx tsx scripts/repair-apple-health-distance.ts --unit=mi --apply
*/
import "dotenv/config";
import { prisma } from "@/lib/db";
import {
applyAppleHealthDistanceRepair,
parseArchiveUnit,
planAppleHealthDistanceRepair,
repairFactorFor,
rollupRepairCommand,
} from "@/lib/measurements/repair-apple-health-distance";
function readArchiveUnit(): "km" | "mi" {
const flag = process.argv.find((arg) => arg.startsWith("--unit="));
if (!flag) return "km";
const parsed = parseArchiveUnit(flag.slice("--unit=".length));
if (!parsed) {
throw new Error(`Unsupported --unit value: ${flag}. Use km or mi.`);
}
return parsed;
}
async function main(): Promise<void> {
const apply = process.argv.includes("--apply");
const archiveUnit = readArchiveUnit();
const factor = repairFactorFor(archiveUnit);
const plans = await planAppleHealthDistanceRepair(prisma, { archiveUnit });
if (plans.length === 0) {
console.log(
"No export-imported walking-distance rows found — nothing to repair.",
);
return;
}
console.log(
`Archive unit ${archiveUnit} (x${factor} to metres). ` +
`${plans.length} account(s) carry export-imported walking distances.\n`,
);
let totalRepairable = 0;
let totalUpdated = 0;
let totalSkipped = 0;
let refused = 0;
let rollupsPending = 0;
for (const plan of plans) {
if (plan.alreadyRepairedAt) {
console.log(
` • ${plan.userId}: already repaired on ` +
`${plan.alreadyRepairedAt.toISOString()} — ${plan.candidateCount} row(s) left alone.`,
);
if (plan.rollupsPending) {
rollupsPending += 1;
console.log(
" ! rollups PENDING from that run: the DAY/WEEK/MONTH/YEAR " +
"buckets and the cached status insights still hold the " +
"pre-repair numbers.",
);
console.log(` re-run: ${rollupRepairCommand(plan.userId)}`);
}
continue;
}
totalRepairable += plan.repairable.length;
const first = plan.repairable[0];
const last = plan.repairable[plan.repairable.length - 1];
console.log(
` • ${plan.userId}: ${plan.repairable.length} row(s)` +
(first && last
? ` from ${first.measuredAt.toISOString().slice(0, 10)} to ` +
`${last.measuredAt.toISOString().slice(0, 10)}`
: "") +
(plan.outOfRange.length > 0
? `, ${plan.outOfRange.length} out of plausible range (skipped)`
: ""),
);
if (first) {
console.log(
` e.g. ${first.id}: ${first.value} m -> ${first.value * factor} m`,
);
}
for (const row of plan.outOfRange) {
console.log(
` ! ${row.id} (${row.measuredAt.toISOString().slice(0, 10)}): ` +
`${row.value} m x ${factor} leaves the plausible range`,
);
}
if (plan.outOfRange.length > 0) {
console.log(
" ! this account will be REFUSED — see the reason under --apply",
);
}
if (!apply) continue;
const outcome = await applyAppleHealthDistanceRepair(prisma, plan, {
archiveUnit,
});
totalUpdated += outcome.updated;
totalSkipped += outcome.skipped;
if (outcome.refusedReason) {
refused += 1;
console.log(` ✗ refused: ${outcome.refusedReason}`);
continue;
}
console.log(
` ✓ repaired ${outcome.updated} row(s), skipped ${outcome.skipped}`,
);
if (outcome.rollupsPending) {
rollupsPending += 1;
console.error(
` ! ${plan.userId}: rows repaired, but the rollup refresh failed.`,
);
console.error(outcome.rollupError);
console.error(
" The DAY/WEEK/MONTH/YEAR buckets and the cached status " +
"insights still hold the pre-repair numbers.",
);
console.error(` re-run: ${rollupRepairCommand(plan.userId)}`);
}
}
console.log(
apply
? `\nDone. ${totalUpdated} row(s) repaired, ${totalSkipped} skipped, ` +
`${refused} account(s) refused.`
: `\nDry run: ${totalRepairable} row(s) would be repaired. ` +
"Read the worked examples above and decide before re-running with " +
"--apply.",
);
if (rollupsPending > 0) {
// The rows are right and the derived tier is not, which no later run of
// this script will notice — it skips a repaired account. Fail loudly.
console.error(
`\n${rollupsPending} account(s) need the rollup backfill re-run ` +
"(commands above).",
);
process.exitCode = 1;
}
}
main()
.catch((err) => {
console.error(err);
process.exitCode = 1;
})
.finally(() => prisma.$disconnect());
@@ -0,0 +1,246 @@
/**
* Structural guard on the Apple Health mapping table's unit column
* (issue #944).
*
* `AppleHealthMapping.hkUnit` records the unit HealthLog ASSUMES a
* HealthKit reading arrives in, and `convertToDbUnit` is written against
* that assumption. Apple's `export.xml` does not honour it: every quantity
* `<Record>` carries the account's own display unit, so a metric archive
* reports walking distance in `km` and an imperial one in `mi`. For ten
* years those readings were stored as if the number were already metres —
* the whole history understated by a factor of a thousand — because the
* mapper accepted `input.unit` and never read it.
*
* So an entry now has exactly two legal shapes:
*
* 1. its `hkUnit` is one `convertHkValue()` knows a factor for, in which
* case `mapAppleHealthEntry()` converts a differing record unit into
* it before `convertToDbUnit` runs; or
* 2. it carries a written `unitFixedReason` saying why no unit attribute
* Apple could write would change the reading (a dimensionless count,
* a pinned event, a logarithmic dB level, a compound rate).
*
* The conversion is opt-in per caller (`convertRecordUnit`), and only the
* archive path opts in: `POST /api/measurements/batch` documents its `unit`
* as captured-not-validated and nothing pins the native client's strings.
* The last block below holds that boundary, so the batch contract cannot be
* widened by accident.
*
* The reason arm alone would be a guard that cannot fail: an entry could
* declare a convertible unit while the mapper quietly ignored it. The
* behavioural block below therefore feeds every convertible entry a
* reading in a SIBLING unit and asserts the stored number moved by the
* documented factor — so deleting the conversion turns this guard red,
* not just the one reproduction test.
*
* A tripwire, not a proof: a reviewer who waves through a wrong factor
* defeats it.
*/
import { describe, it, expect } from "vitest";
import { readFileSync } from "node:fs";
import { join } from "node:path";
import {
APPLE_HEALTH_TYPE_MAP,
mapAppleHealthEntry,
} from "@/lib/measurements/apple-health-mapping";
import {
convertHkValue,
isConvertibleHkUnit,
} from "@/lib/measurements/hk-units";
const ENTRIES = Object.values(APPLE_HEALTH_TYPE_MAP);
/**
* Units from every family the table touches. The behavioural block picks
* the first one that converts into an entry's `hkUnit` and actually moves
* the number, so a new family only needs one representative here.
*/
const SIBLING_UNITS = [
"km",
"mi",
"cm",
"lb",
"st",
"kJ",
"cal",
"degF",
"K",
"mmol<180.156>/L",
"kPa",
"km/hr",
"hr",
"ms",
];
function siblingUnitFor(hkUnit: string): string | null {
for (const candidate of SIBLING_UNITS) {
const converted = convertHkValue(1, candidate, hkUnit);
if (converted !== null && converted !== 1) return candidate;
}
return null;
}
describe("Apple Health mapping — the record's own unit (issue #944)", () => {
it("examines the whole mapping table", () => {
// An empty match set is the failure mode this repo keeps rediscovering:
// a guard is green because it matched nothing, not because nothing is
// wrong. Pin the floor.
expect(ENTRIES.length).toBeGreaterThan(40);
});
it.each(ENTRIES)(
"$hkIdentifier either converts its record unit or says why it cannot",
(mapping) => {
if (isConvertibleHkUnit(mapping.hkUnit)) {
expect(mapping.unitFixedReason).toBeUndefined();
return;
}
expect(
mapping.unitFixedReason,
`${mapping.hkIdentifier} pins hkUnit "${mapping.hkUnit}" with no conversion and no written reason`,
).toBeTypeOf("string");
// Long enough to be a sentence rather than a shrug.
expect(mapping.unitFixedReason!.length).toBeGreaterThan(40);
},
);
const convertible = ENTRIES.filter((mapping) =>
isConvertibleHkUnit(mapping.hkUnit),
);
it("has convertible entries to exercise", () => {
expect(convertible.length).toBeGreaterThan(10);
});
it.each(convertible)(
"$hkIdentifier honours a sibling unit on the record itself",
(mapping) => {
const sibling = siblingUnitFor(mapping.hkUnit);
expect(
sibling,
`no sibling unit listed for the family of "${mapping.hkUnit}"`,
).not.toBeNull();
const base = {
hkIdentifier: mapping.hkIdentifier,
value: 2,
startDate: "2026-05-14T08:00:00.000Z",
endDate: "2026-05-14T08:30:00.000Z",
};
const native = mapAppleHealthEntry(
{ ...base, unit: mapping.hkUnit },
{ convertRecordUnit: true },
);
const foreign = mapAppleHealthEntry(
{ ...base, unit: sibling! },
{ convertRecordUnit: true },
);
expect(native).not.toBeNull();
expect(foreign).not.toBeNull();
const expected = mapping.convertToDbUnit(
convertHkValue(2, sibling!, mapping.hkUnit)!,
);
expect(foreign!.value).toBeCloseTo(expected, 9);
expect(foreign!.value).not.toBeCloseTo(native!.value, 9);
},
);
it("keeps a unit it cannot place from rescaling the reading", () => {
const out = mapAppleHealthEntry(
{
hkIdentifier: "HKQuantityTypeIdentifierDistanceWalkingRunning",
value: 1234,
unit: "furlong",
startDate: "2026-05-14T08:00:00.000Z",
endDate: "2026-05-14T08:30:00.000Z",
},
{ convertRecordUnit: true },
);
expect(out?.value).toBe(1234);
});
it("leaves the batch path's readings alone whatever unit they name", () => {
// The batch contract: `unit` is captured, never read. Every convertible
// entry has to behave that way by DEFAULT, or a native client stamping
// the person's display unit silently rescales their history.
for (const mapping of convertible) {
const sibling = siblingUnitFor(mapping.hkUnit);
if (!sibling) continue;
const base = {
hkIdentifier: mapping.hkIdentifier,
value: 2,
startDate: "2026-05-14T08:00:00.000Z",
endDate: "2026-05-14T08:30:00.000Z",
};
const declared = mapAppleHealthEntry({ ...base, unit: mapping.hkUnit });
const foreign = mapAppleHealthEntry({ ...base, unit: sibling });
expect(declared).not.toBeNull();
expect(
foreign!.value,
`${mapping.hkIdentifier} rescaled a batch reading from "${sibling}"`,
).toBeCloseTo(declared!.value, 9);
}
});
it("has only the archive path opting into the conversion", () => {
// A grep, not a proof — but the wire change this pins is exactly the
// kind that lands as a one-word argument in an unrelated diff.
const optIn = "convertRecordUnit: true";
const archive = readFileSync(
join(process.cwd(), "src/lib/measurements/import-apple-health-export.ts"),
"utf8",
);
expect(archive).toContain(optIn);
const batch = readFileSync(
join(process.cwd(), "src/app/api/measurements/batch/route.ts"),
"utf8",
);
expect(
batch,
"the batch route opted into record-unit conversion",
).not.toContain("convertRecordUnit");
});
it("leaves the km/mi factors in one shared module", () => {
// The workout path used to carry its own copy of the km/mi branch; the
// duplicate is what let the `<Record>` path stay wrong for ten years.
//
// `* 1000` alone is not the offence — these files also turn seconds into
// milliseconds — so a rescale only counts when it stands in a LENGTH
// context: a distance identifier on the same line, or within the five
// lines above it (which covers a rescale in the body of a
// distance-named helper). The limit is the usual one for a matcher of
// this shape: a length conversion inside a helper named nothing like a
// distance, more than five lines below its own signature, slips it.
const shared = join(process.cwd(), "src/lib/measurements/hk-units.ts");
const owners = [
"src/lib/measurements/apple-health-mapping.ts",
"src/lib/measurements/import-apple-health-export.ts",
];
expect(readFileSync(shared, "utf8")).toContain("1609.344");
const RESCALE = /\*\s*1_?000(?!\d)/;
const LENGTH = /distance|metre|meter|\bkm\b/i;
for (const rel of owners) {
const source = readFileSync(join(process.cwd(), rel), "utf8");
expect(source, `${rel} re-implements the mile factor`).not.toContain(
"1609.344",
);
const lines = source.split("\n");
const offenders = lines
.map((line, i) => ({ line, i }))
.filter(
({ line, i }) =>
RESCALE.test(line) &&
lines.slice(Math.max(0, i - 5), i + 1).some((l) => LENGTH.test(l)),
)
.map(({ line, i }) => `${rel}:${i + 1}: ${line.trim()}`);
expect(
offenders,
`${rel} re-implements a length factor — the km/mi conversion belongs in hk-units.ts`,
).toEqual([]);
}
});
});
@@ -390,6 +390,38 @@ describe("mapAppleHealthEntry", () => {
expect(out!.takenAt.toISOString()).toBe(ts);
});
it("stores a batch reading in the unit the table assumes, whatever `unit` says", () => {
// issue #944 — the batch route's `unit` is captured for audit and never
// read. A client that stamps the person's DISPLAY unit into the field
// must not have its metres multiplied by a thousand: the conversion is
// opt-in and only the `export.xml` path opts in.
const out = mapAppleHealthEntry({
hkIdentifier: "HKQuantityTypeIdentifierDistanceWalkingRunning",
value: 2484,
unit: "km",
startDate: ts,
endDate: ts,
});
expect(out).not.toBeNull();
expect(out!.value).toBe(2484);
expect(out!.unit).toBe("m");
});
it("converts out of the record's own unit when the caller opts in", () => {
const out = mapAppleHealthEntry(
{
hkIdentifier: "HKQuantityTypeIdentifierDistanceWalkingRunning",
value: 2.484,
unit: "km",
startDate: ts,
endDate: ts,
},
{ convertRecordUnit: true },
);
expect(out).not.toBeNull();
expect(out!.value).toBeCloseTo(2484, 9);
});
it("applies the unit conversion for fraction-shaped quantities", () => {
const out = mapAppleHealthEntry({
hkIdentifier: "HKQuantityTypeIdentifierOxygenSaturation",
@@ -0,0 +1,77 @@
import { describe, it, expect } from "vitest";
import {
convertHkValue,
hkDistanceToMetres,
isConvertibleHkUnit,
} from "../hk-units";
describe("convertHkValue", () => {
it("converts the km reading Apple writes on a metric archive", () => {
expect(convertHkValue(2.484, "km", "m")).toBeCloseTo(2484, 9);
});
it("converts the mi reading Apple writes on an imperial archive", () => {
expect(convertHkValue(1.543, "mi", "m")).toBeCloseTo(2483.217792, 9);
});
it("returns the reading untouched when the units already agree", () => {
expect(convertHkValue(2.484, "m", "m")).toBe(2.484);
});
it("converts mass, energy and temperature the same way", () => {
expect(convertHkValue(180, "lb", "kg")).toBeCloseTo(81.6466266, 6);
expect(convertHkValue(1000, "kJ", "kcal")).toBeCloseTo(239.005736, 6);
expect(convertHkValue(98.6, "degF", "degC")).toBeCloseTo(37, 9);
});
it("strips the molar mass Apple annotates a glucose unit with", () => {
expect(convertHkValue(5.5, "mmol<180.156>/L", "mg/dL")).toBeCloseTo(
99.0858,
4,
);
});
it("keeps the small calorie apart from the large one", () => {
expect(convertHkValue(1, "Cal", "kcal")).toBe(1);
expect(convertHkValue(1000, "cal", "kcal")).toBeCloseTo(1, 9);
});
it("refuses a unit it cannot place rather than guessing", () => {
expect(convertHkValue(1, "furlong", "m")).toBeNull();
expect(convertHkValue(1, "", "m")).toBeNull();
expect(convertHkValue(1, undefined, "m")).toBeNull();
});
it("refuses to cross families", () => {
expect(convertHkValue(1, "kg", "m")).toBeNull();
expect(convertHkValue(1, "min", "kcal")).toBeNull();
});
it("refuses a non-finite reading", () => {
expect(convertHkValue(Number.NaN, "km", "m")).toBeNull();
});
});
describe("hkDistanceToMetres", () => {
it("converts the workout distance units Apple ships", () => {
expect(hkDistanceToMetres(6.5, "km")).toBeCloseTo(6500, 9);
expect(hkDistanceToMetres(3, "mi")).toBeCloseTo(4828.032, 9);
expect(hkDistanceToMetres(500, "m")).toBe(500);
});
it("falls back to the raw number for a missing unit attribute", () => {
expect(hkDistanceToMetres(500, "")).toBe(500);
expect(hkDistanceToMetres(500, undefined)).toBe(500);
});
});
describe("isConvertibleHkUnit", () => {
it("recognises the units the mapping table pins", () => {
expect(isConvertibleHkUnit("m")).toBe(true);
expect(isConvertibleHkUnit("kg")).toBe(true);
expect(isConvertibleHkUnit("count")).toBe(false);
expect(isConvertibleHkUnit("event")).toBe(false);
expect(isConvertibleHkUnit("%")).toBe(false);
});
});
@@ -1278,3 +1278,67 @@ describe("streamParseExportXml — the archive's own export stamp", () => {
expect(stamps).toEqual([]);
});
});
describe("streamParseExportXml — the record's own unit (issue #944)", () => {
it("stores a km-scale walking distance in metres", async () => {
const { prisma } = await importCumulativeFixture(
cumulativeExportXml(
"HKQuantityTypeIdentifierDistanceWalkingRunning",
"km",
[{ value: 2.484, sourceName: "iPhone", hour: 8 }],
),
);
expect(prisma._measurements).toHaveLength(1);
expect(prisma._measurements[0]).toMatchObject({
type: "WALKING_RUNNING_DISTANCE",
unit: "m",
});
expect(prisma._measurements[0].value).toBeCloseTo(2484, 6);
});
it("stores a kilojoule workout burn in kilocalories", async () => {
const tmp = mkdtempSync(join(tmpdir(), "healthlog-parser-energy-test-"));
const xmlPath = join(tmp, "export.xml");
writeFileSync(
xmlPath,
`<?xml version="1.0" encoding="UTF-8"?>
<HealthData locale="en_AU">
<Workout workoutActivityType="HKWorkoutActivityTypeRunning"
duration="42.0"
durationUnit="min"
totalDistance="6.5"
totalDistanceUnit="km"
totalEnergyBurned="1724"
totalEnergyBurnedUnit="kJ"
startDate="2026-05-14 18:00:00 +0200"
endDate="2026-05-14 18:42:00 +0200"
sourceName="Apple Watch"/>
</HealthData>`,
);
const prisma = makeFakePrisma();
await streamParseExportXml({
xmlPath,
userId: "user-energy",
userTimezone: "Europe/Berlin",
prisma: prisma as unknown as PrismaClient,
});
expect(prisma._workouts).toHaveLength(1);
expect(prisma._workouts[0].totalDistanceM).toBeCloseTo(6500, 6);
expect(prisma._workouts[0].totalEnergyKcal).toBeCloseTo(412.0458891, 6);
});
it("stores a mile-scale walking distance in metres", async () => {
const { prisma } = await importCumulativeFixture(
cumulativeExportXml(
"HKQuantityTypeIdentifierDistanceWalkingRunning",
"mi",
[{ value: 1.543, sourceName: "iPhone", hour: 8 }],
),
);
expect(prisma._measurements).toHaveLength(1);
expect(prisma._measurements[0].value).toBeCloseTo(2483.217792, 6);
});
});
+117 -4
View File
@@ -28,6 +28,7 @@ import type {
SleepStage,
RhythmClassification,
} from "@/generated/prisma/client";
import { convertHkValue } from "./hk-units";
/**
* How a batch of HealthKit samples for the same identifier should be
@@ -47,6 +48,22 @@ export interface AppleHealthMapping {
measurementType: MeasurementType;
/** Apple's HK unit string (e.g. `kg`, `count/min`, `mL/(kg*min)`). */
hkUnit: string;
/**
* issue #944 — why this entry's `hkUnit` can be trusted as fixed.
*
* Apple's `export.xml` writes the ACCOUNT's display unit on every
* quantity `<Record>`, not the unit this table assumes, so
* `mapAppleHealthEntry()` converts the reading from the record's own
* unit into `hkUnit` before `convertToDbUnit` runs — on the archive
* path, which is the caller that opts in. That only works for a unit
* `convertHkValue()` knows a factor for. An entry whose `hkUnit`
* is NOT convertible (a dimensionless count, a pinned event, a
* logarithmic dB level, a compound rate) must say here why no record
* unit can change its reading — the structural guard
* `src/__tests__/apple-health-unit-audit-guard.test.ts` fails on an
* entry that offers neither.
*/
unitFixedReason?: string;
/** HealthLog canonical DB unit (must match `getUnitForType()`). */
dbUnit: string;
/** Convert a HK sample value into the canonical DB unit. Identity for most. */
@@ -189,6 +206,28 @@ export const APPLE_HEALTH_SLEEP_STAGE_MAP: Record<number, SleepStage> = {
* metric: check Apple's HK unit; percent → ×100 path, m/m·s/kg/etc
* → identity path.
*/
/**
* issue #944 — the written reasons a mapping entry may cite for keeping a
* fixed `hkUnit`. Each one answers the same question: what could Apple
* possibly write in this record's `unit` attribute that would change the
* reading? Shared constants rather than free text so the audit reads as
* one decision per class instead of forty near-duplicates.
*/
const UNIT_FIXED_DIMENSIONLESS_COUNT =
"Dimensionless tally: Apple emits `count` in every locale and HKUnit has no sibling spelling for it, so no record unit can rescale the reading.";
const UNIT_FIXED_PER_MINUTE =
"Per-minute rate: Apple emits `count/min` in every locale and the Health app offers no alternative display unit, so there is nothing to convert from.";
const UNIT_FIXED_PERCENT_FRACTION =
"Apple writes the percent unit but the VALUE rides as a 0..1 fraction; the x100 in `convertToDbUnit` already covers both spellings, so branching on the record unit would double-scale it.";
const UNIT_FIXED_EVENT_PIN =
"Categorical event: `convertToDbUnit` pins the stored value to 1 fired event regardless of the reading, so no unit attribute can move it.";
const UNIT_FIXED_AUDIO_SPL =
"Sound-pressure level: Apple emits `dBASPL` only, and a decibel is logarithmic — a sibling unit would need its own formula, never a factor.";
const UNIT_FIXED_COMPOSITE_VO2 =
"Compound rate Apple emits verbatim as `mL/(kg*min)`; the Health app exposes no other spelling and the compound has no sibling unit to convert from.";
const UNIT_FIXED_SLEEP_CATEGORY =
"Sleep samples carry a symbolic category value, not a quantity: the importer derives the duration in minutes from the sample's own dates, so the record's unit attribute (empty in Apple's archive) says nothing about the reading.";
export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
// ── Body composition ────────────────────────────────────────
HKQuantityTypeIdentifierBodyMass: {
@@ -203,6 +242,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierBodyFatPercentage",
measurementType: "BODY_FAT",
hkUnit: "%",
unitFixedReason: UNIT_FIXED_PERCENT_FRACTION,
dbUnit: "%",
// Apple ships 0..1 fraction; HealthLog stores 0..100.
convertToDbUnit: (v) => v * 100,
@@ -254,6 +294,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierHeartRate",
measurementType: "PULSE",
hkUnit: "count/min",
unitFixedReason: UNIT_FIXED_PER_MINUTE,
dbUnit: "bpm",
convertToDbUnit: (v) => v,
aggregation: "latest",
@@ -262,6 +303,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierRestingHeartRate",
measurementType: "RESTING_HEART_RATE",
hkUnit: "count/min",
unitFixedReason: UNIT_FIXED_PER_MINUTE,
dbUnit: "bpm",
convertToDbUnit: (v) => v,
aggregation: "latest",
@@ -284,6 +326,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
// duplication; see `.planning/phase-W1-v1423-research.md`).
measurementType: "ACTIVITY_STEPS",
hkUnit: "count",
unitFixedReason: UNIT_FIXED_DIMENSIONLESS_COUNT,
dbUnit: "steps",
convertToDbUnit: (v) => v,
aggregation: "sum",
@@ -300,6 +343,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierFlightsClimbed",
measurementType: "FLIGHTS_CLIMBED",
hkUnit: "count",
unitFixedReason: UNIT_FIXED_DIMENSIONLESS_COUNT,
dbUnit: "flights",
convertToDbUnit: (v) => v,
aggregation: "sum",
@@ -318,6 +362,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierVO2Max",
measurementType: "VO2_MAX",
hkUnit: "mL/(kg*min)",
unitFixedReason: UNIT_FIXED_COMPOSITE_VO2,
dbUnit: "mL/(kg·min)",
convertToDbUnit: (v) => v,
aggregation: "latest",
@@ -337,6 +382,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierOxygenSaturation",
measurementType: "OXYGEN_SATURATION",
hkUnit: "fraction",
unitFixedReason: UNIT_FIXED_PERCENT_FRACTION,
dbUnit: "%",
// Apple ships 0..1 fraction; HealthLog stores 0..100.
convertToDbUnit: (v) => v * 100,
@@ -348,6 +394,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKCategoryTypeIdentifierSleepAnalysis",
measurementType: "SLEEP_DURATION",
hkUnit: "category",
unitFixedReason: UNIT_FIXED_SLEEP_CATEGORY,
dbUnit: "minutes",
// value here is duration-in-minutes (caller computes
// `endDate - startDate` and converts to minutes); the per-stage
@@ -367,6 +414,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierEnvironmentalAudioExposure",
measurementType: "AUDIO_EXPOSURE_ENV",
hkUnit: "dBASPL",
unitFixedReason: UNIT_FIXED_AUDIO_SPL,
dbUnit: "dBA",
convertToDbUnit: (v) => v,
aggregation: "mean",
@@ -379,6 +427,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierHeadphoneAudioExposure",
measurementType: "AUDIO_EXPOSURE_HEADPHONE",
hkUnit: "dBASPL",
unitFixedReason: UNIT_FIXED_AUDIO_SPL,
dbUnit: "dBA",
convertToDbUnit: (v) => v,
aggregation: "mean",
@@ -403,6 +452,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierAppleWalkingSteadiness",
measurementType: "WALKING_STEADINESS",
hkUnit: "%",
unitFixedReason: UNIT_FIXED_PERCENT_FRACTION,
dbUnit: "%",
// Apple ships 0..1 fraction; HealthLog stores 0..100.
convertToDbUnit: (v) => v * 100,
@@ -416,6 +466,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKCategoryTypeIdentifierEnvironmentalAudioExposureEvent",
measurementType: "AUDIO_EXPOSURE_EVENT",
hkUnit: "count",
unitFixedReason: UNIT_FIXED_EVENT_PIN,
dbUnit: "count",
convertToDbUnit: () => 1,
aggregation: "sum",
@@ -427,6 +478,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKCategoryTypeIdentifierHeadphoneAudioExposureEvent",
measurementType: "AUDIO_EXPOSURE_EVENT",
hkUnit: "count",
unitFixedReason: UNIT_FIXED_EVENT_PIN,
dbUnit: "count",
convertToDbUnit: () => 1,
aggregation: "sum",
@@ -449,6 +501,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKCategoryTypeIdentifierIrregularHeartRhythmEvent",
measurementType: "IRREGULAR_RHYTHM_NOTIFICATION",
hkUnit: "event",
unitFixedReason: UNIT_FIXED_EVENT_PIN,
dbUnit: "event",
convertToDbUnit: () => 1,
aggregation: "sum",
@@ -462,6 +515,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKCategoryTypeIdentifierHighHeartRateEvent",
measurementType: "HIGH_HEART_RATE_EVENT",
hkUnit: "event",
unitFixedReason: UNIT_FIXED_EVENT_PIN,
dbUnit: "event",
convertToDbUnit: () => 1,
aggregation: "sum",
@@ -474,6 +528,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKCategoryTypeIdentifierLowHeartRateEvent",
measurementType: "LOW_HEART_RATE_EVENT",
hkUnit: "event",
unitFixedReason: UNIT_FIXED_EVENT_PIN,
dbUnit: "event",
convertToDbUnit: () => 1,
aggregation: "sum",
@@ -487,6 +542,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKCategoryTypeIdentifierAppleWalkingSteadinessEvent",
measurementType: "WALKING_STEADINESS_EVENT",
hkUnit: "event",
unitFixedReason: UNIT_FIXED_EVENT_PIN,
dbUnit: "event",
convertToDbUnit: () => 1,
aggregation: "sum",
@@ -501,6 +557,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKCategoryTypeIdentifierSleepApneaEvent",
measurementType: "BREATHING_DISTURBANCE_EVENT",
hkUnit: "event",
unitFixedReason: UNIT_FIXED_EVENT_PIN,
dbUnit: "event",
convertToDbUnit: () => 1,
aggregation: "sum",
@@ -524,6 +581,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierRespiratoryRate",
measurementType: "RESPIRATORY_RATE",
hkUnit: "count/min",
unitFixedReason: UNIT_FIXED_PER_MINUTE,
dbUnit: "breaths/min",
convertToDbUnit: (v) => v,
aggregation: "mean",
@@ -537,6 +595,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierBodyMassIndex",
measurementType: "BODY_MASS_INDEX",
hkUnit: "count",
unitFixedReason: UNIT_FIXED_DIMENSIONLESS_COUNT,
dbUnit: "kg/m²",
convertToDbUnit: (v) => v,
aggregation: "latest",
@@ -557,6 +616,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierWalkingHeartRateAverage",
measurementType: "WALKING_HEART_RATE_AVERAGE",
hkUnit: "count/min",
unitFixedReason: UNIT_FIXED_PER_MINUTE,
dbUnit: "bpm",
convertToDbUnit: (v) => v,
aggregation: "mean",
@@ -570,6 +630,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierWalkingAsymmetryPercentage",
measurementType: "WALKING_ASYMMETRY",
hkUnit: "%",
unitFixedReason: UNIT_FIXED_PERCENT_FRACTION,
dbUnit: "%",
convertToDbUnit: (v) => v * 100,
aggregation: "latest",
@@ -580,6 +641,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierWalkingDoubleSupportPercentage",
measurementType: "WALKING_DOUBLE_SUPPORT",
hkUnit: "%",
unitFixedReason: UNIT_FIXED_PERCENT_FRACTION,
dbUnit: "%",
convertToDbUnit: (v) => v * 100,
aggregation: "latest",
@@ -622,6 +684,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierHeartRateRecoveryOneMinute",
measurementType: "CARDIO_RECOVERY",
hkUnit: "count/min",
unitFixedReason: UNIT_FIXED_PER_MINUTE,
dbUnit: "bpm",
convertToDbUnit: (v) => v,
aggregation: "latest",
@@ -645,6 +708,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierNumberOfTimesFallen",
measurementType: "FALL_COUNT",
hkUnit: "count",
unitFixedReason: UNIT_FIXED_DIMENSIONLESS_COUNT,
dbUnit: "count",
convertToDbUnit: (v) => v,
aggregation: "sum",
@@ -685,6 +749,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
hkIdentifier: "HKQuantityTypeIdentifierAppleSleepingBreathingDisturbances",
measurementType: "BREATHING_DISTURBANCES",
hkUnit: "count",
unitFixedReason: UNIT_FIXED_DIMENSIONLESS_COUNT,
dbUnit: "count",
convertToDbUnit: (v) => v,
aggregation: "latest",
@@ -1140,7 +1205,14 @@ export interface AppleHealthEntryInput {
hkIdentifier: string;
/** Numeric value as Apple delivers it (pre-conversion). */
value: number;
/** Apple's unit string, captured for audit; not currently validated. */
/**
* Apple's unit string for this reading, captured for audit.
*
* Read ONLY when the caller opts in with `convertRecordUnit` — see
* `AppleHealthEntryOptions`. Every other caller gets the historical
* contract: the field is recorded and the reading is taken to already
* be in the table's `hkUnit`.
*/
unit: string;
/** ISO timestamp string (e.g. HealthKit `startDate`). */
startDate: string;
@@ -1175,6 +1247,33 @@ export interface AppleHealthEntryInput {
valueMax?: number;
}
/**
* Per-call options for `mapAppleHealthEntry()`.
*/
export interface AppleHealthEntryOptions {
/**
* Convert the reading out of `input.unit` and into the table's `hkUnit`
* before `convertToDbUnit` runs. Default FALSE.
*
* issue #944 — only the `export.xml` archive path passes `true`, because
* only there is the unit attribute known to be authoritative: Apple
* stamps every quantity `<Record>` with the ACCOUNT's display unit, so a
* metric archive reports walking distance in `km` and the number is
* meaningless without it.
*
* `POST /api/measurements/batch` deliberately does NOT opt in. Its `unit`
* is documented as captured-for-audit, not validated, and the native
* client's unit strings for the seventeen convertible identifiers are not
* pinned by anything in this repository — no fixture, no contract test, no
* coordination note. A client that reads a sample in metres and stamps the
* person's DISPLAY unit (`km`) into the field would have every walking
* distance multiplied by a thousand on ingest. Reading the field there is
* a wire change, and a wire change needs the client's strings pinned
* first.
*/
convertRecordUnit?: boolean;
}
/** Output of `mapAppleHealthEntry()`. */
export interface AppleHealthEntryOutput {
type: MeasurementType;
@@ -1214,6 +1313,7 @@ export interface AppleHealthEntryOutput {
*/
export function mapAppleHealthEntry(
input: AppleHealthEntryInput,
options: AppleHealthEntryOptions = {},
): AppleHealthEntryOutput | null {
const mapping = APPLE_HEALTH_TYPE_MAP[input.hkIdentifier];
if (!mapping) return null;
@@ -1221,7 +1321,20 @@ export function mapAppleHealthEntry(
const takenAt = new Date(input.endDate);
if (Number.isNaN(takenAt.getTime())) return null;
const value = mapping.convertToDbUnit(input.value);
// issue #944 — `convertToDbUnit` is written against the table's fixed
// `hkUnit`, but Apple's `export.xml` stamps each `<Record>` with the
// ACCOUNT's own display unit: a metric archive reports walking distance
// in `km`, an imperial one in `mi`, body mass in `lb`, energy in `kJ`.
// The archive path opts in and the reading is normalised into `hkUnit`
// first; a unit we cannot place leaves the value untouched (see
// `convertHkValue`). Every other caller keeps the identity behaviour —
// see `AppleHealthEntryOptions.convertRecordUnit` for why.
const toHkUnit = (raw: number): number =>
options.convertRecordUnit
? (convertHkValue(raw, input.unit, mapping.hkUnit) ?? raw)
: raw;
const value = mapping.convertToDbUnit(toHkUnit(input.value));
const out: AppleHealthEntryOutput = {
type: mapping.measurementType,
@@ -1235,10 +1348,10 @@ export function mapAppleHealthEntry(
// hourly HR bucket). A finite check guards a malformed payload; the
// caller only persists these on the HR-bucket row.
if (typeof input.valueMin === "number" && Number.isFinite(input.valueMin)) {
out.valueMin = mapping.convertToDbUnit(input.valueMin);
out.valueMin = mapping.convertToDbUnit(toHkUnit(input.valueMin));
}
if (typeof input.valueMax === "number" && Number.isFinite(input.valueMax)) {
out.valueMax = mapping.convertToDbUnit(input.valueMax);
out.valueMax = mapping.convertToDbUnit(toHkUnit(input.valueMax));
}
if (mapping.sleepStageMap) {
+185
View File
@@ -0,0 +1,185 @@
/**
* HealthKit unit normalisation (issue #944).
*
* Apple writes a `unit` attribute on every quantity `<Record>` of an
* `export.xml` archive, and the string it writes is the account's OWN
* display unit rather than the unit HealthLog's mapping table assumes:
* a metric account exports walking distance as `km`, an imperial one as
* `mi`, and the same split shows up on body mass (`kg` / `lb` / `st`),
* energy (`kcal` / `kJ`) and temperature (`degC` / `degF`).
*
* Before this module the export import fed the raw number straight into
* `AppleHealthMapping.convertToDbUnit`, which is written against the
* table's fixed `hkUnit` — so a 2.484 km day was stored as 2.484 m. The
* workout path had a hand-rolled km/mi→m branch of its own; both paths
* now share `convertHkValue()`.
*
* Contract, deliberately conservative:
*
* - Same unit in and out → the value is returned untouched, so an
* archive (or an iOS batch) that already speaks the table's unit is
* byte-for-byte unchanged.
* - Unknown unit, empty unit, or two units from different families →
* `null`, and every caller falls back to the raw value. A unit we
* cannot prove a factor for must never silently rescale a reading.
* - Only units Apple's `HKUnit` can actually emit are listed. A unit
* string is matched exactly (after trimming and after stripping the
* molar-mass annotation Apple writes on glucose), never lower-cased:
* `cal` (small calorie) and `Cal` (kilocalorie) are different units
* that differ only in case.
*/
interface HkUnitSpec {
/** Units convert only within one family. */
family: string;
/** This unit → the family's base unit. */
toBase: (value: number) => number;
/** The family's base unit → this unit. */
fromBase: (value: number) => number;
}
/** A unit that is a plain multiple of its family's base unit. */
function scaled(family: string, perBase: number): HkUnitSpec {
return {
family,
toBase: (value) => value * perBase,
fromBase: (value) => value / perBase,
};
}
/**
* Every unit string Apple emits that HealthLog can convert, keyed by the
* exact `HKUnit.unitString` spelling. Base units: metre, kilogram,
* kilocalorie, second, degree Celsius, mmHg, mg/dL, metres per second.
*/
const HK_UNITS: Record<string, HkUnitSpec> = {
// ── Length (base: metre) ───────────────────────────────────
m: scaled("length", 1),
km: scaled("length", 1000),
cm: scaled("length", 0.01),
mm: scaled("length", 0.001),
in: scaled("length", 0.0254),
ft: scaled("length", 0.3048),
yd: scaled("length", 0.9144),
mi: scaled("length", 1609.344),
// ── Mass (base: kilogram) ──────────────────────────────────
kg: scaled("mass", 1),
g: scaled("mass", 0.001),
lb: scaled("mass", 0.45359237),
oz: scaled("mass", 0.028349523125),
st: scaled("mass", 6.35029318),
// ── Energy (base: kilocalorie) ─────────────────────────────
kcal: scaled("energy", 1),
Cal: scaled("energy", 1), // Apple's large calorie == kilocalorie
cal: scaled("energy", 0.001), // small calorie — case is load-bearing
kJ: scaled("energy", 1 / 4.184),
J: scaled("energy", 1 / 4184),
// ── Time (base: second) ────────────────────────────────────
s: scaled("time", 1),
ms: scaled("time", 0.001),
min: scaled("time", 60),
hr: scaled("time", 3600),
d: scaled("time", 86_400),
// ── Pressure (base: mmHg) ──────────────────────────────────
mmHg: scaled("pressure", 1),
kPa: scaled("pressure", 7.500_616_827_041_698),
Pa: scaled("pressure", 0.007_500_616_827_041_698),
// ── Blood glucose (base: mg/dL) ────────────────────────────
// Apple annotates the molar unit with glucose's molar mass, which the
// normaliser strips before the lookup.
"mg/dL": scaled("glucose", 1),
"mmol/L": scaled("glucose", 18.0156), // 180.156 g/mol ÷ 10
// ── Speed (base: metres per second) ────────────────────────
"m/s": scaled("speed", 1),
"km/hr": scaled("speed", 1000 / 3600),
"mi/hr": scaled("speed", 1609.344 / 3600),
// ── Temperature (base: degree Celsius) — affine, not scaled ─
degC: {
family: "temperature",
toBase: (value) => value,
fromBase: (value) => value,
},
degF: {
family: "temperature",
toBase: (value) => ((value - 32) * 5) / 9,
fromBase: (value) => (value * 9) / 5 + 32,
},
K: {
family: "temperature",
toBase: (value) => value - 273.15,
fromBase: (value) => value + 273.15,
},
};
/**
* Apple writes molar units with the substance's molar mass inline —
* `mmol<180.156>/L` for blood glucose. The annotation is informational;
* strip it so the lookup sees the plain unit.
*/
const MOLAR_MASS_ANNOTATION = /<[^>]*>/g;
function normaliseUnitString(unit: string): string {
return unit.trim().replace(MOLAR_MASS_ANNOTATION, "");
}
function lookup(unit: string | null | undefined): HkUnitSpec | null {
if (typeof unit !== "string") return null;
const normalised = normaliseUnitString(unit);
if (!normalised) return null;
return HK_UNITS[normalised] ?? null;
}
/**
* `true` when this module knows a conversion factor for the unit — i.e.
* a `<Record>` that carries a sibling unit from the same family will be
* converted rather than trusted blindly. The mapping-table guard in
* `src/__tests__/apple-health-unit-audit-guard.test.ts` uses this to
* decide whether an entry's fixed `hkUnit` needs a written reason.
*/
export function isConvertibleHkUnit(unit: string | null | undefined): boolean {
return lookup(unit) !== null;
}
/**
* Convert a HealthKit value between two `HKUnit` strings. Returns `null`
* — never a guess — when either unit is unknown or the two belong to
* different families; callers keep the raw value in that case.
*/
export function convertHkValue(
value: number,
fromUnit: string | null | undefined,
toUnit: string,
): number | null {
if (!Number.isFinite(value)) return null;
const from = lookup(fromUnit);
if (!from) return null;
const to = lookup(toUnit);
if (!to) return null;
if (from.family !== to.family) return null;
// Same unit: return the reading untouched rather than round-tripping it
// through the base and picking up a float error.
if (normaliseUnitString(String(fromUnit)) === normaliseUnitString(toUnit)) {
return value;
}
return to.fromBase(from.toBase(value));
}
/**
* Convert a distance a HealthKit archive reports in its own unit into
* metres, the canonical DB unit. Falls back to the raw number for a unit
* this module cannot place (including the empty attribute an archive may
* write) — the pre-#944 behaviour of the workout path, preserved.
*/
export function hkDistanceToMetres(
value: number,
unit: string | null | undefined,
): number {
return convertHkValue(value, unit, "m") ?? value;
}
@@ -42,6 +42,10 @@ import {
} from "@/lib/measurements/drain-per-sample-cumulative";
import { reconcileExternalMeasurement } from "@/lib/measurements/reconcile-external-measurement";
import { resolveHkWorkoutSportType } from "@/lib/measurements/hk-workout-activity-type-map";
import {
convertHkValue,
hkDistanceToMetres,
} from "@/lib/measurements/hk-units";
import { emitInsertedMeasurementArrivals } from "@/lib/arrivals/measurement-emit";
import { maybeEnqueueMorningRefresh } from "@/lib/daily/morning-refresh-trigger";
import { emitDataArrival } from "@/lib/arrivals/emit-shared";
@@ -851,14 +855,21 @@ export async function streamParseExportXml(
return;
}
const mapped = mapAppleHealthEntry({
hkIdentifier: hkType,
value: parsedValue.value,
unit: attrs.unit ?? mapping.hkUnit,
startDate: attrs.startDate,
endDate: attrs.endDate,
sleepStage: parsedValue.sleepStage,
});
const mapped = mapAppleHealthEntry(
{
hkIdentifier: hkType,
value: parsedValue.value,
unit: attrs.unit ?? mapping.hkUnit,
startDate: attrs.startDate,
endDate: attrs.endDate,
sleepStage: parsedValue.sleepStage,
},
// issue #944 — the archive is the one caller whose unit attribute
// is authoritative: Apple writes the account's own display unit on
// every quantity `<Record>`, so `km` here means kilometres and the
// reading has to be converted before it is stored.
{ convertRecordUnit: true },
);
if (!mapped) {
unknown[`${hkType}::map_failed`] =
(unknown[`${hkType}::map_failed`] ?? 0) + 1;
@@ -947,14 +958,13 @@ export async function streamParseExportXml(
const totalDistance = Number.parseFloat(attrs.totalDistance ?? "");
const totalEnergy = Number.parseFloat(attrs.totalEnergyBurned ?? "");
const distanceUnit = attrs.totalDistanceUnit ?? "";
// Apple ships km for HKWorkout.totalDistance by default; metres
// is the canonical DB unit. Convert when the unit is km.
const energyUnit = attrs.totalEnergyBurnedUnit ?? "";
// Apple ships the ACCOUNT's own length unit on HKWorkout.totalDistance
// (km for a metric account, mi for an imperial one); metres is the
// canonical DB unit. issue #944 — the same conversion the `<Record>`
// path now runs, so the two can never drift apart again.
const distanceM = Number.isFinite(totalDistance)
? distanceUnit === "km"
? totalDistance * 1000
: distanceUnit === "mi"
? totalDistance * 1609.344
: totalDistance
? hkDistanceToMetres(totalDistance, distanceUnit)
: null;
const externalId = hashSampleKey(
@@ -970,7 +980,13 @@ export async function streamParseExportXml(
startedAt: startDate,
endedAt: endDate,
durationSec,
totalEnergyKcal: Number.isFinite(totalEnergy) ? totalEnergy : null,
totalEnergyKcal: Number.isFinite(totalEnergy)
? // issue #944 — the energy attribute carries the account's own
// unit too: a Health app set to kilojoules writes `kJ` here, and
// the column is kilocalories. Same shared conversion; an unknown
// unit leaves the number alone.
(convertHkValue(totalEnergy, energyUnit, "kcal") ?? totalEnergy)
: null,
totalDistanceM: distanceM,
externalId,
externalSourceVersion: attrs.sourceVersion ?? null,
@@ -0,0 +1,394 @@
/**
* Repair for walking distances the `export.zip` import stored 1000x too
* small (issue #944).
*
* THE DEFECT (fixed in code from this release): `mapAppleHealthEntry()`
* accepted the `<Record unit="…">` attribute and never read it, so a
* `HKQuantityTypeIdentifierDistanceWalkingRunning` record — which Apple
* writes in the account's own display unit, `km` on a metric archive —
* was folded into the day's total as if the number were already metres.
* Every day of an imported history reads as a couple of metres walked.
*
* THE CODE FIX converts the record's own unit into the table's `hkUnit`
* before storage, so every FUTURE import is right. This module heals what
* is already in the database.
*
* ── The criterion ──────────────────────────────────────────────────
*
* A row is a candidate only when it carries the stamp the export importer
* itself writes, never because its value "looks small":
*
* type = WALKING_RUNNING_DISTANCE
* source = APPLE_HEALTH
* aggregationProvenance = EXPORT_XML_SOURCE_MAX
* externalId LIKE stats:HKQuantityTypeIdentifierDistanceWalkingRunning:%
* deletedAt IS NULL
*
* `EXPORT_XML_SOURCE_MAX` is written in exactly one place — the archive
* fold in `import-apple-health-export.ts` — and the `stats:<HK>:<day>`
* externalId is minted by `dailyStatsExternalId()` for that same fold. A
* row a native iOS sync has since overwritten carries
* `HEALTHKIT_STATISTICS` and its value came from HealthKit's own
* statistics query, so it is correct and out of scope. A row whose
* provenance is `LEGACY_UNKNOWN` or NULL cannot be PROVEN to have come
* from the archive path and is therefore left alone: re-importing the
* archive on the fixed build repairs those, and is the authoritative
* repair for any account that still has its `export.zip`.
*
* ── The refusal ────────────────────────────────────────────────────
*
* The criterion proves an archive ORIGIN. It cannot prove the archive was
* imported on the broken build — an account re-imported since carries the
* same stamp with the right numbers, and its rest days still pass the
* plausibility check after another x1000. So any candidate whose repaired
* value would leave the range refuses the whole account: nothing is
* written, the rows are reported, and the operator decides. Re-importing
* the archive is the authoritative repair and needs no script.
*
* ── Idempotency ────────────────────────────────────────────────────
*
* Applying the repair writes one `AuditLog` row per account inside the
* same transaction as the update. An account that already carries that
* row is skipped, so a second run finds nothing to do — the multiplier
* can never be applied twice.
*/
import type { PrismaClient } from "@/generated/prisma/client";
import { auditLog } from "@/lib/auth/audit";
import { afterMeasurementMutation } from "@/lib/rollups/after-measurement-mutation";
import { validateMeasurementRange } from "@/lib/validations/measurement";
import { dailyStatsExternalId } from "./apple-health-mapping";
import { convertHkValue } from "./hk-units";
/** The audit action that marks an account's distances as repaired. */
export const APPLE_HEALTH_DISTANCE_REPAIR_ACTION =
"measurement.apple_health_distance.repaired";
/** Minted by the same helper the export fold uses, so the two cannot drift. */
export const EXPORT_DISTANCE_EXTERNAL_ID_PREFIX = dailyStatsExternalId(
"HKQuantityTypeIdentifierDistanceWalkingRunning",
"",
);
/**
* The unit the affected archive was written in. Apple writes `km` on a
* metric account and `mi` on an imperial one, and the stored row keeps no
* record of which — the importer discarded the attribute, which is the
* defect. The operator names it; `km` is the default because it is what
* the Health app writes for every metric locale.
*/
export type ArchiveDistanceUnit = "km" | "mi";
export interface RepairCandidateRow {
id: string;
measuredAt: Date;
value: number;
}
export interface AccountRepairPlan {
userId: string;
/** Candidate rows that would be multiplied. */
repairable: RepairCandidateRow[];
/**
* Candidates whose repaired value would leave the plausibility range
* (`validateMeasurementRange`). One row here refuses the whole account:
* it is evidence the rows are NOT the 1000x class.
*/
outOfRange: RepairCandidateRow[];
/** Non-null when this account already carries the repair's audit row. */
alreadyRepairedAt: Date | null;
/**
* True when a previous run wrote the rows but its rollup tail did not
* finish — the audit row still says `rollupsRefreshed: false`, so the
* DAY/WEEK/MONTH/YEAR buckets and the cached status insights still hold
* the pre-repair numbers. Meaningless unless `alreadyRepairedAt` is set.
*/
rollupsPending: boolean;
/** Candidate rows found, whatever the disposition above. */
candidateCount: number;
}
/** The DB surface this module needs — the script hands it the real client. */
export type RepairClient = Pick<
PrismaClient,
"measurement" | "auditLog" | "$transaction"
>;
const REPAIRED_TYPE = "WALKING_RUNNING_DISTANCE" as const;
/** metres per unit of the archive's own length unit. */
export function repairFactorFor(unit: ArchiveDistanceUnit): number {
const factor = convertHkValue(1, unit, "m");
// The two archive units are length units by construction; the shared
// table is the single source of the factor either way.
if (factor === null) throw new Error(`no metre factor for unit "${unit}"`);
return factor;
}
/**
* List the repair candidates per account. Read-only — this is what the
* script's dry run prints.
*/
export async function planAppleHealthDistanceRepair(
client: RepairClient,
options: { archiveUnit: ArchiveDistanceUnit },
): Promise<AccountRepairPlan[]> {
const factor = repairFactorFor(options.archiveUnit);
const rows = await client.measurement.findMany({
where: {
type: REPAIRED_TYPE,
source: "APPLE_HEALTH",
aggregationProvenance: "EXPORT_XML_SOURCE_MAX",
externalId: { startsWith: EXPORT_DISTANCE_EXTERNAL_ID_PREFIX },
deletedAt: null,
},
select: { id: true, userId: true, value: true, measuredAt: true },
orderBy: [{ userId: "asc" }, { measuredAt: "asc" }],
});
if (rows.length === 0) return [];
const userIds = Array.from(new Set(rows.map((row) => row.userId)));
const repaired = await client.auditLog.findMany({
where: {
action: APPLE_HEALTH_DISTANCE_REPAIR_ACTION,
userId: { in: userIds },
},
select: { userId: true, createdAt: true, details: true },
orderBy: { createdAt: "asc" },
});
const repairedAt = new Map<string, Date>();
const rollupsPending = new Set<string>();
for (const entry of repaired) {
if (!entry.userId) continue;
repairedAt.set(entry.userId, entry.createdAt);
if (!rollupsRefreshedIn(entry.details)) rollupsPending.add(entry.userId);
}
const plans = new Map<string, AccountRepairPlan>();
for (const row of rows) {
let plan = plans.get(row.userId);
if (!plan) {
plan = {
userId: row.userId,
repairable: [],
outOfRange: [],
alreadyRepairedAt: repairedAt.get(row.userId) ?? null,
rollupsPending: rollupsPending.has(row.userId),
candidateCount: 0,
};
plans.set(row.userId, plan);
}
plan.candidateCount += 1;
if (plan.alreadyRepairedAt) continue;
const candidate: RepairCandidateRow = {
id: row.id,
measuredAt: row.measuredAt,
value: row.value,
};
if (validateMeasurementRange(REPAIRED_TYPE, row.value * factor) !== null) {
plan.outOfRange.push(candidate);
} else {
plan.repairable.push(candidate);
}
}
return Array.from(plans.values());
}
/**
* The command that heals the rollup tail when it did not run.
*
* Named here rather than in the script so the message an operator reads is
* the same one the module can be asked for.
*/
export function rollupRepairCommand(userId: string): string {
return `pnpm dlx tsx scripts/backfill-rollups.ts --user ${userId}`;
}
/** Did a repair audit row record a finished rollup tail? */
function rollupsRefreshedIn(details: string | null): boolean {
if (!details) return false;
try {
const parsed: unknown = JSON.parse(details);
return (
typeof parsed === "object" &&
parsed !== null &&
(parsed as { rollupsRefreshed?: unknown }).rollupsRefreshed === true
);
} catch {
return false;
}
}
export interface RepairOutcome {
userId: string;
updated: number;
skipped: number;
/**
* Why the account was left alone, when it was. Null on a run that wrote.
* The script prints it verbatim.
*/
refusedReason: string | null;
/** The out-of-range candidates — what a refusal was decided on. */
skippedRows: RepairCandidateRow[];
/**
* True when the rows were written but the rollup tail was not. The audit
* row still reads `rollupsRefreshed: false`, so a later run can tell this
* apart from a finished repair; the caller must surface
* `rollupRepairCommand(userId)` and exit non-zero.
*/
rollupsPending: boolean;
/** What the rollup tail threw, when it threw. */
rollupError: unknown;
}
/**
* The refusal an out-of-range candidate earns.
*
* The provenance criterion proves an archive origin; it cannot prove the
* archive was imported on the BROKEN build. An account that has since been
* re-imported on the fixed build carries the same stamp with the right
* numbers, and its rest days — under 200 m, phone left at home — pass the
* plausibility check after another x1000 and would be written. So a single
* row that would leave the range is treated as what it is: evidence the
* account is not the 1000x class. The whole account stops.
*/
function outOfRangeRefusal(plan: AccountRepairPlan, factor: number): string {
return (
`${plan.outOfRange.length} of ${plan.candidateCount} candidate row(s) ` +
`leave the plausible range after x${factor}, so these rows are not the ` +
"1000x class — the account was most likely re-imported on the fixed " +
"build, or its archive was never in the affected unit. Nothing written. " +
"Check the dry run's worked examples against the account's Health app " +
"before forcing anything."
);
}
/**
* Apply one account's plan: every repairable row multiplied by the
* archive unit's metre factor in ONE transaction, together with the audit
* row that makes a second run a no-op. Returns what it did.
*/
export async function applyAppleHealthDistanceRepair(
client: RepairClient,
plan: AccountRepairPlan,
options: { archiveUnit: ArchiveDistanceUnit },
): Promise<RepairOutcome> {
const factor = repairFactorFor(options.archiveUnit);
const skipped = plan.outOfRange.length;
const base = {
userId: plan.userId,
updated: 0,
skipped,
skippedRows: plan.outOfRange,
rollupsPending: false,
rollupError: null,
};
if (plan.alreadyRepairedAt) {
return {
...base,
refusedReason: `already repaired on ${plan.alreadyRepairedAt.toISOString()}`,
};
}
// Refuse the ACCOUNT, not just the row: see `outOfRangeRefusal`.
if (plan.outOfRange.length > 0) {
return { ...base, refusedReason: outOfRangeRefusal(plan, factor) };
}
if (plan.repairable.length === 0) {
return { ...base, refusedReason: "no repairable rows" };
}
const ids = plan.repairable.map((row) => row.id);
const details = {
type: REPAIRED_TYPE,
archiveUnit: options.archiveUnit,
factor,
rows: ids.length,
skippedOutOfRange: skipped,
issue: 944,
// Flipped to true only once the rollup tail below has finished. A row
// left at false says "rows repaired, rollups still pre-repair", which is
// what a re-run has to be able to see: the account is skipped from here
// on, so nothing else would ever notice.
rollupsRefreshed: false,
};
const auditId = await client.$transaction(async (tx): Promise<string> => {
// One statement per chunk rather than one per row: an account with a
// decade of history carries thousands of rows and an interactive
// transaction has a clock on it. `sync_version` + `updated_at` are
// bumped by hand because raw SQL bypasses Prisma's `@updatedAt`, and
// the iOS delta feed pages on exactly those two columns.
for (let i = 0; i < ids.length; i += 500) {
const chunk = ids.slice(i, i + 500);
await tx.$executeRaw`
UPDATE measurements
SET value = value * ${factor},
unit = 'm',
sync_version = sync_version + 1,
updated_at = NOW()
WHERE id = ANY(${chunk})`;
}
await auditLog(APPLE_HEALTH_DISTANCE_REPAIR_ACTION, {
client: tx,
userId: plan.userId,
actorUserId: null,
details,
});
// Read the row back rather than widening `auditLog`'s return type for
// ~600 call sites. Inside the same transaction, and an account can hold
// at most one of these rows — the second run is refused above — so the
// lookup is exact, not a guess at the newest.
const entry = await tx.auditLog.findFirstOrThrow({
where: {
action: APPLE_HEALTH_DISTANCE_REPAIR_ACTION,
userId: plan.userId,
},
select: { id: true },
});
return entry.id;
});
// The tail: the DAY/WEEK/MONTH/YEAR rollups and the cached status insights
// hold the pre-repair numbers until they are recomputed. It runs outside
// the transaction, so a throw here leaves the rows repaired and the derived
// tier stale — and the account is skipped from the next run on. Hence the
// flag: it is flipped only after the tail returned, and the caller is told
// to run the backfill and to exit non-zero.
let rollupsPending = false;
let rollupError: unknown = null;
try {
await afterMeasurementMutation(
plan.userId,
plan.repairable.map((row) => ({
type: REPAIRED_TYPE,
measuredAt: row.measuredAt,
})),
"repair-apple-health-distance",
);
await client.auditLog.update({
where: { id: auditId },
data: { details: JSON.stringify({ ...details, rollupsRefreshed: true }) },
});
} catch (err) {
// Either leg failing leaves the flag at false, which is the safe
// direction: an operator re-running the backfill needlessly costs time,
// one who never learns it is due keeps wrong numbers on every chart.
rollupsPending = true;
rollupError = err;
}
return {
userId: plan.userId,
updated: ids.length,
skipped,
refusedReason: null,
skippedRows: plan.outOfRange,
rollupsPending,
rollupError,
};
}
/** Narrowing helper for the script's `--unit` flag. */
export function parseArchiveUnit(raw: string): ArchiveDistanceUnit | null {
return raw === "km" || raw === "mi" ? raw : null;
}
+7 -1
View File
@@ -53,7 +53,13 @@ const batchEntrySchema = z
.describe(
"v1.19.2 (iOS #34 extension) — the bucket's MAXIMUM bpm for a 10-minute heart-rate bucket (see `externalId`). Persisted ONLY on a well-formed `stats:HKQuantityTypeIdentifierHeartRate:<bucket-start>` row; ignored (stored null) on every other entry. Omit on a pre-v1.19.2 client — the bucket keeps the avg-only contract.",
),
unit: z.string().min(1).max(60),
unit: z
.string()
.min(1)
.max(60)
.describe(
"Apple's `HKUnit` string for `value`, captured for audit. The server does NOT convert from it: `value` is read in the unit HealthLog stores the identifier in (`m` for walking distance, `kg` for body mass, `kcal` for energy, `degC` for temperature). Send the sample in that unit.",
),
startDate: z.iso.datetime({ offset: true }),
endDate: z.iso.datetime({ offset: true }),
sleepStage: z
@@ -0,0 +1,317 @@
/**
* Integration coverage for the export-import walking-distance repair
* (issue #944).
*
* The defect stored every archive-imported `WALKING_RUNNING_DISTANCE` row a
* thousand times too small. `src/lib/measurements/repair-apple-health-distance.ts`
* heals what is already in the database, and everything that matters about it
* is a database property: which rows the criterion actually selects, that the
* multiply lands in one transaction with its audit row, and that a second run
* finds nothing. None of that can be proven against a mock, so this runs
* against the testcontainer Postgres (Docker / OrbStack; `pnpm test:integration`).
*/
import { beforeEach, describe, expect, it } from "vitest";
import { getPrismaClient, truncateAllTables } from "./setup";
import {
APPLE_HEALTH_DISTANCE_REPAIR_ACTION,
applyAppleHealthDistanceRepair,
planAppleHealthDistanceRepair,
} from "@/lib/measurements/repair-apple-health-distance";
import { settleBackgroundTasks } from "@/lib/logging/background-tasks";
import type {
MeasurementAggregationProvenance,
MeasurementSource,
MeasurementType,
PrismaClient,
} from "@/generated/prisma/client";
const DISTANCE_PREFIX = "stats:HKQuantityTypeIdentifierDistanceWalkingRunning:";
let dayCounter = 0;
async function seedRow(
prisma: PrismaClient,
userId: string,
overrides: {
value: number;
type?: MeasurementType;
source?: MeasurementSource;
provenance?: MeasurementAggregationProvenance | null;
externalId?: string;
},
): Promise<string> {
dayCounter += 1;
const measuredAt = new Date(Date.UTC(2026, 0, dayCounter, 12, 0, 0));
const day = measuredAt.toISOString().slice(0, 10);
const row = await prisma.measurement.create({
data: {
userId,
type: overrides.type ?? "WALKING_RUNNING_DISTANCE",
value: overrides.value,
unit: "m",
source: overrides.source ?? "APPLE_HEALTH",
measuredAt,
externalId: overrides.externalId ?? `${DISTANCE_PREFIX}${day}`,
aggregationProvenance:
overrides.provenance === undefined
? "EXPORT_XML_SOURCE_MAX"
: overrides.provenance,
},
select: { id: true },
});
return row.id;
}
async function readRow(prisma: PrismaClient, id: string) {
return prisma.measurement.findUniqueOrThrow({
where: { id },
select: { value: true, unit: true, syncVersion: true },
});
}
async function makeUser(prisma: PrismaClient, tag: string): Promise<string> {
const user = await prisma.user.create({
data: {
username: `distance-repair-${tag}`,
email: `distance-repair-${tag}@example.test`,
role: "USER",
},
});
return user.id;
}
describe("Apple Health export distance repair — integration", () => {
beforeEach(async () => {
await truncateAllTables(getPrismaClient());
dayCounter = 0;
});
it("selects only rows the archive importer stamped, and repairs them once", async () => {
const prisma = getPrismaClient();
const userId = await makeUser(prisma, "km");
const kmDay = await seedRow(prisma, userId, { value: 2.484 });
const otherDay = await seedRow(prisma, userId, { value: 7.02 });
// Decoys, one per arm of the criterion.
const nativeStats = await seedRow(prisma, userId, {
value: 8123,
provenance: "HEALTHKIT_STATISTICS",
});
const legacyProvenance = await seedRow(prisma, userId, {
value: 3.4,
provenance: null,
});
const manualRow = await seedRow(prisma, userId, {
value: 4.1,
source: "MANUAL",
});
const otherType = await seedRow(prisma, userId, {
value: 9000,
type: "ACTIVITY_STEPS",
externalId: "stats:HKQuantityTypeIdentifierStepCount:2026-01-09",
});
const perSample = await seedRow(prisma, userId, {
value: 1.2,
externalId: "sample:abc123",
});
const plans = await planAppleHealthDistanceRepair(prisma, {
archiveUnit: "km",
});
expect(plans).toHaveLength(1);
expect(plans[0].userId).toBe(userId);
expect(plans[0].repairable.map((row) => row.id).sort()).toEqual(
[kmDay, otherDay].sort(),
);
expect(plans[0].alreadyRepairedAt).toBeNull();
const outcome = await applyAppleHealthDistanceRepair(prisma, plans[0], {
archiveUnit: "km",
});
await settleBackgroundTasks();
expect(outcome).toMatchObject({ userId, updated: 2, skipped: 0 });
expect((await readRow(prisma, kmDay)).value).toBeCloseTo(2484, 6);
expect((await readRow(prisma, otherDay)).value).toBeCloseTo(7020, 6);
// The delta feed pages on sync_version / updated_at, so a repaired row
// has to look mutated to a paired client.
expect((await readRow(prisma, kmDay)).syncVersion).toBe(2);
expect((await readRow(prisma, kmDay)).unit).toBe("m");
// Every decoy untouched.
expect((await readRow(prisma, nativeStats)).value).toBe(8123);
expect((await readRow(prisma, legacyProvenance)).value).toBe(3.4);
expect((await readRow(prisma, manualRow)).value).toBe(4.1);
expect((await readRow(prisma, otherType)).value).toBe(9000);
expect((await readRow(prisma, perSample)).value).toBe(1.2);
const audit = await prisma.auditLog.findMany({
where: { action: APPLE_HEALTH_DISTANCE_REPAIR_ACTION, userId },
});
expect(audit).toHaveLength(1);
expect(JSON.parse(audit[0].details ?? "{}")).toMatchObject({
archiveUnit: "km",
factor: 1000,
rows: 2,
issue: 944,
// Flipped only after the rollup tail returned. A run that repaired the
// rows and lost the tail leaves this false, and the account is skipped
// from here on — so it is the only place the gap can still be seen.
rollupsRefreshed: true,
});
expect(outcome.rollupsPending).toBe(false);
const second = await planAppleHealthDistanceRepair(prisma, {
archiveUnit: "km",
});
expect(second[0].rollupsPending).toBe(false);
});
it("is idempotent — a second run finds nothing to repair", async () => {
const prisma = getPrismaClient();
const userId = await makeUser(prisma, "twice");
const rowId = await seedRow(prisma, userId, { value: 2.484 });
const first = await planAppleHealthDistanceRepair(prisma, {
archiveUnit: "km",
});
await applyAppleHealthDistanceRepair(prisma, first[0], {
archiveUnit: "km",
});
await settleBackgroundTasks();
const second = await planAppleHealthDistanceRepair(prisma, {
archiveUnit: "km",
});
expect(second).toHaveLength(1);
expect(second[0].repairable).toHaveLength(0);
expect(second[0].candidateCount).toBe(1);
expect(second[0].alreadyRepairedAt).toBeInstanceOf(Date);
const outcome = await applyAppleHealthDistanceRepair(prisma, second[0], {
archiveUnit: "km",
});
expect(outcome.updated).toBe(0);
expect((await readRow(prisma, rowId)).value).toBeCloseTo(2484, 6);
expect((await readRow(prisma, rowId)).syncVersion).toBe(2);
});
it("uses the archive's own unit", async () => {
const prisma = getPrismaClient();
const userId = await makeUser(prisma, "mi");
const mile = await seedRow(prisma, userId, { value: 1.543 });
const plans = await planAppleHealthDistanceRepair(prisma, {
archiveUnit: "mi",
});
expect(plans[0].repairable.map((row) => row.id)).toEqual([mile]);
const outcome = await applyAppleHealthDistanceRepair(prisma, plans[0], {
archiveUnit: "mi",
});
await settleBackgroundTasks();
expect(outcome).toMatchObject({ updated: 1, skipped: 0 });
expect((await readRow(prisma, mile)).value).toBeCloseTo(2483.217792, 6);
});
it("refuses the whole account when one row would leave the range", async () => {
const prisma = getPrismaClient();
const userId = await makeUser(prisma, "mixed");
const plausible = await seedRow(prisma, userId, { value: 1.543 });
const absurd = await seedRow(prisma, userId, { value: 250 });
const plans = await planAppleHealthDistanceRepair(prisma, {
archiveUnit: "mi",
});
expect(plans[0].repairable.map((row) => row.id)).toEqual([plausible]);
expect(plans[0].outOfRange.map((row) => row.id)).toEqual([absurd]);
const outcome = await applyAppleHealthDistanceRepair(prisma, plans[0], {
archiveUnit: "mi",
});
await settleBackgroundTasks();
expect(outcome.updated).toBe(0);
expect(outcome.skipped).toBe(1);
expect(outcome.skippedRows.map((row) => row.id)).toEqual([absurd]);
expect(outcome.refusedReason).toMatch(/leave the plausible range/);
// Nothing written at all — not the "repairable" row, not the audit row
// that would make a later, correct run a no-op.
expect((await readRow(prisma, plausible)).value).toBe(1.543);
expect((await readRow(prisma, plausible)).syncVersion).toBe(1);
expect((await readRow(prisma, absurd)).value).toBe(250);
expect(
await prisma.auditLog.count({
where: { action: APPLE_HEALTH_DISTANCE_REPAIR_ACTION, userId },
}),
).toBe(0);
});
it("leaves an account re-imported on the fixed build untouched", async () => {
// The scenario the criterion cannot see: the operator re-imported
// `export.zip` on the fixed build, so the rows are already right. They
// still carry `EXPORT_XML_SOURCE_MAX` + the `stats:` external id and
// still have no audit row, so they are selected exactly like a broken
// account's rows. The rest day survives another x1000; the ordinary days
// do not, and that is what has to stop the account.
const prisma = getPrismaClient();
const userId = await makeUser(prisma, "reimported");
const ordinary = await seedRow(prisma, userId, { value: 2484 });
const longWalk = await seedRow(prisma, userId, { value: 7020 });
const restDay = await seedRow(prisma, userId, { value: 150 });
const plans = await planAppleHealthDistanceRepair(prisma, {
archiveUnit: "km",
});
expect(plans).toHaveLength(1);
// The rest day would pass the plausibility check on its own.
expect(plans[0].repairable.map((row) => row.id)).toEqual([restDay]);
expect(plans[0].outOfRange.map((row) => row.id).sort()).toEqual(
[ordinary, longWalk].sort(),
);
const outcome = await applyAppleHealthDistanceRepair(prisma, plans[0], {
archiveUnit: "km",
});
await settleBackgroundTasks();
expect(outcome.updated).toBe(0);
expect(outcome.refusedReason).not.toBeNull();
for (const [id, value] of [
[ordinary, 2484],
[longWalk, 7020],
[restDay, 150],
] as const) {
const row = await readRow(prisma, id);
expect(row.value).toBe(value);
expect(row.syncVersion).toBe(1);
}
expect(
await prisma.auditLog.count({
where: { action: APPLE_HEALTH_DISTANCE_REPAIR_ACTION, userId },
}),
).toBe(0);
});
it("plans one entry per account", async () => {
const prisma = getPrismaClient();
const first = await makeUser(prisma, "a");
const second = await makeUser(prisma, "b");
await seedRow(prisma, first, { value: 1.5 });
await seedRow(prisma, second, { value: 2.5 });
await seedRow(prisma, second, { value: 3.5 });
const plans = await planAppleHealthDistanceRepair(prisma, {
archiveUnit: "km",
});
expect(
plans.map((plan) => [plan.userId, plan.repairable.length]).sort(),
).toEqual(
[
[first, 1],
[second, 2],
].sort(),
);
});
});