mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-10 13:22:14 +00:00
Merge the Apple Health record-unit fix and the distance repair (#944)
This commit is contained in:
commit
071b110e8b
14 files changed
+1821
-21
No files matched your search
@@ -21350,6 +21350,9 @@ components:
|
||||
type: string
|
||||
minLength: 1
|
||||
maxLength: 60
|
||||
description: "Apple's `HKUnit` string for `value`, captured for audit. The server does NOT convert from it: `value` is
|
||||
read in the unit HealthLog stores the identifier in (`m` for walking distance, `kg` for body mass, `kcal`
|
||||
for energy, `degC` for temperature). Send the sample in that unit."
|
||||
startDate:
|
||||
type: string
|
||||
format: date-time
|
||||
|
||||
@@ -0,0 +1,156 @@
|
||||
# Apple Health walking-distance repair
|
||||
|
||||
`scripts/repair-apple-health-distance.ts` heals walking distances that the
|
||||
`export.zip` import stored a thousand times too small.
|
||||
|
||||
## The defect
|
||||
|
||||
Apple stamps every quantity `<Record>` in an `export.xml` with the account's
|
||||
own display unit: a metric archive reports `HKQuantityTypeIdentifierDistance-
|
||||
WalkingRunning` in `km`, an imperial one in `mi`. Until this release the record
|
||||
mapping accepted that attribute and never read it, so the day's total was
|
||||
folded as if the number were already metres. A 2.484 km day landed as
|
||||
`2.484 m`, and an imported decade reads as a couple of metres walked. The
|
||||
workout path was never affected — it converted its own distance unit.
|
||||
|
||||
The code fix converts the record's unit before storage, so every import from
|
||||
this release on is correct. This script is for the rows already in the
|
||||
database.
|
||||
|
||||
## Prefer a re-import
|
||||
|
||||
If the account still has its `export.zip`, **re-import it**. The archive fold
|
||||
UPSERTs the same `stats:<identifier>:<day>` rows, so a re-import on the fixed
|
||||
build rewrites every affected day with the right number, needs no flags, and
|
||||
also repairs rows this script deliberately leaves alone (see the criterion
|
||||
below). The script exists for accounts that no longer have the archive.
|
||||
|
||||
**An account that has been re-imported must not then be repaired.** The
|
||||
re-import already fixed it. Its rows keep the same `EXPORT_XML_SOURCE_MAX`
|
||||
stamp and the same `stats:` external id, and they carry no audit row, so the
|
||||
criterion selects them exactly as it selects a broken account — the stamp
|
||||
proves where a row came from, not which build imported it. Multiplying them
|
||||
again turns a 2.484 km day into 2484 km. The script refuses such an account on
|
||||
its own (any day whose value would leave the plausible range after the
|
||||
multiply stops the whole account), but that guard fires on the big days: keep
|
||||
the two apart yourself before you type `--apply`.
|
||||
|
||||
## Which rows it touches
|
||||
|
||||
Only rows carrying the stamp the archive importer itself writes:
|
||||
|
||||
| Column | Value |
|
||||
| ------------------------ | ------------------------------------------------------------ |
|
||||
| `type` | `WALKING_RUNNING_DISTANCE` |
|
||||
| `source` | `APPLE_HEALTH` |
|
||||
| `aggregation_provenance` | `EXPORT_XML_SOURCE_MAX` |
|
||||
| `external_id` | `stats:HKQuantityTypeIdentifierDistanceWalkingRunning:<day>` |
|
||||
| `deleted_at` | `NULL` |
|
||||
|
||||
`EXPORT_XML_SOURCE_MAX` is written in exactly one place in the tree — the
|
||||
archive fold — and the `stats:` external id is minted for that same fold, so
|
||||
the pair is a proof of origin rather than a guess. Nothing is ever selected
|
||||
because a value "looks too small".
|
||||
|
||||
Deliberately out of scope:
|
||||
|
||||
- Rows a native iOS sync has since overwritten (`HEALTHKIT_STATISTICS`): their
|
||||
value came from HealthKit's own statistics query and is correct.
|
||||
- Rows whose provenance is `LEGACY_UNKNOWN` or NULL: an archive origin cannot
|
||||
be proven for them. Re-import the archive to repair those.
|
||||
|
||||
## Run a dry-run first
|
||||
|
||||
Always, and read it. The dry run is where the decision is made, not a
|
||||
formality in front of a decision already taken: it is the only place the
|
||||
script shows you real numbers from the account before it changes them. If the
|
||||
worked example does not match what that person's Health app shows for that day,
|
||||
the run is wrong and `--apply` will make it permanent.
|
||||
|
||||
The default mode reports and never writes:
|
||||
|
||||
```bash
|
||||
# from a source checkout, with DATABASE_URL pointing at the instance
|
||||
pnpm dlx tsx scripts/repair-apple-health-distance.ts
|
||||
```
|
||||
|
||||
It prints one line per account: how many rows would be repaired, the date span
|
||||
they cover, one worked example (`2.484 m -> 2484 m`), and any row whose
|
||||
repaired value would leave the plausibility range.
|
||||
|
||||
This script imports the application dependency graph, so it runs from a source
|
||||
checkout, not through the production image's `healthlog-tsx` launcher.
|
||||
|
||||
## Name the archive's unit
|
||||
|
||||
The stored row keeps no record of the unit its archive was written in — that
|
||||
is the defect — so the operator names it. `--unit=km` is the default and is
|
||||
what the Health app writes in every metric locale; pass `--unit=mi` for an
|
||||
imperial archive:
|
||||
|
||||
```bash
|
||||
pnpm dlx tsx scripts/repair-apple-health-distance.ts --unit=mi
|
||||
```
|
||||
|
||||
Check the dry-run's worked example against what the account's Health app shows
|
||||
for that day before applying. If the two units are mixed across accounts, run
|
||||
the script once per unit with `--apply` and confirm each account's example
|
||||
first.
|
||||
|
||||
## Apply
|
||||
|
||||
Take a backup first — `docs/ops/backup-restore.md`. The multiply is in place
|
||||
and has no reverse operation: once an account is written wrong, the database
|
||||
copy you took beforehand is the only way back.
|
||||
|
||||
```bash
|
||||
pnpm dlx tsx scripts/repair-apple-health-distance.ts --apply
|
||||
```
|
||||
|
||||
Per account, in one transaction: every selected row multiplied by the unit's
|
||||
metre factor, `sync_version` incremented and `updated_at` bumped so paired iOS
|
||||
clients pick the rows up on their next delta sync, and an audit row
|
||||
(`measurement.apple_health_distance.repaired`) recording the unit, the factor
|
||||
and the row count. The DAY/WEEK/MONTH/YEAR rollups and the cached status
|
||||
insights for the touched days are recomputed afterwards, outside that
|
||||
transaction.
|
||||
|
||||
If that recompute fails, the rows are right and every chart is not, and no
|
||||
later run of this script would notice — it skips a repaired account. So the
|
||||
audit row is written with `rollupsRefreshed: false` and only flipped to `true`
|
||||
once the recompute has returned. A run that could not finish it prints the
|
||||
account and the command to re-run, exits non-zero, and a later dry run repeats
|
||||
the warning until the backfill has been run:
|
||||
|
||||
```bash
|
||||
pnpm dlx tsx scripts/backfill-rollups.ts --user <userId>
|
||||
```
|
||||
|
||||
A row whose repaired value would leave the plausible range (0–200 000 m)
|
||||
refuses the **whole account**: nothing is written, the offending rows and the
|
||||
reason are printed, and the run moves on to the next account. One such row is
|
||||
evidence the account is not the 1000x class — a re-imported account looks
|
||||
exactly like this — and repairing the rest of it would multiply days that were
|
||||
already right.
|
||||
|
||||
## It runs once per account
|
||||
|
||||
The audit row is written inside the same transaction as the update, and an
|
||||
account that carries it is skipped. A second run reports the account as already
|
||||
repaired and changes nothing, so the multiplier can never be applied twice.
|
||||
|
||||
## Verify
|
||||
|
||||
```sql
|
||||
SELECT date_trunc('year', measured_at) AS year,
|
||||
round(avg(value)) AS avg_metres_per_day
|
||||
FROM measurements
|
||||
WHERE user_id = '<userId>'
|
||||
AND type = 'WALKING_RUNNING_DISTANCE'
|
||||
AND aggregation_provenance = 'EXPORT_XML_SOURCE_MAX'
|
||||
GROUP BY 1 ORDER BY 1;
|
||||
```
|
||||
|
||||
A plausible daily average is thousands of metres. Single digits mean the rows
|
||||
were not repaired; millions mean a factor was applied twice, which the audit
|
||||
row is there to prevent.
|
||||
@@ -24,6 +24,7 @@ typecheck.
|
||||
- **`assert-deploy.ts`** — post-deploy `/api/version` assertion.
|
||||
- **`rotate-encryption-key.ts`** — AES-256-GCM key rotation; see `docs/ops/encryption-key-rotation.md`.
|
||||
- **`restore-backup.ts`** — restore an off-host AES-GCM backup from an S3-compatible bucket.
|
||||
- **`repair-apple-health-distance.ts`** — repair walking-distance rows the `export.zip` import stored 1000x too small; dry-run by default, see `docs/ops/apple-health-distance-repair.md`.
|
||||
- **`repair-intake-anomalies.ts`** — repair historic medication-ledger defects (duplicate dose-slot rows, implausible `taken_at`); dry-run by default, see `docs/ops/intake-repair.md`.
|
||||
- **`backfill-rollups.ts`**, **`drain-per-sample-cumulative.ts`**, **`backfill-mood-note-column.ts`** — historical data backfills (note: the boot-time `rollup-full-backfill` queue handles new accounts automatically; the CLI is the manual fallback).
|
||||
- **`seed-demo.ts`** + **`seed-demo-v15.sql`** — populate a demo tenant.
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
/**
|
||||
* scripts/repair-apple-health-distance.ts — operator repair for walking
|
||||
* distances the `export.zip` import stored 1000x too small (issue #944).
|
||||
*
|
||||
* THE DEFECT (fixed in code from this release): the `<Record>` mapping read
|
||||
* the value but never the record's own `unit` attribute, so a walking-distance
|
||||
* record Apple writes in the account's display unit (`km` on a metric archive,
|
||||
* `mi` on an imperial one) was folded into the day's total as if the number
|
||||
* were already metres. Ten years of history read as a couple of metres a day.
|
||||
*
|
||||
* WHICH ROWS: only the ones carrying the stamp the archive importer writes —
|
||||
* `WALKING_RUNNING_DISTANCE` + `APPLE_HEALTH` + provenance
|
||||
* `EXPORT_XML_SOURCE_MAX` + a `stats:HKQuantityTypeIdentifierDistanceWalking-
|
||||
* Running:<day>` externalId. Never a guess from "the value looks small". Rows
|
||||
* a native iOS sync has since overwritten carry `HEALTHKIT_STATISTICS` and are
|
||||
* correct; rows whose provenance cannot prove an archive origin are left alone.
|
||||
*
|
||||
* If the account still has its `export.zip`, RE-IMPORTING it on the fixed build
|
||||
* is the authoritative repair — the fold UPSERTs the same `stats:` rows with
|
||||
* the right numbers. This script is for the accounts that no longer have it.
|
||||
*
|
||||
* SAFETY:
|
||||
* - Dry-run by DEFAULT: lists the candidate rows per account and exits.
|
||||
* - `--apply` writes, one transaction per account, and records an audit row
|
||||
* in that same transaction. An account carrying that row is skipped, so a
|
||||
* second run finds nothing.
|
||||
* - A row whose repaired value would leave the plausibility range refuses
|
||||
* the WHOLE account: one such row is evidence the rows are not the 1000x
|
||||
* class (an account re-imported on the fixed build looks exactly like
|
||||
* this), and its rest days would otherwise be multiplied again.
|
||||
* - The rollup/status tail runs after the transaction. If it throws, the
|
||||
* rows are repaired and the derived tier is not; the script names the
|
||||
* backfill to re-run, the audit row keeps `rollupsRefreshed: false` so a
|
||||
* later run can still see it, and the exit code is non-zero.
|
||||
* - `--unit=mi` for an imperial archive (default `km`). The stored row keeps
|
||||
* no record of the archive's unit — that is the defect — so the operator
|
||||
* names it; a mismatched unit is why the flag exists rather than a guess.
|
||||
*
|
||||
* RUN (never `pnpm tsx` — the standalone image strips tsx):
|
||||
* pnpm dlx tsx scripts/repair-apple-health-distance.ts # dry run
|
||||
* pnpm dlx tsx scripts/repair-apple-health-distance.ts --apply
|
||||
* pnpm dlx tsx scripts/repair-apple-health-distance.ts --unit=mi --apply
|
||||
*/
|
||||
import "dotenv/config";
|
||||
|
||||
import { prisma } from "@/lib/db";
|
||||
import {
|
||||
applyAppleHealthDistanceRepair,
|
||||
parseArchiveUnit,
|
||||
planAppleHealthDistanceRepair,
|
||||
repairFactorFor,
|
||||
rollupRepairCommand,
|
||||
} from "@/lib/measurements/repair-apple-health-distance";
|
||||
|
||||
function readArchiveUnit(): "km" | "mi" {
|
||||
const flag = process.argv.find((arg) => arg.startsWith("--unit="));
|
||||
if (!flag) return "km";
|
||||
const parsed = parseArchiveUnit(flag.slice("--unit=".length));
|
||||
if (!parsed) {
|
||||
throw new Error(`Unsupported --unit value: ${flag}. Use km or mi.`);
|
||||
}
|
||||
return parsed;
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const apply = process.argv.includes("--apply");
|
||||
const archiveUnit = readArchiveUnit();
|
||||
const factor = repairFactorFor(archiveUnit);
|
||||
|
||||
const plans = await planAppleHealthDistanceRepair(prisma, { archiveUnit });
|
||||
if (plans.length === 0) {
|
||||
console.log(
|
||||
"No export-imported walking-distance rows found — nothing to repair.",
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
console.log(
|
||||
`Archive unit ${archiveUnit} (x${factor} to metres). ` +
|
||||
`${plans.length} account(s) carry export-imported walking distances.\n`,
|
||||
);
|
||||
|
||||
let totalRepairable = 0;
|
||||
let totalUpdated = 0;
|
||||
let totalSkipped = 0;
|
||||
let refused = 0;
|
||||
let rollupsPending = 0;
|
||||
|
||||
for (const plan of plans) {
|
||||
if (plan.alreadyRepairedAt) {
|
||||
console.log(
|
||||
` • ${plan.userId}: already repaired on ` +
|
||||
`${plan.alreadyRepairedAt.toISOString()} — ${plan.candidateCount} row(s) left alone.`,
|
||||
);
|
||||
if (plan.rollupsPending) {
|
||||
rollupsPending += 1;
|
||||
console.log(
|
||||
" ! rollups PENDING from that run: the DAY/WEEK/MONTH/YEAR " +
|
||||
"buckets and the cached status insights still hold the " +
|
||||
"pre-repair numbers.",
|
||||
);
|
||||
console.log(` re-run: ${rollupRepairCommand(plan.userId)}`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
totalRepairable += plan.repairable.length;
|
||||
const first = plan.repairable[0];
|
||||
const last = plan.repairable[plan.repairable.length - 1];
|
||||
console.log(
|
||||
` • ${plan.userId}: ${plan.repairable.length} row(s)` +
|
||||
(first && last
|
||||
? ` from ${first.measuredAt.toISOString().slice(0, 10)} to ` +
|
||||
`${last.measuredAt.toISOString().slice(0, 10)}`
|
||||
: "") +
|
||||
(plan.outOfRange.length > 0
|
||||
? `, ${plan.outOfRange.length} out of plausible range (skipped)`
|
||||
: ""),
|
||||
);
|
||||
if (first) {
|
||||
console.log(
|
||||
` e.g. ${first.id}: ${first.value} m -> ${first.value * factor} m`,
|
||||
);
|
||||
}
|
||||
for (const row of plan.outOfRange) {
|
||||
console.log(
|
||||
` ! ${row.id} (${row.measuredAt.toISOString().slice(0, 10)}): ` +
|
||||
`${row.value} m x ${factor} leaves the plausible range`,
|
||||
);
|
||||
}
|
||||
if (plan.outOfRange.length > 0) {
|
||||
console.log(
|
||||
" ! this account will be REFUSED — see the reason under --apply",
|
||||
);
|
||||
}
|
||||
|
||||
if (!apply) continue;
|
||||
const outcome = await applyAppleHealthDistanceRepair(prisma, plan, {
|
||||
archiveUnit,
|
||||
});
|
||||
totalUpdated += outcome.updated;
|
||||
totalSkipped += outcome.skipped;
|
||||
if (outcome.refusedReason) {
|
||||
refused += 1;
|
||||
console.log(` ✗ refused: ${outcome.refusedReason}`);
|
||||
continue;
|
||||
}
|
||||
console.log(
|
||||
` ✓ repaired ${outcome.updated} row(s), skipped ${outcome.skipped}`,
|
||||
);
|
||||
if (outcome.rollupsPending) {
|
||||
rollupsPending += 1;
|
||||
console.error(
|
||||
` ! ${plan.userId}: rows repaired, but the rollup refresh failed.`,
|
||||
);
|
||||
console.error(outcome.rollupError);
|
||||
console.error(
|
||||
" The DAY/WEEK/MONTH/YEAR buckets and the cached status " +
|
||||
"insights still hold the pre-repair numbers.",
|
||||
);
|
||||
console.error(` re-run: ${rollupRepairCommand(plan.userId)}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log(
|
||||
apply
|
||||
? `\nDone. ${totalUpdated} row(s) repaired, ${totalSkipped} skipped, ` +
|
||||
`${refused} account(s) refused.`
|
||||
: `\nDry run: ${totalRepairable} row(s) would be repaired. ` +
|
||||
"Read the worked examples above and decide before re-running with " +
|
||||
"--apply.",
|
||||
);
|
||||
|
||||
if (rollupsPending > 0) {
|
||||
// The rows are right and the derived tier is not, which no later run of
|
||||
// this script will notice — it skips a repaired account. Fail loudly.
|
||||
console.error(
|
||||
`\n${rollupsPending} account(s) need the rollup backfill re-run ` +
|
||||
"(commands above).",
|
||||
);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
|
||||
main()
|
||||
.catch((err) => {
|
||||
console.error(err);
|
||||
process.exitCode = 1;
|
||||
})
|
||||
.finally(() => prisma.$disconnect());
|
||||
@@ -0,0 +1,246 @@
|
||||
/**
|
||||
* Structural guard on the Apple Health mapping table's unit column
|
||||
* (issue #944).
|
||||
*
|
||||
* `AppleHealthMapping.hkUnit` records the unit HealthLog ASSUMES a
|
||||
* HealthKit reading arrives in, and `convertToDbUnit` is written against
|
||||
* that assumption. Apple's `export.xml` does not honour it: every quantity
|
||||
* `<Record>` carries the account's own display unit, so a metric archive
|
||||
* reports walking distance in `km` and an imperial one in `mi`. For ten
|
||||
* years those readings were stored as if the number were already metres —
|
||||
* the whole history understated by a factor of a thousand — because the
|
||||
* mapper accepted `input.unit` and never read it.
|
||||
*
|
||||
* So an entry now has exactly two legal shapes:
|
||||
*
|
||||
* 1. its `hkUnit` is one `convertHkValue()` knows a factor for, in which
|
||||
* case `mapAppleHealthEntry()` converts a differing record unit into
|
||||
* it before `convertToDbUnit` runs; or
|
||||
* 2. it carries a written `unitFixedReason` saying why no unit attribute
|
||||
* Apple could write would change the reading (a dimensionless count,
|
||||
* a pinned event, a logarithmic dB level, a compound rate).
|
||||
*
|
||||
* The conversion is opt-in per caller (`convertRecordUnit`), and only the
|
||||
* archive path opts in: `POST /api/measurements/batch` documents its `unit`
|
||||
* as captured-not-validated and nothing pins the native client's strings.
|
||||
* The last block below holds that boundary, so the batch contract cannot be
|
||||
* widened by accident.
|
||||
*
|
||||
* The reason arm alone would be a guard that cannot fail: an entry could
|
||||
* declare a convertible unit while the mapper quietly ignored it. The
|
||||
* behavioural block below therefore feeds every convertible entry a
|
||||
* reading in a SIBLING unit and asserts the stored number moved by the
|
||||
* documented factor — so deleting the conversion turns this guard red,
|
||||
* not just the one reproduction test.
|
||||
*
|
||||
* A tripwire, not a proof: a reviewer who waves through a wrong factor
|
||||
* defeats it.
|
||||
*/
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
|
||||
import {
|
||||
APPLE_HEALTH_TYPE_MAP,
|
||||
mapAppleHealthEntry,
|
||||
} from "@/lib/measurements/apple-health-mapping";
|
||||
import {
|
||||
convertHkValue,
|
||||
isConvertibleHkUnit,
|
||||
} from "@/lib/measurements/hk-units";
|
||||
|
||||
const ENTRIES = Object.values(APPLE_HEALTH_TYPE_MAP);
|
||||
|
||||
/**
|
||||
* Units from every family the table touches. The behavioural block picks
|
||||
* the first one that converts into an entry's `hkUnit` and actually moves
|
||||
* the number, so a new family only needs one representative here.
|
||||
*/
|
||||
const SIBLING_UNITS = [
|
||||
"km",
|
||||
"mi",
|
||||
"cm",
|
||||
"lb",
|
||||
"st",
|
||||
"kJ",
|
||||
"cal",
|
||||
"degF",
|
||||
"K",
|
||||
"mmol<180.156>/L",
|
||||
"kPa",
|
||||
"km/hr",
|
||||
"hr",
|
||||
"ms",
|
||||
];
|
||||
|
||||
function siblingUnitFor(hkUnit: string): string | null {
|
||||
for (const candidate of SIBLING_UNITS) {
|
||||
const converted = convertHkValue(1, candidate, hkUnit);
|
||||
if (converted !== null && converted !== 1) return candidate;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
describe("Apple Health mapping — the record's own unit (issue #944)", () => {
|
||||
it("examines the whole mapping table", () => {
|
||||
// An empty match set is the failure mode this repo keeps rediscovering:
|
||||
// a guard is green because it matched nothing, not because nothing is
|
||||
// wrong. Pin the floor.
|
||||
expect(ENTRIES.length).toBeGreaterThan(40);
|
||||
});
|
||||
|
||||
it.each(ENTRIES)(
|
||||
"$hkIdentifier either converts its record unit or says why it cannot",
|
||||
(mapping) => {
|
||||
if (isConvertibleHkUnit(mapping.hkUnit)) {
|
||||
expect(mapping.unitFixedReason).toBeUndefined();
|
||||
return;
|
||||
}
|
||||
expect(
|
||||
mapping.unitFixedReason,
|
||||
`${mapping.hkIdentifier} pins hkUnit "${mapping.hkUnit}" with no conversion and no written reason`,
|
||||
).toBeTypeOf("string");
|
||||
// Long enough to be a sentence rather than a shrug.
|
||||
expect(mapping.unitFixedReason!.length).toBeGreaterThan(40);
|
||||
},
|
||||
);
|
||||
|
||||
const convertible = ENTRIES.filter((mapping) =>
|
||||
isConvertibleHkUnit(mapping.hkUnit),
|
||||
);
|
||||
|
||||
it("has convertible entries to exercise", () => {
|
||||
expect(convertible.length).toBeGreaterThan(10);
|
||||
});
|
||||
|
||||
it.each(convertible)(
|
||||
"$hkIdentifier honours a sibling unit on the record itself",
|
||||
(mapping) => {
|
||||
const sibling = siblingUnitFor(mapping.hkUnit);
|
||||
expect(
|
||||
sibling,
|
||||
`no sibling unit listed for the family of "${mapping.hkUnit}"`,
|
||||
).not.toBeNull();
|
||||
|
||||
const base = {
|
||||
hkIdentifier: mapping.hkIdentifier,
|
||||
value: 2,
|
||||
startDate: "2026-05-14T08:00:00.000Z",
|
||||
endDate: "2026-05-14T08:30:00.000Z",
|
||||
};
|
||||
const native = mapAppleHealthEntry(
|
||||
{ ...base, unit: mapping.hkUnit },
|
||||
{ convertRecordUnit: true },
|
||||
);
|
||||
const foreign = mapAppleHealthEntry(
|
||||
{ ...base, unit: sibling! },
|
||||
{ convertRecordUnit: true },
|
||||
);
|
||||
expect(native).not.toBeNull();
|
||||
expect(foreign).not.toBeNull();
|
||||
|
||||
const expected = mapping.convertToDbUnit(
|
||||
convertHkValue(2, sibling!, mapping.hkUnit)!,
|
||||
);
|
||||
expect(foreign!.value).toBeCloseTo(expected, 9);
|
||||
expect(foreign!.value).not.toBeCloseTo(native!.value, 9);
|
||||
},
|
||||
);
|
||||
|
||||
it("keeps a unit it cannot place from rescaling the reading", () => {
|
||||
const out = mapAppleHealthEntry(
|
||||
{
|
||||
hkIdentifier: "HKQuantityTypeIdentifierDistanceWalkingRunning",
|
||||
value: 1234,
|
||||
unit: "furlong",
|
||||
startDate: "2026-05-14T08:00:00.000Z",
|
||||
endDate: "2026-05-14T08:30:00.000Z",
|
||||
},
|
||||
{ convertRecordUnit: true },
|
||||
);
|
||||
expect(out?.value).toBe(1234);
|
||||
});
|
||||
|
||||
it("leaves the batch path's readings alone whatever unit they name", () => {
|
||||
// The batch contract: `unit` is captured, never read. Every convertible
|
||||
// entry has to behave that way by DEFAULT, or a native client stamping
|
||||
// the person's display unit silently rescales their history.
|
||||
for (const mapping of convertible) {
|
||||
const sibling = siblingUnitFor(mapping.hkUnit);
|
||||
if (!sibling) continue;
|
||||
const base = {
|
||||
hkIdentifier: mapping.hkIdentifier,
|
||||
value: 2,
|
||||
startDate: "2026-05-14T08:00:00.000Z",
|
||||
endDate: "2026-05-14T08:30:00.000Z",
|
||||
};
|
||||
const declared = mapAppleHealthEntry({ ...base, unit: mapping.hkUnit });
|
||||
const foreign = mapAppleHealthEntry({ ...base, unit: sibling });
|
||||
expect(declared).not.toBeNull();
|
||||
expect(
|
||||
foreign!.value,
|
||||
`${mapping.hkIdentifier} rescaled a batch reading from "${sibling}"`,
|
||||
).toBeCloseTo(declared!.value, 9);
|
||||
}
|
||||
});
|
||||
|
||||
it("has only the archive path opting into the conversion", () => {
|
||||
// A grep, not a proof — but the wire change this pins is exactly the
|
||||
// kind that lands as a one-word argument in an unrelated diff.
|
||||
const optIn = "convertRecordUnit: true";
|
||||
const archive = readFileSync(
|
||||
join(process.cwd(), "src/lib/measurements/import-apple-health-export.ts"),
|
||||
"utf8",
|
||||
);
|
||||
expect(archive).toContain(optIn);
|
||||
const batch = readFileSync(
|
||||
join(process.cwd(), "src/app/api/measurements/batch/route.ts"),
|
||||
"utf8",
|
||||
);
|
||||
expect(
|
||||
batch,
|
||||
"the batch route opted into record-unit conversion",
|
||||
).not.toContain("convertRecordUnit");
|
||||
});
|
||||
|
||||
it("leaves the km/mi factors in one shared module", () => {
|
||||
// The workout path used to carry its own copy of the km/mi branch; the
|
||||
// duplicate is what let the `<Record>` path stay wrong for ten years.
|
||||
//
|
||||
// `* 1000` alone is not the offence — these files also turn seconds into
|
||||
// milliseconds — so a rescale only counts when it stands in a LENGTH
|
||||
// context: a distance identifier on the same line, or within the five
|
||||
// lines above it (which covers a rescale in the body of a
|
||||
// distance-named helper). The limit is the usual one for a matcher of
|
||||
// this shape: a length conversion inside a helper named nothing like a
|
||||
// distance, more than five lines below its own signature, slips it.
|
||||
const shared = join(process.cwd(), "src/lib/measurements/hk-units.ts");
|
||||
const owners = [
|
||||
"src/lib/measurements/apple-health-mapping.ts",
|
||||
"src/lib/measurements/import-apple-health-export.ts",
|
||||
];
|
||||
expect(readFileSync(shared, "utf8")).toContain("1609.344");
|
||||
|
||||
const RESCALE = /\*\s*1_?000(?!\d)/;
|
||||
const LENGTH = /distance|metre|meter|\bkm\b/i;
|
||||
for (const rel of owners) {
|
||||
const source = readFileSync(join(process.cwd(), rel), "utf8");
|
||||
expect(source, `${rel} re-implements the mile factor`).not.toContain(
|
||||
"1609.344",
|
||||
);
|
||||
const lines = source.split("\n");
|
||||
const offenders = lines
|
||||
.map((line, i) => ({ line, i }))
|
||||
.filter(
|
||||
({ line, i }) =>
|
||||
RESCALE.test(line) &&
|
||||
lines.slice(Math.max(0, i - 5), i + 1).some((l) => LENGTH.test(l)),
|
||||
)
|
||||
.map(({ line, i }) => `${rel}:${i + 1}: ${line.trim()}`);
|
||||
expect(
|
||||
offenders,
|
||||
`${rel} re-implements a length factor — the km/mi conversion belongs in hk-units.ts`,
|
||||
).toEqual([]);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -390,6 +390,38 @@ describe("mapAppleHealthEntry", () => {
|
||||
expect(out!.takenAt.toISOString()).toBe(ts);
|
||||
});
|
||||
|
||||
it("stores a batch reading in the unit the table assumes, whatever `unit` says", () => {
|
||||
// issue #944 — the batch route's `unit` is captured for audit and never
|
||||
// read. A client that stamps the person's DISPLAY unit into the field
|
||||
// must not have its metres multiplied by a thousand: the conversion is
|
||||
// opt-in and only the `export.xml` path opts in.
|
||||
const out = mapAppleHealthEntry({
|
||||
hkIdentifier: "HKQuantityTypeIdentifierDistanceWalkingRunning",
|
||||
value: 2484,
|
||||
unit: "km",
|
||||
startDate: ts,
|
||||
endDate: ts,
|
||||
});
|
||||
expect(out).not.toBeNull();
|
||||
expect(out!.value).toBe(2484);
|
||||
expect(out!.unit).toBe("m");
|
||||
});
|
||||
|
||||
it("converts out of the record's own unit when the caller opts in", () => {
|
||||
const out = mapAppleHealthEntry(
|
||||
{
|
||||
hkIdentifier: "HKQuantityTypeIdentifierDistanceWalkingRunning",
|
||||
value: 2.484,
|
||||
unit: "km",
|
||||
startDate: ts,
|
||||
endDate: ts,
|
||||
},
|
||||
{ convertRecordUnit: true },
|
||||
);
|
||||
expect(out).not.toBeNull();
|
||||
expect(out!.value).toBeCloseTo(2484, 9);
|
||||
});
|
||||
|
||||
it("applies the unit conversion for fraction-shaped quantities", () => {
|
||||
const out = mapAppleHealthEntry({
|
||||
hkIdentifier: "HKQuantityTypeIdentifierOxygenSaturation",
|
||||
|
||||
@@ -0,0 +1,77 @@
|
||||
import { describe, it, expect } from "vitest";
|
||||
|
||||
import {
|
||||
convertHkValue,
|
||||
hkDistanceToMetres,
|
||||
isConvertibleHkUnit,
|
||||
} from "../hk-units";
|
||||
|
||||
describe("convertHkValue", () => {
|
||||
it("converts the km reading Apple writes on a metric archive", () => {
|
||||
expect(convertHkValue(2.484, "km", "m")).toBeCloseTo(2484, 9);
|
||||
});
|
||||
|
||||
it("converts the mi reading Apple writes on an imperial archive", () => {
|
||||
expect(convertHkValue(1.543, "mi", "m")).toBeCloseTo(2483.217792, 9);
|
||||
});
|
||||
|
||||
it("returns the reading untouched when the units already agree", () => {
|
||||
expect(convertHkValue(2.484, "m", "m")).toBe(2.484);
|
||||
});
|
||||
|
||||
it("converts mass, energy and temperature the same way", () => {
|
||||
expect(convertHkValue(180, "lb", "kg")).toBeCloseTo(81.6466266, 6);
|
||||
expect(convertHkValue(1000, "kJ", "kcal")).toBeCloseTo(239.005736, 6);
|
||||
expect(convertHkValue(98.6, "degF", "degC")).toBeCloseTo(37, 9);
|
||||
});
|
||||
|
||||
it("strips the molar mass Apple annotates a glucose unit with", () => {
|
||||
expect(convertHkValue(5.5, "mmol<180.156>/L", "mg/dL")).toBeCloseTo(
|
||||
99.0858,
|
||||
4,
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps the small calorie apart from the large one", () => {
|
||||
expect(convertHkValue(1, "Cal", "kcal")).toBe(1);
|
||||
expect(convertHkValue(1000, "cal", "kcal")).toBeCloseTo(1, 9);
|
||||
});
|
||||
|
||||
it("refuses a unit it cannot place rather than guessing", () => {
|
||||
expect(convertHkValue(1, "furlong", "m")).toBeNull();
|
||||
expect(convertHkValue(1, "", "m")).toBeNull();
|
||||
expect(convertHkValue(1, undefined, "m")).toBeNull();
|
||||
});
|
||||
|
||||
it("refuses to cross families", () => {
|
||||
expect(convertHkValue(1, "kg", "m")).toBeNull();
|
||||
expect(convertHkValue(1, "min", "kcal")).toBeNull();
|
||||
});
|
||||
|
||||
it("refuses a non-finite reading", () => {
|
||||
expect(convertHkValue(Number.NaN, "km", "m")).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("hkDistanceToMetres", () => {
|
||||
it("converts the workout distance units Apple ships", () => {
|
||||
expect(hkDistanceToMetres(6.5, "km")).toBeCloseTo(6500, 9);
|
||||
expect(hkDistanceToMetres(3, "mi")).toBeCloseTo(4828.032, 9);
|
||||
expect(hkDistanceToMetres(500, "m")).toBe(500);
|
||||
});
|
||||
|
||||
it("falls back to the raw number for a missing unit attribute", () => {
|
||||
expect(hkDistanceToMetres(500, "")).toBe(500);
|
||||
expect(hkDistanceToMetres(500, undefined)).toBe(500);
|
||||
});
|
||||
});
|
||||
|
||||
describe("isConvertibleHkUnit", () => {
|
||||
it("recognises the units the mapping table pins", () => {
|
||||
expect(isConvertibleHkUnit("m")).toBe(true);
|
||||
expect(isConvertibleHkUnit("kg")).toBe(true);
|
||||
expect(isConvertibleHkUnit("count")).toBe(false);
|
||||
expect(isConvertibleHkUnit("event")).toBe(false);
|
||||
expect(isConvertibleHkUnit("%")).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -1278,3 +1278,67 @@ describe("streamParseExportXml — the archive's own export stamp", () => {
|
||||
expect(stamps).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("streamParseExportXml — the record's own unit (issue #944)", () => {
|
||||
it("stores a km-scale walking distance in metres", async () => {
|
||||
const { prisma } = await importCumulativeFixture(
|
||||
cumulativeExportXml(
|
||||
"HKQuantityTypeIdentifierDistanceWalkingRunning",
|
||||
"km",
|
||||
[{ value: 2.484, sourceName: "iPhone", hour: 8 }],
|
||||
),
|
||||
);
|
||||
|
||||
expect(prisma._measurements).toHaveLength(1);
|
||||
expect(prisma._measurements[0]).toMatchObject({
|
||||
type: "WALKING_RUNNING_DISTANCE",
|
||||
unit: "m",
|
||||
});
|
||||
expect(prisma._measurements[0].value).toBeCloseTo(2484, 6);
|
||||
});
|
||||
|
||||
it("stores a kilojoule workout burn in kilocalories", async () => {
|
||||
const tmp = mkdtempSync(join(tmpdir(), "healthlog-parser-energy-test-"));
|
||||
const xmlPath = join(tmp, "export.xml");
|
||||
writeFileSync(
|
||||
xmlPath,
|
||||
`<?xml version="1.0" encoding="UTF-8"?>
|
||||
<HealthData locale="en_AU">
|
||||
<Workout workoutActivityType="HKWorkoutActivityTypeRunning"
|
||||
duration="42.0"
|
||||
durationUnit="min"
|
||||
totalDistance="6.5"
|
||||
totalDistanceUnit="km"
|
||||
totalEnergyBurned="1724"
|
||||
totalEnergyBurnedUnit="kJ"
|
||||
startDate="2026-05-14 18:00:00 +0200"
|
||||
endDate="2026-05-14 18:42:00 +0200"
|
||||
sourceName="Apple Watch"/>
|
||||
</HealthData>`,
|
||||
);
|
||||
const prisma = makeFakePrisma();
|
||||
await streamParseExportXml({
|
||||
xmlPath,
|
||||
userId: "user-energy",
|
||||
userTimezone: "Europe/Berlin",
|
||||
prisma: prisma as unknown as PrismaClient,
|
||||
});
|
||||
|
||||
expect(prisma._workouts).toHaveLength(1);
|
||||
expect(prisma._workouts[0].totalDistanceM).toBeCloseTo(6500, 6);
|
||||
expect(prisma._workouts[0].totalEnergyKcal).toBeCloseTo(412.0458891, 6);
|
||||
});
|
||||
|
||||
it("stores a mile-scale walking distance in metres", async () => {
|
||||
const { prisma } = await importCumulativeFixture(
|
||||
cumulativeExportXml(
|
||||
"HKQuantityTypeIdentifierDistanceWalkingRunning",
|
||||
"mi",
|
||||
[{ value: 1.543, sourceName: "iPhone", hour: 8 }],
|
||||
),
|
||||
);
|
||||
|
||||
expect(prisma._measurements).toHaveLength(1);
|
||||
expect(prisma._measurements[0].value).toBeCloseTo(2483.217792, 6);
|
||||
});
|
||||
});
|
||||
@@ -28,6 +28,7 @@ import type {
|
||||
SleepStage,
|
||||
RhythmClassification,
|
||||
} from "@/generated/prisma/client";
|
||||
import { convertHkValue } from "./hk-units";
|
||||
|
||||
/**
|
||||
* How a batch of HealthKit samples for the same identifier should be
|
||||
@@ -47,6 +48,22 @@ export interface AppleHealthMapping {
|
||||
measurementType: MeasurementType;
|
||||
/** Apple's HK unit string (e.g. `kg`, `count/min`, `mL/(kg*min)`). */
|
||||
hkUnit: string;
|
||||
/**
|
||||
* issue #944 — why this entry's `hkUnit` can be trusted as fixed.
|
||||
*
|
||||
* Apple's `export.xml` writes the ACCOUNT's display unit on every
|
||||
* quantity `<Record>`, not the unit this table assumes, so
|
||||
* `mapAppleHealthEntry()` converts the reading from the record's own
|
||||
* unit into `hkUnit` before `convertToDbUnit` runs — on the archive
|
||||
* path, which is the caller that opts in. That only works for a unit
|
||||
* `convertHkValue()` knows a factor for. An entry whose `hkUnit`
|
||||
* is NOT convertible (a dimensionless count, a pinned event, a
|
||||
* logarithmic dB level, a compound rate) must say here why no record
|
||||
* unit can change its reading — the structural guard
|
||||
* `src/__tests__/apple-health-unit-audit-guard.test.ts` fails on an
|
||||
* entry that offers neither.
|
||||
*/
|
||||
unitFixedReason?: string;
|
||||
/** HealthLog canonical DB unit (must match `getUnitForType()`). */
|
||||
dbUnit: string;
|
||||
/** Convert a HK sample value into the canonical DB unit. Identity for most. */
|
||||
@@ -189,6 +206,28 @@ export const APPLE_HEALTH_SLEEP_STAGE_MAP: Record<number, SleepStage> = {
|
||||
* metric: check Apple's HK unit; percent → ×100 path, m/m·s/kg/etc
|
||||
* → identity path.
|
||||
*/
|
||||
/**
|
||||
* issue #944 — the written reasons a mapping entry may cite for keeping a
|
||||
* fixed `hkUnit`. Each one answers the same question: what could Apple
|
||||
* possibly write in this record's `unit` attribute that would change the
|
||||
* reading? Shared constants rather than free text so the audit reads as
|
||||
* one decision per class instead of forty near-duplicates.
|
||||
*/
|
||||
const UNIT_FIXED_DIMENSIONLESS_COUNT =
|
||||
"Dimensionless tally: Apple emits `count` in every locale and HKUnit has no sibling spelling for it, so no record unit can rescale the reading.";
|
||||
const UNIT_FIXED_PER_MINUTE =
|
||||
"Per-minute rate: Apple emits `count/min` in every locale and the Health app offers no alternative display unit, so there is nothing to convert from.";
|
||||
const UNIT_FIXED_PERCENT_FRACTION =
|
||||
"Apple writes the percent unit but the VALUE rides as a 0..1 fraction; the x100 in `convertToDbUnit` already covers both spellings, so branching on the record unit would double-scale it.";
|
||||
const UNIT_FIXED_EVENT_PIN =
|
||||
"Categorical event: `convertToDbUnit` pins the stored value to 1 fired event regardless of the reading, so no unit attribute can move it.";
|
||||
const UNIT_FIXED_AUDIO_SPL =
|
||||
"Sound-pressure level: Apple emits `dBASPL` only, and a decibel is logarithmic — a sibling unit would need its own formula, never a factor.";
|
||||
const UNIT_FIXED_COMPOSITE_VO2 =
|
||||
"Compound rate Apple emits verbatim as `mL/(kg*min)`; the Health app exposes no other spelling and the compound has no sibling unit to convert from.";
|
||||
const UNIT_FIXED_SLEEP_CATEGORY =
|
||||
"Sleep samples carry a symbolic category value, not a quantity: the importer derives the duration in minutes from the sample's own dates, so the record's unit attribute (empty in Apple's archive) says nothing about the reading.";
|
||||
|
||||
export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
// ── Body composition ────────────────────────────────────────
|
||||
HKQuantityTypeIdentifierBodyMass: {
|
||||
@@ -203,6 +242,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierBodyFatPercentage",
|
||||
measurementType: "BODY_FAT",
|
||||
hkUnit: "%",
|
||||
unitFixedReason: UNIT_FIXED_PERCENT_FRACTION,
|
||||
dbUnit: "%",
|
||||
// Apple ships 0..1 fraction; HealthLog stores 0..100.
|
||||
convertToDbUnit: (v) => v * 100,
|
||||
@@ -254,6 +294,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierHeartRate",
|
||||
measurementType: "PULSE",
|
||||
hkUnit: "count/min",
|
||||
unitFixedReason: UNIT_FIXED_PER_MINUTE,
|
||||
dbUnit: "bpm",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "latest",
|
||||
@@ -262,6 +303,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierRestingHeartRate",
|
||||
measurementType: "RESTING_HEART_RATE",
|
||||
hkUnit: "count/min",
|
||||
unitFixedReason: UNIT_FIXED_PER_MINUTE,
|
||||
dbUnit: "bpm",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "latest",
|
||||
@@ -284,6 +326,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
// duplication; see `.planning/phase-W1-v1423-research.md`).
|
||||
measurementType: "ACTIVITY_STEPS",
|
||||
hkUnit: "count",
|
||||
unitFixedReason: UNIT_FIXED_DIMENSIONLESS_COUNT,
|
||||
dbUnit: "steps",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "sum",
|
||||
@@ -300,6 +343,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierFlightsClimbed",
|
||||
measurementType: "FLIGHTS_CLIMBED",
|
||||
hkUnit: "count",
|
||||
unitFixedReason: UNIT_FIXED_DIMENSIONLESS_COUNT,
|
||||
dbUnit: "flights",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "sum",
|
||||
@@ -318,6 +362,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierVO2Max",
|
||||
measurementType: "VO2_MAX",
|
||||
hkUnit: "mL/(kg*min)",
|
||||
unitFixedReason: UNIT_FIXED_COMPOSITE_VO2,
|
||||
dbUnit: "mL/(kg·min)",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "latest",
|
||||
@@ -337,6 +382,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierOxygenSaturation",
|
||||
measurementType: "OXYGEN_SATURATION",
|
||||
hkUnit: "fraction",
|
||||
unitFixedReason: UNIT_FIXED_PERCENT_FRACTION,
|
||||
dbUnit: "%",
|
||||
// Apple ships 0..1 fraction; HealthLog stores 0..100.
|
||||
convertToDbUnit: (v) => v * 100,
|
||||
@@ -348,6 +394,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKCategoryTypeIdentifierSleepAnalysis",
|
||||
measurementType: "SLEEP_DURATION",
|
||||
hkUnit: "category",
|
||||
unitFixedReason: UNIT_FIXED_SLEEP_CATEGORY,
|
||||
dbUnit: "minutes",
|
||||
// value here is duration-in-minutes (caller computes
|
||||
// `endDate - startDate` and converts to minutes); the per-stage
|
||||
@@ -367,6 +414,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierEnvironmentalAudioExposure",
|
||||
measurementType: "AUDIO_EXPOSURE_ENV",
|
||||
hkUnit: "dBASPL",
|
||||
unitFixedReason: UNIT_FIXED_AUDIO_SPL,
|
||||
dbUnit: "dBA",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "mean",
|
||||
@@ -379,6 +427,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierHeadphoneAudioExposure",
|
||||
measurementType: "AUDIO_EXPOSURE_HEADPHONE",
|
||||
hkUnit: "dBASPL",
|
||||
unitFixedReason: UNIT_FIXED_AUDIO_SPL,
|
||||
dbUnit: "dBA",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "mean",
|
||||
@@ -403,6 +452,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierAppleWalkingSteadiness",
|
||||
measurementType: "WALKING_STEADINESS",
|
||||
hkUnit: "%",
|
||||
unitFixedReason: UNIT_FIXED_PERCENT_FRACTION,
|
||||
dbUnit: "%",
|
||||
// Apple ships 0..1 fraction; HealthLog stores 0..100.
|
||||
convertToDbUnit: (v) => v * 100,
|
||||
@@ -416,6 +466,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKCategoryTypeIdentifierEnvironmentalAudioExposureEvent",
|
||||
measurementType: "AUDIO_EXPOSURE_EVENT",
|
||||
hkUnit: "count",
|
||||
unitFixedReason: UNIT_FIXED_EVENT_PIN,
|
||||
dbUnit: "count",
|
||||
convertToDbUnit: () => 1,
|
||||
aggregation: "sum",
|
||||
@@ -427,6 +478,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKCategoryTypeIdentifierHeadphoneAudioExposureEvent",
|
||||
measurementType: "AUDIO_EXPOSURE_EVENT",
|
||||
hkUnit: "count",
|
||||
unitFixedReason: UNIT_FIXED_EVENT_PIN,
|
||||
dbUnit: "count",
|
||||
convertToDbUnit: () => 1,
|
||||
aggregation: "sum",
|
||||
@@ -449,6 +501,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKCategoryTypeIdentifierIrregularHeartRhythmEvent",
|
||||
measurementType: "IRREGULAR_RHYTHM_NOTIFICATION",
|
||||
hkUnit: "event",
|
||||
unitFixedReason: UNIT_FIXED_EVENT_PIN,
|
||||
dbUnit: "event",
|
||||
convertToDbUnit: () => 1,
|
||||
aggregation: "sum",
|
||||
@@ -462,6 +515,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKCategoryTypeIdentifierHighHeartRateEvent",
|
||||
measurementType: "HIGH_HEART_RATE_EVENT",
|
||||
hkUnit: "event",
|
||||
unitFixedReason: UNIT_FIXED_EVENT_PIN,
|
||||
dbUnit: "event",
|
||||
convertToDbUnit: () => 1,
|
||||
aggregation: "sum",
|
||||
@@ -474,6 +528,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKCategoryTypeIdentifierLowHeartRateEvent",
|
||||
measurementType: "LOW_HEART_RATE_EVENT",
|
||||
hkUnit: "event",
|
||||
unitFixedReason: UNIT_FIXED_EVENT_PIN,
|
||||
dbUnit: "event",
|
||||
convertToDbUnit: () => 1,
|
||||
aggregation: "sum",
|
||||
@@ -487,6 +542,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKCategoryTypeIdentifierAppleWalkingSteadinessEvent",
|
||||
measurementType: "WALKING_STEADINESS_EVENT",
|
||||
hkUnit: "event",
|
||||
unitFixedReason: UNIT_FIXED_EVENT_PIN,
|
||||
dbUnit: "event",
|
||||
convertToDbUnit: () => 1,
|
||||
aggregation: "sum",
|
||||
@@ -501,6 +557,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKCategoryTypeIdentifierSleepApneaEvent",
|
||||
measurementType: "BREATHING_DISTURBANCE_EVENT",
|
||||
hkUnit: "event",
|
||||
unitFixedReason: UNIT_FIXED_EVENT_PIN,
|
||||
dbUnit: "event",
|
||||
convertToDbUnit: () => 1,
|
||||
aggregation: "sum",
|
||||
@@ -524,6 +581,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierRespiratoryRate",
|
||||
measurementType: "RESPIRATORY_RATE",
|
||||
hkUnit: "count/min",
|
||||
unitFixedReason: UNIT_FIXED_PER_MINUTE,
|
||||
dbUnit: "breaths/min",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "mean",
|
||||
@@ -537,6 +595,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierBodyMassIndex",
|
||||
measurementType: "BODY_MASS_INDEX",
|
||||
hkUnit: "count",
|
||||
unitFixedReason: UNIT_FIXED_DIMENSIONLESS_COUNT,
|
||||
dbUnit: "kg/m²",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "latest",
|
||||
@@ -557,6 +616,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierWalkingHeartRateAverage",
|
||||
measurementType: "WALKING_HEART_RATE_AVERAGE",
|
||||
hkUnit: "count/min",
|
||||
unitFixedReason: UNIT_FIXED_PER_MINUTE,
|
||||
dbUnit: "bpm",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "mean",
|
||||
@@ -570,6 +630,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierWalkingAsymmetryPercentage",
|
||||
measurementType: "WALKING_ASYMMETRY",
|
||||
hkUnit: "%",
|
||||
unitFixedReason: UNIT_FIXED_PERCENT_FRACTION,
|
||||
dbUnit: "%",
|
||||
convertToDbUnit: (v) => v * 100,
|
||||
aggregation: "latest",
|
||||
@@ -580,6 +641,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierWalkingDoubleSupportPercentage",
|
||||
measurementType: "WALKING_DOUBLE_SUPPORT",
|
||||
hkUnit: "%",
|
||||
unitFixedReason: UNIT_FIXED_PERCENT_FRACTION,
|
||||
dbUnit: "%",
|
||||
convertToDbUnit: (v) => v * 100,
|
||||
aggregation: "latest",
|
||||
@@ -622,6 +684,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierHeartRateRecoveryOneMinute",
|
||||
measurementType: "CARDIO_RECOVERY",
|
||||
hkUnit: "count/min",
|
||||
unitFixedReason: UNIT_FIXED_PER_MINUTE,
|
||||
dbUnit: "bpm",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "latest",
|
||||
@@ -645,6 +708,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierNumberOfTimesFallen",
|
||||
measurementType: "FALL_COUNT",
|
||||
hkUnit: "count",
|
||||
unitFixedReason: UNIT_FIXED_DIMENSIONLESS_COUNT,
|
||||
dbUnit: "count",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "sum",
|
||||
@@ -685,6 +749,7 @@ export const APPLE_HEALTH_TYPE_MAP: Record<string, AppleHealthMapping> = {
|
||||
hkIdentifier: "HKQuantityTypeIdentifierAppleSleepingBreathingDisturbances",
|
||||
measurementType: "BREATHING_DISTURBANCES",
|
||||
hkUnit: "count",
|
||||
unitFixedReason: UNIT_FIXED_DIMENSIONLESS_COUNT,
|
||||
dbUnit: "count",
|
||||
convertToDbUnit: (v) => v,
|
||||
aggregation: "latest",
|
||||
@@ -1140,7 +1205,14 @@ export interface AppleHealthEntryInput {
|
||||
hkIdentifier: string;
|
||||
/** Numeric value as Apple delivers it (pre-conversion). */
|
||||
value: number;
|
||||
/** Apple's unit string, captured for audit; not currently validated. */
|
||||
/**
|
||||
* Apple's unit string for this reading, captured for audit.
|
||||
*
|
||||
* Read ONLY when the caller opts in with `convertRecordUnit` — see
|
||||
* `AppleHealthEntryOptions`. Every other caller gets the historical
|
||||
* contract: the field is recorded and the reading is taken to already
|
||||
* be in the table's `hkUnit`.
|
||||
*/
|
||||
unit: string;
|
||||
/** ISO timestamp string (e.g. HealthKit `startDate`). */
|
||||
startDate: string;
|
||||
@@ -1175,6 +1247,33 @@ export interface AppleHealthEntryInput {
|
||||
valueMax?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-call options for `mapAppleHealthEntry()`.
|
||||
*/
|
||||
export interface AppleHealthEntryOptions {
|
||||
/**
|
||||
* Convert the reading out of `input.unit` and into the table's `hkUnit`
|
||||
* before `convertToDbUnit` runs. Default FALSE.
|
||||
*
|
||||
* issue #944 — only the `export.xml` archive path passes `true`, because
|
||||
* only there is the unit attribute known to be authoritative: Apple
|
||||
* stamps every quantity `<Record>` with the ACCOUNT's display unit, so a
|
||||
* metric archive reports walking distance in `km` and the number is
|
||||
* meaningless without it.
|
||||
*
|
||||
* `POST /api/measurements/batch` deliberately does NOT opt in. Its `unit`
|
||||
* is documented as captured-for-audit, not validated, and the native
|
||||
* client's unit strings for the seventeen convertible identifiers are not
|
||||
* pinned by anything in this repository — no fixture, no contract test, no
|
||||
* coordination note. A client that reads a sample in metres and stamps the
|
||||
* person's DISPLAY unit (`km`) into the field would have every walking
|
||||
* distance multiplied by a thousand on ingest. Reading the field there is
|
||||
* a wire change, and a wire change needs the client's strings pinned
|
||||
* first.
|
||||
*/
|
||||
convertRecordUnit?: boolean;
|
||||
}
|
||||
|
||||
/** Output of `mapAppleHealthEntry()`. */
|
||||
export interface AppleHealthEntryOutput {
|
||||
type: MeasurementType;
|
||||
@@ -1214,6 +1313,7 @@ export interface AppleHealthEntryOutput {
|
||||
*/
|
||||
export function mapAppleHealthEntry(
|
||||
input: AppleHealthEntryInput,
|
||||
options: AppleHealthEntryOptions = {},
|
||||
): AppleHealthEntryOutput | null {
|
||||
const mapping = APPLE_HEALTH_TYPE_MAP[input.hkIdentifier];
|
||||
if (!mapping) return null;
|
||||
@@ -1221,7 +1321,20 @@ export function mapAppleHealthEntry(
|
||||
const takenAt = new Date(input.endDate);
|
||||
if (Number.isNaN(takenAt.getTime())) return null;
|
||||
|
||||
const value = mapping.convertToDbUnit(input.value);
|
||||
// issue #944 — `convertToDbUnit` is written against the table's fixed
|
||||
// `hkUnit`, but Apple's `export.xml` stamps each `<Record>` with the
|
||||
// ACCOUNT's own display unit: a metric archive reports walking distance
|
||||
// in `km`, an imperial one in `mi`, body mass in `lb`, energy in `kJ`.
|
||||
// The archive path opts in and the reading is normalised into `hkUnit`
|
||||
// first; a unit we cannot place leaves the value untouched (see
|
||||
// `convertHkValue`). Every other caller keeps the identity behaviour —
|
||||
// see `AppleHealthEntryOptions.convertRecordUnit` for why.
|
||||
const toHkUnit = (raw: number): number =>
|
||||
options.convertRecordUnit
|
||||
? (convertHkValue(raw, input.unit, mapping.hkUnit) ?? raw)
|
||||
: raw;
|
||||
|
||||
const value = mapping.convertToDbUnit(toHkUnit(input.value));
|
||||
|
||||
const out: AppleHealthEntryOutput = {
|
||||
type: mapping.measurementType,
|
||||
@@ -1235,10 +1348,10 @@ export function mapAppleHealthEntry(
|
||||
// hourly HR bucket). A finite check guards a malformed payload; the
|
||||
// caller only persists these on the HR-bucket row.
|
||||
if (typeof input.valueMin === "number" && Number.isFinite(input.valueMin)) {
|
||||
out.valueMin = mapping.convertToDbUnit(input.valueMin);
|
||||
out.valueMin = mapping.convertToDbUnit(toHkUnit(input.valueMin));
|
||||
}
|
||||
if (typeof input.valueMax === "number" && Number.isFinite(input.valueMax)) {
|
||||
out.valueMax = mapping.convertToDbUnit(input.valueMax);
|
||||
out.valueMax = mapping.convertToDbUnit(toHkUnit(input.valueMax));
|
||||
}
|
||||
|
||||
if (mapping.sleepStageMap) {
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
/**
|
||||
* HealthKit unit normalisation (issue #944).
|
||||
*
|
||||
* Apple writes a `unit` attribute on every quantity `<Record>` of an
|
||||
* `export.xml` archive, and the string it writes is the account's OWN
|
||||
* display unit rather than the unit HealthLog's mapping table assumes:
|
||||
* a metric account exports walking distance as `km`, an imperial one as
|
||||
* `mi`, and the same split shows up on body mass (`kg` / `lb` / `st`),
|
||||
* energy (`kcal` / `kJ`) and temperature (`degC` / `degF`).
|
||||
*
|
||||
* Before this module the export import fed the raw number straight into
|
||||
* `AppleHealthMapping.convertToDbUnit`, which is written against the
|
||||
* table's fixed `hkUnit` — so a 2.484 km day was stored as 2.484 m. The
|
||||
* workout path had a hand-rolled km/mi→m branch of its own; both paths
|
||||
* now share `convertHkValue()`.
|
||||
*
|
||||
* Contract, deliberately conservative:
|
||||
*
|
||||
* - Same unit in and out → the value is returned untouched, so an
|
||||
* archive (or an iOS batch) that already speaks the table's unit is
|
||||
* byte-for-byte unchanged.
|
||||
* - Unknown unit, empty unit, or two units from different families →
|
||||
* `null`, and every caller falls back to the raw value. A unit we
|
||||
* cannot prove a factor for must never silently rescale a reading.
|
||||
* - Only units Apple's `HKUnit` can actually emit are listed. A unit
|
||||
* string is matched exactly (after trimming and after stripping the
|
||||
* molar-mass annotation Apple writes on glucose), never lower-cased:
|
||||
* `cal` (small calorie) and `Cal` (kilocalorie) are different units
|
||||
* that differ only in case.
|
||||
*/
|
||||
|
||||
interface HkUnitSpec {
|
||||
/** Units convert only within one family. */
|
||||
family: string;
|
||||
/** This unit → the family's base unit. */
|
||||
toBase: (value: number) => number;
|
||||
/** The family's base unit → this unit. */
|
||||
fromBase: (value: number) => number;
|
||||
}
|
||||
|
||||
/** A unit that is a plain multiple of its family's base unit. */
|
||||
function scaled(family: string, perBase: number): HkUnitSpec {
|
||||
return {
|
||||
family,
|
||||
toBase: (value) => value * perBase,
|
||||
fromBase: (value) => value / perBase,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Every unit string Apple emits that HealthLog can convert, keyed by the
|
||||
* exact `HKUnit.unitString` spelling. Base units: metre, kilogram,
|
||||
* kilocalorie, second, degree Celsius, mmHg, mg/dL, metres per second.
|
||||
*/
|
||||
const HK_UNITS: Record<string, HkUnitSpec> = {
|
||||
// ── Length (base: metre) ───────────────────────────────────
|
||||
m: scaled("length", 1),
|
||||
km: scaled("length", 1000),
|
||||
cm: scaled("length", 0.01),
|
||||
mm: scaled("length", 0.001),
|
||||
in: scaled("length", 0.0254),
|
||||
ft: scaled("length", 0.3048),
|
||||
yd: scaled("length", 0.9144),
|
||||
mi: scaled("length", 1609.344),
|
||||
|
||||
// ── Mass (base: kilogram) ──────────────────────────────────
|
||||
kg: scaled("mass", 1),
|
||||
g: scaled("mass", 0.001),
|
||||
lb: scaled("mass", 0.45359237),
|
||||
oz: scaled("mass", 0.028349523125),
|
||||
st: scaled("mass", 6.35029318),
|
||||
|
||||
// ── Energy (base: kilocalorie) ─────────────────────────────
|
||||
kcal: scaled("energy", 1),
|
||||
Cal: scaled("energy", 1), // Apple's large calorie == kilocalorie
|
||||
cal: scaled("energy", 0.001), // small calorie — case is load-bearing
|
||||
kJ: scaled("energy", 1 / 4.184),
|
||||
J: scaled("energy", 1 / 4184),
|
||||
|
||||
// ── Time (base: second) ────────────────────────────────────
|
||||
s: scaled("time", 1),
|
||||
ms: scaled("time", 0.001),
|
||||
min: scaled("time", 60),
|
||||
hr: scaled("time", 3600),
|
||||
d: scaled("time", 86_400),
|
||||
|
||||
// ── Pressure (base: mmHg) ──────────────────────────────────
|
||||
mmHg: scaled("pressure", 1),
|
||||
kPa: scaled("pressure", 7.500_616_827_041_698),
|
||||
Pa: scaled("pressure", 0.007_500_616_827_041_698),
|
||||
|
||||
// ── Blood glucose (base: mg/dL) ────────────────────────────
|
||||
// Apple annotates the molar unit with glucose's molar mass, which the
|
||||
// normaliser strips before the lookup.
|
||||
"mg/dL": scaled("glucose", 1),
|
||||
"mmol/L": scaled("glucose", 18.0156), // 180.156 g/mol ÷ 10
|
||||
|
||||
// ── Speed (base: metres per second) ────────────────────────
|
||||
"m/s": scaled("speed", 1),
|
||||
"km/hr": scaled("speed", 1000 / 3600),
|
||||
"mi/hr": scaled("speed", 1609.344 / 3600),
|
||||
|
||||
// ── Temperature (base: degree Celsius) — affine, not scaled ─
|
||||
degC: {
|
||||
family: "temperature",
|
||||
toBase: (value) => value,
|
||||
fromBase: (value) => value,
|
||||
},
|
||||
degF: {
|
||||
family: "temperature",
|
||||
toBase: (value) => ((value - 32) * 5) / 9,
|
||||
fromBase: (value) => (value * 9) / 5 + 32,
|
||||
},
|
||||
K: {
|
||||
family: "temperature",
|
||||
toBase: (value) => value - 273.15,
|
||||
fromBase: (value) => value + 273.15,
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* Apple writes molar units with the substance's molar mass inline —
|
||||
* `mmol<180.156>/L` for blood glucose. The annotation is informational;
|
||||
* strip it so the lookup sees the plain unit.
|
||||
*/
|
||||
const MOLAR_MASS_ANNOTATION = /<[^>]*>/g;
|
||||
|
||||
function normaliseUnitString(unit: string): string {
|
||||
return unit.trim().replace(MOLAR_MASS_ANNOTATION, "");
|
||||
}
|
||||
|
||||
function lookup(unit: string | null | undefined): HkUnitSpec | null {
|
||||
if (typeof unit !== "string") return null;
|
||||
const normalised = normaliseUnitString(unit);
|
||||
if (!normalised) return null;
|
||||
return HK_UNITS[normalised] ?? null;
|
||||
}
|
||||
|
||||
/**
|
||||
* `true` when this module knows a conversion factor for the unit — i.e.
|
||||
* a `<Record>` that carries a sibling unit from the same family will be
|
||||
* converted rather than trusted blindly. The mapping-table guard in
|
||||
* `src/__tests__/apple-health-unit-audit-guard.test.ts` uses this to
|
||||
* decide whether an entry's fixed `hkUnit` needs a written reason.
|
||||
*/
|
||||
export function isConvertibleHkUnit(unit: string | null | undefined): boolean {
|
||||
return lookup(unit) !== null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a HealthKit value between two `HKUnit` strings. Returns `null`
|
||||
* — never a guess — when either unit is unknown or the two belong to
|
||||
* different families; callers keep the raw value in that case.
|
||||
*/
|
||||
export function convertHkValue(
|
||||
value: number,
|
||||
fromUnit: string | null | undefined,
|
||||
toUnit: string,
|
||||
): number | null {
|
||||
if (!Number.isFinite(value)) return null;
|
||||
const from = lookup(fromUnit);
|
||||
if (!from) return null;
|
||||
const to = lookup(toUnit);
|
||||
if (!to) return null;
|
||||
if (from.family !== to.family) return null;
|
||||
// Same unit: return the reading untouched rather than round-tripping it
|
||||
// through the base and picking up a float error.
|
||||
if (normaliseUnitString(String(fromUnit)) === normaliseUnitString(toUnit)) {
|
||||
return value;
|
||||
}
|
||||
return to.fromBase(from.toBase(value));
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a distance a HealthKit archive reports in its own unit into
|
||||
* metres, the canonical DB unit. Falls back to the raw number for a unit
|
||||
* this module cannot place (including the empty attribute an archive may
|
||||
* write) — the pre-#944 behaviour of the workout path, preserved.
|
||||
*/
|
||||
export function hkDistanceToMetres(
|
||||
value: number,
|
||||
unit: string | null | undefined,
|
||||
): number {
|
||||
return convertHkValue(value, unit, "m") ?? value;
|
||||
}
|
||||
@@ -42,6 +42,10 @@ import {
|
||||
} from "@/lib/measurements/drain-per-sample-cumulative";
|
||||
import { reconcileExternalMeasurement } from "@/lib/measurements/reconcile-external-measurement";
|
||||
import { resolveHkWorkoutSportType } from "@/lib/measurements/hk-workout-activity-type-map";
|
||||
import {
|
||||
convertHkValue,
|
||||
hkDistanceToMetres,
|
||||
} from "@/lib/measurements/hk-units";
|
||||
import { emitInsertedMeasurementArrivals } from "@/lib/arrivals/measurement-emit";
|
||||
import { maybeEnqueueMorningRefresh } from "@/lib/daily/morning-refresh-trigger";
|
||||
import { emitDataArrival } from "@/lib/arrivals/emit-shared";
|
||||
@@ -851,14 +855,21 @@ export async function streamParseExportXml(
|
||||
return;
|
||||
}
|
||||
|
||||
const mapped = mapAppleHealthEntry({
|
||||
hkIdentifier: hkType,
|
||||
value: parsedValue.value,
|
||||
unit: attrs.unit ?? mapping.hkUnit,
|
||||
startDate: attrs.startDate,
|
||||
endDate: attrs.endDate,
|
||||
sleepStage: parsedValue.sleepStage,
|
||||
});
|
||||
const mapped = mapAppleHealthEntry(
|
||||
{
|
||||
hkIdentifier: hkType,
|
||||
value: parsedValue.value,
|
||||
unit: attrs.unit ?? mapping.hkUnit,
|
||||
startDate: attrs.startDate,
|
||||
endDate: attrs.endDate,
|
||||
sleepStage: parsedValue.sleepStage,
|
||||
},
|
||||
// issue #944 — the archive is the one caller whose unit attribute
|
||||
// is authoritative: Apple writes the account's own display unit on
|
||||
// every quantity `<Record>`, so `km` here means kilometres and the
|
||||
// reading has to be converted before it is stored.
|
||||
{ convertRecordUnit: true },
|
||||
);
|
||||
if (!mapped) {
|
||||
unknown[`${hkType}::map_failed`] =
|
||||
(unknown[`${hkType}::map_failed`] ?? 0) + 1;
|
||||
@@ -947,14 +958,13 @@ export async function streamParseExportXml(
|
||||
const totalDistance = Number.parseFloat(attrs.totalDistance ?? "");
|
||||
const totalEnergy = Number.parseFloat(attrs.totalEnergyBurned ?? "");
|
||||
const distanceUnit = attrs.totalDistanceUnit ?? "";
|
||||
// Apple ships km for HKWorkout.totalDistance by default; metres
|
||||
// is the canonical DB unit. Convert when the unit is km.
|
||||
const energyUnit = attrs.totalEnergyBurnedUnit ?? "";
|
||||
// Apple ships the ACCOUNT's own length unit on HKWorkout.totalDistance
|
||||
// (km for a metric account, mi for an imperial one); metres is the
|
||||
// canonical DB unit. issue #944 — the same conversion the `<Record>`
|
||||
// path now runs, so the two can never drift apart again.
|
||||
const distanceM = Number.isFinite(totalDistance)
|
||||
? distanceUnit === "km"
|
||||
? totalDistance * 1000
|
||||
: distanceUnit === "mi"
|
||||
? totalDistance * 1609.344
|
||||
: totalDistance
|
||||
? hkDistanceToMetres(totalDistance, distanceUnit)
|
||||
: null;
|
||||
|
||||
const externalId = hashSampleKey(
|
||||
@@ -970,7 +980,13 @@ export async function streamParseExportXml(
|
||||
startedAt: startDate,
|
||||
endedAt: endDate,
|
||||
durationSec,
|
||||
totalEnergyKcal: Number.isFinite(totalEnergy) ? totalEnergy : null,
|
||||
totalEnergyKcal: Number.isFinite(totalEnergy)
|
||||
? // issue #944 — the energy attribute carries the account's own
|
||||
// unit too: a Health app set to kilojoules writes `kJ` here, and
|
||||
// the column is kilocalories. Same shared conversion; an unknown
|
||||
// unit leaves the number alone.
|
||||
(convertHkValue(totalEnergy, energyUnit, "kcal") ?? totalEnergy)
|
||||
: null,
|
||||
totalDistanceM: distanceM,
|
||||
externalId,
|
||||
externalSourceVersion: attrs.sourceVersion ?? null,
|
||||
|
||||
@@ -0,0 +1,394 @@
|
||||
/**
|
||||
* Repair for walking distances the `export.zip` import stored 1000x too
|
||||
* small (issue #944).
|
||||
*
|
||||
* THE DEFECT (fixed in code from this release): `mapAppleHealthEntry()`
|
||||
* accepted the `<Record unit="…">` attribute and never read it, so a
|
||||
* `HKQuantityTypeIdentifierDistanceWalkingRunning` record — which Apple
|
||||
* writes in the account's own display unit, `km` on a metric archive —
|
||||
* was folded into the day's total as if the number were already metres.
|
||||
* Every day of an imported history reads as a couple of metres walked.
|
||||
*
|
||||
* THE CODE FIX converts the record's own unit into the table's `hkUnit`
|
||||
* before storage, so every FUTURE import is right. This module heals what
|
||||
* is already in the database.
|
||||
*
|
||||
* ── The criterion ──────────────────────────────────────────────────
|
||||
*
|
||||
* A row is a candidate only when it carries the stamp the export importer
|
||||
* itself writes, never because its value "looks small":
|
||||
*
|
||||
* type = WALKING_RUNNING_DISTANCE
|
||||
* source = APPLE_HEALTH
|
||||
* aggregationProvenance = EXPORT_XML_SOURCE_MAX
|
||||
* externalId LIKE stats:HKQuantityTypeIdentifierDistanceWalkingRunning:%
|
||||
* deletedAt IS NULL
|
||||
*
|
||||
* `EXPORT_XML_SOURCE_MAX` is written in exactly one place — the archive
|
||||
* fold in `import-apple-health-export.ts` — and the `stats:<HK>:<day>`
|
||||
* externalId is minted by `dailyStatsExternalId()` for that same fold. A
|
||||
* row a native iOS sync has since overwritten carries
|
||||
* `HEALTHKIT_STATISTICS` and its value came from HealthKit's own
|
||||
* statistics query, so it is correct and out of scope. A row whose
|
||||
* provenance is `LEGACY_UNKNOWN` or NULL cannot be PROVEN to have come
|
||||
* from the archive path and is therefore left alone: re-importing the
|
||||
* archive on the fixed build repairs those, and is the authoritative
|
||||
* repair for any account that still has its `export.zip`.
|
||||
*
|
||||
* ── The refusal ────────────────────────────────────────────────────
|
||||
*
|
||||
* The criterion proves an archive ORIGIN. It cannot prove the archive was
|
||||
* imported on the broken build — an account re-imported since carries the
|
||||
* same stamp with the right numbers, and its rest days still pass the
|
||||
* plausibility check after another x1000. So any candidate whose repaired
|
||||
* value would leave the range refuses the whole account: nothing is
|
||||
* written, the rows are reported, and the operator decides. Re-importing
|
||||
* the archive is the authoritative repair and needs no script.
|
||||
*
|
||||
* ── Idempotency ────────────────────────────────────────────────────
|
||||
*
|
||||
* Applying the repair writes one `AuditLog` row per account inside the
|
||||
* same transaction as the update. An account that already carries that
|
||||
* row is skipped, so a second run finds nothing to do — the multiplier
|
||||
* can never be applied twice.
|
||||
*/
|
||||
import type { PrismaClient } from "@/generated/prisma/client";
|
||||
import { auditLog } from "@/lib/auth/audit";
|
||||
import { afterMeasurementMutation } from "@/lib/rollups/after-measurement-mutation";
|
||||
import { validateMeasurementRange } from "@/lib/validations/measurement";
|
||||
|
||||
import { dailyStatsExternalId } from "./apple-health-mapping";
|
||||
import { convertHkValue } from "./hk-units";
|
||||
|
||||
/** The audit action that marks an account's distances as repaired. */
|
||||
export const APPLE_HEALTH_DISTANCE_REPAIR_ACTION =
|
||||
"measurement.apple_health_distance.repaired";
|
||||
|
||||
/** Minted by the same helper the export fold uses, so the two cannot drift. */
|
||||
export const EXPORT_DISTANCE_EXTERNAL_ID_PREFIX = dailyStatsExternalId(
|
||||
"HKQuantityTypeIdentifierDistanceWalkingRunning",
|
||||
"",
|
||||
);
|
||||
|
||||
/**
|
||||
* The unit the affected archive was written in. Apple writes `km` on a
|
||||
* metric account and `mi` on an imperial one, and the stored row keeps no
|
||||
* record of which — the importer discarded the attribute, which is the
|
||||
* defect. The operator names it; `km` is the default because it is what
|
||||
* the Health app writes for every metric locale.
|
||||
*/
|
||||
export type ArchiveDistanceUnit = "km" | "mi";
|
||||
|
||||
export interface RepairCandidateRow {
|
||||
id: string;
|
||||
measuredAt: Date;
|
||||
value: number;
|
||||
}
|
||||
|
||||
export interface AccountRepairPlan {
|
||||
userId: string;
|
||||
/** Candidate rows that would be multiplied. */
|
||||
repairable: RepairCandidateRow[];
|
||||
/**
|
||||
* Candidates whose repaired value would leave the plausibility range
|
||||
* (`validateMeasurementRange`). One row here refuses the whole account:
|
||||
* it is evidence the rows are NOT the 1000x class.
|
||||
*/
|
||||
outOfRange: RepairCandidateRow[];
|
||||
/** Non-null when this account already carries the repair's audit row. */
|
||||
alreadyRepairedAt: Date | null;
|
||||
/**
|
||||
* True when a previous run wrote the rows but its rollup tail did not
|
||||
* finish — the audit row still says `rollupsRefreshed: false`, so the
|
||||
* DAY/WEEK/MONTH/YEAR buckets and the cached status insights still hold
|
||||
* the pre-repair numbers. Meaningless unless `alreadyRepairedAt` is set.
|
||||
*/
|
||||
rollupsPending: boolean;
|
||||
/** Candidate rows found, whatever the disposition above. */
|
||||
candidateCount: number;
|
||||
}
|
||||
|
||||
/** The DB surface this module needs — the script hands it the real client. */
|
||||
export type RepairClient = Pick<
|
||||
PrismaClient,
|
||||
"measurement" | "auditLog" | "$transaction"
|
||||
>;
|
||||
|
||||
const REPAIRED_TYPE = "WALKING_RUNNING_DISTANCE" as const;
|
||||
|
||||
/** metres per unit of the archive's own length unit. */
|
||||
export function repairFactorFor(unit: ArchiveDistanceUnit): number {
|
||||
const factor = convertHkValue(1, unit, "m");
|
||||
// The two archive units are length units by construction; the shared
|
||||
// table is the single source of the factor either way.
|
||||
if (factor === null) throw new Error(`no metre factor for unit "${unit}"`);
|
||||
return factor;
|
||||
}
|
||||
|
||||
/**
|
||||
* List the repair candidates per account. Read-only — this is what the
|
||||
* script's dry run prints.
|
||||
*/
|
||||
export async function planAppleHealthDistanceRepair(
|
||||
client: RepairClient,
|
||||
options: { archiveUnit: ArchiveDistanceUnit },
|
||||
): Promise<AccountRepairPlan[]> {
|
||||
const factor = repairFactorFor(options.archiveUnit);
|
||||
|
||||
const rows = await client.measurement.findMany({
|
||||
where: {
|
||||
type: REPAIRED_TYPE,
|
||||
source: "APPLE_HEALTH",
|
||||
aggregationProvenance: "EXPORT_XML_SOURCE_MAX",
|
||||
externalId: { startsWith: EXPORT_DISTANCE_EXTERNAL_ID_PREFIX },
|
||||
deletedAt: null,
|
||||
},
|
||||
select: { id: true, userId: true, value: true, measuredAt: true },
|
||||
orderBy: [{ userId: "asc" }, { measuredAt: "asc" }],
|
||||
});
|
||||
if (rows.length === 0) return [];
|
||||
|
||||
const userIds = Array.from(new Set(rows.map((row) => row.userId)));
|
||||
const repaired = await client.auditLog.findMany({
|
||||
where: {
|
||||
action: APPLE_HEALTH_DISTANCE_REPAIR_ACTION,
|
||||
userId: { in: userIds },
|
||||
},
|
||||
select: { userId: true, createdAt: true, details: true },
|
||||
orderBy: { createdAt: "asc" },
|
||||
});
|
||||
const repairedAt = new Map<string, Date>();
|
||||
const rollupsPending = new Set<string>();
|
||||
for (const entry of repaired) {
|
||||
if (!entry.userId) continue;
|
||||
repairedAt.set(entry.userId, entry.createdAt);
|
||||
if (!rollupsRefreshedIn(entry.details)) rollupsPending.add(entry.userId);
|
||||
}
|
||||
|
||||
const plans = new Map<string, AccountRepairPlan>();
|
||||
for (const row of rows) {
|
||||
let plan = plans.get(row.userId);
|
||||
if (!plan) {
|
||||
plan = {
|
||||
userId: row.userId,
|
||||
repairable: [],
|
||||
outOfRange: [],
|
||||
alreadyRepairedAt: repairedAt.get(row.userId) ?? null,
|
||||
rollupsPending: rollupsPending.has(row.userId),
|
||||
candidateCount: 0,
|
||||
};
|
||||
plans.set(row.userId, plan);
|
||||
}
|
||||
plan.candidateCount += 1;
|
||||
if (plan.alreadyRepairedAt) continue;
|
||||
const candidate: RepairCandidateRow = {
|
||||
id: row.id,
|
||||
measuredAt: row.measuredAt,
|
||||
value: row.value,
|
||||
};
|
||||
if (validateMeasurementRange(REPAIRED_TYPE, row.value * factor) !== null) {
|
||||
plan.outOfRange.push(candidate);
|
||||
} else {
|
||||
plan.repairable.push(candidate);
|
||||
}
|
||||
}
|
||||
return Array.from(plans.values());
|
||||
}
|
||||
|
||||
/**
|
||||
* The command that heals the rollup tail when it did not run.
|
||||
*
|
||||
* Named here rather than in the script so the message an operator reads is
|
||||
* the same one the module can be asked for.
|
||||
*/
|
||||
export function rollupRepairCommand(userId: string): string {
|
||||
return `pnpm dlx tsx scripts/backfill-rollups.ts --user ${userId}`;
|
||||
}
|
||||
|
||||
/** Did a repair audit row record a finished rollup tail? */
|
||||
function rollupsRefreshedIn(details: string | null): boolean {
|
||||
if (!details) return false;
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(details);
|
||||
return (
|
||||
typeof parsed === "object" &&
|
||||
parsed !== null &&
|
||||
(parsed as { rollupsRefreshed?: unknown }).rollupsRefreshed === true
|
||||
);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
export interface RepairOutcome {
|
||||
userId: string;
|
||||
updated: number;
|
||||
skipped: number;
|
||||
/**
|
||||
* Why the account was left alone, when it was. Null on a run that wrote.
|
||||
* The script prints it verbatim.
|
||||
*/
|
||||
refusedReason: string | null;
|
||||
/** The out-of-range candidates — what a refusal was decided on. */
|
||||
skippedRows: RepairCandidateRow[];
|
||||
/**
|
||||
* True when the rows were written but the rollup tail was not. The audit
|
||||
* row still reads `rollupsRefreshed: false`, so a later run can tell this
|
||||
* apart from a finished repair; the caller must surface
|
||||
* `rollupRepairCommand(userId)` and exit non-zero.
|
||||
*/
|
||||
rollupsPending: boolean;
|
||||
/** What the rollup tail threw, when it threw. */
|
||||
rollupError: unknown;
|
||||
}
|
||||
|
||||
/**
|
||||
* The refusal an out-of-range candidate earns.
|
||||
*
|
||||
* The provenance criterion proves an archive origin; it cannot prove the
|
||||
* archive was imported on the BROKEN build. An account that has since been
|
||||
* re-imported on the fixed build carries the same stamp with the right
|
||||
* numbers, and its rest days — under 200 m, phone left at home — pass the
|
||||
* plausibility check after another x1000 and would be written. So a single
|
||||
* row that would leave the range is treated as what it is: evidence the
|
||||
* account is not the 1000x class. The whole account stops.
|
||||
*/
|
||||
function outOfRangeRefusal(plan: AccountRepairPlan, factor: number): string {
|
||||
return (
|
||||
`${plan.outOfRange.length} of ${plan.candidateCount} candidate row(s) ` +
|
||||
`leave the plausible range after x${factor}, so these rows are not the ` +
|
||||
"1000x class — the account was most likely re-imported on the fixed " +
|
||||
"build, or its archive was never in the affected unit. Nothing written. " +
|
||||
"Check the dry run's worked examples against the account's Health app " +
|
||||
"before forcing anything."
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply one account's plan: every repairable row multiplied by the
|
||||
* archive unit's metre factor in ONE transaction, together with the audit
|
||||
* row that makes a second run a no-op. Returns what it did.
|
||||
*/
|
||||
export async function applyAppleHealthDistanceRepair(
|
||||
client: RepairClient,
|
||||
plan: AccountRepairPlan,
|
||||
options: { archiveUnit: ArchiveDistanceUnit },
|
||||
): Promise<RepairOutcome> {
|
||||
const factor = repairFactorFor(options.archiveUnit);
|
||||
const skipped = plan.outOfRange.length;
|
||||
const base = {
|
||||
userId: plan.userId,
|
||||
updated: 0,
|
||||
skipped,
|
||||
skippedRows: plan.outOfRange,
|
||||
rollupsPending: false,
|
||||
rollupError: null,
|
||||
};
|
||||
if (plan.alreadyRepairedAt) {
|
||||
return {
|
||||
...base,
|
||||
refusedReason: `already repaired on ${plan.alreadyRepairedAt.toISOString()}`,
|
||||
};
|
||||
}
|
||||
// Refuse the ACCOUNT, not just the row: see `outOfRangeRefusal`.
|
||||
if (plan.outOfRange.length > 0) {
|
||||
return { ...base, refusedReason: outOfRangeRefusal(plan, factor) };
|
||||
}
|
||||
if (plan.repairable.length === 0) {
|
||||
return { ...base, refusedReason: "no repairable rows" };
|
||||
}
|
||||
|
||||
const ids = plan.repairable.map((row) => row.id);
|
||||
const details = {
|
||||
type: REPAIRED_TYPE,
|
||||
archiveUnit: options.archiveUnit,
|
||||
factor,
|
||||
rows: ids.length,
|
||||
skippedOutOfRange: skipped,
|
||||
issue: 944,
|
||||
// Flipped to true only once the rollup tail below has finished. A row
|
||||
// left at false says "rows repaired, rollups still pre-repair", which is
|
||||
// what a re-run has to be able to see: the account is skipped from here
|
||||
// on, so nothing else would ever notice.
|
||||
rollupsRefreshed: false,
|
||||
};
|
||||
const auditId = await client.$transaction(async (tx): Promise<string> => {
|
||||
// One statement per chunk rather than one per row: an account with a
|
||||
// decade of history carries thousands of rows and an interactive
|
||||
// transaction has a clock on it. `sync_version` + `updated_at` are
|
||||
// bumped by hand because raw SQL bypasses Prisma's `@updatedAt`, and
|
||||
// the iOS delta feed pages on exactly those two columns.
|
||||
for (let i = 0; i < ids.length; i += 500) {
|
||||
const chunk = ids.slice(i, i + 500);
|
||||
await tx.$executeRaw`
|
||||
UPDATE measurements
|
||||
SET value = value * ${factor},
|
||||
unit = 'm',
|
||||
sync_version = sync_version + 1,
|
||||
updated_at = NOW()
|
||||
WHERE id = ANY(${chunk})`;
|
||||
}
|
||||
await auditLog(APPLE_HEALTH_DISTANCE_REPAIR_ACTION, {
|
||||
client: tx,
|
||||
userId: plan.userId,
|
||||
actorUserId: null,
|
||||
details,
|
||||
});
|
||||
// Read the row back rather than widening `auditLog`'s return type for
|
||||
// ~600 call sites. Inside the same transaction, and an account can hold
|
||||
// at most one of these rows — the second run is refused above — so the
|
||||
// lookup is exact, not a guess at the newest.
|
||||
const entry = await tx.auditLog.findFirstOrThrow({
|
||||
where: {
|
||||
action: APPLE_HEALTH_DISTANCE_REPAIR_ACTION,
|
||||
userId: plan.userId,
|
||||
},
|
||||
select: { id: true },
|
||||
});
|
||||
return entry.id;
|
||||
});
|
||||
|
||||
// The tail: the DAY/WEEK/MONTH/YEAR rollups and the cached status insights
|
||||
// hold the pre-repair numbers until they are recomputed. It runs outside
|
||||
// the transaction, so a throw here leaves the rows repaired and the derived
|
||||
// tier stale — and the account is skipped from the next run on. Hence the
|
||||
// flag: it is flipped only after the tail returned, and the caller is told
|
||||
// to run the backfill and to exit non-zero.
|
||||
let rollupsPending = false;
|
||||
let rollupError: unknown = null;
|
||||
try {
|
||||
await afterMeasurementMutation(
|
||||
plan.userId,
|
||||
plan.repairable.map((row) => ({
|
||||
type: REPAIRED_TYPE,
|
||||
measuredAt: row.measuredAt,
|
||||
})),
|
||||
"repair-apple-health-distance",
|
||||
);
|
||||
await client.auditLog.update({
|
||||
where: { id: auditId },
|
||||
data: { details: JSON.stringify({ ...details, rollupsRefreshed: true }) },
|
||||
});
|
||||
} catch (err) {
|
||||
// Either leg failing leaves the flag at false, which is the safe
|
||||
// direction: an operator re-running the backfill needlessly costs time,
|
||||
// one who never learns it is due keeps wrong numbers on every chart.
|
||||
rollupsPending = true;
|
||||
rollupError = err;
|
||||
}
|
||||
|
||||
return {
|
||||
userId: plan.userId,
|
||||
updated: ids.length,
|
||||
skipped,
|
||||
refusedReason: null,
|
||||
skippedRows: plan.outOfRange,
|
||||
rollupsPending,
|
||||
rollupError,
|
||||
};
|
||||
}
|
||||
|
||||
/** Narrowing helper for the script's `--unit` flag. */
|
||||
export function parseArchiveUnit(raw: string): ArchiveDistanceUnit | null {
|
||||
return raw === "km" || raw === "mi" ? raw : null;
|
||||
}
|
||||
@@ -53,7 +53,13 @@ const batchEntrySchema = z
|
||||
.describe(
|
||||
"v1.19.2 (iOS #34 extension) — the bucket's MAXIMUM bpm for a 10-minute heart-rate bucket (see `externalId`). Persisted ONLY on a well-formed `stats:HKQuantityTypeIdentifierHeartRate:<bucket-start>` row; ignored (stored null) on every other entry. Omit on a pre-v1.19.2 client — the bucket keeps the avg-only contract.",
|
||||
),
|
||||
unit: z.string().min(1).max(60),
|
||||
unit: z
|
||||
.string()
|
||||
.min(1)
|
||||
.max(60)
|
||||
.describe(
|
||||
"Apple's `HKUnit` string for `value`, captured for audit. The server does NOT convert from it: `value` is read in the unit HealthLog stores the identifier in (`m` for walking distance, `kg` for body mass, `kcal` for energy, `degC` for temperature). Send the sample in that unit.",
|
||||
),
|
||||
startDate: z.iso.datetime({ offset: true }),
|
||||
endDate: z.iso.datetime({ offset: true }),
|
||||
sleepStage: z
|
||||
|
||||
@@ -0,0 +1,317 @@
|
||||
/**
|
||||
* Integration coverage for the export-import walking-distance repair
|
||||
* (issue #944).
|
||||
*
|
||||
* The defect stored every archive-imported `WALKING_RUNNING_DISTANCE` row a
|
||||
* thousand times too small. `src/lib/measurements/repair-apple-health-distance.ts`
|
||||
* heals what is already in the database, and everything that matters about it
|
||||
* is a database property: which rows the criterion actually selects, that the
|
||||
* multiply lands in one transaction with its audit row, and that a second run
|
||||
* finds nothing. None of that can be proven against a mock, so this runs
|
||||
* against the testcontainer Postgres (Docker / OrbStack; `pnpm test:integration`).
|
||||
*/
|
||||
import { beforeEach, describe, expect, it } from "vitest";
|
||||
|
||||
import { getPrismaClient, truncateAllTables } from "./setup";
|
||||
import {
|
||||
APPLE_HEALTH_DISTANCE_REPAIR_ACTION,
|
||||
applyAppleHealthDistanceRepair,
|
||||
planAppleHealthDistanceRepair,
|
||||
} from "@/lib/measurements/repair-apple-health-distance";
|
||||
import { settleBackgroundTasks } from "@/lib/logging/background-tasks";
|
||||
import type {
|
||||
MeasurementAggregationProvenance,
|
||||
MeasurementSource,
|
||||
MeasurementType,
|
||||
PrismaClient,
|
||||
} from "@/generated/prisma/client";
|
||||
|
||||
const DISTANCE_PREFIX = "stats:HKQuantityTypeIdentifierDistanceWalkingRunning:";
|
||||
|
||||
let dayCounter = 0;
|
||||
|
||||
async function seedRow(
|
||||
prisma: PrismaClient,
|
||||
userId: string,
|
||||
overrides: {
|
||||
value: number;
|
||||
type?: MeasurementType;
|
||||
source?: MeasurementSource;
|
||||
provenance?: MeasurementAggregationProvenance | null;
|
||||
externalId?: string;
|
||||
},
|
||||
): Promise<string> {
|
||||
dayCounter += 1;
|
||||
const measuredAt = new Date(Date.UTC(2026, 0, dayCounter, 12, 0, 0));
|
||||
const day = measuredAt.toISOString().slice(0, 10);
|
||||
const row = await prisma.measurement.create({
|
||||
data: {
|
||||
userId,
|
||||
type: overrides.type ?? "WALKING_RUNNING_DISTANCE",
|
||||
value: overrides.value,
|
||||
unit: "m",
|
||||
source: overrides.source ?? "APPLE_HEALTH",
|
||||
measuredAt,
|
||||
externalId: overrides.externalId ?? `${DISTANCE_PREFIX}${day}`,
|
||||
aggregationProvenance:
|
||||
overrides.provenance === undefined
|
||||
? "EXPORT_XML_SOURCE_MAX"
|
||||
: overrides.provenance,
|
||||
},
|
||||
select: { id: true },
|
||||
});
|
||||
return row.id;
|
||||
}
|
||||
|
||||
async function readRow(prisma: PrismaClient, id: string) {
|
||||
return prisma.measurement.findUniqueOrThrow({
|
||||
where: { id },
|
||||
select: { value: true, unit: true, syncVersion: true },
|
||||
});
|
||||
}
|
||||
|
||||
async function makeUser(prisma: PrismaClient, tag: string): Promise<string> {
|
||||
const user = await prisma.user.create({
|
||||
data: {
|
||||
username: `distance-repair-${tag}`,
|
||||
email: `distance-repair-${tag}@example.test`,
|
||||
role: "USER",
|
||||
},
|
||||
});
|
||||
return user.id;
|
||||
}
|
||||
|
||||
describe("Apple Health export distance repair — integration", () => {
|
||||
beforeEach(async () => {
|
||||
await truncateAllTables(getPrismaClient());
|
||||
dayCounter = 0;
|
||||
});
|
||||
|
||||
it("selects only rows the archive importer stamped, and repairs them once", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
const userId = await makeUser(prisma, "km");
|
||||
|
||||
const kmDay = await seedRow(prisma, userId, { value: 2.484 });
|
||||
const otherDay = await seedRow(prisma, userId, { value: 7.02 });
|
||||
// Decoys, one per arm of the criterion.
|
||||
const nativeStats = await seedRow(prisma, userId, {
|
||||
value: 8123,
|
||||
provenance: "HEALTHKIT_STATISTICS",
|
||||
});
|
||||
const legacyProvenance = await seedRow(prisma, userId, {
|
||||
value: 3.4,
|
||||
provenance: null,
|
||||
});
|
||||
const manualRow = await seedRow(prisma, userId, {
|
||||
value: 4.1,
|
||||
source: "MANUAL",
|
||||
});
|
||||
const otherType = await seedRow(prisma, userId, {
|
||||
value: 9000,
|
||||
type: "ACTIVITY_STEPS",
|
||||
externalId: "stats:HKQuantityTypeIdentifierStepCount:2026-01-09",
|
||||
});
|
||||
const perSample = await seedRow(prisma, userId, {
|
||||
value: 1.2,
|
||||
externalId: "sample:abc123",
|
||||
});
|
||||
|
||||
const plans = await planAppleHealthDistanceRepair(prisma, {
|
||||
archiveUnit: "km",
|
||||
});
|
||||
expect(plans).toHaveLength(1);
|
||||
expect(plans[0].userId).toBe(userId);
|
||||
expect(plans[0].repairable.map((row) => row.id).sort()).toEqual(
|
||||
[kmDay, otherDay].sort(),
|
||||
);
|
||||
expect(plans[0].alreadyRepairedAt).toBeNull();
|
||||
|
||||
const outcome = await applyAppleHealthDistanceRepair(prisma, plans[0], {
|
||||
archiveUnit: "km",
|
||||
});
|
||||
await settleBackgroundTasks();
|
||||
expect(outcome).toMatchObject({ userId, updated: 2, skipped: 0 });
|
||||
|
||||
expect((await readRow(prisma, kmDay)).value).toBeCloseTo(2484, 6);
|
||||
expect((await readRow(prisma, otherDay)).value).toBeCloseTo(7020, 6);
|
||||
// The delta feed pages on sync_version / updated_at, so a repaired row
|
||||
// has to look mutated to a paired client.
|
||||
expect((await readRow(prisma, kmDay)).syncVersion).toBe(2);
|
||||
expect((await readRow(prisma, kmDay)).unit).toBe("m");
|
||||
|
||||
// Every decoy untouched.
|
||||
expect((await readRow(prisma, nativeStats)).value).toBe(8123);
|
||||
expect((await readRow(prisma, legacyProvenance)).value).toBe(3.4);
|
||||
expect((await readRow(prisma, manualRow)).value).toBe(4.1);
|
||||
expect((await readRow(prisma, otherType)).value).toBe(9000);
|
||||
expect((await readRow(prisma, perSample)).value).toBe(1.2);
|
||||
|
||||
const audit = await prisma.auditLog.findMany({
|
||||
where: { action: APPLE_HEALTH_DISTANCE_REPAIR_ACTION, userId },
|
||||
});
|
||||
expect(audit).toHaveLength(1);
|
||||
expect(JSON.parse(audit[0].details ?? "{}")).toMatchObject({
|
||||
archiveUnit: "km",
|
||||
factor: 1000,
|
||||
rows: 2,
|
||||
issue: 944,
|
||||
// Flipped only after the rollup tail returned. A run that repaired the
|
||||
// rows and lost the tail leaves this false, and the account is skipped
|
||||
// from here on — so it is the only place the gap can still be seen.
|
||||
rollupsRefreshed: true,
|
||||
});
|
||||
expect(outcome.rollupsPending).toBe(false);
|
||||
|
||||
const second = await planAppleHealthDistanceRepair(prisma, {
|
||||
archiveUnit: "km",
|
||||
});
|
||||
expect(second[0].rollupsPending).toBe(false);
|
||||
});
|
||||
|
||||
it("is idempotent — a second run finds nothing to repair", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
const userId = await makeUser(prisma, "twice");
|
||||
const rowId = await seedRow(prisma, userId, { value: 2.484 });
|
||||
|
||||
const first = await planAppleHealthDistanceRepair(prisma, {
|
||||
archiveUnit: "km",
|
||||
});
|
||||
await applyAppleHealthDistanceRepair(prisma, first[0], {
|
||||
archiveUnit: "km",
|
||||
});
|
||||
await settleBackgroundTasks();
|
||||
|
||||
const second = await planAppleHealthDistanceRepair(prisma, {
|
||||
archiveUnit: "km",
|
||||
});
|
||||
expect(second).toHaveLength(1);
|
||||
expect(second[0].repairable).toHaveLength(0);
|
||||
expect(second[0].candidateCount).toBe(1);
|
||||
expect(second[0].alreadyRepairedAt).toBeInstanceOf(Date);
|
||||
|
||||
const outcome = await applyAppleHealthDistanceRepair(prisma, second[0], {
|
||||
archiveUnit: "km",
|
||||
});
|
||||
expect(outcome.updated).toBe(0);
|
||||
expect((await readRow(prisma, rowId)).value).toBeCloseTo(2484, 6);
|
||||
expect((await readRow(prisma, rowId)).syncVersion).toBe(2);
|
||||
});
|
||||
|
||||
it("uses the archive's own unit", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
const userId = await makeUser(prisma, "mi");
|
||||
const mile = await seedRow(prisma, userId, { value: 1.543 });
|
||||
|
||||
const plans = await planAppleHealthDistanceRepair(prisma, {
|
||||
archiveUnit: "mi",
|
||||
});
|
||||
expect(plans[0].repairable.map((row) => row.id)).toEqual([mile]);
|
||||
|
||||
const outcome = await applyAppleHealthDistanceRepair(prisma, plans[0], {
|
||||
archiveUnit: "mi",
|
||||
});
|
||||
await settleBackgroundTasks();
|
||||
expect(outcome).toMatchObject({ updated: 1, skipped: 0 });
|
||||
expect((await readRow(prisma, mile)).value).toBeCloseTo(2483.217792, 6);
|
||||
});
|
||||
|
||||
it("refuses the whole account when one row would leave the range", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
const userId = await makeUser(prisma, "mixed");
|
||||
const plausible = await seedRow(prisma, userId, { value: 1.543 });
|
||||
const absurd = await seedRow(prisma, userId, { value: 250 });
|
||||
|
||||
const plans = await planAppleHealthDistanceRepair(prisma, {
|
||||
archiveUnit: "mi",
|
||||
});
|
||||
expect(plans[0].repairable.map((row) => row.id)).toEqual([plausible]);
|
||||
expect(plans[0].outOfRange.map((row) => row.id)).toEqual([absurd]);
|
||||
|
||||
const outcome = await applyAppleHealthDistanceRepair(prisma, plans[0], {
|
||||
archiveUnit: "mi",
|
||||
});
|
||||
await settleBackgroundTasks();
|
||||
expect(outcome.updated).toBe(0);
|
||||
expect(outcome.skipped).toBe(1);
|
||||
expect(outcome.skippedRows.map((row) => row.id)).toEqual([absurd]);
|
||||
expect(outcome.refusedReason).toMatch(/leave the plausible range/);
|
||||
|
||||
// Nothing written at all — not the "repairable" row, not the audit row
|
||||
// that would make a later, correct run a no-op.
|
||||
expect((await readRow(prisma, plausible)).value).toBe(1.543);
|
||||
expect((await readRow(prisma, plausible)).syncVersion).toBe(1);
|
||||
expect((await readRow(prisma, absurd)).value).toBe(250);
|
||||
expect(
|
||||
await prisma.auditLog.count({
|
||||
where: { action: APPLE_HEALTH_DISTANCE_REPAIR_ACTION, userId },
|
||||
}),
|
||||
).toBe(0);
|
||||
});
|
||||
|
||||
it("leaves an account re-imported on the fixed build untouched", async () => {
|
||||
// The scenario the criterion cannot see: the operator re-imported
|
||||
// `export.zip` on the fixed build, so the rows are already right. They
|
||||
// still carry `EXPORT_XML_SOURCE_MAX` + the `stats:` external id and
|
||||
// still have no audit row, so they are selected exactly like a broken
|
||||
// account's rows. The rest day survives another x1000; the ordinary days
|
||||
// do not, and that is what has to stop the account.
|
||||
const prisma = getPrismaClient();
|
||||
const userId = await makeUser(prisma, "reimported");
|
||||
const ordinary = await seedRow(prisma, userId, { value: 2484 });
|
||||
const longWalk = await seedRow(prisma, userId, { value: 7020 });
|
||||
const restDay = await seedRow(prisma, userId, { value: 150 });
|
||||
|
||||
const plans = await planAppleHealthDistanceRepair(prisma, {
|
||||
archiveUnit: "km",
|
||||
});
|
||||
expect(plans).toHaveLength(1);
|
||||
// The rest day would pass the plausibility check on its own.
|
||||
expect(plans[0].repairable.map((row) => row.id)).toEqual([restDay]);
|
||||
expect(plans[0].outOfRange.map((row) => row.id).sort()).toEqual(
|
||||
[ordinary, longWalk].sort(),
|
||||
);
|
||||
|
||||
const outcome = await applyAppleHealthDistanceRepair(prisma, plans[0], {
|
||||
archiveUnit: "km",
|
||||
});
|
||||
await settleBackgroundTasks();
|
||||
expect(outcome.updated).toBe(0);
|
||||
expect(outcome.refusedReason).not.toBeNull();
|
||||
|
||||
for (const [id, value] of [
|
||||
[ordinary, 2484],
|
||||
[longWalk, 7020],
|
||||
[restDay, 150],
|
||||
] as const) {
|
||||
const row = await readRow(prisma, id);
|
||||
expect(row.value).toBe(value);
|
||||
expect(row.syncVersion).toBe(1);
|
||||
}
|
||||
expect(
|
||||
await prisma.auditLog.count({
|
||||
where: { action: APPLE_HEALTH_DISTANCE_REPAIR_ACTION, userId },
|
||||
}),
|
||||
).toBe(0);
|
||||
});
|
||||
|
||||
it("plans one entry per account", async () => {
|
||||
const prisma = getPrismaClient();
|
||||
const first = await makeUser(prisma, "a");
|
||||
const second = await makeUser(prisma, "b");
|
||||
await seedRow(prisma, first, { value: 1.5 });
|
||||
await seedRow(prisma, second, { value: 2.5 });
|
||||
await seedRow(prisma, second, { value: 3.5 });
|
||||
|
||||
const plans = await planAppleHealthDistanceRepair(prisma, {
|
||||
archiveUnit: "km",
|
||||
});
|
||||
expect(
|
||||
plans.map((plan) => [plan.userId, plan.repairable.length]).sort(),
|
||||
).toEqual(
|
||||
[
|
||||
[first, 1],
|
||||
[second, 2],
|
||||
].sort(),
|
||||
);
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user