fix(google-health): bound the backfill by progress and a horizon, not pages

A resumable collection counted the pages it had written across every
backfill attempt and stopped at its ceiling (10,000 for heart rate). A
watch that records heart rate every few seconds reaches that count two to
three years back, and since it never reset, every later slice stopped at
the same place, failed as stuck, and the backfill was never stamped
complete.

A walk is now stopped only when it stops moving: 20 pages in a row that
reach no further back than the pages before them end it. A history
backfill reads back at most GOOGLE_HEALTH_BACKFILL_HORIZON_YEARS (20 when
unset). A data type that ends either way counts as finished, and where and
why it ended is kept in backfillProgress past the completion stamp. An
oldest-first list is read whole, as before. The slice chain's ceiling is
derived from sixty days of back-to-back slices instead of the old page
count. An account left at the old ceiling continues from its edge with
the next slice.

Refs #1220
This commit is contained in:
Marc-André Bombeck committed 2026-10-19 17:25:48 +02:00
1 parent 01bf8fbb64
commit 72c259d85c
13 files changed
+1351 -108

No files matched your search

+41 -2
View File
@@ -126,8 +126,11 @@ Only one sync runs per account at a time. **Sync now** reports that a sync is
already in progress while the history backfill that follows a connect is
running, and shows its progress instead of starting a second one.
The backfill reads the account's whole history, and for a watch that records
heart rate every minute that is a thousand-odd pages per year. It runs in
The backfill reads the account's whole history. Heart rate is by far the
largest part: a page holds up to 1,000 readings, so a watch that records once
a minute fills some 525 pages a year, and one that records every few seconds
(as many Pixel and Fitbit watches do during the day) several thousand; one
account measured about 4,100 pages a year. It runs in
slices: a slice reads for up to five minutes or 500 pages, writes what it
read, notes how far it got and lets go of the account. The next slice starts
a minute later where the last one stopped. Two kinds of note are kept: a data
@@ -171,6 +174,35 @@ three times before they count as a failure. None of these mark the connection
as needing a reconnect. Access tokens last an hour, and a long backfill renews
its token as it goes.
### How far back the backfill reads
The backfill is not bounded by a number of pages, because how many pages a
year of history fills depends on the device. Two bounds apply instead:
- **The horizon.** It reads back at most 20 years. Set
`GOOGLE_HEALTH_BACKFILL_HORIZON_YEARS` to a whole number from 1 to 100 to
change that (it is on the compose `environment:` list; any other value
falls back to 20). A data type whose history goes back further is imported
up to the horizon and ends there.
- **Progress.** A walk has to reach older readings as it goes. If Google
hands out 20 pages in a row that reach no further back than the ones before
them (the same page again, or a page token that goes round in a circle),
the walk stops, and that data type's history ends at the oldest reading
written so far. Retrying would only meet the same wall, so it is not
retried; a reconnect starts the backfill over.
Either way the data type counts as finished, the backfill completes once
every other type is in, and the integration card stops saying that older
history is loading. Instead it says, per data type, how far back the import
reaches and why it ends there.
Before 1.43.6 every page-by-page data type had a ceiling of pages counted
across all slices (10,000 for heart rate, 1,000 for the others). A dense
heart-rate history reaches 10,000 pages two to three years back, the count
never reset, and from then on every slice stopped at the same place without
the backfill ever completing. A backfill left in that state continues from
where it stopped with the first slice after the update.
A slice that moved nothing forward (no data type finished, no reading older
than before) is a real failure: the job fails with the reason, pg-boss retries
it a few times, and after that the next restart offers the backfill again.
@@ -215,6 +247,13 @@ reading:
`edgeFrom` and `edgeTo`, the oldest reading written before and after the
slice.
- `edges`: where every unfinished page-by-page data type stands now.
- `ended`: data types whose history ended short of its beginning, each
with its measurement `type`, the instant it ends at (`at`) and the
`reason` (`horizon` or `no_progress`).
- `meta["googleHealth.backfill.ended"]` appears on the slice where a data
type ended short, with the same entries. A walk stopped for lack of
progress also leaves a warning:
`fetchHeartRate: 20 pages in a row reached no further back than <instant>`.
- `meta["googleHealth.backfill.edgeFilterUnsupported"]` appears on the slice
where Google refused the upper-bound filter, listing the data types noted
as refusing it.