mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-10 13:22:14 +00:00
fix(google-health): bound the backfill by progress and a horizon, not pages
A resumable collection counted the pages it had written across every backfill attempt and stopped at its ceiling (10,000 for heart rate). A watch that records heart rate every few seconds reaches that count two to three years back, and since it never reset, every later slice stopped at the same place, failed as stuck, and the backfill was never stamped complete. A walk is now stopped only when it stops moving: 20 pages in a row that reach no further back than the pages before them end it. A history backfill reads back at most GOOGLE_HEALTH_BACKFILL_HORIZON_YEARS (20 when unset). A data type that ends either way counts as finished, and where and why it ended is kept in backfillProgress past the completion stamp. An oldest-first list is read whole, as before. The slice chain's ceiling is derived from sixty days of back-to-back slices instead of the old page count. An account left at the old ceiling continues from its edge with the next slice. Refs #1220
This commit is contained in:
1 parent
01bf8fbb64
commit
72c259d85c
13 files changed
+1351
-108
No files matched your search
@@ -126,8 +126,11 @@ Only one sync runs per account at a time. **Sync now** reports that a sync is
|
||||
already in progress while the history backfill that follows a connect is
|
||||
running, and shows its progress instead of starting a second one.
|
||||
|
||||
The backfill reads the account's whole history, and for a watch that records
|
||||
heart rate every minute that is a thousand-odd pages per year. It runs in
|
||||
The backfill reads the account's whole history. Heart rate is by far the
|
||||
largest part: a page holds up to 1,000 readings, so a watch that records once
|
||||
a minute fills some 525 pages a year, and one that records every few seconds
|
||||
(as many Pixel and Fitbit watches do during the day) several thousand; one
|
||||
account measured about 4,100 pages a year. It runs in
|
||||
slices: a slice reads for up to five minutes or 500 pages, writes what it
|
||||
read, notes how far it got and lets go of the account. The next slice starts
|
||||
a minute later where the last one stopped. Two kinds of note are kept: a data
|
||||
@@ -171,6 +174,35 @@ three times before they count as a failure. None of these mark the connection
|
||||
as needing a reconnect. Access tokens last an hour, and a long backfill renews
|
||||
its token as it goes.
|
||||
|
||||
### How far back the backfill reads
|
||||
|
||||
The backfill is not bounded by a number of pages, because how many pages a
|
||||
year of history fills depends on the device. Two bounds apply instead:
|
||||
|
||||
- **The horizon.** It reads back at most 20 years. Set
|
||||
`GOOGLE_HEALTH_BACKFILL_HORIZON_YEARS` to a whole number from 1 to 100 to
|
||||
change that (it is on the compose `environment:` list; any other value
|
||||
falls back to 20). A data type whose history goes back further is imported
|
||||
up to the horizon and ends there.
|
||||
- **Progress.** A walk has to reach older readings as it goes. If Google
|
||||
hands out 20 pages in a row that reach no further back than the ones before
|
||||
them (the same page again, or a page token that goes round in a circle),
|
||||
the walk stops, and that data type's history ends at the oldest reading
|
||||
written so far. Retrying would only meet the same wall, so it is not
|
||||
retried; a reconnect starts the backfill over.
|
||||
|
||||
Either way the data type counts as finished, the backfill completes once
|
||||
every other type is in, and the integration card stops saying that older
|
||||
history is loading. Instead it says, per data type, how far back the import
|
||||
reaches and why it ends there.
|
||||
|
||||
Before 1.43.6 every page-by-page data type had a ceiling of pages counted
|
||||
across all slices (10,000 for heart rate, 1,000 for the others). A dense
|
||||
heart-rate history reaches 10,000 pages two to three years back, the count
|
||||
never reset, and from then on every slice stopped at the same place without
|
||||
the backfill ever completing. A backfill left in that state continues from
|
||||
where it stopped with the first slice after the update.
|
||||
|
||||
A slice that moved nothing forward (no data type finished, no reading older
|
||||
than before) is a real failure: the job fails with the reason, pg-boss retries
|
||||
it a few times, and after that the next restart offers the backfill again.
|
||||
@@ -215,6 +247,13 @@ reading:
|
||||
`edgeFrom` and `edgeTo`, the oldest reading written before and after the
|
||||
slice.
|
||||
- `edges`: where every unfinished page-by-page data type stands now.
|
||||
- `ended`: data types whose history ended short of its beginning, each
|
||||
with its measurement `type`, the instant it ends at (`at`) and the
|
||||
`reason` (`horizon` or `no_progress`).
|
||||
- `meta["googleHealth.backfill.ended"]` appears on the slice where a data
|
||||
type ended short, with the same entries. A walk stopped for lack of
|
||||
progress also leaves a warning:
|
||||
`fetchHeartRate: 20 pages in a row reached no further back than <instant>`.
|
||||
- `meta["googleHealth.backfill.edgeFilterUnsupported"]` appears on the slice
|
||||
where Google refused the upper-bound filter, listing the data types noted
|
||||
as refusing it.
|
||||
|
||||
Reference in new issue
Block a user