mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-17 23:52:24 +00:00
fix(google-health): resume a backfill inside a collection and run it in short slices
A backfill noted progress only once a whole collection was read to the end. A heart-rate history of years that one attempt could not finish (its budget, a rate limit it could not wait out, a single failed request among thousands of pages) started the next attempt from the newest page again, finished nothing, and failed with "marker not stamped" on every attempt. - Every spot-reading collection read page by page now records, after each page it writes, the oldest sample it has written (backfillProgress.cursors). The next attempt asks Google only for samples at or beyond that edge. The edge is held back once a page's write fails, so no rows are skipped. - The backfill runs in slices of five minutes or 500 pages. A slice that finished a collection or moved an edge continues with the next one; only a slice that moved nothing fails, and its error names each unfinished collection and why the slice stopped. The slice ceiling is derived from the collections and the page ceilings of the resumable ones. - A rate limit Google will not lift within a minute ends the slice instead of failing the walk, and the next slice starts five minutes later. Server errors, timeouts and dropped connections are retried three times. - The heart-rate stream is read last, so the light collections finish in the first slice. - The hourly sync waits up to eight minutes for a running slice instead of skipping the account for as long as the import runs. - Each slice annotates googleHealth.backfill.slice with pages, elapsed time, stop reason, per-collection status and edges, and reports progress on the job.progress line.
This commit is contained in:
1 parent
82d8960044
commit
2d03eba587
16 files changed
+1399
-165
No files matched your search
@@ -122,22 +122,87 @@ The Google Cloud client id/secret are per-user (Settings), not env vars.
|
||||
|
||||
## Backfill, the hourly sync and Google's rate limit
|
||||
|
||||
Only one sync runs per account at a time. While the history backfill that
|
||||
follows a connect is running, the hourly sync leaves that account alone, and
|
||||
**Sync now** reports that a sync is already in progress and shows its progress
|
||||
instead of starting a second one.
|
||||
Only one sync runs per account at a time. **Sync now** reports that a sync is
|
||||
already in progress while the history backfill that follows a connect is
|
||||
running, and shows its progress instead of starting a second one.
|
||||
|
||||
The backfill reads the account's whole history, and for a watch that records
|
||||
heart rate every minute that is a thousand-odd pages per year. It runs in
|
||||
slices: a slice reads for up to five minutes or 500 pages, writes what it
|
||||
read, notes how far it got and lets go of the account. The next slice starts
|
||||
a minute later where the last one stopped. Two kinds of note are kept: a data
|
||||
type read to the end is not read again, and a data type read page by page
|
||||
(heart rate, weight, body fat, blood glucose, core temperature) keeps the
|
||||
time of the oldest reading it has written, so the next slice asks Google only
|
||||
for what is older. A restart or an update in the middle loses at most the page
|
||||
that was being read.
|
||||
|
||||
The hourly sync waits for the slice that is running (up to eight minutes)
|
||||
instead of skipping the account, so recent data keeps arriving while an old
|
||||
history is still coming in.
|
||||
|
||||
Google limits how many requests one account may make per minute. When a sync
|
||||
reaches that limit, Google answers `429`, and HealthLog waits and repeats the
|
||||
request; the connection is never marked as needing a reconnect because of it.
|
||||
Access tokens last an hour, and a long backfill renews its token as it goes.
|
||||
A backfill that stops part-way (a restart, the job's time budget) picks up
|
||||
with the data types it has not finished yet on its next attempt.
|
||||
reaches that limit Google answers `429`, and HealthLog waits and repeats the
|
||||
request. If Google asks for a longer wait than a minute, the slice ends there
|
||||
and the next one starts five minutes later. A server error (`500`, `502`,
|
||||
`503`, `504`), a request that times out and a dropped connection are repeated
|
||||
three times before they count as a failure. None of these mark the connection
|
||||
as needing a reconnect. Access tokens last an hour, and a long backfill renews
|
||||
its token as it goes.
|
||||
|
||||
A slice that moved nothing forward (no data type finished, no reading older
|
||||
than before) is a real failure: the job fails with the reason, pg-boss retries
|
||||
it a few times, and after that the next restart offers the backfill again.
|
||||
Everything already written stays.
|
||||
|
||||
If an older version left your Google Health card asking for a reconnect after
|
||||
a backfill, the first sync after updating checks the stored grant and resumes
|
||||
on its own when the grant is still valid.
|
||||
|
||||
### Reading the backfill in the logs
|
||||
|
||||
Every slice writes one log line when it ends, with
|
||||
`"background":{"task_name":"job.integration_backfill"}`. The parts worth
|
||||
reading:
|
||||
|
||||
- `meta.kind` is `google-health-backfill`, `meta.slice` the slice number
|
||||
(0 for the first), `meta.job_id` the pg-boss job.
|
||||
- `external_calls` lists the requests the slice made to Google, one entry per
|
||||
request: `method` names the data type and page (`fetchHeartRate(page=212)`),
|
||||
`status` the HTTP status, `duration_ms` how long it took. A request that was
|
||||
repeated carries a `note` (`rate_limited_retrying_in_ms=4000`,
|
||||
`server_error_retrying_in_ms=2000`, `no_response_retrying_in_ms=2000`,
|
||||
`token_rejected_retrying_with_fresh_token`); one that failed for good
|
||||
carries an `error`. Successful requests are listed up to 60 per line and
|
||||
everything up to 200; `external_calls_omitted` counts the rest.
|
||||
- `meta["googleHealth.backfill.slice"]` sums the slice up:
|
||||
- `pages`, `elapsedMs` and `imported`: what the slice read, how long it
|
||||
took, how many readings it wrote.
|
||||
- `stopReason`: why it ended early. `time_budget` and `page_budget` are the
|
||||
normal end of a slice, `rate_limited` means Google would not take more
|
||||
requests for now, `job_budget` that the job itself ran out of time.
|
||||
`null` means the slice read everything it started.
|
||||
- `collectionsDone` of `collectionsTotal`: data types read to the end so
|
||||
far.
|
||||
- `collections`: one entry per data type the slice touched, with its
|
||||
`status` (`done`, `stopped` short on a budget or a rate limit,
|
||||
`deferred` to the next slice, or `failed`), its `pages`, and for the
|
||||
page-by-page types
|
||||
`edgeFrom` and `edgeTo`, the oldest reading written before and after the
|
||||
slice.
|
||||
- `edges`: where every unfinished page-by-page data type stands now.
|
||||
- `action.name` is `google_health.backfill.continued` for a slice that made
|
||||
progress and queued the next one, `google_health.backfill.complete` for the
|
||||
one that finished the history, and `google_health.backfill.stuck` for one
|
||||
that moved nothing. A stuck slice also fails the job with a message that
|
||||
names each unfinished data type and its status, for example
|
||||
`no progress in slice 4, stopped on rate_limited; fetchHeartRate:stopped`.
|
||||
|
||||
While a slice runs, the usual `job.progress` line for the
|
||||
`integration-backfill-admission` queue reports
|
||||
`google_health_backfill_slice`, `google_health_backfill_pages` and
|
||||
`google_health_backfill_collections_done` every two minutes.
|
||||
|
||||
## The 7-day re-consent caveat
|
||||
|
||||
While the OAuth consent screen stays in **Testing** publishing mode, Google
|
||||
|
||||
Reference in new issue
Block a user