mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 12:32:43 +00:00
Marc-André Bombeck
2d03eba587
A backfill noted progress only once a whole collection was read to the end. A heart-rate history of years that one attempt could not finish (its budget, a rate limit it could not wait out, a single failed request among thousands of pages) started the next attempt from the newest page again, finished nothing, and failed with "marker not stamped" on every attempt. - Every spot-reading collection read page by page now records, after each page it writes, the oldest sample it has written (backfillProgress.cursors). The next attempt asks Google only for samples at or beyond that edge. The edge is held back once a page's write fails, so no rows are skipped. - The backfill runs in slices of five minutes or 500 pages. A slice that finished a collection or moved an edge continues with the next one; only a slice that moved nothing fails, and its error names each unfinished collection and why the slice stopped. The slice ceiling is derived from the collections and the page ceilings of the resumable ones. - A rate limit Google will not lift within a minute ends the slice instead of failing the walk, and the next slice starts five minutes later. Server errors, timeouts and dropped connections are retried three times. - The heart-rate stream is read last, so the light collections finish in the first slice. - The hourly sync waits up to eight minutes for a running slice instead of skipping the account for as long as the import runs. - Each slice annotates googleHealth.backfill.slice with pages, elapsed time, stop reason, per-collection status and edges, and reports progress on the job.progress line.