Files
HealthLog/src/lib/google-health/sync-lock.ts
T
Marc-André Bombeck 2d03eba587 fix(google-health): resume a backfill inside a collection and run it in short slices
A backfill noted progress only once a whole collection was read to the end.
A heart-rate history of years that one attempt could not finish (its budget,
a rate limit it could not wait out, a single failed request among thousands
of pages) started the next attempt from the newest page again, finished
nothing, and failed with "marker not stamped" on every attempt.

- Every spot-reading collection read page by page now records, after each
  page it writes, the oldest sample it has written (backfillProgress.cursors).
  The next attempt asks Google only for samples at or beyond that edge. The
  edge is held back once a page's write fails, so no rows are skipped.
- The backfill runs in slices of five minutes or 500 pages. A slice that
  finished a collection or moved an edge continues with the next one; only a
  slice that moved nothing fails, and its error names each unfinished
  collection and why the slice stopped. The slice ceiling is derived from the
  collections and the page ceilings of the resumable ones.
- A rate limit Google will not lift within a minute ends the slice instead of
  failing the walk, and the next slice starts five minutes later. Server
  errors, timeouts and dropped connections are retried three times.
- The heart-rate stream is read last, so the light collections finish in the
  first slice.
- The hourly sync waits up to eight minutes for a running slice instead of
  skipping the account for as long as the import runs.
- Each slice annotates googleHealth.backfill.slice with pages, elapsed time,
  stop reason, per-collection status and edges, and reports progress on the
  job.progress line.
2026-10-15 11:26:49 +02:00

61 lines
2.7 KiB
TypeScript

/**
* One Google Health run per account at a time, across processes.
*
* The hourly poll, the manual trigger, the full-history backfill and the
* sleep repair all read the same account's collections with the same grant.
* Run side by side they share Google's per-user, per-minute request quota:
* a backfill deep in a dense heart-rate walk and an hourly sync starting
* beside it pushed the account over the quota, and every request either of
* them made in that minute answered 429. They also write the same rows, so
* two runs racing one night's sleep segments can each move a row the other
* is about to move.
*
* So every run for one account holds the same advisory lock for as long as
* it reads and writes (`withJobLock`, a connection of its own, released when
* the run ends or the process dies). A run that finds it taken does no work:
* the hourly poll skips the account for this tick, the manual trigger answers
* that a sync is already running, and the backfill and the repair wait a
* bounded time for a short run to finish before handing their job back to
* pg-boss to retry. The hourly poll waits too, for about one backfill slice:
* a backfill lets go of the account between its slices, so the hourly sync
* gets its turn instead of being skipped for as long as the import runs.
*/
import { withJobLock, type GuardedRun } from "@/lib/jobs/job-lock";
/** The lock key for one account's Google Health runs. */
export function googleHealthSyncLockKey(userId: string): string {
return `google-health-sync:${userId}`;
}
/**
* How long a backfill or repair waits for an hourly or manual run to finish.
* Those runs read a day of history at most and take a minute or two, so ten
* minutes covers one that is also sitting out a rate limit.
*/
export const GOOGLE_HEALTH_SYNC_LOCK_WAIT_MS = 10 * 60 * 1000;
/**
* How long the hourly poll waits for an account another run holds. A backfill
* slice walks for five minutes and then lets go
* (`GOOGLE_HEALTH_BACKFILL_SLICE_MS`); the page in flight, a rate-limit wait
* and the slice's own bookkeeping can add a minute or two, and the next slice
* starts a minute after this one ends. Eight minutes covers that, and leaves
* the hourly job well inside its fifteen-minute expiry.
*/
export const GOOGLE_HEALTH_POLL_LOCK_WAIT_MS = 8 * 60 * 1000;
/**
* Run `run` while holding the account's Google Health lock. Resolves
* `{ ran: false }` without calling `run` when another run holds it and does
* not let go within `waitMs` (zero: give up at once).
*/
export function withGoogleHealthSyncLock<T>(
userId: string,
run: () => Promise<T>,
options: { waitMs?: number } = {},
): Promise<GuardedRun<T>> {
return withJobLock(googleHealthSyncLockKey(userId), run, undefined, {
waitMs: options.waitMs ?? 0,
});
}