Commit Graph
3 Commits
Author SHA1 Message Date
Marc-André Bombeck 2556da645b test(jobs): wait for the expired import's retry instead of a fixed window
The retry is fetched about a second and a half after the overrun ends, later
on a loaded machine, and the twelve-second window turned that lateness into a
red run. Both cases now wait for the state they assert, bounded by a
deadline, and read the call count once the retry's own run has finished.
2026-10-25 03:19:54 +01:00
Marc-André Bombeck 5199df56db test(jobs): restate the backfill lane's expiry behaviour for pg-boss 12.38
Up to 12.37 pg-boss retried a job whose handler outlived its expiry in
the same worker beside the run it had given up on, and the lane test
pinned that as an executable statement. 12.38 fails the job and fires
the signal as before, but the worker fetches nothing more until the
handler returns, and the retry starts after it. The test now pins that:
one run at a time, the retry starting after the overrun ends, and the
lane's handler importing once per delivery with never two at once. The
lane's lock stays, as the guard for a second worker or instance.
2026-10-22 07:33:01 +02:00
Marc-André Bombeck 029d6d881c fix(jobs): keep a full-history import from running beside its own retry
The shared admission queue sent its jobs without an expiry, so every one
took pg-boss's fifteen minutes. Against pg-boss 12.34, a handler that runs
past its expiry is not stopped: the job is failed, its abort signal fires,
the worker and group slots are freed, and the retry of the same job starts
in the same worker beside the run it replaced. A per-minute heart-rate
history is thousands of pages, so on a slow link one import turned into up
to four copies of itself, on the one queue meant to run a single import at
a time.

- The admission job now carries a sixteen-hour expiry, sized for the
  longest legitimate import and under pg-boss's 24-hour ceiling.
- Each run holds a session advisory lock on its identity (kind, provider,
  account) on a connection of its own. A delivery that finds it taken does
  nothing and says so in the log; the running import stamps its marker or
  is offered again at the next boot. Ending the connection or the process
  releases the lock.
- The Google Health backfill runs under the job's budget and abort signal.
  The page walk asks before each request, stops there, and reports the run
  truncated, so it throws without the completion stamp and is retried.
- The drain moves out of the registrar into its own module so it can be
  driven against a real pg-boss.

The integration test pins the pg-boss behaviour and runs the lane's
handler with a two-second expiry: without the lock the import ran twice.

Refs #1023
2026-09-30 23:11:03 +02:00