The retry is fetched about a second and a half after the overrun ends, later
on a loaded machine, and the twelve-second window turned that lateness into a
red run. Both cases now wait for the state they assert, bounded by a
deadline, and read the call count once the retry's own run has finished.
Up to 12.37 pg-boss retried a job whose handler outlived its expiry in
the same worker beside the run it had given up on, and the lane test
pinned that as an executable statement. 12.38 fails the job and fires
the signal as before, but the worker fetches nothing more until the
handler returns, and the retry starts after it. The test now pins that:
one run at a time, the retry starting after the overrun ends, and the
lane's handler importing once per delivery with never two at once. The
lane's lock stays, as the guard for a second worker or instance.
The shared admission queue sent its jobs without an expiry, so every one
took pg-boss's fifteen minutes. Against pg-boss 12.34, a handler that runs
past its expiry is not stopped: the job is failed, its abort signal fires,
the worker and group slots are freed, and the retry of the same job starts
in the same worker beside the run it replaced. A per-minute heart-rate
history is thousands of pages, so on a slow link one import turned into up
to four copies of itself, on the one queue meant to run a single import at
a time.
- The admission job now carries a sixteen-hour expiry, sized for the
longest legitimate import and under pg-boss's 24-hour ceiling.
- Each run holds a session advisory lock on its identity (kind, provider,
account) on a connection of its own. A delivery that finds it taken does
nothing and says so in the log; the running import stamps its marker or
is offered again at the next boot. Ending the connection or the process
releases the lock.
- The Google Health backfill runs under the job's budget and abort signal.
The page walk asks before each request, stops there, and reports the run
truncated, so it throws without the completion stamp and is retried.
- The drain moves out of the registrar into its own module so it can be
driven against a real pg-boss.
The integration test pins the pg-boss behaviour and runs the lane's
handler with a two-second expiry: without the lock the import ran twice.
Refs #1023