fix(jobs): load every job handler once at worker boot and report failures

Since handlers load on their queue's first job, a module the image
cannot load no longer fails the worker's boot. The worker comes up
healthy, the queue burns its retries when it first has work, and a
one-shot job (a restore, an import tail) ends failed with nobody told at
deploy time.

After binding every queue the worker now runs `preloadHandlers` in the
background, without awaiting it, so readiness is unchanged. It loads
each handler once through the same memoised loader its queue is bound
to, so a module that loads stays loaded for the first job. Each failure
is reported on the `job.run.failed` line at error, with the queue's name
and `reason: "handler_load_failed"`, plus the worker error report
(stderr and GlitchTip). A summary line on stderr names every queue that
failed. The sweep never throws, and a failing report does not stop it.

docs/ops/deploy.md says what to look for after a deploy.
This commit is contained in:
Marc-André Bombeck committed 2026-10-25 00:35:36 +02:00
1 parent f7b89f96e1
commit c6bd27f06f
6 files changed
+305 -15

No files matched your search

+15
View File
@@ -237,6 +237,21 @@ Pass `--only=prod` / `--only=demo` to check a single target, or
`--attempts=N --interval=ms` to widen the retry window while a slow
pull settles.
The worker loads each job's handler module on that job's first run, so a
module the image cannot load does not stop the worker from starting. Once
it has bound its queues, the worker loads every handler a single time in
the background and reports each one that fails, normally within a minute
of start. Each failure is a `job.run.failed` event at `error` with
`reason: "handler_load_failed"` and the queue's name, and stderr also gets
`[pg-boss] [registry] preloaded N/M job handlers; failed to load: <queues>`.
If either shows up after a deploy, the image is broken for those queues,
and their jobs will fail when they run. Roll back (see Rollback above)
instead of waiting for the retries to run out:
```sh
docker compose logs --since 5m app 2>&1 | grep -E 'handler_load_failed|failed to load:'
```
## Walk the synthetic journey
The version check proves the instance serves the right build. It does not