A wide event carries one action and the last annotation wins, so a hop
that failed before the next provider answered, one failed account among
six, or a caught error the code carried on past was either a meta key on
someone else's info line or nothing at all. `emitSignal` writes one extra
line under a stable action and level, joined to the surrounding event by
request and trace id; `logCaught` / `caughtAs` are its warn-level shape
for a catch that degrades on purpose.
Every failed pg-boss run now writes `job.run.failed` at error with the
queue and how many runs of it in a row have failed, counted from pg-boss's
own rows (`readConsecutiveRunFailures`). `reportJobRun` takes a fan-out
pass's whole candidate list: an all-failed run is `job.run.failed`, a
partial one `job.run.partial` at warn, and the run reports `all_failed` in
its facts so a completed night with nothing done still counts toward the
streak. The repeated-failure alert keys on that one action and number.
Retention is per queue now. `failureReaderRetention` refuses a queue
retention under 96 hours (the 72-hour reader window plus a day), and a
guard refuses any `deleteAfterSeconds` that is neither that helper nor a
literal at or above the floor.
A refusal the request was meant to get (any error carrying a 4xx status)
is recorded at warn instead of error. Expired-token 401s were most of the
error lines and hid the real failures between them.
A job that ran into its pg-boss expiry left no trace in the app log. Most
handlers annotate a wide event nobody emits, pg-boss records an expiry only
in its own table, and a worker that dies of heap exhaustion writes one
FATAL line that names no queue. On a large account three queues failed
with "job timed out" for days and the log said nothing about them (#1031).
Every handler now runs inside `observeJob` (through `runJob`), which writes
`job.progress` every two minutes while a handler runs (elapsed time, heap,
RSS and the last progress the pass reported), `job.expired` when the job
passes its expiry while the handler is still running, and `job.finished`
when a long or expired run settles. At worker boot, `worker.boot.jobs_cut_off`
names every job a previous process was still running when it stopped.
The admin card's "job timed out" now says what the limit was and that the
cause is either a run too slow for it or a restart while it ran.
The intake import built its replay key from the row's optional `zaehler`
field. That reads as a per-row counter — the documented example showed one —
but nothing required it to be unique and nothing checked. A reported export
carried the dose quantity there instead: 1 tablet, on all 28 rows. Every entry
queued under the same key, one dose reached the ledger, and the other 27 came
back reported as duplicates of a dose they had nothing to do with.
The key is the instant now, and `zaehler` is out of the contract rather than
accepted-and-ignored. Nothing is lost by dropping it: the live-row unique on
`(user_id, medication_id, scheduled_for, source)` already makes an imported
dose unique per instant, so a re-import of a file first imported under the old
key still dedups.
The telling is the other half. Two different facts were counted as one number:
entries of a submission that collide with each other, and doses the ledger
already holds. They are separate reasons now, grouped with a count, and the
dialog renders them through the same classifier and the same outcome line as
the CSV and JSON import cards — so an import that wrote nothing can no longer
answer in success tone. That rule shipped for the settings import panel and
this surface was not part of it; the guard now lists every import surface by
path, so a fourth one has to be added deliberately. `skippedInvalid` is gone:
it was hardcoded to zero on every path and reported a capability the importer
never had.
Alongside it, the other half of the environment report — a scheduled job that
fails every run has to say so. `recordError()` increments one unnamed counter,
so the admin status page could show a number and not which queue produced it,
and the module's own screen showed a zero day count with nothing to
distinguish "nothing stored yet" from "every run failed". pg-boss has held the
answer the whole time: a job that exhausts its retries keeps its queue name,
its error and its timestamp. The admin status now names the failing queues
with counts and the last error, the environment overview reports its own
account's failed runs, and neither needed a change at any of the thirty
`recordError()` call sites — a visibility fix that has to be remembered per
call site is the same failure one level up.
Refs #650, #646
Marc-André Bombeck