mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-16 07:13:29 +00:00
pg-boss gives a job fifteen minutes unless the send says otherwise, then fails it and retries it while the handler keeps running. Several passes can legitimately need longer and had no protection at all: the seven nightly status passes and the pre-generation (a provider call per account), the period-narrative warm, the record-detection cron (every account's history, every thirty minutes), the key rotation (every encrypted column), the content-index backfill (up to 200 transcriptions) and the on-host backup. Each of them now carries an explicit expiry on every send and schedule, stops between units once three quarters of it is spent (jobBudget), and runs under a session advisory lock keyed on queue and identity (lockedPass), so the retry pg-boss starts after the expiry does nothing beside a run that is still going. The lock is the one the full-history imports already held, moved into job-lock.ts. Where a pass stops early it says so (stopped_early) and the next run continues: the status passes and the narrative warm leave the rest to the next night, the pre-generation and the narratives walk the stalest accounts first, the on-host backup now backs up the oldest copies first, the key rotation reports a partial run in its audit row and skips rotated rows when run again, and the dose-history import fails the delivery so its retry resumes from the committed chunk. The off-host backup, which already stopped on its budget, gets a four-hour expiry and the lock.
59 lines
2.3 KiB
TypeScript
59 lines
2.3 KiB
TypeScript
/**
|
|
* The binding for a queue whose job can outlast pg-boss's default expiry.
|
|
*
|
|
* Three things together make a long job safe, and `queue-runtime.ts` names
|
|
* every queue that needs them:
|
|
*
|
|
* 1. An explicit `expireInSeconds` on every send and schedule, so pg-boss
|
|
* does not declare the job dead at fifteen minutes while it is working.
|
|
* 2. A time budget (`jobBudget` / `jobDeadline` in `job-budget.ts`) the
|
|
* handler checks between units of work, so the pass stops on its own
|
|
* before that expiry and the next run picks up what is left.
|
|
* 3. A lock the running handler holds (`withJobLock`), so the retry pg-boss
|
|
* starts once the expiry has passed does nothing beside a run that is
|
|
* still going. Queue policies cannot do this: they constrain job rows,
|
|
* and an expired job's row is no longer active.
|
|
*
|
|
* `lockedPass` is the third. It wraps a handler so each job runs under the
|
|
* lock `<queue>:<identity>`, where the identity is what two runs must not
|
|
* share: the account for a per-user job, a constant for a pass over everyone.
|
|
* A delivery that finds the lock held reports `already_running` and succeeds,
|
|
* because the run that holds it is doing the work.
|
|
*/
|
|
import type { Job } from "pg-boss";
|
|
|
|
import { withJobLock } from "@/lib/jobs/job-lock";
|
|
import { jobDone, type JobOutcome } from "@/lib/jobs/job-outcome";
|
|
import { annotate } from "@/lib/logging/context";
|
|
|
|
/** The identity of a pass that walks every account: one run at a time. */
|
|
export const WHOLE_PASS = "pass";
|
|
|
|
export function lockedPass<J extends Job<unknown>>(
|
|
queue: string,
|
|
identityOf: (job: J) => string,
|
|
handler: (jobs: J[]) => Promise<JobOutcome>,
|
|
): (jobs: J[]) => Promise<JobOutcome> {
|
|
return async (jobs) => {
|
|
let outcome: JobOutcome = jobDone({ jobs: 0 });
|
|
for (const job of jobs) {
|
|
const run = await withJobLock(`job:${queue}:${identityOf(job)}`, () =>
|
|
handler([job]),
|
|
);
|
|
if (!run.ran) {
|
|
annotate({
|
|
action: { name: "job.long_pass.already_running" },
|
|
meta: { queue },
|
|
});
|
|
outcome = jobDone({ already_running: 1 });
|
|
continue;
|
|
}
|
|
// The first failed job fails the batch; pg-boss hands one job per
|
|
// batch here, so this is that job's own outcome.
|
|
if (!run.result.ok) return run.result;
|
|
outcome = run.result;
|
|
}
|
|
return outcome;
|
|
};
|
|
}
|