The per-account copies roll one account back on the same server; nothing
the app made could rebuild a lost one. A scheduled job now runs pg_dump
over the whole database (every schema, pg-boss included) into
DB_BACKUP_DIR as healthlog-<UTC time>-daily.dump, with a manifest beside
it: app version and build, newest migration, Postgres version, size and
SHA-256, exact row counts of every table, and the fingerprints of the
keys the database needs.
The counts read the same exported snapshot pg_dump reads, so a dump
taken while the app writes still matches its manifest. The archive is
written as .partial, listed back with pg_restore --list, and renamed
after its manifest; a failed run leaves nothing named like a good dump.
Retention removes only files this job wrote, oldest first, and never
the newest good one (DB_BACKUP_KEEP, default 7; two pre-migration dumps).
With the off-host bucket configured, each dump and its manifest go up
under database/ in the envelope the account copies use, streamed. Every
run lands in the new database_backup_runs ledger (migration 0486), and
getDatabaseBackupHealth() answers the last good dump, the last failure,
whether the directory is writable and whether off-host is configured.
The boot readiness summary says where dumps go, or that none can be kept.
The dump, naming, retention and restore rules live in one plain ESM
module so the image's own scripts can share them without tsx.
pg-boss gives a send that names no retry options the queue's defaults:
two retries, at once, without a backoff. A job that reached a model and
failed then ran again immediately, twice, against the provider that had
just failed it. Two model-bound queues still relied on those defaults:
the Coach memory refresh send and the nudge cron. Both now ask for the
same two retries, a minute apart and backed off.
ai-job-retry-bound-guard holds the rule: every send to a queue whose jobs
reach a model names retryLimit (at most 3) and, unless it is 0,
retryBackoff with a retryDelay of at least 30 s; every such cron carries
the same in its schedule options. The queues come from the model-run
sites the expiry guard already keeps complete (now a shared helper), plus
the job files that call a provider directly, which the new guard keeps
complete itself. A planted case runs the checker over sources shaped like
the tree before this change. The queue golden records the nudge cron's
new options.
Refs #1279
A status run (a metric card, the batched assessment, a derived score, a
workout line, a Coach memory pass) is one model call plus the one repeat
the Local and gateway clients send when a thinking model answered with
reasoning alone, each with the person's response timeout. The run's outer
bound was a single call, so the repeat was cut off, and the open card
stopped polling after 48 s, so a note from a model that needs minutes
never reached the page. The bound is now the per-call timeout times
STATUS_MODEL_CALLS, and the card's poll ceiling follows the published
response timeout (fast for the first 48 s, every 15 s after), only while
the route reports the note in flight; the fallback text stays meanwhile.
The status queue is exclusive, so the longer poll cannot queue a second
generation beside a running one.
The morning refresh is one comprehensive generation of up to five calls
in a row, with no outer bound and pg-boss's fifteen-minute expiry, so a
large setting had pg-boss fail and retry it while it was still running.
It now runs behind the same bound as the warms, and the job carries an
expiry derived from that worst case. The per-metric note, the workout line
and the Coach memory refresh get expiries derived the same way.
queue-runtime.ts gains a bounded class for these jobs: model time read off
model-time.ts, an expiry above it, carried on every send. A new guard
names every caller of runStatusCompletion and generateComprehensiveInsight
with the queues that reach it, and holds each such queue to bounded above
its runs or long under a run lock.
Refs #1274
Marc-André Bombeck