The off-host job was said to benefit from the streaming work the weekly
in-database backup got, and it did not. Only the payload builder was
shared: `runOffhostBackup` called `buildFullBackupJson` for the whole
document, `gzipSync` over that whole string, a whole-buffer
`createCipheriv` pass over the result, and handed the finished buffer to
a single `PutObject`. Four full copies of the record alive at once, on
the same arithmetic the weekly pass had already been fixed for.
Configuring an S3 target on a long-lived record therefore restarted the
container. On an account of 445 000 measurements the JSON alone is
242 MB, and against a 1 GB container's 524 MB heap the run died of heap
exhaustion seventeen seconds in — taking every signed-in session with
it, because the job shares the app process.
The write path is a stream end to end now: `streamFullBackupJson`
produces the document a page at a time, gzip and an incremental
AES-256-GCM writer consume it as it arrives, and `@aws-sdk/lib-storage`
puts it up in two 8 MB parts. What the process holds is fixed by the
pipeline's shape rather than by the record. Measured on that same
445 000-measurement account under `--max-old-space-size=450`, with the
heap pre-loaded to what a warm server holds: the old path dies with
`Ineffective mark-compacts near heap limit`, the new one finishes with a
peak 79 MB above its baseline and writes a 9.1 MB object.
The stored envelope gains version 3, which moves the GCM tag from in
front of the ciphertext to the end. That is the only change the stream
required — the tag exists only once the last block is in, so a leading
one means the whole object must exist before its first byte can be sent.
Nothing about the authentication moves: the tag still covers every
ciphertext byte and is verified before a byte of plaintext comes back.
Versions 1 and 2 still read, so every object already in a bucket
restores through the same reader with no flag to tell them apart.
The incremental writer is the crypto module's, split into its byte-level
half so the base64 form the in-database blob uses and the raw form an
object needs are one implementation under two framings. The off-host key
handling is unchanged: a separate `BACKUP_ENCRYPTION_KEY`, 64 hex or
32-byte base64, passed in rather than looked up.
A run that uploaded nothing for anybody is a failed job now, not
`ok: true` with an empty bucket behind it. Wrong credentials, a missing
bucket and an unreachable endpoint all fail every account rather than
one, and the target's own sentence rides out as the cause. A run where
some account got a copy still succeeds; retrying the whole cohort over
one object would re-upload everybody's.
The one ceiling left is structural rather than a memory bound, and it is
counted rather than discovered: a multipart upload carries 10 000 parts,
so an object past 80 GB is refused for that account with a clear message
instead of failing halfway through with an SDK error about part numbers.
`tests/integration/offhost-backup-streaming-memory.test.ts` pins the
budget from both sides — what the streaming uploader holds, and that the
materialising path does not fit the same budget on the same fixture in
the same process, so the number cannot pass by measuring nothing.
The writer added in v1.38.6 compared the process's live heap usage against
80 % of V8's heap limit and aborted above it. That reading is not the
backup's footprint; it is the whole process's, garbage included. A
long-running app sits at 400 MB of largely collectable heap, so on a 1 GB
container (a 419 MB budget) the weekly pass aborted the first flush of every
account: four out of four in seven seconds, the smallest of them a demo
record whose entire stored copy is 1.2 MB, each told its record was too large
and its operator to buy memory.
Drop the gauge. What the writer holds is bounded by its shape and needs no
reading to prove: the three unbounded tables are pulled a page at a time,
every other section is released as soon as its JSON is in the sink, and at
most one flush of text is pending. The one copy that does grow with the
record is the stored blob, so the bound moves there — counted in the
ciphertext bytes actually produced, against a fifth of this process's heap
limit, which is what two copies of the answer plus a live set have to fit
inside. The message names both numbers.
A pass that wrote nothing for anybody now fails its job instead of reporting
`ok: true` with `backed: 0`, so the failing-queue panel and the backups page
read the same thing an operator would. A pass where some accounts were
written keeps passing with the failures as counts.
Measured under --max-old-space-size=450 from a heap deliberately dirtied
first: 445 000 measurements peak at 28 MB held by the backup, a one-row
record at under 1 MB.
The compression that landed last release was real and was measured in the
wrong place. On a development machine with a multi-gigabyte heap the pass
finished; the container the job runs in is capped at 1 GB, which gives V8
a 524 MB limit, and there the same pass died of memory about fifteen
seconds in. Because the job shares the application process, that was not
one failed backup — it was a restart, and every signed-in session on the
host went with it.
Reproduced before touching anything, under `--max-old-space-size=450`
against a seeded account of 445 000 measurements, 30 000 mood entries and
60 000 intake events: `FATAL ERROR: Reached heap limit` at 34 s. The same
fixture doubled to 890 000 measurements now completes inside a 296 MB
limit, holding 16 MB of live heap where the old writer held 168 MB.
The document is written incrementally now. `buildFullBackupPayload` takes
a `deferBulk` mode that declares the three unbounded tables instead of
reading them, and `streamFullBackupJson` walks the payload in key order,
pulls those tables through page by page, and releases every other section
as soon as its JSON reaches the sink. `packBackupBlobStreaming` feeds that
into gzip and an AES-256-GCM stream, so nothing between the database rows
and the stored value ever exists whole. Both writers go through the same
per-row serialisers, and the round-trip suite pins their output byte for
byte rather than trusting that they agree.
The intake and mood reads are ordered totally now. A keyset cursor needs
it, and a partial order also let two exports of one account differ over a
tie, which every multi-dose day has.
The streamed ciphertext is a new shape at rest, because GCM only produces
its tag once the last block is in and a leading tag is what makes the old
layout unstreamable. It carries a marker no older writer can produce, so
the two families are disjoint by construction rather than by sniffing.
All three stored shapes — plain, compressed, streamed — restore, proved
both ways.
The writer also watches its own live heap and stops at 80 % of V8's
limit. An account too large for its host is now one failed backup,
counted in the run's meta, instead of a restart for everybody on it.
The stored column is the ceiling that is left: `data_backups.data` is a
single `text` value, so the finished artifact has to exist as one string,
and that is the only part of the job still growing with the record. The
operator runbook says so, with the number at which it starts to matter.
The weekly pass built the whole account in memory, serialised it whole,
encrypted the string whole, and handed the result to the database whole.
That is four full copies of the record at the same moment — the object
graph, the JSON, the base64 ciphertext at 1.33x the JSON, and the copy the
driver makes of that parameter. Measured against a seeded account of
445 000 readings, the JSON alone is 242 MB and the run dies of heap
exhaustion; under a modest heap it never reaches the insert. The pass was
not slow, it could not finish.
A health record's JSON is extremely repetitive, so gzip takes that 242 MB
to 14 MB and every copy after it shrinks with it. The same run completes
in about twenty seconds with a peak heap of 434 MB and a stored row of
25 MB instead of 322 MB — which also matters because the copy lives inside
the database it would be needed to restore. Both backup jobs now hand the
serialised form back from a frame of its own, so the payload graph is
released before the compress step allocates anything.
The marker sits inside the ciphertext, so the stored column keeps the one
shape `decrypt()` already understands and rows written before this still
restore. The off-host object gets the same treatment through the version
byte its envelope always carried; version 1 objects still read back, which
both the restore drill and the restore script depend on.
The obvious saving — dropping soft-deleted readings, four rows in five on
a real record — is not taken, and the round-trip test now seeds a
tombstone and asserts it makes the journey. A tombstone inside the
retention window is what tells a synced client the row is gone and what
makes a re-sent sample a no-op; a restore without them brings back every
measurement the account ever deleted on the next device sync.
The `data-backup` queue was registered without send options, so every job
it minted took pg-boss's queue defaults: a fifteen-minute expiration and
two redeliveries. A snapshot that serialises every account's whole record
does not fit in fifteen minutes on a large one, so the pass was killed
part-way through, redelivered twice against the same record, and reached
`failed` with "job timed out" about three quarters of an hour after it
started — having never once completed.
Two hours, on the schedule rather than on the queue: `create_queue()` ends
in ON CONFLICT DO NOTHING, so a queue-level default would apply to a fresh
database and be inert on every instance that already has the queue. The
admin run-now route sends with the same options, or an ad-hoc snapshot
would silently keep the default the cron was rescued from.
The default two retries stay. The old redeliveries were pointless because
the first attempt could not succeed either; a transient failure genuinely
benefits from a second go, and the alternative is losing the week.
Every background path that resolved a user row's locale collapsed a
NULL column to English: the nightly briefing warm, the status crons,
the period-narrative warm, reaction lines, the morning refresh, the
briefing push, coach nudges, the reminder workers, and the document and
workout summaries each carried their own copy of
`locales.includes(x) ? x : defaultLocale` or called
`normalizeLocale(user.locale)`. None consulted the instance's configured
default language, so an account that never touched the language picker
got English prose and English pushes on a German-language instance.
Add resolveJobLocale (src/lib/i18n/job-locale.ts): stored user locale
when valid, else the operator's app_settings.default_locale when valid,
else English. The settings read lives beside the other app-settings
accessors. Every job and daily path resolves through it once, where it
has the row; the template helpers keep coerceLocale for a value that is
already resolved, and queue payloads keep normalizeLocale.
The briefing push hands its resolved locale down to the digest, and the
digest to the snapshot read, so the push copy and the digest lines are
in the same language instead of two different fallbacks.
A structural test walks src/lib/jobs and src/lib/daily and fails on a
hand-rolled coercion, a validator applied to a row column, or a row
reader that does not import the helper. It was red against the previous
tree on all three counts.
The sent-latch was a module-level boolean, so every worker boot re-sent
the same notice while the host's configuration never changed (#851).
Anchor the notice in the notification_events ledger instead, through the
same claimNotificationEvent helper the reminder schedulers use. The
anchor is unwindowed: entering the unconfigured state sends once per
admin, staying in it is silent across any number of restarts, and the
anchor is released when the state exits — databases configured, or
IP_GEO_LOOKUP_DISABLED set — so a genuine re-entry notifies again.
Three consequences worth naming:
- A host with lookups disabled is no longer notified at all. With no
egress there is nothing the notice warns about, and the disabled state
counts as an exit, matching the notice's own remediation advice.
- The evaluation now keys on offlineGeoReady() rather than the ASN
reader, so a host with only the ASN database mounted (city lookups
still egress) hears about it, where before it stayed silent.
- The retention prune excludes the new ADMIN_CONFIG_NOTICE event type:
a state-scoped anchor has a reader for as long as the state holds, and
the 90-day horizon delete would have re-opened the repeat on a slower
clock.
Refs #851
The knip blind spot (ignoreExportsUsedInFile) left eight symbols carrying
an export nothing outside their file consumes — the modifier goes, the
symbols stay. The score series module goes whole: its only importer was
its own guard test, whose composite- and config-guarding suites survive
in place.
DocumentContentIndex.tokenizerVersion gains the consumer that was the
point of storing it: the content-index job now re-tokenises rows whose
version trails CONTENT_TOKENIZER_VERSION, from the decrypted index text
along the same path the rotation script uses, keyset-paginated and
fault-isolated per row — a future tokenizer bump migrates the index
instead of silently splitting it. ImportJob.uploadSha256 turned out to
already have its dedupe consumer (both kick-off routes short-circuit on
same-bytes non-failed jobs); its comment now names that instead of
pretending to be write-only. The remaining write-only columns carry
their honest annotations in the shape the audit sweep parses.
- The provider-chain walker now checks the day's recorded spend against
OPERATOR_COST_CAP before invoking an operator-funded admin-* entry. A
request whose primary provider runs on the user's own key reserves
under the generous user-plan ceiling; a fallback hop (or health-ledger
reorder) onto the operator's key could then spend past the operator's
cap unchecked. An exhausted cap skips the candidate like a failed hop.
- summaryState PENDING gets a giver-upper: the detail serialiser
degrades a PENDING older than an hour to UNAVAILABLE (which offers the
manual generate action), and an hourly reaper persists the same heal
following the orphan-ImportJob reconcile precedent — a job dead past
its retry budget no longer leaves the sheet saying "generating"
forever.
- The daily Coach sweep now charges each ignored day against
surfaceCount and auto-dismisses at three — the cap the schema comment
promised since v1.22 but nothing enforced.
- The welcome gate compares the stored disclaimer acknowledgment
version against the current one, so revised disclaimer text
re-prompts; /api/auth/me carries the version for the client.
- slope90 and anomalyCount leave the analytics wire: slope90 had no
reader anywhere (computed on every request, in SQL twice), and the
slim path could never compute anomalyCount (it always reported 0).
anomalyCount stays as an in-process field on summarize() for the
insights feature builder, which is its only real consumer.
A Coach reminder captured with NEXT_BP_LOGGED, NEXT_WEIGHT_LOGGED,
NEXT_SLEEP_LOGGED or NEXT_APP_OPEN confirmed to the user and then never
fired: the daily sweep skipped context triggers and no other evaluator
existed.
Build the evaluator. The three logging cues resolve on evidence — a
live matching-type measurement logged after capture — and ride the same
eventful reminder-satisfy hook every ingest path already fires, with the
daily sweep re-running them as the idempotent backstop. NEXT_APP_OPEN
resolves on the nudge-status read (the poll that mounts with the app
chrome), owner-only so a delegate's visit never satisfies it, with a
capture grace so the session that created the reminder does not count as
the next open. Every trigger delivers through the surfacing transaction
extracted from the sweep: conversation + assistant message + status
flip, all-or-nothing.
Integration syncs (Fitbit, Oura, Polar, WHOOP, Strava, Nightscout) now
mark the per-user analytics cells stale and sweep the workouts cache at
their sync tails — imported rows used to stay invisible to the cached
dashboard, analytics and workout readers until the TTL lapsed. The WHOOP
per-resource webhook/cron driver and the Polar workouts leg carry the
same call, matching the posture Google Health and the workout batch
route already had.
Illness, cycle and encounter write routes hard-evict the analytics
bucket through the new invalidateUserHealthContext helper: Rest Mode,
the cycle context and the Today-rail appointment feed all read from
cached cells under that bucket, and an interactive write must be true on
the very next read.
The content-index backfill built its candidate MIME list from the
provider's native PDF support, so a provider that reads images but not
PDFs never saw a single PDF in "index all documents" — while the
per-document path already rasterises a PDF to page images for exactly
that provider class. The prefilter was stricter than the authority it
fronts.
PDFs are always candidates now; prepareVisionInput stays the per
document authority (native pass-through or rasterised pages). The run
summary also stops folding outcomes into one number: skipped (nothing
billable happened, retried later) and failed (a provider call missed)
are counted separately, carried through the wide event and the worker
log, and reruns stay idempotent because indexed documents drop out of
the candidate set.
Refs #776
The offline geo tier only reached a self-host two ways: baked into the
image at build time, or mounted from the host. Both need something the
operator of a stock `:latest` pull does not have. Add a third: set
MAXMIND_LICENSE_KEY and the worker downloads GeoLite2-City + GeoLite2-ASN
itself into GEOLITE2_DIR, so the offline tier starts working with no
rebuild and no manual mount.
A worker-only pg-boss queue drives it. Boot discovery fetches once when a
key is set and no database is present yet; a monthly cron (past MaxMind's
first-Tuesday reissue) keeps a keyed host current. The fetch is fail-soft:
each edition extracts to a temp file in the target directory and is
promoted with an atomic rename only after both are in hand, so a bad key
or an upstream error leaves the previous databases exactly as they were
and never writes a partial `.mmdb`. After a successful place the reader
cache resets so a running process picks up the files without a restart.
Extraction is pure Node (zlib + a minimal tar reader), not a shell-out:
this module is reachable from the worker-boot import chain, and a
child_process import there makes Next's file tracer pull the whole project
into the standalone output. zlib keeps the trace bounded and drops the
busybox runtime dependency. The download goes through safeFetch with the
public-host pin and redirect-following, since the MaxMind permalink 302s
to a signed CDN.
The runtime image now owns /opt/geolite2 as the unprivileged app user so
the worker can write it; a read-only bring-your-own mount at that path
still wins, because the fetch cannot write over it and fails soft.
Refs #659
A job, never a route. A forecast written as a side effect of opening a
page is a function of browsing rather than of data, and the prognosis
also has to be a value the person cannot overwrite — so one file writes
the table and a guard holds it there.
The cohort is the union of two sets and the second one carries the
weight: accounts with enough recent rated days, plus every account that
already holds rows. Without the second, an account that deleted its
entries would fall out of the sweep and keep last month's answer under a
fresh-looking date, with nothing on the surface to say so.
A pass replaces rather than accumulates. Every account it touches ends
holding exactly the rows that run wrote, or none: below the threshold,
module off, no pattern in the data, no recent day carrying everything
the fit needs — every one of those clears what was there. Per-account
failures are counted and the pass continues; only an all-accounts
failure fails the job.
Runs at 04:35, after the inputs have settled and clear of the geo
backfill on the hour's fortieth minute. The arithmetic is a ridge
regression and nothing else: no provider, no key, no egress, no consent
gate, and the module says so at the top so nobody wires one that does
not apply or assumes one that is not there.
The Coach button showed its unread dot over an empty conversation, and
opening the Coach neither showed anything nor cleared the dot.
Two paths fed one indicator. The nightly sweep flipped a "remind me"
reminder from active to due and stopped there — no conversation, no
message, and `surfaced` had no writer at all. The launcher lit its dot off
a second query that asked only whether anything was due. So the dot
promised content that was never written, and because opening the Coach
clears the message-based signal and not the reminder one, the only way out
was to resolve the reminder in Settings, which nothing on the badge
suggested. A reminder minted for a plan review put a user who had never
typed in the Coach in exactly that state.
The sweep now writes the reminder into the user's Coach thread as a real
assistant message and marks it surfaced in the same transaction: no
message, no status change, and nothing to point at. The badge derives from
that message like every other proactive one, so opening the Coach shows the
note and clears the dot. The launcher's second query is gone.
Reminders an older build left stuck on due are picked up by the same pass,
so an indicator that has been sitting on an empty conversation resolves
into the message it always implied.
No provider call and no push: the body is a deterministic localized
template around the user's own note text, six languages.
Proven end to end against Postgres — the row exists, the badge reads that
row, opening clears it, a second tick writes nothing, and a failed write
leaves the reminder untouched with the badge dark. All five fail against
the previous sweep.
The second indicator in the report, under the bottom bar's overflow entry,
is the active-route marker for that group and not a badge.
Refs #715
- Commit a final authorization claim under the lifecycle lock\n- Keep provider calls and retry outcomes outside database transactions\n- Retain authorization claims with notification delivery diagnostics
A GET must not have side effects — api-handler.ts says so next to
READ_HTTP_METHODS, where an MCP-audience token is admitted on the safe
methods precisely because they are assumed to write nothing. Three routes
broke that: the sync handshake bumped User.lastSyncedAt, the achievements
read inserted unlock rows, and the cycle calendar fired a discarded promise
at the forecast cache so a failed write left no trace anywhere.
Each write moves to the place that causes it.
The sync checkpoint is now stamped by the batch ingest endpoints for
measurements, mood and intakes. The handshake reports it. That also fixes
the watermark: the checkpoint used to advance past a window the client had
not drained, and a client trusting it would skip that window. It can only
trail a real sync now.
The unlock rows move to a nightly achievement-unlock-sweep. The badge grid
never needed them — unlocked comes from the live metrics and the completion
date is derived — so the payload is unchanged; the rows pin that date and
carry it into the backup, and the sweep reaches accounts that never open
the page.
The cycle forecast moves to a nightly cycle-prediction-refresh. The
CyclePrediction row is not a cache of a read: it is the index the reminder
cron scans, so which reminders an account received depended on whether
somebody had opened the calendar. persistPredictionCache no longer
swallows its errors either — the job reports them.
- select active canonical recurrence slots instead of rebuilding from tick day
- scope intake suppression and dedup metadata to the occurrence local date
The 15-minute reminder tick computed windowDuration ONCE per schedule
as windowEnd - windowStart, then applied that SAME span to every
timeOfDay slot in the schedule. For a single-dose schedule that span
genuinely is the one slot's window, but for a multi-dose schedule
(e.g. windowStart=08:00/windowEnd=18:00 with timesOfDay=[08:00,18:00])
it silently gave the 08:00 slot a 600-minute grace before even
entering ORANGE phase — ten hours nobody configured — while an
identically-intended single-dose schedule with a narrow legacy span
(08:00-09:00) correctly escalated on schedule but then had only a
30-minute radius for a logged-but-late dose to suppress its reminder.
Same misapplied field, opposite-looking symptoms: one medication
silently never reminded, the other kept escalating despite being
logged as taken.
resolveSlotWindowDurationMinutes resolves each slot's duration
independently, checked in priority order: an explicit per-dose
doseWindows entry for that exact timeOfDay (the actual per-dose rule
set in the dose-window editor) > the schedule's reminderGraceMinutes
(documented for exactly this purpose but never actually read by the
reminder tick until now — the write path already reads it via
graceToleranceMs) > the same default an unconfigured "point" dose
gets everywhere else in the app, bounded by the gap to the nearest
sibling slot so two doses can never share a window. A single-dose
schedule with neither override keeps the exact legacy windowEnd -
windowStart behaviour, unchanged, for backward compatibility with
schedules that predate the per-dose doseWindows system.
The core property this restores: a dose's resolved window never
depends on how many sibling doses share its schedule. Two
identically-configured doses now resolve identically regardless of
whether one of them happens to have a second dose elsewhere in the
day.
Full suite (18k+ tests) passes with no regressions; 13 new tests
cover both real-world cases that surfaced this (a single 08:00-09:00
schedule and a twice-daily 08:00/18:00 one) plus the sibling-
independence property directly.
Thirty-three suites read a registrar as source text and matched
`boss.work` next to a queue constant. They now match `createAndWork`,
and the seven schedule tuples that grew a `cronIsTheRetry` third element
assert that element rather than accepting any arity — a guard that
accepts more than it did before is a guard that notices less.
Two negative assertions were tightened while they were open: the
"has no cron" checks on the arrival and workout-insight queues ended at
a literal `]`, so a cron added WITH send options would have slipped past.
The behavioural repairs carry the real change. Handlers that resolved to
undefined now resolve to their outcome, so the assertions name the counts
instead of asserting nothing. The reconcile tick's test said it swallows
a failure instead of throwing; it reports a failed outcome now, and the
test says that, with the comment pointing at the retryLimit that carries
the old anti-retry-loop concern.
All 104 bindings across the five registrars now go through createAndWork,
whose handler type is (jobs) => Promise<JobOutcome>. The forty-two inline
handlers return what they did. A handler that falls off the end no longer
compiles, and the healthlog/job-handler-outcome rule refuses a bare
boss.work( anywhere under src/lib/jobs except the one module that owns
the wrapper. The rule's header states its own limit: it catches the
omission, not a handler that claims ok:true after a swallow.
Retry policy, reviewed queue by queue because the rethrow changes what
runs, not only what is visible:
- Twenty-one cron passes carry cronIsTheRetry (retryLimit 0). They are
the retention DELETEs, the OAuth-state prunes, the hourly maintenance
sweeps and the nightly per-sample drain — every one of them a
statement that will fail identically a second later. pg-boss defaults
to two retries with no delay, so converting them without this would
have run the same doomed statement three times a night. The next tick
is the retry and it arrives with the whole gap as backoff.
- The policy rides on the schedule, not on createQueue: create_queue()
ends in ON CONFLICT DO NOTHING, so a queue-level default would be
inert on every already-running instance.
- Everything else keeps its retries deliberately. A TLS probe, an
off-host upload, a restore drill and a provider poll all fail
transiently, and the reminder ticks only fail at job level when the
pass itself could not run.
Sixty-two handler functions across twenty-four modules now resolve to a
JobOutcome instead of undefined. The counts they already wrote to their
wide event ride out in `did`, so a run that deleted nothing and a run
that could not run are no longer the same shape.
Three rules decided each conversion:
- a catch that swallowed becomes jobFailed, keeping its warning line;
- a catch that already rethrew keeps its throw, because the throw
already fails the job and rewrapping it buries the stack;
- a fan-out pass is judged on the pass. One account's revoked grant
belongs on that account's integration ledger, not in a queue-wide
failure that retries the whole cohort, so per-leg failures ride out
as a users_failed count — visible for the first time.
The retention cleanups are the behaviour change worth naming: every one
of them used to warn and return, so a purge that had not run for weeks
looked like a purge with nothing to do. They fail now. Their registrar
sets retryLimit accordingly in the commit that converts the bindings.
JobFailure carries the thrown error's stack; without it every one of the
101 queues would report the same runJob frame.
Six provider syncs returned a bare count, which narrowed their failures away:
a Nightscout window in which every SGV row was refused by a constraint
returned the same 0 as a quiet night, and an Oura or Polar run whose
collections all failed returned 0 too. The routes then answered 200 with a
count and the settings card drew a green tick over it.
Every sync now reports what it imported AND whether any part of it did not
land, and the route resolves the outcome server-side so the web card and the
native client cannot disagree. Google Health keeps its 502 for a run that
failed and wrote nothing, but a run that failed after writing some resources
is now a partial rather than a discarded result. The labs OCR commit carries
the same resolved outcome.
A pg-boss handler signalled success by falling off the end of a function.
Every binding in this tree resolved to undefined and nothing read the
value, so a handler that caught an error, logged a warning and returned
was indistinguishable from one that finished its work: pg-boss marked
both completed and the failed-job readers saw nothing.
JobOutcome makes the completion a value the handler has to construct.
runJob turns ok:false into a reported, rethrown failure so the queue's
retry policy applies. createAndWork is the one binding path; its handler
type is (jobs) => Promise<JobOutcome>, so a handler that returns nothing
no longer compiles.
The type cannot make a handler honest — jobDone() after a swallow still
compiles. It converts omission into commission, which is the smaller and
far more reviewable class.
The per-phase "already sent today" guard read the Telegram message
ledger, and that table is written only when a Telegram send succeeds. A
user on email, APNs or ntfy never had a row, so the 15-minute tick
re-derived the same overdue slot and sent it again on every pass — around
fifty mails for a single missed dose.
Anchor the guard in `push_attempts` instead, the ledger the daily
briefing and the illness escalation already use. Two properties are
deliberate: the anchor is stamped for every dispatched slot regardless of
what any channel returned, because keying it on a delivered push would
re-open the flood the moment a channel starts failing; and it is stamped
before the dispatch, so a crash costs one phase's notice rather than a
repeat every quarter hour.
The key carries the schedule id and the slot's real HH:mm alongside the
phase and the local date. Both are load-bearing: two legacy
single-window schedules on one medication share the empty first-class
time-of-day, so a coarser key would collapse the evening dose into the
morning one. The tests assert the other direction too — the next phase
still escalates, and the next local day starts clean.
The Telegram ledger keeps its message-tracking duties; it is no longer
the dedup key. Anchor rows carry a sentinel channel and stay out of the
operator's notification health and diagnostic reads, which count
deliveries, not decisions.
`medication.clientManaged` and `measurementReminder.clientManaged` mean
the iOS client schedules the reminder locally, so the server should not
push it. That is what the OpenAPI description, the settings copy and the
iOS contract all say. What the crons did was skip the dispatch entirely,
so a user who flipped the switch also lost Telegram, ntfy, webhook,
email and Web Push — channels they may well read on a desktop while the
phone holding the local schedule sits in a drawer.
Move the gate into the dispatcher's channel loop, where it applies to
APNs and nothing else. The skip is not counted as an attempt, so
`dispatched` keeps meaning "something reached the user", and it stays
countable: the same wide-event keys the crons emitted are emitted here,
plus a `skipped` row in the push-attempt ledger.
The Vorsorge path is the sharper half. It suppressed every channel and
then advanced `nextDueAt`, which dropped the reminder out of the
preventive-care card and out of the daily digest: a reminder that never
reached anyone was recorded as handled. The advance now happens only
where it always belonged, after a dispatch that a channel actually
delivered. When nothing delivers, the reminder stays overdue and visible.
A sync failure the classifier bucketed as `transient` had no exit. The park
test read `kind === "persistent"` and the persistent streak anchor was only
ever stamped by a persistent failure, so a streak could run for a thousand
attempts with the persistent counter at zero, the state frozen at
`error_transient`, and no ladder it could climb. `transient` is a hypothesis
about retryability, and the classifier defaults every timeout, 429, 5xx and
unrecognised response to it precisely because the wire cannot tell a reboot
from a permanent break. That default is right for one attempt and wrong for a
thousand: an unbroken streak retried hourly for a full day has refuted it by
experiment. Record it as `persistent` from there, which feeds the park ladder
that already exists — 48 h of unbroken failure end to end before an
integration stops attempting.
The counters could not accumulate anyway. `IntegrationStatus` is keyed
(user, integration) while Withings runs four legs on four crons and WHOOP four
resources on four more, and a success cleared the error, every bucket, the
streak anchor and the state whatever leg had recorded them. The ECG tick at
:41 wiped what the sleep tick recorded at :15 — including a reauth park — so
the strike ladder was reset several times an hour and the card painted a green
"connected · 12 minutes ago" over a data class that had stopped arriving. A
success now clears only what it is entitled to clear.
Two columns carry it: `failing_since_at` dates the streak, `failing_leg` names
its owner. The migration backfills the anchor for rows already in an error
state so a connection broken for weeks escalates on its next attempt rather
than restarting its clock.
The nightly tombstone purge asked for `deleted_at IS NOT NULL AND deleted_at
< cutoff` and the audit-log purge asked for `created_at < cutoff`. Neither
predicate had an index behind it: every deleted_at index in the schema is
partial on IS NULL, the live-rows predicate, and every audit_logs index leads
with user_id or action. Both purges planned a sequential scan and then paid
index maintenance per removed row, in one unbounded statement, on a
connection carrying statement_timeout=60000. Past a certain backlog the
statement aborts, the transaction rolls back, nothing is purged, and the next
night runs the identical statement against the same rows. The mechanism that
gives deletion a horizon could not close on the instances that needed it.
Migration 0277 adds the indexes the predicates ask for; the three tombstone
ones are partial on IS NOT NULL, so on a healthy instance they cover only the
tombstones and shrink as the purge drains. The purges now walk the backlog in
batches of 5 000 by primary key, capped at 200 000 rows a run.
Both handlers used to swallow every error into a wide-event warning and
return, so pg-boss recorded a successful job either way: a purge failing
every night looked exactly like a purge with nothing to do. They now rethrow,
and a run that stops at the cap records that a backlog remains.
Mood has been a first-class module here for a long time, with entries,
structured tags, rated factors and notes. The bridge to the external moodLog
service had nothing left to add, and its Settings card was withdrawn in
a56e6fe61 on the promise of a dedicated removal. This delivers it.
Removed: the hourly `moodlog-sync` queue, cron and handler binding; the
inbound webhook and its shared-secret handling; the reverse push and its four
call sites; the pull-sync module; the per-user credential routes and the
instance-wide toggle; the entry on the `/api/integrations/status` envelope,
the `IntegrationKey` union, the cadence map and the display-name table; the
boot-time secret rotation pass; 22 message keys per locale.
Migration 0271 drops the five `users` credential columns and
`app_settings.mood_log_global`, clears the ledger row for the provider, and
deletes the retired pg-boss schedule so no cron keeps enqueuing work no
handler drains.
Mood data is untouched. `mood_entries` is not named anywhere in the migration
and cannot be reached from it: the dropped columns carry no index, constraint
or foreign key, and `mood_entries.user_id` references `users(id)`, not any of
them. Rows the bridge imported keep `source = 'MOODLOG'`, and every read path
- list filter, source badge, exports, backups - still resolves that value.
`src/lib/validations/moodlog.ts` held the native mood-entry schemas alongside
the bridge's own; it is now `validations/mood.ts` and carries only the former.
Fitbit re-scores a night after the fact, and the per-segment externalId carried
a positional index plus the stage label. Both moved on a re-score: renumbered
segments minted fresh keys, the previous scoring's rows stayed live because
nothing swept them, and the night total counted both copies from then on.
Key the segment on the stable session anchor plus its own start instant
(`<logId>:sleep:<segment-start>`), and add the replace-by-window sweep every
other segment-writing provider already has. The two ship together with the
natural-key rescue that has been sitting in the Fitbit upsert unused: the sweep
tombstones the old-keyed rows first, so the rescue re-keys those very rows in
place instead of inserting a parallel copy or losing the insert to
`skipDuplicates` against the natural unique index.
A one-shot repair queue re-reads the bounded sleep history once per connection
so no user has to do anything; it stamps `sleepRepairedAt` (migration 0270) only
on a clean run, never touches the sync watermark, and reports the rows it
removed. Nights older than the 365-day backfill horizon keep their old keys and
whatever duplicates they already hold — they can no longer gain new ones.
Alongside: `fitbitGet` now honours a `Retry-After` inside a 120 s cap by waiting
once and retrying once, and fails fast above it so a long hold cannot starve the
rest of the poll cohort or outlive a job's expiration. The measurement upsert
skips a write whose payload is unchanged, which takes the hourly overlap off the
delta feed for every Fitbit resource.
These four were the only connected providers with no way to ask for a
sync; they ran on their scheduled pass and nothing else. Each gets a POST
sync route on the hardened Fitbit and Google Health template: session auth,
a 5-per-minute per-user bucket, and a fixed 502 on failure. No body and no
full-sync flag, because none of the four entry points has a full-history
arm to call. Polar goes through the shared legs entry point so both legs
run with the attribution rules built for them.
The Nightscout error arm is the one that matters: a base URL carries the
API token as a query parameter, so the caught message can hold the user's
secret and never reaches the response body. The classified failure is
already on the ledger, recorded with the provider's own redaction.
Connecting also starts the first sync now. The poll handler has accepted a
single-user payload since it was written and never had a producer, so a
fresh connection sat empty for up to an hour waiting for the schedule. The
Oura and Polar callbacks and the Nightscout connect route enqueue one
targeted job each, best effort, with the hourly tick still the safety net.
Strava needs none: its callback already enqueues the history backfill,
which covers the recent window.
The four poll-queue names move into their own module so a request-path
caller can name a queue without dragging the whole worker handler graph
into the request bundle, following the backfill-queue precedent. Creation,
scheduling and handler binding stay in the registrar, and the dead-queue
guard follows the declarations.
Restructure the Polar vitals sync from a single Promise.all over the five
collections into a Promise.allSettled over a per-collection spec, each closure
fetching and mapping so a throwing mapper is contained exactly like a failing
fetch. The healthy collections import; a failing one is recorded once as a
partial failure that names it, without stamping success and without rethrowing,
so a single unavailable data type — SpO2 on a watch without the sensor, a
higher-tier cardio-load read an account lacks — stops blanking recharge, sleep,
and activity on every tick.
A 401 is now a per-collection failure classified reauth_required, not a
whole-leg throw; the write-path escape still surfaces unmarked for the poll
boundary to record once. Mirrors the shipped Oura isolation, minus the
token-refresh leg Polar has no path for.
Add an hourly ECG catch-net cron so a watch-only account with no scale pulls its
recordings without a webhook: the queue was registered and bound but never
scheduled, and the handler now grows a cohort arm that walks every connection on
a payload-less tick, mirroring the activity and sleep handlers.
Make the activity and sleep write paths honest. A swallowed activity batch-write
fell through to the success stamp — a green pill over lost data, green forever on
a persistent write bug; it now records a transient failure and refuses the
stamp. A reconcile throw out of either sleep transaction propagated past the
stamp and was swallowed by both callers, leaving neither success nor failure
recorded; the source now records a transient failure and rethrows so the cron
and webhook paths stay honest without changing their warn-and-continue
behaviour.
Port the hard-fail ledger the Fitbit transport already carries. A revoked WHOOP
grant makes every resource short-circuit and return zero rows without throwing,
so the cycle read as clean: it stamped success, un-parked the reconnect state,
reset the failure streak, and the hourly cohort hammered the dead refresh token
on every tick. Record the three dead-token null-return paths on the ledger and
fold that verdict into both drivers — the full orchestrator and the per-resource
cron path that runs most often — so a dead-token cycle refuses the success
stamp and the park holds.
Return an imported-plus-failed verdict from the orchestrator and gate the
history backfill on it: an interrupted full-history walk now throws for a
pg-boss retry instead of being stamped complete, and a connection parked at
reauth returns without burning the retry budget. Thread the same verdict into
the one-shot sleep-timeline backfill.
Also catch a write-path throw on both drivers so a persistent measurement-write
failure surfaces on the status ledger instead of leaving a stale-green pill, and
widen the cycle overlap to the seven-day re-score window the other collections
use so a late-revised day's strain and energy are re-fetched rather than
stepped past.
Introduce a recorded-marker primitive on the integration ledger so a provider
that records-then-rethrows a sync failure is not recorded a second time at the
poll-cohort boundary, while an unrecorded escape (an Oura or Polar write-path
throw, or any future path) lands on the ledger exactly once, correctly
classified and redacted by the source that owns it. Without the marker a
boundary-blind recorder would double-count the failure streak and could persist
a Nightscout token-bearing URL into the stored error.
Give the poll-cohort factory a required per-provider failure recorder and wire
one for Oura, Polar, Nightscout, and Strava.
Nightscout now counts only freshly inserted readings instead of insert plus
re-confirm (the inflated count fired the reminder trigger every tick), records a
transient failure and skips the success stamp when some rows fail to write, and
emits the measurement-arrival spine for its inserts. Correct the docstring that
promised a far-history backfill that does not exist: the hourly window imports
about 48 hours and nothing deeper.
Strava now enqueues its history backfill at connect time, mirroring the WHOOP
and Fitbit callbacks, so deep history imports within the hour instead of waiting
for the next worker restart.