A database dump is the whole database at one moment, so an account deleted
since stays in the dumps taken before until they age out. The database row
on the backups card now says so with the numbers that decide it: how many
nightly dumps this server keeps, and after how many days the bucket's
lifecycle rule removes the off-host copies.
The lifecycle probe used to answer with the shortest expiry of any enabled
rule, whatever its prefix, so a rule scoped to the account copies read as
covering the dumps too. It now also answers for `database/` alone, counting
only rules whose filter takes in a dump's key (prefix, legacy prefix, an
And filter; a tag filter never matches untagged objects), and the card
warns when no rule covers that prefix: dumps there, and the deleted accounts
in them, would never be removed. The reading moved into its own module.
A lost host takes its local dumps with it, and the copy left is the one in
the bucket. `docker compose run --rm db-restore latest --from-bucket` now
lists `database/` in the bucket the BACKUP_S3_* variables name, takes the
newest dump whose manifest is there too (or the one named), downloads both
with the S3 client the image already ships, and continues as for any
off-host copy: opened with BACKUP_ENCRYPTION_KEY, checked against the
manifest, restored into an empty database. `list --from-bucket` shows what
is there. The sealed download is removed once it is opened.
The image build now proves the S3 client resolves from the shared module's
place in the tree, so the restore cannot lose it unnoticed.
The per-account copies roll one account back on the same server; nothing
the app made could rebuild a lost one. A scheduled job now runs pg_dump
over the whole database (every schema, pg-boss included) into
DB_BACKUP_DIR as healthlog-<UTC time>-daily.dump, with a manifest beside
it: app version and build, newest migration, Postgres version, size and
SHA-256, exact row counts of every table, and the fingerprints of the
keys the database needs.
The counts read the same exported snapshot pg_dump reads, so a dump
taken while the app writes still matches its manifest. The archive is
written as .partial, listed back with pg_restore --list, and renamed
after its manifest; a failed run leaves nothing named like a good dump.
Retention removes only files this job wrote, oldest first, and never
the newest good one (DB_BACKUP_KEEP, default 7; two pre-migration dumps).
With the off-host bucket configured, each dump and its manifest go up
under database/ in the envelope the account copies use, streamed. Every
run lands in the new database_backup_runs ledger (migration 0486), and
getDatabaseBackupHealth() answers the last good dump, the last failure,
whether the directory is writable and whether off-host is configured.
The boot readiness summary says where dumps go, or that none can be kept.
The dump, naming, retention and restore rules live in one plain ESM
module so the image's own scripts can share them without tsx.
Three readers grouped the same rows twice and joined the second grouping
back to the first: the live buckets joined each hourly-mean day's hour
means to the per-source days, the all-time extremes joined the day
weights the same way, and the doctor report's dense buckets joined each
bucket's latest reading and hour means to the bucket aggregates. On a
measurements table without planner statistics (a new install, or an
import under the autoanalyze threshold) the planner estimated one row on
each side and ran every one of those joins as a nested loop that re-ran
the inner grouping for each outer row.
On ten years of readings over six types (51,000 rows, no statistics) the
dense buckets and the every-type live buckets ran into the 60 s statement
timeout and the all-time extremes took 11 s. With statistics all three
answered in 0.1 to 0.2 s.
Each is now one chain of folds over one scan. The live buckets fold the
readings per local hour and the hours per local day, carrying the day of
hours beside the counts. The all-time extremes take the day weights from
the same windowed rows they count, the shape the pre-fold remainder
already has. The dense buckets carry each bucket's latest reading and each
hour's count and mean on the rows as window values, and one grouping folds
them all. Same fixture without statistics: 0.2 s, 0.1 s and 0.1 s, the same
as with them.
A parity suite runs the joined statements as the reference against the
readers across ladders, zones with whole, half-hour and negative offsets,
grains, overlapping and off-ladder sources, glucose contexts and device
types, tombstones and both DST changes; sums and means agree to
floating-point summation order, everything else exactly. The
unanalysed-table guard now runs the three readers under EXPLAIN ANALYZE:
the joins measured 200 to 3,000 row visits per row stored, the folds about
10.
The dense-bucket loader drops under the 200-line function cap, so its
baseline entry goes.
The blood-pressure series chose each UTC day's source with DISTINCT ON
over the day's floor-passing systolic rows and joined the choices back
to the readings. That is the shape the analytics pick had, with the
same weakness: on a measurements table without planner statistics the
planner estimated one row on each side and ran the join as a nested
loop that rescanned the pick for every reading.
The ten-year series over 6,000 freshly written rows from one source
took 0.57 s for the statement alone, 0.86 s with a second source on
every third day; the plan visited about 4,900 rows per row stored, so
the cost grows with the square of the history.
The pick is now one window over one scan: the day's rows ordered
systolic first, then by ladder rank, then by source name, and the first
row's source taken only when it is a systolic one. The scan starts at
the window's first UTC day, so the pick still sees the whole edge day,
and a day without a floor-passing systolic still shows nothing. Both
fixtures above read in about 10 ms, with or without statistics.
A parity suite runs the old statement as the reference against the new
one across overlapping and off-ladder sources, half readings, sub-floor
rows, tombstones, both DST changes, window starts on and either side of
UTC midnight, and four session zones. The unanalysed-table guard now
covers the series' ten-year, quarter and week reads.
The canonical-source pick grouped the window to (type, day, source),
chose each day's source with DISTINCT ON, and joined the choices back to
the rows. Whether that join ran as a hash join or as a nested loop that
rescanned the whole pick for every row was the planner's call, made from
table statistics. On a measurements table without them (a new install,
or an import still under the autoanalyze threshold) it estimated one row
on each side and took the nested loop.
Binding the request's clock as a parameter instead of reading NOW()
flipped which side of that loop the pick landed on: the analytics read
over 6,000 freshly written blood-pressure rows went from 1.6 s to 8.4 s,
and the demo seed's dashboard headline to 20 s. On a ten-year account
without statistics the 90-day read took 4 s with either clock and the
all-time read ran past two minutes.
The pick is now one window function over one scan: FIRST_VALUE of the
source per (type, UTC day) in ladder order, keeping the rows of that
source. Same rows, same order of preference, so every day keeps its
source; the parity suite against the DISTINCT ON statement holds that.
The analytics read on the fixture above takes about 0.1 s, the headline
0.4 s. On the analysed ten-year account the 90-day and all-time reads
are as fast as the join was, and they no longer depend on statistics.
The rows carry one extra column, canonical_source; every reader selects
its columns by name.
A new integration test runs every reader shape under EXPLAIN ANALYZE on
a table with no statistics and bounds the rows the plan produces or
discards per row seeded. The statement count never moved, so a statement
budget could not see this; the join measured 200 to 8,600 row visits
per row seeded and the window stays at 4.
A status run is one completion, so its ceiling was the completion ceiling
under a second name. The Local client also re-exported a JSON-mode helper
nobody imported from it.
The backup export is totally ordered so two exports of the same account are
byte-identical; the alias relation the import-key ledger reads was the one
nested read without an order.
The lab scan runs only in the background worker, which hands it the
run's deadline, but its body still took the schedule as optional and
planned a missing one without a bound: a twenty-page report as forty
model calls. Nothing calls it that way today; the signature allowed the
next caller to, and so did every read planner beneath it.
The scan and the planners (maxReadsFor, planTextReads, planVisionReads,
planInboundTextReads, planInboundReads) now require a schedule. A run
with room for every read says so with an unbounded one; a request uses
SYNCHRONOUS_READ_SCHEDULE. No route, wire or response changes.
The index route without Prefer: respond-async, which the iPhone app
calls, continues into the lab staging with no run schedule. The staging
then planned a long report's text as up to twenty reads with a retry
each: forty model calls inside one HTTP request, the same class the
extract route was bounded against.
The staging now plans a missing schedule as one request does
(SYNCHRONOUS_READ_SCHEDULE, at most three reads): a longer report has
its text shared out between them, and what does not fit is left out and
reported as extractionIncomplete on the document. The request bound
moves into ocr-plan beside the schedule it bounds, so the extract run
and the staging share one definition. Background runs and the automatic
index job keep planning against their own deadline.
The backup carried neither the import ledger nor the tombstones of
deleted documents, so after a restore onto a fresh instance a token
importer stored every imported document the person had deleted on
purpose again.
The document filing section now carries documentImportKeys: the
ledger's rows plus the own and alias keys of the documents waiting in
Recently deleted, which still do not travel as documents. The restore
writes them back, minus any key a restored live document holds, and
counts them in the restore summary and the preview. DocumentImportKey
moves from the excluded models to the two-ended ones, and the round
trip, the column audit and the parity goldens carry a ledger row.
The boot discovery for the documents 0479 marked took the first thousand
by id and skipped those whose account has the documents module off.
Such an account keeps its marks, so with enough of them it held every
place of every boot and the accounts after it never got their retry.
The marks are walked by id past the last row read, an account found
with the module off is left out of the query from then on, and the
boot stops at a thousand documents enqueued rather than read.
Refs #1277
The index-all backfill billed the read and wrote the index in one try:
an index write that failed after the read was settled again in the
catch, handing the reservation back twice. The read is settled once,
and a failed write charges nothing back.
The backfill also checked consent and the operator's switches once, at
the pick, then read up to two hundred documents. The wire is asked
again before every read; a refusal hands that document's reservation
back and ends the walk with no-consent or unavailable.
The extraction merged facts across reads on a key whose only date was
the fact's own effective date. Facts without one collapsed across reads
of different reports, so the same value on two days staged once. An
undated fact is now keyed on its read's report date, falling back to
the document's, as the lab scan keys a row.
Refs #1257
Without Prefer: respond-async the extract route ran the read with no
schedule, so a twenty-page report read a page at a time, with a retry
each, could make forty model calls in one HTTP request. The iPhone app
sends no Prefer header and reads the document detail from the answer,
so the request keeps its 200 and its body.
An extraction without a run deadline now plans at most three reads (six
calls): a longer document has its pages or text shared out between
them, and what does not fit is reported as extractionIncomplete on the
document. The background run keeps reading it a page at a time.
Refs #1257
A long document read in parts can end with a part cut at the provider's
output limit, a part never read, or more facts than one review takes.
That was told once, as a notice when the run finished; a review opened
later looked complete.
InboundDocument.extractionIncomplete (migration 0483) is written with
the facts by both stagings, the extract route or run and lab
auto-staging. The document detail publishes it and the review shows the
same note whenever it is opened. The disaster-recovery backup carries
it; an older file restores it as false.
Refs #1257
A gateway or local entry can carry its own reasoning setting (#1126), sent
on every call that has no level of its own. An endpoint that refuses
reasoning_effort answered each of those calls with a 400 that was never
learned, because only the call's own level counted as sent: every status
note, every summary and every other level-less call failed, for as long as
the setting stood.
The refusal is now learned whichever level put the field on the wire, by
the same path as any refused field: the call is asked once more without it
inside its two-request budget, later calls send no field for that endpoint
and model, a model seen reasoning unasked keeps its thinking room, and one
ai.provider.reasoning_field_refused line says so per lesson instead of a
failure per call.
Refs #1279
The nightly warm, the forced warm and the morning refresh bound a
comprehensive generation and abort it when the bound fires, but the abort
only reached the generator's final check before the cache write. The
provider calls never saw it: the call in flight ran on detached, and a
reply that landed after the bound could still start the JSON-shape,
grounding or screen retry, each a full generation nobody was waiting for.
runBriefingCompletion now takes the caller's signal, hands it to the
provider call, and reserves and sends nothing once it has fired. The
generator passes it at all five of its calls (the generation, its three
retries and the phrasing re-roll). The on-demand route passes none on
purpose: it finishes and writes its briefing after the browser stopped
waiting.
Refs #1279
Tighten the warm-up comments in the summaries slice and the comprehensive
aggregator so neither file grows past its line baseline, drop a release
opener from the sharing-surface note, and record the version stamp that
left the settings section list.
The sweep keeps going when reporting a load failure itself throws, and
now says so on stderr; the error-level guard refuses a silent catch in
src/lib/jobs. The preload test reads the worker source through
readSource with anchors.
The per-account preview switch was kept for the topic overview pages,
which were dropped before they shipped. Its one flag was stored, served
and backed up, and read by nothing; the column-reader audit listed it as
a pending consumer that will not come.
Removed completely:
- `/api/settings/preview-features` (GET, PATCH) is retired: it answers
410 Gone with `meta.errorCode: route.retired` and `removedIn: 1.46.0`,
and the contract publishes it under Retired. The route, its validation
schema, the `PreviewFeatures` OpenAPI schemas and the settings module
are deleted.
- Migration 0485 drops `users.preview_features_json` under a bounded lock
wait, with the retry marker.
- The backup no longer writes the field. A file from v1.44 or v1.45 that
carries it still restores: the schema accepts it and no codec reads it,
so nothing is refused or written for it. The backup-parity goldens
lose the field and apply one setting fewer.
- The data-wipe plan entry and the sharing-surface actor entry go with
the route (frozen count 273 -> 272).
No client build reads the route: neither iOS repository nor any local
checkout references it.
The health ledger records failures per account and provider, and since
background work leaves out a benched provider, an operator who fixed the
instance-wide provider in the admin AI settings still saw every account's
background work sit out the pause the broken setup had earned, up to a day.
Only a person's own provider settings and the Codex sign-ins forgot the
record.
A successful save of the admin AI settings now forgets what the ledger
recorded for admin-openai on every account and closes the status notes'
negative windows, the same reset a person's own save makes for theirs
(clearStatusNegativeWindows without an account covers every record). The
ledger comments that still said a benched row only deprioritises now say
what it does for background work.
Refs #1279
A status run whose whole chain was benched for background work sent no
request, yet the caller opened the note's negative window as for any
failure and raised its failure streak. The streak doubles the window up to
two hours, so a note kept waiting long after its provider recovered.
AllProvidersFailedError now says when no hop sent anything (sentNothing:
every provider benched, or refused by the operator cost cap). The status
chokepoint then holds the note for the first window without touching the
streak (holdStatusNoteWhileBenched), and the caller's own failure write
lands inside that window and neither moves nor counts it. A run that
reached a provider counts as before.
Refs #1279
End course tombstones the pending placeholders at or after its instant.
Moving or removing the end clears the instant, but the tombstoned rows
kept their unique slots, so the today projector treated them as present
and minted nothing: the later dose of that day was neither listed,
reminded nor missed.
The window's one writer now revives the placeholders the End stamped
with its instant once that instant is gone, when the slot is inside the
courses as they stand and no live row holds it.
Refs #1280
`afterProviderBackfill` refolded the rollups over the touched range and
re-warmed the status cards, but never marked the analytics caches stale,
so after a Fitbit or Google Health history backfill the dashboard and
analytics reads served the pre-backfill body until its TTL ran out. The
per-write tail has always done this through `afterMeasurementMutation`.
The backfill tail now calls `invalidateUserMeasurements` with
`touchedFrom` set to the first touched day, after the fold and also when
the fold fails, since the rows changed either way.
Since handlers load on their queue's first job, a module the image
cannot load no longer fails the worker's boot. The worker comes up
healthy, the queue burns its retries when it first has work, and a
one-shot job (a restore, an import tail) ends failed with nobody told at
deploy time.
After binding every queue the worker now runs `preloadHandlers` in the
background, without awaiting it, so readiness is unchanged. It loads
each handler once through the same memoised loader its queue is bound
to, so a module that loads stays loaded for the first job. Each failure
is reported on the `job.run.failed` line at error, with the queue's name
and `reason: "handler_load_failed"`, plus the worker error report
(stderr and GlitchTip). A summary line on stderr names every queue that
failed. The sweep never throws, and a failing report does not stop it.
docs/ops/deploy.md says what to look for after a deploy.
`/api/sync/changes` read the `measurementMirror` domain with one query
that ordered every token-linked reading by the joined token's
`mirror_changed_at`. No index on `measurements` can serve that order, so
every poll read and sorted every row of every token ever switched, even
when nothing had changed past the client's watermark.
The walk now reads the account's tokens with `mirror_changed_at` at or
past the watermark first. Nearly every poll finds none and stops there.
Otherwise it walks each change stamp's readings by
`(ingest_token_id, id)`, the order the index holds, with tokens that
share a stamp walked together so rows interleave by id as before. Rows,
order, page boundaries and the cursor are unchanged; the wire contract
is untouched.
EXPLAIN (ANALYZE, BUFFERS) on a seeded Postgres 16: one account, a
token with 300,000 readings switched at T0, a second token with 50
readings switched at T1, 900,000 untokened readings, page size 200.
steady-state poll (watermark past every switch)
before: 53 ms, Merge Join over ~300,000 index rows, 10,827 buffers
after: 0.07 ms token lookup + 0.02 ms resume probe, 10 buffers
first page after the switch
before: 51 ms, top-N heapsort over ~300,000 rows, 10,948 buffers
after: 0.08 ms + 0.17 ms index scan on (ingest_token_id, id)
resume in the middle of the big token
before: 53 ms
after: 0.02 ms + 0.20 ms, Index Cond on (ingest_token_id, id)
A new integration file pages both forms side by side (tied stamps, a
watermark inside a tie, soft-deleted rows, another account's token,
page sizes from 1 to 100) and asserts identical pages, and that a poll
with nothing switched past its watermark reads no reading at all.
The mood rollup warm-up on the dashboard summary was started with a bare
`void`, so `settleBackgroundTasks()` never saw it. On a stale account the
warm-up started a full recompute that could still be writing when the
statement counter closed its window, and the hot-route budget read 30
against 28 depending on where it landed.
Every read-path start of that shape now goes through
`trackBackgroundTask(runDetached(...))`: the measurement and mood rollup
warm-ups (seven call sites), the dashboard's today-briefing warm, and the
two delegated-access bookkeeping writes in the record guard. A guard
refuses a bare `void` on any of them.
`trackBackgroundTask` now wraps its handle in `Promise.resolve`: a test
double that returns nothing used to leave `undefined` in the set, and
settle spun on it at full CPU.
The budget fixture mints the mood rollups in its seed, as it already did
for the measurement rollups. Eight serial runs of
query-budget-hot-routes read the same counts every time (dashboard
summary 28, snapshot 99, day 47, health-status 59, correlations 43).
A chain whose first provider reasoned without answering and whose second
one answered billed only the answer: result.tokensUsed is the answering
call's alone, and the failed hop's spend, already reported on its
FallbackHop, was dropped by every caller that reconciles the day's ledger.
The raw runners now report tokensSpent on success too, the sum the failed
hops before the answer reported. The briefing, status, Coach (no-tools path
and every tool-loop round) and medication-extract reconciles charge it on
top of the answer. On failure, the Coach turn ledger's close keeps what the
failed chain reported spending instead of refunding everything, and the
medication-extract and document-chat routes charge it too, as the status
and briefing paths already did. Document indexing resolves a single provider
and already charged its failed call.
Refs #1279
canStartCourse answered true on the day a course was ended with End
course, while a course starting that day shares its last day and the
start refused it with 422 overlap. The start is now offered from the
day after the last course's end, the first day validation and the
engine accept.
A course ended at an instant is over for every rule that asks about
today, whatever the zone it is read from: after a move west the course
rows read CURRENT while the medication's own status read ENDED. The
course rows, the coach summary, the overlap rule and the previous-end
read take the instant-aware reading.
Refs #1280
calculateCompliance and buildTimelineForWindow built the cadence
engine's context field by field and left out endedAt, so a dose slot
after End course counted as missed in the streak and the timeline while
the rate left it out.
Both now take the context from one builder, endedAt is required on
ComplianceMedicationContext and CadenceEngineContext, and the course
anchor guard holds every builder of either type to naming it.
Refs #1280
POST /api/medications/{id}/courses/end resolved the caller with
requireAuth(), so a delegate managing the record got 403 while the
medication PUT that used to carry the end date admits MANAGE, and the
reads still served canEndCourse true. The action now resolves the record
at MANAGE over medications and files the previous end (C4) with the
count of tombstoned placeholders.
The medication reads narrow canEndCourse and canStartCourse to the
caller: End needs MANAGE, starting a course stays owner-only. The list
cell stays per record and the narrowing runs after it. Create and edit
answers pass the grant they were resolved under instead of null.
Registered in the sharing surface guard, the MANAGE handler matrix and
the activity feed's verb lines.
Refs #1280
The on-demand briefing (POST /api/insights/generate) runs through the
briefing chokepoint, which named its chain walk "job" for the background
share of the day's ceiling. Since background work leaves out a provider the
health ledger benched, a person pressing "Re-run analysis" got no request at
all: the call failed with a connection refusal that never happened, and a
refused credential was never reported back.
The chain runners now take the walk apart from the budget surface: a call
can be rationed as a job and still have a person waiting on it
(interactive). runBriefingCompletion takes the flag and the on-demand
route's calls (the first pass and the grounding retry) pass it, so they keep
the job share and walk a benched provider at the tail, as background-bench
promises for interactive walks. Every other job-surface caller is a worker
or is enqueued; the status, narrative and warm routes only read and queue,
and document reads and lab scans resolve a single provider without a chain.
Refs #1279
A reply that ran into the token ceiling with unlabelled text was read as
reasoning the server did not separate whenever the ceiling held room for
reasoning, which is every call that carries a level, Off included. A model
that does not reason at all (llama3.1) lost its own truncated answer: the
call failed as reasoning_only, the briefing never reached its JSON-shape
retry, the Coach named the wrong error, the failure counted toward the
bench, and the client learned to ask with "none" for an hour.
Such text is now reasoning only once the endpoint and model have shown that
they reason: a reasoning field or a think tag in an earlier reply, or a
lesson that says so (knownReasoner). A sighting on a request that asked to
reason speaks only for requests that ask too. Otherwise the text is the
answer, cut off, with finishReason "length".
A lone closing </think> now splits a reply only where the chat templates put
it, at the start of a line, or anywhere once the model is known to reason. An
ordinary answer that mentions the tag in a sentence stays whole.
Two expectations encoded the old rule and move: the Coach loop case whose
first reply is unlabelled template-opened text from a model never seen
reasoning, and the Local case that read a never-seen model's cut text as a
thought. The streamed think-block splitting moved into local-reasoning.ts.
Refs #1276
A gateway that refuses reasoning_effort outright (LiteLLM in front of
Ollama: "does not support parameters: ['reasoning_effort']") in front of a
model that reasons without answering sent requests until something outside
stopped it. The answerless rule learned to ask with "none", the refusal of
that "none" was learned as "send no field" over it, the next request came
back answerless again, and each lesson undid the other.
Each call of the Local and the OpenAI-compatible gateway clients now carries
one request budget through every repeat (request-hops.ts): at most two
requests, after which the call ends with the failure it has. A structural
guard holds both clients to sending only through the budget and to handing
it to every repeat.
The refusal is now learned as what it is: an endpoint that refuses the
field, in front of a model already seen reasoning unasked, keeps both facts
(FIELD_REFUSED_REASONS_REGARDLESS), so it is never asked with "none" again
and keeps its thinking room. An answerless reply from an endpoint that
refuses the field is final. When the refused "none" was the repeat of an
answerless reply, the call fails as reasoning_only, which is what it ran
into. The refusal handling of both clients lives in wire-refusal.ts.
A status run that hits its bound now aborts the call it gave up on instead
of leaving it running detached.
Refs #1279
The ledger stamped last_failure_at, last_ok_at, next_retry_at and
updated_at with the database's NOW(), while every reader compares them
with the app's clock: the runner's skip hints and the background bench,
the admin provider-health card, the server-health tri-state and the
support report. A backoff or pause therefore sat wherever the skew
between the two clocks put it. With the app clock pinned, a five-minute
backoff lasted until the database's clock plus five minutes, months away,
or was over before it began.
Both writes now bind the instant they took as a parameter
(`$N::timestamp`, the naive UTC columns), the same one-clock rule the
summary reads follow. The escalating pause arithmetic is unchanged.
The pause integration test pins the app clock years behind and ahead of
the database's and checks the stored instants, a backoff that lifts after
five minutes, a pause that benches background work for its full length,
and a success the server health reads as recent.
Refs #1279
"End course" now posts to `POST /api/medications/{id}/courses/end`, which
ends the running course at that moment: its last day becomes today on the
person's clock and the instant is stored as `endedAt`, so a dose later today
is not due, not reminded and never missed, while one taken earlier still
counts. Pending placeholders already minted for later slots are tombstoned.
A medication without dates gets the course its window describes, ended now.
End stays its own verb instead of `PUT { endsOn }`: an end date set through
the PUT or the course editor keeps meaning the whole day, and the PUT keeps
accepting it on any course, since moving an end is a legitimate edit.
The list and detail reads serve `canEndCourse`, true exactly while a course
is running, and the action refuses with 409 `medications.course.notRunning`
when it is false, so no client can reopen an ended course or invent a
one-day course from a planned one by ending it. The web controls render End
from the served flag; the contributed tests keep their cases against it.
Refs #1280
A course's end date is a calendar day and counts in full: the doses of that
day stay due, reminded and expected until local midnight. That is right for
an end date set in the editor and wrong for "End course", whose dialog
promises that nothing more is due.
A course now carries `endedAt` (migration 0482, additive, nullable), the
instant it was ended on its `endsOn` day, projected onto the medication next
to `startsOn` / `endsOn` by the one course writer. From that instant:
- the schedule engine emits no slot, so a dose later that day is not due,
not reminded, not projected onto the today list and never expected by
compliance, while a dose earlier that day stays inside the course;
- the course reads ENDED (`courseStatusAt` / `courseStatusOn`), so the dose
actions, `remindersActive` and the reminder gate stop at once, follow-ups
of an earlier dose included;
- every engine context builder, compliance context, dose history, Coach
snapshot, doctor report and the adherence storyline read it, through one
select and one picker for the medication window.
Any write that moves a course's `endsOn` clears the instant, so an end date
chosen in the editor keeps meaning the whole day. Backups carry it in both
purposes and a restore projects it again. `courseStatusAt` moves next to the
other course rules.
Refs #1280
Drives the status-card path the way an idle instance runs it: a data sync
every minute dirtying five cards, the production enqueue with its queue
name, singleton key and retry options, the queue's consumer, the card's
single call and fallback, the status chokepoint, the chain walker with the
health ledger's in-memory twin, the gateway client and the note store's
negative windows. Only the wire, pg-boss, the clock and the unrelated
Postgres reads are faked; the wire answers every request with reasoning
and finish_reason "length".
Against the code before this series the hour counted 151 requests, 150 of
them sent with "none", none charged to the day's ledger, and 300 when the
reasoning lesson did not survive from one job to the next (the pattern
behind the reported log). Now it counts six, charges all of them, sends
"none" once, and logs the pause once per window.
Refs #1279
pg-boss gives a send that names no retry options the queue's defaults:
two retries, at once, without a backoff. A job that reached a model and
failed then ran again immediately, twice, against the provider that had
just failed it. Two model-bound queues still relied on those defaults:
the Coach memory refresh send and the nudge cron. Both now ask for the
same two retries, a minute apart and backed off.
ai-job-retry-bound-guard holds the rule: every send to a queue whose jobs
reach a model names retryLimit (at most 3) and, unless it is 0,
retryBackoff with a retryDelay of at least 30 s; every such cron carries
the same in its schedule options. The queues come from the model-run
sites the expiry guard already keeps complete (now a shared helper), plus
the job files that call a provider directly, which the new guard keeps
complete itself. A planted case runs the checker over sources shaped like
the tree before this change. The queue golden records the nudge cron's
new options.
Refs #1279
The content-index backfill capped a run at 200 documents, but counted
only the ones it indexed. Against a provider that failed every call (a
model that reasons and never answers) the cap was never reached, so the
walk transcribed every un-indexed document in turn, one provider call
each, until the job's time ran out after more than two hours, and the
refunded failures never reached the day's token ceiling.
The cap now counts provider calls, the walk stops with reason
provider-failing after three failed calls in a row (one document the
model cannot read is still walked past), and a failed read charges what
its reply reported spending. The per-document read moves into its own
function.
Refs #1279
A status note whose generation stalled or failed opened a fixed five-minute
negative window, and only the read path honoured it. The ingest
invalidator counted only a stored note as fresh, so every data sync
re-enqueued each note whose last generation had just failed, and the
worker generated it again whatever the window said. With a model that
never answers, every sync bought another round of calls per note.
The window now grows with the generations in a row that ended without a
note (failure_streak, migration 0480): five minutes, doubling per failure,
up to two hours. A written or re-dated note resets it. A failure inside an
open window neither moves nor counts it.
runStatusCompletion waits the window out before any provider work,
whichever path selected the note again (a sync, a warm pass, a queued
job), and the ingest invalidator no longer enqueues a note inside its
window. A derived-score assessment that fails now opens the same window.
The person's own regenerate closes the record's windows first, and saving
the AI provider settings closes them and forgets the provider's health
rows, so a fixed setup does not sit out a pause the old one earned.
The window code moves to status-negative-window.ts; the status parity
golden records that the mood note screened earlier in the same minute is
no longer generated again inside its window.
Refs #1279
Marc-André Bombeck