The Coach read lab results from the last twelve months only, even with
the lookback set to its widest option, so a slow change in LDL or PSA
across years stayed out of reach. Labs are sparse, so they are no
longer held to the one-year window that bounds the dense readings: they
reach back over the whole record, and a lookback limit the person set
still applies.
The snapshot block keeps the latest value per marker, now over that
wider window, at its old size: it shares the prompt's size cap and is
never shed. get_labs with an analyte adds each matching marker's
history, summarised so it stays bounded: the newest five earlier
readings in full and older ones as one mean / min / max / count per
year, ten years at most, older years counted. With a heavy fixture
(30 markers, quarterly for 20 years) the overview stays at 4.3k
characters and a named LDL read grows from 0.2k to 1.2k characters for
80 readings. The MCP get_labs read and the lab resource follow, since
they run the same tool.
The widest lookback option no longer reads "All": it names what it
reads, twelve months in detail and every lab result, in every locale.
Refs #1263
A batch that carried a type with duplicates only moved its
lastReceivedAt, lastTrigger and lastBackgroundAt, so a type whose live
delivery had stalled kept reading as freshly arrived for as long as a
replay, a retry or an older client re-sent its old samples.
The arrival ledger now moves those stamps only for a type the batch
brought at least one new or changed row for. An unchanged stats:
re-post and an hour posted again with its stored value count as
duplicates here; the wire keeps answering updated for them. A
duplicates-only type stamps the new last_seen_duplicate_at column on
its existing row and nothing else. The intake bulk route's dose entry
follows the same rule through the shared recorder. lastSyncedAt and
healthKitLastSyncedAt are unchanged.
GET /api/integrations/healthkit serves the stamp additively as
metricFreshness[].lastSeenDuplicateAt, and the Apple Health card names a
re-send that came after the last arrival.
Refs #1259
The grant expiry case gave the grant 400 ms and slept 500: on a loaded
runner accepting, signing in and switching took longer than the grant
lived, and the grant expired before it was accepted. The grant now runs
30 seconds and the case moves the clock the resolver reads to the
instant the grant names, so the row still does not change and nothing
sleeps.
The key check's start-up probe is bounded at ten seconds and two per
query. Under load a probe query ran past two seconds and the check
answered inconclusive where the data says mismatch. The verdict tests
now run the check with time to finish; the cases that test the bounds
pass their own budget or sampler and keep them.
/api/auth/me 18 to 13, dashboard snapshot 103 to 99, summary 86 to 84,
the 365- and 3650-day pulse series 6 to 5, the day view 48 to 47,
insights health status 60 to 59, correlations 44 to 43 and a 500-row
batch ingest 31 to 28.
Readers that tag a cell as a closed block no longer import the
invalidation module, which several tests replace wholesale. The cache
route test counts the slice's pre-fold cell apart from the route's two
cells, and the provenance test reads the shared pick's ORDER BY over its
grouped rows.
Re-importing an Apple Health export the account already holds sent one
UPDATE per row the insert skipped, about two million statements for a
large export, nearly all writing back what was stored. One read per chunk
now finds the rows whose stored values equal what the update would write;
they count as updated as before, without the statement. A key the chunk
carries twice is written in order as before. The natural-key rescue moves
into the same module.
A long-window read of a dense series sent two statements over the same
canonical rows, the day or hour buckets and the strip's figures, so a
ten-year pulse read picked the canonical source per day and scanned the
readings twice. Both aggregates now run in one statement over the window
materialised once, with the same SQL each had (dense-series-read.ts).
An integration test keeps the two old statements as the reference and
requires the same buckets and figures for pulse (two ranked sources),
glucose and oxygen saturation, day and hour grains, in UTC,
Europe/Berlin, Asia/Kolkata and America/New_York, and the same empty
answer for a type without readings.
A table read without an order comes back in whatever order its last
updates left on disk, so the column fill could pick a different row of a
two-row table from run to run. The audit rows of a restore are now found
by the copy's id instead of by time, since the database clock and the
test's clock do not agree to the millisecond. The golden copies are
written again from the code before the restore and export moved.
Up to 12.37 pg-boss retried a job whose handler outlived its expiry in
the same worker beside the run it had given up on, and the lane test
pinned that as an executable statement. 12.38 fails the job and fires
the signal as before, but the worker fetches nothing more until the
handler returns, and the retry starts after it. The test now pins that:
one run at a time, the retry starting after the overrun ends, and the
lane's handler importing once per delivery with never two at once. The
lane's lock stays, as the guard for a second worker or instance.
The rollup parity case writes over half a million readings and folds them
into DAY buckets: about 35 s on a CI runner alone. Run beside two other heavy
files on the same Postgres it took 1.7 times as long and crossed the 60 s
default. Nothing in the case waits on a lock: its advisory locks are scoped
to the worker's own database, and the measurement with the database capped
at two CPUs (14.6 s alone, 25.3 s beside the sharing matrix and the restore
memory test) shows plain contention. The case gets a 150 s budget, twice the
contended estimate. HEALTHLOG_INTEGRATION_PG_CPUS caps the database
container so a CI runner's contention can be reproduced locally.
The 0412 lock-retry case let go of the lock on a 2.5 s timer. On a loaded
runner the migrate CLI took longer than that to start, so the first attempt
found the lock already free, applied cleanly and never produced the 55P03
the case is about. The lock is now released once the attempt is seen queued
behind it in pg_stat_activity and gone again, which is the moment its own
timeout fired, and the case fails outright if no attempt ever queued. It
holds with the CLI start delayed by four seconds.
POST /api/admin/backups/upload stores a backup file as a new copy, the
copy whose id the published restore takes, and it refuses a file for
reasons an operator has to be able to look up: a key this server does
not hold (backup.key.missing, with the key ids), an owner account that
does not exist here, a newer schema, a copy too large to store
(backup.upload.too_large). It was exempted from the contract as console
plumbing while its two siblings were published for exactly that kind
of refusal.
It is registered with the three bodies it reads (raw JSON, raw gzip, a
small multipart form) and every status it answers; the summary is the
three identity fields plus one integer count per class, so a class the
backup format gains later does not break a decoder. The round-trip
test calls it through the contract now.
The six /api/auth/me scalar writes (unit preference, glucose unit,
automatic document reading, local lab OCR, workout routes, the central
Codex switch) read their body through safeJson with a 1 KB cap, so each
answers 415 for a body that is not JSON by content type and 400 for one
that does not parse, beside the 413 they already declared. Their
descriptions said so and the responses did not. All three now come from
one block.
The modules and disable-coach route tests had been skipping the
contract check over a 400 and a 409 the registry already declares; the
skips are gone.
A /api/fhir/* handler answers its own refusals as FHIR: a bad search
parameter is a 400 OperationOutcome, the hourly fhir bucket a 429
OperationOutcome (issue code throttled), both served as
application/fhir+json, the same way the 404 on a read already was. The
contract declared no 400 and the generic JSON 429 for the 429, and no
403 for the doctorReport module gate, which every data route runs.
The searches now declare the 400, every data route the 403
module.disabled envelope and the FHIR 429 with the rate-limit headers
apiHandler attaches to it. The module gate keeps answering the JSON
envelope: it is the errorCode every client already branches on for a
disabled module, and it runs before the FHIR face does. The route tests
drop their skips and assert the 429's content type and Retry-After.
requireAuth answers 403 auth.scope.insufficient to a narrow Bearer token
on a route that names no scope it holds, and requireAdmin answers 403
auth.admin.required to a session that is not an administrator's. Both
run before any route body, and the contract only said so in the prose
of the shared 401, so a client generated from it had no typed branch
for either. The route kit accepted the scope refusal on any operation
that declared a 401 for the same reason.
The 403 is added to the assembled path table rather than to
stdResponses. That set is spread into most operations, about three
hundred of them carry a 403 of their own, and most spread the set after
it, so a shared 403 there would have overwritten every module-off and
sharing refusal in the document. withScopeRefusal adds the shared
response where an operation has none and prefixes an operation's own
403 with the gate's code, keeping the sharing sentence last. Operations
that take no credential, and the three surfaces that authenticate by a
credential of their own, are left as they were.
The kit no longer tolerates the refusal on an operation's behalf, and a
guard holds every operation behind the gate to declaring it.
The phone re-sends the same daily totals on every sync. Each went through
the reconciler, about ten round trips in its own transaction, to write the
same values back, and as an update it started the DAY refold, the rollup
jobs, PR detection and a cache stale for the account.
One read over the batch's stats: ids now finds the rows whose live stored
row already holds every value the update would write; those are reported
updated exactly as before, per entry and in the summary, and nothing is
written or started for them. A changed column, a tombstoned row, a key
that occurs twice in the batch or a value the plausibility gate refuses
still goes through the reconciler.
The shared canonical-source CTE sorted every raw reading in its window by
a CASE expression no index can supply; on the benchmark account the pulse
pick over ten years sorted 260,000 rows and spilled to disk. Its order
depends only on (type, source), so the pick now groups the window to its
distinct (type, day, source) triples first and runs DISTINCT ON over
those. The picked source per day is the same by construction; an
integration test runs the old statement beside the new one over a fixture
with overlapping sources, sources off the ladder, tombstones, empty days
and readings either side of midnight, under default and reordered ladders,
for every reader shape, and requires identical rows.
Every app session now sets jit=off. Postgres compiled the larger folds to
machine code, which took 100 to 140 ms of a 220 ms statement and saved
nothing for statements that run once and are bound by reading rows. JIT
never changes a result.
The SETTING verdicts are now the keys of the codec table, so a column
becomes a setting by getting a codec and nothing else has to name it. The
verdicts for every other User column move beside it into
account-settings-classes.ts, and the restore half into
account-settings-restore.ts. The backup plan keeps the model verdicts.
The classification guard now also refuses a column that has both a codec
and another verdict, since the composed table would quietly carry it as a
setting.
Seed one account as far as the schema allows (a row of every two-ended
model, every column of those rows and of the account row, the instance
settings, a sharing grant), export it both ways and restore each file
into a fresh account, over the live account, and from a version-1 file
in the old shapes. The files, counts, parsed payload, preview, restore
outcome, reported phases, audit rows, wide-event fields and every row the
account holds afterwards are compared with golden copies after ids,
ciphertext and run-time timestamps are normalised. Refusals, skip reports
and the schema's issue list for malformed files are pinned the same way.
The seed moves out of the round-trip test into a shared fixture module
so both files fill the same account.
The environment check pinned the database a worker talks to as the single
shared one. Each worker now runs against its own clone, so the check names
the clone and asserts it is not the migrated template the clones copy.
Vitest's --shard cut the file list into slices of equal length, which gave
the integration shards 954 s to 1 449 s. A sequencer now deals files out by
their measured duration, plus a fixed per-file cost for import and setup,
longest first to the shard with the least work, and starts the longest files
first inside a shard. Timings live in tests/durations/*.json and are
regenerated from Vitest JSON reports with scripts/test-durations.mjs; every
CI shard uploads its report.
Integration shards run three workers each, one cloned database per worker.
The unit suite runs as three projects decided per file from its source:
guards and unit share a module graph inside a worker, isolated keeps a fresh
one for every file that mocks, spies, stubs or resets. In CI it is three
duration-balanced shards next to a job for lint, format, types and the
OpenAPI drift check.
The required check names stay as they were. "Lint, Typecheck & Test" is now
the job that answers for the static checks and every unit shard, the way
"integration" already answers for its shards. The browser-install guard
recognises a direct, flags-only vitest run as a run of the unit suite.
tests/integration/query-counter.ts counts the statements the app sends
while a function runs, by wrapping Client.prototype.query on the copy of pg
the Prisma adapter itself resolves (so a second pg in node_modules cannot
leave it reading zero; a window with no statement throws). Background tasks
are settled before the window closes, so fire-and-forget writes count with
the request that started them.
query-budget-hot-routes.test.ts seeds a small account (four months of the
main types with overlapping sources, stats rows, sleep, glucose, a
medication, workouts and mood), warms each route once, evicts the user's
caches and counts the next call. Budgets, measured on this fixture:
/api/auth/me 18, dashboard summary 86, dashboard snapshot 103, pulse series
30/365/3650 days 6 each, day view 48, insights health status 60,
correlations 44, a 500-row batch ingest 31. A failure prints every
statement grouped by text.
Proof: one extra `SELECT 1` before every handler in apiHandler failed all
ten budgets by exactly one (18 → 19, 86 → 87, … 31 → 32), which also shows
the counts are deterministic; with the probe removed all ten pass.
scripts/bench/seed-large-account.ts (moved from scripts/, extended) writes
one synthetic account of ten years: 1,265,561 measurement rows across the
main types and five sources (Apple Health dense spot rows, Withings blood
pressure and scale, a weekly manual weight, three years of Oura nightly
values, 90 days of Nightscout CGM), `stats:` and per-sample cumulative rows,
nightly sleep stages, 3,020 workouts, 1,095 mood entries and three
medications with 3,285 intake events. Everything goes in through set-based
INSERT … SELECT generate_series; values and timestamps are functions of the
row index and `--anchor`, so a run is reproducible. The rollup tier is
folded with recomputeUserRollups, as the boot backfill does, and the tables
are analysed. About 80 s on the Mac mini.
scripts/bench/run.ts boots the production build once with every role (so
the queues exist), then measures against a web-only process under the
production heap ceiling: p50, p95 and max latency and SQL statements per
request (pg_stat_statements, read once nothing is in flight and the count
holds still) for /api/auth/me, the dashboard summary and snapshot, the pulse
series over 30, 365 and 3650 days, the day view, insights health status and
correlations, each cold (caches evicted by a one-row manual-sync ingest
before every request) and warm, plus a 5,000-row batch ingest as ten
requests of 500. It writes JSON and a markdown table.
docs/contributing/benchmarks.md describes the setup against a throwaway
Postgres in Docker with the production pg_stat_statements settings.
tests/quarantine/quarantine.json is the only way to take a flaky unit or
integration test out of a run. An entry names the file, the full test name,
the issue, the reason, the day it was added and the day it expires, at most
14 days later. Both setup files skip a listed test while its entry is live,
reported as skipped with its issue rather than retried into green. Past its
expiry date the test runs again, and quarantine-guard.test.ts fails until the
entry is removed; it also fails on an entry naming a file or test that does
not exist. The list starts empty.
fast-check properties with one shared configuration (tests/property/): a
fixed seed on every pull request, a fresh one and more cases at night through
FC_SEED and FC_NUM_RUNS, and a printed seed with the shrunk counterexample on
failure.
- Time zones: wall clocks agree with Intl in twenty awkward zones, a wall
clock maps back to an instant showing it, a skipped hour resolves forward
on the same day, every instant lies inside the window of the day it is
filed under, and day keys shift and count as inverses.
- Statistics: every median in the tree says the same number, medians and
spreads obey order, shift and scale, a least-squares fit recovers an exact
line and composes from stored sums exactly, and the segmented regression
recovers a noiseless interrupted line.
- Cadence and compliance: weekly and daily schedules expect exactly what a
day-by-day oracle expects, in any zone and on transition days, and the
rate stays within bounds and never falls when a missed dose is taken.
- Encryption: every codec opens what it sealed under the key it names,
rotation moves the key id and nothing else, and a tampered value never
opens to anything but the sealed bytes.
- Backups: randomly generated accounts come back from a disaster-recovery
restore with an identical canonical dump of every generated table.
Phase config, cadence, side effects, inventory, intake purge, bulk delete and
the import status run through the real handlers against the database. The
advisory-lock ordering the mocked tests read off call order is shown with a
real concurrent transaction holding the inventory lock, cache eviction through
the next list read with a control that the cached list would otherwise stay
stale, and soft deletes on the tombstone and sync counter they leave.
The channel settings routes, their test-message routes, the reminder
thresholds and the workout detail, delete and batch-scope tests run through
the real handlers against the database. Only the far ends are faked (the
relay, the ntfy server, the Telegram Bot API, SMTP), and the tests assert what
those received. The private-origin source guard stays a unit test and now
points at the two files that carry its runtime proof.
The effective tag read, hide and show, custom tags and groups, and the tag
layout are tested through the real handlers on stored rows instead of a
mocked Prisma client: the migration-seeded catalogue served after a truncate,
owner scoping that leaves another account's tag byte-identical, the caps,
sealed labels, soft delete against purge, a deleted group re-homing its tags,
and the layout's optimistic token from 200 through 409 to a successful retry.
The Strava writer had eight mocked test files and no test against a
database. strava-sync.test.ts runs the real sync behind a fake Strava API at
the safeFetch boundary: field mapping, the after= watermark, idempotent
re-sync on the (user, source, external id) key, in-place overwrites that keep
what only the detail read carries, a cursor that never moves backwards or
past a row that failed to write, the paged backfill with its resume point
and requeue, token refresh with both rotated tokens stored sealed, and the
integration ledger after transient and permanent failures. The sync, the
disconnect and the credential routes are covered through the kit.
The mocked cases that asserted call counts and mock ordering are removed;
the pure retry-after parsing stays a unit test.
Fourteen route tests for the /api/auth/me preference family replaced Prisma
with a mock and asserted the calls it received. Their claims are now shown
through the real handler on the stored row and the response: validation
refusals with their codes, normalisation and defaults on read, optimistic
lock conflicts against a real second writer, rate limits driven to their
first refusal in the Postgres limiter, audit rows read back, cache entries
evicted, and the jobs a change queues observed on a real pg-boss queue.
Assertions that were only about the shape of a Prisma call are replaced by
the behaviour they stood for, and two that described a state only a mock
can reach (a missing column) by the stored value that state resolves to.
DELETE /api/auth/me/sessions/{id} takes the session's public handle, an HMAC
of the row id, and compared that handle with the caller's row id to decide
whether the caller signed out the session it was using. The two never match,
so every revoke was audited and annotated with self: false, including the
ones where a person ended their own session. The handle is compared with the
handle of the caller's session now.
The mocked route tests for the session list and revoke, the device list and
revoke, the security-activity feed and the FHIR surface move onto the kit.
Each claim is now shown on the stored rows and on the next request: a
revoked session, device login or token is refused when it is presented again,
another account's session or device answers 404 and stays signed in, the
current-session and current-device flags follow the credential that anchors
them, and no row id, full IP or push token reaches a response.
The FHIR read set, covered against a database only by the scope check until
now, is driven with seeded readings and medications: every resource carries
the seeded values, another account's rows never appear, a fhir:read token
reaches every data route and an unrelated narrow token is refused, the module
gate closes all eleven data routes, and $everything serves the same entries
as the document builder with references that resolve inside the response.
The module-gate inventory now names the new file as its behavioural proof.
One table holds an entry per write route, and every User column the backup
classifies as a setting appears in exactly one entry; a new setting column
fails the inventory case until it has one. For each entry the file sets a
non-default value through its write route, reads it back through its read
route and through /api/auth/me where that publishes it, and checks the stored
column moved off what a fresh account holds.
The backup leg runs through the routes too: the portable full backup a
person downloads, the operator's upload, and the restore into an account
recreated under the same id. Every setting must read back identically on the
restored account, through the same routes and the account payload, and every
stored column must come back with its value.
tests/kit/ holds typed fixture factories and callRoute(handler, options).
The factories write through the application's Prisma client, so a fixture has
the shape production stores and the compiler checks it; sessions and Bearer
tokens are minted by createSession and issueApiToken, so a fixture cannot
drift into a shape authentication no longer accepts.
callRoute builds a real NextRequest, fills the next/headers stand-in from the
auth fixture and runs the exported handler, apiHandler included. Every call is
also a contract check: the status must be declared for that path and method in
the OpenAPI registry, a JSON body must parse with the registered schema, and a
key the schema does not declare fails the call. The scope gate's generic 403,
documented on the shared 401, is the one answer accepted without its own
entry. kit-contract.test.ts proves each refusal can fire.
The token list and revoke move onto the kit. The mocked unit test asserted
the list's where clause and proved the API switch is not consulted by reading
api-handler.ts for a string; the kit test authenticates a Bearer token while
the switch is off, revokes it, and shows it refused on the next request.
Global setup migrates one template database and reads what a fresh install
holds: its schemas, its tables and the rows the migrations seed. Each Vitest
worker clones the template on its first file with CREATE DATABASE ... TEMPLATE
and keeps the clone for the rest of the run, so files run in parallel without
seeing each other's rows, advisory locks or queue schema. Schemas created at
runtime (pg-boss, migration probes) are dropped when the next file starts.
The truncate list is the catalogue now, every table in public except the
migration bookkeeping, instead of a hand-kept list of 42 of 166 tables. The
seeded catalogues are put back after each truncate in foreign-key order, so a
test starts from what a fresh install has. The settings singleton is the one
seeded table a test starts without, declared with its reason.
next/headers and the boot compatibility probe are mocked once in the setup
file. Two lock-wait probes now look at their own database only, since other
workers' backends share pg_stat_activity.
The free-text backfill no longer has an insights cache step since the
readable cached-text column was dropped, so its summary carries no
insightsCachesMigrated count. The expected summaries follow.
GPS tracks are kept only after the account turns storeWorkoutRoutes on.
The two route-storing cases opt in first, and a new case proves that an
account which has not opted in keeps the workout, answers it as
inserted, and stores no track.
The two provenance reads take required query parameters and had no
entry, so the run could not call them. Give them a day and a range on
the steps, seed a second steps source for the last week so the pick and
the comparison have two sources to explain, and save a labs list order
so the module list layout read answers with a row.
The restore admits activity goals key by key against the goals schema,
so the generic { roundTrip } JSON fill was refused as an unknown goal
rather than carried. Fill the column with real goals and pin the
per-goal admission (valid goals kept, unknown keys and out-of-bounds
goals named) in the unit suite.
The medication CSV import created the medication and then set its
category in a second write. A crash between the two left a medication
without a category, and a retry skipped it as a duplicate, so it never
got one. Both writes now share one transaction.
Every accepted row was its own insert, so a 20 000-row file meant
20 000 round trips and an import cut off part way left an arbitrary
prefix behind. Each 200-row chunk is now one createMany: it lands whole
or not at all, and uploading the same file again skips the written
chunks as duplicates. The importers report each committed write, and
the audit row is written in a finally block, marked partial with the
rows written when the import did not finish.
The rows were parsed in the query's timezone while the duplicate keys
used the account's, so two draws on separate days of the file's zone
could collapse into one, or one day split in two.
The health-status read scanned five vitals over 90 days for a dated,
sustained level shift. It now scans every vital with a stable daily level
(adding RMSSD, diastolic pressure, pulse, body fat, respiratory rate,
oxygen saturation, walking heart rate and VO2 max) over 180 days, reads one
value per day per metric, leaves a metric unread when its module is off,
and reports how many days stand on each side of the break. The Coach keeps
its five-metric, 90-day scan unchanged.
Marc-André Bombeck