The personal-record pass, every half hour for every account, read every
live row of each cumulative type into memory to find the best day. With
784 000 active-energy rows on one account (the per-sample rows pile up
when the nightly drain cannot finish, #1031) it ran a 524 MB heap out of
memory; the old code needed 1.1 GB of heap for that account.
It now reads the type in bounded keyset pages and handles one local day
at a time. The canonical-source pick it applies is per day, so the day
totals and the winner are the same; a tie still keeps the earliest day.
Measured on the same account: 189 MB peak heap, the same record. The
warm-up gate counts only up to its threshold instead of counting a whole
history.
`iterateSourcePages` and `iterateDayBuckets` in the consolidation base
take the row type as a parameter, and the day grouping can keep every row.
An account of 1.25 million measurements could not export its data or be
restored from any backup (#1031). Measured in a 1 GB container:
- The full export and the passphrase export built the whole payload and
its JSON string in memory; the app ran out of heap and restarted.
- The weekly backup's JSON is 662 MB, longer than any string V8 can hold,
so `unpackBackupBlob` threw on every read. Restore, download and the
restore preview all answered "could not be decrypted".
- Upload accepted 10 MB, a fraction of a large export.
Every reader and writer of a backup file now streams it:
- `scanBackupJson` reads a backup's bytes once, keeps every section but
the named bulk arrays, and hands their elements out one at a time.
`readStreamedBackup` uses it twice: a first pass that validates each
measurement against the element schema and keeps the rest of the file,
and a second pass that hands the measurements to the writer in batches.
Stored copies and off-host objects are authenticated whole on their
compressed ciphertext before any plaintext is read; only the
decompression is streamed.
- Restore writes the measurements with one `INSERT … FROM unnest` per
batch. Prisma caches a query plan per `createMany` row count, and one
1 000-row plan kept about 145 MB alive, which alone put the restore past
the heap limit. The transaction budget grows with the measurement count.
The awards and ECG sections are handed the ids they reference that were
written, not every written id.
- The export routes write the file into the response as it is produced.
The passphrase export spools its ciphertext to a temporary file because
the HLX1 format puts the tag first; the format is unchanged, so existing
archives open as before.
- Upload accepts the raw body, JSON or gzip, up to the 512 MB request
ceiling, checks it as it arrives and stores it in the same pass; the admin
page compresses the file in the browser before sending it. Multipart stays
for small files.
- Download streams the stored JSON; the preview counts measurements from
the stream. The monthly restore drill and `scripts/restore-backup.ts`
read the off-host object the same way; the script can write `.json.gz`.
On the seeded account in a 1 GB container: export 25 s at 389 MB peak,
passphrase export 25 s, both uploads 19 s, restore of the weekly backup
94 s at 554 MB peak, and export, upload and restore round-trip to exactly
the same rows. The encrypted-column guard now also recognises the raw SQL
statement that writes `DataBackup.data`.
`scripts/seed-large-account.ts` inserts about 1.25 million measurements
for one user, shaped like an Apple Health export plus the iOS sync: dense
heart rate, HRV and SpO2, the high-frequency mean types, export `stats:`
daily rows, recent per-sample cumulative rows and nightly sleep stages.
Everything is generated in Postgres, so a run takes seconds, and it ends
with ANALYZE. It is how the maintenance jobs in #1031 were reproduced and
measured; never point it at a database holding real data.
The backup streamed its JSON through gzip and the cipher, but kept the
stored copy as one string and handed that to Postgres in one parameter.
On a seeded account of 1.25 million measurements the copy is 64 MB of
base64, and the pieces, the joined string and the driver's copies of it
grew the process from 309 MB to 946 MB resident. A bare process needed a
heap limit above 400 MB to finish it (the new path finishes under 220 MB).
With the dense retention running beside it in a 1 GB container the
process died of heap exhaustion, and the backup job sat "active" until its
two-hour limit marked it "job timed out" (#1031).
`packBackupBlobInto` hands the stored copy on in pieces of up to 4 MB as
it is produced, and `storeBackupBlob` writes them into a temporary table
on its own transaction and assembles the row with one `string_agg`. The
process never holds the whole copy. The account's previous copy stays in
place until the new one is complete: a failure rolls back to it.
Measured on the seeded account in a 1 GB container, web and worker in one
process: the backup completes in about 30 s with the dense retention
running beside it, where v1.39.0 crashed within 30 s.
The streamed JSON also reports its progress to the job every 50 000 rows.
pg-boss marks a job "job timed out" at its expiry and fires its abort
signal, but does not stop the handler. A drain that could not finish kept
writing after its job was declared dead, and the retry then ran beside it
over the same days.
The nightly drain tick and the per-user dense-retention and daily-mean
jobs now stop between days once 75 % of the job's expiry has passed, or
as soon as pg-boss aborts the job, and complete with `stopped_early`
instead of failing. What they folded is committed. A nightly tick that
stopped early after folding something sends itself again a minute later,
up to 24 times, so a first-time backlog after a large import drains the
same night; a run that folded nothing does not hand on, so a pass that
cannot move does not loop.
The expiries themselves stay at their defaults: with the passes bounded
per day and budgeted, a longer limit would only delay the signal.
The consolidation drains (cumulative, daily mean, dense intra-day
retention, legacy steps) gathered a type's whole history into one array
before folding the first day. On an account of 1.25 million rows the
heart-rate array alone held about 230 MB, and the dense retention peaked
at 460 MB of heap against a 524 MB limit; in a 1 GB container running the
web server and the worker together the process died of heap exhaustion
within a minute and the job was later marked "job timed out" (#1031).
- Pages are now handed on one at a time and each local day is folded as
soon as its last row has been read, so a pass holds one page and one
day.
- The keyset cursor gains a redundant `measuredAt >= cursor` bound.
Without it Postgres could not use the cursor as an index condition and
re-read every earlier row on each page (47 ms and 248 000 filtered rows
for one page halfway through the heart-rate history; 5 ms now).
- The dense fold resolves a day's hourly rows with one lookup and one
insert instead of two lookups and a write per hour. On the seeded
account the pass went from about 290 000 statements to 54 000.
- `shouldStop` lets a caller end a pass between days; the result says
`stoppedEarly`. Days commit one by one and a folded day leaves the
scan, so the next run continues at the first day not reached.
A job that ran into its pg-boss expiry left no trace in the app log. Most
handlers annotate a wide event nobody emits, pg-boss records an expiry only
in its own table, and a worker that dies of heap exhaustion writes one
FATAL line that names no queue. On a large account three queues failed
with "job timed out" for days and the log said nothing about them (#1031).
Every handler now runs inside `observeJob` (through `runJob`), which writes
`job.progress` every two minutes while a handler runs (elapsed time, heap,
RSS and the last progress the pass reported), `job.expired` when the job
passes its expiry while the handler is still running, and `job.finished`
when a long or expired run settles. At worker boot, `worker.boot.jobs_cut_off`
names every job a previous process was still running when it stopped.
The admin card's "job timed out" now says what the limit was and that the
cause is either a run too slow for it or a restart while it ran.
The cycle delete now runs the shared removal, which finds live starts only.
A second delete of the same cycle read nothing to remove; the route now says
so explicitly and answers 204 before the transaction, and the tombstone the
first delete left stays as it was.
The live daily fold ranked every row by source before folding, which sorted
the whole window twice. A year of per-minute heart rate (525,600 rows) took
about 0.55 s per series on Postgres 16, with both sorts spilling to disk.
The fold now aggregates per source and bucket in one hashed pass and then
keeps each bucket's ladder-canonical source. Folding the canonical source's
rows is the same as folding that source's group, so the numbers do not
change; the same year reads in about 0.29 s. A new integration case pins the
canonical pick, mean, count and spread for a day with three sources.
A start logged within one period before an existing start folds that start
into itself and soft-deletes it, since the two are one period entered out of
order. Nothing recorded which start was folded, so taking the earlier start
back could not bring the later one back: a mis-tap on the 5th with a real
start on the 10th, then deleted, left the 10th gone and the cycle before it
reopened.
Migration 0345 adds menstrual_cycles.absorbed_into_id. The folded row names
the start that absorbed it; removing that start restores it, re-derives its
own end and hands its days back to it. The column is needed because a
soft-deleted row alone cannot tell a folded start from one the person
deleted, and the audit trail is not state a removal can read. Deleting a
cycle directly now runs the same shared removal as deleting its day-log.
The measurement form, and with it the add menu's Measurement entry,
listed sleep and glucose with those modules switched off. It now filters
its types through the surface map's type ownership, which a test holds
equal to the ownership the server gates measurement types on, and a
default or last-used type that is switched off falls back to the first
offered one.
The runtime accepts "+05:30" as a zone id, so isValidTimezone let it through,
but Postgres reads a bare offset in AT TIME ZONE with the POSIX sign (east
of UTC is negative), and the day-grain SQL aggregates would cut every day on
the wrong side of UTC.
isValidTimezone now accepts named zones only: whatever the runtime resolves
to a bare offset is refused, while named zones, their aliases, UTC and the
Etc/GMT family stay accepted. Every write that already validated through it
refuses offsets now, and the screener and mood linked-context request schemas
validate their zone too. A value stored before this change can still sit in
the database, so the SQL call sites that read the stored zone directly (the
dashboard summary, achievements, the doctor report, the tiered AI context)
fall back rather than splice it, as the resolver and the day aggregates
already did.
For a cycle-tracking account the strip painted Strain first and swapped
it for the cycle ring once the calendar read landed, moving the row
under the reader. The ring's decision now reports while it is pending,
and the strip keeps its skeleton until then.
Settings → Coach's sources pointer and the About-me follow-up questions
linked to /coach whatever the Coach's state. Without a provider or
consent the Coach page sends the visitor straight back to Insights, so
the link only bounced. Both now follow the Coach capability; the
questions themselves stay readable.
The confirmation said the AI features stop straight away, and the
withdrawn line that no health data goes to an AI provider. The Coach and
Insights need the receipt only when they use the AI access the server
provides, so on an own key or a local model they keep working, and the
model-written notes are kept while another consent still covers
Insights. Both strings now say so in every locale, and a guard pins the
copy next to the resolver rule it describes.
The aboutMeQuestions capability admits an ai_insights_only receipt, but the
question generator ran its own gate that demanded a Coach receipt, so a
person whose capability read available still got the fallback questions on
an operator-held chain.
The generator now runs the shared wire re-check (aiEgressRefusal) for the
aboutMeQuestions key, so the receipt kinds and the rule for when one is
needed come from the capability table alone.
Migration 0343 moved the operator's Coach answer from module availability
into the Coach switch and folded every live row. Restoring the instance
settings from an older file wrote the dead coach key back into module
availability and left the Coach switch on, so a Coach the operator had off
came back.
The restore now applies the same fold: coach false in the file's module
availability switches the Coach off, and the key is dropped either way.
The visit form's lab and episode pickers fetched with their own page
sizes under the same keys as the Labs list and the illness pages, which
fetch with others. Whichever read landed first served the other: the
picker could see 50 episodes instead of 100, or the Labs list a 200-row
page as the whole list. The picker reads now have keys of their own,
under the lab and illness roots so writes still evict them.
Three assertions in the AI-optional overview spec named slots that no
longer exist anywhere in the tree, so they held whatever the page did.
They now name the error primitives the overview paints on a refused read
and the Coach hand-offs that paint only while the Coach is available, and
the no-provider journey checks that /coach goes back to the overview.
A consent withdrawal purges the stored status notes, briefing, reaction
lines and workout paragraphs in its own transaction, but a provider call
that was already in flight still persisted its reply afterwards, bringing
the text back under a consent that no longer existed.
The consent rule now runs once more after the reply arrives, against the
provider that actually answered and with a fresh receipt read: in
runStatusCompletion (status notes, workout paragraphs, narratives, derived
assessments), before the briefing commit, and before the reaction-line
commit. A refused reply is dropped; the spent tokens stay on the ledger.
A dose or a visit can link documents from its own form, and a
document's sheet can link doses and visits; both ends write the same
table, and the sheet writes the whole set on every tap. Dose and visit
writes left the document reads alone, and the sheet seeded its pickers
from a detail cached for five minutes, so the first tap there after
linking from the dose's form wrote a set without the new link and
removed it.
Dose and visit writes now evict the document root, the sheet reads the
document again on every open, and its pickers wait for that read before
they offer a change.
With no provider set up, a fresh self-host's default, the Coach module
is on but /coach redirects to /insights because the Coach is not
available. The tour read the raw module map, kept its Coach stop, pushed
/coach, was sent back to /insights and pushed /coach again on the
pathname change: a loop.
The tour now reads the same capability-folded map as the navigation, so
the Coach stop is not in the list. And a stop whose page redirects is
dropped after one navigation, in the direction the person was going,
instead of being pushed again.
The daily sweep and the measurement-ingest hook surfaced due reminders into
a Coach conversation whatever the record's coach capability said, and the nag
cap then dismissed them after three ticks although no badge ever showed. The
app-open path already waited on the capability.
surfaceCoachReminders now holds every row while the coach capability is
unavailable (the same answer that keeps the badge dark), and the nag cap
neither charges nor dismisses a held user's surfaced rows. They surface on
the first tick after the Coach is back. The integration fixture gives its
users a provider and consent, and a held case covers the sweep, the nag cap
and the ingest hook.
Every AI surface reads its capability from the ai block on
/api/auth/me, but setting up or removing a provider, signing in to or out
of Codex, saving the fallback chain, the central-Codex opt-in, the
local-OCR switch, a minted consent receipt, the admin's server key and
central Codex, and a guardian's save inside a managed record evicted only
their own reads. The surfaces stayed on the old answer until a reload.
All of these writes, and the ones that already refreshed the payload
(consent, modules, Hide Coach, the operator's switches), now evict one
bundle, aiInputDependentKeys. A structural test holds every client write
to those endpoints to it.
A background job opens one wide event for its whole run, and the per-request
memo cached the operator switches, module availability, module preferences,
provider presence and consent kinds on it. The nightly status passes loop
over every candidate inside that one event, so a switch turned off mid-run
did not stop the remaining users, and the wire re-check read the same cached
answer.
memoizePerRequest takes a freshInBackground option; every input that decides
whether AI work may run sets it, so a background caller reads them on every
call while a request still reads them once.
The page gate matched /coach/plans and /coach/conversations to the Coach
module by the nav prefix, so with the Coach switched off by the operator
or hidden by the person both pages were replaced by the module notice and
never reached their read-only mode. Both are now stored-record pages the
gate leaves alone.
With the Coach off nothing in the navigation led there either. Settings
now carries a plans card, shown when plans exist, and the conversations
card links to the full list.
Reminders are the person's own record, like the stored facts,
conversations and plans. The list and delete routes no longer ask the
Coach module; creating, confirming, resolving or rescheduling one stays
Coach use and is refused by POST and PATCH as before.
Settings -> Coach now always mounts the reminders card (shown with Hide
Coach on only when reminders exist), and Settings -> AI lists them through
the stored-memory block while Settings -> Coach is out of reach. With the
Coach unavailable the card is read-only: every reminder can be read and
deleted, while keep / done / dismiss are not offered.
Plans are the person's own record, like the stored facts and
conversations. The list and delete routes no longer ask the Coach
module, and the plans page turns read-only instead of redirecting to
Insights: every plan can be read and removed, while confirming or moving
one stays Coach use and is refused by the PATCH route as before.
Adds the AI-optional journeys: the overview with AI switched off by the
operator and with no provider (no AI requests, no refusals, one setup
hint, remembered dismissal), and on an account of its own, withdrawing AI
consent deletes the stored briefing, and a hidden Coach's conversation is
read and deleted from Settings, AI.
Specs that drive AI surfaces against mocked routes now serve an available
capability map through /api/auth/me, since the web no longer assumes AI
is on until told otherwise.
The AI consent card turned any failed read of the receipt (a rate limit, a
network drop) into "Withdrawn" and offered a grant. It now says the read
failed, with a retry. The Coach history note is one sentence.
The scores strip reserved a cell for the cycle ring, and dropped Strain to
make room, whenever the cycle module was on. Without an active cycle the
ring renders nothing, so the row showed a gap where Strain used to be. The
overview now asks the same calendar read whether there is a dial today.
The web no longer reads GET /api/feature-flags: useFeatureFlags and its
query key are gone, and use-ai-capability gains the two readers the Coach
pages and the operator notice need. A structural guard fails when web code
reads the retired projection, and when a module requests an AI route
without being registered with the capability that gates it.
Adds the copy for every new state in all seven locales and drops the
per-card no-provider strings.
Settings, AI says which AI features the server's operator switched off and
shows the one setup hint while no provider is configured. The consent
withdrawal says what it deletes and what stays, and a document-reading
consent on file gets its own withdrawal. Regenerate follows the briefing
capability and the connection test names an operator refusal.
The Coach memory (facts and stored conversations) is never gated on the
Coach: Settings, Coach always shows it, and Settings, AI shows it while
the Coach page is out of reach, whenever rows exist.
Onboarding offers AI only where it can be set up here: the done-screen
panel and the getting-started row go away while the operator has AI off
or inside somebody else's record.
Reading a document, scanning a lab report and turning a medication
description into a schedule follow documentAi, labsOcr and
medicationExtract. Unavailable, the action is not offered. With only the
document-reading consent missing, the place that reads asks for it and
grants exactly that (ai_extraction), nothing wider.
Refusals are read by meta.errorCode: a 403 is no longer always called a
missing consent, and a spent daily budget no longer shows the rate-limit
sentence.
The daily briefing, the hero's briefing paragraph, the trend sentences and
the status notes follow the briefing and statusText capabilities: hidden
while unavailable, never an error card or a per-card "connect a provider"
state. The advisor and status reads only fire while their capability is
available; a 403 settles as an outcome instead of throwing.
A failed comprehensive read no longer replaces the overview, and every
overview section is fenced so a section that throws becomes one error row.
The period review always renders its composed narrative. Cycle, breathing
and lab-change sections go through the surface map. One calm setup hint
appears on the overview while no provider is configured and one could be.
Signals, rhythm events and the mood patterns card say a failed read with a
retry, the score anatomy no longer calls a failed read insufficient data,
the catalog marks only the rows whose read failed, and workouts are not
probed while the module is off.
The floating button, the drawer, the launch pills, the nav entry and the
Coach pages read the coach capability from /api/auth/me instead of
combining the operator switch with Hide Coach on their own. A missing
provider or consent now hides the Coach too, instead of opening one that
can only answer with an error.
Stored conversations stay reachable: while the Coach is unavailable the
conversation list turns read-only, opens a transcript in place and keeps
delete. Refusals before the stream opens are read by meta.errorCode and
each code has its own copy.
The connect and sign-in hand-offs moved from window.location writes onto
the loadDocument helper, and the guard stopped seeing them. Its reference
floor caught the drop. Watched red with a fenced thumbnail route handed to
loadDocument.
The provider mock lacked `probeProviderChain`, so the capability loader
threw on every wire and read `check_failed`. It now answers "no
provider" like the rest of the mock, and the derived payload pins
`no_provider` so a starved loader fails the file.
The bulk-delete, restore and by-external-ids suites mocked
`invalidateStatusInsightsForTypes` on the comprehensive-generate barrel,
while the shared post-mutation tail imports it from status-invalidation.
The mock was never called, the real re-warm ran the capability loader
against a stub database, and the "no invalidate" assertion could not
fail. The mock now sits on the imported module; the positive cases
assert the re-warm with the distinct types, the zero-row cases assert
its absence.
A structural check over src/ and tests/: every `vi.mock` / `vi.doMock`
of `@/lib/feature-flags` either uses the shared switches factory or
names `loadAssistantSwitches`. The matcher is whitespace-tolerant,
asserts a non-zero match count, and checks itself against planted
one-line, split, automock and `import()` forms.
Eight files mocked `@/lib/feature-flags` with `getAssistantFlags` alone.
The capability loader reads `loadAssistantSwitches`, so wherever it ran
it threw and every capability resolved to `check_failed`. A shared
factory now points both readers at the function the test drives.
Only the config-change chain reached the loader. Its fixture means AI
switched off by the operator; it now supplies the loader's other inputs
and asserts `operator_disabled` on the digest, so a starved loader turns
it red instead of passing on the failure path.
Nine places assigned window.location.href directly: OAuth connects and
sign-in hand-offs that land on API routes answering with a redirect, and
the reload after account deletion. The newer Next lint rule reads each as
an in-app page change. They now go through one helper that says why a
document load is the right tool there.
Marc-André Bombeck