Under a loaded full run the wrapper around pg_dump and pg_restore crashed
with an unhandled stream error, and the dump under test reported only the
tail of that stack. The wrapper now leaves the exit code to the tool and
writes the reason to stderr.
The count rides out in the job's outcome facts, whose vocabulary is an
allowlist, and is kept as a failure tally so the catch that feeds it is
not a silent one.
The check whether an account's full rollup backfill is queued or
running answered false on any error, so a failed read let the dense
fold delete raw samples the backfill still had to read. A missing queue
table still answers false, since a web-only deployment has no backfill;
any other failure now counts as pending and is annotated. A backfill
that failed for good is still not pending, as the comment now says.
The admin recompute for one account called the full fold without a
statement budget, so on the large accounts it is meant for each yearly
statement ran under the database's ordinary limit and timed out, as the
boot job had before it got its own. It now passes the same per-statement
budget.
The MFA status read listed oidc among the proof methods for a Bearer
caller too, although the provider round trip stamps a browser session
and nothing else, and the API contract says a token is never offered
it. The status read now leaves it out on the Bearer path.
The re-proof accepted an auth_time up to five minutes old and then
stamped the session at callback time, so the proof held for close to
ten minutes after the person signed in at the provider. It now stamps
the authentication instant, clamped to now for a provider clock that
runs ahead, and the gates' five-minute window runs from there.
One staged fact that would not decrypt threw out of the whole repair, so
the job failed, retried, and was enqueued again on every boot. It is now
skipped and counted as undecryptable.
The candidate query read an account zone Postgres does not know as UTC,
while the repair resolves it to the instance default and then
Europe/Berlin. A value confirmed near midnight stayed a candidate the
repair always left alone, so the set never converged. The query now
falls back the same way.
The alert added every attempt of a failed job to the streak, so one job
with two retries read as three failures. It also judged a queue by its
newest job across all accounts: on a queue that runs one job per
account, one account failing every hour beside accounts that succeed
never reached the streak, or flipped between failing and recovered.
The streak now counts jobs, and is kept per account (the job's userId,
or the whole queue for jobs without one). A queue is failing while any
account has failed three jobs in a row, and recovered only when a
failing account has itself run successfully since. A reminder needs a
failure newer than the notice before it, so a one-off job that failed
and does not run again is announced once.
The dense fold deleted Google Health and Oura samples outright, on the
reasoning that no client pulls them. The change feed serves every
source, so a client mirroring it kept those samples next to the hourly
means with nothing to tell it they were gone.
The nightly increment of a server-pulled source is tombstoned again: the
feed reports each sample and the tombstone cleanup purges them after the
retention. A backlog day, such as an account's first fold, is still
deleted outright, and the account's delta cursors expire instead, since
tombstoning millions of rows would undo the point of the fold. The
compaction-tombstone class leaves these sources out, so the feed, the
backlog purge and the restore treat their tombstones like any deletion.
Apple Health and Health Connect are unchanged.
The status prompts carry the unattended flag for background calls, the
queue list holds the lab date repair, and an account whose only way to
confirm is its identity provider is now offered that way.
The reading delete found the cuff's pulse by owner, source and identical
instant. The dense fold anchors an hour of heart rate at the local HH:30
under the samples' source, so a reading taken at half past took that hour
with it. Aggregate (stats:) rows are never part of a reading now, for the
delete and for the restore.
readBestGranularityRollups opened its window at the wall clock, while the
live fallback beside it used the caller's now. A read anchored on another
moment saw different days depending on whether the account already had
DAY buckets; the Today overview's integration test turned red once its
fixed morning fell out of the wall-clock window. Every caller that holds a
now passes it through.
420 000 heart-rate readings, one every five minutes over about four
years, written through the real Google Health backfill with Google's
API faked at the safeFetch seam. The account's DAY rollups lose their
reading extremes, as rows folded before 0417 have them; the boot
discovery finds it, and the windowed full backfill refills every bucket
in the fold window with the raw readings' own extremes. The dense fold
then thins everything older than 90 days to hourly Google Health rows
(420 000 rows to about 59 000) while the DAY tier keeps its figures,
and a second full-history backfill over the same samples inserts
nothing. Without the ingest guard it re-inserts about 394 000 of them.
Each step reports its duration on stderr. On a developer machine:
backfill about 85 s, discovery 0.2 s, full backfill 4 s, fold 40 s,
second backfill 4 s.
Nothing covered the operator's position after an erasure: a backup file,
an admin session, and a server that no longer knows the account. Export
the shared backup seed, delete the account through the settings route
under a 14-day window and run the purge job past its end, then:
- scan every text and JSON column for the account's id; only a
rate-limit bucket that expires within a day may remain;
- upload the file as an admin: refused as owner_not_found, nothing
stored (the documented contract);
- recreate the account row under its id, upload and restore: every
section's count matches the export;
- point the file at a freshly created account, upload and restore: every
count matches and nothing names the erased account afterwards.
Every backup test exports and restores with the same code, so a change
that moves both ends at once passes all of them and still breaks a file
a self-hoster downloaded last month. Commit the disaster-recovery file
the 1.46.1 exporter writes for the shared backup seed, with a manifest of
its counts and SHA-256, and restore each committed file through the admin
upload and restore routes into an empty account, comparing the re-exported
counts section by section.
The folder takes one file per release (GOLDEN_BACKUP_WRITE=1 writes the
current one; the README says when). The seed's own illness-symptom
catalogue key is swapped for a shipped one so every link resolves on a
fresh instance. Ciphertext is sealed under the suite's fixed key. A
measurement tombstone that ages past retention may only lower its count.
A translated label that runs a little longer than the English one wraps,
and a wrapping header grows; every English-only check stays green while
one locale ships a taller dashboard header. The new sweep switches its own
account through each locale the app ships (via the locale API, with
<html lang> asserted before measuring), then compares the top bar and the
dashboard's page-header row on a 390 px phone and a desktop window with
the height English gives them, within 1 px.
The locale list is read from the i18n config, so a new language is swept
without editing the spec. It runs in the desktop project only because the
account's language is one value shared by its tests.
What an account with tens of millions of readings looks like when it
runs into the per-statement limit (empty cards, job.run.failed with
57014), what the full backfill, the hourly heart-rate fold and the
admin alert now do about it, and when raising
DATABASE_STATEMENT_TIMEOUT_MS still makes sense. The variable was only
named in the compose file and the example env until now (#1298).
A full rollup backfill that failed on every boot showed only on the
admin page's card of failing background jobs. From the account's side
the only symptom was empty dashboard cards, and nothing pointed the
operator at the card (#1298).
A new hourly evaluation reads the newest terminal runs of every queue
whose failure has no account-facing surface (the system audience in
subsystem-surface.ts) from pg-boss's own ledger. A failed run adds its
attempts, retries included, and the first successful run ends the
streak. Three failed attempts in a row send every admin a SYSTEM_ALERT
once, a reminder at most once a day while the queue keeps failing, and
one notice when it runs successfully again: the decision the backup
health notices use, with the same compare-and-set claim, so two workers
never send a notice twice.
The memory is one row per announced queue in system_job_alert_notices
(migration 0490). The notice names the queue, the streak and the last
error, and opens the admin status page.
The createdAt field and the locale import grew files the size ratchet
holds. The account payload's insurance-number decryption moves into
its own helper, the import joins the zod one, and two comments lose a
wrapped line, so no baseline has to rise.
The full rollup backfill refills each day's figures from the raw
readings, and the dense fold deletes those readings once it has their
hourly means. On a large account the backfill can run longer than the
ten minutes the fold's boot job waits, and a year it reached after the
fold had taken it would compute its days from hourly means: the day
count, minimum and maximum of every folded day would change.
The consolidation walk now asks a per-account deferUser hook before it
reads an account, and both entries of the dense fold (the per-account
boot job and the nightly drain) pass isRollupFullBackfillPending, which
looks for a queued or running full fold of the account in pg-boss. The
account is folded on a later run, once its rollup tier is rebuilt.
The French bundle mixed the familiar and the formal address: about a
third of the strings said "tu", "ton" or "Réessaie", the rest "vous",
sometimes both on one screen and once in one string.
Every string that addressed the reader as "tu" now uses "vous" with the
matching conjugation, possessives and agreement; placeholders, plural
branches and punctuation are unchanged. The Coach prompts a person
sends address the Coach the same way, the casual "Salut" greeting of
the nudges becomes "Bonjour", and "après que" takes the indicative.
A Dutch reader whose screener answer raised the safety path saw only
the international set, because no Dutch set existed.
The Netherlands gets 113 Zelfmoordpreventie (113, the free 0800-0113
and the chat on 113.nl), Belgium the Zelfmoordlijn (1813,
zelfmoord1813.be), both behind 112. The app's one Dutch bundle is read
in both countries and the account keeps no country, so a bare "nl"
lands on a combined set that names each line with its country, the way
the German and Portuguese sets already do; an explicit nl-NL, nl-BE or
region "be" gets its own country's set.
The Coach chat, the document chat and the fenced chat declared their
locale field as de/en, so a body naming French, Korean, Brazilian
Portuguese or any other shipped language was refused with 422 before
resolveServerLocale, which carries every locale, ever saw it. The same
pair sat on the profile update, so an account could not store any other
language there either, and the stored language is what the cookie-blind
background work (the Coach nudge, notification texts) writes in.
All four now take z.enum(locales), and the published contract for the
metric, insight and biomarker status queries and the profile update
lists every UI locale instead of two.
The header always said "welcome back", including to an account the
setup flow had created a few minutes earlier.
/api/auth/me now publishes the account's createdAt, and the greeting
welcomes an account in its first day ("Welcome, name.") and welcomes
back every other one, including an account whose older server does not
send the field. Until the account is known, during the SSR pass and the
hydration render, the line is the time-of-day greeting alone, so
nothing claims "back" before the client knows it is true.
There was no way for an admin to remove someone else's account; the
users list only offered the undo for a deletion the account itself or a
Guardian had asked for.
DELETE /api/admin/users/{id} takes the account's own deletion path,
now shared as requestAccountDeletion: with a grace window the account
is locked at once (credentials revoked, record untouched) and erased by
the purge job when the window ends, and the existing undo gives it back
until then; with ACCOUNT_DELETION_GRACE_DAYS=0 it is erased in the
request. The route is cookie-only behind requireAdmin and the
recent-proof gate, takes the typed DELETE_ACCOUNT confirmation, and
refuses the caller's own account, a managed profile (its Guardians
delete it), an account already pending deletion, the last admin who
can sign in and the last Guardian of a managed profile, the last three
decided under their locks inside the transaction. Both the admin and
the account get an audit row.
The users list says who asked for a pending deletion, now including an
admin, and flags managed profiles. Each other account's row carries a
delete button whose confirmation names the purge date, or says the
erasure is immediate when the server keeps no window.
From a HealthLog container with its stored ChatGPT sign-in, gpt-6.1-sol,
gpt-6-sol and gpt-6-luna each answered text and read a lab-report image,
and the account's own model list offered them with image input.
gpt-6.1-sol refuses effort none and names low as its lowest. The spec said
the gpt-6 rungs had never been probed live, and still listed gpt-5.4,
gpt-5.4 mini, gpt-5.3-codex and gpt-5.2 as accepted, all of which OpenAI
has retired for ChatGPT sign-in, with the exact refusal the backend sends.
Also note why a fixed chain goes stale: once GPT-5.5 retired, every model
on the chain shipped at the time was retired with it.
OpenAI retired the 5.x line from Codex for ChatGPT sign-in in steps: GPT-5.2
and GPT-5.3-Codex on 2026-05-26, GPT-5.4 and GPT-5.4 mini on 2026-08-31,
and GPT-5.5 on 2026-10-14. The backend answers a retired model with 400
model_not_found and "not supported when using Codex with a ChatGPT
account".
Pin that exact refusal as a reason to try the next slug, pin that an
account refused the head of the default list walks on to the next rung
and caches it, and keep the long-retired models off the default list.
gpt-6.1-sol, a rung on the default list, answers reasoning "off"
(effort none) with 400 unsupported_value and lists low as its lowest
effort (seen 2026-10-24). The client must stay on the slug, retry with
low and remember it, not walk on as if the model were retired.
Background jobs ask for medium reasoning at most, and status cards ask for
none. A model that cannot be switched down (GLM behind a vLLM gateway, for
instance) ignores both, spends the whole room thinking and returns only
reasoning, so the daily briefing and the status cards fail as
reasoning_only and the chain pauses. The Coach on High works because it
leaves 12k tokens to think in.
The chain runner now marks every call of a job walk as unattended, in one
place rather than at each call site. The gateway and local clients give
such a call at least the High thinking room once its model has answered
with reasoning alone. The room has to ride on the #1267 repeat with none:
one call may send two requests, and a once-a-day job meets an expired
lesson every morning, so a room that waited for the unswitchable lesson
would never reach the first briefing of the day. It is a ceiling, not a
charge: a model that does stop at none never touches it. Calls a person is
reading, the Coach included, keep the room of their own level, and the
requested effort is unchanged.
The room calculation both clients shared is now one function.
Reported by @Tio61 in #1276.
The dense hourly fold only took Apple Health and Health Connect samples,
so heart rate from the provider syncs that store one row per sample
stayed raw forever. One self-hosted account held about 25 million
Google Health pulse rows after a full-history backfill, and every
reader and fold that falls back to the raw table paid for it (#1297).
The fold now walks a map of sources to the dense types each stores
sample by sample: every dense type for Apple Health and Health Connect,
heart rate for Google Health and Oura. Their daily SpO2 values are no
intraday stream and stay as they are; Fitbit does not sync intraday
heart rate, and WHOOP, Withings and Polar write dense types only once
per recovery, night or spot check. Each source folds into its own
hourly stats rows, and the DAY rollup is still taken from the raw
samples before they go.
The fold deletes the samples it averages, and the provider syncs write
first-write-wins on the externalId, so a backfill or full sync would
have put every folded sample back. Google Health would even have
re-keyed an hourly row into a sample that fell on its anchor through
the natural-key rescue. The folded_window rule now has one hour rule
per fold source, and both syncs drop the samples of an hour the fold
already took before they write. Recent samples cost no lookup.
The boot discovery of the fold probes each account and source for the
first raw sample it would take instead of grouping every such row, and
the one-time fold repair stays on the two sources it was written for.
The provider sent PUT /api/auth/me/locale on mount and on every switch,
signed in or not. Signed out the route answers 401, the once-per-tab
marker was never set, and every signed-out page load repeated the
refused write.
The mirror moves out of the provider into LocaleServerSyncBridge, which
watches the account payload without fetching it and writes only once an
account is known. The choice made on the sign-in page therefore reaches
the account right after signing in, and the marker is keyed on the
account as well as the language, so another account signing in on the
same tab is written for too. The browser test stub no longer treats the
PUT as ambient, which is what lets its test see a stray write.
An account created through OIDC holds no password, passkey or second
factor, so the recent-proof gate in front of token minting, MCP
connections, exports and the admin reset had nothing to ask it for and
the web could only say "sign in again" (#1296).
GET /api/auth/oidc/reproof sends the browser back to the configured
provider with prompt=login and max_age=0, its own state, nonce and PKCE
verifier, and the signed-in account and session sealed into the state
cookie. The shared callback branches on that intent: it refuses unless
the browser still carries the bound session (checked before the code is
redeemed), the ID token names the account's linked issuer and subject,
and auth_time lies inside the gate's five-minute window. It then stamps
Session.reproofAt exactly as a password re-proof does, creates no
session, and returns to the page it left with ?reproof=done|failed.
recentProofMethods offers "oidc" only to an account without a second
factor whose identity is linked while a provider is configured, and the
fresh-factor gate never lists it. GET /api/auth/me/mfa publishes the
list as proofMethods, so the passkey card offers what the account can
actually give, and register-options takes { method: "oidc" } on a fresh
provider re-proof, so such an account can add a passkey as a second way
to confirm. Both re-proof dialogs offer the provider and a toast reports
the outcome on return.
The callback's missing-state branch and the passkey card's enrollment
flow move into their own functions to keep both handlers under the size
ratchet.
Systolic, diastolic and the cuff's pulse are three rows with no link
column, and the single DELETE tombstoned only the row it was given, so
deleting a reading's systolic left its diastolic and pulse behind as half
a reading. The bulk delete had the same gap.
A reading is the rows with the same owner, the same source and the
identical measuredAt: that is how the entry form, the MCP write, a
Withings measure group and a HealthKit correlation all store one, and how
the FHIR read pairs a diastolic onto its systolic. The looser five-minute
pairing of the chart reads is right for drawing a point and wrong for
taking rows away.
Deleting a systolic or diastolic now tombstones the rest of its reading in
the same transaction with the same deletedAt, each row with its own
syncVersion bump so paired clients see every tombstone. The restore route
brings back the rows deleted together with a named half (identical
deletedAt), so the undo, which names only the row that was tapped, puts
the whole reading back, from the web list and from any client that calls
it the same way. A pulse is still deleted on its own, and a row deleted at
another moment stays deleted. The response shapes do not change; the
OpenAPI descriptions say what the counts now include.
On an account with tens of millions of readings the boot full backfill
never finished. It ran four aggregates over the whole five-year window,
each held to the connection's 60 s statement limit, so every attempt
was cancelled with 57014, the reading extremes added by 0417 stayed
NULL, and the readers that need them fell back to the raw table and
timed out as well (#1298).
The full backfill now folds the window one UTC calendar year per pass,
newest first, and every statement runs under its own five-minute limit,
the shape the pulse day-mean rebuild already uses. Calendar-year cuts
write exactly the buckets one pass writes: day, month and year buckets
fall inside one window and a week across New Year is folded whole by
both. Each window commits on its own, so a run that reaches its job
budget hands the remaining years to a follow-up job instead of failing
and starting over on the retry.
The boot discovery read every live reading in the window to find
uncovered days, the same scan that kept the fold from finishing. It
now walks the days per account and type off the live index, so its
cost follows the number of days with readings, and runs under the same
limit. The full backfill code moves to its own module.
Before the confirm step dated an undated value by its document, a lab
value confirmed from a report that prints its collection date only in the
header was stored with takenAt set to the moment of confirmation. The fix
covers new confirms; the values already filed that way stay on the wrong
day until something moves them.
A one-shot maintenance job, sent at boot only while there is something to
repair, moves exactly those values to the document's report date (else its
filing date) at noon UTC, the same rule the confirm step now uses. A value
qualifies only when the chain is certain: written by the document confirm
path, named by exactly one approved observation fact, takenAt still within
five seconds of its own createdAt, no calendar date of its own in the
decrypted fact, and a document day that differs from the day the value is
filed under in the owner's zone. A value re-dated since, one really taken
on its confirmation day, or one whose document has no date is left alone.
A moved value no longer sits at its creation moment, so the pass converges
and a second run changes nothing.
The job definition lives in a new registry file beside maintenance.ts so
that file stays under the size ratchet.
knownSignInSource compared a trusted device's expiry with new Date()
while the throttle around it times its waits with Date.now(). The test
mocks Date.now() only, so it set the device to expire 30 days after its
fixed clock, which the real date passed on 2026-10-26 at noon UTC; from
then on the remembered browser read as expired and the test failed on
every run. One clock decides both now, and the test's expiry is fixed
against it. A further case pins that an expired device gets no
exemption.
The narrow-chart rule lived in HealthChart's body and grew it past its
size baseline. The overlays hook, which already sizes the room beside
the plot, now takes the chart's width, keeps no room on a narrow chart
and marks each target zone whose label sits inside it; the x axis
helper works out its own tick layout. The tiered series unit test mocks
the lookup for readings older than the rollups and covers the head the
read puts in front of them.
Marc-André Bombeck