Under a loaded full run the wrapper around pg_dump and pg_restore crashed
with an unhandled stream error, and the dump under test reported only the
tail of that stack. The wrapper now leaves the exit code to the tool and
writes the reason to stderr.
The check whether an account's full rollup backfill is queued or
running answered false on any error, so a failed read let the dense
fold delete raw samples the backfill still had to read. A missing queue
table still answers false, since a web-only deployment has no backfill;
any other failure now counts as pending and is annotated. A backfill
that failed for good is still not pending, as the comment now says.
The admin recompute for one account called the full fold without a
statement budget, so on the large accounts it is meant for each yearly
statement ran under the database's ordinary limit and timed out, as the
boot job had before it got its own. It now passes the same per-statement
budget.
The MFA status read listed oidc among the proof methods for a Bearer
caller too, although the provider round trip stamps a browser session
and nothing else, and the API contract says a token is never offered
it. The status read now leaves it out on the Bearer path.
One staged fact that would not decrypt threw out of the whole repair, so
the job failed, retried, and was enqueued again on every boot. It is now
skipped and counted as undecryptable.
The candidate query read an account zone Postgres does not know as UTC,
while the repair resolves it to the instance default and then
Europe/Berlin. A value confirmed near midnight stayed a candidate the
repair always left alone, so the set never converged. The query now
falls back the same way.
The alert added every attempt of a failed job to the streak, so one job
with two retries read as three failures. It also judged a queue by its
newest job across all accounts: on a queue that runs one job per
account, one account failing every hour beside accounts that succeed
never reached the streak, or flipped between failing and recovered.
The streak now counts jobs, and is kept per account (the job's userId,
or the whole queue for jobs without one). A queue is failing while any
account has failed three jobs in a row, and recovered only when a
failing account has itself run successfully since. A reminder needs a
failure newer than the notice before it, so a one-off job that failed
and does not run again is announced once.
The dense fold deleted Google Health and Oura samples outright, on the
reasoning that no client pulls them. The change feed serves every
source, so a client mirroring it kept those samples next to the hourly
means with nothing to tell it they were gone.
The nightly increment of a server-pulled source is tombstoned again: the
feed reports each sample and the tombstone cleanup purges them after the
retention. A backlog day, such as an account's first fold, is still
deleted outright, and the account's delta cursors expire instead, since
tombstoning millions of rows would undo the point of the fold. The
compaction-tombstone class leaves these sources out, so the feed, the
backlog purge and the restore treat their tombstones like any deletion.
Apple Health and Health Connect are unchanged.
The status prompts carry the unattended flag for background calls, the
queue list holds the lab date repair, and an account whose only way to
confirm is its identity provider is now offered that way.
The reading delete found the cuff's pulse by owner, source and identical
instant. The dense fold anchors an hour of heart rate at the local HH:30
under the samples' source, so a reading taken at half past took that hour
with it. Aggregate (stats:) rows are never part of a reading now, for the
delete and for the restore.
420 000 heart-rate readings, one every five minutes over about four
years, written through the real Google Health backfill with Google's
API faked at the safeFetch seam. The account's DAY rollups lose their
reading extremes, as rows folded before 0417 have them; the boot
discovery finds it, and the windowed full backfill refills every bucket
in the fold window with the raw readings' own extremes. The dense fold
then thins everything older than 90 days to hourly Google Health rows
(420 000 rows to about 59 000) while the DAY tier keeps its figures,
and a second full-history backfill over the same samples inserts
nothing. Without the ingest guard it re-inserts about 394 000 of them.
Each step reports its duration on stderr. On a developer machine:
backfill about 85 s, discovery 0.2 s, full backfill 4 s, fold 40 s,
second backfill 4 s.
Nothing covered the operator's position after an erasure: a backup file,
an admin session, and a server that no longer knows the account. Export
the shared backup seed, delete the account through the settings route
under a 14-day window and run the purge job past its end, then:
- scan every text and JSON column for the account's id; only a
rate-limit bucket that expires within a day may remain;
- upload the file as an admin: refused as owner_not_found, nothing
stored (the documented contract);
- recreate the account row under its id, upload and restore: every
section's count matches the export;
- point the file at a freshly created account, upload and restore: every
count matches and nothing names the erased account afterwards.
Every backup test exports and restores with the same code, so a change
that moves both ends at once passes all of them and still breaks a file
a self-hoster downloaded last month. Commit the disaster-recovery file
the 1.46.1 exporter writes for the shared backup seed, with a manifest of
its counts and SHA-256, and restore each committed file through the admin
upload and restore routes into an empty account, comparing the re-exported
counts section by section.
The folder takes one file per release (GOLDEN_BACKUP_WRITE=1 writes the
current one; the README says when). The seed's own illness-symptom
catalogue key is swapped for a shipped one so every link resolves on a
fresh instance. Ciphertext is sealed under the suite's fixed key. A
measurement tombstone that ages past retention may only lower its count.
A full rollup backfill that failed on every boot showed only on the
admin page's card of failing background jobs. From the account's side
the only symptom was empty dashboard cards, and nothing pointed the
operator at the card (#1298).
A new hourly evaluation reads the newest terminal runs of every queue
whose failure has no account-facing surface (the system audience in
subsystem-surface.ts) from pg-boss's own ledger. A failed run adds its
attempts, retries included, and the first successful run ends the
streak. Three failed attempts in a row send every admin a SYSTEM_ALERT
once, a reminder at most once a day while the queue keeps failing, and
one notice when it runs successfully again: the decision the backup
health notices use, with the same compare-and-set claim, so two workers
never send a notice twice.
The memory is one row per announced queue in system_job_alert_notices
(migration 0490). The notice names the queue, the streak and the last
error, and opens the admin status page.
The full rollup backfill refills each day's figures from the raw
readings, and the dense fold deletes those readings once it has their
hourly means. On a large account the backfill can run longer than the
ten minutes the fold's boot job waits, and a year it reached after the
fold had taken it would compute its days from hourly means: the day
count, minimum and maximum of every folded day would change.
The consolidation walk now asks a per-account deferUser hook before it
reads an account, and both entries of the dense fold (the per-account
boot job and the nightly drain) pass isRollupFullBackfillPending, which
looks for a queued or running full fold of the account in pg-boss. The
account is folded on a later run, once its rollup tier is rebuilt.
There was no way for an admin to remove someone else's account; the
users list only offered the undo for a deletion the account itself or a
Guardian had asked for.
DELETE /api/admin/users/{id} takes the account's own deletion path,
now shared as requestAccountDeletion: with a grace window the account
is locked at once (credentials revoked, record untouched) and erased by
the purge job when the window ends, and the existing undo gives it back
until then; with ACCOUNT_DELETION_GRACE_DAYS=0 it is erased in the
request. The route is cookie-only behind requireAdmin and the
recent-proof gate, takes the typed DELETE_ACCOUNT confirmation, and
refuses the caller's own account, a managed profile (its Guardians
delete it), an account already pending deletion, the last admin who
can sign in and the last Guardian of a managed profile, the last three
decided under their locks inside the transaction. Both the admin and
the account get an audit row.
The users list says who asked for a pending deletion, now including an
admin, and flags managed profiles. Each other account's row carries a
delete button whose confirmation names the purge date, or says the
erasure is immediate when the server keeps no window.
The dense hourly fold only took Apple Health and Health Connect samples,
so heart rate from the provider syncs that store one row per sample
stayed raw forever. One self-hosted account held about 25 million
Google Health pulse rows after a full-history backfill, and every
reader and fold that falls back to the raw table paid for it (#1297).
The fold now walks a map of sources to the dense types each stores
sample by sample: every dense type for Apple Health and Health Connect,
heart rate for Google Health and Oura. Their daily SpO2 values are no
intraday stream and stay as they are; Fitbit does not sync intraday
heart rate, and WHOOP, Withings and Polar write dense types only once
per recovery, night or spot check. Each source folds into its own
hourly stats rows, and the DAY rollup is still taken from the raw
samples before they go.
The fold deletes the samples it averages, and the provider syncs write
first-write-wins on the externalId, so a backfill or full sync would
have put every folded sample back. Google Health would even have
re-keyed an hourly row into a sample that fell on its anchor through
the natural-key rescue. The folded_window rule now has one hour rule
per fold source, and both syncs drop the samples of an hour the fold
already took before they write. Recent samples cost no lookup.
The boot discovery of the fold probes each account and source for the
first raw sample it would take instead of grouping every such row, and
the one-time fold repair stays on the two sources it was written for.
The provider sent PUT /api/auth/me/locale on mount and on every switch,
signed in or not. Signed out the route answers 401, the once-per-tab
marker was never set, and every signed-out page load repeated the
refused write.
The mirror moves out of the provider into LocaleServerSyncBridge, which
watches the account payload without fetching it and writes only once an
account is known. The choice made on the sign-in page therefore reaches
the account right after signing in, and the marker is keyed on the
account as well as the language, so another account signing in on the
same tab is written for too. The browser test stub no longer treats the
PUT as ambient, which is what lets its test see a stray write.
An account created through OIDC holds no password, passkey or second
factor, so the recent-proof gate in front of token minting, MCP
connections, exports and the admin reset had nothing to ask it for and
the web could only say "sign in again" (#1296).
GET /api/auth/oidc/reproof sends the browser back to the configured
provider with prompt=login and max_age=0, its own state, nonce and PKCE
verifier, and the signed-in account and session sealed into the state
cookie. The shared callback branches on that intent: it refuses unless
the browser still carries the bound session (checked before the code is
redeemed), the ID token names the account's linked issuer and subject,
and auth_time lies inside the gate's five-minute window. It then stamps
Session.reproofAt exactly as a password re-proof does, creates no
session, and returns to the page it left with ?reproof=done|failed.
recentProofMethods offers "oidc" only to an account without a second
factor whose identity is linked while a provider is configured, and the
fresh-factor gate never lists it. GET /api/auth/me/mfa publishes the
list as proofMethods, so the passkey card offers what the account can
actually give, and register-options takes { method: "oidc" } on a fresh
provider re-proof, so such an account can add a passkey as a second way
to confirm. Both re-proof dialogs offer the provider and a toast reports
the outcome on return.
The callback's missing-state branch and the passkey card's enrollment
flow move into their own functions to keep both handlers under the size
ratchet.
Systolic, diastolic and the cuff's pulse are three rows with no link
column, and the single DELETE tombstoned only the row it was given, so
deleting a reading's systolic left its diastolic and pulse behind as half
a reading. The bulk delete had the same gap.
A reading is the rows with the same owner, the same source and the
identical measuredAt: that is how the entry form, the MCP write, a
Withings measure group and a HealthKit correlation all store one, and how
the FHIR read pairs a diastolic onto its systolic. The looser five-minute
pairing of the chart reads is right for drawing a point and wrong for
taking rows away.
Deleting a systolic or diastolic now tombstones the rest of its reading in
the same transaction with the same deletedAt, each row with its own
syncVersion bump so paired clients see every tombstone. The restore route
brings back the rows deleted together with a named half (identical
deletedAt), so the undo, which names only the row that was tapped, puts
the whole reading back, from the web list and from any client that calls
it the same way. A pulse is still deleted on its own, and a row deleted at
another moment stays deleted. The response shapes do not change; the
OpenAPI descriptions say what the counts now include.
On an account with tens of millions of readings the boot full backfill
never finished. It ran four aggregates over the whole five-year window,
each held to the connection's 60 s statement limit, so every attempt
was cancelled with 57014, the reading extremes added by 0417 stayed
NULL, and the readers that need them fell back to the raw table and
timed out as well (#1298).
The full backfill now folds the window one UTC calendar year per pass,
newest first, and every statement runs under its own five-minute limit,
the shape the pulse day-mean rebuild already uses. Calendar-year cuts
write exactly the buckets one pass writes: day, month and year buckets
fall inside one window and a week across New Year is folded whole by
both. Each window commits on its own, so a run that reaches its job
budget hands the remaining years to a follow-up job instead of failing
and starting over on the retry.
The boot discovery read every live reading in the window to find
uncovered days, the same scan that kept the fold from finishing. It
now walks the days per account and type off the live index, so its
cost follows the number of days with readings, and runs under the same
limit. The full backfill code moves to its own module.
Before the confirm step dated an undated value by its document, a lab
value confirmed from a report that prints its collection date only in the
header was stored with takenAt set to the moment of confirmation. The fix
covers new confirms; the values already filed that way stay on the wrong
day until something moves them.
A one-shot maintenance job, sent at boot only while there is something to
repair, moves exactly those values to the document's report date (else its
filing date) at noon UTC, the same rule the confirm step now uses. A value
qualifies only when the chain is certain: written by the document confirm
path, named by exactly one approved observation fact, takenAt still within
five seconds of its own createdAt, no calendar date of its own in the
decrypted fact, and a document day that differs from the day the value is
filed under in the owner's zone. A value re-dated since, one really taken
on its confirmation day, or one whose document has no date is left alone.
A moved value no longer sits at its creation moment, so the pass converges
and a second run changes nothing.
The job definition lives in a new registry file beside maintenance.ts so
that file stays under the size ratchet.
A range longer than a year reads its series from the week or month
rollups, which are kept for the last five years. When the rollups held
any rows the read returned them alone, so on a record that goes back
further the All range started where the rollups start: a weight history
from 2015 charted from late 2021. The readings older than the first
rollup day are folded live at the tier's grain and lead the series; the
head ends where the first tier bucket begins, so no month is counted
twice. One indexed lookup when there are none.
Against a real Postgres and the real confirm route: an undated value takes
the report date over the filing date, a value's own date is kept, an
impossible own date counts as none, and a document without a report date
dates the value by its filing date.
The forwarder ran every tool as docker exec -i with its own standard
input piped through, also for pg_dump, which reads none. With -i and an
input already at its end the docker client may close the stream before
all output has arrived; one CI run saw the second of three nightly dumps
fail where every local run and the two runs before passed. Attach stdin
only for a file fed to pg_restore, and put the run's error into the
assertion so a failure says why.
realisedFamily probed for the closest two slots up to range.to only, so
the answer moved with the range. A Saturday + Sunday plan started the
weekend before the autumn DST change, read around its Saturday dose: the
±24 h range ends at 06:00 UTC on the Sunday, whose 08:00 slot is at 07:00
UTC after the change. Cut off, the closest slots were six days apart
(weekly); the wide re-mint saw the 25 h gap (daily). The two passes
disagreed, no band survived, and the dose was filed as unscheduled, so a
second backdated dose within a minute came back as the first one (200).
Probe the same reach on both sides of the range's middle.
The list ended 24 hours after local midnight. On the 25-hour day of an
autumn DST change that left the last hour out, so a dose due then could
not be offered from it; on the 23-hour spring day it reached into the
next day. Take the day's bounds from getUserTodayBounds, as the dashboard
tile does. Anchor the health-status test's history to local noon: now
minus whole days is still today on the long day.
The new backup, dump and deletion files were unknown to the duration
sequencer and gathered in one shard, which ran past its job timeout
while the other three finished in under fifteen minutes.
The list reads ordered schedules by their window and id, but the detail,
create, update, compliance, dose-history, GLP-1 and efficacy reads took
them in whatever order Postgres found them, so two schedules created in
the same millisecond could swap places between two answers. Every read
now uses MEDICATION_SCHEDULE_ORDER, and the guard checks every
schedules include, not only the list reads.
A background AI run's deadline is armed in model time plus a fixed margin.
Time spent waiting in the document reader's queue, and the read itself, came
out of that margin, so a person's lab scan or document read could be timed
out by the reaper because other work held the reader. When the reader grants
the slot it now moves the run's deadline by the wait plus the most the read
may take, and gives back the unused part when the read ends, never past the
job's own expiry and only while the run is running. The interactive queue
wait stays within the margin, which covers the run until the slot is
granted.
The per-account free-text encryption backfill seals what is already stored
and changes nothing the record holds, which is why its module is exempt from
the frozen-account rule; the new pending-deletion gate dropped its per-user
jobs anyway. A queue can now say it acts for pending accounts
(actsForPendingAccounts), and this one does.
Also types the ingest route test's token fixture to the row it now reads,
and drops an unused variable from the deletion grace test.
A cookie named healthlog_session can be set by any host under the same
registrable domain, so a page on a sibling subdomain could plant a session of
its choosing in a visitor's browser. Wherever the session cookie is Secure it
is now issued as __Host-healthlog_session, which a browser accepts only from
this exact host. Plain-HTTP installs, which cannot hold a __Host- cookie,
keep the old name.
Nobody is signed out: the old name is still read after the new one, a
session found under it is re-issued under the new name and the old cookie
expired, and a browser holding the new cookie is never answered from the old
one. Every reader and writer goes through one module, which the proxy, the
native-caller check, the native handoff and the OIDC callback use; the
session-surface guard follows the new reader.
A drill that stopped on its time budget before the first object recorded
ok with zero accounts checked, and the readiness tile said passed. Such a
run now records its time and counts and no verdict, so the last pass or
failure stands, and readiness calls it attention (nothing_checked) in every
locale.
The upsert that keeps a pass and a failure apart was stubbed in the drill's
tests and tested nowhere; it now has an integration test read back through
the readiness rule. The drill test that claimed to keep a failed verdict
when keeping it fails is named for what it checks.
Restoring the copy a data wipe took withdrew every held data_wiped purge of
the account. With two wipes in one window, restoring the first copy also
withdrew the second wipe's purge, and the off-host copies of what the person
entered between the wipes, and wiped, stayed until the bucket's rule removed
them. Only the request written after this copy and before the next one goes
now. The withdrawal had no test; it has one against Postgres.
The recovery kit, "Keys the backups still need" and safeToDropRetiredKeys
read only the stored copies and the off-host account objects. After a
rotation the page called the old key safe to drop while every dump taken
before it, the pre-migration dumps that are the way back from an update
among them, could only be restored with that key: db-restore refuses a dump
whose active key is missing.
A dump now counts for the key its rows were written under, the manifest's
active key: a dump on this server while it is kept (read from the manifests),
an off-host one until the bucket's rule for database/ expires it, or
indefinitely when that rule cannot be read. The nightly run records the key
on its ledger row (migration 0489, database_backup_runs.active_key_id). The
encryption view and the kit list the dumps per key, in every locale, and the
rotation and backup runbooks say so.
The enumeration guard read any query that named userId as scoped, including
an orderBy tiebreak, so the nightly recovery, strain and stress score jobs
grouped every account's measurements, wrote scores into frozen records and
spent cap slots on them. It also looked only at findMany and groupBy under
lib/jobs.
The guard now reads a scope from a query's filter only (orderBy, by,
distinct, select and include are dropped first), covers count, aggregate,
updateMany, deleteMany and executeRaw, and walks the boot discoveries and
sweeps that live beside their domain: the rollup and mood rollup discovery,
the Fitbit jobs, the cumulative record rederivation, the intake slot dedup,
the inventory expiry and the climb backfill. What it then found is filtered
(the score cohorts through excludePendingOwners, which filters before the
cap; Fitbit sync, backfill and sleep repair; coach proposal lapsing and
reminder nagging; expired coach reminders; snooze clearing; inventory
expiry) or says why it needn't be (retention and expiry sweeps over rows,
writes scoped by id).
Jobs queued before a deletion ran anyway, since handlers act on the userId
they carry without asking again. The binding of every queue now completes a
job whose payload names an account pending deletion without loading its
handler (pending-deletion-gate). The silent medication intake push, which
does not go through the dispatcher, refuses a pending account itself.
A handover link stayed claimable after its profile was deleted: the claim
gave the profile credentials that every sign-in then refused, and the purge
erased the account someone had just taken over. Locking an account for
deletion now revokes the open links for that profile and the links that
account minted, and creating, previewing and claiming a link refuse a profile
pending deletion, the claim under the record lock.
The last-Guardian rule counted Guardians pending deletion and profiles
pending deletion. A Guardian who had deleted their only profile could not
delete their own account; two Guardians sharing a profile could both leave,
and the second one's erasure was then refused on every run. The rule now
counts only Guardians whose account is not pending deletion, ignores profiles
pending deletion, and is decided under the record lock for account deletion,
data wipe, revoking and renouncing a Guardian. An admin's undo of a managed
profile with no such Guardian left is refused (409
account.deletion.no_guardian).
A run that kept its dump on the volume and failed the upload is the newest
good dump and the newest failure at the same instant. Compared with ">", the
card showed no failure and readiness said fresh, night after night, while no
dump left the host.
The health read now carries the off-host leg separately: when the newest dump
reached the bucket, and the newest one that did not. One pure reading of it
(databaseDumpProblems) feeds both the backups card and the readiness model.
A configured bucket that missed the newest dump is attention (offhost_failed)
with the upload's error, and failing once the newest dump there is older than
36 hours (offhost_stale) or none ever arrived (offhost_never). The card shows
the refused upload and when the bucket last got a dump. A failure at the
instant of the newest dump for another reason, retention throwing, is now
shown too.
The reproductive-health page promised that deleting all data or the
account erases at once. Both wait for the grace window now, the nightly
dumps keep a record until they age out, and the old pgdata volume is a
full copy until it is removed; the page says so and how to make deletion
final. The README, the getting-started note on majors, the move guide and
the Unraid and Portainer templates catch up with Postgres 18, the backups
volume and the recovery kit. The demo seed's spread bound leaves room for
the seed's own one-in-150 spread.
Only the backups page said when the bucket's lifecycle rules left
database/ out, so every dump, and every deleted account inside it, stayed
in the bucket forever without anyone being told. Readiness gains a
dump_retention signal that reads the same cached lifecycle probe: attention
when dumps go off-host and no rule covers database/, ok with the expiry
when one does, quiet when the bucket will not say, inactive without a
bucket or without off-host dumps. The hourly evaluation pushes it to the
admins like every other signal.
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.
A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.
The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
Marc-André Bombeck