A restore clears every row of the account, tombstones included, inside one
long transaction under the account's exclusive restore lock. The nightly
purge deleted expired tombstones across accounts, one statement per batch,
without that lock: on an account under restore it waited on the restore's
row locks for as long as the restore ran, past its statement timeout, or
deadlocked with it once each side held rows the other needed.
Each batch is now deleted one account at a time, in a transaction that
first takes the restore lock in shared mode, the same order the
consolidation passes keep. An account under restore is refused at once,
kept out of the rest of the run, and purged the next night. Still one
delete per account per batch, never one per row. The job reports how many
accounts it deferred.
The integration test holds a stand-in restore on one account while the
purge runs: the purge returns without waiting, leaves that account alone
and purges the other. With the lock call removed it waits out the hold.
Refs #1031
The shared admission queue sent its jobs without an expiry, so every one
took pg-boss's fifteen minutes. Against pg-boss 12.34, a handler that runs
past its expiry is not stopped: the job is failed, its abort signal fires,
the worker and group slots are freed, and the retry of the same job starts
in the same worker beside the run it replaced. A per-minute heart-rate
history is thousands of pages, so on a slow link one import turned into up
to four copies of itself, on the one queue meant to run a single import at
a time.
- The admission job now carries a sixteen-hour expiry, sized for the
longest legitimate import and under pg-boss's 24-hour ceiling.
- Each run holds a session advisory lock on its identity (kind, provider,
account) on a connection of its own. A delivery that finds it taken does
nothing and says so in the log; the running import stamps its marker or
is offered again at the next boot. Ending the connection or the process
releases the lock.
- The Google Health backfill runs under the job's budget and abort signal.
The page walk asks before each request, stops there, and reports the run
truncated, so it throws without the completion stamp and is retried.
- The drain moves out of the registrar into its own module so it can be
driven against a real pg-boss.
The integration test pins the pg-boss behaviour and runs the lane's
handler with a two-second expiry: without the lock the import ran twice.
Refs #1023
A sensor reading every five minutes puts about 26 000 glucose readings in
a 90-day window, and the per-kind series sent every one of them raw. The
hourly bucketing that already applies to a dense heart-rate stream now
applies to glucose too, with the same 10 000-reading cap: the hour's mean
converted once to the user's unit, statistics over every raw reading.
A fingerstick meter stays far under the cap and keeps its raw points.
The doctor report keeps glucose raw inside 90 days. Its clinical panel is
computed from the raw readings of every source, while the day path works
from canonical-source day buckets, so the two do not give the same figures
when a meter and a sensor overlap.
The restore dialog asked GET /api/admin/backups/{id}/summary for the
contents of a copy, and the route read the whole stored copy to answer:
every piece opened, every measurement parsed, the rest validated. On an
account of 1.8 million readings on slow hardware that took about two
minutes. The dialog stopped waiting after 15 s and said the contents
could not be read, and its automatic retry made the server read the
whole copy a second time.
The copy passes through the process once anyway while it is written, so
the preview is worked out there and kept on the row (DataBackup.preview,
migration 0364): the summarizeBackup counts, the schema version, and
the inner key ids with the sections they sit in and one sample value
each. Every creation path goes through storeBackupBlob (the weekly and
manual pass, and an upload), so every new copy carries one. The column
is plain JSON: counts and dates only, and the samples are ciphertext
that stands for what is inside the copy, which a key rotation must not
re-seal.
The route answers from it without opening the copy. Both verdicts are
taken again on every read, from what the preview keeps: whether this
release restores the schema version, and whether this server holds the
keys the copy needs. The pieces are still counted against the row, so
a copy that lost one is refused as the restore and the download refuse
it. A preview names the copy it describes and is ignored once the copy
changes; converting a single stored value into pieces carries it over.
A copy written before this has no preview. It is read whole once, as
before, and the result is kept on the row, so it is slow only the first
time; concurrent requests for the same copy share the one read.
Only a file that fails the schema answers 422 "Backup payload failed
schema validation". Read refusals keep their codes; anything else is
the server failing and answers 500.
The dialog no longer retries the preview, waits up to 55 s, and when it
stops waiting on an older copy it says the copy is still being read and
to open the dialog again in a few minutes.
Refs #1031
A bridge that pushes workouts from a watch vendor had no credential that
reached POST /api/workouts/batch: the route called requireAuth() with no
scope, so under the fail-closed default only a cookie session or a
wildcard token got through, and a measurements:write token was refused
403. The route's header still described the pre-fail-closed behaviour.
Add a separate workouts:write scope rather than widening
measurements:write, so a token already handed to a scale does not start
accepting workouts.
- POST /api/tokens/workouts mints it, mirroring the measurement and
document mints: cookie session plus a fresh proof, a literal permission
array, 10 mints a minute, at most 10 live tokens (409
tokens.workouts.ceiling_reached), a one-year default lifetime, audited.
- POST /api/workouts/batch names the scope. Sessions and wildcard tokens
behave as before. A scoped caller's workouts are stored as EXTERNAL,
resolved from the credential, and a batch in which any entry names a
source is refused 422 (workout.batch.source_not_permitted), the same
rule the measurement batch applies. EXTERNAL is on no source ladder, so
when the phone recorded the same session the phone's copy is shown.
- Settings > API offers a workout token card next to the measurement one,
and the endpoint list shows the batch route.
- Guards: the mint site, the declared scope vocabulary, the recent-proof
routes, and a new check that only the batch declares the scope.
- OpenAPI documents the mint, the batch's narrow-token behaviour and both
error codes; docs/api/README.md lists the workout and document mints.
Refs #1054
A restore on a freshly booted instance failed its clearing step with a
deadlock (40P01) while the boot-time step consolidation was folding the
same account. A consolidation day updates or mints the day's total and
then soft-deletes the day's per-sample rows; the restore deletes every
reading of the account in one statement. When the restore reached a
day's samples before its total, each side held what the other needed
next.
The restore already took an advisory lock per account before its first
delete. Every day transaction of the passes built on runConsolidation
(step and mean consolidation, the per-sample cumulative drain, the dense
intraday retention) now takes the same lock in shared mode as its first
statement. A restore that starts mid-day waits for that day to commit;
a day that would start while a restore runs is refused at once, and the
pass leaves that account for its next run instead of folding rows the
restore is about to replace. The passes still share the lock with each
other.
A new integration test builds the lock order that deadlocked, with a
row trigger holding the consolidation on the day's total while the
restore starts; before this change the consolidation was cancelled with
a deadlock. A guard test requires every transaction in a pass on
runConsolidation to open with the shared lock.
Refs #1031
Prisma's debug logger keeps the arguments of its last hundred calls in
memory whether or not logging is on, and the client and the Postgres
adapter both log every statement with its parameters. A restore that
wrote 1 000 readings per statement therefore kept a copy of about the
last 100 000 readings alive for the whole measurement write: about
60 MB on 1.8 million Apple Health readings, next to a V8 heap limit of
524 MB in the default 1 GB container.
The bulk insert now splits whatever it is handed into statements of at
most 100 rows, which holds about 7 MB. The Apple Health export import
goes through the same helper and gets the same bound. The write costs a
little time: 94 s instead of 76 s for 1.8 million readings on a fast
host.
Refs #1031
A type served from day buckets (pulse or glucose past 90 days, and since
the previous change a dense pulse stream on any window) is drawn one point
per day, each the day's mean at the day's start. The summary line counted
those points as measurements, and the FHIR export stamped the last day's
mean as the latest observation: a value and a time nobody measured.
The day-bucket stats now carry when their newest reading was taken
(`latestAt`). `latestReading` returns that reading when it is there, and
the summary line counts each type's readings from its stats. Raw series
are unchanged: their stats count equals their points and their last point
is the newest reading.
Past 90 days every pulse reader already folded in SQL. Inside it, three
still read every raw heart-rate row, which on a watch that records a
reading a minute is 130 000 rows per read (#1023):
- The insights page's resting-pulse proxy, used when an account has no
resting heart rate, walked the whole window to take one percentile per
day and held the event loop for over a second per view. It now folds in
Postgres (`readRestingPulseProxy`) with the same percentile, rounding,
minimum-sample rule and day anchor.
- The per-kind series sent every reading: 14.7 MB for 90 days. A pulse
window with more than 10 000 readings is now bucketed per local hour,
mean plus low/high band, with stats still over the readings.
- The doctor report drew every reading into one sparkline. A pulse window
over the same cap now takes the per-day path longer reports already use.
Sparse pulse (a cuff, hourly imports, a watch sampling every few minutes)
stays under the cap and keeps its raw shape.
A full-history backfill read every heart-rate point of an account into one
array, mapped it into a second, and wrote the lot in one call; the cycle's
deferred-rollup ledger then kept one entry per written reading. On a watch
that records a reading a minute that is over a million points, and the
backfill ran a 1 GB heap out of memory within seconds, restarting the
instance it shares with the app (#1023).
The metric walk now hands each page to its write before requesting the
next, and both the Google Health and Fitbit ledgers collapse to one key per
touched type-day as they record. The heart-rate walk's page ceiling rises
from 1000 to 10000: at one point a minute the old ceiling is under two
years, a longer history always ended truncated, and the backfill retried
the whole walk on every attempt and every boot without completing.
The clinical category a person picks for a medication lives in a side table
created at runtime, outside the schema. The backup builder and the restore
only walk schema models, so no backup ever carried it: after a restore every
medication read as "Other".
Both backup formats now carry the category inside its medication, and the
restore writes it back in the same transaction as the medication. A file
from before this change restores as before; a category this server does not
know reads as Other instead of refusing the file.
The backup plan gains a list for tables created with raw SQL, and a check
that every such table in the source either maps to a schema model or has a
verdict there, so the next one cannot drop out of backups unnoticed.
The All range is read from the WEEK or MONTH rollup tier once a history is
long enough, from DAY rows for a short one, and from the live table on a
coverage miss. The chart folds every path into the average day of each
week or month, but a tier row carried a cumulative metric's whole total
and a level's reading-weighted mean. A month of steps therefore read about
thirty times higher for a long history than for a short one, under a
caption that said average.
A tier row now carries the average day of its bucket on every tier: a
cumulative metric's total over the days that hold a reading, a level's
mean of its daily means. The rollup fold counts the bucket's days and adds
up their means for it. Only the web chart's series reads (the chart
endpoint and the dashboard batch) use these rows.
A restore writes rows back with their original updatedAt, behind every
paired client's watermarks, and removes rows without leaving tombstones.
A client that kept its cursor would miss both on its next incremental
pull. The same holds after the person deletes all their data.
`users.sync_reset_at` (migration 0363) records when the record was last
replaced: the restore stamps it as its last write and again right after
the commit, the per-account and the admin wipe stamp it inside their
transactions. Cursors now carry the time they were issued, and
/api/sync/changes answers its existing `cursorExpired` to a cursor issued
before the reset, or one without an issue time on an account that has
one. The comparison is on the issue time, because after a restore a
freshly drained cursor still carries old watermarks.
A restore now leaves out measurement tombstones deleted longer ago than
the tombstone retention. The nightly purge removes exactly those rows,
the delta feed answers cursorExpired for every cursor that could still
be served one, and the consolidation repairs look only inside the same
horizon. Younger tombstones keep their readers (a lagging paired client,
the undo of a recent delete, the repairs, the duplicate guard on
re-uploaded samples) and are written as before. The backup file is
unchanged. On a seeded account of 1.89 million readings whose 1.68
million tombstones were past retention, the restore on a slowed database
took 5.4 min instead of 15.
The rollup fold after the commit rebuilt five years of buckets under the
60 s request statement limit and could be dropped on a slow disk, leaving
the charts empty until the next boot. `recomputeUserRollups` takes a
`statementTimeoutMs` that scopes each aggregate and persist transaction
with SET LOCAL, and the restore passes its after-commit allowance.
Fixture tombstones in two round-trip tests were dated in July 2026 and
had aged past the retention; they are now relative to the run.
A restore is refused before it deletes anything when its transaction limit
would run past three quarters of the job's expiry. At two hours that refused
a same-size replacement of about four million readings. The expiry is now six
hours, which admits ten million over ten million; a dead worker still frees
the account within minutes through the heartbeat, so the longer expiry does
not hold an account longer.
The refusal now names both counts, the readings in the backup and the ones the
account holds now (removed readings included until the nightly clean-up),
since clearing the latter costs time too.
The encryption backfill now also removes the two copies of a practitioner's
contact details that outlived the sealing. An appointment reminder's copy of
the address is cleared, since the reminder resolves it from the practitioner
when it fires. An address-book edit's audit row written before the sealing
kept the old phone number and address in its previous values; those values
leave the row and the changed fields stay named. Rows of a deleted account,
which no per-account job reaches, are scrubbed by the discovery pass itself.
A route whose readable geometry is a JSON null rather than SQL NULL had
nothing to seal but matched discovery every night. The pass now sets it to
SQL NULL so it drops out.
Sealing the practitioner's address left the appointment arm reading the old
readable column, which is now empty, so a booked visit's push lost the address
on booking, rescheduling and restoring a deleted visit.
The reminder no longer keeps its own copy of the address. The reminder tick
resolves it from the owning visit's practitioner when it builds the push and
decrypts it there, so an edit of the address also reaches a nudge that is
already booked. MeasurementReminder.location stays NULL on appointment rows:
booking writes none, a re-anchor clears one an earlier release wrote, and a
backup neither carries nor restores one. A checkup's own free-text location is
unchanged.
get_metric_table reads one metric as days, weeks or months through the
chart reader, cut in the user's zone and collapsed to one source per day,
with null for a period without a reading. The model reads a bounded
summary; the full table rides the turn, encrypted at rest. show_result
shows a table an earlier reply of the same conversation fetched, resolved
only against that conversation's own messages. Workouts, labs, sleep and
adherence results project into tables, earlier tables are named in the
context by metadata alone, and result:rN marks are stripped from the reply.
A restore of an account with 1.9 million readings, most of them deleted
by the nightly consolidation and still awaiting the purge, was cancelled
in its clearing step: `DELETE FROM measurements` ran past the 60 s
statement timeout every pooled connection carries, and the whole restore
rolled back with a message that named no cause (#1031).
Most of that statement was not the delete. Two columns point into
`measurements` with ON DELETE SET NULL and had no index, so Postgres
scanned `personal_records` and `ecg_recordings` once per deleted reading:
38 of 40 s on a seeded account with 300 personal records. Migration 0362
indexes both, which also speeds up the tombstone purge and account
deletion.
The restore transaction now sets its own statement and idle-in-transaction
limits to the transaction budget with SET LOCAL, and the budget counts
the rows the account holds as well as the rows the file carries. A
rolled-back restore names what the database refused it over (timeout,
lock, constraint, storage, connection, other) on the job row, the
console, the audit row and the wide event.
Each later piece of the dialog gets its own module, answering nothing for
now: the step builder, the table projection and chart choice, the result
ref marks, the follow-up derivation, sentinel, resolution and continue
chip, the reuse turn, the clarification parse and answer, the method line,
and the four prompt addenda. The turn pipeline calls them where they
belong, so the frames go out in the documented order (step, token,
provenance, result, cards, clarification, followUps, done) and a blocked
turn drops its tables, chips and clarification. With every piece empty
the stream and the prompt stay byte for byte as before.
The home location of the environment module was rounded to 2 decimals,
about 1 km, which places a home in a neighbourhood rather than the town the
schema comment promised. The weather archive it feeds is gridded at 9 to
25 km, so the extra precision bought nothing.
Every stored coordinate of the module now goes through the same 1-decimal
rounding (about 11 km north-south, 7 km east-west at 50° N): the home, the
travel periods, the geocoder's results, the per-day weather rows, and a
restore of a file written with finer values. Migration 0361 rounds the
existing users, travel periods and per-day rows; running it twice changes
nothing more. The answers keep their shape, plain numbers. The API
descriptions and the schema comment say 1 decimal now.
A new nullable column, coach_messages.results_encrypted, holds the tables
of values a turn read as AES-GCM ciphertext; only their metadata rides the
plaintext provenance. readMessageResults serves them to the owner, each
table whole or withheld with a reason. The provenance restore learns the
steps, method, table metadata, chips, clarification and the forced-answer
marker, entry by entry against the wire schema.
The column is registered for key rotation, carried through both backup
formats (ciphertext in a disaster-recovery file, readable JSON in a
portable one) and goes with its row on conversation delete and wipe.
The GPS track of an outdoor workout was stored readable, and its first and
last points are usually the person's front door. The phone number and address
in the address book of doctors and practices were readable too, while the
note beside them was already encrypted.
Migration 0361 adds workout_routes.geometry_encrypted (the geometry loses
its NOT NULL) and practitioners.phone_encrypted / location_encrypted, plus
partial indexes on the backfill's discovery predicates. A track is sealed as
one binary AES-256-GCM value bound to its purpose; the contact fields use the
same Bytes codec as the note. The practitioner's name, practice and specialty
stay readable, because the picker searches and sorts on them.
Every writer stores ciphertext only: workout batch ingest, practitioner
create and edit, and both restore formats. Readers prefer the ciphertext and
fall back to the old column for rows not yet migrated: the workout detail
(map, splits, GPX), the activity insight's climb figure, the practitioner
DTO behind the address book, visits and vaccinations. The wire shape does not
change. An edit's audit row now names a changed phone number or address
without recording either value.
free-text-encryption-backfill covers the three columns in the same pass,
finding a route through its workout. All three are registered for key
rotation (registry, CLI script, and the in-app rotation). A disaster-recovery
backup carries the contact fields as ciphertext only; a portable export
carries them readable. Workout routes are still not part of a backup, as
before. The old columns are marked for removal in a later release.
The purge requests and the off-host key-use ledger are not keyed to a
user, so truncating users never reached them. A file that reads them
whole, the purge job or the encryption status view, saw rows earlier
files had left and failed in a full serial run.
The per-account password limit refused every attempt once ten failures
had landed in fifteen minutes, counted before the password was checked.
Ten guesses from two addresses therefore locked a password-only account,
the app's password sign-in included, and repeating them kept it locked
for as long as anyone cared to.
A place the account has signed in from before is now neither counted nor
held back: a remembered browser, a phone presenting the device id a login
of this account was issued to, or an address a session or device login of
this account came from in the last thirty days. Anywhere else gets five
free failures a day and then a wait that doubles with each failure, from
thirty seconds to at most fifteen minutes, instead of a wall. The per-IP
limit is unchanged, and an unknown identifier is limited the same way on
its hash.
Under an SSO-only policy, a password sign-in to an account other than the
one a pending link names is now refused before the password is checked,
so that exception no longer tells right passwords from wrong ones.
Offering access to the record took a live session alone, and an
invitation outlives the session that sent it: whoever held a stolen
session could invite their own account and keep a way into the record
after the owner signed everything out. POST /api/account/grants now takes
the recent-proof gate at every level (a sign-in or re-proof within five
minutes on the browser, with the second factor on an account that has one;
a step-up elevation on a token, which no shipped client uses here), and
the invite form opens the re-proof dialog and sends again.
Sign out everywhere now withdraws every invitation the account offered
that nobody has accepted. Accepted grants are left standing on purpose:
each is a person the owner chose, often a carer who still needs the
record, and a sign-out button must not quietly cut them off. The response
lists them as grantsKept and the sessions card shows each with a one-click
End access.
A medication's start and end dates are calendar dates, but the report
compared them against the window's instants. West of UTC the window
begins hours after UTC midnight, so a course that ended on the first day
was left out, and one that started the day after the window was listed.
Both now compare against the window's first and last day in the report
zone.
A data wipe during the nightly upload could leave the pre-wipe copy in
the bucket: the purge the wipe started removed its request at once, and
the upload still in flight checked that request after finishing and found
nothing. A wipe's request now stays until a nightly run has walked the
account since the wipe. Until then each purge removes only the copies
dated up to the wipe's day, so the copy of what the account holds after
the wipe stays, and the admin card no longer counts the waiting request
as copies still in the bucket.
The passkey sign-in, security-key sign-in and security-key registration
verifiers read the challenge, verified the response, and deleted the
challenge afterwards. Two requests carrying the same assertion could both
read it before either deleted it, and both be accepted.
A challenge is now claimed with one guarded DELETE ... RETURNING that also
checks its ceremony, its account and its expiry, before anything is
verified; a concurrent second request finds nothing. A passkey sign-in also
refuses a challenge that was begun for a different account than the passkey
belongs to.
Off-host objects are now written as envelope version 4: the header names
the off-host key by fingerprint, and the object's key in the bucket
(<date>/user-<id>.json.enc) is authenticated as GCM associated data, so
a copy moved to another account or another night no longer opens.
Objects of versions 1 to 3 still read. BACKUP_ENCRYPTION_PREVIOUS_KEYS
(comma-separated, on the compose whitelist and in the env manifest)
keeps retired off-host keys readable, which is the rotation path the
off-host key did not have.
The admin backup upload counted the bytes that arrived and none after
the gunzip, so a small gzip could expand without limit into the JSON
scanner and take the app down. The inflated bytes are now held to 100
times the compressed bytes read (at least 256 MB, at most 4 GB) and a
file past that is refused with 413 before anything is stored.
The extractor held the inflated export.xml to a flat 8 GiB whatever the
member occupied, and the header ratio check reads sizes the archive
writes itself: a member declaring an honest ratio could inflate at
1000:1, so an 8 MB upload could fill the temp directory. The streamed
byte count is now held to 200 times the member's compressed size on
disk (never less than 64 MiB, never more than 8 GiB), and a stored
member may not grow at all.
An account may have one Apple Health import queued or running. A
second upload with different bytes is refused with 409
import.apple_health.busy and the running job's id; the check and the
new row are one transaction under an advisory lock.
The staged upload is the person's export in plain text. It is now
removed on every exit that leaves no worker to read it: a refused or
broken multipart body, no worker running, a queue that takes no job
(the row is failed as well), the admin route's missing or unknown
target account, and an account deleted before the worker reached it.
The worker removes the files before recording a failure, so a row that
went with a deleted account no longer strands them. The 15-minute
import reconcile pass also sweeps staged files older than the longest
an import may run.
Marc-André Bombeck