Key rotation re-sealed a single-value backup in place, which holds the
stored string, its bytes, the plaintext and the new ciphertext at once; a
copy can be over a hundred megabytes and the in-app rotation walked 25 of
them per page, enough to run a 1 GB container out of memory. Rotation now
converts each such copy into pieces under the active key instead, one row
at a time. The ~hlgcm1. stream of v1.38.6 to v1.39.1 is read a megabyte
at a time with its tag checked at the end, inside one transaction, so a
copy that fails the check is left untouched; the copy keeps its date. The
scan reads only the first characters of each value. Measured on a copy of
a 2.6 million-reading account (126 MB stored): converted in 25 s with the
heap flat at 57 MB under a 200 MB limit, where re-sealing in place ran out
of heap.
The weekly and manual backup leave an account alone while a restore of
its current copy is queued or running, and count it as skipped, not
failed. A reader whose copy is replaced mid-read answers 409
backup_changed, "replaced by a newer one while it was being read", rather
than the undecryptable error; a restore rolls back with the same code.
Pieces are sealed with a fixed associated-data label, so a value the same
key sealed for another purpose never opens as a piece; rotation re-seals
under the same label. A row holding both a single value and pieces, which
only an older release writing after a downgrade produces, is refused with
a message saying so, and the next weekly run replaces it. Rotation treats a
row deleted between its read and its write as gone rather than an error.
A stored backup was one value in data_backups.data. v1.39.1 wrote it a
piece at a time but still joined the pieces into that one value, and one
value has to pass through the app whole, so a copy was capped at a fifth of
the heap: 105 MB in the default 1 GB container. An account with 1.75
million readings is past that and could not be backed up at all (#1031).
A copy is now kept as ordered pieces of about a megabyte in the new
data_backup_chunks table (migration 0352). Each piece is sealed on its own
with AES-256-GCM, and its sealed header names the copy (a random stream id
kept on the row), its position and whether it is the last. A reader checks
every piece before it releases a byte, so a piece that is missing, moved,
altered, taken from another copy, or a copy cut short is refused with
backup.payload.undecryptable before a restore deletes anything, and the
second read during the restore checks each piece again.
The write is one transaction, so the previous copy stays whole and readable
until the new one commits. Pieces are written through the model API: a raw
query with a large parameter is kept alive by Prisma's query plan cache,
which left 160 MB of heap behind per hundred one-megabyte inserts.
Restore, preview, download, the restore job and the admin list read both
forms; copies written before this change keep their single value and still
restore. The heap-derived limit is gone. The one limit left is on storage,
BACKUP_MAX_STORED_MB (default 2048), and its message names the setting. Key
rotation covers the pieces, the wipe and account deletion take them with
their backup, and the backup plan lists the table as not carried.
Measured on an account of 2.6 million readings in a 1 GB container with web
and worker in one process: backup 87 s, 97 MiB in 96 pieces, preview 20 s,
download 25 s, restore 264 s with every reading back exactly, peak 567 MiB,
no restart.
A weekly or uploaded backup written by v1.39.x is a single ~hlgcm1. stream
in data_backups.data. Its key id sits behind the stream marker, which the
string codec's parser does not read, so key rotation filed every such row
as legacy, failed to decrypt it and counted it as an error. On a host that
ran v1.39.x that was every weekly copy, and the rotation never finished
clean.
Rotation now reads the key id of a streamed value and re-seals it as a
stream under the active key, without touching what it holds.
A current disaster-recovery backup writes aggregationProvenance for every
measurement, null included. The restore treated null like a missing field
and marked every Apple Health stats: row without a provenance as
LEGACY_UNKNOWN, which is meant only for backups too old to carry the field.
The nightly daily-mean fold writes exactly such rows, so on a large Apple
Health account a restore changed tens of thousands of readings it was
supposed to put back as they were (#1031).
Only an absent field now takes the legacy path.
Bump @simplewebauthn/server 13.3.3 -> 14.0.2 and @simplewebauthn/browser
13.3.0 -> 14.0.0, and keep React on an exact pin at 19.3.0 with react-dom
and both type packages moved alongside.
SimpleWebAuthn 14 changes one default the server relied on: the verifier
used to accept every algorithm the library knew (EdDSA, ES256, ES512,
PS256/384/512, RS256/384/512, RS1) and now accepts only the three it
offers. Pass both lists explicitly so a registration that succeeded
before still succeeds, and so the offered list stays EdDSA/ES256/RS256
on a runtime that would otherwise prepend ML-DSA-44. The option JSON of
all four ceremonies, and verification against stored credentials, match
the 13.x snapshots unchanged.
Every existing passkey test mocks @simplewebauthn/server, so a library
upgrade could change the options a browser or the iOS app receives, or
the algorithms the verifier accepts, without any test noticing. Drive
the real library with a software authenticator: snapshot the option
JSON of all four ceremonies, register and authenticate with each of the
ten algorithms the verifier has accepted, and verify an assertion
against a credential row stored the way earlier releases wrote it.
On the rollup path the all-time count, min, max and mean are composed
from DAY buckets plus one raw aggregate of the readings older than the
fold window. The bucket half read every DAY bucket the account had, the
raw half every reading before now minus five years, so a reading on any
day that had a bucket before that instant was counted in both. Such
buckets are common: the per-write hook folds whatever day it touches,
the Apple Health import folded whole spans until this release
clamped it, every fold snaps its start down to a whole day, and
nothing removes a bucket once the moving window has passed it. The
count came out too high and the mean leaned towards the doubled days,
while the live fallback reported the true figures for the same account.
Both halves now meet at the start of the window's first UTC day:
buckets from there on, raw readings before it. A reading lands in
exactly one half whichever buckets exist, so buckets already written
before the window need no repair.
The snapshot's second phase (blood pressure in target, the per-context
glucose tiles, the Health Score) waits until weight, both blood-pressure
halves and steps have DAY buckets. The fold writes buckets only inside
the five-year window, so a type whose every reading is older than that
never gets one and the phase stayed null for good. Since the Apple Health
import now folds only inside the window too, an account that imported
years-old blood pressure or steps and nothing since lost those tiles and
its score with no explanation.
An uncovered gate type now holds the phase back only when it has a
reading inside the window, which is the case the backfill will still
reach. The extra query runs on the miss path alone and asks one indexed
EXISTS per uncovered type.
The projector floored today's slots only on an empty archived era, the
reminder worker and the next-due reads on the newest revision boundary,
and the auto-miss pass on the creation instant alone. Switching tracking
back on and then moving a dose time the same afternoon therefore put the
morning dose on the card, and the next day's pass stamped it a miss; a
same-day times edit on any medication did the same.
One helper now names the live era start (the newest active revision's
boundary) and every path floors on it: the projector, the worker, the
doses card, the medication list, the MCP schedule tool and the auto-miss
pass, which leaves a placeholder minted after the switch for a slot
before it.
Three readers compute a rate straight from intake rows without reading a
schedule: the weekly compliance nudge, the adherence series behind the
general status, and the BP-versus-compliance correlation. They now read
tracked medications only. The structural guard covers intake-row readers
as well as schedule readers, with a whitespace-tolerant matcher, a floor
on the match count and a reviewed allowlist.
A medication's units per dose, a schedule's override and a restored
backup now accept any value above 0 and at most 100 with at most four
decimal places, which is exactly what the Decimal(10,4) column stores.
Before, only whole numbers 1-100 and the fractions 1/4, 1/3, 1/2, 2/3,
3/4 passed, so one and a half tablets could not be recorded. A restore
now refuses a value no write route would accept.
The medication editor keeps its buttons and adds an Other field that
reads 1.5, 1,5, 1 1/2 and 1½, holds the step while the text does not
read, and previews the value it will save. The supply tab's packaging
dialog uses the same reader; it refused every non-whole value before,
so a medication at half a tablet could not have its packaging saved.
Cards and the supply tab show a whole number plus a fraction as 1½.
The rule, the input reader and the display format live in one module
without imports so the editor and the validators share them.
Refs #1034
The shipped iPhone app cannot see the schedule of a medication kept as a
record, because it is served with an empty schedules list. Its editor opens
such a medication as daily at 08:00 and, once any schedule field is touched,
sends a full replacement built from that default, plus asNeeded and oneShot,
and never trackIntake. On a record-only medication the update route now
drops schedules, asNeeded and oneShot from a write that does not name
trackIntake and applies the rest. The web schedule editor and the wizard
name the flag on every save.
The corpus walk and the rotation script paged with Prisma's cursor and skip.
Prisma resolves that cursor by reading the page's last row again, so when
that row was hard-deleted before the next page (a backup replaced, a cached
response expired), the next page came back empty and the walk ended early.
The scan then counted fewer rows than the database holds, which could read
as nothing left on the old key while rows it never reached still were.
Both walks now ask for the rows whose id is greater than the last one read,
which needs no row to still exist.
A per-medication trackIntake flag (migration 0351, default on for every
existing and new row). With it off the dose, dates and schedule stay stored,
but nothing derives a due dose from them: the reminder worker, the manual
reminder check, the today projector, the auto-miss pass, the low-stock pass,
the doses card, the today list, the outstanding-dose badge, compliance and
its heatmap, the doctor report, the Coach, the BP-status gate, insights,
targets, achievements, the efficacy view and the MCP schedule tool all skip
it.
The medication wire serves such a medication with schedules: [] and the
stored rows in an additive recordedSchedules field, so a client that arms
local reminders from schedules stays silent without an update. Switching
tracking off or back on archives a schedule revision at that instant, the
untracked stretch with an empty payload, so it never counts as missed and
the projector does not put a dose from before the switch back on the card.
A structural guard freezes the set of schedule readers: each one filters on
the flag, derives from dueSchedules, shapes the wire with scheduleWireFields,
or is allowlisted with the reason it derives nothing due. The flag travels
through backup, restore and the medication CSV export.
The detail page leads the schedule tab with the switch and a one-line
explanation of each state; with tracking off it drops the reminder card,
the next-dose and adherence rows, and shows the stored schedule as the
record. The wizard summary carries the same switch. Cards, the table and
the detail header mark a record-only medication, which leaves the take and
skip buttons, the take-all set, the log-dose dialog and the dashboard quick
add. Strings in all seven locales.
Portable files exported before v1.39.1 carry no creation date for a
medication, so the restore stamped the restore time. The dose history reads
that date as the day the medication began to exist, and every miss before
it, including the day before the restore, dropped out of the record.
When the file has no creation date, the restore now uses the earliest thing
the file records for the medication: its first dose (scheduled, taken or
written), the start of its first schedule revision, its course start, its
first pause, dose change or side effect, and never a date after the restore.
A file that records nothing about the medication keeps the old default.
An uploaded file whose encrypted copy passes the size one stored backup may
have on this host (a fifth of the heap limit) stopped the store with
BackupBlobTooLargeError, which the route let through as a 500 and reported
to the error tracker. It is now a 413 with the numbers and the remedy in the
message and meta.errorCode backup.upload.too_large, and the refusal is
audited like the others. The store rolls back, so nothing is kept.
A restore commits its transaction and then rebuilds the chart tiers. A
worker that stopped during that rebuild left the job running with a stale
heartbeat, and the boot sweep and the admission check treated it like any
interrupted job: the sweep queued it again, so the second run deleted and
re-wrote the account and lost whatever was written in between, and a second
interruption closed it with "rolled back, nothing was changed", which was
false.
The job now records `committedAt` (added to migration 0347, which has not
shipped) and awaits that write before the rebuild starts. The sweep and the
admission check close a committed job as `failed_after_commit`, never queue
it again, and evict the account's caches. The rebuild itself is left to the
nightly and boot passes that already rebuild missing tiers.
The module comment now says what an interrupted restore does (restarted
once, then failed) and what happens to other writes while a restore runs:
new rows are written at once, updates to rows the restore already deleted
wait on their lock until it commits or rolls back.
New structured leaf SURGICAL_HISTORY in the history group and in the
standard template: every procedure that took place, oldest first, with
its date, what was done (the visit's reason), the body site with its
side and the outcome. Lifetime reference data like the immunization
record, so it ignores the report window. The PDF prints it as a table
and the clinician share page renders it; saved selections do not widen.
The FHIR bundle carries no Procedure resource, on the same terms as
visits: no REST route or capability entry exists for it.
Refs #1025
A visit can be filed as a procedure or surgery (new EncounterKind
PROCEDURE) and carry a body site in the person's own words, encrypted at
rest, plus an optional side (LEFT, RIGHT, BOTH). An existing visit
switches to the new kind by an ordinary edit, keeping its date, reason,
outcome and links.
- migration 0349: enum value, laterality type, two nullable columns
- GET /api/encounters/procedures: the DONE procedures, newest first,
searched after the decrypt over site, side and reason, with body-site
choices and a total computed over the whole history
- create and edit accept bodySite and laterality on any kind
- /checkups gains a Procedures tab with a body-site filter; the visit
form offers the site for a procedure and keeps it in view on any
visit that carries one
- backup and restore carry both fields; the body site is registered as
ciphertext and rotates with the key
- MCP get_visits narrows by kind and reads the whole record for
procedures when no window is named; the Coach visit block names a
procedure's site
- i18n in all seven locales, OpenAPI regenerated
Refs #1025
A bridge such as Tasker or Home Assistant could only send sleep stages
through the batch route, in HealthKit's terms: the category identifier,
a stage codepoint from 0 to 5, and a value it had to work out in
minutes. It can now post one segment to `POST /api/measurements` with
`type: "SLEEP_DURATION"`, a `sleepStage` of `IN_BED`, `ASLEEP`, `AWAKE`,
`CORE`, `DEEP` or `REM`, and either `startDate` + `endDate` or
`measuredAt` (the end) + `value` (minutes).
The row is the one the batch route writes for the same segment, and the
dedupe is the batch route's: the opposite-source same-reading check,
then one reconcile over both unique identities. Without an `externalId`
the server derives one from the stage and both instants, so a re-post
answers 200 `duplicate`, a corrected start answers 200 `updated`, and a
new segment answers 201 `inserted`. The narrow `measurements:write`
token is accepted and still writes only MANUAL rows. Every other body
on this route behaves as before.
Requested in #972.
`/api/auth/me` has reported the resolved zone, the stored one or the
instance default when the stored value is unusable, while
`/api/user/profile` returned the raw column. A client reading the
profile could cut days differently from the server, and sending the
value back in a PATCH was refused.
Both reads, and the profile PATCH answer, now go through one
`resolveStoredTimezone`. The field stays a string and always holds a
valid IANA name. The web account editor already reads the resolved
zone from `/api/auth/me` and saves through the validating timezone
route, so it shows the zone in use and can only write a valid one.
The nutrient batch docs told a client to log and drop an `upsert_failed`
entry. The route means the opposite: a write group failed on the
server's side and nothing was stored. The docs now say to send the
entry again on a later sync, and keep the three validation skips
terminal.
A retry under the same Idempotency-Key used to replay the cached
failure, because the 200 carrying `upsert_failed` was cached like any
other. Both the nutrient and the cycle bulk routes now mark that
response `no-store`, as the measurement batch already does for
`failed`, so the retry writes again.
The cycle bulk route already answered its validation refusal with
`cycle.bulk.invalid` and an over-cap batch with `cycle.bulk.too_large`.
Both are now pinned by tests and named in the route's 422 response.
The validation 422 from `POST /api/measurements/batch` now carries
`meta.errorCode: "measurement.batch.invalid"`, beside the existing
`too_large` and `source_not_permitted` codes, so a client can file the
rows as permanently refused instead of holding its anchor on a guess.
A `value_out_of_range` entry now also carries `convertedValue`, the
reading after conversion into the stored unit, and `range`
(`min`, `max`, `unit`), the band it was checked against. Both appear
only on that skip. A fraction sent where a percent was expected is now
visible in the response instead of looking like a sensor glitch.
Status codes, the envelope and every existing field are unchanged.
Each `createManyAndReturn` flush of the export import left about 5 MB of
heap behind until some two dozen calls had piled up. A 300 000-record
synthetic import under a 524 MB heap settled at 53 MB retained; a run of
back-to-back 600-row calls with nothing between them reached 212 MB.
Spot rows now go through the column-array `INSERT … SELECT FROM unnest`
the restore already uses, with `ON CONFLICT DO NOTHING RETURNING`, which
is what `createMany({ skipDuplicates })` emits. The same import holds
11.5 MB and finishes in 35 s instead of 55 s. Ids are minted in the
cuid() shape, defaults are written as the client would write them, and
integration tests compare the returned rows and the resulting table with
`createManyAndReturn` for fresh rows, conflicts on either unique
identity, and duplicates inside one call.
Restoring a large account was one request: 94 s for 1.25 million readings.
A reverse proxy that cuts a request at 60 s (nginx's default) showed the
operator an error for a restore that went on to finish on the server.
The restore request now checks the typed confirmation, that the backup
exists and that its stored copy opens, admits one job for the account and
answers 202 with its id. The `backup-restore` queue runs the restore and
records its phase, counts and outcome on a `backup_restore_jobs` row
(migration 0347), which `GET /api/admin/backups/restores` and
`GET /api/admin/backups/restores/{id}` read back.
What a restore checks and writes is unchanged. The body moved from the
route into `restoreBackup` (src/lib/export/restore-backup.ts): every
refusal still happens before the first delete and the replacement is still
one transaction, so a failure part-way leaves the account as it was. While
it runs the account keeps showing its current data, and other writes are
neither refused nor held, as before.
- One restore per account: a partial unique index admits one queued or
running job, and a second request gets 409 `backup.restore.active` naming
the running job. A per-account advisory lock inside the transaction holds
even if a worker thought gone is still writing.
- Time budget: the job's expiry is two hours and a file whose transaction
limit would run past three quarters of it is refused before anything is
deleted (`jobDeadline`, beside `jobBudget`).
- Resuming: the job writes a heartbeat every two seconds. A boot sweep
queues a job whose worker stopped again from the start, at most twice,
then fails it as interrupted. The admission check releases an abandoned
job so it never holds the account's slot.
- The staged file is the stored `DataBackup` row, already encrypted at
rest. It stays whatever the outcome, so a retry needs no new upload. The
job records a digest of the copy and refuses one that was replaced while
it waited.
- Failures carry a stable code and a sentence that says what to do.
- The console follows the job: phase and counts while it runs, the outcome
after, and after a reload it finds the job again from the server. A poll
that fails on the network is never shown as a failed restore. A web
process separate from the worker evicts the account's caches when it
first reports the restore finished. The success line goes through the
outcome module, so an empty file is not drawn as a tick.
The model is registered in the backup plan (not carried) and the wipe plan
(wiped with the account); the queue in the maintenance registrar, the
subsystem surface and the job-fact allowlist; the new code in the error
catalogue; the routes in the OpenAPI contract. The tests that pin restore
semantics now drive the real route and the real job through one helper, and
`backup-restore-job.test.ts` pins the asynchronous contract itself.
The calendar grid carried flow, symptoms, BBT, the ovulation test and the
mucus reading, and dropped the rest of the day log: intercourse, spotting,
the pregnancy and progesterone tests, contraception and whether a note
exists. The published contract also promised the three cervix signs while
the route never sent them.
The intent fields resolve through the same reader the day-log read uses,
so an encrypted envelope reads the same on both, and the grid is served
under the same cycle grant as the day log. Additive fields; the contract
test pins the wire keys to the published schema.
Review caught a behaviour change this branch causes but never wrote down.
`pickCanonicalSourceRows` takes the first ladder source present in a day's
bucket and filters the day to that source. Before, a typed reading and a
bridged one were both MANUAL, so both survived. Now MANUAL ranks, EXTERNAL
does not, and the bridged row leaves the canonical set on any day that also
has a typed one.
That reaches every surface needing one source per day — the daily totals
behind the charts, the health score, the doctor report, correlations, mood
crosstabs and personal-record detection. A day with only bridged readings is
untouched: nothing ranked is present, so the picker's pass-through keeps
everything.
The changelog says so in the user's terms, including the part that is a fix
rather than a change: two indistinguishable MANUAL rows were both counted,
which inflated a running total on a day somebody did both.
Four cases pin it, because it is emergent from a rank ladder and stated in
no single place: the contested day, the bridged-only day, the both-MANUAL
shape the change moves away from, and a stored ladder that names EXTERNAL
being honoured. Ranking EXTERNAL turns the first two red, which is the
review anybody adding it to a ladder should have to pass.
The measurements list already badges any row whose source is not MANUAL and
already builds its source filter from the shared enum, so both pick the new
value up on their own. What it needs is a name: `EXTERNAL` reads as a leaked
enum constant, and it is the only new value a user is expected to operate,
since it is an option in the filter and a badge on every bridged row.
Both label catalogues gain an entry. They are keyed by the enum, so a new
source without a label is a type error and the coverage guard fails the
suite — which is exactly what happened here, and is the reason this carries
a settings label too even though EXTERNAL sits on no priority ladder and so
never renders there.
The settings card's scope note and the Home Assistant guide both claimed
readings arrive "as if you had entered them by hand", which is no longer
true. Both now say the rows are marked as coming from an external device and
are still yours to correct, and the guide spells out that the payload must
not name a source at all.
The changelog entries go under Unreleased as Changed: v1.38.1 shipped the
token, so these describe a change to something people already have. They say
what that means in practice — rows written before this keep the old label
and cannot be sorted out afterwards, a batch re-push can duplicate because
the dedup key now carries the source, and an automation sending an explicit
`source` field has to drop it.
Adds the `EXTERNAL_SOURCE` leaf beside `scopes.ts` and mirrors the value in
`measurementSourceEnum` so the read shapes and the sync decoder see it. It
stays out of `WRITABLE_MEASUREMENT_SOURCES`: a source a client could assert
would prove nothing about where a reading came from, and that constant is
published to clients as `ingest.writeAllowlist`.
The value-edit gate in `PUT /api/measurements/[id]` needs the other question
answered, though — whether the number is the user's own to fix — so it now
reads a separate `USER_CORRECTABLE_MEASUREMENT_SOURCES`. The 409 that
protects a Withings row exists because the value is the provider's reading;
the hardware behind an ingest token is the user's own scale, and locking
those rows would contradict what the settings card and the Home Assistant
guide promise. That is the only site reading the wider set.
The one-shot hourly rebuild of already-folded heart-rate, HRV and SpO2
days walks one account's whole folded history in one job with the
default fifteen-minute expiry, about fifty statements per day. Like the
drains it now stops between days once 75 % of the expiry has passed or
pg-boss aborts the job, and fails so pg-boss retries it (the boot send
allows three retries); a rebuilt day leaves the candidate set, so the
retry starts at the first day not yet rebuilt.
The off-host backup walked every account in one job with the default
fifteen-minute expiry, and a retry started again from the first account.
On an instance with one large record next to others, an attempt that
ran out of time spent its retry uploading the accounts it had already
done (#1031).
The run now stops between accounts once the job's time budget is spent
and fails, so pg-boss retries it. Every attempt of a job carries the same
creation time; an attempt skips the accounts whose ledger shows a success
since then, and keys its objects on that day, so a retry after midnight
still writes under the run's date. A new job uploads every account again,
and a retry whose accounts were all done passes.
After an Apple Health import the worker folded the rollup table from the
account's first measurement to its last, whatever the import carried, and
so also folded history older than the five-year window that the boot
backfill, its discovery and every read path keep to.
The import now records the oldest and newest instant it wrote
(`measuredSpan` on the job result), and the fold covers that span,
clamped to the fold window. On the seeded account of 1.25 million rows the
full fold took 12 s, the slowest statement 2.7 s against the 60 s
statement limit, so this is about the window rather than time.
Key rotation and the rotation status read every column that was not
flagged as a blob with one `findMany` over the whole table.
`Measurement.notesEncrypted` and `MoodEntry.noteEncrypted` are such
columns, so rotation loaded every measurement of the instance: 1.25
million rows on one large account (#1031), and the rotation status page
did the same on every view.
Every column is now walked with the id-cursor pages the blob columns
already used: 25 rows a page for blobs, 5 000 for everything else. The
maintenance script's own per-column walks page the same way. On the seeded
account with 1 460 notes to rotate, peak heap fell from 272 MB to 153 MB,
and it no longer grows with the table.
`scanned` now counts the rows that hold ciphertext for every column, as
it already did for the blob columns, rather than every row of the table
for some columns and only the ciphertext rows for others. The runbook
says so.
The repair for the inflated cumulative records deleted the suspect rows,
committed, and then re-ran the detector. When the detector failed, which
a timeout on a large account or a restart can cause, the records stayed
deleted, and since discovery looks for the suspect rows the account was
never repaired again (#1031).
The delete and the re-derivation now run in one transaction, with the
detector reading through it, so they commit together or not at all. The
transaction gets five minutes; the detector took 19.8 s on the largest
seeded account.
The personal-record pass, every half hour for every account, read every
live row of each cumulative type into memory to find the best day. With
784 000 active-energy rows on one account (the per-sample rows pile up
when the nightly drain cannot finish, #1031) it ran a 524 MB heap out of
memory; the old code needed 1.1 GB of heap for that account.
It now reads the type in bounded keyset pages and handles one local day
at a time. The canonical-source pick it applies is per day, so the day
totals and the winner are the same; a tie still keeps the earliest day.
Measured on the same account: 189 MB peak heap, the same record. The
warm-up gate counts only up to its threshold instead of counting a whole
history.
`iterateSourcePages` and `iterateDayBuckets` in the consolidation base
take the row type as a parameter, and the day grouping can keep every row.
An account of 1.25 million measurements could not export its data or be
restored from any backup (#1031). Measured in a 1 GB container:
- The full export and the passphrase export built the whole payload and
its JSON string in memory; the app ran out of heap and restarted.
- The weekly backup's JSON is 662 MB, longer than any string V8 can hold,
so `unpackBackupBlob` threw on every read. Restore, download and the
restore preview all answered "could not be decrypted".
- Upload accepted 10 MB, a fraction of a large export.
Every reader and writer of a backup file now streams it:
- `scanBackupJson` reads a backup's bytes once, keeps every section but
the named bulk arrays, and hands their elements out one at a time.
`readStreamedBackup` uses it twice: a first pass that validates each
measurement against the element schema and keeps the rest of the file,
and a second pass that hands the measurements to the writer in batches.
Stored copies and off-host objects are authenticated whole on their
compressed ciphertext before any plaintext is read; only the
decompression is streamed.
- Restore writes the measurements with one `INSERT … FROM unnest` per
batch. Prisma caches a query plan per `createMany` row count, and one
1 000-row plan kept about 145 MB alive, which alone put the restore past
the heap limit. The transaction budget grows with the measurement count.
The awards and ECG sections are handed the ids they reference that were
written, not every written id.
- The export routes write the file into the response as it is produced.
The passphrase export spools its ciphertext to a temporary file because
the HLX1 format puts the tag first; the format is unchanged, so existing
archives open as before.
- Upload accepts the raw body, JSON or gzip, up to the 512 MB request
ceiling, checks it as it arrives and stores it in the same pass; the admin
page compresses the file in the browser before sending it. Multipart stays
for small files.
- Download streams the stored JSON; the preview counts measurements from
the stream. The monthly restore drill and `scripts/restore-backup.ts`
read the off-host object the same way; the script can write `.json.gz`.
On the seeded account in a 1 GB container: export 25 s at 389 MB peak,
passphrase export 25 s, both uploads 19 s, restore of the weekly backup
94 s at 554 MB peak, and export, upload and restore round-trip to exactly
the same rows. The encrypted-column guard now also recognises the raw SQL
statement that writes `DataBackup.data`.
The backup streamed its JSON through gzip and the cipher, but kept the
stored copy as one string and handed that to Postgres in one parameter.
On a seeded account of 1.25 million measurements the copy is 64 MB of
base64, and the pieces, the joined string and the driver's copies of it
grew the process from 309 MB to 946 MB resident. A bare process needed a
heap limit above 400 MB to finish it (the new path finishes under 220 MB).
With the dense retention running beside it in a 1 GB container the
process died of heap exhaustion, and the backup job sat "active" until its
two-hour limit marked it "job timed out" (#1031).
`packBackupBlobInto` hands the stored copy on in pieces of up to 4 MB as
it is produced, and `storeBackupBlob` writes them into a temporary table
on its own transaction and assembles the row with one `string_agg`. The
process never holds the whole copy. The account's previous copy stays in
place until the new one is complete: a failure rolls back to it.
Measured on the seeded account in a 1 GB container, web and worker in one
process: the backup completes in about 30 s with the dense retention
running beside it, where v1.39.0 crashed within 30 s.
The streamed JSON also reports its progress to the job every 50 000 rows.
Marc-André Bombeck