Commit Graph
6757 Commits
Author SHA1 Message Date
Marc-André Bombeck 0095fda216 test(medications): count only a use of the intake-tracking vocabulary, not its import 2026-09-25 09:01:01 +02:00
Marc-André Bombeck b4c2706ec9 feat(medications): keep a medication as a record without tracking intake
A per-medication trackIntake flag (migration 0351, default on for every
existing and new row). With it off the dose, dates and schedule stay stored,
but nothing derives a due dose from them: the reminder worker, the manual
reminder check, the today projector, the auto-miss pass, the low-stock pass,
the doses card, the today list, the outstanding-dose badge, compliance and
its heatmap, the doctor report, the Coach, the BP-status gate, insights,
targets, achievements, the efficacy view and the MCP schedule tool all skip
it.

The medication wire serves such a medication with schedules: [] and the
stored rows in an additive recordedSchedules field, so a client that arms
local reminders from schedules stays silent without an update. Switching
tracking off or back on archives a schedule revision at that instant, the
untracked stretch with an empty payload, so it never counts as missed and
the projector does not put a dose from before the switch back on the card.

A structural guard freezes the set of schedule readers: each one filters on
the flag, derives from dueSchedules, shapes the wire with scheduleWireFields,
or is allowlisted with the reason it derives nothing due. The flag travels
through backup, restore and the medication CSV export.
2026-09-25 08:57:10 +02:00
Marc-André Bombeck 700de6d228 feat(medications): add the track-intake switch to the medication pages
The detail page leads the schedule tab with the switch and a one-line
explanation of each state; with tracking off it drops the reminder card,
the next-dose and adherence rows, and shows the stored schedule as the
record. The wizard summary carries the same switch. Cards, the table and
the detail header mark a record-only medication, which leaves the take and
skip buttons, the take-all set, the log-dose dialog and the dashboard quick
add. Strings in all seven locales.
2026-09-25 08:57:10 +02:00
Marc-André Bombeck 934131ee3e Merge branch 'feat/restore-background-job' into release/v1.39.1 2026-09-25 08:10:06 +02:00
Marc-André Bombeck cb69f662ab fix(measurements): attribute named sleep stages from a bridge token to EXTERNAL
The named-stage arm took the body's source and so wrote MANUAL under a
scoped credential, while every other arm of the route resolves such rows
to EXTERNAL. It now resolves the source from the credential as well, and
a bridge segment stays apart from the same reading from Apple Health.
2026-09-25 08:04:31 +02:00
Marc-André Bombeck d0e321c54a Merge branch 'fix/v1391-contract-and-ingest' into release/v1.39.1
# Conflicts:
#	docs/api/openapi.yaml
#	src/lib/openapi/routes/measurements.ts
2026-09-25 08:04:30 +02:00
Marc-André Bombeck 7e654175b6 test(contract): pin the v1.39.1 wire changes as additive for shipped clients
Decoders written against the shapes the shipped iPhone builds read still decode every payload this release changes, and the old shapes still satisfy the new document: the new batch-entry fields and the error code stay optional, the profile time zone stays a required string, and the named sleep-stage answer sits beside the old create response rather than in place of it.
2026-09-25 07:39:07 +02:00
Marc-André Bombeck f443ab6615 feat(measurements): accept sleep stages by name on POST /api/measurements
A bridge such as Tasker or Home Assistant could only send sleep stages
through the batch route, in HealthKit's terms: the category identifier,
a stage codepoint from 0 to 5, and a value it had to work out in
minutes. It can now post one segment to `POST /api/measurements` with
`type: "SLEEP_DURATION"`, a `sleepStage` of `IN_BED`, `ASLEEP`, `AWAKE`,
`CORE`, `DEEP` or `REM`, and either `startDate` + `endDate` or
`measuredAt` (the end) + `value` (minutes).

The row is the one the batch route writes for the same segment, and the
dedupe is the batch route's: the opposite-source same-reading check,
then one reconcile over both unique identities. Without an `externalId`
the server derives one from the stage and both instants, so a re-post
answers 200 `duplicate`, a corrected start answers 200 `updated`, and a
new segment answers 201 `inserted`. The narrow `measurements:write`
token is accepted and still writes only MANUAL rows. Every other body
on this route behaves as before.

Requested in #972.
2026-09-25 07:39:07 +02:00
Marc-André Bombeck be112df972 fix(profile): report the time zone the server uses on /api/user/profile
`/api/auth/me` has reported the resolved zone, the stored one or the
instance default when the stored value is unusable, while
`/api/user/profile` returned the raw column. A client reading the
profile could cut days differently from the server, and sending the
value back in a PATCH was refused.

Both reads, and the profile PATCH answer, now go through one
`resolveStoredTimezone`. The field stays a string and always holds a
valid IANA name. The web account editor already reads the resolved
zone from `/api/auth/me` and saves through the validating timezone
route, so it shows the zone in use and can only write a valid one.
2026-09-25 07:39:07 +02:00
Marc-André Bombeck 0cc73d3c88 fix(ingest): treat upsert_failed as a retry in the nutrient and cycle batches
The nutrient batch docs told a client to log and drop an `upsert_failed`
entry. The route means the opposite: a write group failed on the
server's side and nothing was stored. The docs now say to send the
entry again on a later sync, and keep the three validation skips
terminal.

A retry under the same Idempotency-Key used to replay the cached
failure, because the 200 carrying `upsert_failed` was cached like any
other. Both the nutrient and the cycle bulk routes now mark that
response `no-store`, as the measurement batch already does for
`failed`, so the retry writes again.

The cycle bulk route already answered its validation refusal with
`cycle.bulk.invalid` and an over-cap batch with `cycle.bulk.too_large`.
Both are now pinned by tests and named in the route's 422 response.
2026-09-25 07:39:07 +02:00
Marc-André Bombeck a7682a9fed fix(measurements): name the batch validation refusal and the band an entry broke
The validation 422 from `POST /api/measurements/batch` now carries
`meta.errorCode: "measurement.batch.invalid"`, beside the existing
`too_large` and `source_not_permitted` codes, so a client can file the
rows as permanently refused instead of holding its anchor on a guess.

A `value_out_of_range` entry now also carries `convertedValue`, the
reading after conversion into the stored unit, and `range`
(`min`, `max`, `unit`), the band it was checked against. Both appear
only on that skip. A fraction sent where a percent was expected is now
visible in the response instead of looking like a sensor glitch.

Status codes, the envelope and every existing field are unchanged.
2026-09-25 07:39:07 +02:00
Marc-André Bombeck 74dedee645 fix(import): write Apple Health spot rows without Prisma's per-call heap
Each `createManyAndReturn` flush of the export import left about 5 MB of
heap behind until some two dozen calls had piled up. A 300 000-record
synthetic import under a 524 MB heap settled at 53 MB retained; a run of
back-to-back 600-row calls with nothing between them reached 212 MB.

Spot rows now go through the column-array `INSERT … SELECT FROM unnest`
the restore already uses, with `ON CONFLICT DO NOTHING RETURNING`, which
is what `createMany({ skipDuplicates })` emits. The same import holds
11.5 MB and finishes in 35 s instead of 55 s. Ids are minted in the
cuid() shape, defaults are written as the client would write them, and
integration tests compare the returned rows and the resulting table with
`createManyAndReturn` for fresh rows, conflicts on either unique
identity, and duplicates inside one call.
2026-09-25 07:39:07 +02:00
Marc-André Bombeck 47e9f7e81c feat(backup): run a restore as a background job with progress
Restoring a large account was one request: 94 s for 1.25 million readings.
A reverse proxy that cuts a request at 60 s (nginx's default) showed the
operator an error for a restore that went on to finish on the server.

The restore request now checks the typed confirmation, that the backup
exists and that its stored copy opens, admits one job for the account and
answers 202 with its id. The `backup-restore` queue runs the restore and
records its phase, counts and outcome on a `backup_restore_jobs` row
(migration 0347), which `GET /api/admin/backups/restores` and
`GET /api/admin/backups/restores/{id}` read back.

What a restore checks and writes is unchanged. The body moved from the
route into `restoreBackup` (src/lib/export/restore-backup.ts): every
refusal still happens before the first delete and the replacement is still
one transaction, so a failure part-way leaves the account as it was. While
it runs the account keeps showing its current data, and other writes are
neither refused nor held, as before.

- One restore per account: a partial unique index admits one queued or
  running job, and a second request gets 409 `backup.restore.active` naming
  the running job. A per-account advisory lock inside the transaction holds
  even if a worker thought gone is still writing.
- Time budget: the job's expiry is two hours and a file whose transaction
  limit would run past three quarters of it is refused before anything is
  deleted (`jobDeadline`, beside `jobBudget`).
- Resuming: the job writes a heartbeat every two seconds. A boot sweep
  queues a job whose worker stopped again from the start, at most twice,
  then fails it as interrupted. The admission check releases an abandoned
  job so it never holds the account's slot.
- The staged file is the stored `DataBackup` row, already encrypted at
  rest. It stays whatever the outcome, so a retry needs no new upload. The
  job records a digest of the copy and refuses one that was replaced while
  it waited.
- Failures carry a stable code and a sentence that says what to do.
- The console follows the job: phase and counts while it runs, the outcome
  after, and after a reload it finds the job again from the server. A poll
  that fails on the network is never shown as a failed restore. A web
  process separate from the worker evicts the account's caches when it
  first reports the restore finished. The success line goes through the
  outcome module, so an empty file is not drawn as a tick.

The model is registered in the backup plan (not carried) and the wipe plan
(wiped with the account); the queue in the maintenance registrar, the
subsystem surface and the job-fact allowlist; the new code in the error
catalogue; the routes in the OpenAPI contract. The tests that pin restore
semantics now drive the real route and the real job through one helper, and
`backup-restore-job.test.ts` pins the asynchronous contract itself.
2026-09-25 07:39:06 +02:00
Marc-André Bombeck ee23409b77 Merge branch 'fix/v1391-cycle-and-links' into release/v1.39.1 2026-09-25 07:38:34 +02:00
Marc-André Bombeck 17c31c37a5 fix(links): offer the unlink undo inside the sheet
The picker lives inside a modal sheet, and a toast sits outside it where
its button cannot be pressed while the sheet is open. The undo is now a
line under the chips that stays until the next change. The calendar's
intercourse diamond also grows a step so it reads as a diamond on a
phone rather than a dot.
2026-09-25 07:30:36 +02:00
Marc-André Bombeck 92ad5513c0 chore(db): renumber the placeholder repair migration to 0350
0346 is taken by the external-source migration that reached main first.
The repair touches only intake rows and depends on no other migration of
this release.
2026-09-25 07:25:29 +02:00
Marc-André Bombeck 702a13873d Merge branch 'main' into release/v1.39.1 2026-09-25 07:21:34 +02:00
Marc-André Bombeck 47e2520b9f Merge pull request #892 from Antiheld86/feat/measurement-source-external
Give ingested measurements their own source value
2026-09-25 07:21:30 +02:00
Marc-André Bombeck 6e05f86e7b fix(vaccinations): show a dose's linked documents and keep them on edit
A dose opened from the list carried no document links, since only the
detail read has them. The edit sheet showed an empty picker, and saving
any change sent that empty set, which the route reads as unlink
everything. The sheet now reads the dose's own links before it shows
them, and a save leaves the links alone unless they were changed.

A link to one dose or visit (/vaccinations?dose=, /checkups?visit=) now
scrolls to it and opens it for someone who may edit it.
2026-09-25 07:10:56 +02:00
Marc-André Bombeck 31f0d7c9fb fix(links): make the X the only unlink and let a chip open its record
A link chip was one button with an X drawn inside it, so tapping the
chip to open the linked record removed the link. The label now opens the
record (a dose, a visit, a condition or a document) and a separate X,
labelled with what it removes, unlinks, with an undo in the toast. Each
chip also carries its record's date, so several doses of one vaccine no
longer read as identical chips. The fix sits in the shared picker, which
is every chip list with this shape.
2026-09-25 07:10:50 +02:00
Marc-André Bombeck 0390d124cb fix(cycle): mark intercourse and other logged entries on the calendar
An intercourse entry left no trace on the month grid and the legend had
no entry for it. The day now carries a diamond marker, and anything else
logged that day with no marker of its own (a test, contraception,
spotting, a temperature, a mucus or cervix reading, a note) shows a
hollow ring, so a day with one of those logged no longer looks empty.
Both are named in the legend and in the cell's accessible label.
2026-09-25 07:05:03 +02:00
Marc-André Bombeck adec5bd045 fix(cycle): carry every logged day entry on the calendar read
The calendar grid carried flow, symptoms, BBT, the ovulation test and the
mucus reading, and dropped the rest of the day log: intercourse, spotting,
the pregnancy and progesterone tests, contraception and whether a note
exists. The published contract also promised the three cervix signs while
the route never sent them.

The intent fields resolve through the same reader the day-log read uses,
so an encrypted envelope reads the same on both, and the grid is served
under the same cycle grant as the day log. Additive fields; the contract
test pins the wire keys to the published schema.
2026-09-25 07:02:14 +02:00
Marc-André Bombeck 510dbfe273 chore(db): renumber the external-source migration to 0346
0342 sorts before migrations 0343 to 0345, which are already applied on
v1.39.0 instances. The next free number keeps the order linear.
2026-09-25 06:55:22 +02:00
Marc-André Bombeck 1205540585 chore(db): renumber the external-source migration to 0342
0336 was taken by the off-host backup ledger while this branch waited; the
schema change itself is unchanged.
2026-09-25 06:55:18 +02:00
Marc-André Bombeck e7597c7ac1 docs(changelog): leave the entry to the release commit
The text is unchanged and waits with the other release drafts; keeping it
in the branch made CHANGELOG.md collide with every release that shipped
while this waited.
2026-09-25 06:55:18 +02:00
Antiheld86 a099d813ee Document what EXTERNAL does to a contested day, and pin it
Review caught a behaviour change this branch causes but never wrote down.
`pickCanonicalSourceRows` takes the first ladder source present in a day's
bucket and filters the day to that source. Before, a typed reading and a
bridged one were both MANUAL, so both survived. Now MANUAL ranks, EXTERNAL
does not, and the bridged row leaves the canonical set on any day that also
has a typed one.

That reaches every surface needing one source per day — the daily totals
behind the charts, the health score, the doctor report, correlations, mood
crosstabs and personal-record detection. A day with only bridged readings is
untouched: nothing ranked is present, so the picker's pass-through keeps
everything.

The changelog says so in the user's terms, including the part that is a fix
rather than a change: two indistinguishable MANUAL rows were both counted,
which inflated a running total on a day somebody did both.

Four cases pin it, because it is emergent from a rank ladder and stated in
no single place: the contested day, the bridged-only day, the both-MANUAL
shape the change moves away from, and a stored ladder that names EXTERNAL
being honoured. Ranking EXTERNAL turns the first two red, which is the
review anybody adding it to a ladder should have to pass.
2026-09-25 06:55:17 +02:00
Antiheld86 9fc1413181 docs(measurements): label EXTERNAL and update the guide and contract
The measurements list already badges any row whose source is not MANUAL and
already builds its source filter from the shared enum, so both pick the new
value up on their own. What it needs is a name: `EXTERNAL` reads as a leaked
enum constant, and it is the only new value a user is expected to operate,
since it is an option in the filter and a badge on every bridged row.

Both label catalogues gain an entry. They are keyed by the enum, so a new
source without a label is a type error and the coverage guard fails the
suite — which is exactly what happened here, and is the reason this carries
a settings label too even though EXTERNAL sits on no priority ladder and so
never renders there.

The settings card's scope note and the Home Assistant guide both claimed
readings arrive "as if you had entered them by hand", which is no longer
true. Both now say the rows are marked as coming from an external device and
are still yours to correct, and the guide spells out that the payload must
not name a source at all.

The changelog entries go under Unreleased as Changed: v1.38.1 shipped the
token, so these describe a change to something people already have. They say
what that means in practice — rows written before this keep the old label
and cannot be sorted out afterwards, a batch re-push can duplicate because
the dedup key now carries the source, and an automation sending an explicit
`source` field has to drop it.
2026-09-25 06:55:17 +02:00
Antiheld86 112a029443 test(measurements): cover EXTERNAL provenance on both write paths
The write-smoke suite is `satisfies Record<MeasurementSource, SourceCase>`,
so growing the enum is a compile error until a case exists; the new one
goes through the ordinary single-entry POST, which is the production path.
It carries the trap this suite sets for you written down: `beforeEach`
seeds a session cookie and the caller resolver is cookie-first, so arming
only the authorization header resolves as that session and quietly writes
MANUAL — the case would then pass without testing anything. Both hardcoded
totals move 14 → 15.

In the bearer-scope suite the two rows that asserted MANUAL now assert
EXTERNAL, and three cases join them: a body naming MANUAL is refused on
each route (the widening from "names APPLE_HEALTH" to "names anything",
which nothing else covered), and a bridge-written row is still value-
editable by its owner through PUT. That last one guards the split between
the write allowlist and the correctable set — folding the two constants
back together value-locks every bridge row behind a 409, and this is what
notices.
2026-09-25 06:55:17 +02:00
Antiheld86 f94d358ade feat(measurements): write EXTERNAL from both ingest write paths
Under a `measurements:write` credential the row's source is now decided by
the server on both routes, and the body may not name one at all.

The 422 widens from "names APPLE_HEALTH" to "names any source". It was
narrow only because the forced value was MANUAL, so a body saying MANUAL
was never actually overridden. Now it would be — and overriding a client's
explicit assertion quietly is the thing both routes' comments rule out,
while honouring it would defeat the change: a bridge that kept sending
MANUAL would stay indistinguishable from a typed reading. Status and
errorCode are unchanged, and the documented Home Assistant payload sends no
source at all.

The single-POST predicate keeps reading the raw body: the create schema
defaults `source` to MANUAL, so after parsing "said MANUAL" and "said
nothing" are the same value. Both of that route's create sites resolve the
source, the array arm included — that is the combined BP + Pulse path a
bridge uses, and missing it would leave exactly those rows unbadged.

Two consequences worth naming. Bridge rows no longer join the
MANUAL/APPLE_HEALTH cross-source merge, so they can't be swallowed by a
rule written for the iOS adopt-on-pair case. And a bridge posting at the
same instant as a hand entry now lands both rows instead of a 409, which is
honest and does not double-count: the rollup collapse is per
(type, day, source) and MANUAL outranks the unlisted EXTERNAL.
2026-09-25 06:55:17 +02:00
Antiheld86 5ec361abec feat(measurements): mirror EXTERNAL in Zod, keep those rows correctable
Adds the `EXTERNAL_SOURCE` leaf beside `scopes.ts` and mirrors the value in
`measurementSourceEnum` so the read shapes and the sync decoder see it. It
stays out of `WRITABLE_MEASUREMENT_SOURCES`: a source a client could assert
would prove nothing about where a reading came from, and that constant is
published to clients as `ingest.writeAllowlist`.

The value-edit gate in `PUT /api/measurements/[id]` needs the other question
answered, though — whether the number is the user's own to fix — so it now
reads a separate `USER_CORRECTABLE_MEASUREMENT_SOURCES`. The 409 that
protects a Withings row exists because the value is the provider's reading;
the hardware behind an ingest token is the user's own scale, and locking
those rows would contradict what the settings card and the Home Assistant
guide promise. That is the only site reading the wider set.
2026-09-25 06:55:17 +02:00
Antiheld86 c2999c955d feat(measurements): add EXTERNAL measurement source
Rows pushed in through a `measurements:write` Bearer token are attributed
`MANUAL` today, so afterwards nobody can tell which readings came off the
scale and which were typed by hand. Give that write surface its own
`MeasurementSource` value, the way 0189 did for TELEGRAM and 0207 did for
MCP.

Enum extension only: purely additive, no row touched, `IF NOT EXISTS` makes
a rerun safe, and the value is not USED in the migration so Postgres'
same-transaction restriction does not bite. Declared last in the datamodel
so the Prisma member order and the physical order stay identical.

No backfill, and none is possible — `measurements` carries no actor or
token column, so a bridge-written MANUAL row is byte-identical to a typed
one. Existing rows stay MANUAL.
2026-09-25 06:55:17 +02:00
Marc-André Bombeck d81518b5bf Merge branch 'fix/large-account-jobs' into release/v1.39.1 2026-09-25 06:49:16 +02:00
Marc-André Bombeck e9449ece61 Merge branch 'fix/dose-history-pre-creation-slots' into release/v1.39.1 2026-09-25 06:49:15 +02:00
Marc-André Bombeck c757981638 fix(jobs): give the hourly rebuild the same time budget
The one-shot hourly rebuild of already-folded heart-rate, HRV and SpO2
days walks one account's whole folded history in one job with the
default fifteen-minute expiry, about fifty statements per day. Like the
drains it now stops between days once 75 % of the expiry has passed or
pg-boss aborts the job, and fails so pg-boss retries it (the boot send
allows three retries); a rebuilt day leaves the candidate set, so the
retry starts at the first day not yet rebuilt.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck b7a26d3a4d fix(backup): resume an off-host run where its last attempt stopped
The off-host backup walked every account in one job with the default
fifteen-minute expiry, and a retry started again from the first account.
On an instance with one large record next to others, an attempt that
ran out of time spent its retry uploading the accounts it had already
done (#1031).

The run now stops between accounts once the job's time budget is spent
and fails, so pg-boss retries it. Every attempt of a job carries the same
creation time; an attempt skips the accounts whose ledger shows a success
since then, and keys its objects on that day, so a retry after midnight
still writes under the run's date. A new job uploads every account again,
and a retry whose accounts were all done passes.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck 4242b454ce fix(import): fold rollups over the span the import wrote
After an Apple Health import the worker folded the rollup table from the
account's first measurement to its last, whatever the import carried, and
so also folded history older than the five-year window that the boot
backfill, its discovery and every read path keep to.

The import now records the oldest and newest instant it wrote
(`measuredSpan` on the job result), and the fold covers that span,
clamped to the fold window. On the seeded account of 1.25 million rows the
full fold took 12 s, the slowest statement 2.7 s against the 60 s
statement limit, so this is about the window rather than time.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck 0a54a43799 fix(crypto): walk every encrypted column in bounded pages
Key rotation and the rotation status read every column that was not
flagged as a blob with one `findMany` over the whole table.
`Measurement.notesEncrypted` and `MoodEntry.noteEncrypted` are such
columns, so rotation loaded every measurement of the instance: 1.25
million rows on one large account (#1031), and the rotation status page
did the same on every view.

Every column is now walked with the id-cursor pages the blob columns
already used: 25 rows a page for blobs, 5 000 for everything else. The
maintenance script's own per-column walks page the same way. On the seeded
account with 1 460 notes to rotate, peak heap fell from 272 MB to 153 MB,
and it no longer grows with the table.

`scanned` now counts the rows that hold ciphertext for every column, as
it already did for the blob columns, rather than every row of the table
for some columns and only the ciphertext rows for others. The runbook
says so.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck 7071fa91af fix(records): repair cumulative records in one transaction
The repair for the inflated cumulative records deleted the suspect rows,
committed, and then re-ran the detector. When the detector failed, which
a timeout on a large account or a restart can cause, the records stayed
deleted, and since discovery looks for the suspect rows the account was
never repaired again (#1031).

The delete and the re-derivation now run in one transaction, with the
detector reading through it, so they commit together or not at all. The
transaction gets five minutes; the detector took 19.8 s on the largest
seeded account.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck 77bfa2f4e1 perf(records): find the best cumulative day a day at a time
The personal-record pass, every half hour for every account, read every
live row of each cumulative type into memory to find the best day. With
784 000 active-energy rows on one account (the per-sample rows pile up
when the nightly drain cannot finish, #1031) it ran a 524 MB heap out of
memory; the old code needed 1.1 GB of heap for that account.

It now reads the type in bounded keyset pages and handles one local day
at a time. The canonical-source pick it applies is per day, so the day
totals and the winner are the same; a tie still keeps the earliest day.
Measured on the same account: 189 MB peak heap, the same record. The
warm-up gate counts only up to its threshold instead of counting a whole
history.

`iterateSourcePages` and `iterateDayBuckets` in the consolidation base
take the row type as a parameter, and the day grouping can keep every row.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck 237acc0002 fix(backup): export, upload and restore a large account as streams
An account of 1.25 million measurements could not export its data or be
restored from any backup (#1031). Measured in a 1 GB container:

- The full export and the passphrase export built the whole payload and
  its JSON string in memory; the app ran out of heap and restarted.
- The weekly backup's JSON is 662 MB, longer than any string V8 can hold,
  so `unpackBackupBlob` threw on every read. Restore, download and the
  restore preview all answered "could not be decrypted".
- Upload accepted 10 MB, a fraction of a large export.

Every reader and writer of a backup file now streams it:

- `scanBackupJson` reads a backup's bytes once, keeps every section but
  the named bulk arrays, and hands their elements out one at a time.
  `readStreamedBackup` uses it twice: a first pass that validates each
  measurement against the element schema and keeps the rest of the file,
  and a second pass that hands the measurements to the writer in batches.
  Stored copies and off-host objects are authenticated whole on their
  compressed ciphertext before any plaintext is read; only the
  decompression is streamed.
- Restore writes the measurements with one `INSERT … FROM unnest` per
  batch. Prisma caches a query plan per `createMany` row count, and one
  1 000-row plan kept about 145 MB alive, which alone put the restore past
  the heap limit. The transaction budget grows with the measurement count.
  The awards and ECG sections are handed the ids they reference that were
  written, not every written id.
- The export routes write the file into the response as it is produced.
  The passphrase export spools its ciphertext to a temporary file because
  the HLX1 format puts the tag first; the format is unchanged, so existing
  archives open as before.
- Upload accepts the raw body, JSON or gzip, up to the 512 MB request
  ceiling, checks it as it arrives and stores it in the same pass; the admin
  page compresses the file in the browser before sending it. Multipart stays
  for small files.
- Download streams the stored JSON; the preview counts measurements from
  the stream. The monthly restore drill and `scripts/restore-backup.ts`
  read the off-host object the same way; the script can write `.json.gz`.

On the seeded account in a 1 GB container: export 25 s at 389 MB peak,
passphrase export 25 s, both uploads 19 s, restore of the weekly backup
94 s at 554 MB peak, and export, upload and restore round-trip to exactly
the same rows. The encrypted-column guard now also recognises the raw SQL
statement that writes `DataBackup.data`.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck 79ed62b7f9 chore(scripts): add a seed for an account the size of a multi-year import
`scripts/seed-large-account.ts` inserts about 1.25 million measurements
for one user, shaped like an Apple Health export plus the iOS sync: dense
heart rate, HRV and SpO2, the high-frequency mean types, export `stats:`
daily rows, recent per-sample cumulative rows and nightly sleep stages.
Everything is generated in Postgres, so a run takes seconds, and it ends
with ANALYZE. It is how the maintenance jobs in #1031 were reproduced and
measured; never point it at a database holding real data.
2026-09-25 00:32:10 +02:00
Marc-André Bombeck 9411a485c8 fix(backup): store the weekly and manual backup piece by piece
The backup streamed its JSON through gzip and the cipher, but kept the
stored copy as one string and handed that to Postgres in one parameter.
On a seeded account of 1.25 million measurements the copy is 64 MB of
base64, and the pieces, the joined string and the driver's copies of it
grew the process from 309 MB to 946 MB resident. A bare process needed a
heap limit above 400 MB to finish it (the new path finishes under 220 MB).
With the dense retention running beside it in a 1 GB container the
process died of heap exhaustion, and the backup job sat "active" until its
two-hour limit marked it "job timed out" (#1031).

`packBackupBlobInto` hands the stored copy on in pieces of up to 4 MB as
it is produced, and `storeBackupBlob` writes them into a temporary table
on its own transaction and assembles the row with one `string_agg`. The
process never holds the whole copy. The account's previous copy stays in
place until the new one is complete: a failure rolls back to it.

Measured on the seeded account in a 1 GB container, web and worker in one
process: the backup completes in about 30 s with the dense retention
running beside it, where v1.39.0 crashed within 30 s.

The streamed JSON also reports its progress to the job every 50 000 rows.
2026-09-25 00:32:05 +02:00
Marc-André Bombeck 1b45ebb99d fix(jobs): stop the long drains on a budget instead of at the expiry
pg-boss marks a job "job timed out" at its expiry and fires its abort
signal, but does not stop the handler. A drain that could not finish kept
writing after its job was declared dead, and the retry then ran beside it
over the same days.

The nightly drain tick and the per-user dense-retention and daily-mean
jobs now stop between days once 75 % of the job's expiry has passed, or
as soon as pg-boss aborts the job, and complete with `stopped_early`
instead of failing. What they folded is committed. A nightly tick that
stopped early after folding something sends itself again a minute later,
up to 24 times, so a first-time backlog after a large import drains the
same night; a run that folded nothing does not hand on, so a pass that
cannot move does not loop.

The expiries themselves stay at their defaults: with the passes bounded
per day and budgeted, a longer limit would only delay the signal.
2026-09-25 00:31:48 +02:00
Marc-André Bombeck f4b51cf134 perf(consolidation): fold a day at a time and batch the hourly writes
The consolidation drains (cumulative, daily mean, dense intra-day
retention, legacy steps) gathered a type's whole history into one array
before folding the first day. On an account of 1.25 million rows the
heart-rate array alone held about 230 MB, and the dense retention peaked
at 460 MB of heap against a 524 MB limit; in a 1 GB container running the
web server and the worker together the process died of heap exhaustion
within a minute and the job was later marked "job timed out" (#1031).

- Pages are now handed on one at a time and each local day is folded as
  soon as its last row has been read, so a pass holds one page and one
  day.
- The keyset cursor gains a redundant `measuredAt >= cursor` bound.
  Without it Postgres could not use the cursor as an index condition and
  re-read every earlier row on each page (47 ms and 248 000 filtered rows
  for one page halfway through the heart-rate history; 5 ms now).
- The dense fold resolves a day's hourly rows with one lookup and one
  insert instead of two lookups and a write per hour. On the seeded
  account the pass went from about 290 000 statements to 54 000.
- `shouldStop` lets a caller end a pass between days; the result says
  `stoppedEarly`. Days commit one by one and a folded day leaves the
  scan, so the next run continues at the first day not reached.
2026-09-25 00:31:40 +02:00
Marc-André Bombeck 32c3248b57 feat(jobs): log long-running, expired and cut-off jobs by queue
A job that ran into its pg-boss expiry left no trace in the app log. Most
handlers annotate a wide event nobody emits, pg-boss records an expiry only
in its own table, and a worker that dies of heap exhaustion writes one
FATAL line that names no queue. On a large account three queues failed
with "job timed out" for days and the log said nothing about them (#1031).

Every handler now runs inside `observeJob` (through `runJob`), which writes
`job.progress` every two minutes while a handler runs (elapsed time, heap,
RSS and the last progress the pass reported), `job.expired` when the job
passes its expiry while the handler is still running, and `job.finished`
when a long or expired run settles. At worker boot, `worker.boot.jobs_cut_off`
names every job a previous process was still running when it stopped.

The admin card's "job timed out" now says what the limit was and that the
cause is either a run too slow for it or a restart while it ran.
2026-09-25 00:31:30 +02:00
Marc-André Bombeck 47c7c71802 fix(medications): read a pinned dose the same as the dose on its own
Automatic matching counts a dose taken up to an hour before a slot's
on-time window as on time. A dose pinned to the slot was judged from the
window's start instead, so the same dose read on time unpinned and late
once pinned. A pinned dose now uses the same early grace, and the
confirmation says so plainly: the dose counts as if it had been logged for
the slot, on time or late depending on when it was taken.

Refs #1028
2026-09-24 19:54:37 +02:00
Marc-André Bombeck eafef5adeb fix(medications): keep a recorded miss from before the medication's creation
The history view treated every auto-miss anchored before the medication's
creation as the today projector's placeholder and dropped it. That is true
only on the creation day. History that arrives from before the creation, a
restore of an older file for one, can carry real misses, and those vanished
from the history and the rates.

The creation stays the boundary of expected doses: an earlier one, the
first recorded dose or the course start date, would read every day between
it and the creation that no row names as missed, and an import carries
takes and skips only, so those misses would be invented. Before it, every
recorded take, skip and miss claims its slot. Only rows that record nothing
are dropped: a pending row, and an auto-miss within the day before the
creation, which is where the projector's placeholders sit.

Refs #1028
2026-09-24 19:53:42 +02:00
Marc-André Bombeck f028d9c55b fix(medications): never stamp a miss on a slot before the medication existed
The today projector mints a pending placeholder for every slot of the
current day, so a medication added in the afternoon carries placeholders
for that day's earlier slots. The hourly auto-miss pass stamped them as
forgotten doses a day later, and every reader that counts raw rows saw a
miss on a day before the medication existed: the miss-free streak broke on
the creation day. The dose history, the rates and the Coach already ignore
such a placeholder; the streak read the row as it was.

The pass now stamps a miss only on slots from the medication's creation on.
A placeholder before it stays pending, so the web and the app can still
offer the slot for logging and a dose recorded for it converges onto it. A
migration returns the placeholders an earlier pass already stamped. It
touches only rows that can be nothing else: server-minted, never acted on,
anchored before the creation but within the day before it, which is where
the projector's placeholders sit. An older auto-miss carried in with
restored history keeps its state.

Refs #1028
2026-09-24 19:52:39 +02:00
Marc-André Bombeck 8642d63b82 fix(backup): keep a medication's creation date through a portable restore
A portable export left the medication's creation instant out, so a restore
stamped the restore time instead. The creation instant is where a
medication's expected doses start, so the whole restored history then read
as predating the medication: its missed doses dropped out of every
adherence rate. The portable file now carries the creation instant as the
disaster-recovery file already did, and the restore, which already honoured
it, brings it back. A round trip pins both file purposes.

Refs #1028
2026-09-24 19:48:01 +02:00
Marc-André Bombeck a6b8c61a82 fix(medications): count only a recorded dose as taken in the calendar
The adherence calendar counted every off-schedule ledger row as a dose taken
on time. An orphaned skip or auto-miss, a row on an instant that is no slot
of the schedule, carries no taken time, yet coloured its day as taken. Such
a row now counts as taken only with a taken time; an orphaned skip reads as
skipped, and an orphaned auto-miss counts nothing, as it already does in the
rate.

Refs #1028
2026-09-24 19:46:24 +02:00