Commit Graph
172 Commits
Author SHA1 Message Date
Marc-André Bombeck 32de7ecf37 Merge pull request #1135 from MBombeck/fix/cycle-backup-and-med-put
fix(backup): carry encrypted free text readable in portable backups
2026-10-04 12:23:35 +02:00
Marc-André Bombeck befac59576 feat(ai): send a per-entry reasoning effort to Local and gateway providers
Reasoning models on Ollama and similar servers think by default and can
spend the whole max_tokens budget before the visible answer starts, so
the call comes back empty or times out. The Local and OpenAI-compatible
chain entries now carry an optional reasoning setting: Default sends
nothing, Off sends reasoning_effort "none", Low / Medium / High send
their own value. It is stored on the entry in aiProviderChain, so a
chain can mix a thinking model with one that does not know the field.

- Every exported resolver binds the entry's value onto the client, the
  connection test included, keyed on the instance type so the operator
  provider, OpenAI, Anthropic and Codex never carry it.
- PATCH /api/user/ai-provider accepts localReasoningEffort and
  compatReasoningEffort and GET reports them. A provider absent from
  the chain holds its value on a disabled entry; restating Default on a
  chain that was never customised writes nothing.
- PUT /api/insights/provider-chain accepts reasoningEffort on local and
  openai-compatible entries, refuses it on any other, and keeps the
  stored value when the body leaves it out. Its schema is now shared
  with the OpenAPI registry.
- A Default request body is byte-identical to the one sent before.

Refs #1126
2026-10-04 10:42:59 +02:00
Marc-André Bombeck b36190b2df Restore custom mood tags without mood entries
The account's own mood categories, tags and hidden set were written only
inside the restore's mood-entry branch, so a vocabulary set up before the
first entry did not come back. They are written whether or not the file
carries an entry.
2026-10-04 10:19:32 +02:00
Marc-André Bombeck 8ac3fddc3b Carry visit and vaccination free text readable in a portable backup
A practitioner's note, a visit's reason, outcome and body site and a
vaccination's note travelled as stored ciphertext in a portable file, the
same defect the cycle and mood sections had. They now go through the
shared sealed-text helpers: readable in a portable file, sealed under the
receiving host's key on restore, ciphertext kept only where this host
opens it and otherwise reported as visitCiphertext or
vaccinationCiphertext. The practitioner address and phone ciphertext of a
disaster-recovery file gets the same check, since the key preflight now
leaves these sections to it.
2026-10-04 10:19:31 +02:00
Marc-André Bombeck 43a53255c9 Carry custom mood labels readable in a portable backup
The labels of the mood tags and mood categories an account created
travelled as the stored ciphertext in a portable file, so a restore onto
a host with another key refused the file or wrote labels nobody could
open. They now follow the cycle note: readable in a portable file, sealed
under the receiving host's key on restore, ciphertext kept only where
this host opens it and otherwise reported as moodLabelCiphertext.

The sealing helpers move to one module both sections share.
2026-10-04 08:31:11 +02:00
Marc-André Bombeck 8121935642 Carry cycle free text readable in a portable backup
A portable file carried the day-log note, the sensitive-category envelope
and custom symptom labels as the stored ciphertext, so a restore onto a
host with another key either refused the file or wrote values nobody
could open. The portable file now carries them readable and the restore
seals them under the receiving host's key; a disaster-recovery file keeps
the ciphertext.

A file written before this still restores: each ciphertext value is
opened with this host's keys first, and one that does not open is kept
out of the row and reported in the skip summary as cycleCiphertext. The
key preflight leaves the cycle sections to that per-value check.
2026-10-04 08:17:34 +02:00
Marc-André Bombeck eada82f61e Derive a backfilled course in the account's time zone
The backfill and the restore of an older file read the creation day of
an end-only window in UTC while every reader of the window uses the
account's zone; both now use the account's zone (the default zone when
the stored one is unknown to Postgres). A window whose start lies after
its end, which the API refuses but older rows hold, becomes a one-day
course on the end day: every reader treated it as empty and ended, and
a swap would have invented expected doses. One shared rule
(courseFromWindow) serves the restore and the course writer.
2026-10-03 23:36:08 +02:00
Marc-André Bombeck 1494409209 Merge remote-tracking branch 'origin/main' into wip/v140-H
# Conflicts:
#	messages/de.json
#	messages/en.json
#	messages/es.json
#	messages/fr.json
#	messages/it.json
#	messages/ko.json
#	messages/pl.json
#	src/__tests__/sharing-surface-guard.test.ts
#	src/__tests__/v137-wave-zero-fixtures.test.ts
#	tests/integration/sharing-manage-handler-matrix.test.ts
2026-10-03 22:52:49 +02:00
Marc-André Bombeck a14174fe4d Let a medication have several courses
A medication taken now and then (#1024) keeps one row and gains course
rows: calendar spans, both ends inclusive, an optional encrypted note, no
schedule or dose of their own. Courses never overlap and every course but
the latest has ended, so the latest course is always the one the
medication's startsOn/endsOn project. src/lib/medications/courses.ts is
the only writer of those two columns for a medication with courses; it
writes under a per-medication advisory lock and re-projects in the same
transaction, so reminders, the next-due line and intakeActionable keep
reading the columns and stay right.

- Migration 0368 adds medication_courses and backfills one course per
  medication window (an end-only window starts on the creation day, or
  on its end day when that lies earlier); a medication without a window
  stays course-less. Idempotent, data-preserving.
- GET/POST /api/medications/{id}/courses and PATCH/DELETE
  .../courses/{courseId} (owner-only); create seeds the first course
  from a window; the medication PUT edits the latest course and refuses
  a window that would break the rules (medications.course.*).
- List, detail, create and update carry courses, courseCount,
  previousCourseEndedOn and canStartCourse, resolved on the server.
- Compliance expands over the span of all courses and drops the slots
  between them; every compliance read selects the courses. A rolling
  cadence is not anchored by an intake from a previous course.
- The card names the last course's end and the running course's number;
  the detail Lifecycle group lists the courses with start, edit and
  delete. The Coach compliance block, the MCP resource, the doctor
  report window and table, the clinician view and FHIR (one
  MedicationStatement per course with effectivePeriod and status) read
  the courses.
- Backup carries courses per medication and the restore writes them and
  re-projects; an older file derives the one course its window
  describes. Wipe plan, encrypted columns, rotation and OpenAPI follow.
2026-10-03 22:39:25 +02:00
Marc-André Bombeck 4cb8118802 Let a person name their own medication categories
A medication can be filed under a category the person named, next to the
built-in ones (#1041). The labels live in medication_category_labels,
encrypted at rest, capped at 20 per account; a medication filed under one
carries its custom:<uuid> key in the category column.

- GET/POST /api/medications/categories (POST idempotent, 30/min per actor,
  admitted at MANAGE like the cycle symptom vocabulary) and owner-only
  PATCH/DELETE /api/medications/categories/{key}. DELETE moves every
  medication filed under the category to OTHER in the same transaction
  and answers movedCount.
- category is widened to a built-in value or custom:<uuid> on the wire,
  the create and update bodies refuse a key that is not the caller's
  (422 medications.category.unknown), and list, detail, the insights
  medication tiles and the MCP medications resource carry categoryLabel.
  A key with no owned label reads as OTHER and is counted on the event.
- The wizard's class step lists the person's categories under the
  built-in rows with an inline add row; the list page gains a category
  filter (session only) and a manage sheet for rename, hide and delete.
- Backup carries customMedicationCategories (decrypted labels) and the
  restore writes them before the medications; wipe plan, encrypted
  column registry, rotation script, sharing guards and OpenAPI follow.
2026-10-03 21:38:03 +02:00
Marc-André Bombeck d6571c4e9c feat(symptoms): add person-defined symptoms and timestamped occurrences
A symptom of one's own ("aura", "headache") is a definition with an
encrypted name and an icon from the cycle custom-symptom allowlist; each
occurrence carries a time, a 0-10 intensity (the pain score's scale), an
optional encrypted note and an optional link to one of the record's open
illness episodes. Both ride the illness module: every route answers
illness.disabled when it is off.

- Migration 0369 (idempotent): symptom_definitions and symptom_events,
  intensity CHECK 0-10, episode link ON DELETE SET NULL.
- /api/symptoms/definitions{,/[id]} and /api/symptoms/events{,/[id]}:
  eight active definitions per record (a hidden one frees its slot,
  showing it again re-checks), hide or purge, events create/list/edit/
  delete, a new link must name an open episode of the same record
  (422 symptoms.episode.notOpen). Error group symptoms.*.
- Sharing: reads at READ, logging an occurrence at WRITE, defining and
  editing at MANAGE; C1/C3/C4 audit details; frozen lists, the handler
  matrix and the activity verbs updated.
- Both encrypted columns registered and rotated; both models in the wipe
  plan; both carried at both backup ends with occurrences nested in their
  definition, plaintext in a portable file and ciphertext in a recovery
  file, and a link to an episode the file lacks dropped and named
  (symptomEpisodeReference).
- OpenAPI regenerated.
2026-10-03 21:13:45 +02:00
Marc-André Bombeck 350c646463 Adopt the medication category table into the schema
The clinical category lived in a side table the app created at runtime,
after migration 0004 had dropped it as orphaned. It is now the
MedicationCategoryAssignment model. Migration 0368 creates the table
IF NOT EXISTS with the runtime DDL, so an existing table converges with
its rows untouched (verified with prisma migrate diff against a database
holding the runtime table and against a fresh one).

The helpers keep their names and run on the Prisma delegate; the runtime
CREATE TABLE and the pre-transaction ensure call in the restore are gone.
The backup plan judges the table as a schema model (backed up, two-ended,
seeded in the round trip) instead of under RAW_SQL_TABLES, which is now
empty; its guard proves its matcher on a fixture string instead.
2026-10-03 20:45:02 +02:00
Marc-André Bombeck 816eabb3c3 Merge remote-tracking branch 'origin/main' into feat/v140-fixes
# Conflicts:
#	docs/api/openapi.yaml
2026-10-03 19:22:56 +02:00
Marc-André Bombeck 0acf3da575 feat(documents): read documents and lab scans in the background
"Read with AI" and the lab scan used to run their model calls inside the
request. At the default response time a read of three calls outlasted the
100 s a proxy in front of the server commonly allows, and the person saw a
cut connection for a read the server was still doing.

Both now run as a background run (`DocumentAiRun`, migration 0366):

- `POST /api/documents/inbound/{id}/index` with `Prefer: respond-async`
  answers 202 with a run id once every quick refusal has passed. Without
  the header it answers exactly as before, which the iPhone app relies on.
  A second request while a read of the same document is queued or running
  attaches to that run.
- `POST /api/labs/ocr/extract` always answers 202. Only the web client
  calls it.
- `GET /api/ai-runs/{id}` serves the run: the same body the synchronous
  route answers with, or the failure code and status it would have given.
  Scoped to the caller; another account's run is a 404.

The worker (`document-ai-run`, a long queue with a one-hour expiry, one
job per run under `lockedPass`) claims the run, asks the capability again
for the record and re-checks the wire for the provider it picks, sets the
run's deadline from the person's AI response time and refuses a read that
cannot fit the job's expiry. Sync route and worker run one body per kind,
so the two paths cannot drift. Budget is reserved at enqueue and settled
once by whoever ends the run.

Input and result are sealed with labelled AES-256-GCM, registered for key
rotation; the input is dropped when the run ends. A five-minute reaper
fails a run no worker took within fifteen minutes (and hands its
reservation back), fails a run past its deadline, and deletes a finished
run after an hour. Runs are outside backups and inside the data wipe.

The web client polls through the query-key factory and says, calmly, that
the read continues in the background, and when no worker has picked it up.
2026-10-03 18:18:56 +02:00
Marc-André Bombeck ab081d2a85 fix(workouts): keep a deleted workout's insight slot spent
The insight claim row is now the daily-cap ledger. Its workout_id is
nullable with ON DELETE SET NULL (migration 0367), so deleting a workout
no longer frees the day's slot. The cap counts claims that reached the
provider plus live leases, and drops the separate count of insight rows.
A daily workout-insight-claim-cleanup queue removes claim rows untouched
for seven days.
2026-10-03 17:53:36 +02:00
Marc-André Bombeck ff8277695a test(backup): round-trip every account setting through a real restore
Fill every User column on the source account, export through the streaming
writer, recreate the account with credentials of its own, restore through the
route, and compare: each setting column must come back with the value it left
with, and every identity, credential and bookkeeping column must still hold
the target's value. The section in the file is checked to carry nothing but
setting columns.
2026-10-01 13:12:25 +02:00
Marc-André Bombeck 7abaa25c14 feat(backup): export the account settings and restore them onto the account
Both backup purposes now carry an accountSettings section. The restore writes
it to the account being restored inside the same transaction, names every
refused value under the new accountSetting skip catalogue, and re-warms the
stored texts after the transaction when a unit preference changed, the way the
unit settings routes do. A file without the section leaves the account row as
it is, which is how every older backup restores. The instance settings
restore stays its own opt-in.
2026-10-01 13:04:41 +02:00
Marc-André Bombeck d731a1b8fe feat(backup): classify every User column and carry the account settings
Every column of the account row now has a backup verdict in
USER_COLUMN_BACKUP_CLASS: SETTING, CREDENTIAL, IDENTITY or OPERATIONAL. A
schema-derived guard fails on an unclassified column, and the setting columns
each get a codec in account-settings-backup.ts, so a column classified as a
setting without a way to travel does not compile.

The restore end checks each value the way the settings route that writes it
does: AI endpoints against this host's private-origin policy, languages, units
and time zones against the known vocabularies, threshold bands against their
bounds, the avatar against the upload's sniff and limits. A refused value is
left out and named; credentials, identity and the host's own cursors are never
admitted.
2026-10-01 13:04:36 +02:00
Marc-André Bombeck 383323b542 Merge remote-tracking branch 'origin/main' into fix/v1397-meta 2026-10-01 11:29:08 +02:00
Marc-André Bombeck d1a7406ff7 test: pin the nutrient source marker and arrival time in the payload builder test 2026-10-01 11:24:25 +02:00
Marc-André Bombeck 3f5bcbd441 fix(backup): carry four columns the restore was dropping
A column-level round trip over every two-ended model found four values
that left with the export and came back empty or re-stamped:

- MenstrualCycle.absorbedIntoId: the disaster-recovery file carried the
  folded, soft-deleted start but not the start that absorbed it, so after
  a restore removing the absorbing start could no longer bring the folded
  one back. Carried in the disaster-recovery arm and written back only
  when the file carries the start it names.
- Workout.externalSourceVersion: the source's version marker, stored so
  the sync can replay it unchanged, was in neither end. Carried beside
  metadata in the disaster-recovery arm.
- NutrientIntakeDay.externalSourceVersion and createdAt: neither was
  selected, so a restored day lost its source marker and its first-arrival
  time. Carried in both arms.

Files written before this change parse as before; every new field is
optional on the way in.
2026-10-01 11:17:44 +02:00
Marc-André Bombeck 7c5ff89def feat(lint): refuse UTC day keys outside the time zone module
`instant.toISOString().slice(0, 10)` and parsing a key back as UTC
midnight are the idioms behind every day-boundary fix this year: the
evening west of UTC and the small hours east of it land on the
neighbouring day. A new rule, healthlog/no-utc-day-key, refuses them in
src/ outside src/lib/tz and tests, and refuses a disable directive for
itself that is not next-line or gives no reason.

The time zone module now names every legitimate form: userDayKey for a
person's day, dateOnlyKey / dateOnlyAtNoonUtc / dayKeyAsUtcMidnight for
date-only values and day labels, daysBetweenDateKeys / weekdayOfDateKey /
isCalendarDateKey / statedDateKey for key arithmetic and parsing.

Of the 133 sites the rule found, 84 are gone. Most moved onto a named
helper with no change in behaviour; these changed behaviour:

- cycle calendar, insights, Coach snapshot and prediction refresh start
  their lookbacks at the first instant of the user's day
- a model-transcribed date like "June 10, 2026" no longer lands a day
  early in a process east of UTC (documents and lab scans)
- a custom report range covers exactly the picked days in the profile
  zone instead of starting the evening before west of UTC
- date-of-birth pickers cap at the user's today, not the UTC one
- an allergy onset date is stored at noon UTC like other date-only values
- the report's cycle summary, the Coach's usual BP range and the intake
  projector's rollup recompute use the user's day
- vaccination dates render as dates instead of the previous evening

49 remain, each with its reason on the line above: 39 UTC by design (15
file name stamps, rollup and object keys, the UTC-day score engines) and
10 marked baseline. A guard test requires a classified reason on every
directive and holds the baseline to a ceiling that only goes down.
2026-10-01 09:59:57 +02:00
Marc-André Bombeck a2fceb5585 fix(backups): answer the restore preview from counts kept with the copy
The restore dialog asked GET /api/admin/backups/{id}/summary for the
contents of a copy, and the route read the whole stored copy to answer:
every piece opened, every measurement parsed, the rest validated. On an
account of 1.8 million readings on slow hardware that took about two
minutes. The dialog stopped waiting after 15 s and said the contents
could not be read, and its automatic retry made the server read the
whole copy a second time.

The copy passes through the process once anyway while it is written, so
the preview is worked out there and kept on the row (DataBackup.preview,
migration 0364): the summarizeBackup counts, the schema version, and
the inner key ids with the sections they sit in and one sample value
each. Every creation path goes through storeBackupBlob (the weekly and
manual pass, and an upload), so every new copy carries one. The column
is plain JSON: counts and dates only, and the samples are ciphertext
that stands for what is inside the copy, which a key rotation must not
re-seal.

The route answers from it without opening the copy. Both verdicts are
taken again on every read, from what the preview keeps: whether this
release restores the schema version, and whether this server holds the
keys the copy needs. The pieces are still counted against the row, so
a copy that lost one is refused as the restore and the download refuse
it. A preview names the copy it describes and is ignored once the copy
changes; converting a single stored value into pieces carries it over.

A copy written before this has no preview. It is read whole once, as
before, and the result is kept on the row, so it is slow only the first
time; concurrent requests for the same copy share the one read.

Only a file that fails the schema answers 422 "Backup payload failed
schema validation". Read refusals keep their codes; anything else is
the server failing and answers 500.

The dialog no longer retries the preview, waits up to 55 s, and when it
stops waiting on an older copy it says the copy is still being read and
to open the dialog again in a few minutes.

Refs #1031
2026-09-30 22:51:02 +02:00
Marc-André Bombeck add6d1fadc fix(restore): order the restore and the consolidation passes by one lock
A restore on a freshly booted instance failed its clearing step with a
deadlock (40P01) while the boot-time step consolidation was folding the
same account. A consolidation day updates or mints the day's total and
then soft-deletes the day's per-sample rows; the restore deletes every
reading of the account in one statement. When the restore reached a
day's samples before its total, each side held what the other needed
next.

The restore already took an advisory lock per account before its first
delete. Every day transaction of the passes built on runConsolidation
(step and mean consolidation, the per-sample cumulative drain, the dense
intraday retention) now takes the same lock in shared mode as its first
statement. A restore that starts mid-day waits for that day to commit;
a day that would start while a restore runs is refused at once, and the
pass leaves that account for its next run instead of folding rows the
restore is about to replace. The passes still share the lock with each
other.

A new integration test builds the lock order that deadlocked, with a
row trigger holding the consolidation on the day's total while the
restore starts; before this change the consolidation was cancelled with
a deadlock. A guard test requires every transaction in a pass on
runConsolidation to open with the shared lock.

Refs #1031
2026-09-28 00:34:11 +02:00
Marc-André Bombeck f2569b6fdf fix(restore): carry at most a hundred readings per insert statement
Prisma's debug logger keeps the arguments of its last hundred calls in
memory whether or not logging is on, and the client and the Postgres
adapter both log every statement with its parameters. A restore that
wrote 1 000 readings per statement therefore kept a copy of about the
last 100 000 readings alive for the whole measurement write: about
60 MB on 1.8 million Apple Health readings, next to a V8 heap limit of
524 MB in the default 1 GB container.

The bulk insert now splits whatever it is handed into statements of at
most 100 rows, which holds about 7 MB. The Apple Health export import
goes through the same helper and gets the same bound. The write costs a
little time: 94 s instead of 76 s for 1.8 million readings on a fast
host.

Refs #1031
2026-09-28 00:06:47 +02:00
Marc-André Bombeck 206da07e8c Merge branch 'fix/meds-ended-and-mental-health' into release/v1.39.4 2026-09-27 14:18:09 +02:00
Marc-André Bombeck 6f37cbdb23 fix(backup): carry each medication's category through backup and restore
The clinical category a person picks for a medication lives in a side table
created at runtime, outside the schema. The backup builder and the restore
only walk schema models, so no backup ever carried it: after a restore every
medication read as "Other".

Both backup formats now carry the category inside its medication, and the
restore writes it back in the same transaction as the medication. A file
from before this change restores as before; a category this server does not
know reads as Other instead of refusing the file.

The backup plan gains a list for tables created with raw SQL, and a check
that every such table in the source either maps to a schema model or has a
verdict there, so the next one cannot drop out of backups unnoticed.
2026-09-27 14:17:42 +02:00
Marc-André Bombeck c7e53176c1 Merge branch 'fix/v1394-review-correctness' into release/v1.39.4 2026-09-27 14:00:04 +02:00
Marc-André Bombeck eda87fd4a7 fix(sync): expire delta cursors issued before a restore or a data wipe
A restore writes rows back with their original updatedAt, behind every
paired client's watermarks, and removes rows without leaving tombstones.
A client that kept its cursor would miss both on its next incremental
pull. The same holds after the person deletes all their data.

`users.sync_reset_at` (migration 0363) records when the record was last
replaced: the restore stamps it as its last write and again right after
the commit, the per-account and the admin wipe stamp it inside their
transactions. Cursors now carry the time they were issued, and
/api/sync/changes answers its existing `cursorExpired` to a cursor issued
before the reset, or one without an issue time on an account that has
one. The comparison is on the issue time, because after a restore a
freshly drained cursor still carries old watermarks.
2026-09-27 13:46:38 +02:00
Marc-André Bombeck daf386fe7f fix(backup): use the renamed after-transaction allowance for the chart rebuild 2026-09-27 13:32:58 +02:00
Marc-André Bombeck 5173049130 Merge branch 'release/v1.39.4' into fix/v1394-review-correctness 2026-09-27 13:27:59 +02:00
Marc-André Bombeck fc13b9de5c fix(restore): skip expired tombstones and give the chart rebuild its own limit
A restore now leaves out measurement tombstones deleted longer ago than
the tombstone retention. The nightly purge removes exactly those rows,
the delta feed answers cursorExpired for every cursor that could still
be served one, and the consolidation repairs look only inside the same
horizon. Younger tombstones keep their readers (a lagging paired client,
the undo of a recent delete, the repairs, the duplicate guard on
re-uploaded samples) and are written as before. The backup file is
unchanged. On a seeded account of 1.89 million readings whose 1.68
million tombstones were past retention, the restore on a slowed database
took 5.4 min instead of 15.

The rollup fold after the commit rebuilt five years of buckets under the
60 s request statement limit and could be dropped on a slow disk, leaving
the charts empty until the next boot. `recomputeUserRollups` takes a
`statementTimeoutMs` that scopes each aggregate and persist transaction
with SET LOCAL, and the restore passes its after-commit allowance.

Fixture tombstones in two round-trip tests were dated in July 2026 and
had aged past the retention; they are now relative to the run.
2026-09-27 13:21:18 +02:00
Marc-André Bombeck 874d02294e fix(backup): let the restore job's budget admit large accounts
A restore is refused before it deletes anything when its transaction limit
would run past three quarters of the job's expiry. At two hours that refused
a same-size replacement of about four million readings. The expiry is now six
hours, which admits ten million over ten million; a dead worker still frees
the account within minutes through the heartbeat, so the longer expiry does
not hold an account longer.

The refusal now names both counts, the readings in the backup and the ones the
account holds now (removed readings included until the nightly clean-up),
since clearing the latter costs time too.
2026-09-27 13:13:50 +02:00
Marc-André Bombeck 3482581ce1 fix(encounters): name the practice address on an appointment nudge again
Sealing the practitioner's address left the appointment arm reading the old
readable column, which is now empty, so a booked visit's push lost the address
on booking, rescheduling and restoring a deleted visit.

The reminder no longer keeps its own copy of the address. The reminder tick
resolves it from the owning visit's practitioner when it builds the push and
decrypts it there, so an edit of the address also reaches a nudge that is
already booked. MeasurementReminder.location stays NULL on appointment rows:
booking writes none, a re-anchor clears one an earlier release wrote, and a
backup neither carries nor restores one. A checkup's own free-text location is
unchanged.
2026-09-27 13:07:10 +02:00
Marc-André Bombeck 02801bdfda Merge branch 'fix/restore-large-account-clearing' into release/v1.39.4 2026-09-27 12:14:04 +02:00
Marc-André Bombeck e0b4f3c427 fix(restore): size the restore budget by the rows written, and keep test housekeeping off the short limit 2026-09-27 11:57:37 +02:00
Marc-André Bombeck d44b875ec6 test(restore): stub the reading count the restore budget now reads 2026-09-27 11:37:51 +02:00
Marc-André Bombeck 93b6c086b4 fix(restore): let a large restore outlast the request statement timeout
A restore of an account with 1.9 million readings, most of them deleted
by the nightly consolidation and still awaiting the purge, was cancelled
in its clearing step: `DELETE FROM measurements` ran past the 60 s
statement timeout every pooled connection carries, and the whole restore
rolled back with a message that named no cause (#1031).

Most of that statement was not the delete. Two columns point into
`measurements` with ON DELETE SET NULL and had no index, so Postgres
scanned `personal_records` and `ecg_recordings` once per deleted reading:
38 of 40 s on a seeded account with 300 personal records. Migration 0362
indexes both, which also speeds up the tombstone purge and account
deletion.

The restore transaction now sets its own statement and idle-in-transaction
limits to the transaction budget with SET LOCAL, and the budget counts
the rows the account holds as well as the rows the file carries. A
rolled-back restore names what the database refused it over (timeout,
lock, constraint, storage, connection, other) on the job row, the
console, the audit row and the wide event.
2026-09-27 11:24:41 +02:00
Marc-André Bombeck 78fc22d0f8 Merge branch 'coach/wpF-contracts' into release/v1.39.4 2026-09-27 11:10:37 +02:00
Marc-André Bombeck f2e32afb24 fix(privacy): store environment coordinates at 1 decimal instead of 2
The home location of the environment module was rounded to 2 decimals,
about 1 km, which places a home in a neighbourhood rather than the town the
schema comment promised. The weather archive it feeds is gridded at 9 to
25 km, so the extra precision bought nothing.

Every stored coordinate of the module now goes through the same 1-decimal
rounding (about 11 km north-south, 7 km east-west at 50° N): the home, the
travel periods, the geocoder's results, the per-day weather rows, and a
restore of a file written with finer values. Migration 0361 rounds the
existing users, travel periods and per-day rows; running it twice changes
nothing more. The answers keep their shape, plain numbers. The API
descriptions and the schema comment say 1 decimal now.
2026-09-27 10:37:37 +02:00
Marc-André Bombeck a8b3e5d7f4 feat(coach): store a turn's result tables encrypted beside the message
A new nullable column, coach_messages.results_encrypted, holds the tables
of values a turn read as AES-GCM ciphertext; only their metadata rides the
plaintext provenance. readMessageResults serves them to the owner, each
table whole or withheld with a reason. The provenance restore learns the
steps, method, table metadata, chips, clarification and the forced-answer
marker, entry by entry against the wire schema.

The column is registered for key rotation, carried through both backup
formats (ciphertext in a disaster-recovery file, readable JSON in a
portable one) and goes with its row on conversation delete and wipe.
2026-09-27 10:36:27 +02:00
Marc-André Bombeck 5eb7956bad fix(privacy): encrypt workout GPS tracks and practitioner contact details at rest
The GPS track of an outdoor workout was stored readable, and its first and
last points are usually the person's front door. The phone number and address
in the address book of doctors and practices were readable too, while the
note beside them was already encrypted.

Migration 0361 adds workout_routes.geometry_encrypted (the geometry loses
its NOT NULL) and practitioners.phone_encrypted / location_encrypted, plus
partial indexes on the backfill's discovery predicates. A track is sealed as
one binary AES-256-GCM value bound to its purpose; the contact fields use the
same Bytes codec as the note. The practitioner's name, practice and specialty
stay readable, because the picker searches and sorts on them.

Every writer stores ciphertext only: workout batch ingest, practitioner
create and edit, and both restore formats. Readers prefer the ciphertext and
fall back to the old column for rows not yet migrated: the workout detail
(map, splits, GPX), the activity insight's climb figure, the practitioner
DTO behind the address book, visits and vaccinations. The wire shape does not
change. An edit's audit row now names a changed phone number or address
without recording either value.

free-text-encryption-backfill covers the three columns in the same pass,
finding a route through its workout. All three are registered for key
rotation (registry, CLI script, and the in-app rotation). A disaster-recovery
backup carries the contact fields as ciphertext only; a portable export
carries them readable. Workout routes are still not part of a backup, as
before. The old columns are marked for removal in a later release.
2026-09-27 09:59:34 +02:00
Marc-André Bombeck 66b1aef40c Merge branch 'fix/encrypt-free-text-and-local-ai' into release/v1.39.3
# Conflicts:
#	src/app/api/import/csv/route.ts
2026-09-26 21:50:05 +02:00
Marc-André Bombeck 4409529c89 Merge branch 'fix/backup-rotation-and-deletion' into release/v1.39.3
# Conflicts:
#	docs/api/openapi.yaml
#	src/lib/jobs/apple-health-import-worker.ts
2026-09-26 21:48:20 +02:00
Marc-André Bombeck 76398c80c2 test(backup): drop an import the as-stored rewrite left unused 2026-09-26 20:59:09 +02:00
Marc-André Bombeck 94aa7c6bd4 fix(backup): read nothing shorter than a ciphertext as needing a key
The shortest value either codec writes is 40 base64 characters. A
shorter string under an ...Encrypted member is not ciphertext, and
reading it as a legacy value would have made a restore demand the v1
key for a field that holds none.
2026-09-26 20:41:52 +02:00
Marc-André Bombeck 7da778760e fix(backup): leave the admin wipe's off-host copies in place, and correct the backup and rotation runbooks
The admin wipe of every account is the operator clearing their own host,
and the bucket's copies are the documented way back from a wipe pressed
by mistake, so it writes no off-host deletion request. Deleting an
account, deleting a managed profile and "Delete all data" still do.

The backup runbook no longer says the worker never deletes an object: it
lists the grant the worker needs (DeleteObject included), when objects
are deleted, the version-4 envelope, the lifecycle check and how to
rotate the off-host key. The rotation runbook no longer says backups are
covered: rotation reaches a backup's envelope, not the content a restore
writes back, and a retired key stays needed until no backup lists it.
The privacy notice beside account deletion says the same about off-host
copies.

A failed import records its failure in a plain try block, and the
extraction test that inflates past the 64 MiB floor gets the time it
needs on a loaded runner.
2026-09-26 20:33:30 +02:00
Marc-André Bombeck 849748c4cd fix(backup): bind new off-host objects to their key and account, and cap what an uploaded backup may inflate to
Off-host objects are now written as envelope version 4: the header names
the off-host key by fingerprint, and the object's key in the bucket
(<date>/user-<id>.json.enc) is authenticated as GCM associated data, so
a copy moved to another account or another night no longer opens.
Objects of versions 1 to 3 still read. BACKUP_ENCRYPTION_PREVIOUS_KEYS
(comma-separated, on the compose whitelist and in the env manifest)
keeps retired off-host keys readable, which is the rotation path the
off-host key did not have.

The admin backup upload counted the bytes that arrived and none after
the gunzip, so a small gzip could expand without limit into the JSON
scanner and take the app down. The inflated bytes are now held to 100
times the compressed bytes read (at least 256 MB, at most 4 GB) and a
file past that is refused with 413 before anything is stored.
2026-09-26 20:18:55 +02:00
Marc-André Bombeck 46eb8c8b94 fix(backup): carry a not-yet-migrated title or note as stored
Sealing a readable Coach title or custom-metric note while building a
disaster-recovery file gave each export a fresh IV, so the streaming
writer and the materialising builder no longer produced the same file.
A row the free-text backfill has not reached yet now rides as stored,
readable, inside the encrypted backup; the restore seals it, and a
migrated row carries its ciphertext verbatim as before.
2026-09-26 19:58:49 +02:00
Marc-André Bombeck 1c7785c23b fix(backup): take off-host copies out with the account, and say which keys backups still need
Deleting an account, deleting a managed profile, "Delete all data" and the
admin wipe now write an off-host deletion request in the same transaction
as the deletion. A new job (offhost-backup-purge, kicked straight away and
run nightly as a backstop) lists the bucket once, deletes every
<date>/user-<id>.json.enc object of each pending account and removes the
request only when nothing of the account is left. A refused delete keeps
the request with the bucket's answer and is retried; the off-host card
shows how many deletions are outstanding and why. A copy the nightly run
was writing while the account was deleted is taken back after the upload.

The off-host card also reads the bucket's lifecycle rule and says plainly
when there is none (every nightly copy kept forever) or when the
credential may not read it.

Every stored backup now records the key ids its content was written
under, and every off-host upload records the same per key. The admin
encryption view lists, per key, the stored copies that need it with the
oldest date, and when the last off-host copy needing it expires. The
"safe to drop" badge now requires that no backup still needs a retired
key, instead of only that every row was rotated.

Migration 0358 adds data_backups.inner_key_ids, the key-use table and the
deletion requests.
2026-09-26 19:54:40 +02:00