330 Commits
Author SHA1 Message Date
Marc-André Bombeck 7d42cf7122 fix(coach): keep updatedAt through storage-only rewrites
The title encryption backfill moved each Coach conversation's title into
ciphertext with a plain Prisma update, which stamps updatedAt. The Coach
panel orders and groups conversations by that column, so threads untouched
for months showed up under Today. The memory summary refresh did the same
in the background.

Storage-only rewrites now carry the row's own updatedAt and match on it, so
a turn that lands meanwhile wins and the row is retried on the next pass:
conversation titles, practitioner contacts, measurement and mood notes
(where updatedAt is the sync cursor), inventory notes, and the mood-note
script. The summary refresh writes through raw SQL. A guard freezes both
rules: every backfill write on a model with @updatedAt names the column or
is listed with the reason the stamp is wanted, and every conversation
write outside the person's own actions keeps it.
2026-10-05 12:40:55 +02:00
Marc-André Bombeck adb0e7c3b1 fix(demo): seal the Coach titles the way the app writes them
A readable title is moved to title_encrypted by the boot backfill, and
that update also sets updated_at to the moment it ran, so every seeded
thread sorted under "Today" a minute after the restart. The seed now
writes the ciphertext directly.
2026-10-05 11:39:15 +02:00
Marc-André Bombeck 25d486f956 fix(demo): end glucose at yesterday and steady the sleep average
The vitals check holds a day's glucose mean against the band of full
days, so a part day seeded up to the current hour read "below your
range". Spot readings and the CGM stream now end with yesterday.

The aggregate sleep walk drifted to almost nine hours, against which the
~7.5 h per-stage nights read as an hour short. A tighter walk keeps the
trailing average near the nights it is compared with.
2026-10-05 11:28:40 +02:00
Marc-André Bombeck 2b5d22e93d fix(demo): seed the surfaces the start page and Coach now show
The demo seed predates the Today overview, the grouped Coach history and
the procedure filter on documents, and it carried a few defects that a
fresh seed made visible:

- Store every Date as UTC. node-postgres serialises a Date in the host's
  zone and the cast to timestamp drops the offset, so a seed run from a
  machine on Berlin time wrote every instant two hours ahead.
- Keep today's later glucose readings, CGM samples and the evening mood
  entry out of the future.
- Close the seasonal allergy and its flare in the spring. Any open
  episode puts the account in Rest Mode, and the open one read
  "Rest mode, day 401" on the start page.
- Pin today's vitals to the median of the prior four weeks, so the
  Today overview does not report a random tail value as out of range.
- Confirm the key-backup step for the key the demo runs with, which
  removes the backup banner every visitor saw above the dashboard.
- Add two practitioners, a knee arthroscopy with its operative report
  and discharge letter linked (small generated PDFs), its follow-up and
  a check-up booked for tomorrow.
- Replace the single Coach exchange with five conversations from today
  back over a few weeks, the newest with a full answer and its steps.
- Drop the layout field `heroVisible`, which the layout type no longer
  has.
2026-10-05 11:23:28 +02:00
Marc-André Bombeck d07f5796d7 Seed the demo Coach answer with valid metric tokens, steps and token meta 2026-10-04 20:02:49 +02:00
Marc-André Bombeck db9618ee7b Merge remote-tracking branch 'origin/main' into wip/v140-H 2026-10-04 00:43:10 +02:00
Marc-André Bombeck 69bae8bdb4 Add ENCRYPTION_KEY_CHECK=warn as an escape hatch for the boot key check
At a first start there is no recorded key yet, so an operator facing a
refusal they are certain is false has no row to delete. With
ENCRYPTION_KEY_CHECK=warn (default enforce) the check still runs and logs
its full outcome, /api/health names a mismatch as a warning field without
failing, requests are served, and no key is recorded unless every key id
passes.

The variable is on the compose whitelist, in both example files, the env
manifest, the Unraid and Portainer templates and the published-env guard.
The log block, the mismatch page and the self-hosting pages offer it as the
last resort after the three real fixes, and say that it disables a safety
check.
2026-10-03 23:46:40 +02:00
Marc-André Bombeck 2c295bbea8 Probe the oldest stored values before recording an encryption key
The boot key check recorded whatever key was configured whenever its probe
found nothing to try. It only read string columns, so an install whose
ciphertext sat in Bytes columns (notes, documents, ECG, symptoms) read as
empty; a probe that ran out of time did the same; and rows written by a
process that served under a wrong key could open on the next boot and get
that key recorded as the right one.

The probe now reads every registered column, string and Bytes, in all four
stored layouts, and tries the oldest values first, ordered by the row's
updatedAt (else createdAt, else primary key). A key is recorded only when
the oldest value that opens is preceded by at most one that does not, or
when no column holds a single value under the key id. A time-out, an
unreadable table, oversized values or mixed results are inconclusive: the
process serves, records nothing and checks again at the next start.

When the first insert loses a race, the stored record is read back and must
open under this process's key, else the outcome is a mismatch.

A rotation that walked every column without errors removes the record of
each previous key id that no longer holds any value. The refusal log block
and the self-hosting pages explain how to reset the check for a key id.
2026-10-03 23:29:55 +02:00
Marc-André Bombeck 1494409209 Merge remote-tracking branch 'origin/main' into wip/v140-H
# Conflicts:
#	messages/de.json
#	messages/en.json
#	messages/es.json
#	messages/fr.json
#	messages/it.json
#	messages/ko.json
#	messages/pl.json
#	src/__tests__/sharing-surface-guard.test.ts
#	src/__tests__/v137-wave-zero-fixtures.test.ts
#	tests/integration/sharing-manage-handler-matrix.test.ts
2026-10-03 22:52:49 +02:00
Marc-André Bombeck a14174fe4d Let a medication have several courses
A medication taken now and then (#1024) keeps one row and gains course
rows: calendar spans, both ends inclusive, an optional encrypted note, no
schedule or dose of their own. Courses never overlap and every course but
the latest has ended, so the latest course is always the one the
medication's startsOn/endsOn project. src/lib/medications/courses.ts is
the only writer of those two columns for a medication with courses; it
writes under a per-medication advisory lock and re-projects in the same
transaction, so reminders, the next-due line and intakeActionable keep
reading the columns and stay right.

- Migration 0368 adds medication_courses and backfills one course per
  medication window (an end-only window starts on the creation day, or
  on its end day when that lies earlier); a medication without a window
  stays course-less. Idempotent, data-preserving.
- GET/POST /api/medications/{id}/courses and PATCH/DELETE
  .../courses/{courseId} (owner-only); create seeds the first course
  from a window; the medication PUT edits the latest course and refuses
  a window that would break the rules (medications.course.*).
- List, detail, create and update carry courses, courseCount,
  previousCourseEndedOn and canStartCourse, resolved on the server.
- Compliance expands over the span of all courses and drops the slots
  between them; every compliance read selects the courses. A rolling
  cadence is not anchored by an intake from a previous course.
- The card names the last course's end and the running course's number;
  the detail Lifecycle group lists the courses with start, edit and
  delete. The Coach compliance block, the MCP resource, the doctor
  report window and table, the clinician view and FHIR (one
  MedicationStatement per course with effectivePeriod and status) read
  the courses.
- Backup carries courses per medication and the restore writes them and
  re-projects; an older file derives the one course its window
  describes. Wipe plan, encrypted columns, rotation and OpenAPI follow.
2026-10-03 22:39:25 +02:00
Marc-André Bombeck ef0028c020 Merge remote-tracking branch 'origin/main' into wip/v140-I
# Conflicts:
#	docs/api/openapi.yaml
2026-10-03 22:16:29 +02:00
Marc-André Bombeck 1564e7c396 Add the sick note document kind
A certificate of incapacity for work (in Germany the
Arbeitsunfähigkeitsbescheinigung) is now its own document kind,
SICK_NOTE, instead of Other. The enum value rides migration 0370.

Uploads, edits, the kind filter, bulk edits, the import script and the
backup format accept it; the vault files it with a thermometer glyph
between prescriptions and referrals; the filing suggestion can propose
it and names the words a sick note carries. The extraction's own kind
field keeps its three values.
2026-10-03 21:40:01 +02:00
Marc-André Bombeck 4cb8118802 Let a person name their own medication categories
A medication can be filed under a category the person named, next to the
built-in ones (#1041). The labels live in medication_category_labels,
encrypted at rest, capped at 20 per account; a medication filed under one
carries its custom:<uuid> key in the category column.

- GET/POST /api/medications/categories (POST idempotent, 30/min per actor,
  admitted at MANAGE like the cycle symptom vocabulary) and owner-only
  PATCH/DELETE /api/medications/categories/{key}. DELETE moves every
  medication filed under the category to OTHER in the same transaction
  and answers movedCount.
- category is widened to a built-in value or custom:<uuid> on the wire,
  the create and update bodies refuse a key that is not the caller's
  (422 medications.category.unknown), and list, detail, the insights
  medication tiles and the MCP medications resource carry categoryLabel.
  A key with no owned label reads as OTHER and is counted on the event.
- The wizard's class step lists the person's categories under the
  built-in rows with an inline add row; the list page gains a category
  filter (session only) and a manage sheet for rename, hide and delete.
- Backup carries customMedicationCategories (decrypted labels) and the
  restore writes them before the medications; wipe plan, encrypted
  column registry, rotation script, sharing guards and OpenAPI follow.
2026-10-03 21:38:03 +02:00
Marc-André Bombeck d6571c4e9c feat(symptoms): add person-defined symptoms and timestamped occurrences
A symptom of one's own ("aura", "headache") is a definition with an
encrypted name and an icon from the cycle custom-symptom allowlist; each
occurrence carries a time, a 0-10 intensity (the pain score's scale), an
optional encrypted note and an optional link to one of the record's open
illness episodes. Both ride the illness module: every route answers
illness.disabled when it is off.

- Migration 0369 (idempotent): symptom_definitions and symptom_events,
  intensity CHECK 0-10, episode link ON DELETE SET NULL.
- /api/symptoms/definitions{,/[id]} and /api/symptoms/events{,/[id]}:
  eight active definitions per record (a hidden one frees its slot,
  showing it again re-checks), hide or purge, events create/list/edit/
  delete, a new link must name an open episode of the same record
  (422 symptoms.episode.notOpen). Error group symptoms.*.
- Sharing: reads at READ, logging an occurrence at WRITE, defining and
  editing at MANAGE; C1/C3/C4 audit details; frozen lists, the handler
  matrix and the activity verbs updated.
- Both encrypted columns registered and rotated; both models in the wipe
  plan; both carried at both backup ends with occurrences nested in their
  definition, plaintext in a portable file and ciphertext in a recovery
  file, and a link to an episode the file lacks dropped and named
  (symptomEpisodeReference).
- OpenAPI regenerated.
2026-10-03 21:13:45 +02:00
Marc-André Bombeck 37793298a8 Merge remote-tracking branch 'origin/main' into feat/key-backup-and-transport
# Conflicts:
#	docs/api/openapi.yaml
2026-10-03 19:22:01 +02:00
Marc-André Bombeck 0acf3da575 feat(documents): read documents and lab scans in the background
"Read with AI" and the lab scan used to run their model calls inside the
request. At the default response time a read of three calls outlasted the
100 s a proxy in front of the server commonly allows, and the person saw a
cut connection for a read the server was still doing.

Both now run as a background run (`DocumentAiRun`, migration 0366):

- `POST /api/documents/inbound/{id}/index` with `Prefer: respond-async`
  answers 202 with a run id once every quick refusal has passed. Without
  the header it answers exactly as before, which the iPhone app relies on.
  A second request while a read of the same document is queued or running
  attaches to that run.
- `POST /api/labs/ocr/extract` always answers 202. Only the web client
  calls it.
- `GET /api/ai-runs/{id}` serves the run: the same body the synchronous
  route answers with, or the failure code and status it would have given.
  Scoped to the caller; another account's run is a 404.

The worker (`document-ai-run`, a long queue with a one-hour expiry, one
job per run under `lockedPass`) claims the run, asks the capability again
for the record and re-checks the wire for the provider it picks, sets the
run's deadline from the person's AI response time and refuses a read that
cannot fit the job's expiry. Sync route and worker run one body per kind,
so the two paths cannot drift. Budget is reserved at enqueue and settled
once by whoever ends the run.

Input and result are sealed with labelled AES-256-GCM, registered for key
rotation; the input is dropped when the run ends. A five-minute reaper
fails a run no worker took within fifteen minutes (and hands its
reservation back), fails a run past its deadline, and deletes a finished
run after an hour. Runs are outside backups and inside the data wipe.

The web client polls through the query-key factory and says, calmly, that
the read continues in the background, and when no worker has picked it up.
2026-10-03 18:18:56 +02:00
Marc-André Bombeck 308470f229 feat(encryption): back up the key, and refuse to serve when it cannot open the database
Every configured key id now has a canary in encryption_key_canaries,
checked at boot. A key that cannot open its canary, or existing data
written before the canary existed, puts the process in a refusal state:
every API route answers 503 encryption.key_mismatch, /api/health reports
reason encryption_key_mismatch, the pages render one explanation, and
neither the queue producer nor the worker starts. The process stays up
so the reason is visible instead of hiding in a crash loop.

The admin Encryption page gains a "Back up your encryption key" step:
key id and fingerprint (never the key), where the key lives per
platform, a check of a pasted copy that is compared and discarded, and
a confirmation bound to the active key id and fingerprint. The overview
gets a tile and admins get a dashboard banner while it is due.

HEALTHLOG_PLATFORM picks the platform the step opens on. Adds the
TrueNAS self-hosting page and key backup sections in the docs.
2026-10-03 18:06:28 +02:00
Marc-André Bombeck c2ab9cda11 test(coach): put the dialog scenarios to a real model nightly
The live evaluation now captures the tool calls a generation makes, with
their arguments, and hands them to the judge. runScenarioLive puts a
scenario to the model in the chat turn's own context (system prompt,
tool-mode and dialog addenda, DATA INVENTORY, EARLIER TABLES, transcript)
over the real tool catalogue, answers each call with a stand-in result,
and reads the reply through the turn's clarification and chip parsers.
runScenarioJudge grades each scenario a few times with the same
evaluateScenario the deterministic suite uses and reports a pass rate per
scenario; the server-only scenarios and checks are left out.

Like the existing judge it runs only with COACH_EVAL_API_KEY, from the
nightly workflow's non-blocking step, and never throws.
2026-09-27 13:07:25 +02:00
Marc-André Bombeck 78fc22d0f8 Merge branch 'coach/wpF-contracts' into release/v1.39.4 2026-09-27 11:10:37 +02:00
Marc-André Bombeck a8b3e5d7f4 feat(coach): store a turn's result tables encrypted beside the message
A new nullable column, coach_messages.results_encrypted, holds the tables
of values a turn read as AES-GCM ciphertext; only their metadata rides the
plaintext provenance. readMessageResults serves them to the owner, each
table whole or withheld with a reason. The provenance restore learns the
steps, method, table metadata, chips, clarification and the forced-answer
marker, entry by entry against the wire schema.

The column is registered for key rotation, carried through both backup
formats (ciphertext in a disaster-recovery file, readable JSON in a
portable one) and goes with its row on conversation delete and wipe.
2026-09-27 10:36:27 +02:00
Marc-André Bombeck b8e78a78b3 chore(openapi): regenerate for the practitioner backup ciphertext fields
The practitioner entry of the backup payload gains the optional
locationEncrypted and phoneEncrypted fields. The rotation script names each
new column once, at its call, so a dropped call is noticed rather than
covered by a comment.
2026-09-27 10:21:48 +02:00
Marc-André Bombeck 5eb7956bad fix(privacy): encrypt workout GPS tracks and practitioner contact details at rest
The GPS track of an outdoor workout was stored readable, and its first and
last points are usually the person's front door. The phone number and address
in the address book of doctors and practices were readable too, while the
note beside them was already encrypted.

Migration 0361 adds workout_routes.geometry_encrypted (the geometry loses
its NOT NULL) and practitioners.phone_encrypted / location_encrypted, plus
partial indexes on the backfill's discovery predicates. A track is sealed as
one binary AES-256-GCM value bound to its purpose; the contact fields use the
same Bytes codec as the note. The practitioner's name, practice and specialty
stay readable, because the picker searches and sorts on them.

Every writer stores ciphertext only: workout batch ingest, practitioner
create and edit, and both restore formats. Readers prefer the ciphertext and
fall back to the old column for rows not yet migrated: the workout detail
(map, splits, GPX), the activity insight's climb figure, the practitioner
DTO behind the address book, visits and vaccinations. The wire shape does not
change. An edit's audit row now names a changed phone number or address
without recording either value.

free-text-encryption-backfill covers the three columns in the same pass,
finding a route through its workout. All three are registered for key
rotation (registry, CLI script, and the in-app rotation). A disaster-recovery
backup carries the contact fields as ciphertext only; a portable export
carries them readable. Workout routes are still not part of a backup, as
before. The old columns are marked for removal in a later release.
2026-09-27 09:59:34 +02:00
Marc-André Bombeck ed05772a8e test(e2e): run the delegated-writes journey and fail on silent skips
The delegated-writes spec checked for the invitation form's level control
right after page load, before the form had painted, found nothing and
skipped every test on every run, CI included.

- Give the journey its own account pair, seeded and reset in global setup,
  so it no longer invites the same pair as the account-sharing journey
  (a second live grant between two accounts is refused with 409).
- Wait for the form through its data-slot and fail when the WRITE level
  control is missing instead of skipping.
- Revoke the journey's grant in afterAll.
- Keep it out of the mobile project in the config rather than through a
  runtime skip.

Add two checks so a skip cannot hide a spec again:

- e2e-conditional-skip-guard.test.ts requires every conditional
  test.skip/test.fixme in e2e/ to carry a reason and refuses a bare
  test.skip().
- scripts/check-e2e-skipped-specs.mjs reads the Playwright JSON reports of
  both shards in the e2e gate job and fails when a spec file skipped all of
  its tests across every project, apart from an allowlist with reasons.
2026-09-27 04:48:08 +02:00
Marc-André Bombeck 66b1aef40c Merge branch 'fix/encrypt-free-text-and-local-ai' into release/v1.39.3
# Conflicts:
#	src/app/api/import/csv/route.ts
2026-09-26 21:50:05 +02:00
Marc-André Bombeck 4409529c89 Merge branch 'fix/backup-rotation-and-deletion' into release/v1.39.3
# Conflicts:
#	docs/api/openapi.yaml
#	src/lib/jobs/apple-health-import-worker.ts
2026-09-26 21:48:20 +02:00
Marc-André Bombeck c86f23c2c0 fix(security): keep ALLOW_LOCAL_AI_PRIVATE_HOSTS=true for the operator's own AI, deprecated
Making `=true` grant nothing would stop a single-user self-host's local
model on a patch update. It is deprecated for one release cycle instead,
in a narrowed and safe form: it still grants a private host, but only to
the operator's own configurations (the instance-wide admin provider and
AI settings saved on an admin account), and only through the pinned
operator-approved dispatcher, so redirects stay forbidden and metadata,
link-local and unspecified addresses stay refused. A non-admin account's
own base URL needs an exact origin in AI_PRIVATE_ORIGINS.

Provider resolution passes the owning account's role to the client
(`operatorTrusted`); the save route and the connection test apply the
same rule to the account being configured. The boot summary now says the
setting is deprecated and names the replacement, and the admin AI
settings page shows the exact AI_PRIVATE_ORIGINS line for the private
origins saved on the instance (every account's Local and gateway URL and
the admin base URL, scheme, host and port only), so the operator can copy
it and drop the old setting.
2026-09-26 20:59:10 +02:00
Marc-André Bombeck 849748c4cd fix(backup): bind new off-host objects to their key and account, and cap what an uploaded backup may inflate to
Off-host objects are now written as envelope version 4: the header names
the off-host key by fingerprint, and the object's key in the bucket
(<date>/user-<id>.json.enc) is authenticated as GCM associated data, so
a copy moved to another account or another night no longer opens.
Objects of versions 1 to 3 still read. BACKUP_ENCRYPTION_PREVIOUS_KEYS
(comma-separated, on the compose whitelist and in the env manifest)
keeps retired off-host keys readable, which is the rotation path the
off-host key did not have.

The admin backup upload counted the bytes that arrived and none after
the gunzip, so a small gzip could expand without limit into the JSON
scanner and take the app down. The inflated bytes are now held to 100
times the compressed bytes read (at least 256 MB, at most 4 GB) and a
file past that is refused with 413 before anything is stored.
2026-09-26 20:18:55 +02:00
Marc-André Bombeck 1a674cf4b0 fix(security): grant private AI endpoints by exact origin, dialled pinned
ALLOW_LOCAL_AI_PRIVATE_HOSTS=true let any user of the instance point the
Local provider or the OpenAI-compatible gateway at any private address
the server could reach, the metadata range included, and a granted host
was dialled with no connect-time pin at all.

AI_PRIVATE_ORIGINS grants a private AI endpoint with the exact-origin
grammar NOTIFICATION_PRIVATE_ORIGINS and NIGHTSCOUT_PRIVATE_ORIGINS use.
A granted call goes through the pinned operator-approved dispatcher:
redirects forbidden, and metadata, link-local and unspecified answers
dropped at dial time even for a granted name. Everything else stays on
the public pin. The Local client (buffered and streaming) and the
gateway and admin-key tags of the OpenAI client all take their policy
from one function.

The host-list form of ALLOW_LOCAL_AI_PRIVATE_HOSTS keeps working with the
same pinning; a never-grantable literal or a host:port entry is refused
with a one-time warning. Its "true" form grants nothing any more and is
reported in the boot readiness summary with the replacement.

A failed lab-report scan no longer puts the endpoint's response body into
the wide event; status and model stay. The response to the caller never
carried it.
2026-09-26 19:46:11 +02:00
Marc-André Bombeck f7b3ce6ee0 fix(privacy): encrypt Coach conversation titles and custom-metric notes at rest
The title of a Coach conversation is the first 80 characters of the
opening message, and the note on a custom-metric reading is free text;
both were stored readable while every other note was encrypted.

Migration 0357 adds coach_conversations.title_encrypted and
custom_metric_entries.note_encrypted (the title loses its NOT NULL) plus
partial indexes on the backfill's discovery predicate. Every writer now
stores ciphertext only: conversation create, proactive nudge, reminder
surfacing, rename, entry create and edit, and both restore formats.
Readers prefer the ciphertext and fall back to the old column for rows
not yet migrated; a ciphertext that does not open throws instead of
falling back. The history title search decrypts in memory, since SQL can
no longer match the title.

free-text-encryption-backfill is a boot-time, idempotent pg-boss job
(staggered past the startup storm, daily discovery tick) that seals each
remaining readable value and nulls it in one transaction per row. A pass
that sees a row twice stops with an error instead of looping.

Both columns are registered for key rotation (registry, CLI script, and
the in-app rotation that walks the registry). A disaster-recovery backup
carries ciphertext only, sealing a not-yet-migrated row on the way out;
a portable export carries readable text. The old columns are marked for
removal in a later release.
2026-09-26 19:46:01 +02:00
Marc-André Bombeck 026e8fc1ed Merge branch 'fix/compose-env-and-claims' into release/v1.39.3 2026-09-26 17:00:46 +02:00
Marc-André Bombeck bbf6a0aeb9 build: name the OCR language packages literally in the staging script 2026-09-26 15:45:28 +02:00
Marc-André Bombeck 2c88acc5f7 fix(labs): serve the in-browser OCR engine from this origin
Local lab OCR reads a report photo on the device with tesseract.js, whose
defaults load the worker, the WebAssembly core and the language data
from cdn.jsdelivr.net. The production CSP allows workers and fetches only
from this origin, so the engine could not load; where it did load (dev),
the browser of anyone scanning a report contacted a third-party CDN.

The worker, the three LSTM core builds and the deu and eng best_int
language data (about 17 MB) are now copied from pinned packages into
public/tesseract/ at prebuild and predev, and createWorker is pointed at
them with the worker started from its URL rather than a blob. A test
holds the options against the staged file list.
2026-09-26 15:37:13 +02:00
Marc-André Bombeck afcb37e76c feat(config): let the operator turn off the breach and update checks
Two requests left the server without anything being connected: the
HaveIBeenPwned range query on every password set (registration, change,
admin reset) and the daily GitHub releases request behind the About page
and the admin version tile. Neither could be turned off, which made
"nothing leaves" untrue for an instance that wanted it to be.

PASSWORD_BREACH_CHECK_DISABLED and UPDATE_CHECK_DISABLED (1, true, yes or
on) stop each request. Both stay on by default. The breach check keeps
its fail-open contract: switched off, a password is simply not looked up.
The update route answers status "unknown" with reason "disabled" and
the admin tile says the check is off. The password routes reach the
check only through the switch, and a test holds that import line.

Both are on the compose whitelist, in the env manifest under a new
"Outbound requests" group with IP_GEO_LOOKUP_DISABLED, and documented in
both env examples and the getting-started guide.
2026-09-26 15:33:34 +02:00
Marc-André Bombeck d69e12bac2 fix(scripts): skip documents the source already says are too large once HealthLog has named its limit, and show small sizes in KB 2026-09-26 14:47:12 +02:00
Marc-André Bombeck 1eac05b454 fix(documents): key imports by source instance, keep a token on its origin, and harden the import path
- Source keys carry the normalised origin of the Paperless-ngx or Papra
  they came from (source_instance on the document, the alias and the
  purge ledger; unique NULLS NOT DISTINCT). A key stored without one
  (v1.39.2 imports) matches any instance of its system, and a lookup
  without one matches any stored instance. The upload route, the lookup
  route and the import script accept or send it; backups carry it.
- Saving a connection reuses the stored token only for the same origin;
  a new origin needs the token again (422 tokenRequired).
- Document ids are checked against what each system issues (Paperless
  digits, Papra [A-Za-z0-9_-], never . or ..) before any URL is built;
  the Papra organization id likewise.
- Paperless API 9 filters dates on created__gte / created__lte, in the
  picker and the script. A search page the local re-check empties reads
  on, up to five source pages. Tags are read past the first 250.
- The import's key pre-check uses the lookup allowance, a failed fetch
  no longer charges the upload allowance, and one import per key runs at
  a time under an advisory lock, so a double click downloads once.
- Reading and deleting one's own connection works with the documents
  module off and the operator list unset.
2026-09-26 14:20:17 +02:00
Marc-André Bombeck fdc60f69c2 feat(documents): search and import from Paperless-ngx and Papra on request
A person stores one connection per system (base address, Papra
organization id, API token encrypted at rest) and can search it by
document name with tag and date filters, then import picked documents
one per request through the same ingest path as an upload: source-key
answer before any download, content dedup, deleted stays deleted, quota,
provenance, and the person's own AI reading setting. An import can link
the document to the visit, condition or vaccination it was picked for.

The operator lists the instances in DOCUMENT_SOURCE_ORIGINS; unset, the
feature is off. Only listed origins are contacted, through safeFetch with
the origin pinned, redirects refused, timeouts and byte caps. The routes
are cookie-only and owner-only. The token is registered for rotation,
removed by Delete all data, and not carried in backups.
2026-09-26 13:02:36 +02:00
Marc-André Bombeck 1667e9fdf5 Merge branch 'feat/document-import-token' into release/v1.39.2 2026-09-26 09:25:56 +02:00
Marc-André Bombeck 91acd9853f fix(scripts): send the source key in the form only, so a spent lookup never blocks the upload 2026-09-26 08:59:37 +02:00
Marc-André Bombeck e5214265f6 Merge branch 'feat/document-import-token' into release/v1.39.2
# Conflicts:
#	docs/api/openapi.yaml
#	docs/integrations/document-import.md
2026-09-26 08:55:30 +02:00
Marc-André Bombeck 39b56ee4e2 Merge branch 'fix/backup-chunked-storage' into release/v1.39.2 2026-09-26 01:49:22 +02:00
Marc-André Bombeck 8ad4610370 fix(backup): make rotation, concurrent writes and old rows safe for pieces
Key rotation re-sealed a single-value backup in place, which holds the
stored string, its bytes, the plaintext and the new ciphertext at once; a
copy can be over a hundred megabytes and the in-app rotation walked 25 of
them per page, enough to run a 1 GB container out of memory. Rotation now
converts each such copy into pieces under the active key instead, one row
at a time. The ~hlgcm1. stream of v1.38.6 to v1.39.1 is read a megabyte
at a time with its tag checked at the end, inside one transaction, so a
copy that fails the check is left untouched; the copy keeps its date. The
scan reads only the first characters of each value. Measured on a copy of
a 2.6 million-reading account (126 MB stored): converted in 25 s with the
heap flat at 57 MB under a 200 MB limit, where re-sealing in place ran out
of heap.

The weekly and manual backup leave an account alone while a restore of
its current copy is queued or running, and count it as skipped, not
failed. A reader whose copy is replaced mid-read answers 409
backup_changed, "replaced by a newer one while it was being read", rather
than the undecryptable error; a restore rolls back with the same code.

Pieces are sealed with a fixed associated-data label, so a value the same
key sealed for another purpose never opens as a piece; rotation re-seals
under the same label. A row holding both a single value and pieces, which
only an older release writing after a downgrade produces, is refused with
a message saying so, and the next weekly run replaces it. Rotation treats a
row deleted between its read and its write as gone rather than an error.
2026-09-26 01:49:02 +02:00
Marc-André Bombeck d5a39603ca Merge branch 'fix/backup-chunked-storage' into release/v1.39.2 2026-09-26 01:03:25 +02:00
Marc-André Bombeck 1e09f30eff fix(backup): store a backup in sealed pieces instead of one value
A stored backup was one value in data_backups.data. v1.39.1 wrote it a
piece at a time but still joined the pieces into that one value, and one
value has to pass through the app whole, so a copy was capped at a fifth of
the heap: 105 MB in the default 1 GB container. An account with 1.75
million readings is past that and could not be backed up at all (#1031).

A copy is now kept as ordered pieces of about a megabyte in the new
data_backup_chunks table (migration 0352). Each piece is sealed on its own
with AES-256-GCM, and its sealed header names the copy (a random stream id
kept on the row), its position and whether it is the last. A reader checks
every piece before it releases a byte, so a piece that is missing, moved,
altered, taken from another copy, or a copy cut short is refused with
backup.payload.undecryptable before a restore deletes anything, and the
second read during the restore checks each piece again.

The write is one transaction, so the previous copy stays whole and readable
until the new one commits. Pieces are written through the model API: a raw
query with a large parameter is kept alive by Prisma's query plan cache,
which left 160 MB of heap behind per hundred one-megabyte inserts.

Restore, preview, download, the restore job and the admin list read both
forms; copies written before this change keep their single value and still
restore. The heap-derived limit is gone. The one limit left is on storage,
BACKUP_MAX_STORED_MB (default 2048), and its message names the setting. Key
rotation covers the pieces, the wipe and account deletion take them with
their backup, and the backup plan lists the table as not carried.

Measured on an account of 2.6 million readings in a 1 GB container with web
and worker in one process: backup 87 s, 97 MiB in 96 pieces, preview 20 s,
download 25 s, restore 264 s with every reading back exactly, peak 567 MiB,
no restart.
2026-09-26 01:02:48 +02:00
Marc-André Bombeck f711beab20 fix(scripts): fall through on spent lookups, send the key in both places
A 429 on the lookup no longer waits for the Retry-After; the keyed upload
answers the same question. The source key goes in the query string and the
form. The closing summary and the guide point to Read with AI for deferred
documents instead of Index all.

Refs #1038
2026-09-26 00:38:26 +02:00
Marc-André Bombeck bec79879b5 Merge branch 'feat/document-import-token' into release/v1.39.2 2026-09-26 00:22:43 +02:00
Marc-André Bombeck 4af356f804 fix(scripts): skip known documents before download in the importer
Ask HealthLog whether a document is already held before downloading it
from Paperless-ngx or Papra, and send the source key in the query string.
Pin Paperless API version 9 (Paperless-ngx 2.16 and later), apply --since
locally for Paperless too, fall back to the file name for a blank title,
stop with a clear message when the HealthLog address redirects, and say
only that storage is full when the figures are withheld. The guide names
the minimum versions and the shared OTHER namespace.

Refs #1038
2026-09-25 22:35:12 +02:00
Marc-André Bombeck 100412a3a6 Merge branch 'feat/condition-body-site' into release/v1.39.2 2026-09-25 22:09:01 +02:00
Marc-André Bombeck 14f0fc4d4f feat(body-sites): record a body site on conditions and add a body-site view
A condition (illness episode) can carry where on the body it sits, in the
person's own words and encrypted at rest, plus an optional side (LEFT,
RIGHT, BOTH). Same enum, codec and reasoning as the procedure's site.

- migration 0354: two nullable columns on illness_episodes, reusing the
  laterality type from 0349
- create and edit accept bodySite and laterality; on PATCH an absent key
  keeps the stored value, so an edit from the shipped iPhone app, which
  never sends either field, leaves them untouched
- GET /api/body-sites: the sites the record holds across visits and
  conditions, folded after the decrypt and bounded per kind; with site
  (and laterality) the visits and conditions there, one hop to their
  documents, lab results and visits. Read in the profile section;
  conditions only under an illness grant with the module on, and links
  into ungranted sections come back as unlabelled placeholders
- the link service reads document and visit links from the condition's
  end of the two existing tables
- /checkups gains a Body sites tab; the visit and condition forms share
  one body-site field pair that suggests sites already used; a
  condition's page links its site into the view
- the doctor report, the clinician page and the MCP doctor-visit summary
  name the site on conditions; the Coach illness block carries it
  sanitised, and MCP get_illness_recovery returns it fenced
- backup and restore carry both fields in both file kinds; the site is
  registered as ciphertext and rotates with the key
- i18n in all seven locales, OpenAPI regenerated

Refs #1025
2026-09-25 21:55:14 +02:00
Marc-André Bombeck 0257f2207a feat(scripts): import documents from Paperless-ngx and Papra
A single dependency-free Node 22 script that copies tagged documents from
Paperless-ngx (REST, original files) or Papra (API key, /file) into the
vault through a document token. Serial, one file in memory, waits on 429
Retry-After, and resumable through the server's source-key dedup. Supports
--since, --kind / --kind-map, --dry-run with sizes, and holds back automatic
AI reading unless --ai-read is given.

Refs #1038
2026-09-25 21:48:12 +02:00
Marc-André Bombeck 55a32ff7d1 feat(documents): add a document upload token for imports from other systems
A narrow `documents:write` token, minted cookie-only at
`POST /api/tokens/documents` and accepted by the vault upload route alone.
A scoped caller draws on its own hourly bucket per token
(`DOCUMENT_UPLOAD_LIMIT_PER_HOUR`, default 120) and gets a receipt
`{ id, duplicate }` instead of the stored row.

Uploads take optional `sourceSystem` / `sourceId` fields. The source key is
unique per account across live and deleted documents, and the tombstone
purge carries it to a small ledger, so re-sending a document the owner
deleted answers 200 `deleted: true` and stores nothing. `aiRead=defer`
indexes locally and skips the summary and lab staging.

The detail sheet shows where an imported document came from; backup and
restore carry both fields; Delete All Data clears the ledger.

Refs #1038, #938
2026-09-25 21:48:04 +02:00