Commit Graph
436 Commits
Author SHA1 Message Date
Marc-André Bombeck 72a34b3b47 Merge branch 'fix/v1393-review-data' into release/v1.39.3 2026-09-27 00:18:47 +02:00
Marc-André Bombeck f4b9d8a0e1 fix(backup): list the purge job's awaiting-run count among its outcome facts 2026-09-26 22:35:49 +02:00
Marc-André Bombeck 382c44ee88 fix(import): keep a queued Apple Health upload when sweeping staged files
The staging sweep removed any staged upload older than seven hours. An
import can wait in the queue behind another account's import for longer
than that, and when its turn came the file was gone and the import failed.
The sweep now keeps every upload a queued or running import still names,
and the extracted XML while an import is past the queue. When the queue
cannot be read, it removes nothing that tick.
2026-09-26 22:21:46 +02:00
Marc-André Bombeck b6b51fcb5e fix(medications): let a snooze set before a dose's window hold that dose
Since a snooze holds only the slots whose window had opened when it was
set, a snooze set ahead of a dose, such as "remind me later" in the app
before the dose is due, held nothing and the reminder came anyway. A
snooze set while no window of the medication is open now holds the next
slot to open. One set while a window is open still holds only that slot,
so a morning skip that snoozes to the end of the day leaves the evening
dose reminded.
2026-09-26 22:17:14 +02:00
Marc-André Bombeck 612eedcbbb fix(medications): warn when an as-needed medication runs low
Since the supply runway stopped counting an as-needed schedule as a daily
dose, a medication taken only as needed had no runway and never raised a
low-supply notice. It now warns once when two doses or fewer are left,
with its own line naming what is left, and re-arms after a refill like any
other medication.
2026-09-26 22:14:03 +02:00
Marc-André Bombeck bb12d942c0 fix(backup): keep a wipe's purge request until the next off-host run
A data wipe during the nightly upload could leave the pre-wipe copy in
the bucket: the purge the wipe started removed its request at once, and
the upload still in flight checked that request after finishing and found
nothing. A wipe's request now stays until a nightly run has walked the
account since the wipe. Until then each purge removes only the copies
dated up to the wipe's day, so the copy of what the account holds after
the wipe stays, and the admin card no longer counts the waiting request
as copies still in the bucket.
2026-09-26 22:08:51 +02:00
Marc-André Bombeck e9cf3c1b15 fix: carry the cross-branch rules through offhost keys, the Telegram test mock and the OIDC reason list 2026-09-26 22:07:11 +02:00
Marc-André Bombeck 66b1aef40c Merge branch 'fix/encrypt-free-text-and-local-ai' into release/v1.39.3
# Conflicts:
#	src/app/api/import/csv/route.ts
2026-09-26 21:50:05 +02:00
Marc-André Bombeck 4409529c89 Merge branch 'fix/backup-rotation-and-deletion' into release/v1.39.3
# Conflicts:
#	docs/api/openapi.yaml
#	src/lib/jobs/apple-health-import-worker.ts
2026-09-26 21:48:20 +02:00
Marc-André Bombeck 7da778760e fix(backup): leave the admin wipe's off-host copies in place, and correct the backup and rotation runbooks
The admin wipe of every account is the operator clearing their own host,
and the bucket's copies are the documented way back from a wipe pressed
by mistake, so it writes no off-host deletion request. Deleting an
account, deleting a managed profile and "Delete all data" still do.

The backup runbook no longer says the worker never deletes an object: it
lists the grant the worker needs (DeleteObject included), when objects
are deleted, the version-4 envelope, the lifecycle check and how to
rotate the off-host key. The rotation runbook no longer says backups are
covered: rotation reaches a backup's envelope, not the content a restore
writes back, and a retired key stays needed until no backup lists it.
The privacy notice beside account deletion says the same about off-host
copies.

A failed import records its failure in a plain try block, and the
extraction test that inflates past the 64 MiB floor gets the time it
needs on a loaded runner.
2026-09-26 20:33:30 +02:00
Marc-André Bombeck 849748c4cd fix(backup): bind new off-host objects to their key and account, and cap what an uploaded backup may inflate to
Off-host objects are now written as envelope version 4: the header names
the off-host key by fingerprint, and the object's key in the bucket
(<date>/user-<id>.json.enc) is authenticated as GCM associated data, so
a copy moved to another account or another night no longer opens.
Objects of versions 1 to 3 still read. BACKUP_ENCRYPTION_PREVIOUS_KEYS
(comma-separated, on the compose whitelist and in the env manifest)
keeps retired off-host keys readable, which is the rotation path the
off-host key did not have.

The admin backup upload counted the bytes that arrived and none after
the gunzip, so a small gzip could expand without limit into the JSON
scanner and take the app down. The inflated bytes are now held to 100
times the compressed bytes read (at least 256 MB, at most 4 GB) and a
file past that is refused with 413 before anything is stored.
2026-09-26 20:18:55 +02:00
Marc-André Bombeck 1b4594e69f fix(import): bound what an export may inflate to, allow one import per account, and never leave the upload behind
The extractor held the inflated export.xml to a flat 8 GiB whatever the
member occupied, and the header ratio check reads sizes the archive
writes itself: a member declaring an honest ratio could inflate at
1000:1, so an 8 MB upload could fill the temp directory. The streamed
byte count is now held to 200 times the member's compressed size on
disk (never less than 64 MiB, never more than 8 GiB), and a stored
member may not grow at all.

An account may have one Apple Health import queued or running. A
second upload with different bytes is refused with 409
import.apple_health.busy and the running job's id; the check and the
new row are one transaction under an advisory lock.

The staged upload is the person's export in plain text. It is now
removed on every exit that leaves no worker to read it: a refused or
broken multipart body, no worker running, a queue that takes no job
(the row is failed as well), the admin route's missing or unknown
target account, and an account deleted before the worker reached it.
The worker removes the files before recording a failure, so a row that
went with a deleted account no longer strands them. The 15-minute
import reconcile pass also sweeps staged files older than the longest
an import may run.
2026-09-26 20:11:45 +02:00
Marc-André Bombeck 1c7785c23b fix(backup): take off-host copies out with the account, and say which keys backups still need
Deleting an account, deleting a managed profile, "Delete all data" and the
admin wipe now write an off-host deletion request in the same transaction
as the deletion. A new job (offhost-backup-purge, kicked straight away and
run nightly as a backstop) lists the bucket once, deletes every
<date>/user-<id>.json.enc object of each pending account and removes the
request only when nothing of the account is left. A refused delete keeps
the request with the bucket's answer and is retried; the off-host card
shows how many deletions are outstanding and why. A copy the nightly run
was writing while the account was deleted is taken back after the upload.

The off-host card also reads the bucket's lifecycle rule and says plainly
when there is none (every nightly copy kept forever) or when the
credential may not read it.

Every stored backup now records the key ids its content was written
under, and every off-host upload records the same per key. The admin
encryption view lists, per key, the stored copies that need it with the
oldest date, and when the last off-host copy needing it expires. The
"safe to drop" badge now requires that no backup still needs a retired
key, instead of only that every row was rotated.

Migration 0358 adds data_backups.inner_key_ids, the key-use table and the
deletion requests.
2026-09-26 19:54:40 +02:00
Marc-André Bombeck f7b3ce6ee0 fix(privacy): encrypt Coach conversation titles and custom-metric notes at rest
The title of a Coach conversation is the first 80 characters of the
opening message, and the note on a custom-metric reading is free text;
both were stored readable while every other note was encrypted.

Migration 0357 adds coach_conversations.title_encrypted and
custom_metric_entries.note_encrypted (the title loses its NOT NULL) plus
partial indexes on the backfill's discovery predicate. Every writer now
stores ciphertext only: conversation create, proactive nudge, reminder
surfacing, rename, entry create and edit, and both restore formats.
Readers prefer the ciphertext and fall back to the old column for rows
not yet migrated; a ciphertext that does not open throws instead of
falling back. The history title search decrypts in memory, since SQL can
no longer match the title.

free-text-encryption-backfill is a boot-time, idempotent pg-boss job
(staggered past the startup storm, daily discovery tick) that seals each
remaining readable value and nulls it in one transaction per row. A pass
that sees a row twice stops with an error instead of looping.

Both columns are registered for key rotation (registry, CLI script, and
the in-app rotation that walks the registry). A disaster-recovery backup
carries ciphertext only, sealing a not-yet-migrated row on the way out;
a portable export carries readable text. The old columns are marked for
removal in a later release.
2026-09-26 19:46:01 +02:00
Marc-André Bombeck 405c996b6a fix(backup): refuse a restore that needs a dropped key or reaches into another account
A backup carries every encrypted column as the stored ciphertext, and a
restore writes it back verbatim. Key rotation re-seals the stored copy's
envelope but cannot reach inside it, so a copy taken before a rotation
still needs the retired key. Without it the restore went through and
left rows no reader can open.

The restore, the restore preview, the admin upload and the monthly
restore drill now collect the key ids of every inner ciphertext value,
hold them against ENCRYPTION_KEYS and decrypt one value per key. A
missing key, or an id configured with different key material, is
refused before anything is deleted, with the key id named. The restore
status and the preview say which key to put back.

The restore also checks, inside its transaction, that it added no
reference from the account's rows to rows another account owns. The
check is derived from the schema's foreign keys rather than written per
section, so the tag and category upserts keyed on a global key are
covered along with every link table. The illness parent update is
scoped to the owner as well.

The off-host listing now follows every page: the first page held only
the oldest dates, so the drill could read an old object and report the
chain as stale.
2026-09-26 19:22:03 +02:00
Marc-André Bombeck 314e9c1aa3 refactor(tz): take the medication and reminder trees off the zone literal
The medication and measurement-reminder code now imports DEFAULT_TIMEZONE
from the tz module like every other fallback, so the lint rule no longer
exempts those paths.
2026-09-26 17:03:19 +02:00
Marc-André Bombeck e2814336e5 Merge branch 'fix/aggregation-day-boundaries' into release/v1.39.3
# Conflicts:
#	src/app/api/medications/[id]/route.ts
2026-09-26 17:00:54 +02:00
Marc-André Bombeck 026e8fc1ed Merge branch 'fix/compose-env-and-claims' into release/v1.39.3 2026-09-26 17:00:46 +02:00
Marc-André Bombeck bd8cf5d83f Merge branch 'fix/medication-schedule-timezones' into release/v1.39.3 2026-09-26 17:00:46 +02:00
Marc-André Bombeck 9d2fd1db14 fix(tz): one default zone, and the user's own calendar where Berlin was hard-coded
The status assessments folded their recent days, weeks, months and years on
Berlin days whatever the user's zone, the mood and cycle crosstabs keyed
their metric days in Berlin, the resting-pulse proxy defaulted to Berlin
days, the briefing's opener rotated at Berlin midnight, and every
assessment prompt labelled today's date as Berlin's. They now use the
user's zone. Rollup months and years are labelled by their UTC start,
which is how they are cut; read in a zone west of UTC the label named the
month before.

"Europe/Berlin" was written out in some eighty places. DEFAULT_TIMEZONE in
lib/tz/format is now the one definition: the second copy in mood/date-key,
DISPLAY_TIMEZONE, the Berlin-only day key helpers and the per-route copies
are gone. An in-repo lint rule rejects the literal outside lib/tz and
tests; four areas another change owns are listed as pending.
2026-09-26 16:23:14 +02:00
Marc-André Bombeck c88754a0b5 fix(measurements): re-warm status assessments from every measurement write
The MCP writes, the Telegram reply, the CSV and backup importers and the
Apple export import refreshed the rollups by hand and never re-warmed the
cached status assessments, so a reading logged there kept the day's old
assessment until the nightly pass. They now run the shared post-write tail,
and the export import re-warms every type it wrote. The WHOOP body sync
removed its legacy weight row without refolding that day; it does now.

A structural test lists every file that writes measurements and how it keeps
both readers current, so a new write site fails until it says.
2026-09-26 16:07:51 +02:00
Marc-André Bombeck 62cef974bf fix(medications): scope a snooze to its slot and key report adherence by id
A snooze lives on the medication row, and the reminder tick applied it to
every slot of that medication. A Telegram "skip" of the morning dose
snoozes to the end of the day, so the evening dose of the same medication
was neither reminded nor marked missed by the reminder path. A snooze now
holds only slots whose reminder window had opened when it was set, dated by
the row's updatedAt that the snooze write stamps (the same reading the skip
and pin suppression already uses). A later edit to the row can only widen
the hold back to the old behaviour.

The doctor report kept adherence in a map keyed by medication name, so two
medications with the same name (an ended course and a new one, two
strengths) collapsed into one row and the second overwrote the first. The
map is keyed by medication id now and carries the name as the label; the
report's medication list carries the id. The PDF, the FHIR adherence
observations, the MCP visit summary, the clinician view and the GLP-1
section read it that way.
2026-09-26 15:38:20 +02:00
Marc-André Bombeck 91571d4674 fix(medications): count the whole cadence in the supply runway
The runway estimate read FREQ=MONTHLY as one dose every 30 days whatever
its INTERVAL, so a quarterly injection with two doses left showed about 60
days of supply instead of 180. FREQ=DAILY;INTERVAL=2 counted as daily, a
yearly rule ignored INTERVAL and BYMONTH, and a monthly rule ignored extra
BYMONTHDAY values. A CYCLIC schedule was counted as if it never paused,
and a PRN schedule as if it were due every day. The rate now honours
INTERVAL for every frequency, counts the listed days and months, scales a
cyclic schedule by on / (on + off) and leaves PRN out. The low-stock job
selects the schedule type and cycle lengths so it reads the same rate.

Whole-dose counts no longer lose one to binary floating point: 2.4 mL at
0.8 mL per dose is 2.9999999999999996 in IEEE 754 and used to read as two
doses. The stock summary, both runway estimators, the list and detail
routes, the GLP-1 route, the Coach snapshot and the inventory readout now
floor through one helper that treats anything within 1e-9 of the next whole
number as that number.
2026-09-26 15:32:29 +02:00
Marc-André Bombeck 888b738546 fix(reminders): send a multi-hour protocol at every hour it names
A reminder with FREQ=DAILY;BYHOUR=7,19 (the morning-and-evening blood
pressure course) was only ever sent in the morning. The tick opened its
send window at notifyHour alone, so the 19:00 slot stayed due until 07:00
the next day, and the claim was keyed per local day, so a second send that
day would have been refused anyway.

The window is now the hour of the current slot: the local hour of
nextDueAt when it is one of the rule's BYHOUR values, else notifyHour. The
early skip for a reminder already sent for its slot uses the same hour, so
it stays consistent with the hold-open rule and lastNotifiedAt. A
multi-hour reminder claims per slot hour; a single-hour reminder keeps the
per-day key.
2026-09-26 15:27:25 +02:00
Marc-André Bombeck 13db7935f6 fix(image): ship the operator scripts the runbooks run in the container
The password-reset runbook tells an operator to run
`docker compose exec app node scripts/reset-password.mjs`, and the
intake-repair runbook runs `healthlog-tsx scripts/repair-intake-anomalies.ts`
inside the container. The runner stage copied nothing from scripts/, so
both failed with "Cannot find module", the reset at the moment an admin
is locked out.

The runner stage now copies both scripts and the Argon2 parameter module
the reset imports, and proves at build time that each reaches its own
first check with every import resolved. A structural test holds every
in-container command in docs/ and the Dockerfile copies in step.

Key rotation is documented the way it works on the published image: the
Rotate now button under Admin > Encryption is the primary path. The
rotation script imports the application graph and runs from a source
checkout of the same release; the runbook, the admin hint in all seven
languages and the decrypt error now say so.
2026-09-26 15:26:03 +02:00
Marc-André Bombeck 3c3e5ceab7 fix(config): treat an empty environment variable as unset
docker-compose.yml forwards its whitelist as "${NAME:-}", so a variable
the operator never set reaches the process as the empty string. Reads
written as a nullish fallback kept that string: the weather client built
an empty base URL and every geocode failed with ERR_INVALID_URL, and the
IP geo lookup requested a relative path.

Reads with a fallback now go through src/lib/env.ts (envValue, envOr,
envFlag), which treats empty and whitespace as unset. IP_GEO_LOOKUP_DISABLED
accepts 1, true, yes and on instead of only "1". A structural test refuses
a ?? fallback written directly against process.env anywhere in src/, and
the compose whitelist guard counts the helper's call shape as a read.
2026-09-26 15:20:42 +02:00
Marc-André Bombeck 1eac05b454 fix(documents): key imports by source instance, keep a token on its origin, and harden the import path
- Source keys carry the normalised origin of the Paperless-ngx or Papra
  they came from (source_instance on the document, the alias and the
  purge ledger; unique NULLS NOT DISTINCT). A key stored without one
  (v1.39.2 imports) matches any instance of its system, and a lookup
  without one matches any stored instance. The upload route, the lookup
  route and the import script accept or send it; backups carry it.
- Saving a connection reuses the stored token only for the same origin;
  a new origin needs the token again (422 tokenRequired).
- Document ids are checked against what each system issues (Paperless
  digits, Papra [A-Za-z0-9_-], never . or ..) before any URL is built;
  the Papra organization id likewise.
- Paperless API 9 filters dates on created__gte / created__lte, in the
  picker and the script. A search page the local re-check empties reads
  on, up to five source pages. Tags are read past the first 250.
- The import's key pre-check uses the lookup allowance, a failed fetch
  no longer charges the upload allowance, and one import per key runs at
  a time under an advisory lock, so a double click downloads once.
- Reading and deleting one's own connection works with the documents
  module off and the operator list unset.
2026-09-26 14:20:17 +02:00
Marc-André Bombeck 31e4c95487 fix(reminders): one cycle rule for the tick and the start page
The start page read every reminder due before the end of the day, so a
daily weigh-in at 20:00 or a twice-daily blood-pressure course showed
"check-up due" from the morning on and took a pinned place on the rail.

`holdsOpenAfterReminder` is now the one definition of a reminder that
stays due after it is sent: an interval longer than seven days, a rule
whose next gap is longer than seven days, a rule with no occurrence
left, or no cadence; never an appointment. The tick uses it to decide
whether a delivered reminder rolls on, and the start page uses it to
show such a reminder for its whole due day and keep it under the cap.
A short-cycle reminder shows only once its time has come and is not
pinned.

The tick also skips a reminder already sent for its current slot
outside its notify hour before the auto-resolve read. The eventful
satisfy worker still resolves it the moment a reading or lab result
lands, and the poll still runs for it once a day, in the notify hour;
the weekly hold stays after that poll so the safety net is daily.
2026-09-26 10:36:15 +02:00
Marc-André Bombeck fb960fb72e fix(reminders): decide what stays due by the cycle, not the measurement type
The first cut kept only free-text check-ups due after their reminder
and let every reminder with a measurement type roll on. A fortnightly
PHQ-9 or GAD-7 reminder is a measurement reminder, and each one still
jumped two weeks forward the morning it was sent without the
questionnaire being filled.

The rule is now the cycle length. A reminder whose next slot is more
than seven days away, or that has no cadence, stays due after its
reminder until it is satisfied (a matching reading, a lab result or
"done"), skipped or snoozed, and is sent again at most once a week. A
reminder on a weekly or shorter cycle rolls on as before. Appointments
stay one-shot. A reading that lands on an open measurement reminder
still sets lastSatisfiedAt and schedules the next cycle from itself.

The 0355 repair drops its "no measurement type" condition for the same
cycle rule, so the fortnightly questionnaires in the reported shape
are put back on the day they were reminded.
2026-09-26 10:33:13 +02:00
Marc-André Bombeck 54b46f4679 fix(reminders): keep an open check-up due after its reminder
The reminder tick rolled every delivered reminder on to its next slot.
For a check-up (a reminder with no measurement type, made by the person
or suggested by the Coach) that filed an open task under next month or
next year the moment it was reminded, although nobody had done it. The
start page and the check-up card then showed a yearly check-up due
today as scheduled for next year, and a check-up with no cadence
dropped off every surface.

A check-up now keeps its due date after the reminder until it is
marked done, skipped or snoozed, and is reminded again at most once a
week while it stays open. A new `last_notified_at` column records the
last delivered reminder; a repeat is held while that send was for the
current slot and fewer than seven calendar days have passed, so a
snooze, a skip or a completion opens a fresh slot that fires on its own
day. The per-day claim stays the guard against two sends in one day.
Measurement reminders keep rolling on (the next reading satisfies them
and their next slot comes round soon), as do check-ups on a weekly or
shorter cadence, whose next slot is the next reminder anyway.
Appointment reminders stay one-shot.

Migration 0355 adds the column and puts back check-ups the old tick
rolled on: a live, enabled check-up whose most recent reminder claim is
followed within ten minutes by its last write, carrying a due date more
than a week past that send (or none, for a one-shot), not done, skipped
or snoozed since. Its due date returns to the instant of that send.
Anything else is left as it is. The same migration adds `upcoming_visit`
to start-page item choices stored before visits were one of them, which
otherwise read as visits switched off.
2026-09-26 10:01:27 +02:00
Marc-André Bombeck e5214265f6 Merge branch 'feat/document-import-token' into release/v1.39.2
# Conflicts:
#	docs/api/openapi.yaml
#	docs/integrations/document-import.md
2026-09-26 08:55:30 +02:00
Marc-André Bombeck 39b56ee4e2 Merge branch 'fix/backup-chunked-storage' into release/v1.39.2 2026-09-26 01:49:22 +02:00
Marc-André Bombeck 8ad4610370 fix(backup): make rotation, concurrent writes and old rows safe for pieces
Key rotation re-sealed a single-value backup in place, which holds the
stored string, its bytes, the plaintext and the new ciphertext at once; a
copy can be over a hundred megabytes and the in-app rotation walked 25 of
them per page, enough to run a 1 GB container out of memory. Rotation now
converts each such copy into pieces under the active key instead, one row
at a time. The ~hlgcm1. stream of v1.38.6 to v1.39.1 is read a megabyte
at a time with its tag checked at the end, inside one transaction, so a
copy that fails the check is left untouched; the copy keeps its date. The
scan reads only the first characters of each value. Measured on a copy of
a 2.6 million-reading account (126 MB stored): converted in 25 s with the
heap flat at 57 MB under a 200 MB limit, where re-sealing in place ran out
of heap.

The weekly and manual backup leave an account alone while a restore of
its current copy is queued or running, and count it as skipped, not
failed. A reader whose copy is replaced mid-read answers 409
backup_changed, "replaced by a newer one while it was being read", rather
than the undecryptable error; a restore rolls back with the same code.

Pieces are sealed with a fixed associated-data label, so a value the same
key sealed for another purpose never opens as a piece; rotation re-seals
under the same label. A row holding both a single value and pieces, which
only an older release writing after a downgrade produces, is refused with
a message saying so, and the next weekly run replaces it. Rotation treats a
row deleted between its read and its write as gone rather than an error.
2026-09-26 01:49:02 +02:00
Marc-André Bombeck d5a39603ca Merge branch 'fix/backup-chunked-storage' into release/v1.39.2 2026-09-26 01:03:25 +02:00
Marc-André Bombeck 1e09f30eff fix(backup): store a backup in sealed pieces instead of one value
A stored backup was one value in data_backups.data. v1.39.1 wrote it a
piece at a time but still joined the pieces into that one value, and one
value has to pass through the app whole, so a copy was capped at a fifth of
the heap: 105 MB in the default 1 GB container. An account with 1.75
million readings is past that and could not be backed up at all (#1031).

A copy is now kept as ordered pieces of about a megabyte in the new
data_backup_chunks table (migration 0352). Each piece is sealed on its own
with AES-256-GCM, and its sealed header names the copy (a random stream id
kept on the row), its position and whether it is the last. A reader checks
every piece before it releases a byte, so a piece that is missing, moved,
altered, taken from another copy, or a copy cut short is refused with
backup.payload.undecryptable before a restore deletes anything, and the
second read during the restore checks each piece again.

The write is one transaction, so the previous copy stays whole and readable
until the new one commits. Pieces are written through the model API: a raw
query with a large parameter is kept alive by Prisma's query plan cache,
which left 160 MB of heap behind per hundred one-megabyte inserts.

Restore, preview, download, the restore job and the admin list read both
forms; copies written before this change keep their single value and still
restore. The heap-derived limit is gone. The one limit left is on storage,
BACKUP_MAX_STORED_MB (default 2048), and its message names the setting. Key
rotation covers the pieces, the wipe and account deletion take them with
their backup, and the backup plan lists the table as not carried.

Measured on an account of 2.6 million readings in a 1 GB container with web
and worker in one process: backup 87 s, 97 MiB in 96 pieces, preview 20 s,
download 25 s, restore 264 s with every reading back exactly, peak 567 MiB,
no restart.
2026-09-26 01:02:48 +02:00
Marc-André Bombeck 499b7dd9a7 fix(documents): keep deferred imports local and meter every re-send
A document uploaded with aiRead=defer is now indexed locally by any
automatic path, because the indexer reads the marker from the row rather
than from the job. Index all for search leaves such documents out and no
longer counts them. Only an explicit Read with AI or Generate summary ends
the hold.

Uploads that send a body keep their slot of the hourly limit, duplicates
included; handing it back let a token send the same bytes under id after
id for free. One document takes at most 20 further source ids, beyond
which the upload answers 409 documents.inbound.sourceAliasLimit. The
query-string key check is metered on the lookup allowance it shares with
GET /api/documents/inbound/source.

Refs #1038
2026-09-26 00:38:26 +02:00
Marc-André Bombeck 10a1c3ca2e fix(documents): keep every import key deleted, answer re-sends cheaply
A keyed upload that matched a stored document by its bytes never recorded
its key, so deleting that document let the next import run store it again.
Such keys are now kept as aliases of the document, the source-key check
reads them, and the purge carries them into the ledger with the row's own.

A source key may ride the query string and is then answered before the
body is read or the hourly allowance charged; an upload answered as a
duplicate hands its slot back. GET /api/documents/inbound/source answers
whether a key is held, under the same documents:write scope, so an
importer can skip a download.

aiRead=defer is stored on the row. The summary catch-up, the summary job
and lab auto-staging skip such documents until an explicit Read with AI
or Generate summary clears the marker; backup and restore carry it.

A documents:write caller no longer sees the quota figures on a full vault.

Refs #1038
2026-09-25 22:35:11 +02:00
Marc-André Bombeck 55a32ff7d1 feat(documents): add a document upload token for imports from other systems
A narrow `documents:write` token, minted cookie-only at
`POST /api/tokens/documents` and accepted by the vault upload route alone.
A scoped caller draws on its own hourly bucket per token
(`DOCUMENT_UPLOAD_LIMIT_PER_HOUR`, default 120) and gets a receipt
`{ id, duplicate }` instead of the stored row.

Uploads take optional `sourceSystem` / `sourceId` fields. The source key is
unique per account across live and deleted documents, and the tombstone
purge carries it to a small ledger, so re-sending a document the owner
deleted answers 200 `deleted: true` and stores nothing. `aiRead=defer`
indexes locally and skips the summary and lab staging.

The detail sheet shows where an imported document came from; backup and
restore carry both fields; Delete All Data clears the ledger.

Refs #1038, #938
2026-09-25 21:48:04 +02:00
Marc-André Bombeck 37c6b0e144 fix(medications): never mint a slot from before the live schedule era
The projector floored today's slots only on an empty archived era, the
reminder worker and the next-due reads on the newest revision boundary,
and the auto-miss pass on the creation instant alone. Switching tracking
back on and then moving a dose time the same afternoon therefore put the
morning dose on the card, and the next day's pass stamped it a miss; a
same-day times edit on any medication did the same.

One helper now names the live era start (the newest active revision's
boundary) and every path floors on it: the projector, the worker, the
doses card, the medication list, the MCP schedule tool and the auto-miss
pass, which leaves a placeholder minted after the switch for a slot
before it.
2026-09-25 10:03:08 +02:00
Marc-André Bombeck 1d1570733e fix(medications): leave record-only doses out of the nudge, the status model and the BP correlation
Three readers compute a rate straight from intake rows without reading a
schedule: the weekly compliance nudge, the adherence series behind the
general status, and the BP-versus-compliance correlation. They now read
tracked medications only. The structural guard covers intake-row readers
as well as schedule readers, with a whitespace-tolerant matcher, a floor
on the match count and a reviewed allowlist.
2026-09-25 10:03:08 +02:00
Marc-André Bombeck e03cf7c2c8 Merge branch 'feat/medication-record-only' into release/v1.39.1 2026-09-25 09:37:27 +02:00
Marc-André Bombeck b4c2706ec9 feat(medications): keep a medication as a record without tracking intake
A per-medication trackIntake flag (migration 0351, default on for every
existing and new row). With it off the dose, dates and schedule stay stored,
but nothing derives a due dose from them: the reminder worker, the manual
reminder check, the today projector, the auto-miss pass, the low-stock pass,
the doses card, the today list, the outstanding-dose badge, compliance and
its heatmap, the doctor report, the Coach, the BP-status gate, insights,
targets, achievements, the efficacy view and the MCP schedule tool all skip
it.

The medication wire serves such a medication with schedules: [] and the
stored rows in an additive recordedSchedules field, so a client that arms
local reminders from schedules stays silent without an update. Switching
tracking off or back on archives a schedule revision at that instant, the
untracked stretch with an empty payload, so it never counts as missed and
the projector does not put a dose from before the switch back on the card.

A structural guard freezes the set of schedule readers: each one filters on
the flag, derives from dueSchedules, shapes the wire with scheduleWireFields,
or is allowlisted with the reason it derives nothing due. The flag travels
through backup, restore and the medication CSV export.
2026-09-25 08:57:10 +02:00
Marc-André Bombeck 3abbad048e fix(backup): never run a restore again after its data committed
A restore commits its transaction and then rebuilds the chart tiers. A
worker that stopped during that rebuild left the job running with a stale
heartbeat, and the boot sweep and the admission check treated it like any
interrupted job: the sweep queued it again, so the second run deleted and
re-wrote the account and lost whatever was written in between, and a second
interruption closed it with "rolled back, nothing was changed", which was
false.

The job now records `committedAt` (added to migration 0347, which has not
shipped) and awaits that write before the rebuild starts. The sweep and the
admission check close a committed job as `failed_after_commit`, never queue
it again, and evict the account's caches. The rebuild itself is left to the
nightly and boot passes that already rebuild missing tiers.

The module comment now says what an interrupted restore does (restarted
once, then failed) and what happens to other writes while a restore runs:
new rows are written at once, updates to rows the restore already deleted
wait on their lock until it commits or rolls back.
2026-09-25 08:49:45 +02:00
Marc-André Bombeck 47e9f7e81c feat(backup): run a restore as a background job with progress
Restoring a large account was one request: 94 s for 1.25 million readings.
A reverse proxy that cuts a request at 60 s (nginx's default) showed the
operator an error for a restore that went on to finish on the server.

The restore request now checks the typed confirmation, that the backup
exists and that its stored copy opens, admits one job for the account and
answers 202 with its id. The `backup-restore` queue runs the restore and
records its phase, counts and outcome on a `backup_restore_jobs` row
(migration 0347), which `GET /api/admin/backups/restores` and
`GET /api/admin/backups/restores/{id}` read back.

What a restore checks and writes is unchanged. The body moved from the
route into `restoreBackup` (src/lib/export/restore-backup.ts): every
refusal still happens before the first delete and the replacement is still
one transaction, so a failure part-way leaves the account as it was. While
it runs the account keeps showing its current data, and other writes are
neither refused nor held, as before.

- One restore per account: a partial unique index admits one queued or
  running job, and a second request gets 409 `backup.restore.active` naming
  the running job. A per-account advisory lock inside the transaction holds
  even if a worker thought gone is still writing.
- Time budget: the job's expiry is two hours and a file whose transaction
  limit would run past three quarters of it is refused before anything is
  deleted (`jobDeadline`, beside `jobBudget`).
- Resuming: the job writes a heartbeat every two seconds. A boot sweep
  queues a job whose worker stopped again from the start, at most twice,
  then fails it as interrupted. The admission check releases an abandoned
  job so it never holds the account's slot.
- The staged file is the stored `DataBackup` row, already encrypted at
  rest. It stays whatever the outcome, so a retry needs no new upload. The
  job records a digest of the copy and refuses one that was replaced while
  it waited.
- Failures carry a stable code and a sentence that says what to do.
- The console follows the job: phase and counts while it runs, the outcome
  after, and after a reload it finds the job again from the server. A poll
  that fails on the network is never shown as a failed restore. A web
  process separate from the worker evicts the account's caches when it
  first reports the restore finished. The success line goes through the
  outcome module, so an empty file is not drawn as a tick.

The model is registered in the backup plan (not carried) and the wipe plan
(wiped with the account); the queue in the maintenance registrar, the
subsystem surface and the job-fact allowlist; the new code in the error
catalogue; the routes in the OpenAPI contract. The tests that pin restore
semantics now drive the real route and the real job through one helper, and
`backup-restore-job.test.ts` pins the asynchronous contract itself.
2026-09-25 07:39:06 +02:00
Marc-André Bombeck d81518b5bf Merge branch 'fix/large-account-jobs' into release/v1.39.1 2026-09-25 06:49:16 +02:00
Marc-André Bombeck c757981638 fix(jobs): give the hourly rebuild the same time budget
The one-shot hourly rebuild of already-folded heart-rate, HRV and SpO2
days walks one account's whole folded history in one job with the
default fifteen-minute expiry, about fifty statements per day. Like the
drains it now stops between days once 75 % of the expiry has passed or
pg-boss aborts the job, and fails so pg-boss retries it (the boot send
allows three retries); a rebuilt day leaves the candidate set, so the
retry starts at the first day not yet rebuilt.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck b7a26d3a4d fix(backup): resume an off-host run where its last attempt stopped
The off-host backup walked every account in one job with the default
fifteen-minute expiry, and a retry started again from the first account.
On an instance with one large record next to others, an attempt that
ran out of time spent its retry uploading the accounts it had already
done (#1031).

The run now stops between accounts once the job's time budget is spent
and fails, so pg-boss retries it. Every attempt of a job carries the same
creation time; an attempt skips the accounts whose ledger shows a success
since then, and keys its objects on that day, so a retry after midnight
still writes under the run's date. A new job uploads every account again,
and a retry whose accounts were all done passes.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck 4242b454ce fix(import): fold rollups over the span the import wrote
After an Apple Health import the worker folded the rollup table from the
account's first measurement to its last, whatever the import carried, and
so also folded history older than the five-year window that the boot
backfill, its discovery and every read path keep to.

The import now records the oldest and newest instant it wrote
(`measuredSpan` on the job result), and the fold covers that span,
clamped to the fold window. On the seeded account of 1.25 million rows the
full fold took 12 s, the slowest statement 2.7 s against the 60 s
statement limit, so this is about the window rather than time.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck 237acc0002 fix(backup): export, upload and restore a large account as streams
An account of 1.25 million measurements could not export its data or be
restored from any backup (#1031). Measured in a 1 GB container:

- The full export and the passphrase export built the whole payload and
  its JSON string in memory; the app ran out of heap and restarted.
- The weekly backup's JSON is 662 MB, longer than any string V8 can hold,
  so `unpackBackupBlob` threw on every read. Restore, download and the
  restore preview all answered "could not be decrypted".
- Upload accepted 10 MB, a fraction of a large export.

Every reader and writer of a backup file now streams it:

- `scanBackupJson` reads a backup's bytes once, keeps every section but
  the named bulk arrays, and hands their elements out one at a time.
  `readStreamedBackup` uses it twice: a first pass that validates each
  measurement against the element schema and keeps the rest of the file,
  and a second pass that hands the measurements to the writer in batches.
  Stored copies and off-host objects are authenticated whole on their
  compressed ciphertext before any plaintext is read; only the
  decompression is streamed.
- Restore writes the measurements with one `INSERT … FROM unnest` per
  batch. Prisma caches a query plan per `createMany` row count, and one
  1 000-row plan kept about 145 MB alive, which alone put the restore past
  the heap limit. The transaction budget grows with the measurement count.
  The awards and ECG sections are handed the ids they reference that were
  written, not every written id.
- The export routes write the file into the response as it is produced.
  The passphrase export spools its ciphertext to a temporary file because
  the HLX1 format puts the tag first; the format is unchanged, so existing
  archives open as before.
- Upload accepts the raw body, JSON or gzip, up to the 512 MB request
  ceiling, checks it as it arrives and stores it in the same pass; the admin
  page compresses the file in the browser before sending it. Multipart stays
  for small files.
- Download streams the stored JSON; the preview counts measurements from
  the stream. The monthly restore drill and `scripts/restore-backup.ts`
  read the off-host object the same way; the script can write `.json.gz`.

On the seeded account in a 1 GB container: export 25 s at 389 MB peak,
passphrase export 25 s, both uploads 19 s, restore of the weekly backup
94 s at 554 MB peak, and export, upload and restore round-trip to exactly
the same rows. The encrypted-column guard now also recognises the raw SQL
statement that writes `DataBackup.data`.
2026-09-25 02:05:19 +02:00
Marc-André Bombeck 9411a485c8 fix(backup): store the weekly and manual backup piece by piece
The backup streamed its JSON through gzip and the cipher, but kept the
stored copy as one string and handed that to Postgres in one parameter.
On a seeded account of 1.25 million measurements the copy is 64 MB of
base64, and the pieces, the joined string and the driver's copies of it
grew the process from 309 MB to 946 MB resident. A bare process needed a
heap limit above 400 MB to finish it (the new path finishes under 220 MB).
With the dense retention running beside it in a 1 GB container the
process died of heap exhaustion, and the backup job sat "active" until its
two-hour limit marked it "job timed out" (#1031).

`packBackupBlobInto` hands the stored copy on in pieces of up to 4 MB as
it is produced, and `storeBackupBlob` writes them into a temporary table
on its own transaction and assembles the row with one `string_agg`. The
process never holds the whole copy. The account's previous copy stays in
place until the new one is complete: a failure rolls back to it.

Measured on the seeded account in a 1 GB container, web and worker in one
process: the backup completes in about 30 s with the dense retention
running beside it, where v1.39.0 crashed within 30 s.

The streamed JSON also reports its progress to the job every 50 000 rows.
2026-09-25 00:32:05 +02:00