mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 12:32:43 +00:00
b180d09fae021fa5f6dfbe7fb4fe4e330cceedd4
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Marc-André Bombeck
|
26a9efdf52 |
fix(recovery): the database dumps count in the keys the backups still need
The recovery kit, "Keys the backups still need" and safeToDropRetiredKeys read only the stored copies and the off-host account objects. After a rotation the page called the old key safe to drop while every dump taken before it, the pre-migration dumps that are the way back from an update among them, could only be restored with that key: db-restore refuses a dump whose active key is missing. A dump now counts for the key its rows were written under, the manifest's active key: a dump on this server while it is kept (read from the manifests), an off-host one until the bucket's rule for database/ expires it, or indefinitely when that rule cannot be read. The nightly run records the key on its ledger row (migration 0489, database_backup_runs.active_key_id). The encryption view and the kit list the dumps per key, in every locale, and the rotation and backup runbooks say so. |
||
|
Marc-André Bombeck
|
792fc89a93 |
fix(auth): count a passkey as the second factor on the managed-profile acts
An account whose only strong credential is a primary passkey was refused auth.stepup.mfa_not_enrolled on every action behind requireFreshMfa, so it could not create, edit, hand over or delete a managed profile or manage its guardians, although a passkey sign-in or re-proof already stamps the session the gate reads. requireFreshMfa now defaults to the any-possession proof source: a passkey makes the gate reachable, and a passkey re-proof through the dialog clears it. The refusal offers passkey in meta.methods, never the password. hasSecondFactorEnrolled keeps its narrow meaning on purpose. It decides whether the conditional gates apply at all (account deletion, the data reset, the password change, the export, offering manage access, the recent-proof branches and the native Bearer rules), so none of those change, and password sign-in still challenges only TOTP and security keys. The MFA-management routes keep the second-factor source. The decision and its limits are written beside FreshFactorProofSource. Copy, the OpenAPI description and the sharing and key-rotation docs say "second factor or passkey" where this gate applies. Refs #959 |
||
|
Marc-André Bombeck
|
2c295bbea8 |
Probe the oldest stored values before recording an encryption key
The boot key check recorded whatever key was configured whenever its probe found nothing to try. It only read string columns, so an install whose ciphertext sat in Bytes columns (notes, documents, ECG, symptoms) read as empty; a probe that ran out of time did the same; and rows written by a process that served under a wrong key could open on the next boot and get that key recorded as the right one. The probe now reads every registered column, string and Bytes, in all four stored layouts, and tries the oldest values first, ordered by the row's updatedAt (else createdAt, else primary key). A key is recorded only when the oldest value that opens is preceded by at most one that does not, or when no column holds a single value under the key id. A time-out, an unreadable table, oversized values or mixed results are inconclusive: the process serves, records nothing and checks again at the next start. When the first insert loses a race, the stored record is read back and must open under this process's key, else the outcome is a mismatch. A rotation that walked every column without errors removes the record of each previous key id that no longer holds any value. The refusal log block and the self-hosting pages explain how to reset the check for a key id. |
||
|
Marc-André Bombeck
|
4409529c89 |
Merge branch 'fix/backup-rotation-and-deletion' into release/v1.39.3
# Conflicts: # docs/api/openapi.yaml # src/lib/jobs/apple-health-import-worker.ts |
||
|
Marc-André Bombeck
|
7da778760e |
fix(backup): leave the admin wipe's off-host copies in place, and correct the backup and rotation runbooks
The admin wipe of every account is the operator clearing their own host, and the bucket's copies are the documented way back from a wipe pressed by mistake, so it writes no off-host deletion request. Deleting an account, deleting a managed profile and "Delete all data" still do. The backup runbook no longer says the worker never deletes an object: it lists the grant the worker needs (DeleteObject included), when objects are deleted, the version-4 envelope, the lifecycle check and how to rotate the off-host key. The rotation runbook no longer says backups are covered: rotation reaches a backup's envelope, not the content a restore writes back, and a retired key stays needed until no backup lists it. The privacy notice beside account deletion says the same about off-host copies. A failed import records its failure in a plain try block, and the extraction test that inflates past the 64 MiB floor gets the time it needs on a loaded runner. |
||
|
Marc-André Bombeck
|
13db7935f6 |
fix(image): ship the operator scripts the runbooks run in the container
The password-reset runbook tells an operator to run `docker compose exec app node scripts/reset-password.mjs`, and the intake-repair runbook runs `healthlog-tsx scripts/repair-intake-anomalies.ts` inside the container. The runner stage copied nothing from scripts/, so both failed with "Cannot find module", the reset at the moment an admin is locked out. The runner stage now copies both scripts and the Argon2 parameter module the reset imports, and proves at build time that each reaches its own first check with every import resolved. A structural test holds every in-container command in docs/ and the Dockerfile copies in step. Key rotation is documented the way it works on the published image: the Rotate now button under Admin > Encryption is the primary path. The rotation script imports the application graph and runs from a source checkout of the same release; the runbook, the admin hint in all seven languages and the decrypt error now say so. |
||
|
Marc-André Bombeck
|
0a54a43799 |
fix(crypto): walk every encrypted column in bounded pages
Key rotation and the rotation status read every column that was not flagged as a blob with one `findMany` over the whole table. `Measurement.notesEncrypted` and `MoodEntry.noteEncrypted` are such columns, so rotation loaded every measurement of the instance: 1.25 million rows on one large account (#1031), and the rotation status page did the same on every view. Every column is now walked with the id-cursor pages the blob columns already used: 25 rows a page for blobs, 5 000 for everything else. The maintenance script's own per-column walks page the same way. On the seeded account with 1 460 notes to rotate, peak heap fell from 272 MB to 153 MB, and it no longer grows with the table. `scanned` now counts the rows that hold ciphertext for every column, as it already did for the blob columns, rather than every row of the table for some columns and only the ciphertext rows for others. The runbook says so. |
||
|
Marc-André Bombeck
|
45d0fe442f |
fix(crypto): rotate the backup blob and stop keying the registry on names
`DataBackup.data` holds every weekly disaster-recovery snapshot and every uploaded pack, AES-256-GCM at rest like everything else, under a column called `data`. The rotation registry was keyed on the `*Encrypted` naming convention, so the script never read a backup, reported zero rows remaining, and the runbook read that zero as permission to retire the previous key. Following the documented procedure destroyed every stored backup — the copy an operator reaches for precisely when nothing else works. Register it, and register the idempotent-replay response cache the same sweep turned up. The backup column walks in bounded batches (one row is a whole compressed account) and re-seals the ciphertext without reading the plaintext, so both stored envelopes come back byte-identical and so will any later one. The cache is marked disposable: a row it cannot read is dropped rather than counted as an error, because keeping it only guarantees the next replay gets a body it cannot parse. Running the script against a seeded database rather than reading it turned up two more. It died on the mood day-context note, whose table is keyed on the entry rather than on an `id`, and abandoned every column after it — the backups included. And the document index's verbatim text rotated without ever being named in the summary. Make the naming assumption stop carrying weight. A new guard traces the encryption helpers through their wrappers, walks every Prisma write payload, and fails when a column receives ciphertext without being registered. It asserts an anchored, non-zero match count so a matcher that stops matching fails instead of passing. A second check pins each model's primary key against the schema. The script now ends by naming the registered columns it did NOT walk and exiting non-zero when any is missing: a zero that means "nothing left" and a zero that means "never looked" must not print the same. The runbook's retire-the-key step was rewritten around that signal, and tells an operator who already rotated on an older release how to get their backups back. |
||
|
Marc-André Bombeck
|
fa7e1ee739 |
Bring the self-hosting guides in line with the shipped code
- geolite2: quote the real commented compose mount (./geolite2 to /opt/geolite2, the GEOLITE2_DIR default, so no env var is needed for the default path) - getting-started: healthcheck probes /api/health, not /api/version; the environment module reaches archive-api and geocoding-api .open-meteo.com, relevant for egress allowlists - mcp: the module is opt-in and off by default, but no longer the only such module - reverse-proxy: the Coach stream is SSE over plain HTTP, not WebSockets; note that TRUST_PROXY_HOPS is not on the bundled compose whitelist and must be added by hand - scaling: /api/health healthcheck; worker liveness is in-memory and invisible across a web/worker split; pool sizing now goes through getConnectionBudget with DB_CONNECTION_LIMIT as the compose knob and a 2-connection pg-boss share - sso: the native SSO handoff shipped, OIDC_ONLY works with app builds that carry it - unraid: geo databases are baked in only when the build had a licence key, or mounted; replaceable either way |
||
|
Marc-André Bombeck
|
c986efab30 |
Correct operator runbooks against the current code
- attempt-number-semantics: name the real bucket column
(consecutiveFailuresByKind) and drop a drifted line citation
- backup-restore: show the smoke-test response inside the standard
{ data, error } envelope
- deploy: the manual migration step now uses the bundled Prisma CLI
via node (the image strips pnpm); WHOOP vars are already on the
shipped compose whitelist
- encryption-key-rotation: point at the encrypted-column registry
instead of a five-model list several releases behind
- env-check: the CI gate shipped (env-check.yml); the partial-APNs
example now shows the synthetic all-or-none row and the non-zero exit
- password-reset: the admin API reset route exists; only the
self-service route does not
- tls-cert-pin: stop asserting client-repo pin-set details that cannot
be audited from this repository
- drop v141-followup-issues.md: a v1.4.1 punch list whose items are
fixed, removed, or superseded; keeping it under docs/ops invites
re-fixing shipped work
|
||
|
Marc-André Bombeck
|
a4feabec81 |
chore(release): v1.28.30 — the daily briefing stops silently skipping days
The recurring no-briefing day had a chain of causes, closed end to end: Every failure path in the nightly pregenerate + the comprehensive generator now annotates a queryable cause (several classes — features-error, payload-too-large, aborted, generator-failed — previously bumped a counter with zero trace). A failed nightly warm enqueues one bounded retry ~45 minutes later (singleton per user, existing budgets apply), so a transient provider hiccup heals before morning. A cached payload that carries NO briefing no longer satisfies the 24h POST short-circuit nor the unchanged-content hash gate — a once-stripped briefing could previously block its own regeneration indefinitely. The card surfaces the omission reason persistently (grounding strip, generation failure, not attempted) with a forcing retry. Documentation corrected against code truth: Withings credentials are per-user (the documented env path never existed; the dead example entries and manifest rows are gone, and WITHINGS_WEBHOOK_SECRET is now actually whitelisted in compose — stock deploys could never deliver it); the classic Fitbit portal no longer accepts new registrations; the backup-restore example no longer mints a fresh encryption key and uses pnpm dlx tsx; the TLS guide describes the real client model (two CA-level SPKI pins, any-match, leaf renewals need no build); scaling/admin references fixed; the operator-shared subscription provider is documented. |
||
|
Marc-André BombeckandMarc-André Bombeck
|
d7778c1085 |
feat(v1.4): multi-tenant prep — off-host backup, encryption-key versioning, worker/web split, native-Bearer UA gating (#135)
* feat(crypto): G2 — encryption-key versioning + rotation CLI
Adds backwards-compatible support for ENCRYPTION_KEYS (JSON map of
{ "id": "<key>" }) plus ENCRYPTION_ACTIVE_KEY_ID. Ciphertext format is now
"<keyId>.<base64payload>"; legacy bare-base64 ciphertext still decrypts
under the synthetic "v1" id derived from the existing ENCRYPTION_KEY.
scripts/rotate-encryption-key.ts re-encrypts every encrypted column on
User / WithingsConnection / AppSettings with the active key. Idempotent —
rows already on the active id are skipped.
Why: enables zero-downtime rotation when a key is suspected leaked,
without forcing every encrypted row through a single big-bang rewrite.
Co-Authored-By: Marc-André Bombeck <noreply@anthropic.com>
* feat(auth): G4 — native-Bearer UA gating + rotating refresh tokens
Web callers (UA contains 'Mozilla/') keep the legacy 90d Bearer when they
explicitly opt in via X-Client-Type: native. Native callers (HealthLog-iOS,
HealthLog-iPad, n8n, Health-Connect, unrecognised UAs) now receive a 24h
access token plus a 60d rotating refresh token in the login response.
New /api/auth/refresh endpoint atomically validates + rotates the refresh
token (one-time-use), revokes the previously paired access token, and
audit-logs every refresh + revoke. Reuse of an already-consumed refresh
token revokes the entire family for the user/device — defence against a
stolen refresh token.
Migration 0025_refresh_tokens adds the table; the only DDL is CREATE
TABLE so it is forward-compatible. Down migration documented at
docs/ops/migrations/0025-down.sql.
Why: the V3 audit (still-V2-A) flagged that 90d tokens are too long for
native callers when refresh-token rotation is now table-stakes for mobile
apps. Web flows are unchanged so existing browser sessions keep working.
Co-Authored-By: Marc-André Bombeck <noreply@anthropic.com>
* feat(infra): G3 — web/worker container split via HEALTHLOG_PROCESS_TYPE
Adds a tiny startup gate (src/lib/process-type.ts) so the same Docker image
can run as web-only, worker-only, or 'all' (default — single-container,
backwards compatible). assertSubsystemEnabled() refuses to boot the
reminder worker when HEALTHLOG_PROCESS_TYPE=web, preventing accidental
double-cron when an operator splits the deployment.
docker-compose.yml ships the optional app-worker service block commented
out so 'docker compose up' keeps working out of the box; uncommenting it
+ flipping app to HEALTHLOG_PROCESS_TYPE=web is the only change to scale
HTTP horizontally. Healthchecks are isolated per service so neither
startup deadlocks the other; both wait on db: service_healthy.
docs/self-hosting/scaling.md walks through when + how to split.
Co-Authored-By: Marc-André Bombeck <noreply@anthropic.com>
* feat(jobs): G1 — daily off-host encrypted backup to S3-compatible storage
New pg-boss queue 'data-backup-offhost' runs at 02:30 Europe/Berlin, dumps
each user's measurements + medications + intake events + mood entries,
encrypts the JSON under a SEPARATE key (BACKUP_ENCRYPTION_KEY, isolated
from the application ENCRYPTION_KEY), and uploads to any S3-compatible
target via @aws-sdk/client-s3.
Wire format documented in docs/ops/backup-restore.md:
magic "HLBK" || version 0x01 || iv(12) || authTag(16) || ciphertext
The worker prunes objects older than BACKUP_RETENTION_DAYS (default 30);
operators are encouraged to also configure a bucket-level lifecycle rule.
Admin endpoint POST /api/admin/backup/test issues a 1-byte PUT+GET
roundtrip and returns endpoint + bucket + per-call latency. Credentials
are never returned.
scripts/restore-backup.ts downloads + decrypts a single object so
operators can verify the chain end-to-end without standing up a separate
restore harness.
Why: GDPR Art. 32 + V3 audit "data durability" — without an off-host copy
the only line of defence against a Postgres volume loss is the in-DB
DataBackup model on the same host. Separate-key isolation means a leak of
the application encryption key does not also expose every user's archive.
Co-Authored-By: Marc-André Bombeck <noreply@anthropic.com>
* fix(multi-tenant): address review findings — rotation coverage, hard decrypt errors, web/worker gates
Multi-agent review on PR #135 caught several CRITICAL + HIGH issues that
would have shipped silent data corruption or weakened the threat model:
Encryption + rotation
- scripts/rotate-encryption-key.ts: stop selecting `bugReportTokenEncrypted`
(column doesn't exist in the schema; the script crashed on the AppSettings
step). Add the three encrypted columns it was missing —
NotificationChannel.config, PushSubscription.p256dh, PushSubscription.auth
— so rotating to a new key actually re-encrypts every encrypted row.
Without this, dropping the old key id from ENCRYPTION_KEYS would leave
notification channels and web-push subscriptions permanently
undecryptable.
- src/lib/crypto.ts: refuse to silently fall back to the active key when a
legacy bare-base64 row is decrypted but no v1 entry is configured. Throws
a clear "Restore ENCRYPTION_KEY (or add a v1 entry to ENCRYPTION_KEYS)
and run scripts/rotate-encryption-key.ts before removing it" so an
operator who pulled the v1 too early notices immediately instead of
hitting opaque GCM tag errors.
- docs/ops/encryption-key-rotation.md: full step-by-step + rollback note +
troubleshooting section. Spells out that rotation is a forward-only
operation and what to do if the rollback path is needed.
Web / worker split (G3)
- src/proxy.ts: refuse HTTP traffic with a 503 +
X-HealthLog-Process-Type: worker header when HEALTHLOG_PROCESS_TYPE=worker.
The worker container (which still runs `node server.js` for now) no
longer accepts requests, so a misrouted client gets a clear hint instead
of a partially-served response.
- src/instrumentation.ts: short-circuit before importing the reminder
worker when HEALTHLOG_PROCESS_TYPE=web — avoids the noisy
"[instrumentation] CRITICAL" stderr line we emitted on every web boot.
- /api/version added to PUBLIC_PATHS so the compose healthcheck doesn't
silently 302-redirect through the auth gate (the prior wget treated the
login page as a 200 success and would never notice an unhealthy backend).
Refresh-token rotation (G4)
- src/lib/auth/refresh-token.ts: reuse-detection now revokes ALL active
refresh tokens for the user, not only those scoped to the same deviceId.
An attacker who rotated without an X-Device-Id header would otherwise
keep their family alive on the no-device branch. Defence-in-depth.
- revokeRefreshToken() also revokes the paired access token so a leaked
access token cannot outlive the refresh-token sibling that the caller
just killed (logout-on-device).
Off-host backup (G1)
- src/lib/jobs/offhost-backup.ts: drop the worker-side DeleteObject path.
Retention is the bucket's lifecycle-policy job; the worker keeps a
PutObject + GetObject grant only. A compromised worker can no longer
wipe the backup history. Per-user errors are now surfaced as Wide
Event warnings + a structured failures[] array on the report so an
operator can tell which user failed and why.
- scripts/restore-backup.ts: schema-validate the decrypted payload before
writing to disk, plus an optional --user-id=<id> flag that refuses to
proceed unless the dump's userId matches. Prevents silent
cross-tenant restores when an operator points the script at the wrong
S3 key.
Idempotency
- src/lib/idempotency.ts: the leak-guard already blocked `hlk_`
(access-token prefix); also block `hlr_` so a future caller wrapping
the new /api/auth/refresh endpoint with withIdempotency() can't
accidentally cache a freshly-issued refresh token.
Migration / schema
- prisma/migrations/0025_refresh_tokens/migration.sql: wrap in a single
transaction; switch to CREATE { TABLE | INDEX } IF NOT EXISTS so a
partial-deploy retry is a no-op instead of "relation already exists";
guard the FK ADD with information_schema lookup for the same reason.
- Schema replaces the single-column `@@index([deviceId])` with a
composite `@@index([userId, deviceId])` (every lookup also filters on
userId; the old index would never have been used). Comment on
accessTokenHash corrected to HMAC-SHA-256.
Container / ops
- Dockerfile: install tzdata + ENV TZ=Europe/Berlin so the daily
pg-boss schedule (`30 2 * * *` Europe/Berlin) actually fires at 02:30
Berlin instead of silently falling back to UTC on the alpine base.
- docker-compose.yml: tighten the healthcheck to `wget --spider` (matches
the Dockerfile HEALTHCHECK style, doesn't pipe the body), pass through
BACKUP_RETENTION_DAYS so a custom retention env reaches the container.
- .env.example: document every new env var added in this PR
(HEALTHLOG_PROCESS_TYPE, ENCRYPTION_KEYS, ENCRYPTION_ACTIVE_KEY_ID,
BACKUP_S3_*, BACKUP_ENCRYPTION_KEY, BACKUP_RETENTION_DAYS) with
comments and the all-or-none semantics for the backup block.
491 tests still green, typecheck clean.
* chore: retrigger CI
---------
Co-authored-by: Marc-André Bombeck <noreply@anthropic.com>
|