mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 04:32:42 +00:00
a7faae8ded7e11ebd71aeedf1f0349ba767ea5e5
16
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Marc-André Bombeck
|
25f0310596 |
feat(backups): tell an empty ledger apart from an empty bucket
On the release that adds the ledger the table is empty, so on a host that has been uploading nightly for months every account read Never, under a sentence saying no copy had ever reached the bucket. That sentence is untrue on exactly the morning an operator opens the new card for the first time, and it is a claim the host cannot make: what is empty is the record, not the bucket. The run now records every account it walks, not only the ones an object came of. A walk that produced nothing moves the attempt instant and leaves the success columns alone, so the two cases separate: no row at all is no record yet, a row without a success is never. Never now means what the word says — a run looked at this account and has never put anything there. The card names the last check on the never row, so the state is a fact with a timestamp rather than an assertion. Runbook and all seven bundles carry the same two sentences. The integration file proves both arms against real Postgres, including that a failed run leaves the previous good copy on the row. |
||
|
Marc-André Bombeck
|
cfc19a211c |
fix(backups): give the off-host verdicts slack for a Berlin night that is 25 hours long
The threshold was the schedule period with no budget on top of it. Two things spend that budget on a host where nothing is wrong. The cron runs 02:30 Europe/Berlin, so on the DST fall-back Sunday the gap between two runs is 25 hours and every account on every host would cross one period on the same morning. And the ledger instant is when that account's object landed, by design, so on a cohort walked one account at a time an account reached an hour later tonight than last night drifts past 24 with two clean runs behind it. The schedule period and the verdict threshold are now separate numbers. The period is still what the card quotes; the threshold is the period plus six hours, applied to both steps, so due still means one run produced nothing and stale still means two. The fall-back night is pinned in the unit file by its real dates rather than by an arithmetic 25, and a genuinely missed night at 31 hours still reads due. The runbook says why the slack is there. |
||
|
Marc-André Bombeck
|
1c10bbbde8 |
feat(admin): say per account how fresh the off-host copy is
The nightly off-host job reports counts — uploaded, failed, total — and a count cannot name the account it is failing for. A cohort where one object has been refused every night for a month reads as "99 uploaded, 1 failed" each time, which looks like weather rather than like an account with no disaster-recovery copy at all. The worker now writes a ledger row when an object lands (account, instant, size), and Admin → Backups gains a second card that rates every account against the nightly schedule: fresh inside one run, due when one run produced nothing, stale when two did not, never when no object has ever landed. On a host without BACKUP_S3_* the card says off-host backup is not configured and stops there. The verdicts are computed from the ledger, never from a bucket listing: the worker's grant is deliberately PutObject + GetObject, and an admin page must not put the operator's credentials on its render path to state a number the database already holds. |
||
|
Marc-André Bombeck
|
6e4f3fb4c4 |
docs(ops): what a restore replaces, and the console path
Two gaps in `backup-restore.md`. It said the restore "replaces the account's data tables" only in a paragraph about grants, so nothing stated the consequence for the account's own record: the rows it gained after the snapshot was taken are deleted with the rest of their class. And the console restore was named once in passing — `## Restore` documents only the off-host script — so an operator restoring from the browser had no procedure to read. Both are written down now, including which button does what, that the restore targets the snapshot's owner rather than the admin running it, that instance-wide settings ride along only when the dialog's opt-in is ticked, and what a copy this instance can no longer decrypt answers. |
||
|
Marc-André Bombeck
|
7f54f339b2 |
refactor(backup): drop the retention value nothing could ever read
E2. `loadOffhostConfig` parsed BACKUP_RETENTION_DAYS, clamped it to a minimum of one day and put it on the config object, where no caller read it. Neither annotation fits: it is not deliberately internal, because there is no internal use, and no consumer is pending. The design forbids one outright — the worker's grant excludes DeleteObject on purpose, so nothing here can act on a retention window without giving a compromised worker the ability to wipe the history. A clamp that guards nothing only read as an enforcer the worker is not. So the field and the clamp go, and the reason sits where the parse was. The variable keeps its place in the two example files and on the compose whitelist: it is the number the operator sets the bucket's own lifecycle rule to. The docs table said "defaults to 30", which implied the app had an opinion; it now says the app does not read it. |
||
|
Marc-André Bombeck
|
8ad5b90f88 |
docs(ops): the off-host worker never prunes; the bucket lifecycle does
The runbook said the worker prunes objects older than BACKUP_RETENTION_DAYS. It never calls DeleteObject; its grant is PutObject, GetObject and AbortMultipartUpload, on purpose, so a compromised worker cannot wipe the history. The sentence now says what expires objects and what the variable is for. |
||
|
Marc-André Bombeck
|
13a9aacf20 |
fix(backup): stream the nightly off-host upload instead of building it
The off-host job was said to benefit from the streaming work the weekly in-database backup got, and it did not. Only the payload builder was shared: `runOffhostBackup` called `buildFullBackupJson` for the whole document, `gzipSync` over that whole string, a whole-buffer `createCipheriv` pass over the result, and handed the finished buffer to a single `PutObject`. Four full copies of the record alive at once, on the same arithmetic the weekly pass had already been fixed for. Configuring an S3 target on a long-lived record therefore restarted the container. On an account of 445 000 measurements the JSON alone is 242 MB, and against a 1 GB container's 524 MB heap the run died of heap exhaustion seventeen seconds in — taking every signed-in session with it, because the job shares the app process. The write path is a stream end to end now: `streamFullBackupJson` produces the document a page at a time, gzip and an incremental AES-256-GCM writer consume it as it arrives, and `@aws-sdk/lib-storage` puts it up in two 8 MB parts. What the process holds is fixed by the pipeline's shape rather than by the record. Measured on that same 445 000-measurement account under `--max-old-space-size=450`, with the heap pre-loaded to what a warm server holds: the old path dies with `Ineffective mark-compacts near heap limit`, the new one finishes with a peak 79 MB above its baseline and writes a 9.1 MB object. The stored envelope gains version 3, which moves the GCM tag from in front of the ciphertext to the end. That is the only change the stream required — the tag exists only once the last block is in, so a leading one means the whole object must exist before its first byte can be sent. Nothing about the authentication moves: the tag still covers every ciphertext byte and is verified before a byte of plaintext comes back. Versions 1 and 2 still read, so every object already in a bucket restores through the same reader with no flag to tell them apart. The incremental writer is the crypto module's, split into its byte-level half so the base64 form the in-database blob uses and the raw form an object needs are one implementation under two framings. The off-host key handling is unchanged: a separate `BACKUP_ENCRYPTION_KEY`, 64 hex or 32-byte base64, passed in rather than looked up. A run that uploaded nothing for anybody is a failed job now, not `ok: true` with an empty bucket behind it. Wrong credentials, a missing bucket and an unreachable endpoint all fail every account rather than one, and the target's own sentence rides out as the cause. A run where some account got a copy still succeeds; retrying the whole cohort over one object would re-upload everybody's. The one ceiling left is structural rather than a memory bound, and it is counted rather than discovered: a multipart upload carries 10 000 parts, so an object past 80 GB is refused for that account with a clear message instead of failing halfway through with an SDK error about part numbers. `tests/integration/offhost-backup-streaming-memory.test.ts` pins the budget from both sides — what the streaming uploader holds, and that the materialising path does not fit the same budget on the same fixture in the same process, so the number cannot pass by measuring nothing. |
||
|
Marc-André Bombeck
|
24f3afd464 |
fix(backup): bound the stored copy, not the process heap
The writer added in v1.38.6 compared the process's live heap usage against 80 % of V8's heap limit and aborted above it. That reading is not the backup's footprint; it is the whole process's, garbage included. A long-running app sits at 400 MB of largely collectable heap, so on a 1 GB container (a 419 MB budget) the weekly pass aborted the first flush of every account: four out of four in seven seconds, the smallest of them a demo record whose entire stored copy is 1.2 MB, each told its record was too large and its operator to buy memory. Drop the gauge. What the writer holds is bounded by its shape and needs no reading to prove: the three unbounded tables are pulled a page at a time, every other section is released as soon as its JSON is in the sink, and at most one flush of text is pending. The one copy that does grow with the record is the stored blob, so the bound moves there — counted in the ciphertext bytes actually produced, against a fifth of this process's heap limit, which is what two copies of the answer plus a live set have to fit inside. The message names both numbers. A pass that wrote nothing for anybody now fails its job instead of reporting `ok: true` with `backed: 0`, so the failing-queue panel and the backups page read the same thing an operator would. A pass where some accounts were written keeps passing with the failures as counts. Measured under --max-old-space-size=450 from a heap deliberately dirtied first: 445 000 measurements peak at 28 MB held by the backup, a one-row record at under 1 MB. |
||
|
Marc-André Bombeck
|
3118cf240c |
fix(backup): stream the weekly backup instead of building it whole
The compression that landed last release was real and was measured in the wrong place. On a development machine with a multi-gigabyte heap the pass finished; the container the job runs in is capped at 1 GB, which gives V8 a 524 MB limit, and there the same pass died of memory about fifteen seconds in. Because the job shares the application process, that was not one failed backup — it was a restart, and every signed-in session on the host went with it. Reproduced before touching anything, under `--max-old-space-size=450` against a seeded account of 445 000 measurements, 30 000 mood entries and 60 000 intake events: `FATAL ERROR: Reached heap limit` at 34 s. The same fixture doubled to 890 000 measurements now completes inside a 296 MB limit, holding 16 MB of live heap where the old writer held 168 MB. The document is written incrementally now. `buildFullBackupPayload` takes a `deferBulk` mode that declares the three unbounded tables instead of reading them, and `streamFullBackupJson` walks the payload in key order, pulls those tables through page by page, and releases every other section as soon as its JSON reaches the sink. `packBackupBlobStreaming` feeds that into gzip and an AES-256-GCM stream, so nothing between the database rows and the stored value ever exists whole. Both writers go through the same per-row serialisers, and the round-trip suite pins their output byte for byte rather than trusting that they agree. The intake and mood reads are ordered totally now. A keyset cursor needs it, and a partial order also let two exports of one account differ over a tie, which every multi-dose day has. The streamed ciphertext is a new shape at rest, because GCM only produces its tag once the last block is in and a leading tag is what makes the old layout unstreamable. It carries a marker no older writer can produce, so the two families are disjoint by construction rather than by sniffing. All three stored shapes — plain, compressed, streamed — restore, proved both ways. The writer also watches its own live heap and stops at 80 % of V8's limit. An account too large for its host is now one failed backup, counted in the run's meta, instead of a restart for everybody on it. The stored column is the ceiling that is left: `data_backups.data` is a single `text` value, so the finished artifact has to exist as one string, and that is the only part of the job still growing with the record. The operator runbook says so, with the number at which it starts to matter. |
||
|
Marc-André Bombeck
|
c986efab30 |
Correct operator runbooks against the current code
- attempt-number-semantics: name the real bucket column
(consecutiveFailuresByKind) and drop a drifted line citation
- backup-restore: show the smoke-test response inside the standard
{ data, error } envelope
- deploy: the manual migration step now uses the bundled Prisma CLI
via node (the image strips pnpm); WHOOP vars are already on the
shipped compose whitelist
- encryption-key-rotation: point at the encrypted-column registry
instead of a five-model list several releases behind
- env-check: the CI gate shipped (env-check.yml); the partial-APNs
example now shows the synthetic all-or-none row and the non-zero exit
- password-reset: the admin API reset route exists; only the
self-service route does not
- tls-cert-pin: stop asserting client-repo pin-set details that cannot
be audited from this repository
- drop v141-followup-issues.md: a v1.4.1 punch list whose items are
fixed, removed, or superseded; keeping it under docs/ops invites
re-fixing shipped work
|
||
|
Marc-André Bombeck
|
c82ff767a3 |
Say in the runbook what a backup does not carry
Every credential-shaped row is left out on purpose — API tokens, trusted devices, step-up elevations, known devices, clinician share links, and the grants behind shared record access. The per-model reasoning already lives in backup-plan.ts with a structural test behind it, but an operator planning a recovery does not read that file. The consequence differs by target and the difference is the part worth knowing. Restoring onto the same instance changes nothing: the restore replaces data tables and does not touch grants, so shared access carries on as it was. Restoring onto a fresh instance brings none of it, so both people invite and accept again before sharing works. That is the fail-safe direction, and it is better planned for than discovered. |
||
|
Marc-André Bombeck
|
085aa3f5c5 | fix: harden pnpm supply chain and runtime image (#572) | ||
|
Marc-André Bombeck
|
a4feabec81 |
chore(release): v1.28.30 — the daily briefing stops silently skipping days
The recurring no-briefing day had a chain of causes, closed end to end: Every failure path in the nightly pregenerate + the comprehensive generator now annotates a queryable cause (several classes — features-error, payload-too-large, aborted, generator-failed — previously bumped a counter with zero trace). A failed nightly warm enqueues one bounded retry ~45 minutes later (singleton per user, existing budgets apply), so a transient provider hiccup heals before morning. A cached payload that carries NO briefing no longer satisfies the 24h POST short-circuit nor the unchanged-content hash gate — a once-stripped briefing could previously block its own regeneration indefinitely. The card surfaces the omission reason persistently (grounding strip, generation failure, not attempted) with a forcing retry. Documentation corrected against code truth: Withings credentials are per-user (the documented env path never existed; the dead example entries and manifest rows are gone, and WITHINGS_WEBHOOK_SECRET is now actually whitelisted in compose — stock deploys could never deliver it); the classic Fitbit portal no longer accepts new registrations; the backup-restore example no longer mints a fresh encryption key and uses pnpm dlx tsx; the TLS guide describes the real client model (two CA-level SPKI pins, any-match, leaf renewals need no build); scaling/admin references fixed; the operator-shared subscription provider is documented. |
||
|
Marc-André Bombeck
|
ce3b4c70c8 |
chore(ops): signed images, a secret gate before commit, and a monthly restore drill
Published manifests are now cosign-signed keyless with a documented digest-pinning path, a lefthook pre-commit runs gitleaks over staged changes, responses opt into a credentialless embedder policy, the deploy webhook can require a fresh timestamp, and a monthly drill decrypts and parses the newest offhost backup so a broken key or format surfaces before an emergency does. |
||
|
Marc-André Bombeck
|
4f6d2ccbeb |
docs: drop per-release audit notes + legacy spec, scrub phase markers
- /docs/audit/ moved out of the public docs surface (23 per-release forensic notes). They were linked from the README but read as internal traceability artefacts, not user-facing documentation. Kept local for cross-release lookup; added to .gitignore so new ones don't re-track. - /docs/api/openapi-v1422-legacy.yaml removed — superseded by the current openapi.yaml on the v1.5 line. - /docs/api/openapi.yaml info.version bumped 1.4.23 → 1.5.0 so the spec matches package.json (the doc claims the version is mirrored from the package; previously the mirror had drifted six versions). - /docs/ops/deploy.md: image name corrected from `ghcr.io/marc-bombeck/healthlog` → `ghcr.io/mbombeck/healthlog` (the previous command would have 404'd at GHCR); release-process framing dropped from a release-pair note. - /docs/ops/backup-restore.md + /docs/self-hosting/scaling.md: trailing phase tags (`v1.4 G1`, `v1.4 G3`, `v1.4.40 W-POOL`) stripped from section headers + body. Replaced an internal-notes pointer with a concrete `getPoolMax()` reference. The README pointer to /docs/audit/ is removed in the next commit along with the rest of the README accuracy pass. |
||
|
Marc-André BombeckandMarc-André Bombeck
|
d7778c1085 |
feat(v1.4): multi-tenant prep — off-host backup, encryption-key versioning, worker/web split, native-Bearer UA gating (#135)
* feat(crypto): G2 — encryption-key versioning + rotation CLI
Adds backwards-compatible support for ENCRYPTION_KEYS (JSON map of
{ "id": "<key>" }) plus ENCRYPTION_ACTIVE_KEY_ID. Ciphertext format is now
"<keyId>.<base64payload>"; legacy bare-base64 ciphertext still decrypts
under the synthetic "v1" id derived from the existing ENCRYPTION_KEY.
scripts/rotate-encryption-key.ts re-encrypts every encrypted column on
User / WithingsConnection / AppSettings with the active key. Idempotent —
rows already on the active id are skipped.
Why: enables zero-downtime rotation when a key is suspected leaked,
without forcing every encrypted row through a single big-bang rewrite.
Co-Authored-By: Marc-André Bombeck <noreply@anthropic.com>
* feat(auth): G4 — native-Bearer UA gating + rotating refresh tokens
Web callers (UA contains 'Mozilla/') keep the legacy 90d Bearer when they
explicitly opt in via X-Client-Type: native. Native callers (HealthLog-iOS,
HealthLog-iPad, n8n, Health-Connect, unrecognised UAs) now receive a 24h
access token plus a 60d rotating refresh token in the login response.
New /api/auth/refresh endpoint atomically validates + rotates the refresh
token (one-time-use), revokes the previously paired access token, and
audit-logs every refresh + revoke. Reuse of an already-consumed refresh
token revokes the entire family for the user/device — defence against a
stolen refresh token.
Migration 0025_refresh_tokens adds the table; the only DDL is CREATE
TABLE so it is forward-compatible. Down migration documented at
docs/ops/migrations/0025-down.sql.
Why: the V3 audit (still-V2-A) flagged that 90d tokens are too long for
native callers when refresh-token rotation is now table-stakes for mobile
apps. Web flows are unchanged so existing browser sessions keep working.
Co-Authored-By: Marc-André Bombeck <noreply@anthropic.com>
* feat(infra): G3 — web/worker container split via HEALTHLOG_PROCESS_TYPE
Adds a tiny startup gate (src/lib/process-type.ts) so the same Docker image
can run as web-only, worker-only, or 'all' (default — single-container,
backwards compatible). assertSubsystemEnabled() refuses to boot the
reminder worker when HEALTHLOG_PROCESS_TYPE=web, preventing accidental
double-cron when an operator splits the deployment.
docker-compose.yml ships the optional app-worker service block commented
out so 'docker compose up' keeps working out of the box; uncommenting it
+ flipping app to HEALTHLOG_PROCESS_TYPE=web is the only change to scale
HTTP horizontally. Healthchecks are isolated per service so neither
startup deadlocks the other; both wait on db: service_healthy.
docs/self-hosting/scaling.md walks through when + how to split.
Co-Authored-By: Marc-André Bombeck <noreply@anthropic.com>
* feat(jobs): G1 — daily off-host encrypted backup to S3-compatible storage
New pg-boss queue 'data-backup-offhost' runs at 02:30 Europe/Berlin, dumps
each user's measurements + medications + intake events + mood entries,
encrypts the JSON under a SEPARATE key (BACKUP_ENCRYPTION_KEY, isolated
from the application ENCRYPTION_KEY), and uploads to any S3-compatible
target via @aws-sdk/client-s3.
Wire format documented in docs/ops/backup-restore.md:
magic "HLBK" || version 0x01 || iv(12) || authTag(16) || ciphertext
The worker prunes objects older than BACKUP_RETENTION_DAYS (default 30);
operators are encouraged to also configure a bucket-level lifecycle rule.
Admin endpoint POST /api/admin/backup/test issues a 1-byte PUT+GET
roundtrip and returns endpoint + bucket + per-call latency. Credentials
are never returned.
scripts/restore-backup.ts downloads + decrypts a single object so
operators can verify the chain end-to-end without standing up a separate
restore harness.
Why: GDPR Art. 32 + V3 audit "data durability" — without an off-host copy
the only line of defence against a Postgres volume loss is the in-DB
DataBackup model on the same host. Separate-key isolation means a leak of
the application encryption key does not also expose every user's archive.
Co-Authored-By: Marc-André Bombeck <noreply@anthropic.com>
* fix(multi-tenant): address review findings — rotation coverage, hard decrypt errors, web/worker gates
Multi-agent review on PR #135 caught several CRITICAL + HIGH issues that
would have shipped silent data corruption or weakened the threat model:
Encryption + rotation
- scripts/rotate-encryption-key.ts: stop selecting `bugReportTokenEncrypted`
(column doesn't exist in the schema; the script crashed on the AppSettings
step). Add the three encrypted columns it was missing —
NotificationChannel.config, PushSubscription.p256dh, PushSubscription.auth
— so rotating to a new key actually re-encrypts every encrypted row.
Without this, dropping the old key id from ENCRYPTION_KEYS would leave
notification channels and web-push subscriptions permanently
undecryptable.
- src/lib/crypto.ts: refuse to silently fall back to the active key when a
legacy bare-base64 row is decrypted but no v1 entry is configured. Throws
a clear "Restore ENCRYPTION_KEY (or add a v1 entry to ENCRYPTION_KEYS)
and run scripts/rotate-encryption-key.ts before removing it" so an
operator who pulled the v1 too early notices immediately instead of
hitting opaque GCM tag errors.
- docs/ops/encryption-key-rotation.md: full step-by-step + rollback note +
troubleshooting section. Spells out that rotation is a forward-only
operation and what to do if the rollback path is needed.
Web / worker split (G3)
- src/proxy.ts: refuse HTTP traffic with a 503 +
X-HealthLog-Process-Type: worker header when HEALTHLOG_PROCESS_TYPE=worker.
The worker container (which still runs `node server.js` for now) no
longer accepts requests, so a misrouted client gets a clear hint instead
of a partially-served response.
- src/instrumentation.ts: short-circuit before importing the reminder
worker when HEALTHLOG_PROCESS_TYPE=web — avoids the noisy
"[instrumentation] CRITICAL" stderr line we emitted on every web boot.
- /api/version added to PUBLIC_PATHS so the compose healthcheck doesn't
silently 302-redirect through the auth gate (the prior wget treated the
login page as a 200 success and would never notice an unhealthy backend).
Refresh-token rotation (G4)
- src/lib/auth/refresh-token.ts: reuse-detection now revokes ALL active
refresh tokens for the user, not only those scoped to the same deviceId.
An attacker who rotated without an X-Device-Id header would otherwise
keep their family alive on the no-device branch. Defence-in-depth.
- revokeRefreshToken() also revokes the paired access token so a leaked
access token cannot outlive the refresh-token sibling that the caller
just killed (logout-on-device).
Off-host backup (G1)
- src/lib/jobs/offhost-backup.ts: drop the worker-side DeleteObject path.
Retention is the bucket's lifecycle-policy job; the worker keeps a
PutObject + GetObject grant only. A compromised worker can no longer
wipe the backup history. Per-user errors are now surfaced as Wide
Event warnings + a structured failures[] array on the report so an
operator can tell which user failed and why.
- scripts/restore-backup.ts: schema-validate the decrypted payload before
writing to disk, plus an optional --user-id=<id> flag that refuses to
proceed unless the dump's userId matches. Prevents silent
cross-tenant restores when an operator points the script at the wrong
S3 key.
Idempotency
- src/lib/idempotency.ts: the leak-guard already blocked `hlk_`
(access-token prefix); also block `hlr_` so a future caller wrapping
the new /api/auth/refresh endpoint with withIdempotency() can't
accidentally cache a freshly-issued refresh token.
Migration / schema
- prisma/migrations/0025_refresh_tokens/migration.sql: wrap in a single
transaction; switch to CREATE { TABLE | INDEX } IF NOT EXISTS so a
partial-deploy retry is a no-op instead of "relation already exists";
guard the FK ADD with information_schema lookup for the same reason.
- Schema replaces the single-column `@@index([deviceId])` with a
composite `@@index([userId, deviceId])` (every lookup also filters on
userId; the old index would never have been used). Comment on
accessTokenHash corrected to HMAC-SHA-256.
Container / ops
- Dockerfile: install tzdata + ENV TZ=Europe/Berlin so the daily
pg-boss schedule (`30 2 * * *` Europe/Berlin) actually fires at 02:30
Berlin instead of silently falling back to UTC on the alpine base.
- docker-compose.yml: tighten the healthcheck to `wget --spider` (matches
the Dockerfile HEALTHCHECK style, doesn't pipe the body), pass through
BACKUP_RETENTION_DAYS so a custom retention env reaches the container.
- .env.example: document every new env var added in this PR
(HEALTHLOG_PROCESS_TYPE, ENCRYPTION_KEYS, ENCRYPTION_ACTIVE_KEY_ID,
BACKUP_S3_*, BACKUP_ENCRYPTION_KEY, BACKUP_RETENTION_DAYS) with
comments and the all-or-none semantics for the backup block.
491 tests still green, typecheck clean.
* chore: retrigger CI
---------
Co-authored-by: Marc-André Bombeck <noreply@anthropic.com>
|