mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 12:32:43 +00:00
Marc-André Bombeck
13a9aacf20
The off-host job was said to benefit from the streaming work the weekly in-database backup got, and it did not. Only the payload builder was shared: `runOffhostBackup` called `buildFullBackupJson` for the whole document, `gzipSync` over that whole string, a whole-buffer `createCipheriv` pass over the result, and handed the finished buffer to a single `PutObject`. Four full copies of the record alive at once, on the same arithmetic the weekly pass had already been fixed for. Configuring an S3 target on a long-lived record therefore restarted the container. On an account of 445 000 measurements the JSON alone is 242 MB, and against a 1 GB container's 524 MB heap the run died of heap exhaustion seventeen seconds in — taking every signed-in session with it, because the job shares the app process. The write path is a stream end to end now: `streamFullBackupJson` produces the document a page at a time, gzip and an incremental AES-256-GCM writer consume it as it arrives, and `@aws-sdk/lib-storage` puts it up in two 8 MB parts. What the process holds is fixed by the pipeline's shape rather than by the record. Measured on that same 445 000-measurement account under `--max-old-space-size=450`, with the heap pre-loaded to what a warm server holds: the old path dies with `Ineffective mark-compacts near heap limit`, the new one finishes with a peak 79 MB above its baseline and writes a 9.1 MB object. The stored envelope gains version 3, which moves the GCM tag from in front of the ciphertext to the end. That is the only change the stream required — the tag exists only once the last block is in, so a leading one means the whole object must exist before its first byte can be sent. Nothing about the authentication moves: the tag still covers every ciphertext byte and is verified before a byte of plaintext comes back. Versions 1 and 2 still read, so every object already in a bucket restores through the same reader with no flag to tell them apart. The incremental writer is the crypto module's, split into its byte-level half so the base64 form the in-database blob uses and the raw form an object needs are one implementation under two framings. The off-host key handling is unchanged: a separate `BACKUP_ENCRYPTION_KEY`, 64 hex or 32-byte base64, passed in rather than looked up. A run that uploaded nothing for anybody is a failed job now, not `ok: true` with an empty bucket behind it. Wrong credentials, a missing bucket and an unreachable endpoint all fail every account rather than one, and the target's own sentence rides out as the cause. A run where some account got a copy still succeeds; retrying the whole cohort over one object would re-upload everybody's. The one ceiling left is structural rather than a memory bound, and it is counted rather than discovered: a multipart upload carries 10 000 parts, so an object past 80 GB is refused for that account with a clear message instead of failing halfway through with an SDK error about part numbers. `tests/integration/offhost-backup-streaming-memory.test.ts` pins the budget from both sides — what the streaming uploader holds, and that the materialising path does not fit the same budget on the same fixture in the same process, so the number cannot pass by measuring nothing.
docs/ — internal reference + operator playbooks
The user-facing documentation lives at https://docs.healthlog.dev — that's the canonical surface for self-hosting guides, integration walkthroughs, AI-provider setup, and the doctor-report workflow.
The files in this directory are the internal operator playbooks, audit notes, and machine-readable specs that ship alongside the source:
api/— the OpenAPI 3.1 spec for the native-client subset (iOS DTO codegen target). Seeapi/README.mdfor the preview commands. Also home to the MCP-server contract:api/mcp-capabilities.md(tools, resources, prompts, write model) andapi/mcp-skills.md(building a connector / skill).ops/— operator runbooks: deploy, backup / restore, encryption-key rotation, password reset and account recovery, env checks, TLS pinning, data repair. These are what an operator reaches for during an incident.self-hosting/— horizontal-scaling notes (HEALTHLOG_PROCESS_TYPE=web|worker|all) and the deploy-pipeline recipe. Routine install steps stay ondocs.healthlog.dev. Includesself-hosting/mcp.md— enabling the off-by-default MCP connector and pointing Claude.ai / ChatGPT / Claude Desktop at it, andself-hosting/sso.md— OIDC SSO login: IdP setup, the identity-pinning security model,OIDC_ONLYconsequences, and the break-glass runbook. Alsoself-hosting/geolite2.md— bringing your own offline GeoLite2 databases viaGEOLITE2_DIRinstead of sending login IPs to a third-party lookup. Andself-hosting/account-sharing.md— letting one account on the instance open another's health record: the three access levels, the eight sections a grant can be narrowed to, managed profiles for somebody with no login and the guardians who look after them, what revocation does and does not undo, and which trust properties of the instance are unchanged by any of it.migration/— migration notes. Currently only v1.3 to v1.4; later releases are covered byCHANGELOG.mdand https://docs.healthlog.dev.integrations/— per-integration reference (Withings, Fitbit, Google Health, WHOOP, Apple Health, AI providers, data import).adr/— architecture decision records.security/— threat-model notes.diagrams/— architecture diagrams rendered through docs.healthlog.dev.releases/— release imagery.apple-store-connect-checklist.md— the iOS submission gate.codex-protocol-spec.md— Codex / ChatGPT OAuth protocol reverse-engineering notes.ui-guidelines.md— UI / a11y house style.
If you're a self-hoster looking to spin up an instance, start at https://docs.healthlog.dev or jump to the Quick Start in the project README.