mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 20:42:43 +00:00
Marc-André Bombeck
900a5c4f1f
The nightly folds soft-deleted every raw Apple Health sample they folded into an hourly or daily stats row and kept it for 75 days. Nothing ever read those rows as deletions; they only held a re-upload of the sample off, and they were most of the measurements table and its indexes. Ingest now refuses such a re-upload directly: a raw sample of a dense or mean type, old enough to have been folded, whose hour or day a live stats row covers, is a duplicate with reason folded_window. The batch route and the Apple Health export import apply the same rule. With that in place the dense retention and the daily-mean consolidation delete the raw rows outright, and a new compaction-tombstone-purge job removes the backlog: 5,000 rows a batch, one account at a time under the restore lock, a short pause between batches, resumable because the predicate is idempotent, queued at every boot and continuing itself while it makes progress. A person's own deletions are never in its predicate. The sync feed leaves compaction tombstones out and a restore does not write them back; all three classify through the same function. The batch route answers an exact re-sent sample without a transaction: one read over the batch's sample ids, live or deleted, decides what the reconciler would have decided, at the same point in the loop, so every status, reason, counter and the cross-source merge stay as they were. A batch of duplicates only no longer queues PR detection, a reminder sweep or the detection audit row. Caches are evicted only for a foreground or manual sync or a batch that wrote a hand-entered reading, and marked stale otherwise; the targets cache follows the same split. Also: - POST /api/admin/maintenance/measurements queues VACUUM (ANALYZE) and a REINDEX INDEX CONCURRENTLY per index, largest first, on a connection of its own; it refuses while the purge runs, drops invalid leftover copies and stops at the first failed rebuild. Runbook in docs/ops. - The pure volume queues keep their finished jobs for 96 hours instead of seven days, never less than the failure window plus a day. - work_mem is set per app connection from DATABASE_WORK_MEM (16MB), also when the statement timeout is disabled. - The PR-detection fallback tick re-scans only accounts whose readings or workouts changed since the previous tick. - Type discovery for the status refill walks the live-rows index as a loose index scan instead of grouping every live row. - The host metrics chart polls every five minutes, like its sampler. - The batch route records per-type HealthKit arrivals.
docs/ — internal reference + operator playbooks
The user-facing documentation lives at https://docs.healthlog.dev — that's the canonical surface for self-hosting guides, integration walkthroughs, AI-provider setup, and the doctor-report workflow.
The files in this directory are the internal operator playbooks, audit notes, and machine-readable specs that ship alongside the source:
api/— the OpenAPI 3.1 spec for the native-client subset (iOS DTO codegen target). Seeapi/README.mdfor the preview commands. Also home to the MCP-server contract:api/mcp-capabilities.md(tools, resources, prompts, write model) andapi/mcp-skills.md(building a connector / skill).ops/— operator runbooks: deploy, backup / restore, encryption-key rotation, password reset and account recovery, env checks, TLS pinning, data repair. These are what an operator reaches for during an incident.self-hosting/— horizontal-scaling notes (HEALTHLOG_PROCESS_TYPE=web|worker|all) and the deploy-pipeline recipe. Routine install steps stay ondocs.healthlog.dev. Includesself-hosting/mcp.md— enabling the off-by-default MCP connector and pointing Claude.ai / ChatGPT / Claude Desktop at it, andself-hosting/sso.md— OIDC SSO login: IdP setup, the identity-pinning security model,OIDC_ONLYconsequences, and the break-glass runbook. Alsoself-hosting/geolite2.md— bringing your own offline GeoLite2 databases viaGEOLITE2_DIRinstead of sending login IPs to a third-party lookup. Andself-hosting/account-sharing.md— letting one account on the instance open another's health record: the three access levels, the eight sections a grant can be narrowed to, managed profiles for somebody with no login and the guardians who look after them, what revocation does and does not undo, and which trust properties of the instance are unchanged by any of it.migration/— migration notes. Currently only v1.3 to v1.4; later releases are covered byCHANGELOG.mdand https://docs.healthlog.dev.integrations/— per-integration reference (Withings, Fitbit, Google Health, WHOOP, Apple Health, AI providers, data import).adr/— architecture decision records.security/— threat-model notes.diagrams/— architecture diagrams rendered through docs.healthlog.dev.releases/— release imagery.apple-store-connect-checklist.md— the iOS submission gate.codex-protocol-spec.md— Codex / ChatGPT OAuth protocol reverse-engineering notes.ui-guidelines.md— UI / a11y house style.
If you're a self-hoster looking to spin up an instance, start at https://docs.healthlog.dev or jump to the Quick Start in the project README.