mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-14 14:40:52 +00:00
dcfec1f35de140db0c5ac8bdd257b8de34f545eb
9
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Marc-André Bombeck
|
996d2519d1 |
feat(import): read Health Connect exports
Upload the ZIP the Android Health Connect app writes and import its `health_connect_export.db` in the background, the same way an Apple Health export is imported: streamed to disk with a SHA-256, capped at 1 GiB, one import per account, a job row of kind `health_connect` that the shared reconcile cron and staging sweep now look up on its own queue. The worker extracts the database (4 GiB ceiling, zip-bomb checks) and opens it read-only with `immutable=1`, `trusted_schema=OFF` and `query_only`. A view in place of a record table refuses the file, columns are found by name, and schema versions below 9 are refused. Records are read in batches of 500 and aggregated in SQLite: steps, energy and distance as one total per day from the app the person's Health Connect priority list ranks first, heart rate per minute for 90 days and as hourly means before that, sleep stages with the unknown stage counted only in sessions without real stages, workouts, cycle days, water and micronutrients. Apps whose integration the account has connected are left out, and a reading already stored under another source is skipped. Rows carry source HEALTH_CONNECT, `hc:<uuid>` or the `stats:` ids, so a second import of the same export writes nothing. Both staged files are removed on every way out; the job result holds counts per type and per app only. Settings gain a Health Connect card with the export steps, progress and the outcome, in all seven languages, and the import is documented in docs/integrations/health-connect.md. |
||
|
Marc-André Bombeck
|
1b4594e69f |
fix(import): bound what an export may inflate to, allow one import per account, and never leave the upload behind
The extractor held the inflated export.xml to a flat 8 GiB whatever the member occupied, and the header ratio check reads sizes the archive writes itself: a member declaring an honest ratio could inflate at 1000:1, so an 8 MB upload could fill the temp directory. The streamed byte count is now held to 200 times the member's compressed size on disk (never less than 64 MiB, never more than 8 GiB), and a stored member may not grow at all. An account may have one Apple Health import queued or running. A second upload with different bytes is refused with 409 import.apple_health.busy and the running job's id; the check and the new row are one transaction under an advisory lock. The staged upload is the person's export in plain text. It is now removed on every exit that leaves no worker to read it: a refused or broken multipart body, no worker running, a queue that takes no job (the row is failed as well), the admin route's missing or unknown target account, and an account deleted before the worker reached it. The worker removes the files before recording a failure, so a row that went with a deleted account no longer strands them. The 15-minute import reconcile pass also sweeps staged files older than the longest an import may run. |
||
|
Marc-André Bombeck
|
c405f15ef0 |
fix(import): find the export XML whatever iOS named it
The archive member was matched against the English lowercase spelling alone. iOS writes that file name in the phone's language and some versions capitalise it, so an ordinary export could arrive as Export.xml or a translated name and was refused with a message telling the reader their export was invalid. The basename now matches in any capitalisation, and a single XML member at the top of the archive is taken as the translated spelling. Apple's clinical-document file never qualifies: it sits beside the real export and would parse to nothing. When no member qualifies the error lists the XML members the archive does hold, so the next report arrives with the answer in it. Refs #1008 |
||
|
Marc-André Bombeck
|
17fb291aaf |
fix(import): draw the extraction temp name from crypto randomness
The extracted export.xml lands in the shared system tmpdir under a name built from the clock and Math.random, which is predictable enough for a local pre-creation or symlink attack to aim at. The name comes from crypto.randomBytes now. Flagged by code scanning on the import path. |
||
|
Marc-André Bombeck
|
dd02f983cb |
fix(import): bound Apple Health import memory independent of archive size
The export.zip extractor buffered the whole archive through readFileSync — hundreds of MB resident for the entire unpack phase on a real multi-year export. The central directory is now located through a bounded tail read on a file handle, read as one size-capped slice, and the export.xml member streams through a byte-range read stream into the inflate pipeline, so peak extraction memory no longer scales with archive size. The cumulative fold hashed the raw device attribute into its per-day source buckets. Apple serialises that attribute from the live HKDevice object including its runtime address (<<HKDevice: 0x283a08640>, ...), which is not stable across the export enumeration — the map fragmented toward one bucket per record and an 11-million-record export grew it past the default 1 GB Node heap, killing the worker mid-parse (#775). Stripping the volatile address before hashing restores the documented invariant: parser state bounded by actual source cardinality. Also carry split multi-byte UTF-8 sequences across read-chunk boundaries with a StringDecoder — a bare per-chunk toString decayed a split umlaut in a source name to U+FFFD. |
||
|
Marc-André Bombeck
|
79843f8770 |
fix(import): stream the Apple Health ZIP extraction and sweep orphaned import jobs
Closes #588. The ZIP extractor decompressed export.xml synchronously into one in-memory buffer, then wrote it with a blocking writeFileSync — on the same event loop the web server runs in the default single-container deployment. A real multi-year export could pin the Node main thread for minutes, freezing every other request, while briefly holding roughly 2x the decompressed size in heap. A process that died mid-extraction never reached the failure path, so the ImportJob row stayed stuck in "unpacking" with "0 rows imported" forever and no error ever surfaced. Extraction now streams through zlib's async inflate into a file write stream, moving the decompression work off the event loop and bounding memory to the chunk size. The zip-bomb ceiling moves from inflateRawSync's maxOutputLength to an equivalent byte-counting transform on the streamed output. Orphan-ImportJob reconciliation used to run once, at worker boot. docker-compose's restart:unless-stopped does bring a crashed worker back, but a fast restart can land before the stuck row's heartbeat goes stale or pg-boss stops reporting the job as live — and because the boot-time pass never re-runs on a healthy worker, that row was never revisited again. A 15-minute cron now re-runs the same reconcile so a genuinely abandoned import converges to a visible failure instead of staying silently stuck. |
||
|
Marc-André Bombeck
|
1f4ba5939d |
style: apply prettier across the tree
One-shot prettier pass so the new format:check gate starts from a clean baseline. Formatting only — no behaviour change. Generator-owned and vendored artifacts stay untouched via .prettierignore. |
||
|
Marc-André BombeckandClaude Opus 4.7
|
165a14afda |
fix(import): refuse zip-bomb expansions on the Apple Health export
The hand-rolled extractor called `inflateRawSync(compressed)` without
a `maxOutputLength` option and never inspected the central directory's
advertised uncompressed size, so a 1 KB ZIP could expand to tens of
gigabytes during inflate and OOM the shared Next.js process. The route
already caps the upload at 1.5 GB and rate-limits to 3 / min per user,
but a single successful inflate would still take the whole container
with it.
Two layers of defence land here:
- Pre-flight refusal on the central directory's `uncompressedSize`
when it exceeds 8 GiB or when the advertised compression ratio
exceeds 200×. This is a coarse signal — the metadata is attacker-
controlled — but it catches honest-but-oversized payloads early.
- `inflateRawSync(compressed, { maxOutputLength: 8 GiB })` instructs
`node:zlib` to throw rather than expand past the cap, which holds
even when the metadata lies about the uncompressed size.
8 GiB leaves a wide ceiling for legitimate multi-year heavy-user
exports (the largest real-world ones settle in the low single-digit
GB range), while making the bomb class infeasible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Marc-André Bombeck
|
affede31a4 |
v1.4.34 — apple-health import + reliability + web-freeze
The final functional web release before the iOS native client lands.
Headline work: a streaming Apple Health `export.zip` importer for
prospective iOS-migrating accounts, a server-side aggregation cache
that collapses the three hottest dashboard reads onto in-process LRU
slots with single-flight coalescing and per-user invalidation, a
broader compliance classifier with a dedicated `early` bucket so
ahead-of-window doses count as compliant, the Coach launch surface
hoisted onto every authed page with a dashboard-hero CTA, and a
trimmer Settings shelf that folds Sources into Targets. The release
also lights up the `cache.<name>.outcome` annotation on the active
wide event so production logs carry the cache hit-ratio signal
without leaking userIds.
### Added
- **Server-side aggregation cache.** A new `ServerCache<T>` primitive
(`src/lib/cache/server-cache.ts`) extends the v1.4.33 Coach snapshot
LRU shape into a reusable per-route layer with TTL expiry,
capacity-bounded LRU eviction, single-flight read coalescing, and
per-instance hit / miss / eviction / stampede counters. Wires the
three hottest dashboard reads — `/api/analytics` (slim + default),
`/api/gamification/achievements`, `/api/medications/intake?scope=compliance`
— and bolts per-user invalidation onto every measurement / mood /
medication / workout / dashboard-widget write endpoint so the next
read paints fresh data instead of waiting out the TTL. Every cache
hit / miss surfaces on the active wide event as
`cache.<name>.outcome` + `cache.<name>.key_hash` so production logs
carry the hit-ratio signal without leaking userIds.
- **Apple Health `export.zip` import.** New endpoint
`POST /api/import/apple-health-export` (synchronous multipart
upload, asynchronous ingest via a dedicated pg-boss
`apple-health-import` worker, per-`MeasurementType` ingestion
stats). Streams the upload straight to disk so a multi-gigabyte
export never lands in V8 heap, hashes the bytes inline for
content-based idempotency, and unpacks `apple_health_export/export.xml`
with a hand-rolled ZIP central-directory walker that handles Zip64
for archives past the 4 GB barrier. The parser folds every
`<Record>`, `<Workout>`, `<Correlation>`, and `<ClinicalRecord>`
into the existing `Measurement` and `Workout` row shapes — spot
rows keyed by `HKMetadataKeyExternalUUID` (or a deterministic
`sample:<sha256>` fallback), cumulative HK types collapsed into
one `stats:<HKType>:<YYYY-MM-DD>` row per user-local day to match
the iOS daily-aggregation convention. Live per-stage progress
surfaces through `GET /api/import/apple-health-export/{jobId}/status`.
An admin variant at `POST /api/admin/import-apple-health-export`
imports on behalf of a target user (cookie-only `requireAdmin()`
gate; Bearer tokens never elevate).
- **`ImportJob` schema model + migration.** New Prisma model captures
the per-upload state machine (`queued | unpacking | parsing |
upserting | done | failed`), the content-hash for idempotency
short-circuits, and the per-`MeasurementType` ingestion counters
the status route surfaces back to the client.
- **`lastSeenByType` on `/api/analytics`.** Both the slim
(`?slice=summaries`) and default slices return a per-type
`{ lastSeenAt, daysAgo }` map. The dashboard trend tiles wire 12
mounts to a new `tileStaleDays()` helper so an "X days / weeks /
months ago" hint paints under any per-metric tile whose last
sample crosses the 7-day floor. Six locales gained additive
week / month plural keys; Polish uses the genitive-plural form
that covers every non-1 count the bucket math can produce.
- **Dashboard "Ask the coach" hero CTA.** A new pill next to the
existing "Hinzufügen" / "Add" launches the Coach drawer directly
from the dashboard hero. The `<CoachLaunchProvider>` hoisted from
the insights layout to the auth shell so the drawer is reachable
from every authed route; the floating action button stays scoped
to `/insights/**` so it cannot distract on the dashboard.
- **Shared achievements query hook.** `useAchievementsQuery()`
centralises the three previous consumers — recent-achievements
card, the `/achievements` mother page, and the unlock notifier —
onto one TanStack queryKey + cache slot so dashboard cold mount
fires the endpoint once instead of twice.
- **Typed authed Cache-Control presets.** New
`src/lib/http/cache-headers.ts` module exports
`NO_STORE_BUT_BFCACHE = "private, max-age=0, must-revalidate"`,
`SHORT_LIVED_PUBLIC`, and an `applyAuthedHeaders()` helper. The
presets land at the framework level via a `next.config.ts`
`headers()` rule that stamps the bfcache-friendly directive on
every authed HTML response, restoring Chromium bfcache eligibility
for in-app back / forward navigation.
- **`scripts/print-bundle-report.mjs`.** Restores the at-a-glance
"top client chunks by parsed size" signal Turbopack dropped from
`next build`; reads `.next/analyze/client.json` (written by
`@next/bundle-analyzer` when `pnpm analyze` runs with `ANALYZE=1`)
and prints a sorted table plus totals.
### Changed
- **Settings sidebar: "Sources" folded into "Targets & Sources".**
Per-metric threshold ranges and per-metric source priority now
share one `/settings/thresholds` page so the same metric's
threshold and device-source preference sit together instead of
one sidebar entry away. `/settings/sources` keeps a
`permanentRedirect` so external bookmarks and docs links follow
through unchanged. Section count: 11 → 10.
- **Insights tab strip: vital pills collapse under a "Vitals"
parent.** Five vital pills (HRV, Resting HR, Oxygen, Body
Temperature, Active Energy) hide behind one parent pill that
opens a popover sub-list. Parent-pill active state mirrors the
URL so spatial orientation survives the collapse; each sub-page
keeps its own URL and bookmark resolves unchanged. Strip
footprint: 14 → 10 entries when every metric has data.
- **Coolify env-var audit.** `mcp__coolify-apps01__env_vars`
inspection captured the section-1 / section-2 duplicates that
have accumulated under apps-01 since v1.3.1. Audit pinned at
`.planning/round-v1434-iwa-coolify-env-audit.md`; no env-var
deletes performed (operator action).
### Fixed
- **Compliance classifier: early intakes count as compliant.** The
classifier picked up a dedicated `early` bucket (window-start
minus three hours through window-start), the `on_time` post-window
tolerance widened from one hour to three hours, and the `late`
band sits between the new `on_time` ceiling and the configurable
`lateMinutes` knob. Heatmap consumers route the new bucket
through the compliant path; the dedicated `early` counter rides
on `DailyComplianceEntry` for downstream consumers that want to
distinguish ahead-of-window from on-window intakes.
- **Compliance heatmap fallback retired.** The v1.4.33 defensive
`looksClassifierBug` fallthrough is gone now that the underlying
classifier widening removes the every-dose-very-late mode the
fallback covered.
### Performance
- **`/api/gamification/achievements` consumer collapse.** Two of
three previous consumers shared the same TanStack queryKey
literal; the third carried a per-user discriminator so the cache
treated it as a fresh slot. Dashboard cold mount fired the
endpoint twice. Collapsing onto the shared hook trims that to a
single request; with the new server-side cache slot warm, the
duplicate-mount worst case rides single-flight coalescing.
- **GHCR build fires once per release tag.** The
`docker-publish.yml` workflow lost its `push.branches: [main]`
trigger; the `:latest` raw-tag enable rule moved onto the tag
ref so each release produces exactly one multi-arch build that
refreshes both the semver tags and the `:latest` alias.
- **NFT-trace warnings silenced.** `next.config.ts` gained an
`outputFileTracingExcludes` block that narrows the Turbopack
tracer away from the `MAXMIND_LICENSE_KEY` env-access path so
the trace-report no longer warns on every dependent route file.
### Refactor
- **Dashboard analytics consumers.** The two cards that previously
decided their own freshness copy now lean on the typed
`tileStaleDays()` helper and the additive `lastSeenByType` field
on the analytics response. Mood and BD-Zielbereich tiles keep the
default `null` because they have no underlying per-type freshness
signal.
### Accessibility
- **Dashboard hero CTA hits the WCAG 2.5.5 touch-target floor.**
The new "Ask the coach" pill carries `min-h-11` on mobile and
the matching `sm:min-h-9` desktop floor so the touch target
stays above 44 px on the Pixel 5 boundary.
- **Insights tab-strip Vitals parent reads correctly to assistive
tech.** Parent pill carries `aria-current="page"` whenever the
current URL matches one of its child vital sub-pages, plus a
`data-slot="insights-tab-strip-group"` hook for visual-regression
testing.
### Internal
- **Two e2e flake windows tightened.** `onboarding-flicker` swapped
its 12-sample 50 ms poll loop for a single Playwright
auto-retrying `toBeHidden({ timeout: 700 })` assertion so the
1-2 ms race window between the analytics-pending shell and
`useAuth().user` resolving collapses to a single retry slot.
`mobile-viewport` dropped `nav a[href]` from the touch-target
sweep (the bottom-nav owns its own WCAG spec) and gated the
44-px floor on `matchMedia('(min-width: 640px)').matches === false`
so the Pixel 5 viewport never tripped into the `sm:` tier during
WebKit render commits.
- **Server-cache observability.** Every `cached()` invocation passes
an `annotate` callback that lands two keys on the wide event:
`cache.<name>.outcome` ∈ `{ hit | miss | stampede }` and
`cache.<name>.key_hash` (non-reversible djb2 32-bit hash). Ops
can grep `cache.analytics.outcome` over any time window to
compute the hit ratio per deployment.
- **AASA followups verified.** The v1.4.33 `/.well-known/apple-app-site-association`
handler serves a direct 200 on every fronting origin; Apple's
CDN has ingested matching bodies for all three domains.
### Web freeze
v1.4.34 marks the last functional release of the web codebase before
the iOS native client launches. Subsequent v1.4.x tags carry security
fixes, dependency bumps, and hotfix-only corrections. New feature work
is paused until the iOS app clears Apple review, at which point a
v1.5.0 version-bump-only release tags the milestone. The Prisma
schema head comment and `.planning/v15-strategic-plan.md` §5
decision-log row pin the freeze trigger in-tree.
|