mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 12:32:43 +00:00
f29995a18cfd3bbf35201c19b0c7adddea3a350f
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Marc-André Bombeck
|
17fb291aaf |
fix(import): draw the extraction temp name from crypto randomness
The extracted export.xml lands in the shared system tmpdir under a name built from the clock and Math.random, which is predictable enough for a local pre-creation or symlink attack to aim at. The name comes from crypto.randomBytes now. Flagged by code scanning on the import path. |
||
|
Marc-André Bombeck
|
50ff90a409 |
feat(import): refuse an Apple Health import the heap cannot carry
Before the multi-minute unpack starts, compare the archive's declared uncompressed export.xml size against the V8 heap limit. The streaming parse keeps resident state span-bounded, so the model is generous — a fixed 192 MiB base plus 1/64 of the XML size; the 8 GiB extraction cap needs ~320 MiB, which any default heap clears. The check only refuses the truly impossible (an operator-pinned tiny heap against a huge export) and does so with an actionable failureReason naming NODE_OPTIONS=--max-old-space-size, instead of an OOM-killed worker and a job stuck in a running state (#775). Document the memory model, the insufficient_memory refusal, and the interrupted-import reconcile in the self-hosting scaling guide. |
||
|
Marc-André Bombeck
|
dd02f983cb |
fix(import): bound Apple Health import memory independent of archive size
The export.zip extractor buffered the whole archive through readFileSync — hundreds of MB resident for the entire unpack phase on a real multi-year export. The central directory is now located through a bounded tail read on a file handle, read as one size-capped slice, and the export.xml member streams through a byte-range read stream into the inflate pipeline, so peak extraction memory no longer scales with archive size. The cumulative fold hashed the raw device attribute into its per-day source buckets. Apple serialises that attribute from the live HKDevice object including its runtime address (<<HKDevice: 0x283a08640>, ...), which is not stable across the export enumeration — the map fragmented toward one bucket per record and an 11-million-record export grew it past the default 1 GB Node heap, killing the worker mid-parse (#775). Stripping the volatile address before hashing restores the documented invariant: parser state bounded by actual source cardinality. Also carry split multi-byte UTF-8 sequences across read-chunk boundaries with a StringDecoder — a bare per-chunk toString decayed a split umlaut in a source name to U+FFFD. |
||
|
Marc-André Bombeck
|
4e95bd28f8 |
fix(import): stop the Apple Health card ticking an archive that landed nothing
Same shape as the CSV and JSON cards: reaching the done state rendered the success tick whether or not a single row was upserted. The archive's count picks the tone now, and an archive that wrote nothing says so. The clinical-record count stays out of that decision. Skipping electrocardiograms and lab documents is a documented, deliberate exclusion, not a refused row, so it must not read as a warning. Refs #640 |
||
|
Marc-André Bombeck
|
0c9832eb8d |
fix(import): accept a blood-glucose reading that carries no context
A CSV import of a continuous-sensor export wrote nothing and rendered a green success state. Three faults compounded, and the reporter took all three apart before filing. The parser refused every BLOOD_GLUCOSE row with a blank glucoseContext. Behind it, the CHECK constraint from migration 0021 read both ways at once: a glucose row MUST carry a context, every other type MUST NOT. The second half is right and stays. The first was written when the only glucose writer was the entry form, where a person picks fasting or post-meal at the moment of the reading. Every machine writer that arrived since has no such answer per sample, so the Nightscout pull, the MCP log tool, the Telegram capture and both bulk importers were all building rows the database then refused. A missing context now stores as NULL, never as a fifth enum member. Every per-context reader selects by equality against the four members, so the reading stays out of the doctor report's fasting rows, the matching FHIR observations and the per-context targets, while the chart, the all-time summary and the clinical panel keep it in full. Hand entry still asks for a context, because at that moment you know it. The import cards now read the counts rather than the status code. Nothing written is a failure, a mixed file is a warning, and the refused rows are grouped by reason with a count instead of fifty identical lines, with the per-line detail one disclosure away. Both cards render through one outcome table, so a tick cannot reach a result that wrote nothing. Refs #640 |
||
|
Marc-André Bombeck
|
15b5f6f92f |
fix(ingest): hold the stability floor for identities no request schema sees
The dedup key on the medication intake surfaces is not only `externalId`. `MedicationIntakeEvent.idempotency_key` is a `@unique` column persisted with the dose row forever, with no TTL and no sweep, matched by plain equality on the per-medication POST, the bulk drain and the external ingest bridge. That makes it an identity, not a replay token, and it carries the same failure mode: a key that rotates per client launch never matches its own earlier post, so every retry logs the dose again. The blank case is sharper still — the column is unique table-wide, so an empty key resolves to whichever unrelated row stored it first. Refuse both shapes at the schema on the single-entry paths and per entry on the bulk drain. The MCP write path hashes a host-supplied key into the `externalId` half of its dedup key, so the shape is laundered before anything downstream could see it and the check has to run on the input. That closes a live one: every tool handler falls back to `""` when the host omits the argument, and `""` digests to a constant, so each keyless write after the first reported `already_logged` and was silently dropped. The HTTP `Idempotency-Key` header stays out, and now says why in the guard: `IdempotencyKey.expiresAt` bounds it and the cleanup job sweeps it, so it is a windowed replay token and rotation is its design. Extend the structural guard with a provenance rule for identities that never pass a Zod request schema — a spreadsheet column, an MCP tool argument. Inside the import and MCP trees, a module handling an identity must apply the floor itself or import a module that does. That is a real chain rather than a list of known offenders: a new importer that mints an `externalId` from file input and imports nothing checked fails it, with no allowlist to escape into. |
||
|
Marc-André Bombeck
|
2ab66ab2e3 |
fix(ingest): refuse an external id that cannot be a stable identity
An external id is half of every ingest dedup key — (userId, source, externalId), (userId, externalSource, externalId), (userId, type, source, externalId). Re-posting the same triple is idempotent and returns the existing row, which holds only while the id is stable across client launches. A client that serialised an opaque platform object with its default object description sent a per-allocation memory address instead, so no sweep ever matched its own earlier row and each one minted a fresh record: a live instance grew 23 phantom medications in a day, none of which could collect a dose, because the dose side resolved through the same rotating string. Add one shared classifier for identifiers that provably cannot be an identity — an angle-bracketed object description carrying a hex pointer, a bare 0x address, and a blank value — and wire it at every surface that accepts a client-supplied externalId. Single-entry creates refuse through the schema, so the client gets the standard 422 multi-issue envelope with a message naming the shape it sent and what to send instead. Batch and bulk surfaces classify per entry and report the offending row as skipped with reason unstable_external_id, leaving the rest of the payload to land: one bad row must not stop a sync. Every refusal carries pinned wide-event meta (count, shape, surface) and never the id itself. A structural guard freezes both routes into the floor: a request-side externalId declaration either applies the field check or carries a marker naming the file that checks it per entry, and the guard verifies that file really does. The backup-restore payload is the one exemption, with its reason recorded — it replays ids the server itself wrote. |
||
|
Marc-André Bombeck
|
dd7213db5c |
fix(measurements): honor the unit preference at entry and in the record list
Add the affine display-transform registry (kg↔lb, cm↔in, °C↔°F with the temperature offset carried as an absolute-only shift), a single client hook that reads the preference, and a shared inbound unit-alias resolver. Wire the manual entry form to label, prefill, and convert at the boundary so an imperial reading persists in canonical SI, and route the record list value cells and edit dialog through the same choke point. Close the one write path that could persist a non-canonical unit: the MCP log_measurement tool now resolves a unit hint to canonical or refuses it, reusing the resolver the CSV importer already relied on. |
||
|
Marc-André Bombeck
|
fb50335459 |
fix(insights): stop the pulse page from silently swapping to resting heart rate
Closes #584. `/insights/pulse` computed hasRestingHr from RESTING_HEART_RATE presence and used it to swap the ENTIRE primary chart and the Coach read strip to that series whenever any resting row existed, while the title, stat strip, capture action, and "show all values" link kept saying PULSE. A user could see a resting-only chart under a page that claimed to be pulse throughout. The chart now always includes PULSE as its first series; when resting data exists, RESTING_HEART_RATE renders as a second, clearly-labelled series (the same multi-series pattern the blood-pressure page uses for systolic/diastolic) instead of replacing it. The Coach read strip stays pinned to PULSE. The page also drops the resting-calibrated target band it used to paint behind the chart — that band is calibrated to steady-state resting readings and would misreport an expected workout spike in raw PULSE data as "out of target"; the personal target belongs on the dedicated resting-pulse page against the clean resting series. |
||
|
Marc-André Bombeck
|
79843f8770 |
fix(import): stream the Apple Health ZIP extraction and sweep orphaned import jobs
Closes #588. The ZIP extractor decompressed export.xml synchronously into one in-memory buffer, then wrote it with a blocking writeFileSync — on the same event loop the web server runs in the default single-container deployment. A real multi-year export could pin the Node main thread for minutes, freezing every other request, while briefly holding roughly 2x the decompressed size in heap. A process that died mid-extraction never reached the failure path, so the ImportJob row stayed stuck in "unpacking" with "0 rows imported" forever and no error ever surfaced. Extraction now streams through zlib's async inflate into a file write stream, moving the decompression work off the event loop and bounding memory to the chunk size. The zip-bomb ceiling moves from inflateRawSync's maxOutputLength to an equivalent byte-counting transform on the streamed output. Orphan-ImportJob reconciliation used to run once, at worker boot. docker-compose's restart:unless-stopped does bring a crashed worker back, but a fast restart can land before the stuck row's heartbeat goes stale or pg-boss stops reporting the job as live — and because the boot-time pass never re-runs on a healthy worker, that row was never revisited again. A 15-minute cron now re-runs the same reconcile so a genuinely abandoned import converges to a visible failure instead of staying silently stuck. |
||
|
Marc-André Bombeck
|
1f4ba5939d |
style: apply prettier across the tree
One-shot prettier pass so the new format:check gate starts from a clean baseline. Formatting only — no behaviour change. Generator-owned and vendored artifacts stay untouched via .prettierignore. |
||
|
Marc-André Bombeck
|
fbdcb31202 |
feat(import): add CSV measurement import with per-row status and dry-run
A self-hoster migrating from a spreadsheet or a meter export overwhelmingly has CSV, which neither the JSON nor the Apple Health import covers. Add `POST /api/import/csv` reusing the existing write loop and rollup re-fold. The documented header is `type,value,unit,measuredAt[,glucoseContext,notes, externalId]`, order-independent. measuredAt must carry an explicit ISO-8601 offset — a row without one is skipped rather than guessed — and is bounded by the shared entry-instant validator. Glucose accepts mmol/L (converted to canonical mg/dL) and weight accepts lb (converted to kg); any other non-canonical unit is skipped, never mis-stored. An optional externalId column makes a re-upload idempotent; without it a re-upload duplicates. The response carries a per-row status envelope so a 2000-row file with three bad rows imports 1997 and names the three. `?dryRun=1` validates and previews the outcome without writing. The parse, validation, and unit conversion live in a pure module so they are unit-tested without a database. Settings -> Export & Import gains a third CSV card mirroring the JSON and Apple Health cards, with upload, paste, preview, and a downloadable example. The new route is documented in the OpenAPI contract. |
||
|
Marc-André BombeckandClaude Opus 4.7
|
165a14afda |
fix(import): refuse zip-bomb expansions on the Apple Health export
The hand-rolled extractor called `inflateRawSync(compressed)` without
a `maxOutputLength` option and never inspected the central directory's
advertised uncompressed size, so a 1 KB ZIP could expand to tens of
gigabytes during inflate and OOM the shared Next.js process. The route
already caps the upload at 1.5 GB and rate-limits to 3 / min per user,
but a single successful inflate would still take the whole container
with it.
Two layers of defence land here:
- Pre-flight refusal on the central directory's `uncompressedSize`
when it exceeds 8 GiB or when the advertised compression ratio
exceeds 200×. This is a coarse signal — the metadata is attacker-
controlled — but it catches honest-but-oversized payloads early.
- `inflateRawSync(compressed, { maxOutputLength: 8 GiB })` instructs
`node:zlib` to throw rather than expand past the cap, which holds
even when the metadata lies about the uncompressed size.
8 GiB leaves a wide ceiling for legitimate multi-year heavy-user
exports (the largest real-world ones settle in the low single-digit
GB range), while making the bomb class infeasible.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Marc-André Bombeck
|
affede31a4 |
v1.4.34 — apple-health import + reliability + web-freeze
The final functional web release before the iOS native client lands.
Headline work: a streaming Apple Health `export.zip` importer for
prospective iOS-migrating accounts, a server-side aggregation cache
that collapses the three hottest dashboard reads onto in-process LRU
slots with single-flight coalescing and per-user invalidation, a
broader compliance classifier with a dedicated `early` bucket so
ahead-of-window doses count as compliant, the Coach launch surface
hoisted onto every authed page with a dashboard-hero CTA, and a
trimmer Settings shelf that folds Sources into Targets. The release
also lights up the `cache.<name>.outcome` annotation on the active
wide event so production logs carry the cache hit-ratio signal
without leaking userIds.
### Added
- **Server-side aggregation cache.** A new `ServerCache<T>` primitive
(`src/lib/cache/server-cache.ts`) extends the v1.4.33 Coach snapshot
LRU shape into a reusable per-route layer with TTL expiry,
capacity-bounded LRU eviction, single-flight read coalescing, and
per-instance hit / miss / eviction / stampede counters. Wires the
three hottest dashboard reads — `/api/analytics` (slim + default),
`/api/gamification/achievements`, `/api/medications/intake?scope=compliance`
— and bolts per-user invalidation onto every measurement / mood /
medication / workout / dashboard-widget write endpoint so the next
read paints fresh data instead of waiting out the TTL. Every cache
hit / miss surfaces on the active wide event as
`cache.<name>.outcome` + `cache.<name>.key_hash` so production logs
carry the hit-ratio signal without leaking userIds.
- **Apple Health `export.zip` import.** New endpoint
`POST /api/import/apple-health-export` (synchronous multipart
upload, asynchronous ingest via a dedicated pg-boss
`apple-health-import` worker, per-`MeasurementType` ingestion
stats). Streams the upload straight to disk so a multi-gigabyte
export never lands in V8 heap, hashes the bytes inline for
content-based idempotency, and unpacks `apple_health_export/export.xml`
with a hand-rolled ZIP central-directory walker that handles Zip64
for archives past the 4 GB barrier. The parser folds every
`<Record>`, `<Workout>`, `<Correlation>`, and `<ClinicalRecord>`
into the existing `Measurement` and `Workout` row shapes — spot
rows keyed by `HKMetadataKeyExternalUUID` (or a deterministic
`sample:<sha256>` fallback), cumulative HK types collapsed into
one `stats:<HKType>:<YYYY-MM-DD>` row per user-local day to match
the iOS daily-aggregation convention. Live per-stage progress
surfaces through `GET /api/import/apple-health-export/{jobId}/status`.
An admin variant at `POST /api/admin/import-apple-health-export`
imports on behalf of a target user (cookie-only `requireAdmin()`
gate; Bearer tokens never elevate).
- **`ImportJob` schema model + migration.** New Prisma model captures
the per-upload state machine (`queued | unpacking | parsing |
upserting | done | failed`), the content-hash for idempotency
short-circuits, and the per-`MeasurementType` ingestion counters
the status route surfaces back to the client.
- **`lastSeenByType` on `/api/analytics`.** Both the slim
(`?slice=summaries`) and default slices return a per-type
`{ lastSeenAt, daysAgo }` map. The dashboard trend tiles wire 12
mounts to a new `tileStaleDays()` helper so an "X days / weeks /
months ago" hint paints under any per-metric tile whose last
sample crosses the 7-day floor. Six locales gained additive
week / month plural keys; Polish uses the genitive-plural form
that covers every non-1 count the bucket math can produce.
- **Dashboard "Ask the coach" hero CTA.** A new pill next to the
existing "Hinzufügen" / "Add" launches the Coach drawer directly
from the dashboard hero. The `<CoachLaunchProvider>` hoisted from
the insights layout to the auth shell so the drawer is reachable
from every authed route; the floating action button stays scoped
to `/insights/**` so it cannot distract on the dashboard.
- **Shared achievements query hook.** `useAchievementsQuery()`
centralises the three previous consumers — recent-achievements
card, the `/achievements` mother page, and the unlock notifier —
onto one TanStack queryKey + cache slot so dashboard cold mount
fires the endpoint once instead of twice.
- **Typed authed Cache-Control presets.** New
`src/lib/http/cache-headers.ts` module exports
`NO_STORE_BUT_BFCACHE = "private, max-age=0, must-revalidate"`,
`SHORT_LIVED_PUBLIC`, and an `applyAuthedHeaders()` helper. The
presets land at the framework level via a `next.config.ts`
`headers()` rule that stamps the bfcache-friendly directive on
every authed HTML response, restoring Chromium bfcache eligibility
for in-app back / forward navigation.
- **`scripts/print-bundle-report.mjs`.** Restores the at-a-glance
"top client chunks by parsed size" signal Turbopack dropped from
`next build`; reads `.next/analyze/client.json` (written by
`@next/bundle-analyzer` when `pnpm analyze` runs with `ANALYZE=1`)
and prints a sorted table plus totals.
### Changed
- **Settings sidebar: "Sources" folded into "Targets & Sources".**
Per-metric threshold ranges and per-metric source priority now
share one `/settings/thresholds` page so the same metric's
threshold and device-source preference sit together instead of
one sidebar entry away. `/settings/sources` keeps a
`permanentRedirect` so external bookmarks and docs links follow
through unchanged. Section count: 11 → 10.
- **Insights tab strip: vital pills collapse under a "Vitals"
parent.** Five vital pills (HRV, Resting HR, Oxygen, Body
Temperature, Active Energy) hide behind one parent pill that
opens a popover sub-list. Parent-pill active state mirrors the
URL so spatial orientation survives the collapse; each sub-page
keeps its own URL and bookmark resolves unchanged. Strip
footprint: 14 → 10 entries when every metric has data.
- **Coolify env-var audit.** `mcp__coolify-apps01__env_vars`
inspection captured the section-1 / section-2 duplicates that
have accumulated under apps-01 since v1.3.1. Audit pinned at
`.planning/round-v1434-iwa-coolify-env-audit.md`; no env-var
deletes performed (operator action).
### Fixed
- **Compliance classifier: early intakes count as compliant.** The
classifier picked up a dedicated `early` bucket (window-start
minus three hours through window-start), the `on_time` post-window
tolerance widened from one hour to three hours, and the `late`
band sits between the new `on_time` ceiling and the configurable
`lateMinutes` knob. Heatmap consumers route the new bucket
through the compliant path; the dedicated `early` counter rides
on `DailyComplianceEntry` for downstream consumers that want to
distinguish ahead-of-window from on-window intakes.
- **Compliance heatmap fallback retired.** The v1.4.33 defensive
`looksClassifierBug` fallthrough is gone now that the underlying
classifier widening removes the every-dose-very-late mode the
fallback covered.
### Performance
- **`/api/gamification/achievements` consumer collapse.** Two of
three previous consumers shared the same TanStack queryKey
literal; the third carried a per-user discriminator so the cache
treated it as a fresh slot. Dashboard cold mount fired the
endpoint twice. Collapsing onto the shared hook trims that to a
single request; with the new server-side cache slot warm, the
duplicate-mount worst case rides single-flight coalescing.
- **GHCR build fires once per release tag.** The
`docker-publish.yml` workflow lost its `push.branches: [main]`
trigger; the `:latest` raw-tag enable rule moved onto the tag
ref so each release produces exactly one multi-arch build that
refreshes both the semver tags and the `:latest` alias.
- **NFT-trace warnings silenced.** `next.config.ts` gained an
`outputFileTracingExcludes` block that narrows the Turbopack
tracer away from the `MAXMIND_LICENSE_KEY` env-access path so
the trace-report no longer warns on every dependent route file.
### Refactor
- **Dashboard analytics consumers.** The two cards that previously
decided their own freshness copy now lean on the typed
`tileStaleDays()` helper and the additive `lastSeenByType` field
on the analytics response. Mood and BD-Zielbereich tiles keep the
default `null` because they have no underlying per-type freshness
signal.
### Accessibility
- **Dashboard hero CTA hits the WCAG 2.5.5 touch-target floor.**
The new "Ask the coach" pill carries `min-h-11` on mobile and
the matching `sm:min-h-9` desktop floor so the touch target
stays above 44 px on the Pixel 5 boundary.
- **Insights tab-strip Vitals parent reads correctly to assistive
tech.** Parent pill carries `aria-current="page"` whenever the
current URL matches one of its child vital sub-pages, plus a
`data-slot="insights-tab-strip-group"` hook for visual-regression
testing.
### Internal
- **Two e2e flake windows tightened.** `onboarding-flicker` swapped
its 12-sample 50 ms poll loop for a single Playwright
auto-retrying `toBeHidden({ timeout: 700 })` assertion so the
1-2 ms race window between the analytics-pending shell and
`useAuth().user` resolving collapses to a single retry slot.
`mobile-viewport` dropped `nav a[href]` from the touch-target
sweep (the bottom-nav owns its own WCAG spec) and gated the
44-px floor on `matchMedia('(min-width: 640px)').matches === false`
so the Pixel 5 viewport never tripped into the `sm:` tier during
WebKit render commits.
- **Server-cache observability.** Every `cached()` invocation passes
an `annotate` callback that lands two keys on the wide event:
`cache.<name>.outcome` ∈ `{ hit | miss | stampede }` and
`cache.<name>.key_hash` (non-reversible djb2 32-bit hash). Ops
can grep `cache.analytics.outcome` over any time window to
compute the hit ratio per deployment.
- **AASA followups verified.** The v1.4.33 `/.well-known/apple-app-site-association`
handler serves a direct 200 on every fronting origin; Apple's
CDN has ingested matching bodies for all three domains.
### Web freeze
v1.4.34 marks the last functional release of the web codebase before
the iOS native client launches. Subsequent v1.4.x tags carry security
fixes, dependency bumps, and hotfix-only corrections. New feature work
is paused until the iOS app clears Apple review, at which point a
v1.5.0 version-bump-only release tags the milestone. The Prisma
schema head comment and `.planning/v15-strategic-plan.md` §5
decision-log row pin the freeze trigger in-tree.
|