mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-16 23:33:29 +00:00
The final functional web release before the iOS native client lands.
Headline work: a streaming Apple Health `export.zip` importer for
prospective iOS-migrating accounts, a server-side aggregation cache
that collapses the three hottest dashboard reads onto in-process LRU
slots with single-flight coalescing and per-user invalidation, a
broader compliance classifier with a dedicated `early` bucket so
ahead-of-window doses count as compliant, the Coach launch surface
hoisted onto every authed page with a dashboard-hero CTA, and a
trimmer Settings shelf that folds Sources into Targets. The release
also lights up the `cache.<name>.outcome` annotation on the active
wide event so production logs carry the cache hit-ratio signal
without leaking userIds.
### Added
- **Server-side aggregation cache.** A new `ServerCache<T>` primitive
(`src/lib/cache/server-cache.ts`) extends the v1.4.33 Coach snapshot
LRU shape into a reusable per-route layer with TTL expiry,
capacity-bounded LRU eviction, single-flight read coalescing, and
per-instance hit / miss / eviction / stampede counters. Wires the
three hottest dashboard reads — `/api/analytics` (slim + default),
`/api/gamification/achievements`, `/api/medications/intake?scope=compliance`
— and bolts per-user invalidation onto every measurement / mood /
medication / workout / dashboard-widget write endpoint so the next
read paints fresh data instead of waiting out the TTL. Every cache
hit / miss surfaces on the active wide event as
`cache.<name>.outcome` + `cache.<name>.key_hash` so production logs
carry the hit-ratio signal without leaking userIds.
- **Apple Health `export.zip` import.** New endpoint
`POST /api/import/apple-health-export` (synchronous multipart
upload, asynchronous ingest via a dedicated pg-boss
`apple-health-import` worker, per-`MeasurementType` ingestion
stats). Streams the upload straight to disk so a multi-gigabyte
export never lands in V8 heap, hashes the bytes inline for
content-based idempotency, and unpacks `apple_health_export/export.xml`
with a hand-rolled ZIP central-directory walker that handles Zip64
for archives past the 4 GB barrier. The parser folds every
`<Record>`, `<Workout>`, `<Correlation>`, and `<ClinicalRecord>`
into the existing `Measurement` and `Workout` row shapes — spot
rows keyed by `HKMetadataKeyExternalUUID` (or a deterministic
`sample:<sha256>` fallback), cumulative HK types collapsed into
one `stats:<HKType>:<YYYY-MM-DD>` row per user-local day to match
the iOS daily-aggregation convention. Live per-stage progress
surfaces through `GET /api/import/apple-health-export/{jobId}/status`.
An admin variant at `POST /api/admin/import-apple-health-export`
imports on behalf of a target user (cookie-only `requireAdmin()`
gate; Bearer tokens never elevate).
- **`ImportJob` schema model + migration.** New Prisma model captures
the per-upload state machine (`queued | unpacking | parsing |
upserting | done | failed`), the content-hash for idempotency
short-circuits, and the per-`MeasurementType` ingestion counters
the status route surfaces back to the client.
- **`lastSeenByType` on `/api/analytics`.** Both the slim
(`?slice=summaries`) and default slices return a per-type
`{ lastSeenAt, daysAgo }` map. The dashboard trend tiles wire 12
mounts to a new `tileStaleDays()` helper so an "X days / weeks /
months ago" hint paints under any per-metric tile whose last
sample crosses the 7-day floor. Six locales gained additive
week / month plural keys; Polish uses the genitive-plural form
that covers every non-1 count the bucket math can produce.
- **Dashboard "Ask the coach" hero CTA.** A new pill next to the
existing "Hinzufügen" / "Add" launches the Coach drawer directly
from the dashboard hero. The `<CoachLaunchProvider>` hoisted from
the insights layout to the auth shell so the drawer is reachable
from every authed route; the floating action button stays scoped
to `/insights/**` so it cannot distract on the dashboard.
- **Shared achievements query hook.** `useAchievementsQuery()`
centralises the three previous consumers — recent-achievements
card, the `/achievements` mother page, and the unlock notifier —
onto one TanStack queryKey + cache slot so dashboard cold mount
fires the endpoint once instead of twice.
- **Typed authed Cache-Control presets.** New
`src/lib/http/cache-headers.ts` module exports
`NO_STORE_BUT_BFCACHE = "private, max-age=0, must-revalidate"`,
`SHORT_LIVED_PUBLIC`, and an `applyAuthedHeaders()` helper. The
presets land at the framework level via a `next.config.ts`
`headers()` rule that stamps the bfcache-friendly directive on
every authed HTML response, restoring Chromium bfcache eligibility
for in-app back / forward navigation.
- **`scripts/print-bundle-report.mjs`.** Restores the at-a-glance
"top client chunks by parsed size" signal Turbopack dropped from
`next build`; reads `.next/analyze/client.json` (written by
`@next/bundle-analyzer` when `pnpm analyze` runs with `ANALYZE=1`)
and prints a sorted table plus totals.
### Changed
- **Settings sidebar: "Sources" folded into "Targets & Sources".**
Per-metric threshold ranges and per-metric source priority now
share one `/settings/thresholds` page so the same metric's
threshold and device-source preference sit together instead of
one sidebar entry away. `/settings/sources` keeps a
`permanentRedirect` so external bookmarks and docs links follow
through unchanged. Section count: 11 → 10.
- **Insights tab strip: vital pills collapse under a "Vitals"
parent.** Five vital pills (HRV, Resting HR, Oxygen, Body
Temperature, Active Energy) hide behind one parent pill that
opens a popover sub-list. Parent-pill active state mirrors the
URL so spatial orientation survives the collapse; each sub-page
keeps its own URL and bookmark resolves unchanged. Strip
footprint: 14 → 10 entries when every metric has data.
- **Coolify env-var audit.** `mcp__coolify-apps01__env_vars`
inspection captured the section-1 / section-2 duplicates that
have accumulated under apps-01 since v1.3.1. Audit pinned at
`.planning/round-v1434-iwa-coolify-env-audit.md`; no env-var
deletes performed (operator action).
### Fixed
- **Compliance classifier: early intakes count as compliant.** The
classifier picked up a dedicated `early` bucket (window-start
minus three hours through window-start), the `on_time` post-window
tolerance widened from one hour to three hours, and the `late`
band sits between the new `on_time` ceiling and the configurable
`lateMinutes` knob. Heatmap consumers route the new bucket
through the compliant path; the dedicated `early` counter rides
on `DailyComplianceEntry` for downstream consumers that want to
distinguish ahead-of-window from on-window intakes.
- **Compliance heatmap fallback retired.** The v1.4.33 defensive
`looksClassifierBug` fallthrough is gone now that the underlying
classifier widening removes the every-dose-very-late mode the
fallback covered.
### Performance
- **`/api/gamification/achievements` consumer collapse.** Two of
three previous consumers shared the same TanStack queryKey
literal; the third carried a per-user discriminator so the cache
treated it as a fresh slot. Dashboard cold mount fired the
endpoint twice. Collapsing onto the shared hook trims that to a
single request; with the new server-side cache slot warm, the
duplicate-mount worst case rides single-flight coalescing.
- **GHCR build fires once per release tag.** The
`docker-publish.yml` workflow lost its `push.branches: [main]`
trigger; the `:latest` raw-tag enable rule moved onto the tag
ref so each release produces exactly one multi-arch build that
refreshes both the semver tags and the `:latest` alias.
- **NFT-trace warnings silenced.** `next.config.ts` gained an
`outputFileTracingExcludes` block that narrows the Turbopack
tracer away from the `MAXMIND_LICENSE_KEY` env-access path so
the trace-report no longer warns on every dependent route file.
### Refactor
- **Dashboard analytics consumers.** The two cards that previously
decided their own freshness copy now lean on the typed
`tileStaleDays()` helper and the additive `lastSeenByType` field
on the analytics response. Mood and BD-Zielbereich tiles keep the
default `null` because they have no underlying per-type freshness
signal.
### Accessibility
- **Dashboard hero CTA hits the WCAG 2.5.5 touch-target floor.**
The new "Ask the coach" pill carries `min-h-11` on mobile and
the matching `sm:min-h-9` desktop floor so the touch target
stays above 44 px on the Pixel 5 boundary.
- **Insights tab-strip Vitals parent reads correctly to assistive
tech.** Parent pill carries `aria-current="page"` whenever the
current URL matches one of its child vital sub-pages, plus a
`data-slot="insights-tab-strip-group"` hook for visual-regression
testing.
### Internal
- **Two e2e flake windows tightened.** `onboarding-flicker` swapped
its 12-sample 50 ms poll loop for a single Playwright
auto-retrying `toBeHidden({ timeout: 700 })` assertion so the
1-2 ms race window between the analytics-pending shell and
`useAuth().user` resolving collapses to a single retry slot.
`mobile-viewport` dropped `nav a[href]` from the touch-target
sweep (the bottom-nav owns its own WCAG spec) and gated the
44-px floor on `matchMedia('(min-width: 640px)').matches === false`
so the Pixel 5 viewport never tripped into the `sm:` tier during
WebKit render commits.
- **Server-cache observability.** Every `cached()` invocation passes
an `annotate` callback that lands two keys on the wide event:
`cache.<name>.outcome` ∈ `{ hit | miss | stampede }` and
`cache.<name>.key_hash` (non-reversible djb2 32-bit hash). Ops
can grep `cache.analytics.outcome` over any time window to
compute the hit ratio per deployment.
- **AASA followups verified.** The v1.4.33 `/.well-known/apple-app-site-association`
handler serves a direct 200 on every fronting origin; Apple's
CDN has ingested matching bodies for all three domains.
### Web freeze
v1.4.34 marks the last functional release of the web codebase before
the iOS native client launches. Subsequent v1.4.x tags carry security
fixes, dependency bumps, and hotfix-only corrections. New feature work
is paused until the iOS app clears Apple review, at which point a
v1.5.0 version-bump-only release tags the milestone. The Prisma
schema head comment and `.planning/v15-strategic-plan.md` §5
decision-log row pin the freeze trigger in-tree.
290 lines
9.2 KiB
TypeScript
290 lines
9.2 KiB
TypeScript
/**
|
|
* Stream a `multipart/form-data` body to disk without buffering the
|
|
* whole payload in memory.
|
|
*
|
|
* Next.js 16's `request.formData()` is fine up to ~10 MB but allocates
|
|
* the full body on a 1 GB upload. The Apple Health export ingest
|
|
* therefore reads `request.body` directly (a `ReadableStream<Uint8Array>`
|
|
* per Web Streams) and walks the multipart boundary by hand.
|
|
*
|
|
* Only the named `file` field of the upload is captured; any other
|
|
* field-name parts are read and discarded so the stream drains
|
|
* cleanly. The captured field's bytes are written to a temp file and
|
|
* a SHA-256 digest is computed in the same pass — the digest powers
|
|
* the content-hash idempotency check in the kick-off endpoint.
|
|
*
|
|
* Locks per `.planning/research/v1434-r-1-xml-import.md` §5.4.
|
|
*/
|
|
import { createHash, randomUUID } from "node:crypto";
|
|
import { createWriteStream } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import { join } from "node:path";
|
|
import { Writable } from "node:stream";
|
|
|
|
export interface MultipartFileResult {
|
|
/** Absolute path the captured file landed at. */
|
|
filePath: string;
|
|
/** Total bytes streamed. */
|
|
bytes: number;
|
|
/** SHA-256 hex digest of the captured bytes. */
|
|
sha256: string;
|
|
/** Optional original filename pulled from `Content-Disposition`. */
|
|
originalFilename: string | null;
|
|
/** Other named text fields encountered in the multipart body. */
|
|
textFields: Record<string, string>;
|
|
}
|
|
|
|
export interface MultipartStreamOptions {
|
|
/** Maximum allowed file size in bytes (hard cap; throws on overflow). */
|
|
maxBytes: number;
|
|
/** Required multipart field name to capture. */
|
|
fieldName: string;
|
|
/** Temp directory; defaults to OS temp dir. */
|
|
tmpDir?: string;
|
|
/** Optional filename prefix; defaults to `healthlog-upload`. */
|
|
tmpPrefix?: string;
|
|
}
|
|
|
|
const CRLF = Buffer.from("\r\n");
|
|
const DOUBLE_CRLF = Buffer.from("\r\n\r\n");
|
|
|
|
/**
|
|
* Parse the multipart boundary from a `Content-Type` header. Returns
|
|
* `null` if the header is missing or unparseable.
|
|
*/
|
|
export function parseBoundary(contentType: string | null): string | null {
|
|
if (!contentType) return null;
|
|
const match = contentType.match(/boundary=(?:"([^"]+)"|([^;,\s]+))/i);
|
|
if (!match) return null;
|
|
return match[1] ?? match[2] ?? null;
|
|
}
|
|
|
|
/**
|
|
* Stream a multipart body from `req.body` to disk, returning the
|
|
* captured file's path + a SHA-256 digest. Throws on:
|
|
* - Missing / unparseable boundary.
|
|
* - Missing required field.
|
|
* - Size cap exceeded.
|
|
*/
|
|
export async function streamMultipartToDisk(
|
|
body: ReadableStream<Uint8Array>,
|
|
contentType: string | null,
|
|
opts: MultipartStreamOptions,
|
|
): Promise<MultipartFileResult> {
|
|
const boundary = parseBoundary(contentType);
|
|
if (!boundary) {
|
|
throw new Error("Multipart body missing boundary parameter");
|
|
}
|
|
const boundaryMarker = Buffer.from(`--${boundary}`);
|
|
const tmpDirectory = opts.tmpDir ?? tmpdir();
|
|
const filePath = join(
|
|
tmpDirectory,
|
|
`${opts.tmpPrefix ?? "healthlog-upload"}-${randomUUID()}.bin`,
|
|
);
|
|
|
|
const hash = createHash("sha256");
|
|
let bytesWritten = 0;
|
|
let originalFilename: string | null = null;
|
|
const textFields: Record<string, string> = {};
|
|
|
|
// Lazily created sink — opens once we have entered the matching part.
|
|
let fileSink: Writable | null = null;
|
|
let fileSinkOpen = false;
|
|
|
|
const ensureFileSink = (): Writable => {
|
|
if (!fileSink) {
|
|
fileSink = createWriteStream(filePath);
|
|
fileSinkOpen = true;
|
|
}
|
|
return fileSink;
|
|
};
|
|
|
|
// Buffer holding bytes awaiting boundary scan. Capped at twice the
|
|
// boundary length so we never accumulate more than necessary.
|
|
let buffer = Buffer.alloc(0);
|
|
// Current parser state:
|
|
// "preamble" — before the first boundary
|
|
// "headers" — inside a part's header block
|
|
// "body-file" — inside the captured file's body
|
|
// "body-text" — inside a text field's body
|
|
// "epilogue" — after the closing boundary
|
|
type ParserState = "preamble" | "headers" | "body-file" | "body-text" | "epilogue";
|
|
let state: ParserState = "preamble";
|
|
let currentFieldName: string | null = null;
|
|
let currentTextBuffer: Buffer[] = [];
|
|
|
|
const writeFileChunk = (chunk: Buffer): Promise<void> => {
|
|
return new Promise((resolve, reject) => {
|
|
if (bytesWritten + chunk.length > opts.maxBytes) {
|
|
reject(
|
|
new Error(
|
|
`Upload exceeds size cap of ${opts.maxBytes} bytes`,
|
|
),
|
|
);
|
|
return;
|
|
}
|
|
bytesWritten += chunk.length;
|
|
hash.update(chunk);
|
|
const sink = ensureFileSink();
|
|
sink.write(chunk, (err) => {
|
|
if (err) reject(err);
|
|
else resolve();
|
|
});
|
|
});
|
|
};
|
|
|
|
const closeFileSink = (): Promise<void> => {
|
|
return new Promise((resolve) => {
|
|
if (!fileSink || !fileSinkOpen) {
|
|
resolve();
|
|
return;
|
|
}
|
|
fileSinkOpen = false;
|
|
fileSink.end(() => resolve());
|
|
});
|
|
};
|
|
|
|
const reader = body.getReader();
|
|
let streamDone = false;
|
|
|
|
// The classic SAX-style boundary scanner: accumulate into `buffer`,
|
|
// search for the boundary marker, emit bytes up to it.
|
|
const processBuffer = async (): Promise<void> => {
|
|
while (true) {
|
|
if (state === "preamble") {
|
|
const idx = buffer.indexOf(boundaryMarker);
|
|
if (idx === -1) {
|
|
// Need more bytes to find a boundary; retain a small tail
|
|
// so a boundary split across chunks is still detectable.
|
|
if (buffer.length > boundaryMarker.length) {
|
|
buffer = buffer.slice(buffer.length - boundaryMarker.length);
|
|
}
|
|
return;
|
|
}
|
|
// Drop everything up to and including the boundary marker.
|
|
buffer = buffer.slice(idx + boundaryMarker.length);
|
|
// After the first boundary we expect "\r\n" or "--" (closing).
|
|
if (buffer.length < 2) return;
|
|
if (buffer[0] === 0x2d && buffer[1] === 0x2d) {
|
|
state = "epilogue";
|
|
return;
|
|
}
|
|
if (buffer[0] === 0x0d && buffer[1] === 0x0a) {
|
|
buffer = buffer.slice(2);
|
|
state = "headers";
|
|
continue;
|
|
}
|
|
return;
|
|
}
|
|
|
|
if (state === "headers") {
|
|
const eoh = buffer.indexOf(DOUBLE_CRLF);
|
|
if (eoh === -1) return;
|
|
const headerBlock = buffer.slice(0, eoh).toString("utf8");
|
|
buffer = buffer.slice(eoh + DOUBLE_CRLF.length);
|
|
|
|
const disposition = headerBlock
|
|
.split(/\r?\n/)
|
|
.find((l) => l.toLowerCase().startsWith("content-disposition:"));
|
|
if (!disposition) {
|
|
// Unparseable part — skip until next boundary as text.
|
|
state = "body-text";
|
|
currentFieldName = null;
|
|
continue;
|
|
}
|
|
const nameMatch = disposition.match(/name="([^"]+)"/i);
|
|
const filenameMatch = disposition.match(/filename="([^"]*)"/i);
|
|
currentFieldName = nameMatch?.[1] ?? null;
|
|
if (currentFieldName === opts.fieldName) {
|
|
state = "body-file";
|
|
originalFilename = filenameMatch?.[1] ?? null;
|
|
} else {
|
|
state = "body-text";
|
|
currentTextBuffer = [];
|
|
}
|
|
continue;
|
|
}
|
|
|
|
// body-file / body-text: emit bytes up to the next CRLF--boundary.
|
|
const sep = Buffer.concat([CRLF, boundaryMarker]);
|
|
const idx = buffer.indexOf(sep);
|
|
if (idx === -1) {
|
|
// Emit everything except a trailing slice that might contain
|
|
// the start of the next boundary marker.
|
|
const keep = Math.min(buffer.length, sep.length - 1);
|
|
const emit = buffer.slice(0, buffer.length - keep);
|
|
if (emit.length > 0) {
|
|
if (state === "body-file") {
|
|
await writeFileChunk(emit);
|
|
} else {
|
|
currentTextBuffer.push(emit);
|
|
}
|
|
}
|
|
buffer = buffer.slice(buffer.length - keep);
|
|
return;
|
|
}
|
|
|
|
const emit = buffer.slice(0, idx);
|
|
if (emit.length > 0) {
|
|
if (state === "body-file") {
|
|
await writeFileChunk(emit);
|
|
} else {
|
|
currentTextBuffer.push(emit);
|
|
}
|
|
}
|
|
// Finalise the text field if applicable.
|
|
if (state === "body-text" && currentFieldName) {
|
|
textFields[currentFieldName] = Buffer.concat(
|
|
currentTextBuffer,
|
|
).toString("utf8");
|
|
currentTextBuffer = [];
|
|
}
|
|
if (state === "body-file") {
|
|
await closeFileSink();
|
|
}
|
|
buffer = buffer.slice(idx + sep.length);
|
|
// After the boundary we expect "\r\n" or "--".
|
|
if (buffer.length < 2) {
|
|
state = "preamble";
|
|
return;
|
|
}
|
|
if (buffer[0] === 0x2d && buffer[1] === 0x2d) {
|
|
state = "epilogue";
|
|
return;
|
|
}
|
|
if (buffer[0] === 0x0d && buffer[1] === 0x0a) {
|
|
buffer = buffer.slice(2);
|
|
state = "headers";
|
|
continue;
|
|
}
|
|
return;
|
|
}
|
|
};
|
|
|
|
while (!streamDone) {
|
|
const { value, done } = await reader.read();
|
|
if (done) {
|
|
streamDone = true;
|
|
break;
|
|
}
|
|
buffer = Buffer.concat([buffer, Buffer.from(value)]);
|
|
await processBuffer();
|
|
}
|
|
await processBuffer();
|
|
await closeFileSink();
|
|
|
|
if (bytesWritten === 0) {
|
|
throw new Error(
|
|
`Multipart body did not include a '${opts.fieldName}' file field`,
|
|
);
|
|
}
|
|
|
|
return {
|
|
filePath,
|
|
bytes: bytesWritten,
|
|
sha256: hash.digest("hex"),
|
|
originalFilename,
|
|
textFields,
|
|
};
|
|
}
|