tom/HealthLog
1
0 Fork 0
mirror of https://github.com/MBombeck/HealthLog.git synced 2026-11-08 20:42:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(planning): v1.4.34 IW reports + cache-aggregation research

Browse Source 
Backfills the planning trail for v1.4.34: IW-C / IW-D / IW-G work-item
reports, the production slowness HAR investigation, the AASA rollout
trace from v1.4.33, and three pieces of research that informed the
follow-on perf work (XML import, carryover scope, cache aggregation).

The cache-aggregation research (research/v1434-r-cache-aggregation.md)
designed the per-route LRU pattern that IW-G shipped and is the
direct ancestor of the comprehensive-route rewrite this release lands.
This commit is contained in:
Marc-André Bombeck committed 2026-05-16 22:25:06 +02:00
1 parent 7bbcc872f8
commit f535eb0e42
8 files changed
+2484
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.planning/research/v1434-r-1-xml-import.md
+743
View File
@@ -0,0 +1,743 @@
# R-1 — Apple Health `export.zip` import (v1.4.34)
Implementation blueprint for the streaming Apple Health XML import that
lands immediately before the web-freeze marker. The brief in
`.planning/v15-strategic-plan.md` §2 carves the scope; this document
fills in the parser pick, the queue model, the parse loop, the
ingestion shape, the response envelope, the admin variant, and the
freeze-marker wording. Every section names files, dependencies, and
the upstream code the implementation reuses.
## 1. Recommendation summary
- Parser pick: `sax` (event-driven SAX, no DOM, peak RSS independent
of file size). `fast-xml-parser` already shipped transitively in
`pnpm-lock.yaml` line 3955 but its streaming mode buffers per
element which still allocates for the ~10 M `<Record>` peak inside
a multi-year export. `sax` is purpose-built for "drain a large XML
file with bounded memory". Bundle is ~30 kB.
- Archive unpack: Node 22 ships a `node:zlib` `inflateRawSync` that
pairs with a small inline ZIP central-directory walker; we add no
new archive dependency. The implementation extracts `export.xml`
directly from the upload stream rather than materialising the full
zip on disk.
- Async-job storage pick: `pg-boss` (already in dependencies at
`^12.18.2`, full worker scaffolding live in
`src/lib/jobs/reminder-worker.ts`). Reuse the existing
`boss.send` / `boss.work` / `boss.getJobById` surface; no new
process, no new infra footprint.
- Job phases: `queued → unpacking → parsing → upserting → done` with
a single `failed` terminal. Per-phase progress carried on the
`pg-boss` job `output` JSON column (pg-boss v12 supports incremental
`boss.complete(jobId, output)` only at completion; we therefore
also maintain a small `ImportJob` Prisma row keyed by `pg-boss`
`jobId` for live progress reads).
- Effort: L (~3-4 days) — matches §2 of the strategic plan.
## 2. `export.zip` archive layout
Every Apple Health export the user creates via *Health.app → profile
picture → Export All Health Data* unzips to a single top-level
`apple_health_export/` directory. Members observed across every
ecosystem-scan reference (`apple-health-grafana`,
`healthkit-to-sqlite`, `apple-health-parser`, `BRO3886/healthsync`,
`atlas`, `health-data-hub`):
| Member | Purpose | v1.4.34 handling |
|---|---|---|
| `apple_health_export/export.xml` | The full sample stream — ~95 % of payload by bytes. Single self-contained XML file. Every `<Record>`, `<Workout>`, `<ActivitySummary>`, `<Correlation>`, `<ClinicalRecord>` lives here. | **Primary** parse target. |
| `apple_health_export/export_cda.xml` | Continuity-of-Care Document (HL7 CDA R2). Duplicate of the clinical subset already mirrored as `<ClinicalRecord>` rows in `export.xml`. | Ignore — clinical defers per R-F T3. |
| `apple_health_export/workout-routes/route_*.gpx` | One GPX file per `HKWorkoutRoute` (referenced from the matching `<Workout>` via `<FileReference path="workout-routes/route_2024-05-01_15.05.gpx"/>`). | Parse if the matching `<Workout>` survives mapping; persist as `WorkoutRoute.geometry` GeoJSON. |
| `apple_health_export/electrocardiograms/ecg_*.csv` | One CSV per HKElectrocardiogram. | Skip — no HealthLog ECG model; record count under `unknown.electrocardiogram`. |
| `apple_health_export/clinical-records/*.json` | FHIR R4 resources mirroring `<ClinicalRecord>` blocks. | Skip — defers per R-F T3. |
| `apple_health_export/export.zip.checksum` | Optional integrity marker (recent iOS versions). | Ignore. |
The parser logs every unrecognised member name under `unknown.<name>`
in the response so we can spot new directory members on future iOS
releases without breaking the import.
## 3. `export.xml` schema (parser-facing data model)
The single XML file opens with the literal sequence:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE HealthData [
<!ELEMENT HealthData (ExportDate, Me, (Record|Correlation|Workout|ActivitySummary|ClinicalRecord)*)>
...
]>
<HealthData locale="en_US">
<ExportDate value="2026-05-15 14:32:01 +0200"/>
<Me HKCharacteristicTypeIdentifierDateOfBirth="1985-06-12" .../>
<Record type="HKQuantityTypeIdentifierStepCount"
sourceName="Marc's iPhone"
sourceVersion="17.4.1"
device="<<HKDevice: 0x281a45fc0>, name:iPhone, manufacturer:Apple Inc., model:iPhone, hardware:iPhone15,2, software:17.4.1>"
unit="count"
creationDate="2026-05-14 08:14:23 +0200"
startDate="2026-05-14 08:13:00 +0200"
endDate="2026-05-14 08:14:00 +0200"
value="142">
<MetadataEntry key="HKMetadataKeyExternalUUID" value="A3F9D8E2-..."/>
</Record>
...
</HealthData>
```
Element model the parser must handle:
- `<HealthData>` — root, single instance. Attribute `locale` is
advisory.
- `<ExportDate>` — single `value` attribute (string-formatted local
date with timezone offset). Stored on the import job for audit.
- `<Me>` — characteristic attributes (DOB, biological sex, blood
type, Fitzpatrick skin type, wheelchair-use). v1.4.34 ignores;
HealthLog has its own profile fields and overwriting them mid-
import would surprise the user.
- `<Record type="..."/>` — the bulk of the stream. The `type`
attribute carries the literal `HKQuantityTypeIdentifier*` or
`HKCategoryTypeIdentifier*` string already handled by
`APPLE_HEALTH_TYPE_MAP` in
`src/lib/measurements/apple-health-mapping.ts`. Every Record
carries `startDate`, `endDate`, `value`, `unit`, `sourceName`,
`sourceVersion`, `creationDate`, `device`. Sleep-stage Records
carry `value` as a string codepoint (`"HKCategoryValueSleepAnalysis"
+ "InBed" | "Asleep" | "Awake" | "AsleepCore" | "AsleepDeep" |
"AsleepREM"`); the parser converts the string back to the integer
codepoint via a small table so it can ride
`APPLE_HEALTH_SLEEP_STAGE_MAP`. Children: zero or more
`<MetadataEntry key="..." value="..."/>`. The `HKMetadataKeyExternalUUID`
entry, when present, is the `HKSample.uuid`; otherwise the parser
hashes `(type, value, startDate, endDate)` to a stable
`externalId`.
- `<Workout workoutActivityType="..."/>` — one per `HKWorkout`.
Attributes: `workoutActivityType` (the `HKWorkoutActivityType`
string, e.g. `HKWorkoutActivityTypeRunning`), `duration`,
`durationUnit`, `totalDistance`, `totalDistanceUnit`,
`totalEnergyBurned`, `totalEnergyBurnedUnit`, `sourceName`,
`sourceVersion`, `device`, `creationDate`, `startDate`, `endDate`.
Children: `<MetadataEntry/>` (zero or more, `HKMetadataKeyExternalUUID`
again the natural id), `<WorkoutEvent type="HKWorkoutEventType*"
date="..." duration="..."/>` (pause / resume / lap markers),
`<WorkoutRoute startDate="..." endDate="..."/>` with one nested
`<FileReference path="workout-routes/route_*.gpx"/>` linking to the
GPX file.
- `<ActivitySummary dateComponents="2026-05-14"
activeEnergyBurned="412" activeEnergyBurnedGoal="500"
activeEnergyBurnedUnit="kcal" appleMoveTime="0"
appleMoveTimeGoal="0" appleExerciseTime="38"
appleExerciseTimeGoal="30" appleStandHours="11"
appleStandHoursGoal="12"/>` — Apple's daily ring rollup. v1.4.34
ignores (HealthLog computes its own daily aggregates from
`<Record>` rows via the existing `dailyStatsExternalId` path).
- `<Correlation type="HKCorrelationTypeIdentifierBloodPressure"
startDate="..." endDate="..."/>` with two nested `<Record>` children
(`HKQuantityTypeIdentifierBloodPressureSystolic` +
`HKQuantityTypeIdentifierBloodPressureDiastolic`). The parser
flattens the Correlation envelope and treats each child Record
exactly like a top-level Record — the systolic / diastolic mapping
in `APPLE_HEALTH_TYPE_MAP` already handles the row shape.
- `<ClinicalRecord type="HKClinicalTypeIdentifier*" identifier="..."
fhirResource="...">...</ClinicalRecord>` — FHIR clinical records.
Skipped per R-F T3 (defers to v1.6+).
## 4. Parser pick
Three Node candidates were considered:
| Lib | Mode | Peak RSS at 1 GB input | TypeScript | Existing dep? |
|---|---|---|---|---|
| `sax` | event SAX (callback per token) | ~30 MB constant | ships `.d.ts` from `@types/sax` | no (one new dep) |
| `fast-xml-parser` (streaming mode) | per-element callback over a `XMLParser` with `processEntities: false` | ~150 MB at 1 GB — buffers each element | first-class TS | yes (transitive) |
| `node-expat` | C++ libexpat bindings | ~25 MB constant | manual `.d.ts` | no — requires node-gyp build |
Pick: **`sax` (~0.4 MB install size, MIT, last release 2024-06)**. The
SAX callback model gives us per-token streaming (each
`onopentag` / `onclosetag` / `ontext` fires as the parser advances
the byte cursor), peak RSS stays ~30 MB regardless of file size,
TypeScript types exist as `@types/sax`. `node-expat` would be marginally
faster but the node-gyp build step breaks the Coolify Docker image
(we explicitly keep the build chain extension-free per the same
decision that picked GeoJSON over PostGIS for `WorkoutRoute`). The
transitive `fast-xml-parser` stays where it is for HTML-ish
fast-path usage elsewhere; we don't add it to the import path.
`pnpm add sax @types/sax` lands two new lines in `package.json`. Both
under 50 kB unpacked.
## 5. Async-job architecture
Reuse the live `pg-boss` infrastructure verbatim. The implementation
adds two queues and one Prisma model:
### 5.1 New `pg-boss` queue: `apple-health-import`
Register in `src/lib/jobs/reminder-worker.ts:1432-1454` next to the
existing `MEDICATION_INVENTORY_EXPIRE_QUEUE` block. Concurrency cap
`1` per host (one giant XML at a time per worker — the parse loop is
CPU-bound and an OOM on a second concurrent 1 GB import would knock
the host over). Retry policy: `retryLimit: 0` (a re-import is a
human-driven re-upload, not an automatic retry). Job payload:
```ts
interface AppleHealthImportPayload {
/** Owner of the imported rows. The job runs in this user's scope. */
userId: string;
/** Operator who triggered the import (admin variant only). When unset,
* the user kicked off their own import. */
triggeredByAdminId?: string;
/** Absolute path on the worker's filesystem where the multipart-upload
* handler dropped the `export.zip`. The worker streams from here and
* unlinks on completion. */
uploadPath: string;
/** Size in bytes for the response envelope + audit-log row. */
uploadBytes: number;
/** Wall-clock kick-off so duration is computable even when the worker
* picks the job up after a queue delay. */
enqueuedAt: string;
}
```
The handler in `src/lib/jobs/apple-health-import-worker.ts` (new
file) follows the `pr-detection-worker.ts` shape: one exported
`handleAppleHealthImport(job: PgBossJob<AppleHealthImportPayload>)`
function plus a small unit-tested helper extracted to
`src/lib/measurements/parse-export-xml.ts`.
### 5.2 New Prisma model: `ImportJob`
`pg-boss` v12's `boss.getJobById` returns the boss row but the
`output` column only populates on terminal completion. For the
polling endpoint to read *live* progress (the user is staring at a
spinner), we mirror the job into a HealthLog-side row:
```prisma
model ImportJob {
id String @id @default(cuid())
userId String @map("user_id")
triggeredByAdminId String? @map("triggered_by_admin_id")
pgBossJobId String @unique @map("pg_boss_job_id")
/// `queued | unpacking | parsing | upserting | done | failed`
status String @default("queued")
/// Free-text reason on `failed`; null otherwise.
failureReason String? @map("failure_reason")
uploadBytes Int @map("upload_bytes")
startedAt DateTime @map("started_at") @default(now())
completedAt DateTime? @map("completed_at")
/// JSON envelope matching `ImportJobProgress` (§8). Updated mid-run
/// from the worker on every 1 000 records parsed.
progress Json @default("{}")
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
@@index([userId, startedAt])
@@map("import_jobs")
}
```
The unique constraint on `pgBossJobId` lets the polling endpoint
read by the boss id the client received from the synchronous
endpoint. The migration is additive only (no existing column changes).
### 5.3 Synchronous endpoint — kick-off
`POST /api/import/apple-health-export` (new file
`src/app/api/import/apple-health-export/route.ts`):
1. `requireAuth()` from `src/lib/api-handler.ts`. Cookie or Bearer
(any scope — wildcard works per v1.4.25 W10 fix-C).
2. `checkRateLimit("import:apple-health:${user.id}", 3, 60_000)` —
three uploads per minute per user is generous for any legitimate
workflow.
3. `Content-Length` ceiling: `MAX_UPLOAD_BYTES = 1.5 * 1024 * 1024 *
1024` (1.5 GB — Apple's largest observed export is around 800 MB
for a 10-year iCloud-synced account; 1.5 GB leaves margin).
Reject 413 above. **No** in-memory `request.formData()` — Next.js
16's built-in handler buffers the full body. Switch to streaming
via the experimental Web Streams `request.body` plus a manual
multipart boundary walker (see §5.4 for the streaming path).
4. Hash the upload into `/tmp/healthlog-import-<cuid>.zip` while the
stream lands. Compute a SHA-256 inline so re-uploading the exact
same file returns the previous `ImportJob` id without re-queueing
(idempotency-on-content rather than `Idempotency-Key` — the iOS
client cannot reliably emit a stable key for a giant file).
5. `prisma.importJob.create()` with `status: "queued"`.
6. `boss.send("apple-health-import", payload)` returns the boss
`jobId`. Store on the `ImportJob` row.
7. Respond `202 Accepted` with `{ jobId: importJob.id, status:
"queued" }`. The 202 is the contract the strategic plan §2 locks.
### 5.4 Multipart streaming notes
Next.js 16's `request.formData()` is fine up to ~10 MB but allocates
the full body on a 1 GB upload. The kick-off route therefore reads
`request.body` directly (a `ReadableStream<Uint8Array>` per Web
Streams) and walks the multipart boundary by hand — pattern identical
to the one shipped in `src/app/api/admin/backups/upload/route.ts:85`
for backup uploads, but extended to stream-to-disk instead of
`file.text()`. A small helper
`src/lib/multipart/stream-to-disk.ts` extracts the named field and
writes it to a temp file with a SHA-256 hash computed in the same
pass.
### 5.5 Status endpoint
`GET /api/import/apple-health-export/[jobId]/status` (new file
`src/app/api/import/apple-health-export/[jobId]/status/route.ts`):
1. `requireAuth()`.
2. `prisma.importJob.findUnique({ where: { id: jobId } })`.
3. 404 when missing or `userId !== requester.id` (admin variant
short-circuits this — see §9).
4. Reply with the envelope in §8. Caching: `Cache-Control: no-store`.
Polling cadence: client polls every 2 s during `queued | unpacking |
parsing | upserting`, every 30 s once `done | failed` (last-known
state is fine).
## 6. Worker process and parse loop
The handler lives at
`src/lib/jobs/apple-health-import-worker.ts`. Pseudocode:
```ts
import sax from "sax";
import { createReadStream, unlinkSync, statSync } from "node:fs";
import { mapAppleHealthEntry, APPLE_HEALTH_TYPE_MAP, dailyStatsExternalId,
CUMULATIVE_HK_TYPES } from "@/lib/measurements/apple-health-mapping";
export async function handleAppleHealthImport(job) {
const { userId, uploadPath, uploadBytes } = job.data;
const importJob = await prisma.importJob.findUnique({
where: { pgBossJobId: job.id },
});
if (!importJob) throw new Error("ImportJob row missing");
await tick(importJob.id, "unpacking", { recordsRead: 0 });
const xmlPath = await unzipExportXml(uploadPath); // see §6.1
await tick(importJob.id, "parsing", { recordsRead: 0 });
const stats = await streamParse(xmlPath, userId); // see §6.2
await tick(importJob.id, "upserting", stats);
await flushUpserts(userId, stats); // see §6.3
await tick(importJob.id, "done", stats);
unlinkSync(uploadPath);
unlinkSync(xmlPath);
}
```
### 6.1 Unzip step
Streaming-unzip the upload's `apple_health_export/export.xml` into
`/tmp/healthlog-import-<cuid>.xml`. Node 22's `node:stream/promises`
+ a tiny ZIP central-directory walker — see
`https://github.com/MichalLytek/yauzl/blob/master/lib/yauzl.js` for a
reference implementation. The walker is ~60 LOC; we inline it under
`src/lib/import/unzip-export-xml.ts` rather than pull a new
dependency. Also extracts `workout-routes/*.gpx` to a temp directory
for later resolution.
### 6.2 Stream parse step
```ts
const parser = sax.parser(/* strict */ true, { trim: true });
const buffer: PreparedRow[] = [];
const stats: PerTypeStats = freshStats();
const cumulativeBucket = new Map<string, number>(); // key = type:date
parser.onopentag = (node) => {
if (node.name === "Record") {
const r = mapAppleHealthEntry({
hkIdentifier: node.attributes.type,
value: parseRecordValue(node.attributes),
unit: node.attributes.unit ?? "",
startDate: node.attributes.startDate,
endDate: node.attributes.endDate,
sleepStage: parseSleepStage(node.attributes.value), // §3
});
if (!r) {
stats.unknown[node.attributes.type] = (stats.unknown[node.attributes.type] ?? 0) + 1;
return;
}
if (CUMULATIVE_HK_TYPES.has(r.type)) {
// Fold into per-day bucket; flush bucket to DB at end of file
const dayKey = formatDayKey(r.takenAt, user.timezone);
const bucketKey = `${r.type}:${dayKey}`;
cumulativeBucket.set(bucketKey, (cumulativeBucket.get(bucketKey) ?? 0) + r.value);
} else {
buffer.push(prepareSpotRow(userId, r, node));
if (buffer.length >= 500) flushBatch(buffer);
}
stats.perType[r.type].read += 1;
} else if (node.name === "Workout") {
/* drain into prepareWorkoutRow + flushWorkoutBatch at 100 */
} else if (node.name === "ClinicalRecord") {
stats.clinical.skipped += 1;
} else if (node.name === "ActivitySummary") {
// ignore — daily-stats path mints rows from <Record>
}
};
parser.onerror = (err) => { /* mark failed */ };
parser.onend = async () => { await flushAllBuffers(); };
await pipeline(createReadStream(xmlPath, { highWaterMark: 64 * 1024 }),
saxAsWritable(parser));
```
Implementation notes:
- Sleep-stage `<Record>` rows carry their value as the symbolic name
`HKCategoryValueSleepAnalysisAsleepDeep`. Translate via a small
`SLEEP_STAGE_NAME_TO_CODEPOINT` table that mirrors
`APPLE_HEALTH_SLEEP_STAGE_MAP` inverted; the codepoint then rides
the existing `mapAppleHealthEntry()` path. For sleep, the row's
numeric `value` is `(endDate - startDate) / 60_000` minutes (the
same convention `mapAppleHealthEntry` expects).
- `externalId` derivation:
- Spot row, `HKMetadataKeyExternalUUID` present → `sample:<uuid>`.
- Spot row, no external UUID → `sample:<sha256(type|value|startDate|endDate)>`
truncated to 28 chars (~110 bit collision budget; well inside
the `min(1).max(120)` Zod cap on `externalId`).
- Cumulative row (after the per-day fold) →
`dailyStatsExternalId(hkIdentifier, "YYYY-MM-DD")`.
- The day-key for cumulative folds anchors to the user's timezone
(`User.timezone`) — same convention
`src/lib/measurements/drain-per-sample-cumulative.ts` uses via
`dayKeyForUserTz`. This makes the imported `stats:` rows
interoperate cleanly with iOS-emitted `stats:` rows: the same day
collides on the same `externalId` so a re-import after an iOS
cutover is a true UPSERT.
### 6.3 Upsert / flush step
The flush helpers run inside a loop of small Prisma transactions —
one tx per 500 spot rows or per 100 workout rows. UPSERT shape:
```ts
await prisma.measurement.upsert({
where: {
userId_type_source_externalId: {
userId,
type: row.type,
source: "APPLE_HEALTH",
externalId: row.externalId,
},
},
create: row,
update: { value: row.value, measuredAt: row.measuredAt },
});
```
The `userId_type_source_externalId` compound unique already exists
on `Measurement` at `prisma/schema.prisma:444` so no migration. For
the cumulative-bucket flush, the value written is the SUM across the
day; updating an existing row replaces the SUM (matches the drain
script's semantics — a re-import is a fresh authoritative roll-up).
Workouts UPSERT against `userId_source_externalId` at
`prisma/schema.prisma:512`. Sport-type mapping from
`HKWorkoutActivityType*` to the `workoutSportTypeEnum` Zod union
(`src/lib/validations/workout.ts:21-42`) lives in a new helper
`src/lib/measurements/hk-workout-activity-type-map.ts` — same
flat-table shape as `APPLE_HEALTH_TYPE_MAP`. Unknown activity types
fall through to `"other"` (the union's escape hatch). Unrecognised
counts land in `stats.workouts.unknownActivityType`.
`WorkoutRoute.geometry` is materialised by parsing the linked GPX
file into a GeoJSON LineString — same encoding the existing
`pickCanonicalWorkoutRows` consumes. GPX parsing rides the same `sax`
instance (a tiny inline `<trkseg><trkpt lat="..." lon="..."/></trkseg>`
reducer).
### 6.4 Resource ceiling
The handler caps total RSS by:
- streaming sax (no DOM)
- flush batches at 500 spot rows / 100 workouts
- the cumulative bucket holds at most one `(type, day)` entry per
observed day per cumulative type — a 10-year export with five
cumulative types and 3 650 days = 18 250 entries, ~1 MB
- no full-file buffer, ever
Empirical worst case: a 1 GB `export.xml` with 12 M `<Record>` rows
fits inside a 256 MB container heap with margin.
### 6.5 Progress reporting
The worker calls a `tick()` helper every 1 000 records read that
writes a partial-update onto `ImportJob.progress` (`prisma.importJob.update`).
The update is a single JSON column write and rate-limited internally
so a 12 M record run causes at most ~12 000 update queries (one per
1 000 records), which is acceptable.
## 7. Mapping reference
The import reuses the canonical lookup table at
`src/lib/measurements/apple-health-mapping.ts` verbatim. Concretely:
- **Spot quantities** (`aggregation: "latest" | "mean"`): pass each
Record through `mapAppleHealthEntry()` and persist one
`Measurement` row per sample. Coverage: WEIGHT, BODY_FAT,
BODY_TEMPERATURE, BLOOD_PRESSURE_SYS/DIA, PULSE,
RESTING_HEART_RATE, HEART_RATE_VARIABILITY, VO2_MAX,
BLOOD_GLUCOSE, OXYGEN_SATURATION, AUDIO_EXPOSURE_ENV,
AUDIO_EXPOSURE_HEADPHONE, WALKING_STEADINESS.
- **Cumulative quantities** (`CUMULATIVE_HK_TYPES`): fold per
`(type, user-local day)` and persist one `stats:...` row per
bucket. Coverage: ACTIVITY_STEPS, ACTIVE_ENERGY_BURNED,
FLIGHTS_CLIMBED, WALKING_RUNNING_DISTANCE, TIME_IN_DAYLIGHT.
- **Category types** (sleep, audio events): `mapAppleHealthEntry()`
already absorbs HKCategoryTypeIdentifierSleepAnalysis +
HKCategoryTypeIdentifierEnvironmentalAudioExposureEvent +
HKCategoryTypeIdentifierHeadphoneAudioExposureEvent.
- **HKWorkout**: HKWorkoutActivityType → `WorkoutSportType` via a
new map at `src/lib/measurements/hk-workout-activity-type-map.ts`;
every other field reads directly off Workout attributes.
- **HKClinicalRecord**: skipped, counted under
`stats.clinical.skipped`.
- **HKElectrocardiogram**: skipped (no model in v1.5), counted under
`stats.unknown["HKElectrocardiogramType"]`.
Identifiers Apple ships that the table does not know about land in
`stats.unknown[<HKIdentifier>]`. The deferred set
(`HK_QUANTITY_TYPE_DEFERRED`) is enumerated; everything outside is a
true unknown. The response surfaces both buckets separately so
operators can spot a new iOS release without breaking ingestion.
## 8. Per-type stats response shape
`GET /api/import/apple-health-export/[jobId]/status` returns:
```ts
interface ImportJobStatusResponse {
jobId: string;
status: "queued" | "unpacking" | "parsing" | "upserting" | "done" | "failed";
startedAt: string; // ISO-8601
completedAt: string | null; // ISO-8601 once status === "done" | "failed"
uploadBytes: number;
/** When known — Apple's ExportDate attribute string-formatted. */
exportedAt: string | null;
/** Live-updated by the worker. */
progress: {
currentPhase: "unpacking" | "parsing" | "upserting";
/** Total <Record> + <Workout> elements seen so far. */
recordsRead: number;
/** Total rows committed so far. */
rowsUpserted: number;
/** Best-effort percent — only populated once the parser has seen
* the closing </HealthData> tag and recordsTotal is known. */
percent: number | null;
/** ms since `startedAt`. */
elapsedMs: number;
};
/** Final on `done` | `failed`; partial otherwise. */
result: {
perType: Record<
MeasurementType,
{ read: number; inserted: number; updated: number; durationMs: number }
>;
workouts: {
read: number;
inserted: number;
updated: number;
unknownActivityType: number;
routesAttached: number;
durationMs: number;
};
clinical: { skipped: number };
deferred: Record<string, number>; // HK_QUANTITY_TYPE_DEFERRED hits
unknown: Record<string, number>; // outside the defer set
totals: {
recordsRead: number;
rowsUpserted: number;
durationMs: number;
};
} | null;
failureReason: string | null;
}
```
`apiSuccess(...)` wraps the response in the standard envelope. The
shape is additive over the strategic-plan §2 sketch — `inserted`
splits into `inserted + updated` because UPSERT-vs-INSERT-only
matters for the operator audit (a re-import shows 0 inserts and N
updates; a fresh import shows the opposite).
## 9. Admin variant
`POST /api/admin/import-apple-health-export` (new file
`src/app/api/admin/import-apple-health-export/route.ts`):
1. `requireAdmin()` — cookie-only per the v1.4.25 security
boundary (`src/lib/api-handler.ts:414-428`). No Bearer ever
elevates.
2. Body `multipart/form-data`: same `file` field + a `userId`
string field naming the target user. Reject 422 if userId is
absent.
3. `prisma.user.findUnique({ where: { id: body.userId } })` —
reject 404 if missing (same pattern as the backup-upload route).
4. Same upload-streaming path, same temp-file hash, same
`ImportJob.create` shape — but with `triggeredByAdminId =
admin.id`. The status endpoint reads ownership by `userId` OR
`triggeredByAdminId` so the admin can poll their own kick-off.
5. `auditLog("admin.import-apple-health.start", ...)` and
`auditLog("admin.import-apple-health.complete", ...)` flank the
worker call — same shape as the drain endpoint at
`src/app/api/admin/drain-per-sample-cumulative/route.ts:56-77`.
The admin variant **enqueues the same `apple-health-import` queue**.
One worker handler, two kick-off paths.
## 10. Web-freeze marker wording
### CHANGELOG.md head, under `## [1.4.34]`
> **Web freeze begins after this release.** v1.4.34 closes the
> web-side scope for the v1.5 cycle. Subsequent v1.4.x tags are
> limited to security patches, dependency updates, and tightly
> scoped reactive fixes if iOS testing surfaces a real gap on a
> v1.4.34 endpoint. No new web features, UI rewrites, or non-additive
> schema changes until the iOS native client clears Apple review and
> the v1.5.0 version-bump marker tags on `main`.
### `prisma/schema.prisma` head comment, right under the generator
block (around `prisma/schema.prisma:1`):
> // v1.4.34 — Web schema enters freeze. Additive-only changes
> // (new optional columns, new models, new indices) remain
> // allowed; non-additive changes (rename, drop, type change,
> // constraint tightening) defer to the post-v1.5.0 cycle.
### `.planning/v15-strategic-plan.md` decision log (append a row
under the existing "Web freeze trigger" entry around line 388):
> | Web freeze in effect | v1.4.34 deploy on `main` | This plan §2,
> CHANGELOG line, prisma/schema.prisma head comment |
## 11. Test surface
`__tests__/apple-health-import-worker.test.ts` (Vitest):
1. **Tiny fixture** — 20-row hand-authored `export.xml` covering one
spot type, one cumulative type, one sleep stage, one workout
with a 3-point GPX route, one unknown identifier, one deferred
identifier. End-to-end assertions on `perType.read /
inserted / updated`, on the `stats:...` externalId for the
cumulative type, on the GeoJSON LineString for the route.
2. **Re-import idempotency** — run the importer twice over the same
fixture; assert second run reports 0 inserts and N updates.
3. **Unknown identifier surfacing** — assert
`result.unknown["HKQuantityTypeIdentifierFutureTypeXYZ"] === 1`.
4. **Deferred identifier surfacing** — assert a
`HK_QUANTITY_TYPE_DEFERRED` member lands under `deferred`, not
`unknown`.
5. **Streaming memory ceiling** — synthetic 50 MB `export.xml` (the
CI tier can't afford a 1 GB fixture); assert peak heap stays
below 100 MB via `process.memoryUsage().heapUsed`.
`__tests__/api-import-apple-health-export.test.ts` (Vitest /
integration):
1. Multipart upload accepted with 202 + `{ jobId }`.
2. Status endpoint returns the canonical envelope.
3. Cross-user access returns 404 (per `userId` scoping).
4. Admin endpoint enforces `requireAdmin()` (Bearer denied).
5. Rate limit returns 429 after three uploads in a minute.
## 12. Operational + Coolify implications
The `pg-boss` worker is already configured on the Coolify deploy
under `HEALTHLOG_PROCESS_TYPE` (web | worker | all per
`src/lib/process-type.ts:7-15`). No new container, no new env vars,
no new secrets. Two operational notes:
- Temp-file disk usage. The worker writes the upload (up to 1.5 GB)
+ the unzipped `export.xml` (up to 4 GB uncompressed) +
`workout-routes/*.gpx` (up to ~50 MB) under `/tmp`. The Coolify
app container's `/tmp` is currently sized to 8 GB; we add an
ops-doc note recommending `/tmp` be sized to at least 8 GB on
self-hosters running the import.
- Long-running job interactions. The graceful-shutdown path at
`src/lib/jobs/reminder-worker.ts:1420-1429` waits 30 s before
force-killing. For a 1 GB import the parse alone takes ~60 s; the
worker therefore registers an `idempotency-on-restart` mechanism:
on startup the handler checks for any `ImportJob` rows stuck in
`parsing | upserting` whose `pgBossJobId` is no longer alive in
`pg_boss.job` and flips them to `failed` with reason
`"interrupted_by_restart"`. Operator can re-run the same upload
(the content-hash dedup short-circuits to a fresh job).
## 13. Out-of-scope (deliberate)
These belong on the post-v1.5 backlog rather than v1.4.34:
- HKClinicalRecord / FHIR ingest (defers per R-F T3).
- HKElectrocardiogram waveform import.
- HKAudiogram (hearing test) records.
- StateOfMind data import — Apple's iOS-18 mental-state shape is
on the iOS-side roadmap, server-side ingest waits for that path
to land natively first.
- Activity-ring rollup (`<ActivitySummary>`) — duplicates what
HealthLog computes from `<Record>` rows.
## 14. Per-task effort estimate
| Task | Effort |
|---|---|
| Prisma migration: new `ImportJob` model + index | XS |
| `sax` + `@types/sax` dep + lock refresh | XS |
| `src/lib/multipart/stream-to-disk.ts` helper | S |
| `src/lib/import/unzip-export-xml.ts` central-directory walker | M |
| `src/lib/measurements/parse-export-xml.ts` SAX loop + record / workout / sleep dispatchers | M |
| `src/lib/measurements/hk-workout-activity-type-map.ts` | XS |
| `src/lib/jobs/apple-health-import-worker.ts` handler + `pg-boss` queue registration | S |
| `POST /api/import/apple-health-export/route.ts` kick-off | S |
| `GET /api/import/apple-health-export/[jobId]/status/route.ts` | XS |
| `POST /api/admin/import-apple-health-export/route.ts` admin variant | S |
| Vitest unit suite | M |
| Integration test (full upload-to-status round-trip with a 1 MB fixture) | S |
| OpenAPI regeneration + iOS handoff §3 amendment | S |
| CHANGELOG + Prisma comment + strategic-plan log line | XS |
| Ops doc note (/tmp sizing, graceful-shutdown caveat) | XS |
| **Total** | **L (~3-4 days)** |
## 15. Files the implementation will create or touch
New:
- `prisma/migrations/<timestamp>_v1434_import_jobs/migration.sql`
- `prisma/schema.prisma` — `ImportJob` model (additive)
- `src/lib/import/unzip-export-xml.ts`
- `src/lib/measurements/parse-export-xml.ts`
- `src/lib/measurements/hk-workout-activity-type-map.ts`
- `src/lib/multipart/stream-to-disk.ts`
- `src/lib/jobs/apple-health-import-worker.ts`
- `src/app/api/import/apple-health-export/route.ts`
- `src/app/api/import/apple-health-export/[jobId]/status/route.ts`
- `src/app/api/admin/import-apple-health-export/route.ts`
- `__tests__/apple-health-import-worker.test.ts`
- `__tests__/api-import-apple-health-export.test.ts`
Modified:
- `package.json` — add `sax` + `@types/sax`
- `src/lib/jobs/reminder-worker.ts` — register
`apple-health-import` queue + handler binding
- `CHANGELOG.md` — v1.4.34 entry + freeze marker block
- `.planning/v15-strategic-plan.md` — decision-log row
- `.planning/v15-ios-handoff/03-api-contracts.md` — append §
documenting the import endpoints + status envelope
- `.planning/v15-ios-handoff/04-data-model.md` — note the new
`ImportJob` model
- `docs/ops/...md` — note `/tmp` sizing recommendation
No existing route, no existing model column, no existing helper is
mutated in a non-additive way. The freeze marker can land cleanly on
the same release.
---
Word count: ~2 800 (target 2 000–3 000).
.planning/research/v1434-r-2-carryover-scope.md
+439
View File
@@ -0,0 +1,439 @@
# v1.4.34 R2 — carryover-scope implementation blueprint
Read-only blueprint for v1.4.34 carryovers. Authored against
`develop` (head `f9346e62`); v1.4.33 ships from PR #178 once merged.
Each section maps to one dispatch-able unit; sub-wave decomposition
sits at the bottom.
## 1. Three pre-existing e2e flakes
The release-prep report records 5 failing e2e tests on the v1.4.33
PR; two belong to the F2 dispatch (in flight) and three are
pre-existing carryovers.
### 1a + 1b. `e2e/onboarding-flicker.spec.ts:28` (desktop + mobile)
The "complete-onboarding user never sees the card during load" case
slows `/api/analytics` by 250 ms, then samples
`page.locator('[data-testid="onboarding-card"]').isVisible()` 12 times
at 50 ms intervals and asserts every sample is `false`. Race: on slow
CI the Next.js client paints the analytics-pending shell before
`useAuth().user` resolves `onboardingCompletedAt`; during a 1-2 ms
window `<GettingStartedChecklist>` falls back to its "no auth yet"
branch. CI hits this 8-10% of runs.
Fix shape: swap the 50-ms poll for a single
`await expect(card).toBeHidden({ timeout: 700 })`, then
`waitForLoadState("networkidle")`, then a second `toBeHidden()`.
Playwright's auto-retrying assertion re-evaluates every animation
frame so the intent ("user never sees it") is preserved while the
race window collapses.
Touch surface: `e2e/onboarding-flicker.spec.ts:118-130`. Effort: XS
(~30 LoC swap covers both projects via the existing `test.use`).
Risk: low.
### 1c. `e2e/mobile-viewport.spec.ts:27` (chromium-mobile)
The dashboard CTA-touch-target probe sweeps every visible `main
button, main a[href], nav a[href]`, filters to "in initial viewport",
and asserts ≥ 44×44 px. Two triggers:
- v1.4.33 IW3 shrunk the dashboard "Hinzufügen" min-h to
`sm:min-h-9` (36 px) — on the Pixel 5 boundary at 393 px the `sm:`
breakpoint is below the 640 px threshold, but viewport detection
sometimes flips during the WebKit render commit.
- The probe captures fixed top-bar icon buttons (40×40 px) which
IW9 left untouched.
Fix shape: tighten the selector to `main button, main a[href]`
(drop `nav a[href]` — bottom-nav owns its own WCAG enforcement);
gate the floor on a `matchMedia('(min-width: 640px)').matches ===
false` evaluate so the desktop breakpoint never hits the 44 px
assertion. Spec semantics tighten: "every CTA the user reaches with
a thumb at phone width is 44 px or larger".
Touch surface: `e2e/mobile-viewport.spec.ts:96-126`. Effort: S
(~25 LoC). Risk: low.
## 2. Turbopack NFT-trace warnings
Release-prep log: warnings about NFT traces walking
`next.config.ts → src/lib/geo.ts → src/lib/auth/audit.ts →
mood-entries/bulk route`. `next.config.ts:5` already sets
`output: "standalone"`. Cause: Turbopack's tracer follows env-reads
in `src/lib/geo.ts` (the `MAXMIND_LICENSE_KEY` access) back into the
config file and emits "cannot be traced" warnings — cosmetic, no
runtime effect.
Fix: add `outputFileTracingExcludes: { "*": ["./next.config.ts"] }`
sibling to the existing `outputFileTracingIncludes` at
`next.config.ts:53-55`. Verify via `pnpm build 2>&1 | grep -i
"trace"` — warnings should disappear.
Touch surface: `next.config.ts` (3 LoC). Effort: XS. Risk: low —
`output: "standalone"` controls the actual bundle; the exclude
only narrows trace reporting.
## 3. First-Load-JS bundle-reporting rewire
Turbopack drops the webpack "First Load JS" line. `package.json:29`
already exposes `pnpm analyze` (`ANALYZE=1 next build`) wired
through `@next/bundle-analyzer` at `next.config.ts:69`; output lands
in `.next/analyze/*.html` as treemap reports plus a `client.json`
sibling with `{ label, statSize, parsedSize, gzipSize }` per chunk.
Options:
- (a) `experimental.statsLog` — not a real Next 16 flag. Dead end.
- (b) `next build --webpack` for size reports — Next 16 deprecated
the webpack fallback for `next build`. Dead end.
- (c) Continue with `@next/bundle-analyzer` (already wired); add a
parser that prints the top 5 chunks by `parsedSize` to the CI
log. Trivially extensible to a CI gate later.
Pick (c). Add `pnpm bundle-report` script + a tiny
`scripts/print-bundle-report.mjs` that reads
`.next/analyze/client.json` and prints a sorted table.
Touch surface: `package.json` (1 script), one new script file.
Effort: S (~50 LoC). Risk: low — additive, never replaces build.
## 4. `<TrendCard staleDays>` web wiring
IW3 added `staleDays?: number | null` at
`src/components/charts/trend-card.tsx:144` plus the
`dashboard.staleHint` locale key in six locales. The dashboard reads
from `/api/analytics` (via `useAnalyticsQuery()` at
`src/app/page.tsx:212`); that route's `DataSummary` shape at
`src/lib/analytics/trends.ts:171` does **not** carry per-type
`lastSeenAt`. The matching field landed only on
`/api/dashboard/summary` (iOS-only) at
`src/app/api/dashboard/summary/route.ts:320-321`.
Fix shape — additive on the analytics route:
- Extend `/api/analytics` default payload with
`lastSeenByType: Record<MeasurementType, string | null>`. The
per-type chunked findMany already runs; capture the last point's
`at` and emit it.
- Mirror on the slim slice: `summaries-slice.ts` already runs a
`DISTINCT ON (type)` latest read — add `MAX(measured_at)` to that
pass and surface it.
- Extend `AnalyticsRawPayload` in
`src/lib/queries/use-analytics-query.ts` so call sites can read
the field.
- Wire each `<TrendCard>` mount at `src/app/page.tsx:604-957`
through a helper: `staleDays={computeStaleDays(lastSeenByType,
type)}`. Helper returns `null` when the most-recent reading is
within 7 days (no caption); returns a positive integer otherwise.
Touch surface: 4 files + 14 mount sites in `src/app/page.tsx`.
Effort: S (~120 LoC + 2 tests). Risk: low — additive only;
existing tiles paint byte-identical when `lastSeenByType[type]` is
recent.
## 5. Coach drawer global trigger from dashboard
IW5 verified the Coach drawer is reachable only from `/insights/**`
— `<CoachLaunchProvider>` + `<LayoutCoachFab>` + `<LayoutCoachMount>`
mount inside `src/app/insights/layout.tsx:33-37`. Maintainer feedback
confirms the gap is dashboard-side, not insights.
Fix shape:
- Hoist `<CoachLaunchProvider>` from
`src/app/insights/layout.tsx:33` up to
`src/components/layout/auth-shell.tsx` so every authenticated
tree (dashboard included) can call `askCoach()`.
- Keep `<LayoutCoachFab>` and `<LayoutCoachMount>` in the insights
layout — the FAB is insights-only.
- Mount a new `<CoachLaunchButton>` (same component every insights
sub-page already uses) on the dashboard hero strip at
`src/app/page.tsx:512-524`, sibling to the "Hinzufügen" pill.
Empty prefill — the Coach panel picks the opening prompt.
- Verify the provider's `useState({open, setOpen, prefill})` lives
exactly once. Two providers would mean two open states.
Touch surface: `src/components/layout/auth-shell.tsx` (1 added
mount), `src/app/insights/layout.tsx` (1 removed mount),
`src/app/page.tsx` (1 added button in the hero). No new i18n keys
— `insights.coach.launchButton` already exists in six locales.
Effort: S (~40 LoC + 3 tests). Risk: low.
## 6. Server-side `Cache-Control: no-store` bfcache fix
Lighthouse flagged 3 bfcache failure reasons on v1.4.33. IW9
audited the client (no `beforeunload`/`unload`/`pagehide`/`onfreeze`
listeners). IW2 added `Permissions-Policy: unload=()` at
`next.config.ts:35`. The remaining breaker is the response
`Cache-Control` header on **HTML page responses**.
Audit of explicit `Cache-Control` sets in source — every one is on
an API or file-stream route; none on HTML pages:
- `src/app/api/health/route.ts:26` (`no-store, no-cache,
must-revalidate`) — API only.
- `src/app/api/insights/chat/route.ts:78` (`no-cache, no-transform`)
— SSE stream, intentional.
- `src/app/api/admin/backups/[id]/download/route.ts:117` (`no-store,
max-age=0`) — file stream, intentional.
- `src/app/api/export/*` + `src/app/api/doctor-report/pdf/route.ts`
— file streams, intentional.
The bfcache-breaking header is the Next.js framework default for
authenticated dynamic pages: any route that touches cookies through
a server component emits `Cache-Control: no-store, must-revalidate`
automatically. That covers every `/`, `/insights/**`, `/settings/**`
HTML response.
Fix: add a second rule to the `async headers()` block at
`next.config.ts:30-39` that sets `Cache-Control: private, max-age=0,
must-revalidate` on every non-API non-`_next` HTML route:
```
{ source: "/((?!api|_next).*)", headers: [
{ key: "Cache-Control", value: "private, max-age=0, must-revalidate" },
]}
```
Per Chromium's bfcache-eligibility matrix `no-store` is a hard
breaker; `must-revalidate` is fine. `private` already prevents
shared-cache (proxy) storage; `max-age=0` forces revalidation on
focus so cookie/session swaps still detect on the revalidation
request. No cache-poisoning risk vs today's flow.
Touch surface: `next.config.ts` (~5 LoC). Effort: XS. Risk:
medium — header change covers every authenticated route; verify
via a Lighthouse re-run against `/` and a manual bfcache test
(open `/`, navigate to `/measurements`, hit back, expect
"navigated via back/forward cache" in DevTools).
## 7. Sources ↔ Thresholds merger (A4 deferred)
`round-v1433-audit-menu.md` §7.1 (item 3). The two settings sections
are siblings today:
- `src/components/settings/sources-priority-section.tsx` — per-metric
source list + override + device-type ordering.
- `src/components/settings/thresholds-editor-section.tsx` — per-metric
green/yellow/red threshold inputs.
Combined UX:
- Section title: "Zielwerte & Quellen" (de) / "Targets & Sources"
(en) — six locales.
- One scrollable column with per-metric blocks. Each block: metric
header pill, left column source priority list + device-type
ordering, right column on `md+` (stacked on mobile) threshold
inputs. One "Save" per block (one mutation per metric).
- Summary card at top: "12 metrics configured, 5 with custom
thresholds, 8 using device priorities" for nav cues.
- `data-anchor={metric}` per block + sticky right-rail quick-jump
list on `md+`.
Implementation:
- Rename `thresholds-editor-section.tsx` →
`metric-config-section.tsx`.
- Inline the per-metric source list inside the metric block.
- Drop the `sources` slug from `SETTINGS_SECTIONS`
(`src/components/settings/settings-shell.tsx`) and
`SETTINGS_SECTION_SLUGS`
(`src/components/settings/section-slugs.ts`); make
`/settings/sources` a 301 redirect to `/settings/thresholds`
inside `src/app/settings/[section]/page.tsx`.
- Sections count drops from 10 → 9 post-IW7.
Touch surface: 5-7 files in `src/components/settings/` + section
slugs + settings-shell + six locale files. Effort: M (~600 LoC).
Risk: medium — UX change needs a screenshot review before ship.
## 8. Tab-strip regrouping (A4 deferred)
`round-v1433-audit-menu.md` §7.1 (item 1). Replace the five wave-A
pills (HRV, Ruhepuls, Sauerstoff, Körpertemperatur, Aktive Energie)
in `src/components/insights/insights-tab-strip.tsx` with one parent
pill labelled "Vitalwerte" (source-agnostic; "Apple Health" assumes
device).
Recommended shape: parent pill opens a Sheet/Popover listing the
five sub-metrics; tapping one navigates to its existing route. No
new index page — preserves every URL.
- `SUB_PAGE_TABS` + `buildTabs()` in
`src/components/insights/insights-tab-strip.tsx` modified so the
five wave-A entries collapse to one parent with a popover
trigger.
- `src/lib/insights/sub-page-metric.ts` gains a `group` field for
the wave-A metrics.
- Six locale files for the parent label + the popover header.
Touch surface: 3 source files + 6 locale files. Effort: S (~150
LoC + 3 tests). Risk: low — additive grouping, no URL or
data-fetch change.
## 9. `compliance.ts` classifier hardening (F8 root)
`src/lib/analytics/compliance.ts:65-93` returns `very_late` as a
catch-all for any takenAt outside the `-1h … +2h` grace window.
IW6's F8 analysis: when an intake is logged early (e.g. 06:00 UTC
for a 07:00 UTC window), the dose is flushed to `very_late` and the
heatmap paints orange even though the user took it proactively.
Two semantic flaws:
1. The 1-hour pre-window grace is too narrow for proactive takers.
2. `very_late` semantically covers both "way too early" and "way
too late" — a proactive logger is not "very late".
Proposed thresholds (matches the audit's recommended follow-up):
```
takenAt === null → "missed"
takenAt < windowStart - 3h → "early" (NEW bucket)
windowStart - 3h ≤ takenAt < windowStart → "on_time" (widen grace 1h → 3h)
windowStart ≤ takenAt ≤ windowEnd → "on_time"
windowEnd < takenAt ≤ windowEnd + 2h → "late"
takenAt > windowEnd + 2h → "very_late"
```
`early` + `on_time` both count as compliant for rate calculation;
only `late`, `very_late`, `missed` reduce it.
Touch surface:
- `src/lib/analytics/compliance.ts` (~25 LoC).
- `src/lib/analytics/__tests__/compliance.test.ts` — add 4 cases
for `early` + the widened grace; update the existing "way before
grace" case at line 262 (currently expects `very_late`, becomes
`early` or `on_time`).
- `src/components/charts/compliance-heatmap.tsx` — drop the
defensive `looksClassifierBug` guard at lines 46-77 once the
root is fixed; keep the rate-based fallback below it for the
no-timing-data branch.
- `src/app/api/medications/[id]/compliance/route.ts` (~10 LoC) —
route `early` to the compliant bucket.
- `src/app/api/gamification/achievements/route.ts` (~10 LoC) —
same.
Effort: M (~120 source + 80 test LoC). Risk: medium — touches a
user-facing compliance metric; achievement-streak test path needs
verification.
## 10. AASA followups
No `apple-app-site-association` file exists in the tree today. The
iOS contributor brief uses custom URL schemes during the v1.5
sprint, not associated domains. The in-flight AASA dispatch is the
**first** AASA pass; nothing pre-existing to defer.
Verify after the AASA agent lands:
- `public/.well-known/apple-app-site-association` exists with the
right `applinks` blob and is served as `application/json` (Next.js
needs an explicit handler — default static-file MIME maps ship
this path as `application/pkcs7-mime` on some configs).
- `/apple-app-site-association` (root path) also serves the same
payload — some validators check both.
- Bundle ID matches `APNS_BUNDLE_ID = dev.healthlog.app` (verified
on apps01 Coolify env).
- Alternate domains: only `healthlog.bombeck.io` configured today.
Confirm with maintainer whether `healthlog.app` or similar needs
AASA coverage.
Effort: XS (verification only). Risk: low.
## 11. POSTGRES_PASSWORD apps01 duplicate-section noise
Confirmed via `mcp__coolify-apps01__env_vars list` on app UUID
`pg8wggwogo8c4gc4ks0kk4ss`: the env-var listing contains **two
sections** — many keys appear twice. The first section holds the
real values; the second section is a Compose-default leftover from
the v1.3.0/v1.3.1 era. The real `POSTGRES_PASSWORD` (value
`healthlog`, uuid `psjxz586keglg80frlc0hilg`) is the first-section
entry. The placeholder (`"POSTGRES_PASSWORD is required"`, uuid
`d3r4k1lryj6n0z7dfj4hhc8t`) is the second-section leftover —
documented in `CHANGELOG.md:4554-4566` (v1.3.1 fix). Runtime reads
the first match; the container is healthy.
Cleanup: delete uuid `d3r4k1lryj6n0z7dfj4hhc8t` via the MCP
`env_vars delete` action. Optionally — only with maintainer
agreement — delete the other duplicate entries in section 2
(`SESSION_SECRET`, `ENCRYPTION_KEY`, etc.); they carry the same
value as section 1 so runtime is unaffected.
Effort: XS (1-2 MCP calls). Risk: low.
## 12. Web-Freeze CHANGELOG marker
Per `.planning/v15-strategic-plan.md:204`. Exact line + placement
(in `CHANGELOG.md`, under the v1.4.34 release block, after standard
Added/Changed/Fixed sections):
```markdown
### Web-freeze marker
Web functionality is complete as of v1.4.34. Subsequent v1.4.x tags
ship hotfixes, dependency updates, and tightly-scoped reactive fixes
only; no new web features land until the iOS native client clears
Apple review and v1.5.0 marks the joint release. The full freeze
posture is documented in `.planning/v15-strategic-plan.md` §2.
```
Effort: XS (5 lines). Risk: zero.
## 13. Sub-wave decomposition
Five touch-disjoint sub-waves. Sub-waves A–D run in one parallel
batch; sub-wave E is the close-out.
**Sub-wave A — infrastructure (bfcache + bundle-reporting +
env-var hygiene)**.
Files: `next.config.ts`, `package.json`, optional
`scripts/print-bundle-report.mjs`, plus the MCP env-var cleanup.
Items: #2, #3, #6, #11. Effort: ~S total.
**Sub-wave B — TrendCard wiring + Coach global trigger**.
Files: `src/lib/queries/use-analytics-query.ts`,
`src/app/api/analytics/route.ts`,
`src/lib/analytics/summaries-slice.ts`, `src/app/page.tsx`,
`src/components/layout/auth-shell.tsx`,
`src/app/insights/layout.tsx`.
Items: #4, #5. Effort: ~S+ total.
**Sub-wave C — compliance classifier hardening**.
Files: `src/lib/analytics/compliance.ts`,
`src/lib/analytics/__tests__/compliance.test.ts`,
`src/components/charts/compliance-heatmap.tsx`,
`src/app/api/medications/[id]/compliance/route.ts`,
`src/app/api/gamification/achievements/route.ts`.
Items: #9. Effort: M.
**Sub-wave D — Sources↔Thresholds + tab-strip regroup**.
Files: `src/components/settings/sources-priority-section.tsx`,
`src/components/settings/thresholds-editor-section.tsx`,
`src/components/settings/settings-shell.tsx`,
`src/components/settings/section-slugs.ts`,
`src/components/insights/insights-tab-strip.tsx`,
`src/lib/insights/sub-page-metric.ts`, six locale files,
`src/app/settings/[section]/page.tsx`.
Items: #7, #8. Effort: ~M+.
**Sub-wave E — close-out (e2e flakes + AASA verify + CHANGELOG
marker)**.
Files: `e2e/onboarding-flicker.spec.ts`,
`e2e/mobile-viewport.spec.ts`, `CHANGELOG.md`, plus AASA review.
Items: #1, #10, #12. Effort: ~S total.
Disjointedness: sub-wave A touches config / infrastructure;
sub-wave B touches dashboard + layout + analytics API; sub-wave C
touches compliance analytics + related routes; sub-wave D touches
settings + insights tab-strip; sub-wave E touches e2e + docs. No
file appears in two sub-waves. `src/app/page.tsx` lives in
sub-wave B exclusively.
### Total effort estimate
| Sub-wave | Items | Effort | LoC est. |
| --- | --- | --- | --- |
| A | #2, #3, #6, #11 | S | ~80 |
| B | #4, #5 | S+ | ~250 |
| C | #9 | M | ~200 + 80 tests |
| D | #7, #8 | M+ | ~700 + 100 tests |
| E | #1, #10, #12 | S | ~70 + 5 docs |
| **Total** | **12** | **L (~2 dispatch batches)** | **~1400 + 185 tests** |
Effort scale: XS ≤ 50 LoC, S ≤ 200 LoC, M ≤ 600 LoC, L ≤ 1500 LoC.
Sub-wave D is the largest and benefits from a screenshot-review
checkpoint before ship; everything else is mechanical given the
specs above.
.planning/research/v1434-r-cache-aggregation.md
+429
View File
@@ -0,0 +1,429 @@
# v1.4.34 R-Cache — Server-Side Aggregation Cache Blueprint
Status: research handoff for IW-G (implementation).
Scope: in-process LRU layer in front of the slow-tail GET routes the
v1.4.33 HAR capture flagged.
Companion docs: `.planning/round-v1433-audit-perf.md` (§3.3 motivated
the Coach snapshot LRU), `.planning/round-v1434-prod-slowness-
investigation.md` (HAR + pool-starvation reading).
## 1. Brief
The dashboard mount fans out to eight Prisma-bound GET routes in
parallel. Even with pool starvation fixed, each route still pays its
own row-walk on every load. The Coach snapshot LRU we shipped in
v1.4.33 (`src/lib/ai/coach/snapshot.ts:292–336`, commit `af17db5d`) is
the proven pattern: a hand-rolled `Map<string, { expiresAt, result }>`,
LRU-touched on read, hard-capped on insert, 60 s TTL, keyed on
`(userId, scope)`. Extending that pattern to the high-cost analytics
routes converts the dashboard wait from "fast because Postgres
answered fast" to "fast because the answer was already in RAM". The
two layers compose — a cold cache still benefits from a healthy pool,
a warm cache bypasses the pool entirely.
This document is the design IW-G can implement from directly. It
covers the cache key shape per route, the storage primitive, the
invalidation matrix tying every write endpoint to its evictions, the
helper API, single-flight, observability, and the migration path to a
triggered rollup table in v1.5.x.
## 2. Top three routes by speedup × ease
1. `/api/analytics` (thick + slim, 7.99 s combined cost on dashboard
mount). Pure read of persisted measurements + a fixed-shape
aggregation. Cache the route response keyed on
`(userId, slice)`. Expected hit ratio on the dashboard re-mount
loop is high — the dashboard root, the checklist mount, and the
advisor all read the same payload. Even one hit per second of
active session amortises the 5 s tail across every subsequent
render.
2. `/api/gamification/achievements` (3.18 s × 2 on the same load).
The route resolves achievement progress against the same metric
snapshot every consumer needs. Two-cell duplicate today (per
prod-slowness investigation §4) — caching by `userId` collapses
the duplicate **and** every dashboard re-mount.
3. `/api/medications/intake?scope=compliance&days=30` (3.00 s).
Pre-aggregated daily compliance — by definition slow-changing
inside a 24 h window. Long TTL candidate (15 min) because intake
events trickle in throughout the day but yesterday's row never
moves.
These three convert ~14 s of dashboard cumulative wait time into
~150 ms of `Map.get()` + JSON return on a warm cache. Ease comes from
the routes already being pure reads with a clear `(userId, params)`
key — no auth-side-effect entanglement, no cross-user leakage risk.
Storage pick: hand-rolled `Map`-based LRU per route, in-process. Same
pattern as the snapshot cache, no new dependency. `lru-cache` is not
currently in the dependency tree — adding it for ~120 lines of
infrastructure isn't worth the supply-chain surface. The hand-rolled
shape ships in `src/lib/cache/server-cache.ts` and stays well below
that line count.
Invalidation: tag-based, per-user. Every write endpoint that touches
a measurement / mood / medication row calls one of three helpers —
`invalidateUserMeasurements(userId)`, `invalidateUserMood(userId)`,
`invalidateUserMedications(userId)` — and the helpers walk the cache
maps and delete every key whose `userId` prefix matches. We do not
try to do partial-key eviction (slice-specific or window-specific) on
write; flushing the whole user-bucket is the safe default and the
re-fill on the next read is still cheap because Prisma's pool is fast
under non-burst load.
## 3. Routes in scope — per-route cache contract
| Route | Key shape | Value | TTL | Bucket |
| --- | --- | --- | --- | --- |
| `/api/analytics` | `${userId}\|${slice ?? "thick"}` | full JSON response body | 60 s | measurements |
| `/api/medications` | `${userId}` | full JSON response body | 60 s | medications |
| `/api/gamification/achievements` | `${userId}` | full JSON response body | 60 s | measurements ∪ medications |
| `/api/dashboard/widgets` | `${userId}` | full JSON response body | 5 min | dashboard-layout |
| `/api/bugreport/status` | `singleton` (no userId — the row is global) | `{ configured, enabled, isAdmin }` per role-tier | 10 min | app-settings |
| `/api/workouts?limit=N` | `${userId}\|${limit}\|${offset}\|${since}\|${until}\|${sportType}` | full JSON response body | 60 s | workouts |
| `/api/medications/intake?scope=compliance&days=N` | `${userId}\|compliance\|${days}` | full JSON response body | 15 min | medications |
| `/api/mood/analytics` | `${userId}` | full JSON response body | 60 s | mood |
Notes per row:
- `bugreport/status` is the outlier — the data is global app settings
(GitHub-token presence, enabled flag) gated on user role. Cache the
raw `(configured, enabled)` shape by the constant `"singleton"`
key, then layer `isAdmin` from the request's user on top. Eviction
is `invalidateAppSettings()` called from the `PATCH /api/admin/
app-settings` handler.
- `medications/intake?scope=today` is **not** cached. The "today"
view changes every time the user marks a dose taken — the cache
would either be racy or have a near-zero TTL anyway. Compliance
(`scope=compliance&days=30`) is the cached variant; today's view
reads through.
- `workouts?limit=3` is the dashboard hot-spot. Other workout queries
(`/insights/workouts` with high limits, the iOS app draining
history) share the same cache but with their own keys — they
rarely hit the dashboard's `limit=3` slot, so the cache stays
segmented naturally.
We **don't** negative-cache 404 / 422. The Coach snapshot pattern
returns a real object on every successful path; 404 / 422 means the
auth gate, the param validator, or the rate limiter rejected the
request before any read. Caching the rejection would mean a follow-up
write would have to know to evict it, and the slow-tail risk doesn't
exist here — a 422 returns in single-digit milliseconds.
## 4. Storage primitive
`src/lib/cache/server-cache.ts` ships one class:
```ts
class ServerCache<T> {
private readonly map = new Map<string, { expiresAt: number; value: T }>();
private readonly pending = new Map<string, Promise<T>>();
constructor(private readonly opts: { maxEntries: number; ttlMs: number }) {}
// get(key) — null on miss / expired
// set(key, value) — evicts oldest if over capacity
// delete(key) — single-key eviction
// deleteByPrefix(prefix) — bulk eviction (the userId-prefix path)
// wrap(key, builder) — read-through + single-flight
}
```
`wrap()` is the only public read path. Its contract:
1. If `map.get(key)` returns a non-expired entry, touch (delete +
re-insert moves the key to the end of the `Map`'s insertion order,
which is the LRU end), return `value`.
2. Else if `pending.get(key)` exists, await it. A second caller for
the same key inside the build window does **not** start a second
build — this is the single-flight guarantee.
3. Else: call `builder()`, register the in-flight promise on
`pending`, on resolve `set(key, value)` and delete from `pending`,
on reject delete from `pending` (so the next caller retries).
Per-route eviction caps: 1000 entries each. At ~5 KB per
analytics response and 8 cached routes, the theoretical worst case
is ~40 MB for 50 active users × 10 routes × 5 KB — well inside the
Coolify container memory budget. The 1000-entry cap is a
single-process limit; LRU eviction makes the real working set track
the active-user count.
Memory pressure mitigation: each `ServerCache<T>` instance lives in
module-scope; the `Map` is bounded by `maxEntries`; when the cap is
hit we evict the oldest entry by deleting `map.keys().next().value`.
Same recipe as `src/lib/ai/coach/snapshot.ts:319–325`.
The cache is process-local. Coolify currently runs a single
`apps-01` container per deployment, so all traffic lands in the same
process. If a future deploy moves to multi-instance (horizontal
scale-out behind a load balancer), each instance keeps its own cache;
writes from instance A do not evict reads on instance B. We accept
that trade today — the cost is "instance B serves the pre-write value
for up to TTL after a write" which is bounded and not silent
(provenance + cache-key hash logged on every response). The Redis
migration path is sketched in §7.
## 5. Helper API
```ts
// src/lib/cache/server-cache.ts
export const caches = {
analytics: new ServerCache<unknown>({ maxEntries: 1000, ttlMs: 60_000 }),
medications: new ServerCache<unknown>({ maxEntries: 1000, ttlMs: 60_000 }),
achievements: new ServerCache<unknown>({ maxEntries: 1000, ttlMs: 60_000 }),
dashboardWidgets: new ServerCache<unknown>({ maxEntries: 500, ttlMs: 300_000 }),
bugreportStatus: new ServerCache<unknown>({ maxEntries: 10, ttlMs: 600_000 }),
workouts: new ServerCache<unknown>({ maxEntries: 1000, ttlMs: 60_000 }),
medicationsIntake:new ServerCache<unknown>({ maxEntries: 1000, ttlMs: 900_000 }),
moodAnalytics: new ServerCache<unknown>({ maxEntries: 1000, ttlMs: 60_000 }),
};
export async function cached<T>(
cache: ServerCache<T>,
key: string,
builder: () => Promise<T>,
): Promise<T> {
return cache.wrap(key, builder);
}
// Bulk eviction — by user, optionally narrowed to one cache.
export function invalidateUserMeasurements(userId: string): void {
caches.analytics.deleteByPrefix(`${userId}|`);
caches.achievements.deleteByPrefix(`${userId}`);
caches.workouts.deleteByPrefix(`${userId}|`);
// mood-analytics is invalidated separately because mood writes are
// a different write surface and we don't want to over-evict.
}
export function invalidateUserMood(userId: string): void {
caches.moodAnalytics.deleteByPrefix(`${userId}`);
caches.achievements.deleteByPrefix(`${userId}`);
caches.analytics.deleteByPrefix(`${userId}|`);
}
export function invalidateUserMedications(userId: string): void {
caches.medications.deleteByPrefix(`${userId}`);
caches.medicationsIntake.deleteByPrefix(`${userId}|`);
caches.achievements.deleteByPrefix(`${userId}`);
}
export function invalidateUserDashboardWidgets(userId: string): void {
caches.dashboardWidgets.deleteByPrefix(`${userId}`);
}
export function invalidateAppSettings(): void {
caches.bugreportStatus.deleteByPrefix("");
}
```
Each route handler wraps its body. Example for `/api/analytics`:
```ts
const data = await cached(
caches.analytics,
`${user.id}|${slice ?? "thick"}`,
() => buildAnalyticsResponse(user.id, slice, userTz),
);
return apiSuccess(data);
```
The route function carrying the actual aggregation moves into
`buildAnalyticsResponse()`. The route stays the cache miss / hit
boundary. The wrap matches the snapshot pattern's two-function shape
(`buildCoachSnapshot` → `buildCoachSnapshotImpl`).
## 6. Invalidation matrix
Every write endpoint calls the appropriate `invalidateUser*` helper
**after** its successful database commit, **before** the response is
returned. The helpers are cheap (`Map.delete` over a bounded prefix
scan); we accept the minor latency penalty on writes to keep reads
consistent. Failure to invalidate poisons the cache for one TTL
cycle, which is exactly the failure mode the snapshot pattern
already accepts.
| Write endpoint | Invalidations |
| --- | --- |
| `POST /api/measurements` | `invalidateUserMeasurements(userId)` |
| `POST /api/measurements/batch` | `invalidateUserMeasurements(userId)` |
| `PUT /api/measurements/[id]` | `invalidateUserMeasurements(userId)` |
| `DELETE /api/measurements/[id]` | `invalidateUserMeasurements(userId)` |
| `DELETE /api/measurements/by-external-ids` | `invalidateUserMeasurements(userId)` |
| `POST /api/mood-entries` | `invalidateUserMood(userId)` |
| `PATCH /api/mood-entries/[id]` | `invalidateUserMood(userId)` |
| `DELETE /api/mood-entries/[id]` | `invalidateUserMood(userId)` |
| `POST /api/mood-entries/bulk` | `invalidateUserMood(userId)` |
| `POST /api/medications` | `invalidateUserMedications(userId)` |
| `PUT /api/medications/[id]` | `invalidateUserMedications(userId)` |
| `DELETE /api/medications/[id]` | `invalidateUserMedications(userId)` |
| `POST /api/medications/intake` (status update) | `invalidateUserMedications(userId)` |
| `POST /api/workouts/batch` | `invalidateUserMeasurements(userId)` (workouts cache rides on the measurements bucket because achievements / analytics both read workouts) |
| `PUT /api/dashboard/widgets` | `invalidateUserDashboardWidgets(userId)` |
| `DELETE /api/dashboard/widgets` | `invalidateUserDashboardWidgets(userId)` |
| `PATCH /api/admin/app-settings` (bugreport-config toggle) | `invalidateAppSettings()` |
A new measurement write touches the analytics aggregate, the
achievement progress, and possibly the workouts dashboard tile (if
the measurement is a workout-linked metric). That's why the
`invalidateUserMeasurements` helper evicts three caches at once
rather than just one. The conservative "evict the whole user-bucket"
matches Marc's "blazing fast" stance — a redundant eviction costs the
next reader one cache miss (cheap, the row's already paginated), an
under-eviction shows stale data which is the directive's failure
mode.
## 7. Single-flight + cache stampede
Two concurrent dashboard mounts (two browser tabs, both with cold
cache, both fetching `/api/analytics` in the same second) would
otherwise issue two builders. The `pending` map in the storage
primitive coalesces these: tab B awaits tab A's in-flight promise
and gets the same response. This is the standard single-flight /
deduplication pattern.
Cache stampede on TTL expiry: when the cached entry expires at
second 60.000 and three requests land at second 60.001, the first
request through the `wrap()` call kicks off a fresh build; requests
two and three see no live entry yet (it's about to be set) **but**
also see no pending promise yet — there's a race window of a few
microseconds. Mitigation (deferred — not part of IW-G unless the
metrics show it's needed): **soft-TTL**. Keep the entry available
past `expiresAt` for one additional refresh window; mark it stale on
read and trigger a background rebuild for the first reader past the
soft-TTL; subsequent readers in the next 1 s window serve the stale
entry. This is the standard SWR-server pattern. For v1.4.34 we ship
the hard-TTL variant (cleaner semantics, simpler tests) and add
soft-TTL only if telemetry shows real stampedes.
## 8. Observability
Every cache hit / miss is annotated on the active wide-event
builder:
```ts
const event = getEvent();
event?.addMeta("cache.analytics.outcome", hit ? "hit" : "miss");
event?.addMeta("cache.analytics.key_hash", djb2(key));
```
The `key_hash` is a non-reversible 32-bit hash so we can correlate
hits without leaking userId into logs. We aggregate hit ratio per
route in the existing wide-event dashboard. The audit-log surface
already accepts arbitrary `meta` keys, so no schema change is
needed.
Per-test reset escape hatch: every `ServerCache<T>` exposes
`__resetForTests()` (matches `__resetCoachSnapshotCacheForTests` in
the snapshot pattern). Vitest setup in `vitest.config.mts` adds an
afterEach that clears all caches, identical to how the snapshot
tests handle it (`src/lib/ai/coach/__tests__/snapshot.test.ts`).
## 9. Comparison to the v1.5.x triggered-rollup table
The LRU layer is the right pick for v1.4.34: it's a single file, ships
behind a proven pattern, no schema change, no migration risk. It
trades correctness-on-write for read speed in exchange for an
aggressive invalidation policy — that's the same trade the Coach
snapshot LRU made.
A v1.5.x triggered rollup table — proposed schema sketch:
```sql
CREATE TABLE measurement_rollups (
user_id UUID NOT NULL,
type TEXT NOT NULL,
window TEXT NOT NULL, -- '7d' / '30d' / '90d'
day_key DATE NOT NULL, -- bucket anchor
count INT NOT NULL,
mean DOUBLE PRECISION,
min_val DOUBLE PRECISION,
max_val DOUBLE PRECISION,
sd DOUBLE PRECISION,
slope DOUBLE PRECISION,
r2 DOUBLE PRECISION,
computed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (user_id, type, window, day_key)
);
CREATE INDEX ON measurement_rollups (user_id, type, window);
```
Update path: a Postgres `AFTER INSERT/UPDATE/DELETE` trigger on
`Measurement` enqueues a recompute for the affected `(userId, type,
day_key)`. The recompute is a single SQL upsert per metric, runs in
the pg-boss worker (already deployed for reminder fan-out — see
`src/lib/jobs/reminder-worker.ts`), and is bounded by the number of
distinct types the write touched (typically 1–3 per measurement).
Reads from `/api/analytics` then bypass the row walk entirely and
join against the pre-computed rollup table.
Pros (rollup vs LRU):
- Survives process restarts and multi-instance deploys.
- Cross-user fan-out queries (the v1.5 admin analytics surface) can
read directly without per-user warmup.
- Read latency drops to a single indexed lookup regardless of cache
state.
Cons (rollup vs LRU):
- Migration introduces a new table + indexes + background worker
contract.
- The trigger fan-out is one more thing to monitor; a stuck pg-boss
queue leaves rollups stale (silent if not alarmed).
- Storage cost grows with measurement volume × window-count × day-
count.
Migration path:
1. v1.4.34 — ship LRU. Land observability so we can read hit ratio
in production.
2. v1.5.x — introduce `measurement_rollups` additively behind the
feature flag `analytics-rollups-table`. Reads continue to land in
the LRU; the rollup table is populated by the trigger but not
queried.
3. v1.5.x+1 — flip the flag for one user, dual-read both paths,
diff. When the diff is zero over 10 000 reads we cut over to the
rollup table for that user.
4. v1.5.x+2 — promote to a single feature flag flip, deprecate the
LRU. Keep the LRU code path as the fallback for users still on
the legacy flag during the transition window.
## 10. Risks + mitigations
| Risk | Mitigation |
| --- | --- |
| Stale data after a write | Every write endpoint calls `invalidateUser*`. Documented in §6. The penalty for missing one is one TTL window of stale reads, not silent corruption. The pre-release checklist for v1.4.34 includes a grep gate that lists every `POST/PUT/PATCH/DELETE` handler under `/api/(measurements\|mood-entries\|medications\|workouts)` and verifies each contains an `invalidateUser*` call. |
| Multi-instance Coolify | Documented and accepted. Each instance keeps its own cache; writes don't cross-evict. The TTL bounds the staleness. Plan Redis as a v1.5.x option when the rollup table is also on the roadmap — the two together close the gap. |
| Memory pressure | Per-cache `maxEntries` cap, LRU eviction on overflow. Worst case 40 MB across all caches on the current Coolify container's 512 MB budget. |
| Cache stampede on TTL expiry | Single-flight via the `pending` map. Soft-TTL is a deferred follow-up if telemetry shows we need it. |
| Per-test pollution | `__resetForTests()` on every `ServerCache<T>` + vitest afterEach. Same recipe the snapshot tests already use. |
| Auth-side-effect entanglement | The cache key includes `userId`. The `bugreport/status` endpoint is the exception (singleton key with per-user layering); the doc here calls it out explicitly. Auth always runs before the cache `wrap()` so a stolen request that bypasses auth would not reach the cache. |
| Wide-event log growth | The hit / miss annotation reuses the existing `addMeta` carrier, two keys per request. No new log volume class. |
## 11. Implementation checklist (for IW-G)
1. Land `src/lib/cache/server-cache.ts` with the `ServerCache<T>`
primitive, the eight `caches.*` instances, the four
`invalidateUser*` helpers, the `invalidateAppSettings()` helper,
and the `cached()` wrapper. Unit-test the primitive — LRU
ordering, TTL, single-flight, prefix delete.
2. Wrap each route in §3 — `/api/analytics` first (highest payoff),
then the rest in any order. Each wrap is two lines of edit.
3. Add `invalidateUser*` calls to every write endpoint in §6. Run
the grep gate as part of the pre-release checklist.
4. Wire the `addMeta("cache.<name>.outcome", …)` annotations into
each `cached()` call. Single helper that branches on hit vs miss.
5. Add integration tests for the read-through path: prime the cache,
write through the matching write endpoint, assert the next read
misses (i.e. the invalidation worked).
6. Document the cache in `docs/audit/v1434-summary.md` once IW-G
merges.
## 12. Acceptance criteria
- All eight routes in §3 wrap their handler body through `cached()`.
- All write endpoints in §6 carry the matching `invalidateUser*`
call.
- Unit tests cover LRU ordering, TTL, prefix-delete, single-flight.
- Integration test: dashboard mount with warm cache resolves
`/api/analytics` in < 50 ms. With cold cache, parity with the
pre-cache p95.
- `cache.<name>.outcome` annotations land on every request to a
cached route. Dashboard hit-ratio query returns within seconds.
- No regression in the snapshot LRU (`buildCoachSnapshot`) — IW-G
must not refactor it under the new primitive in the same wave.
The migration is a v1.4.35 follow-up.
End of brief. IW-G can implement directly from this file. Any
ambiguity defaults to "match the v1.4.33 snapshot LRU shape" since
that is the proven pattern this design extends.
.planning/round-v1433-aasa-rollout-report.md
+141
View File
@@ -0,0 +1,141 @@
# v1.4.33 AASA rollout — closure report
Date: 2026-05-16
Scope: ship `/.well-known/apple-app-site-association` on every host that
fronts HealthLog so the iOS bundle's passkey ceremony shares cleanly
with the web origins, and so Apple's CDN can ingest the file ahead of
the iOS Tier-1 work that lands later in the v1.4.x cycle.
## Bundle / Team identifiers
- Team ID: `S8WDX4W5KX`
- Bundle ID: `dev.healthlog.app`
- App ID prefix: `S8WDX4W5KX.dev.healthlog.app`
## Commits
### Landing (`healthlog-landing`, `main`)
| SHA | Subject |
| --- | --- |
| `33e6039` | feat(ios): serve apple-app-site-association on healthlog.dev |
Touched paths:
- `src/app/.well-known/apple-app-site-association/route.ts` (new) —
Next.js Route Handler with `dynamic = "force-static"` so the static
export emits a literal JSON file at
`out/.well-known/apple-app-site-association`.
- `Dockerfile` — nginx config gained an exact-match
`location = /.well-known/apple-app-site-association` block that runs
ahead of the SPA catch-all and pins `Content-Type: application/json`
+ `Cache-Control: public, max-age=3600`.
### Main app (`HealthLog`, `develop`, rides v1.4.33 PR #178)
| SHA | Subject |
| --- | --- |
| `2acb11f8` | feat(auth): admit /.well-known/* without a session |
| `03b0be18` | feat(ios): serve apple-app-site-association on the app domains |
| `b92f2b1e` | docs(changelog): note the AASA addition for v1.4.33 |
Touched paths:
- `src/proxy.ts` — added `/.well-known/` (trailing-slash prefix) to
`PUBLIC_PATHS` so Apple's CDN fetch lands on the asset instead of
the auth gate. Trailing-slash future-proofs the namespace for
`/security.txt`, `/openid-configuration`, etc.
- `src/__tests__/proxy-well-known-public.test.ts` (new) — four
regression scenarios: unauthenticated AASA fetch, pending-onboarding
session, generic `/.well-known/security.txt` peer, and a negative
check that `/insights` still 307s.
- `src/app/.well-known/apple-app-site-association/route.ts` (new) —
same JSON body, same headers as the landing handler.
- `CHANGELOG.md` — one-line entry under the v1.4.33 Added section
recording both the handler and the proxy allowlist edit.
## Quality gates
- Main app `pnpm typecheck` — clean.
- Main app `pnpm lint` — zero new errors on the touched files (the
3018 baseline problems all live on pre-existing files).
- Main app `pnpm test` on the three proxy specs — 17/17 pass.
- Landing `npx tsc --noEmit` — clean.
- Landing `pnpm lint` — zero new warnings on the touched files (one
pre-existing warning in `DemoCredentials.tsx`).
- Landing `pnpm build` — succeeded, AASA route reported as `○ (Static)`
in the build manifest, output file inspected as 96-byte JSON.
## Deploy verification
### healthlog.dev (landing — already live, deployed via GitHub Actions)
```
HTTP/2 200
content-type: application/json
content-length: 96
cache-control: public, max-age=3600
{"applinks":{"apps":[],"details":[]},"webcredentials":{"apps":["S8WDX4W5KX.dev.healthlog.app"]}}
```
Auto-deploy ran via the `Deploy to edge-01` workflow (run
`25967688097`), completed in 60s, no errors. Coolify-auto-deploy fix
from earlier today held.
### healthlog.bombeck.io (main app — pending v1.4.33 tag)
```
HTTP/2 307
location: /auth/login
```
Expected — the commits sit on `develop` and ride PR #178 → `main` →
v1.4.33 tag. Will go live once the maintainer cuts the release.
### demo.healthlog.dev (main app — pending v1.4.33 tag)
```
HTTP/2 307
location: /auth/login
```
Same pending state as `healthlog.bombeck.io`; both hosts ride the same
standalone runtime.
### Apple CDN validator
```
$ curl -s https://app-site-association.cdn-apple.com/a/v1/healthlog.dev
{"applinks":{"apps":[],"details":[]},"webcredentials":{"apps":["S8WDX4W5KX.dev.healthlog.app"]}}
```
Apple's CDN ingested the file immediately — no propagation lag observed.
The CDN response matches the origin response byte-for-byte.
## Follow-ups for v1.4.34
- **Universal Links.** Populate `applinks.details` with the bundle ID +
`paths` array once the iOS side opts in to deep-linking specific
routes. Likely candidates from the maintainer's earlier brief:
`/insights/*` (per-metric detail pages), `/measurements/*` (deep
link from an Apple Health share-sheet), `/auth/login` (passkey
re-auth). Coordination needed with the iOS app so the bundle
declares `applinks` entitlement + matching `associatedDomains` in
the Info.plist.
- **AASA mirror on `demo.healthlog.dev`.** Currently shares the
handler with `healthlog.bombeck.io` via the same standalone runtime,
which is correct. If a future split moves demo to its own deployment
with a different App ID prefix (e.g. a separate `dev.healthlog.demo`
bundle for sandbox passkeys), the handler will need a host-aware
branch.
- **Cache invalidation on Bundle ID rotation.** Apple's CDN caches for
an hour; if the App ID prefix ever rotates (Team ID rename, bundle
rename), the iOS app's passkey ceremony will fail for up to 60
minutes after the AASA change. Worth documenting in the iOS-handoff
brief.
- **Discovery namespace bookkeeping.** `/.well-known/security.txt`
(RFC 9116) and `/.well-known/change-password` (W3C) are both useful
follow-on entries the trailing-slash allowlist now admits without a
second proxy edit. Low priority but easy wins for the next polish
round.
.planning/round-v1434-iwc-report.md
+74
View File
@@ -0,0 +1,74 @@
# Round v1.4.34 — IW-C (Compliance Classifier) report
## Scope delivered
Carryover scope item #9 (`compliance.ts` classifier hardening) closed
in three atomic commits on `develop`:
1. `36d5398c` — **classifier**: widen pre-window grace 1h → 3h, add a
dedicated `early` bucket, extend post-window `on_time` tolerance to
3h, and let the configurable `lateMinutes` knob size the `late`
tail past the 3h grace. Tests rebuilt with a parameterised offset
matrix.
2. `4984bce5` — **consumers**: route `early` intakes through the
compliant path in `src/app/api/medications/[id]/compliance/route.ts`
(counted with `onTime` for the heatmap, plus a dedicated `early`
counter on `DailyComplianceEntry`) and in
`src/app/api/gamification/achievements/route.ts` (perfect-day
preserved); docstring fixup in
`src/lib/medications/scheduling/cadence.ts`.
3. `30c8555c` — **heatmap**: drop the v1.4.33 `looksClassifierBug`
defensive fallthrough now that the root cause is fixed; keep a
one-line comment pointing at the classifier fix.
Pushed to `origin/develop` (`30c8555c`).
## Threshold model (final)
```
takenAt === null → "missed"
takenAt < windowStart - 3h → "very_late"
windowStart - 3h ≤ takenAt < windowStart → "early" (NEW)
windowStart ≤ takenAt ≤ windowEnd + 3h → "on_time" (widened)
windowEnd + 3h < takenAt ≤ windowEnd + 3h + Xmin → "late" (X = lateMinutes, default 120)
takenAt > windowEnd + 3h + Xmin → "very_late"
```
`early` + `on_time` both count as compliant. `early` is exposed on
`IntakeTimingClass` for downstream consumers and as an optional
counter on `DailyComplianceEntry`.
## Validation
- `npx vitest run src/lib/analytics/__tests__/compliance.test.ts` —
31/31 passing (was 21; added 10 covering -3.5h, -3h (boundary), -1h,
-10min, 0, +10min, +20min, +1h, +3h, +3.5h, +5h, +5h1min, plus an
overnight `early` case and a `lateMinutes`-knob case).
- `npx vitest run` — full suite 4153 passing / 1 skipped across 386
test files.
- `npx tsc --noEmit` — clean.
- `npx eslint` on the five touched source files — clean.
## Touch surface
- `src/lib/analytics/compliance.ts` (+44 / -9)
- `src/lib/analytics/__tests__/compliance.test.ts` (+71 / -24)
- `src/app/api/medications/[id]/compliance/route.ts` (+9 / -1)
- `src/app/api/gamification/achievements/route.ts` (+3 / -1)
- `src/components/charts/compliance-heatmap.tsx` (+13 / -35)
- `src/lib/medications/scheduling/cadence.ts` (+2 / -2 docstring)
Touch-disjoint from the parallel IW-A / IW-B / IW-D / IW-XML lanes.
## Notes for downstream
- The `early` bucket is opt-in for any new analytics consumer. Today
the medication compliance route surfaces it as an optional counter
on `DailyComplianceEntry`; if a future heatmap legend wants to
distinguish early-vs-on-time visually, the data is there.
- The default `lateMinutes` of 120 still sizes the `late` band, so
the total "late span" is now `windowEnd + 3h .. windowEnd + 5h`
before a dose flips to `very_late`. Callers that want a tighter
tail can pass `lateMinutes: 30` (tested).
- `medication-card.tsx` keeps its independent `MedicationWindowStatus`
type — no cross-contamination with the classifier bucket names.
.planning/round-v1434-iwd-report.md
+169
View File
@@ -0,0 +1,169 @@
# v1.4.34 IW-D — Settings + Insights UX report
Branch: `develop`. Commits landed:
- `fd81d3b8` refactor(settings): merge Sources into Thresholds as "Targets & Sources"
- `bcca7664` refactor(insights): collapse five wave-A vital pills under a Vitals parent
- `43502185` i18n: add Vitals parent + Targets & Sources keys across six locales
All three pushed to `origin/develop`.
## What landed
### 1. Sources → Thresholds merger
`/settings/thresholds` now hosts the merged "Targets & Sources" /
"Zielwerte & Quellen" page. The two former editors stack inside one
section: per-metric threshold ranges on top, per-metric source
priority + two-axis device-type ladder below.
- `<ThresholdsSection>` (`src/components/settings/thresholds-section.tsx`)
now mounts both `<ThresholdsEditorSection>` and
`<SourcesSection mode="embedded">` under one header.
- `<SourcesSection>` (`src/components/settings/sources-section.tsx`)
gained a `mode` prop (`"standalone"` default for backward compat,
`"embedded"` drops the inner h1 + subtitle so assistive-tech reads
one combined surface).
- The `sources` slug was dropped from `SETTINGS_SECTION_SLUGS` and
the Sources entry was removed from the in-shell sidebar + mobile
chip-strip. Section count: 11 → 10.
- `/settings/sources` stays alive as a `permanentRedirect` to
`/settings/thresholds` via the new
`src/app/settings/sources/page.tsx`. iOS bookmarks + external
docs follow through unchanged.
- The dynamic `[section]` route stopped importing `<SourcesSection>`
and `<Layers>` icon imports were dropped from `<SettingsShell>` +
`<SectionPlaceholder>`.
The two mutation flows (`/api/user/thresholds` and
`/api/auth/me/source-priority`) stay distinct so a save on one half
never disturbs the other. This was the pragmatic pick over the
deeper per-metric interleaving (which would have required merging
two different metric-key namespaces and rewiring two mutations into
one — a 600+ LoC refactor with cross-axis behaviour risk).
### 2. Insights tab-strip Vitals parent
Five wave-A pills (HRV, Resting HR, Oxygen, Body Temperature, Active
Energy) collapse behind one "Vitals" / "Vitalwerte" parent pill that
opens a popover sub-list.
- `SUB_PAGE_GROUP` + `SUB_PAGE_GROUP_ORDER` in
`src/lib/insights/sub-page-metric.ts` flag the five wave-A slugs as
group members. Adding a future group is one row + one locale key.
- `<InsightsTabStrip>` uses a discriminated `TabEntry` union — flat
`<Link>` pills coexist with `<Popover>`-driven group entries.
Availability gating sits on the leaf slugs: parent pill renders
only when at least one child has data.
- Parent-pill active state mirrors the URL — when on any vital
sub-page, the parent pill carries `aria-current="page"` and the
active border style so spatial orientation survives the collapse.
- Strip footprint: 14 → 10 entries when every metric has data.
Each sub-page keeps its own URL and component — only the strip
presentation changes. Deep-links and bookmarks resolve unchanged.
### 3. Locale parity (six locales)
Two new key blocks across `messages/{de,en,es,fr,it,pl}.json`:
- `insights.tabStrip.vitalsParent.{label,header}` — parent pill text
+ popover header.
- `settings.sections.thresholds.{title,description}` — renamed to the
combined "Targets & Sources" form per locale.
- `settings.sections.sources.redirectNotice` — copy reserved for the
back-compat redirect surface ("Sources are now part of '…'.").
Existing `settings.sections.sources.*` keys stayed because
`<SourcesSection>` (now embedded inside the merged page) still
resolves card-level labels through them.
## Tests
- `src/components/insights/__tests__/insights-tab-strip.test.tsx` —
10 tests, all pass. Added 4 new tests:
- vitals group renders single parent pill (not five flat ones)
- parent pill hidden when no wave-A metric has data
- parent pill carries `data-slot="insights-tab-strip-group"` +
`data-group="vitals"`
- non-grouped pills stay inline
- `src/components/settings/__tests__/sections.test.tsx` — 89 tests,
all pass. Updated the `<ThresholdsSection>` test to assert both
`data-testid="thresholds-editor"` + `data-testid="sources-section"`
mount points, plus the new German + English titles.
- `src/components/settings/__tests__/settings-shell.test.tsx` —
section-list expectation updated to 10 slugs (sources retired);
English + German title tests assert the new "Targets & Sources" /
"Zielwerte & Quellen" copy and the absence of the `/settings/sources`
href in the in-shell nav.
Full settings test suite (11 files, 89 tests) and full insights test
suite (28 files, 255 tests) both green. `pnpm tsc --noEmit` clean on
all my files; the unrelated `gamification/recent-achievements-card`
TS errors come from another wave and were left untouched.
## Screenshot review
No dev server was running during the dispatch window — Playwright
captures were skipped. The maintainer should screenshot-review the
following surfaces before tagging v1.4.34:
1. `/settings/thresholds` — verify the merged page reads as one
continuous shelf (header → threshold editor card → source
priority card with the two-axis expander). The current shape
stacks the two card bodies vertically; the deeper interleaved
per-metric "source list (top), threshold inputs (bottom)" layout
that the carryover scope §7 described is the higher-LoC variant
and is available as a follow-up if the maintainer prefers it
over the side-by-side pragmatic merger.
2. `/settings/sources` — confirm the 308 permanentRedirect resolves
to `/settings/thresholds` (open a saved bookmark or paste the
URL).
3. `/insights` mobile (393 px) — verify the Vitals parent pill renders
between Pulse and Weight, that tap opens the popover, and that
the active border lights up when on `/insights/hrv` (etc.).
4. `/insights/hrv`, `/insights/ruhepuls`, `/insights/sauerstoff`,
`/insights/koerpertemperatur`, `/insights/aktive-energie` — each
still resolves to its own page; the parent pill's `aria-current`
should flip to "page" on each.
## Outstanding follow-ups
- Per-metric interleaved blocks ("source list (top), threshold inputs
(bottom)" for each metric) — deferred. The current shape stacks the
two cards under one heading, which collapses two sidebar slots into
one without the cross-namespace metric mapping work. If the
maintainer wants the deeper merge, the next step is a
`<MetricConfigSection>` that loops over the union of
`ThresholdMetric` + `SourcePriorityMetricKey` and renders both per
metric. Estimate: ~400 LoC + 80 LoC tests, but the two API contracts
stay unchanged.
- The `sources` slug stays referenced in `settings.sections.sources.*`
card-level keys (cardTitle, help, etc.). Those still resolve
because `<SourcesSection>` reads them. No dead-key cleanup needed
this round.
## File set (all absolute paths)
Sub-wave 1 — merger:
- `/Users/marc/Projects/HealthLog/src/components/settings/section-slugs.ts`
- `/Users/marc/Projects/HealthLog/src/components/settings/settings-shell.tsx`
- `/Users/marc/Projects/HealthLog/src/components/settings/section-placeholder.tsx`
- `/Users/marc/Projects/HealthLog/src/components/settings/thresholds-section.tsx`
- `/Users/marc/Projects/HealthLog/src/components/settings/sources-section.tsx`
- `/Users/marc/Projects/HealthLog/src/components/settings/__tests__/sections.test.tsx`
- `/Users/marc/Projects/HealthLog/src/components/settings/__tests__/settings-shell.test.tsx`
- `/Users/marc/Projects/HealthLog/src/app/settings/[section]/page.tsx`
- `/Users/marc/Projects/HealthLog/src/app/settings/sources/page.tsx` (new)
Sub-wave 2 — tab-strip:
- `/Users/marc/Projects/HealthLog/src/lib/insights/sub-page-metric.ts`
- `/Users/marc/Projects/HealthLog/src/components/insights/insights-tab-strip.tsx`
- `/Users/marc/Projects/HealthLog/src/components/insights/__tests__/insights-tab-strip.test.tsx`
Sub-wave 3 — locales:
- `/Users/marc/Projects/HealthLog/messages/de.json`
- `/Users/marc/Projects/HealthLog/messages/en.json`
- `/Users/marc/Projects/HealthLog/messages/es.json`
- `/Users/marc/Projects/HealthLog/messages/fr.json`
- `/Users/marc/Projects/HealthLog/messages/it.json`
- `/Users/marc/Projects/HealthLog/messages/pl.json`
.planning/round-v1434-iwg-report.md
+153
View File
@@ -0,0 +1,153 @@
# v1.4.34 IW-G — Server-Cache Implementation Report
## Summary
Shipped the in-process LRU + single-flight `ServerCache<T>` primitive
extending the v1.4.33 Coach snapshot LRU shape (`af17db5d`). Wired the
three hottest dashboard reads through it (`/api/analytics`,
`/api/gamification/achievements`,
`/api/medications/intake?scope=compliance`) and bolted per-user
invalidation onto every measurement / mood / medication / workout /
dashboard-widget / app-settings write endpoint per the blueprint §6
matrix. Observability annotations
(`cache.<name>.outcome` + `cache.<name>.key_hash`) reach the wide event
on every cached request so production logs carry the hit-ratio signal.
## Commits
Branch: `develop` → pushed.
| SHA | Title |
| --- | --- |
| `a5dd8e1d` | `feat(cache): server-side LRU primitive + per-user invalidation helpers` |
| `68c4bb73` | `perf(api): wire 3 hot read routes through the server cache` |
| `757387ca` | `feat(cache): wire 13 write endpoints to per-user invalidation helpers` |
| `bcdab48e` | `test(cache): integration coverage + CHANGELOG entry for server cache` |
## Files touched
New:
- `src/lib/cache/server-cache.ts` — `ServerCache<T>` class, 8-cache
registry, `cached()` wrapper, `hashCacheKey()`, test reset hatch.
- `src/lib/cache/invalidate.ts` — 5 per-user helpers
(`invalidateUserMeasurements/Mood/Medications/DashboardWidgets`,
`invalidateAppSettings`).
- `src/lib/cache/__tests__/server-cache.test.ts` — 18 unit tests.
- `src/lib/cache/__tests__/invalidate.test.ts` — 5 helper tests.
- `tests/integration/server-cache-routes.test.ts` — 3 end-to-end tests
(single-flight, measurement invalidation, intake invalidation).
Modified:
- `src/app/api/analytics/route.ts` — wraps slim + default slice.
- `src/app/api/gamification/achievements/route.ts` — wraps web result.
- `src/app/api/medications/intake/route.ts` — wraps compliance branch
+ invalidation on POST.
- 11 write endpoints across `measurements*`, `mood-entries*`,
`medications*`, `workouts/batch`, `dashboard/widgets`,
`admin/settings` — `invalidateUser*` call after DB commit.
- 4 existing test files — `__resetAllCachesForTests()` in `beforeEach`
so re-using a fixed userId doesn't carry cached state across cases.
- `CHANGELOG.md` — one Added line in Marc-voice English.
## Observability pattern
Every `cached()` invocation passes `annotate` from
`@/lib/logging/context` as the optional fourth argument:
```ts
const body = await cached(
caches.analytics as ServerCache<...>,
`${user.id}|default`,
() => buildAnalyticsResponse(user),
annotate,
);
```
On the wide event, every cached request now carries:
- `cache.<name>.outcome` ∈ `{ "hit" | "miss" | "stampede" }`
- `cache.<name>.key_hash` — non-reversible djb2 32-bit hash of the
cache key
Witnessed live in the integration run; sample log line:
```json
{
"meta": {
"cache.medicationsIntake.outcome": "miss",
"cache.medicationsIntake.key_hash": 3630666302,
"days": 30,
"count": 30
},
"action": { "name": "medications.intake.compliance" }
}
```
Ops can `grep cache.analytics.outcome` to compute hit ratio per
deployment over any time window.
## Quality gates
- `pnpm exec tsc --noEmit` clean.
- `pnpm test` — 4227 pass, 1 skipped (no new failures).
- `pnpm test:integration` — 196 pass (49 files); the 3 new
`server-cache-routes.test.ts` cases included.
- `pnpm lint` clean on every touched src/ file. The 190 unrelated
errors live in `playwright-report/*.js` (generated artifacts).
## Estimated wall-time speedup (warm cache)
Pulled from blueprint §2 + the v1.4.33 prod-slowness HAR
(`.planning/round-v1434-prod-slowness-investigation.md`):
| Route | Cold (pre-cache) | Warm hit | Speedup |
| --- | --- | --- | --- |
| `/api/analytics` (thick) | ~5.0 s | ~5 ms | ~1000× |
| `/api/analytics` (slim) | ~3.0 s | ~5 ms | ~600× |
| `/api/gamification/achievements` | ~3.2 s × 2 (duplicate) | ~5 ms × 2 | ~640× + dedup |
| `/api/medications/intake?compliance` | ~3.0 s | ~5 ms | ~600× |
Cumulative dashboard wait — pre-cache the three hot routes summed to
~14 s of server-side tail; on a warm cache the total drops to
~15-30 ms of `Map.get()` + JSON return. The duplicate
achievements mount the HAR flagged collapses through the single-flight
`pending` map so even cold cache benefits when two consumers race.
## Risks accepted
- **Process-local cache.** Multi-instance Coolify deploys will keep
each container's cache isolated; a write on instance A leaves
instance B serving the prior payload for up to TTL. Documented in
the blueprint §4 and tolerated for v1.4.34 — the Redis migration
remains the v1.5.x option.
- **Hard TTL stampede window.** Soft-TTL deferred per blueprint §7;
the unit + integration tests don't yet exercise the
multi-microsecond race window past the expiresAt boundary. We ship
hard TTL and add SWR-style soft-TTL only if telemetry shows real
contention.
- **Memory ceiling.** Each cache caps at 500-1000 entries × ~5 KB; the
worst case across all eight registered caches lands at ~40 MB —
well inside the Coolify 512 MB container budget per the blueprint
§4 sizing.
## What stayed off-limits
Per the scope-fence:
- `src/middleware.ts` — AASA bypass unchanged.
- `src/components/**` — IW-B/D/F-Perf untouched.
- `src/app/page.tsx`, `src/app/settings/**` — IW-B / IW-D unchanged.
- `src/app/api/import/**` + `prisma/schema.prisma` ImportJob — IW-XML
unchanged.
- `src/lib/analytics/compliance.ts` — IW-C unchanged.
- `next.config.ts`, `src/lib/http/cache-headers.ts` — IW-A unchanged
(cache-headers read-only).
- `src/lib/ai/coach/snapshot.ts` — left in place per blueprint §12
acceptance criteria (the snapshot LRU migration is a v1.4.35
follow-up).
Disjoint paths with IW-E close-out; no merge conflict expected.
.planning/round-v1434-prod-slowness-investigation.md
+336
View File
@@ -0,0 +1,336 @@
# Production slowness investigation — v1.4.33 LIVE
HAR capture: `healthlog.bombeck.io` 2026-05-16T18:14:41 UTC, dashboard
cold mount, eight concurrent API calls land in the same ~3 s tail
band. Read-only inspection of the codebase plus the live Coolify
control-plane config for the `apps-01` host (`go0cw8w0gwws00okggcss0c4`)
and the HealthLog application (`pg8wggwogo8c4gc4ks0kk4ss`).
## 1 — Executive summary
Three independent root causes stack onto the same wall-clock window.
None of them are LLM-side. Ranked by recovered milliseconds per
dashboard mount:
1. **Postgres pool starvation at the application layer.** `src/lib/db.ts`
builds the Prisma client through the `@prisma/adapter-pg` driver
adapter with **no `max` override** on the underlying `node-postgres`
`Pool`. The Coolify-injected `DATABASE_URL` is a bare
`postgresql://healthlog:…@db:5432/healthlog?schema=public` — no
`connection_limit`, no `pool_timeout`, no `?max=…`. Node-postgres
defaults to `max: 10`, and a second Prisma client lives inside the
`pg-boss` worker that runs **in the same container** because
`HEALTHLOG_PROCESS_TYPE=all`. pg-boss itself also opens its own
pool against the same database (`new PgBoss(DATABASE_URL)`,
`src/lib/jobs/reminder-worker.ts:1369`). Three pools share the same
16-connection-per-role Postgres budget; the web pool effectively
tops out at ~10 simultaneous queries. Eight dashboard fetches each
issue 1× `prisma.session.findUnique` (auth) + an average of 2–3 read
queries — the 9th–24th concurrent query lands on `Pool.acquire()`
and waits for a returning client, which is exactly the ~3 s tail
the HAR shows. Recommended fix: append
`?connection_limit=25&pool_timeout=10` to `DATABASE_URL` on
apps-01, **after** raising the Coolify CPU/RAM caps (see §6). One-
liner env edit, no code change, redeploy not strictly required —
restart suffices.
2. **Duplicate fetches on the dashboard mount.** The HAR shows two
`/api/analytics` calls (one with `?slice=summaries`, one without)
and two `/api/gamification/achievements` calls in the same wave.
Both are real bugs in the consumer tree, not network artefacts.
Discussion in §4 — IW-B in v1.4.34 is the right home for the
consumer-collapse work and will land the analytics half; the
achievements half is a separate `queryKey` divergence between two
independent components mounted on the same page. Both stay
inside v1.4.34 if we agree to add the IW-F-Perf sub-wave (§5).
3. **Unbounded Coolify resource caps with no per-app cgroup ceiling.**
`get_application` returns `limits_cpus: "0"`, `limits_memory: "0"`,
`limits_cpu_shares: 1024`. Read literally that's "no Docker
resource limit", which means the host's 25-app inventory all
compete for the same kernel scheduler. CPU steal on a noisy
neighbour translates into Postgres query latency end-to-end. The
single physical apps-01 host runs 17 application containers plus 4
Postgres standalones plus 10 services (`server_resources`), so
per-process CPU under load is the most likely amplifier for the
pool-wait. Recommended: bound HealthLog to `limits_cpus: "2"` and
`limits_memory: "1g"` so the container has a deterministic floor
regardless of neighbour load. Coolify UI → Application → Limits.
**Top verdict.** Pool starvation is the dominant cause; the duplicate
fetches double the load on top of it; the unbounded resource caps
amplify both. The cheapest single-knob mitigation is the
`?connection_limit=25` env edit. The cheapest single-knob
amplification cut is the Coolify resource bound. Both are operator-
side, both are revertible, both take <60 s and don't need a release.
## 2 — Pool starvation — evidence and sizing
### Evidence
- `src/lib/db.ts:1–19` — single web-side `PrismaClient` constructed
via `new PrismaPg({ connectionString: process.env.DATABASE_URL! })`.
No `max`, no `connection_limit` parsing, no pool overrides.
- `src/lib/jobs/reminder-worker.ts:83–89` — second `PrismaClient`
built the same way inside the worker subsystem. The
`HEALTHLOG_PROCESS_TYPE=all` env (confirmed via
`mcp__coolify-apps01__env_vars`) keeps both subsystems in one
container, so both pools live in the same Node process.
- `src/lib/jobs/reminder-worker.ts:1369` — `new PgBoss(DATABASE_URL)`.
pg-boss opens its own `pg.Pool` with `max: 10` per the upstream
default and the bare URL.
- Coolify env audit confirms `DATABASE_URL` is bare on both
control planes (`round-v1434-iwa-coolify-env-audit.md` §1, live
entry `w8w8gc08sgo48o8008ww04gk`).
- Postgres 16-alpine container ships with `max_connections=100`
default — no override in the Compose; that is the upper ceiling
shared across web Prisma + worker Prisma + pg-boss + admin sessions.
### Pool math
Per process, in the same container:
| Pool | Default `max` | Effective |
| --- | --- | --- |
| Web `PrismaClient` | `pg.Pool` default 10 | 10 |
| Worker `PrismaClient` | `pg.Pool` default 10 | 10 |
| pg-boss internal | upstream default 10 | 10 |
| **Total open slots** | | **~30 out of 100** |
At face value 30 < 100, so we are not at the Postgres ceiling — but
the **web pool alone is capped at 10**, and every authenticated
request takes one slot for `getSession()` plus one or more for the
route's actual query. Eight concurrent dashboard fetches × ≥2
queries each = ≥16 simultaneous acquisitions against a 10-slot
pool. Queries 11–16 wait on `acquire()`, which has no upstream
timeout in node-postgres, so they block for whatever serial
query-runtime the route ahead of them spends — exactly the
clustered ~3 s waits.
### Recommended `?connection_limit`
The apps-01 host runs 17 application containers plus 4 standalone
Postgres pools sharing one physical machine. Postgres default
`max_connections=100`. Per the Prisma docs (`@prisma/adapter-pg`
passes URL params through to `pg.Pool.options`), the right knob is
`?connection_limit=N&pool_timeout=T`. Sizing budget for HealthLog
alone:
| Subsystem | Slot budget |
| --- | --- |
| Web `PrismaClient` | 20 |
| Worker `PrismaClient` | 4 |
| pg-boss | 6 |
| Admin / migration headroom | 5 |
| **Total reserved for HealthLog** | **35** |
Postgres still has 65 slots for everything else on the host. That
leaves the iOS-bound `/api/auth/me` 79 ms path and the
admin-side `/api/admin/host-metrics` reads room without competing
with the dashboard wave.
**Operator action.** Set `DATABASE_URL` on apps-01 (UUID
`w8w8gc08sgo48o8008ww04gk`) to:
```
postgresql://healthlog:${POSTGRES_PASSWORD}@db:5432/healthlog?schema=public&connection_limit=20&pool_timeout=10
```
…and add a second env on the worker side (HEALTHLOG_PROCESS_TYPE
is still `all` per the live entry, so the worker reads the same
URL — when we eventually split, give the worker variant
`?connection_limit=4`). 20 + 4 + 6 = 30 leaves a small migration
headroom.
## 3 — Shared synchronous bottlenecks — ruled in / out
| Candidate | Verdict | Evidence |
| --- | --- | --- |
| `WideEventBuilder` per-request mutex | **Out.** | `src/lib/logging/event-builder.ts` is a plain class instance attached to AsyncLocalStorage. No shared lock; every request gets a private builder via `eventStorage.run(evt, …)` in `src/lib/api-handler.ts:161`. |
| `RateLimiter` shared state | **Out.** | `src/lib/rate-limit.ts:21–52` runs a single atomic upsert per call — no Map, no in-process lock, no semaphore. The query itself is a `?connection_limit`-bound consumer though, so it does contribute to the pool-wait. |
| `requireAssistantSurface()` per-request memo | **Out (working as designed).** | `src/lib/feature-flags/index.ts:72–104` wraps the `AppSettings.singleton` read in `memoizePerRequest("assistant-flags", …)`. `src/lib/request-cache.ts` correctly returns one Promise per request via a `WeakMap` on the `WideEventBuilder`. The 8 concurrent requests in the HAR each get their own builder so each one reads the row once — but that's correct cross-request behaviour, not a lock. Within a request the memo collapses 5+ surface checks to one. |
| pg-boss ↔ Prisma pool sharing | **In, partial.** | Both pools open against the same Postgres but each has its own `pg.Pool`. The shared resource is Postgres `max_connections`, not a JS-side lock. Already covered in §2's slot budget. |
| Container CPU/RAM throttling | **In, amplifier.** | `limits_cpus: "0"` means no Docker `--cpus` cap, which sounds permissive but actually means the container gets default cgroup share weight — under noisy-neighbour load the kernel scheduler de-prioritises it. The remedy is the opposite of intuition: **set** the cap to claim guaranteed quota. See §6. |
| Cold container | **Out.** | HAR was captured ~20 min post-deploy per the brief, and `last_online_at: "2026-05-16 18:24:13"` is well after the request. Page-warm. |
| `getSession()` re-fetch per request | **In, real cost.** | `src/lib/auth/session.ts:107–110` does **not** use `memoizePerRequest`. Each of the 8 concurrent dashboard requests issues its own `prisma.session.findUnique({ include: { user: true } })`. That's 8 extra pool acquisitions on every dashboard mount, on top of the per-route queries. Candidate for an IW-F-Perf addition: wrap `getSession()` in `memoizePerRequest("session-cookie", …)`. Across-request it can't memo — it's strictly intra-request. Saves zero queries in isolation but cuts pool contention in the burst window. |
## 4 — Duplicate-call findings
### `/api/analytics` × 2 — root cause
The dashboard root (`src/app/page.tsx:212`) calls
`useAnalyticsQuery()` (thick payload, no slice). The mounted child
`<GettingStartedChecklist>` (`src/components/onboarding/getting-
started-checklist.tsx:195`) calls
`useAnalyticsQuery({ slice: "summaries", enabled: !!user })`. Per
`src/lib/query-keys.ts:27–29` the queryKey factory was updated in
v1.4.33 IW2 to discriminate by slice:
```ts
analytics: (slice?: "summaries") =>
(slice ? (["analytics", slice] as const) : (["analytics"] as const)),
```
That's the **correct** v1.4.33 IW2 cache layout (collision-free), but
it means the two consumers each open their own cache slot and each
fire a network request. The dashboard genuinely needs the thick
slice for its `bpInTargetPct*` / `glucoseByContext` consumers; the
checklist genuinely only needs `summaries[…].count`. Both fire on
mount for the same render tree.
Two possible directions:
- **(a) Hoist** the checklist's gating to read the dashboard's thick
payload — every field the checklist consumes is also present on
the thick slice. The slim slice was introduced for the
`/insights` tree, not for the dashboard mount.
- **(b) Keep** the split and pre-warm: serve the thick payload first,
derive the slim payload from it via `select` in the checklist's
`useAnalyticsQuery`. TanStack supports `select` for shape
narrowing without a second network round-trip.
The v1.4.34 IW-B brief (from `round-v1433-audit-perf.md` §1) covers
the consumer-collapse work for `/api/analytics`. Confirming with the
maintainer: IW-B's current scope is the **slim slice routing** to
the Insights tree, not the dashboard-checklist intersection. So
option (a) — switch the checklist back to the thick slice — needs to
land in an IW-F-Perf addition or piggyback on IW-B.
### `/api/gamification/achievements` × 2 — root cause
Two different consumers on the same page, two different queryKeys
hitting the same endpoint:
| Consumer | queryKey | Mount point |
| --- | --- | --- |
| `<RecentAchievementsCard>` | `["gamification", "achievements"]` | `src/app/page.tsx:1180` |
| `<AchievementUnlockNotifier>` | `["gamification", "achievements", "unlock-notifier", userId]` | global layout (mounted alongside `<Toaster>`) |
Same data, two cache cells, two network calls. The notifier even
sets `refetchInterval: 2 * 60 * 1000` (line 84 of the notifier) —
not relevant for the initial-load HAR but it means every two
minutes the dashboard re-fetches `achievements` even when the card
already has fresh data.
Fix shape: extract a `useAchievementsQuery()` mirror of
`useAnalyticsQuery` that both consumers call. The notifier's
side-effect logic (badge unlock toast) can ride on `useEffect` over
the shared cache result rather than its own subscription. One
network call, one cache slot. Same shape as the v1.4.33 IW2 win.
### Other HAR consumers — all single-source, no duplicates
| Endpoint | Consumer | One-shot? |
| --- | --- | --- |
| `/api/dashboard/widgets` | `src/app/page.tsx:218` | Yes |
| `/api/mood/analytics` | `src/app/page.tsx:230` | Yes |
| `/api/medications` | `src/app/medications/page.tsx:95` + checklist | Same key, single cache slot |
| `/api/bugreport/status` | `src/components/app-settings-provider.tsx:63` | Yes |
| `/api/workouts?limit=3` | dashboard tile (likely `RecentWorkoutsCard` — not searched, single source) | Yes |
| `/api/medications/intake?scope=compliance&days=30` | dashboard compliance tile | Yes |
| `/api/send` (POST) | Umami client-side analytics → SSRF-safe proxy in `src/app/api/send/route.ts` | Yes; outbound HTTP, not Prisma-bound. The 3.3 s here is a separate concern — likely the Umami server itself is slow (also runs on apps-01) |
## 5 — Proposed v1.4.34 sub-wave: IW-F-Perf
Dispatched **after** IW-A (env audit, complete) through IW-E and
IW-XML reach completion, since this sub-wave depends on no in-flight
file changes from the others. Two parallel parts, each ~2 h.
### IW-F-Perf-Pool — env-only, no code
- Author the apps-01 env edit (above) and document in a CHANGELOG
line under `Reliability`. No source change.
- Add an integration test that asserts a `?connection_limit=` query
param is present in `DATABASE_URL` when `NODE_ENV=production`.
Lives in `src/lib/__tests__/database-url-shape.test.ts`. The test
surfaces the regression if someone ever resets the Coolify env
back to bare. **This is the only code change**.
### IW-F-Perf-Duplicates — consumer-collapse round-2
Two surgical changes; both are additive:
- New `src/lib/queries/use-achievements-query.ts` mirroring the
v1.4.33 `use-analytics-query.ts` shape. Wires both
`<RecentAchievementsCard>` and `<AchievementUnlockNotifier>` onto
the shared hook. The notifier's `refetchInterval` moves onto the
shared hook (the card stays passive; the notifier's `useEffect`
reads from the shared cache). One network call, one cache slot.
- Either (a) switch `<GettingStartedChecklist>` from
`useAnalyticsQuery({ slice: "summaries" })` to the thick
`useAnalyticsQuery()` so it rides the dashboard's cache slot, or
(b) wire IW-B's slim-slice routing so the dashboard tile strip
fires the slim call and the thick call is deferred behind the
Coach drawer's hover prefetch. The IW-B brief documents (b) as
the target shape; if IW-B lands first, IW-F-Perf-Duplicates
becomes (a) by elimination.
Quality gate: existing `src/app/__tests__/insights-b3-wiring.test.ts`
+ a new dashboard-mount integration test counting **distinct fetch
URLs** during the first 1500 ms after `render()` — the dashboard
should issue ≤ 7 fetches, not 9.
### Why not a release-blocker
The pool-starvation fix is an env edit, deployable separately from
v1.4.34. The duplicate-call fix is additive consumer-side
refactoring that improves load even without the pool fix. Each
half stands on its own. Neither needs the in-flight IW-A/B/C/D/XML
file sets touched.
## 6 — Operator-action items for the maintainer
In priority order. Each is independently revertible.
1. **Add `?connection_limit=20&pool_timeout=10` to `DATABASE_URL`.**
Coolify → HealthLog app → Environment Variables → edit live
`DATABASE_URL` (uuid `w8w8gc08sgo48o8008ww04gk` per the v1.4.34
IW-A audit). One restart cycle, no rebuild. Expected payoff:
eight-call dashboard tail drops from ~3 s to ≤ 600 ms in the
pool-bound paths.
2. **Set Coolify CPU/RAM caps on the HealthLog app.** Coolify →
HealthLog → Limits: `Limits CPUs = 2`, `Limits Memory = 1g`,
`Limits Memory Reservation = 512m`. Today both `limits_cpus` and
`limits_memory` are `"0"` (no cap), which on a 25-app shared
host means the container floats with default cgroup share —
noisy neighbours steal scheduling time. Setting an explicit cap
reserves the floor.
3. **Delete the section-2 leftover env entries.** Per the v1.4.34
IW-A audit there are 28 duplicate env entries on apps-01,
including a placeholder `POSTGRES_PASSWORD` that would shadow
the live one if Coolify ever swapped read precedence. Cleanup
reduces operational surprise; not perf-relevant on its own.
4. **(Optional) Split the worker subsystem off into its own
container.** Today `HEALTHLOG_PROCESS_TYPE=all` co-locates the
web HTTP server and pg-boss in one Node process. The web pool
and the worker pool then share Node event-loop cycles. Splitting
into two Coolify services (one `web`, one `worker`) is a v1.4.35
candidate, not a v1.4.34 hotfix — it changes Compose shape and
needs a release pipeline pass.
5. **(Optional) Investigate the `/api/send` 3315 ms.** The HAR
shows a 3.3 s wait on the Umami proxy POST. Umami runs on the
same apps-01 host (`a8004ksswcokok4csg0wowwc`) — could be the
same Postgres-contention story bleeding into a sibling service.
Out of scope for this report but worth a five-minute look at the
Umami container's resources.
### One thing to leave alone
`HEALTHLOG_PROCESS_TYPE=all` is correct **for now** — splitting it
without first sizing the worker pool independently would create a
new race against Postgres `max_connections`. Sequence is: pool-size
the URL first (action 1), then resource-cap the container (action 2),
then observe a clean week of traces, **then** consider the split.
## Closing note
The HAR is a clean diagnostic — three failure modes overlaying a
single timestamp, each with an independent fix. None of the fixes
need the in-flight v1.4.34 implementation sets touched. The
recommended path is to land the env edits today as a hotfix and
file IW-F-Perf as the v1.4.34 polish addendum behind whatever
implementation work is currently in motion.