A module mock that leaves out an export the code under test reaches throws
at the call, and where that code catches and logs (a best-effort tail, a
fail-soft read) the test went on green while the path it meant to cover
never ran. The Apple Health import tests mocked the rollup module without
its fold window, so the import tail's rollup fold and cache invalidation
never ran in any of them; four route test files mocked the logging context
or the rollups without what the measurement tail reaches.
Both setup files now watch the console and fail the test in which such a
line was logged. The ECG import test keeps the real rollup module and
stubs only the fold, and the four unit mocks gain what they lacked.
The lab, medication and measurement CSV routes ignored parameters they
did not know, so a misspelt dryrun=1 fell back to a real import and
wrote the file the caller meant only to check. Their query schemas are
strict now: an unknown parameter answers 422 before anything is read
or a rate bucket is spent.
Two streamed endpoints read a spreadsheet record by record with a
column mapping from the query: separator, decimal mark, date order and
zone are stated, never guessed per row. Lab rows resolve to markers the
way scanned reports do, convert units where the conversion is known and
keep what the file said; medication rows map name, dose, daily times,
as-needed and a dated window, checked against the create schema, with
reminders off unless asked for. A dry run writes nothing, a re-upload
only finds duplicates, and every refused row comes back with its line
and reason.
Every provider sync and the shared mutation tail refolded its touched
(type, day) buckets one at a time: an aggregate, a transaction and three
queue sends per day. They now hand the whole set to
recomputeBucketsForKeys, which leaves the DAY tier exactly as the per-key
hook did (pinned by a parity test against it) at the cost of one
aggregate per chunk of keys and one coarse job per run of buckets.
An upload refused for its size, a broken stream or a stopped worker is an
operational outcome, not something done to the account. The admin import
keeps its audit rows, since an admin acting on another account is.
Output of `node scripts/strip-planning-tags.mjs --write`: 3,157 comment
lines in 1,308 files lose the work-item identifier at their head. Release
versions and issue references stay; nothing outside comment text
changes, which the script checks per file against a comment-free
printout. healthlog/no-planning-tags is enabled at error level with this
commit, so the tree it lands on is clean.
A sync from a watch or an integration now waits the instance's sync gap
(AppSettings.aiSyncRegenGapHours, else AI_SYNC_REGEN_GAP_HOURS, else 6 h)
before it may regenerate a status card again; a hand entry keeps the hour.
Every caller of afterMeasurementMutation and invalidateStatusInsightsForTypes
names its source, so a new ingest path cannot skip the choice.
Refs #1217
A dash between two parts of a sentence ("{what} — tomorrow", "Could not
record {name} — try again", "End {name} — …") stands in for a comma, a
colon or a full stop the sentence should say. Every such string in the
seven bundles now says it in plain words. A range keeps its unspaced en
dash ("70–100 mmHg", "{start}–{end}"); that is how a range is written,
not a pause, and the spaced range forms became unspaced ones.
Labels the components put together themselves follow the same rule:
aria-labels read "Move up: Ramipril", a medication and its dose read
"Ramipril 5 mg", page titles use "|", Coach prefills are two sentences,
and the API's user-facing error messages lost their dashes too.
The bundle guard now refuses " — " and " – " beside " · ", with an empty
allowlist, and a sibling guard walks src/components and src/app for a
string or template literal that joins its parts with a spaced dash.
Upload the ZIP the Android Health Connect app writes and import its
`health_connect_export.db` in the background, the same way an Apple
Health export is imported: streamed to disk with a SHA-256, capped at
1 GiB, one import per account, a job row of kind `health_connect` that
the shared reconcile cron and staging sweep now look up on its own
queue.
The worker extracts the database (4 GiB ceiling, zip-bomb checks) and
opens it read-only with `immutable=1`, `trusted_schema=OFF` and
`query_only`. A view in place of a record table refuses the file,
columns are found by name, and schema versions below 9 are refused.
Records are read in batches of 500 and aggregated in SQLite: steps,
energy and distance as one total per day from the app the person's
Health Connect priority list ranks first, heart rate per minute for 90
days and as hourly means before that, sleep stages with the unknown
stage counted only in sessions without real stages, workouts, cycle
days, water and micronutrients. Apps whose integration the account has
connected are left out, and a reading already stored under another
source is skipped. Rows carry source HEALTH_CONNECT, `hc:<uuid>` or the
`stats:` ids, so a second import of the same export writes nothing.
Both staged files are removed on every way out; the job result holds
counts per type and per app only. Settings gain a Health Connect card
with the export steps, progress and the outcome, in all seven
languages, and the import is documented in
docs/integrations/health-connect.md.
Land every file the v1.42 work shares before any of it is built, so the
parts built side by side never edit the same file.
Schema and migrations 0379-0386: the HEALTH_CONNECT measurement source,
ImportJob.kind, the air-quality columns and the per-user switch on the
environment tables, sealed environment locations with nullable plaintext,
managed-profile handover links, the per-type HealthKit arrival ledger, lab
unit-conversion provenance, and user-defined vaccines. Every new column is
annotated for the column-reader sweep.
HEALTH_CONNECT reaches every exhaustive list: the source labels, the
read enum (not the write allowlist), the source-priority ladder cap, now
the number of sources rather than a fixed eight, and the write-path smoke
test. RMSSD gets its own priority class and a 300 ms ceiling. The batch
syncTrigger gains `manual`, with its label on the Apple Health card.
Queues for the Health Connect import, the compaction-tombstone purge and
the measurement-table maintenance are registered with stub handlers;
host-metric sampling moves to every five minutes. Environment and work_mem
variables reach the compose whitelist, the production example and the
manifest. The proxy admits the handover link and redirects it at the edge,
and the Health Connect upload bypasses the proxy body limit.
The new routes answer 501 behind their final authentication and are
published in the contract; backup, wipe and key rotation classify the new
models and columns; the seven message bundles carry the base copy. Guards
that list release-pending keys and columns fail once an entry is wired, so
those lists empty themselves before the tag.
Also re-anchor a login-throttle fixture to the wall clock, where its
expiry is compared, and re-base the aggregate bundle cap on a measured
12 KB of new locale strings.
A re-import that moved a reading to another instant re-folded only the day
it moved to, so the day it left kept the reading in its rollups. And a
write chunk that failed part-way ended the request before the re-fold ran;
a retry then read the landed rows as duplicates and touched nothing, so
those days kept their pre-import rollups for good. The re-fold now covers
the old instant of a moved row and runs whether or not the writes finish.
The extractor held the inflated export.xml to a flat 8 GiB whatever the
member occupied, and the header ratio check reads sizes the archive
writes itself: a member declaring an honest ratio could inflate at
1000:1, so an 8 MB upload could fill the temp directory. The streamed
byte count is now held to 200 times the member's compressed size on
disk (never less than 64 MiB, never more than 8 GiB), and a stored
member may not grow at all.
An account may have one Apple Health import queued or running. A
second upload with different bytes is refused with 409
import.apple_health.busy and the running job's id; the check and the
new row are one transaction under an advisory lock.
The staged upload is the person's export in plain text. It is now
removed on every exit that leaves no worker to read it: a refused or
broken multipart body, no worker running, a queue that takes no job
(the row is failed as well), the admin route's missing or unknown
target account, and an account deleted before the worker reached it.
The worker removes the files before recording a failure, so a row that
went with a deleted account no longer strands them. The 15-minute
import reconcile pass also sweeps staged files older than the longest
an import may run.
Re-uploading a large export with an externalId column ran one
updateMany per row inside a single interactive transaction; on a
9 000-row file that outlived Prisma's 5 s timeout and the request ended
in a 500.
A re-imported externalId row whose stored value, unit, instant, glucose
context and note already match the line is now a duplicate: not written,
reported as skipped / duplicate. Changed rows still update, a tombstoned
one is still resurrected. The existence probes run per measurement type
in chunks of 1 000 keys, and the writes commit in chunks of 200 (creates
via createManyAndReturn, updates as one batch transaction per chunk)
instead of one transaction around the whole file. A failure part-way
leaves earlier chunks written, and re-uploading the same file completes
the rest.
The OpenAPI description now says what a re-upload does, including the
natural-key dedup for rows without an externalId, which it used to call
a duplicate.
Next reads the body of every non-GET request the proxy matches into
memory before the route runs. The ceiling was 512 MB so the Apple Health
importer could receive whole archives, which let any anonymous POST to any
API path make the server hold that much.
The ceiling is now 1 MB (proxyClientMaxBodySize). The fifteen routes whose
real bodies are larger are left out of the proxy matcher and read their
own body after authenticating; apiHandler gives them the proxy's refusals
(worker-only, demo mode), security headers and a CSP. safeJson now counts
bytes while it reads and stops at its cap instead of reading everything
first, the CSV and dose-history imports do the same, and a multipart
backup upload is counted while it is read. A test compiles the matcher the way
Next does and checks it skips exactly the listed routes and every route
with a larger body is listed.
Local lab OCR runs its WebAssembly engine in a worker served from
/tesseract/. Those responses, and the /labs and /documents pages that start
the OCR, now carry 'wasm-unsafe-eval' in script-src; no other route does.
The status assessments folded their recent days, weeks, months and years on
Berlin days whatever the user's zone, the mood and cycle crosstabs keyed
their metric days in Berlin, the resting-pulse proxy defaulted to Berlin
days, the briefing's opener rotated at Berlin midnight, and every
assessment prompt labelled today's date as Berlin's. They now use the
user's zone. Rollup months and years are labelled by their UTC start,
which is how they are cut; read in a zone west of UTC the label named the
month before.
"Europe/Berlin" was written out in some eighty places. DEFAULT_TIMEZONE in
lib/tz/format is now the one definition: the second copy in mood/date-key,
DISPLAY_TIMEZONE, the Berlin-only day key helpers and the per-route copies
are gone. An in-repo lint rule rejects the literal outside lib/tz and
tests; four areas another change owns are listed as pending.
The MCP writes, the Telegram reply, the CSV and backup importers and the
Apple export import refreshed the rollups by hand and never re-warmed the
cached status assessments, so a reading logged there kept the day's old
assessment until the nightly pass. They now run the shared post-write tail,
and the export import re-warms every type it wrote. The WHOOP body sync
removed its legacy weight row without refolding that day; it does now.
A structural test lists every file that writes measurements and how it keeps
both readers current, so a new write site fails until it says.
Fifty-nine 422s raised off a Zod `safeParse` answered with one opaque
sentence and nothing a client could map onto a field, while the other
166 sent `details.issues` — the multi-issue envelope that exists so a
caller with three bad fields fixes all three in one round-trip. Twelve
of the fifty-nine put a dotted machine token in `error` where the
sentence belongs, so `error` was sometimes text to show a person and
sometimes a token that must not be shown, with nothing in the payload
to tell them apart.
Every one of them now answers through `apiValidationError`, which keeps
the existing `error` string and the existing `meta` byte for byte and
adds the issue list beside them. The dotted tokens keep their place in
`error` for the clients that already match on them and are published in
`meta.errorCode` as well, which is the field a machine code belongs in
and the one a new client should read.
A guard reads the route tree per refusal site rather than per file —
per file is what let this grow, since a route that used the helper once
looked compliant to a `grep -l` sweep — and asserts it still finds the
sites it judges, so a matcher that stops matching fails instead of
reporting a clean zero.
Three assertions repeated 18.016 and the product it gave. They were the
reason the two factors could sit in the tree at once looking correct. They
take the exported constant now, and the parser test asserts the round trip
a reader on mmol/L experiences rather than a magic product.
- one helper resolves the level-A values for a write; the five writers
call it instead of carrying their own copy of the derivation
- the create schema takes a1 to a5, and a value the user set wins over
the derived one; the batch request shape is untouched
- both arms of both upserts restate the values, so a re-post cannot
leave a stale reading beside a mood that has since changed
- the JSON import derives from the row's own label rather than the
file's score, which is the one place the two can disagree
- correcting an entry's label re-derives pleasantness the way it
re-derives the score; an explicit null takes an answer back
- a round trip through all five writers against real Postgres asserts
the rows agree and that nothing invents the other four values
Same shape as the CSV and JSON cards: reaching the done state rendered the
success tick whether or not a single row was upserted. The archive's count
picks the tone now, and an archive that wrote nothing says so.
The clinical-record count stays out of that decision. Skipping
electrocardiograms and lab documents is a documented, deliberate exclusion,
not a refused row, so it must not read as a warning.
Refs #640
The dedup key on the medication intake surfaces is not only `externalId`.
`MedicationIntakeEvent.idempotency_key` is a `@unique` column persisted
with the dose row forever, with no TTL and no sweep, matched by plain
equality on the per-medication POST, the bulk drain and the external
ingest bridge. That makes it an identity, not a replay token, and it
carries the same failure mode: a key that rotates per client launch never
matches its own earlier post, so every retry logs the dose again. The
blank case is sharper still — the column is unique table-wide, so an
empty key resolves to whichever unrelated row stored it first. Refuse
both shapes at the schema on the single-entry paths and per entry on the
bulk drain.
The MCP write path hashes a host-supplied key into the `externalId` half
of its dedup key, so the shape is laundered before anything downstream
could see it and the check has to run on the input. That closes a live
one: every tool handler falls back to `""` when the host omits the
argument, and `""` digests to a constant, so each keyless write after the
first reported `already_logged` and was silently dropped.
The HTTP `Idempotency-Key` header stays out, and now says why in the
guard: `IdempotencyKey.expiresAt` bounds it and the cleanup job sweeps
it, so it is a windowed replay token and rotation is its design.
Extend the structural guard with a provenance rule for identities that
never pass a Zod request schema — a spreadsheet column, an MCP tool
argument. Inside the import and MCP trees, a module handling an identity
must apply the floor itself or import a module that does. That is a real
chain rather than a list of known offenders: a new importer that mints an
`externalId` from file input and imports nothing checked fails it, with
no allowlist to escape into.
An external id is half of every ingest dedup key — (userId, source,
externalId), (userId, externalSource, externalId), (userId, type,
source, externalId). Re-posting the same triple is idempotent and
returns the existing row, which holds only while the id is stable
across client launches. A client that serialised an opaque platform
object with its default object description sent a per-allocation
memory address instead, so no sweep ever matched its own earlier row
and each one minted a fresh record: a live instance grew 23 phantom
medications in a day, none of which could collect a dose, because the
dose side resolved through the same rotating string.
Add one shared classifier for identifiers that provably cannot be an
identity — an angle-bracketed object description carrying a hex
pointer, a bare 0x address, and a blank value — and wire it at every
surface that accepts a client-supplied externalId.
Single-entry creates refuse through the schema, so the client gets the
standard 422 multi-issue envelope with a message naming the shape it
sent and what to send instead. Batch and bulk surfaces classify per
entry and report the offending row as skipped with reason
unstable_external_id, leaving the rest of the payload to land: one bad
row must not stop a sync. Every refusal carries pinned wide-event meta
(count, shape, surface) and never the id itself.
A structural guard freezes both routes into the floor: a request-side
externalId declaration either applies the field check or carries a
marker naming the file that checks it per entry, and the guard verifies
that file really does. The backup-restore payload is the one exemption,
with its reason recorded — it replays ids the server itself wrote.
Mood has been a first-class module here for a long time, with entries,
structured tags, rated factors and notes. The bridge to the external moodLog
service had nothing left to add, and its Settings card was withdrawn in
a56e6fe61 on the promise of a dedicated removal. This delivers it.
Removed: the hourly `moodlog-sync` queue, cron and handler binding; the
inbound webhook and its shared-secret handling; the reverse push and its four
call sites; the pull-sync module; the per-user credential routes and the
instance-wide toggle; the entry on the `/api/integrations/status` envelope,
the `IntegrationKey` union, the cadence map and the display-name table; the
boot-time secret rotation pass; 22 message keys per locale.
Migration 0271 drops the five `users` credential columns and
`app_settings.mood_log_global`, clears the ledger row for the provider, and
deletes the retired pg-boss schedule so no cron keeps enqueuing work no
handler drains.
Mood data is untouched. `mood_entries` is not named anywhere in the migration
and cannot be reached from it: the dropped columns carry no index, constraint
or foreign key, and `mood_entries.user_id` references `users(id)`, not any of
them. Rows the bridge imported keep `source = 'MOODLOG'`, and every read path
- list filter, source badge, exports, backups - still resolves that value.
`src/lib/validations/moodlog.ts` held the native mood-entry schemas alongside
the bridge's own; it is now `validations/mood.ts` and carries only the former.
The v1.28.33 rollup + staging fixes were correct but never reached the
reporter's case: the upload dedup keys on the content SHA-256 and the
lookup matched a previously FAILED job too, so re-uploading the same
export.zip returned the dead job's id and replayed its stale
failureReason without ever staging or sending again. The dedup now
excludes failed rows (short-circuits only on queued/unpacking/parsing/
upserting/done); the same bytes retry cleanly. A redundant staged upload
is unlinked on a dedup hit instead of leaking in the staging dir.
reconcileOrphanImportJobs no longer unconditionally fails every
non-terminal ImportJob at boot: it now checks pg-boss job state plus a
progress heartbeat (new import_jobs.updated_at, migration 0242), so a
split or multi-worker deployment cannot kill an import running in
another worker. The missing-staging message names the separate-container
cause explicitly, with a matching note in docs/self-hosting/scaling.md.
Rollup writes were delete-then-createMany inside a transaction; under READ
COMMITTED a concurrent recompute racing the import's full fold on the same
(user, type, granularity, bucket, source) partition tripped the composite
PK — and the P2002 swallow dropped the loser's freshly computed aggregate,
leaving a stale bucket. persistRollupRows now writes chunked
INSERT … ON CONFLICT DO UPDATE with deterministic (type, source, bucket)
lock ordering inside one transaction: a race can neither fail the import
nor lose an aggregate, and previously-staled buckets self-heal on their
next recompute. The vanished-source delete and in-transaction guarantees
are preserved; the now-unreachable P2002 swallow is removed.
The masked ENOENT is gone with it: long imports outlived the queue's
15-minute expiry, pg-boss redelivered the still-running job, and the
redelivery re-opened the consumed staging file — overwriting the real
terminal state with a raw ENOENT. Import sends now carry retryLimit 0 +
6h expiry (a retry after the staging file is consumed can only mask),
deliveries for terminal ImportJob rows are no-ops, a genuinely missing
staging file reports an honest re-upload message, and the extracted XML
is cleaned up on failure too.
Platform-wide hardening: the failure classes found live this week hunted down
across every integration.
Fitbit carried the same tombstone wedge fixed for Google — probe now matches
soft-deleted rows and the update resurrects them; the comment describing a
partial unique index that never existed is corrected. Sleep segment ids are
stable across re-scoring for Withings (session+segment-start), WHOOP/Polar
(one segment per stage, no positional index) and Oura (run-start keyed), each
with a session-scoped sweep that soft-deletes rows a re-score orphaned — so
legacy and affected nights heal as they are re-read.
One platform rule for deleted rows: source-owned readings resurrect on
re-import (Withings, WHOOP, Oura, Polar, Nightscout, mood webhook, CSV
re-imports, Apple stats day-totals); Apple sample-grain deletions stick per
the paired-client contract, and the mood bulk route's tombstone match is now
a true no-op reported as duplicate. Import counts reconcile against actual
createMany counts.
Google lifecycle: a dead token registers on the cycle's hard-fail ledger, so
an expired connection parks instead of stamping success hourly; backfill and
sleep-repair markers stamp only after a clean pass (pg-boss retries are now
real); failed measurement writes hold the watermark; no-op updates are
skipped; mapping.md tells the current truth.
Scale: doctor-report reads only consumed columns and enabled sections;
exports and backups keyset-paginate; achievements tally via SQL buckets;
series charts day-bucket dense kinds beyond 90 days; BP pairing is a linear
two-pointer; distinct-type scans run server-side; the graded-series cold
fallback is bounded; the fitbit defer tracker loops instead of spreading.
The CSV importer ran a serial prisma.upsert/create per accepted row, so
an N-row file cost N round-trips and a few-thousand-row import could
approach the request timeout. Adopt the /measurements/batch shape:
pre-fetch existing rows under the IMPORT source in one query per
dedup-key shape (externalId and natural-key paths), partition into bulk
createMany + grouped updateMany, and reconstruct the per-line
inserted/updated/duplicate envelope from the partition. In-file repeats
and re-import idempotency preserved; response contract unchanged.
Lay the account-security and data-sovereignty substrate for v1.23. No
user-facing behaviour yet — schema, crypto registry, redaction, and the
note-encryption data path only.
Schema + migrations:
- 0199 — User totp_secret_encrypted / totp_confirmed_at / mfa_enforced /
passkey_upgrade_nudge_dismissed; Session mfa_verified_at / last_active_at;
Passkey last_used_at. All additive, nullable / defaulted.
- 0200 — mfa_challenges (single-use step-up ticket; only the ticket hash is
stored, so no session or token is ever minted before the second factor),
mfa_recovery_codes (Argon2id at rest), webauthn_mfa_credentials (second-factor
keys, kept separate from passkeys, which remain the passwordless-primary set).
- 0201 — mood_entries.note_encrypted + measurements.notes_encrypted (BYTEA),
matching the existing free-text encrypted-note convention.
Free-text health notes:
- New writes encrypt the mood / measurement note into the *_encrypted column and
null the legacy plaintext column; reads prefer the ciphertext and fall back to
plaintext only for not-yet-migrated rows, via a single read/write helper.
- A boot-time job migrates existing plaintext rows in an idempotent,
transactional, fail-closed pass (encrypt, then null the plaintext, per row in
one transaction; safe to re-run; a row with content is never left both-null).
- The legacy plaintext columns are kept this release and dropped in a follow-up
once the backfill reports zero remaining rows — the same boundary the
key-rotation playbook uses before dropping a legacy key.
Crypto coverage:
- Register the three new encrypted columns and wire them into the key-rotation
script so a rotation never orphans them.
- Extend the observability and log redaction denylists so TOTP secrets, MFA /
recovery / backup codes, and export passphrases never reach a wide event.
One-shot prettier pass so the new format:check gate starts from a clean
baseline. Formatting only — no behaviour change. Generator-owned and
vendored artifacts stay untouched via .prettierignore.
A self-hoster migrating from a spreadsheet or a meter export overwhelmingly
has CSV, which neither the JSON nor the Apple Health import covers. Add
`POST /api/import/csv` reusing the existing write loop and rollup re-fold.
The documented header is `type,value,unit,measuredAt[,glucoseContext,notes,
externalId]`, order-independent. measuredAt must carry an explicit ISO-8601
offset — a row without one is skipped rather than guessed — and is bounded by
the shared entry-instant validator. Glucose accepts mmol/L (converted to
canonical mg/dL) and weight accepts lb (converted to kg); any other
non-canonical unit is skipped, never mis-stored. An optional externalId
column makes a re-upload idempotent; without it a re-upload duplicates.
The response carries a per-row status envelope so a 2000-row file with three
bad rows imports 1997 and names the three. `?dryRun=1` validates and previews
the outcome without writing. The parse, validation, and unit conversion live
in a pure module so they are unit-tested without a database.
Settings -> Export & Import gains a third CSV card mirroring the JSON and
Apple Health cards, with upload, paste, preview, and a downloadable example.
The new route is documented in the OpenAPI contract.
The generic JSON import validated measurement `measuredAt` with a bare
`z.string().datetime()` — no ceiling — so a crafted payload could forward-date
a reading, the gap the single-POST plausibility bound already closed. Route
the measurement and mood instant fields through the shared
`validateEntryInstant` (no future beyond a 5-minute skew, no instant before
1900).
Measurement rows were also always created with a NULL externalId, which is
distinct in the `(userId, type, source, externalId)` unique key, so a
re-import duplicated the whole set. Give measurements the same optional
externalId upsert mood already has: present -> idempotent upsert, absent ->
first-write-wins create.
The client bundle carries only the active locale — resolved on the
server and handed to the provider with no hydration flash, halving the
dashboard route to under a megabyte parsed (288 KiB gzip). The auth
shell renders the app chrome immediately instead of a spinner while the
session resolves in parallel, the widgets request folds into the
snapshot it duplicated, and insight pills prefetch their sub-pages —
a first visit to a sub-page lands in about ninety milliseconds instead
of nearly a second.
Raising the middleware body cap to 512 MB for archive imports removed
the implicit ceiling that protected JSON routes, so any bulk endpoint
would buffer an arbitrarily large body before validation ran. Every
JSON body read under /api now carries an explicit byte cap sized to its
legitimate payload (64 KB for single-record routes up to 16 MB for the
measurement import), webhooks cap before signature work, and oversized
bodies return 413.
Marc-André Bombeck