The batch route counted arrivals inline and grew past its size baseline
with the new-sample rule. tallyHealthKitArrivals now owns the fold, the
route hands it each landed entry and hour with whether it brought a new
value, and the size baselines move down to what the route is now.
Refs #1259
A batch that carried a type with duplicates only moved its
lastReceivedAt, lastTrigger and lastBackgroundAt, so a type whose live
delivery had stalled kept reading as freshly arrived for as long as a
replay, a retry or an older client re-sent its old samples.
The arrival ledger now moves those stamps only for a type the batch
brought at least one new or changed row for. An unchanged stats:
re-post and an hour posted again with its stored value count as
duplicates here; the wire keeps answering updated for them. A
duplicates-only type stamps the new last_seen_duplicate_at column on
its existing row and nothing else. The intake bulk route's dose entry
follows the same rule through the shared recorder. lastSyncedAt and
healthKitLastSyncedAt are unchanged.
GET /api/integrations/healthkit serves the stamp additively as
metricFreshness[].lastSeenDuplicateAt, and the Apple Health card names a
re-send that came after the last arrival.
Refs #1259
Every sync marked all of an account's analytics cells stale, so the slice
re-read the pre-fold remainder (every reading older than five years,
folded per source and day, the slice's largest statement on a long
history) and the day view re-read its past history blocks, although a
sync of today's readings cannot change either.
A cache key can now name the instant its value's readings end before
(closedBlockKeyPart). invalidateUserMeasurements takes touchedFrom, the
earliest measuredAt a write created, changed or removed, and leaves such
blocks in place when it is not earlier; without it every block goes as
before. The batch route passes it from the identities it wrote. The
pre-fold remainder moves into its own module and is cached as a closed
block; the day view's past history blocks carry their end, with a day of
margin for a night across midnight.
The batch route's two up-front reads move into batch-stored-rows.ts with
the once-per-batch key rule they share.
The phone re-sends the same daily totals on every sync. Each went through
the reconciler, about ten round trips in its own transaction, to write the
same values back, and as an update it started the DAY refold, the rollup
jobs, PR detection and a cache stale for the account.
One read over the batch's stats: ids now finds the rows whose live stored
row already holds every value the update would write; those are reported
updated exactly as before, per entry and in the summary, and nothing is
written or started for them. A changed column, a tombstoned row, a key
that occurs twice in the batch or a value the plausibility gate refuses
still goes through the reconciler.
POST /api/measurements/batch ran one interactive reconciler transaction
per fresh sample: a savepoint, three advisory locks, two candidate reads
and the insert, about ten round trips. A 500-entry upload of new samples
was 500 of those, the 2 to 15 second outliers. Fresh samples now land
through INSERT ... ON CONFLICT DO NOTHING under the same per-account
advisory lock. Both measurement identities are full unique indexes, so
the statement inserts exactly what the reconciler would have inserted,
and every skipped row still goes through the reconciler for its own
verdict. Only rows whose external and natural keys each occur once in
the batch are buffered, and the buffer is flushed before any other row
is reconciled, so the order of effects stays the sequential one. The
contract and every per-entry status are unchanged; a golden test runs
each batch through both paths against the same table.
/api/health took seconds after a start because register() awaited the
whole pg-boss worker start (queues, schedules, boot discovery) before
the server took a request. The worker, or a web process's producer, now
starts in the background after the blocking boot checks; the probe
answers at once with reason "starting" until it is up, and gives the
database two seconds before calling it not answering.
169 of 364 mutating handlers met no rate limit, every PATCH and DELETE on
an [id] among them. The shared per-account write bucket (300 a minute,
keyed on the acting account) was called by hand in a dozen creates; it is
now charged by the identity resolvers for any POST, PUT, PATCH or DELETE,
once per request, so a route is metered by resolving its caller.
The bulk ingest routes the native client syncs through in bursts
(measurements, workouts, intake, mood and cycle bulk, nutrients, the ECG
sweep) declare writeBucket: "own" and keep their dedicated ceilings, so a
large HealthKit backfill cannot lock the person out of logging a dose. A
read-over-POST stays free.
write-rate-limit-guard fails on any mutating route that neither resolves
its caller through a metering resolver nor calls a limiter, unless it is
on a short reasoned exception list; an opt-out must be on its own reviewed
list and call its own limiter.
Output of `node scripts/strip-planning-tags.mjs --write`: 3,157 comment
lines in 1,308 files lose the work-item identifier at their head. Release
versions and issue references stay; nothing outside comment text
changes, which the script checks per file against a comment-free
printout. healthlog/no-planning-tags is enabled at error level with this
commit, so the tree it lands on is clean.
A *.validation-failed row was written on every malformed request, the
caller's own included, and the PR detection enqueue wrote one per batch.
Neither is a security event and nothing reads them. auditLog now files a
refused attempt only when a delegate made it, for the owner's activity
feed; otherwise it rides the wide event as meta.rejected_attempt. The PR
detection enqueue moves to annotate().
A sync from a watch or an integration now waits the instance's sync gap
(AppSettings.aiSyncRegenGapHours, else AI_SYNC_REGEN_GAP_HOURS, else 6 h)
before it may regenerate a status card again; a hand entry keeps the hour.
Every caller of afterMeasurementMutation and invalidateStatusInsightsForTypes
names its source, so a new ingest path cannot skip the choice.
Refs #1217
Migration 0401 turns healthkit_type_syncs.type into text so the ledger can
hold MEDICATION_DOSE, and adds last_background_at (seeded from rows whose
latest arrival was a background one). POST /api/medications/intake/bulk
takes syncTrigger and records APPLE_HEALTH dose arrivals; the Apple Health
status read serves a MEDICATION_DOSE freshness entry and lastBackgroundAt
on every entry with arrival facts.
- Publish timezone, glucoseUnit and a features block on the account
payload; glucoseUnit reads as the closed pair the contract names, and
features.trackIntake tells a separately released client that
medications take the record-only switch.
- Refuse a new dose for a record-only medication with
medication.intake.notTracked on every dose-writing route; the bulk
route reports it per entry as intake_not_tracked, and correcting an
already resolved event stays open.
- Answer passkey sign-in failures by name instead of a 500:
passkey.challenge.expired, passkey.unknown, passkey.response.invalid
and passkey.verification.failed, each with an audit row.
- Carry one account-wide adherence figure in meta.aggregate on the
batched compliance read, weighted by expected doses, and accept
?days=30|60|90|180|365 for a report window.
- Accept ?days=30|90|180|365 on the mood insights read so stability
covers the period shown, and name the window it covers.
- Keep only the reminder that announces a dose off APNs when the phone
owns medication reminders; follow-ups for a dose still open are
pushed again, marked by a followUp flag the reminder cron sets.
- Document which batch validation issues belong to an entry and pin it
with a test that splits and resends a batch; publish rejectedFields
items as ValidationIssue.
The nightly folds soft-deleted every raw Apple Health sample they folded
into an hourly or daily stats row and kept it for 75 days. Nothing ever
read those rows as deletions; they only held a re-upload of the sample
off, and they were most of the measurements table and its indexes.
Ingest now refuses such a re-upload directly: a raw sample of a dense or
mean type, old enough to have been folded, whose hour or day a live
stats row covers, is a duplicate with reason folded_window. The batch
route and the Apple Health export import apply the same rule. With that
in place the dense retention and the daily-mean consolidation delete the
raw rows outright, and a new compaction-tombstone-purge job removes the
backlog: 5,000 rows a batch, one account at a time under the restore
lock, a short pause between batches, resumable because the predicate is
idempotent, queued at every boot and continuing itself while it makes
progress. A person's own deletions are never in its predicate. The sync
feed leaves compaction tombstones out and a restore does not write them
back; all three classify through the same function.
The batch route answers an exact re-sent sample without a transaction:
one read over the batch's sample ids, live or deleted, decides what the
reconciler would have decided, at the same point in the loop, so every
status, reason, counter and the cross-source merge stay as they were. A
batch of duplicates only no longer queues PR detection, a reminder sweep
or the detection audit row. Caches are evicted only for a foreground or
manual sync or a batch that wrote a hand-entered reading, and marked
stale otherwise; the targets cache follows the same split.
Also:
- POST /api/admin/maintenance/measurements queues VACUUM (ANALYZE) and a
REINDEX INDEX CONCURRENTLY per index, largest first, on a connection of
its own; it refuses while the purge runs, drops invalid leftover copies
and stops at the first failed rebuild. Runbook in docs/ops.
- The pure volume queues keep their finished jobs for 96 hours instead
of seven days, never less than the failure window plus a day.
- work_mem is set per app connection from DATABASE_WORK_MEM (16MB), also
when the statement timeout is disabled.
- The PR-detection fallback tick re-scans only accounts whose readings
or workouts changed since the previous tick.
- Type discovery for the status refill walks the live-rows index as a
loose index scan instead of grouping every live row.
- The host metrics chart polls every five minutes, like its sampler.
- The batch route records per-type HealthKit arrivals.
Land every file the v1.42 work shares before any of it is built, so the
parts built side by side never edit the same file.
Schema and migrations 0379-0386: the HEALTH_CONNECT measurement source,
ImportJob.kind, the air-quality columns and the per-user switch on the
environment tables, sealed environment locations with nullable plaintext,
managed-profile handover links, the per-type HealthKit arrival ledger, lab
unit-conversion provenance, and user-defined vaccines. Every new column is
annotated for the column-reader sweep.
HEALTH_CONNECT reaches every exhaustive list: the source labels, the
read enum (not the write allowlist), the source-priority ladder cap, now
the number of sources rather than a fixed eight, and the write-path smoke
test. RMSSD gets its own priority class and a 300 ms ceiling. The batch
syncTrigger gains `manual`, with its label on the Apple Health card.
Queues for the Health Connect import, the compaction-tombstone purge and
the measurement-table maintenance are registered with stub handlers;
host-metric sampling moves to every five minutes. Environment and work_mem
variables reach the compose whitelist, the production example and the
manifest. The proxy admits the handover link and redirects it at the edge,
and the Health Connect upload bypasses the proxy body limit.
The new routes answer 501 behind their final authentication and are
published in the contract; backup, wipe and key rotation classify the new
models and columns; the seven message bundles carry the base copy. Guards
that list release-pending keys and columns fail once an entry is wired, so
those lists empty themselves before the tag.
Also re-anchor a login-throttle fixture to the wall clock, where its
expiry is compared, and re-base the aggregate bundle cap on a measured
12 KB of new locale strings.
The validation 422 from `POST /api/measurements/batch` now carries
`meta.errorCode: "measurement.batch.invalid"`, beside the existing
`too_large` and `source_not_permitted` codes, so a client can file the
rows as permanently refused instead of holding its anchor on a guess.
A `value_out_of_range` entry now also carries `convertedValue`, the
reading after conversion into the stored unit, and `range`
(`min`, `max`, `unit`), the band it was checked against. Both appear
only on that skip. A fraction sent where a percent was expected is now
visible in the response instead of looking like a sensor glitch.
Status codes, the envelope and every existing field are unchanged.
Under a `measurements:write` credential the row's source is now decided by
the server on both routes, and the body may not name one at all.
The 422 widens from "names APPLE_HEALTH" to "names any source". It was
narrow only because the forced value was MANUAL, so a body saying MANUAL
was never actually overridden. Now it would be — and overriding a client's
explicit assertion quietly is the thing both routes' comments rule out,
while honouring it would defeat the change: a bridge that kept sending
MANUAL would stay indistinguishable from a typed reading. Status and
errorCode are unchanged, and the documented Home Assistant payload sends no
source at all.
The single-POST predicate keeps reading the raw body: the create schema
defaults `source` to MANUAL, so after parsing "said MANUAL" and "said
nothing" are the same value. Both of that route's create sites resolve the
source, the array arm included — that is the combined BP + Pulse path a
bridge uses, and missing it would leave exactly those rows unbadged.
Two consequences worth naming. Bridge rows no longer join the
MANUAL/APPLE_HEALTH cross-source merge, so they can't be swallowed by a
rule written for the iOS adopt-on-pair case. And a bridge posting at the
same instant as a hand entry now lands both rows instead of a 409, which is
honest and does not double-count: the rollup collapse is per
(type, day, source) and MANUAL outranks the unlisted EXTERNAL.
Fifty-nine 422s raised off a Zod `safeParse` answered with one opaque
sentence and nothing a client could map onto a field, while the other
166 sent `details.issues` — the multi-issue envelope that exists so a
caller with three bad fields fixes all three in one round-trip. Twelve
of the fifty-nine put a dotted machine token in `error` where the
sentence belongs, so `error` was sometimes text to show a person and
sometimes a token that must not be shown, with nothing in the payload
to tell them apart.
Every one of them now answers through `apiValidationError`, which keeps
the existing `error` string and the existing `meta` byte for byte and
adds the issue list beside them. The dotted tokens keep their place in
`error` for the clients that already match on them and are published in
`meta.errorCode` as well, which is the field a machine code belongs in
and the one a new client should read.
A guard reads the route tree per refusal site rather than per file —
per file is what let this grow, since a route that used the helper once
looked compliant to a `grep -l` sweep — and asserts it still finds the
sites it judges, so a matcher that stops matching fails instead of
reporting a clean zero.
A refused client had nothing to back off against. `rateLimitHeaders()`
emitted `X-RateLimit-Remaining` and an ISO `X-RateLimit-Reset`, no
`Retry-After` and no cap, and only about 70 of the 199 route files that
consult a limiter attached even those two. The rest answered with a bare
prose envelope, including every endpoint an offline replay queue hits:
the four batch ingests, `/api/sync/changes` and `/api/auth/refresh`. A
first-run Apple Health import trips the 60-per-minute batch ceiling
reliably, and the only strategy left to the client was a blind backoff.
Threading headers through 129 handlers is 129 edits the next new route
forgets, so the verdict is carried instead: `checkRateLimit` records it
in a request-scoped cell and `apiHandler` dresses any 429 that leaves
without headers. A route that already built its own is left alone —
`Retry-After` is the presence test, and it comes from the same builder.
`Retry-After` is whole seconds rounding UP (RFC 9110 §10.2.3), floored
at zero. `X-RateLimit-Limit` is new and needs the cap, so the limiter
result carries it. `X-RateLimit-Reset` keeps its ISO form: it is the
wrong convention for the name, but it is on the wire today and changing
it is a decision of its own.
The cell lives in its own leaf module because 177 test files replace
`@/lib/rate-limit` wholesale with a mock factory, and `apiHandler` must
not depend on anything those factories stand in for.
Since v1.30.17 there has been no self-service path to a token that can
write a measurement. That release made the Bearer scope default
fail-closed, which closed a real privilege-escalation hole, and removed
the generic mint at POST /api/tokens along with it. The generic mint had
been the only way to get a token that could reach /api/measurements, so
a local sensor pushed through Home Assistant or Node-RED lost its door
while the OAuth integrations kept theirs.
Add a narrow scope accepted by exactly two routes, both writes, both on
the holder's own record.
The scope constant is a leaf module beside the surface that consumes it,
matching FHIR_READ_SCOPE and the MCP scopes. requireRecordAuth grows an
options object rather than a sibling function, because the sharing
surface guard keys every leg on that one identifier and a second name
would mean two copies of the delegable declaration. AuthContext carries
the admitting scope as a single nullable string rather than the token's
permission array, so nothing downstream can re-derive an authorisation
decision from it.
A scoped credential is refused a record selector before any grant is
read. Placing the refusal above the sharing check rather than beside it
means no future widening of what a grant permits can reach one of these
tokens.
Rows are attributed MANUAL. An entry that explicitly names APPLE_HEALTH
is refused 422 rather than relabelled: that source is half the
(userId, type, source, externalId) dedup key the iOS client writes into,
it participates in the cross-source merge, and it gates the HealthKit
sync fields. For the same reason a scoped write no longer advances
User.lastSyncedAt, which the phone reads to size its undelivered window.
Also closes a narrower gap found while building this. The idempotency
wrapper resolved a replayed delegated response against a live grant
without considering which credential was presenting it, so a cached
response about a shared record could be returned to a token the handler
itself would refuse. No write was reachable that way; the response body
was.
Guards: T3 gains the mint site, T4's vocabulary gains the scope and
learns to read the requireRecordAuth form (without which this change
would open a way to widen a route that the closed vocabulary could not
see), and a new T5 freezes the set of delegable routes naming a scope.
The sharing surface guard's AST matcher reads only the first two
arguments, so it needed prose but no code.
The six measurement routes carried eight copy-pasted rollup-recompute +
status-invalidate blocks — the documented two-ended failure class
waiting for the next surface to remember one leg and forget the other.
All of them now call afterMeasurementMutation (the helper the deletion
fix introduced), and the two write surfaces that bypassed the hook
entirely join it: the nightly Recovery/Stress/Strain score rows and the
PHQ-9/GAD-7 assessment projection now recompute their (type, day)
bucket on the tick that writes them.
Every interactive path, every import, MCP and Telegram checked an incoming
measurement against VALUE_RANGES before storing it. The provider sync writers
did not, so a decode slip or a glitched sensor landed a row the application
itself calls impossible, and every mean, median and personal band derived from
it inherited the impossibility. Filtering the read side hid the row. It kept
landing.
The gate is one module reading the domain the app already declares, applied at
two kinds of seam. reconcileExternalMeasurement is the shared identity writer
for Polar, Oura, WHOOP, the Withings sleep leg, the Apple export importer and
the batch endpoint, so gating there means a provider added later inherits the
refusal rather than having to remember a check it never knew about; the new
rejected_range verdict carries no row, which is what makes the compiler force
every caller to handle it. The five writers that hold Prisma themselves gate at
their entry, before the existence probe decides insert versus overwrite, so an
impossible reading can no longer replace a good stored value.
An out-of-band value is dropped. Not clamped, because a clamp invents a reading
nobody took. Not stored behind a flag, because a quarantined row is still a row
and every consumer would have to learn the flag. Not fatal to the batch, because
one bad sample must not strand the run's other readings behind a held watermark.
Absence reads as absence.
A silent drop, though, is indistinguishable from a provider that sent nothing,
so every drop is tallied on the wide event under measurement.range_rejected with
the metric, the provider and the edge it broke. Counts only: the reading itself
is health data and never enters the log line, and the direction plus the
metric's own declared bound is what a diagnosis actually needs. The tally
accumulates rather than replacing, because a provider gates once per resource
leg and the reconciler once per row. One warning per provider lifts the event to
warn so a drop does not hide inside a green line.
A structural guard enumerates all twenty measurement writers with the reason
each is allowed to write, so a new one has to answer the question before it
ships. scripts/diagnose-implausible-measurements.ts reports the rows already
stored, aggregated per metric and source with no value beyond the bucket's own
extremes. It repairs nothing: whether a stored impossible row is a decode slip
or a band that is too narrow is not a call a script gets to make.
A GET must not have side effects — api-handler.ts says so next to
READ_HTTP_METHODS, where an MCP-audience token is admitted on the safe
methods precisely because they are assumed to write nothing. Three routes
broke that: the sync handshake bumped User.lastSyncedAt, the achievements
read inserted unlock rows, and the cycle calendar fired a discarded promise
at the forecast cache so a failed write left no trace anywhere.
Each write moves to the place that causes it.
The sync checkpoint is now stamped by the batch ingest endpoints for
measurements, mood and intakes. The handshake reports it. That also fixes
the watermark: the checkpoint used to advance past a window the client had
not drained, and a client trusting it would skip that window. It can only
trail a real sync now.
The unlock rows move to a nightly achievement-unlock-sweep. The badge grid
never needed them — unlocked comes from the live metrics and the completion
date is derived — so the payload is unchanged; the rows pin that date and
carry it into the backup, and the sweep reaches accounts that never open
the page.
The cycle forecast moves to a nightly cycle-prediction-refresh. The
CyclePrediction row is not a cache of a read: it is the index the reminder
cron scans, so which reminders an account received depended on whether
somebody had opened the calendar. persistPredictionCache no longer
swallows its errors either — the job reports them.
The batch route has accepted a `syncTrigger` since v1.32.8 and put it on the
wide event, where no human reads it. Half of what was promised got built: the
value is written and appears in three places in the codebase, none of which is
a screen anyone looks at.
Persist it on the account, alongside the instant of the most recent batch that
arrived without the app being opened, and publish both from
`GET /api/integrations/healthkit`. The card now names how the last sync arrived
and when data last came in on its own. `healthKitLastSyncedAt` alone reads
identically whether the phone delivers by itself or only while someone has the
app open, which is why "is background delivery working" could not be settled
from either side.
A batch that declares no trigger writes null rather than leaving the previous
one standing — the field describes this sync, and an older client's silence is
not evidence about it. A foreground batch leaves the background timestamp
alone; overwriting it would erase the only answer the card has.
Also here: WHOOP, Fitbit and Google Health join the workout-freshness table.
All three write workouts, and without an entry their cards could report every
measurement type fresh while the workout leg had been silent for weeks. And the
status pill drops a generic `aria-label` that replaced its own accessible name,
so a screen reader announced the same words for a healthy connection as for a
broken one.
`failing_leg` was one slot, so two legs of the same provider failing in
overlapping windows overwrote each other. The row remembered whichever failed
last, and that leg recovering then cleared an error the other one was still
causing — the ledger forgot a live failure and painted the card green over it.
The slot becomes a set. A failure adds its leg, a success removes its own, and
the error state, the per-kind buckets and the streak anchor clear only once the
set is empty. The array is ordered oldest failure first, so the last element is
the leg that wrote the message `last_error` holds; when that leg recovers while
others are still failing, the message is cleared rather than left describing a
leg that is now healthy.
An unattributed failure adds nothing and removes nothing. A single-leg provider
therefore behaves exactly as before, and a multi-leg provider no longer loses
what its other legs already reported.
The column never reached a client — it is absent from the status serialiser and
from the OpenAPI contract — so it is replaced rather than shimmed.
An external id is half of every ingest dedup key — (userId, source,
externalId), (userId, externalSource, externalId), (userId, type,
source, externalId). Re-posting the same triple is idempotent and
returns the existing row, which holds only while the id is stable
across client launches. A client that serialised an opaque platform
object with its default object description sent a per-allocation
memory address instead, so no sweep ever matched its own earlier row
and each one minted a fresh record: a live instance grew 23 phantom
medications in a day, none of which could collect a dose, because the
dose side resolved through the same rotating string.
Add one shared classifier for identifiers that provably cannot be an
identity — an angle-bracketed object description carrying a hex
pointer, a bare 0x address, and a blank value — and wire it at every
surface that accepts a client-supplied externalId.
Single-entry creates refuse through the schema, so the client gets the
standard 422 multi-issue envelope with a message naming the shape it
sent and what to send instead. Batch and bulk surfaces classify per
entry and report the offending row as skipped with reason
unstable_external_id, leaving the rest of the payload to land: one bad
row must not stop a sync. Every refusal carries pinned wide-event meta
(count, shape, surface) and never the id itself.
A structural guard freezes both routes into the floor: a request-side
externalId declaration either applies the field check or carries a
marker naming the file that checks it per entry, and the guard verifies
that file really does. The backup-restore payload is the one exemption,
with its reason recorded — it replays ids the server itself wrote.
Migration 0263 added three columns beside the Apple Health `stats:*`
authority ladder. Only `aggregation_provenance` is ever read back: the
external-measurement reconciler selects it and refuses a lower-authority write
against a higher-authority canonical row, and the cumulative drain reads it
too. That column stays.
`aggregation_contributor_count` and `aggregation_selected_source_hash` are
written by three call sites and read by none. No select, no raw SQL, no export,
no backup, no API contract, no repair path. The import-time selection keeps its
deterministic tiebreak, which compares an in-memory map key computed per flush
rather than a value read back from the table.
Both are plain nullable scalars with no index, unique, check or foreign key, so
the drop is a catalog-only operation that touches no measured value.
Route-level tests that run the real apiHandler + requireAuth so the
cookie-vs-Bearer derivation is exercised end to end, not papered over by a
unit mock:
- a cookie POST records WEB, a Bearer POST records API, and a body that
forges `source` is ignored;
- the dose-history ledger selects and returns `source` per row, null on a
legacy row;
- a batch carrying `syncTrigger:"background"` rides the ingest annotation,
a batch without it annotates null, an invalid value is 422, and the field
leaves the per-row insert/update outcomes byte-identical.
Root-cause the intake source attribution and give the HealthKit batch a
diagnostic sync-trigger, one attribution surface for both.
AuthContext now carries a read-only authMethod ("cookie" | "bearer"),
derived from the same branch that sets the wide event's auth_method. The
medication-intake route maps it to the stored IntakeSource: a Bearer
(native) call records API, a cookie (browser) call records WEB. The other
producer values are untouched — a converge onto a pending reminder row
updates through applyCanonicalSlotWrite, which never rewrites source, so
REMINDER / IMPORT / APPLE_HEALTH provenance is preserved. The value is
never client-asserted, the same posture as userId being narrowed from auth.
The dose-history ledger now selects and returns source per intake row so a
client can label how each dose was recorded. The endpoint gains its first
OpenAPI entry (path plus response schema) so the field lands in the
iOS-consumed contract.
The measurements batch accepts an optional per-request syncTrigger
(foreground | background | push). It is diagnostic-only: recorded on the
measurement.batch.ingest wide event and nothing else — never persisted,
never part of the dedup key, never an input to attribution or how a sample
is stored. Sending it or omitting it produces byte-identical row outcomes.
Regenerated docs/api/openapi.yaml.
Merge the eight reviewed audit fixes that were cut a week ago and never
landed. Main has moved on since, so each fix was re-checked against the
current tree and each of the five conflicts resolved by understanding
both sides.
Kept, still open on main:
- Proxy early exits now carry the transport security header subset via a
shared helper. Reconciled with main's /api/health worker-only carve-out.
- Batch measurement ingest bounds the entry instant so a future-dated row
can no longer win "latest reading" or poison the rollup buckets.
- Web Push dials through the pinned safeFetch dispatcher with
requirePublicHost, closing the last user-controlled-host egress.
- Mood bulk upsert and its tag/factor links run in one transaction, so a
rejected rating rolls the mood row back instead of leaving a tagless row.
- Full backup carries NutrientIntakeDay with its unit and source. Folded
into main's portable-export / disaster-recovery split.
- Live daily-series fallback restores the min/max spread band the tiered
read already emitted.
- Coach stream aborts its in-flight reader on unmount and resets send state
on a conversation switch.
- Manual water writes share the day-bound helper, nutrient add clamps after
the increment, and the thiamin cap drops to a real supplement ceiling.
- Compliance point window widens symmetrically instead of wrapping overnight.
- Oura, Polar and Fitbit token helpers classify an empty or non-JSON 2xx
body as a handled transient error.
- WebAuthn relying-party fallback to localhost is gated to development.
- Chart-token strip list is derived from the allowlist and pinned by a test.
- Regression tests pin the snapshot-cache tenant key, the GlitchTip
path-secret redaction and the Withings credentials delete.
Superseded by main, tests re-pointed at the current shape:
- The intake CSV import was rewritten on main to run through a background
job with a single-active-job admission gate. The future-date bound is
re-applied at request validation and the per-user limiter is kept for
parity with the bulk twin; the old synchronous-write test is replaced.
Related to #585. Investigated the intraday pulse chart's read path
end to end: the ZIP-import raw-sample fold, the live-sync uploaded
10-min bucket fold, their merge, and the chart's render logic. All of
it treats the two ingestion shapes consistently — the chart has no
shape awareness at all, it paints whatever bucket series it receives.
No read-side or display-side defect explains a systematic density
difference between the two ingestion paths.
What was missing is visibility into the write and read sides that
would let a future report actually be diagnosed instead of
re-investigated from scratch. The batch endpoint's wide-event trace
never broke a skip down by reason, and never distinguished an
HR-bucket write from any other entry in the same request. The
intraday route's own trace surfaced only the final bucket count, with
no visibility into which shape (raw sample rows vs uploaded bucket
rows vs folded hourly rows) a day's data actually came from. Both
traces now carry that breakdown, so the next report of "the chart
looks sparser after a given date" is answerable by grep instead of by
re-deriving the whole read path again.
`classifyIntakeTiming` treated `windowStart === windowEnd` as an overnight
wrap, inflating a degenerate point window into a ~30h on-time band — a dose
taken many hours late still classified `on_time` and fed a perfect-compliance
streak. Widen the point window symmetrically instead, matching the treatment
`resolveLegacyWindowBounds` already applies.
`/api/measurements/batch` never applied the entry-instant plausibility bound
its single-entry twin has carried since v1.17 W1b, so a future-dated row landed
and permanently won 'latest reading' while poisoning the rollup buckets and the
canonical picker. Skip per entry, like the range guard. The past floor stays at
1900 — a historical backfill is a legitimate shape here.
The intake CSV import had neither an upper date bound nor a rate limiter, while
running up to 1000 sequential find + create + consume iterations per call. Add
both, on the bulk twin's bucket shape and budget.
Every ingest path in the product can now emit a typed, salience-classified
event into pg-boss, consumed by one worker that claims a durable per-day
reaction marker and fans out to the reaction surfaces.
The recency classifier runs at the seam, before anything is enqueued, so a
mass backfill — a multi-year Apple Health export, a provider re-sync, a
30-day catch-up — emits nothing at all. Non-salient writes are annotated and
dropped.
The worker makes no provider call by construction: it imports no provider
module, writes only markers, and enqueues nothing that is not budget-gated
downstream. Business refusals return a status rather than throwing, so a
refusal is never retried against a ceiling that will not move until the local
day rolls over.
Seams: sleep (one hook, inherited by all seven transports), workouts (the
batch route, the four provider syncs, the export importer), weight and blood
pressure (both measurement routes), labs (manual, OCR commit, document fact
commit).
The provider workout syncs could not previously tell a created row from an
updated one, so a re-sync of a recent session would have re-fired on every
poll; they now recover the distinction from the row the upsert already
returns.
Reaction lines are AES-256-GCM at rest, following the InsightNarrative
precedent, and the column is registered for key rotation. Markers are pruned
after 14 days by a daily maintenance job.
Each case fails on the pre-fix code: the rollup tests on an unaligned
recompute window, the batch tests on the older stats: snapshot winning,
and the resting test on a series that stops at the fold horizon.
Two entries carrying the same stats: key in one batch both miss the
existing-row probe, so the overwrite branch never runs, both land in the
bulk insert, and skipDuplicates keeps whichever the unique index saw
first. The older snapshot of a per-day cumulative total or of a filling
ten-minute heart-rate bucket therefore won and the newer, larger figure
was dropped silently — both entries reported inserted, the counters
still summed, and the client checkpointed past both.
stats: rows carry overwrite semantics by contract, so within one batch
the last entry for a key is the authoritative one. Resolve it up front
and report the superseded earlier entries as duplicate with a
superseded_in_batch reason, a status the client already checkpoints
past.
Two data-integrity gaps found in review of the v1.30.7 contract:
- The bucket externalId guard accepted a non-canonical timestamp that V8
silently rolls over (T24:00, 04-31, non-leap 02-29). Such a row would key
under a literal string the canonical client never emits, so the two would
never collapse on the overwrite key, and it would plot on the wrong local
day. Require canonical round-trip (toISOString() === suffix) in both the
guard and the bucket-start parser.
- The per-bucket min/max bypassed the plausibility-range guard that value
passes; now that they drive the DAY min/max band and the intraday envelope,
an out-of-range or mis-ordered spread poisoned the aggregate. Persist the
spread only when both extremes are in range and ordered around value;
otherwise drop to null and keep the average.
Refine the go-forward aggregated HR wire contract (iOS #34) from the never-
adopted hourly bucket to a 10-minute UTC bucket, so one contract serves both
the measurements-list de-explosion and the intraday-pulse 10-min curve.
- Batch route accepts stats:HKQuantityTypeIdentifierHeartRate:<10-min ISO-Z>
with avg + min/max, overwrite semantics, malformed-suffix rejection; the
validator is identifier-generic (allowlist) so respiratoryRate/SpO2 can join
later without a grammar change.
- The intraday reader classifies a day's PULSE rows three ways by externalId
shape (raw / uploaded 10-min bucket / server-folded hourly), builds the
10-min curve directly from uploaded buckets with the min/max envelope and
tension detection enabled, and overlays buckets over raw on the one
cutover-straddling day.
- DAY rollup min/max read COALESCE(value_min, value) so a bucket-native day's
band reflects the true intra-bucket extremes.
No schema change (value_min/value_max exist since v1.19.2). Reconciliation is
go-forward cutover; the nightly retention fold handles the raw backlog.
Marc-André Bombeck