The nightly folds soft-deleted every raw Apple Health sample they folded
into an hourly or daily stats row and kept it for 75 days. Nothing ever
read those rows as deletions; they only held a re-upload of the sample
off, and they were most of the measurements table and its indexes.
Ingest now refuses such a re-upload directly: a raw sample of a dense or
mean type, old enough to have been folded, whose hour or day a live
stats row covers, is a duplicate with reason folded_window. The batch
route and the Apple Health export import apply the same rule. With that
in place the dense retention and the daily-mean consolidation delete the
raw rows outright, and a new compaction-tombstone-purge job removes the
backlog: 5,000 rows a batch, one account at a time under the restore
lock, a short pause between batches, resumable because the predicate is
idempotent, queued at every boot and continuing itself while it makes
progress. A person's own deletions are never in its predicate. The sync
feed leaves compaction tombstones out and a restore does not write them
back; all three classify through the same function.
The batch route answers an exact re-sent sample without a transaction:
one read over the batch's sample ids, live or deleted, decides what the
reconciler would have decided, at the same point in the loop, so every
status, reason, counter and the cross-source merge stay as they were. A
batch of duplicates only no longer queues PR detection, a reminder sweep
or the detection audit row. Caches are evicted only for a foreground or
manual sync or a batch that wrote a hand-entered reading, and marked
stale otherwise; the targets cache follows the same split.
Also:
- POST /api/admin/maintenance/measurements queues VACUUM (ANALYZE) and a
REINDEX INDEX CONCURRENTLY per index, largest first, on a connection of
its own; it refuses while the purge runs, drops invalid leftover copies
and stops at the first failed rebuild. Runbook in docs/ops.
- The pure volume queues keep their finished jobs for 96 hours instead
of seven days, never less than the failure window plus a day.
- work_mem is set per app connection from DATABASE_WORK_MEM (16MB), also
when the statement timeout is disabled.
- The PR-detection fallback tick re-scans only accounts whose readings
or workouts changed since the previous tick.
- Type discovery for the status refill walks the live-rows index as a
loose index scan instead of grouping every live row.
- The host metrics chart polls every five minutes, like its sampler.
- The batch route records per-type HealthKit arrivals.
A restore writes rows back with their original updatedAt, behind every
paired client's watermarks, and removes rows without leaving tombstones.
A client that kept its cursor would miss both on its next incremental
pull. The same holds after the person deletes all their data.
`users.sync_reset_at` (migration 0363) records when the record was last
replaced: the restore stamps it as its last write and again right after
the commit, the per-account and the admin wipe stamp it inside their
transactions. Cursors now carry the time they were issued, and
/api/sync/changes answers its existing `cursorExpired` to a cursor issued
before the reset, or one without an issue time on an account that has
one. The comparison is on the issue time, because after a restore a
freshly drained cursor still carries old watermarks.
Fifty-nine 422s raised off a Zod `safeParse` answered with one opaque
sentence and nothing a client could map onto a field, while the other
166 sent `details.issues` — the multi-issue envelope that exists so a
caller with three bad fields fixes all three in one round-trip. Twelve
of the fifty-nine put a dotted machine token in `error` where the
sentence belongs, so `error` was sometimes text to show a person and
sometimes a token that must not be shown, with nothing in the payload
to tell them apart.
Every one of them now answers through `apiValidationError`, which keeps
the existing `error` string and the existing `meta` byte for byte and
adds the issue list beside them. The dotted tokens keep their place in
`error` for the clients that already match on them and are published in
`meta.errorCode` as well, which is the field a machine code belongs in
and the one a new client should read.
A guard reads the route tree per refusal site rather than per file —
per file is what let this grow, since a route that used the helper once
looked compliant to a `grep -l` sweep — and asserts it still finds the
sites it judges, so a matcher that stops matching fails instead of
reporting a clean zero.
- /api/sync/changes carries a1 to a5 and the day context under the same keys
the write path accepts, so a client reads back exactly what it can post
- BulkMoodEntry accepts both, all optional; a1 stays server-derived from the
five-point label, so a build that sends only a label still writes a complete
row and needs no change to stay correct
- the batch parses each entry's context on its own: one row carrying a value
the vocabulary does not name is skipped and the rest of the batch lands
- docs/api/openapi.yaml regenerated alongside
The multi-domain feed at GET /api/sync/changes read cycleDayLog and
menstrualCycle without consulting the module gate, so an account with
cycle tracking off still received period days and per-day symptom rows
over the same token that every /api/cycle/* route refuses with a 403.
Resolve the cycle module once per request and skip both queries when it
is off, rather than filtering after the read. The blocks come back empty
and the request still succeeds: the feed multiplexes five domains over
one cursor, so refusing it outright would wedge measurement, mood and
intake catch-up for every account that has cycle tracking off. The two
cycle watermarks stay untouched while the module is off, so re-enabling
resumes where the client left off.
A client that synced cycle rows before the module was turned off keeps
its stale local mirror. Tombstones would be the wrong signal there --
they mean deleted, and disabling a module must leave the rows intact --
so purging a paired mirror needs an explicit signal in the feed contract
and is left for a coordinated contract cycle.
Wire shape unchanged.
Lay the account-security and data-sovereignty substrate for v1.23. No
user-facing behaviour yet — schema, crypto registry, redaction, and the
note-encryption data path only.
Schema + migrations:
- 0199 — User totp_secret_encrypted / totp_confirmed_at / mfa_enforced /
passkey_upgrade_nudge_dismissed; Session mfa_verified_at / last_active_at;
Passkey last_used_at. All additive, nullable / defaulted.
- 0200 — mfa_challenges (single-use step-up ticket; only the ticket hash is
stored, so no session or token is ever minted before the second factor),
mfa_recovery_codes (Argon2id at rest), webauthn_mfa_credentials (second-factor
keys, kept separate from passkeys, which remain the passwordless-primary set).
- 0201 — mood_entries.note_encrypted + measurements.notes_encrypted (BYTEA),
matching the existing free-text encrypted-note convention.
Free-text health notes:
- New writes encrypt the mood / measurement note into the *_encrypted column and
null the legacy plaintext column; reads prefer the ciphertext and fall back to
plaintext only for not-yet-migrated rows, via a single read/write helper.
- A boot-time job migrates existing plaintext rows in an idempotent,
transactional, fail-closed pass (encrypt, then null the plaintext, per row in
one transaction; safe to re-run; a row with content is never left both-null).
- The legacy plaintext columns are kept this release and dropped in a follow-up
once the backfill reports zero remaining rows — the same boundary the
key-rotation playbook uses before dropping a legacy key.
Crypto coverage:
- Register the three new encrypted columns and wire them into the key-rotation
script so a rotation never orphans them.
- Extend the observability and log redaction denylists so TOTP secrets, MFA /
recovery / backup codes, and export passphrases never reach a wide event.
One-shot prettier pass so the new format:check gate starts from a clean
baseline. Formatting only — no behaviour change. Generator-owned and
vendored artifacts stay untouched via .prettierignore.
RECOVERY_SCORE is written by two sources for the same day: the WHOOP-native
percentage from the device sync and the COMPUTED proxy from the nightly
readiness job. The wellness reader hard-filtered to source COMPUTED, silently
dropping every ingested native row, and the doctor PDF blended both sources
into one min/avg/max. The COMPUTED proxy also equals the readiness blend
verbatim, so the Coach prompt carried the same number under both READINESS and
RECOVERY_SCORE.
Add resolveCanonicalRecovery: per day, keep the WHOOP row when present, else
the COMPUTED proxy. Drop the COMPUTED-only filter for RECOVERY in the wellness
reader and resolve to the canonical row; STRESS and STRAIN stay COMPUTED-only.
Resolve the doctor PDF wellness summary the same way so the report reads the
native value, not a proxy/native blend. Dedup the Coach snapshot: omit the
recovery line when it mirrors readiness (the no-native case), keep it when a
WHOOP-native value differs. Document the WHOOP-over-COMPUTED resolution on the
sync delta feed, which already mirrors each row with its source so the client
resolves one canonical series rather than charting two.
A WHOOP-connected user's recovery tile, chart, and PDF flip from the proxy to
the native value.
Add the `/api/cycle/*` capture, calendar, history, and settings routes
plus the offline-sync delta extension for the v1.15 cycle-tracking
feature, all against the locked iOS contract.
- Enable flag: add nullable `cycleTrackingEnabled` to CycleProfile
(migration 0132). A null toggle derives from gender (FEMALE → on).
`isCycleEnabled()` is the single gate resolver; every `/api/cycle/*`
route 403s `{ errorCode: "cycle.disabled" }` when off. The resolved
flag surfaces additively on GET /api/auth/me.
- Routes (apiHandler + Zod + envelope + gate + annotate + auditLog):
POST/PATCH/DELETE day-logs (single), POST day-logs/bulk (idempotent,
capped 500, rate-limited 60/min, per-entry inserted/updated/duplicate/
skipped), POST period boundary, GET calendar (runs the deterministic
engine, debounced stale-while-revalidate cache, goal-gated fertile
window), GET cycles (history + median/MAD stats), GET profile,
DELETE cycles, PATCH /api/auth/me/cycle-prefs (deep-merge).
- Day-log note encrypts at rest (notesEncrypted); intent fields stay
queryable plaintext. Upsert keys on (userId, source, externalId) when
externalId present, else (userId, date).
- Offline sync: extend GET /api/sync/changes + the cursor with the
`cycleDays` + `cycles` domains (CURSOR_VERSION → 2). cycleDays
tombstones key on externalId else id; cycles on id.
- OpenAPI: register every route + DTO; regenerate the committed spec.
- i18n: cycle prediction disclaimer across all six locales.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Marc-André Bombeck