Mood has been a first-class module here for a long time, with entries,
structured tags, rated factors and notes. The bridge to the external moodLog
service had nothing left to add, and its Settings card was withdrawn in
a56e6fe61 on the promise of a dedicated removal. This delivers it.
Removed: the hourly `moodlog-sync` queue, cron and handler binding; the
inbound webhook and its shared-secret handling; the reverse push and its four
call sites; the pull-sync module; the per-user credential routes and the
instance-wide toggle; the entry on the `/api/integrations/status` envelope,
the `IntegrationKey` union, the cadence map and the display-name table; the
boot-time secret rotation pass; 22 message keys per locale.
Migration 0271 drops the five `users` credential columns and
`app_settings.mood_log_global`, clears the ledger row for the provider, and
deletes the retired pg-boss schedule so no cron keeps enqueuing work no
handler drains.
Mood data is untouched. `mood_entries` is not named anywhere in the migration
and cannot be reached from it: the dropped columns carry no index, constraint
or foreign key, and `mood_entries.user_id` references `users(id)`, not any of
them. Rows the bridge imported keep `source = 'MOODLOG'`, and every read path
- list filter, source badge, exports, backups - still resolves that value.
`src/lib/validations/moodlog.ts` held the native mood-entry schemas alongside
the bridge's own; it is now `validations/mood.ts` and carries only the former.
Mirrors the Google sleep suites name for name: the mapper keeps its externalIds
across a re-score that re-classifies and renumbers segments, the sweep stays
bounded to the night's window and never touches the fresh keep-set, a null
window issues no delete at all, and a cleanup failure never fails the sync.
The end-to-end fixture runs one night stored under the old keys through
`syncUserSleep` and pins the ordering the heal depends on — sweep first, then
the rescue re-keying the tombstoned rows in place, no parallel insert — plus a
second pass that writes nothing at all.
Also covers the repair's verdict gates (bounded 365-day window, marker stamped
only on a clean run, watermark untouched, dead token and reauth park returning
without a stamp), the unchanged-value skip including the re-classified segment
it must not swallow, and both Retry-After paths.
Fitbit re-scores a night after the fact, and the per-segment externalId carried
a positional index plus the stage label. Both moved on a re-score: renumbered
segments minted fresh keys, the previous scoring's rows stayed live because
nothing swept them, and the night total counted both copies from then on.
Key the segment on the stable session anchor plus its own start instant
(`<logId>:sleep:<segment-start>`), and add the replace-by-window sweep every
other segment-writing provider already has. The two ship together with the
natural-key rescue that has been sitting in the Fitbit upsert unused: the sweep
tombstones the old-keyed rows first, so the rescue re-keys those very rows in
place instead of inserting a parallel copy or losing the insert to
`skipDuplicates` against the natural unique index.
A one-shot repair queue re-reads the bounded sleep history once per connection
so no user has to do anything; it stamps `sleepRepairedAt` (migration 0270) only
on a clean run, never touches the sync watermark, and reports the rows it
removed. Nights older than the 365-day backfill horizon keep their old keys and
whatever duplicates they already hold — they can no longer gain new ones.
Alongside: `fitbitGet` now honours a `Retry-After` inside a 120 s cap by waiting
once and retrying once, and fails fast above it so a long hold cannot starve the
rest of the poll cohort or outlive a job's expiration. The measurement upsert
skips a write whose payload is unchanged, which takes the hourly overlap off the
delta feed for every Fitbit resource.
These four were the only connected providers with no way to ask for a
sync; they ran on their scheduled pass and nothing else. Each gets a POST
sync route on the hardened Fitbit and Google Health template: session auth,
a 5-per-minute per-user bucket, and a fixed 502 on failure. No body and no
full-sync flag, because none of the four entry points has a full-history
arm to call. Polar goes through the shared legs entry point so both legs
run with the attribution rules built for them.
The Nightscout error arm is the one that matters: a base URL carries the
API token as a query parameter, so the caught message can hold the user's
secret and never reaches the response body. The classified failure is
already on the ledger, recorded with the provider's own redaction.
Connecting also starts the first sync now. The poll handler has accepted a
single-user payload since it was written and never had a producer, so a
fresh connection sat empty for up to an hour waiting for the schedule. The
Oura and Polar callbacks and the Nightscout connect route enqueue one
targeted job each, best effort, with the hourly tick still the safety net.
Strava needs none: its callback already enqueues the history backfill,
which covers the recent window.
The four poll-queue names move into their own module so a request-path
caller can name a queue without dragging the whole worker handler graph
into the request bundle, following the backfill-queue precedent. Creation,
scheduling and handler binding stay in the registrar, and the dead-queue
guard follows the declarations.
Restructure the Polar vitals sync from a single Promise.all over the five
collections into a Promise.allSettled over a per-collection spec, each closure
fetching and mapping so a throwing mapper is contained exactly like a failing
fetch. The healthy collections import; a failing one is recorded once as a
partial failure that names it, without stamping success and without rethrowing,
so a single unavailable data type — SpO2 on a watch without the sensor, a
higher-tier cardio-load read an account lacks — stops blanking recharge, sleep,
and activity on every tick.
A 401 is now a per-collection failure classified reauth_required, not a
whole-leg throw; the write-path escape still surfaces unmarked for the poll
boundary to record once. Mirrors the shipped Oura isolation, minus the
token-refresh leg Polar has no path for.
Add an hourly ECG catch-net cron so a watch-only account with no scale pulls its
recordings without a webhook: the queue was registered and bound but never
scheduled, and the handler now grows a cohort arm that walks every connection on
a payload-less tick, mirroring the activity and sleep handlers.
Make the activity and sleep write paths honest. A swallowed activity batch-write
fell through to the success stamp — a green pill over lost data, green forever on
a persistent write bug; it now records a transient failure and refuses the
stamp. A reconcile throw out of either sleep transaction propagated past the
stamp and was swallowed by both callers, leaving neither success nor failure
recorded; the source now records a transient failure and rethrows so the cron
and webhook paths stay honest without changing their warn-and-continue
behaviour.
Port the hard-fail ledger the Fitbit transport already carries. A revoked WHOOP
grant makes every resource short-circuit and return zero rows without throwing,
so the cycle read as clean: it stamped success, un-parked the reconnect state,
reset the failure streak, and the hourly cohort hammered the dead refresh token
on every tick. Record the three dead-token null-return paths on the ledger and
fold that verdict into both drivers — the full orchestrator and the per-resource
cron path that runs most often — so a dead-token cycle refuses the success
stamp and the park holds.
Return an imported-plus-failed verdict from the orchestrator and gate the
history backfill on it: an interrupted full-history walk now throws for a
pg-boss retry instead of being stamped complete, and a connection parked at
reauth returns without burning the retry budget. Thread the same verdict into
the one-shot sleep-timeline backfill.
Also catch a write-path throw on both drivers so a persistent measurement-write
failure surfaces on the status ledger instead of leaving a stale-green pill, and
widen the cycle overlap to the seven-day re-score window the other collections
use so a late-revised day's strain and energy are re-fetched rather than
stepped past.
Introduce a recorded-marker primitive on the integration ledger so a provider
that records-then-rethrows a sync failure is not recorded a second time at the
poll-cohort boundary, while an unrecorded escape (an Oura or Polar write-path
throw, or any future path) lands on the ledger exactly once, correctly
classified and redacted by the source that owns it. Without the marker a
boundary-blind recorder would double-count the failure streak and could persist
a Nightscout token-bearing URL into the stored error.
Give the poll-cohort factory a required per-provider failure recorder and wire
one for Oura, Polar, Nightscout, and Strava.
Nightscout now counts only freshly inserted readings instead of insert plus
re-confirm (the inflated count fired the reminder trigger every tick), records a
transient failure and skips the success stamp when some rows fail to write, and
emits the measurement-arrival spine for its inserts. Correct the docstring that
promised a far-history backfill that does not exist: the hourly window imports
about 48 hours and nothing deeper.
Strava now enqueues its history backfill at connect time, mirroring the WHOOP
and Fitbit callbacks, so deep history imports within the hour instead of waiting
for the next worker restart.
Compliance card: a dose-change imperative and a fabricated risk figure are
withheld to the deterministic fallback (no model text persisted, updatedAt
stays null), while a clean grounded summary still renders and persists.
Hero lines: a well-formed, truncated, or fenced JSON envelope resolves to the
deterministic lead/template for both the arrival reaction and the coach nudge;
a plain-text line still ships verbatim.
Route the medication-compliance status card through finalizeStatusSummary so
it enforces the same dose / risk / causal outbound contracts its seven sibling
status cards do, not only the unparseable-envelope withhold it already had. A
dose-change imperative or a fabricated figure is now withheld to the controlled
fallback on this surface too.
Add the status cards' structured-envelope detection to the two background hero
lines (arrival reaction, coach nudge): a brace-led or fenced JSON reply, or a
truncated envelope under the char cap, now resolves to the deterministic lead
rather than rendering raw.
Fold in the residuals from the same sweep: strip a truncated SUGGEST-ACTION
marker from the Coach prose tail, drop an unreachable zero-rate branch in the
adherence fold, and document why the internal conversation summary stays off
the render-boundary screen.
New sync-dead-token suite mirrors the Google Health parity test: the three
getValidToken null-return paths each register on the hard-fail ledger, a
successful refresh leaves it empty, and a dead-token cycle returns
{ imported: 0, failed: true } without stamping markSynced or recordSyncSuccess.
Extends the upsert suite with the two write-catch labels (measurements:create,
measurements:update) and the sync orchestrator suite with a write-label
watermark hold. Rewrites the backfill suite to mirror google-health-backfill:
a clean run stamps, a failed verdict throws without stamping, a parked
connection returns without running the sync.
A revoked or expired Fitbit refresh token used to fail the cycle's status
ledger and then stamp the same cycle a success: the connection flipped back
to connected, the watermark advanced, and after a reconnect the 24h overlap
could no longer reach the missed days. getValidToken's three null-return
paths (credentials missing, refresh failure, vanished connection) now register
on the per-cycle hard-fail ledger, so the cycle's verdict fails, the park at
error_reauth holds, and lastSyncedAt does not move. The two measurement write
catches do the same, so fetched-but-unwritten rows are re-fetched next tick
instead of aging out of the overlap.
syncUserFitbit now returns { imported, failed }; the manual-sync route, the
poll cohort adapter, and the backfill consume the new shape. The backfill
stamps backfillCompletedAt only after a clean full-history walk: a parked
connection returns without stamping, and a failed verdict throws so pg-boss
retries on the already-configured retry budget.
Parity suite proves a Polar exercise and an equivalent Strava activity map to a
downstream-indistinguishable Workout row (sport bucket, start instant, duration,
distance, calories, HR, null-vs-zero). Regression guard asserts the exercises
request hits GET /v3/exercises with no userId path segment. Dedup ladder tests
mirror the Strava suite (overwrite-in-place, Apple+Polar twin collapses, Polar
above Strava). Composite test proves a failure in one Polar leg leaves the
other leg's result intact on the shared ledger. Defensive-shape tests cover the
four flagged assumptions: numeric-vs-string id, UTC-offset composition (positive
and negative), bare-array-vs-paged body, and a 403 scope error as a soft error.
Refs #568
Pull the athlete's exercises from the stateless GET /v3/exercises list (last
30 days of uploads, no transaction, no userId in the path) and upsert each into
the Workout table as source POLAR, keyed on (userId, source, externalId) so a
re-fetch or Polar-side re-analysis overwrites in place. A new table-driven
sport map normalises the raw Polar sport labels into the canonical
WorkoutSportType bucket at ingest, mirroring the Strava/WHOOP mappers; the raw
labels and training load ride metadata for provenance.
The exercise leg rides the existing hourly polar-sync cohort tick alongside the
vitals leg, with independent attribution on the shared polar ledger: each leg
is attempted and records its own failure, and neither can mask the other's
result. Polar takes its place in the workout source ladder between Withings and
Strava. No migration and no OpenAPI change: MeasurementSource.POLAR and the
full workouts source enum already exist.
Refs #568
The persistent mood-rollup writer bucketed on the UTC truncation of
`mood_logged_at` and ignored the canonical per-row `MoodEntry.date`
label. Every coverage-miss fallback keys on `entry.date`, so the rollup
tier disagreed with the fallback and with the day the user logged the
mood: a late-evening or after-midnight local mood landed on the wrong
calendar day, and the label shifted the moment the boot backfill flipped
a user from the live fallback onto the rollup tier.
Key every bucket off the stored `date` label instead — DAY, WEEK, MONTH
and YEAR — with `bucket_start` set to that label's UTC-midnight instant.
The rollup is now byte-identical to the live `entry.date` fallback and
timezone-independent by construction: DST correctness stays where it
already lives, in `moodDateKey` at write time. No consumer changes, no
schema change; every rollup reader already formats `bucket_start` as a
UTC day label.
Migration 0269 empties `mood_entry_rollups` so the boot-time
full-backfill worker re-mints every user's buckets under the corrected
keying; the worker now deletes-then-refolds so it is self-cleaning
against any row left by an older instance. While the table is empty,
reads fall back to the live `date`-keyed walk, which was already
correct, so there is no window where a wrong day is shown.
Closes#588. The ZIP extractor decompressed export.xml synchronously
into one in-memory buffer, then wrote it with a blocking writeFileSync
— on the same event loop the web server runs in the default
single-container deployment. A real multi-year export could pin the
Node main thread for minutes, freezing every other request, while
briefly holding roughly 2x the decompressed size in heap. A process
that died mid-extraction never reached the failure path, so the
ImportJob row stayed stuck in "unpacking" with "0 rows imported"
forever and no error ever surfaced.
Extraction now streams through zlib's async inflate into a file write
stream, moving the decompression work off the event loop and bounding
memory to the chunk size. The zip-bomb ceiling moves from
inflateRawSync's maxOutputLength to an equivalent byte-counting
transform on the streamed output.
Orphan-ImportJob reconciliation used to run once, at worker boot.
docker-compose's restart:unless-stopped does bring a crashed worker
back, but a fast restart can land before the stuck row's heartbeat
goes stale or pg-boss stops reporting the job as live — and because
the boot-time pass never re-runs on a healthy worker, that row was
never revisited again. A 15-minute cron now re-runs the same reconcile
so a genuinely abandoned import converges to a visible failure instead
of staying silently stuck.
The harness landed in v1.30.35, after the slices that built this
release were already cut — so the reaction line's prompt module was
never registered, and its own docblock's warning against the
two-locale collapse was untested. Registering it surfaced a real gap:
the meaning-first instruction lived only in the system prompt's own
words, never in the user prompt in the family's shared phrasing, which
is where the rest of the surfaces carry it and where the tone contract
checks for it. The German body carried no such instruction at all.
Threads locale through the user-prompt builder so the fixed
instruction line matches the reader's own reply language, and wires
the per-user per-day opener-archetype rotation the milestone spec
called for and the first implementation had not connected.
Nothing warms this queue and no read path triggers it, so an unregistered name
fails open in the worst way: every send resolves, every job vanishes, and the
symptom is indistinguishable from the surface's honest empty state.
The disable-route suite only ever supplied a cookie-shaped context, which is why
the token-id-as-session-id defect was invisible to it. It now covers both arms
and asserts the credential shape handed to the session revoker, plus that the
elevation is spent before the teardown and not spent when the factor code is
wrong.
The projection's central claim was carried by a comment. It is now walked
before it is returned: any non-numeric leaf at any depth throws, and the only
two string leaves are allowlisted by shape — the sport type against the schema
enum itself, and the day key against its format.
This matters because sport_type is a TEXT column whose API write path is
enum-constrained but whose backup-restore path accepts any non-empty string, so
a restored row can carry arbitrary text. It is re-asserted at the projection
boundary rather than trusted from the column. The heart-rate provenance string
is dropped rather than allowlisted, and route geometry stays consumed-not-carried
— it yields one number, and location history may not ride a prompt.
Own-history rows now collapse cross-source twins through the canonical picker,
as the detail page's comparison already did; a session recorded by two paired
watches was inflating the sample size and biasing every median.
Three properties, each with a mutation-verified guard:
The chip and the line run on DIFFERENT clocks and must keep doing so. The chip
expires after three hours; the line stands for the rest of the local day.
Collapsing them onto one clock reads like a simplification and silently deletes
the day's read at lunchtime.
Degradation is the headline. For every reason a line can fail to exist — no
provider, no consent, an exhausted budget, a dead provider, an unusable output
— the row stays line-less, the reservation is reconciled, and the chip and the
provisional-to-final flip still work. The degraded digest is built through the
real composer rather than a hand-shaped fixture, so the assertions cannot agree
with themselves.
The server pass carries no wall-clock time. The hero paints on the SSR pass
because it is the LCP element, so a locale-formatted time inside it would differ
from the browser's — React #418. The tests render through the server snapshot,
which is exactly the pass that must stay clock-free.
The seeded end-to-end spec lands a sleep batch through the real ingest route,
writes the arrival marker, and asserts the chip and the phase flip in place on
a refetch trigger the digest hook already declares — no reload, no new cadence.
One test per gate, each asserting the refusal reason AND that the provider
entry was never reached — a gate that refuses but still resolves a provider
would pass a reason-only assertion while costing what the gate exists to
prevent. The cap and the hash are asserted independently of each other, since
the singleton queue key that normally collapses a burst is best-effort.
The backfill fixture walks the real chain: 750 historical sessions plus 40
re-synced ones plus one fresh, through the live classifier and the live spine
worker, yielding exactly one generation.
The peak detector changed under test: on a flat ride the ninetieth percentile
sits a beat above the mean, so a bpm of sampling noise read as two efforts the
rider did not do. Peaks now need an absolute prominence over the session mean
as well.
The spine dispatches a generator-free enqueue; the worker resolves the
provider. Five gates stand in front of it in ascending cost order — both
modules on, ten-minute duration floor, a hard count of four a day in the
user's own timezone, an input hash over the deterministic evidence, and the
token ledger inside the shared status chokepoint. Each refusal returns a
status rather than throwing, because retrying against a ceiling that does not
move until midnight is an unbounded loop.
The detail route only reads: a workout with no row serves null and renders no
card, which is every workout that predates this and every re-synced one. An
undecryptable paragraph degrades to no card rather than failing the page.
Today gains two additive-nullable digest fields. `justIn` names the day's
newest arrival while it is still news — under three hours old, after which the
record is simply current rather than new. `reactionLine` carries a single
generated sentence about that arrival and replaces the hero's standing lead for
the rest of the local day, never appending a second paragraph, so the hero stays
one lead line tall and nothing below it shifts.
Nothing here polls. The open dashboard picks the change up on the cadence it
already runs — a state change, not a notification.
The generated sentence is bounded before it is written: the arrival spine's
unique row per kind per local day IS the throttle, the worker re-checks the
same claim before spending, and the reservation is reconciled on every exit
including each failure path. Without a provider, without consent, or against an
exhausted budget the row stays line-less and the surface falls back to the
deterministic lead — the chip and the provisional-to-final flip are unaffected.
The chip's time string is formatted client-side after mount; `justIn.at`
crosses the wire as an ISO instant so the server never renders a local
wall-clock time the browser would disagree with.
A step-up elevation minted from the account password satisfied the fresh-factor
routes. That is a second-factor bypass: a stolen token plus a known password
could rotate the recovery codes — which take no factor in the body — and then
spend one of them to disable the second factor outright. The web has never
allowed that chain, because a password login does not stamp mfaVerifiedAt.
The method is now an authorisation input. Only totp, webauthn, and passkey
satisfy a fresh-factor gate, which is exactly the set of ceremonies the web
stamps a session for; password reaches what a plain cookie session reaches and
stops. The mint gains TOTP and security-key arms so a TOTP-only account can
still manage its factor natively, and the response says which reach it bought.
The gate also returns a discriminated context. session.id meant a session row on
one arm and a token row on the other, and the disable route passed the token id
to a session-scoped query — excluding nothing, so it deleted every browser
session and revoked the caller's own device login. The transport is now in the
type and the compiler refuses the confusion.
Redemption splits into validate-then-commit so a 429, a 422, or a wrong code no
longer burns a proof against a five-per-fifteen-minute mint ceiling. The claim
is UPDATE ... RETURNING, so the statement that decides is the one that reports.
The mint is transactional behind a partial unique index. Sign-out-everywhere and
both password-reset paths clear elevations. The status read drops to plain
requireAuth — it carries no credential material — and every mutation stays gated.
The batch route now also resolves the user's timezone and module map through
the arrival seam, so a raw call count on user.findUnique no longer isolates
the read this guard is about. Filtering by the SELECT shape keeps the
once-per-batch contract pinned while letting the unrelated cached lookups
through.
Backfill immunity is proved by replay, not by a hand-picked case: a ten-year
Apple Health export (12772 writes) and a 30-day provider re-sync (58 writes)
must both emit zero events, while a fresh sleep night, workout and first
weight of the day emit exactly one each.
The provider-isolation guard walks the worker's static import graph and fails
if any provider client or resolver becomes reachable. It carries two positive
controls, so a broken entry path cannot make it pass against an empty graph.
The integration test covers what unit mocks cannot: the unique index, the
ON CONFLICT DO NOTHING claim, the losing side of ten concurrent claims, the
forward-only marker, ciphertext at rest, the delete cascade, and the
retention window.
Every ingest path in the product can now emit a typed, salience-classified
event into pg-boss, consumed by one worker that claims a durable per-day
reaction marker and fans out to the reaction surfaces.
The recency classifier runs at the seam, before anything is enqueued, so a
mass backfill — a multi-year Apple Health export, a provider re-sync, a
30-day catch-up — emits nothing at all. Non-salient writes are annotated and
dropped.
The worker makes no provider call by construction: it imports no provider
module, writes only markers, and enqueues nothing that is not budget-gated
downstream. Business refusals return a status rather than throwing, so a
refusal is never retried against a ceiling that will not move until the local
day rolls over.
Seams: sleep (one hook, inherited by all seven transports), workouts (the
batch route, the four provider syncs, the export importer), weight and blood
pressure (both measurement routes), labs (manual, OCR commit, document fact
commit).
The provider workout syncs could not previously tell a created row from an
updated one, so a re-sync of a recent session would have re-fired on every
poll; they now recover the distinction from the row the upsert already
returns.
Reaction lines are AES-256-GCM at rest, following the InsightNarrative
precedent, and the column is registered for key rotation. Markers are pruned
after 14 days by a daily maintenance job.
The per-document summary job is enqueued at upload time and no-ops while the
documentsAutoAiRead opt-in is OFF. Nothing ever re-enqueued it, so the switch
only applied to documents uploaded AFTER the flip — a user who filled the vault
first and opted in later had documents that would never be read, and the
setting looked inert.
A genuine OFF->ON transition now schedules one bounded catch-up pass over the
user's un-summarised documents. It grants nothing of its own: it only enqueues
the ordinary summary job, which still re-reads the opt-in, re-asserts the
per-document egress consent and reserves the daily token budget. The pass
itself re-reads the opt-in first, so a toggle flipped back OFF before it runs
egresses nothing.
Bounded by an id-cursor walk capped at 200 documents per pass (a summary is a
provider call, unlike the local-compute thumbnail backfill). Idempotent at four
levels: the trigger fires only on a real transition, the per-user singleton
collapses a double toggle, the candidate set excludes summarised documents, and
the summary job skips one that already has a summary.
Opening a document said "the summary is being prepared" and kept saying it.
The view had one fallback for a null summary column and used it for every
outcome: a document nobody ever enqueued, a missing vision provider, a spent
budget, a withdrawn consent, an unreadable file, a provider error, and a
summary the outbound screen withheld all rendered as work in progress.
The common case is not the screen. The background job is enqueued at exactly
one place — a fresh upload with the auto-read opt-in already on — and no
backfill reaches anything else. Every document uploaded before the toggle went
on has a null column no job will ever fill, and the view promised all of them a
summary that was never coming.
Record the outcome as `summary_state` (migration 0254, additive, defaults to
NONE which is truthful for every existing row) and branch on it. Only PENDING,
set when a job genuinely exists, may claim generation is running. NONE,
WITHHELD and UNAVAILABLE each say what happened and offer to generate one.
The withhold keeps its rule and loses its dead end. Blocked prose is still
never shown and never stored — only the fact of the refusal is kept, and it is
no longer terminal, so asking again re-runs the gauntlet. That was the original
intent; leaving the column null defeated it, because null read as "generating"
and the guard meant nothing would ever overwrite it.
The on-demand summary now persists what it produces. It is the only repair path
for the documents the job never ran for, and re-deriving the same paragraph on
every open costs a provider call for nothing. Generation is still only ever a
click — nothing runs because a view rendered. Transcription stays session-only.
Also drop the "read by AI" line from the detail footer.