Deleting an account, a guardian deleting a managed profile, and "Delete
all data" were final the moment they were confirmed: the account row
cascaded into every table and every stored copy, and the off-host objects
were purged in the same breath. Nothing short of rolling the whole
instance back could return one account.
ACCOUNT_DELETION_GRACE_DAYS (14 by default, at most 30, 0 erases at once
as before) now sets a window:
- Account and managed-profile deletion lock the account (migration 0488:
deletion_requested_at, deletion_purge_at, deletion_requested_by_id).
Every session, refresh token, API token, connector, share link and
step-up elevation is revoked in the same transaction, and the record,
its stored copies and its off-host objects are left as they are.
- Sign-in is refused with 403 auth.account.deletion_pending and the
erasure date, after the credential was proved and before a second
factor is asked for. The session and Bearer resolvers and the acting-
account resolver refuse a pending account too, so a credential minted
in the moment before the lock is useless.
- POST /api/admin/users/{id}/undo-deletion (cookie-only, audit-logged)
clears the lock. The admin users list reports the pending state.
- The hourly account-deletion-purge job erases accounts whose window has
ended, exactly as the immediate deletion did, with the off-host purge
request committed alongside. The Guardian invariant is decided again;
a refused erasure stays pending and is reported.
- A wipe first stores the whole record as a PRE_WIPE_<ms> copy through
the writer the weekly copy uses, keeps it out of the wipe, and holds
its off-host purge back with a new not_before on the request. The
ordinary restore brings the record back; restoring that copy withdraws
the held purge. The purge job deletes the copy when its window ends.
Background work skips accounts pending deletion through one shared
predicate (src/lib/account-deletion/active-account.ts): every reminder,
digest, Coach, sync, backfill, rollup, score and backup enumeration
filters with it, and account-deletion-enumeration-guard.test.ts fails on
a query in background work that neither filters nor says why it need
not. Notification delivery and the AI capability loader refuse a pending
account as chokepoints behind it, and the Withings, WHOOP and Telegram
webhooks and the notification quick actions ignore one.
The privacy summary publishes the window. Two pieces of the reminder tick
and the off-host pass moved into modules of their own to keep the files
inside their size ratchet.
Three readers grouped the same rows twice and joined the second grouping
back to the first: the live buckets joined each hourly-mean day's hour
means to the per-source days, the all-time extremes joined the day
weights the same way, and the doctor report's dense buckets joined each
bucket's latest reading and hour means to the bucket aggregates. On a
measurements table without planner statistics (a new install, or an
import under the autoanalyze threshold) the planner estimated one row on
each side and ran every one of those joins as a nested loop that re-ran
the inner grouping for each outer row.
On ten years of readings over six types (51,000 rows, no statistics) the
dense buckets and the every-type live buckets ran into the 60 s statement
timeout and the all-time extremes took 11 s. With statistics all three
answered in 0.1 to 0.2 s.
Each is now one chain of folds over one scan. The live buckets fold the
readings per local hour and the hours per local day, carrying the day of
hours beside the counts. The all-time extremes take the day weights from
the same windowed rows they count, the shape the pre-fold remainder
already has. The dense buckets carry each bucket's latest reading and each
hour's count and mean on the rows as window values, and one grouping folds
them all. Same fixture without statistics: 0.2 s, 0.1 s and 0.1 s, the same
as with them.
A parity suite runs the joined statements as the reference against the
readers across ladders, zones with whole, half-hour and negative offsets,
grains, overlapping and off-ladder sources, glucose contexts and device
types, tombstones and both DST changes; sums and means agree to
floating-point summation order, everything else exactly. The
unanalysed-table guard now runs the three readers under EXPLAIN ANALYZE:
the joins measured 200 to 3,000 row visits per row stored, the folds about
10.
The dense-bucket loader drops under the 200-line function cap, so its
baseline entry goes.
The blood-pressure series chose each UTC day's source with DISTINCT ON
over the day's floor-passing systolic rows and joined the choices back
to the readings. That is the shape the analytics pick had, with the
same weakness: on a measurements table without planner statistics the
planner estimated one row on each side and ran the join as a nested
loop that rescanned the pick for every reading.
The ten-year series over 6,000 freshly written rows from one source
took 0.57 s for the statement alone, 0.86 s with a second source on
every third day; the plan visited about 4,900 rows per row stored, so
the cost grows with the square of the history.
The pick is now one window over one scan: the day's rows ordered
systolic first, then by ladder rank, then by source name, and the first
row's source taken only when it is a systolic one. The scan starts at
the window's first UTC day, so the pick still sees the whole edge day,
and a day without a floor-passing systolic still shows nothing. Both
fixtures above read in about 10 ms, with or without statistics.
A parity suite runs the old statement as the reference against the new
one across overlapping and off-ladder sources, half readings, sub-floor
rows, tombstones, both DST changes, window starts on and either side of
UTC midnight, and four session zones. The unanalysed-table guard now
covers the series' ten-year, quarter and week reads.
The canonical-source pick grouped the window to (type, day, source),
chose each day's source with DISTINCT ON, and joined the choices back to
the rows. Whether that join ran as a hash join or as a nested loop that
rescanned the whole pick for every row was the planner's call, made from
table statistics. On a measurements table without them (a new install,
or an import still under the autoanalyze threshold) it estimated one row
on each side and took the nested loop.
Binding the request's clock as a parameter instead of reading NOW()
flipped which side of that loop the pick landed on: the analytics read
over 6,000 freshly written blood-pressure rows went from 1.6 s to 8.4 s,
and the demo seed's dashboard headline to 20 s. On a ten-year account
without statistics the 90-day read took 4 s with either clock and the
all-time read ran past two minutes.
The pick is now one window function over one scan: FIRST_VALUE of the
source per (type, UTC day) in ladder order, keeping the rows of that
source. Same rows, same order of preference, so every day keeps its
source; the parity suite against the DISTINCT ON statement holds that.
The analytics read on the fixture above takes about 0.1 s, the headline
0.4 s. On the analysed ten-year account the 90-day and all-time reads
are as fast as the join was, and they no longer depend on statistics.
The rows carry one extra column, canonical_source; every reader selects
its columns by name.
A new integration test runs every reader shape under EXPLAIN ANALYZE on
a table with no statistics and bounds the rows the plan produces or
discards per row seeded. The statement count never moved, so a statement
budget could not see this; the join measured 200 to 8,600 row visits
per row seeded and the window stays at 4.
The summaries slice and the comprehensive aggregator took their JS windows
(the DAY-bucket bounds, the rollup slope anchors, lastSeen staleness) from
the process clock and their SQL windows (avg7, avg30, the month before, the
median, the 90-day canonical pick, the cold-path slope evidence) from
NOW(), the database's clock. One response mixed two clocks, so a reading
on a window's edge fell in or out with whichever host answered, and with
the process clock pinned the analytics parity walk changed its avg7 with
the hour it ran at.
Each read now takes one `now` and binds it into its statements
(`request-clock-sql.ts`): `$N::timestamp`, which keeps the driver's
offset-free UTC string as written in any session zone. The canonical-pick
helper takes its window as an instant plus an interval instead of reading
NOW(); the series and calendar readers anchor it on their own `since`, the
BP-in-target read on its 365-day start. The window-mean and live-slope
columns the two aggregates each spelled out now come from one builder.
The streak's local day series read its bounds through `::timestamptz`,
which follows the session zone: outside a UTC session today's reading fell
out of the streak. It reads them as UTC first.
The parity walks for analytics, the dashboard summary and the
comprehensive overview pin a fixed day instead of today: their records are
laid out from the anchor day, so the chronotype's work and free nights and
the weight-by-weekday points moved with the weekday the walk ran on. The
snapshots are unchanged. A new suite pins the clock to four times of day,
far from the database clock, with readings ten minutes either side of
every edge, on both read paths, in a UTC and an Auckland session.
PATCH /api/tokens/{id} sets mirrorToAppleHealth on a measurement token:
cookie session only, the caller's own tokens only, revoked ones included,
any other token refused with tokens.mirror.not_measurement_token. It
stamps mirrorChangedAt and audits only when the value changes. The mint
accepts the setting (off by default), the token list returns it, and
GET /api/tokens/measurements counts the EXTERNAL readings no token
reaches.
A reading sent with a measurements:write token keeps its EXTERNAL source
and now points at the token that carried it (measurements.ingest_token_id).
Every stored row answers a server-resolved appleHealthMirror: true for
WITHINGS, IMPORT and MANUAL, the token's choice for EXTERNAL, false
otherwise. The plain list accepts appleHealthMirror=true so a client can
page exactly those rows.
Migration 0476 adds the columns and links existing EXTERNAL readings only
where one measurement token provably sent them; 0477 builds the index
concurrently. Backups carry the link and a restore keeps it only for a
token the account still holds. A guard holds every EXTERNAL writer to
setting the link.
Work a request starts and does not wait for runs inside the request's
context, so an annotate({ action }) from it landed on the request's event
whenever it ran before the event was emitted. The measurement tail's
status re-warm logged a hand-entered reading as
insights.status.invalidate.skipped on some requests and as
measurement.create on others.
runDetached starts such work so that its action is dropped for the event
it was detached from; meta, warnings and the auth it reads stay as they
were, and an event the work opens for itself keeps its own action.
fireAndForget now takes a function and starts it detached, and every
trackBackgroundTask handle on a request path is built with runDetached,
which a structural guard holds.
GET /api/measurements/series keeps authentication, the query and its
refusal; readSeries (lib/measurements/series/read-series.ts) picks the
reader in the old order (sleep nights, paired blood pressure, the bucketed
read of a dense kind, the raw rows) and adds the direction, change,
reference lines and stat strip. The kinds, units and point shape live in
kinds.ts. The route falls from 752 to 95 lines.
The parity walk recorded before the move is unchanged. The guards on the
verdict resolver, the spread, the pulse day mean and the unit literals read
the new files, the last two with their scope widened to the series folder.
The dayKey drill-down of GET /api/measurements answered the stored rows as
they were, notesEncrypted ciphertext included and the note itself unread.
It now selects the contract's columns by name and shapes each row like the
plain page: the note decrypted onto notes, the ciphertext never on the wire.
A route test walks the measurement reads over rows with a sealed note
(the page, the drill-down, the day-grouped list, an aggregate, the detail)
and fails on any sealed field; it failed on the drill-down before this
change. The parity walk drops its contract exemption for the drill-down.
GET /api/measurements keeps authentication and the query refusal and hands
the validated query to measurementListResponse (lib/measurements/list),
which tries the list modes in their old order. POST keeps authentication,
the body, the bridge-source refusal and the three validation answers; the
single and combined writes with their shared tail move to
lib/measurements/create/create-readings.ts, the named sleep-stage write to
create-sleep-stage.ts. The route falls from 938 to 274 lines.
The parity walk recorded before the move is unchanged; the guards that name
the measurement writers, the write tail and the bounded reads follow the
code.
postBatch was one 1255-line function. The route keeps authentication, the
rate limit, the body, the entry cap, the schema and the bridge-source
refusal; the ingest moves verbatim into lib/measurements/batch-ingest/, one
module per stage in the order they always ran: the first pass (prepare),
the measurement writes with their prefilters and set-based insert, the
hourly sums, the sync checkpoint, the audit and wide-event report, and the
cache and arrival tail. The route is 274 lines; no function in the folder
is over 200.
The parity walk recorded before the move (measurements-batch.parity) is
unchanged, as is the set-insert golden. The guards that read the route for
the per-entry external-id check, the plausibility validator, the write
tail, the dense-read bound and the unit audit now read the service files.
The cross-source picker anchors each duplicate cluster on its first start,
so which cluster a workout lands in depends on the workouts around it. The
detail route still read the old fixed five-minute clock slot, and every
other reader handed the picker only the rows of its own range. The detail
of a twin that straddled a slot boundary declared itself canonical while
the list hid it, and runs at 09:58, 10:02 and 10:06 were grouped one way by
a read starting at 09:55 and another by a read starting at 10:00.
Every reader that decides which workout is canonical (list, detail and its
previous session, activity calendar, day view, doctor report, Coach,
briefing and correlation blocks, strain, per-sport context, the activity
insight's history, the batch write) now reads through one windowed reader.
It reads a duplicate window past the end of the range and reaches back
until every sport in the range starts at a workout that is an anchor
whatever came before it, then picks over everything read and keeps the
rows that start inside the range. The per-sport baseline and the insight
history leave out the whole cluster of the current session instead of the
row alone, which left its twin behind as a session of its own. The batch
write clusters its rows together with the stored workouts around them and
drops only a row that loses to another row of the same batch.
A property test checks the windowed read against a whole-history read for
random dense histories and ranges; route tests pin list, detail, deep link,
previous session and the batch write against the database.
A batch that carried a type with duplicates only moved its
lastReceivedAt, lastTrigger and lastBackgroundAt, so a type whose live
delivery had stalled kept reading as freshly arrived for as long as a
replay, a retry or an older client re-sent its old samples.
The arrival ledger now moves those stamps only for a type the batch
brought at least one new or changed row for. An unchanged stats:
re-post and an hour posted again with its stored value count as
duplicates here; the wire keeps answering updated for them. A
duplicates-only type stamps the new last_seen_duplicate_at column on
its existing row and nothing else. The intake bulk route's dose entry
follows the same rule through the shared recorder. lastSyncedAt and
healthKitLastSyncedAt are unchanged.
GET /api/integrations/healthkit serves the stamp additively as
metricFreshness[].lastSeenDuplicateAt, and the Apple Health card names a
re-send that came after the last arrival.
Refs #1259
Re-importing an Apple Health export the account already holds sent one
UPDATE per row the insert skipped, about two million statements for a
large export, nearly all writing back what was stored. One read per chunk
now finds the rows whose stored values equal what the update would write;
they count as updated as before, without the statement. A key the chunk
carries twice is written in order as before. The natural-key rescue moves
into the same module.
A long-window read of a dense series sent two statements over the same
canonical rows, the day or hour buckets and the strip's figures, so a
ten-year pulse read picked the canonical source per day and scanned the
readings twice. Both aggregates now run in one statement over the window
materialised once, with the same SQL each had (dense-series-read.ts).
An integration test keeps the two old statements as the reference and
requires the same buckets and figures for pulse (two ranked sources),
glucose and oxygen saturation, day and hour grains, in UTC,
Europe/Berlin, Asia/Kolkata and America/New_York, and the same empty
answer for a type without readings.
Every sync marked all of an account's analytics cells stale, so the slice
re-read the pre-fold remainder (every reading older than five years,
folded per source and day, the slice's largest statement on a long
history) and the day view re-read its past history blocks, although a
sync of today's readings cannot change either.
A cache key can now name the instant its value's readings end before
(closedBlockKeyPart). invalidateUserMeasurements takes touchedFrom, the
earliest measuredAt a write created, changed or removed, and leaves such
blocks in place when it is not earlier; without it every block goes as
before. The batch route passes it from the identities it wrote. The
pre-fold remainder moves into its own module and is cached as a closed
block; the day view's past history blocks carry their end, with a day of
margin for a night across midnight.
The batch route's two up-front reads move into batch-stored-rows.ts with
the once-per-batch key rule they share.
Two copies of one session were collapsed only when both starts fell in the
same fixed five-minute clock slot, so a pair thirty seconds apart that
straddled a slot boundary (08:04:50 and 08:05:20) stayed as two workouts in
the list, the batch write, the calendar, strain and the Coach.
Workouts of one user and sport are now clustered by start time: a workout
joins the open cluster while it starts less than five minutes after the
cluster's first start. Wherever no pair within five minutes straddles a slot
boundary the clusters are exactly the old slots, which a parity test checks
against the previous picker on thousands of generated inputs, strain impulses
included. A property test shows any offset under five minutes now collapses,
five minutes or more never does, and a uniform shift of every start keeps the
picked set.
A converted value that rounds to zero from below came out as -0, and
Intl.NumberFormat prints that as "-0": 0 °F typed and shown back read
"-0 °F", and a body-temperature deviation of a few hundredths below zero read
"-0.0 °F". The rounding now returns plain zero. Found by the new unit
round-trip properties, which check that every value a person can type in
their own unit is shown back exactly as typed.
The value domain read the plausibility band out of the measurement
validation module, so every metric sub-page that asks whether a metric
can be negative or is integer-valued loaded the measurement schemas and
Zod with them. The band and its check live in a Zod-free module now,
re-exported by the validation module for the server writers, and the
values list checks its segment against the exhaustive type label map
instead of the Zod enum. About fifty metric routes drop roughly 88 KB
gzip from their first load. The client-route guard walks a metric page
and the values list too.
The training impulse summed every stored workout row of the day, and the
briefing's workout tally and the day view's workout section listed every
row, while the workout list collapses a session recorded by the watch and
synced again through a second service. All three now run the same
cross-source picker. The strain engine keeps the heart-rate series of a
dropped twin when the picked row has none, so the collapse never turns a
scored session into an unscored one.
POST /api/measurements/batch ran one interactive reconciler transaction
per fresh sample: a savepoint, three advisory locks, two candidate reads
and the insert, about ten round trips. A 500-entry upload of new samples
was 500 of those, the 2 to 15 second outliers. Fresh samples now land
through INSERT ... ON CONFLICT DO NOTHING under the same per-account
advisory lock. Both measurement identities are full unique indexes, so
the statement inserts exactly what the reconciler would have inserted,
and every skipped row still goes through the reconciler for its own
verdict. Only rows whose external and natural keys each occur once in
the batch are buffered, and the buffer is flushed before any other row
is reconciled, so the order of effects stays the sequential one. The
contract and every per-entry status are unchanged; a golden test runs
each batch through both paths against the same table.
/api/health took seconds after a start because register() awaited the
whole pg-boss worker start (queues, schedules, boot discovery) before
the server took a request. The worker, or a web process's producer, now
starts in the background after the blocking boot checks; the probe
answers at once with reason "starting" until it is up, and gives the
database two seconds before calling it not answering.
Room climate from a home automation system gets three types of its own:
INDOOR_CO2 (ppm), INDOOR_TEMPERATURE (celsius, shown in Fahrenheit for
imperial readers) and INDOOR_HUMIDITY (percent). They arrive through the
existing ingest token (measurements:write, source EXTERNAL), sit in the
environment cluster beside daylight, and the daily rollup carries their
mean, low and high like any spot reading. They never mix with the body
temperatures or with the outdoor environment channels.
The Home Assistant page documents the three types, a rest_command that
sends a room as one request, a single-room automation, and an
importable blueprint (docs/integrations/blueprints) that skips sensors
reporting unknown or non-numeric states, converts Fahrenheit, and
stamps readings to the minute so a retried push is refused rather than
written twice.
Six Apple Health signals leave the deferred list and ride
POST /api/measurements/batch and the export.zip import end to end:
EXERCISE_MINUTES, STAND_MINUTES and STAND_HOURS (the exercise and stand
rings), MINDFUL_MINUTES, STATE_OF_MIND_VALENCE (iOS 18 State of Mind,
-1..1) and AFIB_BURDEN (AFib History, percent of the week). The ring
types and mindful minutes are cumulative, so the daily rollup is their
sum; AFib burden is scaled strictly from Apple's fraction and a value
above 1 is refused rather than guessed at. The archive import takes
mindful minutes from the session's dates and skips idle stand hours.
Every registry carries them: units, ranges, labels in all locales,
categories, chart tokens, report catalogue (mindful minutes and the
valence fenced as sensitive), signal registry with FHIR codings, and
MCP exposure for the three ring types only.
Workout GPS tracks are now an opt-in, off by default
(PATCH /api/auth/me/workout-routes, a switch on Data & Privacy). While
it is off, the workout batch route and the export import keep every
workout and drop its track with the same per-entry status. With it on,
the import reads the referenced GPX files from the archive one bounded
member at a time, thins each to the point cap and stores it sealed.
The climb a track carries is written to Workout.elevationM at write
time, so the workout insight no longer opens the track, and the AI
egress guard now freezes the set of files that read route geometry.
The cycle, mood-tag and mood-factor boards each kept a table of unit
words, and the German cycle board printed S/min where every other screen
says bpm. A board row now names the measurement it reads; MetricCrosstab
resolves the unit through measurementUnitLabel after the reader's unit
preference, and the callers use the same reading for their detail line.
A rating or score shows no unit word, as everywhere else. The boards' own
unit keys are gone from all bundles, and unit-label.test.ts refuses a
component that keeps a unit table again. The pair label wraps instead of
being cut off.
The vitals tiles printed the canonical storage token beside the value, so
a German screen read "15,4 breaths/min". The list view had a private map
for a few count types; every other path passed the token through.
useUnitDisplay().unitFor now resolves the token through one module
(lib/measurements/unit-label): word tokens map to bundle keys, "celsius"
to its symbol, kind markers (score, rating, count, event, level, ratio) to
nothing. The list, the day view and the vitals tiles all read it. A guard
walks every measurement type and every display source and refuses a unit
that is neither a shared symbol nor resolved through the bundle.
A day's sleep is the main night plus its naps (daySleepMinutes). The
series, the sleep score and the composition already read it that way; the
sleep list, the CSV export and the correlation sleep channel read the main
night alone, so a day of naps listed as 0 h and an average over the list
counted it as a night without sleep. The list now carries the day total in
value and the main night alone in the new mainAsleepMinutes field; the nap
caption reads as included.
GET /api/sleep/night without a date answered the latest day holding any
sleep, so a nap after the last night left the hypnogram with no night to
draw. It now walks back to the latest night with a main sleep; nap days
stay reachable by date.
The SQL night page bounded a night by its asleep sessions only, so an
in-bed-only session of the same night fell outside a chunked read that the
one-night read counted. The span now covers every session of the night.
Output of `node scripts/strip-planning-tags.mjs --write`: 3,157 comment
lines in 1,308 files lose the work-item identifier at their head. Release
versions and issue references stay; nothing outside comment text
changes, which the script checks per file against a comment-free
printout. healthlog/no-planning-tags is enabled at error level with this
commit, so the tree it lands on is clean.
Three pickers decided which of two overlapping workouts to keep: the
read-time picker the list, the detail page and the Coach use, a
write-time batch picker with its own 90-second window, and an unwired
field-merge picker. They used different source orders (the `steps`
ladder, or a separate workout default when no order was saved), so the
batch route could keep a row the list then ranked below another.
The batch route now runs the read-time picker over the rows it is about
to insert, with the ladder from `workoutSourceLadder`, and reports what
it hides as duplicates. A session HealthKit hands over twice under two
ids is stored once, as the list already showed it. The write-time and
the unwired picker are removed.
The 1500-line list route carried the day-grouped list, the sleep list
and its SQL night page, each with its own copy of the window-total
paging. They now live in src/lib/measurements/list/ with one paging
helper, and the route dispatches.
The sleep list gives every page night a row. A night whose
reconstruction held no main sleep (a day of naps alone) used to be
dropped while meta.total still counted it, which left short pages; its
row now carries 0 minutes and its naps. The SQL night cut splices the
asleep stages from ASLEEP_STAGES and folds rows into hour buckets
before the window functions, which cannot move a session boundary and
sorts a handful of rows per night instead of every stage. A drill-down
reads its night the way the night view does.
groupBy=day refuses valueMin / valueMax with 422: a value range filters
single readings, and the rows promise whole days.
An integration test runs the SQL page and reconstructSleepSessions over
one set of awkward shapes and holds them to the same nights, and the
list, the night view and the series to one answer per day. The unit
test of the day-grouped branch now pins only the branch; what the rows
hold is pinned against Postgres.
The latest canonical reading per type ran as `DISTINCT ON (type)` over
every live row of the account in five places (summaries slice warm and
cold path, comprehensive aggregator warm and cold path, targets). With no
skip scan in Postgres 16 that sorted the whole account to keep one row
per type. `readLatestCanonical` drives the read from the type list
instead: one index probe for each type's latest instant, then a closed
one-day window ranked by the source ladder. The closed window also keeps
the plan under `jit_above_cost`. The targets read reuses `readLatestEver`
with a floor.
The summaries slice was built inside three cache cells (the tile-strip
route, the thick analytics body, the dashboard snapshot). It now has one
cell, read through `readSummariesSlice`; composing callers read it
fresh-only so a refresh after a write cannot cache the pre-write slice
again, and the snapshot folds the water tile into a copy instead of the
cached object.
zod 4.6 grew the shared Zod chunk from 63 to 89 KB gz (the new schema
compiler and its memoizer ride along because every schema goes through
the `z` namespace), and the dashboard, measurements, mood and settings
routes each carried it for values that need no validation.
Move the measurement source list, the notes limit, the screening check
and the Coach source list into value-only modules the schemas are built
from, import the settings classifier and response check from their
leaves, and load the gate's managed settings and anamnesis sections on
demand. Re-base the four route baselines to the measured values and the
aggregate cap to 4975, with the reason in the budget comment.
A sync from a watch or an integration now waits the instance's sync gap
(AppSettings.aiSyncRegenGapHours, else AI_SYNC_REGEN_GAP_HOURS, else 6 h)
before it may regenerate a status card again; a hand entry keeps the hour.
Every caller of afterMeasurementMutation and invalidateStatusInsightsForTypes
names its source, so a new ingest path cannot skip the choice.
Refs #1217
Every raw-SQL clock read against a timestamp column without a zone now
writes NOW() AT TIME ZONE 'UTC', so passkey and login challenges, rate
limits, provider pauses, coach usage stamps, tombstone and measurement
windows no longer pass through the session time zone even where the
per-connection pin cannot apply. The regression-window bounds stay naive
UTC timestamps instead of being turned into timestamptz. A structural
guard keeps bare NOW() / CURRENT_TIMESTAMP out of raw SQL outside a short
list of timestamptz targets. The integration test sets the database
default to Europe/Berlin and drives challenge claims, session expiry, the
native sign-in handoff freshness check and the rate-limit window through
the app's client, plus the challenge claim through an unpinned client.
Self-hosting docs describe the pin for an existing or shared Postgres.
The fold leftovers were told from a person's deletions by syncVersion = 1,
but raw samples are raised in place before a fold takes them (the collision
merge, a changed Health Connect value). The repair then recomputed windows
without those samples and could overwrite a correct mean, also in windows
one run had folded whole.
A tombstone of fold age now counts as a leftover when it carries
syncVersion = 1 or shares its deletion instant with such a tombstone of the
same day; anything else of that age is ambiguous and leaves its window as
stored. Only a window taken in two parts (fold runs at least an hour apart,
or a run and later live samples) is recomputed.
Once the repair has been through an account, the purge may remove class-A
leftovers inside the horizon, so the folds' re-fold of a stored day or hour
leaves it as stored, and the purge takes the fold lock per account. The
fold-lock transactions get an explicit timeout and maxWait, since the lock
wait counts against Prisma's five-second default.
The fold repair and the regular folds rewrote an existing daily or hourly
stats: mean from the samples still in the table. Fold tombstones are purged
75 days after they were written, so for an older window those samples can
be a fragment: a re-upload after an anchor reset, the rows of one later
fold run. A mean over them replaced a stored mean that had been taken from
the whole day, and on old days it moved a lot.
A stored mean is now recomputed only for a day that starts at or after the
tombstone horizon (foldConstituentHorizon): every fold tombstone of such a
day was written after dayStart + the fold threshold and is still inside the
retention, so the live samples plus the fold leftovers are the whole day.
That is the last 75 days for daily means and, since dense days fold 90 days
late, the 75 days before the raw window for hourly means. A window also
needs at least one fold leftover; without one the rows in hand are not
known to be all of it.
Beyond the horizon:
- The repair leaves the day as it is and counts it per type as
windowsSkippedBeyondHorizon.
- The daily-mean consolidation and the dense hourly fold leave a day or
hour that already has a live stats: row exactly as it is, the stored
mean and its live samples alike (daysLeftAsStored, hoursLeftAsStored).
Up to v1.41 the fold overwrote such a row with the mean of the live rows
alone, and the first v1.42 cut with the mean of those plus whatever
tombstones were left; both are a mean of part of a day. Dropping the
live rows instead would lose samples that may not be in the stored
mean, so they stay as synced. They are few, the folded_window guard
refuses new ones, and the nightly pass walks over them without writing.
A day or hour without a stored mean still folds as before.
Inside the horizon, a pass that recomputes a stored mean soft-deletes the
live samples it took in as fold leftovers instead of deleting them, so a
second pass computes the same mean from the same samples. Before, a fold
that deleted them left the repair a fragment, and the repair then rewrote
the mean from the tombstones alone.
The repair, the mean consolidation and the dense fold now take a per-account
advisory lock after the restore lock in every day transaction, and the
repair reads the day's samples inside it, so the two never compute the
same window from two views of its samples or write the same row at once.
The repair also stops at the fold boundary: a day the fold has not
finished is the fold's to complete.
The daily-mean consolidation and the dense hourly fold cut at
`now - threshold` instead of a local day boundary. A day was folded in two
runs and the second run overwrote the stored mean with the mean of the
later part alone: every mean-type day, and the boundary hour of every
heart-rate, HRV and SpO2 day, plus the derived resting figure and the DAY
rollup of such a day. The Health Connect importer split its raw window at
the same unaligned instant.
- Fold up to the start of the account's local day the threshold ends in
(`foldBoundary`), for every consolidation pass and the Health Connect
importer's raw/hourly split.
- A window that already has a live `stats:` row is refolded from every
sample of it, live rows plus the fold's own tombstones; with none left
the stored mean stays.
- `folded_window` refuses a raw sample only when its local day lies before
the aligned boundary and a live `stats:` row of its source covers it,
also within reach of the sample after a timezone change.
- A deletion inside a day the fold had not finished is class B.
- Add a one-time, resumable fold repair per account that recomputes each
hourly and daily mean from its remaining samples and writes only values
that differ, recorded in `measurement_fold_repairs` (migration 0389).
- The compaction-tombstone purge waits for the repair per account, and the
boot sends no purge run before one account is repaired.
- The restore keys live `stats:` ids by source, and writes compaction
tombstones back from a file older than the account's repair, resetting
the repair.
The nightly dense-intraday fold scanned Apple Health rows only, so the
per-minute heart rate a Health Connect import writes for the last 90 days
never aged into hourly means and every new export added another quarter
of raw samples.
The fold now walks each fold source in turn, Apple Health first and
unchanged, and folds Health Connect samples into hourly stats: rows under
their own source, deleting the raw rows outright. The folded_window rule
covers both sources for the hour window and looks a covering row up under
the sample's own source; the day window stays Apple Health only. The
Health Connect importer applies the same rule to raw samples, so an
export that still holds folded minutes does not put them back.
Marc-André Bombeck