get_metric_table reads one metric as days, weeks or months through the
chart reader, cut in the user's zone and collapsed to one source per day,
with null for a period without a reading. The model reads a bounded
summary; the full table rides the turn, encrypted at rest. show_result
shows a table an earlier reply of the same conversation fetched, resolved
only against that conversation's own messages. Workouts, labs, sleep and
adherence results project into tables, earlier tables are named in the
context by metadata alone, and result:rN marks are stripped from the reply.
A restore of an account with 1.9 million readings, most of them deleted
by the nightly consolidation and still awaiting the purge, was cancelled
in its clearing step: `DELETE FROM measurements` ran past the 60 s
statement timeout every pooled connection carries, and the whole restore
rolled back with a message that named no cause (#1031).
Most of that statement was not the delete. Two columns point into
`measurements` with ON DELETE SET NULL and had no index, so Postgres
scanned `personal_records` and `ecg_recordings` once per deleted reading:
38 of 40 s on a seeded account with 300 personal records. Migration 0362
indexes both, which also speeds up the tombstone purge and account
deletion.
The restore transaction now sets its own statement and idle-in-transaction
limits to the transaction budget with SET LOCAL, and the budget counts
the rows the account holds as well as the rows the file carries. A
rolled-back restore names what the database refused it over (timeout,
lock, constraint, storage, connection, other) on the job row, the
console, the audit row and the wide event.
Each later piece of the dialog gets its own module, answering nothing for
now: the step builder, the table projection and chart choice, the result
ref marks, the follow-up derivation, sentinel, resolution and continue
chip, the reuse turn, the clarification parse and answer, the method line,
and the four prompt addenda. The turn pipeline calls them where they
belong, so the frames go out in the documented order (step, token,
provenance, result, cards, clarification, followUps, done) and a blocked
turn drops its tables, chips and clarification. With every piece empty
the stream and the prompt stay byte for byte as before.
The home location of the environment module was rounded to 2 decimals,
about 1 km, which places a home in a neighbourhood rather than the town the
schema comment promised. The weather archive it feeds is gridded at 9 to
25 km, so the extra precision bought nothing.
Every stored coordinate of the module now goes through the same 1-decimal
rounding (about 11 km north-south, 7 km east-west at 50° N): the home, the
travel periods, the geocoder's results, the per-day weather rows, and a
restore of a file written with finer values. Migration 0361 rounds the
existing users, travel periods and per-day rows; running it twice changes
nothing more. The answers keep their shape, plain numbers. The API
descriptions and the schema comment say 1 decimal now.
A new nullable column, coach_messages.results_encrypted, holds the tables
of values a turn read as AES-GCM ciphertext; only their metadata rides the
plaintext provenance. readMessageResults serves them to the owner, each
table whole or withheld with a reason. The provenance restore learns the
steps, method, table metadata, chips, clarification and the forced-answer
marker, entry by entry against the wire schema.
The column is registered for key rotation, carried through both backup
formats (ciphertext in a disaster-recovery file, readable JSON in a
portable one) and goes with its row on conversation delete and wipe.
The GPS track of an outdoor workout was stored readable, and its first and
last points are usually the person's front door. The phone number and address
in the address book of doctors and practices were readable too, while the
note beside them was already encrypted.
Migration 0361 adds workout_routes.geometry_encrypted (the geometry loses
its NOT NULL) and practitioners.phone_encrypted / location_encrypted, plus
partial indexes on the backfill's discovery predicates. A track is sealed as
one binary AES-256-GCM value bound to its purpose; the contact fields use the
same Bytes codec as the note. The practitioner's name, practice and specialty
stay readable, because the picker searches and sorts on them.
Every writer stores ciphertext only: workout batch ingest, practitioner
create and edit, and both restore formats. Readers prefer the ciphertext and
fall back to the old column for rows not yet migrated: the workout detail
(map, splits, GPX), the activity insight's climb figure, the practitioner
DTO behind the address book, visits and vaccinations. The wire shape does not
change. An edit's audit row now names a changed phone number or address
without recording either value.
free-text-encryption-backfill covers the three columns in the same pass,
finding a route through its workout. All three are registered for key
rotation (registry, CLI script, and the in-app rotation). A disaster-recovery
backup carries the contact fields as ciphertext only; a portable export
carries them readable. Workout routes are still not part of a backup, as
before. The old columns are marked for removal in a later release.
The purge requests and the off-host key-use ledger are not keyed to a
user, so truncating users never reached them. A file that reads them
whole, the purge job or the encryption status view, saw rows earlier
files had left and failed in a full serial run.
The per-account password limit refused every attempt once ten failures
had landed in fifteen minutes, counted before the password was checked.
Ten guesses from two addresses therefore locked a password-only account,
the app's password sign-in included, and repeating them kept it locked
for as long as anyone cared to.
A place the account has signed in from before is now neither counted nor
held back: a remembered browser, a phone presenting the device id a login
of this account was issued to, or an address a session or device login of
this account came from in the last thirty days. Anywhere else gets five
free failures a day and then a wait that doubles with each failure, from
thirty seconds to at most fifteen minutes, instead of a wall. The per-IP
limit is unchanged, and an unknown identifier is limited the same way on
its hash.
Under an SSO-only policy, a password sign-in to an account other than the
one a pending link names is now refused before the password is checked,
so that exception no longer tells right passwords from wrong ones.
Offering access to the record took a live session alone, and an
invitation outlives the session that sent it: whoever held a stolen
session could invite their own account and keep a way into the record
after the owner signed everything out. POST /api/account/grants now takes
the recent-proof gate at every level (a sign-in or re-proof within five
minutes on the browser, with the second factor on an account that has one;
a step-up elevation on a token, which no shipped client uses here), and
the invite form opens the re-proof dialog and sends again.
Sign out everywhere now withdraws every invitation the account offered
that nobody has accepted. Accepted grants are left standing on purpose:
each is a person the owner chose, often a carer who still needs the
record, and a sign-out button must not quietly cut them off. The response
lists them as grantsKept and the sessions card shows each with a one-click
End access.
A medication's start and end dates are calendar dates, but the report
compared them against the window's instants. West of UTC the window
begins hours after UTC midnight, so a course that ended on the first day
was left out, and one that started the day after the window was listed.
Both now compare against the window's first and last day in the report
zone.
A data wipe during the nightly upload could leave the pre-wipe copy in
the bucket: the purge the wipe started removed its request at once, and
the upload still in flight checked that request after finishing and found
nothing. A wipe's request now stays until a nightly run has walked the
account since the wipe. Until then each purge removes only the copies
dated up to the wipe's day, so the copy of what the account holds after
the wipe stays, and the admin card no longer counts the waiting request
as copies still in the bucket.
The passkey sign-in, security-key sign-in and security-key registration
verifiers read the challenge, verified the response, and deleted the
challenge afterwards. Two requests carrying the same assertion could both
read it before either deleted it, and both be accepted.
A challenge is now claimed with one guarded DELETE ... RETURNING that also
checks its ceremony, its account and its expiry, before anything is
verified; a concurrent second request finds nothing. A passkey sign-in also
refuses a challenge that was begun for a different account than the passkey
belongs to.
Off-host objects are now written as envelope version 4: the header names
the off-host key by fingerprint, and the object's key in the bucket
(<date>/user-<id>.json.enc) is authenticated as GCM associated data, so
a copy moved to another account or another night no longer opens.
Objects of versions 1 to 3 still read. BACKUP_ENCRYPTION_PREVIOUS_KEYS
(comma-separated, on the compose whitelist and in the env manifest)
keeps retired off-host keys readable, which is the rotation path the
off-host key did not have.
The admin backup upload counted the bytes that arrived and none after
the gunzip, so a small gzip could expand without limit into the JSON
scanner and take the app down. The inflated bytes are now held to 100
times the compressed bytes read (at least 256 MB, at most 4 GB) and a
file past that is refused with 413 before anything is stored.
The extractor held the inflated export.xml to a flat 8 GiB whatever the
member occupied, and the header ratio check reads sizes the archive
writes itself: a member declaring an honest ratio could inflate at
1000:1, so an 8 MB upload could fill the temp directory. The streamed
byte count is now held to 200 times the member's compressed size on
disk (never less than 64 MiB, never more than 8 GiB), and a stored
member may not grow at all.
An account may have one Apple Health import queued or running. A
second upload with different bytes is refused with 409
import.apple_health.busy and the running job's id; the check and the
new row are one transaction under an advisory lock.
The staged upload is the person's export in plain text. It is now
removed on every exit that leaves no worker to read it: a refused or
broken multipart body, no worker running, a queue that takes no job
(the row is failed as well), the admin route's missing or unknown
target account, and an account deleted before the worker reached it.
The worker removes the files before recording a failure, so a row that
went with a deleted account no longer strands them. The 15-minute
import reconcile pass also sweeps staged files older than the longest
an import may run.
"Sign out everywhere" ended web sessions and device logins but left every
credential that works without signing in: a connected AI assistant kept its
OAuth connection and minted the next access token from it, a connector or
ingest token kept working, and a clinician share link kept opening the
record. A password change revoked the tokens but not the assistant's
connection or the share links. Anything a stolen session had minted
outlived the remedy.
- Sign out everywhere now ends every other sign-in, every AI-assistant
connection, every programmatic token except the caller's own, and every
clinician share link. Share links are revoked by default; the web offers
a checkbox to keep them (keepShareLinks=1), and a client that sends
nothing, such as the shipped app, gets the safe default.
- A password change and an admin password reset also end AI-assistant
connections, share links and step-up elevations.
- Turning on the first second factor (TOTP or a security key) ends every
other sign-in, as a password change does. Adding a further factor does
not.
destroyOtherSessions takes an explicit reach ("sign-ins" or "everything")
so each caller says how far it goes.
Re-uploading a large export with an externalId column ran one
updateMany per row inside a single interactive transaction; on a
9 000-row file that outlived Prisma's 5 s timeout and the request ended
in a 500.
A re-imported externalId row whose stored value, unit, instant, glucose
context and note already match the line is now a duplicate: not written,
reported as skipped / duplicate. Changed rows still update, a tombstoned
one is still resurrected. The existence probes run per measurement type
in chunks of 1 000 keys, and the writes commit in chunks of 200 (creates
via createManyAndReturn, updates as one batch transaction per chunk)
instead of one transaction around the whole file. A failure part-way
leaves earlier chunks written, and re-uploading the same file completes
the rest.
The OpenAPI description now says what a re-upload does, including the
natural-key dedup for rows without an externalId, which it used to call
a duplicate.
Deleting an account, deleting a managed profile, "Delete all data" and the
admin wipe now write an off-host deletion request in the same transaction
as the deletion. A new job (offhost-backup-purge, kicked straight away and
run nightly as a backstop) lists the bucket once, deletes every
<date>/user-<id>.json.enc object of each pending account and removes the
request only when nothing of the account is left. A refused delete keeps
the request with the bucket's answer and is retried; the off-host card
shows how many deletions are outstanding and why. A copy the nightly run
was writing while the account was deleted is taken back after the upload.
The off-host card also reads the bucket's lifecycle rule and says plainly
when there is none (every nightly copy kept forever) or when the
credential may not read it.
Every stored backup now records the key ids its content was written
under, and every off-host upload records the same per key. The admin
encryption view lists, per key, the stored copies that need it with the
oldest date, and when the last off-host copy needing it expires. The
"safe to drop" badge now requires that no backup still needs a retired
key, instead of only that every row was rotated.
Migration 0358 adds data_backups.inner_key_ids, the key-use table and the
deletion requests.
Exporting the whole record, making a clinician share link, minting a token,
and the admin backup download, upload and restore, data wipe and password
reset ran on a live session alone. The plaintext full backup did, while the
encrypted one beside it asked for a second factor. Whoever held a stolen
session could take the record, or leave behind a credential that outlives
the session.
These now go through one gate, requireRecentProof (assertRecentCookieProof
for the routes that resolve their caller themselves). A browser session
passes after a sign-in or re-proof within five minutes: a second factor on
an account that has one, otherwise the password or a passkey. The refusal is
401 auth.reproof.required with meta.methods, and the web opens the re-proof
dialog, posts the proof to the new POST /api/auth/reproof and retries. A
password proof stamps the new Session.reproofAt (migration 0359), never
mfaVerifiedAt, so it cannot stand in for a second factor anywhere.
On Bearer, the full backup asks for an X-Step-Up elevation; the encrypted
export asks for a second-factor elevation on an account that has one. Share
links and connector tokens keep taking the token alone, because the shipped
app calls them without an elevation and reads a 401 there as a dead
session; the guard test names them.
Key rotation keeps its stricter second-factor gate.
The title of a Coach conversation is the first 80 characters of the
opening message, and the note on a custom-metric reading is free text;
both were stored readable while every other note was encrypted.
Migration 0357 adds coach_conversations.title_encrypted and
custom_metric_entries.note_encrypted (the title loses its NOT NULL) plus
partial indexes on the backfill's discovery predicate. Every writer now
stores ciphertext only: conversation create, proactive nudge, reminder
surfacing, rename, entry create and edit, and both restore formats.
Readers prefer the ciphertext and fall back to the old column for rows
not yet migrated; a ciphertext that does not open throws instead of
falling back. The history title search decrypts in memory, since SQL can
no longer match the title.
free-text-encryption-backfill is a boot-time, idempotent pg-boss job
(staggered past the startup storm, daily discovery tick) that seals each
remaining readable value and nulls it in one transaction per row. A pass
that sees a row twice stops with an error instead of looping.
Both columns are registered for key rotation (registry, CLI script, and
the in-app rotation that walks the registry). A disaster-recovery backup
carries ciphertext only, sealing a not-yet-migrated row on the way out;
a portable export carries readable text. The old columns are marked for
removal in a later release.
A backup carries every encrypted column as the stored ciphertext, and a
restore writes it back verbatim. Key rotation re-seals the stored copy's
envelope but cannot reach inside it, so a copy taken before a rotation
still needs the retired key. Without it the restore went through and
left rows no reader can open.
The restore, the restore preview, the admin upload and the monthly
restore drill now collect the key ids of every inner ciphertext value,
hold them against ENCRYPTION_KEYS and decrypt one value per key. A
missing key, or an id configured with different key material, is
refused before anything is deleted, with the key id named. The restore
status and the preview say which key to put back.
The restore also checks, inside its transaction, that it added no
reference from the account's rows to rows another account owns. The
check is derived from the schema's foreign keys rather than written per
section, so the tag and category upserts keyed on a global key are
covered along with every link table. The illness parent update is
scoped to the owner as well.
The off-host listing now follows every page: the first page held only
the oldest dates, so the drill could read an old object and report the
chain as stale.
Building the coarse buckets from the DAY tier read every day of the window
into the process: a five-year yearly read moved thousands of rows per type,
and the Coach's history bands read several such windows per request. The
per-day source pick and the fold now run in Postgres (DISTINCT ON the day,
ordered by the ladder rank and then the source name, the same order the
in-process pick uses), so a coarse read returns one row per bucket. The
long-range chart picks its tier from the span of the window's days and reads
that tier once. The Coach snapshot hands its already-loaded source order to
the history bands instead of each band reading it again.
A report whose end date lies in the past listed medications that started
after it, the dose history up to today and the latest injection even when it
came later. It now lists a medication only if its course had begun by the
end date (or it had a dose scheduled inside the window) and had not ended
before the start, and dose history and the last injection stop at the end
date.
The status assessments folded their recent days, weeks, months and years on
Berlin days whatever the user's zone, the mood and cycle crosstabs keyed
their metric days in Berlin, the resting-pulse proxy defaulted to Berlin
days, the briefing's opener rotated at Berlin midnight, and every
assessment prompt labelled today's date as Berlin's. They now use the
user's zone. Rollup months and years are labelled by their UTC start,
which is how they are cut; read in a zone west of UTC the label named the
month before.
"Europe/Berlin" was written out in some eighty places. DEFAULT_TIMEZONE in
lib/tz/format is now the one definition: the second copy in mood/date-key,
DISPLAY_TIMEZONE, the Berlin-only day key helpers and the per-route copies
are gone. An in-repo lint rule rejects the literal outside lib/tz and
tests; four areas another change owns are listed as pending.
The SSO callback bound a verified provider email to the local account with
the same address on the spot. The local address is not verified: anyone
could register with it, or type it into their profile, and the SSO user
then landed in an account the other person still held the password to.
The callback now links nothing. It records the link it would make in a
sealed, ten-minute, HttpOnly cookie and sends the browser to the sign-in
page, and the link is made only when that browser then signs in to that
same account with its password (and second factor) or passkey. Under an
SSO-only policy the password form opens for exactly that account. The
native sign-in answers oidc_link_required; the link is confirmed once on
the web.
Moving an account to a new email address now also needs a fresh proof: a
sign-in or second factor in the last five minutes, or the current password
sent alongside the change. Without it the address is left as it was and
reported in rejectedFields; the rest of the save still lands.
The all-time weight, blood-pressure and pulse figures the insight features
carry averaged every raw row, so a device that samples densely outweighed the
readings the source ladder prefers on the same days. They now keep one source
per day, like the dashboard's all-time figures.
The period-over-period range delta averaged readings. For step-like totals
the current window still holds the last day and a half as many small
samples, so the average collapsed; it now compares average daily totals. For
sleep it averaged stage segments and in-bed envelopes; it now compares
average nights of time asleep. The API descriptions say which.
A browser session on its own could start TOTP enrollment or register a
security key. Combined with a pending secret counting as a factor, that let
a stolen session give itself a second factor, register a passkey with it,
and keep access after the owner changed the password.
TOTP setup and security-key registration now pass on the cookie path only
when the session signed in or completed a second factor in the last five
minutes, or when the body carries a proof in the step-up shape (password,
a confirmed authenticator code, or a passkey or security-key assertion).
The settings cards ask for that proof when the server answers
auth.reproof.required and retry the same call with it. The Bearer path is
unchanged: its elevation already is the proof.
Every route that checks a password or a factor as a re-proof now uses one
verifier and one per-account budget of five attempts per fifteen minutes,
shared with the step-up mint, and records each refused proof as
auth.mfa.failed. Passkey enrollment and password change had neither the
limit (passkey enrollment) nor the audit row (both). A structural test
freezes the set of credential-verifying routes and requires a throttle and
an audit call in each.
The rollup writer keeps one row per bucket and source. Collapsing the stored
WEEK / MONTH / YEAR rows picked one source for the whole bucket, so a month
of manual blood-pressure readings plus one synced reading reported the synced
day alone, and a step month that switched devices lost every day the losing
device carried. The live weekly and monthly chart fold made the same pick.
Every coarse bucket is now folded from the canonical DAY buckets: the source
ladder picks one source per day, then the week, month or year counts every
one of its days. Spread, slope and r² are composed from the summed day
accumulators. The live fold picks per local day before folding into weeks
and months. Ties between sources without a ladder rank break alphabetically
on the SQL paths too, matching the rollup reader.
The login challenge read its attempt count, verified the factor, and only
then incremented. Requests sent in parallel all read the same count below
the cap, so one ticket allowed more guesses than the cap. Each attempt is
now reserved by a single guarded UPDATE before the factor is evaluated,
and a failure only settles the reserved attempt.
A pending secret from an enrollment that was never confirmed counted as a
second factor wherever a code was verified: step-up minting, the passkey
enrollment re-proof, MFA disable and login completion. Anyone holding a
session can create a pending secret, so a code from one proves nothing
about the account owner. verifyMfaFactor now refuses both TOTP and
recovery codes until totpConfirmedAt is set.
Keyed uploads and picker imports also check deleted, not yet purged
documents by content and answer deleted, remembering the new key on the
tombstone so the purge carries it into the ledger. A hand upload without
a key still stores a new copy. No content hash of a deleted document is
kept past the purge.
- Source keys carry the normalised origin of the Paperless-ngx or Papra
they came from (source_instance on the document, the alias and the
purge ledger; unique NULLS NOT DISTINCT). A key stored without one
(v1.39.2 imports) matches any instance of its system, and a lookup
without one matches any stored instance. The upload route, the lookup
route and the import script accept or send it; backups carry it.
- Saving a connection reuses the stored token only for the same origin;
a new origin needs the token again (422 tokenRequired).
- Document ids are checked against what each system issues (Paperless
digits, Papra [A-Za-z0-9_-], never . or ..) before any URL is built;
the Papra organization id likewise.
- Paperless API 9 filters dates on created__gte / created__lte, in the
picker and the script. A search page the local re-check empties reads
on, up to five source pages. Tags are read past the first 250.
- The import's key pre-check uses the lookup allowance, a failed fetch
no longer charges the upload allowance, and one import per key runs at
a time under an advisory lock, so a double click downloads once.
- Reading and deleting one's own connection works with the documents
module off and the operator list unset.
A snooze that had already run out before the send is history, not a
choice about the reminded slot, so it no longer keeps a row from being
repaired. The "no due date" arm matched nothing, because a reminder
cannot be created without a cadence; it now covers a rule whose last
occurrence was the one sent, which is open unless it was done or
skipped since, while a course whose window closed before the send stays
closed through the end-date guard. The start-page step reads a stored
item list only when it is a list, so a malformed layout can no longer
fail the migration.
The first cut kept only free-text check-ups due after their reminder
and let every reminder with a measurement type roll on. A fortnightly
PHQ-9 or GAD-7 reminder is a measurement reminder, and each one still
jumped two weeks forward the morning it was sent without the
questionnaire being filled.
The rule is now the cycle length. A reminder whose next slot is more
than seven days away, or that has no cadence, stays due after its
reminder until it is satisfied (a matching reading, a lab result or
"done"), skipped or snoozed, and is sent again at most once a week. A
reminder on a weekly or shorter cycle rolls on as before. Appointments
stay one-shot. A reading that lands on an open measurement reminder
still sets lastSatisfiedAt and schedules the next cycle from itself.
The 0355 repair drops its "no measurement type" condition for the same
cycle rule, so the fortnightly questionnaires in the reported shape
are put back on the day they were reminded.
The snoozed fixture sat inside a week of the send, so the one-week clause already excluded it and dropping the snooze guard from the migration left the suite green. Snooze it further out, a minute after the send, so only the snooze cursor tells it apart from a roll-on.
The reminder tick rolled every delivered reminder on to its next slot.
For a check-up (a reminder with no measurement type, made by the person
or suggested by the Coach) that filed an open task under next month or
next year the moment it was reminded, although nobody had done it. The
start page and the check-up card then showed a yearly check-up due
today as scheduled for next year, and a check-up with no cadence
dropped off every surface.
A check-up now keeps its due date after the reminder until it is
marked done, skipped or snoozed, and is reminded again at most once a
week while it stays open. A new `last_notified_at` column records the
last delivered reminder; a repeat is held while that send was for the
current slot and fewer than seven calendar days have passed, so a
snooze, a skip or a completion opens a fresh slot that fires on its own
day. The per-day claim stays the guard against two sends in one day.
Measurement reminders keep rolling on (the next reading satisfies them
and their next slot comes round soon), as do check-ups on a weekly or
shorter cadence, whose next slot is the next reminder anyway.
Appointment reminders stay one-shot.
Migration 0355 adds the column and puts back check-ups the old tick
rolled on: a live, enabled check-up whose most recent reminder claim is
followed within ten minutes by its last write, carrying a due date more
than a week past that send (or none, for a one-shot), not done, skipped
or snoozed since. Its due date returns to the instant of that send.
Anything else is left as it is. The same migration adds `upcoming_visit`
to start-page item choices stored before visits were one of them, which
otherwise read as visits switched off.
Marc-André Bombeck