Several export reads sorted only by a column two rows can share (a
creation time, a day, a name), and a few to-many reads (medication
schedules, mood tag links, cycle and illness symptom links) had no order
at all. Tied rows then came back in whatever order they sat on disk, so
two exports of one account could differ, and the parity goldens flapped
once the fixture's child rows shared their pinned timestamps.
Each read now ends its order in the model's primary key: the existing
sort key plus `id`, or the remaining key columns of a compound key
(nutrient `source`, hidden tag `moodTagId`, link `symptomId` /
`moodTagId` / `documentId`). Schedules sort by window start, then id.
The restore re-export reads through the same modules, so it inherits
the order.
The parity goldens are unchanged: they already hold the tie-broken
order, so no regeneration was needed and they still pin the output of
the pre-split exporter.
A new integration test runs both purposes, built and streamed, through
a client that records every findMany and every nested to-many read, and
fails on any whose order does not cover the primary key (keys pinned by
an equality in `where`, or by the foreign key to the parent, count as
covered). A second case writes the same tied rows in forward and
reverse physical order and requires byte-identical files.
Add the shared contract the reasoning release builds on, with no change
to how a turn runs today.
- Schema: operator reasoning controls on app_settings (0376), the
encrypted trail text on coach_messages (0377), and source,
last_used_at and source_message_id on coach_facts (0378).
- Provider contract: CompletionParams.reasoning and onReasoning,
CompletionResult.reasoning and providerState, AiMessage.providerState.
- Reasoning levels, thinking budgets and the background-job table in
src/lib/ai/reasoning/levels.ts; coachPrefsJson.reasoning, absent
meaning medium, salvaged field by field like every other pref.
- Stream: activity, memoryNote and planProposal frames, done.stop and
done.withheldResults, result.interim, a compare chart kind, eight
table refs, three more clarification kinds with an assumed choice,
the change_assumption chip, and assumptions, activity, stop,
memoryNote and planProposal on the provenance. Zod mirror, type
parity and the native-client key rule cover every new frame.
- Chat request: memoryDecision and planDecision, parsed and not yet
acted on.
- Stub contracts for the memory functions and the activity recorder,
inert until their implementations land.
- Backup, restore and rotation carry the new columns; a fact's message
reference is resolved against the restored messages like its
conversation reference.
- Message keys for the trail, memory, plans, assumptions, replies,
interim tables, comparison charts and reasoning settings in all seven
locales, each reachable from a catalog and checked both ways.
A new nullable column, coach_messages.results_encrypted, holds the tables
of values a turn read as AES-GCM ciphertext; only their metadata rides the
plaintext provenance. readMessageResults serves them to the owner, each
table whole or withheld with a reason. The provenance restore learns the
steps, method, table metadata, chips, clarification and the forced-answer
marker, entry by entry against the wire schema.
The column is registered for key rotation, carried through both backup
formats (ciphertext in a disaster-recovery file, readable JSON in a
portable one) and goes with its row on conversation delete and wipe.
Sealing a readable Coach title or custom-metric note while building a
disaster-recovery file gave each export a fresh IV, so the streaming
writer and the materialising builder no longer produced the same file.
A row the free-text backfill has not reached yet now rides as stored,
readable, inside the encrypted backup; the restore seals it, and a
migrated row carries its ciphertext verbatim as before.
The title of a Coach conversation is the first 80 characters of the
opening message, and the note on a custom-metric reading is free text;
both were stored readable while every other note was encrypted.
Migration 0357 adds coach_conversations.title_encrypted and
custom_metric_entries.note_encrypted (the title loses its NOT NULL) plus
partial indexes on the backfill's discovery predicate. Every writer now
stores ciphertext only: conversation create, proactive nudge, reminder
surfacing, rename, entry create and edit, and both restore formats.
Readers prefer the ciphertext and fall back to the old column for rows
not yet migrated; a ciphertext that does not open throws instead of
falling back. The history title search decrypts in memory, since SQL can
no longer match the title.
free-text-encryption-backfill is a boot-time, idempotent pg-boss job
(staggered past the startup storm, daily discovery tick) that seals each
remaining readable value and nulls it in one transaction per row. A pass
that sees a row twice stops with an error instead of looping.
Both columns are registered for key rotation (registry, CLI script, and
the in-app rotation that walks the registry). A disaster-recovery backup
carries ciphertext only, sealing a not-yet-migrated row on the way out;
a portable export carries readable text. The old columns are marked for
removal in a later release.
A lab note or a biomarker context written under a key this instance no
longer holds came out of a portable export as `null` — byte-identical to a
column that was never written. The restore importer then wrote nothing, so
the text was gone and nothing said it had ever been there. The whole loss
fit inside a fail-soft `catch` written for a list view, where returning null
is the right answer.
Both now export a short marker naming what happened, and the marker survives
a restore. The Coach transcript already did this with a placeholder sentence
and the doctor report with a rendered flag; the marker is now one constant
the transcript shares, and its comment says why an empty string will not do.
The unit test that required `note: null` here pinned the loss. It keeps the
half that matters — one bad row must never abort the export — and asserts
the marker instead. The round trip is proven end to end against a real
restore.
The transcript came back in the previous change; what the Coach keeps
BETWEEN threads did not. A restored account had every word of every
conversation and a Coach that has to be told its own history again — the
register said as much for all three: "a restore makes the account
re-tell its history", "keeps the talk and loses the commitment", "a
restored account silently stops sending them".
Tombstones ride only in a disaster-recovery file. All three soft-delete,
and a deleted fact is hidden from injection while kept for audit — a
distinction only the owning instance can act on. A portable file
carrying them would hand a person rows their own app refuses to show
them, and a restore that resurrected one would put a fact back into the
Coach's mouth that the person had deliberately taken out of it.
Neither reference here is a foreign key. sourceConversationId, on all
three, and relatedPlanId on the reminder are bare id columns, so nothing
in the database refuses a value pointing at nothing. That makes them
more dangerous than a real key rather than less: a dangling reference
costs no error, it just quietly stops meaning anything while the row
goes on looking complete. The restore resolves both against what it
actually wrote, nulls what it cannot resolve, and reports the count.
Nulling rather than dropping is the point. A live reminder must not be
lost because the plan it mentions was tombstoned out of a portable file.
The reminder is still the person's and it still fires; what it loses is
a pointer that was already going nowhere.
Order follows from that: threads, then plans, then reminders. Running
the memory before the transcript would cut every fact loose from the
conversation it came out of and still report success.
Verified by breaking it: making the conversation lookup always return
null fails the round trip on the fact's own source field.
A restored account got a Coach that had never spoken to it. The
conversations, every turn in them, and the record of which vault
documents a thread was grounded in were all classified as backed up and
travelled with nothing. The register's own entry for the messages called
them the largest volume of free text an account owns after its
documents.
The three travel together because two of them mean nothing alone: a turn
without its thread has nowhere to hang, and an attachment names a
pairing rather than a thing.
documentScoped is the field that makes this more than lost history. It
is a permanent fence — set when a conversation is created through the
fenced endpoint or gets its first document, and lowered by no code path
anywhere, not by detaching every document and not by deleting one. A
restore that let it default to false would hand the tool loop back to
precisely the conversations the fence exists to keep away from it, and
nothing in the interface would look wrong afterwards. So it is carried
verbatim, restored verbatim, and asserted per thread rather than in
aggregate: a restore that wrote false everywhere would still return both
threads and every turn and satisfy every count.
Prose follows the contract the rest of the payload already uses. A
disaster-recovery file carries the stored ciphertext; a portable file
carries the text, because a portable export exists to be readable by the
person who owns it, exactly as it already decrypts medication notes,
mood notes and document summaries. A single row under a key the instance
no longer holds is named rather than thrown, so one unreadable turn
cannot cost someone the rest of the transcript.
Ids ride in both modes, as the reminders do, so an attachment resolves
and the facts, plans and reminders still owed have something real to
point at. The restore runs after the documents for the same reason, and
an attachment naming a document the file does not carry is dropped and
reported instead of failing the whole file over one link.
Verified by breaking it: forcing documentScoped to false on the way back
fails the round trip on that field by name.
Marc-André Bombeck