Probe the oldest stored values before recording an encryption key

The boot key check recorded whatever key was configured whenever its probe
found nothing to try. It only read string columns, so an install whose
ciphertext sat in Bytes columns (notes, documents, ECG, symptoms) read as
empty; a probe that ran out of time did the same; and rows written by a
process that served under a wrong key could open on the next boot and get
that key recorded as the right one.

The probe now reads every registered column, string and Bytes, in all four
stored layouts, and tries the oldest values first, ordered by the row's
updatedAt (else createdAt, else primary key). A key is recorded only when
the oldest value that opens is preceded by at most one that does not, or
when no column holds a single value under the key id. A time-out, an
unreadable table, oversized values or mixed results are inconclusive: the
process serves, records nothing and checks again at the next start.

When the first insert loses a race, the stored record is read back and must
open under this process's key, else the outcome is a mismatch.

A rotation that walked every column without errors removes the record of
each previous key id that no longer holds any value. The refusal log block
and the self-hosting pages explain how to reset the check for a key id.

docs/ops/encryption-key-rotation.md

@@ -103,6 +103,13 @@ under `v1` (the synthetic id assigned to the existing `ENCRYPTION_KEY`).
    a full `Columns walked` line. Both read the same registry the rotation
    walks.
 
+   A run that walked every column without an error also removes the boot
+   key check's record of each previous key id that no longer holds a single
+   value (`encryption_key_canaries`); the script prints which ids it removed
+   and, for an id that still holds values, the columns that do. With the record
+   gone, the id can later carry a different key without the start-up check
+   refusing it.
+
    A zero only means "nothing left" when the run also says it walked every
    registered column. A zero from a run that skipped columns means "never
    looked", and dropping the old key on it makes those rows permanently

docs/self-hosting/getting-started.md

@@ -267,6 +267,34 @@ every API request answers `503` with `meta.errorCode`
 Restore the original key, point `DATABASE_URL` at the matching database, or
 start with an empty database; then restart.
 
+The check remembers the key it first saw for each key id in the table
+`encryption_key_canaries`. Before it records a key, it reads the oldest
+encrypted values in the database and records the key only if they open, or
+if the database holds no encrypted value under that key id at all. When the
+stored data proves neither (a single unreadable value, the oldest values
+failing while newer ones open, or a probe that ran out of time), the app
+keeps serving, records nothing, logs a warning that starts with
+`[boot] Encryption key check inconclusive`, and checks again at the next
+start.
+
+If you are certain the configured key is the one your data was written
+with, and the check still refuses because it recorded a different key
+earlier, reset the check for that key id and restart. The refusal in the log
+prints the exact statement; for the default key id it is:
+
+```bash
+docker compose exec -T db psql -U healthlog healthlog \
+  -c "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';"
+docker compose restart app
+```
+
+The next start probes the stored data again and records the key only if
+that data opens with it. Do this only when you are sure: if the key is in
+fact wrong, the app serves errors on every encrypted value instead of the
+one clear refusal. After a key rotation the rotation script removes the
+record of a retired key id itself, once it reports that no values remain
+under that id.
+
 ## Backup and restore
 
 Your database is the only stateful piece — back it up, and back up the

docs/self-hosting/truenas.md

@@ -80,6 +80,32 @@ Fix it one of three ways, then restart the app:
 3. Start fresh with an empty database. The old data and its backups need the
    old key.
 
+Before it records a key as the right one, the check reads the oldest
+encrypted values in the database and records the key only if they open (or if
+the database holds no encrypted value under that key id yet). When the data
+proves neither, the app keeps serving, records nothing, writes a warning
+starting with `[boot] Encryption key check inconclusive` to the log, and
+checks again at the next start.
+
+### Resetting the check
+
+If you are certain the configured key is the one the data was written with,
+and the app still refuses because the check recorded a different key earlier,
+remove that record and restart. Open a shell on the database container (Apps,
+HealthLog, the database container's Shell) and run, with the key id the log
+names (`v1` unless you use `ENCRYPTION_KEYS`):
+
+```bash
+psql -U healthlog healthlog \
+  -c "DELETE FROM encryption_key_canaries WHERE key_id = 'v1';"
+```
+
+Then restart the app. It probes the stored data again and records the key
+only if that data opens with it. If the key is in fact wrong, the app then
+fails on every encrypted value instead of refusing clearly, so only do this
+when you are sure. After a key rotation you never need this for the old key
+id: the rotation script removes its record once no values remain under it.
+
 ## Updates
 
 Pick the new tag, take a snapshot of the database dataset first, and update.