HealthLog/docs/ops/encryption-key-rotation.md

# Encryption-key rotation

HealthLog encrypts sensitive at-rest data (Withings tokens, AI provider keys,
notification channel configs, web-push subscription secrets, VAPID private
keys, etc.) with AES-256-GCM under a per-deployment key. v1.4 introduces a
versioned key format so the key can be rotated without downtime and without
re-encrypting every row by hand.

## Format

| Layout          | Marker                                          | Where                       |
| --------------- | ----------------------------------------------- | --------------------------- |
| Versioned (new) | `<keyId>.<base64(iv \|\| tag \|\| ciphertext)>` | All new writes              |
| Legacy (v1.3.x) | `<base64(iv \|\| tag \|\| ciphertext)>`         | Existing rows until rotated |

`<keyId>` matches `[A-Za-z0-9_-]{1,32}` and indexes into the `ENCRYPTION_KEYS`
JSON map. Decryption tries the versioned format first; if no `.` is present
or the prefix isn't a known id, the row is treated as legacy and decrypted
under `v1` (the synthetic id assigned to the existing `ENCRYPTION_KEY`).

> **Why a separate `v1` is required.** Legacy ciphertexts have no key id, so
> the only way to identify them is "no `.` in the value". If you remove the
> `v1` entry from `ENCRYPTION_KEYS` _before_ every legacy row has been
> rotated, those rows can't be decrypted any more — the active key won't
> match the original ciphertext. The decrypt path now refuses to silently
> fall back to the active key in that scenario; it throws a clear error.

## Rotating from v1.3.x to a new key

1. **Generate the new key** on a machine with `openssl`:
   ```
   openssl rand -hex 32
   ```
2. **Update environment variables.** Keep the existing `ENCRYPTION_KEY` in
   place — it's still needed to decrypt legacy rows during the transition:
   ```
   ENCRYPTION_KEY="<old key, unchanged>"
   ENCRYPTION_KEYS='{"v1":"<old key>","v2":"<new key>"}'
   ENCRYPTION_ACTIVE_KEY_ID="v2"
   ```
   Restart the app. New writes are now encrypted under `v2`; existing
   `v1`-keyed and legacy bare rows still decrypt because the `v1` entry is
   retained.
3. **Re-encrypt the stored data under the new key.** Open **Admin →
   Encryption** and press **Rotate now**. The button needs an admin account
   with a second factor or a passkey, confirmed recently; it queues one background run
   (`POST /api/admin/encryption/rotate`) on the app's own worker, so it
   works on the published image with nothing else installed. The same page
   shows the per-column counts and the result of the last run.

   Without a second factor or a passkey on any admin account, run the rotation script
   instead. It imports the application code and the generated database
   client, so it runs from a source checkout of the same release, not
   inside the production container (the image has no package manager and
   does not contain `scripts/rotate-encryption-key.ts`). It needs the same
   `ENCRYPTION_KEYS` and `ENCRYPTION_ACTIVE_KEY_ID` as the app and a
   `DATABASE_URL` that reaches the database:

   ```
   git checkout vX.Y.Z   # the release the instance runs
   pnpm install && pnpm db:generate
   pnpm dlx tsx scripts/rotate-encryption-key.ts v2
   ```

   The bundled `db` service publishes no port. On a stock compose stack,
   either publish `5432` for the duration, or run the checkout in a
   throwaway `node:22` container attached to the compose network.

   Both paths are idempotent: running again is a no-op for rows already
   prefixed with `v2.`. Both rotate every column in the canonical registry
   (`src/lib/crypto/encrypted-columns.ts`), which covers the `*Encrypted`
   columns plus the ones whose names say nothing about their contents:
   `IntegrationStatus.lastError`, `CoachMessage.encryptedContent`,
   `NotificationChannel.config`, the OAuth `accessToken` / `refreshToken`
   columns, the web-push `p256dh` / `auth` secrets, the idempotent-replay
   `IdempotencyKey.responseBody`, and `DataBackup.data` — the whole-account
   backup blob.

   When you used the script, read three things off its output before going
   further:

   - the per-column line, `scanned` / `rotated` / `errors` / `dropped`.
     `scanned` counts the rows that hold ciphertext in that column, not every
     row of the table. Treat `errors > 0` as a hard failure and re-run after
     fixing the cause. A
     `dropped` count is only ever a cache row the run could not read and
     deleted rather than leave unreadable.
   - `Columns walked: N/M registered`. The two numbers must match.
   - a `NOT WALKED` block, if one appears. It lists registered columns this
     run skipped, and the run exits non-zero. Rotation is incomplete; do not
     go on to step 4.

   Two guard tests keep the registry honest in CI: one scans the schema for
   `*Encrypted` columns, the other derives the ciphertext-bearing columns from
   the Prisma write payloads, which is what catches a column that holds
   ciphertext under an ordinary name.

4. **Confirm nothing is left on the old key, THEN drop it.** This is the step
   that destroys data if it is taken on a bad signal, so check the corpus
   rather than the absence of complaints. Open **Admin → Encryption** and
   confirm the status view reports zero rows outside the active key, or
   re-run the script: a clean second pass reports `rotated=0`, `errors=0` and
   a full `Columns walked` line. Both read the same registry the rotation
   walks.

   A run that walked every column without an error also removes the boot
   key check's record of each previous key id that no longer holds a single
   value (`encryption_key_canaries`); the script prints which ids it removed
   and, for an id that still holds values, the columns that do. With the record
   gone, the id can later carry a different key without the start-up check
   refusing it.

   A zero only means "nothing left" when the run also says it walked every
   registered column. A zero from a run that skipped columns means "never
   looked", and dropping the old key on it makes those rows permanently
   undecryptable — `decrypt()` is fail-closed and there is no recovery path.
   Only once the corpus reads clean:

   ```
   ENCRYPTION_KEYS='{"v2":"<new key>"}'
   ENCRYPTION_ACTIVE_KEY_ID="v2"
   # ENCRYPTION_KEY can now be removed
   ```

   Restart. The legacy single-key fallback is now disconnected; only `v2`
   exists.

> **The rows in the database are covered. The content of a backup is not.**
> Rotation re-encrypts every registered column, including `DataBackup.data`
> and the pieces in `DataBackupChunk`, so every stored copy opens under the
> new key. What it cannot change is what a disaster-recovery backup carries
> inside: the database's ciphertext as it was stored when the copy was taken,
> which a restore writes back verbatim. A copy taken before the rotation still
> needs the old key for its notes, documents and coach history, and so does
> every copy in the off-host bucket and every backup file you downloaded.
>
> So before step 4, also check **Admin → Encryption → Keys the backups still
> need**. It lists, per key id, the stored copies that need it with the oldest
> date, and when the last off-host copy needing it expires under the bucket's
> lifecycle rule. The database dumps are listed too: a dump is restored only
> with the key that was active when it was taken (`db-restore` refuses
> without it), so the old key stays listed while a nightly or pre-migration
> dump taken under it is on this server, and until the bucket's rule for
> `database/` has expired the off-host copies. Keep the old key in
> `ENCRYPTION_KEYS` until it is no longer listed there (the weekly copy
> replaces itself within a week; delete old uploaded copies you no longer
> want, and wait out the dump and off-host retention), or accept that those
> copies cannot be restored. Copies written before v1.39.3
> did not record their keys and are listed as such: treat them as needing
> every key that existed when they were written.
>
> Dropping the key early no longer fails silently: a restore, a restore
> preview or an upload of a copy that needs a missing key is refused with the
> key id named, and nothing is changed. Putting the key back makes the copy
> restorable again.
>
> If you rotated on a release before v1.38.6 and dropped the previous key,
> the stored backups themselves are encrypted under the key you removed: put
> that key back into `ENCRYPTION_KEYS` and re-run the rotation on this release
> before removing it again.

## Adding a third key (v2 → v3)

Same procedure, just shift the labels — keep `v2` in the map until a full
run (every registered column walked) reports zero `v2.`-prefixed rows
remaining.

```
ENCRYPTION_KEYS='{"v2":"<old>","v3":"<new>"}'
ENCRYPTION_ACTIVE_KEY_ID="v3"
# restart, then Admin → Encryption → Rotate now
# (or: pnpm dlx tsx scripts/rotate-encryption-key.ts v3 from a checkout)
ENCRYPTION_KEYS='{"v3":"<new>"}'
```

## Rollback

> **Important.** Once a rotation has run, ciphertexts in the
> database start with `v2.` (or whatever the active id is). The pre-PR
> v1.3.x image cannot read that prefix — it expects bare base64 — and
> calling `decrypt()` on those rows will throw.

If you need to revert to the pre-rotation image:

- Either keep the new image. The new code reads both formats, so most
  rollback scenarios don't need to undo rotation.
- Or, if you must run the old code, restore a database backup taken
  _before_ the rotation ran. There is no script to convert
  `v2.`-prefixed rows back to legacy format — by design, rotation is a
  forward-only operation.

This is why we recommend running the rotation in a window where you have a
fresh DB snapshot and the new image has been smoke-tested in production for
at least 24 hours.

## Troubleshooting

- `Encryption key id 'v1' is not configured` — the database still contains
  `v1.`-prefixed rows but `v1` was removed from `ENCRYPTION_KEYS`. Re-add
  the key, run the rotation again, then remove it.
- `Found a legacy-format ciphertext but no v1 key is configured` — same
  cause for legacy bare-base64 rows. Restore `ENCRYPTION_KEY` (or add a
  `v1` entry to `ENCRYPTION_KEYS`) and run the rotation again before
  removing it.
- `Refusing to rotate: argv key id ... does not match the currently active
id ...` — pass the same id you set in `ENCRYPTION_ACTIVE_KEY_ID`. The
  guard prevents accidental re-encryption to a non-current key.