HealthLog/docs/ops/tls-cert-pin.md

TLS SPKI pinning — coupling, alarm, and re-pin runbook

The native client SPKI-pins the server's TLS chain at the CA level. It ships a small pin set — currently the issuing intermediate plus the root — and accepts a connection only when the served chain passes baseline X.509 evaluation AND at least one certificate anywhere in the chain matches a pin (any-match). The leaf is deliberately not pinned, so routine leaf renewals are a client no-op. The outage risk is a chain change: the served chain stops terminating in the pinned CA keys — the server moves to a different CA, or the CA rotates its intermediate / root keys. This page explains the coupling, the server-side alarm that watches the served leaf, and the operator / release-owner steps when it fires.

The coupling

A SPKI pin is base64(sha256(DER subjectPublicKeyInfo)) — a hash of a certificate's public key, not of the whole certificate. The native client carries a set of these pins (build-time config, ≥ 2 well-formed pins enforced for release builds) and rejects any TLS connection whose validated chain contains no pinned key. Hosts outside the client's pinned-host list fall back to plain system-trust validation, which is the documented model for self-hosted instances that ship their own chain.

The app host's certificate auto-renews roughly every 90 days and each renewal mints a new leaf keypair. Because the client pins the CA keys rather than the leaf, those renewals pass without a client build. What DOES break every pinned client is a chain that no longer terminates in a pinned key — and there is no server-side signal that clients are pinning the served chain at all. The alarm below exists to catch exactly that early: it watches the served leaf, the most frequently rotating element, as the canary for any chain movement.

The alarm

A scheduled job (src/lib/jobs/tls-pin-monitor.ts) probes the served leaf every 6 hours and fails loudly when it changes:

• It derives the public host from APP_URL (falling back to NEXT_PUBLIC_APP_URL) and opens a raw TLS socket to it — observing exactly the certificate a pinned client would, regardless of how the reverse proxy terminates TLS.
• It reads the served leaf, computes its SPKI pin (base64(sha256(DER subjectPublicKeyInfo))), and compares it against the operator's known-good set in TLS_LEAF_SPKI_PINS.
• On a served pin that is not in the known set it emits a tls.pin.leaf_changed wide-event annotation (old set + served pin), writes a system.tls.pin_changed audit-log row, and fans out a high-priority SYSTEM_ALERT to every admin user through the notification dispatcher (Telegram / ntfy / Web Push / APNs, whichever the operator has wired).

A transient TLS or network failure during the probe is annotated (tls.pin.probe_failed) and never alarms — the alarm fires only on a successful probe returning a pin genuinely outside the known set.

Baseline source: an env var, not auto-learning

TLS_LEAF_SPKI_PINS (comma-separated) is the known-good set of served leaf pins. It is the operator's explicit record of what the server is expected to serve — distinct from the client's CA-level pin set, which lives in the client build config. The baseline is an env var on purpose, not a persisted "last-seen" row:

• An explicit baseline forces the operator to acknowledge every leaf rotation and check the new chain against the client's pinned CA keys before the alarm goes quiet.
• A persisted last-seen baseline would silently adopt the first rotated pin and suppress the very alarm the pinned client needs.
• When the env var is unset, the monitor fails loud, not open: it logs the served pin (so you can seed the baseline) and warns "not configured", but never auto-adopts what it observed.

Setting the baseline

Extract the current leaf SPKI pin from the served certificate:

openssl s_client -connect "$APP_HOST:443" -servername "$APP_HOST" </dev/null 2>/dev/null \
  | openssl x509 -noout -pubkey \
  | openssl pkey -pubin -outform der 2>/dev/null \
  | openssl dgst -sha256 -binary \
  | openssl base64

$APP_HOST is the host in your configured APP_URL (the host the client connects to). The output is the base64(sha256(DER subjectPublicKeyInfo)) pin. Set it in the deployment env:

TLS_LEAF_SPKI_PINS="<current-leaf-pin>"

The variable is on the compose environment: whitelist and the scripts/env-manifest.json optional groups, so pnpm check-env will report whether it is set. It is optional: without it the monitor still runs and logs the served pin, but cannot alarm on a change.

When the alarm fires

When you see a tls.pin.leaf_changed alert / system.tls.pin_changed audit row, the served leaf has rotated. First determine which of two cases you are in:

1. Routine renewal, same chain. Inspect the new chain (the openssl pipeline above, or the alert payload). If the issuing intermediate and root are unchanged, the client's CA-level pins still match and no client is at risk. Update TLS_LEAF_SPKI_PINS to the new served-leaf pin and re-deploy so the alarm goes quiet. No client build.
2. Chain change. The new leaf chains through a different intermediate / root (CA switch, or the CA rotated its keys). Every pinned client will stop connecting once the old chain is no longer served — run the re-pin procedure below inside the old chain's remaining validity, with margin for app review and client roll-out (the alert body and audit row carry the new leaf's validTo).

Re-pin procedure (chain change)

1. Extract the new chain's CA pins (intermediate + root) with the openssl pipeline / the client repo's SPKI-extraction script.
2. Dual-pin. Add the new CA pins to the client's pin set alongside the old ones (do not replace them yet). Clients on the old build keep working as long as the old chain is still served; the dual-pinned build works against both chains. Update TLS_LEAF_SPKI_PINS to the currently served leaf pin(s) so the server-side alarm reflects reality.
3. Ship the client build carrying the dual-pin set to TestFlight (and onward to release). Give it long enough to roll out to the installed base before the old chain disappears.
4. Retire the old pins once the old chain is no longer served and the dual-pinned client build has reached the installed base.

CA-level pins move rarely — intermediates and roots carry multi-year lifetimes and rotations are announced well ahead. Track the pinned certificates' expiry dates and pre-stage the successor pins before the cutover, so a chain change never catches a single-chain client.

Environment assumption

The monitor speaks TLS directly to the public host from APP_URL / NEXT_PUBLIC_APP_URL, which is correct regardless of the reverse-proxy topology — it observes the same leaf a real client does. The one assumption is that the host the client connects to is the host in that env var. If the client is pinned against a different hostname (e.g. a vanity domain that fronts the configured APP_URL), point the monitor at that host by setting APP_URL to it, or extend the job to probe the additional host. Plain-HTTP self-hosts have no leaf to pin and the monitor no-ops there by design.