HealthLog/docs/ops/tls-cert-pin.md

TLS leaf SPKI pin — coupling, alarm, and re-pin runbook

The native client SPKI-pins the server's TLS leaf certificate. This is a deliberate hardening choice on the client side, but it couples the client to the exact leaf keypair the server serves: when that leaf rotates, a client that was only ever shipped the old pin refuses to connect. This page explains the coupling, the server-side alarm that watches for a rotation, and the operator / release-owner steps when it fires.

The coupling

A SPKI pin is base64(sha256(DER subjectPublicKeyInfo)) — a hash of the certificate's public key, not of the whole certificate. The native client carries a set of these pins and rejects any TLS connection whose leaf public key is not in the set.

The app host's certificate is issued by Google Trust Services (GTS) and auto-renews roughly every 90 days. Each renewal mints a new leaf keypair, so the SPKI pin changes on every renewal. Intermediate and root rolls have the same effect on the leaf chain. Because the client pins the leaf, every such change is a hard outage for any client that hasn't been shipped the new pin — and there is otherwise no server-side signal that a client is pinning the served leaf at all.

The alarm

A scheduled job (src/lib/jobs/tls-pin-monitor.ts) probes the served leaf every 6 hours and fails loudly when it changes:

• It derives the public host from APP_URL (falling back to NEXT_PUBLIC_APP_URL) and opens a raw TLS socket to it — observing exactly the certificate a pinned client would, regardless of how the reverse proxy terminates TLS.
• It reads the served leaf, computes its SPKI pin (base64(sha256(DER subjectPublicKeyInfo))), and compares it against the operator's known-good set in TLS_LEAF_SPKI_PINS.
• On a served pin that is not in the known set it emits a tls.pin.leaf_changed wide-event annotation (old set + served pin), writes a system.tls.pin_changed audit-log row, and fans out a high-priority SYSTEM_ALERT to every admin user through the notification dispatcher (Telegram / ntfy / Web Push / APNs, whichever the operator has wired).

A transient TLS or network failure during the probe is annotated (tls.pin.probe_failed) and never alarms — the alarm fires only on a successful probe returning a pin genuinely outside the known set.

Baseline source: an env var, not auto-learning

TLS_LEAF_SPKI_PINS (comma-separated) is the known-good set. The baseline is an env var on purpose, not a persisted "last-seen" row:

• The pinned client only trusts the pins it was shipped, so the operator must derive the client's pin set from a single explicit source of truth. The env var is that source — the same value goes into the client's pin set and into TLS_LEAF_SPKI_PINS.
• A persisted last-seen baseline would silently adopt the first rotated pin and suppress the very alarm the pinned client needs.
• When the env var is unset, the monitor fails loud, not open: it logs the served pin (so you can seed the baseline) and warns "not configured", but never auto-adopts what it observed.

Setting the baseline

Extract the current leaf SPKI pin from the served certificate:

openssl s_client -connect "$APP_HOST:443" -servername "$APP_HOST" </dev/null 2>/dev/null \
  | openssl x509 -noout -pubkey \
  | openssl pkey -pubin -outform der 2>/dev/null \
  | openssl dgst -sha256 -binary \
  | openssl base64

$APP_HOST is the host in your configured APP_URL (the host the client connects to). The output is the base64(sha256(DER subjectPublicKeyInfo)) pin. Set it in the deployment env:

TLS_LEAF_SPKI_PINS="<current-leaf-pin>"

The variable is on the compose environment: whitelist and the scripts/env-manifest.json optional groups, so pnpm check-env will report whether it is set. It is optional: without it the monitor still runs and logs the served pin, but cannot alarm on a change.

When the alarm fires — re-pin procedure

When you see a tls.pin.leaf_changed alert / system.tls.pin_changed audit row, the served leaf has rotated. The pinned client will stop connecting once its shipped pin no longer matches a served leaf. Act inside the certificate's remaining validity, with margin for app review and client roll-out — target ≥ 11 days before the old pin's certificate expires (the alert body and audit row both carry the new leaf's validTo).

1. Re-extract the new leaf pin with the openssl pipeline above. It should equal the servedPin in the alert.
2. Dual-pin. Add the new pin to the client's pin set alongside the old one (do not replace it yet), and update TLS_LEAF_SPKI_PINS to the comma-separated pair, e.g. TLS_LEAF_SPKI_PINS="<old-pin>,<new-pin>". Holding both pins means clients on the old build keep working through the cutover and the server-side alarm goes quiet because the served pin is now in the known set. Re-deploy the server so the new env value takes effect.
3. Ship the client build carrying the dual-pin set to TestFlight (and onward to release). Give it long enough to roll out to the installed base before the next renewal.
4. Retire the old pin once the old leaf is no longer served and the dual-pinned client build has reached the installed base: drop the old pin from both the client pin set and TLS_LEAF_SPKI_PINS, leaving only the current leaf pin.

Treat the dual-pin window as standing practice: always ship the next pin before the current leaf rotates, so a renewal never catches a single-pinned client. If GTS renews on a fixed cadence, pre-extracting and dual-pinning the upcoming leaf ahead of the renewal turns every rotation into a no-op.

Environment assumption

The monitor speaks TLS directly to the public host from APP_URL / NEXT_PUBLIC_APP_URL, which is correct regardless of the reverse-proxy topology — it observes the same leaf a real client does. The one assumption is that the host the client connects to is the host in that env var. If the client is pinned against a different hostname (e.g. a vanity domain that fronts the configured APP_URL), point the monitor at that host by setting APP_URL to it, or extend the job to probe the additional host. Plain-HTTP self-hosts have no leaf to pin and the monitor no-ops there by design.