mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 20:42:43 +00:00
Marc-André Bombeck
a4feabec81
The recurring no-briefing day had a chain of causes, closed end to end: Every failure path in the nightly pregenerate + the comprehensive generator now annotates a queryable cause (several classes — features-error, payload-too-large, aborted, generator-failed — previously bumped a counter with zero trace). A failed nightly warm enqueues one bounded retry ~45 minutes later (singleton per user, existing budgets apply), so a transient provider hiccup heals before morning. A cached payload that carries NO briefing no longer satisfies the 24h POST short-circuit nor the unchanged-content hash gate — a once-stripped briefing could previously block its own regeneration indefinitely. The card surfaces the omission reason persistently (grounding strip, generation failure, not attempted) with a forcing retry. Documentation corrected against code truth: Withings credentials are per-user (the documented env path never existed; the dead example entries and manifest rows are gone, and WITHINGS_WEBHOOK_SECRET is now actually whitelisted in compose — stock deploys could never deliver it); the classic Fitbit portal no longer accepts new registrations; the backup-restore example no longer mints a fresh encryption key and uses pnpm dlx tsx; the TLS guide describes the real client model (two CA-level SPKI pins, any-match, leaf renewals need no build); scaling/admin references fixed; the operator-shared subscription provider is documented.
145 lines
7.2 KiB
Markdown
145 lines
7.2 KiB
Markdown
# TLS SPKI pinning — coupling, alarm, and re-pin runbook
|
|
|
|
The native client SPKI-pins the server's TLS chain at the **CA level**. It
|
|
ships a small pin set — currently the issuing intermediate plus the root —
|
|
and accepts a connection only when the served chain passes baseline X.509
|
|
evaluation AND at least one certificate anywhere in the chain matches a pin
|
|
(any-match). The leaf is deliberately **not** pinned, so routine leaf
|
|
renewals are a client no-op. The outage risk is a **chain change**: the
|
|
served chain stops terminating in the pinned CA keys — the server moves to a
|
|
different CA, or the CA rotates its intermediate / root keys. This page
|
|
explains the coupling, the server-side alarm that watches the served leaf,
|
|
and the operator / release-owner steps when it fires.
|
|
|
|
## The coupling
|
|
|
|
A SPKI pin is `base64(sha256(DER subjectPublicKeyInfo))` — a hash of a
|
|
certificate's public key, not of the whole certificate. The native client
|
|
carries a set of these pins (build-time config, ≥ 2 well-formed pins
|
|
enforced for release builds) and rejects any TLS connection whose validated
|
|
chain contains no pinned key. Hosts outside the client's pinned-host list
|
|
fall back to plain system-trust validation, which is the documented model
|
|
for self-hosted instances that ship their own chain.
|
|
|
|
The app host's certificate auto-renews roughly every 90 days and each
|
|
renewal mints a new leaf keypair. Because the client pins the CA keys rather
|
|
than the leaf, those renewals pass without a client build. What DOES break
|
|
every pinned client is a chain that no longer terminates in a pinned key —
|
|
and there is no server-side signal that clients are pinning the served chain
|
|
at all. The alarm below exists to catch exactly that early: it watches the
|
|
served **leaf**, the most frequently rotating element, as the canary for any
|
|
chain movement.
|
|
|
|
## The alarm
|
|
|
|
A scheduled job (`src/lib/jobs/tls-pin-monitor.ts`) probes the served leaf
|
|
every 6 hours and fails loudly when it changes:
|
|
|
|
- It derives the public host from `APP_URL` (falling back to
|
|
`NEXT_PUBLIC_APP_URL`) and opens a raw TLS socket to it — observing exactly
|
|
the certificate a pinned client would, regardless of how the reverse proxy
|
|
terminates TLS.
|
|
- It reads the served leaf, computes its SPKI pin
|
|
(`base64(sha256(DER subjectPublicKeyInfo))`), and compares it against the
|
|
operator's known-good set in `TLS_LEAF_SPKI_PINS`.
|
|
- On a served pin that is **not** in the known set it emits a
|
|
`tls.pin.leaf_changed` wide-event annotation (old set + served pin), writes
|
|
a `system.tls.pin_changed` audit-log row, and fans out a high-priority
|
|
`SYSTEM_ALERT` to every admin user through the notification dispatcher
|
|
(Telegram / ntfy / Web Push / APNs, whichever the operator has wired).
|
|
|
|
A transient TLS or network failure during the probe is annotated
|
|
(`tls.pin.probe_failed`) and never alarms — the alarm fires only on a
|
|
successful probe returning a pin genuinely outside the known set.
|
|
|
|
### Baseline source: an env var, not auto-learning
|
|
|
|
`TLS_LEAF_SPKI_PINS` (comma-separated) is the known-good set of **served
|
|
leaf** pins. It is the operator's explicit record of what the server is
|
|
expected to serve — distinct from the client's CA-level pin set, which lives
|
|
in the client build config. The baseline is an env var on purpose, **not** a
|
|
persisted "last-seen" row:
|
|
|
|
- An explicit baseline forces the operator to acknowledge every leaf
|
|
rotation and check the new chain against the client's pinned CA keys
|
|
before the alarm goes quiet.
|
|
- A persisted last-seen baseline would silently adopt the first rotated pin
|
|
and suppress the very alarm the pinned client needs.
|
|
- When the env var is unset, the monitor fails **loud, not open**: it logs
|
|
the served pin (so you can seed the baseline) and warns "not configured",
|
|
but never auto-adopts what it observed.
|
|
|
|
## Setting the baseline
|
|
|
|
Extract the current leaf SPKI pin from the served certificate:
|
|
|
|
```sh
|
|
openssl s_client -connect "$APP_HOST:443" -servername "$APP_HOST" </dev/null 2>/dev/null \
|
|
| openssl x509 -noout -pubkey \
|
|
| openssl pkey -pubin -outform der 2>/dev/null \
|
|
| openssl dgst -sha256 -binary \
|
|
| openssl base64
|
|
```
|
|
|
|
`$APP_HOST` is the host in your configured `APP_URL` (the host the client
|
|
connects to). The output is the `base64(sha256(DER subjectPublicKeyInfo))`
|
|
pin. Set it in the deployment env:
|
|
|
|
```
|
|
TLS_LEAF_SPKI_PINS="<current-leaf-pin>"
|
|
```
|
|
|
|
The variable is on the compose `environment:` whitelist and the
|
|
`scripts/env-manifest.json` optional groups, so `pnpm check-env` will report
|
|
whether it is set. It is optional: without it the monitor still runs and logs
|
|
the served pin, but cannot alarm on a change.
|
|
|
|
## When the alarm fires
|
|
|
|
When you see a `tls.pin.leaf_changed` alert / `system.tls.pin_changed` audit
|
|
row, the served leaf has rotated. First determine which of two cases you are
|
|
in:
|
|
|
|
1. **Routine renewal, same chain.** Inspect the new chain (the `openssl`
|
|
pipeline above, or the alert payload). If the issuing intermediate and
|
|
root are unchanged, the client's CA-level pins still match and no client
|
|
is at risk. Update `TLS_LEAF_SPKI_PINS` to the new served-leaf pin and
|
|
re-deploy so the alarm goes quiet. No client build.
|
|
2. **Chain change.** The new leaf chains through a different intermediate /
|
|
root (CA switch, or the CA rotated its keys). Every pinned client will
|
|
stop connecting once the old chain is no longer served — run the re-pin
|
|
procedure below inside the old chain's remaining validity, with margin
|
|
for app review and client roll-out (the alert body and audit row carry
|
|
the new leaf's `validTo`).
|
|
|
|
### Re-pin procedure (chain change)
|
|
|
|
1. **Extract the new chain's CA pins** (intermediate + root) with the
|
|
`openssl` pipeline / the client repo's SPKI-extraction script.
|
|
2. **Dual-pin.** Add the new CA pins to the client's pin set **alongside**
|
|
the old ones (do not replace them yet). Clients on the old build keep
|
|
working as long as the old chain is still served; the dual-pinned build
|
|
works against both chains. Update `TLS_LEAF_SPKI_PINS` to the currently
|
|
served leaf pin(s) so the server-side alarm reflects reality.
|
|
3. **Ship the client build** carrying the dual-pin set to TestFlight (and
|
|
onward to release). Give it long enough to roll out to the installed
|
|
base before the old chain disappears.
|
|
4. **Retire the old pins** once the old chain is no longer served and the
|
|
dual-pinned client build has reached the installed base.
|
|
|
|
CA-level pins move rarely — intermediates and roots carry multi-year
|
|
lifetimes and rotations are announced well ahead. Track the pinned
|
|
certificates' expiry dates and pre-stage the successor pins before the
|
|
cutover, so a chain change never catches a single-chain client.
|
|
|
|
## Environment assumption
|
|
|
|
The monitor speaks TLS directly to the public host from `APP_URL` /
|
|
`NEXT_PUBLIC_APP_URL`, which is correct regardless of the reverse-proxy
|
|
topology — it observes the same leaf a real client does. The one assumption
|
|
is that the host the client connects to is the host in that env var. If the
|
|
client is pinned against a different hostname (e.g. a vanity domain that
|
|
fronts the configured `APP_URL`), point the monitor at that host by setting
|
|
`APP_URL` to it, or extend the job to probe the additional host. Plain-HTTP
|
|
self-hosts have no leaf to pin and the monitor no-ops there by design.
|