chore(release): v1.28.30 — the daily briefing stops silently skipping days

The recurring no-briefing day had a chain of causes, closed end to end:

Every failure path in the nightly pregenerate + the comprehensive generator
now annotates a queryable cause (several classes — features-error,
payload-too-large, aborted, generator-failed — previously bumped a counter
with zero trace). A failed nightly warm enqueues one bounded retry ~45
minutes later (singleton per user, existing budgets apply), so a transient
provider hiccup heals before morning. A cached payload that carries NO
briefing no longer satisfies the 24h POST short-circuit nor the
unchanged-content hash gate — a once-stripped briefing could previously
block its own regeneration indefinitely. The card surfaces the omission
reason persistently (grounding strip, generation failure, not attempted)
with a forcing retry.

Documentation corrected against code truth: Withings credentials are
per-user (the documented env path never existed; the dead example entries
and manifest rows are gone, and WITHINGS_WEBHOOK_SECRET is now actually
whitelisted in compose — stock deploys could never deliver it); the classic
Fitbit portal no longer accepts new registrations; the backup-restore
example no longer mints a fresh encryption key and uses pnpm dlx tsx; the
TLS guide describes the real client model (two CA-level SPKI pins,
any-match, leaf renewals need no build); scaling/admin references fixed;
the operator-shared subscription provider is documented.
This commit is contained in:
Marc-André Bombeck committed 2026-07-12 11:08:59 +02:00
1 parent 14a517387e
commit a4feabec81
26 files changed
+1190 -148

No files matched your search

+14 -8
View File
@@ -68,20 +68,26 @@ The credentials are never returned.
## Restore
Pick a key (e.g. `2026-05-08/user-clx123.json.enc`) from the bucket
and run:
and run the restore script with the same backup credentials and
encryption key the backup was written under — a freshly generated
`BACKUP_ENCRYPTION_KEY` cannot decrypt any existing object:
```bash
BACKUP_S3_ENDPOINT=https://...r2.cloudflarestorage.com \
BACKUP_S3_BUCKET=healthlog-backups \
BACKUP_S3_ACCESS_KEY=... \
BACKUP_S3_SECRET_KEY=... \
BACKUP_S3_REGION=auto \
BACKUP_ENCRYPTION_KEY=$(openssl rand -hex 32) \
pnpm tsx scripts/restore-backup.ts \
BACKUP_S3_ENDPOINT=https://...r2.cloudflarestorage.com \
BACKUP_S3_BUCKET=healthlog-backups \
BACKUP_S3_ACCESS_KEY=... \
BACKUP_S3_SECRET_KEY=... \
BACKUP_S3_REGION=auto \
BACKUP_ENCRYPTION_KEY=<the key the backup was written under> \
pnpm dlx tsx scripts/restore-backup.ts \
2026-05-08/user-clx123.json.enc \
/tmp/restored.json
```
The production standalone image strips `tsx`, so a bare
`pnpm tsx scripts/...` fails inside the container — always invoke
one-shot scripts via `pnpm dlx tsx`.
The script downloads the object, decrypts it, and writes the JSON dump
to disk. Importing the JSON back into a HealthLog instance is left to
the operator (use `prisma db seed` or a custom script).
+2 -2
View File
@@ -44,7 +44,7 @@ under `v1` (the synthetic id assigned to the existing `ENCRYPTION_KEY`).
3. **Run the rotation script** to re-encrypt every existing encrypted column
under the new active key:
```
pnpm tsx scripts/rotate-encryption-key.ts v2
pnpm dlx tsx scripts/rotate-encryption-key.ts v2
```
The script is idempotent — running it again is a no-op for rows already
prefixed with `v2.`. It rotates User, WithingsConnection, AppSettings,
@@ -69,7 +69,7 @@ script reports zero `v2.`-prefixed rows remaining.
```
ENCRYPTION_KEYS='{"v2":"<old>","v3":"<new>"}'
ENCRYPTION_ACTIVE_KEY_ID="v3"
pnpm tsx scripts/rotate-encryption-key.ts v3
pnpm dlx tsx scripts/rotate-encryption-key.ts v3
ENCRYPTION_KEYS='{"v3":"<new>"}'
```
+65 -48
View File
@@ -1,26 +1,34 @@
# TLS leaf SPKI pin — coupling, alarm, and re-pin runbook
# TLS SPKI pinning — coupling, alarm, and re-pin runbook
The native client SPKI-pins the server's TLS **leaf** certificate. This is a
deliberate hardening choice on the client side, but it couples the client to
the exact leaf keypair the server serves: when that leaf rotates, a client
that was only ever shipped the old pin refuses to connect. This page explains
the coupling, the server-side alarm that watches for a rotation, and the
operator / release-owner steps when it fires.
The native client SPKI-pins the server's TLS chain at the **CA level**. It
ships a small pin set — currently the issuing intermediate plus the root —
and accepts a connection only when the served chain passes baseline X.509
evaluation AND at least one certificate anywhere in the chain matches a pin
(any-match). The leaf is deliberately **not** pinned, so routine leaf
renewals are a client no-op. The outage risk is a **chain change**: the
served chain stops terminating in the pinned CA keys — the server moves to a
different CA, or the CA rotates its intermediate / root keys. This page
explains the coupling, the server-side alarm that watches the served leaf,
and the operator / release-owner steps when it fires.
## The coupling
A SPKI pin is `base64(sha256(DER subjectPublicKeyInfo))` — a hash of the
A SPKI pin is `base64(sha256(DER subjectPublicKeyInfo))` — a hash of a
certificate's public key, not of the whole certificate. The native client
carries a set of these pins and rejects any TLS connection whose leaf public
key is not in the set.
carries a set of these pins (build-time config, ≥ 2 well-formed pins
enforced for release builds) and rejects any TLS connection whose validated
chain contains no pinned key. Hosts outside the client's pinned-host list
fall back to plain system-trust validation, which is the documented model
for self-hosted instances that ship their own chain.
The app host's certificate is issued by Google Trust Services (GTS) and
auto-renews roughly every 90 days. Each renewal mints a **new leaf keypair**,
so the SPKI pin changes on every renewal. Intermediate and root rolls have
the same effect on the leaf chain. Because the client pins the leaf, every
such change is a hard outage for any client that hasn't been shipped the new
pin — and there is otherwise no server-side signal that a client is pinning
the served leaf at all.
The app host's certificate auto-renews roughly every 90 days and each
renewal mints a new leaf keypair. Because the client pins the CA keys rather
than the leaf, those renewals pass without a client build. What DOES break
every pinned client is a chain that no longer terminates in a pinned key —
and there is no server-side signal that clients are pinning the served chain
at all. The alarm below exists to catch exactly that early: it watches the
served **leaf**, the most frequently rotating element, as the canary for any
chain movement.
## The alarm
@@ -46,13 +54,15 @@ successful probe returning a pin genuinely outside the known set.
### Baseline source: an env var, not auto-learning
`TLS_LEAF_SPKI_PINS` (comma-separated) is the known-good set. The baseline is
an env var on purpose, **not** a persisted "last-seen" row:
`TLS_LEAF_SPKI_PINS` (comma-separated) is the known-good set of **served
leaf** pins. It is the operator's explicit record of what the server is
expected to serve — distinct from the client's CA-level pin set, which lives
in the client build config. The baseline is an env var on purpose, **not** a
persisted "last-seen" row:
- The pinned client only trusts the pins it was shipped, so the operator must
derive the client's pin set from a single explicit source of truth. The env
var **is** that source — the same value goes into the client's pin set and
into `TLS_LEAF_SPKI_PINS`.
- An explicit baseline forces the operator to acknowledge every leaf
rotation and check the new chain against the client's pinned CA keys
before the alarm goes quiet.
- A persisted last-seen baseline would silently adopt the first rotated pin
and suppress the very alarm the pinned client needs.
- When the env var is unset, the monitor fails **loud, not open**: it logs
@@ -84,36 +94,43 @@ The variable is on the compose `environment:` whitelist and the
whether it is set. It is optional: without it the monitor still runs and logs
the served pin, but cannot alarm on a change.
## When the alarm fires — re-pin procedure
## When the alarm fires
When you see a `tls.pin.leaf_changed` alert / `system.tls.pin_changed` audit
row, the served leaf has rotated. The pinned client will stop connecting once
its shipped pin no longer matches a served leaf. Act inside the certificate's
remaining validity, with margin for app review and client roll-out — target
**≥ 11 days before the old pin's certificate expires** (the alert body and
audit row both carry the new leaf's `validTo`).
row, the served leaf has rotated. First determine which of two cases you are
in:
1. **Re-extract the new leaf pin** with the `openssl` pipeline above. It
should equal the `servedPin` in the alert.
2. **Dual-pin.** Add the new pin to the client's pin set **alongside** the old
one (do not replace it yet), and update `TLS_LEAF_SPKI_PINS` to the
comma-separated pair, e.g.
`TLS_LEAF_SPKI_PINS="<old-pin>,<new-pin>"`. Holding both pins means clients
on the old build keep working through the cutover and the server-side
alarm goes quiet because the served pin is now in the known set. Re-deploy
the server so the new env value takes effect.
1. **Routine renewal, same chain.** Inspect the new chain (the `openssl`
pipeline above, or the alert payload). If the issuing intermediate and
root are unchanged, the client's CA-level pins still match and no client
is at risk. Update `TLS_LEAF_SPKI_PINS` to the new served-leaf pin and
re-deploy so the alarm goes quiet. No client build.
2. **Chain change.** The new leaf chains through a different intermediate /
root (CA switch, or the CA rotated its keys). Every pinned client will
stop connecting once the old chain is no longer served — run the re-pin
procedure below inside the old chain's remaining validity, with margin
for app review and client roll-out (the alert body and audit row carry
the new leaf's `validTo`).
### Re-pin procedure (chain change)
1. **Extract the new chain's CA pins** (intermediate + root) with the
`openssl` pipeline / the client repo's SPKI-extraction script.
2. **Dual-pin.** Add the new CA pins to the client's pin set **alongside**
the old ones (do not replace them yet). Clients on the old build keep
working as long as the old chain is still served; the dual-pinned build
works against both chains. Update `TLS_LEAF_SPKI_PINS` to the currently
served leaf pin(s) so the server-side alarm reflects reality.
3. **Ship the client build** carrying the dual-pin set to TestFlight (and
onward to release). Give it long enough to roll out to the installed base
before the next renewal.
4. **Retire the old pin** once the old leaf is no longer served and the
dual-pinned client build has reached the installed base: drop the old pin
from both the client pin set and `TLS_LEAF_SPKI_PINS`, leaving only the
current leaf pin.
onward to release). Give it long enough to roll out to the installed
base before the old chain disappears.
4. **Retire the old pins** once the old chain is no longer served and the
dual-pinned client build has reached the installed base.
Treat the dual-pin window as standing practice: always ship the **next** pin
before the current leaf rotates, so a renewal never catches a single-pinned
client. If GTS renews on a fixed cadence, pre-extracting and dual-pinning the
upcoming leaf ahead of the renewal turns every rotation into a no-op.
CA-level pins move rarely — intermediates and roots carry multi-year
lifetimes and rotations are announced well ahead. Track the pinned
certificates' expiry dates and pre-stage the successor pins before the
cutover, so a chain change never catches a single-chain client.
## Environment assumption