mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-11 13:52:20 +00:00
Marc-André Bombeck
a4feabec81
The recurring no-briefing day had a chain of causes, closed end to end: Every failure path in the nightly pregenerate + the comprehensive generator now annotates a queryable cause (several classes — features-error, payload-too-large, aborted, generator-failed — previously bumped a counter with zero trace). A failed nightly warm enqueues one bounded retry ~45 minutes later (singleton per user, existing budgets apply), so a transient provider hiccup heals before morning. A cached payload that carries NO briefing no longer satisfies the 24h POST short-circuit nor the unchanged-content hash gate — a once-stripped briefing could previously block its own regeneration indefinitely. The card surfaces the omission reason persistently (grounding strip, generation failure, not attempted) with a forcing retry. Documentation corrected against code truth: Withings credentials are per-user (the documented env path never existed; the dead example entries and manifest rows are gone, and WITHINGS_WEBHOOK_SECRET is now actually whitelisted in compose — stock deploys could never deliver it); the classic Fitbit portal no longer accepts new registrations; the backup-restore example no longer mints a fresh encryption key and uses pnpm dlx tsx; the TLS guide describes the real client model (two CA-level SPKI pins, any-match, leaf renewals need no build); scaling/admin references fixed; the operator-shared subscription provider is documented.
126 lines
5.7 KiB
Markdown
126 lines
5.7 KiB
Markdown
# Scaling: web/worker container split
|
||
|
||
The default `docker compose up` launches a single container that runs
|
||
both the Next.js HTTP server AND the pg-boss worker. That works fine
|
||
for a personal deployment but is hard to scale horizontally because
|
||
every replica also runs the full reminder/insight cron schedule.
|
||
|
||
v1.4 introduces an environment-variable gate so the same image can be
|
||
deployed in three modes:
|
||
|
||
| `HEALTHLOG_PROCESS_TYPE` | Web | Worker | Use when |
|
||
| ------------------------ | --- | ------ | ------------------------- |
|
||
| `all` (default) | ✅ | ✅ | single-host self-hosting |
|
||
| `web` | ✅ | ❌ | scaling HTTP horizontally |
|
||
| `worker` | ❌ | ✅ | dedicated job worker |
|
||
|
||
## How to split
|
||
|
||
1. Edit `docker-compose.yml`: set `HEALTHLOG_PROCESS_TYPE=web` on the
|
||
`app` service and uncomment the `app-worker` service block at the
|
||
bottom (it ships with `HEALTHLOG_PROCESS_TYPE=worker` baked in).
|
||
2. Make sure both containers see the same `DATABASE_URL`,
|
||
`ENCRYPTION_KEY` (or `ENCRYPTION_KEYS`), and `API_TOKEN_HMAC_KEY`.
|
||
3. `docker compose up -d --build`.
|
||
|
||
Both containers connect to the same Postgres. pg-boss claims jobs
|
||
atomically via row-level locking (`FOR UPDATE SKIP LOCKED`), so
|
||
running multiple worker replicas is **safe** — no job runs twice —
|
||
but it doubles the DB load and muddies telemetry without adding
|
||
throughput a personal instance needs (`src/lib/process-type.ts`).
|
||
Run **one** worker container and scale the `web` containers instead.
|
||
|
||
## Healthchecks
|
||
|
||
- `app` (web) healthcheck: `wget /api/version` every 30s.
|
||
- `app-worker` does not expose a port; its liveness is reported into
|
||
the `worker_status` table on every reminder pass and surfaced through
|
||
`/api/admin/status`.
|
||
|
||
The web container does NOT depend on the worker, and vice versa, so
|
||
neither container's startup can deadlock the other. The shared
|
||
dependency is Postgres; both wait on `db: condition: service_healthy`.
|
||
|
||
## Caveats
|
||
|
||
- Background tasks that touch user-scoped Wide Events (e.g. reminder
|
||
notifications) emit telemetry from the worker; configure `LOKI_*`
|
||
env vars on the worker if you want them.
|
||
- The off-host backup job runs in the worker only. The admin
|
||
`POST /api/admin/backup/test` endpoint runs in the web container and
|
||
exercises the same S3 credentials — set `BACKUP_*` env vars on BOTH.
|
||
- The startup gate (`assertSubsystemEnabled`) refuses to boot the
|
||
reminder worker when `HEALTHLOG_PROCESS_TYPE=web`, so an accidental
|
||
cross-mode invocation aborts immediately instead of silently doubling
|
||
cron load.
|
||
|
||
## Postgres connection-pool sizing
|
||
|
||
Every container that boots the HealthLog image opens its own
|
||
`pg.Pool` against the configured `DATABASE_URL`. The pool ceiling is
|
||
**20 connections per container by default** (raised from the library
|
||
default of 10 in v1.4.40 to keep the dashboard's analytics fan-out
|
||
from starving every other query during a cold mount).
|
||
|
||
Plan total Postgres slots as **container_count × `DATABASE_POOL_MAX`**.
|
||
A stock Postgres 16 container ships with `max_connections = 100`, so
|
||
the safe envelope is:
|
||
|
||
| Web replicas | Worker replicas | Total containers | Pool slots used | Headroom under 100 |
|
||
| ------------ | --------------- | ---------------- | --------------- | ------------------ |
|
||
| 1 | 0 | 1 | 20 | 80 |
|
||
| 1 | 1 | 2 | 40 | 60 |
|
||
| 2 | 1 | 3 | 60 | 40 |
|
||
| 3 | 1 | 4 | 80 | 20 |
|
||
| 4 | 1 | 5 | 100 | 0 (do not exceed) |
|
||
|
||
Once the table tips into the "0 headroom" row, every other client of
|
||
the same Postgres — `psql` sessions, ad-hoc backups, the Prisma CLI
|
||
during a migration deploy — will get `FATAL: sorry, too many clients
|
||
already`. Either raise `max_connections` on the Postgres side, lower
|
||
`DATABASE_POOL_MAX`, or front the database with PgBouncer in
|
||
transaction-pooling mode.
|
||
|
||
### Overriding the per-container pool ceiling
|
||
|
||
Set `DATABASE_POOL_MAX` to a positive integer on every container that
|
||
needs a non-default ceiling — both web and worker pick up the same
|
||
env var:
|
||
|
||
```yaml
|
||
# docker-compose.yml — example for a 6-container deployment
|
||
services:
|
||
app:
|
||
image: ghcr.io/mbombeck/healthlog:latest
|
||
environment:
|
||
DATABASE_POOL_MAX: "12" # 6 containers × 12 = 72 < 100
|
||
HEALTHLOG_PROCESS_TYPE: web
|
||
```
|
||
|
||
Rules of thumb:
|
||
|
||
- Each web replica's hottest path (`/api/analytics` fan-out, capped at
|
||
`p-limit(4)`) consumes 4 concurrent slots. Keep
|
||
`DATABASE_POOL_MAX ≥ 8` so a single power-user request never
|
||
exhausts the pool of one container.
|
||
- Worker replicas mostly use 1 connection per active pg-boss job;
|
||
`DATABASE_POOL_MAX = 8` is plenty unless you raised the pg-boss
|
||
`teamConcurrency`.
|
||
- If you raise Postgres `max_connections` past 100, prefer raising
|
||
`DATABASE_POOL_MAX` over adding more containers — fewer, fatter
|
||
Node processes amortise the V8 footprint better than many thin
|
||
ones.
|
||
|
||
### Why the default is 20
|
||
|
||
The v1.4.39 empirical cold-mount trace showed thick `/api/analytics`
|
||
holding ≥ 8 of the 10 default pool slots for 6.5 s on a 347 k-row
|
||
tenant, starving every other dashboard query for the duration. The
|
||
20-slot default — paired with the `p-limit(4)` cap on the analytics
|
||
fan-out — keeps 16 slots free for the rest of the dashboard while
|
||
still sitting well under Postgres's 100-slot stock ceiling.
|
||
|
||
The implementation lives in `src/lib/db.ts → getPoolMax()`; the
|
||
20-slot default was chosen after measuring on a 4-CPU production
|
||
container — see the inline rationale in `getPoolMax()`.
|