HealthLog/docs/self-hosting/scaling.md

Scaling: web/worker container split

The default docker compose up launches a single container that runs both the Next.js HTTP server AND the pg-boss worker. That works fine for a personal deployment but is hard to scale horizontally because every replica also runs the full reminder/insight cron schedule.

v1.4 introduces an environment-variable gate so the same image can be deployed in three modes:

HEALTHLOG_PROCESS_TYPE	Web	Worker	Use when
all (default)	✅	✅	single-host self-hosting
web	✅	❌	scaling HTTP horizontally
worker	❌	✅	dedicated job worker

How to split

1. Edit docker-compose.yml: set HEALTHLOG_PROCESS_TYPE=web on the app service and uncomment the app-worker service block at the bottom (it ships with HEALTHLOG_PROCESS_TYPE=worker baked in).
2. Make sure both containers see the same DATABASE_URL, ENCRYPTION_KEY (or ENCRYPTION_KEYS), and API_TOKEN_HMAC_KEY.
3. docker compose up -d --build.

Both containers connect to the same Postgres. pg-boss claims jobs atomically via row-level locking, so it is safe to run multiple worker replicas — they will share the load.

Healthchecks

• app (web) healthcheck: wget /api/version every 30s.
• app-worker does not expose a port; its liveness is reported into the worker_status table on every reminder pass and surfaced through /api/admin/worker-status.

The web container does NOT depend on the worker, and vice versa, so neither container's startup can deadlock the other. The shared dependency is Postgres; both wait on db: condition: service_healthy.

Caveats

• Background tasks that touch user-scoped Wide Events (e.g. reminder notifications) emit telemetry from the worker; configure LOKI_* env vars on the worker if you want them.
• The off-host backup job runs in the worker only. The admin POST /api/admin/backup/test endpoint runs in the web container and exercises the same S3 credentials — set BACKUP_* env vars on BOTH.
• The startup gate (assertSubsystemEnabled) refuses to boot the reminder worker when HEALTHLOG_PROCESS_TYPE=web, so an accidental cross-mode invocation aborts immediately instead of silently doubling cron load.

Postgres connection-pool sizing

Every container that boots the HealthLog image opens its own pg.Pool against the configured DATABASE_URL. The pool ceiling is 20 connections per container by default (raised from the library default of 10 in v1.4.40 to keep the dashboard's analytics fan-out from starving every other query during a cold mount).

Plan total Postgres slots as container_count × DATABASE_POOL_MAX. A stock Postgres 16 container ships with max_connections = 100, so the safe envelope is:

Web replicas	Worker replicas	Total containers	Pool slots used	Headroom under 100
1	0	1	20	80
1	1	2	40	60
2	1	3	60	40
3	1	4	80	20
4	1	5	100	0 (do not exceed)

Once the table tips into the "0 headroom" row, every other client of the same Postgres — psql sessions, ad-hoc backups, the Prisma CLI during a migration deploy — will get FATAL: sorry, too many clients already. Either raise max_connections on the Postgres side, lower DATABASE_POOL_MAX, or front the database with PgBouncer in transaction-pooling mode.

Overriding the per-container pool ceiling

Set DATABASE_POOL_MAX to a positive integer on every container that needs a non-default ceiling — both web and worker pick up the same env var:

# docker-compose.yml — example for a 6-container deployment
services:
  app:
    image: ghcr.io/mbombeck/healthlog:latest
    environment:
      DATABASE_POOL_MAX: "12" # 6 containers × 12 = 72 < 100
      HEALTHLOG_PROCESS_TYPE: web

Rules of thumb:

• Each web replica's hottest path (/api/analytics fan-out, capped at p-limit(4)) consumes 4 concurrent slots. Keep DATABASE_POOL_MAX ≥ 8 so a single power-user request never exhausts the pool of one container.
• Worker replicas mostly use 1 connection per active pg-boss job; DATABASE_POOL_MAX = 8 is plenty unless you raised the pg-boss teamConcurrency.
• If you raise Postgres max_connections past 100, prefer raising DATABASE_POOL_MAX over adding more containers — fewer, fatter Node processes amortise the V8 footprint better than many thin ones.

Why the default is 20

The v1.4.39 empirical cold-mount trace showed thick /api/analytics holding ≥ 8 of the 10 default pool slots for 6.5 s on a 347 k-row tenant, starving every other dashboard query for the duration. The 20-slot default — paired with the p-limit(4) cap on the analytics fan-out — keeps 16 slots free for the rest of the dashboard while still sitting well under Postgres's 100-slot stock ceiling.

The implementation lives in src/lib/db.ts → getPoolMax(); the 20-slot default was chosen after measuring on a 4-CPU production container — see the inline rationale in getPoolMax().