HealthLog/docs/self-hosting/scaling.md

Scaling: web/worker container split

The default docker compose up launches a single container that runs both the Next.js HTTP server AND the pg-boss worker. That works fine for a personal deployment but is hard to scale horizontally because every replica also runs the full reminder/insight cron schedule.

v1.4 introduces an environment-variable gate so the same image can be deployed in three modes:

HEALTHLOG_PROCESS_TYPE	Web	Worker	Use when
all (default)	✅	✅	single-host self-hosting
web	✅	❌	scaling HTTP horizontally
worker	❌	✅	dedicated job worker

How to split

1. Edit docker-compose.yml: set HEALTHLOG_PROCESS_TYPE=web on the app service and uncomment the app-worker service block at the bottom (it ships with HEALTHLOG_PROCESS_TYPE=worker baked in).
2. Make sure both containers see the same DATABASE_URL, ENCRYPTION_KEY (or ENCRYPTION_KEYS), and API_TOKEN_HMAC_KEY.
3. docker compose up -d --build.

Both containers connect to the same Postgres. pg-boss claims jobs atomically via row-level locking (FOR UPDATE SKIP LOCKED), so running multiple worker replicas is safe — no job runs twice — but it doubles the DB load and muddies telemetry without adding throughput a personal instance needs (src/lib/process-type.ts). Run one worker container and scale the web containers instead.

Apple Health import staging (shared volume required for a split)

The Apple Health export.zip import streams the upload to a staging file on local disk (under the container's temp dir) and hands the worker only the file path. In single-container mode (HEALTHLOG_PROCESS_TYPE=all, the default) the web handler and the worker share the same filesystem, so the handoff just works.

In a web/worker split — or any topology where the container that accepts the upload is not the one that runs the job (a web + worker split, extra all replicas, or old+new containers coexisting during a rolling deploy) — the worker cannot see the web container's staging file. The import then fails with a staging-file-missing error ("the import staging file is not visible to the worker"). Two things to know:

• A pure web container has no pg-boss handle, so an import kicked off there currently returns 503 Background worker is not running. Run imports from a container that also runs the worker until send-only boss support lands.
• If you run web and worker as separate containers, they must share the import staging directory — mount the same named volume at the temp path on both — or run imports in single-container (all) mode. Without a shared staging volume, imports in split mode cannot complete.

Healthchecks

• app (web) healthcheck: wget /api/version every 30s.
• app-worker does not expose a port; its liveness is reported into the worker_status table on every reminder pass and surfaced through /api/admin/status.

The web container does NOT depend on the worker, and vice versa, so neither container's startup can deadlock the other. The shared dependency is Postgres; both wait on db: condition: service_healthy.

Caveats

• Background tasks that touch user-scoped Wide Events (e.g. reminder notifications) emit telemetry from the worker; configure LOKI_* env vars on the worker if you want them.
• The off-host backup job runs in the worker only. The admin POST /api/admin/backup/test endpoint runs in the web container and exercises the same S3 credentials — set BACKUP_* env vars on BOTH.
• The startup gate (assertSubsystemEnabled) refuses to boot the reminder worker when HEALTHLOG_PROCESS_TYPE=web, so an accidental cross-mode invocation aborts immediately instead of silently doubling cron load.

Postgres connection-pool sizing

Every container that boots the HealthLog image opens its own pg.Pool against the configured DATABASE_URL. The pool ceiling is 20 connections per container by default (raised from the library default of 10 in v1.4.40 to keep the dashboard's analytics fan-out from starving every other query during a cold mount).

Plan total Postgres slots as container_count × DATABASE_POOL_MAX. A stock Postgres 16 container ships with max_connections = 100, so the safe envelope is:

Web replicas	Worker replicas	Total containers	Pool slots used	Headroom under 100
1	0	1	20	80
1	1	2	40	60
2	1	3	60	40
3	1	4	80	20
4	1	5	100	0 (do not exceed)

Once the table tips into the "0 headroom" row, every other client of the same Postgres — psql sessions, ad-hoc backups, the Prisma CLI during a migration deploy — will get FATAL: sorry, too many clients already. Either raise max_connections on the Postgres side, lower DATABASE_POOL_MAX, or front the database with PgBouncer in transaction-pooling mode.

Overriding the per-container pool ceiling

Set DATABASE_POOL_MAX to a positive integer on every container that needs a non-default ceiling — both web and worker pick up the same env var:

# docker-compose.yml — example for a 6-container deployment
services:
  app:
    image: ghcr.io/mbombeck/healthlog:latest
    environment:
      DATABASE_POOL_MAX: "12" # 6 containers × 12 = 72 < 100
      HEALTHLOG_PROCESS_TYPE: web

Rules of thumb:

• Each web replica's hottest path (/api/analytics fan-out, capped at p-limit(4)) consumes 4 concurrent slots. Keep DATABASE_POOL_MAX ≥ 8 so a single power-user request never exhausts the pool of one container.
• Worker replicas mostly use 1 connection per active pg-boss job; DATABASE_POOL_MAX = 8 is plenty unless you raised the pg-boss teamConcurrency.
• If you raise Postgres max_connections past 100, prefer raising DATABASE_POOL_MAX over adding more containers — fewer, fatter Node processes amortise the V8 footprint better than many thin ones.

Why the default is 20

The v1.4.39 empirical cold-mount trace showed thick /api/analytics holding ≥ 8 of the 10 default pool slots for 6.5 s on a 347 k-row tenant, starving every other dashboard query for the duration. The 20-slot default — paired with the p-limit(4) cap on the analytics fan-out — keeps 16 slots free for the rest of the dashboard while still sitting well under Postgres's 100-slot stock ceiling.

The implementation lives in src/lib/db.ts → getPoolMax(); the 20-slot default was chosen after measuring on a 4-CPU production container — see the inline rationale in getPoolMax().