HealthLog/docs/self-hosting/scaling.md

Scaling: web/worker container split

The default docker compose up launches a single container that runs both the Next.js HTTP server AND the pg-boss worker. That works fine for a personal deployment but is hard to scale horizontally because every replica also runs the full reminder/insight cron schedule.

v1.4 introduces an environment-variable gate so the same image can be deployed in three modes:

HEALTHLOG_PROCESS_TYPE	Web	Worker	Use when
all (default)	✅	✅	single-host self-hosting
web	✅	❌	scaling HTTP horizontally
worker	❌	✅	dedicated job worker

How to split

1. Edit docker-compose.yml: set HEALTHLOG_PROCESS_TYPE=web on the app service and uncomment the app-worker service block at the bottom (it ships with HEALTHLOG_PROCESS_TYPE=worker baked in).
2. Make sure both containers see the same DATABASE_URL, ENCRYPTION_KEY (or ENCRYPTION_KEYS), and API_TOKEN_HMAC_KEY.
3. docker compose up -d --build.

Both containers connect to the same Postgres. pg-boss claims jobs atomically via row-level locking (FOR UPDATE SKIP LOCKED), so running multiple worker replicas is safe — no job runs twice — but it doubles the DB load and muddies telemetry without adding throughput a personal instance needs (src/lib/process-type.ts). Run one worker container and scale the web containers instead.

Apple Health import staging (shared volume required for a split)

The Apple Health export.zip import streams the upload to a staging file on local disk (under the container's temp dir) and hands the worker only the file path. In single-container mode (HEALTHLOG_PROCESS_TYPE=all, the default) the web handler and the worker share the same filesystem, so the handoff just works.

In a web/worker split — or any topology where the container that accepts the upload is not the one that runs the job (a web + worker split, extra all replicas, or old+new containers coexisting during a rolling deploy) — the worker cannot see the web container's staging file. The import then fails with a staging-file-missing error ("the import staging file is not visible to the worker"). Two things to know:

• A pure web container has no pg-boss handle, so an import kicked off there currently returns 503 Background worker is not running. Run imports from a container that also runs the worker until send-only boss support lands.
• If you run web and worker as separate containers, they must share the import staging directory — mount the same named volume at the temp path on both — or run imports in single-container (all) mode. Without a shared staging volume, imports in split mode cannot complete.

Apple Health import memory

The import parses export.xml as a stream: the archive is never buffered whole, records flush to the database in fixed-size batches, and what stays resident scales with the data's calendar span, not its record count. Multi-year exports with millions of records parse within the default Node.js heap.

Before parsing starts, the worker compares the archive's declared uncompressed size against the runtime heap limit. If the limit is too low to carry the export — realistically only when --max-old-space-size was pinned far below the default — the import fails immediately with a failureReason starting with insufficient_memory, naming the required and available amounts. Raise the limit on the container that runs the worker and re-upload:

# docker-compose.yml
services:
  app:
    environment:
      NODE_OPTIONS: --max-old-space-size=1024

An import interrupted by a container restart or an out-of-memory kill does not stay stuck: the worker reconciles non-terminal import jobs at boot and every 15 minutes, flipping orphaned rows to failed (interrupted_by_restart) so the UI offers a retry instead of an endless spinner.

Healthchecks

• app (web) healthcheck: wget /api/version every 30s.
• app-worker does not expose a port; its liveness is reported into the worker_status table on every reminder pass and surfaced through /api/admin/status.

The web container does NOT depend on the worker, and vice versa, so neither container's startup can deadlock the other. The shared dependency is Postgres; both wait on db: condition: service_healthy.

Caveats

• Background tasks that touch user-scoped Wide Events (e.g. reminder notifications) emit telemetry from the worker; configure LOKI_* env vars on the worker if you want them.
• The off-host backup job runs in the worker only. The admin POST /api/admin/backup/test endpoint runs in the web container and exercises the same S3 credentials — set BACKUP_* env vars on BOTH.
• The startup gate (assertSubsystemEnabled) refuses to boot the reminder worker when HEALTHLOG_PROCESS_TYPE=web, so an accidental cross-mode invocation aborts immediately instead of silently doubling cron load.

Postgres connection-pool sizing

Every container that boots the HealthLog image opens its own pg.Pool against the configured DATABASE_URL. The pool ceiling is 20 connections per container by default (raised from the library default of 10 in v1.4.40 to keep the dashboard's analytics fan-out from starving every other query during a cold mount).

Plan total Postgres slots as container_count × DATABASE_POOL_MAX. A stock Postgres 16 container ships with max_connections = 100, so the safe envelope is:

Web replicas	Worker replicas	Total containers	Pool slots used	Headroom under 100
1	0	1	20	80
1	1	2	40	60
2	1	3	60	40
3	1	4	80	20
4	1	5	100	0 (do not exceed)

Once the table tips into the "0 headroom" row, every other client of the same Postgres — psql sessions, ad-hoc backups, the Prisma CLI during a migration deploy — will get FATAL: sorry, too many clients already. Either raise max_connections on the Postgres side, lower DATABASE_POOL_MAX, or front the database with PgBouncer in transaction-pooling mode.

Overriding the per-container pool ceiling

Set DATABASE_POOL_MAX to a positive integer on every container that needs a non-default ceiling — both web and worker pick up the same env var:

# docker-compose.yml — example for a 6-container deployment
services:
  app:
    image: ghcr.io/mbombeck/healthlog:latest
    environment:
      DATABASE_POOL_MAX: "12" # 6 containers × 12 = 72 < 100
      HEALTHLOG_PROCESS_TYPE: web

Rules of thumb:

• Each web replica's hottest path (/api/analytics fan-out, capped at p-limit(4)) consumes 4 concurrent slots. Keep DATABASE_POOL_MAX ≥ 8 so a single power-user request never exhausts the pool of one container.
• Worker replicas mostly use 1 connection per active pg-boss job; DATABASE_POOL_MAX = 8 is plenty unless you raised the pg-boss teamConcurrency.
• If you raise Postgres max_connections past 100, prefer raising DATABASE_POOL_MAX over adding more containers — fewer, fatter Node processes amortise the V8 footprint better than many thin ones.

Why the default is 20

The v1.4.39 empirical cold-mount trace showed thick /api/analytics holding ≥ 8 of the 10 default pool slots for 6.5 s on a 347 k-row tenant, starving every other dashboard query for the duration. The 20-slot default — paired with the p-limit(4) cap on the analytics fan-out — keeps 16 slots free for the rest of the dashboard while still sitting well under Postgres's 100-slot stock ceiling.

The implementation lives in src/lib/db.ts → getPoolMax(); the 20-slot default was chosen after measuring on a 4-CPU production container — see the inline rationale in getPoolMax().