HealthLog/docs/self-hosting/scaling.md

Scaling: web/worker container split

The default docker compose up launches a single container that runs both the Next.js HTTP server AND the pg-boss worker. That works fine for a personal deployment but is hard to scale horizontally because every replica also runs the full reminder/insight cron schedule.

v1.4 introduces an environment-variable gate so the same image can be deployed in three modes:

HEALTHLOG_PROCESS_TYPE	Web	Worker	Use when
all (default)	✅	✅	single-host self-hosting
web	✅	❌	scaling HTTP horizontally
worker	❌	✅	dedicated job worker

How to split

1. Edit docker-compose.yml: set HEALTHLOG_PROCESS_TYPE=web on the app service and uncomment the app-worker service block at the bottom (it ships with HEALTHLOG_PROCESS_TYPE=worker baked in).
2. Make sure both containers see the same DATABASE_URL, ENCRYPTION_KEY (or ENCRYPTION_KEYS), and API_TOKEN_HMAC_KEY.
3. docker compose up -d --build.

Both containers connect to the same Postgres. pg-boss claims jobs atomically via row-level locking (FOR UPDATE SKIP LOCKED), so running multiple worker replicas is safe — no job runs twice — but it doubles the DB load and muddies telemetry without adding throughput a personal instance needs (src/lib/process-type.ts). Run one worker container and scale the web containers instead.

Apple Health import staging (shared volume required for a split)

The Apple Health export.zip import streams the upload to a staging file on local disk (under the container's temp dir) and hands the worker only the file path. In single-container mode (HEALTHLOG_PROCESS_TYPE=all, the default) the web handler and the worker share the same filesystem, so the handoff just works.

In a web/worker split — or any topology where the container that accepts the upload is not the one that runs the job (a web + worker split, extra all replicas, or old+new containers coexisting during a rolling deploy) — the worker cannot see the web container's staging file. The import then fails with a staging-file-missing error ("the import staging file is not visible to the worker"). Two things to know:

• A pure web container opens a send-only pg-boss connection at boot (src/instrumentation.ts, startGlobalBossProducer), so an import kicked off there is accepted and queued for the worker like any other job. It answers 503 Background worker is not running only when that producer connection could not be opened (the boot log then shows Failed to start pg-boss producer).
• If you run web and worker as separate containers, they must share the import staging directory — mount the same named volume at the temp path on both — or run imports in single-container (all) mode. Without a shared staging volume, imports in split mode cannot complete.

Apple Health import memory

The import parses export.xml as a stream: the archive is never buffered whole, records flush to the database in fixed-size batches, and what stays resident scales with the data's calendar span, not its record count. Multi-year exports with millions of records parse within the default Node.js heap.

Before parsing starts, the worker compares the archive's declared uncompressed size against the runtime heap limit. If the limit is too low to carry the export — realistically only when --max-old-space-size was pinned far below the default — the import fails immediately with a failureReason starting with insufficient_memory, naming the required and available amounts. Raise the limit on the container that runs the worker and re-upload:

# docker-compose.yml
services:
  app:
    environment:
      NODE_OPTIONS: --max-old-space-size=1024 --report-on-fatalerror --report-filename=stderr --report-exclude-env --report-exclude-network

Your NODE_OPTIONS replaces the image's, so carry the four --report-* flags over: they make an out-of-memory exit leave a diagnostic report in the container log (see docs/ops/heap-diagnostics.md).

An import interrupted by a container restart or an out-of-memory kill does not stay stuck: the worker reconciles non-terminal import jobs at boot and every 15 minutes, flipping orphaned rows to failed (interrupted_by_restart) so the UI offers a retry instead of an endless spinner.

Document and lab-scan reads run on the worker (v1.40)

Since v1.40, every AI read of a document or a lab report runs as a background job on the worker, not inside the HTTP request:

• reading a document for search (the content index),
• a document summary, stored or shown once,
• filing suggestions (title, type, date),
• extracting facts from a stored document for review,
• a lab scan (photo or PDF of a lab report).

The request only queues the read and answers 202; the client then polls the run until it ends. That is what keeps a slow model from being cut off by a proxy timeout in front of the server. When the enqueue itself fails (no pg-boss connection at all), the request answers 503 straight away, and nothing is charged.

"Waiting for the background worker" is the line the app shows when a read has sat in the queue for more than 30 seconds without a worker picking it up. On a single all container that is rare and short (the worker is busy with another job). If it does not go away, no worker process is consuming the queue: the worker container is stopped, crashing on boot, or pointed at another database. A run that no worker picks up within 15 minutes fails as aiRuns.workerUnavailable and its AI budget reservation is handed back. That sweep is itself a worker job, so with no worker running at all the run simply stays queued until one starts.

In a web/worker split, the worker must run the same version as the web containers. The web container queues the read; only a worker that knows the document-ai-run queue can execute it. A worker left on a pre-v1.40 image never takes these jobs, so every document read and lab scan waits and then fails as above, while everything else looks healthy. Deploy both from the same image tag (HEALTHLOG_IMAGE_REF is shared by app and the app-worker block) and restart the worker together with the web containers.

Database role

Both containers run the entrypoint, and both set up the database role the app connects as (database-role.md). They take turns on an advisory lock and derive the same password from API_TOKEN_HMAC_KEY and the database's name, which the shared environment block gives both, so a worker that starts after the web container changes nothing the web container relies on. Rotating API_TOKEN_HMAC_KEY changes that password: restart all containers together.

Healthchecks

• app (web) healthcheck: a node -e fetch of /api/health every 30s.
• app-worker does not expose a port; its liveness is tracked in-memory inside the process (src/lib/jobs/worker-status.ts) and read back by /api/admin/status. In a web/worker split this means the web container cannot see the separate worker container's liveness: /api/admin/status only reports the worker embedded in the container that serves the request. Watch the worker container's own logs and restart policy instead.

The web container does NOT depend on the worker, and vice versa, so neither container's startup can deadlock the other. The shared dependency is Postgres; both wait on db: condition: service_healthy.

Caveats

• Background tasks that touch user-scoped Wide Events (e.g. reminder notifications) emit telemetry from the worker; configure LOKI_* env vars on the worker if you want them.
• The off-host backup job runs in the worker only. The admin POST /api/admin/backup/test endpoint runs in the web container and exercises the same S3 credentials — set BACKUP_* env vars on BOTH.
• The startup gate (assertSubsystemEnabled) refuses to boot the reminder worker when HEALTHLOG_PROCESS_TYPE=web, so an accidental cross-mode invocation aborts immediately instead of silently doubling cron load.

Postgres connection-pool sizing

Every container that boots the HealthLog image owns a connection budget against the configured DATABASE_URL. The budget is 20 connections per container by default and is split between the Prisma pool and pg-boss (2 connections). The bundled compose exposes the knob as DB_CONNECTION_LIMIT, baked into DATABASE_URL as connection_limit; DATABASE_POOL_MAX remains a working explicit override. Resolution order: DB_CONNECTION_LIMIT, then DATABASE_POOL_MAX, then the connection_limit URL parameter, then the default of 20. In the bundled single-container setup (HEALTHLOG_PROCESS_TYPE=all) the web requests and the background worker run in one process and share that one Prisma pool, so a budget sized for the web traffic alone leaves the worker's jobs queueing for the same connections.

Plan total Postgres slots as container_count × budget. A stock Postgres container ships with max_connections = 100, so the safe envelope is:

Web replicas	Worker replicas	Total containers	Pool slots used	Headroom under 100
1	0	1	20	80
1	1	2	40	60
2	1	3	60	40
3	1	4	80	20
4	1	5	100	0 (do not exceed)

Once the table tips into the "0 headroom" row, every other client of the same Postgres — psql sessions, ad-hoc backups, the Prisma CLI during a migration deploy — will get FATAL: sorry, too many clients already. Either raise max_connections on the Postgres side, lower DATABASE_POOL_MAX, or front the database with PgBouncer in transaction-pooling mode.

The app sets its statement timeout, work_mem and TimeZone=UTC on every connection through the libpq options startup parameter. PgBouncer refuses that parameter (unsupported startup parameter: options) unless it is listed in ignore_startup_parameters. Either list it there, or set DATABASE_SESSION_OPTIONS_DISABLED=1 so the app sends no session settings; set the timeout and work_mem on the database role instead (ALTER ROLE healthlog SET work_mem = '16MB') if you still want them. The time zone is the exception: with the options off the app pins it with SET TIME ZONE 'UTC' on each new connection, which PgBouncer tracks per client, and warns at startup if a connection still reports another zone. The measurement maintenance job (Admin, System status) then applies its own settings with SET after connecting. Its concurrent index rebuild needs a session, so run it through a session-mode pool or a direct connection.

Overriding the per-container pool ceiling

With the bundled compose, set DB_CONNECTION_LIMIT in .env; it flows into DATABASE_URL's connection_limit. On a hand-rolled setup, DATABASE_POOL_MAX as a plain env var works the same way on every container, web and worker alike:

# docker-compose.yml — example for a 6-container deployment
services:
  app:
    image: ghcr.io/mbombeck/healthlog:latest
    environment:
      DATABASE_POOL_MAX: "12" # 6 containers × 12 = 72 < 100
      HEALTHLOG_PROCESS_TYPE: web

Rules of thumb:

• Each web replica's hottest path (/api/analytics fan-out, capped at p-limit(4)) consumes 4 concurrent slots. Keep DATABASE_POOL_MAX ≥ 8 so a single power-user request never exhausts the pool of one container.
• Worker replicas mostly use 1 connection per active pg-boss job; DATABASE_POOL_MAX = 8 is plenty unless you raised the pg-boss teamConcurrency.
• If you raise Postgres max_connections past 100, prefer raising DATABASE_POOL_MAX over adding more containers — fewer, fatter Node processes amortise the V8 footprint better than many thin ones.

Why the default is 20

The v1.4.39 empirical cold-mount trace showed thick /api/analytics holding ≥ 8 of the 10 default pool slots for 6.5 s on a 347 k-row tenant, starving every other dashboard query for the duration. The 20-slot default — paired with the p-limit(4) cap on the analytics fan-out — keeps 16 slots free for the rest of the dashboard while still sitting well under Postgres's 100-slot stock ceiling.

The implementation lives in src/lib/db.ts → getConnectionBudget() (split into getPrismaPoolMax() and getPgBossPoolMax()); the 20-slot default was chosen after measuring on a 4-CPU production container — see the inline rationale in src/lib/db.ts.

Large accounts

An account that holds tens of millions of readings, most often a full-history Google Health backfill of minute-by-minute heart rate, runs into the per-statement limit the app sets on every database connection: 60 seconds, from DATABASE_STATEMENT_TIMEOUT_MS (unset means 60000, 0 turns the limit off).

What it looks like

• Dashboard cards stay empty, or /api/daily/digest and other reads answer 500.
• The log shows job.run.failed for rollup-full-backfill with consecutiveFailures counting up, and Postgres error 57014: canceling statement due to statement timeout.
• The admin status page lists rollup-full-backfill under failing background jobs, and every admin gets a push once a system job has failed three jobs in a row (retries of one job count once; a job that runs per account counts per account), a reminder at most once a day while it keeps failing, and a notice when the failing job runs again. A one-off job that failed and does not run again is announced once.

What the app does about it

• The full rollup backfill, which runs at boot for any account whose summary tier is incomplete (and after a release that adds a column to it), folds the five-year window one calendar year at a time. Every statement of it runs under its own limit of five minutes, whatever DATABASE_STATEMENT_TIMEOUT_MS says, and a run that reaches its time budget hands the remaining years to a follow-up job. The readers that fall back to the raw readings while that tier is incomplete are fast again once it has finished.
• Heart rate older than 90 days is folded into one mean per hour, for Apple Health, Health Connect, Google Health and Oura alike. The DAY summaries keep the figures taken from the raw samples, and a later backfill or full sync does not write the folded samples back. On an account with a dense history the first fold works through the backlog over a few nights, one day per transaction, and leaves an account alone while its summary tier is still being rebuilt. Postgres reuses the freed table pages for new rows; to give the index space back to the disk, run the index rebuild of the measurement maintenance job from Admin, System status, once the backlog is through (docs/ops/measurement-maintenance.md).

When to raise the limit

Normally never: the long background passes bring their own limits. Raise DATABASE_STATEMENT_TIMEOUT_MS only when request reads still time out on a large account after the backfill has finished (the failing-jobs card is empty), or when something between the app and Postgres sets a lower limit. In that case also check SHOW statement_timeout; as the database role HealthLog connects with: a limit set on the role or the database, or by a pooler, applies too. Recreate the container after changing the variable so every connection picks it up:

# docker-compose.yml
services:
  app:
    environment:
      DATABASE_STATEMENT_TIMEOUT_MS: "180000" # three minutes

A higher limit lets one slow statement hold a pool connection longer, so raise it in steps rather than turning it off.