mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 20:42:43 +00:00
Marc-André Bombeck
fa7e1ee739
- geolite2: quote the real commented compose mount (./geolite2 to /opt/geolite2, the GEOLITE2_DIR default, so no env var is needed for the default path) - getting-started: healthcheck probes /api/health, not /api/version; the environment module reaches archive-api and geocoding-api .open-meteo.com, relevant for egress allowlists - mcp: the module is opt-in and off by default, but no longer the only such module - reverse-proxy: the Coach stream is SSE over plain HTTP, not WebSockets; note that TRUST_PROXY_HOPS is not on the bundled compose whitelist and must be added by hand - scaling: /api/health healthcheck; worker liveness is in-memory and invisible across a web/worker split; pool sizing now goes through getConnectionBudget with DB_CONNECTION_LIMIT as the compose knob and a 2-connection pg-boss share - sso: the native SSO handoff shipped, OIDC_ONLY works with app builds that carry it - unraid: geo databases are baked in only when the build had a licence key, or mounted; replaceable either way
192 lines
8.8 KiB
Markdown
192 lines
8.8 KiB
Markdown
# Scaling: web/worker container split
|
||
|
||
The default `docker compose up` launches a single container that runs
|
||
both the Next.js HTTP server AND the pg-boss worker. That works fine
|
||
for a personal deployment but is hard to scale horizontally because
|
||
every replica also runs the full reminder/insight cron schedule.
|
||
|
||
v1.4 introduces an environment-variable gate so the same image can be
|
||
deployed in three modes:
|
||
|
||
| `HEALTHLOG_PROCESS_TYPE` | Web | Worker | Use when |
|
||
| ------------------------ | --- | ------ | ------------------------- |
|
||
| `all` (default) | ✅ | ✅ | single-host self-hosting |
|
||
| `web` | ✅ | ❌ | scaling HTTP horizontally |
|
||
| `worker` | ❌ | ✅ | dedicated job worker |
|
||
|
||
## How to split
|
||
|
||
1. Edit `docker-compose.yml`: set `HEALTHLOG_PROCESS_TYPE=web` on the
|
||
`app` service and uncomment the `app-worker` service block at the
|
||
bottom (it ships with `HEALTHLOG_PROCESS_TYPE=worker` baked in).
|
||
2. Make sure both containers see the same `DATABASE_URL`,
|
||
`ENCRYPTION_KEY` (or `ENCRYPTION_KEYS`), and `API_TOKEN_HMAC_KEY`.
|
||
3. `docker compose up -d --build`.
|
||
|
||
Both containers connect to the same Postgres. pg-boss claims jobs
|
||
atomically via row-level locking (`FOR UPDATE SKIP LOCKED`), so
|
||
running multiple worker replicas is **safe** — no job runs twice —
|
||
but it doubles the DB load and muddies telemetry without adding
|
||
throughput a personal instance needs (`src/lib/process-type.ts`).
|
||
Run **one** worker container and scale the `web` containers instead.
|
||
|
||
## Apple Health import staging (shared volume required for a split)
|
||
|
||
The Apple Health `export.zip` import streams the upload to a staging
|
||
file on local disk (under the container's temp dir) and hands the
|
||
worker only the file **path**. In single-container mode
|
||
(`HEALTHLOG_PROCESS_TYPE=all`, the default) the web handler and the
|
||
worker share the same filesystem, so the handoff just works.
|
||
|
||
In a web/worker split — or any topology where the container that
|
||
accepts the upload is not the one that runs the job (a `web` +
|
||
`worker` split, extra `all` replicas, or old+new containers coexisting
|
||
during a rolling deploy) — the worker cannot see the web container's
|
||
staging file. The import then fails with a staging-file-missing error
|
||
("the import staging file is not visible to the worker"). Two things to
|
||
know:
|
||
|
||
- A pure `web` container has no pg-boss handle, so an import kicked off
|
||
there currently returns `503 Background worker is not running`. Run
|
||
imports from a container that also runs the worker until send-only
|
||
boss support lands.
|
||
- If you run web and worker as separate containers, they **must share
|
||
the import staging directory** — mount the same named volume at the
|
||
temp path on both — or run imports in single-container (`all`) mode.
|
||
Without a shared staging volume, imports in split mode cannot
|
||
complete.
|
||
|
||
## Apple Health import memory
|
||
|
||
The import parses `export.xml` as a stream: the archive is never
|
||
buffered whole, records flush to the database in fixed-size batches,
|
||
and what stays resident scales with the data's calendar span, not its
|
||
record count. Multi-year exports with millions of records parse within
|
||
the default Node.js heap.
|
||
|
||
Before parsing starts, the worker compares the archive's declared
|
||
uncompressed size against the runtime heap limit. If the limit is too
|
||
low to carry the export — realistically only when
|
||
`--max-old-space-size` was pinned far below the default — the import
|
||
fails immediately with a `failureReason` starting with
|
||
`insufficient_memory`, naming the required and available amounts.
|
||
Raise the limit on the container that runs the worker and re-upload:
|
||
|
||
```yaml
|
||
# docker-compose.yml
|
||
services:
|
||
app:
|
||
environment:
|
||
NODE_OPTIONS: --max-old-space-size=1024
|
||
```
|
||
|
||
An import interrupted by a container restart or an out-of-memory kill
|
||
does not stay stuck: the worker reconciles non-terminal import jobs at
|
||
boot and every 15 minutes, flipping orphaned rows to `failed`
|
||
(`interrupted_by_restart`) so the UI offers a retry instead of an
|
||
endless spinner.
|
||
|
||
## Healthchecks
|
||
|
||
- `app` (web) healthcheck: `wget /api/health` every 30s.
|
||
- `app-worker` does not expose a port; its liveness is tracked
|
||
in-memory inside the process (`src/lib/jobs/worker-status.ts`) and
|
||
read back by `/api/admin/status`. In a web/worker split this means
|
||
the web container cannot see the separate worker container's
|
||
liveness: `/api/admin/status` only reports the worker embedded in
|
||
the container that serves the request. Watch the worker container's
|
||
own logs and restart policy instead.
|
||
|
||
The web container does NOT depend on the worker, and vice versa, so
|
||
neither container's startup can deadlock the other. The shared
|
||
dependency is Postgres; both wait on `db: condition: service_healthy`.
|
||
|
||
## Caveats
|
||
|
||
- Background tasks that touch user-scoped Wide Events (e.g. reminder
|
||
notifications) emit telemetry from the worker; configure `LOKI_*`
|
||
env vars on the worker if you want them.
|
||
- The off-host backup job runs in the worker only. The admin
|
||
`POST /api/admin/backup/test` endpoint runs in the web container and
|
||
exercises the same S3 credentials — set `BACKUP_*` env vars on BOTH.
|
||
- The startup gate (`assertSubsystemEnabled`) refuses to boot the
|
||
reminder worker when `HEALTHLOG_PROCESS_TYPE=web`, so an accidental
|
||
cross-mode invocation aborts immediately instead of silently doubling
|
||
cron load.
|
||
|
||
## Postgres connection-pool sizing
|
||
|
||
Every container that boots the HealthLog image owns a connection
|
||
budget against the configured `DATABASE_URL`. The budget is
|
||
**20 connections per container by default** and is split between the
|
||
Prisma pool and pg-boss (2 connections). The bundled compose exposes
|
||
the knob as `DB_CONNECTION_LIMIT`, baked into `DATABASE_URL` as
|
||
`connection_limit`; `DATABASE_POOL_MAX` remains a working explicit
|
||
override. Resolution order: `DB_CONNECTION_LIMIT`, then
|
||
`DATABASE_POOL_MAX`, then the `connection_limit` URL parameter, then
|
||
the default of 20.
|
||
|
||
Plan total Postgres slots as **container_count × budget**.
|
||
A stock Postgres 16 container ships with `max_connections = 100`, so
|
||
the safe envelope is:
|
||
|
||
| Web replicas | Worker replicas | Total containers | Pool slots used | Headroom under 100 |
|
||
| ------------ | --------------- | ---------------- | --------------- | ------------------ |
|
||
| 1 | 0 | 1 | 20 | 80 |
|
||
| 1 | 1 | 2 | 40 | 60 |
|
||
| 2 | 1 | 3 | 60 | 40 |
|
||
| 3 | 1 | 4 | 80 | 20 |
|
||
| 4 | 1 | 5 | 100 | 0 (do not exceed) |
|
||
|
||
Once the table tips into the "0 headroom" row, every other client of
|
||
the same Postgres — `psql` sessions, ad-hoc backups, the Prisma CLI
|
||
during a migration deploy — will get `FATAL: sorry, too many clients
|
||
already`. Either raise `max_connections` on the Postgres side, lower
|
||
`DATABASE_POOL_MAX`, or front the database with PgBouncer in
|
||
transaction-pooling mode.
|
||
|
||
### Overriding the per-container pool ceiling
|
||
|
||
With the bundled compose, set `DB_CONNECTION_LIMIT` in `.env`; it
|
||
flows into `DATABASE_URL`'s `connection_limit`. On a hand-rolled
|
||
setup, `DATABASE_POOL_MAX` as a plain env var works the same way on
|
||
every container, web and worker alike:
|
||
|
||
```yaml
|
||
# docker-compose.yml — example for a 6-container deployment
|
||
services:
|
||
app:
|
||
image: ghcr.io/mbombeck/healthlog:latest
|
||
environment:
|
||
DATABASE_POOL_MAX: "12" # 6 containers × 12 = 72 < 100
|
||
HEALTHLOG_PROCESS_TYPE: web
|
||
```
|
||
|
||
Rules of thumb:
|
||
|
||
- Each web replica's hottest path (`/api/analytics` fan-out, capped at
|
||
`p-limit(4)`) consumes 4 concurrent slots. Keep
|
||
`DATABASE_POOL_MAX ≥ 8` so a single power-user request never
|
||
exhausts the pool of one container.
|
||
- Worker replicas mostly use 1 connection per active pg-boss job;
|
||
`DATABASE_POOL_MAX = 8` is plenty unless you raised the pg-boss
|
||
`teamConcurrency`.
|
||
- If you raise Postgres `max_connections` past 100, prefer raising
|
||
`DATABASE_POOL_MAX` over adding more containers — fewer, fatter
|
||
Node processes amortise the V8 footprint better than many thin
|
||
ones.
|
||
|
||
### Why the default is 20
|
||
|
||
The v1.4.39 empirical cold-mount trace showed thick `/api/analytics`
|
||
holding ≥ 8 of the 10 default pool slots for 6.5 s on a 347 k-row
|
||
tenant, starving every other dashboard query for the duration. The
|
||
20-slot default — paired with the `p-limit(4)` cap on the analytics
|
||
fan-out — keeps 16 slots free for the rest of the dashboard while
|
||
still sitting well under Postgres's 100-slot stock ceiling.
|
||
|
||
The implementation lives in `src/lib/db.ts → getConnectionBudget()`
|
||
(split into `getPrismaPoolMax()` and `getPgBossPoolMax()`); the
|
||
20-slot default was chosen after measuring on a 4-CPU production
|
||
container — see the inline rationale in `src/lib/db.ts`.
|