mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 20:42:43 +00:00
Marc-André Bombeck
82aa66b400
The v1.28.33 rollup + staging fixes were correct but never reached the reporter's case: the upload dedup keys on the content SHA-256 and the lookup matched a previously FAILED job too, so re-uploading the same export.zip returned the dead job's id and replayed its stale failureReason without ever staging or sending again. The dedup now excludes failed rows (short-circuits only on queued/unpacking/parsing/ upserting/done); the same bytes retry cleanly. A redundant staged upload is unlinked on a dedup hit instead of leaking in the staging dir. reconcileOrphanImportJobs no longer unconditionally fails every non-terminal ImportJob at boot: it now checks pg-boss job state plus a progress heartbeat (new import_jobs.updated_at, migration 0242), so a split or multi-worker deployment cannot kill an import running in another worker. The missing-staging message names the separate-container cause explicitly, with a matching note in docs/self-hosting/scaling.md.
152 lines
7.0 KiB
Markdown
152 lines
7.0 KiB
Markdown
# Scaling: web/worker container split
|
||
|
||
The default `docker compose up` launches a single container that runs
|
||
both the Next.js HTTP server AND the pg-boss worker. That works fine
|
||
for a personal deployment but is hard to scale horizontally because
|
||
every replica also runs the full reminder/insight cron schedule.
|
||
|
||
v1.4 introduces an environment-variable gate so the same image can be
|
||
deployed in three modes:
|
||
|
||
| `HEALTHLOG_PROCESS_TYPE` | Web | Worker | Use when |
|
||
| ------------------------ | --- | ------ | ------------------------- |
|
||
| `all` (default) | ✅ | ✅ | single-host self-hosting |
|
||
| `web` | ✅ | ❌ | scaling HTTP horizontally |
|
||
| `worker` | ❌ | ✅ | dedicated job worker |
|
||
|
||
## How to split
|
||
|
||
1. Edit `docker-compose.yml`: set `HEALTHLOG_PROCESS_TYPE=web` on the
|
||
`app` service and uncomment the `app-worker` service block at the
|
||
bottom (it ships with `HEALTHLOG_PROCESS_TYPE=worker` baked in).
|
||
2. Make sure both containers see the same `DATABASE_URL`,
|
||
`ENCRYPTION_KEY` (or `ENCRYPTION_KEYS`), and `API_TOKEN_HMAC_KEY`.
|
||
3. `docker compose up -d --build`.
|
||
|
||
Both containers connect to the same Postgres. pg-boss claims jobs
|
||
atomically via row-level locking (`FOR UPDATE SKIP LOCKED`), so
|
||
running multiple worker replicas is **safe** — no job runs twice —
|
||
but it doubles the DB load and muddies telemetry without adding
|
||
throughput a personal instance needs (`src/lib/process-type.ts`).
|
||
Run **one** worker container and scale the `web` containers instead.
|
||
|
||
## Apple Health import staging (shared volume required for a split)
|
||
|
||
The Apple Health `export.zip` import streams the upload to a staging
|
||
file on local disk (under the container's temp dir) and hands the
|
||
worker only the file **path**. In single-container mode
|
||
(`HEALTHLOG_PROCESS_TYPE=all`, the default) the web handler and the
|
||
worker share the same filesystem, so the handoff just works.
|
||
|
||
In a web/worker split — or any topology where the container that
|
||
accepts the upload is not the one that runs the job (a `web` +
|
||
`worker` split, extra `all` replicas, or old+new containers coexisting
|
||
during a rolling deploy) — the worker cannot see the web container's
|
||
staging file. The import then fails with a staging-file-missing error
|
||
("the import staging file is not visible to the worker"). Two things to
|
||
know:
|
||
|
||
- A pure `web` container has no pg-boss handle, so an import kicked off
|
||
there currently returns `503 Background worker is not running`. Run
|
||
imports from a container that also runs the worker until send-only
|
||
boss support lands.
|
||
- If you run web and worker as separate containers, they **must share
|
||
the import staging directory** — mount the same named volume at the
|
||
temp path on both — or run imports in single-container (`all`) mode.
|
||
Without a shared staging volume, imports in split mode cannot
|
||
complete.
|
||
|
||
## Healthchecks
|
||
|
||
- `app` (web) healthcheck: `wget /api/version` every 30s.
|
||
- `app-worker` does not expose a port; its liveness is reported into
|
||
the `worker_status` table on every reminder pass and surfaced through
|
||
`/api/admin/status`.
|
||
|
||
The web container does NOT depend on the worker, and vice versa, so
|
||
neither container's startup can deadlock the other. The shared
|
||
dependency is Postgres; both wait on `db: condition: service_healthy`.
|
||
|
||
## Caveats
|
||
|
||
- Background tasks that touch user-scoped Wide Events (e.g. reminder
|
||
notifications) emit telemetry from the worker; configure `LOKI_*`
|
||
env vars on the worker if you want them.
|
||
- The off-host backup job runs in the worker only. The admin
|
||
`POST /api/admin/backup/test` endpoint runs in the web container and
|
||
exercises the same S3 credentials — set `BACKUP_*` env vars on BOTH.
|
||
- The startup gate (`assertSubsystemEnabled`) refuses to boot the
|
||
reminder worker when `HEALTHLOG_PROCESS_TYPE=web`, so an accidental
|
||
cross-mode invocation aborts immediately instead of silently doubling
|
||
cron load.
|
||
|
||
## Postgres connection-pool sizing
|
||
|
||
Every container that boots the HealthLog image opens its own
|
||
`pg.Pool` against the configured `DATABASE_URL`. The pool ceiling is
|
||
**20 connections per container by default** (raised from the library
|
||
default of 10 in v1.4.40 to keep the dashboard's analytics fan-out
|
||
from starving every other query during a cold mount).
|
||
|
||
Plan total Postgres slots as **container_count × `DATABASE_POOL_MAX`**.
|
||
A stock Postgres 16 container ships with `max_connections = 100`, so
|
||
the safe envelope is:
|
||
|
||
| Web replicas | Worker replicas | Total containers | Pool slots used | Headroom under 100 |
|
||
| ------------ | --------------- | ---------------- | --------------- | ------------------ |
|
||
| 1 | 0 | 1 | 20 | 80 |
|
||
| 1 | 1 | 2 | 40 | 60 |
|
||
| 2 | 1 | 3 | 60 | 40 |
|
||
| 3 | 1 | 4 | 80 | 20 |
|
||
| 4 | 1 | 5 | 100 | 0 (do not exceed) |
|
||
|
||
Once the table tips into the "0 headroom" row, every other client of
|
||
the same Postgres — `psql` sessions, ad-hoc backups, the Prisma CLI
|
||
during a migration deploy — will get `FATAL: sorry, too many clients
|
||
already`. Either raise `max_connections` on the Postgres side, lower
|
||
`DATABASE_POOL_MAX`, or front the database with PgBouncer in
|
||
transaction-pooling mode.
|
||
|
||
### Overriding the per-container pool ceiling
|
||
|
||
Set `DATABASE_POOL_MAX` to a positive integer on every container that
|
||
needs a non-default ceiling — both web and worker pick up the same
|
||
env var:
|
||
|
||
```yaml
|
||
# docker-compose.yml — example for a 6-container deployment
|
||
services:
|
||
app:
|
||
image: ghcr.io/mbombeck/healthlog:latest
|
||
environment:
|
||
DATABASE_POOL_MAX: "12" # 6 containers × 12 = 72 < 100
|
||
HEALTHLOG_PROCESS_TYPE: web
|
||
```
|
||
|
||
Rules of thumb:
|
||
|
||
- Each web replica's hottest path (`/api/analytics` fan-out, capped at
|
||
`p-limit(4)`) consumes 4 concurrent slots. Keep
|
||
`DATABASE_POOL_MAX ≥ 8` so a single power-user request never
|
||
exhausts the pool of one container.
|
||
- Worker replicas mostly use 1 connection per active pg-boss job;
|
||
`DATABASE_POOL_MAX = 8` is plenty unless you raised the pg-boss
|
||
`teamConcurrency`.
|
||
- If you raise Postgres `max_connections` past 100, prefer raising
|
||
`DATABASE_POOL_MAX` over adding more containers — fewer, fatter
|
||
Node processes amortise the V8 footprint better than many thin
|
||
ones.
|
||
|
||
### Why the default is 20
|
||
|
||
The v1.4.39 empirical cold-mount trace showed thick `/api/analytics`
|
||
holding ≥ 8 of the 10 default pool slots for 6.5 s on a 347 k-row
|
||
tenant, starving every other dashboard query for the duration. The
|
||
20-slot default — paired with the `p-limit(4)` cap on the analytics
|
||
fan-out — keeps 16 slots free for the rest of the dashboard while
|
||
still sitting well under Postgres's 100-slot stock ceiling.
|
||
|
||
The implementation lives in `src/lib/db.ts → getPoolMax()`; the
|
||
20-slot default was chosen after measuring on a 4-CPU production
|
||
container — see the inline rationale in `getPoolMax()`.
|