From 70c5026812673a99513379041befdc731fd20b3c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc-Andr=C3=A9=20Bombeck?= Date: Thu, 21 May 2026 17:16:04 +0200 Subject: [PATCH] docs(operator): pg.Pool max scaling guidance for multi-container deploys MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Append a Postgres connection-pool sizing section to the self-hosting scaling guide. Documents the v1.4.40 W-POOL default (20 slots per container), the container_count × DATABASE_POOL_MAX × max_connections envelope (with a 1-5 container ready-reckoner table), the env-var override path, and PgBouncer as the escape hatch for deployments that outgrow the per-process pool model. Cross-references src/lib/db.ts and the empirical trace that motivated the bump. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../phase-W-PROCESS-DOCS-v1441-report.md | 141 ++++++++++++++++++ docs/self-hosting/scaling.md | 71 +++++++++ 2 files changed, 212 insertions(+) create mode 100644 .planning/phase-W-PROCESS-DOCS-v1441-report.md diff --git a/.planning/phase-W-PROCESS-DOCS-v1441-report.md b/.planning/phase-W-PROCESS-DOCS-v1441-report.md new file mode 100644 index 000000000..2ea320fcb --- /dev/null +++ b/.planning/phase-W-PROCESS-DOCS-v1441-report.md @@ -0,0 +1,141 @@ +# Phase W-PROCESS-DOCS — v1.4.41 report + +Wave **W-PROCESS-DOCS** delivers three process / DX hardening items +identified in the v1.4.40 audit + W-INFRA closure report. + +## Item 1 — knip exports + types enforcement (STAGED, not yet flipped) + +**Goal**: promote `exports` + `types` from informational-only to +enforcing in `.github/workflows/knip.yml`. + +**Status**: gate **staged** — comment block + flip instructions in +place, but the `--include` filter still excludes `exports,types` +because the cleanup waves they depend on (W-SIMPLIFIER, +W-INSIGHTS-HOT) had not yet landed on `develop` at the time +W-PROCESS-DOCS ran. + +**Baseline at hand-off** (from `pnpm knip --reporter compact` on +HEAD `4cca62dc…`): + +- Unused exports: 48 +- Unused exported types: 52 + +Most of the remaining surface is public-API zod schemas and type +aliases re-exported for downstream consumers — many will legitimately +stay after cleanup. The promotion criterion is **zero offenders on a +clean `pnpm knip --reporter compact`**, NOT "drive both counts to +zero unconditionally". Where a public schema must remain exported, +the next wave should: + +1. Verify the export has an external consumer (test file, API route, + downstream package). +2. Add a `// knip-ignore` marker tag on the export, OR add the file + to `knip.json`'s `ignore` block with a one-line rationale. + +**Flip recipe** for the wave that lands after W-SIMPLIFIER + +W-INSIGHTS-HOT: + +```yaml +# .github/workflows/knip.yml — drop the --include filter +- run: pnpm knip --reporter compact +``` + +A revert is a one-line restoration of the `--include` filter — risk +is low and reversible. + +## Item 2 — ESLint custom rule `healthlog/queryKey-factory` (SHIPPED) + +**Goal**: promote the test-guard substitute (v1.4.40 W-RSC, +`src/lib/__tests__/query-keys.test.ts` factory-bypass guard) to a +real ESLint rule so editor + CI both fail fast on a literal-array +`queryKey: [ … ]` declaration inside guarded files. + +**Files**: + +- `eslint-plugins/healthlog/queryKey-factory.js` — new flat-config + plugin. AST rule scans every `Property` node and flags + `queryKey: ArrayExpression` / `mutationKey: ArrayExpression` only + when the surrounding file is on the guarded list. Identifiers, + `queryKeys.foo()` call expressions, and conditional expressions + all pass — only the bypass shape from audit-H1 fails. +- `eslint.config.mjs` — registers the plugin and wires the rule at + `"error"` severity. + +**Guarded surface** (mirrors `guardedRoots` in the test-guard +substitute): + +- `src/components/charts/**` +- `src/components/comparison/**` +- `src/app/page.tsx` +- `src/hooks/use-auth.ts` + +Files outside the guarded surface are intentionally exempt — extend +the list in lockstep with the test-guard substitute as future waves +migrate `src/components/admin/**`, `src/components/settings/**`, +`src/components/medications/**`, and `src/components/integrations/**` +to the factory. + +**Whitelist** for the factory home itself: + +- `src/lib/query-keys.ts` (the factory definition) +- `src/lib/__tests__/query-keys.test.ts` (asserts factory output + shape against literal keys — intentional) + +**Verification**: `pnpm lint` reports 0 errors / 5 pre-existing +warnings on HEAD. The new rule loads cleanly and matches zero +offenders, confirming W-RSC + W-FRONTEND-FACTORY already routed +every guarded file through the factory. + +## Item 3 — pg.Pool max × N container deploy doc (SHIPPED) + +**Goal**: document the v1.4.40 W-POOL change (`pg.Pool max = 20` per +container, env-overridable via `DATABASE_POOL_MAX`) in deploy notes +so operators can compute the per-deployment pool envelope without +reading source. + +**File**: `docs/self-hosting/scaling.md` — appended a +`## Postgres connection-pool sizing (v1.4.40 W-POOL)` section +covering: + +- The 20-slot default rationale (cold-mount trace, `p-limit(4)` + pairing). +- A `container_count × DATABASE_POOL_MAX` ready-reckoner table from + 1 to 5 containers against the stock `max_connections = 100` + ceiling, including the headroom column and the "do not exceed" + callout for 5 containers. +- An override example (`docker-compose.yml`) and three rules of + thumb for picking the right per-container ceiling. +- A pointer to PgBouncer transaction pooling for deployments that + outgrow the per-process pool model. +- A cross-reference to `src/lib/db.ts → getPoolMax()` and the + empirical trace that motivated the bump. + +The page already covered the web/worker split from v1.4 G3, so the +new section sits naturally at the end as the next "what to know +when you scale horizontally" topic. + +## Touch-disjoint guards honoured + +- Did NOT touch `src/lib/query-keys.ts` (W-FRONTEND-FACTORY territory). +- Did NOT touch `src/lib/__tests__/query-keys.test.ts` (W-FRONTEND-FACTORY + may extend its allowlist; the ESLint rule's allowlist mirrors the + test-guard substitute and can be updated in lockstep without merge + conflicts). +- Coordinated via `git pull --ff-only` mid-flight; the eslint plugin + + config + knip workflow edits landed on top of W-FRONTEND-FACTORY's + `0bf07afb refactor(query-keys): expand factory and migrate auth, + notifications, about` commit without conflicts. + +## Quality gates + +- `pnpm lint` — 0 errors, 5 pre-existing warnings (unchanged). +- `pnpm tsc --noEmit` — clean. +- `pnpm knip --reporter compact --include files,dependencies,binaries,unlisted` + — clean (matches current CI gate). + +## Hand-off to the next wave + +When W-SIMPLIFIER + W-INSIGHTS-HOT close and `pnpm knip --reporter +compact` reports zero unused exports + zero unused types, flip the +`--include` filter off in `.github/workflows/knip.yml` to enforce +both tiers. Everything else here is ready to ship as-is. diff --git a/docs/self-hosting/scaling.md b/docs/self-hosting/scaling.md index 581ea43e3..3b3c975a3 100644 --- a/docs/self-hosting/scaling.md +++ b/docs/self-hosting/scaling.md @@ -50,3 +50,74 @@ dependency is Postgres; both wait on `db: condition: service_healthy`. reminder worker when `HEALTHLOG_PROCESS_TYPE=web`, so an accidental cross-mode invocation aborts immediately instead of silently doubling cron load. + +## Postgres connection-pool sizing (v1.4.40 W-POOL) + +Every container that boots the HealthLog image opens its own +`pg.Pool` against the configured `DATABASE_URL`. The pool ceiling is +**20 connections per container by default** (raised from the library +default of 10 in v1.4.40 to keep the dashboard's analytics fan-out +from starving every other query during a cold mount). + +Plan total Postgres slots as **container_count × `DATABASE_POOL_MAX`**. +A stock Postgres 16 container ships with `max_connections = 100`, so +the safe envelope is: + +| Web replicas | Worker replicas | Total containers | Pool slots used | Headroom under 100 | +| ------------ | --------------- | ---------------- | --------------- | ------------------ | +| 1 | 0 | 1 | 20 | 80 | +| 1 | 1 | 2 | 40 | 60 | +| 2 | 1 | 3 | 60 | 40 | +| 3 | 1 | 4 | 80 | 20 | +| 4 | 1 | 5 | 100 | 0 (do not exceed) | + +Once the table tips into the "0 headroom" row, every other client of +the same Postgres — `psql` sessions, ad-hoc backups, the Prisma CLI +during a migration deploy — will get `FATAL: sorry, too many clients +already`. Either raise `max_connections` on the Postgres side, lower +`DATABASE_POOL_MAX`, or front the database with PgBouncer in +transaction-pooling mode. + +### Overriding the per-container pool ceiling + +Set `DATABASE_POOL_MAX` to a positive integer on every container that +needs a non-default ceiling — both web and worker pick up the same +env var: + +```yaml +# docker-compose.yml — example for a 6-container deployment +services: + app: + image: ghcr.io/mbombeck/healthlog:latest + environment: + DATABASE_POOL_MAX: "12" # 6 containers × 12 = 72 < 100 + HEALTHLOG_PROCESS_TYPE: web +``` + +Rules of thumb: + +- Each web replica's hottest path (`/api/analytics` fan-out, capped at + `p-limit(4)` per v1.4.40 W-POOL) consumes 4 concurrent slots. Keep + `DATABASE_POOL_MAX ≥ 8` so a single power-user request never + exhausts the pool of one container. +- Worker replicas mostly use 1 connection per active pg-boss job; + `DATABASE_POOL_MAX = 8` is plenty unless you raised the pg-boss + `teamConcurrency`. +- If you raise Postgres `max_connections` past 100, prefer raising + `DATABASE_POOL_MAX` over adding more containers — fewer, fatter + Node processes amortise the V8 footprint better than many thin + ones. + +### Why the default is 20 + +The v1.4.39 empirical cold-mount trace showed thick `/api/analytics` +holding ≥ 8 of the 10 default pool slots for 6.5 s on a 347 k-row +tenant, starving every other dashboard query for the duration. The +20-slot default — paired with the `p-limit(4)` cap on the analytics +fan-out — keeps 16 slots free for the rest of the dashboard while +still sitting well under Postgres's 100-slot stock ceiling. + +The implementation lives in `src/lib/db.ts → getPoolMax()`; the +rationale and the audit trail live in +`.planning/round-v1439-empirical-trace.md § B2` and the v1.4.40 +W-POOL phase report.