mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-11 13:52:20 +00:00
docs(operator): pg.Pool max scaling guidance for multi-container deploys
Append a Postgres connection-pool sizing section to the self-hosting scaling guide. Documents the v1.4.40 W-POOL default (20 slots per container), the container_count × DATABASE_POOL_MAX × max_connections envelope (with a 1-5 container ready-reckoner table), the env-var override path, and PgBouncer as the escape hatch for deployments that outgrow the per-process pool model. Cross-references src/lib/db.ts and the empirical trace that motivated the bump. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
8a56f48259
commit
70c5026812
2 files changed
+212
No files matched your search
@@ -0,0 +1,141 @@
|
||||
# Phase W-PROCESS-DOCS — v1.4.41 report
|
||||
|
||||
Wave **W-PROCESS-DOCS** delivers three process / DX hardening items
|
||||
identified in the v1.4.40 audit + W-INFRA closure report.
|
||||
|
||||
## Item 1 — knip exports + types enforcement (STAGED, not yet flipped)
|
||||
|
||||
**Goal**: promote `exports` + `types` from informational-only to
|
||||
enforcing in `.github/workflows/knip.yml`.
|
||||
|
||||
**Status**: gate **staged** — comment block + flip instructions in
|
||||
place, but the `--include` filter still excludes `exports,types`
|
||||
because the cleanup waves they depend on (W-SIMPLIFIER,
|
||||
W-INSIGHTS-HOT) had not yet landed on `develop` at the time
|
||||
W-PROCESS-DOCS ran.
|
||||
|
||||
**Baseline at hand-off** (from `pnpm knip --reporter compact` on
|
||||
HEAD `4cca62dc…`):
|
||||
|
||||
- Unused exports: 48
|
||||
- Unused exported types: 52
|
||||
|
||||
Most of the remaining surface is public-API zod schemas and type
|
||||
aliases re-exported for downstream consumers — many will legitimately
|
||||
stay after cleanup. The promotion criterion is **zero offenders on a
|
||||
clean `pnpm knip --reporter compact`**, NOT "drive both counts to
|
||||
zero unconditionally". Where a public schema must remain exported,
|
||||
the next wave should:
|
||||
|
||||
1. Verify the export has an external consumer (test file, API route,
|
||||
downstream package).
|
||||
2. Add a `// knip-ignore` marker tag on the export, OR add the file
|
||||
to `knip.json`'s `ignore` block with a one-line rationale.
|
||||
|
||||
**Flip recipe** for the wave that lands after W-SIMPLIFIER +
|
||||
W-INSIGHTS-HOT:
|
||||
|
||||
```yaml
|
||||
# .github/workflows/knip.yml — drop the --include filter
|
||||
- run: pnpm knip --reporter compact
|
||||
```
|
||||
|
||||
A revert is a one-line restoration of the `--include` filter — risk
|
||||
is low and reversible.
|
||||
|
||||
## Item 2 — ESLint custom rule `healthlog/queryKey-factory` (SHIPPED)
|
||||
|
||||
**Goal**: promote the test-guard substitute (v1.4.40 W-RSC,
|
||||
`src/lib/__tests__/query-keys.test.ts` factory-bypass guard) to a
|
||||
real ESLint rule so editor + CI both fail fast on a literal-array
|
||||
`queryKey: [ … ]` declaration inside guarded files.
|
||||
|
||||
**Files**:
|
||||
|
||||
- `eslint-plugins/healthlog/queryKey-factory.js` — new flat-config
|
||||
plugin. AST rule scans every `Property` node and flags
|
||||
`queryKey: ArrayExpression` / `mutationKey: ArrayExpression` only
|
||||
when the surrounding file is on the guarded list. Identifiers,
|
||||
`queryKeys.foo()` call expressions, and conditional expressions
|
||||
all pass — only the bypass shape from audit-H1 fails.
|
||||
- `eslint.config.mjs` — registers the plugin and wires the rule at
|
||||
`"error"` severity.
|
||||
|
||||
**Guarded surface** (mirrors `guardedRoots` in the test-guard
|
||||
substitute):
|
||||
|
||||
- `src/components/charts/**`
|
||||
- `src/components/comparison/**`
|
||||
- `src/app/page.tsx`
|
||||
- `src/hooks/use-auth.ts`
|
||||
|
||||
Files outside the guarded surface are intentionally exempt — extend
|
||||
the list in lockstep with the test-guard substitute as future waves
|
||||
migrate `src/components/admin/**`, `src/components/settings/**`,
|
||||
`src/components/medications/**`, and `src/components/integrations/**`
|
||||
to the factory.
|
||||
|
||||
**Whitelist** for the factory home itself:
|
||||
|
||||
- `src/lib/query-keys.ts` (the factory definition)
|
||||
- `src/lib/__tests__/query-keys.test.ts` (asserts factory output
|
||||
shape against literal keys — intentional)
|
||||
|
||||
**Verification**: `pnpm lint` reports 0 errors / 5 pre-existing
|
||||
warnings on HEAD. The new rule loads cleanly and matches zero
|
||||
offenders, confirming W-RSC + W-FRONTEND-FACTORY already routed
|
||||
every guarded file through the factory.
|
||||
|
||||
## Item 3 — pg.Pool max × N container deploy doc (SHIPPED)
|
||||
|
||||
**Goal**: document the v1.4.40 W-POOL change (`pg.Pool max = 20` per
|
||||
container, env-overridable via `DATABASE_POOL_MAX`) in deploy notes
|
||||
so operators can compute the per-deployment pool envelope without
|
||||
reading source.
|
||||
|
||||
**File**: `docs/self-hosting/scaling.md` — appended a
|
||||
`## Postgres connection-pool sizing (v1.4.40 W-POOL)` section
|
||||
covering:
|
||||
|
||||
- The 20-slot default rationale (cold-mount trace, `p-limit(4)`
|
||||
pairing).
|
||||
- A `container_count × DATABASE_POOL_MAX` ready-reckoner table from
|
||||
1 to 5 containers against the stock `max_connections = 100`
|
||||
ceiling, including the headroom column and the "do not exceed"
|
||||
callout for 5 containers.
|
||||
- An override example (`docker-compose.yml`) and three rules of
|
||||
thumb for picking the right per-container ceiling.
|
||||
- A pointer to PgBouncer transaction pooling for deployments that
|
||||
outgrow the per-process pool model.
|
||||
- A cross-reference to `src/lib/db.ts → getPoolMax()` and the
|
||||
empirical trace that motivated the bump.
|
||||
|
||||
The page already covered the web/worker split from v1.4 G3, so the
|
||||
new section sits naturally at the end as the next "what to know
|
||||
when you scale horizontally" topic.
|
||||
|
||||
## Touch-disjoint guards honoured
|
||||
|
||||
- Did NOT touch `src/lib/query-keys.ts` (W-FRONTEND-FACTORY territory).
|
||||
- Did NOT touch `src/lib/__tests__/query-keys.test.ts` (W-FRONTEND-FACTORY
|
||||
may extend its allowlist; the ESLint rule's allowlist mirrors the
|
||||
test-guard substitute and can be updated in lockstep without merge
|
||||
conflicts).
|
||||
- Coordinated via `git pull --ff-only` mid-flight; the eslint plugin
|
||||
+ config + knip workflow edits landed on top of W-FRONTEND-FACTORY's
|
||||
`0bf07afb refactor(query-keys): expand factory and migrate auth,
|
||||
notifications, about` commit without conflicts.
|
||||
|
||||
## Quality gates
|
||||
|
||||
- `pnpm lint` — 0 errors, 5 pre-existing warnings (unchanged).
|
||||
- `pnpm tsc --noEmit` — clean.
|
||||
- `pnpm knip --reporter compact --include files,dependencies,binaries,unlisted`
|
||||
— clean (matches current CI gate).
|
||||
|
||||
## Hand-off to the next wave
|
||||
|
||||
When W-SIMPLIFIER + W-INSIGHTS-HOT close and `pnpm knip --reporter
|
||||
compact` reports zero unused exports + zero unused types, flip the
|
||||
`--include` filter off in `.github/workflows/knip.yml` to enforce
|
||||
both tiers. Everything else here is ready to ship as-is.
|
||||
@@ -50,3 +50,74 @@ dependency is Postgres; both wait on `db: condition: service_healthy`.
|
||||
reminder worker when `HEALTHLOG_PROCESS_TYPE=web`, so an accidental
|
||||
cross-mode invocation aborts immediately instead of silently doubling
|
||||
cron load.
|
||||
|
||||
## Postgres connection-pool sizing (v1.4.40 W-POOL)
|
||||
|
||||
Every container that boots the HealthLog image opens its own
|
||||
`pg.Pool` against the configured `DATABASE_URL`. The pool ceiling is
|
||||
**20 connections per container by default** (raised from the library
|
||||
default of 10 in v1.4.40 to keep the dashboard's analytics fan-out
|
||||
from starving every other query during a cold mount).
|
||||
|
||||
Plan total Postgres slots as **container_count × `DATABASE_POOL_MAX`**.
|
||||
A stock Postgres 16 container ships with `max_connections = 100`, so
|
||||
the safe envelope is:
|
||||
|
||||
| Web replicas | Worker replicas | Total containers | Pool slots used | Headroom under 100 |
|
||||
| ------------ | --------------- | ---------------- | --------------- | ------------------ |
|
||||
| 1 | 0 | 1 | 20 | 80 |
|
||||
| 1 | 1 | 2 | 40 | 60 |
|
||||
| 2 | 1 | 3 | 60 | 40 |
|
||||
| 3 | 1 | 4 | 80 | 20 |
|
||||
| 4 | 1 | 5 | 100 | 0 (do not exceed) |
|
||||
|
||||
Once the table tips into the "0 headroom" row, every other client of
|
||||
the same Postgres — `psql` sessions, ad-hoc backups, the Prisma CLI
|
||||
during a migration deploy — will get `FATAL: sorry, too many clients
|
||||
already`. Either raise `max_connections` on the Postgres side, lower
|
||||
`DATABASE_POOL_MAX`, or front the database with PgBouncer in
|
||||
transaction-pooling mode.
|
||||
|
||||
### Overriding the per-container pool ceiling
|
||||
|
||||
Set `DATABASE_POOL_MAX` to a positive integer on every container that
|
||||
needs a non-default ceiling — both web and worker pick up the same
|
||||
env var:
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml — example for a 6-container deployment
|
||||
services:
|
||||
app:
|
||||
image: ghcr.io/mbombeck/healthlog:latest
|
||||
environment:
|
||||
DATABASE_POOL_MAX: "12" # 6 containers × 12 = 72 < 100
|
||||
HEALTHLOG_PROCESS_TYPE: web
|
||||
```
|
||||
|
||||
Rules of thumb:
|
||||
|
||||
- Each web replica's hottest path (`/api/analytics` fan-out, capped at
|
||||
`p-limit(4)` per v1.4.40 W-POOL) consumes 4 concurrent slots. Keep
|
||||
`DATABASE_POOL_MAX ≥ 8` so a single power-user request never
|
||||
exhausts the pool of one container.
|
||||
- Worker replicas mostly use 1 connection per active pg-boss job;
|
||||
`DATABASE_POOL_MAX = 8` is plenty unless you raised the pg-boss
|
||||
`teamConcurrency`.
|
||||
- If you raise Postgres `max_connections` past 100, prefer raising
|
||||
`DATABASE_POOL_MAX` over adding more containers — fewer, fatter
|
||||
Node processes amortise the V8 footprint better than many thin
|
||||
ones.
|
||||
|
||||
### Why the default is 20
|
||||
|
||||
The v1.4.39 empirical cold-mount trace showed thick `/api/analytics`
|
||||
holding ≥ 8 of the 10 default pool slots for 6.5 s on a 347 k-row
|
||||
tenant, starving every other dashboard query for the duration. The
|
||||
20-slot default — paired with the `p-limit(4)` cap on the analytics
|
||||
fan-out — keeps 16 slots free for the rest of the dashboard while
|
||||
still sitting well under Postgres's 100-slot stock ceiling.
|
||||
|
||||
The implementation lives in `src/lib/db.ts → getPoolMax()`; the
|
||||
rationale and the audit trail live in
|
||||
`.planning/round-v1439-empirical-trace.md § B2` and the v1.4.40
|
||||
W-POOL phase report.
|
||||
Reference in new issue
Block a user