Files
HealthLog/src/lib/db.ts
T
Marc-André Bombeck 900a5c4f1f perf: stop keeping folded samples and re-reading duplicates
The nightly folds soft-deleted every raw Apple Health sample they folded
into an hourly or daily stats row and kept it for 75 days. Nothing ever
read those rows as deletions; they only held a re-upload of the sample
off, and they were most of the measurements table and its indexes.

Ingest now refuses such a re-upload directly: a raw sample of a dense or
mean type, old enough to have been folded, whose hour or day a live
stats row covers, is a duplicate with reason folded_window. The batch
route and the Apple Health export import apply the same rule. With that
in place the dense retention and the daily-mean consolidation delete the
raw rows outright, and a new compaction-tombstone-purge job removes the
backlog: 5,000 rows a batch, one account at a time under the restore
lock, a short pause between batches, resumable because the predicate is
idempotent, queued at every boot and continuing itself while it makes
progress. A person's own deletions are never in its predicate. The sync
feed leaves compaction tombstones out and a restore does not write them
back; all three classify through the same function.

The batch route answers an exact re-sent sample without a transaction:
one read over the batch's sample ids, live or deleted, decides what the
reconciler would have decided, at the same point in the loop, so every
status, reason, counter and the cross-source merge stay as they were. A
batch of duplicates only no longer queues PR detection, a reminder sweep
or the detection audit row. Caches are evicted only for a foreground or
manual sync or a batch that wrote a hand-entered reading, and marked
stale otherwise; the targets cache follows the same split.

Also:
- POST /api/admin/maintenance/measurements queues VACUUM (ANALYZE) and a
  REINDEX INDEX CONCURRENTLY per index, largest first, on a connection of
  its own; it refuses while the purge runs, drops invalid leftover copies
  and stops at the first failed rebuild. Runbook in docs/ops.
- The pure volume queues keep their finished jobs for 96 hours instead
  of seven days, never less than the failure window plus a day.
- work_mem is set per app connection from DATABASE_WORK_MEM (16MB), also
  when the statement timeout is disabled.
- The PR-detection fallback tick re-scans only accounts whose readings
  or workouts changed since the previous tick.
- Type discovery for the status refill walks the live-rows index as a
  loose index scan instead of grouping every live row.
- The host metrics chart polls every five minutes, like its sampler.
- The batch route records per-type HealthKit arrivals.
2026-10-08 08:41:25 +02:00

157 lines
6.2 KiB
TypeScript

import { Prisma, PrismaClient } from "@/generated/prisma/client";
import { PrismaPg } from "@prisma/adapter-pg";
const globalForPrisma = globalThis as unknown as {
prisma: PrismaClient | undefined;
};
const DEFAULT_CONNECTION_BUDGET = 20;
const DEFAULT_POOL_TIMEOUT_SECONDS = 20;
const PG_BOSS_CONNECTIONS = 2;
function positiveInteger(raw: string | undefined): number | null {
if (!raw) return null;
const parsed = Number.parseInt(raw, 10);
return Number.isFinite(parsed) && parsed > 0 ? parsed : null;
}
function databaseUrlParameter(name: string): string | undefined {
const rawUrl = process.env.DATABASE_URL;
if (!rawUrl) return undefined;
try {
return new URL(rawUrl).searchParams.get(name) ?? undefined;
} catch {
return undefined;
}
}
/**
* Total PostgreSQL connections this process may own. Compose already exposes
* this as DB_CONNECTION_LIMIT through DATABASE_URL's `connection_limit`.
* DATABASE_POOL_MAX remains a backward-compatible explicit override.
*/
export function getConnectionBudget(): number {
const configured =
positiveInteger(process.env.DB_CONNECTION_LIMIT) ??
positiveInteger(process.env.DATABASE_POOL_MAX) ??
positiveInteger(databaseUrlParameter("connection_limit")) ??
DEFAULT_CONNECTION_BUDGET;
return Math.max(2, configured);
}
export function getPgBossPoolMax(): number {
return Math.min(PG_BOSS_CONNECTIONS, getConnectionBudget() - 1);
}
export function getPrismaPoolMax(): number {
return getConnectionBudget() - getPgBossPoolMax();
}
export function getPoolConnectionTimeoutMs(): number {
const seconds =
positiveInteger(process.env.DB_POOL_TIMEOUT) ??
positiveInteger(databaseUrlParameter("pool_timeout")) ??
DEFAULT_POOL_TIMEOUT_SECONDS;
return seconds * 1_000;
}
/**
* Per-session DB statement timeout in milliseconds.
*
* Without this a single pathological query or lock wait holds one of the
* (default 20) pool slots indefinitely; 20 such queries permanently exhaust
* the pool and every DB-backed route stops serving — the single highest-impact
* availability gap (A-1). `statement_timeout` caps how long the server runs one
* statement; `idle_in_transaction_session_timeout` reaps a connection wedged
* mid-transaction (a client that opened a tx and stalled). Both are applied at
* connection-establishment via the libpq `options` startup parameter, so every
* pooled session inherits them.
*
* Env-overridable via `DATABASE_STATEMENT_TIMEOUT_MS`. Set to `0` to disable
* (legacy unbounded behaviour). Default 60 s — bounds a genuinely runaway query
* (the A-1 goal: never hold a pool slot forever) while leaving ample headroom
* for the heaviest legitimate read, the live-aggregate analytics fallback on a
* coverage miss over a large account, which can run well past 15 s. Heavy admin
* one-shots (drain/backfill) can still dial their own timeout if needed.
*/
export function getStatementTimeoutMs(): number {
const raw = process.env.DATABASE_STATEMENT_TIMEOUT_MS;
if (raw !== undefined && raw !== "") {
const parsed = Number.parseInt(raw, 10);
if (Number.isFinite(parsed) && parsed >= 0) return parsed;
}
return 60_000;
}
/** `work_mem` when `DATABASE_WORK_MEM` is unset. */
export const DEFAULT_WORK_MEM = "16MB";
/**
* A Postgres memory size: digits and an optional unit. The value is spliced
* into the libpq `options` string, where a space would start another `-c`
* flag, so anything else is refused and the default used.
*/
const WORK_MEM_SHAPE = /^[1-9][0-9]{0,6}(kB|MB|GB)?$/;
/**
* Per-session `work_mem` (v1.42), from `DATABASE_WORK_MEM`.
*
* The server default of 4 MB made the larger sorts and hash aggregates of the
* analytics reads spill to disk. `work_mem` is a USERSET setting, so it rides
* the same startup `options` as the timeouts and applies to this app's
* connections only, without a database restart. Each sort or hash step of
* each connection may use this much, so raise it with the pool size in mind.
* An unparsable value falls back to the default rather than reaching libpq.
*/
export function getWorkMem(): string {
const raw = process.env.DATABASE_WORK_MEM?.trim();
return raw && WORK_MEM_SHAPE.test(raw) ? raw : DEFAULT_WORK_MEM;
}
/**
* Build the libpq `options` startup string applying the session settings.
* Passed straight through `PrismaPg` to the underlying `pg.Pool`, which
* forwards it as the connection's `options` startup parameter so every
* session carries them from the first query.
*
* The timeouts are left out when disabled (timeout 0); `work_mem` is always
* set. Up to v1.41 the whole string was `undefined` at timeout 0, which would
* have taken `work_mem` with it.
*/
export function buildSessionOptions(): string {
const timeoutMs = getStatementTimeoutMs();
const timeouts =
timeoutMs > 0
? `-c statement_timeout=${timeoutMs} -c idle_in_transaction_session_timeout=${timeoutMs} `
: "";
return `${timeouts}-c work_mem=${getWorkMem()}`;
}
function createPrismaClient() {
const adapter = new PrismaPg({
connectionString: process.env.DATABASE_URL!,
max: getPrismaPoolMax(),
connectionTimeoutMillis: getPoolConnectionTimeoutMs(),
options: buildSessionOptions(),
});
return new PrismaClient({ adapter });
}
// One client per process, in production too. A production build gives every
// server bundle its own copy of this module (the route handlers, the RSC/SSR
// renderer, and the instrumentation bundle that boots the workers), so a
// module-scoped client alone meant three of them: three query-compiler states
// of roughly 28 MB each on a heap that V8 caps near 520 MB in a 1 GiB
// container, and three pg pools each sized to the whole per-process
// connection budget above. Parking the client on `globalThis` makes every
// copy of the module resolve the same one.
export const prisma = globalForPrisma.prisma ?? createPrismaClient();
globalForPrisma.prisma = prisma;
// Prisma's `InputJsonValue` requires an explicit index signature that
// typed application shapes (Zod-validated, hand-written interfaces)
// don't carry. Every JSON-column write would otherwise repeat the same
// `value as unknown as Prisma.InputJsonValue` escape hatch — this one
// helper centralises the cast so the WHY stays in a single place.
export const toJson = <T>(v: T) => v as unknown as Prisma.InputJsonValue;