diff --git a/docs/ops/measurement-maintenance.md b/docs/ops/measurement-maintenance.md index 2576b3fcd..8429f15f8 100644 --- a/docs/ops/measurement-maintenance.md +++ b/docs/ops/measurement-maintenance.md @@ -137,7 +137,8 @@ it. ## Reading the result The run's wide event `job.measurement_maintenance` carries -`maintenance_outcome` (`completed`, `refused_purge_running`, `failed`), the +`maintenance_outcome` (`completed`, `nothing_due`, `refused_purge_running`, +`failed`), the live and dead rows the run started from, the table and index sizes before and after, and `maintenance_steps`, one entry per step with its target, size before and after, and duration. The same sizes from `psql`: @@ -147,6 +148,38 @@ SELECT pg_size_pretty(pg_total_relation_size('measurements')) AS total, pg_size_pretty(pg_indexes_size('measurements')) AS indexes; ``` +## The nightly check + +Since v1.43.7 the same queue also runs by itself at 03:40 (Europe/Berlin). +That run names no steps: it reads the table's live and dead rows and runs a +plain `VACUUM (ANALYZE)` only when there are at least 10,000 dead rows and +more than one for every ten live ones. It never rebuilds an index, and a +plain vacuum stops neither reads nor writes. It shares the singleton key with +the run you start, so the two never overlap. The admin card keeps showing +only the runs you started. + +The check also measures how far the table file has grown past what its live +rows need (`maintenance_bloat_factor` on the wide event, `bloat_factor` in the +job result). Past 3 it adds a warning to the event: a plain vacuum makes the +space reusable but does not shrink the file, and only `VACUUM FULL` does (see +below). + +Why the table grew in the first place: until v1.43.7 an update that changed +nothing still moved `updated_at`, which sits in an index, so no update could +stay on its page. Migration 0412 keeps `updated_at` and `sync_version` on such +an update and gives the table `fillfactor = 85` and a 5 % autovacuum +threshold. To check that updates stay on their page: + +```sql +SELECT n_tup_upd, n_tup_hot_upd, n_live_tup, n_dead_tup, + pg_size_pretty(pg_relation_size('measurements')) AS heap +FROM pg_stat_user_tables WHERE relname = 'measurements'; +``` + +`n_tup_hot_upd` should now grow along with `n_tup_upd`. The fill factor +applies to pages written after the migration; a `VACUUM FULL` after the +upgrade applies it to the whole table. + ## "could not resize shared memory segment" Up to v1.42.0 the run could stop at once with `could not resize shared memory @@ -160,6 +193,6 @@ runs single-process and needs no shared segment, and the bundled ## Not part of this `VACUUM FULL` would shrink the table file as well, but it locks the table for -reads and writes while it runs. After the purge the remaining table is small -and the space inside it is reused by new rows, so it is not needed. If you -want it anyway, run it yourself in a maintenance window with the app stopped. +reads and writes while it runs, so no job starts it. When the nightly check +warns about the bloat factor, run it yourself in a maintenance window with the +app stopped. diff --git a/prisma/migrations/0412_measurement_hot_updates/migration.sql b/prisma/migrations/0412_measurement_hot_updates/migration.sql new file mode 100644 index 000000000..b1123119c --- /dev/null +++ b/prisma/migrations/0412_measurement_hot_updates/migration.sql @@ -0,0 +1,48 @@ +-- Let an update of `measurements` that changes nothing stay a HOT update. +-- +-- Every write path that re-reads a provider window (Withings, Fitbit, Oura, +-- the Apple Health `stats:` overwrites, the reconcile paths) updates rows +-- whose values did not change, and each such update moved `updated_at` +-- (Prisma's `@updatedAt`) and `sync_version`. `updated_at` sits in the +-- sync-feed index `measurements_user_id_updated_at_id_idx`, so not one of +-- those updates could be heap-only: every one wrote a new row version on +-- another page and an entry in every index, the table grew several times +-- its live size, and every unchanged row went back down the `/api/sync/changes` +-- feed to the phone. +-- +-- 1. A row trigger keeps `updated_at` and `sync_version` as they were when +-- the update changes no other column. Such an update then touches no +-- indexed column, so Postgres can keep it on the page (HOT), and the +-- sync feed stays silent because nothing changed. An update that does +-- change something still moves both stamps, exactly as before. +-- 2. `fillfactor = 85` leaves room on each page for those in-page versions; +-- it applies to pages written from now on (a `VACUUM FULL` rewrites the +-- rest). +-- 3. Autovacuum starts at 5 % dead rows instead of 20 %, so a large table +-- does not wait for a fifth of itself to be dead before it is cleaned. +-- +-- `to_jsonb(row)` compares every column, so a column added later is covered +-- without touching this function. + +CREATE OR REPLACE FUNCTION "measurements_keep_stamps_on_noop"() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + IF (to_jsonb(NEW) - 'updated_at' - 'sync_version') + = (to_jsonb(OLD) - 'updated_at' - 'sync_version') THEN + NEW."updated_at" := OLD."updated_at"; + NEW."sync_version" := OLD."sync_version"; + END IF; + RETURN NEW; +END; +$$; + +CREATE TRIGGER "measurements_noop_update_keeps_stamps" + BEFORE UPDATE ON "measurements" + FOR EACH ROW + EXECUTE FUNCTION "measurements_keep_stamps_on_noop"(); + +ALTER TABLE "measurements" SET ( + fillfactor = 85, + autovacuum_vacuum_scale_factor = 0.05, + autovacuum_analyze_scale_factor = 0.05 +); diff --git a/prisma/schema.prisma b/prisma/schema.prisma index 6abeda554..ece8af7e0 100644 --- a/prisma/schema.prisma +++ b/prisma/schema.prisma @@ -2094,6 +2094,11 @@ model Measurement { // limit+1`. Without this index Postgres filters by `userId` then sorts // the user's entire measurement set per page; the delta drain allows // 120 pulls/min/user. See migration 0096. + // Because `updatedAt` is indexed here, an update that moves it can never + // be heap-only. The trigger `measurements_noop_update_keeps_stamps` + // (migration 0412) keeps `updatedAt` and `syncVersion` on an update that + // changes nothing else, so those stay HOT; the table also carries + // `fillfactor = 85` and a 5 % autovacuum threshold from the same migration. @@index([userId, updatedAt, id]) // `measurements_legacy_step_consolidation_idx` is a partial index // over `type = 'ACTIVITY_STEPS' AND deleted_at IS NULL` — see diff --git a/src/lib/jobs/__tests__/measurement-maintenance-schedule.test.ts b/src/lib/jobs/__tests__/measurement-maintenance-schedule.test.ts new file mode 100644 index 000000000..2a0eb28dc --- /dev/null +++ b/src/lib/jobs/__tests__/measurement-maintenance-schedule.test.ts @@ -0,0 +1,91 @@ +/** + * The nightly measurements check: what it decides from the table's + * statistics. It vacuums only when dead rows pile up and never rebuilds; the + * operator's run keeps the steps it names. + */ +import { describe, expect, it, vi } from "vitest"; + +vi.mock("@/lib/db", () => ({ + prisma: {}, + sessionOptionsDisabled: () => false, +})); + +import { + DEAD_ROW_VACUUM_FLOOR, + resolveSteps, + tableHealth, + vacuumDue, + type TableHealth, +} from "@/lib/jobs/measurement-maintenance"; + +const health = (over: Partial = {}): TableHealth => ({ + liveRows: 1_000_000, + deadRows: 0, + heapBytes: 0, + bloatFactor: null, + ...over, +}); + +describe("vacuumDue", () => { + it("vacuums once dead rows pass a tenth of the live ones", () => { + expect(vacuumDue(health({ deadRows: 100_001 }))).toBe(true); + expect(vacuumDue(health({ deadRows: 99_999 }))).toBe(false); + }); + + it("leaves a small table alone below the floor", () => { + expect( + vacuumDue(health({ liveRows: 100, deadRows: DEAD_ROW_VACUUM_FLOOR - 1 })), + ).toBe(false); + }); +}); + +describe("resolveSteps", () => { + it("gives the nightly tick a vacuum when one is due and never a rebuild", () => { + expect(resolveSteps({}, health({ deadRows: 500_000 }))).toEqual({ + vacuum: true, + reindex: false, + }); + expect(resolveSteps(undefined, health())).toEqual({ + vacuum: false, + reindex: false, + }); + }); + + it("keeps the operator's steps whatever the statistics say", () => { + expect(resolveSteps({ vacuum: false, reindex: true }, health())).toEqual({ + vacuum: false, + reindex: true, + }); + expect(resolveSteps({ vacuum: true, reindex: false }, health())).toEqual({ + vacuum: true, + reindex: false, + }); + }); +}); + +describe("tableHealth", () => { + it("measures the heap against what the live rows need", async () => { + const db = { + query: vi.fn().mockResolvedValue({ + // 1,000 rows of 72 bytes plus 28 of overhead at fill factor 0.85 + // need about 117,647 bytes; the heap holds 1,176,470. + rows: [{ live: "1000", dead: "50", heap: "1176470", width: "72" }], + }), + }; + expect(await tableHealth(db as never)).toEqual({ + liveRows: 1000, + deadRows: 50, + heapBytes: 1176470, + bloatFactor: 10, + }); + }); + + it("has no factor before the table has statistics", async () => { + const db = { + query: vi.fn().mockResolvedValue({ + rows: [{ live: "0", dead: "0", heap: "8192", width: null }], + }), + }; + expect((await tableHealth(db as never)).bloatFactor).toBeNull(); + }); +}); diff --git a/src/lib/jobs/job-outcome.ts b/src/lib/jobs/job-outcome.ts index 4a29444fd..8d450001b 100644 --- a/src/lib/jobs/job-outcome.ts +++ b/src/lib/jobs/job-outcome.ts @@ -57,6 +57,8 @@ export const JOB_FACT_ALLOWLIST: ReadonlySet = new Set([ "assessments_warmed", "auto_resolved", "backed", + // The nightly measurements check: heap size over what the live rows need. + "bloat_factor", "buckets_collapsed", "buckets_recomputed", "budget_blocked", @@ -78,6 +80,7 @@ export const JOB_FACT_ALLOWLIST: ReadonlySet = new Set([ "days_skipped", "days_skipped_no_tombstones", "days_stored", + "dead_rows", // Tombstone purge: accounts left for the next run because a restore of // the account was running. "deferred_accounts", diff --git a/src/lib/jobs/measurement-maintenance-status.ts b/src/lib/jobs/measurement-maintenance-status.ts index f611a025b..92ad00757 100644 --- a/src/lib/jobs/measurement-maintenance-status.ts +++ b/src/lib/jobs/measurement-maintenance-status.ts @@ -1,8 +1,8 @@ /** * What the admin card shows about the measurement table maintenance: the - * newest run and where it stands, whether the compaction-tombstone purge is - * still at work (the run refuses to start while it is), and the table's size - * right now. + * newest run the operator started and where it stands, whether the + * compaction-tombstone purge is still at work (the run refuses to start while + * it is), and the table's size right now. * * Read straight from `pgboss.job`, the same source `job-failures.ts` reads, * and fail-soft the same way: a web-only deployment has no `pgboss` schema, @@ -90,6 +90,9 @@ export async function readMeasurementMaintenanceStatus(): Promise'did'->>'outcome' AS outcome FROM pgboss.job WHERE name = ${MEASUREMENT_MAINTENANCE_QUEUE} + -- The operator's runs name their steps; the nightly check's tick + -- names none and reports through its wide event instead. + AND (data ? 'vacuum' OR data ? 'reindex') ORDER BY created_on DESC LIMIT 1 `, diff --git a/src/lib/jobs/measurement-maintenance.ts b/src/lib/jobs/measurement-maintenance.ts index 34eed50d9..b48ad7b96 100644 --- a/src/lib/jobs/measurement-maintenance.ts +++ b/src/lib/jobs/measurement-maintenance.ts @@ -1,12 +1,22 @@ /** - * Operator-triggered maintenance of the `measurements` table (v1.42). + * Maintenance of the `measurements` table (v1.42). * * `VACUUM (ANALYZE)` and a `REINDEX INDEX CONCURRENTLY` per index, largest * first, after the compaction-tombstone purge has shrunk the table. Neither * statement can run inside a transaction, so this is a job rather than a - * Prisma migration. Started only from `POST /api/admin/maintenance/measurements` - * (cookie-only admin), never on a schedule: the operator picks a quiet window. - * Runbook: `docs/ops/measurement-maintenance.md`. + * Prisma migration. Runbook: `docs/ops/measurement-maintenance.md`. + * + * Two ways in: + * - The operator, from `POST /api/admin/maintenance/measurements` + * (cookie-only admin), with the steps named; the rebuild only ever runs + * this way, in a window the operator picks. + * - The nightly check (`MEASUREMENT_MAINTENANCE_CRON`, a tick without + * steps). It reads the table's dead and live rows and vacuums only when + * the dead ones pass `vacuumDue`. A plain VACUUM takes no lock that stops + * reads or writes. It also measures how far the heap has grown past its + * live rows (`TableHealth.bloatFactor`); only a rewrite (`VACUUM FULL`, + * which locks the table) gives that space back, so past + * `TABLE_BLOAT_WARN_FACTOR` the run warns instead of acting. * * Why the indexes: after the purge the heap is small, but a B-tree does not * give back the pages its deleted entries held. VACUUM marks the heap space @@ -59,12 +69,51 @@ export const MEASUREMENT_MAINTENANCE_SEND_OPTIONS = { singletonKey: MEASUREMENT_MAINTENANCE_SINGLETON_KEY, } as const; -/** Which steps the operator asked for. */ +/** Which steps the operator asked for. The nightly tick names none. */ export interface MeasurementMaintenancePayload { + vacuum?: boolean; + reindex?: boolean; +} + +/** The steps a run takes, once resolved. */ +export interface MaintenanceSteps { vacuum: boolean; reindex: boolean; } +/** 03:40 Europe/Berlin, inside the nightly maintenance band. */ +export const MEASUREMENT_MAINTENANCE_CRON = "40 3 * * *"; + +/** Dead rows, as a share of live ones, past which the nightly check vacuums. */ +export const DEAD_ROW_VACUUM_SHARE = 0.1; + +/** Fewer dead rows than this are never worth a vacuum of their own. */ +export const DEAD_ROW_VACUUM_FLOOR = 10_000; + +/** + * Heap size against what the live rows need, past which the run warns that + * only a rewrite gives the space back. + */ +export const TABLE_BLOAT_WARN_FACTOR = 3; + +/** Bytes a heap row costs beyond its data: header, item pointer, alignment. */ +const ROW_OVERHEAD_BYTES = 28; + +/** The fill factor migration 0412 gives the table. */ +const FILL_FACTOR = 0.85; + +/** What the statistics say about the table's heap. */ +export interface TableHealth { + liveRows: number; + deadRows: number; + heapBytes: number; + /** + * Heap bytes over what the live rows need at the planner's average row + * width and the table's fill factor; null before the table has statistics. + */ + bloatFactor: number | null; +} + /** * Session settings of the maintenance connection. * @@ -113,7 +162,12 @@ export interface MaintenanceStep { } export interface MeasurementMaintenanceSummary { - outcome: "completed" | "refused_purge_running" | "already_running" | "failed"; + outcome: + | "completed" + | "nothing_due" + | "refused_purge_running" + | "already_running" + | "failed"; steps: MaintenanceStep[]; tableBytesBefore: number; tableBytesAfter: number; @@ -143,6 +197,62 @@ async function relationSizes( }; } +/** The table's heap as the statistics collector and the planner see it. */ +export async function tableHealth( + db: MaintenanceConnection, +): Promise { + const { rows } = await db.query<{ + live: string; + dead: string; + heap: string; + width: string | null; + }>( + `SELECT s.n_live_tup::text AS live, + s.n_dead_tup::text AS dead, + pg_relation_size('measurements'::regclass)::text AS heap, + (SELECT sum(avg_width)::text FROM pg_stats + WHERE schemaname = s.schemaname AND tablename = 'measurements') AS width + FROM pg_stat_user_tables s + WHERE s.relid = 'measurements'::regclass`, + ); + const liveRows = Number(rows[0]?.live ?? 0); + const heapBytes = Number(rows[0]?.heap ?? 0); + const width = Number(rows[0]?.width ?? 0); + const needed = (liveRows * (width + ROW_OVERHEAD_BYTES)) / FILL_FACTOR; + return { + liveRows, + deadRows: Number(rows[0]?.dead ?? 0), + heapBytes, + bloatFactor: + liveRows > 0 && width > 0 + ? Math.round((heapBytes / needed) * 10) / 10 + : null, + }; +} + +/** Whether the dead rows are worth a vacuum of their own tonight. */ +export function vacuumDue(health: TableHealth): boolean { + return ( + health.deadRows >= DEAD_ROW_VACUUM_FLOOR && + health.deadRows > health.liveRows * DEAD_ROW_VACUUM_SHARE + ); +} + +/** + * The steps a delivery takes: the operator's as asked, the nightly tick's a + * vacuum when one is due and never a rebuild. + */ +export function resolveSteps( + payload: MeasurementMaintenancePayload | undefined, + health: TableHealth, +): MaintenanceSteps { + const named = + typeof payload?.vacuum === "boolean" || + typeof payload?.reindex === "boolean"; + if (!named) return { vacuum: vacuumDue(health), reindex: false }; + return { vacuum: payload?.vacuum ?? true, reindex: payload?.reindex ?? true }; +} + async function measurementIndexes( db: MaintenanceConnection, ): Promise { @@ -223,7 +333,7 @@ async function dropInvalidCopies( */ export async function runMeasurementMaintenance( db: MaintenanceConnection, - payload: MeasurementMaintenancePayload, + payload: MaintenanceSteps, options: { shouldStop?: () => boolean } = {}, ): Promise { const before = await relationSizes(db); @@ -240,6 +350,8 @@ export async function runMeasurementMaintenance( indexBytesAfter: after.indexes, }); + if (!payload.vacuum && !payload.reindex) return summary("nothing_due"); + // The queue table exists wherever a worker has started; a database without // one has no purge to wait for. const { rows: queueTable } = await db.query<{ present: boolean }>( @@ -327,10 +439,6 @@ export async function handleMeasurementMaintenance( jobs: Job[], ): Promise { const job = jobs[0]; - const payload: MeasurementMaintenancePayload = { - vacuum: job?.data?.vacuum ?? true, - reindex: job?.data?.reindex ?? true, - }; return withBackgroundEvent("job.measurement_maintenance", async (evt) => { const startedAt = Date.now(); const guarded = await withJobLock( @@ -350,7 +458,10 @@ export async function handleMeasurementMaintenance( await client.query(statement); } } - return await runMeasurementMaintenance(client, payload); + const health = await tableHealth(client); + const steps = resolveSteps(job?.data, health); + const summary = await runMeasurementMaintenance(client, steps); + return { summary, health, steps }; } finally { await client .end() @@ -363,20 +474,26 @@ export async function handleMeasurementMaintenance( } }, ); - const result: MeasurementMaintenanceSummary = guarded.ran - ? guarded.result - : { - outcome: "already_running", - steps: [], - tableBytesBefore: 0, - tableBytesAfter: 0, - indexBytesBefore: 0, - indexBytesAfter: 0, - }; + if (!guarded.ran) { + evt.addMeta("maintenance_outcome", "already_running"); + return jobDone({ outcome: "already_running" }); + } + const { summary: result, health, steps } = guarded.result; evt.addMeta("maintenance_outcome", result.outcome); - evt.addMeta("maintenance_vacuum", payload.vacuum); - evt.addMeta("maintenance_reindex", payload.reindex); + evt.addMeta("maintenance_vacuum", steps.vacuum); + evt.addMeta("maintenance_reindex", steps.reindex); + evt.addMeta("maintenance_live_rows", health.liveRows); + evt.addMeta("maintenance_dead_rows", health.deadRows); + evt.addMeta("maintenance_heap_bytes", health.heapBytes); + if (health.bloatFactor !== null) { + evt.addMeta("maintenance_bloat_factor", health.bloatFactor); + if (health.bloatFactor > TABLE_BLOAT_WARN_FACTOR) { + evt.addWarning( + `measurements heap is ${health.bloatFactor}x what its live rows need; only VACUUM FULL in a maintenance window gives the space back`, + ); + } + } evt.addMeta("maintenance_table_bytes_before", result.tableBytesBefore); evt.addMeta("maintenance_table_bytes_after", result.tableBytesAfter); evt.addMeta("maintenance_index_bytes_before", result.indexBytesBefore); @@ -412,6 +529,9 @@ export async function handleMeasurementMaintenance( processed: result.steps.filter((s) => s.ok).length, failed: result.steps.filter((s) => !s.ok).length, duration_ms: Date.now() - startedAt, + dead_rows: health.deadRows, + // Zero before the table has statistics to measure against. + bloat_factor: health.bloatFactor ?? 0, }; if (result.outcome === "failed") { const firstError = result.steps.find((s) => !s.ok); diff --git a/src/lib/jobs/queue-runtime.ts b/src/lib/jobs/queue-runtime.ts index 12d817ce7..e4922f110 100644 --- a/src/lib/jobs/queue-runtime.ts +++ b/src/lib/jobs/queue-runtime.ts @@ -391,7 +391,7 @@ export const QUEUE_RUNTIME: Readonly> = { expiryVia: ["MEASUREMENT_MAINTENANCE_SEND_OPTIONS"], stop: { oneShot: - "VACUUM and REINDEX CONCURRENTLY cannot be resumed mid-statement; the send sets retryLimit 0 and the operator re-triggers.", + "VACUUM and REINDEX CONCURRENTLY cannot be resumed mid-statement; the send and the nightly tick set retryLimit 0, and the next night or the operator runs it again.", }, exclusive: { marker: "retryLimit: 0", @@ -400,7 +400,7 @@ export const QUEUE_RUNTIME: Readonly> = { fn: "MEASUREMENT_MAINTENANCE_SEND_OPTIONS", }, }, - why: "Rebuilds every index of the largest table, one at a time.", + why: "The operator's run rebuilds every index of the largest table, one at a time; the nightly check is one VACUUM at most.", }, "pulse-day-mean": { runtime: "long", diff --git a/src/lib/jobs/reminder/register-maintenance.ts b/src/lib/jobs/reminder/register-maintenance.ts index 4f25c5679..19969b7c2 100644 --- a/src/lib/jobs/reminder/register-maintenance.ts +++ b/src/lib/jobs/reminder/register-maintenance.ts @@ -80,7 +80,9 @@ import { type MeasurementFoldRepairPayload, } from "@/lib/jobs/measurement-fold-repair"; import { + MEASUREMENT_MAINTENANCE_CRON, MEASUREMENT_MAINTENANCE_QUEUE, + MEASUREMENT_MAINTENANCE_SEND_OPTIONS, handleMeasurementMaintenance, type MeasurementMaintenancePayload, } from "@/lib/jobs/measurement-maintenance"; @@ -507,9 +509,9 @@ const allQueues = [ // their window; the purge above waits for it. Without this entry the boot // enqueue silently no-ops and the purge never runs. MEASUREMENT_FOLD_REPAIR_QUEUE, - // v1.42 — operator-triggered VACUUM / REINDEX of `measurements`. No - // cron: the admin route enqueues it. Without this entry the trigger - // silently no-ops. + // v1.42 — VACUUM / REINDEX of `measurements`: the admin route enqueues the + // operator's run, and a nightly check vacuums when dead rows pile up. + // Without this entry both silently no-op. MEASUREMENT_MAINTENANCE_QUEUE, // v1.43 — the pulse day-mean dry run and rebuild, admin-triggered only. PULSE_DAY_MEAN_QUEUE, @@ -634,6 +636,14 @@ const allQueues = [ // `retryLimit: 0` in its own options for the other reason: it pages each // failed account itself, and a retry would page the same accounts again. const schedules: ScheduleEntry[] = [ + // v1.43.7 — nightly 03:40 check of `measurements`: a plain VACUUM when the + // dead rows pass the threshold, never a rebuild. Same singleton key as the + // operator's run, so the two collapse. + [ + MEASUREMENT_MAINTENANCE_QUEUE, + MEASUREMENT_MAINTENANCE_CRON, + { ...MEASUREMENT_MAINTENANCE_SEND_OPTIONS }, + ], // v1.25 (W-ENV) — daily 02:10 Europe/Berlin discovery tick (empty payload) // that fans out one per-user environment fetch per opted-in account. [ENVIRONMENT_FETCH_QUEUE, ENVIRONMENT_FETCH_CRON], @@ -849,11 +859,11 @@ const queuePolicies: QueuePolicyTable = { reason: "Fixed singleton key, admin-triggered. Two concurrent corpus rotations must never overlap; the route already reports a suppressed send back as alreadyQueued.", }, - // v1.42 — operator-triggered VACUUM / REINDEX of `measurements`. + // v1.42 — VACUUM / REINDEX of `measurements`, operator or nightly. [MEASUREMENT_MAINTENANCE_QUEUE]: { policy: "exclusive", reason: - "Fixed singleton key, admin-triggered. Two maintenance passes would rebuild the same indexes side by side; the route reports a suppressed send back as enqueued: false.", + "Fixed singleton key on the admin send and the nightly tick alike. Two maintenance passes would vacuum or rebuild side by side; the route reports a suppressed send back as enqueued: false.", }, // v1.43 — the pulse day-mean dry run and rebuild. [PULSE_DAY_MEAN_QUEUE]: { diff --git a/src/lib/jobs/reminder/registrar-shared.ts b/src/lib/jobs/reminder/registrar-shared.ts index b180aadc3..1b4f7cfce 100644 --- a/src/lib/jobs/reminder/registrar-shared.ts +++ b/src/lib/jobs/reminder/registrar-shared.ts @@ -312,6 +312,7 @@ export const SCHEDULED_AND_SENT_QUEUES: ReadonlySet = new Set([ "geolite2-fetch", "insight-pregenerate", "intake-slot-dedup", + "measurement-maintenance", "med-notes-encryption-backfill", "nightscout-sync", "note-encryption-backfill", diff --git a/tests/integration/measurement-hot-update.test.ts b/tests/integration/measurement-hot-update.test.ts new file mode 100644 index 000000000..7c9963da3 --- /dev/null +++ b/tests/integration/measurement-hot-update.test.ts @@ -0,0 +1,120 @@ +/** + * Migration 0412: an update of `measurements` that changes nothing keeps its + * stamps and stays on its page. + * + * `updated_at` sits in the sync-feed index, so an update that moved it could + * never be heap-only, and every provider re-sync that rewrote unchanged rows + * grew the table by a full row version and an entry in every index. Proven + * here against Postgres: the no-op update keeps `updated_at` and + * `sync_version` and is counted as HOT in the transaction's own statistics, + * a real change still moves both, and the nightly check reads the table's + * statistics and vacuums only when dead rows pass its threshold. + */ +import { Client } from "pg"; +import { beforeEach, describe, expect, it } from "vitest"; + +import { + resolveSteps, + runMeasurementMaintenance, + tableHealth, +} from "@/lib/jobs/measurement-maintenance"; + +import { getPrismaClient, truncateAllTables } from "./setup"; + +const prisma = getPrismaClient(); + +async function seedReading() { + const user = await prisma.user.create({ + data: { + username: "hot-update", + email: "hot-update@example.test", + role: "USER", + timezone: "UTC", + }, + }); + return prisma.measurement.create({ + data: { + userId: user.id, + type: "WEIGHT", + value: 72.3, + unit: "kg", + measuredAt: new Date("2026-10-01T07:00:00.000Z"), + source: "WITHINGS", + }, + }); +} + +beforeEach(async () => { + await truncateAllTables(prisma); +}); + +describe("an update that changes nothing", () => { + it("keeps updated_at and sync_version and stays on its page", async () => { + const row = await seedReading(); + await new Promise((r) => setTimeout(r, 20)); + + const hot = await prisma.$transaction(async (tx) => { + await tx.measurement.update({ + where: { id: row.id }, + data: { value: 72.3, deletedAt: null, syncVersion: { increment: 1 } }, + }); + const [stats] = await tx.$queryRaw>` + SELECT n_tup_hot_upd AS hot, n_tup_upd AS upd + FROM pg_stat_xact_user_tables WHERE relname = 'measurements'`; + return stats; + }); + expect(Number(hot.upd)).toBe(1); + expect(Number(hot.hot)).toBe(1); + + const after = await prisma.measurement.findUniqueOrThrow({ + where: { id: row.id }, + }); + expect(after.updatedAt.getTime()).toBe(row.updatedAt.getTime()); + expect(after.syncVersion).toBe(row.syncVersion); + }); + + it("still moves both stamps when a value changes", async () => { + const row = await seedReading(); + await new Promise((r) => setTimeout(r, 20)); + await prisma.measurement.update({ + where: { id: row.id }, + data: { value: 72.4, syncVersion: { increment: 1 } }, + }); + const after = await prisma.measurement.findUniqueOrThrow({ + where: { id: row.id }, + }); + expect(after.updatedAt.getTime()).toBeGreaterThan(row.updatedAt.getTime()); + expect(after.syncVersion).toBe(row.syncVersion + 1); + }); + + it("gives the table room on each page and an earlier autovacuum", async () => { + const [rel] = await prisma.$queryRaw>` + SELECT reloptions AS options FROM pg_class WHERE relname = 'measurements'`; + expect(rel.options).toEqual( + expect.arrayContaining([ + "fillfactor=85", + "autovacuum_vacuum_scale_factor=0.05", + ]), + ); + }); +}); + +describe("the nightly check", () => { + it("reads the statistics and does nothing while no vacuum is due", async () => { + await seedReading(); + const db = new Client({ connectionString: process.env.DATABASE_URL }); + await db.connect(); + try { + await db.query("ANALYZE measurements"); + const health = await tableHealth(db); + expect(health.heapBytes).toBeGreaterThan(0); + const steps = resolveSteps({}, health); + expect(steps).toEqual({ vacuum: false, reindex: false }); + const summary = await runMeasurementMaintenance(db, steps); + expect(summary.outcome).toBe("nothing_due"); + expect(summary.steps).toEqual([]); + } finally { + await db.end(); + } + }); +});