HealthLog/docs/ops/measurement-maintenance.md

Measurement table maintenance

Since v1.42 the nightly folds of Apple Health data delete the raw samples they fold into hourly or daily rows, instead of keeping them as deleted rows for 75 days. On an instance that has synced Apple Health for a while, those leftovers ("compaction tombstones") were most of the measurements table and of its indexes. Three pieces deal with them, in this order:

1. The fold repair runs by itself, once per account, before anything is deleted. Up to v1.42 the folds could fold a day in two runs, and the second run stored the mean of the later part of the day only. The repair recomputes those hourly and daily means from all of the day's samples, including the compaction tombstones, and corrects a stored mean only when it differs. It does so only where every sample of the day is provably still in the table: the last 75 days for daily means, and for hourly means (folded 90 days late) the 75 days before that. Older days stay as they are, because the deleted rows of their samples have passed the 75-day retention and what is left can be a fragment of the day. When it has finished an account it records that in measurement_fold_repairs.
2. The backlog purge runs by itself, after the repair. Every worker boot queues one compaction-tombstone-purge job. It deletes the leftovers 5,000 rows at a time, one account at a time, pausing briefly between batches, and stops after 40 batches; a run that stopped with work left queues the next one a minute later. Rows a person deleted are never touched by it: they keep the 75-day retention of the nightly tombstone cleanup. The purge leaves an account alone until the repair has finished it, and a boot queues no purge run while the repair has finished no account.
3. The table maintenance is yours to start, after the purge. It runs VACUUM (ANALYZE) on measurements and then rebuilds each of its indexes with REINDEX INDEX CONCURRENTLY, largest first. The purge leaves the indexes full of empty pages, and only a rebuild gives that space back.

Repair before purge

The order matters because the purge deletes what the repair reads. Do not delete compaction tombstones by hand, and do not insert rows into measurement_fold_repairs to hurry the purge along: a mean the repair has not seen keeps its wrong value for good once its tombstones are gone.

The repair reports every run on the wide event job.measurement_fold_repair (fold_repair_means_checked, fold_repair_means_corrected, fold_repair_resting_corrected, fold_repair_skipped_beyond_horizon for the older windows it left as they are, fold_repair_samples_absorbed for live samples it took into a checked mean, and fold_repair_by_type with the same counts per type; counts only, never a value). A second run over an account corrects nothing. It is finished when every account has a row:

SELECT count(*) AS accounts_left
FROM users u
LEFT JOIN measurement_fold_repairs r ON r.user_id = u.id
WHERE r.user_id IS NULL;

Each run works on one account, a day at a time, and stops on its time budget with a follow-up that resumes at the next day; an account under restore is tried again five minutes later. While accounts are left, purge runs report them as compaction_purge_awaiting_repair_accounts.

A restore of a backup exported before the account's repair finished writes the compaction tombstones back, removes the account's row and queues the repair again; the purge follows once the repair is through.

Is the purge finished?

The purge reports every run on the wide event job.compaction_tombstone_purge (compaction_purge_deleted, compaction_purge_drained, compaction_purge_deferred_accounts). It is finished when a run reports drained: true and deletes nothing. From psql:

SELECT state, count(*)
FROM pgboss.job
WHERE name = 'compaction-tombstone-purge'
GROUP BY state;

No created, retry or active rows means nothing is queued or running. The maintenance refuses to start while one is.

Before you start

• Pick a quiet window. Reads and writes keep working during the rebuild, but each rebuild reads the whole table twice and writes a full new copy of the index.

• Free disk. A concurrent rebuild needs room for one more copy of the index it is rebuilding, plus the WAL it writes. Check the largest index:

  SELECT c.relname, pg_size_pretty(pg_relation_size(c.oid))
  FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
  WHERE i.indrelid = 'measurements'::regclass
  ORDER BY pg_relation_size(c.oid) DESC
  LIMIT 3;
  

  Have at least twice the largest size free on the database volume.

• Take a backup: a database dump, as in Updates, as before any operation on the whole table.

Starting it

In the admin area, or with a signed-in admin session:

curl -X POST https://<your-instance>/api/admin/maintenance/measurements \
  -H 'content-type: application/json' \
  --cookie '__Host-healthlog_session=<session>' \
  -d '{"vacuum": true, "reindex": true}'

On a plain-HTTP install the cookie is named healthlog_session.

The answer is 202 with { "enqueued": true }, or enqueued: false when a run is already queued or running. A Bearer token is refused: the route is cookie-only, like every admin route. Without a background worker the route answers 503.

Either step can be left out ("reindex": false runs only the vacuum).

What it does, and what stops it

The run uses a connection of its own: no statement timeout (a rebuild of a large index takes longer than the app's 60 seconds), maintenance_work_mem of 128 MB for this session only, and a lock timeout of five minutes, so a rebuild that cannot get its lock gives up instead of holding writers behind it.

• It refuses to start while the purge is queued or running.
• Before the first rebuild it drops any invalid <index>_ccnew copy an interrupted earlier run left behind.
• If a rebuild fails, most likely for lack of disk, it drops that rebuild's half-built copy at once and stops. The remaining indexes keep working as they are. Free some space and start it again; finished indexes are simply rebuilt once more.
• One run at a time. It is never retried automatically.

Reading the result

The run's wide event job.measurement_maintenance carries maintenance_outcome (completed, nothing_due for the nightly check, refused_purge_running, failed), the live and dead rows the run started from, the table and index sizes before and after, and maintenance_steps, one entry per step with its target, size before and after, and duration. The same sizes from psql:

SELECT pg_size_pretty(pg_total_relation_size('measurements')) AS total,
       pg_size_pretty(pg_indexes_size('measurements')) AS indexes;

The nightly check

Since v1.43.7 the same queue also runs by itself at 03:40 (Europe/Berlin). That run names no steps and changes nothing beyond dropping a retired index (next section): vacuuming the table is autovacuum's job, which migration 0412 starts at 5 % dead rows instead of Postgres's 20 %. The check reads the table's live and dead rows and warns (maintenance_autovacuum_behind on the wide event) when there are at least 10,000 dead rows and more than one for every five live ones, four times the point at which autovacuum should already have run. That means autovacuum is not keeping up: look for a transaction left open for hours (pg_stat_activity, state = 'idle in transaction') or for autovacuum workers busy elsewhere. The check never rebuilds an index. It shares the singleton key with the run you start, so the two never overlap. The admin card keeps showing only the runs you started.

The check also measures how far the table file has grown past what its live rows need (maintenance_bloat_factor on the wide event, as a factor such as 1.3; bloat_percent in the job result, as whole percent such as 130, because a job fact is a whole number). Past 3 it adds a warning to the event: a vacuum makes the space reusable but does not shrink the file, and only VACUUM FULL does (see below).

Why the table grew in the first place: until v1.43.7 an update that changed nothing still moved updated_at, which sits in an index, so no update could stay on its page. Migration 0412 keeps updated_at and sync_version on such an update and gives the table fillfactor = 85 and a 5 % autovacuum threshold. To check that updates stay on their page:

SELECT n_tup_upd, n_tup_hot_upd, n_live_tup, n_dead_tup,
       pg_size_pretty(pg_relation_size('measurements')) AS heap
FROM pg_stat_user_tables WHERE relname = 'measurements';

n_tup_hot_upd should now grow along with n_tup_upd. The fill factor applies to pages written after the migration; a VACUUM FULL after the upgrade applies it to the whole table.

Retired indexes

An index the schema no longer declares has to leave the database too, and on measurements the obvious migration is the wrong one. A plain DROP INDEX needs an exclusive lock on the table; DROP INDEX CONCURRENTLY blocks nobody but waits, without limit, for every transaction already running on the table. A migration runs at boot, so either one waits for whatever holds the table (a pg_dump, a long backfill, a VACUUM FULL); past the container's start period the container is replaced with the migration half done, and the next boot stops at P3009 until someone runs prisma migrate resolve.

So an index is retired in two halves (v1.43.8, migration 0416, for measurements_user_id_type_measured_at_idx):

1. The migration tries a plain DROP INDEX IF EXISTS inside a DO block with a two-second lock_timeout and catches the timeout (EXCEPTION WHEN lock_not_available). When nothing holds the table the index is gone at once; otherwise the migration still succeeds and the index stays. The longest the table's other statements can wait behind the attempt is those two seconds. A new database and Prisma's shadow database always take the lock, so the migration history ends without the index, exactly as the schema does, and prisma migrate dev / migrate diff see no drift and generate nothing.
2. The index name goes into RETIRED_MEASUREMENT_INDEXES in src/lib/jobs/retired-measurement-indexes.mjs. Every run of this job, the nightly 03:40 check included, drops each name still present with DROP INDEX CONCURRENTLY IF EXISTS, waiting at most 60 seconds for its locks. A run that runs out of time leaves the index (possibly marked invalid, which the next drop finishes), sets maintenance_retired_index_pending on the wide event with a warning, and the next night tries again. The index costs writes until then, as it did before the release; nothing reads it.

Until the job has run, the database holds one index the schema does not declare. Nothing compares the two at runtime, and the only place they are compared, a migrate diff against the migration history, never sees it. To check by hand:

SELECT indexname FROM pg_indexes
 WHERE tablename = 'measurements'
   AND indexname = 'measurements_user_id_type_measured_at_idx';

No row: it is gone. To drop it now rather than at night, start a run from the admin card (any steps), or run the same statement yourself:

SET lock_timeout = '60s';
DROP INDEX CONCURRENTLY IF EXISTS "measurements_user_id_type_measured_at_idx";

When the app connects as its restricted role (database-role.md), the job may vacuum and rebuild (MAINTAIN) but not drop: DROP INDEX belongs to the table's owner. A run then skips the drops, of retired indexes and of the invalid copy a failed rebuild leaves, names what it found under maintenance_drops_left_for_boot with a warning, and the next start drops each as the owner with DROP INDEX CONCURRENTLY, waiting at most five seconds. Until then a failed rebuild's copy holds its disk; restart the container, or drop it by hand as the owner with the statement above.

For the next large index, follow the same two halves. src/__tests__/migration-lock-timeout-guard.test.ts refuses a DROP INDEX CONCURRENTLY in any migration and requires every retired name to have a migration that bounds its wait and catches the timeout.

"could not resize shared memory segment"

Up to v1.42.0 the run could stop at once with could not resize shared memory segment ... No space left on device. Postgres ran the vacuum in parallel and kept its working memory in /dev/shm, which Docker limits to 64 MB per container unless shm_size is set. From v1.42.1 the maintenance connection runs single-process and needs no shared segment, and the bundled docker-compose.yml gives the database shm_size: 256mb. On v1.42.0, add shm_size: 256mb to the db service and restart the database container.

Not part of this

VACUUM FULL would shrink the table file as well, but it locks the table for reads and writes while it runs, so no job starts it. When the nightly check warns about the bloat factor, run it yourself in a maintenance window with the app stopped.