HealthLog/docs/ops/measurement-maintenance.md

Measurement table maintenance

Since v1.42 the nightly folds of Apple Health data delete the raw samples they fold into hourly or daily rows, instead of keeping them as deleted rows for 75 days. On an instance that has synced Apple Health for a while, those leftovers ("compaction tombstones") were most of the measurements table and of its indexes. Two pieces clean them up:

1. The backlog purge runs by itself. Every worker boot queues one compaction-tombstone-purge job. It deletes the leftovers 5,000 rows at a time, one account at a time, pausing briefly between batches, and stops after 40 batches; a run that stopped with work left queues the next one a minute later. Rows a person deleted are never touched by it: they keep the 75-day retention of the nightly tombstone cleanup.
2. The table maintenance is yours to start, after the purge. It runs VACUUM (ANALYZE) on measurements and then rebuilds each of its indexes with REINDEX INDEX CONCURRENTLY, largest first. The purge leaves the indexes full of empty pages, and only a rebuild gives that space back.

Is the purge finished?

The purge reports every run on the wide event job.compaction_tombstone_purge (compaction_purge_deleted, compaction_purge_drained, compaction_purge_deferred_accounts). It is finished when a run reports drained: true and deletes nothing. From psql:

SELECT state, count(*)
FROM pgboss.job
WHERE name = 'compaction-tombstone-purge'
GROUP BY state;

No created, retry or active rows means nothing is queued or running. The maintenance refuses to start while one is.

Before you start

• Pick a quiet window. Reads and writes keep working during the rebuild, but each rebuild reads the whole table twice and writes a full new copy of the index.

• Free disk. A concurrent rebuild needs room for one more copy of the index it is rebuilding, plus the WAL it writes. Check the largest index:

  SELECT c.relname, pg_size_pretty(pg_relation_size(c.oid))
  FROM pg_index i JOIN pg_class c ON c.oid = i.indexrelid
  WHERE i.indrelid = 'measurements'::regclass
  ORDER BY pg_relation_size(c.oid) DESC
  LIMIT 3;
  

  Have at least twice the largest size free on the database volume.

• Take a backup (Settings, Data, Backups), as before any operation on the whole table.

Starting it

In the admin area, or with a signed-in admin session:

curl -X POST https://<your-instance>/api/admin/maintenance/measurements \
  -H 'content-type: application/json' \
  --cookie 'healthlog_session=<session>' \
  -d '{"vacuum": true, "reindex": true}'

The answer is 202 with { "enqueued": true }, or enqueued: false when a run is already queued or running. A Bearer token is refused: the route is cookie-only, like every admin route. Without a background worker the route answers 503.

Either step can be left out ("reindex": false runs only the vacuum).

What it does, and what stops it

The run uses a connection of its own: no statement timeout (a rebuild of a large index takes longer than the app's 60 seconds), maintenance_work_mem of 128 MB for this session only, and a lock timeout of five minutes, so a rebuild that cannot get its lock gives up instead of holding writers behind it.

• It refuses to start while the purge is queued or running.
• Before the first rebuild it drops any invalid <index>_ccnew copy an interrupted earlier run left behind.
• If a rebuild fails, most likely for lack of disk, it drops that rebuild's half-built copy at once and stops. The remaining indexes keep working as they are. Free some space and start it again; finished indexes are simply rebuilt once more.
• One run at a time. It is never retried automatically.

Reading the result

The run's wide event job.measurement_maintenance carries maintenance_outcome (completed, refused_purge_running, failed), the table and index sizes before and after, and maintenance_steps, one entry per step with its target, size before and after, and duration. The same sizes from psql:

SELECT pg_size_pretty(pg_total_relation_size('measurements')) AS total,
       pg_size_pretty(pg_indexes_size('measurements')) AS indexes;

Not part of this

VACUUM FULL would shrink the table file as well, but it locks the table for reads and writes while it runs. After the purge the remaining table is small and the space inside it is reused by new rows, so it is not needed. If you want it anyway, run it yourself in a maintenance window with the app stopped.