diff --git a/docs/ops/deploy.md b/docs/ops/deploy.md index 5a12edd69..15ce03a1d 100644 --- a/docs/ops/deploy.md +++ b/docs/ops/deploy.md @@ -140,6 +140,22 @@ An INVALID index whose migration is already recorded applied is not touched at boot. The boot names it on stderr with `WARNING — index … is INVALID`; rebuild it with `REINDEX INDEX CONCURRENTLY "";` when the database is quiet. +### The app's database role + +The steps above run as the owner `DATABASE_URL` names: the dump, the +migrations and, after them, the setup of `healthlog_app`, the role the app +then connects as ([database-role.md](../self-hosting/database-role.md)). The +app never runs a migration. A migration that creates a table needs nothing +more: the setup grants the role every table it lacks before the app starts. + +On the first start of the release that brings the role, the setup hands the +existing `pgboss` tables to it, which takes each table's lock for a moment. +Stop the old container (and a separate worker) before the new one starts, as +for any update with migrations; a lock that does not come within 15 seconds +leaves that start on the owner with a notice, and the next start tries again. +A web container and a worker container that start together wait for each +other on an advisory lock and derive the same password. + ### v1.46.0 notes - **Postgres 18.** The compose file moves the bundled database from diff --git a/docs/ops/disaster-recovery.md b/docs/ops/disaster-recovery.md index 0e9bb9364..a6c40c94c 100644 --- a/docs/ops/disaster-recovery.md +++ b/docs/ops/disaster-recovery.md @@ -147,6 +147,14 @@ different. Sign in. The app's own key check runs at start and refuses to serve on a key that does not open the data. +The restore runs as the owner `DATABASE_URL` names, and the restored tables +belong to it. The app connects as `healthlog_app` +([database-role.md](../self-hosting/database-role.md)): the start in step 5 +grants that role the restored tables and hands it the queue schema before the +app runs. A dump carries grants to `healthlog_app`; `db-restore` leaves them +out (`--no-owner --no-privileges`). Restoring with `pg_restore` by hand, pass +the same two flags, or it fails on a database without the role. + When the dumps are still in the server's `backups` volume (the database volume was lost, the backups volume was not), step 4 is `docker compose run --rm db-restore latest`. diff --git a/docs/ops/measurement-maintenance.md b/docs/ops/measurement-maintenance.md index e0bce3094..53e827ddb 100644 --- a/docs/ops/measurement-maintenance.md +++ b/docs/ops/measurement-maintenance.md @@ -214,7 +214,7 @@ So an index is retired in two halves (v1.43.8, migration 0416, for exactly as the schema does, and `prisma migrate dev` / `migrate diff` see no drift and generate nothing. 2. The index name goes into `RETIRED_MEASUREMENT_INDEXES` in - `src/lib/jobs/measurement-maintenance.ts`. Every run of this job, the + `src/lib/jobs/retired-measurement-indexes.mjs`. Every run of this job, the nightly 03:40 check included, drops each name still present with `DROP INDEX CONCURRENTLY IF EXISTS`, waiting at most 60 seconds for its locks. A run that runs out of time leaves the index (possibly marked @@ -242,6 +242,16 @@ SET lock_timeout = '60s'; DROP INDEX CONCURRENTLY IF EXISTS "measurements_user_id_type_measured_at_idx"; ``` +When the app connects as its restricted role +([database-role.md](../self-hosting/database-role.md)), the job may vacuum and +rebuild (`MAINTAIN`) but not drop: `DROP INDEX` belongs to the table's owner. +A run then skips the drops, of retired indexes and of the invalid copy a +failed rebuild leaves, names what it found under +`maintenance_drops_left_for_boot` with a warning, and the next start drops +each as the owner with `DROP INDEX CONCURRENTLY`, waiting at most five +seconds. Until then a failed rebuild's copy holds its disk; restart the +container, or drop it by hand as the owner with the statement above. + For the next large index, follow the same two halves. `src/__tests__/migration-lock-timeout-guard.test.ts` refuses a `DROP INDEX CONCURRENTLY` in any migration and requires every retired name to diff --git a/docs/self-hosting/database-role.md b/docs/self-hosting/database-role.md new file mode 100644 index 000000000..4b8ea1403 --- /dev/null +++ b/docs/self-hosting/database-role.md @@ -0,0 +1,112 @@ +# The app's database role + +The container connects to Postgres under two names. The user in +`DATABASE_URL` (on the bundled compose file, `healthlog`, which the official +Postgres image makes a superuser) owns the schema: the entrypoint uses it for +the dump before an update, the migrations and the restore. The app itself, +web and worker, connects as `healthlog_app`, a role the entrypoint sets up +after the migrations on every start. Nothing to configure: the first start of +a release with this change creates it. + +The point is the blast radius of a bug. If a dependency of the app were ever +made to run code inside the container, the database connection it found would +be one that can read and write rows, not one that can drop tables, rewrite the +schema or reach the server's files. + +## What the role may and may not do + +| May | May not | +| ------------------------------------------------------------------------- | --------------------------------------------------------------- | +| read, insert, update and delete rows in every table of the app's schema | create, alter or drop tables, indexes, types or functions there | +| use and advance its sequences | write `_prisma_migrations` (it may read it) | +| `VACUUM`, `ANALYZE` and `REINDEX` the `measurements` table (`MAINTAIN`) | create roles, schemas or databases | +| own the `pgboss` schema, where the job queue keeps and changes its tables | act as a superuser: no `COPY … PROGRAM`, no `pg_read_file` | +| create temporary tables | become the owner (`SET ROLE`) | + +The queue schema is the role's because pg-boss creates and migrates its own +tables, functions and per-queue partitions while the app runs. Nothing of the +app's data lives there. The start also takes `CREATE` on the app's schema +away from `PUBLIC`, which Postgres 15 and newer already do for a new database +but a database first created on an older major still allows. + +Its password is derived from `API_TOKEN_HMAC_KEY`, so a separate worker +container ([scaling.md](scaling.md)) arrives at the same one without a new +variable, and it is stored as a SCRAM hash. The owner's URL is replaced, not +kept beside the role's, so it is not in the app's environment. +`docker compose exec app …` still starts with the container's configured +environment, so the operator scripts (`reset-password.mjs` and the others) +run as the owner as before. + +Each start re-applies the grants, so a table the new migrations created is +covered before the app starts, and a restored database is put right on the +start after the restore. + +## What the start log says + +When it is in place: + +```text +HealthLog: the app connects as healthlog_app: rows and the queue, no schema changes (DB_APP_ROLE=off turns this off) +``` + +When it is not, one notice names the reason, and the app runs as the +`DATABASE_URL` user, exactly as every earlier release did: + +```text +HealthLog: notice: the app connects as healthlog, the user DATABASE_URL names, not as the restricted role healthlog_app: . … +``` + +The reasons: + +- **The user may not create roles.** A managed database (RDS, Cloud SQL, + Supabase and the like) or a database server where you gave HealthLog an + account without `CREATEROLE`. Either leave it, or create the split yourself + (below) and set `DB_APP_ROLE=off`. +- **Postgres older than 17.** The maintenance job's `VACUUM` and index rebuild + need the `MAINTAIN` privilege, which Postgres 17 introduced; before it only + the table's owner may run them. The bundled database is Postgres 18; see + [postgres-major-upgrade.md](../ops/postgres-major-upgrade.md) for one you + run yourself. +- **The role cannot log in.** A `pg_hba.conf` that lists users by name needs a + line for `healthlog_app` beside the one for the owner. +- **A role named `healthlog_app` exists that HealthLog did not create.** The + start leaves a role it does not recognise alone. +- **Setting it up failed**, most often a lock it waited 15 seconds for: the + first start hands the existing queue tables to the role, and another + container still using them holds it up. The next start tries again. + +None of these stops the start. + +## Turning it off + +Set `DB_APP_ROLE=off` in `.env` and recreate the container. The app then +connects as the `DATABASE_URL` user. Use it when you manage database roles +yourself, including when `DATABASE_URL` already names a restricted role of +your own; then that role needs what the table above lists, and the +migrations need a user that owns the schema. + +The role stays in the database when you turn it off and does no harm. To +remove it, as the owner: + +```sql +REASSIGN OWNED BY healthlog_app TO healthlog; +DROP OWNED BY healthlog_app; +DROP ROLE healthlog_app; +``` + +## Index drops at start + +`DROP INDEX` belongs to a table's owner; `MAINTAIN` does not include it. The +measurement maintenance therefore no longer drops a retired index or the +invalid copy a failed rebuild leaves; it names them on its run and the next +start drops them as the owner, waiting at most five seconds for each +([measurement-maintenance.md](../ops/measurement-maintenance.md)). + +## Dumps and restores + +The nightly dump runs as the role and reads everything: the app's schema +through its grants, the queue schema as its owner. A dump carries the grants +to `healthlog_app`; `db-restore` leaves them and the ownership out +(`--no-owner --no-privileges`), restores as the owner, and the next start +grants again. A restore of your own with `pg_restore` should do the same, or +it fails on a database without the role. diff --git a/docs/self-hosting/scaling.md b/docs/self-hosting/scaling.md index 6ecac0731..3bc36d23b 100644 --- a/docs/self-hosting/scaling.md +++ b/docs/self-hosting/scaling.md @@ -129,6 +129,15 @@ looks healthy. Deploy both from the same image tag (`HEALTHLOG_IMAGE_REF` is shared by `app` and the `app-worker` block) and restart the worker together with the web containers. +## Database role + +Both containers run the entrypoint, and both set up `healthlog_app`, the +database role the app connects as +([database-role.md](database-role.md)). They take turns on an advisory lock +and derive the same password from `API_TOKEN_HMAC_KEY`, which the shared +environment block gives both, so a worker that starts after the web container +changes nothing the web container relies on. + ## Healthchecks - `app` (web) healthcheck: `wget /api/health` every 30s.