diff --git a/.env.example b/.env.example index 61a0289df..ccdedbb3b 100644 --- a/.env.example +++ b/.env.example @@ -39,8 +39,9 @@ # Left commented here so that append does not create a duplicate key. Uncomment # and set it directly instead if you prefer to edit the file by hand. -# The container runs the app as `healthlog_app`, a database role it creates on -# every start that reads and writes rows but cannot change the schema. The +# The container runs the app as `healthlog_app__`, a database +# role it creates on every start that reads and writes rows but cannot change +# the schema. The # migrations keep the DATABASE_URL user. "off" keeps the app on that user, if # you manage database roles yourself (docs/self-hosting/database-role.md). # DB_APP_ROLE="off" diff --git a/.env.production.example b/.env.production.example index a028871fa..ae0c3eef3 100644 --- a/.env.production.example +++ b/.env.production.example @@ -527,9 +527,9 @@ API_TOKEN_HMAC_KEY="" # settings at all; the pooler's or the database's own defaults then apply. # DATABASE_SESSION_OPTIONS_DISABLED="1" # -# The app runs as `healthlog_app`, a database role the container creates on -# every start: it reads and writes rows but cannot alter or drop tables or -# create roles. Migrations and restores still use the user in DATABASE_URL. +# The app runs as `healthlog_app__`, a database role the +# container creates for this database on every start: it reads and writes +# rows but cannot alter or drop tables, own anything, or create roles. Migrations and restores still use the user in DATABASE_URL. # Where that user may not create roles (a managed database), the start log # says so and the app uses DATABASE_URL as given. Set "off" if you manage # the database roles yourself. See docs/self-hosting/database-role.md. diff --git a/docker-compose.yml b/docker-compose.yml index a19bb63ed..c272075a9 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -81,8 +81,9 @@ x-healthlog-env: &healthlog_env # 1 sends neither of the two settings above (they ride the libpq `options` # startup parameter, which PgBouncer refuses by default). DATABASE_SESSION_OPTIONS_DISABLED: "${DATABASE_SESSION_OPTIONS_DISABLED:-}" - # The app connects as `healthlog_app`, a role the entrypoint creates that - # reads and writes rows but cannot change the schema; the migrations keep + # The app connects as `healthlog_app__`, a role the + # entrypoint creates for this database that reads and writes rows but + # cannot change the schema; the migrations and the queue's install keep # the owner above. off keeps the app on the owner (for an operator who # manages database roles). See docs/self-hosting/database-role.md. DB_APP_ROLE: "${DB_APP_ROLE:-}" diff --git a/docker-entrypoint.sh b/docker-entrypoint.sh index e74be1f56..246a20b2f 100644 --- a/docker-entrypoint.sh +++ b/docker-entrypoint.sh @@ -109,8 +109,8 @@ echo "HealthLog: Migrations complete." # ── The app's database role ────────────────────────────────── # Everything above ran as the owner DATABASE_URL names. The app itself -# connects as `healthlog_app`, which reads and writes rows and runs the queue -# but cannot change the schema (scripts/db-app-role.mjs says what it may and +# connects as `healthlog_app__`, this database's own role, +# which reads and writes rows and runs the queue but cannot change the schema (scripts/db-app-role.mjs says what it may and # may not do, and docs/self-hosting/database-role.md how to opt out). The # script hands the role's URL over on descriptor 3 only, never on stdout or # stderr; nothing handed over means it said why and the app stays on the diff --git a/docs/ops/deploy.md b/docs/ops/deploy.md index 15ce03a1d..381b78f45 100644 --- a/docs/ops/deploy.md +++ b/docs/ops/deploy.md @@ -143,18 +143,21 @@ it with `REINDEX INDEX CONCURRENTLY "";` when the database is quiet. ### The app's database role The steps above run as the owner `DATABASE_URL` names: the dump, the -migrations and, after them, the setup of `healthlog_app`, the role the app -then connects as ([database-role.md](../self-hosting/database-role.md)). The -app never runs a migration. A migration that creates a table needs nothing -more: the setup grants the role every table it lacks before the app starts. +migrations and, after them, the job queue's own install or migration and the +setup of `healthlog_app__`, the role the app then connects as +([database-role.md](../self-hosting/database-role.md)). Neither the app nor +its queue ever runs a migration. A migration that creates a table needs +nothing more: the setup grants the role every table it lacks before the app +starts. -On the first start of the release that brings the role, the setup hands the -existing `pgboss` tables to it, which takes each table's lock for a moment. -Stop the old container (and a separate worker) before the new one starts, as -for any update with migrations; a lock that does not come within 15 seconds -leaves that start on the owner with a notice, and the next start tries again. -A web container and a worker container that start together wait for each -other on an advisory lock and derive the same password. +The queue schema `pgboss` stays the owner's. When an update brings a newer +pg-boss with a schema migration, the start runs it as the owner, including +its index builds, before the app starts; on a large queue that start takes as +long as the builds. Stop the old container (and a separate worker) before the +new one starts, as for any update with migrations. A web container and a +worker container that start together wait for each other on an advisory lock +and derive the same password. Going back to 1.46 runs as the owner, as 1.46 +always did, and finds nothing in the queue schema it did not create. ### v1.46.0 notes diff --git a/docs/ops/disaster-recovery.md b/docs/ops/disaster-recovery.md index a6c40c94c..9cca9bb66 100644 --- a/docs/ops/disaster-recovery.md +++ b/docs/ops/disaster-recovery.md @@ -148,11 +148,12 @@ different. key that does not open the data. The restore runs as the owner `DATABASE_URL` names, and the restored tables -belong to it. The app connects as `healthlog_app` +belong to it. The app connects as its own role, +`healthlog_app__` ([database-role.md](../self-hosting/database-role.md)): the start in step 5 -grants that role the restored tables and hands it the queue schema before the -app runs. A dump carries grants to `healthlog_app`; `db-restore` leaves them -out (`--no-owner --no-privileges`). Restoring with `pg_restore` by hand, pass +grants that role the restored tables, the queue's included, before the app +runs; the queue schema stays the owner's. A dump carries grants to the role; +`db-restore` leaves them out (`--no-owner --no-privileges`). Restoring with `pg_restore` by hand, pass the same two flags, or it fails on a database without the role. When the dumps are still in the server's `backups` volume (the database diff --git a/docs/self-hosting/database-role.md b/docs/self-hosting/database-role.md index 4b8ea1403..3bb9b929a 100644 --- a/docs/self-hosting/database-role.md +++ b/docs/self-hosting/database-role.md @@ -3,39 +3,97 @@ The container connects to Postgres under two names. The user in `DATABASE_URL` (on the bundled compose file, `healthlog`, which the official Postgres image makes a superuser) owns the schema: the entrypoint uses it for -the dump before an update, the migrations and the restore. The app itself, -web and worker, connects as `healthlog_app`, a role the entrypoint sets up -after the migrations on every start. Nothing to configure: the first start of -a release with this change creates it. +the dump before an update, the migrations, the job queue's own install and +migrations, and the restore. The app itself, web and worker, connects as a +role the entrypoint sets up for this database after the migrations on every +start. Nothing to configure: the first start of a release with this change +creates it. The point is the blast radius of a bug. If a dependency of the app were ever made to run code inside the container, the database connection it found would be one that can read and write rows, not one that can drop tables, rewrite the -schema or reach the server's files. +schema, plant code for the owner to run, or reach the server's files. + +## Its name + +`healthlog_app__`: the database's name, cut to letters, +digits and underscores, and eight hex digits of a hash of it. On the bundled +compose file that is `healthlog_app_healthlog_` and the hash. The start log +names it, and so does `pg_stat_activity`. + +Postgres roles belong to the whole server, not to one database. One role per +database keeps two HealthLog instances on one server (production and staging +on a shared Postgres) apart: each has its own role and password, neither +locks the other out, and neither role has any grant in the other database. A +role that holds an explicit `CONNECT` grant on another database is not used +(the start says so). Postgres lets every role connect to a database whose +`CONNECT` is granted to `PUBLIC`, the default; the role finds nothing to read +there. To shut the door as well, revoke it on each HealthLog database: +`REVOKE CONNECT ON DATABASE FROM PUBLIC;` (and grant it back to any +monitoring user that relied on it). ## What the role may and may not do -| May | May not | -| ------------------------------------------------------------------------- | --------------------------------------------------------------- | -| read, insert, update and delete rows in every table of the app's schema | create, alter or drop tables, indexes, types or functions there | -| use and advance its sequences | write `_prisma_migrations` (it may read it) | -| `VACUUM`, `ANALYZE` and `REINDEX` the `measurements` table (`MAINTAIN`) | create roles, schemas or databases | -| own the `pgboss` schema, where the job queue keeps and changes its tables | act as a superuser: no `COPY … PROGRAM`, no `pg_read_file` | -| create temporary tables | become the owner (`SET ROLE`) | +| May | May not | +| ----------------------------------------------------------------------- | ------------------------------------------------------------------- | +| read, insert, update and delete rows in every table of the app's schema | create, alter or drop tables, indexes, types or functions anywhere | +| use and advance its sequences | write `_prisma_migrations` (it may read it) | +| `VACUUM`, `ANALYZE` and `REINDEX` the `measurements` table (`MAINTAIN`) | create triggers or rules, own anything | +| read and write the job queue's rows in `pgboss` | write `pgboss.bam` or the version in `pgboss.version` (it may read) | +| create temporary tables | create roles, schemas or databases; become the owner (`SET ROLE`) | +| | act as a superuser: no `COPY … PROGRAM`, no `pg_read_file` | -The queue schema is the role's because pg-boss creates and migrates its own -tables, functions and per-queue partitions while the app runs. Nothing of the -app's data lives there. The start also takes `CREATE` on the app's schema -away from `PUBLIC`, which Postgres 15 and newer already do for a new database -but a database first created on an older major still allows. +The start also takes `CREATE` on the app's schema away from `PUBLIC`, which +Postgres 15 and newer already do for a new database but a database first +created on an older major still allows. -Its password is derived from `API_TOKEN_HMAC_KEY`, so a separate worker -container ([scaling.md](scaling.md)) arrives at the same one without a new -variable, and it is stored as a SCRAM hash. The owner's URL is replaced, not -kept beside the role's, so it is not in the app's environment. -`docker compose exec app …` still starts with the container's configured -environment, so the operator scripts (`reset-password.mjs` and the others) -run as the owner as before. +## The job queue's schema + +`pgboss` belongs to the owner. pg-boss changes its schema when it installs, +when it migrates (and in the index builds a migration leaves in +`pgboss.bam`), and for queues created with `partition: true`, which HealthLog +does not use. So the start installs and migrates the queue schema as the +owner, with pg-boss's own plans, before the app runs, and the worker starts +pg-boss with `migrate: false`: it only reads and writes rows, and refuses to +run on a schema that is not at its version rather than change it. A new queue +is a row. + +The role may not write the two tables whose contents decide what runs as the +owner: `pgboss.bam` holds SQL that pg-boss's migration runner executes, and +the version in `pgboss.version` decides which migrations run. Owning nothing +in the schema, the role cannot plant a trigger, a rule or a function body +that a later session as the owner would run with the owner's rights: a +rollback to 1.46 (which runs as the owner throughout), `DB_APP_ROLE=off`, a +container that fell back to the owner, or the old container during a deploy. +For the same reason a container that falls back to the owner and one that +runs as the role can share the schema: the owner creates every object in it +either way, and the next start grants the role what is new. + +Every start, on every path (the role, `DB_APP_ROLE=off`, a fallback), also +takes back anything in `pgboss` that a HealthLog-made role owns and drops any +trigger or rule there (pg-boss defines none). A pre-release build handed the +schema to a server-wide `healthlog_app` role; when the start finds that, it +also compares every queue function with pg-boss's own definition and puts +back the ones that differ, retires that role's grants in this database, and +drops it once no database refers to it. The log names each. + +pg-boss rebuilds bloated queue indexes only on tables it owns; under the role +it reports them. The start rebuilds them as the owner, one at a time with a +five-second lock wait. + +## The password + +Derived from `API_TOKEN_HMAC_KEY` and the database's name, so a separate +worker container ([scaling.md](scaling.md)) arrives at the same one without a +new variable, and stored as a SCRAM hash. Rotating `API_TOKEN_HMAC_KEY` with a +web/worker split means restarting all containers together: the first one to +start sets the new password, and a container still on the old key cannot +open new connections until it restarts with the new one. + +The owner's URL is replaced, not kept beside the role's, so it is not in the +app's environment. `docker compose exec app …` still starts with the +container's configured environment, so the operator scripts +(`reset-password.mjs` and the others) run as the owner as before. Each start re-applies the grants, so a table the new migrations created is covered before the app starts, and a restored database is put right on the @@ -46,14 +104,14 @@ start after the restore. When it is in place: ```text -HealthLog: the app connects as healthlog_app: rows and the queue, no schema changes (DB_APP_ROLE=off turns this off) +HealthLog: the app connects as healthlog_app_healthlog_1a2b3c4d: rows and the queue, no schema changes (DB_APP_ROLE=off turns this off) ``` When it is not, one notice names the reason, and the app runs as the `DATABASE_URL` user, exactly as every earlier release did: ```text -HealthLog: notice: the app connects as healthlog, the user DATABASE_URL names, not as the restricted role healthlog_app: . … +HealthLog: notice: the app connects as healthlog, the user DATABASE_URL names, not as a restricted role: . … ``` The reasons: @@ -68,12 +126,14 @@ The reasons: [postgres-major-upgrade.md](../ops/postgres-major-upgrade.md) for one you run yourself. - **The role cannot log in.** A `pg_hba.conf` that lists users by name needs a - line for `healthlog_app` beside the one for the owner. -- **A role named `healthlog_app` exists that HealthLog did not create.** The - start leaves a role it does not recognise alone. -- **Setting it up failed**, most often a lock it waited 15 seconds for: the - first start hands the existing queue tables to the role, and another - container still using them holds it up. The next start tries again. + line for the role beside the one for the owner. +- **The role holds `CONNECT` on another database.** Revoke it there. +- **A role of that name exists that HealthLog did not create** for this + database. The start leaves a role it does not recognise alone. +- **The queue schema is newer than this image's pg-boss**, after going back + to an older image: only the owner runs on it. +- **Setting it up failed**, most often a lock it waited 15 seconds for while + another container held it. The next start tries again. None of these stops the start. @@ -83,17 +143,22 @@ Set `DB_APP_ROLE=off` in `.env` and recreate the container. The app then connects as the `DATABASE_URL` user. Use it when you manage database roles yourself, including when `DATABASE_URL` already names a restricted role of your own; then that role needs what the table above lists, and the -migrations need a user that owns the schema. +migrations and the queue's install need a user that owns the schema. -The role stays in the database when you turn it off and does no harm. To -remove it, as the owner: +The role stays in the database when you turn it off and owns nothing. To +remove it, as the owner, with the name from the start log: ```sql -REASSIGN OWNED BY healthlog_app TO healthlog; -DROP OWNED BY healthlog_app; -DROP ROLE healthlog_app; +DROP OWNED BY healthlog_app_healthlog_1a2b3c4d; +DROP ROLE healthlog_app_healthlog_1a2b3c4d; ``` +## Going back to 1.46 + +1.46 knows nothing of the role and runs as the owner, as it always did. The +queue schema is the owner's, so it finds nothing it did not create. The role +stays, unused, until you drop it as above. + ## Index drops at start `DROP INDEX` belongs to a table's owner; `MAINTAIN` does not include it. The @@ -104,9 +169,8 @@ start drops them as the owner, waiting at most five seconds for each ## Dumps and restores -The nightly dump runs as the role and reads everything: the app's schema -through its grants, the queue schema as its owner. A dump carries the grants -to `healthlog_app`; `db-restore` leaves them and the ownership out +The nightly dump runs as the role and reads everything through its grants. A +dump carries those grants; `db-restore` leaves them and the ownership out (`--no-owner --no-privileges`), restores as the owner, and the next start grants again. A restore of your own with `pg_restore` should do the same, or it fails on a database without the role. diff --git a/docs/self-hosting/scaling.md b/docs/self-hosting/scaling.md index 01d225033..53ae5ec4f 100644 --- a/docs/self-hosting/scaling.md +++ b/docs/self-hosting/scaling.md @@ -131,12 +131,13 @@ and restart the worker together with the web containers. ## Database role -Both containers run the entrypoint, and both set up `healthlog_app`, the -database role the app connects as -([database-role.md](database-role.md)). They take turns on an advisory lock -and derive the same password from `API_TOKEN_HMAC_KEY`, which the shared -environment block gives both, so a worker that starts after the web container -changes nothing the web container relies on. +Both containers run the entrypoint, and both set up the database role the +app connects as ([database-role.md](database-role.md)). They take turns on an +advisory lock and derive the same password from `API_TOKEN_HMAC_KEY` and the +database's name, which the shared environment block gives both, so a worker +that starts after the web container changes nothing the web container relies +on. Rotating `API_TOKEN_HMAC_KEY` changes that password: restart all +containers together. ## Healthchecks diff --git a/scripts/env-manifest.json b/scripts/env-manifest.json index 977a50755..69a0027d0 100644 --- a/scripts/env-manifest.json +++ b/scripts/env-manifest.json @@ -228,7 +228,7 @@ }, { "name": "DB_APP_ROLE", - "purpose": "auto (default) or off. Auto: after the migrations the entrypoint creates the role healthlog_app, which may read and write rows and run the queue but not change the schema, and starts the app with it in place of the DATABASE_URL user. Off keeps the app on the DATABASE_URL user, for an operator who manages database roles.", + "purpose": "auto (default) or off. Auto: after the migrations the entrypoint creates this database's role healthlog_app__, which may read and write rows and run the queue but not change the schema, and starts the app with it in place of the DATABASE_URL user. Off keeps the app on the DATABASE_URL user, for an operator who manages database roles.", "example": "off" } ]