docs: name the per-database role and describe the owner-owned queue schema

.env.example

@@ -39,8 +39,9 @@
 # Left commented here so that append does not create a duplicate key. Uncomment
 # and set it directly instead if you prefer to edit the file by hand.
 
-# The container runs the app as `healthlog_app`, a database role it creates on
+# The container runs the app as `healthlog_app_<database>_<hash>`, a database
-# every start that reads and writes rows but cannot change the schema. The
+# role it creates on every start that reads and writes rows but cannot change
+# the schema. The
 # migrations keep the DATABASE_URL user. "off" keeps the app on that user, if
 # you manage database roles yourself (docs/self-hosting/database-role.md).
 # DB_APP_ROLE="off"

.env.production.example

@@ -527,9 +527,9 @@ API_TOKEN_HMAC_KEY="<openssl rand -hex 32>"
 # settings at all; the pooler's or the database's own defaults then apply.
 # DATABASE_SESSION_OPTIONS_DISABLED="1"
 #
-# The app runs as `healthlog_app`, a database role the container creates on
+# The app runs as `healthlog_app_<database>_<hash>`, a database role the
-# every start: it reads and writes rows but cannot alter or drop tables or
+# container creates for this database on every start: it reads and writes
-# create roles. Migrations and restores still use the user in DATABASE_URL.
+# rows but cannot alter or drop tables, own anything, or create roles. Migrations and restores still use the user in DATABASE_URL.
 # Where that user may not create roles (a managed database), the start log
 # says so and the app uses DATABASE_URL as given. Set "off" if you manage
 # the database roles yourself. See docs/self-hosting/database-role.md.

docker-compose.yml

@@ -81,8 +81,9 @@ x-healthlog-env: &healthlog_env
   # 1 sends neither of the two settings above (they ride the libpq `options`
   # startup parameter, which PgBouncer refuses by default).
   DATABASE_SESSION_OPTIONS_DISABLED: "${DATABASE_SESSION_OPTIONS_DISABLED:-}"
-  # The app connects as `healthlog_app`, a role the entrypoint creates that
+  # The app connects as `healthlog_app_<database>_<hash>`, a role the
-  # reads and writes rows but cannot change the schema; the migrations keep
+  # entrypoint creates for this database that reads and writes rows but
+  # cannot change the schema; the migrations and the queue's install keep
   # the owner above. off keeps the app on the owner (for an operator who
   # manages database roles). See docs/self-hosting/database-role.md.
   DB_APP_ROLE: "${DB_APP_ROLE:-}"

docker-entrypoint.sh

@@ -109,8 +109,8 @@ echo "HealthLog: Migrations complete."
 
 # ── The app's database role ──────────────────────────────────
 # Everything above ran as the owner DATABASE_URL names. The app itself
-# connects as `healthlog_app`, which reads and writes rows and runs the queue
+# connects as `healthlog_app_<database>_<hash>`, this database's own role,
-# but cannot change the schema (scripts/db-app-role.mjs says what it may and
+# which reads and writes rows and runs the queue but cannot change the schema (scripts/db-app-role.mjs says what it may and
 # may not do, and docs/self-hosting/database-role.md how to opt out). The
 # script hands the role's URL over on descriptor 3 only, never on stdout or
 # stderr; nothing handed over means it said why and the app stays on the

docs/ops/deploy.md

@@ -143,18 +143,21 @@ it with `REINDEX INDEX CONCURRENTLY "<index>";` when the database is quiet.
 ### The app's database role
 
 The steps above run as the owner `DATABASE_URL` names: the dump, the
-migrations and, after them, the setup of `healthlog_app`, the role the app
+migrations and, after them, the job queue's own install or migration and the
-then connects as ([database-role.md](../self-hosting/database-role.md)). The
+setup of `healthlog_app_<database>_<hash>`, the role the app then connects as
-app never runs a migration. A migration that creates a table needs nothing
+([database-role.md](../self-hosting/database-role.md)). Neither the app nor
-more: the setup grants the role every table it lacks before the app starts.
+its queue ever runs a migration. A migration that creates a table needs
+nothing more: the setup grants the role every table it lacks before the app
+starts.
 
-On the first start of the release that brings the role, the setup hands the
+The queue schema `pgboss` stays the owner's. When an update brings a newer
-existing `pgboss` tables to it, which takes each table's lock for a moment.
+pg-boss with a schema migration, the start runs it as the owner, including
-Stop the old container (and a separate worker) before the new one starts, as
+its index builds, before the app starts; on a large queue that start takes as
-for any update with migrations; a lock that does not come within 15 seconds
+long as the builds. Stop the old container (and a separate worker) before the
-leaves that start on the owner with a notice, and the next start tries again.
+new one starts, as for any update with migrations. A web container and a
-A web container and a worker container that start together wait for each
+worker container that start together wait for each other on an advisory lock
-other on an advisory lock and derive the same password.
+and derive the same password. Going back to 1.46 runs as the owner, as 1.46
+always did, and finds nothing in the queue schema it did not create.
 
 ### v1.46.0 notes
 

docs/ops/disaster-recovery.md

@@ -148,11 +148,12 @@ different.
    key that does not open the data.
 
 The restore runs as the owner `DATABASE_URL` names, and the restored tables
-belong to it. The app connects as `healthlog_app`
+belong to it. The app connects as its own role,
+`healthlog_app_<database>_<hash>`
 ([database-role.md](../self-hosting/database-role.md)): the start in step 5
-grants that role the restored tables and hands it the queue schema before the
+grants that role the restored tables, the queue's included, before the app
-app runs. A dump carries grants to `healthlog_app`; `db-restore` leaves them
+runs; the queue schema stays the owner's. A dump carries grants to the role;
-out (`--no-owner --no-privileges`). Restoring with `pg_restore` by hand, pass
+`db-restore` leaves them out (`--no-owner --no-privileges`). Restoring with `pg_restore` by hand, pass
 the same two flags, or it fails on a database without the role.
 
 When the dumps are still in the server's `backups` volume (the database

docs/self-hosting/database-role.md

@@ -3,39 +3,97 @@
 The container connects to Postgres under two names. The user in
 `DATABASE_URL` (on the bundled compose file, `healthlog`, which the official
 Postgres image makes a superuser) owns the schema: the entrypoint uses it for
-the dump before an update, the migrations and the restore. The app itself,
+the dump before an update, the migrations, the job queue's own install and
-web and worker, connects as `healthlog_app`, a role the entrypoint sets up
+migrations, and the restore. The app itself, web and worker, connects as a
-after the migrations on every start. Nothing to configure: the first start of
+role the entrypoint sets up for this database after the migrations on every
-a release with this change creates it.
+start. Nothing to configure: the first start of a release with this change
+creates it.
 
 The point is the blast radius of a bug. If a dependency of the app were ever
 made to run code inside the container, the database connection it found would
 be one that can read and write rows, not one that can drop tables, rewrite the
-schema or reach the server's files.
+schema, plant code for the owner to run, or reach the server's files.
+
+## Its name
+
+`healthlog_app_<database>_<hash>`: the database's name, cut to letters,
+digits and underscores, and eight hex digits of a hash of it. On the bundled
+compose file that is `healthlog_app_healthlog_` and the hash. The start log
+names it, and so does `pg_stat_activity`.
+
+Postgres roles belong to the whole server, not to one database. One role per
+database keeps two HealthLog instances on one server (production and staging
+on a shared Postgres) apart: each has its own role and password, neither
+locks the other out, and neither role has any grant in the other database. A
+role that holds an explicit `CONNECT` grant on another database is not used
+(the start says so). Postgres lets every role connect to a database whose
+`CONNECT` is granted to `PUBLIC`, the default; the role finds nothing to read
+there. To shut the door as well, revoke it on each HealthLog database:
+`REVOKE CONNECT ON DATABASE <name> FROM PUBLIC;` (and grant it back to any
+monitoring user that relied on it).
 
 ## What the role may and may not do
 
-| May                                                                       | May not                                                         |
+| May                                                                     | May not                                                             |
-| ------------------------------------------------------------------------- | --------------------------------------------------------------- |
+| ----------------------------------------------------------------------- | ------------------------------------------------------------------- |
-| read, insert, update and delete rows in every table of the app's schema   | create, alter or drop tables, indexes, types or functions there |
+| read, insert, update and delete rows in every table of the app's schema | create, alter or drop tables, indexes, types or functions anywhere  |
-| use and advance its sequences                                             | write `_prisma_migrations` (it may read it)                     |
+| use and advance its sequences                                           | write `_prisma_migrations` (it may read it)                         |
-| `VACUUM`, `ANALYZE` and `REINDEX` the `measurements` table (`MAINTAIN`)   | create roles, schemas or databases                              |
+| `VACUUM`, `ANALYZE` and `REINDEX` the `measurements` table (`MAINTAIN`) | create triggers or rules, own anything                              |
-| own the `pgboss` schema, where the job queue keeps and changes its tables | act as a superuser: no `COPY … PROGRAM`, no `pg_read_file`      |
+| read and write the job queue's rows in `pgboss`                         | write `pgboss.bam` or the version in `pgboss.version` (it may read) |
-| create temporary tables                                                   | become the owner (`SET ROLE`)                                   |
+| create temporary tables                                                 | create roles, schemas or databases; become the owner (`SET ROLE`)   |
+|                                                                         | act as a superuser: no `COPY … PROGRAM`, no `pg_read_file`          |
 
-The queue schema is the role's because pg-boss creates and migrates its own
+The start also takes `CREATE` on the app's schema away from `PUBLIC`, which
-tables, functions and per-queue partitions while the app runs. Nothing of the
+Postgres 15 and newer already do for a new database but a database first
-app's data lives there. The start also takes `CREATE` on the app's schema
+created on an older major still allows.
-away from `PUBLIC`, which Postgres 15 and newer already do for a new database
-but a database first created on an older major still allows.
 
-Its password is derived from `API_TOKEN_HMAC_KEY`, so a separate worker
+## The job queue's schema
-container ([scaling.md](scaling.md)) arrives at the same one without a new
+
-variable, and it is stored as a SCRAM hash. The owner's URL is replaced, not
+`pgboss` belongs to the owner. pg-boss changes its schema when it installs,
-kept beside the role's, so it is not in the app's environment.
+when it migrates (and in the index builds a migration leaves in
-`docker compose exec app …` still starts with the container's configured
+`pgboss.bam`), and for queues created with `partition: true`, which HealthLog
-environment, so the operator scripts (`reset-password.mjs` and the others)
+does not use. So the start installs and migrates the queue schema as the
-run as the owner as before.
+owner, with pg-boss's own plans, before the app runs, and the worker starts
+pg-boss with `migrate: false`: it only reads and writes rows, and refuses to
+run on a schema that is not at its version rather than change it. A new queue
+is a row.
+
+The role may not write the two tables whose contents decide what runs as the
+owner: `pgboss.bam` holds SQL that pg-boss's migration runner executes, and
+the version in `pgboss.version` decides which migrations run. Owning nothing
+in the schema, the role cannot plant a trigger, a rule or a function body
+that a later session as the owner would run with the owner's rights: a
+rollback to 1.46 (which runs as the owner throughout), `DB_APP_ROLE=off`, a
+container that fell back to the owner, or the old container during a deploy.
+For the same reason a container that falls back to the owner and one that
+runs as the role can share the schema: the owner creates every object in it
+either way, and the next start grants the role what is new.
+
+Every start, on every path (the role, `DB_APP_ROLE=off`, a fallback), also
+takes back anything in `pgboss` that a HealthLog-made role owns and drops any
+trigger or rule there (pg-boss defines none). A pre-release build handed the
+schema to a server-wide `healthlog_app` role; when the start finds that, it
+also compares every queue function with pg-boss's own definition and puts
+back the ones that differ, retires that role's grants in this database, and
+drops it once no database refers to it. The log names each.
+
+pg-boss rebuilds bloated queue indexes only on tables it owns; under the role
+it reports them. The start rebuilds them as the owner, one at a time with a
+five-second lock wait.
+
+## The password
+
+Derived from `API_TOKEN_HMAC_KEY` and the database's name, so a separate
+worker container ([scaling.md](scaling.md)) arrives at the same one without a
+new variable, and stored as a SCRAM hash. Rotating `API_TOKEN_HMAC_KEY` with a
+web/worker split means restarting all containers together: the first one to
+start sets the new password, and a container still on the old key cannot
+open new connections until it restarts with the new one.
+
+The owner's URL is replaced, not kept beside the role's, so it is not in the
+app's environment. `docker compose exec app …` still starts with the
+container's configured environment, so the operator scripts
+(`reset-password.mjs` and the others) run as the owner as before.
 
 Each start re-applies the grants, so a table the new migrations created is
 covered before the app starts, and a restored database is put right on the
@@ -46,14 +104,14 @@ start after the restore.
 When it is in place:
 
 ```text
-HealthLog: the app connects as healthlog_app: rows and the queue, no schema changes (DB_APP_ROLE=off turns this off)
+HealthLog: the app connects as healthlog_app_healthlog_1a2b3c4d: rows and the queue, no schema changes (DB_APP_ROLE=off turns this off)
 ```
 
 When it is not, one notice names the reason, and the app runs as the
 `DATABASE_URL` user, exactly as every earlier release did:
 
 ```text
-HealthLog: notice: the app connects as healthlog, the user DATABASE_URL names, not as the restricted role healthlog_app: <reason>. …
+HealthLog: notice: the app connects as healthlog, the user DATABASE_URL names, not as a restricted role: <reason>. …
 ```
 
 The reasons:
@@ -68,12 +126,14 @@ The reasons:
   [postgres-major-upgrade.md](../ops/postgres-major-upgrade.md) for one you
   run yourself.
 - **The role cannot log in.** A `pg_hba.conf` that lists users by name needs a
-  line for `healthlog_app` beside the one for the owner.
+  line for the role beside the one for the owner.
-- **A role named `healthlog_app` exists that HealthLog did not create.** The
+- **The role holds `CONNECT` on another database.** Revoke it there.
-  start leaves a role it does not recognise alone.
+- **A role of that name exists that HealthLog did not create** for this
-- **Setting it up failed**, most often a lock it waited 15 seconds for: the
+  database. The start leaves a role it does not recognise alone.
-  first start hands the existing queue tables to the role, and another
+- **The queue schema is newer than this image's pg-boss**, after going back
-  container still using them holds it up. The next start tries again.
+  to an older image: only the owner runs on it.
+- **Setting it up failed**, most often a lock it waited 15 seconds for while
+  another container held it. The next start tries again.
 
 None of these stops the start.
 
@@ -83,17 +143,22 @@ Set `DB_APP_ROLE=off` in `.env` and recreate the container. The app then
 connects as the `DATABASE_URL` user. Use it when you manage database roles
 yourself, including when `DATABASE_URL` already names a restricted role of
 your own; then that role needs what the table above lists, and the
-migrations need a user that owns the schema.
+migrations and the queue's install need a user that owns the schema.
 
-The role stays in the database when you turn it off and does no harm. To
+The role stays in the database when you turn it off and owns nothing. To
-remove it, as the owner:
+remove it, as the owner, with the name from the start log:
 
 ```sql
-REASSIGN OWNED BY healthlog_app TO healthlog;
+DROP OWNED BY healthlog_app_healthlog_1a2b3c4d;
-DROP OWNED BY healthlog_app;
+DROP ROLE healthlog_app_healthlog_1a2b3c4d;
-DROP ROLE healthlog_app;
 ```
 
+## Going back to 1.46
+
+1.46 knows nothing of the role and runs as the owner, as it always did. The
+queue schema is the owner's, so it finds nothing it did not create. The role
+stays, unused, until you drop it as above.
+
 ## Index drops at start
 
 `DROP INDEX` belongs to a table's owner; `MAINTAIN` does not include it. The
@@ -104,9 +169,8 @@ start drops them as the owner, waiting at most five seconds for each
 
 ## Dumps and restores
 
-The nightly dump runs as the role and reads everything: the app's schema
+The nightly dump runs as the role and reads everything through its grants. A
-through its grants, the queue schema as its owner. A dump carries the grants
+dump carries those grants; `db-restore` leaves them and the ownership out
-to `healthlog_app`; `db-restore` leaves them and the ownership out
 (`--no-owner --no-privileges`), restores as the owner, and the next start
 grants again. A restore of your own with `pg_restore` should do the same, or
 it fails on a database without the role.

docs/self-hosting/scaling.md

@@ -131,12 +131,13 @@ and restart the worker together with the web containers.
 
 ## Database role
 
-Both containers run the entrypoint, and both set up `healthlog_app`, the
+Both containers run the entrypoint, and both set up the database role the
-database role the app connects as
+app connects as ([database-role.md](database-role.md)). They take turns on an
-([database-role.md](database-role.md)). They take turns on an advisory lock
+advisory lock and derive the same password from `API_TOKEN_HMAC_KEY` and the
-and derive the same password from `API_TOKEN_HMAC_KEY`, which the shared
+database's name, which the shared environment block gives both, so a worker
-environment block gives both, so a worker that starts after the web container
+that starts after the web container changes nothing the web container relies
-changes nothing the web container relies on.
+on. Rotating `API_TOKEN_HMAC_KEY` changes that password: restart all
+containers together.
 
 ## Healthchecks
 

scripts/env-manifest.json

@@ -228,7 +228,7 @@
         },
         {
           "name": "DB_APP_ROLE",
-          "purpose": "auto (default) or off. Auto: after the migrations the entrypoint creates the role healthlog_app, which may read and write rows and run the queue but not change the schema, and starts the app with it in place of the DATABASE_URL user. Off keeps the app on the DATABASE_URL user, for an operator who manages database roles.",
+          "purpose": "auto (default) or off. Auto: after the migrations the entrypoint creates this database's role healthlog_app_<database>_<hash>, which may read and write rows and run the queue but not change the schema, and starts the app with it in place of the DATABASE_URL user. Off keeps the app on the DATABASE_URL user, for an operator who manages database roles.",
           "example": "off"
         }
       ]