diff --git a/README.md b/README.md index 214f00045..854eb8cc3 100644 --- a/README.md +++ b/README.md @@ -87,7 +87,7 @@ curl -s http://localhost:3000/api/version # returns version + buildSha + built `SESSION_COOKIE_SECURE=false` is the one setting plain-HTTP self-hosts must set: without it the session cookie carries the `Secure` flag, the browser drops it over HTTP, and login silently fails. Leave it unset (or `true`) only when a TLS reverse proxy serves HTTPS — and then set `NEXT_PUBLIC_APP_URL` and `APP_URL` to the public URL first (see the [self-hosting guide](https://docs.healthlog.dev/self-hosting/)). -Persistent state lives in exactly one named volume (`pgdata`, the Postgres data dir); the app container is stateless, so backing up the database backs up everything. +Persistent state lives in exactly one named volume (`postgres-data`, the Postgres data); the app container is stateless, so backing up the database backs up everything. An install from v1.45 or earlier moves its Postgres 16 database into it on the first start ([details](docs/ops/postgres-major-upgrade.md)). NAS and homelab users have ready-made paths: [Unraid Community Applications](docs/self-hosting/unraid.md), [Portainer app templates](docs/self-hosting/portainer/README.md) and [TrueNAS](docs/self-hosting/truenas.md) (the Community train app is pending catalog review). @@ -105,7 +105,7 @@ The App Store listing carries a small price, which exists to cover Apple's yearl ## Tech stack -Next.js 16 (App Router, React Server Components), TypeScript strict, PostgreSQL 16 with Prisma, pg-boss for jobs, Tailwind CSS 4 + shadcn/ui, Recharts, WebAuthn passkeys, Vitest + Playwright, multi-stage Alpine Docker image. The OpenAPI 3.1 contract for native clients is generated from the Zod schemas at [`docs/api/openapi.yaml`](docs/api/openapi.yaml). +Next.js 16 (App Router, React Server Components), TypeScript strict, PostgreSQL 18 with Prisma, pg-boss for jobs, Tailwind CSS 4 + shadcn/ui, Recharts, WebAuthn passkeys, Vitest + Playwright, multi-stage Alpine Docker image. The OpenAPI 3.1 contract for native clients is generated from the Zod schemas at [`docs/api/openapi.yaml`](docs/api/openapi.yaml). ## Documentation diff --git a/docs/diagrams/01-data-flow.svg b/docs/diagrams/01-data-flow.svg index 9bb7a272b..2a4b04956 100644 --- a/docs/diagrams/01-data-flow.svg +++ b/docs/diagrams/01-data-flow.svg @@ -81,7 +81,7 @@ pre-aggregated buckets - PostgreSQL 16 + PostgreSQL 18 encrypted at rest diff --git a/docs/diagrams/03-self-hosting-topology.svg b/docs/diagrams/03-self-hosting-topology.svg index 658a87b7b..0b860ce2b 100644 --- a/docs/diagrams/03-self-hosting-topology.svg +++ b/docs/diagrams/03-self-hosting-topology.svg @@ -1,6 +1,6 @@ HealthLog self-hosting topology - Internet traffic terminates at a reverse proxy (Caddy, Traefik or Nginx) for TLS, then reaches the Next.js app on port 3000. The app shares a Postgres 16 database with the pg-boss worker process for reminders, insights, backups and the Apple Health import. GHCR ships the multi-arch image; Coolify watches the digest and pulls; off-host S3-compatible storage is optional for backups. + Internet traffic terminates at a reverse proxy (Caddy, Traefik or Nginx) for TLS, then reaches the Next.js app on port 3000. The app shares a Postgres 18 database with the pg-boss worker process for reminders, insights, backups and the Apple Health import. GHCR ships the multi-arch image; Coolify watches the digest and pulls; off-host S3-compatible storage is optional for backups. @@ -47,7 +47,7 @@ - PostgreSQL 16 + PostgreSQL 18 26 Prisma models + pg-boss queues diff --git a/docs/diagrams/README.md b/docs/diagrams/README.md index c462567c7..e690170b0 100644 --- a/docs/diagrams/README.md +++ b/docs/diagrams/README.md @@ -10,7 +10,7 @@ copy. | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `01-data-flow.svg` | Sources (Withings, Apple Health export.zip, iPhone HealthKit, manual, Nightscout) → ingest endpoints → Measurement + MeasurementRollup → reads (rollup probe / live SQL fallback / cache) → surfaces (dashboard, insights, Coach, doctor PDF). | | `02-coach-pipeline.svg` | User question → snapshot builder → prompt assembler → provider chain (ChatGPT OAuth / OpenAI / Anthropic / local / admin-shared) → response parser → cited reply with mini-charts. | -| `03-self-hosting-topology.svg` | Internet → reverse proxy (TLS) → Next.js app (port 3000) ↔ pg-boss worker ↔ Postgres 16, plus GHCR image pull, optional Coolify autodeploy, optional S3 off-host backup. | +| `03-self-hosting-topology.svg` | Internet → reverse proxy (TLS) → Next.js app (port 3000) ↔ pg-boss worker ↔ Postgres 18, plus GHCR image pull, optional Coolify autodeploy, optional S3 off-host backup. | | `04-source-priority.svg` | Three sources for the same metric on the same day → resolver picks one canonical row per (cumulative vs point) ladder. Losing rows stay in the audit trail. | | `05-security-model.svg` | Three concentric layers — auth perimeter (passkey + password fallback), session perimeter (HttpOnly cookie + Postgres sessions), encrypted core (AES-256-GCM versioned keys). Side rails: rate limiter, audit log, HMAC tokens, CSP/HSTS, SSRF guard. | diff --git a/docs/ops/deploy.md b/docs/ops/deploy.md index 032c55730..0a1f8ff7c 100644 --- a/docs/ops/deploy.md +++ b/docs/ops/deploy.md @@ -129,6 +129,26 @@ An INVALID index whose migration is already recorded applied is not touched at boot. The boot names it on stderr with `WARNING — index … is INVALID`; rebuild it with `REINDEX INDEX CONCURRENTLY "";` when the database is quiet. +### v1.46.0 notes + +- **Postgres 18.** The compose file moves the bundled database from + Postgres 16 to 18. The first `docker compose up -d` with the new compose + file runs `db-upgrade`, which copies the database into a new + `postgres-data` volume, checks every table's row count, and only then lets + the app start; the old `pgdata` volume stays as it was, as the way back. + Pulling only the new image with the old compose file keeps 16, which this + release still runs on. Details, rollback and the steps for a database you + run yourself: [postgres-major-upgrade.md](postgres-major-upgrade.md). +- Migration `0485` drops `preview_features_json` from `users`, the same + pattern as `0430` in v1.44.0 below: do not run the previous image beside + the new schema. The sequence above stops the old container first; a + web/worker split stops both roles before the first new container starts. +- The other migrations add columns and one index (built concurrently). + Three touch rows: `0476` links API-written measurements to their token on + accounts that have exactly one measurement token, and `0478`/`0479` give + PDFs whose preview or text read failed one more attempt. On a prod-sized + copy all of them together finished in under a second. + ### v1.45.0 notes A single-container install (`HEALTHLOG_PROCESS_TYPE=all`, the shipped diff --git a/docs/ops/postgres-major-upgrade.md b/docs/ops/postgres-major-upgrade.md index 744213f28..58af2d6c4 100644 --- a/docs/ops/postgres-major-upgrade.md +++ b/docs/ops/postgres-major-upgrade.md @@ -1,32 +1,97 @@ # Moving the database to a newer Postgres major -The compose file ships Postgres 16 and stays there for now, because a major -version change is the one database update that cannot happen by itself on -`docker compose pull`. HealthLog runs on Postgres 18 as well: the full -integration suite runs against 18 (`INTEGRATION_PG_IMAGE=postgres:18-alpine -pnpm test:integration`), and the TrueNAS app starts new installs on 18. +From v1.46 the compose file runs Postgres 18. Installs from v1.45 or earlier +ran 16, and a major version cannot open the data directory of an older one, +so the compose file carries a one-time step that moves the database across +on the first start after the update. Nothing needs doing by hand on a stack +that runs the project's `docker-compose.yml`. -## Why changing the image line is not enough +Installs that bring their own Postgres (Unraid, an existing database server) +are not touched: HealthLog runs on 16 and 18 alike, and the move there is +yours to schedule with the steps under [By hand](#by-hand). -Minor releases (16.14 to 16.15) share the on-disk format. Majors do not, and -the official image refuses to start a data directory written by an older -major. From 18 on, the image also keeps its data one level deeper, under -`/var/lib/postgresql/18/docker`, and expects the volume mounted at -`/var/lib/postgresql` instead of `/var/lib/postgresql/data`. With the old -volume and only the image line changed, the `db` container exits at once with -"there appears to be PostgreSQL data in: /var/lib/postgresql/data". Nothing is -lost, but the stack stays down until the image line goes back to 16. +## What happens on the first start -So the move is a dump from 16 and a restore into a fresh 18 volume. +`db-upgrade` is a short-lived container next to `db`. It starts after the new +Postgres 18 database is up and before the app, and it exits in a second when +there is nothing to move. When the old `pgdata` volume holds a Postgres 16 +database and the new one is empty, it: -## Steps +1. checks there is room on the disk for a second copy; +2. starts Postgres 16 inside its own container on the old volume, reachable + only from inside that container; +3. streams a dump into Postgres 18 in a single transaction; +4. compares the row count of every table between the two; +5. gathers the planner statistics, which a restore does not carry; +6. stops Postgres 16 and leaves a marker file on the old volume. -Take the encryption key's copy out of its safe place first: the dump is -unreadable without it, and this is the moment you would find out. +The app starts once that succeeded, applies the release's migrations and +serves as before. On a typical personal record the whole move takes seconds; +a database of a few gigabytes takes a few minutes. Watch it with +`docker compose logs -f db-upgrade`. + +If any step fails, the new database stays empty, the app does not start, and +the log says what happened. A move cut short by a restart or a full disk is +run again from scratch on the next `docker compose up -d`. + +The old `pgdata` volume is read and never changed, apart from the marker +file. That is the way back. + +## Going back + +Stop the stack, put the two lines of the `db` service back the way v1.45 had +them, and start it with the v1.45 image: + +```yaml +db: + image: postgres:16.15-alpine + volumes: + - pgdata:/var/lib/postgresql/data +``` ```bash -# 1. Stop the app so nothing writes while the dump runs. If you run the -# worker role in its own container, stop it too: docker compose stop app app-worker +docker compose down +HEALTHLOG_IMAGE_REF=:v1.45.0 docker compose up -d +``` + +Anything written after the move lives only in the Postgres 18 volume. + +## Afterwards + +Keep the old volume until the new database has run for a while. When you are +sure, remove it (`docker volume ls` shows its full name, prefixed with the +compose project): + +```bash +docker volume rm _pgdata +``` + +`db-upgrade` then finds nothing to move. A later release drops the service +and the `pgdata` line. + +## Starting over + +When the move has run and the Postgres 18 volume is empty again (deleted by +hand, say), `db-upgrade` refuses to import the old state a second time, since +that would quietly bring back the database as it was on the day of the move. +To do exactly that on purpose, remove the marker and start again: + +```bash +docker compose run --rm --entrypoint rm db-upgrade \ + /var/lib/postgresql/data/HEALTHLOG_MOVED_TO_POSTGRES_18 +docker compose up -d +``` + +## By hand + +For a Postgres you run yourself, or a compose file you have changed so much +that the step does not fit, the move is a dump and a restore. Take the +encryption key's copy out of its safe place first: the dump is unreadable +without it, and this is the moment you would find out. + +```bash +# 1. Stop the app so nothing writes while the dump runs (and app-worker, if +# you run the worker role in its own container). docker compose stop app # 2. Dump the database in Postgres' custom format. @@ -37,56 +102,31 @@ docker compose exec -T db pg_dump -U healthlog -Fc healthlog \ docker compose stop db ``` -4. In `docker-compose.yml`, change the `db` service: - - ```yaml - image: postgres:18.6-alpine # or the current 18 minor - volumes: - - pgdata18:/var/lib/postgresql - ``` - - and add the new volume next to `pgdata` in the top-level `volumes:` block: - - ```yaml - volumes: - pgdata: - pgdata18: - ``` +4. Point the `db` service at Postgres 18 and a fresh volume. From 18 on, the + image keeps its data under `/var/lib/postgresql/18/docker` and expects the + volume mounted one level up, at `/var/lib/postgresql`; mounted at + `.../data` it refuses to start with "there appears to be PostgreSQL data + in: /var/lib/postgresql/data". ```bash -# 5. Start the new database and wait until it reports healthy. +# 5. Start the new database and restore the dump into it. docker compose up -d db -docker compose ps db - -# 6. Restore the dump into it. docker compose exec -T db pg_restore -U healthlog -d healthlog --no-owner \ < healthlog-pg16-.dump -# 7. Gather the planner statistics. A restore brings the rows but not the +# 6. Gather the planner statistics. A restore brings the rows but not the # statistics, and until autovacuum gets round to every table the planner # guesses at their sizes, which can turn a sub-second page into one that # takes many seconds. docker compose exec -T db vacuumdb -U healthlog -d healthlog --analyze-in-stages -# 8. Start the app (and app-worker, if you run it) and confirm it. +# 7. Start the app and confirm it. docker compose up -d app curl -s http://localhost:3000/api/version ``` -The start log should say there are no pending migrations: the dump carried the -migration history with it. Sign in and open a few pages before you go on. - -## Afterwards - -Keep the old `pgdata` volume and the dump until the new database has run for a -few days. Going back means restoring the old `image:` and `volumes:` lines; -the old volume still holds the database as it was before step 2. When you are -sure, remove it (`docker volume ls` shows its full name, prefixed with the -compose project): - -```bash -docker volume rm _pgdata -``` +The start log should say there are no pending migrations beyond the release's +own: the dump carried the migration history with it. If you had run `CREATE EXTENSION pg_stat_statements;` for [query statistics](query-statistics.md), the dump carries it and the compose diff --git a/docs/self-hosting/getting-started.md b/docs/self-hosting/getting-started.md index e9cd933a8..f4fb5d208 100644 --- a/docs/self-hosting/getting-started.md +++ b/docs/self-hosting/getting-started.md @@ -81,7 +81,9 @@ explicitly in `.env` (see the commented example there). ### Using an existing or shared Postgres -HealthLog can live in a database on a Postgres instance you already run. +HealthLog can live in a database on a Postgres instance you already run, +version 16 or newer; the compose file ships 18, and every release is tested +on both. It stores every time as UTC and pins each connection it opens to UTC (`TimeZone=UTC`, sent with the libpq `options` startup parameter), so the server's or the database's own `timezone` setting does not matter: an @@ -105,9 +107,12 @@ ALTER DATABASE healthlog SET timezone TO 'UTC'; docker compose up -d ``` -Compose starts two services: +Compose starts two services and one short-lived step: -- `db` — PostgreSQL 16 with a named volume for persistence. +- `db` — PostgreSQL 18 with a named volume for persistence. +- `db-upgrade` — moves a Postgres 16 database from v1.45 or earlier into + the new one, once, and exits; on a new install it has nothing to do. See + [Moving to a newer Postgres major](../ops/postgres-major-upgrade.md). - `app` — HealthLog itself, listening on `localhost:3000`. The image runs `prisma migrate deploy` on boot so the schema lands automatically; subsequent restarts no-op when the database is diff --git a/docs/self-hosting/portainer/README.md b/docs/self-hosting/portainer/README.md index b9da05db8..aa8dc2f9c 100644 --- a/docs/self-hosting/portainer/README.md +++ b/docs/self-hosting/portainer/README.md @@ -1,7 +1,7 @@ # HealthLog on Portainer HealthLog ships a Portainer **app template** that deploys the app and its -PostgreSQL 16 database as one stack from the project `docker-compose.yml`. +PostgreSQL 18 database as one stack from the project `docker-compose.yml`. ## Add the template @@ -39,7 +39,7 @@ curl -s http://:3000/api/version ## Notes -- Persistent state lives in one named volume (`pgdata`, the Postgres data dir); +- Persistent state lives in one named volume (`postgres-data`, the Postgres data); the app container is stateless. Back up the database to back up everything. - Pin a release tag (`HEALTHLOG_IMAGE_REF=:vX.Y.Z`) for deliberate upgrades, or track `:latest`. Skim the diff --git a/docs/self-hosting/scaling.md b/docs/self-hosting/scaling.md index 77759eeeb..c5f0b0071 100644 --- a/docs/self-hosting/scaling.md +++ b/docs/self-hosting/scaling.md @@ -174,7 +174,7 @@ sized for the web traffic alone leaves the worker's jobs queueing for the same connections. Plan total Postgres slots as **container_count × budget**. -A stock Postgres 16 container ships with `max_connections = 100`, so +A stock Postgres container ships with `max_connections = 100`, so the safe envelope is: | Web replicas | Worker replicas | Total containers | Pool slots used | Headroom under 100 | diff --git a/docs/self-hosting/truenas.md b/docs/self-hosting/truenas.md index 460867eb4..d312e4fb7 100644 --- a/docs/self-hosting/truenas.md +++ b/docs/self-hosting/truenas.md @@ -213,7 +213,7 @@ the menu next to Custom App, Install via YAML, using these values: | User | runs as uid `1001` by default; no extra capabilities, `no-new-privileges` is fine | | Health check | `GET /api/health`, `200` when ready | | Start period | at least 60 s, with more retries than the image's 5 × 30 s: migrations run at every boot | -| Database | PostgreSQL 16 or 18, no extensions, in its own container with its own dataset | +| Database | PostgreSQL 18 (16 works too), no extensions, in its own container with its own dataset | | Volumes | none for the app; the database gets its own dataset | | Variable | Required | Notes | diff --git a/docs/self-hosting/unraid.md b/docs/self-hosting/unraid.md index 77abd18f7..87c0e91df 100644 --- a/docs/self-hosting/unraid.md +++ b/docs/self-hosting/unraid.md @@ -69,10 +69,12 @@ curl -s http://tower:3000/api/version Use this if you prefer the per-app Unraid UI. You install PostgreSQL yourself, then add HealthLog pointed at it. -### Step 1 — install PostgreSQL 16 +### Step 1 — install PostgreSQL 1. In **Apps** (Community Applications), search for a **PostgreSQL** container - (for example jj9987's `postgresql`). Install version 16. + (for example jj9987's `postgresql`). Install version 18; an existing 16 + works too. Moving a 16 to 18 later is a dump and restore, see + [Moving to a newer Postgres major](../ops/postgres-major-upgrade.md#by-hand). 2. Set: - `POSTGRES_USER` = `healthlog` - `POSTGRES_PASSWORD` = your generated password diff --git a/docs/self-hosting/unraid/healthlog.xml b/docs/self-hosting/unraid/healthlog.xml index c6e92f981..c68c70c58 100644 --- a/docs/self-hosting/unraid/healthlog.xml +++ b/docs/self-hosting/unraid/healthlog.xml @@ -2,7 +2,7 @@