mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-08 20:42:43 +00:00
Marc-André Bombeck
1e09f30eff
A stored backup was one value in data_backups.data. v1.39.1 wrote it a piece at a time but still joined the pieces into that one value, and one value has to pass through the app whole, so a copy was capped at a fifth of the heap: 105 MB in the default 1 GB container. An account with 1.75 million readings is past that and could not be backed up at all (#1031). A copy is now kept as ordered pieces of about a megabyte in the new data_backup_chunks table (migration 0352). Each piece is sealed on its own with AES-256-GCM, and its sealed header names the copy (a random stream id kept on the row), its position and whether it is the last. A reader checks every piece before it releases a byte, so a piece that is missing, moved, altered, taken from another copy, or a copy cut short is refused with backup.payload.undecryptable before a restore deletes anything, and the second read during the restore checks each piece again. The write is one transaction, so the previous copy stays whole and readable until the new one commits. Pieces are written through the model API: a raw query with a large parameter is kept alive by Prisma's query plan cache, which left 160 MB of heap behind per hundred one-megabyte inserts. Restore, preview, download, the restore job and the admin list read both forms; copies written before this change keep their single value and still restore. The heap-derived limit is gone. The one limit left is on storage, BACKUP_MAX_STORED_MB (default 2048), and its message names the setting. Key rotation covers the pieces, the wipe and account deletion take them with their backup, and the backup plan lists the table as not carried. Measured on an account of 2.6 million readings in a 1 GB container with web and worker in one process: backup 87 s, 97 MiB in 96 pieces, preview 20 s, download 25 s, restore 264 s with every reading back exactly, peak 567 MiB, no restart.
scripts/
One-shot operator and build scripts. These are not part of the running application — they are run by hand during a deploy, a backfill, or a release, or wired into CI. Recurring work belongs on pg-boss (src/lib/jobs/), not here.
How to run them
From a source checkout:
pnpm dlx tsx scripts/<file>.ts
The production image intentionally removes npm, pnpm, and Corepack. Only
container-safe scripts named by an operator runbook use its pinned
healthlog-tsx launcher; scripts that import the full application dependency
graph must run from a source checkout. scripts/ is excluded from the project
typecheck.
What lives here
generate-openapi.ts— emitsdocs/api/openapi.yamlfrom the registered Zod schemas (pnpm openapi:generate).check-openapi.ts— compares a freshly generated spec against the committed YAML; CI fails on drift (pnpm openapi:check).check-env.ts— pre-deploy env-var sanity check againstenv-manifest.json(pnpm check-env); wired into CI.assert-deploy.ts— post-deploy/api/versionassertion.rotate-encryption-key.ts— AES-256-GCM key rotation; seedocs/ops/encryption-key-rotation.md.restore-backup.ts— restore an off-host AES-GCM backup from an S3-compatible bucket.repair-apple-health-distance.ts— repair walking-distance rows theexport.zipimport stored 1000x too small; dry-run by default, seedocs/ops/apple-health-distance-repair.md.repair-intake-anomalies.ts— repair historic medication-ledger defects (duplicate dose-slot rows, implausibletaken_at); dry-run by default, seedocs/ops/intake-repair.md.backfill-rollups.ts,drain-per-sample-cumulative.ts,backfill-mood-note-column.ts— historical data backfills (note: the boot-timerollup-full-backfillqueue handles new accounts automatically; the CLI is the manual fallback).seed-demo.ts+seed-demo-v15.sql— populate a demo tenant.test-notifications.ts— exercise the notification dispatcher + reminder check.fetch-geolite2.sh— download the MaxMind GeoLite2 MMDBs (needs a licence key) beforedocker build.install-hooks.sh— install the local git hooks.generate-sw-version.mjs,print-bundle-report.mjs— service-worker version stamp and bundle-size report.i18n/— locale tooling;__tests__/— tests for the scripts above.
Conventions
See ../CLAUDE.md "Tests and commands" + "Self-hosting gotchas". Operator runbooks that drive these scripts live in ../docs/ops/.