Restored rows keep the ids they had in the file. Once an erased
account's copy had gone into one account, the same file uploaded into a
second account restored into a primary-key conflict, reported only as
the generic "the database rejected a record" failure.
For a copy put into a chosen account, a primary-key conflict now ends
the restore with backup.restore.rows_exist_elsewhere and a sentence that
says the file was most likely restored into another account already.
The transaction still rolls back, so neither account changes. The code
is in the catalogue, the console names it, and the runbook says a file
goes into one account only.
The upload creates a passwordless account before it reads the file and
deletes it again when the file is refused. The only audit row was
written after a successful store, so a refused upload whose delete also
failed left an account nobody could sign into and no record of where it
came from.
The account and its admin.user.create row (outcome "created") now
commit together, and the discard writes admin.user.delete with outcome
"deleted" or "failed", so a leftover account always has a trace. The
runbook says how to remove one by hand.
A restore read the account settings under the operator's AI endpoint
policy whenever the receiving account was an admin. For a copy put into
a chosen account, the settings were written by the erased account's
owner, not by the operator, yet an admin account could be chosen as the
target, and an ordinary target promoted later got the same trust.
Admin accounts are now refused as a target, in the check and the
picker, with backup.target.admin_account. A copy uploaded into a chosen
account is stored as RETARGETED_UPLOAD_<ms>, and its restore never reads
its settings as the operator's, whatever the account's role is by then.
The empty-account check counted every backed-up model, and signing up
and answering the setup questions always leaves a setup row, plus a
cycle profile when the cycle area is chosen. The account of someone who
has signed up again, the case the restore target is for, was therefore
always refused.
Those two models are one row per account and are deleted and re-created
from the copy by every restore, so they no longer count. A reading or
medication entered on setup's last screen still refuses the account,
because the restore would delete it. A guard holds the exempt models to
that shape, and the erased-account integration file now drives the real
setup routes before the restore.
The runbook said a restore writes into an account that exists under the
same id, which left an operator with an erased account's file and no
way in. It now says where such a copy can go, how the new account is
signed into, why an account with records is refused, and what does not
come back.
DEPLOY_WEBHOOK_REQUIRE_TIMESTAMP had been changed to mean signed requests
only, so an operator who set it for a sender that adds a timestamp but cannot
sign would have had every deploy notification refused after the upgrade. It
means the secret header plus a fresh timestamp again. Signature-only, the
setting that stops a replay, is DEPLOY_WEBHOOK_REQUIRE_SIGNATURE, on the
compose whitelist, in the env manifest and examples, and in the deploy
guide.
A cookie named healthlog_session can be set by any host under the same
registrable domain, so a page on a sibling subdomain could plant a session of
its choosing in a visitor's browser. Wherever the session cookie is Secure it
is now issued as __Host-healthlog_session, which a browser accepts only from
this exact host. Plain-HTTP installs, which cannot hold a __Host- cookie,
keep the old name.
Nobody is signed out: the old name is still read after the new one, a
session found under it is re-issued under the new name and the old cookie
expired, and a browser holding the new cookie is never answered from the old
one. Every reader and writer goes through one module, which the proxy, the
native-caller check, the native handoff and the OIDC callback use; the
session-surface guard follows the new reader.
Off-host dumps were named by their time alone. Two instances sharing a bucket
(nothing forbids it, and it was safe for the account objects) overwrote each
other's dump at the same second of the same default schedule, and
latest --from-bucket restored whichever instance's dump was newest. Each
instance now uploads under database/<host of APP_URL>/, the object key (and so
the encryption's associated data) carries it, and the restore takes only this
instance's dumps: it refuses another's and names the instances it found,
unless --instance=<host> or the dump's full key says which one on purpose.
A restore killed outright (out of memory, docker kill) left its .restore- or
.download- directory, a whole database decrypted, in the backups directory,
where nothing removed it. A restore now removes them before it starts and on
SIGINT and SIGTERM, and the nightly retention sweeps any older than twelve
hours.
The notes said the other migrations only add columns and one concurrent
index. 0487 drops four app_settings columns the previous image reads on
nearly every request, and 0488 builds its index on users without
CONCURRENTLY. The rollback step now names the columns to hand back, and the
notes say what v1.45 does with accounts pending deletion and held wipe
purges, and how to settle them before going back.
Pre-migration dumps were kept by count alone, so an install that updates
rarely kept its last two, and every account deleted since, indefinitely,
while the docs and the backups card stated the nightly retention as the
retention of deleted accounts. A pre-migration dump older than the oldest
nightly dump kept now goes, except the newest one, the way back from the
latest update, which stays until the next update. The card, the disaster
recovery guide and its variable table say so.
The recovery kit, "Keys the backups still need" and safeToDropRetiredKeys
read only the stored copies and the off-host account objects. After a
rotation the page called the old key safe to drop while every dump taken
before it, the pre-migration dumps that are the way back from an update
among them, could only be restored with that key: db-restore refuses a dump
whose active key is missing.
A dump now counts for the key its rows were written under, the manifest's
active key: a dump on this server while it is kept (read from the manifests),
an off-host one until the bucket's rule for database/ expires it, or
indefinitely when that rule cannot be read. The nightly run records the key
on its ledger row (migration 0489, database_backup_runs.active_key_id). The
encryption view and the kit list the dumps per key, in every locale, and the
rotation and backup runbooks say so.
The reproductive-health page promised that deleting all data or the
account erases at once. Both wait for the grace window now, the nightly
dumps keep a record until they age out, and the old pgdata volume is a
full copy until it is removed; the page says so and how to make deletion
final. The README, the getting-started note on majors, the move guide and
the Unraid and Portainer templates catch up with Postgres 18, the backups
volume and the recovery kit. The demo seed's spread bound leaves room for
the seed's own one-in-150 spread.
Going back to Postgres 16 after a move that stopped half-way left the
moving marker, and the next start emptied the live 16 database and
started a second server on its data directory. db-upgrade now waits for
the database service and does nothing unless it runs Postgres 18.
Using 16 again after a finished move and then updating started the app
on the move-day copy without a word. The marker now records the old
database's last checkpoint, and a later start refuses when it moved on;
the guide says how to move the current database instead. The disk check
leaves room for the first pre-migration dump, and the move check seeds
the old database with v1.45's own migrations.
The fourth rollback step had been joined onto the paragraph after the
list; it is a list item again, with the way back to the data after it.
The disaster recovery page now says how the server tells admins about
its copies: the recovery readiness card, the recovery kit and the hourly
Backup health notification. The support page put GlitchTip under Admin,
General; the card is under Admin, Integrations, as the deploy runbook
says. The pre-migration dump notes say that without the backups volume
no dump is taken, also where the image's own directory is writable.
The restore refuses a file that carries any document as metadata only
(findIncompleteDocument), whatever the account holds; the page said it
refused when the account held a document.
Only the backups page said when the bucket's lifecycle rules left
database/ out, so every dump, and every deleted account inside it, stayed
in the bucket forever without anyone being told. Readiness gains a
dump_retention signal that reads the same cached lifecycle probe: attention
when dumps go off-host and no rule covers database/, ok with the expiry
when one does, quiet when the bucket will not say, inactive without a
bucket or without off-host dumps. The hourly evaluation pushes it to the
admins like every other signal.
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.
A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.
The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
The Postgres major-upgrade guide suggested removing the old pgdata volume
"when you are sure". That volume is a full copy of the database as it stood
before the move: account deletions, trash purges and key rotations after it
never reach it. The guide now says so and makes removing it part of the
move.
The deploy-status webhook authenticated with a static secret in a header,
and the timestamp the docs called replay protection was covered by nothing:
whoever captures one request holds the secret and can send a fresh request
with any content. The strict setting, documented as closing the replay
window completely, only made that timestamp mandatory.
A sender can now sign instead: X-Deploy-Webhook-Signature is the
HMAC-SHA256 of the timestamp and the exact body under the shared secret,
with the timestamp within five minutes, and the secret never travels. The
strict setting accepts only signed requests. The secret header still works
by default for senders that cannot sign, and the docs describe its
timestamp as the staleness hint it is.
Several surfaces promised more than the per-account copies deliver:
- The full-backup export card said it had the same shape as the weekly
copy. It is a portable export with documents as metadata only; it now
says so, and that it cannot recreate an account on another server.
- The export page offered to "import from a backup". The JSON import adds
measurements and mood entries; it is now named that way, and its help
states the 16 MB and 10,000-entries-per-list limits from the constants
the route enforces.
- The weekly copy is labelled an account rollback on this server, and the
backups page says a copy cannot rebuild a lost server. The upload says
the account must exist here. Two restore refusals pointed to a
"disaster-recovery" copy; they now name the copies that exist.
- The instance-settings restore option was described as being for
rebuilding a host.
docs/ops/backup-restore.md is rewritten short and accurate: what each
copy is, the deletion grace window, that database dumps keep a deleted
account until they age out, how to restore one account, and a link to
the disaster-recovery procedure for a lost server.
The new-server steps lead with `db-restore latest --from-bucket`, the copy
that survives when the host and its local dumps are gone, and keep the
restore from downloaded files beside it. Both backup pages now say that the
database dumps are not purged per account: a deleted account stays in them
until they age out, which is the retention period an operator states to the
people using the instance.
docs/ops/disaster-recovery.md says what is dumped and where, that local
dumps are unencrypted and why, which keys a restore needs and what each
one costs when missing, how to restore onto a new server or back to the
state before a migration, and how the procedure is tested. The deploy
runbook and the getting-started guide now describe the dump the
entrypoint takes before migrating, and the off-host account copy page
points at the dump for rebuilding a server.
Fix two dead anchors (the MCP trust model, the README quick start) and the
statements that no longer match the app: a guardian can connect a managed
profile's services, the environment module is on by default and reaches the
air-quality host too, its readable location columns were dropped in 0409,
settings sections carry their current names, the Withings card lives on
/settings/integrations, the TrueNAS start period matches the image, the
deploy check names HEALTHLOG_PROD_URL, and the docs index describes what
self-hosting/ and integrations/ hold.
A concurrent build that is still running is INVALID with its migration not
yet finished, exactly like a failed one. With the web and worker roles in
two containers, the second boot dropped the index the first was building,
which cancelled the first one's migration and left it failed until someone
resolved it by hand.
The check and the drop now run in one transaction that first takes Prisma's
migration lock without waiting and does nothing when another instance holds
it; an index Postgres reports as being built is never dropped. One
transaction also keeps lock_timeout on the statement it bounds behind a
transaction pooler, and the applied-migrations lookup quotes the schema, so
a mixed-case schema no longer reads as having none applied.
The compose file stays on 16. Changing only the image line stops the stack:
the 18 image refuses a data directory from an older major and expects its
volume one level up. The runbook walks the dump and restore into a fresh
volume, keeps the old one as the way back, and is linked from the update
section and the compose pin.
A CREATE INDEX CONCURRENTLY that fails half-way leaves an INVALID index,
and the migration re-run after migrate resolve --rolled-back finds it
through IF NOT EXISTS and does nothing: recorded applied, with an index
the planner never uses. 0477 has this shape.
The boot now drops an INVALID index that a not-yet-applied migration
builds concurrently before it runs migrate deploy, with a bounded lock
wait, so the re-run builds it again. An INVALID index whose migration is
recorded applied is named on stderr with the rebuild that mends it. The
deploy runbook says both.
Since handlers load on their queue's first job, a module the image
cannot load no longer fails the worker's boot. The worker comes up
healthy, the queue burns its retries when it first has work, and a
one-shot job (a restore, an import tail) ends failed with nobody told at
deploy time.
After binding every queue the worker now runs `preloadHandlers` in the
background, without awaiting it, so readiness is unchanged. It loads
each handler once through the same memoised loader its queue is bound
to, so a module that loads stays loaded for the first job. Each failure
is reported on the `job.run.failed` line at error, with the queue's name
and `reason: "handler_load_failed"`, plus the worker error report
(stderr and GlitchTip). A summary line on stderr names every queue that
failed. The sweep never throws, and a failing report does not stop it.
docs/ops/deploy.md says what to look for after a deploy.
The TrueNAS page now walks through the catalog install field by field, how
to generate the two secrets, where TrueNAS keeps the key and why that is not
a backup, a dump and restore from the postgres container shell, and updates
with forward-only migrations. The listing is marked as pending review, and
the install-via-YAML values stay for the time until it is live.
The deploy notes list what needs a matching truenas/apps pull request in the
same release cycle: a new or stricter required variable, a port or health
check change, a Postgres major or extension, and any other compose-shape
change.
The nightly measurements check reported its bloat factor, a ratio such as
1.3, as a job fact. A fact is a non-negative whole number, so the outcome
validation refused it on every run, after the vacuum and the reindex had
already done their work, and the run failed. It now reports bloat_percent
(130), and the duration goes through the same rounding.
A malformed fact no longer fails a run whose work is done: runJob drops it
from the result and emits job.outcome.fact_invalid at error level with the
queue and the key, where failing would have made pg-boss retry finished
work. A failed outcome is unchanged. wholeFact and percentFact are the
intended way to report a measured value or a ratio, and a guard reads
every outcome producer and refuses a fact that is a division or names a
ratio unless it is rounded on purpose. The other producers were checked:
their numeric facts are counts and integer millisecond durations.
Refs #1271
A web-only process opens its queue connection without migrating and refuses a job-queue schema older than its own, so in a web/worker split the new worker has to start, and upgrade the schema from 45 to 46, before the new web containers. The v1.45.0 notes say so, name the one additive migration, state that a single-container install needs no action, and point a forked compose file at the new stop grace period.
0440 adds its columns with ADD COLUMN IF NOT EXISTS like 0424 and
0432; a structural test holds every migration from 0422 on to that
form. 0430 clears insights_cached_at where no sealed payload is left,
so the nightly pregenerate refills an account whose only copy was the
readable column instead of keeping an empty card fresh.
The deploy runbook notes that 0430 drops a users column: an old process
fails on its next full user read, so stop before starting rather than
running old and new side by side.
The heap sample named old_space alone, so a heap that stood well above
it could not be explained from the log: a buffer past V8's large-object
threshold lives in large_object_space, compiled code and bytecode in the
code and trusted spaces. The sampled and pressure lines now carry the
used size of every space, with any space the runtime reports that is
not named summed into other_spaces_mb, so the fields account for the
whole used heap whatever the Node version.
A marked migration's DO block holds the locks it takes until it commits,
so an app transaction taking two of its tables in the other order can
deadlock with it. The block then rolled back whole, exactly as on a lock
timeout, so the boot retries 40P01 for marked migrations as it does 55P03.
Corrects 0415's note that the drops hold their lock for an instant.
0416 ran DROP INDEX CONCURRENTLY at boot, which waits without bound for
every transaction on the table; a pg_dump or a backfill outlasted the
start period and the next boot stopped at P3009. The migration now tries
a plain drop with a two-second lock wait and tolerates the timeout, and
the nightly measurement maintenance drops what is left concurrently under
a bounded wait, retrying the next night. A fresh or shadow database takes
the lock at once, so the history still ends where the schema does. The
guard refuses a concurrent drop in a migration and ties every retired
index to a tolerant one.
The entrypoint migrates before the server listens, and a migration that
waits out a held lock retries for up to about eight minutes, while the
health check marked the container unhealthy after two. The start period
is now ten minutes in the compose file and the image; the first answering
probe still ends it, so a normal boot reports healthy as before.
medication_compliance_rollups lost its last reader when compliance moved
to the schedule-anchored computation, yet every dose write still upserted
it, a boot backfill refolded it, and the wipe and backup plans carried it.
The per-dose hook keeps the one derived row that still hangs off a dose
day, the supplement nutrients, under dose-day-refresh. The pending-row
mints (today projection, reminder) no longer call it, since an untaken
dose adds no nutrients. The import job finishes without its per-day
recompute phase; its progress keeps touchedDays and rollupProcessed on
the wire, inert and documented as deprecated. The backfill queue is
retired on boot and migration 0414 drops the table under a bounded lock
wait.
Prisma ignores PGTZ, so a database whose default zone is not UTC ran every
migration in that zone, and a migration mixing timestamp and timestamptz
wrote shifted values. The boot wrapper now appends -c TimeZone=UTC to the
options parameter of DATABASE_URL for its Prisma calls, honouring
DATABASE_SESSION_OPTIONS_DISABLED like the app does.
Marc-André Bombeck