Only the backups page said when the bucket's lifecycle rules left
database/ out, so every dump, and every deleted account inside it, stayed
in the bucket forever without anyone being told. Readiness gains a
dump_retention signal that reads the same cached lifecycle probe: attention
when dumps go off-host and no rule covers database/, ok with the expiry
when one does, quiet when the bucket will not say, inactive without a
bucket or without off-host dumps. The hourly evaluation pushes it to the
admins like every other signal.
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.
A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.
The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
The Postgres major-upgrade guide suggested removing the old pgdata volume
"when you are sure". That volume is a full copy of the database as it stood
before the move: account deletions, trash purges and key rotations after it
never reach it. The guide now says so and makes removing it part of the
move.
The deploy-status webhook authenticated with a static secret in a header,
and the timestamp the docs called replay protection was covered by nothing:
whoever captures one request holds the secret and can send a fresh request
with any content. The strict setting, documented as closing the replay
window completely, only made that timestamp mandatory.
A sender can now sign instead: X-Deploy-Webhook-Signature is the
HMAC-SHA256 of the timestamp and the exact body under the shared secret,
with the timestamp within five minutes, and the secret never travels. The
strict setting accepts only signed requests. The secret header still works
by default for senders that cannot sign, and the docs describe its
timestamp as the staleness hint it is.
Several surfaces promised more than the per-account copies deliver:
- The full-backup export card said it had the same shape as the weekly
copy. It is a portable export with documents as metadata only; it now
says so, and that it cannot recreate an account on another server.
- The export page offered to "import from a backup". The JSON import adds
measurements and mood entries; it is now named that way, and its help
states the 16 MB and 10,000-entries-per-list limits from the constants
the route enforces.
- The weekly copy is labelled an account rollback on this server, and the
backups page says a copy cannot rebuild a lost server. The upload says
the account must exist here. Two restore refusals pointed to a
"disaster-recovery" copy; they now name the copies that exist.
- The instance-settings restore option was described as being for
rebuilding a host.
docs/ops/backup-restore.md is rewritten short and accurate: what each
copy is, the deletion grace window, that database dumps keep a deleted
account until they age out, how to restore one account, and a link to
the disaster-recovery procedure for a lost server.
The new-server steps lead with `db-restore latest --from-bucket`, the copy
that survives when the host and its local dumps are gone, and keep the
restore from downloaded files beside it. Both backup pages now say that the
database dumps are not purged per account: a deleted account stays in them
until they age out, which is the retention period an operator states to the
people using the instance.
docs/ops/disaster-recovery.md says what is dumped and where, that local
dumps are unencrypted and why, which keys a restore needs and what each
one costs when missing, how to restore onto a new server or back to the
state before a migration, and how the procedure is tested. The deploy
runbook and the getting-started guide now describe the dump the
entrypoint takes before migrating, and the off-host account copy page
points at the dump for rebuilding a server.
Fix two dead anchors (the MCP trust model, the README quick start) and the
statements that no longer match the app: a guardian can connect a managed
profile's services, the environment module is on by default and reaches the
air-quality host too, its readable location columns were dropped in 0409,
settings sections carry their current names, the Withings card lives on
/settings/integrations, the TrueNAS start period matches the image, the
deploy check names HEALTHLOG_PROD_URL, and the docs index describes what
self-hosting/ and integrations/ hold.
A concurrent build that is still running is INVALID with its migration not
yet finished, exactly like a failed one. With the web and worker roles in
two containers, the second boot dropped the index the first was building,
which cancelled the first one's migration and left it failed until someone
resolved it by hand.
The check and the drop now run in one transaction that first takes Prisma's
migration lock without waiting and does nothing when another instance holds
it; an index Postgres reports as being built is never dropped. One
transaction also keeps lock_timeout on the statement it bounds behind a
transaction pooler, and the applied-migrations lookup quotes the schema, so
a mixed-case schema no longer reads as having none applied.
The compose file stays on 16. Changing only the image line stops the stack:
the 18 image refuses a data directory from an older major and expects its
volume one level up. The runbook walks the dump and restore into a fresh
volume, keeps the old one as the way back, and is linked from the update
section and the compose pin.
A CREATE INDEX CONCURRENTLY that fails half-way leaves an INVALID index,
and the migration re-run after migrate resolve --rolled-back finds it
through IF NOT EXISTS and does nothing: recorded applied, with an index
the planner never uses. 0477 has this shape.
The boot now drops an INVALID index that a not-yet-applied migration
builds concurrently before it runs migrate deploy, with a bounded lock
wait, so the re-run builds it again. An INVALID index whose migration is
recorded applied is named on stderr with the rebuild that mends it. The
deploy runbook says both.
Since handlers load on their queue's first job, a module the image
cannot load no longer fails the worker's boot. The worker comes up
healthy, the queue burns its retries when it first has work, and a
one-shot job (a restore, an import tail) ends failed with nobody told at
deploy time.
After binding every queue the worker now runs `preloadHandlers` in the
background, without awaiting it, so readiness is unchanged. It loads
each handler once through the same memoised loader its queue is bound
to, so a module that loads stays loaded for the first job. Each failure
is reported on the `job.run.failed` line at error, with the queue's name
and `reason: "handler_load_failed"`, plus the worker error report
(stderr and GlitchTip). A summary line on stderr names every queue that
failed. The sweep never throws, and a failing report does not stop it.
docs/ops/deploy.md says what to look for after a deploy.
The TrueNAS page now walks through the catalog install field by field, how
to generate the two secrets, where TrueNAS keeps the key and why that is not
a backup, a dump and restore from the postgres container shell, and updates
with forward-only migrations. The listing is marked as pending review, and
the install-via-YAML values stay for the time until it is live.
The deploy notes list what needs a matching truenas/apps pull request in the
same release cycle: a new or stricter required variable, a port or health
check change, a Postgres major or extension, and any other compose-shape
change.
The nightly measurements check reported its bloat factor, a ratio such as
1.3, as a job fact. A fact is a non-negative whole number, so the outcome
validation refused it on every run, after the vacuum and the reindex had
already done their work, and the run failed. It now reports bloat_percent
(130), and the duration goes through the same rounding.
A malformed fact no longer fails a run whose work is done: runJob drops it
from the result and emits job.outcome.fact_invalid at error level with the
queue and the key, where failing would have made pg-boss retry finished
work. A failed outcome is unchanged. wholeFact and percentFact are the
intended way to report a measured value or a ratio, and a guard reads
every outcome producer and refuses a fact that is a division or names a
ratio unless it is rounded on purpose. The other producers were checked:
their numeric facts are counts and integer millisecond durations.
Refs #1271
A web-only process opens its queue connection without migrating and refuses a job-queue schema older than its own, so in a web/worker split the new worker has to start, and upgrade the schema from 45 to 46, before the new web containers. The v1.45.0 notes say so, name the one additive migration, state that a single-container install needs no action, and point a forked compose file at the new stop grace period.
0440 adds its columns with ADD COLUMN IF NOT EXISTS like 0424 and
0432; a structural test holds every migration from 0422 on to that
form. 0430 clears insights_cached_at where no sealed payload is left,
so the nightly pregenerate refills an account whose only copy was the
readable column instead of keeping an empty card fresh.
The deploy runbook notes that 0430 drops a users column: an old process
fails on its next full user read, so stop before starting rather than
running old and new side by side.
The heap sample named old_space alone, so a heap that stood well above
it could not be explained from the log: a buffer past V8's large-object
threshold lives in large_object_space, compiled code and bytecode in the
code and trusted spaces. The sampled and pressure lines now carry the
used size of every space, with any space the runtime reports that is
not named summed into other_spaces_mb, so the fields account for the
whole used heap whatever the Node version.
A marked migration's DO block holds the locks it takes until it commits,
so an app transaction taking two of its tables in the other order can
deadlock with it. The block then rolled back whole, exactly as on a lock
timeout, so the boot retries 40P01 for marked migrations as it does 55P03.
Corrects 0415's note that the drops hold their lock for an instant.
0416 ran DROP INDEX CONCURRENTLY at boot, which waits without bound for
every transaction on the table; a pg_dump or a backfill outlasted the
start period and the next boot stopped at P3009. The migration now tries
a plain drop with a two-second lock wait and tolerates the timeout, and
the nightly measurement maintenance drops what is left concurrently under
a bounded wait, retrying the next night. A fresh or shadow database takes
the lock at once, so the history still ends where the schema does. The
guard refuses a concurrent drop in a migration and ties every retired
index to a tolerant one.
The entrypoint migrates before the server listens, and a migration that
waits out a held lock retries for up to about eight minutes, while the
health check marked the container unhealthy after two. The start period
is now ten minutes in the compose file and the image; the first answering
probe still ends it, so a normal boot reports healthy as before.
medication_compliance_rollups lost its last reader when compliance moved
to the schedule-anchored computation, yet every dose write still upserted
it, a boot backfill refolded it, and the wipe and backup plans carried it.
The per-dose hook keeps the one derived row that still hangs off a dose
day, the supplement nutrients, under dose-day-refresh. The pending-row
mints (today projection, reminder) no longer call it, since an untaken
dose adds no nutrients. The import job finishes without its per-day
recompute phase; its progress keeps touchedDays and rollupProcessed on
the wire, inert and documented as deprecated. The backfill queue is
retired on boot and migration 0414 drops the table under a bounded lock
wait.
Prisma ignores PGTZ, so a database whose default zone is not UTC ran every
migration in that zone, and a migration mixing timestamp and timestamptz
wrote shifted values. The boot wrapper now appends -c TimeZone=UTC to the
options parameter of DATABASE_URL for its Prisma calls, honouring
DATABASE_SESSION_OPTIONS_DISABLED like the app does.
migrate deploy runs a migration file statement by statement, so the
top-level SET LOCAL lock_timeout in 0412 did nothing: the migration
waited as long as the lock was held, with every write to measurements
queued behind it, and a file that failed half-way stayed half-applied.
0412 now takes its locks inside one DO block with a statement-scoped
timeout, which rolls back whole, after an idempotent CREATE OR REPLACE.
The entrypoint runs migrate deploy through scripts/migrate-deploy.mjs.
A 55P03 failure of a migration marked -- @retry-on-lock-timeout is
marked rolled back and retried after 15/30/60/120/240 s, read from
Prisma's report or, after P3009, from the stored row. Every other
failure stays fatal. A guard holds migrations to the shape that makes
the marker true, and 0411/0413 add their columns with IF NOT EXISTS so
a manual re-run is safe too.
The audit log is the trail of what a person or an admin did to an account.
Every failed provider sync wrote a row there, burying that trail, and kept
the provider's error message in plain text while the status row seals it.
A failed sync now emits an integrations.sync.failed warning with the same
counts and no message.
Migration 0412 starts autovacuum on the table at 5 % dead rows, and the
nightly check vacuumed at 10 %, a point autovacuum had always passed first.
Two triggers for one task only hid which of them did it. The nightly check
now measures and changes nothing: it warns when the dead rows reach four
times autovacuum's threshold, which means autovacuum is not keeping up, and
when the heap has outgrown its live rows. The operator's run keeps its
steps.
`updated_at` sits in the sync-feed index, so every update that moved it
wrote a new row version on another page and an entry in every index.
Provider re-syncs rewrite many unchanged rows, which grew the table to
several times its live size and sent each unchanged row down the sync
feed again. Migration 0412 adds a row trigger that keeps `updated_at` and
`sync_version` when an update changes nothing else, so such an update is
heap-only and the feed stays silent; it sets `fillfactor = 85` and a 5 %
autovacuum threshold. The maintenance queue gains a nightly check that
vacuums when dead rows pass a tenth of the live ones and warns when the
heap is more than three times what its live rows need, which only a
`VACUUM FULL` in a window gives back. The admin card keeps showing the
operator's runs only.
The pressure report re-armed only after the heap fell under 70 % and
half an hour passed, so a heap that climbed to its limit soon after an
85 % report died with only that older report on record. A 95 % tier
now writes the same report at once, whatever the cooldown, and re-arms
under 85 %.
The image sets NODE_OPTIONS so Node writes its diagnostic report to
stderr when the heap runs out, with environment variables and network
interfaces excluded. The heap-diagnostics and scaling docs say how to
keep the flags when overriding NODE_OPTIONS.
Heap figures share one megabytes helper with the support report.
runtime.heap.sampled is written every minute while the heap is above
70 % of its limit, which turned the 500-entry ring behind the admin log
page over in hours and pushed out the events an operator opens it for.
emitEvent takes a skipMemoryRing option; the sample sets it and still
reaches stdout and Loki, while the pressure and snapshot lines go
everywhere as before.
The snapshot directory was /tmp/healthlog-heap-<pid>, a predictable name
(pid 1 in most containers) that mkdirSync accepted when it already
existed or was a symlink. It now comes from mkdtempSync and is checked
with lstat to be a real directory owned by the process user and closed
to others before the snapshot is written.
runtime.heap.sampled records heap, old space, RSS, external memory and
the work in flight every five minutes, every minute at or above 70 % of
the heap limit. At 85 % runtime.heap.pressure lists the oldest requests
and jobs in flight and the routes most of them share, once, re-armed
only after the heap falls below 70 % and thirty minutes pass. A
heap-limit crash leaves no other record of what the process was doing.
HEALTHLOG_HEAP_SNAPSHOT=1 adds one heap snapshot at that moment into
HEALTHLOG_DIAG_DIR (0700 directory, 0600 file), off by default.
docs/ops/heap-diagnostics.md warns that a snapshot holds decrypted
health data and keys and says how to delete it.
The weekly copy read every stored document and ECG trace whole, then held
their base64, the section JSON and the preview copies beside them. A vault
of 190 MB peaked at 930 MB of heap and took a 524 MB process down on every
run. Documents now leave the database one at a time in 1.5 MB slices, the
traces a page at a time, every other section is written an element at a
time, and a sealed piece never exceeds its chunk size. The restore keeps
document bytes out of its first read and writes each document from a
second one. Same file format, byte for byte.
Refs #1239
An account whose only strong credential is a primary passkey was refused
auth.stepup.mfa_not_enrolled on every action behind requireFreshMfa, so it
could not create, edit, hand over or delete a managed profile or manage its
guardians, although a passkey sign-in or re-proof already stamps the
session the gate reads.
requireFreshMfa now defaults to the any-possession proof source: a passkey
makes the gate reachable, and a passkey re-proof through the dialog clears
it. The refusal offers passkey in meta.methods, never the password.
hasSecondFactorEnrolled keeps its narrow meaning on purpose. It decides
whether the conditional gates apply at all (account deletion, the data
reset, the password change, the export, offering manage access, the
recent-proof branches and the native Bearer rules), so none of those
change, and password sign-in still challenges only TOTP and security keys.
The MFA-management routes keep the second-factor source. The decision and
its limits are written beside FreshFactorProofSource.
Copy, the OpenAPI description and the sharing and key-rotation docs say
"second factor or passkey" where this gate applies.
Refs #959
The repair covers the last 75 days of daily means and the 75 days before
the raw window of hourly means; older days stay as they are. The ops page
names the two new run counters.
The daily-mean consolidation and the dense hourly fold cut at
`now - threshold` instead of a local day boundary. A day was folded in two
runs and the second run overwrote the stored mean with the mean of the
later part alone: every mean-type day, and the boundary hour of every
heart-rate, HRV and SpO2 day, plus the derived resting figure and the DAY
rollup of such a day. The Health Connect importer split its raw window at
the same unaligned instant.
- Fold up to the start of the account's local day the threshold ends in
(`foldBoundary`), for every consolidation pass and the Health Connect
importer's raw/hourly split.
- A window that already has a live `stats:` row is refolded from every
sample of it, live rows plus the fold's own tombstones; with none left
the stored mean stays.
- `folded_window` refuses a raw sample only when its local day lies before
the aligned boundary and a live `stats:` row of its source covers it,
also within reach of the sample after a timezone change.
- A deletion inside a day the fold had not finished is class B.
- Add a one-time, resumable fold repair per account that recomputes each
hourly and daily mean from its remaining samples and writes only values
that differ, recorded in `measurement_fold_repairs` (migration 0389).
- The compaction-tombstone purge waits for the repair per account, and the
boot sends no purge run before one account is repaired.
- The restore keys live `stats:` ids by source, and writes compaction
tombstones back from a file older than the account's repair, resetting
the repair.
The nightly folds soft-deleted every raw Apple Health sample they folded
into an hourly or daily stats row and kept it for 75 days. Nothing ever
read those rows as deletions; they only held a re-upload of the sample
off, and they were most of the measurements table and its indexes.
Ingest now refuses such a re-upload directly: a raw sample of a dense or
mean type, old enough to have been folded, whose hour or day a live
stats row covers, is a duplicate with reason folded_window. The batch
route and the Apple Health export import apply the same rule. With that
in place the dense retention and the daily-mean consolidation delete the
raw rows outright, and a new compaction-tombstone-purge job removes the
backlog: 5,000 rows a batch, one account at a time under the restore
lock, a short pause between batches, resumable because the predicate is
idempotent, queued at every boot and continuing itself while it makes
progress. A person's own deletions are never in its predicate. The sync
feed leaves compaction tombstones out and a restore does not write them
back; all three classify through the same function.
The batch route answers an exact re-sent sample without a transaction:
one read over the batch's sample ids, live or deleted, decides what the
reconciler would have decided, at the same point in the loop, so every
status, reason, counter and the cross-source merge stay as they were. A
batch of duplicates only no longer queues PR detection, a reminder sweep
or the detection audit row. Caches are evicted only for a foreground or
manual sync or a batch that wrote a hand-entered reading, and marked
stale otherwise; the targets cache follows the same split.
Also:
- POST /api/admin/maintenance/measurements queues VACUUM (ANALYZE) and a
REINDEX INDEX CONCURRENTLY per index, largest first, on a connection of
its own; it refuses while the purge runs, drops invalid leftover copies
and stops at the first failed rebuild. Runbook in docs/ops.
- The pure volume queues keep their finished jobs for 96 hours instead
of seven days, never less than the failure window plus a day.
- work_mem is set per app connection from DATABASE_WORK_MEM (16MB), also
when the statement timeout is disabled.
- The PR-detection fallback tick re-scans only accounts whose readings
or workouts changed since the previous tick.
- Type discovery for the status refill walks the live-rows index as a
loose index scan instead of grouping every live row.
- The host metrics chart polls every five minutes, like its sampler.
- The batch route records per-type HealthKit arrivals.
parseCoachPrefs parsed the stored blob as a whole and returned the
defaults on any invalid field. A lookback value this version does not
know therefore dropped the metric exclusions with it. Each field now
falls back on its own, and the exclusion and cluster lists keep their
known entries, so drift narrows what the Coach reads instead of widening
it.
The deploy runbook's rollback section notes the one case this cannot
cover: v1.40.1 and earlier still parse the blob whole, so an account that
saved the new "Last year" lookback loses its exclusions there.
Node sizes its heap from the container's memory cap. The 1g default gave
524 MB, which a long-used account fills in normal operation: on a live
instance 490 to 507 MB stayed live after a full GC, garbage collection
stalled requests for up to 8.5 s, and the process ran out of heap 26 times
in a week, taking the weekly backup with it. 1536m gives 792 MB. The
example env and the backup runbook say how to lower it on a small host.
Marc-André Bombeck