The pressure report re-armed only after the heap fell under 70 % and
half an hour passed, so a heap that climbed to its limit soon after an
85 % report died with only that older report on record. A 95 % tier
now writes the same report at once, whatever the cooldown, and re-arms
under 85 %.
The image sets NODE_OPTIONS so Node writes its diagnostic report to
stderr when the heap runs out, with environment variables and network
interfaces excluded. The heap-diagnostics and scaling docs say how to
keep the flags when overriding NODE_OPTIONS.
Heap figures share one megabytes helper with the support report.
runtime.heap.sampled is written every minute while the heap is above
70 % of its limit, which turned the 500-entry ring behind the admin log
page over in hours and pushed out the events an operator opens it for.
emitEvent takes a skipMemoryRing option; the sample sets it and still
reaches stdout and Loki, while the pressure and snapshot lines go
everywhere as before.
The snapshot directory was /tmp/healthlog-heap-<pid>, a predictable name
(pid 1 in most containers) that mkdirSync accepted when it already
existed or was a symlink. It now comes from mkdtempSync and is checked
with lstat to be a real directory owned by the process user and closed
to others before the snapshot is written.
runtime.heap.sampled records heap, old space, RSS, external memory and
the work in flight every five minutes, every minute at or above 70 % of
the heap limit. At 85 % runtime.heap.pressure lists the oldest requests
and jobs in flight and the routes most of them share, once, re-armed
only after the heap falls below 70 % and thirty minutes pass. A
heap-limit crash leaves no other record of what the process was doing.
HEALTHLOG_HEAP_SNAPSHOT=1 adds one heap snapshot at that moment into
HEALTHLOG_DIAG_DIR (0700 directory, 0600 file), off by default.
docs/ops/heap-diagnostics.md warns that a snapshot holds decrypted
health data and keys and says how to delete it.
The weekly copy read every stored document and ECG trace whole, then held
their base64, the section JSON and the preview copies beside them. A vault
of 190 MB peaked at 930 MB of heap and took a 524 MB process down on every
run. Documents now leave the database one at a time in 1.5 MB slices, the
traces a page at a time, every other section is written an element at a
time, and a sealed piece never exceeds its chunk size. The restore keeps
document bytes out of its first read and writes each document from a
second one. Same file format, byte for byte.
Refs #1239
An account whose only strong credential is a primary passkey was refused
auth.stepup.mfa_not_enrolled on every action behind requireFreshMfa, so it
could not create, edit, hand over or delete a managed profile or manage its
guardians, although a passkey sign-in or re-proof already stamps the
session the gate reads.
requireFreshMfa now defaults to the any-possession proof source: a passkey
makes the gate reachable, and a passkey re-proof through the dialog clears
it. The refusal offers passkey in meta.methods, never the password.
hasSecondFactorEnrolled keeps its narrow meaning on purpose. It decides
whether the conditional gates apply at all (account deletion, the data
reset, the password change, the export, offering manage access, the
recent-proof branches and the native Bearer rules), so none of those
change, and password sign-in still challenges only TOTP and security keys.
The MFA-management routes keep the second-factor source. The decision and
its limits are written beside FreshFactorProofSource.
Copy, the OpenAPI description and the sharing and key-rotation docs say
"second factor or passkey" where this gate applies.
Refs #959
The repair covers the last 75 days of daily means and the 75 days before
the raw window of hourly means; older days stay as they are. The ops page
names the two new run counters.
The daily-mean consolidation and the dense hourly fold cut at
`now - threshold` instead of a local day boundary. A day was folded in two
runs and the second run overwrote the stored mean with the mean of the
later part alone: every mean-type day, and the boundary hour of every
heart-rate, HRV and SpO2 day, plus the derived resting figure and the DAY
rollup of such a day. The Health Connect importer split its raw window at
the same unaligned instant.
- Fold up to the start of the account's local day the threshold ends in
(`foldBoundary`), for every consolidation pass and the Health Connect
importer's raw/hourly split.
- A window that already has a live `stats:` row is refolded from every
sample of it, live rows plus the fold's own tombstones; with none left
the stored mean stays.
- `folded_window` refuses a raw sample only when its local day lies before
the aligned boundary and a live `stats:` row of its source covers it,
also within reach of the sample after a timezone change.
- A deletion inside a day the fold had not finished is class B.
- Add a one-time, resumable fold repair per account that recomputes each
hourly and daily mean from its remaining samples and writes only values
that differ, recorded in `measurement_fold_repairs` (migration 0389).
- The compaction-tombstone purge waits for the repair per account, and the
boot sends no purge run before one account is repaired.
- The restore keys live `stats:` ids by source, and writes compaction
tombstones back from a file older than the account's repair, resetting
the repair.
The nightly folds soft-deleted every raw Apple Health sample they folded
into an hourly or daily stats row and kept it for 75 days. Nothing ever
read those rows as deletions; they only held a re-upload of the sample
off, and they were most of the measurements table and its indexes.
Ingest now refuses such a re-upload directly: a raw sample of a dense or
mean type, old enough to have been folded, whose hour or day a live
stats row covers, is a duplicate with reason folded_window. The batch
route and the Apple Health export import apply the same rule. With that
in place the dense retention and the daily-mean consolidation delete the
raw rows outright, and a new compaction-tombstone-purge job removes the
backlog: 5,000 rows a batch, one account at a time under the restore
lock, a short pause between batches, resumable because the predicate is
idempotent, queued at every boot and continuing itself while it makes
progress. A person's own deletions are never in its predicate. The sync
feed leaves compaction tombstones out and a restore does not write them
back; all three classify through the same function.
The batch route answers an exact re-sent sample without a transaction:
one read over the batch's sample ids, live or deleted, decides what the
reconciler would have decided, at the same point in the loop, so every
status, reason, counter and the cross-source merge stay as they were. A
batch of duplicates only no longer queues PR detection, a reminder sweep
or the detection audit row. Caches are evicted only for a foreground or
manual sync or a batch that wrote a hand-entered reading, and marked
stale otherwise; the targets cache follows the same split.
Also:
- POST /api/admin/maintenance/measurements queues VACUUM (ANALYZE) and a
REINDEX INDEX CONCURRENTLY per index, largest first, on a connection of
its own; it refuses while the purge runs, drops invalid leftover copies
and stops at the first failed rebuild. Runbook in docs/ops.
- The pure volume queues keep their finished jobs for 96 hours instead
of seven days, never less than the failure window plus a day.
- work_mem is set per app connection from DATABASE_WORK_MEM (16MB), also
when the statement timeout is disabled.
- The PR-detection fallback tick re-scans only accounts whose readings
or workouts changed since the previous tick.
- Type discovery for the status refill walks the live-rows index as a
loose index scan instead of grouping every live row.
- The host metrics chart polls every five minutes, like its sampler.
- The batch route records per-type HealthKit arrivals.
parseCoachPrefs parsed the stored blob as a whole and returned the
defaults on any invalid field. A lookback value this version does not
know therefore dropped the metric exclusions with it. Each field now
falls back on its own, and the exclusion and cluster lists keep their
known entries, so drift narrows what the Coach reads instead of widening
it.
The deploy runbook's rollback section notes the one case this cannot
cover: v1.40.1 and earlier still parse the blob whole, so an account that
saved the new "Last year" lookback loses its exclusions there.
Node sizes its heap from the container's memory cap. The 1g default gave
524 MB, which a long-used account fills in normal operation: on a live
instance 490 to 507 MB stayed live after a full GC, garbage collection
stalled requests for up to 8.5 s, and the process ran out of heap 26 times
in a week, taking the weekly backup with it. 1536m gives 792 MB. The
example env and the backup runbook say how to lower it on a small host.
A weekly or nightly backup job whose process died (a restart, a
container out of memory) stayed active until its two- or four-hour
expiry, and its retry walked the accounts in the same order: the account
that killed it has the oldest copy, so it led again, and every account
after it went without a copy for as long as that record stayed too
large.
Both passes now send with a five-minute pg-boss heartbeat, so a dead run
fails within minutes and its retry picks it up. Each pass records when it
started and finished every account (backup_pass_attempts, migration
0373); an account whose last attempt never finished goes behind all the
others. The backups page shows when the run in progress started and names
an account a run died under, for the weekly and the off-host card, and a
heartbeat timeout reads as what it means.
The drill read only the alphabetically last object of the newest date, so
one account stood in for all of them. It now checks every object of that
date one at a time, records a failure against the account it belongs to,
keeps going, and pages once per failing object with its key and how many
accounts were checked.
The replay cell was keyed on (user, key, method, path). A client that sends
two different writes under one key got the first write's cached response for
the second, and the second was never saved. Native clients up to 1.1.0 post
the two halves of a manual blood pressure exactly that way, so the diastolic
value was silently dropped.
Each cell now stores a SHA-256 fingerprint of the request body (JSON with
recursively sorted keys, otherwise the raw bytes; scoped by method and path;
at most the first MiB is read). Lookup replays, or refuses as in flight, only
when the fingerprint matches. A different body, or a row written before the
column existed, runs the handler without claiming or caching and leaves the
existing cell for its own retries; a lost claim race re-checks the winner's
fingerprint the same way. The body itself is never stored.
Migration 0371 adds the nullable idempotency_keys.request_fingerprint column.
The OpenAPI parameter description, the client notes and the Home Assistant
guide say what a replay now requires. docs/ops gains a read-only query that
lists systolic readings left without their diastolic half, held to the
schema by an integration test.
The boot key check recorded whatever key was configured whenever its probe
found nothing to try. It only read string columns, so an install whose
ciphertext sat in Bytes columns (notes, documents, ECG, symptoms) read as
empty; a probe that ran out of time did the same; and rows written by a
process that served under a wrong key could open on the next boot and get
that key recorded as the right one.
The probe now reads every registered column, string and Bytes, in all four
stored layouts, and tries the oldest values first, ordered by the row's
updatedAt (else createdAt, else primary key). A key is recorded only when
the oldest value that opens is preceded by at most one that does not, or
when no column holds a single value under the key id. A time-out, an
unreadable table, oversized values or mixed results are inconclusive: the
process serves, records nothing and checks again at the next start.
When the first insert loses a race, the stored record is read back and must
open under this process's key, else the outcome is a mismatch.
A rotation that walked every column without errors removes the record of
each previous key id that no longer holds any value. The refusal log block
and the self-hosting pages explain how to reset the check for a key id.
Every configured key id now has a canary in encryption_key_canaries,
checked at boot. A key that cannot open its canary, or existing data
written before the canary existed, puts the process in a refusal state:
every API route answers 503 encryption.key_mismatch, /api/health reports
reason encryption_key_mismatch, the pages render one explanation, and
neither the queue producer nor the worker starts. The process stays up
so the reason is visible instead of hiding in a crash loop.
The admin Encryption page gains a "Back up your encryption key" step:
key id and fingerprint (never the key), where the key lives per
platform, a check of a pasted copy that is compared and discarded, and
a confirmation bound to the active key id and fingerprint. The overview
gets a tile and admins get a dashboard banner while it is due.
HEALTHLOG_PLATFORM picks the platform the step opens on. Adds the
TrueNAS self-hosting page and key backup sections in the docs.
The admin wipe of every account is the operator clearing their own host,
and the bucket's copies are the documented way back from a wipe pressed
by mistake, so it writes no off-host deletion request. Deleting an
account, deleting a managed profile and "Delete all data" still do.
The backup runbook no longer says the worker never deletes an object: it
lists the grant the worker needs (DeleteObject included), when objects
are deleted, the version-4 envelope, the lifecycle check and how to
rotate the off-host key. The rotation runbook no longer says backups are
covered: rotation reaches a backup's envelope, not the content a restore
writes back, and a retired key stays needed until no backup lists it.
The privacy notice beside account deletion says the same about off-host
copies.
A failed import records its failure in a plain try block, and the
extraction test that inflates past the 64 MiB floor gets the time it
needs on a loaded runner.
The password-reset runbook tells an operator to run
`docker compose exec app node scripts/reset-password.mjs`, and the
intake-repair runbook runs `healthlog-tsx scripts/repair-intake-anomalies.ts`
inside the container. The runner stage copied nothing from scripts/, so
both failed with "Cannot find module", the reset at the moment an admin
is locked out.
The runner stage now copies both scripts and the Argon2 parameter module
the reset imports, and proves at build time that each reaches its own
first check with every import resolved. A structural test holds every
in-container command in docs/ and the Dockerfile copies in step.
Key rotation is documented the way it works on the published image: the
Rotate now button under Admin > Encryption is the primary path. The
rotation script imports the application graph and runs from a source
checkout of the same release; the runbook, the admin hint in all seven
languages and the decrypt error now say so.
The runbook now says that key rotation converts copies still stored as one
value into pieces a slice at a time, that a backup skips an account whose
copy is being restored, that two backups of one account run one after the
other, what a reader answers when its copy is replaced mid-read, and that
migration 0352 cannot be undone by going back to v1.39.1.
The drill section said the drill needs nothing beyond the uploader's
GetObject and ListBucket, but the documented uploader grant is PutObject,
GetObject and AbortMultipartUpload. The drill lists the bucket to find the
newest object, so ListBucket is an addition, not something already granted.
The weekly-backup section still described one stored value capped at a
fifth of the heap and advised more memory. It now says how a copy is stored
and checked, that its size no longer depends on memory, what
BACKUP_MAX_STORED_MB limits and how to raise it, how older copies are read,
and how key rotation handles both forms. It also states that a 512 MB
container is below the app's own working set during a backup.
Key rotation and the rotation status read every column that was not
flagged as a blob with one `findMany` over the whole table.
`Measurement.notesEncrypted` and `MoodEntry.noteEncrypted` are such
columns, so rotation loaded every measurement of the instance: 1.25
million rows on one large account (#1031), and the rotation status page
did the same on every view.
Every column is now walked with the id-cursor pages the blob columns
already used: 25 rows a page for blobs, 5 000 for everything else. The
maintenance script's own per-column walks page the same way. On the seeded
account with 1 460 notes to rotate, peak heap fell from 272 MB to 153 MB,
and it no longer grows with the table.
`scanned` now counts the rows that hold ciphertext for every column, as
it already did for the blob columns, rather than every row of the table
for some columns and only the ciphertext rows for others. The runbook
says so.
The update runbook ran migrations with a plain docker run and an
.env.production file. No setup guide creates that file, and a plain
docker run starts outside the Compose network, where the database's
service name does not resolve, so the step waited thirty seconds and
gave up with a connection error.
It uses docker compose run now, which starts the one-off container in
the stack with the app's own environment. Refs #961
The badge read the enabled flag plus a DSN that parses. Neither says anything
about the target accepting a report: a wrong public key, a wrong project id and
a host that stopped answering all parse, and every one of them showed green
with a line naming the host. That is the false reassurance an operator finds
out about when they go looking for a crash that was never reported.
Nothing in the tree recorded an outcome, so there was nothing to read. The
sender now stamps one next to the DSN it used — instant on a success, a short
classification on a failure. Never the response body: that is text from a
remote host and has no business in this database or on this page.
The badge is green only while a report has left inside the last fortnight,
amber while a target is set and nothing has come back, and it names the reason
underneath. The clock lives with the ledger, so the verdict is resolved in the
route and the card paints what it is handed. A failure that a later success
superseded is not shown at all. Pressing Test records like any other send, so
the button proves the target on demand.
On the release that adds the ledger the table is empty, so on a host that has
been uploading nightly for months every account read Never, under a sentence
saying no copy had ever reached the bucket. That sentence is untrue on exactly
the morning an operator opens the new card for the first time, and it is a
claim the host cannot make: what is empty is the record, not the bucket.
The run now records every account it walks, not only the ones an object came
of. A walk that produced nothing moves the attempt instant and leaves the
success columns alone, so the two cases separate: no row at all is no record
yet, a row without a success is never. Never now means what the word says —
a run looked at this account and has never put anything there.
The card names the last check on the never row, so the state is a fact with a
timestamp rather than an assertion. Runbook and all seven bundles carry the
same two sentences. The integration file proves both arms against real
Postgres, including that a failed run leaves the previous good copy on the
row.
The threshold was the schedule period with no budget on top of it. Two things
spend that budget on a host where nothing is wrong. The cron runs 02:30
Europe/Berlin, so on the DST fall-back Sunday the gap between two runs is 25
hours and every account on every host would cross one period on the same
morning. And the ledger instant is when that account's object landed, by
design, so on a cohort walked one account at a time an account reached an hour
later tonight than last night drifts past 24 with two clean runs behind it.
The schedule period and the verdict threshold are now separate numbers. The
period is still what the card quotes; the threshold is the period plus six
hours, applied to both steps, so due still means one run produced nothing and
stale still means two. The fall-back night is pinned in the unit file by its
real dates rather than by an arithmetic 25, and a genuinely missed night at 31
hours still reads due. The runbook says why the slack is there.
There is no default error-reporting DSN in this tree and there should not be:
a self-hoster's crash reports must not leave their host because nobody told
them not to. That part was already right, so nothing here adds a default.
What was wrong is what the console said about it. The GlitchTip card showed a
green "Configured" badge as soon as a DSN existed, whether or not the switch
that actually sends anything was on, and it never said where reports go. An
operator reading that badge has been told their errors are being collected
somewhere, which may be true of neither half.
The badge now tracks reports actually leaving, and one line under the switch
states the case: the host they go to, or that they stay here, or that tracking
is on with no target. The host only — the public key in the DSN answers a
different question. The deploy runbook says the same thing for a
maintainer-operated instance.
The nightly off-host job reports counts — uploaded, failed, total — and a
count cannot name the account it is failing for. A cohort where one object
has been refused every night for a month reads as "99 uploaded, 1 failed"
each time, which looks like weather rather than like an account with no
disaster-recovery copy at all.
The worker now writes a ledger row when an object lands (account, instant,
size), and Admin → Backups gains a second card that rates every account
against the nightly schedule: fresh inside one run, due when one run
produced nothing, stale when two did not, never when no object has ever
landed. On a host without BACKUP_S3_* the card says off-host backup is not
configured and stops there.
The verdicts are computed from the ledger, never from a bucket listing: the
worker's grant is deliberately PutObject + GetObject, and an admin page must
not put the operator's credentials on its render path to state a number the
database already holds.
Two gaps in `backup-restore.md`. It said the restore "replaces the account's
data tables" only in a paragraph about grants, so nothing stated the
consequence for the account's own record: the rows it gained after the snapshot
was taken are deleted with the rest of their class. And the console restore was
named once in passing — `## Restore` documents only the off-host script — so an
operator restoring from the browser had no procedure to read.
Both are written down now, including which button does what, that the restore
targets the snapshot's owner rather than the admin running it, that
instance-wide settings ride along only when the dialog's opt-in is ticked, and
what a copy this instance can no longer decrypt answers.
E2. `loadOffhostConfig` parsed BACKUP_RETENTION_DAYS, clamped it to a
minimum of one day and put it on the config object, where no caller read
it. Neither annotation fits: it is not deliberately internal, because
there is no internal use, and no consumer is pending. The design forbids
one outright — the worker's grant excludes DeleteObject on purpose, so
nothing here can act on a retention window without giving a compromised
worker the ability to wipe the history. A clamp that guards nothing only
read as an enforcer the worker is not.
So the field and the clamp go, and the reason sits where the parse was.
The variable keeps its place in the two example files and on the compose
whitelist: it is the number the operator sets the bucket's own lifecycle
rule to. The docs table said "defaults to 30", which implied the app had
an opinion; it now says the app does not read it.
Post-publish verify proves the published image boots. Nothing proved that
on a running instance a person can sign in, write a reading and read it
back — which is exactly the shape of the defects this line keeps
rediscovering: a value written and never read.
`scripts/synthetic-journey.mjs` walks that journey in plain Node 22 with no
dependencies, so it runs from a workstation, a runner or the deploy host.
It warms `/`, `/api/health`, `/api/version` and the measurements list first
and discards those timings (the first hit on each route after a deploy is a
cold start), then asserts five legs in order: the served version equals the
expected one, the password flow signs in through a cookie jar, a weight
reading is written with an Idempotency-Key, the same row reads back with
its value, unit and source, and the reversible delete leaves the account
clean. One line per leg with the elapsed time; the first failure ends the
run non-zero with the status and a body excerpt that has every token shape
redacted out of it.
The workflow is `workflow_dispatch` only, with `base_url` and
`expected_version` as inputs and the credentials read from repository
secrets. It is deliberately not wired into the deploy path: the target
instance and the secrets are the maintainer's call.
Mutation proof, against a production build served from
`.next/standalone/server.js` on a throwaway Postgres.
Green:
warm 35 ms (discarded)
ok version 2 ms serves 1.38.13
ok sign in 78 ms session cookie held
ok write 64 ms wrote cmtup9lwu0007zhovn0mxajdz
ok read back 8 ms 77.7 kg (MANUAL)
ok delete 27 ms account left clean
synthetic journey green
Read leg broken deliberately — the same server behind a throwaway forwarder
that empties the `GET /api/measurements` list and changes nothing else:
ok version 4 ms serves 1.38.13
ok sign in 38 ms session cookie held
ok write 40 ms wrote cmtupa4ck000gzhovwnij64fz
FAILED read back 8 ms read back: the list came back without the reading
status 200
body {"data":{"measurements":[],"meta":{"total":0}},"error":null}
synthetic journey RED on "read back"
Version leg broken the same way, by asking the instance for the previous
tag:
FAILED version 3 ms version: instance serves "1.38.13", expected "v1.38.12"
synthetic journey RED on "version"
`--self-test` carries that proof in the repository: it runs the journey
against an in-process instance nine times, once with nothing broken and
once per broken leg, and fails if a broken leg still comes back green. The
workflow runs it before it touches the real instance.
Gates: typecheck, `node --check`, lint (the three known warnings, no
errors), prettier, the workflow and CI-determinism guards.
The runbook said the worker prunes objects older than BACKUP_RETENTION_DAYS. It never calls DeleteObject; its grant is PutObject, GetObject and AbortMultipartUpload, on purpose, so a compromised worker cannot wipe the history. The sentence now says what expires objects and what the variable is for.
The rollups and the cached status insights are recomputed after the
transaction, so a throw there left the rows repaired and every chart on
the pre-repair numbers — and no later run would notice, because the
audit row makes the account skip from then on.
The tail is wrapped now. The audit row is written with
`rollupsRefreshed: false` and only flipped once the recompute returned,
so a re-run can tell "repaired, rollups pending" from "done" and says so.
A run that could not finish the tail names the account, prints the
backfill to re-run — `scripts/backfill-rollups.ts --user <userId>` — and
exits non-zero.
Review finding M2.
An in-place multiplier with no reverse operation was documented with only
a dry run in front of it. Point at the backup runbook where the write
happens.
Review finding M1.
The provenance criterion proves where a row came from, not which build
imported it. An account re-imported on the fixed build carries the same
`EXPORT_XML_SOURCE_MAX` stamp and the same `stats:` external id with the
right numbers and no audit row, so it is selected exactly like a broken
one — and its rest days, under 200 m because the phone stayed at home,
survive the plausibility check after another thousandfold multiply.
So one candidate row that would leave the range now refuses the whole
account: nothing written, no audit row, the offending rows and a printed
reason instead. The idempotency guard only ever held against the
script's own prior run; this holds against the case it could not see.
The runbook says the same in words — a re-imported account must not be
repaired, the re-import already did it — and says what the dry run is
for: it is where the decision is made, not a formality in front of one
already taken.
Integration coverage for the re-imported account: correct rows, an
`--apply` run, nothing changes and no audit row is left behind.
Review finding H1.
The unit fix only helps the next import. An account that imported an
archive before it carries a decade of days understated 1000x, and the
authoritative repair — re-importing the archive on the fixed build —
needs an archive the account may no longer have.
`scripts/repair-apple-health-distance.ts` is dry-run by default and
selects rows only by the stamp the archive fold itself writes: the
EXPORT_XML_SOURCE_MAX provenance plus the `stats:` external id minted
for that fold. Never by "the value looks small", so a row a native sync
has since overwritten is left alone. `--apply` multiplies one account
per transaction and writes the audit row in the same transaction, which
is what makes a second run a no-op. The archive's own unit is named by
the operator (`--unit=mi`) because the discarded attribute is the defect
itself, and a row whose repaired value would leave the plausible range
is reported rather than written.
Refs #944
The off-host job was said to benefit from the streaming work the weekly
in-database backup got, and it did not. Only the payload builder was
shared: `runOffhostBackup` called `buildFullBackupJson` for the whole
document, `gzipSync` over that whole string, a whole-buffer
`createCipheriv` pass over the result, and handed the finished buffer to
a single `PutObject`. Four full copies of the record alive at once, on
the same arithmetic the weekly pass had already been fixed for.
Configuring an S3 target on a long-lived record therefore restarted the
container. On an account of 445 000 measurements the JSON alone is
242 MB, and against a 1 GB container's 524 MB heap the run died of heap
exhaustion seventeen seconds in — taking every signed-in session with
it, because the job shares the app process.
The write path is a stream end to end now: `streamFullBackupJson`
produces the document a page at a time, gzip and an incremental
AES-256-GCM writer consume it as it arrives, and `@aws-sdk/lib-storage`
puts it up in two 8 MB parts. What the process holds is fixed by the
pipeline's shape rather than by the record. Measured on that same
445 000-measurement account under `--max-old-space-size=450`, with the
heap pre-loaded to what a warm server holds: the old path dies with
`Ineffective mark-compacts near heap limit`, the new one finishes with a
peak 79 MB above its baseline and writes a 9.1 MB object.
The stored envelope gains version 3, which moves the GCM tag from in
front of the ciphertext to the end. That is the only change the stream
required — the tag exists only once the last block is in, so a leading
one means the whole object must exist before its first byte can be sent.
Nothing about the authentication moves: the tag still covers every
ciphertext byte and is verified before a byte of plaintext comes back.
Versions 1 and 2 still read, so every object already in a bucket
restores through the same reader with no flag to tell them apart.
The incremental writer is the crypto module's, split into its byte-level
half so the base64 form the in-database blob uses and the raw form an
object needs are one implementation under two framings. The off-host key
handling is unchanged: a separate `BACKUP_ENCRYPTION_KEY`, 64 hex or
32-byte base64, passed in rather than looked up.
A run that uploaded nothing for anybody is a failed job now, not
`ok: true` with an empty bucket behind it. Wrong credentials, a missing
bucket and an unreachable endpoint all fail every account rather than
one, and the target's own sentence rides out as the cause. A run where
some account got a copy still succeeds; retrying the whole cohort over
one object would re-upload everybody's.
The one ceiling left is structural rather than a memory bound, and it is
counted rather than discovered: a multipart upload carries 10 000 parts,
so an object past 80 GB is refused for that account with a clear message
instead of failing halfway through with an SDK error about part numbers.
`tests/integration/offhost-backup-streaming-memory.test.ts` pins the
budget from both sides — what the streaming uploader holds, and that the
materialising path does not fit the same budget on the same fixture in
the same process, so the number cannot pass by measuring nothing.
The writer added in v1.38.6 compared the process's live heap usage against
80 % of V8's heap limit and aborted above it. That reading is not the
backup's footprint; it is the whole process's, garbage included. A
long-running app sits at 400 MB of largely collectable heap, so on a 1 GB
container (a 419 MB budget) the weekly pass aborted the first flush of every
account: four out of four in seven seconds, the smallest of them a demo
record whose entire stored copy is 1.2 MB, each told its record was too large
and its operator to buy memory.
Drop the gauge. What the writer holds is bounded by its shape and needs no
reading to prove: the three unbounded tables are pulled a page at a time,
every other section is released as soon as its JSON is in the sink, and at
most one flush of text is pending. The one copy that does grow with the
record is the stored blob, so the bound moves there — counted in the
ciphertext bytes actually produced, against a fifth of this process's heap
limit, which is what two copies of the answer plus a live set have to fit
inside. The message names both numbers.
A pass that wrote nothing for anybody now fails its job instead of reporting
`ok: true` with `backed: 0`, so the failing-queue panel and the backups page
read the same thing an operator would. A pass where some accounts were
written keeps passing with the failures as counts.
Measured under --max-old-space-size=450 from a heap deliberately dirtied
first: 445 000 measurements peak at 28 MB held by the backup, a one-row
record at under 1 MB.
`DataBackup.data` holds every weekly disaster-recovery snapshot and every
uploaded pack, AES-256-GCM at rest like everything else, under a column
called `data`. The rotation registry was keyed on the `*Encrypted` naming
convention, so the script never read a backup, reported zero rows
remaining, and the runbook read that zero as permission to retire the
previous key. Following the documented procedure destroyed every stored
backup — the copy an operator reaches for precisely when nothing else
works.
Register it, and register the idempotent-replay response cache the same
sweep turned up. The backup column walks in bounded batches (one row is a
whole compressed account) and re-seals the ciphertext without reading the
plaintext, so both stored envelopes come back byte-identical and so will
any later one. The cache is marked disposable: a row it cannot read is
dropped rather than counted as an error, because keeping it only
guarantees the next replay gets a body it cannot parse.
Running the script against a seeded database rather than reading it turned
up two more. It died on the mood day-context note, whose table is keyed on
the entry rather than on an `id`, and abandoned every column after it —
the backups included. And the document index's verbatim text rotated
without ever being named in the summary.
Make the naming assumption stop carrying weight. A new guard traces the
encryption helpers through their wrappers, walks every Prisma write
payload, and fails when a column receives ciphertext without being
registered. It asserts an anchored, non-zero match count so a matcher that
stops matching fails instead of passing. A second check pins each model's
primary key against the schema.
The script now ends by naming the registered columns it did NOT walk and
exiting non-zero when any is missing: a zero that means "nothing left" and
a zero that means "never looked" must not print the same. The runbook's
retire-the-key step was rewritten around that signal, and tells an
operator who already rotated on an older release how to get their backups
back.
The compression that landed last release was real and was measured in the
wrong place. On a development machine with a multi-gigabyte heap the pass
finished; the container the job runs in is capped at 1 GB, which gives V8
a 524 MB limit, and there the same pass died of memory about fifteen
seconds in. Because the job shares the application process, that was not
one failed backup — it was a restart, and every signed-in session on the
host went with it.
Reproduced before touching anything, under `--max-old-space-size=450`
against a seeded account of 445 000 measurements, 30 000 mood entries and
60 000 intake events: `FATAL ERROR: Reached heap limit` at 34 s. The same
fixture doubled to 890 000 measurements now completes inside a 296 MB
limit, holding 16 MB of live heap where the old writer held 168 MB.
The document is written incrementally now. `buildFullBackupPayload` takes
a `deferBulk` mode that declares the three unbounded tables instead of
reading them, and `streamFullBackupJson` walks the payload in key order,
pulls those tables through page by page, and releases every other section
as soon as its JSON reaches the sink. `packBackupBlobStreaming` feeds that
into gzip and an AES-256-GCM stream, so nothing between the database rows
and the stored value ever exists whole. Both writers go through the same
per-row serialisers, and the round-trip suite pins their output byte for
byte rather than trusting that they agree.
The intake and mood reads are ordered totally now. A keyset cursor needs
it, and a partial order also let two exports of one account differ over a
tie, which every multi-dose day has.
The streamed ciphertext is a new shape at rest, because GCM only produces
its tag once the last block is in and a leading tag is what makes the old
layout unstreamable. It carries a marker no older writer can produce, so
the two families are disjoint by construction rather than by sniffing.
All three stored shapes — plain, compressed, streamed — restore, proved
both ways.
The writer also watches its own live heap and stops at 80 % of V8's
limit. An account too large for its host is now one failed backup,
counted in the run's meta, instead of a restart for everybody on it.
The stored column is the ceiling that is left: `data_backups.data` is a
single `text` value, so the finished artifact has to exist as one string,
and that is the only part of the job still growing with the record. The
operator runbook says so, with the number at which it starts to matter.
Marc-André Bombeck