fix(backup): bound the stored copy, not the process heap

The writer added in v1.38.6 compared the process's live heap usage against
80 % of V8's heap limit and aborted above it. That reading is not the
backup's footprint; it is the whole process's, garbage included. A
long-running app sits at 400 MB of largely collectable heap, so on a 1 GB
container (a 419 MB budget) the weekly pass aborted the first flush of every
account: four out of four in seven seconds, the smallest of them a demo
record whose entire stored copy is 1.2 MB, each told its record was too large
and its operator to buy memory.

Drop the gauge. What the writer holds is bounded by its shape and needs no
reading to prove: the three unbounded tables are pulled a page at a time,
every other section is released as soon as its JSON is in the sink, and at
most one flush of text is pending. The one copy that does grow with the
record is the stored blob, so the bound moves there — counted in the
ciphertext bytes actually produced, against a fifth of this process's heap
limit, which is what two copies of the answer plus a live set have to fit
inside. The message names both numbers.

A pass that wrote nothing for anybody now fails its job instead of reporting
`ok: true` with `backed: 0`, so the failing-queue panel and the backups page
read the same thing an operator would. A pass where some accounts were
written keeps passing with the failures as counts.

Measured under --max-old-space-size=450 from a heap deliberately dirtied
first: 445 000 measurements peak at 28 MB held by the backup, a one-row
record at under 1 MB.
This commit is contained in:
Marc-André Bombeck committed 2026-09-03 09:51:05 +02:00
1 parent 3cf65c2c90
commit 24f3afd464
10 files changed
+569 -104

No files matched your search

+25 -11
View File
@@ -150,15 +150,28 @@ Two things changed, and both matter to an operator:
released as soon as its JSON exists. On the same fixture doubled to 890 000
measurements, the pass completes inside a 296 MB heap limit; before the
change, half that record did not fit in 546 MB.
- **It stops itself.** The writer watches its own live heap and aborts at 80 %
of V8's limit. That backup then fails for that one account, is counted in the
run's `users_failed` and `heap_budget_trips` meta, and the pass carries on
with everybody else. A memory failure is now a failed job rather than a
restart for every user on the host.
- **It stops itself, on its own size.** The envelope writer counts the
encrypted bytes it has produced for an account and gives up on the account
whose stored copy would not fit this process — a fifth of V8's heap limit,
which is 105 MB on the 524 MB limit a 1 GB container gets. That backup then
fails for that one account, is counted in the run's `users_failed` and
`records_oversized` meta, and the pass carries on with everybody else. A
memory failure is a failed job rather than a restart for every user on the
host.
If `heap_budget_trips` is non-zero in `job.data_backup`, the answer is more
If `records_oversized` is non-zero in `job.data_backup`, the answer is more
memory, not a retry: raise the container's limit, or set
`NODE_OPTIONS=--max-old-space-size=<MB>` to something under it.
`NODE_OPTIONS=--max-old-space-size=<MB>` to something under it. The limit is
derived from the heap limit, so it rises with it.
The check deliberately reads bytes it produced and not the process's heap. A
version that read the heap shipped in v1.38.6 and compared the whole process's
live usage — garbage included — against 80 % of the limit. A Next.js server
that has been up for a week sits at 400 MB of largely collectable heap, so the
weekly pass aborted every account on the first chunk, including one whose whole
stored copy is 1.2 MB, and reported success while doing it. If you are on
v1.38.6, a `data-backup` run that finishes in seconds having backed up nothing
is that defect and not your record.
### The stored column is the remaining ceiling
@@ -167,10 +180,11 @@ exist as one value before it can be written — that copy cannot be streamed
away. It is the only thing left in the job that grows with the record: at a
compressed blob of ~42 MB the pass peaks around 236 MB, and the growth from
there is roughly twice the blob's size (the base64 answer, plus the driver's
copy of it on the way to the wire). A blob past roughly 150 MB — an account
several times larger than any seen so far — will hit a 524 MB limit again, and
the fix at that point is to stop storing the artifact in one column: chunk it
across rows, or keep only the off-host copy. Reading it back has the same
copy of it on the way to the wire). That is the ceiling the size check is set
against — a fifth of the heap limit, so two copies plus the process's own live
set still fit — and an account that reaches it fails as a job with a message
naming both numbers. The fix at that point is to stop storing the artifact in
one column: chunk it across rows, or keep only the off-host copy. Reading it back has the same
shape, and worse: a restore parses the whole document, so the read path needs
several times the blob in heap. An operator restoring a very large account
should give the container more memory for the duration.