fix(backup): stream documents and ECG traces through backup and restore

The weekly copy read every stored document and ECG trace whole, then held
their base64, the section JSON and the preview copies beside them. A vault
of 190 MB peaked at 930 MB of heap and took a 524 MB process down on every
run. Documents now leave the database one at a time in 1.5 MB slices, the
traces a page at a time, every other section is written an element at a
time, and a sealed piece never exceeds its chunk size. The restore keeps
document bytes out of its first read and writes each document from a
second one. Same file format, byte for byte.

Refs #1239
This commit is contained in:
Marc-André Bombeck committed 2026-10-18 13:10:19 +02:00
1 parent 9db0f09ba5
commit ac047854ba
12 files changed
+946 -229

No files matched your search

+15
View File
@@ -230,6 +230,21 @@ rather than by the size of the record going through it: measured on the same
died and the new one finished holding tens of megabytes, writing a 9.1 MB
object that restores to the identical record.
Up to 1.43.4 that held for the tables that are long in rows, not for the
ones that are heavy in bytes. The stored documents and the ECG traces were
still read whole, and a vault of 190 MB cost about five times that in heap:
the bytes, their base64, the section's JSON and the preview's copies of each
row. A single account with that many documents took a 524 MB process down on
every weekly run, while 1.25 million measurements beside them fitted easily.
Both writers now read the traces a page at a time and the documents one at a
time, each document's bytes in slices of 1.5 MB. Measured on that account
under `--max-old-space-size=524`, the peak went from 930 MB (a crash) to
209 MB. The restore had the same problem in the other direction: it kept
every document of the file in memory and peaked at 1.25 GB on the same copy.
It now leaves the stored bytes out of its first read and writes the
documents one at a time from a second one, and the copy restores every
document byte for byte inside the 524 MB heap.
One ceiling remains, and it is structural rather than a memory bound: a
multipart upload carries 10 000 parts, so 80 GB is the largest object one
account can produce. Past it the account's backup fails with a clear refusal,