mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-09 21:10:59 +00:00
fix(backup): store a backup in sealed pieces instead of one value
A stored backup was one value in data_backups.data. v1.39.1 wrote it a piece at a time but still joined the pieces into that one value, and one value has to pass through the app whole, so a copy was capped at a fifth of the heap: 105 MB in the default 1 GB container. An account with 1.75 million readings is past that and could not be backed up at all (#1031). A copy is now kept as ordered pieces of about a megabyte in the new data_backup_chunks table (migration 0352). Each piece is sealed on its own with AES-256-GCM, and its sealed header names the copy (a random stream id kept on the row), its position and whether it is the last. A reader checks every piece before it releases a byte, so a piece that is missing, moved, altered, taken from another copy, or a copy cut short is refused with backup.payload.undecryptable before a restore deletes anything, and the second read during the restore checks each piece again. The write is one transaction, so the previous copy stays whole and readable until the new one commits. Pieces are written through the model API: a raw query with a large parameter is kept alive by Prisma's query plan cache, which left 160 MB of heap behind per hundred one-megabyte inserts. Restore, preview, download, the restore job and the admin list read both forms; copies written before this change keep their single value and still restore. The heap-derived limit is gone. The one limit left is on storage, BACKUP_MAX_STORED_MB (default 2048), and its message names the setting. Key rotation covers the pieces, the wipe and account deletion take them with their backup, and the backup plan lists the table as not carried. Measured on an account of 2.6 million readings in a 1 GB container with web and worker in one process: backup 87 s, 97 MiB in 96 pieces, preview 20 s, download 25 s, restore 264 s with every reading back exactly, peak 567 MiB, no restart.
This commit is contained in:
1 parent
b0f9bbe530
commit
1e09f30eff
39 files changed
+1700
-546
No files matched your search
@@ -288,6 +288,11 @@
|
||||
"name": "TRUST_PROXY_HOPS",
|
||||
"purpose": "Trusted reverse-proxy hop count; the client IP is read that many entries from the right of X-Forwarded-For.",
|
||||
"example": "1"
|
||||
},
|
||||
{
|
||||
"name": "BACKUP_MAX_STORED_MB",
|
||||
"purpose": "Largest stored backup of one account the app keeps, in megabytes of encrypted copy (2048 by default). A storage limit, independent of the app's memory: a backup past it fails with a message naming this variable, and the previous copy stays.",
|
||||
"example": "2048"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -855,14 +855,27 @@ async function main() {
|
||||
// operator that zero meant safe to drop the old key, and every stored backup
|
||||
// became undecryptable. Walked in bounded id-cursor batches because one row
|
||||
// is an entire compressed account, and re-sealed WITHOUT reading the
|
||||
// plaintext, so both stored envelopes (the plain backup JSON and the
|
||||
// `HLZ1:` gzip form) — and any later one — rotate unchanged.
|
||||
// plaintext, so every single-value envelope (the plain backup JSON, the
|
||||
// `HLZ1:` gzip form and the v1.39.1 `~hlgcm1.` stream) rotates unchanged.
|
||||
results.push(
|
||||
await rotateRegistryColumn("DataBackup", "data", {
|
||||
dataBackup: prisma.dataBackup,
|
||||
} as unknown as CorpusClient),
|
||||
);
|
||||
|
||||
// ───── Whole-account backup in pieces (Bytes, binary2, batched) ─────
|
||||
// From v1.39.2 a backup is stored as sealed pieces in "data_backup_chunks",
|
||||
// one `encryptBytes()` value each, and the "data" column above holds only
|
||||
// copies written before that. Walked in id-cursor batches of a few pieces
|
||||
// (each is about a megabyte) and re-sealed without reading the backup: the
|
||||
// header that ties a piece to its copy and position is inside the
|
||||
// ciphertext, so it comes through rotation unchanged.
|
||||
results.push(
|
||||
await rotateRegistryColumn("DataBackupChunk", "data", {
|
||||
dataBackupChunk: prisma.dataBackupChunk,
|
||||
} as unknown as CorpusClient),
|
||||
);
|
||||
|
||||
// ───── Idempotent-replay response cache (String, disposable) ─────
|
||||
// `IdempotencyKey.responseBody` is the cached response body, encrypted
|
||||
// because the PHI-returning creates echo their own decrypted DTO. Rotating
|
||||
|
||||
Reference in new issue
Block a user