fix(import): bound what an export may inflate to, allow one import per account, and never leave the upload behind

The extractor held the inflated export.xml to a flat 8 GiB whatever the
member occupied, and the header ratio check reads sizes the archive
writes itself: a member declaring an honest ratio could inflate at
1000:1, so an 8 MB upload could fill the temp directory. The streamed
byte count is now held to 200 times the member's compressed size on
disk (never less than 64 MiB, never more than 8 GiB), and a stored
member may not grow at all.

An account may have one Apple Health import queued or running. A
second upload with different bytes is refused with 409
import.apple_health.busy and the running job's id; the check and the
new row are one transaction under an advisory lock.

The staged upload is the person's export in plain text. It is now
removed on every exit that leaves no worker to read it: a refused or
broken multipart body, no worker running, a queue that takes no job
(the row is failed as well), the admin route's missing or unknown
target account, and an account deleted before the worker reached it.
The worker removes the files before recording a failure, so a row that
went with a deleted account no longer strands them. The 15-minute
import reconcile pass also sweeps staged files older than the longest
an import may run.
This commit is contained in:
Marc-André Bombeck committed 2026-09-26 20:11:45 +02:00
1 parent 1c7785c23b
commit 1b4594e69f
16 files changed
+549 -57

No files matched your search

+34 -1
View File
@@ -73,6 +73,39 @@ const MAX_DECOMPRESSED_BYTES = 8 * 1024 * 1024 * 1024;
*/
const MAX_COMPRESSION_RATIO = 200;
/**
* Floor for the output cap of a small member. The ratio cap alone would
* refuse a tiny, legitimately very compressible export; below this many
* bytes of output the ratio is not what protects the disk.
*/
const MIN_OUTPUT_CAP_BYTES = 64 * 1024 * 1024;
/**
* How many bytes one member may inflate to, from what it ACTUALLY occupies
* in the archive rather than what its header claims. The header is the
* attacker's to write: a member declaring an honest-looking 10:1 ratio can
* still inflate at 1000:1, and with only the flat 8 GiB ceiling an 8 MB
* upload could fill `/tmp` with 8 GB before anything noticed. The streamed
* byte count is held to the ratio the preflight states, applied to the
* compressed bytes on disk, and never above the absolute ceiling. A stored
* member cannot grow at all.
*/
export function memberOutputCap(entry: {
compressionMethod: number;
compressedSize: number;
}): number {
if (entry.compressionMethod === 0) {
return Math.min(entry.compressedSize, MAX_DECOMPRESSED_BYTES);
}
return Math.min(
MAX_DECOMPRESSED_BYTES,
Math.max(
MIN_OUTPUT_CAP_BYTES,
entry.compressedSize * MAX_COMPRESSION_RATIO,
),
);
}
/**
* Cap on the central-directory slice the extractor is willing to hold
* in memory. A real Apple export's central directory is a few KB (one
@@ -636,7 +669,7 @@ async function streamEntryToFile(
}
const dest = createWriteStream(destPath);
const cap = createByteCap(MAX_DECOMPRESSED_BYTES);
const cap = createByteCap(memberOutputCap(entry));
try {
if (entry.compressedSize === 0) {