fix(heap): report again at 95 % and leave a report on an out-of-memory exit

The pressure report re-armed only after the heap fell under 70 % and
half an hour passed, so a heap that climbed to its limit soon after an
85 % report died with only that older report on record. A 95 % tier
now writes the same report at once, whatever the cooldown, and re-arms
under 85 %.

The image sets NODE_OPTIONS so Node writes its diagnostic report to
stderr when the heap runs out, with environment variables and network
interfaces excluded. The heap-diagnostics and scaling docs say how to
keep the flags when overriding NODE_OPTIONS.

Heap figures share one megabytes helper with the support report.
This commit is contained in:
Marc-André Bombeck committed 2026-10-20 03:42:26 +02:00
1 parent 3cf67d05f9
commit b79bd8370d
7 files changed
+101 -14

No files matched your search

+11 -1
View File
@@ -7,7 +7,7 @@ A Node process that reaches its heap limit stops with `JavaScript heap out of me
Two background wide events, no configuration needed:
- `runtime.heap.sampled` (info), every five minutes, and every minute while the heap is at or above 70 % of its limit. It goes to stdout and Loki but not to Admin > App logs, so it cannot push other events out of that page's 500-entry buffer. It carries `heap_used_mb`, `heap_total_mb`, `heap_limit_mb`, `heap_used_ratio`, `old_space_mb`, `rss_mb`, `external_mb`, `array_buffers_mb`, `active_requests`, `active_jobs`, `in_flight_dropped` and `uptime_s`.
- `runtime.heap.pressure` (warn), the first time the heap reaches 85 % of its limit. Besides the figures above it lists the 50 oldest requests and jobs in flight (`in_flight_oldest`: kind, method, route, age in seconds, whether it was a server action) and the routes most of them share (`in_flight_by_route`). Routes are reduced to their shape: no query string, secret path segments replaced by `[REDACTED]`, ids by `:id`, anything unusual by `:x`. It writes again only after the heap has dropped below 70 % and thirty minutes have passed.
- `runtime.heap.pressure` (warn), the first time the heap reaches 85 % of its limit. Besides the figures above it lists the 50 oldest requests and jobs in flight (`in_flight_oldest`: kind, method, route, age in seconds, whether it was a server action) and the routes most of them share (`in_flight_by_route`). Routes are reduced to their shape: no query string, secret path segments replaced by `[REDACTED]`, ids by `:id`, anything unusual by `:x`. It writes again only after the heap has dropped below 70 % and thirty minutes have passed. One exception: when the heap reaches 95 %, the same report is written again at once, with `pressure_ratio: 0.95`, whatever the cooldown; that tier re-arms as soon as the heap is back under 85 %. A heap that climbs to its limit shortly after an 85 % report therefore still leaves a report naming the work that pushed it over.
The heap is read every five seconds, a single call that takes microseconds. Tracking requests costs one map insert and one delete per request.
@@ -17,6 +17,16 @@ To find the last lines before a crash:
docker compose logs app --since 2h | grep '"runtime.heap.'
```
## The report Node writes when the heap runs out
The image sets `NODE_OPTIONS="--report-on-fatalerror --report-filename=stderr --report-exclude-env --report-exclude-network"`. When the process runs out of heap, Node writes a diagnostic report, about 10 KB of JSON with `"trigger": "OOMError"`, to stderr just before it exits: the JavaScript stack at the failed allocation, the native stack, the heap spaces and the open handles. It lands in the container log beside the `runtime.heap.*` lines:
```sh
docker compose logs app --since 2h | grep -A400 '"trigger": "OOMError"'
```
The two exclude flags keep environment variables (keys, passwords, the database URL) and network interfaces out of the report. If you set `NODE_OPTIONS` yourself, for example to raise `--max-old-space-size`, your value replaces the image's; add the four flags to it to keep the report.
## Optional heap snapshot
A snapshot shows what the heap is made of, which the events above cannot. Turn it on only while you are chasing a specific crash:
+5 -1
View File
@@ -79,9 +79,13 @@ Raise the limit on the container that runs the worker and re-upload:
services:
app:
environment:
NODE_OPTIONS: --max-old-space-size=1024
NODE_OPTIONS: --max-old-space-size=1024 --report-on-fatalerror --report-filename=stderr --report-exclude-env --report-exclude-network
```
Your `NODE_OPTIONS` replaces the image's, so carry the four `--report-*`
flags over: they make an out-of-memory exit leave a diagnostic report in
the container log (see `docs/ops/heap-diagnostics.md`).
An import interrupted by a container restart or an out-of-memory kill
does not stay stuck: the worker reconciles non-terminal import jobs at
boot and every 15 minutes, flipping orphaned rows to `failed`