The pressure report re-armed only after the heap fell under 70 % and
half an hour passed, so a heap that climbed to its limit soon after an
85 % report died with only that older report on record. A 95 % tier
now writes the same report at once, whatever the cooldown, and re-arms
under 85 %.
The image sets NODE_OPTIONS so Node writes its diagnostic report to
stderr when the heap runs out, with environment variables and network
interfaces excluded. The heap-diagnostics and scaling docs say how to
keep the flags when overriding NODE_OPTIONS.
Heap figures share one megabytes helper with the support report.
runtime.heap.sampled is written every minute while the heap is above
70 % of its limit, which turned the 500-entry ring behind the admin log
page over in hours and pushed out the events an operator opens it for.
emitEvent takes a skipMemoryRing option; the sample sets it and still
reaches stdout and Loki, while the pressure and snapshot lines go
everywhere as before.
The snapshot directory was /tmp/healthlog-heap-<pid>, a predictable name
(pid 1 in most containers) that mkdirSync accepted when it already
existed or was a symlink. It now comes from mkdtempSync and is checked
with lstat to be a real directory owned by the process user and closed
to others before the snapshot is written.
runtime.heap.sampled records heap, old space, RSS, external memory and
the work in flight every five minutes, every minute at or above 70 % of
the heap limit. At 85 % runtime.heap.pressure lists the oldest requests
and jobs in flight and the routes most of them share, once, re-armed
only after the heap falls below 70 % and thirty minutes pass. A
heap-limit crash leaves no other record of what the process was doing.
HEALTHLOG_HEAP_SNAPSHOT=1 adds one heap snapshot at that moment into
HEALTHLOG_DIAG_DIR (0700 directory, 0600 file), off by default.
docs/ops/heap-diagnostics.md warns that a snapshot holds decrypted
health data and keys and says how to delete it.
Marc-André Bombeck