feat(observability): log heap samples and a one-time pressure report

runtime.heap.sampled records heap, old space, RSS, external memory and
the work in flight every five minutes, every minute at or above 70 % of
the heap limit. At 85 % runtime.heap.pressure lists the oldest requests
and jobs in flight and the routes most of them share, once, re-armed
only after the heap falls below 70 % and thirty minutes pass. A
heap-limit crash leaves no other record of what the process was doing.

HEALTHLOG_HEAP_SNAPSHOT=1 adds one heap snapshot at that moment into
HEALTHLOG_DIAG_DIR (0700 directory, 0600 file), off by default.
docs/ops/heap-diagnostics.md warns that a snapshot holds decrypted
health data and keys and says how to delete it.
This commit is contained in:
Marc-André Bombeck committed 2026-10-19 16:00:53 +02:00
1 parent 5923dac8aa
commit 16024397c8
8 files changed
+597

No files matched your search

+8
View File
@@ -178,6 +178,14 @@ API_TOKEN_HMAC_KEY="REPLACE-WITH-OUTPUT-OF-openssl-rand-hex-32"
# leave the host unencrypted.
# LOG_INCLUDE_STACK="true"
# Heap diagnostics, off by default. HEALTHLOG_HEAP_SNAPSHOT=1 writes ONE heap
# snapshot the first time the heap reaches 85 % of its limit, into
# HEALTHLOG_DIAG_DIR (default /tmp). A snapshot contains decrypted health data
# and keys; enable it only to chase a crash and delete the file afterwards.
# See docs/ops/heap-diagnostics.md.
# HEALTHLOG_HEAP_SNAPSHOT="1"
# HEALTHLOG_DIAG_DIR="/tmp"
# Optional: ship Wide Events to Loki. Leave blank for stdout-only logging.
# LOKI_ENDPOINT="http://loki:3100"
# LOKI_USERNAME=""