feat(observability): log heap samples and a one-time pressure report

runtime.heap.sampled records heap, old space, RSS, external memory and
the work in flight every five minutes, every minute at or above 70 % of
the heap limit. At 85 % runtime.heap.pressure lists the oldest requests
and jobs in flight and the routes most of them share, once, re-armed
only after the heap falls below 70 % and thirty minutes pass. A
heap-limit crash leaves no other record of what the process was doing.

HEALTHLOG_HEAP_SNAPSHOT=1 adds one heap snapshot at that moment into
HEALTHLOG_DIAG_DIR (0700 directory, 0600 file), off by default.
docs/ops/heap-diagnostics.md warns that a snapshot holds decrypted
health data and keys and says how to delete it.
This commit is contained in:
Marc-André Bombeck committed 2026-10-19 16:00:53 +02:00
1 parent 5923dac8aa
commit 16024397c8
8 files changed
+597

No files matched your search

+9
View File
@@ -653,6 +653,15 @@ API_TOKEN_HMAC_KEY="<openssl rand -hex 32>"
# LOG_SLOW_THRESHOLD_MS="3000"
# LOG_INCLUDE_STACK="false"
# Heap diagnostics, off by default. The process always logs its heap
# (runtime.heap.sampled) and, at 85 % of the limit, one report of the work in
# flight (runtime.heap.pressure). HEALTHLOG_HEAP_SNAPSHOT=1 additionally writes
# ONE heap snapshot at that moment into HEALTHLOG_DIAG_DIR (default /tmp). A
# snapshot contains decrypted health data and keys; enable it only to chase a
# crash and delete the file afterwards. See docs/ops/heap-diagnostics.md.
# HEALTHLOG_HEAP_SNAPSHOT="1"
# HEALTHLOG_DIAG_DIR="/tmp"
# Optional log shipping to a Loki instance. LOKI_ENDPOINT turns shipping on.
# Either form works: the base URL (http://loki:3100) or the full push URL
# (http://loki:3100/loki/api/v1/push). A Loki on your own network is fine,