feat(observability): log heap samples and a one-time pressure report

runtime.heap.sampled records heap, old space, RSS, external memory and
the work in flight every five minutes, every minute at or above 70 % of
the heap limit. At 85 % runtime.heap.pressure lists the oldest requests
and jobs in flight and the routes most of them share, once, re-armed
only after the heap falls below 70 % and thirty minutes pass. A
heap-limit crash leaves no other record of what the process was doing.

HEALTHLOG_HEAP_SNAPSHOT=1 adds one heap snapshot at that moment into
HEALTHLOG_DIAG_DIR (0700 directory, 0600 file), off by default.
docs/ops/heap-diagnostics.md warns that a snapshot holds decrypted
health data and keys and says how to delete it.
This commit is contained in:
Marc-André Bombeck committed 2026-10-19 16:00:53 +02:00
1 parent 5923dac8aa
commit 16024397c8
8 files changed
+597

No files matched your search

+7
View File
@@ -409,6 +409,13 @@ x-healthlog-env: &healthlog_env
LOG_SAMPLE_RATE: "${LOG_SAMPLE_RATE:-}"
LOG_SLOW_THRESHOLD_MS: "${LOG_SLOW_THRESHOLD_MS:-}"
LOG_INCLUDE_STACK: "${LOG_INCLUDE_STACK:-}"
# Heap diagnostics, off by default. HEALTHLOG_HEAP_SNAPSHOT=1 writes ONE
# heap snapshot the first time the heap reaches 85 % of its limit, into
# HEALTHLOG_DIAG_DIR (default /tmp inside the container). A snapshot holds
# decrypted health data and keys: turn it on only to chase a crash, then
# delete the file. See docs/ops/heap-diagnostics.md.
HEALTHLOG_HEAP_SNAPSHOT: "${HEALTHLOG_HEAP_SNAPSHOT:-}"
HEALTHLOG_DIAG_DIR: "${HEALTHLOG_DIAG_DIR:-}"
# Optional log shipping to a Loki instance. Set LOKI_ENDPOINT to the base
# URL or the full push URL to turn it on; a Loki on your own network works.
# The username/password pair is only needed when your Loki is behind basic