fix(infra): raise Prometheus retention.size 1GB -> 6GB

Prometheus size-based retention counts the WAL toward its budget. With
retention.size=1GB and a stuck/corrupt WAL that had grown to 2.1GB, the
total was permanently over budget, so Prometheus deleted every
freshly-compacted block the moment it was written. Net effect: only the
in-memory ~2h head block was ever queryable — metrics appeared to "start
this morning" no matter when you looked, and last night's Caddy 502
spike (planner /api/overpass upstream stall, 18:33-18:34 UTC) was
unobservable in Prometheus (logs survived in Loki).

The corrupt WAL was cleared out-of-band on the flagship. This raises the
size backstop to 6GB so compacted blocks survive to the intended 15d
time-based retention. Disk is 38G (43% used), so 6GB is safe headroom.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-06-09 11:52:10 +02:00
parent 001c53294a
commit 114f6c2ce7

View file

@ -201,7 +201,13 @@ services:
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.retention.time=15d"
- "--storage.tsdb.retention.size=1GB"
# Size-based retention is a disk-safety backstop, not the primary
# policy (time=15d is). It counts the WAL toward the budget, so a
# too-small cap makes Prometheus delete freshly-compacted blocks on
# creation — which it did at 1GB, leaving only the ~2h head block
# queryable (a stuck/corrupt WAL had bloated to 2.1GB). 6GB holds
# ~15d of metrics at current cardinality with headroom; disk is 38G.
- "--storage.tsdb.retention.size=6GB"
loki:
image: grafana/loki:latest