perf(infra): scrape every 20s + drop unused cAdvisor metric families

Two cardinality/volume cuts to keep Prometheus storage sustainable
within the size budget:

- scrape_interval 5s -> 20s (evaluation_interval too). 5s tripled the
  sample volume vs the 15s default for no benefit on a single-box
  deployment; all rules use [5m] windows and for: >= 1m.
- Drop high-cardinality cAdvisor families we never chart or alert on
  (container_fs_*, blkio_*, tasks_state, memory_failures_total,
  memory_numa_*, network_tcp*/udp*) on both cadvisor jobs. Dashboards
  only use container CPU, memory, and start_time. This was the bulk of
  the ~12.4k active series.

Validated with `promtool check config`.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-06-09 11:59:27 +02:00
parent 114f6c2ce7
commit f38a224376

View file

@ -1,6 +1,10 @@
global:
scrape_interval: 5s
evaluation_interval: 5s
# 20s is plenty of resolution for a single-box self-hosted deployment.
# The previous 5s tripled (vs the 15s default) the sample volume and
# WAL growth for no practical benefit; all alert/recording rules use
# [5m] windows and `for:` >= 1m, so 20s evaluation is fine.
scrape_interval: 20s
evaluation_interval: 20s
scrape_configs:
- job_name: "journal"
@ -24,6 +28,16 @@ scrape_configs:
- job_name: "cadvisor"
static_configs:
- targets: ["cadvisor:8080"]
# cAdvisor emits ~100 per-container metric families; our dashboards
# and alerts only use CPU, memory, and container_start_time. Drop the
# high-cardinality families we never query (per-filesystem, block-IO,
# per-task state, memory-failure counters, socket tables). These
# dominated the ~12k active series — dropping them is the main
# cardinality cut. Add a name back here if you start charting it.
metric_relabel_configs:
- source_labels: [__name__]
regex: 'container_(fs_.*|blkio_.*|tasks_state|memory_failures_total|memory_numa_.*|network_tcp.*|network_udp.*)'
action: drop
- job_name: "caddy"
static_configs:
@ -38,3 +52,8 @@ scrape_configs:
- targets: ["10.0.1.10:8080"]
labels:
host: "brouter"
# Same high-cardinality drop as the local cadvisor job above.
metric_relabel_configs:
- source_labels: [__name__]
regex: 'container_(fs_.*|blkio_.*|tasks_state|memory_failures_total|memory_numa_.*|network_tcp.*|network_udp.*)'
action: drop