perf(infra): scrape every 20s + drop unused cAdvisor metric families
Two cardinality/volume cuts to keep Prometheus storage sustainable within the size budget: - scrape_interval 5s -> 20s (evaluation_interval too). 5s tripled the sample volume vs the 15s default for no benefit on a single-box deployment; all rules use [5m] windows and for: >= 1m. - Drop high-cardinality cAdvisor families we never chart or alert on (container_fs_*, blkio_*, tasks_state, memory_failures_total, memory_numa_*, network_tcp*/udp*) on both cadvisor jobs. Dashboards only use container CPU, memory, and start_time. This was the bulk of the ~12.4k active series. Validated with `promtool check config`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
114f6c2ce7
commit
f38a224376
1 changed files with 21 additions and 2 deletions
|
|
@ -1,6 +1,10 @@
|
|||
global:
|
||||
scrape_interval: 5s
|
||||
evaluation_interval: 5s
|
||||
# 20s is plenty of resolution for a single-box self-hosted deployment.
|
||||
# The previous 5s tripled (vs the 15s default) the sample volume and
|
||||
# WAL growth for no practical benefit; all alert/recording rules use
|
||||
# [5m] windows and `for:` >= 1m, so 20s evaluation is fine.
|
||||
scrape_interval: 20s
|
||||
evaluation_interval: 20s
|
||||
|
||||
scrape_configs:
|
||||
- job_name: "journal"
|
||||
|
|
@ -24,6 +28,16 @@ scrape_configs:
|
|||
- job_name: "cadvisor"
|
||||
static_configs:
|
||||
- targets: ["cadvisor:8080"]
|
||||
# cAdvisor emits ~100 per-container metric families; our dashboards
|
||||
# and alerts only use CPU, memory, and container_start_time. Drop the
|
||||
# high-cardinality families we never query (per-filesystem, block-IO,
|
||||
# per-task state, memory-failure counters, socket tables). These
|
||||
# dominated the ~12k active series — dropping them is the main
|
||||
# cardinality cut. Add a name back here if you start charting it.
|
||||
metric_relabel_configs:
|
||||
- source_labels: [__name__]
|
||||
regex: 'container_(fs_.*|blkio_.*|tasks_state|memory_failures_total|memory_numa_.*|network_tcp.*|network_udp.*)'
|
||||
action: drop
|
||||
|
||||
- job_name: "caddy"
|
||||
static_configs:
|
||||
|
|
@ -38,3 +52,8 @@ scrape_configs:
|
|||
- targets: ["10.0.1.10:8080"]
|
||||
labels:
|
||||
host: "brouter"
|
||||
# Same high-cardinality drop as the local cadvisor job above.
|
||||
metric_relabel_configs:
|
||||
- source_labels: [__name__]
|
||||
regex: 'container_(fs_.*|blkio_.*|tasks_state|memory_failures_total|memory_numa_.*|network_tcp.*|network_udp.*)'
|
||||
action: drop
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue