9e598fb6a1
Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top of the host compose in #291. ## Observability (section 6) - **Prometheus**: new `brouter-cadvisor` job scraping `10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`. - **cAdvisor sidecar** on the dedicated host's compose (`--docker_only --whitelisted_container_labels=trails.cool.service`) so metrics only cover trails containers, never the operator's unrelated workloads on the shared box. - **Promtail sidecar** on the dedicated host, Docker SD with relabel- drop on missing `trails.cool.service` label, pushing to flagship Loki at `http://10.0.0.2:3100/loki/api/v1/push`. - **Flagship compose**: Loki now publishes port 3100 on the vSwitch IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from the public internet. - **Grafana dashboard**: `brouter.json` — scrape up/down, request rate (from Planner-side `brouter_request_duration_seconds`), p50/p95/p99, container memory/CPU, Loki logs panel. - **Alert**: `brouter-scrape-down` fires on `up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting` so a total scrape failure still pages. Operator needs one UFW rule on the dedicated host for the cAdvisor port — documented in `infrastructure/brouter-host/README.md`. ## Documentation (section 8) - `CLAUDE.md` — hosts table + updated deployment table with SSH targets per workflow; BRouter host SSH is `-p 2232 trails@...`, different key. - `docs/architecture.md` — Hosting section rewritten to cover both hosts, vSwitch boundary, and the observability-scoping rationale for the shared dedicated host. - `docs/deployment.md` (new) — full operator runbook: host layout, first-time BRouter provisioning, SOPS rotation (including the macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with rollback, manual workflow triggers. Task 8.4 (infrastructure/README.md) skipped: that file doesn't exist and the ground is covered by brouter-host/README.md + docs/deployment.md. ## Validation - `docker compose config` on both the flagship and brouter-host compose files — both validate. - `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full turbo cache hits; no code changes in this commit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
95 lines
3.6 KiB
JSON
95 lines
3.6 KiB
JSON
{
|
|
"title": "BRouter (dedicated host)",
|
|
"uid": "trails-brouter",
|
|
"annotations": {
|
|
"list": [
|
|
{
|
|
"name": "Deploys",
|
|
"enable": true,
|
|
"datasource": { "type": "grafana", "uid": "-- Grafana --" },
|
|
"iconColor": "rgba(0, 211, 255, 1)",
|
|
"target": { "limit": 100, "matchAny": false, "tags": ["deploy", "brouter"], "type": "tags" }
|
|
}
|
|
]
|
|
},
|
|
"timezone": "browser",
|
|
"refresh": "30s",
|
|
"panels": [
|
|
{
|
|
"title": "Scrape up/down",
|
|
"type": "stat",
|
|
"gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
|
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
|
"targets": [{ "expr": "up{job=\"brouter-cadvisor\"}", "legendFormat": "{{instance}}" }],
|
|
"options": {
|
|
"reduceOptions": { "values": false, "calcs": ["lastNotNull"] },
|
|
"colorMode": "background"
|
|
},
|
|
"fieldConfig": {
|
|
"defaults": {
|
|
"mappings": [
|
|
{ "type": "value", "options": { "0": { "text": "DOWN", "color": "red" } } },
|
|
{ "type": "value", "options": { "1": { "text": "UP", "color": "green" } } }
|
|
],
|
|
"thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] }
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"title": "BRouter request rate (from Planner)",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 6, "y": 0 },
|
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
|
"targets": [{ "expr": "sum(rate(brouter_request_duration_seconds_count[5m])) by (status)", "legendFormat": "{{status}}" }]
|
|
},
|
|
{
|
|
"title": "BRouter latency (p50/p95/p99, from Planner)",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 18, "x": 0, "y": 8 },
|
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
|
"targets": [
|
|
{ "expr": "histogram_quantile(0.50, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p50" },
|
|
{ "expr": "histogram_quantile(0.95, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p95" },
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p99" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "s" } }
|
|
},
|
|
{
|
|
"title": "Container memory (BRouter + Caddy sidecar)",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
|
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
|
"targets": [
|
|
{
|
|
"expr": "container_memory_usage_bytes{job=\"brouter-cadvisor\",name=~\"trails-brouter.*\"}",
|
|
"legendFormat": "{{name}}"
|
|
}
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "bytes" } }
|
|
},
|
|
{
|
|
"title": "Container CPU (BRouter + Caddy sidecar)",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 },
|
|
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
|
"targets": [
|
|
{
|
|
"expr": "sum by (name) (rate(container_cpu_usage_seconds_total{job=\"brouter-cadvisor\",name=~\"trails-brouter.*\"}[5m]))",
|
|
"legendFormat": "{{name}}"
|
|
}
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "percentunit" } }
|
|
},
|
|
{
|
|
"title": "Recent BRouter logs",
|
|
"type": "logs",
|
|
"gridPos": { "h": 10, "w": 24, "x": 0, "y": 24 },
|
|
"datasource": { "type": "loki", "uid": "loki" },
|
|
"targets": [{ "expr": "{host=\"brouter\", service=\"brouter\"}" }],
|
|
"options": { "showTime": true, "wrapLogMessage": false }
|
|
}
|
|
],
|
|
"schemaVersion": 39,
|
|
"version": 1,
|
|
"time": { "from": "now-1h", "to": "now" }
|
|
}
|