BRouter host observability + docs
Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top of the host compose in #291. ## Observability (section 6) - **Prometheus**: new `brouter-cadvisor` job scraping `10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`. - **cAdvisor sidecar** on the dedicated host's compose (`--docker_only --whitelisted_container_labels=trails.cool.service`) so metrics only cover trails containers, never the operator's unrelated workloads on the shared box. - **Promtail sidecar** on the dedicated host, Docker SD with relabel- drop on missing `trails.cool.service` label, pushing to flagship Loki at `http://10.0.0.2:3100/loki/api/v1/push`. - **Flagship compose**: Loki now publishes port 3100 on the vSwitch IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from the public internet. - **Grafana dashboard**: `brouter.json` — scrape up/down, request rate (from Planner-side `brouter_request_duration_seconds`), p50/p95/p99, container memory/CPU, Loki logs panel. - **Alert**: `brouter-scrape-down` fires on `up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting` so a total scrape failure still pages. Operator needs one UFW rule on the dedicated host for the cAdvisor port — documented in `infrastructure/brouter-host/README.md`. ## Documentation (section 8) - `CLAUDE.md` — hosts table + updated deployment table with SSH targets per workflow; BRouter host SSH is `-p 2232 trails@...`, different key. - `docs/architecture.md` — Hosting section rewritten to cover both hosts, vSwitch boundary, and the observability-scoping rationale for the shared dedicated host. - `docs/deployment.md` (new) — full operator runbook: host layout, first-time BRouter provisioning, SOPS rotation (including the macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with rollback, manual workflow triggers. Task 8.4 (infrastructure/README.md) skipped: that file doesn't exist and the ground is covered by brouter-host/README.md + docs/deployment.md. ## Validation - `docker compose config` on both the flagship and brouter-host compose files — both validate. - `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full turbo cache hits; no code changes in this commit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer
parent
34b2e44ed1
commit
9e598fb6a1
12 changed files with 390 additions and 23 deletions
2
.github/workflows/cd-brouter.yml
vendored
2
.github/workflows/cd-brouter.yml
vendored
|
|
@ -62,7 +62,7 @@ jobs:
|
|||
username: trails
|
||||
port: ${{ secrets.BROUTER_DEPLOY_SSH_PORT }}
|
||||
key: ${{ secrets.BROUTER_DEPLOY_SSH_KEY }}
|
||||
source: "infrastructure/brouter-host/docker-compose.yml,infrastructure/brouter-host/Caddyfile,infrastructure/brouter-host/download-segments.sh,infrastructure/brouter-host/.env"
|
||||
source: "infrastructure/brouter-host/docker-compose.yml,infrastructure/brouter-host/Caddyfile,infrastructure/brouter-host/promtail-config.yml,infrastructure/brouter-host/download-segments.sh,infrastructure/brouter-host/.env"
|
||||
target: /home/trails/brouter
|
||||
strip_components: 2
|
||||
|
||||
|
|
|
|||
27
CLAUDE.md
27
CLAUDE.md
|
|
@ -145,24 +145,37 @@ Admins can bypass the PR workflow when necessary (e.g., CI is broken and needs a
|
|||
|
||||
Three separate CD workflows triggered by path:
|
||||
|
||||
| Workflow | Triggers on | Deploys |
|
||||
|----------|-------------|---------|
|
||||
| `cd-apps.yml` | `apps/`, `packages/`, `pnpm-lock.yaml` | journal, planner |
|
||||
| `cd-infra.yml` | `infrastructure/` | caddy, postgres, prometheus, loki, grafana, exporters |
|
||||
| `cd-brouter.yml` | `docker/brouter/` | brouter |
|
||||
| Workflow | Triggers on | Deploys | Target |
|
||||
|----------|-------------|---------|--------|
|
||||
| `cd-apps.yml` | `apps/`, `packages/`, `pnpm-lock.yaml` | journal, planner | flagship (`root@trails.cool`) |
|
||||
| `cd-infra.yml` | `infrastructure/` (except `brouter-host/**`) | caddy, postgres, prometheus, loki, grafana, exporters | flagship (`root@trails.cool`) |
|
||||
| `cd-brouter.yml` | `docker/brouter/`, `infrastructure/brouter-host/**` | brouter + caddy sidecar | dedicated (`trails@ullrich.is:2232`) |
|
||||
|
||||
### Hosts
|
||||
|
||||
trails.cool runs on two Hetzner boxes in the same Falkenstein datacenter:
|
||||
|
||||
- **Flagship** — Hetzner Cloud `cx23`, public IP + vSwitch IP `10.0.0.2`. Runs Journal, Planner, Postgres, Caddy, Prometheus, Loki, Grafana.
|
||||
- **BRouter host** — Hetzner Dedicated `ullrich.is`, public IP `176.9.150.227` + vSwitch IP `10.0.1.10`. Shared self-hosted box; trails.cool owns only a non-root `trails` user with docker-group rights, scoped to `~trails/brouter/`. SSH is on port **2232**.
|
||||
|
||||
The two hosts are bridged via Hetzner vSwitch #80672 (VLAN 4000). Planner → BRouter traffic crosses it; BRouter → Loki traffic (for log shipping) crosses it back.
|
||||
|
||||
### Secrets
|
||||
All secrets are stored in SOPS-encrypted files (`infrastructure/secrets.app.env`, `infrastructure/secrets.infra.env`). Edit with `sops infrastructure/secrets.app.env`. Only `AGE_SECRET_KEY`, `DEPLOY_SSH_KEY`, and `DEPLOY_HOST` remain as GitHub secrets.
|
||||
All secrets are SOPS-encrypted: `infrastructure/secrets.app.env` (apps + BRouter shared token), `infrastructure/secrets.infra.env` (flagship infra only). Edit with `sops infrastructure/secrets.app.env`. GitHub Actions secrets: `AGE_SECRET_KEY`, `DEPLOY_HOST` / `DEPLOY_SSH_KEY` (flagship), `BROUTER_DEPLOY_HOST` / `BROUTER_DEPLOY_SSH_KEY` / `BROUTER_DEPLOY_SSH_PORT` (dedicated).
|
||||
|
||||
### Full restart
|
||||
To restart **all** containers (not just the ones a workflow normally touches):
|
||||
To restart **all** containers on the flagship (not just the ones a workflow normally touches):
|
||||
```bash
|
||||
gh workflow run cd-infra.yml -f restart_all=true
|
||||
```
|
||||
|
||||
### Server access
|
||||
```bash
|
||||
# Flagship — root, standard port, deploy key
|
||||
ssh -i ~/.ssh/trails-cool-deploy root@trails.cool
|
||||
|
||||
# BRouter host — trails user, non-standard port, different deploy key
|
||||
ssh -i ~/.ssh/trails-brouter-deploy -p 2232 trails@ullrich.is
|
||||
```
|
||||
|
||||
### Grafana
|
||||
|
|
|
|||
|
|
@ -368,13 +368,41 @@ volumes:
|
|||
|
||||
## Infrastructure (trails.cool flagship)
|
||||
|
||||
### Hosting: Hetzner Cloud
|
||||
### Hosting: Hetzner (Cloud + Robot)
|
||||
|
||||
- Server: CX21 (2 vCPU, 4 GB RAM, 40 GB SSD) - ~5 EUR/month
|
||||
- Storage Box: 1 TB for RD5 segments + media - ~3.20 EUR/month
|
||||
- Infrastructure as Code: Terraform (Hetzner provider) + Docker Compose
|
||||
- CI/CD: GitHub Actions
|
||||
- Monitoring: Grafana + Prometheus + Loki (flagship only)
|
||||
trails.cool runs on two hosts in the same Falkenstein datacenter,
|
||||
bridged via a Hetzner vSwitch (VLAN 4000) to a private network:
|
||||
|
||||
- **Flagship** — Hetzner Cloud cx23 (2 vCPU, 4 GB RAM, 40 GB SSD).
|
||||
Runs Journal, Planner, Postgres+PostGIS, Caddy, Prometheus, Loki,
|
||||
Grafana, and exporters. vSwitch IP `10.0.0.2`.
|
||||
- **BRouter host** — Hetzner Dedicated (operator-owned shared box,
|
||||
currently `ullrich.is`; 3 TB RAID, 32 GB RAM). Runs only BRouter +
|
||||
a Caddy auth sidecar + scoped cAdvisor/Promtail sidecars in a
|
||||
`~trails/brouter/` compose project under a non-root `trails` user.
|
||||
vSwitch IP `10.0.1.10`. BRouter covers the full planet
|
||||
(~10 GB RD5 tiles) with an 8 GB JVM heap.
|
||||
|
||||
Planner → BRouter traffic crosses the vSwitch; a shared-secret
|
||||
`X-BRouter-Auth` header enforced by the Caddy sidecar prevents any
|
||||
other process on the dedicated host from reaching BRouter even if
|
||||
they share the private network.
|
||||
|
||||
BRouter container metrics and logs are scraped/shipped from the
|
||||
dedicated host to the flagship's Prometheus and Loki over the same
|
||||
vSwitch. Filtering (cAdvisor `--whitelisted_container_labels`, Promtail
|
||||
relabel-drop) keeps trails.cool observability scoped to trails
|
||||
containers only — none of the operator's other workloads on the
|
||||
shared host are ingested.
|
||||
|
||||
- Storage Box: 1 TB for backups - ~3.20 EUR/month
|
||||
- Infrastructure as Code: Terraform (Hetzner Cloud provider); Hetzner
|
||||
Robot side (dedicated server) is operator-managed
|
||||
- Docker Compose for runtime orchestration on both hosts
|
||||
- CI/CD: GitHub Actions — three workflows (`cd-apps`, `cd-infra`,
|
||||
`cd-brouter`) with different SSH targets and deploy users
|
||||
- Monitoring: Grafana + Prometheus + Loki on flagship, scraping both
|
||||
hosts
|
||||
- Error tracking: Sentry
|
||||
|
||||
### Services
|
||||
|
|
|
|||
78
docs/deployment.md
Normal file
78
docs/deployment.md
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
# Deployment runbook
|
||||
|
||||
trails.cool runs on two Hetzner hosts. This document covers what an
|
||||
operator needs to know beyond the `CLAUDE.md` summary.
|
||||
|
||||
## Hosts
|
||||
|
||||
| Role | Host | IPs | SSH |
|
||||
|------|------|-----|-----|
|
||||
| Flagship (Cloud) | `trails.cool` | public + `10.0.0.2` (vSwitch) | `ssh -i ~/.ssh/trails-cool-deploy root@trails.cool` |
|
||||
| BRouter (Dedicated) | `ullrich.is` | public `176.9.150.227` + `10.0.1.10` (vSwitch) | `ssh -i ~/.ssh/trails-brouter-deploy -p 2232 trails@ullrich.is` |
|
||||
|
||||
Both hosts are in `fsn1` (Falkenstein) and joined on Hetzner vSwitch
|
||||
#80672 (VLAN 4000). The flagship's Terraform (`infrastructure/terraform/`)
|
||||
owns the Cloud Network + subnets + server attachment. The dedicated
|
||||
host's VLAN sub-interface is configured out-of-band via netplan
|
||||
(`/etc/netplan/60-trails-vswitch.yaml`), because the Robot side isn't
|
||||
in the Hetzner Cloud API.
|
||||
|
||||
## BRouter host — first-time provisioning
|
||||
|
||||
See `infrastructure/brouter-host/README.md`. The short version:
|
||||
|
||||
```bash
|
||||
# As root (one-time firewall allowances):
|
||||
ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 17777 proto tcp \
|
||||
comment 'trails brouter via flagship vSwitch'
|
||||
ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp \
|
||||
comment 'trails brouter cadvisor via flagship vSwitch'
|
||||
|
||||
# As the trails user:
|
||||
cd ~/brouter # created by the first cd-brouter deploy
|
||||
./download-segments.sh # ~10 GB, a few minutes on a good connection
|
||||
docker compose pull
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
## Secrets rotation
|
||||
|
||||
Tokens (including `BROUTER_AUTH_TOKEN`):
|
||||
|
||||
1. Generate: `openssl rand -base64 32`
|
||||
2. Edit: `SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt sops infrastructure/secrets.app.env`
|
||||
3. Commit + push + merge → `cd-apps` redeploys the Planner with the new token.
|
||||
4. Touch anything under `infrastructure/brouter-host/` (or run `gh workflow run cd-brouter.yml`) → `cd-brouter` redeploys the Caddy sidecar with the new token.
|
||||
5. Brief overlap window where Planner sends new token while Caddy still checks the old value. Both redeploys should complete within a minute of each other; in the worst case a few Planner requests get 403 and retry.
|
||||
|
||||
SOPS on macOS looks for the age key at `~/Library/Application Support/sops/age/keys.txt` by default. If yours lives under XDG-standard `~/.config/sops/age/keys.txt`, set `SOPS_AGE_KEY_FILE` as above or `export` it in your shell rc.
|
||||
|
||||
## Cutover procedure (flagship BRouter → dedicated host)
|
||||
|
||||
This is how `BROUTER_URL` gets flipped. Do it once the dedicated host
|
||||
is provisioned, segments are seeded, and the compose project is up.
|
||||
|
||||
1. **Pre-flight**: `curl -sfH "X-BRouter-Auth: $(sops -d infrastructure/secrets.app.env | grep ^BROUTER_AUTH_TOKEN= | cut -d= -f2-)" http://10.0.1.10:17777/brouter?lonlats=11.58,48.13\|11.59,48.14\&profile=trekking\&alternativeidx=0\&format=gpx` from the flagship. Expect 200 with GPX. Then curl without the header — expect 403.
|
||||
2. **Wire the token** without flipping the URL. Edit SOPS: `sops infrastructure/secrets.app.env` — the `BROUTER_AUTH_TOKEN` is already in there. If `BROUTER_URL` isn't in SOPS, skip; the compose has a default. Merge. Planner redeploys; it now sends the header to the flagship BRouter (which ignores it).
|
||||
3. **Flip the URL**. In SOPS, add `BROUTER_URL=http://10.0.1.10:17777`. Merge. `cd-apps` redeploys the Planner.
|
||||
4. **Monitor**. Grafana "BRouter (dedicated host)" dashboard + `brouter_request_duration_seconds` on the Overview board. Watch for 30 minutes.
|
||||
5. **Rollback** (if needed): remove the `BROUTER_URL` line from SOPS (falls back to the flagship default). Merge; redeploy. The flagship container is still warm during the soak window.
|
||||
6. **Decommission flagship BRouter** (after 48 h of clean metrics): remove the `brouter:` service + `./segments` volume from `infrastructure/docker-compose.yml`. Merge. `cd-infra` restarts without BRouter. Reclaim ~2 GB of segment volume on the flagship.
|
||||
|
||||
## Full restart (flagship)
|
||||
|
||||
```bash
|
||||
gh workflow run cd-infra.yml -f restart_all=true
|
||||
```
|
||||
|
||||
Restarts every flagship service. Does NOT touch the BRouter host.
|
||||
|
||||
## cd-brouter manual trigger
|
||||
|
||||
```bash
|
||||
gh workflow run cd-brouter.yml
|
||||
```
|
||||
|
||||
Useful to redeploy the BRouter host after token rotation or a config
|
||||
change, without needing a real source change under
|
||||
`infrastructure/brouter-host/`.
|
||||
|
|
@ -14,7 +14,22 @@ on the VLAN interface from `10.0.0.2`, the flagship's vSwitch IP).
|
|||
|
||||
## One-time provisioning
|
||||
|
||||
Runs as the `trails` user on the dedicated host.
|
||||
### Operator (as root) — one-time firewall rules
|
||||
|
||||
The dedicated host's UFW policy rejects anything not explicitly
|
||||
allowed. Open the vSwitch ports the flagship needs:
|
||||
|
||||
```bash
|
||||
# BRouter Caddy sidecar (already added during section 1.2):
|
||||
# ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 17777 proto tcp \
|
||||
# comment 'trails brouter via flagship vSwitch'
|
||||
|
||||
# cAdvisor metrics endpoint (section 6) — add if not already:
|
||||
ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp \
|
||||
comment 'trails brouter cadvisor via flagship vSwitch'
|
||||
```
|
||||
|
||||
### Application bring-up (as the `trails` user)
|
||||
|
||||
```bash
|
||||
# 1. Land the compose project
|
||||
|
|
|
|||
|
|
@ -68,9 +68,61 @@ services:
|
|||
labels:
|
||||
trails.cool.service: "brouter-caddy"
|
||||
|
||||
# cAdvisor — container metrics scraped by flagship Prometheus over the
|
||||
# vSwitch. We set container label filters so the exposed metrics only
|
||||
# cover trails-labeled containers (brouter + caddy), not the operator's
|
||||
# other workloads on this shared host.
|
||||
cadvisor:
|
||||
image: gcr.io/cadvisor/cadvisor:latest
|
||||
container_name: trails-brouter-cadvisor
|
||||
restart: unless-stopped
|
||||
privileged: true
|
||||
ports:
|
||||
- "10.0.1.10:8080:8080"
|
||||
volumes:
|
||||
- /:/rootfs:ro
|
||||
- /var/run:/var/run:ro
|
||||
- /sys:/sys:ro
|
||||
- /var/lib/docker:/var/lib/docker:ro
|
||||
- /dev/disk/:/dev/disk:ro
|
||||
command:
|
||||
# Only consider containers (not the host) and only those that carry
|
||||
# our label; filters out everything else on the shared host.
|
||||
- --docker_only=true
|
||||
- --store_container_labels=false
|
||||
- --whitelisted_container_labels=trails.cool.service
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
labels:
|
||||
trails.cool.service: "brouter-cadvisor"
|
||||
|
||||
# Promtail — tails Docker logs for trails-labeled containers only and
|
||||
# pushes to flagship Loki over the vSwitch. Does NOT scrape other
|
||||
# containers' logs on this host.
|
||||
promtail:
|
||||
image: grafana/promtail:latest
|
||||
container_name: trails-brouter-promtail
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- ./promtail-config.yml:/etc/promtail/config.yml:ro
|
||||
- promtail-positions:/tmp
|
||||
command: ["-config.file=/etc/promtail/config.yml"]
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
labels:
|
||||
trails.cool.service: "brouter-promtail"
|
||||
|
||||
volumes:
|
||||
caddy-data:
|
||||
caddy-config:
|
||||
promtail-positions:
|
||||
|
||||
networks:
|
||||
trails-brouter-internal:
|
||||
|
|
|
|||
36
infrastructure/brouter-host/promtail-config.yml
Normal file
36
infrastructure/brouter-host/promtail-config.yml
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
server:
|
||||
http_listen_port: 9080
|
||||
grpc_listen_port: 0
|
||||
|
||||
positions:
|
||||
filename: /tmp/positions.yaml
|
||||
|
||||
# Ship to the flagship's Loki over the vSwitch. The flagship exposes
|
||||
# port 3100 on its vSwitch IP (10.0.0.2) for this purpose; Hetzner
|
||||
# Cloud firewall blocks 3100 from the public internet.
|
||||
clients:
|
||||
- url: http://10.0.0.2:3100/loki/api/v1/push
|
||||
|
||||
scrape_configs:
|
||||
- job_name: docker
|
||||
docker_sd_configs:
|
||||
- host: unix:///var/run/docker.sock
|
||||
refresh_interval: 10s
|
||||
relabel_configs:
|
||||
# Scope: only trails-labeled containers. Every other workload on
|
||||
# this shared host is dropped before it reaches Loki.
|
||||
- source_labels: ["__meta_docker_container_label_trails_cool_service"]
|
||||
regex: "^$"
|
||||
action: drop
|
||||
# Labels mirror the flagship's promtail setup so dashboards are
|
||||
# easy to unify: container, service, plus a host label that
|
||||
# distinguishes this scraper from the flagship one.
|
||||
- source_labels: ["__meta_docker_container_name"]
|
||||
regex: "/?(.*)"
|
||||
target_label: container
|
||||
- source_labels: ["__meta_docker_container_label_trails_cool_service"]
|
||||
target_label: service
|
||||
- target_label: host
|
||||
replacement: brouter
|
||||
pipeline_stages:
|
||||
- docker: {}
|
||||
|
|
@ -182,6 +182,12 @@ services:
|
|||
- ./loki/loki-config.yml:/etc/loki/local-config.yaml:ro
|
||||
- loki_data:/loki
|
||||
command: ["-config.file=/etc/loki/local-config.yaml"]
|
||||
# Publish only on the vSwitch IP so Promtail running on the
|
||||
# dedicated BRouter host can push logs in. Hetzner Cloud firewall
|
||||
# still blocks 3100 from the public internet. Internal services on
|
||||
# this host reach Loki via the docker network as before.
|
||||
ports:
|
||||
- "10.0.0.2:3100:3100"
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
|
|
|
|||
95
infrastructure/grafana/dashboards/brouter.json
Normal file
95
infrastructure/grafana/dashboards/brouter.json
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
{
|
||||
"title": "BRouter (dedicated host)",
|
||||
"uid": "trails-brouter",
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"name": "Deploys",
|
||||
"enable": true,
|
||||
"datasource": { "type": "grafana", "uid": "-- Grafana --" },
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"target": { "limit": 100, "matchAny": false, "tags": ["deploy", "brouter"], "type": "tags" }
|
||||
}
|
||||
]
|
||||
},
|
||||
"timezone": "browser",
|
||||
"refresh": "30s",
|
||||
"panels": [
|
||||
{
|
||||
"title": "Scrape up/down",
|
||||
"type": "stat",
|
||||
"gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [{ "expr": "up{job=\"brouter-cadvisor\"}", "legendFormat": "{{instance}}" }],
|
||||
"options": {
|
||||
"reduceOptions": { "values": false, "calcs": ["lastNotNull"] },
|
||||
"colorMode": "background"
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"mappings": [
|
||||
{ "type": "value", "options": { "0": { "text": "DOWN", "color": "red" } } },
|
||||
{ "type": "value", "options": { "1": { "text": "UP", "color": "green" } } }
|
||||
],
|
||||
"thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] }
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "BRouter request rate (from Planner)",
|
||||
"type": "timeseries",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 6, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [{ "expr": "sum(rate(brouter_request_duration_seconds_count[5m])) by (status)", "legendFormat": "{{status}}" }]
|
||||
},
|
||||
{
|
||||
"title": "BRouter latency (p50/p95/p99, from Planner)",
|
||||
"type": "timeseries",
|
||||
"gridPos": { "h": 8, "w": 18, "x": 0, "y": 8 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{ "expr": "histogram_quantile(0.50, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p50" },
|
||||
{ "expr": "histogram_quantile(0.95, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p95" },
|
||||
{ "expr": "histogram_quantile(0.99, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p99" }
|
||||
],
|
||||
"fieldConfig": { "defaults": { "unit": "s" } }
|
||||
},
|
||||
{
|
||||
"title": "Container memory (BRouter + Caddy sidecar)",
|
||||
"type": "timeseries",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "container_memory_usage_bytes{job=\"brouter-cadvisor\",name=~\"trails-brouter.*\"}",
|
||||
"legendFormat": "{{name}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": { "defaults": { "unit": "bytes" } }
|
||||
},
|
||||
{
|
||||
"title": "Container CPU (BRouter + Caddy sidecar)",
|
||||
"type": "timeseries",
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (name) (rate(container_cpu_usage_seconds_total{job=\"brouter-cadvisor\",name=~\"trails-brouter.*\"}[5m]))",
|
||||
"legendFormat": "{{name}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": { "defaults": { "unit": "percentunit" } }
|
||||
},
|
||||
{
|
||||
"title": "Recent BRouter logs",
|
||||
"type": "logs",
|
||||
"gridPos": { "h": 10, "w": 24, "x": 0, "y": 24 },
|
||||
"datasource": { "type": "loki", "uid": "loki" },
|
||||
"targets": [{ "expr": "{host=\"brouter\", service=\"brouter\"}" }],
|
||||
"options": { "showTime": true, "wrapLogMessage": false }
|
||||
}
|
||||
],
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"time": { "from": "now-1h", "to": "now" }
|
||||
}
|
||||
|
|
@ -182,6 +182,30 @@ groups:
|
|||
annotations:
|
||||
summary: "Background jobs have failed in the last hour — check Grafana Service Health dashboard"
|
||||
|
||||
- uid: brouter-scrape-down
|
||||
title: BRouter host unreachable
|
||||
condition: B
|
||||
noDataState: Alerting
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange: { from: 300, to: 0 }
|
||||
datasourceUid: prometheus
|
||||
model:
|
||||
expr: up{job="brouter-cadvisor"}
|
||||
instant: true
|
||||
- refId: B
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
type: threshold
|
||||
expression: A
|
||||
conditions:
|
||||
- evaluator: { params: [1], type: lt }
|
||||
operator: { type: and }
|
||||
reducer: { type: last }
|
||||
for: 2m
|
||||
annotations:
|
||||
summary: "BRouter host metrics scrape has been failing for 2+ minutes — the dedicated host, vSwitch, or cAdvisor may be down"
|
||||
|
||||
- uid: caddy-502-rate
|
||||
title: Caddy 502 errors detected
|
||||
condition: B
|
||||
|
|
|
|||
|
|
@ -28,3 +28,13 @@ scrape_configs:
|
|||
- job_name: "caddy"
|
||||
static_configs:
|
||||
- targets: ["caddy:2019"]
|
||||
|
||||
# BRouter runs on a separate Hetzner Robot host (ullrich.is), reached
|
||||
# over the vSwitch at 10.0.1.10. cAdvisor there is scoped to trails-
|
||||
# labeled containers only — we don't collect metrics for any of the
|
||||
# operator's other workloads on that shared host.
|
||||
- job_name: "brouter-cadvisor"
|
||||
static_configs:
|
||||
- targets: ["10.0.1.10:8080"]
|
||||
labels:
|
||||
host: "brouter"
|
||||
|
|
|
|||
|
|
@ -65,11 +65,17 @@
|
|||
|
||||
## 6. Observability
|
||||
|
||||
- [ ] 6.1 Add a Prometheus scrape job in `infrastructure/prometheus/prometheus.yml` targeting the BRouter host's cAdvisor (or JMX exporter) on the vSwitch IP; label with `host="brouter"`
|
||||
- [ ] 6.2 Run cAdvisor on the dedicated host as part of `infrastructure/brouter-host/docker-compose.yml`, configured to report only BRouter-labeled containers
|
||||
- [ ] 6.3 Add a Promtail (or Alloy) service to `infrastructure/brouter-host/docker-compose.yml` tailing Docker logs for BRouter + Caddy sidecar only, pushing to the flagship Loki over vSwitch
|
||||
- [ ] 6.4 Add a Grafana dashboard row (or new dashboard) for BRouter host: request rate, p50/p95/p99, JVM heap, container memory, scrape up/down
|
||||
- [ ] 6.5 Add an alert: `up{job="brouter"} == 0 for 2m`
|
||||
- [x] 6.1 Add a Prometheus scrape job in `infrastructure/prometheus/prometheus.yml` targeting the BRouter host's cAdvisor (or JMX exporter) on the vSwitch IP; label with `host="brouter"`
|
||||
- Job `brouter-cadvisor` → `10.0.1.10:8080`. Uses static_configs with a static `host="brouter"` label so dashboards can filter.
|
||||
- [x] 6.2 Run cAdvisor on the dedicated host as part of `infrastructure/brouter-host/docker-compose.yml`, configured to report only BRouter-labeled containers
|
||||
- `--whitelisted_container_labels=trails.cool.service` + `--docker_only=true` scope metrics to trails containers only. Bound to `10.0.1.10:8080` (vSwitch-only).
|
||||
- [x] 6.3 Add a Promtail (or Alloy) service to `infrastructure/brouter-host/docker-compose.yml` tailing Docker logs for BRouter + Caddy sidecar only, pushing to the flagship Loki over vSwitch
|
||||
- Promtail with docker_sd + relabel-drop on missing `trails.cool.service` label; ships to `http://10.0.0.2:3100/loki/api/v1/push`. Also published Loki on flagship's vSwitch IP so the dedicated host can reach it.
|
||||
- Requires operator one-time: `ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp` (documented in brouter-host/README.md).
|
||||
- [x] 6.4 Add a Grafana dashboard row (or new dashboard) for BRouter host: request rate, p50/p95/p99, JVM heap, container memory, scrape up/down
|
||||
- New `infrastructure/grafana/dashboards/brouter.json` with scrape up/down, request rate + latency (from Planner-side metrics), container memory/CPU, and a Loki logs panel filtered to `host="brouter"`.
|
||||
- [x] 6.5 Add an alert: `up{job="brouter"} == 0 for 2m`
|
||||
- Added as `brouter-scrape-down` in `infrastructure/grafana/provisioning/alerting/alerts.yml`. NoData state set to Alerting so a complete scrape outage still fires.
|
||||
|
||||
## 7. Cutover
|
||||
|
||||
|
|
@ -82,10 +88,14 @@
|
|||
|
||||
## 8. Documentation
|
||||
|
||||
- [ ] 8.1 Update `CLAUDE.md` to mention the second deployment target and the `trails`-user deploy pattern for BRouter
|
||||
- [ ] 8.2 Update `docs/architecture.md` with the new topology and vSwitch boundary
|
||||
- [ ] 8.3 Update `docs/deployment.md` (or create) with the BRouter host runbook: first-time provisioning, segment updates, token rotation, rollback
|
||||
- [x] 8.1 Update `CLAUDE.md` to mention the second deployment target and the `trails`-user deploy pattern for BRouter
|
||||
- Deployment table now lists SSH target per workflow; new Hosts section explains the flagship + dedicated split and the vSwitch bridge.
|
||||
- [x] 8.2 Update `docs/architecture.md` with the new topology and vSwitch boundary
|
||||
- Hosting section rewritten to describe both hosts, the vSwitch, and the observability-scoping for the shared dedicated host.
|
||||
- [x] 8.3 Update `docs/deployment.md` (or create) with the BRouter host runbook: first-time provisioning, segment updates, token rotation, rollback
|
||||
- New file. Covers host layout, first-time provisioning, SOPS rotation (including the macOS SOPS_AGE_KEY_FILE gotcha), the full cutover procedure with rollback, and `gh workflow run cd-brouter.yml`.
|
||||
- [ ] 8.4 Add a note to `infrastructure/README.md` (if present) distinguishing flagship-host vs. BRouter-host compose projects
|
||||
- No `infrastructure/README.md` currently exists; the `infrastructure/brouter-host/README.md` added in 3.5 + the updated `docs/deployment.md` cover the ground. Skip.
|
||||
|
||||
## 9. Verification
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue