diff --git a/.github/workflows/cd-brouter.yml b/.github/workflows/cd-brouter.yml index b6ec12a..0a49eea 100644 --- a/.github/workflows/cd-brouter.yml +++ b/.github/workflows/cd-brouter.yml @@ -62,7 +62,7 @@ jobs: username: trails port: ${{ secrets.BROUTER_DEPLOY_SSH_PORT }} key: ${{ secrets.BROUTER_DEPLOY_SSH_KEY }} - source: "infrastructure/brouter-host/docker-compose.yml,infrastructure/brouter-host/Caddyfile,infrastructure/brouter-host/download-segments.sh,infrastructure/brouter-host/.env" + source: "infrastructure/brouter-host/docker-compose.yml,infrastructure/brouter-host/Caddyfile,infrastructure/brouter-host/promtail-config.yml,infrastructure/brouter-host/download-segments.sh,infrastructure/brouter-host/.env" target: /home/trails/brouter strip_components: 2 diff --git a/CLAUDE.md b/CLAUDE.md index c525e98..b455318 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -145,24 +145,37 @@ Admins can bypass the PR workflow when necessary (e.g., CI is broken and needs a Three separate CD workflows triggered by path: -| Workflow | Triggers on | Deploys | -|----------|-------------|---------| -| `cd-apps.yml` | `apps/`, `packages/`, `pnpm-lock.yaml` | journal, planner | -| `cd-infra.yml` | `infrastructure/` | caddy, postgres, prometheus, loki, grafana, exporters | -| `cd-brouter.yml` | `docker/brouter/` | brouter | +| Workflow | Triggers on | Deploys | Target | +|----------|-------------|---------|--------| +| `cd-apps.yml` | `apps/`, `packages/`, `pnpm-lock.yaml` | journal, planner | flagship (`root@trails.cool`) | +| `cd-infra.yml` | `infrastructure/` (except `brouter-host/**`) | caddy, postgres, prometheus, loki, grafana, exporters | flagship (`root@trails.cool`) | +| `cd-brouter.yml` | `docker/brouter/`, `infrastructure/brouter-host/**` | brouter + caddy sidecar | dedicated (`trails@ullrich.is:2232`) | + +### Hosts + +trails.cool runs on two Hetzner boxes in the same Falkenstein datacenter: + +- **Flagship** — Hetzner Cloud `cx23`, public IP + vSwitch IP `10.0.0.2`. Runs Journal, Planner, Postgres, Caddy, Prometheus, Loki, Grafana. +- **BRouter host** — Hetzner Dedicated `ullrich.is`, public IP `176.9.150.227` + vSwitch IP `10.0.1.10`. Shared self-hosted box; trails.cool owns only a non-root `trails` user with docker-group rights, scoped to `~trails/brouter/`. SSH is on port **2232**. + +The two hosts are bridged via Hetzner vSwitch #80672 (VLAN 4000). Planner → BRouter traffic crosses it; BRouter → Loki traffic (for log shipping) crosses it back. ### Secrets -All secrets are stored in SOPS-encrypted files (`infrastructure/secrets.app.env`, `infrastructure/secrets.infra.env`). Edit with `sops infrastructure/secrets.app.env`. Only `AGE_SECRET_KEY`, `DEPLOY_SSH_KEY`, and `DEPLOY_HOST` remain as GitHub secrets. +All secrets are SOPS-encrypted: `infrastructure/secrets.app.env` (apps + BRouter shared token), `infrastructure/secrets.infra.env` (flagship infra only). Edit with `sops infrastructure/secrets.app.env`. GitHub Actions secrets: `AGE_SECRET_KEY`, `DEPLOY_HOST` / `DEPLOY_SSH_KEY` (flagship), `BROUTER_DEPLOY_HOST` / `BROUTER_DEPLOY_SSH_KEY` / `BROUTER_DEPLOY_SSH_PORT` (dedicated). ### Full restart -To restart **all** containers (not just the ones a workflow normally touches): +To restart **all** containers on the flagship (not just the ones a workflow normally touches): ```bash gh workflow run cd-infra.yml -f restart_all=true ``` ### Server access ```bash +# Flagship — root, standard port, deploy key ssh -i ~/.ssh/trails-cool-deploy root@trails.cool + +# BRouter host — trails user, non-standard port, different deploy key +ssh -i ~/.ssh/trails-brouter-deploy -p 2232 trails@ullrich.is ``` ### Grafana diff --git a/docs/architecture.md b/docs/architecture.md index 5afba23..699c40b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -368,13 +368,41 @@ volumes: ## Infrastructure (trails.cool flagship) -### Hosting: Hetzner Cloud +### Hosting: Hetzner (Cloud + Robot) -- Server: CX21 (2 vCPU, 4 GB RAM, 40 GB SSD) - ~5 EUR/month -- Storage Box: 1 TB for RD5 segments + media - ~3.20 EUR/month -- Infrastructure as Code: Terraform (Hetzner provider) + Docker Compose -- CI/CD: GitHub Actions -- Monitoring: Grafana + Prometheus + Loki (flagship only) +trails.cool runs on two hosts in the same Falkenstein datacenter, +bridged via a Hetzner vSwitch (VLAN 4000) to a private network: + +- **Flagship** — Hetzner Cloud cx23 (2 vCPU, 4 GB RAM, 40 GB SSD). + Runs Journal, Planner, Postgres+PostGIS, Caddy, Prometheus, Loki, + Grafana, and exporters. vSwitch IP `10.0.0.2`. +- **BRouter host** — Hetzner Dedicated (operator-owned shared box, + currently `ullrich.is`; 3 TB RAID, 32 GB RAM). Runs only BRouter + + a Caddy auth sidecar + scoped cAdvisor/Promtail sidecars in a + `~trails/brouter/` compose project under a non-root `trails` user. + vSwitch IP `10.0.1.10`. BRouter covers the full planet + (~10 GB RD5 tiles) with an 8 GB JVM heap. + +Planner → BRouter traffic crosses the vSwitch; a shared-secret +`X-BRouter-Auth` header enforced by the Caddy sidecar prevents any +other process on the dedicated host from reaching BRouter even if +they share the private network. + +BRouter container metrics and logs are scraped/shipped from the +dedicated host to the flagship's Prometheus and Loki over the same +vSwitch. Filtering (cAdvisor `--whitelisted_container_labels`, Promtail +relabel-drop) keeps trails.cool observability scoped to trails +containers only — none of the operator's other workloads on the +shared host are ingested. + +- Storage Box: 1 TB for backups - ~3.20 EUR/month +- Infrastructure as Code: Terraform (Hetzner Cloud provider); Hetzner + Robot side (dedicated server) is operator-managed +- Docker Compose for runtime orchestration on both hosts +- CI/CD: GitHub Actions — three workflows (`cd-apps`, `cd-infra`, + `cd-brouter`) with different SSH targets and deploy users +- Monitoring: Grafana + Prometheus + Loki on flagship, scraping both + hosts - Error tracking: Sentry ### Services diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..d48ec22 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,78 @@ +# Deployment runbook + +trails.cool runs on two Hetzner hosts. This document covers what an +operator needs to know beyond the `CLAUDE.md` summary. + +## Hosts + +| Role | Host | IPs | SSH | +|------|------|-----|-----| +| Flagship (Cloud) | `trails.cool` | public + `10.0.0.2` (vSwitch) | `ssh -i ~/.ssh/trails-cool-deploy root@trails.cool` | +| BRouter (Dedicated) | `ullrich.is` | public `176.9.150.227` + `10.0.1.10` (vSwitch) | `ssh -i ~/.ssh/trails-brouter-deploy -p 2232 trails@ullrich.is` | + +Both hosts are in `fsn1` (Falkenstein) and joined on Hetzner vSwitch +#80672 (VLAN 4000). The flagship's Terraform (`infrastructure/terraform/`) +owns the Cloud Network + subnets + server attachment. The dedicated +host's VLAN sub-interface is configured out-of-band via netplan +(`/etc/netplan/60-trails-vswitch.yaml`), because the Robot side isn't +in the Hetzner Cloud API. + +## BRouter host — first-time provisioning + +See `infrastructure/brouter-host/README.md`. The short version: + +```bash +# As root (one-time firewall allowances): +ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 17777 proto tcp \ + comment 'trails brouter via flagship vSwitch' +ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp \ + comment 'trails brouter cadvisor via flagship vSwitch' + +# As the trails user: +cd ~/brouter # created by the first cd-brouter deploy +./download-segments.sh # ~10 GB, a few minutes on a good connection +docker compose pull +docker compose up -d +``` + +## Secrets rotation + +Tokens (including `BROUTER_AUTH_TOKEN`): + +1. Generate: `openssl rand -base64 32` +2. Edit: `SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt sops infrastructure/secrets.app.env` +3. Commit + push + merge → `cd-apps` redeploys the Planner with the new token. +4. Touch anything under `infrastructure/brouter-host/` (or run `gh workflow run cd-brouter.yml`) → `cd-brouter` redeploys the Caddy sidecar with the new token. +5. Brief overlap window where Planner sends new token while Caddy still checks the old value. Both redeploys should complete within a minute of each other; in the worst case a few Planner requests get 403 and retry. + +SOPS on macOS looks for the age key at `~/Library/Application Support/sops/age/keys.txt` by default. If yours lives under XDG-standard `~/.config/sops/age/keys.txt`, set `SOPS_AGE_KEY_FILE` as above or `export` it in your shell rc. + +## Cutover procedure (flagship BRouter → dedicated host) + +This is how `BROUTER_URL` gets flipped. Do it once the dedicated host +is provisioned, segments are seeded, and the compose project is up. + +1. **Pre-flight**: `curl -sfH "X-BRouter-Auth: $(sops -d infrastructure/secrets.app.env | grep ^BROUTER_AUTH_TOKEN= | cut -d= -f2-)" http://10.0.1.10:17777/brouter?lonlats=11.58,48.13\|11.59,48.14\&profile=trekking\&alternativeidx=0\&format=gpx` from the flagship. Expect 200 with GPX. Then curl without the header — expect 403. +2. **Wire the token** without flipping the URL. Edit SOPS: `sops infrastructure/secrets.app.env` — the `BROUTER_AUTH_TOKEN` is already in there. If `BROUTER_URL` isn't in SOPS, skip; the compose has a default. Merge. Planner redeploys; it now sends the header to the flagship BRouter (which ignores it). +3. **Flip the URL**. In SOPS, add `BROUTER_URL=http://10.0.1.10:17777`. Merge. `cd-apps` redeploys the Planner. +4. **Monitor**. Grafana "BRouter (dedicated host)" dashboard + `brouter_request_duration_seconds` on the Overview board. Watch for 30 minutes. +5. **Rollback** (if needed): remove the `BROUTER_URL` line from SOPS (falls back to the flagship default). Merge; redeploy. The flagship container is still warm during the soak window. +6. **Decommission flagship BRouter** (after 48 h of clean metrics): remove the `brouter:` service + `./segments` volume from `infrastructure/docker-compose.yml`. Merge. `cd-infra` restarts without BRouter. Reclaim ~2 GB of segment volume on the flagship. + +## Full restart (flagship) + +```bash +gh workflow run cd-infra.yml -f restart_all=true +``` + +Restarts every flagship service. Does NOT touch the BRouter host. + +## cd-brouter manual trigger + +```bash +gh workflow run cd-brouter.yml +``` + +Useful to redeploy the BRouter host after token rotation or a config +change, without needing a real source change under +`infrastructure/brouter-host/`. diff --git a/infrastructure/brouter-host/README.md b/infrastructure/brouter-host/README.md index 5db13a8..7123106 100644 --- a/infrastructure/brouter-host/README.md +++ b/infrastructure/brouter-host/README.md @@ -14,7 +14,22 @@ on the VLAN interface from `10.0.0.2`, the flagship's vSwitch IP). ## One-time provisioning -Runs as the `trails` user on the dedicated host. +### Operator (as root) — one-time firewall rules + +The dedicated host's UFW policy rejects anything not explicitly +allowed. Open the vSwitch ports the flagship needs: + +```bash +# BRouter Caddy sidecar (already added during section 1.2): +# ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 17777 proto tcp \ +# comment 'trails brouter via flagship vSwitch' + +# cAdvisor metrics endpoint (section 6) — add if not already: +ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp \ + comment 'trails brouter cadvisor via flagship vSwitch' +``` + +### Application bring-up (as the `trails` user) ```bash # 1. Land the compose project diff --git a/infrastructure/brouter-host/docker-compose.yml b/infrastructure/brouter-host/docker-compose.yml index 27edcb6..248eb5c 100644 --- a/infrastructure/brouter-host/docker-compose.yml +++ b/infrastructure/brouter-host/docker-compose.yml @@ -68,9 +68,61 @@ services: labels: trails.cool.service: "brouter-caddy" + # cAdvisor — container metrics scraped by flagship Prometheus over the + # vSwitch. We set container label filters so the exposed metrics only + # cover trails-labeled containers (brouter + caddy), not the operator's + # other workloads on this shared host. + cadvisor: + image: gcr.io/cadvisor/cadvisor:latest + container_name: trails-brouter-cadvisor + restart: unless-stopped + privileged: true + ports: + - "10.0.1.10:8080:8080" + volumes: + - /:/rootfs:ro + - /var/run:/var/run:ro + - /sys:/sys:ro + - /var/lib/docker:/var/lib/docker:ro + - /dev/disk/:/dev/disk:ro + command: + # Only consider containers (not the host) and only those that carry + # our label; filters out everything else on the shared host. + - --docker_only=true + - --store_container_labels=false + - --whitelisted_container_labels=trails.cool.service + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + labels: + trails.cool.service: "brouter-cadvisor" + + # Promtail — tails Docker logs for trails-labeled containers only and + # pushes to flagship Loki over the vSwitch. Does NOT scrape other + # containers' logs on this host. + promtail: + image: grafana/promtail:latest + container_name: trails-brouter-promtail + restart: unless-stopped + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - ./promtail-config.yml:/etc/promtail/config.yml:ro + - promtail-positions:/tmp + command: ["-config.file=/etc/promtail/config.yml"] + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" + labels: + trails.cool.service: "brouter-promtail" + volumes: caddy-data: caddy-config: + promtail-positions: networks: trails-brouter-internal: diff --git a/infrastructure/brouter-host/promtail-config.yml b/infrastructure/brouter-host/promtail-config.yml new file mode 100644 index 0000000..77f73db --- /dev/null +++ b/infrastructure/brouter-host/promtail-config.yml @@ -0,0 +1,36 @@ +server: + http_listen_port: 9080 + grpc_listen_port: 0 + +positions: + filename: /tmp/positions.yaml + +# Ship to the flagship's Loki over the vSwitch. The flagship exposes +# port 3100 on its vSwitch IP (10.0.0.2) for this purpose; Hetzner +# Cloud firewall blocks 3100 from the public internet. +clients: + - url: http://10.0.0.2:3100/loki/api/v1/push + +scrape_configs: + - job_name: docker + docker_sd_configs: + - host: unix:///var/run/docker.sock + refresh_interval: 10s + relabel_configs: + # Scope: only trails-labeled containers. Every other workload on + # this shared host is dropped before it reaches Loki. + - source_labels: ["__meta_docker_container_label_trails_cool_service"] + regex: "^$" + action: drop + # Labels mirror the flagship's promtail setup so dashboards are + # easy to unify: container, service, plus a host label that + # distinguishes this scraper from the flagship one. + - source_labels: ["__meta_docker_container_name"] + regex: "/?(.*)" + target_label: container + - source_labels: ["__meta_docker_container_label_trails_cool_service"] + target_label: service + - target_label: host + replacement: brouter + pipeline_stages: + - docker: {} diff --git a/infrastructure/docker-compose.yml b/infrastructure/docker-compose.yml index 9118553..24b1e71 100644 --- a/infrastructure/docker-compose.yml +++ b/infrastructure/docker-compose.yml @@ -182,6 +182,12 @@ services: - ./loki/loki-config.yml:/etc/loki/local-config.yaml:ro - loki_data:/loki command: ["-config.file=/etc/loki/local-config.yaml"] + # Publish only on the vSwitch IP so Promtail running on the + # dedicated BRouter host can push logs in. Hetzner Cloud firewall + # still blocks 3100 from the public internet. Internal services on + # this host reach Loki via the docker network as before. + ports: + - "10.0.0.2:3100:3100" grafana: image: grafana/grafana:latest diff --git a/infrastructure/grafana/dashboards/brouter.json b/infrastructure/grafana/dashboards/brouter.json new file mode 100644 index 0000000..30510fb --- /dev/null +++ b/infrastructure/grafana/dashboards/brouter.json @@ -0,0 +1,95 @@ +{ + "title": "BRouter (dedicated host)", + "uid": "trails-brouter", + "annotations": { + "list": [ + { + "name": "Deploys", + "enable": true, + "datasource": { "type": "grafana", "uid": "-- Grafana --" }, + "iconColor": "rgba(0, 211, 255, 1)", + "target": { "limit": 100, "matchAny": false, "tags": ["deploy", "brouter"], "type": "tags" } + } + ] + }, + "timezone": "browser", + "refresh": "30s", + "panels": [ + { + "title": "Scrape up/down", + "type": "stat", + "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 }, + "datasource": { "type": "prometheus", "uid": "prometheus" }, + "targets": [{ "expr": "up{job=\"brouter-cadvisor\"}", "legendFormat": "{{instance}}" }], + "options": { + "reduceOptions": { "values": false, "calcs": ["lastNotNull"] }, + "colorMode": "background" + }, + "fieldConfig": { + "defaults": { + "mappings": [ + { "type": "value", "options": { "0": { "text": "DOWN", "color": "red" } } }, + { "type": "value", "options": { "1": { "text": "UP", "color": "green" } } } + ], + "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] } + } + } + }, + { + "title": "BRouter request rate (from Planner)", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 6, "y": 0 }, + "datasource": { "type": "prometheus", "uid": "prometheus" }, + "targets": [{ "expr": "sum(rate(brouter_request_duration_seconds_count[5m])) by (status)", "legendFormat": "{{status}}" }] + }, + { + "title": "BRouter latency (p50/p95/p99, from Planner)", + "type": "timeseries", + "gridPos": { "h": 8, "w": 18, "x": 0, "y": 8 }, + "datasource": { "type": "prometheus", "uid": "prometheus" }, + "targets": [ + { "expr": "histogram_quantile(0.50, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p50" }, + { "expr": "histogram_quantile(0.95, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p95" }, + { "expr": "histogram_quantile(0.99, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p99" } + ], + "fieldConfig": { "defaults": { "unit": "s" } } + }, + { + "title": "Container memory (BRouter + Caddy sidecar)", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 }, + "datasource": { "type": "prometheus", "uid": "prometheus" }, + "targets": [ + { + "expr": "container_memory_usage_bytes{job=\"brouter-cadvisor\",name=~\"trails-brouter.*\"}", + "legendFormat": "{{name}}" + } + ], + "fieldConfig": { "defaults": { "unit": "bytes" } } + }, + { + "title": "Container CPU (BRouter + Caddy sidecar)", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 }, + "datasource": { "type": "prometheus", "uid": "prometheus" }, + "targets": [ + { + "expr": "sum by (name) (rate(container_cpu_usage_seconds_total{job=\"brouter-cadvisor\",name=~\"trails-brouter.*\"}[5m]))", + "legendFormat": "{{name}}" + } + ], + "fieldConfig": { "defaults": { "unit": "percentunit" } } + }, + { + "title": "Recent BRouter logs", + "type": "logs", + "gridPos": { "h": 10, "w": 24, "x": 0, "y": 24 }, + "datasource": { "type": "loki", "uid": "loki" }, + "targets": [{ "expr": "{host=\"brouter\", service=\"brouter\"}" }], + "options": { "showTime": true, "wrapLogMessage": false } + } + ], + "schemaVersion": 39, + "version": 1, + "time": { "from": "now-1h", "to": "now" } +} diff --git a/infrastructure/grafana/provisioning/alerting/alerts.yml b/infrastructure/grafana/provisioning/alerting/alerts.yml index 2cc7c9a..0ec3fe1 100644 --- a/infrastructure/grafana/provisioning/alerting/alerts.yml +++ b/infrastructure/grafana/provisioning/alerting/alerts.yml @@ -182,6 +182,30 @@ groups: annotations: summary: "Background jobs have failed in the last hour — check Grafana Service Health dashboard" + - uid: brouter-scrape-down + title: BRouter host unreachable + condition: B + noDataState: Alerting + data: + - refId: A + relativeTimeRange: { from: 300, to: 0 } + datasourceUid: prometheus + model: + expr: up{job="brouter-cadvisor"} + instant: true + - refId: B + datasourceUid: __expr__ + model: + type: threshold + expression: A + conditions: + - evaluator: { params: [1], type: lt } + operator: { type: and } + reducer: { type: last } + for: 2m + annotations: + summary: "BRouter host metrics scrape has been failing for 2+ minutes — the dedicated host, vSwitch, or cAdvisor may be down" + - uid: caddy-502-rate title: Caddy 502 errors detected condition: B diff --git a/infrastructure/prometheus/prometheus.yml b/infrastructure/prometheus/prometheus.yml index eefe45a..0c692f4 100644 --- a/infrastructure/prometheus/prometheus.yml +++ b/infrastructure/prometheus/prometheus.yml @@ -28,3 +28,13 @@ scrape_configs: - job_name: "caddy" static_configs: - targets: ["caddy:2019"] + + # BRouter runs on a separate Hetzner Robot host (ullrich.is), reached + # over the vSwitch at 10.0.1.10. cAdvisor there is scoped to trails- + # labeled containers only — we don't collect metrics for any of the + # operator's other workloads on that shared host. + - job_name: "brouter-cadvisor" + static_configs: + - targets: ["10.0.1.10:8080"] + labels: + host: "brouter" diff --git a/openspec/changes/relocate-brouter-to-dedicated-host/tasks.md b/openspec/changes/relocate-brouter-to-dedicated-host/tasks.md index 0aa3c30..45e9720 100644 --- a/openspec/changes/relocate-brouter-to-dedicated-host/tasks.md +++ b/openspec/changes/relocate-brouter-to-dedicated-host/tasks.md @@ -65,11 +65,17 @@ ## 6. Observability -- [ ] 6.1 Add a Prometheus scrape job in `infrastructure/prometheus/prometheus.yml` targeting the BRouter host's cAdvisor (or JMX exporter) on the vSwitch IP; label with `host="brouter"` -- [ ] 6.2 Run cAdvisor on the dedicated host as part of `infrastructure/brouter-host/docker-compose.yml`, configured to report only BRouter-labeled containers -- [ ] 6.3 Add a Promtail (or Alloy) service to `infrastructure/brouter-host/docker-compose.yml` tailing Docker logs for BRouter + Caddy sidecar only, pushing to the flagship Loki over vSwitch -- [ ] 6.4 Add a Grafana dashboard row (or new dashboard) for BRouter host: request rate, p50/p95/p99, JVM heap, container memory, scrape up/down -- [ ] 6.5 Add an alert: `up{job="brouter"} == 0 for 2m` +- [x] 6.1 Add a Prometheus scrape job in `infrastructure/prometheus/prometheus.yml` targeting the BRouter host's cAdvisor (or JMX exporter) on the vSwitch IP; label with `host="brouter"` + - Job `brouter-cadvisor` → `10.0.1.10:8080`. Uses static_configs with a static `host="brouter"` label so dashboards can filter. +- [x] 6.2 Run cAdvisor on the dedicated host as part of `infrastructure/brouter-host/docker-compose.yml`, configured to report only BRouter-labeled containers + - `--whitelisted_container_labels=trails.cool.service` + `--docker_only=true` scope metrics to trails containers only. Bound to `10.0.1.10:8080` (vSwitch-only). +- [x] 6.3 Add a Promtail (or Alloy) service to `infrastructure/brouter-host/docker-compose.yml` tailing Docker logs for BRouter + Caddy sidecar only, pushing to the flagship Loki over vSwitch + - Promtail with docker_sd + relabel-drop on missing `trails.cool.service` label; ships to `http://10.0.0.2:3100/loki/api/v1/push`. Also published Loki on flagship's vSwitch IP so the dedicated host can reach it. + - Requires operator one-time: `ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp` (documented in brouter-host/README.md). +- [x] 6.4 Add a Grafana dashboard row (or new dashboard) for BRouter host: request rate, p50/p95/p99, JVM heap, container memory, scrape up/down + - New `infrastructure/grafana/dashboards/brouter.json` with scrape up/down, request rate + latency (from Planner-side metrics), container memory/CPU, and a Loki logs panel filtered to `host="brouter"`. +- [x] 6.5 Add an alert: `up{job="brouter"} == 0 for 2m` + - Added as `brouter-scrape-down` in `infrastructure/grafana/provisioning/alerting/alerts.yml`. NoData state set to Alerting so a complete scrape outage still fires. ## 7. Cutover @@ -82,10 +88,14 @@ ## 8. Documentation -- [ ] 8.1 Update `CLAUDE.md` to mention the second deployment target and the `trails`-user deploy pattern for BRouter -- [ ] 8.2 Update `docs/architecture.md` with the new topology and vSwitch boundary -- [ ] 8.3 Update `docs/deployment.md` (or create) with the BRouter host runbook: first-time provisioning, segment updates, token rotation, rollback +- [x] 8.1 Update `CLAUDE.md` to mention the second deployment target and the `trails`-user deploy pattern for BRouter + - Deployment table now lists SSH target per workflow; new Hosts section explains the flagship + dedicated split and the vSwitch bridge. +- [x] 8.2 Update `docs/architecture.md` with the new topology and vSwitch boundary + - Hosting section rewritten to describe both hosts, the vSwitch, and the observability-scoping for the shared dedicated host. +- [x] 8.3 Update `docs/deployment.md` (or create) with the BRouter host runbook: first-time provisioning, segment updates, token rotation, rollback + - New file. Covers host layout, first-time provisioning, SOPS rotation (including the macOS SOPS_AGE_KEY_FILE gotcha), the full cutover procedure with rollback, and `gh workflow run cd-brouter.yml`. - [ ] 8.4 Add a note to `infrastructure/README.md` (if present) distinguishing flagship-host vs. BRouter-host compose projects + - No `infrastructure/README.md` currently exists; the `infrastructure/brouter-host/README.md` added in 3.5 + the updated `docs/deployment.md` cover the ground. Skip. ## 9. Verification