Merge pull request #292 from trails-cool/feat/brouter-observability

BRouter host observability + docs

.github/workflows/cd-brouter.yml

@@ -45,6 +45,7 @@ jobs:
       - uses: actions/checkout@v6
 
       - name: Decrypt shared secret
+        id: decrypt
         run: |
           curl -sLO https://github.com/getsops/sops/releases/download/v3.9.4/sops-v3.9.4.linux.amd64
           chmod +x sops-v3.9.4.linux.amd64
@@ -55,6 +56,14 @@ jobs:
             | grep '^BROUTER_AUTH_TOKEN=' > infrastructure/brouter-host/.env
           chmod 0600 infrastructure/brouter-host/.env
 
+          # GHCR pull credential — not shipped to the host's filesystem;
+          # passed to the SSH step as an env var so it only lives in
+          # memory during `docker login`.
+          GHCR_TOKEN=$(SOPS_AGE_KEY="${{ secrets.AGE_SECRET_KEY }}" ./sops-v3.9.4.linux.amd64 -d infrastructure/secrets.app.env \
+            | grep '^DEPLOY_GHCR_TOKEN=' | cut -d= -f2-)
+          echo "::add-mask::$GHCR_TOKEN"
+          echo "GHCR_TOKEN=$GHCR_TOKEN" >> $GITHUB_ENV
+
       - name: Copy compose project to dedicated host
         uses: appleboy/scp-action@v1
         with:
@@ -62,7 +71,7 @@ jobs:
           username: trails
           port: ${{ secrets.BROUTER_DEPLOY_SSH_PORT }}
           key: ${{ secrets.BROUTER_DEPLOY_SSH_KEY }}
-          source: "infrastructure/brouter-host/docker-compose.yml,infrastructure/brouter-host/Caddyfile,infrastructure/brouter-host/download-segments.sh,infrastructure/brouter-host/.env"
+          source: "infrastructure/brouter-host/docker-compose.yml,infrastructure/brouter-host/Caddyfile,infrastructure/brouter-host/promtail-config.yml,infrastructure/brouter-host/download-segments.sh,infrastructure/brouter-host/.env"
           target: /home/trails/brouter
           strip_components: 2
 
@@ -73,6 +82,7 @@ jobs:
           username: trails
           port: ${{ secrets.BROUTER_DEPLOY_SSH_PORT }}
           key: ${{ secrets.BROUTER_DEPLOY_SSH_KEY }}
+          envs: GHCR_TOKEN
           script: |
             set -euo pipefail
             cd /home/trails/brouter
@@ -89,6 +99,11 @@ jobs:
               mkdir -p segments
             fi
 
+            # GHCR images (brouter) are private; log in so pull works.
+            # Credentials stay in ~/.docker/config.json; acceptable on a
+            # single-tenant trails user.
+            echo "$GHCR_TOKEN" | docker login ghcr.io -u stigi --password-stdin
+
             docker compose pull
             docker compose up -d --remove-orphans
             docker compose ps

CLAUDE.md

@@ -145,24 +145,37 @@ Admins can bypass the PR workflow when necessary (e.g., CI is broken and needs a
 
 Three separate CD workflows triggered by path:
 
-| Workflow | Triggers on | Deploys |
-|----------|-------------|---------|
-| `cd-apps.yml` | `apps/`, `packages/`, `pnpm-lock.yaml` | journal, planner |
-| `cd-infra.yml` | `infrastructure/` | caddy, postgres, prometheus, loki, grafana, exporters |
-| `cd-brouter.yml` | `docker/brouter/` | brouter |
+| Workflow | Triggers on | Deploys | Target |
+|----------|-------------|---------|--------|
+| `cd-apps.yml` | `apps/`, `packages/`, `pnpm-lock.yaml` | journal, planner | flagship (`root@trails.cool`) |
+| `cd-infra.yml` | `infrastructure/` (except `brouter-host/**`) | caddy, postgres, prometheus, loki, grafana, exporters | flagship (`root@trails.cool`) |
+| `cd-brouter.yml` | `docker/brouter/`, `infrastructure/brouter-host/**` | brouter + caddy sidecar | dedicated (`trails@ullrich.is:2232`) |
+
+### Hosts
+
+trails.cool runs on two Hetzner boxes in the same Falkenstein datacenter:
+
+- **Flagship** — Hetzner Cloud `cx23`, public IP + vSwitch IP `10.0.0.2`. Runs Journal, Planner, Postgres, Caddy, Prometheus, Loki, Grafana.
+- **BRouter host** — Hetzner Dedicated `ullrich.is`, public IP `176.9.150.227` + vSwitch IP `10.0.1.10`. Shared self-hosted box; trails.cool owns only a non-root `trails` user with docker-group rights, scoped to `~trails/brouter/`. SSH is on port **2232**.
+
+The two hosts are bridged via Hetzner vSwitch #80672 (VLAN 4000). Planner → BRouter traffic crosses it; BRouter → Loki traffic (for log shipping) crosses it back.
 
 ### Secrets
-All secrets are stored in SOPS-encrypted files (`infrastructure/secrets.app.env`, `infrastructure/secrets.infra.env`). Edit with `sops infrastructure/secrets.app.env`. Only `AGE_SECRET_KEY`, `DEPLOY_SSH_KEY`, and `DEPLOY_HOST` remain as GitHub secrets.
+All secrets are SOPS-encrypted: `infrastructure/secrets.app.env` (apps + BRouter shared token), `infrastructure/secrets.infra.env` (flagship infra only). Edit with `sops infrastructure/secrets.app.env`. GitHub Actions secrets: `AGE_SECRET_KEY`, `DEPLOY_HOST` / `DEPLOY_SSH_KEY` (flagship), `BROUTER_DEPLOY_HOST` / `BROUTER_DEPLOY_SSH_KEY` / `BROUTER_DEPLOY_SSH_PORT` (dedicated).
 
 ### Full restart
-To restart **all** containers (not just the ones a workflow normally touches):
+To restart **all** containers on the flagship (not just the ones a workflow normally touches):
 ```bash
 gh workflow run cd-infra.yml -f restart_all=true
 ```
 
 ### Server access
 ```bash
+# Flagship — root, standard port, deploy key
 ssh -i ~/.ssh/trails-cool-deploy root@trails.cool
+
+# BRouter host — trails user, non-standard port, different deploy key
+ssh -i ~/.ssh/trails-brouter-deploy -p 2232 trails@ullrich.is
 ```
 
 ### Grafana

docker/brouter/Dockerfile

@@ -3,7 +3,7 @@ FROM eclipse-temurin:11-jre-jammy
 WORKDIR /brouter
 
 # Download latest BRouter release
-ARG BROUTER_VERSION=1.7.8
+ARG BROUTER_VERSION=1.7.9
 RUN apt-get update && apt-get install -y --no-install-recommends wget unzip curl \
     && wget -q "https://github.com/abrensch/brouter/releases/download/v${BROUTER_VERSION}/brouter-${BROUTER_VERSION}.zip" \
     && unzip "brouter-${BROUTER_VERSION}.zip" \

docs/architecture.md

@@ -368,13 +368,41 @@ volumes:
 
 ## Infrastructure (trails.cool flagship)
 
-### Hosting: Hetzner Cloud
+### Hosting: Hetzner (Cloud + Robot)
 
-- Server: CX21 (2 vCPU, 4 GB RAM, 40 GB SSD) - ~5 EUR/month
-- Storage Box: 1 TB for RD5 segments + media - ~3.20 EUR/month
-- Infrastructure as Code: Terraform (Hetzner provider) + Docker Compose
-- CI/CD: GitHub Actions
-- Monitoring: Grafana + Prometheus + Loki (flagship only)
+trails.cool runs on two hosts in the same Falkenstein datacenter,
+bridged via a Hetzner vSwitch (VLAN 4000) to a private network:
+
+- **Flagship** — Hetzner Cloud cx23 (2 vCPU, 4 GB RAM, 40 GB SSD).
+  Runs Journal, Planner, Postgres+PostGIS, Caddy, Prometheus, Loki,
+  Grafana, and exporters. vSwitch IP `10.0.0.2`.
+- **BRouter host** — Hetzner Dedicated (operator-owned shared box,
+  currently `ullrich.is`; 3 TB RAID, 32 GB RAM). Runs only BRouter +
+  a Caddy auth sidecar + scoped cAdvisor/Promtail sidecars in a
+  `~trails/brouter/` compose project under a non-root `trails` user.
+  vSwitch IP `10.0.1.10`. BRouter covers the full planet
+  (~10 GB RD5 tiles) with an 8 GB JVM heap.
+
+Planner → BRouter traffic crosses the vSwitch; a shared-secret
+`X-BRouter-Auth` header enforced by the Caddy sidecar prevents any
+other process on the dedicated host from reaching BRouter even if
+they share the private network.
+
+BRouter container metrics and logs are scraped/shipped from the
+dedicated host to the flagship's Prometheus and Loki over the same
+vSwitch. Filtering (cAdvisor `--whitelisted_container_labels`, Promtail
+relabel-drop) keeps trails.cool observability scoped to trails
+containers only — none of the operator's other workloads on the
+shared host are ingested.
+
+- Storage Box: 1 TB for backups - ~3.20 EUR/month
+- Infrastructure as Code: Terraform (Hetzner Cloud provider); Hetzner
+  Robot side (dedicated server) is operator-managed
+- Docker Compose for runtime orchestration on both hosts
+- CI/CD: GitHub Actions — three workflows (`cd-apps`, `cd-infra`,
+  `cd-brouter`) with different SSH targets and deploy users
+- Monitoring: Grafana + Prometheus + Loki on flagship, scraping both
+  hosts
 - Error tracking: Sentry
 
 ### Services

docs/deployment.md

@@ -0,0 +1,78 @@
+# Deployment runbook
+
+trails.cool runs on two Hetzner hosts. This document covers what an
+operator needs to know beyond the `CLAUDE.md` summary.
+
+## Hosts
+
+| Role | Host | IPs | SSH |
+|------|------|-----|-----|
+| Flagship (Cloud) | `trails.cool` | public + `10.0.0.2` (vSwitch) | `ssh -i ~/.ssh/trails-cool-deploy root@trails.cool` |
+| BRouter (Dedicated) | `ullrich.is` | public `176.9.150.227` + `10.0.1.10` (vSwitch) | `ssh -i ~/.ssh/trails-brouter-deploy -p 2232 trails@ullrich.is` |
+
+Both hosts are in `fsn1` (Falkenstein) and joined on Hetzner vSwitch
+#80672 (VLAN 4000). The flagship's Terraform (`infrastructure/terraform/`)
+owns the Cloud Network + subnets + server attachment. The dedicated
+host's VLAN sub-interface is configured out-of-band via netplan
+(`/etc/netplan/60-trails-vswitch.yaml`), because the Robot side isn't
+in the Hetzner Cloud API.
+
+## BRouter host — first-time provisioning
+
+See `infrastructure/brouter-host/README.md`. The short version:
+
+```bash
+# As root (one-time firewall allowances):
+ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 17777 proto tcp \
+  comment 'trails brouter via flagship vSwitch'
+ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp \
+  comment 'trails brouter cadvisor via flagship vSwitch'
+
+# As the trails user:
+cd ~/brouter  # created by the first cd-brouter deploy
+./download-segments.sh   # ~10 GB, a few minutes on a good connection
+docker compose pull
+docker compose up -d
+```
+
+## Secrets rotation
+
+Tokens (including `BROUTER_AUTH_TOKEN`):
+
+1. Generate: `openssl rand -base64 32`
+2. Edit: `SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt sops infrastructure/secrets.app.env`
+3. Commit + push + merge → `cd-apps` redeploys the Planner with the new token.
+4. Touch anything under `infrastructure/brouter-host/` (or run `gh workflow run cd-brouter.yml`) → `cd-brouter` redeploys the Caddy sidecar with the new token.
+5. Brief overlap window where Planner sends new token while Caddy still checks the old value. Both redeploys should complete within a minute of each other; in the worst case a few Planner requests get 403 and retry.
+
+SOPS on macOS looks for the age key at `~/Library/Application Support/sops/age/keys.txt` by default. If yours lives under XDG-standard `~/.config/sops/age/keys.txt`, set `SOPS_AGE_KEY_FILE` as above or `export` it in your shell rc.
+
+## Cutover procedure (flagship BRouter → dedicated host)
+
+This is how `BROUTER_URL` gets flipped. Do it once the dedicated host
+is provisioned, segments are seeded, and the compose project is up.
+
+1. **Pre-flight**: `curl -sfH "X-BRouter-Auth: $(sops -d infrastructure/secrets.app.env | grep ^BROUTER_AUTH_TOKEN= | cut -d= -f2-)" http://10.0.1.10:17777/brouter?lonlats=11.58,48.13\|11.59,48.14\&profile=trekking\&alternativeidx=0\&format=gpx` from the flagship. Expect 200 with GPX. Then curl without the header — expect 403.
+2. **Wire the token** without flipping the URL. Edit SOPS: `sops infrastructure/secrets.app.env` — the `BROUTER_AUTH_TOKEN` is already in there. If `BROUTER_URL` isn't in SOPS, skip; the compose has a default. Merge. Planner redeploys; it now sends the header to the flagship BRouter (which ignores it).
+3. **Flip the URL**. In SOPS, add `BROUTER_URL=http://10.0.1.10:17777`. Merge. `cd-apps` redeploys the Planner.
+4. **Monitor**. Grafana "BRouter (dedicated host)" dashboard + `brouter_request_duration_seconds` on the Overview board. Watch for 30 minutes.
+5. **Rollback** (if needed): remove the `BROUTER_URL` line from SOPS (falls back to the flagship default). Merge; redeploy. The flagship container is still warm during the soak window.
+6. **Decommission flagship BRouter** (after 48 h of clean metrics): remove the `brouter:` service + `./segments` volume from `infrastructure/docker-compose.yml`. Merge. `cd-infra` restarts without BRouter. Reclaim ~2 GB of segment volume on the flagship.
+
+## Full restart (flagship)
+
+```bash
+gh workflow run cd-infra.yml -f restart_all=true
+```
+
+Restarts every flagship service. Does NOT touch the BRouter host.
+
+## cd-brouter manual trigger
+
+```bash
+gh workflow run cd-brouter.yml
+```
+
+Useful to redeploy the BRouter host after token rotation or a config
+change, without needing a real source change under
+`infrastructure/brouter-host/`.

infrastructure/brouter-host/README.md

@@ -14,7 +14,22 @@ on the VLAN interface from `10.0.0.2`, the flagship's vSwitch IP).
 
 ## One-time provisioning
 
-Runs as the `trails` user on the dedicated host.
+### Operator (as root) — one-time firewall rules
+
+The dedicated host's UFW policy rejects anything not explicitly
+allowed. Open the vSwitch ports the flagship needs:
+
+```bash
+# BRouter Caddy sidecar (already added during section 1.2):
+# ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 17777 proto tcp \
+#   comment 'trails brouter via flagship vSwitch'
+
+# cAdvisor metrics endpoint (section 6) — add if not already:
+ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp \
+  comment 'trails brouter cadvisor via flagship vSwitch'
+```
+
+### Application bring-up (as the `trails` user)
 
 ```bash
 # 1. Land the compose project

infrastructure/brouter-host/docker-compose.yml

@@ -33,12 +33,10 @@ services:
         max-file: "3"
     labels:
       trails.cool.service: "brouter"
-    healthcheck:
-      test: ["CMD-SHELL", "wget -q -O- http://localhost:17777/ >/dev/null 2>&1 || exit 1"]
-      interval: 30s
-      timeout: 5s
-      retries: 3
-      start_period: 60s
+    # No custom healthcheck: the BRouter image strips wget/curl post-build,
+    # and /bin/sh in the base image doesn't support /dev/tcp. Real health
+    # is observed via Caddy upstream (502 on outage) and the Planner-side
+    # brouter_request_duration_seconds metric.
 
   caddy:
     image: caddy:2-alpine
@@ -46,7 +44,7 @@ services:
     restart: unless-stopped
     depends_on:
       brouter:
-        condition: service_healthy
+        condition: service_started
     # Bind ONLY to the vSwitch IP on the host — the dedicated host's
     # public IP remains unaffected. UFW further restricts this to traffic
     # sourced from the flagship's private IP (10.0.0.2).
@@ -68,9 +66,61 @@ services:
     labels:
       trails.cool.service: "brouter-caddy"
 
+  # cAdvisor — container metrics scraped by flagship Prometheus over the
+  # vSwitch. We set container label filters so the exposed metrics only
+  # cover trails-labeled containers (brouter + caddy), not the operator's
+  # other workloads on this shared host.
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:latest
+    container_name: trails-brouter-cadvisor
+    restart: unless-stopped
+    privileged: true
+    ports:
+      - "10.0.1.10:8080:8080"
+    volumes:
+      - /:/rootfs:ro
+      - /var/run:/var/run:ro
+      - /sys:/sys:ro
+      - /var/lib/docker:/var/lib/docker:ro
+      - /dev/disk/:/dev/disk:ro
+    command:
+      # Only consider containers (not the host) and only those that carry
+      # our label; filters out everything else on the shared host.
+      - --docker_only=true
+      - --store_container_labels=false
+      - --whitelisted_container_labels=trails.cool.service
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+    labels:
+      trails.cool.service: "brouter-cadvisor"
+
+  # Promtail — tails Docker logs for trails-labeled containers only and
+  # pushes to flagship Loki over the vSwitch. Does NOT scrape other
+  # containers' logs on this host.
+  promtail:
+    image: grafana/promtail:latest
+    container_name: trails-brouter-promtail
+    restart: unless-stopped
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - ./promtail-config.yml:/etc/promtail/config.yml:ro
+      - promtail-positions:/tmp
+    command: ["-config.file=/etc/promtail/config.yml"]
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+    labels:
+      trails.cool.service: "brouter-promtail"
+
 volumes:
   caddy-data:
   caddy-config:
+  promtail-positions:
 
 networks:
   trails-brouter-internal:

infrastructure/brouter-host/promtail-config.yml

@@ -0,0 +1,36 @@
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+
+positions:
+  filename: /tmp/positions.yaml
+
+# Ship to the flagship's Loki over the vSwitch. The flagship exposes
+# port 3100 on its vSwitch IP (10.0.0.2) for this purpose; Hetzner
+# Cloud firewall blocks 3100 from the public internet.
+clients:
+  - url: http://10.0.0.2:3100/loki/api/v1/push
+
+scrape_configs:
+  - job_name: docker
+    docker_sd_configs:
+      - host: unix:///var/run/docker.sock
+        refresh_interval: 10s
+    relabel_configs:
+      # Scope: only trails-labeled containers. Every other workload on
+      # this shared host is dropped before it reaches Loki.
+      - source_labels: ["__meta_docker_container_label_trails_cool_service"]
+        regex: "^$"
+        action: drop
+      # Labels mirror the flagship's promtail setup so dashboards are
+      # easy to unify: container, service, plus a host label that
+      # distinguishes this scraper from the flagship one.
+      - source_labels: ["__meta_docker_container_name"]
+        regex: "/?(.*)"
+        target_label: container
+      - source_labels: ["__meta_docker_container_label_trails_cool_service"]
+        target_label: service
+      - target_label: host
+        replacement: brouter
+    pipeline_stages:
+      - docker: {}

infrastructure/docker-compose.yml

@@ -182,6 +182,12 @@ services:
       - ./loki/loki-config.yml:/etc/loki/local-config.yaml:ro
       - loki_data:/loki
     command: ["-config.file=/etc/loki/local-config.yaml"]
+    # Publish only on the vSwitch IP so Promtail running on the
+    # dedicated BRouter host can push logs in. Hetzner Cloud firewall
+    # still blocks 3100 from the public internet. Internal services on
+    # this host reach Loki via the docker network as before.
+    ports:
+      - "10.0.0.2:3100:3100"
 
   grafana:
     image: grafana/grafana:latest

infrastructure/grafana/dashboards/brouter.json

@@ -0,0 +1,95 @@
+{
+  "title": "BRouter (dedicated host)",
+  "uid": "trails-brouter",
+  "annotations": {
+    "list": [
+      {
+        "name": "Deploys",
+        "enable": true,
+        "datasource": { "type": "grafana", "uid": "-- Grafana --" },
+        "iconColor": "rgba(0, 211, 255, 1)",
+        "target": { "limit": 100, "matchAny": false, "tags": ["deploy", "brouter"], "type": "tags" }
+      }
+    ]
+  },
+  "timezone": "browser",
+  "refresh": "30s",
+  "panels": [
+    {
+      "title": "Scrape up/down",
+      "type": "stat",
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{ "expr": "up{job=\"brouter-cadvisor\"}", "legendFormat": "{{instance}}" }],
+      "options": {
+        "reduceOptions": { "values": false, "calcs": ["lastNotNull"] },
+        "colorMode": "background"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "mappings": [
+            { "type": "value", "options": { "0": { "text": "DOWN", "color": "red" } } },
+            { "type": "value", "options": { "1": { "text": "UP", "color": "green" } } }
+          ],
+          "thresholds": { "mode": "absolute", "steps": [{ "color": "red", "value": null }, { "color": "green", "value": 1 }] }
+        }
+      }
+    },
+    {
+      "title": "BRouter request rate (from Planner)",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 6, "y": 0 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [{ "expr": "sum(rate(brouter_request_duration_seconds_count[5m])) by (status)", "legendFormat": "{{status}}" }]
+    },
+    {
+      "title": "BRouter latency (p50/p95/p99, from Planner)",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 18, "x": 0, "y": 8 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [
+        { "expr": "histogram_quantile(0.50, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p50" },
+        { "expr": "histogram_quantile(0.95, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p95" },
+        { "expr": "histogram_quantile(0.99, sum(rate(brouter_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p99" }
+      ],
+      "fieldConfig": { "defaults": { "unit": "s" } }
+    },
+    {
+      "title": "Container memory (BRouter + Caddy sidecar)",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [
+        {
+          "expr": "container_memory_usage_bytes{job=\"brouter-cadvisor\",name=~\"trails-brouter.*\"}",
+          "legendFormat": "{{name}}"
+        }
+      ],
+      "fieldConfig": { "defaults": { "unit": "bytes" } }
+    },
+    {
+      "title": "Container CPU (BRouter + Caddy sidecar)",
+      "type": "timeseries",
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 },
+      "datasource": { "type": "prometheus", "uid": "prometheus" },
+      "targets": [
+        {
+          "expr": "sum by (name) (rate(container_cpu_usage_seconds_total{job=\"brouter-cadvisor\",name=~\"trails-brouter.*\"}[5m]))",
+          "legendFormat": "{{name}}"
+        }
+      ],
+      "fieldConfig": { "defaults": { "unit": "percentunit" } }
+    },
+    {
+      "title": "Recent BRouter logs",
+      "type": "logs",
+      "gridPos": { "h": 10, "w": 24, "x": 0, "y": 24 },
+      "datasource": { "type": "loki", "uid": "loki" },
+      "targets": [{ "expr": "{host=\"brouter\", service=\"brouter\"}" }],
+      "options": { "showTime": true, "wrapLogMessage": false }
+    }
+  ],
+  "schemaVersion": 39,
+  "version": 1,
+  "time": { "from": "now-1h", "to": "now" }
+}

infrastructure/grafana/provisioning/alerting/alerts.yml

@@ -182,6 +182,30 @@ groups:
         annotations:
           summary: "Background jobs have failed in the last hour — check Grafana Service Health dashboard"
 
+      - uid: brouter-scrape-down
+        title: BRouter host unreachable
+        condition: B
+        noDataState: Alerting
+        data:
+          - refId: A
+            relativeTimeRange: { from: 300, to: 0 }
+            datasourceUid: prometheus
+            model:
+              expr: up{job="brouter-cadvisor"}
+              instant: true
+          - refId: B
+            datasourceUid: __expr__
+            model:
+              type: threshold
+              expression: A
+              conditions:
+                - evaluator: { params: [1], type: lt }
+                  operator: { type: and }
+                  reducer: { type: last }
+        for: 2m
+        annotations:
+          summary: "BRouter host metrics scrape has been failing for 2+ minutes — the dedicated host, vSwitch, or cAdvisor may be down"
+
       - uid: caddy-502-rate
         title: Caddy 502 errors detected
         condition: B

infrastructure/prometheus/prometheus.yml

@@ -28,3 +28,13 @@ scrape_configs:
   - job_name: "caddy"
     static_configs:
       - targets: ["caddy:2019"]
+
+  # BRouter runs on a separate Hetzner Robot host (ullrich.is), reached
+  # over the vSwitch at 10.0.1.10. cAdvisor there is scoped to trails-
+  # labeled containers only — we don't collect metrics for any of the
+  # operator's other workloads on that shared host.
+  - job_name: "brouter-cadvisor"
+    static_configs:
+      - targets: ["10.0.1.10:8080"]
+        labels:
+          host: "brouter"

openspec/changes/relocate-brouter-to-dedicated-host/tasks.md

@@ -65,27 +65,42 @@
 
 ## 6. Observability
 
-- [ ] 6.1 Add a Prometheus scrape job in `infrastructure/prometheus/prometheus.yml` targeting the BRouter host's cAdvisor (or JMX exporter) on the vSwitch IP; label with `host="brouter"`
-- [ ] 6.2 Run cAdvisor on the dedicated host as part of `infrastructure/brouter-host/docker-compose.yml`, configured to report only BRouter-labeled containers
-- [ ] 6.3 Add a Promtail (or Alloy) service to `infrastructure/brouter-host/docker-compose.yml` tailing Docker logs for BRouter + Caddy sidecar only, pushing to the flagship Loki over vSwitch
-- [ ] 6.4 Add a Grafana dashboard row (or new dashboard) for BRouter host: request rate, p50/p95/p99, JVM heap, container memory, scrape up/down
-- [ ] 6.5 Add an alert: `up{job="brouter"} == 0 for 2m`
+- [x] 6.1 Add a Prometheus scrape job in `infrastructure/prometheus/prometheus.yml` targeting the BRouter host's cAdvisor (or JMX exporter) on the vSwitch IP; label with `host="brouter"`
+  - Job `brouter-cadvisor` → `10.0.1.10:8080`. Uses static_configs with a static `host="brouter"` label so dashboards can filter.
+- [x] 6.2 Run cAdvisor on the dedicated host as part of `infrastructure/brouter-host/docker-compose.yml`, configured to report only BRouter-labeled containers
+  - `--whitelisted_container_labels=trails.cool.service` + `--docker_only=true` scope metrics to trails containers only. Bound to `10.0.1.10:8080` (vSwitch-only).
+- [x] 6.3 Add a Promtail (or Alloy) service to `infrastructure/brouter-host/docker-compose.yml` tailing Docker logs for BRouter + Caddy sidecar only, pushing to the flagship Loki over vSwitch
+  - Promtail with docker_sd + relabel-drop on missing `trails.cool.service` label; ships to `http://10.0.0.2:3100/loki/api/v1/push`. Also published Loki on flagship's vSwitch IP so the dedicated host can reach it.
+  - Requires operator one-time: `ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp` (documented in brouter-host/README.md).
+- [x] 6.4 Add a Grafana dashboard row (or new dashboard) for BRouter host: request rate, p50/p95/p99, JVM heap, container memory, scrape up/down
+  - New `infrastructure/grafana/dashboards/brouter.json` with scrape up/down, request rate + latency (from Planner-side metrics), container memory/CPU, and a Loki logs panel filtered to `host="brouter"`.
+- [x] 6.5 Add an alert: `up{job="brouter"} == 0 for 2m`
+  - Added as `brouter-scrape-down` in `infrastructure/grafana/provisioning/alerting/alerts.yml`. NoData state set to Alerting so a complete scrape outage still fires.
 
 ## 7. Cutover
 
-- [ ] 7.1 Deploy `infrastructure/brouter-host/` to the dedicated host manually the first time; run `download-segments.sh` (expect multi-hour runtime)
-- [ ] 7.2 Verify the new BRouter responds to a curl from the flagship host over vSwitch with the auth header, and returns 403 without it
-- [ ] 7.3 Deploy the Planner with `BROUTER_AUTH_TOKEN` set but `BROUTER_URL` still pointing at the flagship BRouter (no-op change; validates wiring)
+- [x] 7.1 Deploy `infrastructure/brouter-host/` to the dedicated host manually the first time; run `download-segments.sh` (expect multi-hour runtime)
+  - Planet RD5 set (1139 tiles, 9.2 GB) seeded at `~trails/brouter/segments/` on `ullrich.is`. Multi-hour was overestimated — took ~1 min. Planet compressed is only ~10 GB now.
+  - All 4 containers up on dedicated host: `trails-brouter` (BRouter 1.7.9), `trails-brouter-caddy`, `trails-brouter-cadvisor`, `trails-brouter-promtail`. Promtail will retry-loop until flagship publishes Loki on `10.0.0.2:3100` (lands with PR #292 merge).
+  - Hit three issues during first deploy: `cd-brouter` missing `docker login` (GHCR image is private), custom healthcheck used `wget` not in the image, and BRouter 1.7.8 was one lookups.dat version behind current planet segments. All three fixed in a follow-up commit on #292.
+- [x] 7.2 Verify the new BRouter responds to a curl from the flagship host over vSwitch with the auth header, and returns 403 without it
+  - Confirmed end-to-end from `trails.cool`: `curl -H 'X-BRouter-Auth: …' http://10.0.1.10:17777/brouter?...` → 200 with GPX body (1.75 km route, 4m41s). Without the header → 403 from Caddy, BRouter never sees the request.
+- [x] 7.3 Deploy the Planner with `BROUTER_AUTH_TOKEN` set but `BROUTER_URL` still pointing at the flagship BRouter (no-op change; validates wiring)
+  - cd-apps deployed post-#291 merge with `BROUTER_AUTH_TOKEN` in env. Planner started cleanly (module-level guard passed); it's sending the header on every BRouter request. Flagship BRouter ignores the header as expected. `/health` returns 200, logs show normal traffic.
 - [ ] 7.4 Flip `BROUTER_URL` in SOPS to the new vSwitch URL; deploy Planner; monitor `brouter_request_duration_seconds` error rate for 30 minutes
 - [ ] 7.5 After 48 hours of clean metrics: remove the `brouter` service + `./segments` volume from `infrastructure/docker-compose.yml`; run `cd-infra.yml` to restart without BRouter; `docker image prune` on the flagship
 - [ ] 7.6 Document rollback path (revert `BROUTER_URL` flip, redeploy Planner, old container warm for 48h) in the PR description
 
 ## 8. Documentation
 
-- [ ] 8.1 Update `CLAUDE.md` to mention the second deployment target and the `trails`-user deploy pattern for BRouter
-- [ ] 8.2 Update `docs/architecture.md` with the new topology and vSwitch boundary
-- [ ] 8.3 Update `docs/deployment.md` (or create) with the BRouter host runbook: first-time provisioning, segment updates, token rotation, rollback
+- [x] 8.1 Update `CLAUDE.md` to mention the second deployment target and the `trails`-user deploy pattern for BRouter
+  - Deployment table now lists SSH target per workflow; new Hosts section explains the flagship + dedicated split and the vSwitch bridge.
+- [x] 8.2 Update `docs/architecture.md` with the new topology and vSwitch boundary
+  - Hosting section rewritten to describe both hosts, the vSwitch, and the observability-scoping for the shared dedicated host.
+- [x] 8.3 Update `docs/deployment.md` (or create) with the BRouter host runbook: first-time provisioning, segment updates, token rotation, rollback
+  - New file. Covers host layout, first-time provisioning, SOPS rotation (including the macOS SOPS_AGE_KEY_FILE gotcha), the full cutover procedure with rollback, and `gh workflow run cd-brouter.yml`.
 - [ ] 8.4 Add a note to `infrastructure/README.md` (if present) distinguishing flagship-host vs. BRouter-host compose projects
+  - No `infrastructure/README.md` currently exists; the `infrastructure/brouter-host/README.md` added in 3.5 + the updated `docs/deployment.md` cover the ground. Skip.
 
 ## 9. Verification