trails-cool/trails
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

BRouter host observability + docs

Browse source 
Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top
of the host compose in #291.

## Observability (section 6)

- **Prometheus**: new `brouter-cadvisor` job scraping
  `10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`.
- **cAdvisor sidecar** on the dedicated host's compose
  (`--docker_only --whitelisted_container_labels=trails.cool.service`)
  so metrics only cover trails containers, never the operator's
  unrelated workloads on the shared box.
- **Promtail sidecar** on the dedicated host, Docker SD with relabel-
  drop on missing `trails.cool.service` label, pushing to flagship
  Loki at `http://10.0.0.2:3100/loki/api/v1/push`.
- **Flagship compose**: Loki now publishes port 3100 on the vSwitch
  IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from
  the public internet.
- **Grafana dashboard**: `brouter.json` — scrape up/down, request
  rate (from Planner-side `brouter_request_duration_seconds`),
  p50/p95/p99, container memory/CPU, Loki logs panel.
- **Alert**: `brouter-scrape-down` fires on
  `up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting`
  so a total scrape failure still pages.

Operator needs one UFW rule on the dedicated host for the cAdvisor
port — documented in `infrastructure/brouter-host/README.md`.

## Documentation (section 8)

- `CLAUDE.md` — hosts table + updated deployment table with SSH
  targets per workflow; BRouter host SSH is `-p 2232 trails@...`,
  different key.
- `docs/architecture.md` — Hosting section rewritten to cover both
  hosts, vSwitch boundary, and the observability-scoping rationale
  for the shared dedicated host.
- `docs/deployment.md` (new) — full operator runbook: host layout,
  first-time BRouter provisioning, SOPS rotation (including the
  macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with
  rollback, manual workflow triggers.

Task 8.4 (infrastructure/README.md) skipped: that file doesn't
exist and the ground is covered by brouter-host/README.md +
docs/deployment.md.

## Validation

- `docker compose config` on both the flagship and brouter-host
  compose files — both validate.
- `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full
  turbo cache hits; no code changes in this commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-04-23 23:02:35 +02:00
parent 34b2e44ed1
commit 9e598fb6a1
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
12 changed files with 390 additions and 23 deletions
Download patch file Download diff file Expand all files Collapse all files

40
docs/architecture.md
View file

@ -368,13 +368,41 @@ volumes:
## Infrastructure (trails.cool flagship)
### Hosting: Hetzner Cloud
### Hosting: Hetzner (Cloud + Robot)
- Server: CX21 (2 vCPU, 4 GB RAM, 40 GB SSD) - ~5 EUR/month
- Storage Box: 1 TB for RD5 segments + media - ~3.20 EUR/month
- Infrastructure as Code: Terraform (Hetzner provider) + Docker Compose
- CI/CD: GitHub Actions
- Monitoring: Grafana + Prometheus + Loki (flagship only)
trails.cool runs on two hosts in the same Falkenstein datacenter,
bridged via a Hetzner vSwitch (VLAN 4000) to a private network:
- **Flagship** — Hetzner Cloud cx23 (2 vCPU, 4 GB RAM, 40 GB SSD).
Runs Journal, Planner, Postgres+PostGIS, Caddy, Prometheus, Loki,
Grafana, and exporters. vSwitch IP `10.0.0.2`.
- **BRouter host** — Hetzner Dedicated (operator-owned shared box,
currently `ullrich.is`; 3 TB RAID, 32 GB RAM). Runs only BRouter +
a Caddy auth sidecar + scoped cAdvisor/Promtail sidecars in a
`~trails/brouter/` compose project under a non-root `trails` user.
vSwitch IP `10.0.1.10`. BRouter covers the full planet
(~10 GB RD5 tiles) with an 8 GB JVM heap.
Planner → BRouter traffic crosses the vSwitch; a shared-secret
`X-BRouter-Auth` header enforced by the Caddy sidecar prevents any
other process on the dedicated host from reaching BRouter even if
they share the private network.
BRouter container metrics and logs are scraped/shipped from the
dedicated host to the flagship's Prometheus and Loki over the same
vSwitch. Filtering (cAdvisor `--whitelisted_container_labels`, Promtail
relabel-drop) keeps trails.cool observability scoped to trails
containers only — none of the operator's other workloads on the
shared host are ingested.
- Storage Box: 1 TB for backups - ~3.20 EUR/month
- Infrastructure as Code: Terraform (Hetzner Cloud provider); Hetzner
Robot side (dedicated server) is operator-managed
- Docker Compose for runtime orchestration on both hosts
- CI/CD: GitHub Actions — three workflows (`cd-apps`, `cd-infra`,
`cd-brouter`) with different SSH targets and deploy users
- Monitoring: Grafana + Prometheus + Loki on flagship, scraping both
hosts
- Error tracking: Sentry
### Services

78
docs/deployment.md Normal file
View file

@ -0,0 +1,78 @@
# Deployment runbook
trails.cool runs on two Hetzner hosts. This document covers what an
operator needs to know beyond the `CLAUDE.md` summary.
## Hosts
| Role | Host | IPs | SSH |
|------|------|-----|-----|
| Flagship (Cloud) | `trails.cool` | public + `10.0.0.2` (vSwitch) | `ssh -i ~/.ssh/trails-cool-deploy root@trails.cool` |
| BRouter (Dedicated) | `ullrich.is` | public `176.9.150.227` + `10.0.1.10` (vSwitch) | `ssh -i ~/.ssh/trails-brouter-deploy -p 2232 trails@ullrich.is` |
Both hosts are in `fsn1` (Falkenstein) and joined on Hetzner vSwitch
#80672 (VLAN 4000). The flagship's Terraform (`infrastructure/terraform/`)
owns the Cloud Network + subnets + server attachment. The dedicated
host's VLAN sub-interface is configured out-of-band via netplan
(`/etc/netplan/60-trails-vswitch.yaml`), because the Robot side isn't
in the Hetzner Cloud API.
## BRouter host — first-time provisioning
See `infrastructure/brouter-host/README.md`. The short version:
```bash
# As root (one-time firewall allowances):
ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 17777 proto tcp \
comment 'trails brouter via flagship vSwitch'
ufw allow in on enp4s0.4000 from 10.0.0.2 to any port 8080 proto tcp \
comment 'trails brouter cadvisor via flagship vSwitch'
# As the trails user:
cd ~/brouter # created by the first cd-brouter deploy
./download-segments.sh # ~10 GB, a few minutes on a good connection
docker compose pull
docker compose up -d
```
## Secrets rotation
Tokens (including `BROUTER_AUTH_TOKEN`):
1. Generate: `openssl rand -base64 32`
2. Edit: `SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt sops infrastructure/secrets.app.env`
3. Commit + push + merge → `cd-apps` redeploys the Planner with the new token.
4. Touch anything under `infrastructure/brouter-host/` (or run `gh workflow run cd-brouter.yml`) → `cd-brouter` redeploys the Caddy sidecar with the new token.
5. Brief overlap window where Planner sends new token while Caddy still checks the old value. Both redeploys should complete within a minute of each other; in the worst case a few Planner requests get 403 and retry.
SOPS on macOS looks for the age key at `~/Library/Application Support/sops/age/keys.txt` by default. If yours lives under XDG-standard `~/.config/sops/age/keys.txt`, set `SOPS_AGE_KEY_FILE` as above or `export` it in your shell rc.
## Cutover procedure (flagship BRouter → dedicated host)
This is how `BROUTER_URL` gets flipped. Do it once the dedicated host
is provisioned, segments are seeded, and the compose project is up.
1. **Pre-flight**: `curl -sfH "X-BRouter-Auth: $(sops -d infrastructure/secrets.app.env | grep ^BROUTER_AUTH_TOKEN= | cut -d= -f2-)" http://10.0.1.10:17777/brouter?lonlats=11.58,48.13\|11.59,48.14\&profile=trekking\&alternativeidx=0\&format=gpx` from the flagship. Expect 200 with GPX. Then curl without the header — expect 403.
2. **Wire the token** without flipping the URL. Edit SOPS: `sops infrastructure/secrets.app.env` — the `BROUTER_AUTH_TOKEN` is already in there. If `BROUTER_URL` isn't in SOPS, skip; the compose has a default. Merge. Planner redeploys; it now sends the header to the flagship BRouter (which ignores it).
3. **Flip the URL**. In SOPS, add `BROUTER_URL=http://10.0.1.10:17777`. Merge. `cd-apps` redeploys the Planner.
4. **Monitor**. Grafana "BRouter (dedicated host)" dashboard + `brouter_request_duration_seconds` on the Overview board. Watch for 30 minutes.
5. **Rollback** (if needed): remove the `BROUTER_URL` line from SOPS (falls back to the flagship default). Merge; redeploy. The flagship container is still warm during the soak window.
6. **Decommission flagship BRouter** (after 48 h of clean metrics): remove the `brouter:` service + `./segments` volume from `infrastructure/docker-compose.yml`. Merge. `cd-infra` restarts without BRouter. Reclaim ~2 GB of segment volume on the flagship.
## Full restart (flagship)
```bash
gh workflow run cd-infra.yml -f restart_all=true
```
Restarts every flagship service. Does NOT touch the BRouter host.
## cd-brouter manual trigger
```bash
gh workflow run cd-brouter.yml
```
Useful to redeploy the BRouter host after token rotation or a config
change, without needing a real source change under
`infrastructure/brouter-host/`.