When a user rapidly moves waypoints in the Planner, BRouter's
thread-priority watchdog kills older in-flight requests to free a thread
for newer ones, surfacing to the user as "operation killed by
thread-priority-watchdog" errors.
Two fixes:
1. Bump the BRouter thread pool on the dedicated host from 4 to 16.
`BROUTER_THREADS` is now a Dockerfile ENV (default 4 for
flagship/CI/dev) overridden to 16 in `infrastructure/brouter-host/`.
The host has 32 GB of RAM and idle cores; 4 threads was a flagship-era
constraint that no longer applies.
2. Cancel the previous in-flight `/api/route` request when a new one
starts. `use-routing.ts` now keeps an `AbortController` in a ref,
aborts it at the top of each `computeRoute`, and bails quietly on
`AbortError` so the newer call drives state.
Together these eliminate both the "real" contention (at the BRouter
level) and the wasted work (BRouter computing a route the client no
longer cares about).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three fixes after the first cd-brouter run on the dedicated host:
1. **BRouter 1.7.8 → 1.7.9** (`docker/brouter/Dockerfile`). Planet RD5
segments on brouter.de are now version 11; 1.7.8's `lookups.dat`
is v10, causing `lookup version mismatch (old rd5?)
lookups.dat=10 E10_N45.rd5=11` on every route request.
2. **cd-brouter.yml: docker login to ghcr.io before pull**.
ghcr.io/trails-cool/brouter is private, and the dedicated host's
Docker daemon isn't logged in by default. Extract DEPLOY_GHCR_TOKEN
from SOPS at runner side, pass to the SSH step via envs, and
`docker login` before `docker compose pull`. Credential is
`::add-mask::`-ed so it doesn't show in logs.
3. **Drop custom healthcheck** on the brouter service. The image
strips wget/curl post-build, and /bin/sh in the base doesn't
support /dev/tcp, so there's no in-image way to do an HTTP probe.
Real health is observed via Caddy's upstream 502 behavior on
outage and the Planner-side `brouter_request_duration_seconds`
metric. caddy's `depends_on` drops from service_healthy to
service_started.
End-to-end verified on the dedicated host after applying the compose
fix manually:
- Caddy enforces auth: 403 without header, proxies with.
- BRouter 1.7.9 will resolve the segment-version error once the image
is rebuilt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands sections 6 and 8 of relocate-brouter-to-dedicated-host on top
of the host compose in #291.
## Observability (section 6)
- **Prometheus**: new `brouter-cadvisor` job scraping
`10.0.1.10:8080` over the vSwitch, labeled `host="brouter"`.
- **cAdvisor sidecar** on the dedicated host's compose
(`--docker_only --whitelisted_container_labels=trails.cool.service`)
so metrics only cover trails containers, never the operator's
unrelated workloads on the shared box.
- **Promtail sidecar** on the dedicated host, Docker SD with relabel-
drop on missing `trails.cool.service` label, pushing to flagship
Loki at `http://10.0.0.2:3100/loki/api/v1/push`.
- **Flagship compose**: Loki now publishes port 3100 on the vSwitch
IP only (10.0.0.2:3100) — Hetzner Cloud firewall blocks it from
the public internet.
- **Grafana dashboard**: `brouter.json` — scrape up/down, request
rate (from Planner-side `brouter_request_duration_seconds`),
p50/p95/p99, container memory/CPU, Loki logs panel.
- **Alert**: `brouter-scrape-down` fires on
`up{job="brouter-cadvisor"} < 1 for 2m`; `noDataState: Alerting`
so a total scrape failure still pages.
Operator needs one UFW rule on the dedicated host for the cAdvisor
port — documented in `infrastructure/brouter-host/README.md`.
## Documentation (section 8)
- `CLAUDE.md` — hosts table + updated deployment table with SSH
targets per workflow; BRouter host SSH is `-p 2232 trails@...`,
different key.
- `docs/architecture.md` — Hosting section rewritten to cover both
hosts, vSwitch boundary, and the observability-scoping rationale
for the shared dedicated host.
- `docs/deployment.md` (new) — full operator runbook: host layout,
first-time BRouter provisioning, SOPS rotation (including the
macOS `SOPS_AGE_KEY_FILE` gotcha), cutover procedure with
rollback, manual workflow triggers.
Task 8.4 (infrastructure/README.md) skipped: that file doesn't
exist and the ground is covered by brouter-host/README.md +
docs/deployment.md.
## Validation
- `docker compose config` on both the flagship and brouter-host
compose files — both validate.
- `pnpm typecheck`, `pnpm lint`, `pnpm test` — all clean (full
turbo cache hits; no code changes in this commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lands sections 3-5 of the relocate-brouter-to-dedicated-host change:
everything needed to run BRouter on the dedicated Hetzner Robot host
and have the Planner talk to it with the shared-secret header. Does
NOT flip the cutover — the flagship BRouter stays warm during soak.
## BRouter host compose (section 3)
New `infrastructure/brouter-host/` — a standalone compose project that
runs as the `trails` user on `ullrich.is`:
- `docker-compose.yml` — brouter + caddy sidecar. BRouter has no host
port; caddy binds only to `10.0.1.10:17777` (vSwitch IP). Every
service explicitly overrides the host's default Loki logging driver
to `json-file` so logs don't leak to the operator's personal Loki.
- `Caddyfile` — single-purpose reverse proxy that requires
`X-BRouter-Auth: ${BROUTER_AUTH_TOKEN}` on every request. `auto_https
off` (vSwitch-only); default access log format omits request
headers, so the token is never written to disk.
- `download-segments.sh` — crawls brouter.de, pulls planet-wide RD5
tiles via `wget -N` (incremental). Idempotent, safe to cron.
- `README.md` — one-shot provisioning + token rotation + rollback
notes.
`docker/brouter/Dockerfile` is patched to honor `JAVA_OPTS` (was
hardcoded `-Xmx1024M` in CMD). Default keeps the flagship's current
heap; compose on the dedicated host overrides to `-Xmx8g` for planet
scale on a 32 GB box.
## Planner shared-secret header (section 4)
`apps/planner/app/lib/brouter.ts`:
- Module-level guard: throws at startup in production if
`BROUTER_AUTH_TOKEN` is unset.
- `authHeaders()` helper (reads env at call time, so tests can
`vi.stubEnv` without module reset).
- Header attached on both `computeRoute` (per-segment) and
`computeSegmentGpx`.
3 new unit tests cover header attachment + the no-token path.
`infrastructure/docker-compose.yml` passes `BROUTER_AUTH_TOKEN` to
the Planner service, and makes `BROUTER_URL` overridable via SOPS so
the cutover is a one-variable flip.
## cd-brouter workflow (section 5)
Rewritten to deploy to the dedicated host:
- SSH as `trails@${BROUTER_DEPLOY_HOST}` on port
`${BROUTER_DEPLOY_SSH_PORT}` (2232) using
`${BROUTER_DEPLOY_SSH_KEY}`.
- Decrypts SOPS, extracts ONLY `BROUTER_AUTH_TOKEN` into a `.env`
file, scp'd alongside the compose project.
- `paths:` trigger now includes `infrastructure/brouter-host/**`.
- Segment download is NOT run here — first-time seed is a manual
operator step (multi-hour). Routine re-runs are cron-able on the
dedicated host.
- Grafana annotation step preserved (reaches flagship Grafana as
before).
## What's NOT here
- `brouter:` service on the flagship is intentionally left in place
(removed in section 7.5 after the 48 h soak window post-cutover).
- Observability (section 6) — Prometheus scrape + Loki shipping from
the dedicated host — comes in a follow-up PR.
- Cutover itself (section 7) — flip `BROUTER_URL`, verify, remove the
flagship brouter — is an operator action gated on first-time
provisioning + smoke testing.
## Verification
`pnpm typecheck && pnpm lint && pnpm test` all clean; planner build
passes (the CI regression from #286 was fixed in #290).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
