Add Hetzner vSwitch network for BRouter relocation
Adds an OpenSpec change scoping the relocation of BRouter from the co-located flagship (cx23, 4 GB RAM, 40 GB SSD, Europe segments only) to a dedicated Hetzner Robot host in the same datacenter, with private connectivity over Hetzner vSwitch #80672 (VLAN 4000). This first PR only lays the network prerequisite: - Terraform: a Hetzner Cloud Network (10.0.0.0/16) with a cloud subnet (10.0.0.0/24) hosting the flagship at 10.0.0.2, and a vSwitch subnet (10.0.1.0/24) bridged to Robot VLAN 4000. The dedicated host's VLAN sub-interface (10.0.1.10 on enp4s0.4000) is configured out-of-band via netplan and is not Terraform-managed. - lifecycle { ignore_changes = [user_data] } on the flagship server to prevent the Hetzner provider's post-1.45 user_data hash drift from triggering a spurious full-server replacement on unrelated applies. - OpenSpec change with proposal, design, specs (delta for brouter-integration / infrastructure / observability / security-hardening), and tasks; Section 1 (pre-flight) is checked off with operator notes. Verification: ping both directions across the vSwitch is 0% loss, sub-ms latency; dedicated host's VLAN config persists across reboot (verified ~60 s to restore private reachability). Follow-up PRs will land the BRouter host compose project, Planner shared-secret header, CD workflow retarget, observability wiring, and the cutover. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
566a322dbb
commit
102a744e67
10 changed files with 465 additions and 0 deletions
|
|
@ -54,6 +54,14 @@ resource "hcloud_server" "trails" {
|
|||
SSHD
|
||||
systemctl reload ssh
|
||||
EOF
|
||||
|
||||
# Hetzner Cloud provider's user_data hash algorithm changed in 1.45+,
|
||||
# causing spurious replacement diffs on every plan even when the script
|
||||
# is unchanged. user_data only runs on first boot anyway — changes after
|
||||
# provisioning have no effect — so we pin the attribute to ignore_changes.
|
||||
lifecycle {
|
||||
ignore_changes = [user_data]
|
||||
}
|
||||
}
|
||||
|
||||
resource "hcloud_firewall" "trails" {
|
||||
|
|
@ -86,6 +94,55 @@ resource "hcloud_firewall_attachment" "trails" {
|
|||
server_ids = [hcloud_server.trails.id]
|
||||
}
|
||||
|
||||
# Private network bridging the flagship (Hetzner Cloud) to the dedicated
|
||||
# BRouter host (Hetzner Robot) via vSwitch. Both sides must be in the same
|
||||
# location (fsn1 / eu-central).
|
||||
#
|
||||
# Hetzner requires Cloud servers to attach to a regular "cloud" subnet;
|
||||
# the "vswitch" subnet is VLAN-only and cannot host Cloud NICs. The two
|
||||
# subnets live side by side in the same network and Hetzner routes between
|
||||
# them automatically.
|
||||
#
|
||||
# 10.0.0.0/16 — overall private network
|
||||
# 10.0.0.1 — network gateway (reserved by Hetzner; not usable)
|
||||
# 10.0.0.0/24 — cloud subnet (hosts Cloud server NICs)
|
||||
# 10.0.0.2 — flagship (assigned below)
|
||||
# 10.0.1.0/24 — vSwitch subnet (bridged to Robot VLAN)
|
||||
# 10.0.1.10 — dedicated BRouter host (configured on the Robot side
|
||||
# as a VLAN sub-interface, not managed by Terraform)
|
||||
|
||||
resource "hcloud_network" "private" {
|
||||
name = "trails-cool"
|
||||
ip_range = "10.0.0.0/16"
|
||||
|
||||
labels = {
|
||||
project = "trails-cool"
|
||||
}
|
||||
}
|
||||
|
||||
resource "hcloud_network_subnet" "cloud" {
|
||||
network_id = hcloud_network.private.id
|
||||
type = "cloud"
|
||||
network_zone = "eu-central"
|
||||
ip_range = "10.0.0.0/24"
|
||||
}
|
||||
|
||||
resource "hcloud_network_subnet" "vswitch" {
|
||||
network_id = hcloud_network.private.id
|
||||
type = "vswitch"
|
||||
network_zone = "eu-central"
|
||||
ip_range = "10.0.1.0/24"
|
||||
vswitch_id = var.vswitch_id
|
||||
}
|
||||
|
||||
resource "hcloud_server_network" "trails" {
|
||||
server_id = hcloud_server.trails.id
|
||||
network_id = hcloud_network.private.id
|
||||
ip = "10.0.0.2"
|
||||
|
||||
depends_on = [hcloud_network_subnet.cloud]
|
||||
}
|
||||
|
||||
# Reverse DNS
|
||||
|
||||
resource "hcloud_rdns" "trails_ipv4" {
|
||||
|
|
@ -361,3 +418,8 @@ output "domain" {
|
|||
output "planner_domain" {
|
||||
value = "planner.trails.cool"
|
||||
}
|
||||
|
||||
output "flagship_private_ip" {
|
||||
value = hcloud_server_network.trails.ip
|
||||
description = "Private IP of the flagship on the vSwitch network (reachable from the dedicated BRouter host)"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -7,4 +7,9 @@ variable "hcloud_token" {
|
|||
variable "ssh_public_key" {
|
||||
description = "SSH public key for server access"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "vswitch_id" {
|
||||
description = "Hetzner Robot vSwitch ID bridging the flagship to the dedicated BRouter host (see robot.hetzner.com/vswitch)"
|
||||
type = number
|
||||
}
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
schema: spec-driven
|
||||
created: 2026-04-23
|
||||
122
openspec/changes/relocate-brouter-to-dedicated-host/design.md
Normal file
122
openspec/changes/relocate-brouter-to-dedicated-host/design.md
Normal file
|
|
@ -0,0 +1,122 @@
|
|||
## Context
|
||||
|
||||
The flagship instance today runs on a single Hetzner Cloud cx23 (2 vCPU / 4 GB RAM / 40 GB SSD). `infrastructure/docker-compose.yml` brings up Journal, Planner, Postgres+PostGIS, BRouter, Caddy, Prometheus, Loki, Promtail, Grafana, and exporters in one compose project. BRouter currently serves Europe-only RD5 tiles (see `cd-brouter.yml` tile list) and shares the 4 GB memory budget with Postgres, the Loki stack, and the Node runtimes. The Planner proxies all BRouter requests; clients never talk to BRouter directly (`apps/planner/app/lib/brouter.ts`).
|
||||
|
||||
A second Hetzner host in the same datacenter is already provisioned. It is a general-purpose self-hosted box (3 TB RAID, 32 GB RAM) running unrelated workloads. We own a non-root `trails` user there with docker-group membership; we do not own root, sudo, the firewall, or the host's own observability agents. The host and the flagship Cloud box can be joined on a Hetzner vSwitch.
|
||||
|
||||
Other relevant context:
|
||||
- `cd-brouter.yml` currently SSHes as `root` into `DEPLOY_HOST` and runs `docker compose up -d brouter` against the shared compose project.
|
||||
- Secrets are SOPS-encrypted in `infrastructure/secrets.app.env` / `infrastructure/secrets.infra.env`, with `AGE_SECRET_KEY`, `DEPLOY_SSH_KEY`, `DEPLOY_HOST` as the only GitHub Actions secrets today.
|
||||
- The Planner reads `BROUTER_URL` from a single env var (`apps/planner/app/lib/brouter.ts:1`) and makes unauthenticated HTTP requests. There's no existing auth path.
|
||||
- `brouter_request_duration_seconds` already tracks BRouter latency from the Planner side (see `observability` spec) and surfaces in Grafana.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- BRouter runs on the dedicated host with planet-wide RD5 coverage and a memory budget sized for global routing.
|
||||
- All BRouter traffic flows over a private Hetzner vSwitch; no public port for BRouter.
|
||||
- Planner → BRouter requests are authenticated with a shared secret so that a misconfigured vSwitch or firewall mistake doesn't silently expose the service.
|
||||
- Deployment works end-to-end as the non-root `trails` user with only docker-group privileges.
|
||||
- BRouter container metrics and logs reach the flagship instance's Grafana without polluting trails.cool observability with the shared host's other workloads.
|
||||
- Cutover is reversible for 48 h via `BROUTER_URL` rollback.
|
||||
|
||||
**Non-Goals:**
|
||||
- Failover / multi-upstream BRouter. A separate change (`brouter-failover-or-degradation`, deferred) handles that.
|
||||
- Touching the shared host's firewall, SSH configuration, or other tenants' services. Operator confirms the vSwitch interface is reachable and the host firewall allows the flagship's private IP to hit the BRouter container's published port; no further host-level config is in scope for this change.
|
||||
- Migrating other trails.cool services off the flagship box.
|
||||
- Automated segment updates. Manual re-run of the segment script is fine for now.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Transport: Hetzner vSwitch, same DC
|
||||
|
||||
**Decision:** Join the flagship and the dedicated host on a Hetzner vSwitch. BRouter's Caddy sidecar binds the published port to the vSwitch IP only.
|
||||
|
||||
**Alternatives:**
|
||||
- *WireGuard overlay.* Works without vendor lock-in; adds a daemon, key rotation, and another moving part on a host we don't fully own. vSwitch is simpler when both endpoints are Hetzner in the same DC.
|
||||
- *Public port + firewall allowlist + TLS.* Would require managing certs on a host we don't run Caddy/Let's Encrypt on natively, plus IP allowlist drift if the flagship ever changes IPs.
|
||||
|
||||
**Rationale:** vSwitch is Hetzner-native, low-latency (sub-ms, same DC), encrypted at the hypervisor level, and removes public exposure entirely. The shared-secret header (below) is defense-in-depth against vSwitch misconfiguration.
|
||||
|
||||
### 2. Auth: shared-secret header enforced by a Caddy sidecar
|
||||
|
||||
**Decision:** Run a thin Caddy container on the dedicated host in front of BRouter. Caddy requires `X-BRouter-Auth: <token>` on every request; missing/wrong token → 403 before BRouter ever sees the request. The Planner reads `BROUTER_AUTH_TOKEN` from env and sets the header on every fetch in `apps/planner/app/lib/brouter.ts`.
|
||||
|
||||
**Alternatives:**
|
||||
- *mTLS.* Stronger but requires cert rotation and adds operational weight for a single consumer.
|
||||
- *Host-based IP allowlist alone.* Relies entirely on network config; a vSwitch routing mistake becomes an auth bypass. Layered defense is cheap.
|
||||
- *BRouter-native auth.* BRouter has no auth plugin; forking or patching Java is out of proportion.
|
||||
|
||||
**Rationale:** One 32-byte random token in SOPS, one header on every proxy request, one `@require_header` Caddy matcher. Minimal code and config surface.
|
||||
|
||||
### 3. Deploy user: `trails` with docker group, no sudo
|
||||
|
||||
**Decision:** The deploy workflow SSHes as `trails@<brouter-host>` using a dedicated key. The workflow runs `docker compose` inside `~trails/brouter/`, pulls `ghcr.io/trails-cool/brouter:latest`, and restarts. All files under `~trails/brouter/` are owned by `trails:trails`. Segment downloads also run as `trails` via a script in the same directory.
|
||||
|
||||
**Alternatives:**
|
||||
- *Root SSH.* Matches the current pattern but requires privileges we don't have on the shared host.
|
||||
- *Systemd user service.* Adds another deployment model inconsistent with the rest of the fleet.
|
||||
|
||||
**Rationale:** Docker-group membership is sufficient to run compose. No sudo needed. Keeps the blast radius contained to the `trails` account — consistent with the shared-host constraint.
|
||||
|
||||
### 4. Segment coverage: global from day one
|
||||
|
||||
**Decision:** Download the full planet RD5 set (~60–80 GB) to `~trails/brouter/segments/` on first deploy. Script downloads per-tile, idempotent, resumes where it left off.
|
||||
|
||||
**Alternatives:**
|
||||
- *Europe first, global later.* Lower cutover risk but requires a second migration.
|
||||
- *On-demand segment download.* BRouter supports only mounted segments; on-demand requires a fork.
|
||||
|
||||
**Rationale:** Disk is abundant (3 TB available, segments use <3%). Global from the start matches the motivation of the move and avoids a second cutover.
|
||||
|
||||
### 5. Memory: `-Xmx8g` JVM heap on a 32 GB host
|
||||
|
||||
**Decision:** Set `JAVA_OPTS=-Xmx8g` (or the equivalent `brouter.sh` arg) on the BRouter container. Leave the rest for Linux page cache, which matters more for cold-segment reads than heap.
|
||||
|
||||
**Alternatives:**
|
||||
- *`-Xmx16g`.* Diminishing returns; segment LRU cache at 8 GB already holds a large working set. Page cache is a better spend for the remaining RAM.
|
||||
- *`-Xmx4g`.* Fine for low concurrency; leaves no headroom for bursts across disjoint planet regions.
|
||||
|
||||
**Rationale:** Matches observed sizing of public planet-scale BRouter instances. Revisit if `brouter_request_duration_seconds` p95 degrades under load.
|
||||
|
||||
### 6. Observability: scrape the container, not the host
|
||||
|
||||
**Decision:** From the flagship's Prometheus, scrape the BRouter container's exposed metrics port over vSwitch (either BRouter's own `/metrics` if enabled, or a sidecar JMX exporter). Also scrape a cAdvisor instance running on the dedicated host, filtered by container label to BRouter only. Loki collects logs via a Promtail/Alloy container running as `trails` tailing the BRouter container's stdout only.
|
||||
|
||||
**Alternatives:**
|
||||
- *Shared host `node_exporter`.* Would mix trails.cool metrics with unrelated workloads. Not ours to report on, and noisy.
|
||||
- *Run Prometheus locally on the dedicated host.* Splits observability into two Grafanas. The CLAUDE.md rule is "observability continues to live in trails internal grafana."
|
||||
- *Ship metrics via remote_write only.* Adds complexity without benefit given the vSwitch is already low-latency.
|
||||
|
||||
**Rationale:** Scrape over vSwitch is the simplest path that keeps the data in the existing Grafana. Scoping to the BRouter container (not the host) respects the shared-host boundary.
|
||||
|
||||
### 7. Cutover: parallel stand-up with env-var flip
|
||||
|
||||
**Decision:**
|
||||
1. Stand up BRouter on the dedicated host, seed segments, validate with a curl-over-vSwitch + auth-header test.
|
||||
2. Add `BROUTER_AUTH_TOKEN` to SOPS; deploy Planner with the new token but still pointing at `http://brouter:17777` (old upstream). Verify the token is wired but unused.
|
||||
3. Flip `BROUTER_URL` in the Planner env to the new vSwitch URL. Deploy Planner. Monitor `brouter_request_duration_seconds` for error rate + latency.
|
||||
4. Leave the old BRouter container running on the flagship for 48 h; document the rollback as a single `BROUTER_URL` change.
|
||||
5. After 48 h of clean metrics, remove the `brouter` service from `infrastructure/docker-compose.yml`, delete `/opt/trails-cool/segments/`, and update `cd-infra.yml` to skip BRouter.
|
||||
|
||||
**Rationale:** Single-variable flip with a warm rollback. `BROUTER_URL` is already an env var; no code branching needed.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- **vSwitch misconfig leaves BRouter unreachable.** → The shared-secret header means the service can be public-IP-reachable as a fallback during a vSwitch outage without auth regressions; confirm both hosts ping each other over vSwitch before flipping `BROUTER_URL`.
|
||||
- **Shared host's other workloads interfere with BRouter memory/CPU.** → Pin `-Xmx8g` so JVM behavior is predictable. Monitor host-level memory pressure via an alert on BRouter OOM log lines (shipped via Promtail). Operator can escalate to the shared-host owner.
|
||||
- **Segment download takes hours; first deploy blocks.** → Run the segment script out-of-band from the CI deploy (manual one-shot `ssh trails@host '~/brouter/download-segments.sh'`). Don't let the workflow time out on a 60 GB download.
|
||||
- **Auth token leaks in logs or PR reviews.** → Token only lives in SOPS + GitHub Actions secrets. Caddy access logs mask `X-BRouter-Auth` via a `log.filter`. Rotation is a redeploy of Planner + Caddy with a new token.
|
||||
- **Planner latency increases due to cross-host hop.** → Same-DC vSwitch is ~0.2–0.5 ms extra; p50 impact negligible. If p99 regresses, investigate segment cache hit ratio (likely root cause is global vs. Europe, not the hop).
|
||||
- **Observability from a host we don't own is fragile** — the shared host's Docker daemon, vSwitch interface, or disk filling from other tenants can all break us silently. → Alert on `up{job="brouter"}` = 0 for 2 minutes, plus a synthetic check from Planner side (existing `brouter_request_duration_seconds` error rate).
|
||||
- **Only one consumer of the token for now, but token rotation is still a two-place update** (SOPS + Planner redeploy + Caddy redeploy). → Document rotation in runbook; not worth automating yet.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
See cutover in Decision 7. Rollback: revert the `BROUTER_URL` change in SOPS, redeploy Planner. Old container remains warm for 48 h.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Does the dedicated host's network config actually expose a vSwitch interface today, or does the operator need to enable it on the Hetzner side first? (Operator to confirm before implementation starts.)
|
||||
- Does BRouter expose Prometheus metrics in the current image, or do we need a JMX exporter sidecar? If JMX is needed, add a sub-task.
|
||||
- Segment update cadence (brouter.de publishes weekly). Start manual; a later change can add a cron on the dedicated host if desired.
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
## Why
|
||||
|
||||
BRouter currently runs co-located with the Journal, Planner, PostgreSQL, and the observability stack on a single Hetzner Cloud box (cx23: 2 vCPU / 4 GB RAM / 40 GB SSD). The 40 GB disk and 4 GB RAM budget cap us at a Europe-only segment set; global routing needs ~60–80 GB of RD5 files and a JVM heap that doesn't compete with Postgres and Loki for memory. A larger self-hosted server in the same Hetzner datacenter has 3 TB of RAID storage and 32 GB of RAM available with headroom to spare, and co-locating stays on the private Hetzner network for low latency. Moving BRouter there unlocks global routing without contending for resources on the primary host.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Run BRouter on a second Hetzner host in the same datacenter, owned by a non-root `trails` user that is a member of the `docker` group (no sudo).
|
||||
- Expose BRouter only on a private network interface (Hetzner vSwitch) shared with the primary host; no public ingress. The existing host firewall rejects everything else.
|
||||
- Front the BRouter container with a Caddy sidecar that enforces a shared-secret header (`X-BRouter-Auth: <token>`). The Planner backend adds the header when proxying `/api/route`. Requests without the header are rejected with 403.
|
||||
- Extend BRouter coverage from the current Europe RD5 set to the full planet (~60–80 GB on disk). The dedicated host's disk accommodates this comfortably.
|
||||
- Size the BRouter JVM for planet-scale traffic: `-Xmx8g` heap on a host with 32 GB total RAM, leaving generous page cache for segment files.
|
||||
- **BREAKING** for operators: `cd-brouter.yml` targets a new host/SSH identity (`trails@<brouter-host>`) instead of `root@<main-host>`. The segment-download logic moves with the workflow, and the segment tile list expands to global.
|
||||
- Update the Planner's `BROUTER_URL` to the new host's private vSwitch address and add `BROUTER_AUTH_TOKEN` as a new SOPS-managed secret. The Planner sends the token on every request.
|
||||
- Scrape BRouter-specific metrics (cAdvisor filtered to the BRouter container, JVM exporter if available) and tail BRouter container logs from the primary host's Prometheus/Loki over the vSwitch. Do **not** scrape the shared host's `node_exporter` — its other self-hosted workloads are out of scope for trails.cool observability.
|
||||
- Remove the BRouter service from the primary host's `infrastructure/docker-compose.yml` once cutover is complete; stop shipping BRouter images to the primary via `cd-infra.yml`.
|
||||
- Document the new host in `CLAUDE.md` and `docs/architecture.md` as a second deployment target.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
|
||||
_None._ This change relocates an existing capability rather than introducing a new user-facing one. All new requirements extend existing specs.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
- `brouter-integration`: deployment target changes from co-located Docker Compose service to a remote host reached over a private network; adds a shared-secret auth requirement on the proxy hop; extends segment coverage from Europe to global.
|
||||
- `infrastructure`: introduces a second Hetzner host on a vSwitch, a non-root `trails` deploy user, a global segment management workflow, and split CD targets for app vs. BRouter; updates the "BRouter segment management" and "CD pipeline" requirements accordingly.
|
||||
- `observability`: adds scraping/log shipping for the remote BRouter host over the private network, scoped to BRouter containers only.
|
||||
- `security-hardening`: adds the shared-secret auth requirement for BRouter and the private-network-only exposure rule.
|
||||
|
||||
## Impact
|
||||
|
||||
- **Code**: `apps/planner/app/lib/brouter.ts` (add `X-BRouter-Auth` header, read `BROUTER_AUTH_TOKEN`); planner server env wiring.
|
||||
- **Infrastructure**: `infrastructure/docker-compose.yml` (remove brouter service + segments volume); new `infrastructure/brouter-host/` directory with the remote compose file, Caddy sidecar config, and segment-download script.
|
||||
- **CI/CD**: `.github/workflows/cd-brouter.yml` (retarget host, new secrets `BROUTER_DEPLOY_HOST` / `BROUTER_DEPLOY_SSH_KEY` / `BROUTER_AUTH_TOKEN`); `cd-infra.yml` no longer touches BRouter.
|
||||
- **Secrets**: new entries in `infrastructure/secrets.infra.env` (SOPS) for `BROUTER_AUTH_TOKEN`; new GitHub Actions secrets for the new host's SSH key and hostname.
|
||||
- **Observability**: update `infrastructure/prometheus/prometheus.yml` with a new scrape target over the vSwitch; update `infrastructure/promtail/` to pull logs from the remote host (via Docker socket proxy or a shipped-from agent).
|
||||
- **Documentation**: `CLAUDE.md` (note the second host); `docs/architecture.md` (topology diagram and failure modes); `docs/deployment.md` if present.
|
||||
- **Dependencies**: no npm/package changes. Requires Hetzner vSwitch configured between the two hosts (operator action).
|
||||
- **Operational risk**: BRouter becomes a second SPOF reachable only over vSwitch. Monitored via existing `brouter_request_duration_seconds` histogram in the Planner; cutover includes a 48 h rollback window with the old container kept warm on the primary host.
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: BRouter API proxy
|
||||
The Planner backend SHALL proxy all BRouter API calls. Clients SHALL NOT communicate with BRouter directly. The proxy SHALL attach a `X-BRouter-Auth: <token>` header to every upstream request using the `BROUTER_AUTH_TOKEN` environment variable.
|
||||
|
||||
#### Scenario: Proxied route request
|
||||
- **WHEN** the routing host client requests a route computation
|
||||
- **THEN** the Planner backend forwards the request to BRouter over the configured upstream URL with the `X-BRouter-Auth` header set, applies rate limiting, and returns the response
|
||||
|
||||
#### Scenario: Missing auth token at startup
|
||||
- **WHEN** the Planner starts in production without `BROUTER_AUTH_TOKEN` set
|
||||
- **THEN** the Planner logs a fatal error and refuses to start
|
||||
|
||||
### Requirement: BRouter Docker deployment
|
||||
BRouter SHALL run as a Docker container on a dedicated Hetzner host reached over a private Hetzner vSwitch, with planet-wide RD5 segments mounted as a volume. The BRouter container SHALL NOT be exposed on any public network interface.
|
||||
|
||||
#### Scenario: BRouter container starts
|
||||
- **WHEN** the `docker compose up -d` command runs in `~trails/brouter/` on the dedicated host
|
||||
- **THEN** the BRouter container is reachable only on the vSwitch-bound IP and can compute routes using the mounted planet-wide RD5 segments
|
||||
|
||||
#### Scenario: Public network isolation
|
||||
- **WHEN** a request is sent to the dedicated host's public IP on the BRouter port
|
||||
- **THEN** the request is refused at the host firewall or times out; BRouter does not respond
|
||||
|
||||
#### Scenario: JVM memory sizing
|
||||
- **WHEN** the BRouter container starts
|
||||
- **THEN** the JVM is launched with `-Xmx8g` (or equivalent) so that the heap does not exceed 8 GB on the 32 GB host
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Shared-secret auth on BRouter
|
||||
The BRouter deployment SHALL require a shared-secret header on every request. Requests without a valid `X-BRouter-Auth` header SHALL be rejected before reaching the BRouter process.
|
||||
|
||||
#### Scenario: Valid token
|
||||
- **WHEN** a request arrives at the BRouter host with the correct `X-BRouter-Auth` header
|
||||
- **THEN** the Caddy sidecar forwards the request to BRouter and returns its response
|
||||
|
||||
#### Scenario: Missing or wrong token
|
||||
- **WHEN** a request arrives without an `X-BRouter-Auth` header or with an incorrect value
|
||||
- **THEN** the Caddy sidecar responds with HTTP 403 and does not forward the request to BRouter
|
||||
|
||||
#### Scenario: Token not logged
|
||||
- **WHEN** Caddy emits an access log line for a BRouter request
|
||||
- **THEN** the `X-BRouter-Auth` header value is redacted or omitted from the log line
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: BRouter segment management
|
||||
The infrastructure SHALL support downloading and updating planet-wide RD5 segments from brouter.de to the dedicated BRouter host. Segment files SHALL live under `~trails/brouter/segments/` on the dedicated host and SHALL be owned by the `trails` user.
|
||||
|
||||
#### Scenario: Download segments
|
||||
- **WHEN** the segment download script runs on the dedicated host as the `trails` user
|
||||
- **THEN** all planet-wide RD5 files referenced by the tile list are downloaded to `~trails/brouter/segments/`, skipping files that already exist
|
||||
|
||||
#### Scenario: Segment update
|
||||
- **WHEN** an operator re-runs the segment download script
|
||||
- **THEN** outdated or missing RD5 files are re-fetched from brouter.de and the BRouter container is restarted
|
||||
|
||||
### Requirement: CI/CD pipeline
|
||||
GitHub Actions SHALL use separate workflows for app deployment, infrastructure deployment, and BRouter deployment, with secrets decrypted from a SOPS-encrypted file.
|
||||
|
||||
#### Scenario: App deployment
|
||||
- **WHEN** code changes are pushed to main in apps/ or packages/
|
||||
- **THEN** the cd-apps workflow builds Docker images, pushes to ghcr.io, and deploys app containers to the flagship host
|
||||
|
||||
#### Scenario: Infrastructure deployment
|
||||
- **WHEN** changes are pushed to main in infrastructure/
|
||||
- **THEN** the cd-infra workflow copies configs and restarts infrastructure services on the flagship host without rebuilding app images and without touching the BRouter host
|
||||
|
||||
#### Scenario: BRouter deployment
|
||||
- **WHEN** changes are pushed to main in docker/brouter/ or the BRouter host compose config
|
||||
- **THEN** the cd-brouter workflow SSHes as the `trails` user into the dedicated BRouter host using `BROUTER_DEPLOY_HOST` / `BROUTER_DEPLOY_SSH_KEY` and runs `docker compose up -d` in `~trails/brouter/`
|
||||
|
||||
#### Scenario: Secret decryption at deploy time
|
||||
- **WHEN** any CD workflow runs
|
||||
- **THEN** the SOPS-encrypted secrets file is decrypted and provided to docker-compose as an env file
|
||||
|
||||
#### Scenario: Gitleaks scan
|
||||
- **WHEN** a PR is opened
|
||||
- **THEN** gitleaks scans for committed secrets
|
||||
|
||||
#### Scenario: Dependency audit
|
||||
- **WHEN** CI runs
|
||||
- **THEN** pnpm audit checks for high/critical vulnerabilities
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Private network between flagship and BRouter hosts
|
||||
The flagship host and the dedicated BRouter host SHALL be joined on a Hetzner vSwitch in the same datacenter. All traffic between Planner and BRouter SHALL traverse this private network.
|
||||
|
||||
#### Scenario: vSwitch reachability
|
||||
- **WHEN** the flagship host issues a request to the BRouter host's vSwitch IP on the BRouter service port with a valid `X-BRouter-Auth` header
|
||||
- **THEN** the request succeeds over the private network without traversing the public internet
|
||||
|
||||
#### Scenario: No public BRouter exposure
|
||||
- **WHEN** Hetzner Cloud firewall rules or equivalent host firewall rules are inspected
|
||||
- **THEN** no rule allows inbound traffic to the BRouter service port from any public IP
|
||||
|
||||
### Requirement: Non-root deploy user on the BRouter host
|
||||
The BRouter host SHALL be administered by the trails.cool project through a non-root `trails` user that is a member of the `docker` group. The CD workflow SHALL NOT require sudo or root SSH access on this host.
|
||||
|
||||
#### Scenario: Deploy with trails user
|
||||
- **WHEN** the cd-brouter workflow connects to the BRouter host
|
||||
- **THEN** it authenticates as `trails` and successfully runs `docker compose` commands without invoking sudo
|
||||
|
||||
#### Scenario: Scoped ownership
|
||||
- **WHEN** files are created by the deploy or segment-download scripts
|
||||
- **THEN** they live under `~trails/brouter/` and are owned by `trails:trails`
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
## ADDED Requirements
|
||||
|
||||
### Requirement: Remote BRouter metrics
|
||||
Prometheus on the flagship host SHALL scrape BRouter-specific metrics from the dedicated BRouter host over the vSwitch. Scraping SHALL be scoped to BRouter containers only and SHALL NOT collect host-level metrics from the dedicated host's other (non-trails.cool) workloads.
|
||||
|
||||
#### Scenario: BRouter container metrics collected
|
||||
- **WHEN** Prometheus scrapes the dedicated BRouter host
|
||||
- **THEN** metrics from the BRouter container (and, if enabled, the Caddy sidecar) are collected via cAdvisor filtered by container label, or via a BRouter/JMX metrics endpoint exposed on the vSwitch
|
||||
|
||||
#### Scenario: No shared-host noise
|
||||
- **WHEN** Prometheus is configured with the remote BRouter scrape target
|
||||
- **THEN** no configuration scrapes the dedicated host's `node_exporter` or metrics from containers other than BRouter and its sidecar
|
||||
|
||||
#### Scenario: BRouter scrape failure alert
|
||||
- **WHEN** the BRouter scrape target returns `up == 0` for more than 2 minutes
|
||||
- **THEN** Grafana fires an alert distinct from flagship-side alerts
|
||||
|
||||
### Requirement: Remote BRouter log shipping
|
||||
Loki on the flagship host SHALL receive BRouter container logs from the dedicated BRouter host via a Promtail or Alloy agent running on that host as the `trails` user. Log shipping SHALL be scoped to BRouter-related containers only.
|
||||
|
||||
#### Scenario: BRouter logs visible in Grafana
|
||||
- **WHEN** the BRouter container writes to stdout or stderr
|
||||
- **THEN** the log line is available in Grafana Explore via Loki with container and host labels identifying it as coming from the dedicated BRouter host
|
||||
|
||||
#### Scenario: Non-BRouter logs not shipped
|
||||
- **WHEN** a non-BRouter container on the dedicated host writes logs
|
||||
- **THEN** those logs are not ingested by the flagship Loki instance
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
## ADDED Requirements
|
||||
|
||||
### Requirement: BRouter access control
|
||||
The BRouter service SHALL be reachable only from the flagship host over a private Hetzner vSwitch, and every request SHALL carry a valid `X-BRouter-Auth` shared-secret header. The token SHALL be stored only in SOPS-encrypted secrets and GitHub Actions secrets, never committed in cleartext.
|
||||
|
||||
#### Scenario: Request from public internet
|
||||
- **WHEN** an attacker sends a request to the BRouter host's public IP on the BRouter service port
|
||||
- **THEN** the host firewall refuses the connection
|
||||
|
||||
#### Scenario: Request from vSwitch without token
|
||||
- **WHEN** a request arrives on the BRouter host over the vSwitch without a valid `X-BRouter-Auth` header
|
||||
- **THEN** the Caddy sidecar responds with HTTP 403 and BRouter never sees the request
|
||||
|
||||
#### Scenario: Token storage
|
||||
- **WHEN** `BROUTER_AUTH_TOKEN` is added or rotated
|
||||
- **THEN** the token is written only to `infrastructure/secrets.infra.env` (SOPS-encrypted) and to the GitHub Actions secret store, and is never committed in cleartext to the repository
|
||||
|
||||
#### Scenario: Token rotation
|
||||
- **WHEN** the token is rotated
|
||||
- **THEN** operators update SOPS, redeploy the Planner (new token in outbound header), and redeploy the Caddy sidecar (new token in the matcher), in that order, with zero downtime
|
||||
80
openspec/changes/relocate-brouter-to-dedicated-host/tasks.md
Normal file
80
openspec/changes/relocate-brouter-to-dedicated-host/tasks.md
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
## 1. Pre-flight (operator)
|
||||
|
||||
- [x] 1.1 Confirm Hetzner vSwitch is provisioned between the flagship cx23 and the dedicated host; record both vSwitch IPs
|
||||
- vSwitch #80672 (VLAN 4000, 1 TB), flagship 10.0.0.2 on `enp7s0`, dedicated 10.0.1.10 on `enp4s0.4000` (MTU 1400), Cloud Network 10.0.0.0/16 with cloud subnet 10.0.0.0/24 + vSwitch subnet 10.0.1.0/24 (Terraform)
|
||||
- Dedicated host VLAN persistence: `/etc/netplan/60-trails-vswitch.yaml` on Ubuntu 22.04; verified surviving a reboot (VLAN reachable 64 s after reboot, 0% packet loss steady-state)
|
||||
- Flagship side auto-configured by Hetzner cloud-init on attach (no manual OS config)
|
||||
- [x] 1.2 Confirm the dedicated host's firewall allows inbound TCP from the flagship vSwitch IP to the planned BRouter service port, and blocks that port from the public interface
|
||||
- UFW active with `INPUT` default-DROP. Allow rule: `17777/tcp on enp4s0.4000 from 10.0.0.2`. Public interface is blocked by default-deny (17777 not in the public allowlist: 2232/SSH, 80, 443).
|
||||
- [x] 1.3 Confirm the `trails` user exists on the dedicated host, is in the `docker` group, and can run `docker ps` without sudo
|
||||
- uid 1002, groups include 114(docker); `sudo -iu trails docker ps` returns container list (host runs many other self-hosted services, all user-owned)
|
||||
- [x] 1.4 Generate an SSH keypair for the deploy workflow (`BROUTER_DEPLOY_SSH_KEY`) and install the public key in `~trails/.ssh/authorized_keys`
|
||||
- ed25519 keypair at `~/.ssh/trails-brouter-deploy{,.pub}` on operator laptop; public key installed at `/home/trails/.ssh/authorized_keys` on `ullrich.is` (perms 0600/0700, owner trails:trails)
|
||||
- Verified: `ssh -i ~/.ssh/trails-brouter-deploy -p 2232 trails@ullrich.is 'docker ps'` succeeds with no password prompt
|
||||
- Note: SSH on `ullrich.is` listens on port **2232**, not 22. The CD workflow must use `-p 2232`. Add `BROUTER_DEPLOY_SSH_PORT=2232` as a GitHub Actions secret (or hard-code it in the workflow step).
|
||||
- Before merging: add the contents of `~/.ssh/trails-brouter-deploy` as GitHub Actions secret `BROUTER_DEPLOY_SSH_KEY`; then consider whether to keep or delete the local copy.
|
||||
- [x] 1.5 Verify the dedicated host's Docker daemon version supports the compose file features used in `infrastructure/docker-compose.yml`
|
||||
- Docker Engine 29.1.5 (API 1.52), Compose v5.0.1, storage driver overlay2 — all modern
|
||||
- **Gotcha**: default `LoggingDriver=loki` on this host (routes to user's personal Loki). Our BRouter compose must override `logging:` per service so our logs go to trails.cool's Loki (flagship) instead. See 6.3 for the two viable approaches: (a) direct Loki driver with `loki-url: http://10.0.0.2:3100/loki/api/v1/push`, or (b) json-file + our own Promtail sidecar.
|
||||
|
||||
## 2. Secrets and config
|
||||
|
||||
- [ ] 2.1 Generate a 32-byte random `BROUTER_AUTH_TOKEN`
|
||||
- [ ] 2.2 Add `BROUTER_AUTH_TOKEN` to `infrastructure/secrets.infra.env` (SOPS) and commit the re-encrypted file
|
||||
- [ ] 2.3 Add `BROUTER_AUTH_TOKEN` to `infrastructure/secrets.app.env` (SOPS) for the Planner
|
||||
- [ ] 2.4 Add GitHub Actions secrets: `BROUTER_DEPLOY_HOST`, `BROUTER_DEPLOY_SSH_KEY`
|
||||
- [ ] 2.5 Document the rotation runbook in `docs/deployment.md` (or equivalent)
|
||||
|
||||
## 3. BRouter host compose project
|
||||
|
||||
- [ ] 3.1 Create `infrastructure/brouter-host/docker-compose.yml` with services `brouter` (bound only to the internal Docker network) and `caddy` (published on the vSwitch IP, auth-enforcing)
|
||||
- [ ] 3.2 Create `infrastructure/brouter-host/Caddyfile` that requires `X-BRouter-Auth` equal to the configured token and forwards matching requests to `brouter:17777`; redact the header from access logs
|
||||
- [ ] 3.3 Set `JAVA_OPTS=-Xmx8g` (or equivalent BRouter env) on the `brouter` service
|
||||
- [ ] 3.4 Create `infrastructure/brouter-host/download-segments.sh` that fetches the planet RD5 tile list idempotently into `./segments/`
|
||||
- [ ] 3.5 Add a README in `infrastructure/brouter-host/` with one-shot provisioning notes (`git clone`, first segment download, first compose up)
|
||||
|
||||
## 4. Planner changes
|
||||
|
||||
- [ ] 4.1 Add `BROUTER_AUTH_TOKEN` env var to `apps/planner/app/lib/brouter.ts`; send `X-BRouter-Auth` on every fetch
|
||||
- [ ] 4.2 Fail the Planner startup with a clear error when `NODE_ENV=production` and `BROUTER_AUTH_TOKEN` is unset
|
||||
- [ ] 4.3 Update `infrastructure/docker-compose.yml` Planner service env to pass `BROUTER_AUTH_TOKEN` through from the SOPS env file
|
||||
- [ ] 4.4 Add a unit test covering the header-attachment path in `apps/planner/app/lib/brouter.ts`
|
||||
|
||||
## 5. CD workflow
|
||||
|
||||
- [ ] 5.1 Rewrite `.github/workflows/cd-brouter.yml` deploy job: SSH as `trails@${{ secrets.BROUTER_DEPLOY_HOST }}`, `cd ~trails/brouter`, `docker compose pull && docker compose up -d`
|
||||
- [ ] 5.2 Update workflow `paths:` trigger to include `infrastructure/brouter-host/**`
|
||||
- [ ] 5.3 Move the segment-download logic out of the workflow into the on-host `download-segments.sh`; workflow calls it but tolerates a long-running invocation (or skips on subsequent deploys if segments already present)
|
||||
- [ ] 5.4 Keep the Grafana annotation step, pointing at the flagship Grafana over its existing path
|
||||
- [ ] 5.5 Remove the `brouter:` service from `infrastructure/docker-compose.yml` on the flagship (deferred to cutover step 7.5)
|
||||
|
||||
## 6. Observability
|
||||
|
||||
- [ ] 6.1 Add a Prometheus scrape job in `infrastructure/prometheus/prometheus.yml` targeting the BRouter host's cAdvisor (or JMX exporter) on the vSwitch IP; label with `host="brouter"`
|
||||
- [ ] 6.2 Run cAdvisor on the dedicated host as part of `infrastructure/brouter-host/docker-compose.yml`, configured to report only BRouter-labeled containers
|
||||
- [ ] 6.3 Add a Promtail (or Alloy) service to `infrastructure/brouter-host/docker-compose.yml` tailing Docker logs for BRouter + Caddy sidecar only, pushing to the flagship Loki over vSwitch
|
||||
- [ ] 6.4 Add a Grafana dashboard row (or new dashboard) for BRouter host: request rate, p50/p95/p99, JVM heap, container memory, scrape up/down
|
||||
- [ ] 6.5 Add an alert: `up{job="brouter"} == 0 for 2m`
|
||||
|
||||
## 7. Cutover
|
||||
|
||||
- [ ] 7.1 Deploy `infrastructure/brouter-host/` to the dedicated host manually the first time; run `download-segments.sh` (expect multi-hour runtime)
|
||||
- [ ] 7.2 Verify the new BRouter responds to a curl from the flagship host over vSwitch with the auth header, and returns 403 without it
|
||||
- [ ] 7.3 Deploy the Planner with `BROUTER_AUTH_TOKEN` set but `BROUTER_URL` still pointing at the flagship BRouter (no-op change; validates wiring)
|
||||
- [ ] 7.4 Flip `BROUTER_URL` in SOPS to the new vSwitch URL; deploy Planner; monitor `brouter_request_duration_seconds` error rate for 30 minutes
|
||||
- [ ] 7.5 After 48 hours of clean metrics: remove the `brouter` service + `./segments` volume from `infrastructure/docker-compose.yml`; run `cd-infra.yml` to restart without BRouter; `docker image prune` on the flagship
|
||||
- [ ] 7.6 Document rollback path (revert `BROUTER_URL` flip, redeploy Planner, old container warm for 48h) in the PR description
|
||||
|
||||
## 8. Documentation
|
||||
|
||||
- [ ] 8.1 Update `CLAUDE.md` to mention the second deployment target and the `trails`-user deploy pattern for BRouter
|
||||
- [ ] 8.2 Update `docs/architecture.md` with the new topology and vSwitch boundary
|
||||
- [ ] 8.3 Update `docs/deployment.md` (or create) with the BRouter host runbook: first-time provisioning, segment updates, token rotation, rollback
|
||||
- [ ] 8.4 Add a note to `infrastructure/README.md` (if present) distinguishing flagship-host vs. BRouter-host compose projects
|
||||
|
||||
## 9. Verification
|
||||
|
||||
- [ ] 9.1 `pnpm typecheck && pnpm lint && pnpm test` pass with the new env handling and unit test
|
||||
- [ ] 9.2 `pnpm test:e2e` passes with the Planner hitting the relocated BRouter (or a mocked upstream that enforces the auth header)
|
||||
- [ ] 9.3 A manual smoke test from Grafana confirms BRouter metrics and logs appear under the `brouter` host label after cutover
|
||||
- [ ] 9.4 `openspec archive relocate-brouter-to-dedicated-host` runs cleanly after cutover + documentation are merged
|
||||
Loading…
Add table
Add a link
Reference in a new issue