trails/openspec/changes/relocate-brouter-to-dedicated-host/specs/observability/spec.md at 05b72e91149534dbcb3ffff45e4fe65415bfb2f8

Ullrich Schäfer 102a744e67

Add Hetzner vSwitch network for BRouter relocation

Adds an OpenSpec change scoping the relocation of BRouter from the
co-located flagship (cx23, 4 GB RAM, 40 GB SSD, Europe segments only) to
a dedicated Hetzner Robot host in the same datacenter, with private
connectivity over Hetzner vSwitch #80672 (VLAN 4000).

This first PR only lays the network prerequisite:

- Terraform: a Hetzner Cloud Network (10.0.0.0/16) with a cloud subnet
  (10.0.0.0/24) hosting the flagship at 10.0.0.2, and a vSwitch subnet
  (10.0.1.0/24) bridged to Robot VLAN 4000. The dedicated host's VLAN
  sub-interface (10.0.1.10 on enp4s0.4000) is configured out-of-band via
  netplan and is not Terraform-managed.
- lifecycle { ignore_changes = [user_data] } on the flagship server to
  prevent the Hetzner provider's post-1.45 user_data hash drift from
  triggering a spurious full-server replacement on unrelated applies.
- OpenSpec change with proposal, design, specs (delta for
  brouter-integration / infrastructure / observability /
  security-hardening), and tasks; Section 1 (pre-flight) is checked off
  with operator notes.

Verification: ping both directions across the vSwitch is 0% loss,
sub-ms latency; dedicated host's VLAN config persists across reboot
(verified ~60 s to restore private reachability).

Follow-up PRs will land the BRouter host compose project, Planner
shared-secret header, CD workflow retarget, observability wiring, and
the cutover.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-04-23 22:43:57 +02:00

1.8 KiB

Raw Blame History

ADDED Requirements

Requirement: Remote BRouter metrics

Prometheus on the flagship host SHALL scrape BRouter-specific metrics from the dedicated BRouter host over the vSwitch. Scraping SHALL be scoped to BRouter containers only and SHALL NOT collect host-level metrics from the dedicated host's other (non-trails.cool) workloads.

Scenario: BRouter container metrics collected

WHEN Prometheus scrapes the dedicated BRouter host
THEN metrics from the BRouter container (and, if enabled, the Caddy sidecar) are collected via cAdvisor filtered by container label, or via a BRouter/JMX metrics endpoint exposed on the vSwitch

Scenario: No shared-host noise

WHEN Prometheus is configured with the remote BRouter scrape target
THEN no configuration scrapes the dedicated host's node_exporter or metrics from containers other than BRouter and its sidecar

Scenario: BRouter scrape failure alert

WHEN the BRouter scrape target returns up == 0 for more than 2 minutes
THEN Grafana fires an alert distinct from flagship-side alerts

Requirement: Remote BRouter log shipping

Loki on the flagship host SHALL receive BRouter container logs from the dedicated BRouter host via a Promtail or Alloy agent running on that host as the trails user. Log shipping SHALL be scoped to BRouter-related containers only.

Scenario: BRouter logs visible in Grafana

WHEN the BRouter container writes to stdout or stderr
THEN the log line is available in Grafana Explore via Loki with container and host labels identifying it as coming from the dedicated BRouter host

Scenario: Non-BRouter logs not shipped

WHEN a non-BRouter container on the dedicated host writes logs
THEN those logs are not ingested by the flagship Loki instance

1.8 KiB Raw Blame History