trails/openspec/specs/observability/spec.md

Purpose

Health endpoints, Prometheus metrics, structured logging, Grafana dashboards, and alerting for both apps and the flagship instance.

Requirements

Requirement: Health endpoints

Both apps SHALL expose a /health endpoint returning service and database status.

Scenario: Healthy service

• WHEN the app is running and the database is reachable
• THEN GET /health returns { "status": "ok", "db": "connected", "version": "<string>" } with HTTP 200 (version from SENTRY_RELEASE env var, defaults to "dev")

Scenario: Degraded service

• WHEN the app is running but the database is unreachable
• THEN GET /health returns { "status": "degraded", "db": "unreachable", "version": "<string>" } with HTTP 503

Requirement: Prometheus metrics

Both apps SHALL expose a /metrics endpoint with Prometheus-formatted metrics.

Scenario: Default Node.js metrics

• WHEN Prometheus scrapes /metrics
• THEN it receives event loop lag, heap usage, and GC metrics

Scenario: HTTP request metrics

• WHEN requests are served
• THEN http_request_duration_seconds histogram is updated with route, method, and status labels

Scenario: Planner-specific metrics

• WHEN the Planner is running
• THEN planner_active_sessions and planner_connected_clients gauges reflect current state

Scenario: BRouter latency metrics

• WHEN the Planner proxies a routing request to BRouter
• THEN brouter_request_duration_seconds histogram is updated (buckets: 0.1s to 10s)

Requirement: Structured logging

Both apps SHALL output structured JSON logs in production.

Scenario: Request logging

• WHEN an HTTP request is served in production
• THEN a JSON log line is emitted with method, path, status, duration, and timestamp

Scenario: Dev mode pretty printing

• WHEN running in development
• THEN logs are human-readable (pretty-printed)

Requirement: Grafana dashboards

The flagship instance SHALL have pre-configured Grafana dashboards.

Scenario: Overview dashboard

• WHEN an operator opens Grafana
• THEN they see request rate, error rate, and latency percentiles

Scenario: Infrastructure dashboard

• WHEN an operator checks infrastructure health
• THEN they see CPU, memory, disk usage, and DB connection pool stats

Requirement: Alerting

Grafana SHALL alert on critical conditions.

Scenario: Disk usage alert

• WHEN disk usage exceeds 80% for 15 minutes
• THEN an alert fires

Scenario: App down alert

• WHEN an app returns zero healthy responses for 2 minutes
• THEN an alert fires

Requirement: Log aggregation

Loki SHALL collect and index logs from all Docker containers.

Scenario: Search logs

• WHEN an operator searches logs in Grafana
• THEN they can filter by container name, log level, and time range

Requirement: Remote BRouter metrics

Prometheus on the flagship host SHALL scrape BRouter-specific metrics from the dedicated BRouter host over the vSwitch. Scraping SHALL be scoped to BRouter containers only and SHALL NOT collect host-level metrics from the dedicated host's other (non-trails.cool) workloads.

Scenario: BRouter container metrics collected

• WHEN Prometheus scrapes the dedicated BRouter host
• THEN metrics from the BRouter container (and, if enabled, the Caddy sidecar) are collected via cAdvisor filtered by container label, or via a BRouter/JMX metrics endpoint exposed on the vSwitch

Scenario: No shared-host noise

• WHEN Prometheus is configured with the remote BRouter scrape target
• THEN no configuration scrapes the dedicated host's node_exporter or metrics from containers other than BRouter and its sidecar

Scenario: BRouter scrape failure alert

• WHEN the BRouter scrape target returns up == 0 for more than 2 minutes
• THEN Grafana fires an alert distinct from flagship-side alerts

Requirement: Remote BRouter log shipping

Loki on the flagship host SHALL receive BRouter container logs from the dedicated BRouter host via a Promtail or Alloy agent running on that host as the trails user. Log shipping SHALL be scoped to BRouter-related containers only.

Scenario: BRouter logs visible in Grafana

• WHEN the BRouter container writes to stdout or stderr
• THEN the log line is available in Grafana Explore via Loki with container and host labels identifying it as coming from the dedicated BRouter host

Scenario: Non-BRouter logs not shipped

• WHEN a non-BRouter container on the dedicated host writes logs
• THEN those logs are not ingested by the flagship Loki instance