trails-cool/trails
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
trails/openspec/changes/archive/2026-03-27-observability/proposal.md
Ullrich Schäfer 81d0feffdd
Archive observability change, sync specs to main 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 21:44:07 +01:00

2 KiB
Raw Permalink Blame History

Why

We just had a production outage caused by disk full — and the only way to diagnose it was SSH + manual docker commands. There are no health endpoints, no structured logs, no metrics, and no dashboards. When things break, we're flying blind. The architecture doc specifies a full Grafana + Prometheus + Loki stack for the flagship instance.

What Changes

  • Health endpoints: /health on both apps returning service status + DB connectivity
  • Structured logging: JSON logs from both apps (not plain text console.log)
  • Prometheus metrics: Request latency, active sessions, DB pool stats, BRouter response times, exposed via /metrics endpoint
  • Grafana + Prometheus + Loki stack: Self-hosted on the Hetzner server via Docker Compose, scraping app metrics and collecting container logs
  • Dashboards: Pre-configured Grafana dashboards for request latency, error rates, active Planner sessions, DB performance, disk usage
  • Alerting: Grafana alerts for disk usage > 80%, app down, high error rate
  • Caddy access logging: Enable structured access logs for request visibility

Capabilities

New Capabilities

  • observability: Health endpoints, Prometheus metrics, structured logging, Grafana dashboards, and alerting for the flagship instance

Modified Capabilities

  • infrastructure: Add Grafana, Prometheus, Loki containers to Docker Compose. Configure Caddy access logging.

Impact

  • Infrastructure: 3 new Docker containers (Grafana, Prometheus, Loki) + config files. Adds ~500MB RAM usage.
  • Server: CX21 (4GB RAM) might be tight. May need CX22 (8GB) if memory is an issue.
  • Files: New Prometheus/Grafana/Loki configs, modified docker-compose.yml, new /health and /metrics routes, logging utility
  • Dependencies: prom-client for Prometheus metrics, pino for structured logging
  • Security: Grafana dashboard behind basic auth or restricted to localhost
    • SSH tunnel. Metrics endpoint not publicly accessible.
  • Ports: Grafana on 3100 (internal), Prometheus on 9090 (internal)