The alert was firing with DatasourceNoData because zero 5xx responses
means the numerator returns empty (not 0). Fixed by:
- Adding `or vector(0)` so zero 5xx returns 0% instead of empty
- Setting `noDataState: OK` as a safety net
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The error rate formula (errors/total*100) produces NaN when there's
zero traffic (0/0). Grafana treats NaN as a firing condition.
Use clamp_min on the denominator to return 0% instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The threshold expressions were missing the 'expression' field referencing
the query refId, causing 'no variable specified for refId C' errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Grafana provisioning YAML doesn't support env var substitution.
Replace ${ALERT_EMAIL} with hardcoded admin@trails.cool.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Health endpoints:
- /api/health (Journal) and /health (Planner) with DB connectivity check
- Docker healthchecks updated to use app health endpoints
Structured logging:
- Pino with JSON output in production, pretty-print in dev
- Request logging middleware in Planner (method, path, status, duration)
- Replaced console.log/error with structured logger in email and auth flows
Prometheus metrics:
- prom-client with default Node.js metrics + custom histograms/gauges
- /metrics endpoints on both apps
- http_request_duration, planner_active_sessions, brouter_request_duration
Monitoring stack:
- Prometheus, Loki, Grafana containers in docker-compose
- Grafana provisioned with datasources, dashboards, and alert rules
- Caddy access logging (JSON to stdout for Loki)
- grafana.trails.cool with basic auth via Caddy
Dashboards and alerting:
- Overview: request rate, error rate, latency p50/p95/p99
- Planner: active sessions, connected clients, BRouter latency
- Infrastructure: memory, CPU, event loop lag
- Alerts: disk >80%, app down 2min, error rate >5%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
