The annotation config was missing the target block that tells
Grafana to filter by tags. Without it, the annotation query
was silently ignored.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The error rate formula (errors/total*100) produces NaN when there's
zero traffic (0/0). Grafana treats NaN as a firing condition.
Use clamp_min on the denominator to return 0% instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The threshold expressions were missing the 'expression' field referencing
the query refId, causing 'no variable specified for refId C' errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Panels:
- Stat cards: total users, routes, activities, active planner sessions
- Time series: cumulative user growth, route growth
- Bar charts: signups/day, routes/day, planner sessions/day
- Table: top 20 routes by distance
All queries use the grafana_reader PostgreSQL datasource.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Create grafana_reader role with SELECT-only access on all schemas
- Init script runs on postgres first boot (docker-entrypoint-initdb.d)
- Grafana PostgreSQL datasource provisioned with read-only credentials
- Enables SQL queries in dashboards (user count, routes, etc.)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exporters:
- postgres_exporter: DB connections, transactions, cache hit ratio, query stats
- node_exporter: host CPU, memory, disk, network
- cAdvisor: per-container CPU and memory usage
PostgreSQL:
- Enable pg_stat_statements for query-level performance tracking
- Track index scans vs sequential scans, cache hit ratio
Dashboard (service-health.json):
- DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage
- Host: disk gauge, CPU, memory, network I/O, disk I/O
- BRouter: request latency p50/p95/p99, container CPU + memory
- All containers: CPU and memory comparison
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Alert rules reference datasourceUid: prometheus/loki which requires
explicit UIDs in the provisioning config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Grafana provisioning YAML doesn't support env var substitution.
Replace ${ALERT_EMAIL} with hardcoded admin@trails.cool.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Health endpoints:
- /api/health (Journal) and /health (Planner) with DB connectivity check
- Docker healthchecks updated to use app health endpoints
Structured logging:
- Pino with JSON output in production, pretty-print in dev
- Request logging middleware in Planner (method, path, status, duration)
- Replaced console.log/error with structured logger in email and auth flows
Prometheus metrics:
- prom-client with default Node.js metrics + custom histograms/gauges
- /metrics endpoints on both apps
- http_request_duration, planner_active_sessions, brouter_request_duration
Monitoring stack:
- Prometheus, Loki, Grafana containers in docker-compose
- Grafana provisioned with datasources, dashboards, and alert rules
- Caddy access logging (JSON to stdout for Loki)
- grafana.trails.cool with basic auth via Caddy
Dashboards and alerting:
- Overview: request rate, error rate, latency p50/p95/p99
- Planner: active sessions, connected clients, BRouter latency
- Infrastructure: memory, CPU, event loop lag
- Alerts: disk >80%, app down 2min, error rate >5%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
