Fix blind spot where Caddy 502 errors were invisible: the dashboards and
alerts queried caddy_http_response_duration_seconds (upstream responses only),
missing 502s that Caddy generates itself. Switch to
caddy_http_request_duration_seconds (server-level, all responses).
Add Journal Grafana dashboard with: 502 rate, response codes, request rate
by route, latency percentiles, container restarts/memory/CPU, Node.js event
loop lag and heap, and Loki log panels for errors and Caddy 5xx entries.
Add color coding to Caddy status code panel (green=2xx, blue=3xx, yellow=4xx,
red=5xx). Add log-based error rate panel to the overview dashboard.
New alerts: container restart loop, PostgreSQL connections > 80, application
crash log detection (Loki), and Caddy 502 rate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Dashboard: switch error rate panel from app-level http_request_duration
(never observed) to Caddy's caddy_http_response_duration_seconds_count.
Journal: add custom server.ts (matching planner pattern) that observes
http_request_duration_seconds on every request. Replaces react-router-serve
with a custom Node HTTP server that handles /api/health, /api/metrics,
static assets, and request timing. Removes redundant React Router routes
for health and metrics.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Caddy Response Status Codes: caddy_http_responses_total doesn't exist,
use caddy_http_response_duration_seconds_count which has the code label
- pg_stat_statements: enable extension in init script, add custom queries
config for postgres-exporter, remove invalid datname filter from query
- Add postgres/queries.yml to cd-infra SCP sources
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The alert was firing with DatasourceNoData because zero 5xx responses
means the numerator returns empty (not 0). Fixed by:
- Adding `or vector(0)` so zero 5xx returns 0% instead of empty
- Setting `noDataState: OK` as a safety net
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Grafana file-based provisioning expects flat dashboard JSON, not the
API format with a "dashboard" wrapper. The 4 wrapped dashboards were
silently failing to load — only business.json (already flat) worked.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The type:dashboard field was causing Grafana to filter annotations
by dashboard scope. Our deploy annotations are global (no dashboardUID),
so the target.type:tags is the only filter needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The annotation config was missing the target block that tells
Grafana to filter by tags. Without it, the annotation query
was silently ignored.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The error rate formula (errors/total*100) produces NaN when there's
zero traffic (0/0). Grafana treats NaN as a firing condition.
Use clamp_min on the denominator to return 0% instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The threshold expressions were missing the 'expression' field referencing
the query refId, causing 'no variable specified for refId C' errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Panels:
- Stat cards: total users, routes, activities, active planner sessions
- Time series: cumulative user growth, route growth
- Bar charts: signups/day, routes/day, planner sessions/day
- Table: top 20 routes by distance
All queries use the grafana_reader PostgreSQL datasource.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Create grafana_reader role with SELECT-only access on all schemas
- Init script runs on postgres first boot (docker-entrypoint-initdb.d)
- Grafana PostgreSQL datasource provisioned with read-only credentials
- Enables SQL queries in dashboards (user count, routes, etc.)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Exporters:
- postgres_exporter: DB connections, transactions, cache hit ratio, query stats
- node_exporter: host CPU, memory, disk, network
- cAdvisor: per-container CPU and memory usage
PostgreSQL:
- Enable pg_stat_statements for query-level performance tracking
- Track index scans vs sequential scans, cache hit ratio
Dashboard (service-health.json):
- DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage
- Host: disk gauge, CPU, memory, network I/O, disk I/O
- BRouter: request latency p50/p95/p99, container CPU + memory
- All containers: CPU and memory comparison
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Alert rules reference datasourceUid: prometheus/loki which requires
explicit UIDs in the provisioning config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Grafana provisioning YAML doesn't support env var substitution.
Replace ${ALERT_EMAIL} with hardcoded admin@trails.cool.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Health endpoints:
- /api/health (Journal) and /health (Planner) with DB connectivity check
- Docker healthchecks updated to use app health endpoints
Structured logging:
- Pino with JSON output in production, pretty-print in dev
- Request logging middleware in Planner (method, path, status, duration)
- Replaced console.log/error with structured logger in email and auth flows
Prometheus metrics:
- prom-client with default Node.js metrics + custom histograms/gauges
- /metrics endpoints on both apps
- http_request_duration, planner_active_sessions, brouter_request_duration
Monitoring stack:
- Prometheus, Loki, Grafana containers in docker-compose
- Grafana provisioned with datasources, dashboards, and alert rules
- Caddy access logging (JSON to stdout for Loki)
- grafana.trails.cool with basic auth via Caddy
Dashboards and alerting:
- Overview: request rate, error rate, latency p50/p95/p99
- Planner: active sessions, connected clients, BRouter latency
- Infrastructure: memory, CPU, event loop lag
- Alerts: disk >80%, app down 2min, error rate >5%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
