3.5 KiB
Context
The Hetzner CX21 runs Journal, Planner, BRouter, PostgreSQL, and Caddy. There is no monitoring — the disk-full outage was only discovered when a user reported errors. Sentry covers application errors but not infrastructure health, performance trends, or log aggregation.
Goals / Non-Goals
Goals:
- Health endpoints for uptime monitoring
- Structured JSON logging for searchability
- Prometheus metrics for request latency, DB, sessions, BRouter
- Grafana dashboards for at-a-glance status
- Loki for centralized log aggregation
- Alerts for disk full, app down, high error rate
Non-Goals:
- Distributed tracing (Sentry already does this)
- Custom business metrics (user signups, route counts — later)
- External uptime monitoring service (can add later)
- Monitoring for self-hosted instances (flagship only)
Decisions
D1: Grafana + Prometheus + Loki in Docker Compose
Add all three as services in the existing docker-compose.yml. They share the
Docker network with the app containers. Prometheus scrapes /metrics from
Journal and Planner. Loki collects logs via the Docker logging driver.
Grafana is only accessible via SSH tunnel (ssh -L 3100:localhost:3100) or
Caddy with basic auth on a subdomain (e.g., grafana.trails.cool).
D2: Pino for structured logging
Replace console.log/console.error with Pino. JSON output in production,
pretty-print in dev. Pino is the standard Node.js structured logger — fast,
zero-dep in production, and Loki-compatible.
Create a shared logging utility in packages/logging/ or keep it simple
with per-app lib/logger.server.ts.
D3: prom-client for Prometheus metrics
Use prom-client to expose a /metrics endpoint on each app. Default Node.js
metrics (event loop lag, heap, GC) plus custom:
http_request_duration_seconds(histogram, by route + method + status)planner_active_sessions(gauge)planner_connected_clients(gauge)brouter_request_duration_seconds(histogram)db_pool_active_connections(gauge)
D4: Health endpoint checking DB connectivity
GET /health returns { status: "ok", db: "connected" } or
{ status: "degraded", db: "unreachable" } with appropriate HTTP status.
Used by Docker healthcheck and external monitoring.
D5: Caddy access logs to stdout (Loki picks them up)
Enable Caddy's log directive. Structured JSON access logs go to stdout,
Docker sends them to Loki via the logging driver. No sidecar needed.
D6: Grafana provisioned dashboards
Ship dashboard JSON files in infrastructure/grafana/dashboards/. Grafana
auto-loads them via provisioning config. Dashboards:
- Overview: Request rate, error rate, latency p50/p95/p99
- Planner: Active sessions, connected clients, BRouter latency
- Infrastructure: CPU, memory, disk, DB connections
D7: Alert rules via Grafana
Configure alert rules in provisioned dashboard or as code:
- Disk usage > 80% (fires 15 min)
- Any app returns 0 healthy responses for 2 min
- Error rate > 5% for 5 min
- DB connection pool exhausted
Notifications via email (Resend, once transactional-emails is implemented) or webhook.
Risks / Trade-offs
- Memory pressure → Grafana + Prometheus + Loki add ~500MB. CX21 has 4GB, currently using ~2GB. Tight but workable. Monitor and upgrade to CX22 if needed.
- Disk usage → Prometheus retention default 15 days, Loki retention configurable. Set conservative limits (1GB each).
- Complexity → Three new services to maintain. Mitigated by using official Docker images with minimal config.