7d20dbb12f
Two new OpenSpec changes: - observability (30 tasks): health endpoints, Prometheus, Grafana+Loki, structured logging, dashboards, alerting - security-hardening (24 tasks): Caddy headers, scanner blocking, gitleaks, pnpm audit, dependabot, non-root Docker, fail2ban Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
92 lines
3.5 KiB
Markdown
92 lines
3.5 KiB
Markdown
## Context
|
|
|
|
The Hetzner CX21 runs Journal, Planner, BRouter, PostgreSQL, and Caddy. There
|
|
is no monitoring — the disk-full outage was only discovered when a user
|
|
reported errors. Sentry covers application errors but not infrastructure health,
|
|
performance trends, or log aggregation.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
- Health endpoints for uptime monitoring
|
|
- Structured JSON logging for searchability
|
|
- Prometheus metrics for request latency, DB, sessions, BRouter
|
|
- Grafana dashboards for at-a-glance status
|
|
- Loki for centralized log aggregation
|
|
- Alerts for disk full, app down, high error rate
|
|
|
|
**Non-Goals:**
|
|
- Distributed tracing (Sentry already does this)
|
|
- Custom business metrics (user signups, route counts — later)
|
|
- External uptime monitoring service (can add later)
|
|
- Monitoring for self-hosted instances (flagship only)
|
|
|
|
## Decisions
|
|
|
|
### D1: Grafana + Prometheus + Loki in Docker Compose
|
|
|
|
Add all three as services in the existing docker-compose.yml. They share the
|
|
Docker network with the app containers. Prometheus scrapes `/metrics` from
|
|
Journal and Planner. Loki collects logs via the Docker logging driver.
|
|
|
|
Grafana is only accessible via SSH tunnel (`ssh -L 3100:localhost:3100`) or
|
|
Caddy with basic auth on a subdomain (e.g., `grafana.trails.cool`).
|
|
|
|
### D2: Pino for structured logging
|
|
|
|
Replace `console.log`/`console.error` with Pino. JSON output in production,
|
|
pretty-print in dev. Pino is the standard Node.js structured logger — fast,
|
|
zero-dep in production, and Loki-compatible.
|
|
|
|
Create a shared logging utility in `packages/logging/` or keep it simple
|
|
with per-app `lib/logger.server.ts`.
|
|
|
|
### D3: prom-client for Prometheus metrics
|
|
|
|
Use `prom-client` to expose a `/metrics` endpoint on each app. Default Node.js
|
|
metrics (event loop lag, heap, GC) plus custom:
|
|
- `http_request_duration_seconds` (histogram, by route + method + status)
|
|
- `planner_active_sessions` (gauge)
|
|
- `planner_connected_clients` (gauge)
|
|
- `brouter_request_duration_seconds` (histogram)
|
|
- `db_pool_active_connections` (gauge)
|
|
|
|
### D4: Health endpoint checking DB connectivity
|
|
|
|
`GET /health` returns `{ status: "ok", db: "connected" }` or
|
|
`{ status: "degraded", db: "unreachable" }` with appropriate HTTP status.
|
|
Used by Docker healthcheck and external monitoring.
|
|
|
|
### D5: Caddy access logs to stdout (Loki picks them up)
|
|
|
|
Enable Caddy's `log` directive. Structured JSON access logs go to stdout,
|
|
Docker sends them to Loki via the logging driver. No sidecar needed.
|
|
|
|
### D6: Grafana provisioned dashboards
|
|
|
|
Ship dashboard JSON files in `infrastructure/grafana/dashboards/`. Grafana
|
|
auto-loads them via provisioning config. Dashboards:
|
|
- **Overview**: Request rate, error rate, latency p50/p95/p99
|
|
- **Planner**: Active sessions, connected clients, BRouter latency
|
|
- **Infrastructure**: CPU, memory, disk, DB connections
|
|
|
|
### D7: Alert rules via Grafana
|
|
|
|
Configure alert rules in provisioned dashboard or as code:
|
|
- Disk usage > 80% (fires 15 min)
|
|
- Any app returns 0 healthy responses for 2 min
|
|
- Error rate > 5% for 5 min
|
|
- DB connection pool exhausted
|
|
|
|
Notifications via email (Resend, once transactional-emails is implemented)
|
|
or webhook.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- **Memory pressure** → Grafana + Prometheus + Loki add ~500MB. CX21 has 4GB,
|
|
currently using ~2GB. Tight but workable. Monitor and upgrade to CX22 if
|
|
needed.
|
|
- **Disk usage** → Prometheus retention default 15 days, Loki retention
|
|
configurable. Set conservative limits (1GB each).
|
|
- **Complexity** → Three new services to maintain. Mitigated by using official
|
|
Docker images with minimal config.
|