Archive observability change, sync specs to main

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

openspec/changes/archive/2026-03-27-observability/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-03-25

openspec/changes/archive/2026-03-27-observability/design.md

@@ -0,0 +1,92 @@
+## Context
+
+The Hetzner CX21 runs Journal, Planner, BRouter, PostgreSQL, and Caddy. There
+is no monitoring — the disk-full outage was only discovered when a user
+reported errors. Sentry covers application errors but not infrastructure health,
+performance trends, or log aggregation.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Health endpoints for uptime monitoring
+- Structured JSON logging for searchability
+- Prometheus metrics for request latency, DB, sessions, BRouter
+- Grafana dashboards for at-a-glance status
+- Loki for centralized log aggregation
+- Alerts for disk full, app down, high error rate
+
+**Non-Goals:**
+- Distributed tracing (Sentry already does this)
+- Custom business metrics (user signups, route counts — later)
+- External uptime monitoring service (can add later)
+- Monitoring for self-hosted instances (flagship only)
+
+## Decisions
+
+### D1: Grafana + Prometheus + Loki in Docker Compose
+
+Add all three as services in the existing docker-compose.yml. They share the
+Docker network with the app containers. Prometheus scrapes `/metrics` from
+Journal and Planner. Loki collects logs via the Docker logging driver.
+
+Grafana is only accessible via SSH tunnel (`ssh -L 3100:localhost:3100`) or
+Caddy with basic auth on a subdomain (e.g., `grafana.trails.cool`).
+
+### D2: Pino for structured logging
+
+Replace `console.log`/`console.error` with Pino. JSON output in production,
+pretty-print in dev. Pino is the standard Node.js structured logger — fast,
+zero-dep in production, and Loki-compatible.
+
+Create a shared logging utility in `packages/logging/` or keep it simple
+with per-app `lib/logger.server.ts`.
+
+### D3: prom-client for Prometheus metrics
+
+Use `prom-client` to expose a `/metrics` endpoint on each app. Default Node.js
+metrics (event loop lag, heap, GC) plus custom:
+- `http_request_duration_seconds` (histogram, by route + method + status)
+- `planner_active_sessions` (gauge)
+- `planner_connected_clients` (gauge)
+- `brouter_request_duration_seconds` (histogram)
+- `db_pool_active_connections` (gauge)
+
+### D4: Health endpoint checking DB connectivity
+
+`GET /health` returns `{ status: "ok", db: "connected" }` or
+`{ status: "degraded", db: "unreachable" }` with appropriate HTTP status.
+Used by Docker healthcheck and external monitoring.
+
+### D5: Caddy access logs to stdout (Loki picks them up)
+
+Enable Caddy's `log` directive. Structured JSON access logs go to stdout,
+Docker sends them to Loki via the logging driver. No sidecar needed.
+
+### D6: Grafana provisioned dashboards
+
+Ship dashboard JSON files in `infrastructure/grafana/dashboards/`. Grafana
+auto-loads them via provisioning config. Dashboards:
+- **Overview**: Request rate, error rate, latency p50/p95/p99
+- **Planner**: Active sessions, connected clients, BRouter latency
+- **Infrastructure**: CPU, memory, disk, DB connections
+
+### D7: Alert rules via Grafana
+
+Configure alert rules in provisioned dashboard or as code:
+- Disk usage > 80% (fires 15 min)
+- Any app returns 0 healthy responses for 2 min
+- Error rate > 5% for 5 min
+- DB connection pool exhausted
+
+Notifications via email (Resend, once transactional-emails is implemented)
+or webhook.
+
+## Risks / Trade-offs
+
+- **Memory pressure** → Grafana + Prometheus + Loki add ~500MB. CX21 has 4GB,
+  currently using ~2GB. Tight but workable. Monitor and upgrade to CX22 if
+  needed.
+- **Disk usage** → Prometheus retention default 15 days, Loki retention
+  configurable. Set conservative limits (1GB each).
+- **Complexity** → Three new services to maintain. Mitigated by using official
+  Docker images with minimal config.

openspec/changes/archive/2026-03-27-observability/proposal.md

@@ -0,0 +1,47 @@
+## Why
+
+We just had a production outage caused by disk full — and the only way to
+diagnose it was SSH + manual docker commands. There are no health endpoints,
+no structured logs, no metrics, and no dashboards. When things break, we're
+flying blind. The architecture doc specifies a full Grafana + Prometheus + Loki
+stack for the flagship instance.
+
+## What Changes
+
+- **Health endpoints**: `/health` on both apps returning service status + DB
+  connectivity
+- **Structured logging**: JSON logs from both apps (not plain text console.log)
+- **Prometheus metrics**: Request latency, active sessions, DB pool stats,
+  BRouter response times, exposed via `/metrics` endpoint
+- **Grafana + Prometheus + Loki stack**: Self-hosted on the Hetzner server via
+  Docker Compose, scraping app metrics and collecting container logs
+- **Dashboards**: Pre-configured Grafana dashboards for request latency, error
+  rates, active Planner sessions, DB performance, disk usage
+- **Alerting**: Grafana alerts for disk usage > 80%, app down, high error rate
+- **Caddy access logging**: Enable structured access logs for request visibility
+
+## Capabilities
+
+### New Capabilities
+
+- `observability`: Health endpoints, Prometheus metrics, structured logging,
+  Grafana dashboards, and alerting for the flagship instance
+
+### Modified Capabilities
+
+- `infrastructure`: Add Grafana, Prometheus, Loki containers to Docker Compose.
+  Configure Caddy access logging.
+
+## Impact
+
+- **Infrastructure**: 3 new Docker containers (Grafana, Prometheus, Loki) +
+  config files. Adds ~500MB RAM usage.
+- **Server**: CX21 (4GB RAM) might be tight. May need CX22 (8GB) if memory
+  is an issue.
+- **Files**: New Prometheus/Grafana/Loki configs, modified docker-compose.yml,
+  new `/health` and `/metrics` routes, logging utility
+- **Dependencies**: `prom-client` for Prometheus metrics, `pino` for structured
+  logging
+- **Security**: Grafana dashboard behind basic auth or restricted to localhost
+  + SSH tunnel. Metrics endpoint not publicly accessible.
+- **Ports**: Grafana on 3100 (internal), Prometheus on 9090 (internal)

openspec/changes/archive/2026-03-27-observability/specs/infrastructure/spec.md

@@ -0,0 +1,15 @@
+## MODIFIED Requirements
+
+### Requirement: Docker Compose deployment
+All services SHALL be deployed via Docker Compose, including Grafana, Prometheus, and Loki for the flagship instance.
+
+#### Scenario: Monitoring stack starts
+- **WHEN** `docker compose up -d` is run
+- **THEN** Grafana, Prometheus, and Loki containers start alongside the application containers
+
+### Requirement: Caddy access logging
+Caddy SHALL emit structured JSON access logs for all requests.
+
+#### Scenario: Access log emitted
+- **WHEN** any HTTP request passes through Caddy
+- **THEN** a JSON log line with remote IP, method, path, status, and duration is written to stdout

openspec/changes/archive/2026-03-27-observability/specs/observability/spec.md

@@ -0,0 +1,67 @@
+## ADDED Requirements
+
+### Requirement: Health endpoints
+Both apps SHALL expose a `/health` endpoint returning service and database status.
+
+#### Scenario: Healthy service
+- **WHEN** the app is running and the database is reachable
+- **THEN** `GET /health` returns `{ "status": "ok", "db": "connected" }` with HTTP 200
+
+#### Scenario: Degraded service
+- **WHEN** the app is running but the database is unreachable
+- **THEN** `GET /health` returns `{ "status": "degraded", "db": "unreachable" }` with HTTP 503
+
+### Requirement: Prometheus metrics
+Both apps SHALL expose a `/metrics` endpoint with Prometheus-formatted metrics.
+
+#### Scenario: Default Node.js metrics
+- **WHEN** Prometheus scrapes `/metrics`
+- **THEN** it receives event loop lag, heap usage, and GC metrics
+
+#### Scenario: HTTP request metrics
+- **WHEN** requests are served
+- **THEN** `http_request_duration_seconds` histogram is updated with route, method, and status labels
+
+#### Scenario: Planner-specific metrics
+- **WHEN** the Planner is running
+- **THEN** `planner_active_sessions` and `planner_connected_clients` gauges reflect current state
+
+### Requirement: Structured logging
+Both apps SHALL output structured JSON logs in production.
+
+#### Scenario: Request logging
+- **WHEN** an HTTP request is served in production
+- **THEN** a JSON log line is emitted with method, path, status, duration, and timestamp
+
+#### Scenario: Dev mode pretty printing
+- **WHEN** running in development
+- **THEN** logs are human-readable (pretty-printed)
+
+### Requirement: Grafana dashboards
+The flagship instance SHALL have pre-configured Grafana dashboards.
+
+#### Scenario: Overview dashboard
+- **WHEN** an operator opens Grafana
+- **THEN** they see request rate, error rate, and latency percentiles
+
+#### Scenario: Infrastructure dashboard
+- **WHEN** an operator checks infrastructure health
+- **THEN** they see CPU, memory, disk usage, and DB connection pool stats
+
+### Requirement: Alerting
+Grafana SHALL alert on critical conditions.
+
+#### Scenario: Disk usage alert
+- **WHEN** disk usage exceeds 80% for 15 minutes
+- **THEN** an alert fires
+
+#### Scenario: App down alert
+- **WHEN** an app returns zero healthy responses for 2 minutes
+- **THEN** an alert fires
+
+### Requirement: Log aggregation
+Loki SHALL collect and index logs from all Docker containers.
+
+#### Scenario: Search logs
+- **WHEN** an operator searches logs in Grafana
+- **THEN** they can filter by container name, log level, and time range

openspec/changes/archive/2026-03-27-observability/tasks.md

@@ -0,0 +1,62 @@
+## 1. Health Endpoints
+
+- [x] 1.1 Add `/health` API route to Journal — checks DB with a simple query, returns status JSON
+- [x] 1.2 Add `/health` endpoint to Planner server.ts — checks DB connectivity
+- [x] 1.3 Update Docker healthcheck in docker-compose.yml to use `/health` instead of `pg_isready`
+- [x] 1.4 Add health routes to route configs (journal routes.ts)
+
+## 2. Structured Logging
+
+- [x] 2.1 Add `pino` dependency to both apps
+- [x] 2.2 Create `apps/journal/app/lib/logger.server.ts` — Pino instance, JSON in prod, pretty in dev
+- [x] 2.3 Create `apps/planner/app/lib/logger.server.ts` — same pattern
+- [x] 2.4 Replace key `console.log`/`console.error` calls with logger (auth, DB errors, session lifecycle)
+- [x] 2.5 Add request logging middleware to Planner server.ts (method, path, status, duration)
+
+## 3. Prometheus Metrics
+
+- [x] 3.1 Add `prom-client` dependency to both apps
+- [x] 3.2 Create metrics utility for Journal — default metrics + http_request_duration_seconds histogram
+- [x] 3.3 Create metrics utility for Planner — default metrics + http histogram + planner_active_sessions + planner_connected_clients gauges + brouter_request_duration_seconds histogram
+- [x] 3.4 Add `/metrics` endpoint to Journal (API route)
+- [x] 3.5 Add `/metrics` endpoint to Planner server.ts
+- [x] 3.6 Wire request duration tracking into request handlers
+
+## 4. Caddy Access Logs
+
+- [x] 4.1 Enable Caddy structured JSON access logging in Caddyfile (log directive)
+
+## 5. Monitoring Stack (Docker Compose)
+
+- [x] 5.1 Add Prometheus container to docker-compose.yml with scrape config for journal:3000 and planner:3001
+- [x] 5.2 Add Loki container to docker-compose.yml with Docker logging driver config
+- [x] 5.3 Add Grafana container to docker-compose.yml with provisioned data sources (Prometheus + Loki)
+- [x] 5.4 Create `infrastructure/prometheus/prometheus.yml` — scrape config
+- [x] 5.5 Create `infrastructure/loki/loki-config.yml` — retention, storage
+- [x] 5.6 Create `infrastructure/grafana/provisioning/` — data sources + dashboard provider config
+- [x] 5.7 Expose Grafana via Caddy on grafana.trails.cool with basic auth
+
+## 6. Dashboards
+
+- [x] 6.1 Create overview dashboard JSON — request rate, error rate, latency p50/p95/p99
+- [x] 6.2 Create planner dashboard JSON — active sessions, connected clients, BRouter latency
+- [x] 6.3 Create infrastructure dashboard JSON — node_exporter or cAdvisor metrics (CPU, memory, disk)
+
+## 7. Alerting
+
+- [x] 7.1 Configure Grafana alert rule: disk usage > 80%
+- [x] 7.2 Configure Grafana alert rule: app health check failing for 2 min
+- [x] 7.3 Configure Grafana alert rule: error rate > 5% for 5 min
+- [x] 7.4 Set up alert notification channel (email or webhook)
+
+## 8. DNS & Terraform
+
+- [x] 8.1 Add grafana.trails.cool DNS record
+- [x] 8.2 Add Grafana basic auth credentials to deploy secrets
+
+## 9. Verify
+
+- [x] 9.1 Test /health endpoints locally — verify OK and degraded responses
+- [x] 9.2 Test /metrics endpoints — verify Prometheus format output
+- [x] 9.3 Test Grafana dashboards load with data after deploy
+- [x] 9.4 Test alert fires when simulating disk full condition