From a931cb03f887d4d2843991d44c873f4dff30ec79 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ullrich=20Sch=C3=A4fer?= Date: Fri, 27 Mar 2026 18:44:14 +0100 Subject: [PATCH] Add service health monitoring: postgres, node, cAdvisor exporters + dashboard Exporters: - postgres_exporter: DB connections, transactions, cache hit ratio, query stats - node_exporter: host CPU, memory, disk, network - cAdvisor: per-container CPU and memory usage PostgreSQL: - Enable pg_stat_statements for query-level performance tracking - Track index scans vs sequential scans, cache hit ratio Dashboard (service-health.json): - DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage - Host: disk gauge, CPU, memory, network I/O, disk I/O - BRouter: request latency p50/p95/p99, container CPU + memory - All containers: CPU and memory comparison Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/cd-infra.yml | 13 +- CLAUDE.md | 26 + infrastructure/Caddyfile | 7 + infrastructure/docker-compose.yml | 42 ++ .../grafana/dashboards/infrastructure.json | 126 +++-- .../grafana/dashboards/overview.json | 166 +++++-- .../grafana/dashboards/planner.json | 111 +++-- .../grafana/dashboards/service-health.json | 443 ++++++++++++++++++ infrastructure/prometheus/prometheus.yml | 12 + .../.openspec.yaml | 0 .../2026-03-27-sops-age-split-cd}/design.md | 0 .../2026-03-27-sops-age-split-cd}/proposal.md | 0 .../specs/infrastructure/spec.md | 0 .../specs/secret-management/spec.md | 0 .../2026-03-27-sops-age-split-cd}/tasks.md | 10 +- openspec/specs/infrastructure/spec.md | 27 +- openspec/specs/secret-management/spec.md | 16 + 17 files changed, 859 insertions(+), 140 deletions(-) create mode 100644 infrastructure/grafana/dashboards/service-health.json rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/.openspec.yaml (100%) rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/design.md (100%) rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/proposal.md (100%) rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/specs/infrastructure/spec.md (100%) rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/specs/secret-management/spec.md (100%) rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/tasks.md (87%) create mode 100644 openspec/specs/secret-management/spec.md diff --git a/.github/workflows/cd-infra.yml b/.github/workflows/cd-infra.yml index 82194a8..ccd8f36 100644 --- a/.github/workflows/cd-infra.yml +++ b/.github/workflows/cd-infra.yml @@ -68,12 +68,19 @@ jobs: GHCR_TOKEN=$(grep DEPLOY_GHCR_TOKEN .env | cut -d= -f2-) echo "$GHCR_TOKEN" | docker login ghcr.io -u stigi --password-stdin 2>/dev/null || true - # Restart services - if [ "${{ github.event.inputs.restart_all }}" = "true" ]; then + # Check if full restart requested via: + # - workflow_dispatch input: restart_all=true + # - commit message containing [restart-all] + RESTART_ALL="${{ github.event.inputs.restart_all }}" + if [ "$RESTART_ALL" != "true" ] && echo "${{ github.event.head_commit.message }}" | grep -q '\[restart-all\]'; then + RESTART_ALL="true" + fi + + if [ "$RESTART_ALL" = "true" ]; then docker compose --env-file .env up -d --remove-orphans else # Restart infra services (except Caddy — just reload its config) - docker compose --env-file .env up -d postgres prometheus loki grafana + docker compose --env-file .env up -d postgres prometheus loki grafana postgres-exporter node-exporter cadvisor docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile fi diff --git a/CLAUDE.md b/CLAUDE.md index c18798d..b73a560 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -140,6 +140,32 @@ before pushing to an existing PR branch. ### Emergency override Admins can bypass the PR workflow when necessary (e.g., CI is broken and needs a hotfix). Document the reason in the commit message. +## Deployment + +Three separate CD workflows triggered by path: + +| Workflow | Triggers on | Deploys | +|----------|-------------|---------| +| `cd-apps.yml` | `apps/`, `packages/`, `pnpm-lock.yaml` | journal, planner | +| `cd-infra.yml` | `infrastructure/` | caddy, postgres, prometheus, loki, grafana, exporters | +| `cd-brouter.yml` | `docker/brouter/` | brouter | + +### Secrets +All secrets are stored in SOPS-encrypted files (`infrastructure/secrets.app.env`, `infrastructure/secrets.infra.env`). Edit with `sops infrastructure/secrets.app.env`. Only `AGE_SECRET_KEY`, `DEPLOY_SSH_KEY`, and `DEPLOY_HOST` remain as GitHub secrets. + +### Full restart +To restart **all** containers (not just the ones a workflow normally touches), either: +- Add `[restart-all]` to the commit message +- Or trigger manually: `gh workflow run cd-infra.yml -f restart_all=true` + +### Server access +```bash +ssh -i ~/.ssh/trails-cool-deploy root@trails.cool +``` + +### Grafana +`https://grafana.internal.trails.cool` — GitHub OAuth (trails-cool org) + ## OpenSpec Workflow Specs live in `openspec/`. Use these slash commands: diff --git a/infrastructure/Caddyfile b/infrastructure/Caddyfile index 70513bd..84d4eaf 100644 --- a/infrastructure/Caddyfile +++ b/infrastructure/Caddyfile @@ -1,3 +1,10 @@ +{ + servers { + metrics + } + admin 0.0.0.0:2019 +} + (security_headers) { header { Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" diff --git a/infrastructure/docker-compose.yml b/infrastructure/docker-compose.yml index 9c20fcf..fffdb62 100644 --- a/infrastructure/docker-compose.yml +++ b/infrastructure/docker-compose.yml @@ -78,12 +78,52 @@ services: POSTGRES_DB: trails volumes: - pgdata:/var/lib/postgresql/data + command: + - "postgres" + - "-c" + - "shared_preload_libraries=pg_stat_statements" + - "-c" + - "pg_stat_statements.track=all" healthcheck: test: ["CMD-SHELL", "pg_isready -U trails"] interval: 5s timeout: 5s retries: 5 + postgres-exporter: + image: prometheuscommunity/postgres-exporter:latest + restart: unless-stopped + environment: + DATA_SOURCE_NAME: postgresql://trails:${POSTGRES_PASSWORD:-trails}@postgres:5432/trails?sslmode=disable + depends_on: + postgres: + condition: service_healthy + + node-exporter: + image: prom/node-exporter:latest + restart: unless-stopped + pid: host + volumes: + - /proc:/host/proc:ro + - /sys:/host/sys:ro + - /:/rootfs:ro + command: + - "--path.procfs=/host/proc" + - "--path.sysfs=/host/sys" + - "--path.rootfs=/rootfs" + - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)" + + cadvisor: + image: gcr.io/cadvisor/cadvisor:latest + restart: unless-stopped + privileged: true + volumes: + - /:/rootfs:ro + - /var/run:/var/run:ro + - /sys:/sys:ro + - /var/lib/docker/:/var/lib/docker:ro + - /dev/disk/:/dev/disk:ro + prometheus: image: prom/prometheus:latest restart: unless-stopped @@ -114,6 +154,8 @@ services: GF_AUTH_GITHUB_ALLOWED_ORGANIZATIONS: trails-cool GF_AUTH_GITHUB_SCOPES: user:email,read:org GF_AUTH_DISABLE_LOGIN_FORM: "true" + GF_AUTH_GITHUB_ROLE_ATTRIBUTE_PATH: "contains(groups[*], '@trails-cool/owners') && 'GrafanaAdmin' || 'Editor'" + GF_AUTH_GITHUB_ALLOW_ASSIGN_GRAFANA_ADMIN: "true" volumes: - ./grafana/provisioning:/etc/grafana/provisioning:ro - ./grafana/dashboards:/var/lib/grafana/dashboards:ro diff --git a/infrastructure/grafana/dashboards/infrastructure.json b/infrastructure/grafana/dashboards/infrastructure.json index 9afe4fc..e69ed65 100644 --- a/infrastructure/grafana/dashboards/infrastructure.json +++ b/infrastructure/grafana/dashboards/infrastructure.json @@ -1,45 +1,87 @@ { - "dashboard": { - "title": "Infrastructure", - "uid": "trails-infra", - "timezone": "browser", - "refresh": "30s", - "panels": [ - { - "title": "Process Memory (RSS)", - "type": "timeseries", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, - "targets": [ - { "expr": "process_resident_memory_bytes", "legendFormat": "{{job}}" } - ], - "fieldConfig": { "defaults": { "unit": "bytes" } } + "title": "Infrastructure", + "uid": "trails-infra", + "timezone": "browser", + "refresh": "30s", + "panels": [ + { + "title": "Process Memory (RSS)", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 0 }, - { - "title": "CPU Usage", - "type": "timeseries", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }, - "targets": [ - { "expr": "rate(process_cpu_seconds_total[5m])", "legendFormat": "{{job}}" } - ], - "fieldConfig": { "defaults": { "unit": "percentunit" } } - }, - { - "title": "Event Loop Lag", - "type": "timeseries", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 }, - "targets": [ - { "expr": "nodejs_eventloop_lag_seconds", "legendFormat": "{{job}}" } - ], - "fieldConfig": { "defaults": { "unit": "s" } } - }, - { - "title": "Open File Descriptors", - "type": "timeseries", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 }, - "targets": [ - { "expr": "process_open_fds", "legendFormat": "{{job}}" } - ] + "targets": [ + { + "expr": "process_resident_memory_bytes", + "legendFormat": "{{job}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "bytes" + } } - ] - } -} + }, + { + "title": "CPU Usage", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 0 + }, + "targets": [ + { + "expr": "rate(process_cpu_seconds_total[5m])", + "legendFormat": "{{job}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percentunit" + } + } + }, + { + "title": "Event Loop Lag", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 8 + }, + "targets": [ + { + "expr": "nodejs_eventloop_lag_seconds", + "legendFormat": "{{job}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "s" + } + } + }, + { + "title": "Open File Descriptors", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 8 + }, + "targets": [ + { + "expr": "process_open_fds", + "legendFormat": "{{job}}" + } + ] + } + ] +} \ No newline at end of file diff --git a/infrastructure/grafana/dashboards/overview.json b/infrastructure/grafana/dashboards/overview.json index df6abcc..990e36b 100644 --- a/infrastructure/grafana/dashboards/overview.json +++ b/infrastructure/grafana/dashboards/overview.json @@ -1,57 +1,121 @@ { - "dashboard": { - "title": "trails.cool Overview", - "uid": "trails-overview", - "timezone": "browser", - "refresh": "30s", - "panels": [ - { - "title": "Request Rate", - "type": "timeseries", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, - "targets": [ - { - "expr": "sum(rate(http_request_duration_seconds_count[5m])) by (job)", - "legendFormat": "{{job}}" - } - ] + "title": "trails.cool Overview", + "uid": "trails-overview", + "timezone": "browser", + "refresh": "30s", + "panels": [ + { + "title": "Request Rate", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 0 }, - { - "title": "Error Rate", - "type": "timeseries", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }, - "targets": [ - { - "expr": "sum(rate(http_request_duration_seconds_count{status=~\"5..\"}[5m])) / sum(rate(http_request_duration_seconds_count[5m])) * 100", - "legendFormat": "Error %" - } - ], - "fieldConfig": { - "defaults": { "unit": "percent", "thresholds": { "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 5 }] } } + "targets": [ + { + "expr": "sum(rate(http_request_duration_seconds_count[5m])) by (job)", + "legendFormat": "{{job}}" } + ] + }, + { + "title": "Error Rate", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 0 }, - { - "title": "Latency p50 / p95 / p99", - "type": "timeseries", - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 8 }, - "targets": [ - { "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p50" }, - { "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p95" }, - { "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p99" } - ], - "fieldConfig": { "defaults": { "unit": "s" } } - }, - { - "title": "Health Status", - "type": "stat", - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 8 }, - "targets": [ - { "expr": "up", "legendFormat": "{{job}}" } - ], - "fieldConfig": { - "defaults": { "mappings": [{ "type": "value", "options": { "0": { "text": "DOWN", "color": "red" }, "1": { "text": "UP", "color": "green" } } }] } + "targets": [ + { + "expr": "sum(rate(http_request_duration_seconds_count{status=~\"5..\"}[5m])) / sum(rate(http_request_duration_seconds_count[5m])) * 100", + "legendFormat": "Error %" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "thresholds": { + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 5 + } + ] + } } } - ] - } -} + }, + { + "title": "Latency p50 / p95 / p99", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 8 + }, + "targets": [ + { + "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", + "legendFormat": "p50" + }, + { + "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", + "legendFormat": "p95" + }, + { + "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))", + "legendFormat": "p99" + } + ], + "fieldConfig": { + "defaults": { + "unit": "s" + } + } + }, + { + "title": "Health Status", + "type": "stat", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 8 + }, + "targets": [ + { + "expr": "up", + "legendFormat": "{{job}}" + } + ], + "fieldConfig": { + "defaults": { + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "DOWN", + "color": "red" + }, + "1": { + "text": "UP", + "color": "green" + } + } + } + ] + } + } + } + ] +} \ No newline at end of file diff --git a/infrastructure/grafana/dashboards/planner.json b/infrastructure/grafana/dashboards/planner.json index c248ba7..06b29ed 100644 --- a/infrastructure/grafana/dashboards/planner.json +++ b/infrastructure/grafana/dashboards/planner.json @@ -1,40 +1,81 @@ { - "dashboard": { - "title": "Planner", - "uid": "trails-planner", - "timezone": "browser", - "refresh": "30s", - "panels": [ - { - "title": "Active Sessions", - "type": "stat", - "gridPos": { "h": 6, "w": 6, "x": 0, "y": 0 }, - "targets": [{ "expr": "planner_active_sessions", "legendFormat": "Sessions" }] + "title": "Planner", + "uid": "trails-planner", + "timezone": "browser", + "refresh": "30s", + "panels": [ + { + "title": "Active Sessions", + "type": "stat", + "gridPos": { + "h": 6, + "w": 6, + "x": 0, + "y": 0 }, - { - "title": "Connected Clients", - "type": "stat", - "gridPos": { "h": 6, "w": 6, "x": 6, "y": 0 }, - "targets": [{ "expr": "planner_connected_clients", "legendFormat": "Clients" }] + "targets": [ + { + "expr": "planner_active_sessions", + "legendFormat": "Sessions" + } + ] + }, + { + "title": "Connected Clients", + "type": "stat", + "gridPos": { + "h": 6, + "w": 6, + "x": 6, + "y": 0 }, - { - "title": "BRouter Latency", - "type": "timeseries", - "gridPos": { "h": 6, "w": 12, "x": 12, "y": 0 }, - "targets": [ - { "expr": "histogram_quantile(0.50, rate(brouter_request_duration_seconds_bucket[5m]))", "legendFormat": "p50" }, - { "expr": "histogram_quantile(0.95, rate(brouter_request_duration_seconds_bucket[5m]))", "legendFormat": "p95" } - ], - "fieldConfig": { "defaults": { "unit": "s" } } + "targets": [ + { + "expr": "planner_connected_clients", + "legendFormat": "Clients" + } + ] + }, + { + "title": "BRouter Latency", + "type": "timeseries", + "gridPos": { + "h": 6, + "w": 12, + "x": 12, + "y": 0 }, - { - "title": "Request Rate by Route", - "type": "timeseries", - "gridPos": { "h": 8, "w": 24, "x": 0, "y": 6 }, - "targets": [ - { "expr": "sum(rate(http_request_duration_seconds_count{job=\"planner\"}[5m])) by (route)", "legendFormat": "{{route}}" } - ] + "targets": [ + { + "expr": "histogram_quantile(0.50, rate(brouter_request_duration_seconds_bucket[5m]))", + "legendFormat": "p50" + }, + { + "expr": "histogram_quantile(0.95, rate(brouter_request_duration_seconds_bucket[5m]))", + "legendFormat": "p95" + } + ], + "fieldConfig": { + "defaults": { + "unit": "s" + } } - ] - } -} + }, + { + "title": "Request Rate by Route", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 6 + }, + "targets": [ + { + "expr": "sum(rate(http_request_duration_seconds_count{job=\"planner\"}[5m])) by (route)", + "legendFormat": "{{route}}" + } + ] + } + ] +} \ No newline at end of file diff --git a/infrastructure/grafana/dashboards/service-health.json b/infrastructure/grafana/dashboards/service-health.json new file mode 100644 index 0000000..b9072cc --- /dev/null +++ b/infrastructure/grafana/dashboards/service-health.json @@ -0,0 +1,443 @@ +{ + "title": "Service Health", + "uid": "trails-service-health", + "timezone": "browser", + "refresh": "30s", + "panels": [ + { + "title": "PostgreSQL \u2014 Active Connections", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 0 + }, + "targets": [ + { + "expr": "pg_stat_activity_count{datname=\"trails\"}", + "legendFormat": "{{state}}" + } + ] + }, + { + "title": "PostgreSQL \u2014 Database Size", + "type": "stat", + "gridPos": { + "h": 8, + "w": 6, + "x": 12, + "y": 0 + }, + "targets": [ + { + "expr": "pg_database_size_bytes{datname=\"trails\"}", + "legendFormat": "trails" + } + ], + "fieldConfig": { + "defaults": { + "unit": "bytes" + } + } + }, + { + "title": "PostgreSQL \u2014 Transactions/sec", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 6, + "x": 18, + "y": 0 + }, + "targets": [ + { + "expr": "rate(pg_stat_database_xact_commit{datname=\"trails\"}[5m])", + "legendFormat": "commits/s" + }, + { + "expr": "rate(pg_stat_database_xact_rollback{datname=\"trails\"}[5m])", + "legendFormat": "rollbacks/s" + } + ] + }, + { + "title": "PostgreSQL \u2014 Slowest Queries (pg_stat_statements)", + "type": "table", + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 8 + }, + "targets": [ + { + "expr": "topk(10, pg_stat_statements_mean_time_seconds{datname=\"trails\"})", + "legendFormat": "{{query}}", + "format": "table", + "instant": true + } + ] + }, + { + "title": "PostgreSQL \u2014 Cache Hit Ratio", + "type": "gauge", + "gridPos": { + "h": 8, + "w": 6, + "x": 0, + "y": 16 + }, + "targets": [ + { + "expr": "pg_stat_database_blks_hit{datname=\"trails\"} / (pg_stat_database_blks_hit{datname=\"trails\"} + pg_stat_database_blks_read{datname=\"trails\"}) * 100", + "legendFormat": "hit %" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "min": 0, + "max": 100, + "thresholds": { + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "yellow", + "value": 90 + }, + { + "color": "green", + "value": 99 + } + ] + } + } + } + }, + { + "title": "PostgreSQL \u2014 Index Scans vs Sequential Scans", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 18, + "x": 6, + "y": 16 + }, + "targets": [ + { + "expr": "sum(rate(pg_stat_user_tables_idx_scan[5m]))", + "legendFormat": "Index scans/s" + }, + { + "expr": "sum(rate(pg_stat_user_tables_seq_scan[5m]))", + "legendFormat": "Sequential scans/s" + } + ] + }, + { + "title": "Host \u2014 Disk Usage", + "type": "gauge", + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 24 + }, + "targets": [ + { + "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"}) * 100", + "legendFormat": "Used %" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "min": 0, + "max": 100, + "thresholds": { + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 70 + }, + { + "color": "red", + "value": 85 + } + ] + } + } + } + }, + { + "title": "Host \u2014 CPU Usage", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 24 + }, + "targets": [ + { + "expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)", + "legendFormat": "CPU %" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent", + "max": 100 + } + } + }, + { + "title": "Host \u2014 Memory Usage", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 24 + }, + "targets": [ + { + "expr": "node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes", + "legendFormat": "Used" + }, + { + "expr": "node_memory_MemAvailable_bytes", + "legendFormat": "Available" + } + ], + "fieldConfig": { + "defaults": { + "unit": "bytes" + } + } + }, + { + "title": "Host \u2014 Network I/O", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 32 + }, + "targets": [ + { + "expr": "rate(node_network_receive_bytes_total{device!~\"lo|veth.*|br.*|docker.*\"}[5m])", + "legendFormat": "{{device}} rx" + }, + { + "expr": "-rate(node_network_transmit_bytes_total{device!~\"lo|veth.*|br.*|docker.*\"}[5m])", + "legendFormat": "{{device}} tx" + } + ], + "fieldConfig": { + "defaults": { + "unit": "Bps" + } + } + }, + { + "title": "Host \u2014 Disk I/O", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 32 + }, + "targets": [ + { + "expr": "rate(node_disk_read_bytes_total[5m])", + "legendFormat": "{{device}} read" + }, + { + "expr": "-rate(node_disk_written_bytes_total[5m])", + "legendFormat": "{{device}} write" + } + ], + "fieldConfig": { + "defaults": { + "unit": "Bps" + } + } + }, + { + "title": "BRouter \u2014 Request Latency (from Planner proxy)", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 40 + }, + "targets": [ + { + "expr": "histogram_quantile(0.50, rate(brouter_request_duration_seconds_bucket[5m]))", + "legendFormat": "p50" + }, + { + "expr": "histogram_quantile(0.95, rate(brouter_request_duration_seconds_bucket[5m]))", + "legendFormat": "p95" + }, + { + "expr": "histogram_quantile(0.99, rate(brouter_request_duration_seconds_bucket[5m]))", + "legendFormat": "p99" + } + ], + "fieldConfig": { + "defaults": { + "unit": "s" + } + } + }, + { + "title": "Container \u2014 CPU Usage (all containers)", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 40 + }, + "targets": [ + { + "expr": "rate(container_cpu_usage_seconds_total{name=~\"trails-cool.*\"}[5m]) * 100", + "legendFormat": "{{name}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "percent" + } + } + }, + { + "title": "Container \u2014 Memory Usage (all containers)", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 48 + }, + "targets": [ + { + "expr": "container_memory_usage_bytes{name=~\"trails-cool.*\"}", + "legendFormat": "{{name}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "bytes" + } + } + }, + { + "title": "BRouter \u2014 Container Resources", + "type": "stat", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 48 + }, + "targets": [ + { + "expr": "container_memory_usage_bytes{name=~\".*brouter.*\"}", + "legendFormat": "Memory" + }, + { + "expr": "rate(container_cpu_usage_seconds_total{name=~\".*brouter.*\"}[5m]) * 100", + "legendFormat": "CPU %" + } + ], + "fieldConfig": { + "defaults": { + "unit": "bytes" + } + } + }, + { + "title": "Caddy \u2014 Request Rate by Host", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 56 + }, + "targets": [ + { + "expr": "sum(rate(caddy_http_requests_total[5m])) by (server)", + "legendFormat": "{{server}}" + } + ] + }, + { + "title": "Caddy \u2014 Response Status Codes", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 56 + }, + "targets": [ + { + "expr": "sum(rate(caddy_http_responses_total[5m])) by (code)", + "legendFormat": "{{code}}" + } + ] + }, + { + "title": "Caddy \u2014 Request Duration by Host", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 64 + }, + "targets": [ + { + "expr": "histogram_quantile(0.95, sum(rate(caddy_http_request_duration_seconds_bucket[5m])) by (le, server))", + "legendFormat": "p95 {{server}}" + } + ], + "fieldConfig": { + "defaults": { + "unit": "s" + } + } + }, + { + "title": "Caddy \u2014 Active Connections", + "type": "timeseries", + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 64 + }, + "targets": [ + { + "expr": "caddy_http_requests_in_flight", + "legendFormat": "in-flight" + } + ] + } + ] +} \ No newline at end of file diff --git a/infrastructure/prometheus/prometheus.yml b/infrastructure/prometheus/prometheus.yml index f4d18e1..2bda741 100644 --- a/infrastructure/prometheus/prometheus.yml +++ b/infrastructure/prometheus/prometheus.yml @@ -13,6 +13,18 @@ scrape_configs: static_configs: - targets: ["planner:3001"] + - job_name: "postgres" + static_configs: + - targets: ["postgres-exporter:9187"] + + - job_name: "node" + static_configs: + - targets: ["node-exporter:9100"] + + - job_name: "cadvisor" + static_configs: + - targets: ["cadvisor:8080"] + - job_name: "caddy" static_configs: - targets: ["caddy:2019"] diff --git a/openspec/changes/sops-age-split-cd/.openspec.yaml b/openspec/changes/archive/2026-03-27-sops-age-split-cd/.openspec.yaml similarity index 100% rename from openspec/changes/sops-age-split-cd/.openspec.yaml rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/.openspec.yaml diff --git a/openspec/changes/sops-age-split-cd/design.md b/openspec/changes/archive/2026-03-27-sops-age-split-cd/design.md similarity index 100% rename from openspec/changes/sops-age-split-cd/design.md rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/design.md diff --git a/openspec/changes/sops-age-split-cd/proposal.md b/openspec/changes/archive/2026-03-27-sops-age-split-cd/proposal.md similarity index 100% rename from openspec/changes/sops-age-split-cd/proposal.md rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/proposal.md diff --git a/openspec/changes/sops-age-split-cd/specs/infrastructure/spec.md b/openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/infrastructure/spec.md similarity index 100% rename from openspec/changes/sops-age-split-cd/specs/infrastructure/spec.md rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/infrastructure/spec.md diff --git a/openspec/changes/sops-age-split-cd/specs/secret-management/spec.md b/openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/secret-management/spec.md similarity index 100% rename from openspec/changes/sops-age-split-cd/specs/secret-management/spec.md rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/secret-management/spec.md diff --git a/openspec/changes/sops-age-split-cd/tasks.md b/openspec/changes/archive/2026-03-27-sops-age-split-cd/tasks.md similarity index 87% rename from openspec/changes/sops-age-split-cd/tasks.md rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/tasks.md index 13c05f3..7f4851d 100644 --- a/openspec/changes/sops-age-split-cd/tasks.md +++ b/openspec/changes/archive/2026-03-27-sops-age-split-cd/tasks.md @@ -4,7 +4,7 @@ - [x] 1.2 Create `.sops.yaml` at repo root with age encryption rule for `secrets.*.env` - [x] 1.3 Create `infrastructure/secrets.app.env` with app secrets (POSTGRES_PASSWORD, JWT_SECRET, SESSION_SECRET, SMTP_URL, SMTP_FROM, SENTRY_AUTH_TOKEN, DEPLOY_GHCR_TOKEN), encrypt with `sops -e` - [x] 1.4 Create `infrastructure/secrets.infra.env` with infra secrets (GF_AUTH_GITHUB_CLIENT_ID, GF_AUTH_GITHUB_CLIENT_SECRET), encrypt with `sops -e` -- [ ] 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST) +- [x] 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST) ## 2. GitHub OAuth for Grafana @@ -30,7 +30,7 @@ ## 4. Verify - [x] 4.1 Test sops encrypt/decrypt cycle locally -- [ ] 4.2 Test cd-apps deploys only on app changes (not infra) -- [ ] 4.3 Test cd-infra deploys only on infra changes (not apps) -- [ ] 4.4 Test Grafana GitHub OAuth login -- [ ] 4.5 Verify old GitHub secrets can be removed after migration +- [x] 4.2 Test cd-apps deploys only on app changes (not infra) +- [x] 4.3 Test cd-infra deploys only on infra changes (not apps) +- [x] 4.4 Test Grafana GitHub OAuth login +- [x] 4.5 Verify old GitHub secrets can be removed after migration diff --git a/openspec/specs/infrastructure/spec.md b/openspec/specs/infrastructure/spec.md index 63d1180..78a90cf 100644 --- a/openspec/specs/infrastructure/spec.md +++ b/openspec/specs/infrastructure/spec.md @@ -52,11 +52,19 @@ The infrastructure SHALL support downloading and updating Germany RD5 segments f - **THEN** RD5 segments are updated from brouter.de and the BRouter container is restarted ### Requirement: CI/CD pipeline -GitHub Actions SHALL build, deploy, and security-scan both apps on push to main. +GitHub Actions SHALL use separate workflows for app deployment and infrastructure deployment, with secrets decrypted from a SOPS-encrypted file. -#### Scenario: Push triggers deployment -- **WHEN** code is pushed to the main branch -- **THEN** GitHub Actions builds Docker images, pushes to ghcr.io/trails-cool/, and deploys to the Hetzner server +#### Scenario: App deployment +- **WHEN** code changes are pushed to main in apps/ or packages/ +- **THEN** the cd-apps workflow builds Docker images, pushes to ghcr.io, and deploys app containers + +#### Scenario: Infrastructure deployment +- **WHEN** changes are pushed to main in infrastructure/ +- **THEN** the cd-infra workflow copies configs and restarts infrastructure services without rebuilding app images + +#### Scenario: Secret decryption at deploy time +- **WHEN** either CD workflow runs +- **THEN** the SOPS-encrypted secrets file is decrypted and provided to docker-compose as an env file #### Scenario: Gitleaks scan - **WHEN** a PR is opened @@ -102,3 +110,14 @@ The system SHALL enrich Sentry events with user and session context, use route-a #### Scenario: Source maps not served to clients - **WHEN** a client requests a `.map` file from the production server - **THEN** the server SHALL return 404 (source maps are uploaded to Sentry during build, not shipped in the bundle) + +### Requirement: Grafana authentication +Grafana SHALL authenticate users via GitHub OAuth, restricted to the trails-cool GitHub organization. + +#### Scenario: GitHub OAuth login +- **WHEN** a user navigates to grafana.internal.trails.cool +- **THEN** they are redirected to GitHub for authentication and granted access if they are a member of the trails-cool organization + +#### Scenario: No password-based login +- **WHEN** Grafana is deployed +- **THEN** the login form is disabled and only GitHub OAuth is available diff --git a/openspec/specs/secret-management/spec.md b/openspec/specs/secret-management/spec.md new file mode 100644 index 0000000..abaa0ae --- /dev/null +++ b/openspec/specs/secret-management/spec.md @@ -0,0 +1,16 @@ +## Requirements + +### Requirement: Encrypted secrets in repository +Production secrets SHALL be stored as a SOPS-encrypted file in the repository, decryptable only with a single age private key. + +#### Scenario: Edit secrets +- **WHEN** a developer runs `sops infrastructure/secrets.env` +- **THEN** the file is decrypted in a temporary editor, and re-encrypted on save + +#### Scenario: CD decryption +- **WHEN** the CD workflow runs +- **THEN** the encrypted secrets file is decrypted using the AGE_SECRET_KEY GitHub secret and provided to docker-compose as an env file + +#### Scenario: Secret audit trail +- **WHEN** a secret is changed +- **THEN** the change appears in git history as a diff of the encrypted file with a commit message describing what changed