From 424e692ee0ce0a34d9e895cf38e136c982387039 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ullrich=20Sch=C3=A4fer?= Date: Fri, 27 Mar 2026 18:44:14 +0100 Subject: [PATCH] Add service health monitoring: postgres, node, cAdvisor exporters + dashboard Exporters: - postgres_exporter: DB connections, transactions, cache hit ratio, query stats - node_exporter: host CPU, memory, disk, network - cAdvisor: per-container CPU and memory usage PostgreSQL: - Enable pg_stat_statements for query-level performance tracking - Track index scans vs sequential scans, cache hit ratio Dashboard (service-health.json): - DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage - Host: disk gauge, CPU, memory, network I/O, disk I/O - BRouter: request latency p50/p95/p99, container CPU + memory - All containers: CPU and memory comparison Co-Authored-By: Claude Opus 4.6 (1M context) --- .github/workflows/cd-infra.yml | 13 +- CLAUDE.md | 26 +++ infrastructure/Caddyfile | 7 + infrastructure/docker-compose.yml | 40 ++++ .../grafana/dashboards/service-health.json | 187 ++++++++++++++++++ infrastructure/prometheus/prometheus.yml | 12 ++ .../.openspec.yaml | 0 .../2026-03-27-sops-age-split-cd}/design.md | 0 .../2026-03-27-sops-age-split-cd}/proposal.md | 0 .../specs/infrastructure/spec.md | 0 .../specs/secret-management/spec.md | 0 .../2026-03-27-sops-age-split-cd}/tasks.md | 10 +- openspec/specs/infrastructure/spec.md | 27 ++- openspec/specs/secret-management/spec.md | 16 ++ 14 files changed, 326 insertions(+), 12 deletions(-) create mode 100644 infrastructure/grafana/dashboards/service-health.json rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/.openspec.yaml (100%) rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/design.md (100%) rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/proposal.md (100%) rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/specs/infrastructure/spec.md (100%) rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/specs/secret-management/spec.md (100%) rename openspec/changes/{sops-age-split-cd => archive/2026-03-27-sops-age-split-cd}/tasks.md (87%) create mode 100644 openspec/specs/secret-management/spec.md diff --git a/.github/workflows/cd-infra.yml b/.github/workflows/cd-infra.yml index 82194a8..ccd8f36 100644 --- a/.github/workflows/cd-infra.yml +++ b/.github/workflows/cd-infra.yml @@ -68,12 +68,19 @@ jobs: GHCR_TOKEN=$(grep DEPLOY_GHCR_TOKEN .env | cut -d= -f2-) echo "$GHCR_TOKEN" | docker login ghcr.io -u stigi --password-stdin 2>/dev/null || true - # Restart services - if [ "${{ github.event.inputs.restart_all }}" = "true" ]; then + # Check if full restart requested via: + # - workflow_dispatch input: restart_all=true + # - commit message containing [restart-all] + RESTART_ALL="${{ github.event.inputs.restart_all }}" + if [ "$RESTART_ALL" != "true" ] && echo "${{ github.event.head_commit.message }}" | grep -q '\[restart-all\]'; then + RESTART_ALL="true" + fi + + if [ "$RESTART_ALL" = "true" ]; then docker compose --env-file .env up -d --remove-orphans else # Restart infra services (except Caddy — just reload its config) - docker compose --env-file .env up -d postgres prometheus loki grafana + docker compose --env-file .env up -d postgres prometheus loki grafana postgres-exporter node-exporter cadvisor docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile fi diff --git a/CLAUDE.md b/CLAUDE.md index c18798d..b73a560 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -140,6 +140,32 @@ before pushing to an existing PR branch. ### Emergency override Admins can bypass the PR workflow when necessary (e.g., CI is broken and needs a hotfix). Document the reason in the commit message. +## Deployment + +Three separate CD workflows triggered by path: + +| Workflow | Triggers on | Deploys | +|----------|-------------|---------| +| `cd-apps.yml` | `apps/`, `packages/`, `pnpm-lock.yaml` | journal, planner | +| `cd-infra.yml` | `infrastructure/` | caddy, postgres, prometheus, loki, grafana, exporters | +| `cd-brouter.yml` | `docker/brouter/` | brouter | + +### Secrets +All secrets are stored in SOPS-encrypted files (`infrastructure/secrets.app.env`, `infrastructure/secrets.infra.env`). Edit with `sops infrastructure/secrets.app.env`. Only `AGE_SECRET_KEY`, `DEPLOY_SSH_KEY`, and `DEPLOY_HOST` remain as GitHub secrets. + +### Full restart +To restart **all** containers (not just the ones a workflow normally touches), either: +- Add `[restart-all]` to the commit message +- Or trigger manually: `gh workflow run cd-infra.yml -f restart_all=true` + +### Server access +```bash +ssh -i ~/.ssh/trails-cool-deploy root@trails.cool +``` + +### Grafana +`https://grafana.internal.trails.cool` — GitHub OAuth (trails-cool org) + ## OpenSpec Workflow Specs live in `openspec/`. Use these slash commands: diff --git a/infrastructure/Caddyfile b/infrastructure/Caddyfile index 70513bd..84d4eaf 100644 --- a/infrastructure/Caddyfile +++ b/infrastructure/Caddyfile @@ -1,3 +1,10 @@ +{ + servers { + metrics + } + admin 0.0.0.0:2019 +} + (security_headers) { header { Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" diff --git a/infrastructure/docker-compose.yml b/infrastructure/docker-compose.yml index 9c20fcf..8955e99 100644 --- a/infrastructure/docker-compose.yml +++ b/infrastructure/docker-compose.yml @@ -78,12 +78,52 @@ services: POSTGRES_DB: trails volumes: - pgdata:/var/lib/postgresql/data + command: + - "postgres" + - "-c" + - "shared_preload_libraries=pg_stat_statements" + - "-c" + - "pg_stat_statements.track=all" healthcheck: test: ["CMD-SHELL", "pg_isready -U trails"] interval: 5s timeout: 5s retries: 5 + postgres-exporter: + image: prometheuscommunity/postgres-exporter:latest + restart: unless-stopped + environment: + DATA_SOURCE_NAME: postgresql://trails:${POSTGRES_PASSWORD:-trails}@postgres:5432/trails?sslmode=disable + depends_on: + postgres: + condition: service_healthy + + node-exporter: + image: prom/node-exporter:latest + restart: unless-stopped + pid: host + volumes: + - /proc:/host/proc:ro + - /sys:/host/sys:ro + - /:/rootfs:ro + command: + - "--path.procfs=/host/proc" + - "--path.sysfs=/host/sys" + - "--path.rootfs=/rootfs" + - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)" + + cadvisor: + image: gcr.io/cadvisor/cadvisor:latest + restart: unless-stopped + privileged: true + volumes: + - /:/rootfs:ro + - /var/run:/var/run:ro + - /sys:/sys:ro + - /var/lib/docker/:/var/lib/docker:ro + - /dev/disk/:/dev/disk:ro + prometheus: image: prom/prometheus:latest restart: unless-stopped diff --git a/infrastructure/grafana/dashboards/service-health.json b/infrastructure/grafana/dashboards/service-health.json new file mode 100644 index 0000000..c5d68cb --- /dev/null +++ b/infrastructure/grafana/dashboards/service-health.json @@ -0,0 +1,187 @@ +{ + "dashboard": { + "title": "Service Health", + "uid": "trails-service-health", + "timezone": "browser", + "refresh": "30s", + "panels": [ + { + "title": "PostgreSQL — Active Connections", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }, + "targets": [ + { "expr": "pg_stat_activity_count{datname=\"trails\"}", "legendFormat": "{{state}}" } + ] + }, + { + "title": "PostgreSQL — Database Size", + "type": "stat", + "gridPos": { "h": 8, "w": 6, "x": 12, "y": 0 }, + "targets": [ + { "expr": "pg_database_size_bytes{datname=\"trails\"}", "legendFormat": "trails" } + ], + "fieldConfig": { "defaults": { "unit": "bytes" } } + }, + { + "title": "PostgreSQL — Transactions/sec", + "type": "timeseries", + "gridPos": { "h": 8, "w": 6, "x": 18, "y": 0 }, + "targets": [ + { "expr": "rate(pg_stat_database_xact_commit{datname=\"trails\"}[5m])", "legendFormat": "commits/s" }, + { "expr": "rate(pg_stat_database_xact_rollback{datname=\"trails\"}[5m])", "legendFormat": "rollbacks/s" } + ] + }, + { + "title": "PostgreSQL — Slowest Queries (pg_stat_statements)", + "type": "table", + "gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 }, + "targets": [ + { + "expr": "topk(10, pg_stat_statements_mean_time_seconds{datname=\"trails\"})", + "legendFormat": "{{query}}", + "format": "table", + "instant": true + } + ] + }, + { + "title": "PostgreSQL — Cache Hit Ratio", + "type": "gauge", + "gridPos": { "h": 8, "w": 6, "x": 0, "y": 16 }, + "targets": [ + { "expr": "pg_stat_database_blks_hit{datname=\"trails\"} / (pg_stat_database_blks_hit{datname=\"trails\"} + pg_stat_database_blks_read{datname=\"trails\"}) * 100", "legendFormat": "hit %" } + ], + "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100, "thresholds": { "steps": [{ "color": "red", "value": null }, { "color": "yellow", "value": 90 }, { "color": "green", "value": 99 }] } } } + }, + { + "title": "PostgreSQL — Index Scans vs Sequential Scans", + "type": "timeseries", + "gridPos": { "h": 8, "w": 18, "x": 6, "y": 16 }, + "targets": [ + { "expr": "sum(rate(pg_stat_user_tables_idx_scan[5m]))", "legendFormat": "Index scans/s" }, + { "expr": "sum(rate(pg_stat_user_tables_seq_scan[5m]))", "legendFormat": "Sequential scans/s" } + ] + }, + { + "title": "Host — Disk Usage", + "type": "gauge", + "gridPos": { "h": 8, "w": 8, "x": 0, "y": 24 }, + "targets": [ + { "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/\"} / node_filesystem_size_bytes{mountpoint=\"/\"}) * 100", "legendFormat": "Used %" } + ], + "fieldConfig": { "defaults": { "unit": "percent", "min": 0, "max": 100, "thresholds": { "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 70 }, { "color": "red", "value": 85 }] } } } + }, + { + "title": "Host — CPU Usage", + "type": "timeseries", + "gridPos": { "h": 8, "w": 8, "x": 8, "y": 24 }, + "targets": [ + { "expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)", "legendFormat": "CPU %" } + ], + "fieldConfig": { "defaults": { "unit": "percent", "max": 100 } } + }, + { + "title": "Host — Memory Usage", + "type": "timeseries", + "gridPos": { "h": 8, "w": 8, "x": 16, "y": 24 }, + "targets": [ + { "expr": "node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes", "legendFormat": "Used" }, + { "expr": "node_memory_MemAvailable_bytes", "legendFormat": "Available" } + ], + "fieldConfig": { "defaults": { "unit": "bytes" } } + }, + { + "title": "Host — Network I/O", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 32 }, + "targets": [ + { "expr": "rate(node_network_receive_bytes_total{device!~\"lo|veth.*|br.*|docker.*\"}[5m])", "legendFormat": "{{device}} rx" }, + { "expr": "-rate(node_network_transmit_bytes_total{device!~\"lo|veth.*|br.*|docker.*\"}[5m])", "legendFormat": "{{device}} tx" } + ], + "fieldConfig": { "defaults": { "unit": "Bps" } } + }, + { + "title": "Host — Disk I/O", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 32 }, + "targets": [ + { "expr": "rate(node_disk_read_bytes_total[5m])", "legendFormat": "{{device}} read" }, + { "expr": "-rate(node_disk_written_bytes_total[5m])", "legendFormat": "{{device}} write" } + ], + "fieldConfig": { "defaults": { "unit": "Bps" } } + }, + { + "title": "BRouter — Request Latency (from Planner proxy)", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 40 }, + "targets": [ + { "expr": "histogram_quantile(0.50, rate(brouter_request_duration_seconds_bucket[5m]))", "legendFormat": "p50" }, + { "expr": "histogram_quantile(0.95, rate(brouter_request_duration_seconds_bucket[5m]))", "legendFormat": "p95" }, + { "expr": "histogram_quantile(0.99, rate(brouter_request_duration_seconds_bucket[5m]))", "legendFormat": "p99" } + ], + "fieldConfig": { "defaults": { "unit": "s" } } + }, + { + "title": "Container — CPU Usage (all containers)", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 40 }, + "targets": [ + { "expr": "rate(container_cpu_usage_seconds_total{name=~\"trails-cool.*\"}[5m]) * 100", "legendFormat": "{{name}}" } + ], + "fieldConfig": { "defaults": { "unit": "percent" } } + }, + { + "title": "Container — Memory Usage (all containers)", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 48 }, + "targets": [ + { "expr": "container_memory_usage_bytes{name=~\"trails-cool.*\"}", "legendFormat": "{{name}}" } + ], + "fieldConfig": { "defaults": { "unit": "bytes" } } + }, + { + "title": "BRouter — Container Resources", + "type": "stat", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 48 }, + "targets": [ + { "expr": "container_memory_usage_bytes{name=~\".*brouter.*\"}", "legendFormat": "Memory" }, + { "expr": "rate(container_cpu_usage_seconds_total{name=~\".*brouter.*\"}[5m]) * 100", "legendFormat": "CPU %" } + ], + "fieldConfig": { "defaults": { "unit": "bytes" } } + }, + { + "title": "Caddy — Request Rate by Host", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 56 }, + "targets": [ + { "expr": "sum(rate(caddy_http_requests_total[5m])) by (server)", "legendFormat": "{{server}}" } + ] + }, + { + "title": "Caddy — Response Status Codes", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 56 }, + "targets": [ + { "expr": "sum(rate(caddy_http_responses_total[5m])) by (code)", "legendFormat": "{{code}}" } + ] + }, + { + "title": "Caddy — Request Duration by Host", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 0, "y": 64 }, + "targets": [ + { "expr": "histogram_quantile(0.95, sum(rate(caddy_http_request_duration_seconds_bucket[5m])) by (le, server))", "legendFormat": "p95 {{server}}" } + ], + "fieldConfig": { "defaults": { "unit": "s" } } + }, + { + "title": "Caddy — Active Connections", + "type": "timeseries", + "gridPos": { "h": 8, "w": 12, "x": 12, "y": 64 }, + "targets": [ + { "expr": "caddy_http_requests_in_flight", "legendFormat": "in-flight" } + ] + } + ] + } +} diff --git a/infrastructure/prometheus/prometheus.yml b/infrastructure/prometheus/prometheus.yml index f4d18e1..2bda741 100644 --- a/infrastructure/prometheus/prometheus.yml +++ b/infrastructure/prometheus/prometheus.yml @@ -13,6 +13,18 @@ scrape_configs: static_configs: - targets: ["planner:3001"] + - job_name: "postgres" + static_configs: + - targets: ["postgres-exporter:9187"] + + - job_name: "node" + static_configs: + - targets: ["node-exporter:9100"] + + - job_name: "cadvisor" + static_configs: + - targets: ["cadvisor:8080"] + - job_name: "caddy" static_configs: - targets: ["caddy:2019"] diff --git a/openspec/changes/sops-age-split-cd/.openspec.yaml b/openspec/changes/archive/2026-03-27-sops-age-split-cd/.openspec.yaml similarity index 100% rename from openspec/changes/sops-age-split-cd/.openspec.yaml rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/.openspec.yaml diff --git a/openspec/changes/sops-age-split-cd/design.md b/openspec/changes/archive/2026-03-27-sops-age-split-cd/design.md similarity index 100% rename from openspec/changes/sops-age-split-cd/design.md rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/design.md diff --git a/openspec/changes/sops-age-split-cd/proposal.md b/openspec/changes/archive/2026-03-27-sops-age-split-cd/proposal.md similarity index 100% rename from openspec/changes/sops-age-split-cd/proposal.md rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/proposal.md diff --git a/openspec/changes/sops-age-split-cd/specs/infrastructure/spec.md b/openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/infrastructure/spec.md similarity index 100% rename from openspec/changes/sops-age-split-cd/specs/infrastructure/spec.md rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/infrastructure/spec.md diff --git a/openspec/changes/sops-age-split-cd/specs/secret-management/spec.md b/openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/secret-management/spec.md similarity index 100% rename from openspec/changes/sops-age-split-cd/specs/secret-management/spec.md rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/specs/secret-management/spec.md diff --git a/openspec/changes/sops-age-split-cd/tasks.md b/openspec/changes/archive/2026-03-27-sops-age-split-cd/tasks.md similarity index 87% rename from openspec/changes/sops-age-split-cd/tasks.md rename to openspec/changes/archive/2026-03-27-sops-age-split-cd/tasks.md index 13c05f3..7f4851d 100644 --- a/openspec/changes/sops-age-split-cd/tasks.md +++ b/openspec/changes/archive/2026-03-27-sops-age-split-cd/tasks.md @@ -4,7 +4,7 @@ - [x] 1.2 Create `.sops.yaml` at repo root with age encryption rule for `secrets.*.env` - [x] 1.3 Create `infrastructure/secrets.app.env` with app secrets (POSTGRES_PASSWORD, JWT_SECRET, SESSION_SECRET, SMTP_URL, SMTP_FROM, SENTRY_AUTH_TOKEN, DEPLOY_GHCR_TOKEN), encrypt with `sops -e` - [x] 1.4 Create `infrastructure/secrets.infra.env` with infra secrets (GF_AUTH_GITHUB_CLIENT_ID, GF_AUTH_GITHUB_CLIENT_SECRET), encrypt with `sops -e` -- [ ] 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST) +- [x] 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST) ## 2. GitHub OAuth for Grafana @@ -30,7 +30,7 @@ ## 4. Verify - [x] 4.1 Test sops encrypt/decrypt cycle locally -- [ ] 4.2 Test cd-apps deploys only on app changes (not infra) -- [ ] 4.3 Test cd-infra deploys only on infra changes (not apps) -- [ ] 4.4 Test Grafana GitHub OAuth login -- [ ] 4.5 Verify old GitHub secrets can be removed after migration +- [x] 4.2 Test cd-apps deploys only on app changes (not infra) +- [x] 4.3 Test cd-infra deploys only on infra changes (not apps) +- [x] 4.4 Test Grafana GitHub OAuth login +- [x] 4.5 Verify old GitHub secrets can be removed after migration diff --git a/openspec/specs/infrastructure/spec.md b/openspec/specs/infrastructure/spec.md index 63d1180..78a90cf 100644 --- a/openspec/specs/infrastructure/spec.md +++ b/openspec/specs/infrastructure/spec.md @@ -52,11 +52,19 @@ The infrastructure SHALL support downloading and updating Germany RD5 segments f - **THEN** RD5 segments are updated from brouter.de and the BRouter container is restarted ### Requirement: CI/CD pipeline -GitHub Actions SHALL build, deploy, and security-scan both apps on push to main. +GitHub Actions SHALL use separate workflows for app deployment and infrastructure deployment, with secrets decrypted from a SOPS-encrypted file. -#### Scenario: Push triggers deployment -- **WHEN** code is pushed to the main branch -- **THEN** GitHub Actions builds Docker images, pushes to ghcr.io/trails-cool/, and deploys to the Hetzner server +#### Scenario: App deployment +- **WHEN** code changes are pushed to main in apps/ or packages/ +- **THEN** the cd-apps workflow builds Docker images, pushes to ghcr.io, and deploys app containers + +#### Scenario: Infrastructure deployment +- **WHEN** changes are pushed to main in infrastructure/ +- **THEN** the cd-infra workflow copies configs and restarts infrastructure services without rebuilding app images + +#### Scenario: Secret decryption at deploy time +- **WHEN** either CD workflow runs +- **THEN** the SOPS-encrypted secrets file is decrypted and provided to docker-compose as an env file #### Scenario: Gitleaks scan - **WHEN** a PR is opened @@ -102,3 +110,14 @@ The system SHALL enrich Sentry events with user and session context, use route-a #### Scenario: Source maps not served to clients - **WHEN** a client requests a `.map` file from the production server - **THEN** the server SHALL return 404 (source maps are uploaded to Sentry during build, not shipped in the bundle) + +### Requirement: Grafana authentication +Grafana SHALL authenticate users via GitHub OAuth, restricted to the trails-cool GitHub organization. + +#### Scenario: GitHub OAuth login +- **WHEN** a user navigates to grafana.internal.trails.cool +- **THEN** they are redirected to GitHub for authentication and granted access if they are a member of the trails-cool organization + +#### Scenario: No password-based login +- **WHEN** Grafana is deployed +- **THEN** the login form is disabled and only GitHub OAuth is available diff --git a/openspec/specs/secret-management/spec.md b/openspec/specs/secret-management/spec.md new file mode 100644 index 0000000..abaa0ae --- /dev/null +++ b/openspec/specs/secret-management/spec.md @@ -0,0 +1,16 @@ +## Requirements + +### Requirement: Encrypted secrets in repository +Production secrets SHALL be stored as a SOPS-encrypted file in the repository, decryptable only with a single age private key. + +#### Scenario: Edit secrets +- **WHEN** a developer runs `sops infrastructure/secrets.env` +- **THEN** the file is decrypted in a temporary editor, and re-encrypted on save + +#### Scenario: CD decryption +- **WHEN** the CD workflow runs +- **THEN** the encrypted secrets file is decrypted using the AGE_SECRET_KEY GitHub secret and provided to docker-compose as an env file + +#### Scenario: Secret audit trail +- **WHEN** a secret is changed +- **THEN** the change appears in git history as a diff of the encrypted file with a commit message describing what changed