trails-cool/trails
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
trails/openspec/changes/archive/2026-03-27-sops-age-split-cd/tasks.md
Ullrich Schäfer 424e692ee0
Add service health monitoring: postgres, node, cAdvisor exporters + dashboard 
Exporters:
- postgres_exporter: DB connections, transactions, cache hit ratio, query stats
- node_exporter: host CPU, memory, disk, network
- cAdvisor: per-container CPU and memory usage

PostgreSQL:
- Enable pg_stat_statements for query-level performance tracking
- Track index scans vs sequential scans, cache hit ratio

Dashboard (service-health.json):
- DB: connections, size, transactions/s, slow queries, cache hit ratio, index usage
- Host: disk gauge, CPU, memory, network I/O, disk I/O
- BRouter: request latency p50/p95/p99, container CPU + memory
- All containers: CPU and memory comparison

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 18:50:42 +01:00

2.3 KiB
Raw Blame History

1. SOPS + age Setup

  • 1.1 Generate age key pair (age-keygen), store public key in .sops.yaml, private key as AGE_SECRET_KEY GitHub secret
  • 1.2 Create .sops.yaml at repo root with age encryption rule for secrets.*.env
  • 1.3 Create infrastructure/secrets.app.env with app secrets (POSTGRES_PASSWORD, JWT_SECRET, SESSION_SECRET, SMTP_URL, SMTP_FROM, SENTRY_AUTH_TOKEN, DEPLOY_GHCR_TOKEN), encrypt with sops -e
  • 1.4 Create infrastructure/secrets.infra.env with infra secrets (GF_AUTH_GITHUB_CLIENT_ID, GF_AUTH_GITHUB_CLIENT_SECRET), encrypt with sops -e
  • 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST)

2. GitHub OAuth for Grafana

  • 2.1 Register GitHub OAuth app (callback: grafana.internal.trails.cool/login/github)
  • 2.2 Add GF_AUTH_GITHUB_CLIENT_ID and GF_AUTH_GITHUB_CLIENT_SECRET to secrets.env
  • 2.3 Update docker-compose.yml: add GitHub OAuth env vars to Grafana, remove GF_SECURITY_ADMIN_USER/PASSWORD
  • 2.4 Update Caddyfile: remove basic_auth block for grafana.internal, just reverse_proxy
  • 2.5 Remove Caddy GRAFANA_USER/GRAFANA_PASSWORD_HASH env vars from docker-compose.yml

3. Split CD Workflows

  • 3.1 Create .github/workflows/cd-apps.yml — triggered by apps/, packages/, pnpm-lock.yaml changes
  • 3.2 Create .github/workflows/cd-infra.yml — triggered by infrastructure/ changes
  • 3.3 Add sops + age install step to both workflows
  • 3.4 cd-apps decrypt step: sops -d secrets.app.env > .env, SCP to server
  • 3.5 cd-infra decrypt step: merge secrets.app.env + secrets.infra.env into .env, SCP to server
  • 3.6 cd-apps: build images, push, SSH deploy (pull, migrate, restart apps)
  • 3.7 cd-infra: copy configs, SSH deploy (restart infra services)
  • 3.8 Remove old cd.yml workflow
  • 3.9 Remove GRAFANA_* exports and password reset from deploy script
  • 3.10 Both workflows use docker compose --env-file .env up -d instead of inline exports

4. Verify

  • 4.1 Test sops encrypt/decrypt cycle locally
  • 4.2 Test cd-apps deploys only on app changes (not infra)
  • 4.3 Test cd-infra deploys only on infra changes (not apps)
  • 4.4 Test Grafana GitHub OAuth login
  • 4.5 Verify old GitHub secrets can be removed after migration