SOPS+age secrets, split CD workflows, GitHub OAuth for Grafana

Secrets: - Add .sops.yaml with age encryption config - Add encrypted secrets.app.env (app secrets) and secrets.infra.env (Grafana OAuth) - CD decrypts at deploy time with AGE_SECRET_KEY — all other secrets move out of GitHub Actions into version-controlled encrypted files Split CD: - cd-apps.yml: triggered by apps/packages changes, builds Docker images, deploys apps - cd-infra.yml: triggered by infrastructure/ changes, copies configs, restarts services - Remove monolithic cd.yml Grafana auth: - GitHub OAuth (trails-cool org), disable login form - Remove Caddy basic_auth block and all GRAFANA_* env vars/secrets Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 16:56:01 +01:00 · 2026-03-27 16:56:01 +01:00 · 57094323d2
commit 57094323d2
parent 206f6ac941
15 changed files with 541 additions and 138 deletions
--- a/openspec/changes/sops-age-split-cd/.openspec.yaml
+++ b/openspec/changes/sops-age-split-cd/.openspec.yaml
@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-03-27
--- a/openspec/changes/sops-age-split-cd/design.md
+++ b/openspec/changes/sops-age-split-cd/design.md
@ -0,0 +1,126 @@
+## Context
+
+The current CD workflow has ~10 GitHub Actions secrets passed via `export`
+statements in a deploy script. Some secrets contain `$` characters that get
+mangled. Grafana authentication required managing a bcrypt hash, a plaintext
+password, and a username across three different systems (Caddy, Grafana, CD).
+The single CD workflow rebuilds Docker images even for config-only changes.
+
+## Goals / Non-Goals
+
+**Goals:**
+- All secrets in one encrypted file in the repo (SOPS + age)
+- Only one GitHub secret needed for decryption (AGE_SECRET_KEY)
+- App deploys (Docker build + push) separate from infra deploys (config copy)
+- Grafana login via GitHub OAuth (no passwords)
+- Remove Caddy basic auth for Grafana
+
+**Non-Goals:**
+- Secret rotation automation (manual for now)
+- Per-environment secret files (only production)
+- Grafana RBAC / team-based access (just org membership check)
+- Moving DEPLOY_SSH_KEY or GITHUB_TOKEN into SOPS (these are GitHub-native)
+
+## Decisions
+
+### D1: SOPS + age file-based encryption
+
+Create `.sops.yaml` at the repo root defining age as the encryption method.
+Two encrypted secret files, split by deployment scope:
+
+- **`infrastructure/secrets.app.env`** — app secrets needed by journal/planner.
+  Used by both cd-apps (production) and future staging deploys.
+- **`infrastructure/secrets.infra.env`** — monitoring/Grafana secrets.
+  Used only by cd-infra (production). Staging doesn't run Grafana.
+
+```yaml
+# .sops.yaml
+creation_rules:
+  - path_regex: secrets\..*\.env$
+    age: <public-key>
+```
+
+Workflow:
+- **Edit secrets**: `sops infrastructure/secrets.app.env` (decrypts in editor,
+  re-encrypts on save)
+- **CD decrypts**: Install sops + age, decrypt the relevant file(s) to a temp
+  `.env`, pass to `docker compose --env-file`
+- **Only one GitHub secret**: `AGE_SECRET_KEY` (the private key)
+
+**secrets.app.env** contents (used by cd-apps + cd-infra):
+- POSTGRES_PASSWORD, JWT_SECRET, SESSION_SECRET
+- SMTP_URL, SMTP_FROM
+- SENTRY_AUTH_TOKEN
+- DEPLOY_GHCR_TOKEN
+
+**secrets.infra.env** contents (used by cd-infra only):
+- GF_AUTH_GITHUB_CLIENT_ID, GF_AUTH_GITHUB_CLIENT_SECRET
+
+Secrets that stay as GitHub Actions secrets:
+- DEPLOY_SSH_KEY (used by SCP/SSH actions, not by docker-compose)
+- AGE_SECRET_KEY (chicken-and-egg: can't encrypt the decryption key)
+- DEPLOY_HOST (not really secret, but convenient)
+
+### D2: Split CD into two workflows
+
+**cd-apps.yml** — triggered by changes to `apps/`, `packages/`, `pnpm-lock.yaml`:
+1. Build and push Docker images (journal, planner, brouter)
+2. SSH to server, pull images, run migrations, restart app containers
+
+**cd-infra.yml** — triggered by changes to `infrastructure/`:
+1. Copy config files (docker-compose, Caddyfile, Prometheus, Loki, Grafana)
+2. SSH to server, decrypt secrets, `docker compose up -d`
+
+Both workflows also trigger on `workflow_dispatch` for manual runs.
+Both share the decrypt-secrets step.
+
+### D3: GitHub OAuth for Grafana
+
+Register a GitHub OAuth app at github.com/settings/applications:
+- Callback URL: `https://grafana.internal.trails.cool/login/github`
+- Homepage: `https://grafana.internal.trails.cool`
+
+Grafana config via environment variables:
+```yaml
+GF_AUTH_GITHUB_ENABLED: "true"
+GF_AUTH_GITHUB_CLIENT_ID: <from-sops>
+GF_AUTH_GITHUB_CLIENT_SECRET: <from-sops>
+GF_AUTH_GITHUB_ALLOWED_ORGANIZATIONS: trails-cool
+GF_AUTH_GITHUB_SCOPES: user:email,read:org
+GF_AUTH_DISABLE_LOGIN_FORM: "true"
+```
+
+Remove from Caddyfile:
+- `basic_auth` block for grafana.internal
+- GRAFANA_USER, GRAFANA_PASSWORD, GRAFANA_PASSWORD_HASH env vars
+
+Remove from docker-compose.yml:
+- GF_SECURITY_ADMIN_USER, GF_SECURITY_ADMIN_PASSWORD
+- Caddy GRAFANA_USER, GRAFANA_PASSWORD_HASH env vars
+
+Remove from CD:
+- grafana cli password reset step
+- All GRAFANA_* secret exports
+
+### D4: Server-side secret decryption
+
+The CD deploy step:
+1. Install sops + age on the runner
+2. Decrypt: `SOPS_AGE_KEY=${{ secrets.AGE_SECRET_KEY }} sops -d infrastructure/secrets.env > /tmp/secrets.env`
+3. SCP the decrypted `.env` to server as `/opt/trails-cool/.env`
+4. `docker compose --env-file .env up -d`
+5. Clean up `/tmp/secrets.env` from runner
+
+The server never has the age private key — secrets arrive as a plain `.env`
+file via SCP (same security as current `export` approach, but now
+version-controlled and auditable).
+
+## Risks / Trade-offs
+
+- **age key loss** → If the AGE_SECRET_KEY is lost, secrets can't be decrypted.
+  Mitigate: store a backup of the age key outside GitHub (e.g., password
+  manager).
+- **Encrypted file merge conflicts** → SOPS encrypted files don't merge well.
+  Mitigate: only one person edits secrets at a time (fine for solo/small team).
+- **GitHub OAuth requires internet** → If GitHub is down, Grafana login fails.
+  Acceptable for a monitoring dashboard.
--- a/openspec/changes/sops-age-split-cd/proposal.md
+++ b/openspec/changes/sops-age-split-cd/proposal.md
@ -0,0 +1,44 @@
+## Why
+
+Secrets are scattered across GitHub Actions secrets with no version control,
+no audit trail, and painful manual management (the Grafana password hash saga).
+The CD pipeline is monolithic — changing a Grafana dashboard rebuilds both app
+Docker images. And Grafana authentication requires managing bcrypt hashes and
+basic auth layers.
+
+## What Changes
+
+- **SOPS + age for secrets**: Encrypt a `.env.production` file in the repo.
+  CD decrypts at deploy time with a single age private key stored as one
+  GitHub secret. All other secrets move from GitHub Actions secrets into the
+  encrypted file — version-controlled, diffable, auditable.
+- **Split CD into apps vs infra**: Two workflows triggered by path filters.
+  App changes (apps/, packages/) build Docker images and deploy. Infra changes
+  (infrastructure/) copy configs and restart services. No unnecessary rebuilds.
+- **GitHub OAuth for Grafana**: Replace Caddy basic auth + Grafana login with
+  GitHub OAuth. One login, restricted to the trails-cool GitHub org. Remove
+  GRAFANA_PASSWORD_HASH, GRAFANA_USER, GRAFANA_PASSWORD secrets entirely.
+
+## Capabilities
+
+### New Capabilities
+
+- `secret-management`: SOPS + age encrypted secrets in the repository with
+  CD decryption
+
+### Modified Capabilities
+
+- `infrastructure`: Split CD workflows, GitHub OAuth for Grafana, remove
+  Caddy basic auth for Grafana
+
+## Impact
+
+- **Files**: New `.env.production.enc` (encrypted), `.sops.yaml` config,
+  split `cd-apps.yml` and `cd-infra.yml` workflows, updated docker-compose.yml
+  and Caddyfile
+- **Dependencies**: `sops` and `age` CLI tools in CD runner (install step)
+- **GitHub secrets**: Reduced from ~10 secrets to 2 (AGE_SECRET_KEY +
+  DEPLOY_SSH_KEY). Everything else moves into the encrypted env file.
+- **Grafana**: GitHub OAuth app registration needed (Client ID + Secret go
+  into the SOPS-encrypted file)
+- **Caddy**: Remove basic auth block for grafana.internal, just proxy through
--- a/openspec/changes/sops-age-split-cd/specs/infrastructure/spec.md
+++ b/openspec/changes/sops-age-split-cd/specs/infrastructure/spec.md
@ -0,0 +1,27 @@
+## MODIFIED Requirements
+
+### Requirement: CI/CD pipeline
+GitHub Actions SHALL use separate workflows for app deployment and infrastructure deployment, with secrets decrypted from a SOPS-encrypted file.
+
+#### Scenario: App deployment
+- **WHEN** code changes are pushed to main in apps/ or packages/
+- **THEN** the cd-apps workflow builds Docker images, pushes to ghcr.io, and deploys app containers
+
+#### Scenario: Infrastructure deployment
+- **WHEN** changes are pushed to main in infrastructure/
+- **THEN** the cd-infra workflow copies configs and restarts infrastructure services without rebuilding app images
+
+#### Scenario: Secret decryption at deploy time
+- **WHEN** either CD workflow runs
+- **THEN** the SOPS-encrypted secrets file is decrypted and provided to docker-compose as an env file
+
+### Requirement: Grafana authentication
+Grafana SHALL authenticate users via GitHub OAuth, restricted to the trails-cool GitHub organization.
+
+#### Scenario: GitHub OAuth login
+- **WHEN** a user navigates to grafana.internal.trails.cool
+- **THEN** they are redirected to GitHub for authentication and granted access if they are a member of the trails-cool organization
+
+#### Scenario: No password-based login
+- **WHEN** Grafana is deployed
+- **THEN** the login form is disabled and only GitHub OAuth is available
--- a/openspec/changes/sops-age-split-cd/specs/secret-management/spec.md
+++ b/openspec/changes/sops-age-split-cd/specs/secret-management/spec.md
@ -0,0 +1,16 @@
+## ADDED Requirements
+
+### Requirement: Encrypted secrets in repository
+Production secrets SHALL be stored as a SOPS-encrypted file in the repository, decryptable only with a single age private key.
+
+#### Scenario: Edit secrets
+- **WHEN** a developer runs `sops infrastructure/secrets.env`
+- **THEN** the file is decrypted in a temporary editor, and re-encrypted on save
+
+#### Scenario: CD decryption
+- **WHEN** the CD workflow runs
+- **THEN** the encrypted secrets file is decrypted using the AGE_SECRET_KEY GitHub secret and provided to docker-compose as an env file
+
+#### Scenario: Secret audit trail
+- **WHEN** a secret is changed
+- **THEN** the change appears in git history as a diff of the encrypted file with a commit message describing what changed
--- a/openspec/changes/sops-age-split-cd/tasks.md
+++ b/openspec/changes/sops-age-split-cd/tasks.md
@ -0,0 +1,36 @@
+## 1. SOPS + age Setup
+
+- [x] 1.1 Generate age key pair (`age-keygen`), store public key in `.sops.yaml`, private key as `AGE_SECRET_KEY` GitHub secret
+- [x] 1.2 Create `.sops.yaml` at repo root with age encryption rule for `secrets.*.env`
+- [x] 1.3 Create `infrastructure/secrets.app.env` with app secrets (POSTGRES_PASSWORD, JWT_SECRET, SESSION_SECRET, SMTP_URL, SMTP_FROM, SENTRY_AUTH_TOKEN, DEPLOY_GHCR_TOKEN), encrypt with `sops -e`
+- [x] 1.4 Create `infrastructure/secrets.infra.env` with infra secrets (GF_AUTH_GITHUB_CLIENT_ID, GF_AUTH_GITHUB_CLIENT_SECRET), encrypt with `sops -e`
+- [ ] 1.5 Remove migrated secrets from GitHub Actions (keep only AGE_SECRET_KEY, DEPLOY_SSH_KEY, DEPLOY_HOST)
+
+## 2. GitHub OAuth for Grafana
+
+- [x] 2.1 Register GitHub OAuth app (callback: grafana.internal.trails.cool/login/github)
+- [x] 2.2 Add GF_AUTH_GITHUB_CLIENT_ID and GF_AUTH_GITHUB_CLIENT_SECRET to secrets.env
+- [x] 2.3 Update docker-compose.yml: add GitHub OAuth env vars to Grafana, remove GF_SECURITY_ADMIN_USER/PASSWORD
+- [x] 2.4 Update Caddyfile: remove basic_auth block for grafana.internal, just reverse_proxy
+- [x] 2.5 Remove Caddy GRAFANA_USER/GRAFANA_PASSWORD_HASH env vars from docker-compose.yml
+
+## 3. Split CD Workflows
+
+- [x] 3.1 Create `.github/workflows/cd-apps.yml` — triggered by apps/, packages/, pnpm-lock.yaml changes
+- [x] 3.2 Create `.github/workflows/cd-infra.yml` — triggered by infrastructure/ changes
+- [x] 3.3 Add sops + age install step to both workflows
+- [x] 3.4 cd-apps decrypt step: `sops -d secrets.app.env > .env`, SCP to server
+- [x] 3.5 cd-infra decrypt step: merge `secrets.app.env` + `secrets.infra.env` into `.env`, SCP to server
+- [x] 3.6 cd-apps: build images, push, SSH deploy (pull, migrate, restart apps)
+- [x] 3.7 cd-infra: copy configs, SSH deploy (restart infra services)
+- [x] 3.8 Remove old `cd.yml` workflow
+- [x] 3.9 Remove GRAFANA_* exports and password reset from deploy script
+- [x] 3.10 Both workflows use `docker compose --env-file .env up -d` instead of inline exports
+
+## 4. Verify
+
+- [x] 4.1 Test sops encrypt/decrypt cycle locally
+- [ ] 4.2 Test cd-apps deploys only on app changes (not infra)
+- [ ] 4.3 Test cd-infra deploys only on infra changes (not apps)
+- [ ] 4.4 Test Grafana GitHub OAuth login
+- [ ] 4.5 Verify old GitHub secrets can be removed after migration