ci(deploy): fail loudly — set -e, drizzle output guards, health gates

Hardening from two incidents on 2026-06-06/07:

Schema drift (morning): drizzle-kit push exits 0 even when it aborts
on an interactive prompt it can't render in CI, so cd-staging's
set -euo pipefail never fired and a month of staging schema drift
accumulated silently until new code hit missing columns. All three
drizzle push call sites now tee output and fail the deploy on any
'Error:' line.

Production outage (overnight, ~9h): cd-infra and cd-apps had no
failure handling at all. A network-option change stopped postgres for
a network recreation that then deadlocked on containers from other
compose projects holding trails-shared; the script carried on, the
stack stayed down. Both scripts now run set -euo pipefail and gate on
container health at the end (postgres+journal for cd-infra,
journal+planner for cd-apps) so a deploy that leaves the stack down
is a red X, not a shrug.

docs/deployment.md gains the cross-project manual procedure for
network-changing deploys — the CD workflows only manage their own
compose project and cannot apply those safely.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.github/workflows/cd-apps.yml

@@ -114,6 +114,10 @@ jobs:
           username: root
           key: ${{ secrets.DEPLOY_SSH_KEY }}
           script: |
+            # Abort the deploy on the first failure. Without this, a failed
+            # schema push deploys new code against an old schema (the
+            # 2026-06-06 schema-drift incident).
+            set -euo pipefail
             cd /opt/trails-cool
 
             # Login to ghcr.io
@@ -125,11 +129,34 @@ jobs:
             # Hand-written data migrations (idempotent) run BEFORE drizzle-kit
             # push so unique-key reshapes can collapse duplicate rows first.
             docker compose --env-file app.env run --rm journal node --experimental-strip-types /app/packages/db/src/migrate-data.ts
-            docker compose --env-file app.env run --rm journal npx drizzle-kit push --config /app/packages/db/drizzle.config.ts --force
+            # drizzle-kit exits 0 even when it aborts on an interactive
+            # prompt it can't show (no TTY in CI) — that exact lie hid a
+            # month of staging schema drift. Treat any Error in its output
+            # as a failed deploy.
+            docker compose --env-file app.env run --rm journal npx drizzle-kit push --config /app/packages/db/drizzle.config.ts --force 2>&1 | tee /tmp/drizzle-push.log
+            if grep -q "Error:" /tmp/drizzle-push.log; then
+              echo "drizzle-kit push reported an error — failing the deploy"
+              exit 1
+            fi
             # --remove-orphans cleans up containers whose service was deleted
             # from the compose file, matching cd-infra's behaviour.
             docker compose --env-file app.env up -d --remove-orphans journal planner
 
+            # Gate on container health: a deploy that leaves journal or
+            # planner unhealthy must fail loudly, not report green.
+            for svc in journal planner; do
+              for i in $(seq 1 24); do
+                status=$(docker inspect -f '{{.State.Health.Status}}' "trails-cool-$svc-1" 2>/dev/null || echo missing)
+                [ "$status" = "healthy" ] && break
+                sleep 5
+              done
+              if [ "$status" != "healthy" ]; then
+                echo "$svc did not become healthy (last status: $status)"
+                docker compose --env-file app.env logs "$svc" --tail 50 || true
+                exit 1
+              fi
+            done
+
             # Reload Caddy with the Caddyfile we just scp'd. cd-apps
             # ships infrastructure/Caddyfile alongside docker-compose.yml
             # (see scp step above), but containers don't auto-pick-up

.github/workflows/cd-infra.yml

@@ -64,6 +64,11 @@ jobs:
           username: root
           key: ${{ secrets.DEPLOY_SSH_KEY }}
           script: |
+            # Abort on first failure. The 2026-06-06/07 outage: a network
+            # recreation stopped postgres, a later step failed, and the
+            # deploy left production down for ~9h while the job's partial
+            # progress looked plausible. Fail fast, verify health at the end.
+            set -euo pipefail
             cd /opt/trails-cool
 
             # .env was placed by the SCP step (decrypted app + infra secrets)
@@ -93,6 +98,22 @@ jobs:
 
             docker compose ps
 
+            # Gate on the stack actually being up: postgres healthy and —
+            # since an infra restart bounces the app containers' database —
+            # journal back to healthy too. A deploy that leaves either down
+            # must fail loudly (see the 2026-06-06/07 outage).
+            for ctr in trails-cool-postgres-1 trails-cool-journal-1; do
+              for i in $(seq 1 36); do
+                status=$(docker inspect -f '{{.State.Health.Status}}' "$ctr" 2>/dev/null || echo missing)
+                [ "$status" = "healthy" ] && break
+                sleep 5
+              done
+              if [ "$status" != "healthy" ]; then
+                echo "$ctr did not become healthy (last status: $status)"
+                exit 1
+              fi
+            done
+
             # Annotate deploy in Grafana
             GRAFANA_TOKEN=$(grep GRAFANA_SERVICE_TOKEN .env | cut -d= -f2-)
             if [ -n "$GRAFANA_TOKEN" ]; then

.github/workflows/cd-staging.yml

@@ -181,7 +181,14 @@ jobs:
 
             # Pull and deploy staging containers (journal + planner via "persistent" profile)
             docker compose -f docker-compose.staging.yml -p trails-staging --env-file staging.env --profile persistent pull
-            docker compose -f docker-compose.staging.yml -p trails-staging --env-file staging.env --profile persistent run --rm journal npx drizzle-kit push --config /app/packages/db/drizzle.config.ts --force
+            # drizzle-kit exits 0 even when it aborts on an interactive
+            # prompt (no TTY in CI) — set -e alone can't catch it. That lie
+            # hid a month of staging schema drift (2026-06-06 incident).
+            docker compose -f docker-compose.staging.yml -p trails-staging --env-file staging.env --profile persistent run --rm journal npx drizzle-kit push --config /app/packages/db/drizzle.config.ts --force 2>&1 | tee /tmp/drizzle-push.log
+            if grep -q "Error:" /tmp/drizzle-push.log; then
+              echo "drizzle-kit push reported an error — failing the deploy"
+              exit 1
+            fi
             docker compose -f docker-compose.staging.yml -p trails-staging --env-file staging.env --profile persistent up -d --remove-orphans
 
             # Reload Caddy so new staging routes (or Caddyfile changes shipped
@@ -337,7 +344,13 @@ jobs:
 
             # Pull, migrate, deploy (journal-only — no --profile means planner skipped)
             docker compose -f docker-compose.staging.yml -p "$PROJECT" --env-file "$ENV_FILE" pull journal
-            docker compose -f docker-compose.staging.yml -p "$PROJECT" --env-file "$ENV_FILE" run --rm journal npx drizzle-kit push --config /app/packages/db/drizzle.config.ts --force
+            # Same drizzle-kit exit-code-0-on-error guard as the persistent
+            # staging deploy above.
+            docker compose -f docker-compose.staging.yml -p "$PROJECT" --env-file "$ENV_FILE" run --rm journal npx drizzle-kit push --config /app/packages/db/drizzle.config.ts --force 2>&1 | tee /tmp/drizzle-push.log
+            if grep -q "Error:" /tmp/drizzle-push.log; then
+              echo "drizzle-kit push reported an error — failing the preview deploy"
+              exit 1
+            fi
             docker compose -f docker-compose.staging.yml -p "$PROJECT" --env-file "$ENV_FILE" up -d --remove-orphans journal
 
             # Reload Caddy to pick up the per-PR snippet (writes/replaces it from the SCP step)