trails-cool/trails
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

ci(deploy): fail loudly — set -e, drizzle output guards, health gates

Browse source 
Hardening from two incidents on 2026-06-06/07:

Schema drift (morning): drizzle-kit push exits 0 even when it aborts
on an interactive prompt it can't render in CI, so cd-staging's
set -euo pipefail never fired and a month of staging schema drift
accumulated silently until new code hit missing columns. All three
drizzle push call sites now tee output and fail the deploy on any
'Error:' line.

Production outage (overnight, ~9h): cd-infra and cd-apps had no
failure handling at all. A network-option change stopped postgres for
a network recreation that then deadlocked on containers from other
compose projects holding trails-shared; the script carried on, the
stack stayed down. Both scripts now run set -euo pipefail and gate on
container health at the end (postgres+journal for cd-infra,
journal+planner for cd-apps) so a deploy that leaves the stack down
is a red X, not a shrug.

docs/deployment.md gains the cross-project manual procedure for
network-changing deploys — the CD workflows only manage their own
compose project and cannot apply those safely.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-06-07 08:15:17 +02:00
parent 007845cb59
commit f790da2ed3
4 changed files with 99 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

29
.github/workflows/cd-apps.yml vendored
View file

@ -114,6 +114,10 @@ jobs:
username: root
key: ${{ secrets.DEPLOY_SSH_KEY }}
script: |
# Abort the deploy on the first failure. Without this, a failed
# schema push deploys new code against an old schema (the
# 2026-06-06 schema-drift incident).
set -euo pipefail
cd /opt/trails-cool
# Login to ghcr.io
@ -125,11 +129,34 @@ jobs:
# Hand-written data migrations (idempotent) run BEFORE drizzle-kit
# push so unique-key reshapes can collapse duplicate rows first.
docker compose --env-file app.env run --rm journal node --experimental-strip-types /app/packages/db/src/migrate-data.ts
docker compose --env-file app.env run --rm journal npx drizzle-kit push --config /app/packages/db/drizzle.config.ts --force
# drizzle-kit exits 0 even when it aborts on an interactive
# prompt it can't show (no TTY in CI) — that exact lie hid a
# month of staging schema drift. Treat any Error in its output
# as a failed deploy.
docker compose --env-file app.env run --rm journal npx drizzle-kit push --config /app/packages/db/drizzle.config.ts --force 2>&1 | tee /tmp/drizzle-push.log
if grep -q "Error:" /tmp/drizzle-push.log; then
echo "drizzle-kit push reported an error — failing the deploy"
exit 1
fi
# --remove-orphans cleans up containers whose service was deleted
# from the compose file, matching cd-infra's behaviour.
docker compose --env-file app.env up -d --remove-orphans journal planner
# Gate on container health: a deploy that leaves journal or
# planner unhealthy must fail loudly, not report green.
for svc in journal planner; do
for i in $(seq 1 24); do
status=$(docker inspect -f '{{.State.Health.Status}}' "trails-cool-$svc-1" 2>/dev/null || echo missing)
[ "$status" = "healthy" ] && break
sleep 5
done
if [ "$status" != "healthy" ]; then
echo "$svc did not become healthy (last status: $status)"
docker compose --env-file app.env logs "$svc" --tail 50 || true
exit 1
fi
done
# Reload Caddy with the Caddyfile we just scp'd. cd-apps
# ships infrastructure/Caddyfile alongside docker-compose.yml
# (see scp step above), but containers don't auto-pick-up