Hardening from two incidents on 2026-06-06/07: Schema drift (morning): drizzle-kit push exits 0 even when it aborts on an interactive prompt it can't render in CI, so cd-staging's set -euo pipefail never fired and a month of staging schema drift accumulated silently until new code hit missing columns. All three drizzle push call sites now tee output and fail the deploy on any 'Error:' line. Production outage (overnight, ~9h): cd-infra and cd-apps had no failure handling at all. A network-option change stopped postgres for a network recreation that then deadlocked on containers from other compose projects holding trails-shared; the script carried on, the stack stayed down. Both scripts now run set -euo pipefail and gate on container health at the end (postgres+journal for cd-infra, journal+planner for cd-apps) so a deploy that leaves the stack down is a red X, not a shrug. docs/deployment.md gains the cross-project manual procedure for network-changing deploys — the CD workflows only manage their own compose project and cannot apply those safely. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| adr | ||
| conversations | ||
| ideas | ||
| legal-archive | ||
| architecture.md | ||
| demo-persona.md | ||
| deployment.md | ||
| gpx-extensions.md | ||
| information-architecture.md | ||
| old-trails-analysis.md | ||
| philosophy.md | ||
| roadmap.md | ||
| server-hardening.md | ||
| tooling.md | ||