ci: scheduled image prune + staging-deploy disk hygiene

The 2026-06-07 disk-full outage (flagship 100%, postgres crash-looping
on its pidfile, prod + staging down): cd-apps DOES prune images after
deploys, but it had been failing early all day (migration bug), so its
prune never ran — while ~10 staging/preview deploys kept pulling fresh
images with no prune of their own.

- cd-staging: prune superseded layers (until=1h guard against racing
  in-flight pulls) after persistent staging and preview deploys.
- disk-maintenance.yml: NEW daily scheduled prune (04:30 UTC) that
  also FAILS when the disk is still ≥85% after pruning — a redundant
  alert channel for exactly the case where the Grafana disk alert
  drowns in other noise, as it did during the incident.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-06-07 09:52:49 +02:00
parent d5a8709688
commit 89caedca05
2 changed files with 63 additions and 0 deletions

View file

@ -195,6 +195,13 @@ jobs:
# via cd-infra) are live. Idempotent.
docker compose exec -T caddy caddy reload --config /etc/caddy/Caddyfile || true
# Reclaim superseded image layers. cd-apps prunes after its own
# deploys, but a day of staging/preview deploys while cd-apps is
# red can fill the disk on its own (2026-06-07: 100% full,
# postgres down). The 1h filter avoids racing layers another
# in-flight deploy just pulled.
docker image prune -af --filter "until=1h" || true
docker compose -f docker-compose.staging.yml -p trails-staging --env-file staging.env --profile persistent ps
# ── PR preview deploy ────────────────────────────────────────────────
@ -356,6 +363,10 @@ jobs:
# Reload Caddy to pick up the per-PR snippet (writes/replaces it from the SCP step)
docker compose exec -T caddy caddy reload --config /etc/caddy/Caddyfile
# Same disk-hygiene prune as the persistent staging deploy —
# preview pushes are the highest-volume image source.
docker image prune -af --filter "until=1h" || true
docker compose -f docker-compose.staging.yml -p "$PROJECT" --env-file "$ENV_FILE" ps
# Find any prior preview comment so we can update it in place rather

52
.github/workflows/disk-maintenance.yml vendored Normal file
View file

@ -0,0 +1,52 @@
name: Disk Maintenance
# Daily safety net for flagship disk usage. The deploy workflows prune
# superseded image layers after their own runs, but that protection
# disappears exactly when it's needed most: a deploy that fails early
# never reaches its prune step, while other workflows keep pulling
# fresh images (2026-06-07 incident: cd-apps red all day, ~10 staging
# deploys, disk 100% full, postgres down on a Saturday morning).
#
# Also doubles as a redundant alert channel: the run FAILS when the
# disk is still above the threshold after pruning, so a scheduled-run
# failure email lands even if the Grafana disk alert drowns in other
# noise (which is what happened during the incident).
on:
schedule:
# Daily at 04:30 UTC (offset from staging-cleanup's Monday 04:00)
- cron: "30 4 * * *"
workflow_dispatch: {}
concurrency:
group: disk-maintenance
cancel-in-progress: false
jobs:
prune-flagship:
name: Prune unused images (flagship)
runs-on: ubuntu-latest
environment: production
steps:
- name: Prune and check disk headroom
uses: appleboy/ssh-action@v1
with:
host: ${{ secrets.DEPLOY_HOST }}
username: root
key: ${{ secrets.DEPLOY_SSH_KEY }}
script: |
set -euo pipefail
echo "before: $(df -h / | tail -1)"
# 12h filter: never touch layers a same-day deploy may still
# be assembling; running containers' images are never pruned.
docker image prune -af --filter "until=12h"
echo "after: $(df -h / | tail -1)"
PCT=$(df --output=pcent / | tail -1 | tr -dc '0-9')
if [ "$PCT" -ge 85 ]; then
echo "Disk still at ${PCT}% after pruning — needs a human."
echo "Largest docker consumers:"
docker system df
exit 1
fi
echo "Disk at ${PCT}% — healthy."