ci: scheduled image prune + staging-deploy disk hygiene
The 2026-06-07 disk-full outage (flagship 100%, postgres crash-looping on its pidfile, prod + staging down): cd-apps DOES prune images after deploys, but it had been failing early all day (migration bug), so its prune never ran — while ~10 staging/preview deploys kept pulling fresh images with no prune of their own. - cd-staging: prune superseded layers (until=1h guard against racing in-flight pulls) after persistent staging and preview deploys. - disk-maintenance.yml: NEW daily scheduled prune (04:30 UTC) that also FAILS when the disk is still ≥85% after pruning — a redundant alert channel for exactly the case where the Grafana disk alert drowns in other noise, as it did during the incident. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
d5a8709688
commit
89caedca05
2 changed files with 63 additions and 0 deletions
52
.github/workflows/disk-maintenance.yml
vendored
Normal file
52
.github/workflows/disk-maintenance.yml
vendored
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
name: Disk Maintenance
|
||||
|
||||
# Daily safety net for flagship disk usage. The deploy workflows prune
|
||||
# superseded image layers after their own runs, but that protection
|
||||
# disappears exactly when it's needed most: a deploy that fails early
|
||||
# never reaches its prune step, while other workflows keep pulling
|
||||
# fresh images (2026-06-07 incident: cd-apps red all day, ~10 staging
|
||||
# deploys, disk 100% full, postgres down on a Saturday morning).
|
||||
#
|
||||
# Also doubles as a redundant alert channel: the run FAILS when the
|
||||
# disk is still above the threshold after pruning, so a scheduled-run
|
||||
# failure email lands even if the Grafana disk alert drowns in other
|
||||
# noise (which is what happened during the incident).
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Daily at 04:30 UTC (offset from staging-cleanup's Monday 04:00)
|
||||
- cron: "30 4 * * *"
|
||||
workflow_dispatch: {}
|
||||
|
||||
concurrency:
|
||||
group: disk-maintenance
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
prune-flagship:
|
||||
name: Prune unused images (flagship)
|
||||
runs-on: ubuntu-latest
|
||||
environment: production
|
||||
steps:
|
||||
- name: Prune and check disk headroom
|
||||
uses: appleboy/ssh-action@v1
|
||||
with:
|
||||
host: ${{ secrets.DEPLOY_HOST }}
|
||||
username: root
|
||||
key: ${{ secrets.DEPLOY_SSH_KEY }}
|
||||
script: |
|
||||
set -euo pipefail
|
||||
echo "before: $(df -h / | tail -1)"
|
||||
# 12h filter: never touch layers a same-day deploy may still
|
||||
# be assembling; running containers' images are never pruned.
|
||||
docker image prune -af --filter "until=12h"
|
||||
echo "after: $(df -h / | tail -1)"
|
||||
|
||||
PCT=$(df --output=pcent / | tail -1 | tr -dc '0-9')
|
||||
if [ "$PCT" -ge 85 ]; then
|
||||
echo "Disk still at ${PCT}% after pruning — needs a human."
|
||||
echo "Largest docker consumers:"
|
||||
docker system df
|
||||
exit 1
|
||||
fi
|
||||
echo "Disk at ${PCT}% — healthy."
|
||||
Loading…
Add table
Add a link
Reference in a new issue