ci: don't fail deploys on a transient "prune already running" collision

cd-apps's post-deploy `docker image prune` runs under `set -euo pipefail`
after the containers are already swapped, so when it collides with a
concurrent deploy/disk-maintenance prune ("Error response from daemon: a
prune operation is already running") it fails an otherwise-successful
deploy (observed on the #587 planner deploy). The cleanup is best-effort;
disk-maintenance.yml is the real image-prune safety net.

- cd-apps: `docker image prune -af || true`.
- disk-maintenance: tolerate the same collision on its prune; the disk-%
  threshold check afterward stays the real failure gate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-07-14 09:27:50 +02:00
parent 5debe5643e
commit a75eb1965c
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
2 changed files with 12 additions and 3 deletions

View file

@ -177,8 +177,13 @@ jobs:
# deploy from failing if Caddy itself is unhealthy.
docker compose exec -T caddy caddy reload --config /etc/caddy/Caddyfile || true
# Clean up
docker image prune -af
# Clean up (best-effort). This runs AFTER the containers are
# swapped + Caddy reloaded, so the deploy already succeeded —
# a transient "a prune operation is already running" collision
# with a concurrent deploy/disk-maintenance prune must NOT fail
# an otherwise-green deploy. disk-maintenance.yml is the real
# image-prune safety net.
docker image prune -af || true
docker compose ps
# Annotate deploy in Grafana. GRAFANA_SERVICE_TOKEN lives

View file

@ -39,7 +39,11 @@ jobs:
echo "before: $(df -h / | tail -1)"
# 12h filter: never touch layers a same-day deploy may still
# be assembling; running containers' images are never pruned.
docker image prune -af --filter "until=12h"
# `|| true`: a concurrent deploy's prune can make this collide
# with "a prune operation is already running" — that's benign
# (the other prune is freeing space too), and the disk-% gate
# below is the real assertion, so don't fail on the collision.
docker image prune -af --filter "until=12h" || true
echo "after: $(df -h / | tail -1)"
PCT=$(df --output=pcent / | tail -1 | tr -dc '0-9')