trails-cool/trails
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

fix(infra): cap container log sizes to prevent disk-full outages

Browse source 
Every compose service used docker's default json-file driver with no
rotation, so a service logging in a loop grew unbounded — a 2.6 GB
container log contributed to a 2026-07-14 disk-full outage (/ at 100%,
Postgres down, apps 500ing, deploys failing at the SCP step).

Add an `x-logging` anchor (10 MB x 3 files = 30 MB max per container) and
apply it to all 11 services. Complements disk-maintenance.yml (which prunes
images but can't touch container logs). Takes effect when cd-infra recreates
the containers. Documented the disk-pressure runbook in docs/deployment.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-07-14 08:34:17 +02:00
parent 0328fb7f33
commit 010edae636
No known key found for this signature in database
GPG key ID: A32FF691A0F752D9
2 changed files with 46 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

25
docs/deployment.md
View file

@ -67,6 +67,31 @@ gh workflow run cd-infra.yml -f restart_all=true
Restarts every flagship service. Does NOT touch the BRouter host.
## Flagship disk pressure
The flagship root fs is ~38 GB. When it hits 100%, Postgres can't write and
every app 500s; deploys fail at the SCP step ("error copy file to dest").
Two independent guards keep it in check:
- **Image churn** — `disk-maintenance.yml` prunes unused images daily (04:30
UTC, `until=12h`); the deploy workflows also prune after each run.
- **Container logs** — every compose service caps its `json-file` logs at
30 MB (`x-logging` anchor in `docker-compose.yml`). Before that cap a
single service logging in a loop grew a 2.6 GB log (2026-07-14 outage).
Emergency (disk already full, service down):
```bash
ssh -i ~/.ssh/trails-cool-deploy root@trails.cool
df -h / # confirm 100%
docker image prune -af # safe — never removes volumes/data
docker builder prune -f # build cache
docker system df # what's left; du -xhd1 /var to hunt
# containers self-recover once space frees; re-run the failed deploy after.
```
Never `docker system prune --volumes` — that deletes the Postgres data volume.
## Network-changing deploys (flagship)
Changing options on an existing Docker network (`enable_ipv6`, subnets,