refactor(infra): mount config dirs not single files; reload on deploy

Single-file bind mounts (./foo.yml:/etc/foo.yml) pin to the host file's
inode at container-create time. The CD pipeline scp's a replacement file
(new inode), so the running container keeps reading the OLD inode —
`docker compose up -d` won't recreate on a content-only change, and
neither restart/SIGHUP/`caddy reload` re-reads the new file. Net effect:
config-only infra PRs deployed "successfully" but never took effect
(confirmed with PR #500's prometheus.yml; the WAL retention work only
applied because #498 also changed docker-compose.yml, forcing a
recreate). Caddyfile changes had the same latent gap.

Switch the four single-file config mounts to DIRECTORY mounts, which
resolve children live so a reload/restart picks up the new file:

- prometheus, loki, promtail: mount ./<svc> dir (loki/promtail
  --config.file paths updated to the real filenames).
- caddy: move Caddyfile into caddy/ and mount the dir; ./sites overlays
  /etc/caddy/sites (caddy/sites/.gitkeep keeps the mountpoint). Container
  path /etc/caddy/Caddyfile is unchanged, so every `caddy reload --config
  /etc/caddy/Caddyfile` ref still works. scp source paths in cd-infra and
  cd-apps updated to ship the caddy/ dir.

cd-infra now applies config-only changes after `up -d`: SIGHUP prometheus
(zero downtime), restart loki+promtail (no SIGHUP reload), caddy reload
(graceful). Mirrored prometheus/loki mounts in docker-compose.dev.yml.

Validated: docker compose config (both files), caddy validate from the
new path, read-only-parent + sites-overlay mount mechanics, workflow YAML.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-06-09 13:10:35 +02:00
parent 8e41b09ac2
commit e60c9d7057
6 changed files with 48 additions and 16 deletions

View file

@ -100,7 +100,7 @@ jobs:
host: ${{ secrets.DEPLOY_HOST }}
username: root
key: ${{ secrets.DEPLOY_SSH_KEY }}
source: "infrastructure/docker-compose.yml,infrastructure/Caddyfile"
source: "infrastructure/docker-compose.yml,infrastructure/caddy"
target: /opt/trails-cool
strip_components: 1

View file

@ -43,7 +43,7 @@ jobs:
host: ${{ secrets.DEPLOY_HOST }}
username: root
key: ${{ secrets.DEPLOY_SSH_KEY }}
source: "infrastructure/docker-compose.yml,infrastructure/Caddyfile,infrastructure/prometheus/prometheus.yml,infrastructure/loki/loki-config.yml,infrastructure/promtail/promtail-config.yml,infrastructure/postgres/queries.yml,infrastructure/postgres/init-grafana-user.sql,infrastructure/grafana/provisioning,infrastructure/grafana/dashboards"
source: "infrastructure/docker-compose.yml,infrastructure/caddy,infrastructure/prometheus/prometheus.yml,infrastructure/loki/loki-config.yml,infrastructure/promtail/promtail-config.yml,infrastructure/postgres/queries.yml,infrastructure/postgres/init-grafana-user.sql,infrastructure/grafana/provisioning,infrastructure/grafana/dashboards"
target: /opt/trails-cool
strip_components: 1
@ -88,14 +88,25 @@ jobs:
if [ "${{ github.event.inputs.restart_all }}" = "true" ]; then
docker compose --env-file .env up -d --remove-orphans
else
# Restart infra services (except Caddy — just reload its config).
# Restart infra services (config reloads handled below).
# --remove-orphans cleans up containers whose service was deleted
# from the compose file (e.g., the flagship `brouter` removal in
# PR #297 left an orphan that had to be removed by hand).
docker compose --env-file .env up -d --remove-orphans postgres prometheus loki promtail grafana postgres-exporter node-exporter cadvisor
docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile
fi
# Apply config-only changes that `up -d` skips — it recreates a
# container only when its compose *definition* changes, not when a
# mounted config file's content changes. The configs are mounted as
# directories (not single files), so a reload/restart re-reads the
# freshly scp'd file; a single-file mount would have pinned the old
# inode. Prometheus hot-reloads on SIGHUP (zero downtime); Loki and
# Promtail reload their main config only on restart; Caddy reloads
# gracefully (validates, swaps live, no downtime).
docker compose --env-file .env kill -s SIGHUP prometheus
docker compose --env-file .env restart loki promtail
docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile
docker compose ps
# Gate on the stack actually being up: postgres healthy and —