From 5c4b6fd9af13eb89103e5f2562ce5c592570c630 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ullrich=20Sch=C3=A4fer?= Date: Sun, 26 Apr 2026 11:51:13 +0200 Subject: [PATCH 1/2] Stop the caddy-502-rate alert firing on every deploy MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The journal/planner deploy in cd-apps.yml does `docker compose up -d journal planner`, which stops the old container and starts the new one — Caddy keeps forwarding requests during the ~10–30s gap and returns 502s. The caddy-502-rate alert (threshold > 0 for 2m) correctly trips, every time. Two production changes plus a long-broken workflow detail: - infrastructure/Caddyfile — add `lb_try_duration 30s` / `lb_try_interval 250ms` to the journal and planner reverse_proxy blocks. Caddy now holds and retries the upstream for up to 30s during a restart instead of 502'ing immediately. Real outages (upstream unreachable longer than 30s) still 502 and the alert still fires for those. - infrastructure/grafana/provisioning/alerting/alerts.yml — add a comment documenting why caddy-502-rate stays at threshold > 0: with lb_try_duration in front of it, the alert no longer conflates "deploy in flight" with "real outage." - .github/workflows/cd-apps.yml — fix a long-silent bug: the Grafana deploy-annotation step was reading GRAFANA_SERVICE_TOKEN from `.env`, but the secrets file we scp to /opt/trails-cool is named `app.env`. The token check failed silently and the curl was being skipped on every deploy. Switching to `app.env` so deploys actually annotate Grafana. Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/cd-apps.yml | 10 ++++++++-- infrastructure/Caddyfile | 18 ++++++++++++++++-- .../grafana/provisioning/alerting/alerts.yml | 6 ++++++ 3 files changed, 30 insertions(+), 4 deletions(-) diff --git a/.github/workflows/cd-apps.yml b/.github/workflows/cd-apps.yml index 5ba0f06..6277543 100644 --- a/.github/workflows/cd-apps.yml +++ b/.github/workflows/cd-apps.yml @@ -115,8 +115,14 @@ jobs: docker image prune -af docker compose ps - # Annotate deploy in Grafana - GRAFANA_TOKEN=$(grep GRAFANA_SERVICE_TOKEN .env | cut -d= -f2- 2>/dev/null) + # Annotate deploy in Grafana. The token lives in the + # decrypted SOPS env file we just scp'd to /opt/trails-cool + # — that file is `app.env`, not `.env`. (Pre-fix this read + # the wrong path, so annotations were silently no-op'ing + # every deploy.) `2>/dev/null` keeps a missing token from + # failing the deploy; `|| true` keeps the curl from + # failing the deploy if Grafana itself is unhealthy. + GRAFANA_TOKEN=$(grep GRAFANA_SERVICE_TOKEN app.env 2>/dev/null | cut -d= -f2-) if [ -n "$GRAFANA_TOKEN" ]; then docker compose exec -T grafana curl -sf -X POST \ -H "Authorization: Bearer $GRAFANA_TOKEN" \ diff --git a/infrastructure/Caddyfile b/infrastructure/Caddyfile index f2a61b6..6d80f0a 100644 --- a/infrastructure/Caddyfile +++ b/infrastructure/Caddyfile @@ -28,7 +28,17 @@ output stdout format json } - reverse_proxy journal:3000 + reverse_proxy journal:3000 { + # During an `apps` deploy the journal container is briefly down + # (~10–30s) while compose swaps containers. Without these, + # Caddy returns 502 immediately and the `caddy-502-rate` alert + # trips on every deploy. With them, Caddy holds and retries + # against the upstream for up to 30s — restart becomes + # invisible to clients. A real outage longer than 30s still + # 502s and correctly trips the alert. + lb_try_duration 30s + lb_try_interval 250ms + } } www.{$DOMAIN:trails.cool} { @@ -53,5 +63,9 @@ planner.{$DOMAIN:trails.cool} { output stdout format json } - reverse_proxy planner:3001 + reverse_proxy planner:3001 { + # Same rationale as the journal block — see the comment there. + lb_try_duration 30s + lb_try_interval 250ms + } } diff --git a/infrastructure/grafana/provisioning/alerting/alerts.yml b/infrastructure/grafana/provisioning/alerting/alerts.yml index aaae096..1ce2546 100644 --- a/infrastructure/grafana/provisioning/alerting/alerts.yml +++ b/infrastructure/grafana/provisioning/alerting/alerts.yml @@ -206,6 +206,12 @@ groups: annotations: summary: "BRouter host metrics scrape has been failing for 2+ minutes — the dedicated host, vSwitch, or cAdvisor may be down" + # The threshold here is intentionally `> 0` for 2m — *any* + # sustained 502 stream is real. Deploy-time restarts no longer + # produce 502s thanks to `lb_try_duration` on Caddy's reverse + # proxy (see `infrastructure/Caddyfile`); if 502s appear here + # it means the upstream has been unreachable for longer than + # Caddy's retry window, which is a genuine outage. - uid: caddy-502-rate title: Caddy 502 errors detected condition: B From 55c9154f05f801d282965636b15e9fb63f885ddd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ullrich=20Sch=C3=A4fer?= Date: Sun, 26 Apr 2026 11:53:55 +0200 Subject: [PATCH 2/2] Revert cd-apps annotation path: GRAFANA_SERVICE_TOKEN is in .env, not app.env The token lives in `secrets.infra.env`, which `cd-infra.yml` merges together with `secrets.app.env` into the server's `/opt/trails-cool/.env`. The cd-apps workflow's own `app.env` intentionally does NOT carry the token (apps don't need it at runtime), so the original `grep ... .env` was correct. My earlier edit in this branch swapped the path to `app.env` and would have broken the annotation hook the moment it actually worked. Restored `.env` and updated the inline comment to make the file ownership explicit (cd-infra populates it; cd-apps reads it). Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/cd-apps.yml | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/.github/workflows/cd-apps.yml b/.github/workflows/cd-apps.yml index 6277543..cc6d706 100644 --- a/.github/workflows/cd-apps.yml +++ b/.github/workflows/cd-apps.yml @@ -115,14 +115,16 @@ jobs: docker image prune -af docker compose ps - # Annotate deploy in Grafana. The token lives in the - # decrypted SOPS env file we just scp'd to /opt/trails-cool - # — that file is `app.env`, not `.env`. (Pre-fix this read - # the wrong path, so annotations were silently no-op'ing - # every deploy.) `2>/dev/null` keeps a missing token from - # failing the deploy; `|| true` keeps the curl from - # failing the deploy if Grafana itself is unhealthy. - GRAFANA_TOKEN=$(grep GRAFANA_SERVICE_TOKEN app.env 2>/dev/null | cut -d= -f2-) + # Annotate deploy in Grafana. GRAFANA_SERVICE_TOKEN lives + # in secrets.infra.env (decrypted by cd-infra.yml into the + # merged /opt/trails-cool/.env on the server). cd-apps's + # own app.env intentionally does NOT carry it — apps don't + # need it at runtime. So we read from the merged `.env` + # that cd-infra populated. If cd-infra has never run on + # this host, .env may not exist; the `2>/dev/null` and + # the `if -n` guard make the annotation a silent no-op + # rather than a deploy failure in that case. + GRAFANA_TOKEN=$(grep GRAFANA_SERVICE_TOKEN .env 2>/dev/null | cut -d= -f2-) if [ -n "$GRAFANA_TOKEN" ]; then docker compose exec -T grafana curl -sf -X POST \ -H "Authorization: Bearer $GRAFANA_TOKEN" \