5c4b6fd9af
The journal/planner deploy in cd-apps.yml does `docker compose up -d journal planner`, which stops the old container and starts the new one — Caddy keeps forwarding requests during the ~10–30s gap and returns 502s. The caddy-502-rate alert (threshold > 0 for 2m) correctly trips, every time. Two production changes plus a long-broken workflow detail: - infrastructure/Caddyfile — add `lb_try_duration 30s` / `lb_try_interval 250ms` to the journal and planner reverse_proxy blocks. Caddy now holds and retries the upstream for up to 30s during a restart instead of 502'ing immediately. Real outages (upstream unreachable longer than 30s) still 502 and the alert still fires for those. - infrastructure/grafana/provisioning/alerting/alerts.yml — add a comment documenting why caddy-502-rate stays at threshold > 0: with lb_try_duration in front of it, the alert no longer conflates "deploy in flight" with "real outage." - .github/workflows/cd-apps.yml — fix a long-silent bug: the Grafana deploy-annotation step was reading GRAFANA_SERVICE_TOKEN from `.env`, but the secrets file we scp to /opt/trails-cool is named `app.env`. The token check failed silently and the curl was being skipped on every deploy. Switching to `app.env` so deploys actually annotate Grafana. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
71 lines
2.3 KiB
Caddyfile
71 lines
2.3 KiB
Caddyfile
{
|
||
servers {
|
||
metrics
|
||
}
|
||
admin 0.0.0.0:2019
|
||
}
|
||
|
||
(security_headers) {
|
||
header {
|
||
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
|
||
X-Content-Type-Options "nosniff"
|
||
X-Frame-Options "DENY"
|
||
Referrer-Policy "strict-origin-when-cross-origin"
|
||
Permissions-Policy "camera=(), microphone=(), geolocation=()"
|
||
Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' blob:; worker-src 'self' blob:; style-src 'self' 'unsafe-inline'; img-src 'self' data: https://*.tile.openstreetmap.org; connect-src 'self' wss: https://*.sentry.io https://*.ingest.de.sentry.io; font-src 'self';"
|
||
}
|
||
}
|
||
|
||
(block_scanners) {
|
||
@scanners path /.env* /.git* /wp-* /wp-admin* /admin* /config.* /backup* /.aws* /.docker* /composer* /vendor* *.php *.asp
|
||
respond @scanners 403
|
||
}
|
||
|
||
{$DOMAIN:trails.cool} {
|
||
import security_headers
|
||
import block_scanners
|
||
log {
|
||
output stdout
|
||
format json
|
||
}
|
||
reverse_proxy journal:3000 {
|
||
# During an `apps` deploy the journal container is briefly down
|
||
# (~10–30s) while compose swaps containers. Without these,
|
||
# Caddy returns 502 immediately and the `caddy-502-rate` alert
|
||
# trips on every deploy. With them, Caddy holds and retries
|
||
# against the upstream for up to 30s — restart becomes
|
||
# invisible to clients. A real outage longer than 30s still
|
||
# 502s and correctly trips the alert.
|
||
lb_try_duration 30s
|
||
lb_try_interval 250ms
|
||
}
|
||
}
|
||
|
||
www.{$DOMAIN:trails.cool} {
|
||
redir https://{$DOMAIN:trails.cool}{uri} permanent
|
||
}
|
||
|
||
grafana.internal.{$DOMAIN:trails.cool} {
|
||
reverse_proxy grafana:3000
|
||
}
|
||
|
||
planner.{$DOMAIN:trails.cool} {
|
||
header {
|
||
Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
|
||
X-Content-Type-Options "nosniff"
|
||
X-Frame-Options "DENY"
|
||
Referrer-Policy "strict-origin-when-cross-origin"
|
||
Permissions-Policy "camera=(), microphone=(), geolocation=()"
|
||
Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' blob:; worker-src 'self' blob:; style-src 'self' 'unsafe-inline'; img-src 'self' data: https://*.tile.openstreetmap.org; connect-src 'self' wss: https://{$DOMAIN:trails.cool} https://*.sentry.io https://*.ingest.de.sentry.io; font-src 'self';"
|
||
}
|
||
import block_scanners
|
||
log {
|
||
output stdout
|
||
format json
|
||
}
|
||
reverse_proxy planner:3001 {
|
||
# Same rationale as the journal block — see the comment there.
|
||
lb_try_duration 30s
|
||
lb_try_interval 250ms
|
||
}
|
||
}
|