cd-infra kills Prometheus when it enters the deploy stopped (SIGHUP branch misreads stopped-container ID as 'was running') #23

New issue

Closed

opened 2026-07-27 12:16:47 +00:00 by ullrich · 0 comments

ullrich commented

2026-07-27 12:16:47 +00:00

Owner

cd-infra run #94 (the first infra deploy on main since the Forgejo cutover) failed at the Prometheus readiness gate, and left Prometheus down for ~16 minutes — no metrics, no alert evaluation. Restarted manually; the stack is healthy again.

The readiness gate did its job: it refused to report success. The bug is upstream of it.

Mechanism — confirmed from the run log

12:04:40.266  Container trails-cool-prometheus-1 Starting
12:04:40.527  Container trails-cool-prometheus-1 Started
12:04:41.577  Container trails-cool-prometheus-1 Killing
12:04:41.638  Container trails-cool-prometheus-1 Killed
12:08:00.069  trails-cool-prometheus-1 did not become ready (last status: exited)

The deploy started Prometheus and killed it one second later. The culprit is the config-reload branch:

PROMETHEUS_BEFORE_ID=$(docker inspect -f '{{.Id}}' trails-cool-prometheus-1 2>/dev/null || true)
# ... docker compose up -d ...
PROMETHEUS_AFTER_ID=$(docker inspect -f '{{.Id}}' trails-cool-prometheus-1 2>/dev/null || true)

if [ -n "$PROMETHEUS_BEFORE_ID" ] && [ "$PROMETHEUS_BEFORE_ID" = "$PROMETHEUS_AFTER_ID" ]; then
  docker compose --env-file .env kill -s SIGHUP prometheus
fi

The intent is sound: "if compose did not recreate Prometheus, it hasn't picked up the new config, so SIGHUP it to reload in place."

The flaw is that docker inspect returns an ID for a stopped container too. So when Prometheus enters the deploy already stopped:

BEFORE_ID = the stopped container's ID (non-empty)
up -d starts that same container — same ID
AFTER_ID == BEFORE_ID → the branch fires
SIGHUP lands ~1s into startup, mid WAL replay, before Prometheus installs its SIGHUP handler — and the default disposition for SIGHUP is terminate

So the one case where the reload is entirely unnecessary — Prometheus just started, already holding the new config — is the case where it is fatal. ID equality was standing in for "was already running", and those differ precisely when the container was stopped.

This is why it has never fired before: normally Prometheus is either running (SIGHUP reloads correctly) or recreated (IDs differ, branch skipped). It needs Prometheus to be stopped but present going in.

Proposed fix

Guard on running state, not just identity:

PROMETHEUS_BEFORE_RUNNING=$(docker inspect -f '{{.State.Running}}' trails-cool-prometheus-1 2>/dev/null || echo false)
# ...
if [ "$PROMETHEUS_BEFORE_RUNNING" = "true" ] && [ "$PROMETHEUS_BEFORE_ID" = "$PROMETHEUS_AFTER_ID" ]; then
  docker compose --env-file .env kill -s SIGHUP prometheus
fi

A container compose just started needs no reload, so skipping the SIGHUP there is correct on both counts.

Worth considering alongside: --web.enable-lifecycle + POST /-/reload fails loudly and only when Prometheus is actually up, rather than firing a signal at a process that may not be ready to receive it.

Open question

Prometheus took a SIGTERM at 11:55:07 and several containers (caddy, cadvisor, grafana, postgres, postgres-exporter) were recreated around 11:53 — both before run #94's SSH step began at 12:04:27. So something stopped Prometheus in an earlier partial execution, which is what set up the stopped-but-present state. That earlier activity is unexplained; concurrency: cancel-in-progress: true on deploy-infra cancelling a first execution mid-flight is the obvious suspect. A cancelled deploy leaving the stack half-recreated is worth understanding on its own.

Re-running should now be safe

Prometheus is running again, so a re-run takes the intended path: BEFORE_RUNNING=true, IDs match, SIGHUP reaches a long-settled process and reloads config normally.

Found while verifying #22. That PR's changes deployed correctly and are verified end-to-end (host .prom → node_exporter → Prometheus, node_textfile_scrape_error 0).

`cd-infra` run #94 (the first infra deploy on `main` since the Forgejo cutover) failed at the Prometheus readiness gate, and left **Prometheus down for ~16 minutes** — no metrics, no alert evaluation. Restarted manually; the stack is healthy again. The readiness gate did its job: it refused to report success. The bug is upstream of it. ## Mechanism — confirmed from the run log ``` 12:04:40.266 Container trails-cool-prometheus-1 Starting 12:04:40.527 Container trails-cool-prometheus-1 Started 12:04:41.577 Container trails-cool-prometheus-1 Killing 12:04:41.638 Container trails-cool-prometheus-1 Killed 12:08:00.069 trails-cool-prometheus-1 did not become ready (last status: exited) ``` The deploy **started** Prometheus and **killed it one second later**. The culprit is the config-reload branch: ```bash PROMETHEUS_BEFORE_ID=$(docker inspect -f '{{.Id}}' trails-cool-prometheus-1 2>/dev/null || true) # ... docker compose up -d ... PROMETHEUS_AFTER_ID=$(docker inspect -f '{{.Id}}' trails-cool-prometheus-1 2>/dev/null || true) if [ -n "$PROMETHEUS_BEFORE_ID" ] && [ "$PROMETHEUS_BEFORE_ID" = "$PROMETHEUS_AFTER_ID" ]; then docker compose --env-file .env kill -s SIGHUP prometheus fi ``` The intent is sound: *"if compose did not recreate Prometheus, it hasn't picked up the new config, so SIGHUP it to reload in place."* The flaw is that **`docker inspect` returns an ID for a stopped container too.** So when Prometheus enters the deploy already stopped: 1. `BEFORE_ID` = the stopped container's ID (non-empty) 2. `up -d` **starts** that same container — same ID 3. `AFTER_ID` == `BEFORE_ID` → the branch fires 4. SIGHUP lands ~1s into startup, mid WAL replay, before Prometheus installs its SIGHUP handler — and the default disposition for SIGHUP is *terminate* So the one case where the reload is entirely unnecessary — Prometheus just started, already holding the new config — is the case where it is fatal. ID equality was standing in for "was already running", and those differ precisely when the container was stopped. This is why it has never fired before: normally Prometheus is either running (SIGHUP reloads correctly) or recreated (IDs differ, branch skipped). It needs Prometheus to be *stopped but present* going in. ## Proposed fix Guard on running state, not just identity: ```bash PROMETHEUS_BEFORE_RUNNING=$(docker inspect -f '{{.State.Running}}' trails-cool-prometheus-1 2>/dev/null || echo false) # ... if [ "$PROMETHEUS_BEFORE_RUNNING" = "true" ] && [ "$PROMETHEUS_BEFORE_ID" = "$PROMETHEUS_AFTER_ID" ]; then docker compose --env-file .env kill -s SIGHUP prometheus fi ``` A container compose just started needs no reload, so skipping the SIGHUP there is correct on both counts. Worth considering alongside: `--web.enable-lifecycle` + `POST /-/reload` fails loudly and only when Prometheus is actually up, rather than firing a signal at a process that may not be ready to receive it. ## Open question Prometheus took a SIGTERM at **11:55:07** and several containers (caddy, cadvisor, grafana, postgres, postgres-exporter) were recreated around **11:53** — both *before* run #94's SSH step began at 12:04:27. So something stopped Prometheus in an earlier partial execution, which is what set up the stopped-but-present state. That earlier activity is unexplained; `concurrency: cancel-in-progress: true` on `deploy-infra` cancelling a first execution mid-flight is the obvious suspect. A cancelled deploy leaving the stack half-recreated is worth understanding on its own. ## Re-running should now be safe Prometheus is running again, so a re-run takes the intended path: `BEFORE_RUNNING=true`, IDs match, SIGHUP reaches a long-settled process and reloads config normally. Found while verifying #22. That PR's changes deployed correctly and are verified end-to-end (host `.prom` → node_exporter → Prometheus, `node_textfile_scrape_error 0`).

ullrich referenced this issue

2026-07-27 12:18:57 +00:00

POI index never auto-refreshes — neither refresh timer is installed #19

ullrich referenced this issue from a commit

2026-07-27 12:22:06 +00:00

fix(cd-infra): don't SIGHUP a Prometheus that compose just started

ullrich referenced this issue from a pull request that will close it,

2026-07-27 12:22:38 +00:00

fix(cd-infra): don't SIGHUP a Prometheus that compose just started #24

ullrich closed this issue

2026-07-27 12:33:30 +00:00