SOPS changes to secrets.app.env don't restart app containers #13

New issue

Closed

opened 2026-07-26 13:50:02 +00:00 by ullrich · 0 comments

ullrich commented

2026-07-26 13:50:02 +00:00

Owner

Migrated from GitHub issue #294 (opened 2026-04-24 by @stigi, before the Forgejo migration). Re-verified against the current .forgejo/ workflows on 2026-07-26 — the bug is fully intact, and the ported workflows carried it over verbatim. Line references below are updated to the current files.

Problem

Editing infrastructure/secrets.app.env (SOPS) does not restart the planner/journal containers on the flagship. The BRouter cutover (PR #293) exposed this: the PR only touched SOPS + tasks.md. After merge:

cd-apps.yml did not trigger — its paths: only include apps/**, packages/**, pnpm-lock.yaml.
cd-infra.yml triggered, but its default deploy path only recreates postgres prometheus loki promtail grafana postgres-exporter node-exporter cadvisor (explicitly listed in the docker compose up command). It copies the new .env to the server but leaves the Planner/Journal containers running with the old env.

The workaround was to SSH and run docker compose --env-file .env up -d planner manually to pick up the new BROUTER_URL.

Why it matters

docs/deployment.md:43 §Secrets rotation says "Commit + push + merge → cd-apps redeploys the Planner with the new token" — not true today. Anyone rotating BROUTER_AUTH_TOKEN, SENTRY_AUTH_TOKEN etc. via SOPS will silently leave the old value in the running app containers, with nothing going red.

This is a silent failure, which is what makes it worth fixing rather than documenting.

Fix options

Add infrastructure/secrets.app.env to cd-apps.yml paths:. Pro: simplest; cd-apps already rebuilds+deploys planner/journal. Con: a full image rebuild for a secret-only change is overkill.
Extend cd-infra.yml to restart planner journal when secrets.app.env changed. Pro: fast (no image rebuild). Con: a little more logic (paths-filter or git diff in the workflow).
Document the gap + require a restart_all run after any SOPS app-secret edit. Pro: zero workflow change. Con: easy to forget; makes secret rotation a manual two-step.

Option 2 is probably the right default. Happy to take suggestions.

Current evidence (re-checked 2026-07-26)

.forgejo/workflows/cd-apps.yml — paths: is apps/**, packages/**, pnpm-lock.yaml; no infrastructure/** at all.
.forgejo/workflows/cd-infra.yml:80 — the explicit service list in docker compose --env-file .env up -d --remove-orphans postgres prometheus loki promtail grafana postgres-exporter node-exporter cadvisor still excludes planner/journal. (Line 78 is the restart_all=true branch, which does cover them — that is the current manual escape hatch.)
docs/deployment.md:43 — the documentation claim still does not match behaviour.

Changed since the original report

Workflows moved from .github/workflows/ to .forgejo/workflows/, so any fix belongs there. GitHub Actions is off and the GitHub repo is retired.
docs/deployment.md still tells you to run gh workflow run cd-brouter.yml for step 4, which no longer works — worth fixing in the same pass, since a reader following those steps today gets stuck twice.
The restart_all=true input on cd-infra is the accurate description of today's workaround, so option 3 is at least now expressible without SSH.

Migrated from GitHub issue #294 (opened 2026-04-24 by @stigi, before the Forgejo migration). **Re-verified against the current `.forgejo/` workflows on 2026-07-26 — the bug is fully intact**, and the ported workflows carried it over verbatim. Line references below are updated to the current files. ## Problem Editing `infrastructure/secrets.app.env` (SOPS) does not restart the `planner`/`journal` containers on the flagship. The BRouter cutover (PR #293) exposed this: the PR only touched SOPS + tasks.md. After merge: - `cd-apps.yml` did **not** trigger — its `paths:` only include `apps/**`, `packages/**`, `pnpm-lock.yaml`. - `cd-infra.yml` triggered, but its default deploy path only recreates `postgres prometheus loki promtail grafana postgres-exporter node-exporter cadvisor` (explicitly listed in the `docker compose up` command). It copies the new `.env` to the server but leaves the Planner/Journal containers running with the old env. The workaround was to SSH and run `docker compose --env-file .env up -d planner` manually to pick up the new `BROUTER_URL`. ## Why it matters `docs/deployment.md:43` §Secrets rotation says "Commit + push + merge → `cd-apps` redeploys the Planner with the new token" — **not true today**. Anyone rotating `BROUTER_AUTH_TOKEN`, `SENTRY_AUTH_TOKEN` etc. via SOPS will silently leave the old value in the running app containers, with nothing going red. This is a *silent* failure, which is what makes it worth fixing rather than documenting. ## Fix options 1. **Add `infrastructure/secrets.app.env` to `cd-apps.yml` `paths:`.** Pro: simplest; cd-apps already rebuilds+deploys planner/journal. Con: a full image rebuild for a secret-only change is overkill. 2. **Extend `cd-infra.yml` to restart `planner journal` when `secrets.app.env` changed.** Pro: fast (no image rebuild). Con: a little more logic (paths-filter or git diff in the workflow). 3. **Document the gap + require a `restart_all` run after any SOPS app-secret edit.** Pro: zero workflow change. Con: easy to forget; makes secret rotation a manual two-step. Option 2 is probably the right default. Happy to take suggestions. ## Current evidence (re-checked 2026-07-26) - `.forgejo/workflows/cd-apps.yml` — `paths:` is `apps/**`, `packages/**`, `pnpm-lock.yaml`; no `infrastructure/**` at all. - `.forgejo/workflows/cd-infra.yml:80` — the explicit service list in `docker compose --env-file .env up -d --remove-orphans postgres prometheus loki promtail grafana postgres-exporter node-exporter cadvisor` still excludes planner/journal. (Line 78 is the `restart_all=true` branch, which *does* cover them — that is the current manual escape hatch.) - `docs/deployment.md:43` — the documentation claim still does not match behaviour. ## Changed since the original report - Workflows moved from `.github/workflows/` to `.forgejo/workflows/`, so any fix belongs there. GitHub Actions is off and the GitHub repo is retired. - `docs/deployment.md` still tells you to run `gh workflow run cd-brouter.yml` for step 4, which no longer works — worth fixing in the same pass, since a reader following those steps today gets stuck twice. - The `restart_all=true` input on `cd-infra` is the accurate description of today's workaround, so option 3 is at least now expressible without SSH.

ullrich referenced this issue from a commit

2026-07-26 18:06:10 +00:00

fix(cd): restart the app containers when SOPS app secrets change

ullrich referenced this issue from a pull request that will close it,

2026-07-26 18:06:45 +00:00

fix(cd): restart the app containers when SOPS app secrets change #18

ullrich closed this issue

2026-07-26 18:09:07 +00:00

ullrich referenced this issue

2026-07-26 18:26:12 +00:00

POI index never auto-refreshes — neither refresh timer is installed #19