Single-file bind mounts (./foo.yml:/etc/foo.yml) pin to the host file's
inode at container-create time. The CD pipeline scp's a replacement file
(new inode), so the running container keeps reading the OLD inode —
`docker compose up -d` won't recreate on a content-only change, and
neither restart/SIGHUP/`caddy reload` re-reads the new file. Net effect:
config-only infra PRs deployed "successfully" but never took effect
(confirmed with PR #500's prometheus.yml; the WAL retention work only
applied because #498 also changed docker-compose.yml, forcing a
recreate). Caddyfile changes had the same latent gap.
Switch the four single-file config mounts to DIRECTORY mounts, which
resolve children live so a reload/restart picks up the new file:
- prometheus, loki, promtail: mount ./<svc> dir (loki/promtail
--config.file paths updated to the real filenames).
- caddy: move Caddyfile into caddy/ and mount the dir; ./sites overlays
/etc/caddy/sites (caddy/sites/.gitkeep keeps the mountpoint). Container
path /etc/caddy/Caddyfile is unchanged, so every `caddy reload --config
/etc/caddy/Caddyfile` ref still works. scp source paths in cd-infra and
cd-apps updated to ship the caddy/ dir.
cd-infra now applies config-only changes after `up -d`: SIGHUP prometheus
(zero downtime), restart loki+promtail (no SIGHUP reload), caddy reload
(graceful). Mirrored prometheus/loki mounts in docker-compose.dev.yml.
Validated: docker compose config (both files), caddy validate from the
new path, read-only-parent + sites-overlay mount mechanics, workflow YAML.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wiring follows the IS_FLAGSHIP pattern: the compose env defaults every
FEDERATION_* var to empty (off — the safe self-host default; all
federation surfaces 404), and cd-apps.yml appends
FEDERATION_ENABLED=true + FEDERATION_LOG_LEVEL=info to the flagship's
app.env. FEDERATION_KEY_ENCRYPTION_KEY was already in SOPS.
Rollout 12.2/12.3 soaked on staging against a real Mastodon
(2026-06-06/07); the trails↔trails Accept bug found by the §11 harness
is fixed and deployed. Rollback: drop the two echo lines, merge, rerun
cd-apps — instant off, follow rows persist.
First boot with the flag enqueues backfill-user-keypairs (idempotent)
and registers the federation jobs. Applied via manual cd-apps dispatch
after merge since workflow-file changes don't trigger the path filter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hardening from two incidents on 2026-06-06/07:
Schema drift (morning): drizzle-kit push exits 0 even when it aborts
on an interactive prompt it can't render in CI, so cd-staging's
set -euo pipefail never fired and a month of staging schema drift
accumulated silently until new code hit missing columns. All three
drizzle push call sites now tee output and fail the deploy on any
'Error:' line.
Production outage (overnight, ~9h): cd-infra and cd-apps had no
failure handling at all. A network-option change stopped postgres for
a network recreation that then deadlocked on containers from other
compose projects holding trails-shared; the script carried on, the
stack stayed down. Both scripts now run set -euo pipefail and gate on
container health at the end (postgres+journal for cd-infra,
journal+planner for cd-apps) so a deploy that leaves the stack down
is a red X, not a shrug.
docs/deployment.md gains the cross-project manual procedure for
network-changing deploys — the CD workflows only manage their own
compose project and cannot apply those safely.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Self-hosted instances no longer inherit the trails.cool flagship Sentry
DSNs. Code paths now read DSN strictly from env — unset = no Sentry
init, no events sent.
Flagship continues to report unchanged: cd-apps.yml and cd-staging.yml
inject the public DSNs as workflow env vars, which feed into:
- runtime via \`infrastructure/app.env\` / \`staging.env\`
(\`SENTRY_DSN_JOURNAL\` / \`SENTRY_DSN_PLANNER\` → compose env per service)
- build time via docker build-arg \`VITE_SENTRY_DSN\` (journal only;
planner has no client Sentry init).
Sentry DSNs are public-by-design (transmitted unencrypted from the
client JS bundle), so embedding them as plaintext workflow env vars
is no worse than the runtime exposure. Forks should replace these with
their own DSNs or remove the workflow env lines to ship Sentry-free.
Files changed:
- apps/journal/server.ts: \`process.env.SENTRY_DSN\` only, no fallback
- apps/planner/server.ts: same
- apps/journal/app/lib/sentry.client.ts: \`import.meta.env.VITE_SENTRY_DSN\`
only; falsy = skip init
- apps/journal/Dockerfile: new \`ARG VITE_SENTRY_DSN\` baked into build
- infrastructure/docker-compose.yml: pass \`SENTRY_DSN\` per service
- infrastructure/docker-compose.staging.yml: same
- .github/workflows/cd-apps.yml: workflow env + build-arg + app.env echo
- .github/workflows/cd-staging.yml: same
Full repo: pnpm typecheck, pnpm lint, pnpm test, journal build all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Re-pushing an edited route to Wahoo now updates the existing remote
route via PUT against the stored remote_id instead of POSTing a new
copy. external_id drops the version suffix and identifies the logical
route. sync_pushes is keyed by (user, route, provider) and tracks
last_pushed_version for the "local newer" UI state.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
cd-apps already scp's `infrastructure/Caddyfile` to the server
alongside docker-compose.yml — but the running Caddy container
doesn't auto-pick-up config changes. cd-infra is the workflow that
calls `caddy reload`, and it only triggers on `infrastructure/`
paths. So any Caddyfile edit shipped through cd-apps (e.g. the
`lb_try_duration` block in PR #329, deployed as part of an apps
push) sits on disk unapplied until the next cd-infra run.
This was real today: PR #329 added `lb_try_duration 30s` to silence
deploy-time 502s. PR #331 (a vite.config edit, apps path) merged
right after and triggered cd-apps, which copied the new Caddyfile
but didn't reload Caddy. The very next planner restart in that
deploy promptly produced three 502s — the exact thing #329 was
supposed to prevent. We applied the reload manually via SSH after
the fact.
This commit makes that reload happen on every cd-apps deploy. The
operation is idempotent (Caddy validates first, swaps live, no
downtime) so doing it even when Caddyfile is unchanged costs
nothing. `|| true` keeps the deploy from failing if Caddy is
unhealthy at deploy time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The token lives in `secrets.infra.env`, which `cd-infra.yml` merges
together with `secrets.app.env` into the server's
`/opt/trails-cool/.env`. The cd-apps workflow's own `app.env`
intentionally does NOT carry the token (apps don't need it at
runtime), so the original `grep ... .env` was correct. My earlier
edit in this branch swapped the path to `app.env` and would have
broken the annotation hook the moment it actually worked.
Restored `.env` and updated the inline comment to make the file
ownership explicit (cd-infra populates it; cd-apps reads it).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The journal/planner deploy in cd-apps.yml does `docker compose up -d
journal planner`, which stops the old container and starts the new
one — Caddy keeps forwarding requests during the ~10–30s gap and
returns 502s. The caddy-502-rate alert (threshold > 0 for 2m)
correctly trips, every time.
Two production changes plus a long-broken workflow detail:
- infrastructure/Caddyfile — add `lb_try_duration 30s` /
`lb_try_interval 250ms` to the journal and planner reverse_proxy
blocks. Caddy now holds and retries the upstream for up to 30s
during a restart instead of 502'ing immediately. Real outages
(upstream unreachable longer than 30s) still 502 and the alert
still fires for those.
- infrastructure/grafana/provisioning/alerting/alerts.yml — add a
comment documenting why caddy-502-rate stays at threshold > 0:
with lb_try_duration in front of it, the alert no longer
conflates "deploy in flight" with "real outage."
- .github/workflows/cd-apps.yml — fix a long-silent bug: the
Grafana deploy-annotation step was reading
GRAFANA_SERVICE_TOKEN from `.env`, but the secrets file we scp
to /opt/trails-cool is named `app.env`. The token check failed
silently and the curl was being skipped on every deploy.
Switching to `app.env` so deploys actually annotate Grafana.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The old home was an h1, subtitle, and two auth buttons — visitors
arriving at trails.cool had no idea what it was, and self-hosted
instances had nothing interesting to land on.
The new home is one layout that serves both audiences:
- Hero: a product-describing h1 ("Federated outdoor journal") +
tagline. The site name already lives in the top banner and nav brand,
so the h1 carries the pitch instead of triplicating "trails.cool".
- Auth CTAs: Register (blue) + Sign In (outlined) get primary weight.
Planner demotes to a small "Or try the Planner without an account →"
link below them — it's a nice escape hatch but not the goal.
- Marketing cards: on the flagship only, a 2x2 grid of emoji-icon cards
for Planner / Journal / Federation / Ownership, matching the Planner
home's visual weight. `IS_FLAGSHIP=true` toggles it. Self-hosters get
a "Powered by trails.cool — about the project" link back to the
flagship instead, so they don't have to write their own marketing.
- Public feed: reverse-chronological list of the 20 most recent public
activities on the instance, with owner + distance + date. Empty state
for fresh installs.
Plumbing:
- `listRecentPublicActivities(limit)` in activities.server.ts joins
users so the feed can render "by <displayName>" in one query.
- `IS_FLAGSHIP` env wired through docker-compose.yml; cd-apps and
cd-infra both write `IS_FLAGSHIP=true` alongside `DOMAIN=trails.cool`,
so self-hosters (who deploy without these workflows) default to off.
Specs:
- `activity-feed`: new "Instance-wide public activity feed" requirement
so the visibility contract is pinned (public in feed, unlisted +
private are not).
- `journal-landing`: new small spec capturing the hero / marketing /
feed layout contract and the `IS_FLAGSHIP` toggle, so future
instance-branding or empty-state changes have a stable anchor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compose healthcheck for both apps was `curl -sf http://localhost:.../health`,
but node:25-slim (Debian trixie-slim) ships neither curl nor wget, so
the check always reported the containers as `unhealthy` even while they
were happily serving 200s. Install curl in both Dockerfiles.
Separately, today's flagship BRouter cleanup (PR #297) left an orphan
`brouter` container behind because cd-infra's `docker compose up -d <list>`
doesn't pass `--remove-orphans`. I had to remove it by hand. Pass the
flag in both cd-infra and cd-apps so future service deletions clean up
on the next deploy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BusyBox wget in the Grafana Alpine container doesn't support
--post-data/--header. Grafana's image includes curl.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
GRAFANA_SERVICE_TOKEN is in secrets.infra.env, which the infra
workflow merges into .env on the server. The apps workflow was
reading from app.env (which only has app secrets).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Posts an annotation to Grafana after each successful deploy, tagged
with "deploy" and the service name. Annotations appear as vertical
markers on dashboards, making it easy to correlate deploys with
metric changes.
Uses GRAFANA_SERVICE_TOKEN from SOPS secrets — must be added via
sops infrastructure/secrets.infra.env. Gracefully skips if the
token is not configured.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The token file may have a trailing newline that causes sentry-cli
API requests to fail with "API request failed".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Disable Sentry Vite plugin telemetry (telemetry: false)
- Pass SENTRY_RELEASE as build-arg (not secret) since it's just the git sha
- Keep only SENTRY_AUTH_TOKEN as Docker secret
The release version was showing as *** because Docker secrets get masked
by GitHub Actions. Git sha is not sensitive — safe as a build arg.
The sourcemap warning from react-router is a known upstream issue (ships
without sourcemaps) and is harmless.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Approve @sentry/cli and esbuild build scripts so postinstall runs
- Use --mount=type=secret instead of ARG for SENTRY_AUTH_TOKEN
(fixes Docker warning about secrets in build args)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The decrypted secrets.app.env contains values with angle brackets
(SMTP_FROM) that bash interprets as redirects. The build job only
needs SENTRY_AUTH_TOKEN, so extract it with grep instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Write decrypted env files to infrastructure/ instead of /tmp/ so
strip_components: 1 correctly places them on the server.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Secrets:
- Add .sops.yaml with age encryption config
- Add encrypted secrets.app.env (app secrets) and secrets.infra.env (Grafana OAuth)
- CD decrypts at deploy time with AGE_SECRET_KEY — all other secrets
move out of GitHub Actions into version-controlled encrypted files
Split CD:
- cd-apps.yml: triggered by apps/packages changes, builds Docker images, deploys apps
- cd-infra.yml: triggered by infrastructure/ changes, copies configs, restarts services
- Remove monolithic cd.yml
Grafana auth:
- GitHub OAuth (trails-cool org), disable login form
- Remove Caddy basic_auth block and all GRAFANA_* env vars/secrets
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
