overview.json was missing a comma between the "Health Status" and
"App Error Rate (from logs)" panel objects, making the whole file
invalid JSON. Grafana's file provisioner skips dashboards that fail to
parse, so the entire "trails.cool Overview" dashboard (5 panels) never
loaded — a silent gap in observability. Validated with json.load.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two cardinality/volume cuts to keep Prometheus storage sustainable
within the size budget:
- scrape_interval 5s -> 20s (evaluation_interval too). 5s tripled the
sample volume vs the 15s default for no benefit on a single-box
deployment; all rules use [5m] windows and for: >= 1m.
- Drop high-cardinality cAdvisor families we never chart or alert on
(container_fs_*, blkio_*, tasks_state, memory_failures_total,
memory_numa_*, network_tcp*/udp*) on both cadvisor jobs. Dashboards
only use container CPU, memory, and start_time. This was the bulk of
the ~12.4k active series.
Validated with `promtool check config`.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prometheus size-based retention counts the WAL toward its budget. With
retention.size=1GB and a stuck/corrupt WAL that had grown to 2.1GB, the
total was permanently over budget, so Prometheus deleted every
freshly-compacted block the moment it was written. Net effect: only the
in-memory ~2h head block was ever queryable — metrics appeared to "start
this morning" no matter when you looked, and last night's Caddy 502
spike (planner /api/overpass upstream stall, 18:33-18:34 UTC) was
unobservable in Prometheus (logs survived in Loki).
The corrupt WAL was cleared out-of-band on the flagship. This raises the
size backstop to 6GB so compacted blocks survive to the intended 15d
time-based retention. Disk is 38G (43% used), so 6GB is safe headroom.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Garmin Connect as the third connected-services provider (spec:
garmin-import). The interesting parts:
- Push-first ingestion: Garmin has no list endpoint. The webhook
normalizes ping (callbackURL) and push (inline) notification batches
into events; the slow work (authorized FIT download, FIT→GPX via the
shared converter, activity creation) runs in a garmin-import-activity
pg-boss job so the webhook answers fast. Callback URLs are validated
against Garmin's API host before any fetch (SSRF guard).
- History via backfill requests: /sync/import/garmin is a date-range
requester with honest async progress (no pick list — the concept
doesn't exist in a push model). Ranges chunk to Garmin's 90-day cap;
overlaps are free via sync_imports dedupe. Requests persist in
import_batches via two new nullable columns (range_start/range_end).
- OAuth2 + PKCE on the existing oauth credential kind. Design
correction from apply: the verifier rides a short-lived httpOnly
cookie scoped to the callback path — the state param is visible in
redirect URLs and must never carry it. Manifests opt in via pkce:true.
- Deregistration notifications flip the connection to 'revoked'
(row kept for audit, imports retained, re-connect prompt shown).
- Framework evolutions, all additive: parseWebhook returns
WebhookEvent[] (Garmin batches; Wahoo adapted), manifest gains
configured()/importUrl/pkce, importActivity accepts summary stats
for FIT-less imports, manager gains markRevoked.
- Env-gated: no GARMIN_CLIENT_ID → provider hidden on
/settings/connections. Privacy manifest entry (DE+EN). i18n en+de.
Rollout (§6) stays gated on the Garmin Developer Program application
(submitted 2026-06-07). Fixtures are doc-shaped; the staging soak
swaps in recorded payloads if shapes differ.
Gate: typecheck ✓ lint ✓ unit+integration ✓ e2e 70/72 + both known
flakes green isolated ✓ openspec validate ✓
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Garmin Connect as the third connected-services provider, modeled on
wahoo-import but adapted to Garmin's push-first API: OAuth2+PKCE on the
existing oauth credential kind, ping/push webhook ingestion with async
pg-boss processing and an SSRF allowlist on callback URLs, date-range
backfill instead of a pick list (Garmin has no activity-list endpoint),
mandatory deregistration handling. Route push (Courses API) explicitly
deferred to a follow-up, mirroring wahoo-route-push.
Build is fixtures-first; rollout tasks are gated on Garmin Connect
Developer Program approval, which runs in parallel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flagship ⇄ staging, both directions, 2026-06-07: follows settled to
Accepted in seconds, first polls ingested each side's public
activities, both feeds render remote attribution. The cross-origin
Accept fix (#490) verified working in production.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wiring follows the IS_FLAGSHIP pattern: the compose env defaults every
FEDERATION_* var to empty (off — the safe self-host default; all
federation surfaces 404), and cd-apps.yml appends
FEDERATION_ENABLED=true + FEDERATION_LOG_LEVEL=info to the flagship's
app.env. FEDERATION_KEY_ENCRYPTION_KEY was already in SOPS.
Rollout 12.2/12.3 soaked on staging against a real Mastodon
(2026-06-06/07); the trails↔trails Accept bug found by the §11 harness
is fixed and deployed. Rollback: drop the two echo lines, merge, rerun
cd-apps — instant off, follow rows persist.
First boot with the flag enqueues backfill-user-keypairs (idempotent)
and registers the federation jobs. Applied via manual cd-apps dispatch
after merge since workflow-file changes don't trigger the path filter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
12.1/12.2/12.3/12.6 marked with provenance (additive schema behind the
off flag on prod; the 2026-06-06/07 staging soak against a real
Mastodon covered inbound + push delivery; rollback documented in the
runbook). 12.4 annotated: protocol verified by the two-instance
harness, live staging⇄preview soak queued now that previews federate.
12.5 (prod flag flip) explicitly left as an operator decision.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every PR preview becomes a second live trails instance, so the
trails-to-trails soak (social-federation 12.4) can run against real
DNS + TLS + Caddy: follow from staging.trails.cool to
pr-<N>.staging.trails.cool and back. Same flag + key wiring as
persistent staging; preview teardown leaves the remote side with
ordinary dead-instance delivery failures, which fediverse software
already expires.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 11.4 harness — e2e/federation/: postgres (two DBs) + a Caddy with
an internal CA terminating real HTTPS between two complete journal
containers (journal-a.test / journal-b.test), driven by an opt-in
integration test (FEDERATION_TWO_INSTANCE=1, via run.sh). The driver
seeds users straight into each DB, mints a signed session cookie
(known SESSION_SECRET), follows across instances through the real
/follows/outgoing route, and asserts the full pipeline: Follow →
auto-Accept → settle → first outbox poll → bob's public activity
rendered in alice's /feed with @bob@journal-b.test attribution. Plus a
wire-level 11.3 check: unsigned Create(Note) → 4xx, no DB writes.
And the harness immediately earned its keep — it caught a real
trails↔trails bug Mastodon interop never could: our Follow ids are
fragment URIs on OUR domain, so the Follow embedded in another trails
instance's Accept is cross-origin and Fedify rightly distrusts it;
re-fetching the id returns the actor document, getObject() yields a
Person, and the Accept/Reject/Undo listeners bailed silently — follows
stayed Pending forever. The listeners now fall back to the wire
objectId (captured BEFORE getObject(), which memoizes the fetched
document and changes what objectId reports) validated against our
Follow-id shape + the personal inbox's ctx.recipient. Forgery-safe:
settleOutgoingFollow only matches a Pending row toward the
HTTP-Signature-authenticated sender.
Supporting changes:
- federation.server.ts: env-gated allowPrivateAddress
(FEDERATION_ALLOW_PRIVATE_ADDRESS=true) so the harness's RFC 1918
Docker network is fetchable — testing only, loudly documented.
- Root .dockerignore: local builds shipped a 1GB+ context and
overlaid host (darwin) node_modules over the image's own install
via COPY . . — CI never noticed because it builds from clean
checkouts. Context is now ~5MB.
- tasks.md: 11.1–11.7 marked with provenance notes; design.md gains
the cross-origin embedded-object lesson.
Gate: typecheck ✓ lint ✓ unit+integration (FEDERATION_INTEGRATION=1) ✓
e2e 71/72 + known komoot flake green isolated ✓ harness run clean from
scratch (2/2, teardown included) ✓
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the 7.4 remainder. Fedify's FetchError carries the failed
Response, so a 429 from a remote's outbox now reads Retry-After
(delta-seconds or HTTP-date per RFC 9110), arms a per-host backoff
(15 min default, capped at the 1 h poll interval), and pollRemoteActor
skips backing-off hosts before doing any DB or network work. State is
in-process like the pacing map — a restart costs at most one extra
request that re-arms the backoff. last_polled_at is deliberately not
stamped on a rate-limited poll so the hourly sweep retries.
Unit tests cover the header parser (incl. the V8 footgun where
Date.parse reads bare negative integers as years), default/cap
behavior, and window expiry; an integration test drives the full
FetchError path and asserts the second poll never touches the network.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
8.1/8.2: listSocialFeed is now a UNION ALL of local and remote
branches, sorted on COALESCE(remote_published_at, created_at):
- local rows: visibility='public' from accepted local follows — and
the previously missing accepted_at filter is added (the spec's
'Pending follows contribute nothing' scenario)
- remote rows: gated structurally by joining the viewer's OWN accepted
follow against the originating actor — which is exactly the
followers-only audience rule (a row reaches only viewers whose
follow brought it in); attribution from the remote_actors cache;
cards link outward to the canonical origin page (no local detail
page for remote rows)
9.1: annotated — local Pending button shipped with locked accounts;
remote Pending lives on /follows/outgoing.
9.2: already enforced + tested since §3 (actor/webfinger 404).
9.3 hardening: deliver-activity now re-checks the OWNER's profile
visibility at send time, closing the enqueue→delivery flip window
(enqueue-side and inbound-side gates already existed).
Integration tests: the §8 audience-leak guard (A sees followers-only,
B does not), public remote attribution + outward link, pending
contributes nothing, mixed local/remote COALESCE ordering.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tasks 7.1–7.3, 7.5 (+7.4 pacing; Retry-After-duration backoff noted as
remaining). Resolves the activities.owner_id open question.
Schema (design decision from the open question):
- activities.owner_id nullable + check constraint enforcing exactly
one of (owner_id, remote_actor_iri) — the follows pattern again.
- remote_published_at carries the origin's publish time for the §8
feed sort; index on remote_actor_iri for the feed join.
- Compiler-audited fallout: notification fan-out, the Note object
dispatcher, and the activity detail loader explicitly skip/404
remote rows (their canonical page is the origin; feed links there).
Every other surface joins users on owner_id and excludes remote rows
structurally.
Ingestion:
- federation-ingest.server.ts: parseOutboxItem targets exactly our own
outgoing Create(Note) shape (PropertyValue stats, first-paragraph
name, audience from to/cc); unknown items skipped, never fatal.
- ingestRemoteActivities: replay-safe via unique remote_origin_iri,
conflict-streak early exit (outboxes are newest-first).
- pollRemoteActor: signed fetch (Authorized Fetch via a local
follower's key), outbox resolution via remote_actors cache with
actor-doc fallback (which refreshes the cache), 1 req/5 s per-host
pacing, last_polled_at stamping. Network + pacing injectable.
Jobs: poll-remote-actor (per-actor; the §4 Accept listener already
enqueues it as the first poll) + poll-remote-outboxes (5-min cron
sweep over accepted remote follows not polled within the hour).
Tests: parse-shape units; integration suite for ingestion provenance,
audience→visibility mirroring, replay safety, the DB author invariant,
poll flow (actor-doc → collection → page), signer requirement, and
due-polling selection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
10.1 Privacy manifest (legal/privacy):
- German legal half: Föderation entry under Empfänger/Drittanbieter —
what a public profile exposes, what accepted followers' servers
receive, the loss-of-control over delivered copies, Art. 6(1)(a)
basis, private profiles don't federate.
- English manifest half: dedicated Federation (ActivityPub) section —
actor object contents, push delivery + retraction semantics (incl.
the tombstone caveat), encrypted-at-rest signing keys, remote actor
cache, ingested remote content with the followers-only viewer gate,
inbox logging/rate limits. PRIVACY_LAST_UPDATED bumped.
10.2 docs/deployment.md federation runbook: enabling per environment,
key rotation posture, abuse monitoring, and the troubleshooting
checklist distilled from the 2026-06-06/07 soak (IP families first,
no outbox backfill, tombstones, the 10s delivery timeout, actor
re-fetch).
10.3 Home marketing blurb (en+de) aligned to what actually ships:
Mastodon can follow you + trails-to-trails outbound — no more 'follow
friends anywhere' overstatement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tasks 6.1–6.6. A local user can follow a user on another trails
instance: WebFinger-resolve the handle, verify the target runs
trails.cool via NodeInfo — checked against the ACTOR IRI's host, never
the handle's domain (split-domain constraint) — record a Pending
follow row, deliver a signed Follow. The §4 inbox listeners already
settle (Accept) or drop (Reject) the row.
- federation-outbound.server.ts: followRemoteActor / cancelRemoteFollow
(Undo delivery) / listOutgoingRemoteFollows. Network steps (lookup,
NodeInfo, delivery) injectable for offline integration tests.
Software allowlist: trails-cool.
- /follows/outgoing: follow-by-handle form + pending/accepted list +
cancel; linked from the feed empty state; i18n en+de. Remote actors
have no local profile page, so the remote Pending state lives here
(the profile Pending button from locked accounts covers local).
- /.well-known/trails-cool now publishes software: trails-cool (6.2).
- Clear 4xx codes: invalid_handle, local_handle, remote_resolve_failed,
not_trails (6.3).
- Tests: handle-parser + allowlist units; integration suite for the
Pending lifecycle (create/idempotent/refuse-non-trails/refuse-
unresolvable/own-host/cancel+Undo) with injected network deps; e2e
anonymous-redirect guard for the new page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 2.2.x line was uninstallable when federation landed
(@fedify/webfinger@2.2.x wasn't published); fixed upstream, so move to
latest. Exact pin stays deliberate — a federation library defines the
wire shapes remote instances see.
Verified: typecheck, lint, full unit + integration suites (245 tests,
including the wire-shape assertions: JRD, actor attachment arrays,
PropertyValue fields, dereferenceable Note ids) all green on 2.2.5.
Lockfile diff is a clean 24-line swap of the fedify package family.
Note: a transient cross-suite flake surfaced when running both
integration suites concurrently (shared journal.follows state) —
pre-existing, unrelated to the upgrade; clean re-run green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Prompted by Hollo's split-domain setup. Two parts:
- docs/ideas/split-domain-handles.md: offering split handle/server
domains to self-hosters is config-level work — Fedify's origin
option natively accepts { handleHost, webOrigin } — and handles are
permanent identity, so apex handles matter. Single-domain stays the
default (Decision #18); post-launch polish.
- Interop constraint pinned in social-federation task 6.1 and
route-federation's gating decision: the trails-to-trails NodeInfo
check must run against the actor IRI's host after WebFinger
resolution, never the handle's domain — split-domain remote
instances would otherwise be wrongly refused.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ullrich's notes from the first real federation exchange (2026-06-06/07
staging ↔ Mastodon soak), expanded with the constraints the soak
surfaced: route map images in Notes, fediverse kudos (inbound
Like/Announce), comments from the fediverse, planned routes as
Events (Mobilizon), Wanderer interop, federated explore.
The fourth note item — remote follower counted but not listed — was a
bug, fixed separately in #475.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gap analysis of docs/architecture.md against openspec/ and docs/ideas/
found the envisioned-but-uncaptured remainder. This commit captures it:
New OpenSpec change (validated):
- route-federation — the collaboration half of the federation vision:
routes as dereferenceable trails:Route objects, Create/Update
fan-out, Invite/Accept collaboration mirroring (arch decision #2),
cross-instance Planner edits via HTTP-Signature-requested scoped
tokens (decisions #3/#12), mirror sync healing (#16). Depends on
social-federation §6 + route-sharing; carries the 2026-06-07 soak
lessons as design constraints.
New docs/ideas/ explorations:
- instance-administration — registration toggle, suspend/ban,
federation blocklists, reports (moderation now gates the federated
comments idea)
- social-interactions — local likes + comments (don't exist even
locally; the foundation federated kudos/comments attach to)
- activity-participants — group tagging with confirm/decline +
federated mentions (the participants jsonb column is an untyped stub)
- multi-day-collections — architecture open question #1, directions
evaluated
architecture.md cleanup:
- Mastodon-compat section annotated with shipped/captured state + the
live-soak interop lessons (attachment arrays, tombstones, 10s
timeout, no backfill)
- api.trails.cool removed (contradicted resolved decision #18)
- Phase 1 ticked (shipped); Phase 2/3 items annotated with where each
is tracked; specs/ → openspec/, activity.* → journal.*, cx21 → cx23
- brouter-web open question marked resolved-in-practice; new section
pointing to where the unshipped vision is tracked
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 2026-06-07 disk-full outage (flagship 100%, postgres crash-looping
on its pidfile, prod + staging down): cd-apps DOES prune images after
deploys, but it had been failing early all day (migration bug), so its
prune never ran — while ~10 staging/preview deploys kept pulling fresh
images with no prune of their own.
- cd-staging: prune superseded layers (until=1h guard against racing
in-flight pulls) after persistent staging and preview deploys.
- disk-maintenance.yml: NEW daily scheduled prune (04:30 UTC) that
also FAILS when the disk is still ≥85% after pruning — a redundant
alert channel for exactly the case where the Grafana disk alert
drowns in other noise, as it did during the incident.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
countFollowers counts every accepted follow row, but listFollowers
inner-joined users on follower_id — so a federated follower (NULL
follower_id, follower_actor_iri set) bumped the count while never
appearing in the list. Observed live: profile said '1 follower', list
below was empty. listFollowing had the same latent bug for outbound
trails-to-trails follows.
Both lists now left-join users + remote_actors: local entries link to
the local profile as before; remote entries display cached actor data
(IRI parsing as fallback) and link out to the remote profile.
Integration test pins count/list consistency for a remote follower.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
