Task group 4 of federation-hardening.
4.1 — FEDERATION.md at the repo root: actor discovery (WebFinger, actor,
NodeInfo), object/activity types with real JSON examples (Note, Create,
Delete, the narrow follow-graph inbox), addressing, HTTP-Signature
expectations, the two-layer dedup contract, durable delivery/retry
policy, and blocklist moderation semantics — precise enough for another
implementation to interoperate. Linked from README and docs/architecture.
4.2 — three prom-client metrics + a journal dashboard row:
- `federation_delivery_total{outcome}` — incremented in deliver-activity
(delivered/skipped/failed).
- `federation_inbox_dropped_total{reason}` — incremented at every inbox
drop (duplicate | blocked); this is the counter deferred from task 3.2.
- `federation_queue_depth` — gauge sampled at scrape time in
/api/metrics from PgBossMessageQueue.getDepth(); the restart-loss
regression detector.
Grafana journal.json gains a Federation row (delivery rate, queue depth,
inbox drops); the logs panels shift down to make room.
Verified: dashboard JSON valid; journal typecheck + lint clean; unit
suite 357 passing (route-template guard unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the route-label cardinality fix. The first cut collapsed
dynamic segments via regex (:id/:username/:provider) and capped the
distinct-route set at 200 with a /other overflow. In production that cap
filled almost entirely with vulnerability-scanner junk (`/.ssh/id_rsa`,
`/%00.aws/credentials`, …) on a first-come-first-served basis: only 3 real
templates made it in before the cap saturated, so legitimate routes hit
afterward were misbucketed into /other — the metric became useless for
per-route latency even though memory was bounded.
Replace the regex+cap approach with explicit matching against the journal's
known route templates (mirroring app/routes.ts). A `:param` segment matches
any single path segment; literals are preferred over params (so /routes/new
beats /routes/:id); anything matching no template collapses to /other
immediately — no per-path tracking, no cap race. Cardinality is hard-bounded
to (templates + 1) regardless of traffic, and scanner noise can never crowd
out real routes.
A co-located drift guard flattens the real route config and fails if
ROUTE_TEMPLATES and app/routes.ts ever diverge.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The httpRequestDuration histogram labeled every request with the raw URL
path (`url.split("?")[0]`), so every distinct path — `/activities/<uuid>`,
`/routes/<uuid>`, probed usernames, crawler junk — became a permanent
label-set. prom-client never evicts label-sets, so RSS grew linearly for
the life of the process and reset only on restart. On the flagship this
showed as ~25 MiB/day of unbounded journal memory growth; a live
`/api/metrics` scrape held 2,655 histogram series across 291 distinct
`route` values (344 KB body), dominated by per-UUID activity paths.
Add `normalizeRoute()` in metrics.server.ts: it collapses dynamic path
segments back to the `:param` templates declared in app/routes.ts (UUID
and numeric segments -> `:id`; `:username`/`:provider` slots keyed by
their preceding static segment), strips query/fragment, and enforces a
hard cap (MAX_ROUTES) that funnels anything beyond the known route table
into a single `/other` bucket as an absolute backstop. Cardinality is now
proportional to the route table, not to traffic.
Logs still record the raw path (`logger.info`) — only the metric label is
bounded.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
/users/bruno (and any route that imports demo-bot.server.ts) was
returning 500 on prod after #258 because the route's module graph
loaded metrics.server.ts a second time, re-running collectDefaultMetrics
and `new client.Gauge(...)` — prom-client's global registry rejects
duplicate metric names, so the route module failed to load.
Guard every metric creation via `getSingleMetric` so a second module
load reuses the existing gauges/histograms.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a background demo user (`bruno`) plus two pg-boss jobs that generate
public trekking routes and activities in inner Berlin and prune them
after 14 days. Disabled by default everywhere; flip `DEMO_BOT_ENABLED`
only in prod.
- Schema: `synthetic` boolean on routes + activities
- `ensureDemoUser()` idempotent insert + badge on /users/bruno
- Generation gate: 07-21 Berlin local, p=0.12, 40-route cap in 14d
- Initial 4-walk backfill on first enable; retention via daily prune
- Prometheus gauges `demo_bot_synthetic_routes_total` / `_activities_total`
- Unit + DB-gated integration + E2E tests
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Health endpoints:
- /api/health (Journal) and /health (Planner) with DB connectivity check
- Docker healthchecks updated to use app health endpoints
Structured logging:
- Pino with JSON output in production, pretty-print in dev
- Request logging middleware in Planner (method, path, status, duration)
- Replaced console.log/error with structured logger in email and auth flows
Prometheus metrics:
- prom-client with default Node.js metrics + custom histograms/gauges
- /metrics endpoints on both apps
- http_request_duration, planner_active_sessions, brouter_request_duration
Monitoring stack:
- Prometheus, Loki, Grafana containers in docker-compose
- Grafana provisioned with datasources, dashboards, and alert rules
- Caddy access logging (JSON to stdout for Loki)
- grafana.trails.cool with basic auth via Caddy
Dashboards and alerting:
- Overview: request rate, error rate, latency p50/p95/p99
- Planner: active sessions, connected clients, BRouter latency
- Infrastructure: memory, CPU, event loop lag
- Alerts: disk >80%, app down 2min, error rate >5%
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
