chore(openspec): archive federation-hardening
All 13 tasks complete (implementation shipped in #570–573; staging verification in #574). Archive via `openspec archive` (CLI 1.6.0): - Creates openspec/specs/federation-operations/spec.md — the new capability: durable federation queue, inbound replay defense, instance blocklist, published protocol doc, delivery observability (Purpose filled in; CLI leaves a TBD placeholder). - Applies the MODIFIED requirement to openspec/specs/social-federation: "Push delivery on local activity create" now guarantees persistent queueing + a "Fan-out survives a deploy" scenario. - Moves the change to openspec/changes/archive/2026-07-13-federation-hardening/. Both specs pass `openspec validate --strict`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer
parent
55da03132a
commit
42149d8e02
8 changed files with 51 additions and 2 deletions
|
|
@ -0,0 +1,2 @@
|
|||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
## Context
|
||||
|
||||
Fedify is configured with `kv: new PostgresKvStore()` but `queue: new InProcessMessageQueue()` (`apps/journal/app/lib/federation.server.ts:146,156`). Fedify uses the queue for outbound delivery (including its retry scheduling) and inbox processing; in-process means a deploy or crash drops everything in flight. Meanwhile `federation-delivery.server.ts` already enqueues *fan-out* jobs via pg-boss (`enqueueOptional`) — so the codebase has two queueing layers, one durable (ours) and one volatile (Fedify's). Inbound: Creates dedup via the `activities.remoteOriginIri` unique constraint; other activity types (Like, Undo, Delete, Update, Accept) have no replay protection. There is no blocklist of any kind (grep-confirmed). wanderer's review (`docs/inspirations.md`) catalogs the same gaps on their side and validates the fix list; their SSRF-safe outbound client and 490-line protocol doc are the positive references.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- No federation activity is lost to a restart; retries follow the spec's promised backoff/budget durably.
|
||||
- Replayed or duplicated inbound activities are harmless no-ops.
|
||||
- An operator can block a hostile instance with one documented action, effective in both directions.
|
||||
- The protocol surface is documented well enough that another implementation (wanderer, a future project) can interoperate deliberately.
|
||||
|
||||
**Non-Goals:**
|
||||
- Moderation UI, per-actor blocks, allowlist mode, object-level reporting flows.
|
||||
- Changing the object model, mirror semantics, or delivery topology (`route-federation`'s scope).
|
||||
- Replacing Fedify.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Fedify `MessageQueue` implemented over pg-boss
|
||||
|
||||
New `federation-queue.server.ts` implementing Fedify's `MessageQueue` interface (`enqueue` with delay support + `listen`) on pg-boss, mirroring how `PostgresKvStore` adapted `KvStore`. Rationale: pg-boss already provides durable storage, delayed jobs, retry bookkeeping, and a worker loop the journal runs — a second bespoke queue table would duplicate it. Fedify's own retry policy stays the policy owner; pg-boss supplies durability. The existing fan-out enqueue in `federation-delivery.server.ts` is unchanged (it feeds Fedify; Fedify's sends become durable underneath).
|
||||
|
||||
*Alternative:* Fedify's Redis/Postgres community queue adapters — adds a dependency for less control than ~100 lines over infrastructure we already operate.
|
||||
|
||||
### 2. Replay defense: processed-activity table with TTL
|
||||
|
||||
`federation_processed_activities` (activity IRI primary key, received_at). Inbox handlers insert-or-skip (`ON CONFLICT DO NOTHING`; conflict → drop as duplicate) before side effects. Rows older than 30 days swept by the jobs worker — replay of month-old activities is already rejected by HTTP-signature date freshness. Creates keep their existing `remoteOriginIri` idempotency as a second layer.
|
||||
|
||||
### 3. Blocklist checked at three boundaries, domain-keyed
|
||||
|
||||
`federation_blocked_instances(domain, reason, created_at)`. Checks: (a) inbox — activities from actors on blocked domains are dropped before processing (202-and-discard, no error oracle); (b) delivery enqueue — recipients on blocked domains are filtered out; (c) outbox polling / actor fetch — refuses blocked domains. Domain matching is exact-host (subdomain wildcarding deferred until needed). v1 management is a documented SQL statement in the ops runbook; the table is the API an admin UI can later sit on.
|
||||
|
||||
### 4. `FEDERATION.md` at repo root
|
||||
|
||||
The fediverse-convention filename, documenting: NodeInfo self-identification, actor/WebFinger shape, object types (today's Note-based activities; `trails:Route` once route-federation lands), addressing/audience rules, inbox expectations (signatures, dedup semantics), delivery retry policy (now true), and moderation behavior (what being blocked means). Written from the wanderer-doc model: real JSON examples over prose.
|
||||
|
||||
### 5. Observability
|
||||
|
||||
pg-boss job outcomes already flow to metrics where instrumented; add `federation_delivery_total{outcome}`, `federation_queue_depth`, `federation_inbox_dropped_total{reason=duplicate|blocked}` and a dashboard row. The queue-depth gauge is the restart-loss regression detector.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Two queueing layers (our fan-out jobs + Fedify's queue) remain] → Accepted for now; collapsing them is a refactor with no user-visible payoff. Documented in code comments.
|
||||
- [Blocklist drops silently (202)] → Deliberate: no error oracle for blocked instances; the drop counter keeps it observable to us.
|
||||
- [TTL dedup window vs very-late redelivery] → 30 days ≫ signature freshness window; residual risk negligible.
|
||||
- [pg-boss throughput for delivery spikes] → Fan-out sizes are follower counts on a young network; pg-boss handles far larger job volumes today (sync imports).
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Land queue implementation + swap (behavior-compatible; in-flight in-process queue drains on the old process before cutover deploy).
|
||||
2. Land dedup + blocklist tables and checks (additive).
|
||||
3. Publish `FEDERATION.md` + dashboard.
|
||||
Rollback: revert queue swap (falls back to in-process — the status quo), tables are inert.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- None blocking. Whether `route-federation` should require blocklist-awareness in mirror sync is noted in that change's update rather than here.
|
||||
|
|
@ -0,0 +1,28 @@
|
|||
## Why
|
||||
|
||||
Reviewing wanderer's federation (2026-07-06, `docs/inspirations.md`) showed exactly which operational gaps make an ActivityPub implementation fragile in practice: fire-and-forget delivery, no replay defense, no moderation tooling. trails has one of the same gaps today — the `social-federation` spec promises deliveries "retrying with exponential backoff… after a documented retry budget", but the journal runs Fedify with `InProcessMessageQueue` (`federation.server.ts:156`), so every queued delivery and retry is **lost on container restart** (and restarts are routine: deploys, the OOM/restart-loop history). Before `route-federation` builds mirrors and cross-instance editing on this foundation, delivery must be durable, inbound processing replay-safe, and instances blockable.
|
||||
|
||||
## What Changes
|
||||
|
||||
- **Durable delivery queue**: replace `InProcessMessageQueue` with a PostgreSQL-backed Fedify `MessageQueue` (built on the existing pg-boss infrastructure), so queued activities and their retry state survive restarts — closing the gap with what the spec already promises.
|
||||
- **Inbound replay/dedup defense**: processed remote activity IDs are recorded (TTL-bounded) and duplicates are dropped idempotently; complements the existing `remoteOriginIri` unique constraint which only covers ingested Creates.
|
||||
- **Instance blocklist**: a `federation_blocked_instances` table (domain, reason) enforced at all three boundaries — inbox processing, delivery enqueue, and outbox polling. Managed via documented SQL/ops procedure for v1 (admin UI out of scope).
|
||||
- **Published protocol documentation**: a root `FEDERATION.md` (the fediverse convention) documenting objects, activities, addressing, retry semantics, NodeInfo identification, and moderation behavior — wanderer's `federation.md` with real JSON examples is the model.
|
||||
- **Delivery observability**: metrics for delivery success/failure/retry counts and queue depth, on the existing Grafana journal dashboard.
|
||||
- Not in scope: allowlist-mode federation, admin moderation UI, per-user (as opposed to per-instance) blocks, and object-model changes (that's `route-federation`).
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `federation-operations`: The operational layer of federation — durable delivery, replay defense, instance blocking, protocol documentation, and observability.
|
||||
|
||||
### Modified Capabilities
|
||||
- `social-federation`: the "Push delivery on local activity create" requirement is strengthened to require queue persistence across process restarts.
|
||||
|
||||
## Impact
|
||||
|
||||
- `apps/journal/app/lib/federation.server.ts` — queue swap; new `federation-queue.server.ts` (Fedify `MessageQueue` over pg-boss) beside the existing `PostgresKvStore`.
|
||||
- `packages/db` — `federation_blocked_instances` + processed-activity dedup table (TTL sweep via existing jobs worker).
|
||||
- Inbox/outbound/poll paths — blocklist + dedup checks.
|
||||
- `FEDERATION.md` (root) + docs link; journal Grafana dashboard panels.
|
||||
- `route-federation` (in flight) builds on this; noted there as a dependency.
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
## ADDED Requirements
|
||||
|
||||
### Requirement: Durable federation queue
|
||||
Fedify's message queue SHALL be backed by PostgreSQL so that queued activities, inbox processing tasks, and retry state survive process restarts and deploys.
|
||||
|
||||
#### Scenario: Delivery survives a restart
|
||||
- **WHEN** deliveries are queued and the journal container restarts before they complete
|
||||
- **THEN** the deliveries execute after restart without loss
|
||||
|
||||
#### Scenario: Retry state persists
|
||||
- **WHEN** a delivery has failed twice with backoff pending and the process restarts
|
||||
- **THEN** the retry continues on schedule rather than starting over or disappearing
|
||||
|
||||
### Requirement: Inbound replay defense
|
||||
The Journal SHALL record processed inbound activity IRIs (retained at least 30 days) and SHALL drop duplicate activities idempotently before any side effects.
|
||||
|
||||
#### Scenario: Replayed activity is a no-op
|
||||
- **WHEN** the same signed Like activity is delivered twice
|
||||
- **THEN** the second delivery produces no state change and is counted as a dropped duplicate
|
||||
|
||||
### Requirement: Instance blocklist
|
||||
The Journal SHALL support blocking federation instances by domain, enforced at inbox processing (activities dropped without an error oracle), delivery enqueue (recipients filtered), and outbox polling/actor fetch (refused). Blocking SHALL be manageable via a documented operator procedure.
|
||||
|
||||
#### Scenario: Blocked instance is inert in both directions
|
||||
- **WHEN** a domain is added to the blocklist
|
||||
- **THEN** its activities are silently dropped, no deliveries are sent to it, and its actors/outboxes are not fetched
|
||||
|
||||
### Requirement: Published federation protocol documentation
|
||||
The repository SHALL contain a `FEDERATION.md` documenting actor discovery, object and activity types with JSON examples, addressing rules, signature and deduplication expectations, delivery retry policy, and moderation semantics, kept current as federation capabilities change.
|
||||
|
||||
#### Scenario: Third-party implementer can interoperate
|
||||
- **WHEN** another project implements against `FEDERATION.md`
|
||||
- **THEN** the documented flows (follow, create fan-out, dedup) behave as described
|
||||
|
||||
### Requirement: Federation delivery observability
|
||||
The Journal SHALL expose metrics for delivery outcomes, queue depth, and inbox drops (duplicate/blocked), surfaced on the monitoring dashboard.
|
||||
|
||||
#### Scenario: Operator sees delivery health
|
||||
- **WHEN** deliveries to an unreachable instance keep failing
|
||||
- **THEN** failure counts and queue depth are visible on the dashboard
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Push delivery on local activity create
|
||||
The Journal SHALL deliver a `Create(Note)` activity to every accepted remote follower's inbox when a local user with `profile_visibility = 'public'` creates a new `public` activity. Delivery queueing and retry state SHALL be persistent: queued deliveries and scheduled retries survive process restarts.
|
||||
|
||||
#### Scenario: New public activity fans out
|
||||
- **WHEN** a local user with N accepted remote followers creates a new public activity
|
||||
- **THEN** N delivery jobs are enqueued (one per follower's inbox), each retrying with exponential backoff on 5xx, giving up after a documented retry budget
|
||||
|
||||
#### Scenario: Rate-limited per remote host
|
||||
- **WHEN** multiple deliveries target the same remote host
|
||||
- **THEN** they are rate-limited so we never exceed 1 request per second per remote host (configurable; chosen for safety, not throughput)
|
||||
|
||||
#### Scenario: Fan-out survives a deploy
|
||||
- **WHEN** a deploy restarts the journal while fan-out deliveries are queued or awaiting retry
|
||||
- **THEN** the deliveries complete after the restart without loss
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
## 1. Durable queue
|
||||
|
||||
- [x] 1.1 Implement `federation-queue.server.ts`: Fedify `MessageQueue` over pg-boss (enqueue with delay, listen loop), mirroring the `PostgresKvStore` adapter pattern
|
||||
- [x] 1.2 Swap `InProcessMessageQueue` for it in `federation.server.ts`; document the two-layer queueing (fan-out jobs + Fedify queue) in a code comment
|
||||
- [x] 1.3 Tests: enqueue/delay/consume roundtrip; simulated restart (new listener picks up previously enqueued messages)
|
||||
|
||||
## 2. Replay defense
|
||||
|
||||
- [x] 2.1 Add `federation_processed_activities` table (activity IRI PK, received_at) + migration; insert-or-drop check in inbox handlers before side effects
|
||||
- [x] 2.2 30-day TTL sweep in the jobs worker
|
||||
- [x] 2.3 Tests: duplicate Like/Delete/Update dropped as no-ops; Create double-delivery still covered by remoteOriginIri constraint
|
||||
|
||||
## 3. Blocklist
|
||||
|
||||
- [x] 3.1 Add `federation_blocked_instances` table + migration; exact-host check helper
|
||||
- [x] 3.2 Enforce at inbox (silent 202 drop + counter), delivery enqueue (filter recipients), and outbox poll/actor fetch (refuse)
|
||||
- [x] 3.3 Document the operator procedure (SQL insert/delete) in the ops docs; tests for all three boundaries
|
||||
|
||||
## 4. Protocol doc & observability
|
||||
|
||||
- [x] 4.1 Write `FEDERATION.md` (repo root): NodeInfo, actors/WebFinger, object + activity types with JSON examples, addressing, signatures + dedup expectations, retry policy, moderation semantics; link from README and docs
|
||||
- [x] 4.2 Add `federation_delivery_total{outcome}`, `federation_queue_depth`, `federation_inbox_dropped_total{reason}` metrics + journal dashboard row
|
||||
|
||||
## 5. Verification
|
||||
|
||||
- [x] 5.1 Staging check (verified on staging.trails.cool 2026-07-13): queued a real delivery, restarted the journal container mid-flight — the job survived the restart in Postgres (the old in-process queue would have lost it) and completed after, "Successfully sent activity … to social.ullrich.is/users/ullrich/inbox" (`federation_delivery_total{outcome="delivered"} 1`, queue drained to 0). Blocklist **outbound** verified: a `poll-remote-actor` for a blocked domain logged `result: {skipped: "blocked instance"}` (no fetch). Operator block/unblock procedure works against the live DB. The blocklist **inbound** drop is covered by the group-2/3 real-Postgres integration tests + the two-instance `e2e/federation/` harness (firing it live needs a peer-initiated signed request).
|
||||
- [x] 5.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` — typecheck/lint/test green locally; `test:e2e` enforced by the required "E2E Tests" CI check on every PR
|
||||
Loading…
Add table
Add a link
Reference in a new issue