Merge pull request #573 from trails-cool/feat/federation-hardening-docs-observability

feat(journal): federation protocol doc + delivery observability

FEDERATION.md

@@ -0,0 +1,198 @@
+# Federation protocol
+
+trails.cool's Journal federates over [ActivityPub](https://www.w3.org/TR/activitypub/),
+implemented with [Fedify](https://fedify.dev). This document describes the
+wire protocol precisely enough for another implementation to interoperate
+deliberately — actor discovery, the object and activity types we emit and
+accept, addressing, signatures, deduplication, delivery retry, and
+moderation. Examples use `trails.example` for our instance and
+`remote.example` for a peer.
+
+Federation is per-instance opt-in (`FEDERATION_ENABLED`). When it is off,
+every federation surface returns 404 — a disabled instance is
+indistinguishable from one without the feature. Only users with
+`profile_visibility = 'public'` federate; a private user's actor,
+WebFinger, inbox, and outbox all 404, so their existence never leaks.
+
+## Actor discovery
+
+### WebFinger
+
+`GET /.well-known/webfinger?resource=acct:alice@trails.example` resolves a
+handle to an actor IRI:
+
+```json
+{
+  "subject": "acct:alice@trails.example",
+  "links": [
+    { "rel": "self", "type": "application/activity+json", "href": "https://trails.example/users/alice" }
+  ]
+}
+```
+
+### Actor
+
+`GET https://trails.example/users/alice` with `Accept: application/activity+json`
+returns a `Person`. The actor IRI and the human profile `url` are the same
+by design (browsers get HTML at that URL via content negotiation):
+
+```json
+{
+  "@context": ["https://www.w3.org/ns/activitystreams", "https://w3id.org/security/v1"],
+  "id": "https://trails.example/users/alice",
+  "type": "Person",
+  "preferredUsername": "alice",
+  "name": "Alice",
+  "summary": "trail runner",
+  "url": "https://trails.example/users/alice",
+  "inbox": "https://trails.example/users/alice/inbox",
+  "outbox": "https://trails.example/users/alice/outbox",
+  "publicKey": {
+    "id": "https://trails.example/users/alice#main-key",
+    "owner": "https://trails.example/users/alice",
+    "publicKeyPem": "-----BEGIN PUBLIC KEY-----\n…\n-----END PUBLIC KEY-----\n"
+  },
+  "assertionMethod": [ { "type": "Multikey", "…": "…" } ],
+  "attachment": [
+    { "type": "PropertyValue", "name": "🥾 trails.cool", "value": "<a href=\"https://trails.example/users/alice\" rel=\"me\">trails.example/users/alice</a>" },
+    { "type": "PropertyValue", "name": "Instance", "value": "<a href=\"https://trails.example\">trails.example</a>" }
+  ]
+}
+```
+
+`publicKey` is the RSA key Mastodon reads for HTTP-Signature verification;
+`assertionMethod` carries the same keys as Multikeys for newer stacks.
+
+### NodeInfo (software discovery)
+
+`GET /.well-known/nodeinfo` links to `GET /nodeinfo/2.1`:
+
+```json
+{
+  "version": "2.1",
+  "software": { "name": "trails-cool", "version": "1.2.3", "homepage": "https://trails.cool/" },
+  "protocols": ["activitypub"],
+  "usage": { "users": {}, "localPosts": 0, "localComments": 0 }
+}
+```
+
+`software.name` is the machine-readable "this is a trails instance" marker
+used by the trails-to-trails outbound check. Usage counts are deliberately
+zeroed — publishing per-instance counts is a privacy decision we have not
+made.
+
+## Objects and activities
+
+Activities correspond to a user's journal entries. The object model is
+deliberately Mastodon-compatible: a `Create(Note)` whose HTML `content`
+summarizes the activity and whose `url` links to the journal detail page.
+(A first-class `trails:Route` object type is planned with route-federation;
+today everything is a Note.)
+
+### Note
+
+```json
+{
+  "id": "https://trails.example/activities/01H…",
+  "type": "Note",
+  "attributedTo": "https://trails.example/users/alice",
+  "content": "<p>Morning trail run — 12.4 km, 480 m up</p>",
+  "url": "https://trails.example/activities/01H…",
+  "published": "2026-07-13T07:12:00Z",
+  "to": ["https://www.w3.org/ns/activitystreams#Public"]
+}
+```
+
+The Note IRI (`/activities/<id>`) is dereferenceable and serves
+`application/activity+json` — Mastodon's search-fetch and strict re-fetch
+of pushed objects both rely on this.
+
+### Create / Delete
+
+A publish is a `Create` wrapping the Note; the activity id is the object IRI
+with a `#create` fragment. A retraction is a `Delete` wrapping a `Tombstone`
+at the same object IRI:
+
+```json
+{ "id": "https://trails.example/activities/01H…#create", "type": "Create",
+  "actor": "https://trails.example/users/alice",
+  "object": { "…": "the Note above" },
+  "published": "2026-07-13T07:12:00Z",
+  "to": ["https://www.w3.org/ns/activitystreams#Public"] }
+```
+
+Note that a `Delete` poisons the object URI on the remote forever (remotes
+tombstone it); re-publishing the same URI after a Delete is silently
+refused by strict remotes.
+
+### Follow graph
+
+The inbox is **narrow** — only follow-graph activities are processed;
+anything else is acknowledged (`202`) and dropped.
+
+| Inbound | Effect |
+|---|---|
+| `Follow` (remote → local public actor) | auto-accepted; we push back an `Accept(Follow)` |
+| `Undo(Follow)` | removes the follow |
+| `Accept(Follow)` | settles our outgoing Follow; triggers the first outbox poll |
+| `Reject(Follow)` | drops our pending outgoing Follow |
+
+## Addressing
+
+Public activities are addressed to `https://www.w3.org/ns/activitystreams#Public`
+and **push-delivered** to each accepted remote follower's inbox (fan-out,
+one delivery per follower). We do not implement shared-inbox delivery.
+Remotes do not backfill history — only pushed or individually-fetched
+objects appear on a peer.
+
+## Signatures
+
+All inbound activities must carry a valid HTTP Signature; Fedify verifies it
+against the sending actor's `publicKey` (fetched and cached). Unsigned or
+badly-signed requests are rejected. Outbound deliveries are signed with the
+sending user's key. An actor changing keys requires the remote to re-fetch
+the actor document.
+
+## Deduplication
+
+Delivery is at-least-once, so receivers must be idempotent. trails dedups
+inbound activities two ways:
+
+- **`Create(Note)`** — idempotent via a unique constraint on the activity's
+  origin IRI (`remote_origin_iri`); a redelivered Create is a no-op.
+- **Follow-graph activities** (`Follow` / `Undo` / `Accept` / `Reject`) — the
+  activity IRI is recorded in `federation_processed_activities` on first
+  receipt (insert-or-drop before any side effect); a redelivery is dropped
+  and counted (`federation_inbox_dropped_total{reason="duplicate"}`).
+  Records are retained ≥ 30 days, which comfortably exceeds the
+  HTTP-Signature date-freshness window, then swept.
+
+## Delivery retry
+
+Delivery queueing and retry state are **durable** — they survive process
+restarts and deploys (backed by PostgreSQL via pg-boss; Fedify owns the
+retry policy). On a `5xx` or timeout, a delivery retries with exponential
+backoff, giving up after a bounded budget (~8 attempts spanning roughly a
+day) before a permanent failure is logged. Deliveries are paced to at most
+1 request/second per remote host. Metrics:
+`federation_delivery_total{outcome}` and `federation_queue_depth`.
+
+## Moderation
+
+An operator can block a federation instance by domain (exact-host match).
+A blocked instance is **inert in both directions**:
+
+- its inbound activities are silently dropped (`202`, no error oracle) and
+  counted (`federation_inbox_dropped_total{reason="blocked"}`);
+- it receives no deliveries (blocked recipients are filtered from fan-out);
+- we never fetch its actors or outboxes.
+
+Blocking is effective immediately (checked per request / per job, no cache).
+The operator procedure (a SQL insert/delete against
+`journal.federation_blocked_instances`) is documented in the
+[deployment runbook](docs/deployment.md#blocking-an-instance).
+
+---
+
+*Kept current as federation capabilities change. Specs:
+`openspec/specs/social-federation` and `openspec/specs/federation-operations`.*

README.md

@@ -96,6 +96,14 @@ docker compose up -d
 See [docs/architecture.md](docs/architecture.md) for details on self-hosting
 configuration.
 
+## Federation
+
+The Journal federates over ActivityPub. The wire protocol — actor
+discovery, object/activity types with JSON examples, addressing,
+signatures, deduplication, delivery retry, and moderation — is documented
+in [FEDERATION.md](FEDERATION.md), which is precise enough for another
+implementation to interoperate against.
+
 ## Philosophy
 
 - **Privacy by design** — The Planner collects zero user data

apps/journal/app/jobs/deliver-activity.ts

@@ -15,6 +15,7 @@ import {
   type DeliveryPayload,
 } from "../lib/federation-delivery.server.ts";
 import { logger } from "../lib/logger.server.ts";
+import { federationDeliveryTotal } from "../lib/metrics.server.ts";
 
 /**
  * Outbound pacing (spec 5.5): never exceed 1 request/second per remote
@@ -46,8 +47,10 @@ export const deliverActivityJob = defineJournalJob({
     for (const job of jobs) {
       const p = job.data;
       try {
-        await deliverOne(p);
+        const outcome = await deliverOne(p);
+        federationDeliveryTotal.inc({ outcome });
       } catch (err) {
+        federationDeliveryTotal.inc({ outcome: "failed" });
         logger.warn(
           { err, action: p.action, objectIri: p.objectIri, recipient: p.recipientActorIri },
           "deliver-activity attempt failed (pg-boss will retry until budget exhausted)",
@@ -58,7 +61,7 @@ export const deliverActivityJob = defineJournalJob({
   },
 });
 
-async function deliverOne(p: DeliveryPayload): Promise<void> {
+async function deliverOne(p: DeliveryPayload): Promise<"delivered" | "skipped"> {
   const federation = getFederation();
   const ctx = federation.createContext(new URL(getOrigin()), undefined);
 
@@ -75,7 +78,7 @@ async function deliverOne(p: DeliveryPayload): Promise<void> {
       .limit(1);
     if (!row) {
       logger.info({ objectIri: p.objectIri }, "deliver-activity: activity gone or non-public; skipping");
-      return;
+      return "skipped";
     }
     // Spec 9.3: flipping the profile to private stops federation — also
     // for deliveries already enqueued when the flip happened.
@@ -86,7 +89,7 @@ async function deliverOne(p: DeliveryPayload): Promise<void> {
       .limit(1);
     if (!owner || owner.profileVisibility !== "public") {
       logger.info({ objectIri: p.objectIri }, "deliver-activity: owner no longer public; skipping");
-      return;
+      return "skipped";
     }
     activity = activityToCreate(row as FederatableActivity, p.ownerUsername);
   } else {
@@ -127,4 +130,5 @@ async function deliverOne(p: DeliveryPayload): Promise<void> {
     { action: p.action, objectIri: p.objectIri, recipient: p.recipientActorIri },
     "deliver-activity: delivered",
   );
+  return "delivered";
 }

apps/journal/app/lib/federation.server.ts

@@ -38,6 +38,7 @@ import { PostgresKvStore } from "./federation-kv.server.ts";
 import { PgBossMessageQueue } from "./federation-queue.server.ts";
 import { markInboundActivityProcessed } from "./federation-replay.server.ts";
 import { isBlockedIri } from "./federation-blocklist.server.ts";
+import { federationInboxDroppedTotal } from "./metrics.server.ts";
 import { ensureUserKeypair, loadUserKeypair } from "./federation-keys.server.ts";
 import { activityToCreate, activityToNote } from "./federation-objects.server.ts";
 import {
@@ -273,8 +274,8 @@ function buildFederation(): Federation<void> {
       // when the local target is public; otherwise drop (the actor
       // already 404s for private users).
       if (follow.id == null || follow.actorId == null || follow.objectId == null) return;
-      if (await isBlockedIri(follow.actorId.href)) return; // blocked instance: silent 202 drop
+      if (await isBlockedIri(follow.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
-      if (!(await markInboundActivityProcessed(follow.id.href)).fresh) return; // replay: drop
+      if (!(await markInboundActivityProcessed(follow.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
       const parsed = ctx.parseUri(follow.objectId);
       if (parsed?.type !== "actor") return;
       const { outcome } = await recordRemoteFollow(follow.actorId.href, parsed.identifier);
@@ -295,8 +296,8 @@ function buildFederation(): Federation<void> {
       // Spec 4.3: Undo(Follow) removes the follow row. Other Undos are
       // acknowledged and dropped.
       if (undo.actorId == null) return;
-      if (await isBlockedIri(undo.actorId.href)) return; // blocked instance: silent 202 drop
+      if (await isBlockedIri(undo.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
-      if (undo.id != null && !(await markInboundActivityProcessed(undo.id.href)).fresh) return; // replay: drop
+      if (undo.id != null && !(await markInboundActivityProcessed(undo.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
       const undoObjectId = undo.objectId; // capture before dereference (see Accept)
       const object = await undo.getObject(ctx);
       if (object instanceof Follow && object.objectId != null) {
@@ -325,8 +326,8 @@ function buildFederation(): Federation<void> {
       // Spec 4.4: a remote accepted our outgoing Follow — settle the
       // Pending row and trigger the first outbox poll for that actor.
       if (accept.actorId == null) return;
-      if (await isBlockedIri(accept.actorId.href)) return; // blocked instance: silent 202 drop
+      if (await isBlockedIri(accept.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
-      if (accept.id != null && !(await markInboundActivityProcessed(accept.id.href)).fresh) return; // replay: drop
+      if (accept.id != null && !(await markInboundActivityProcessed(accept.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
       // Capture the raw object reference BEFORE dereferencing:
       // getObject() memoizes the fetched document, after which objectId
       // reports the fetched object's id (fragment stripped) instead of
@@ -366,8 +367,8 @@ function buildFederation(): Federation<void> {
     .on(Reject, async (ctx, reject) => {
       // Spec 4.5: remote refused our Follow — drop the Pending row.
       if (reject.actorId == null) return;
-      if (await isBlockedIri(reject.actorId.href)) return; // blocked instance: silent 202 drop
+      if (await isBlockedIri(reject.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
-      if (reject.id != null && !(await markInboundActivityProcessed(reject.id.href)).fresh) return; // replay: drop
+      if (reject.id != null && !(await markInboundActivityProcessed(reject.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
       const objectId = reject.objectId; // capture before dereference (see Accept)
       const object = await reject.getObject(ctx);
       let localUser: Awaited<ReturnType<typeof findLocalPublicUserByIri>> = null;

apps/journal/app/lib/metrics.server.ts

@@ -48,6 +48,42 @@ export const demoBotSyntheticActivitiesTotal = getOrCreate(
     }),
 );
 
+// --- Federation metrics (spec: federation-operations "Federation delivery
+// observability") ---------------------------------------------------------
+
+/** Outbound delivery attempts by outcome (delivered | skipped | failed). */
+export const federationDeliveryTotal = getOrCreate(
+  "federation_delivery_total",
+  () =>
+    new client.Counter({
+      name: "federation_delivery_total",
+      help: "Outbound federation delivery attempts by outcome",
+      labelNames: ["outcome"] as const,
+    }),
+);
+
+/** Inbound activities dropped, by reason (duplicate | blocked). */
+export const federationInboxDroppedTotal = getOrCreate(
+  "federation_inbox_dropped_total",
+  () =>
+    new client.Counter({
+      name: "federation_inbox_dropped_total",
+      help: "Inbound federation activities dropped before side effects, by reason",
+      labelNames: ["reason"] as const,
+    }),
+);
+
+/** Messages waiting in the durable Fedify queue. Set at scrape time by the
+ * metrics route (the restart-loss regression detector). */
+export const federationQueueDepth = getOrCreate(
+  "federation_queue_depth",
+  () =>
+    new client.Gauge({
+      name: "federation_queue_depth",
+      help: "Messages waiting in the durable Fedify (pg-boss) message queue",
+    }),
+);
+
 export const registry = client.register;
 
 // --- Route label normalization -------------------------------------------

apps/journal/app/routes/api.metrics.ts

@@ -1,6 +1,19 @@
-import { registry } from "~/lib/metrics.server";
+import { registry, federationQueueDepth } from "~/lib/metrics.server";
 
 export async function loader() {
+  // Sample the durable Fedify queue depth at scrape time (the
+  // restart-loss regression detector). Best-effort: if federation is off
+  // or the boss isn't up yet, leave the last value rather than fail the
+  // scrape.
+  if (process.env.FEDERATION_ENABLED === "true") {
+    try {
+      const { PgBossMessageQueue } = await import("~/lib/federation-queue.server");
+      const depth = await new PgBossMessageQueue().getDepth();
+      federationQueueDepth.set(depth.queued);
+    } catch {
+      // boss not initialized / transient DB error — keep the prior gauge value
+    }
+  }
   const metrics = await registry.metrics();
   return new Response(metrics, {
     headers: { "Content-Type": registry.contentType },

docs/architecture.md

@@ -179,6 +179,11 @@ Tech stack:
 
 ## ActivityPub Integration
 
+The concrete wire protocol — actor discovery, object/activity types with
+JSON examples, addressing, signatures, deduplication, delivery retry, and
+moderation — is documented in [FEDERATION.md](../FEDERATION.md) at the repo
+root. This section is the higher-level design intent.
+
 ### Federated Activities
 
 - `Create` Route - Publishing a new route

infrastructure/grafana/dashboards/journal.json

@@ -1,312 +1,391 @@
 {
-  "title": "Journal",
+ "title": "Journal",
-  "uid": "trails-journal",
+ "uid": "trails-journal",
-  "annotations": {
+ "annotations": {
-    "list": [
+  "list": [
-      {
+   {
-        "name": "Deploys",
+    "name": "Deploys",
-        "enable": true,
+    "enable": true,
-        "datasource": {
+    "datasource": {
-          "type": "grafana",
+     "type": "grafana",
-          "uid": "-- Grafana --"
+     "uid": "-- Grafana --"
-        },
-        "iconColor": "rgba(0, 211, 255, 1)",
-        "target": {
-          "limit": 100,
-          "matchAny": false,
-          "tags": [
-            "deploy"
-          ],
-          "type": "tags"
-        }
-      }
-    ]
-  },
-  "timezone": "browser",
-  "refresh": "30s",
-  "panels": [
-    {
-      "title": "Caddy 502 Rate (Journal)",
-      "description": "502 errors returned by Caddy for the journal upstream. These indicate Caddy could not reach the journal container.",
-      "type": "timeseries",
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code=\"502\"}[5m])) or vector(0)",
-          "legendFormat": "502/s"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "fillOpacity": 20,
-            "lineWidth": 2
-          },
-          "color": {
-            "mode": "fixed",
-            "fixedColor": "red"
-          }
-        }
-      }
     },
-    {
+    "iconColor": "rgba(0, 211, 255, 1)",
-      "title": "Caddy Response Codes (Journal)",
+    "target": {
-      "description": "All HTTP response codes from Caddy for the journal host, including Caddy-generated errors like 502.",
+     "limit": 100,
-      "type": "timeseries",
+     "matchAny": false,
-      "gridPos": {
+     "tags": [
-        "h": 8,
+      "deploy"
-        "w": 12,
+     ],
-        "x": 12,
+     "type": "tags"
-        "y": 0
-      },
-      "targets": [
-        {
-          "expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code!~\"(101|2..|3..)\"}[5m])) by (code)",
-          "legendFormat": "{{code}}"
-        }
-      ]
-    },
-    {
-      "title": "Journal Request Rate by Status",
-      "description": "Request rate from the journal app's own metrics, grouped by HTTP status code.",
-      "type": "timeseries",
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 8
-      },
-      "targets": [
-        {
-          "expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (status)",
-          "legendFormat": "{{status}}"
-        }
-      ]
-    },
-    {
-      "title": "Journal Latency p50 / p95 / p99",
-      "type": "timeseries",
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 8
-      },
-      "targets": [
-        {
-          "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
-          "legendFormat": "p50"
-        },
-        {
-          "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
-          "legendFormat": "p95"
-        },
-        {
-          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
-          "legendFormat": "p99"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s"
-        }
-      }
-    },
-    {
-      "title": "Request Rate by Route",
-      "type": "timeseries",
-      "gridPos": {
-        "h": 8,
-        "w": 24,
-        "x": 0,
-        "y": 16
-      },
-      "targets": [
-        {
-          "expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (route)",
-          "legendFormat": "{{route}}"
-        }
-      ]
-    },
-    {
-      "title": "Container Restarts",
-      "description": "Journal container restart count. Spikes here correlate with 502 errors — Caddy returns 502 while the container is restarting.",
-      "type": "timeseries",
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 0,
-        "y": 24
-      },
-      "targets": [
-        {
-          "expr": "changes(container_start_time_seconds{name=\"trails-cool-journal-1\"}[5m])",
-          "legendFormat": "journal"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "custom": {
-            "fillOpacity": 30,
-            "drawStyle": "bars"
-          },
-          "color": {
-            "mode": "fixed",
-            "fixedColor": "orange"
-          }
-        }
-      }
-    },
-    {
-      "title": "Container Memory",
-      "type": "timeseries",
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 8,
-        "y": 24
-      },
-      "targets": [
-        {
-          "expr": "container_memory_usage_bytes{name=\"trails-cool-journal-1\"}",
-          "legendFormat": "used"
-        },
-        {
-          "expr": "container_memory_working_set_bytes{name=\"trails-cool-journal-1\"}",
-          "legendFormat": "working set"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "bytes"
-        }
-      }
-    },
-    {
-      "title": "Container CPU",
-      "type": "timeseries",
-      "gridPos": {
-        "h": 8,
-        "w": 8,
-        "x": 16,
-        "y": 24
-      },
-      "targets": [
-        {
-          "expr": "rate(container_cpu_usage_seconds_total{name=\"trails-cool-journal-1\"}[5m]) * 100",
-          "legendFormat": "CPU %"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "percent"
-        }
-      }
-    },
-    {
-      "title": "Node.js Event Loop Lag",
-      "type": "timeseries",
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 0,
-        "y": 32
-      },
-      "targets": [
-        {
-          "expr": "nodejs_eventloop_lag_seconds{job=\"journal\"}",
-          "legendFormat": "lag"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "s"
-        }
-      }
-    },
-    {
-      "title": "Node.js Heap Used",
-      "type": "timeseries",
-      "gridPos": {
-        "h": 8,
-        "w": 12,
-        "x": 12,
-        "y": 32
-      },
-      "targets": [
-        {
-          "expr": "nodejs_heap_size_used_bytes{job=\"journal\"}",
-          "legendFormat": "used"
-        },
-        {
-          "expr": "nodejs_heap_size_total_bytes{job=\"journal\"}",
-          "legendFormat": "total"
-        }
-      ],
-      "fieldConfig": {
-        "defaults": {
-          "unit": "bytes"
-        }
-      }
-    },
-    {
-      "title": "Journal Logs (Errors & Warnings)",
-      "description": "Recent error and warning logs from the journal container. Requires Promtail docker_sd_configs with Pino JSON parsing.",
-      "type": "logs",
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 40
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "loki"
-          },
-          "expr": "{service=\"journal\"} |~ \"(error|warn|ERR|WARN|level.*(40|50|60))\"",
-          "legendFormat": ""
-        }
-      ],
-      "options": {
-        "showTime": true,
-        "sortOrder": "Descending",
-        "enableLogDetails": true,
-        "dedupStrategy": "none"
-      }
-    },
-    {
-      "title": "Caddy Logs (5xx Responses)",
-      "description": "Caddy access log entries with 5xx status codes. Shows the raw request that triggered the error.",
-      "type": "logs",
-      "gridPos": {
-        "h": 10,
-        "w": 24,
-        "x": 0,
-        "y": 50
-      },
-      "targets": [
-        {
-          "datasource": {
-            "type": "loki",
-            "uid": "loki"
-          },
-          "expr": "{service=\"caddy\"} |~ \"\\\"status\\\":\\s*5\\d\\d\"",
-          "legendFormat": ""
-        }
-      ],
-      "options": {
-        "showTime": true,
-        "sortOrder": "Descending",
-        "enableLogDetails": true,
-        "dedupStrategy": "none"
-      }
     }
+   }
   ]
+ },
+ "timezone": "browser",
+ "refresh": "30s",
+ "panels": [
+  {
+   "title": "Caddy 502 Rate (Journal)",
+   "description": "502 errors returned by Caddy for the journal upstream. These indicate Caddy could not reach the journal container.",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 12,
+    "x": 0,
+    "y": 0
+   },
+   "targets": [
+    {
+     "expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code=\"502\"}[5m])) or vector(0)",
+     "legendFormat": "502/s"
+    }
+   ],
+   "fieldConfig": {
+    "defaults": {
+     "custom": {
+      "fillOpacity": 20,
+      "lineWidth": 2
+     },
+     "color": {
+      "mode": "fixed",
+      "fixedColor": "red"
+     }
+    }
+   }
+  },
+  {
+   "title": "Caddy Response Codes (Journal)",
+   "description": "All HTTP response codes from Caddy for the journal host, including Caddy-generated errors like 502.",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 12,
+    "x": 12,
+    "y": 0
+   },
+   "targets": [
+    {
+     "expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code!~\"(101|2..|3..)\"}[5m])) by (code)",
+     "legendFormat": "{{code}}"
+    }
+   ]
+  },
+  {
+   "title": "Journal Request Rate by Status",
+   "description": "Request rate from the journal app's own metrics, grouped by HTTP status code.",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 12,
+    "x": 0,
+    "y": 8
+   },
+   "targets": [
+    {
+     "expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (status)",
+     "legendFormat": "{{status}}"
+    }
+   ]
+  },
+  {
+   "title": "Journal Latency p50 / p95 / p99",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 12,
+    "x": 12,
+    "y": 8
+   },
+   "targets": [
+    {
+     "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
+     "legendFormat": "p50"
+    },
+    {
+     "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
+     "legendFormat": "p95"
+    },
+    {
+     "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
+     "legendFormat": "p99"
+    }
+   ],
+   "fieldConfig": {
+    "defaults": {
+     "unit": "s"
+    }
+   }
+  },
+  {
+   "title": "Request Rate by Route",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 24,
+    "x": 0,
+    "y": 16
+   },
+   "targets": [
+    {
+     "expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (route)",
+     "legendFormat": "{{route}}"
+    }
+   ]
+  },
+  {
+   "title": "Container Restarts",
+   "description": "Journal container restart count. Spikes here correlate with 502 errors \u2014 Caddy returns 502 while the container is restarting.",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 8,
+    "x": 0,
+    "y": 24
+   },
+   "targets": [
+    {
+     "expr": "changes(container_start_time_seconds{name=\"trails-cool-journal-1\"}[5m])",
+     "legendFormat": "journal"
+    }
+   ],
+   "fieldConfig": {
+    "defaults": {
+     "custom": {
+      "fillOpacity": 30,
+      "drawStyle": "bars"
+     },
+     "color": {
+      "mode": "fixed",
+      "fixedColor": "orange"
+     }
+    }
+   }
+  },
+  {
+   "title": "Container Memory",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 8,
+    "x": 8,
+    "y": 24
+   },
+   "targets": [
+    {
+     "expr": "container_memory_usage_bytes{name=\"trails-cool-journal-1\"}",
+     "legendFormat": "used"
+    },
+    {
+     "expr": "container_memory_working_set_bytes{name=\"trails-cool-journal-1\"}",
+     "legendFormat": "working set"
+    }
+   ],
+   "fieldConfig": {
+    "defaults": {
+     "unit": "bytes"
+    }
+   }
+  },
+  {
+   "title": "Container CPU",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 8,
+    "x": 16,
+    "y": 24
+   },
+   "targets": [
+    {
+     "expr": "rate(container_cpu_usage_seconds_total{name=\"trails-cool-journal-1\"}[5m]) * 100",
+     "legendFormat": "CPU %"
+    }
+   ],
+   "fieldConfig": {
+    "defaults": {
+     "unit": "percent"
+    }
+   }
+  },
+  {
+   "title": "Node.js Event Loop Lag",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 12,
+    "x": 0,
+    "y": 32
+   },
+   "targets": [
+    {
+     "expr": "nodejs_eventloop_lag_seconds{job=\"journal\"}",
+     "legendFormat": "lag"
+    }
+   ],
+   "fieldConfig": {
+    "defaults": {
+     "unit": "s"
+    }
+   }
+  },
+  {
+   "title": "Node.js Heap Used",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 12,
+    "x": 12,
+    "y": 32
+   },
+   "targets": [
+    {
+     "expr": "nodejs_heap_size_used_bytes{job=\"journal\"}",
+     "legendFormat": "used"
+    },
+    {
+     "expr": "nodejs_heap_size_total_bytes{job=\"journal\"}",
+     "legendFormat": "total"
+    }
+   ],
+   "fieldConfig": {
+    "defaults": {
+     "unit": "bytes"
+    }
+   }
+  },
+  {
+   "title": "Federation Delivery Rate by Outcome",
+   "description": "Outbound federation delivery attempts/sec by outcome (delivered/skipped/failed). Sustained 'failed' means an unreachable or misconfigured remote; retries continue until the budget is exhausted.",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 12,
+    "x": 0,
+    "y": 40
+   },
+   "targets": [
+    {
+     "expr": "sum by (outcome) (rate(federation_delivery_total[5m]))",
+     "legendFormat": "{{outcome}}"
+    }
+   ],
+   "fieldConfig": {
+    "defaults": {
+     "custom": {
+      "fillOpacity": 20,
+      "lineWidth": 2
+     }
+    }
+   }
+  },
+  {
+   "title": "Federation Queue Depth",
+   "description": "Messages waiting in the durable Fedify (pg-boss) queue. A depth that climbs and never drains is the restart-loss / stuck-consumer regression detector \u2014 before the pg-boss backing (PR #570) a restart silently zeroed this by dropping the work.",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 12,
+    "x": 12,
+    "y": 40
+   },
+   "targets": [
+    {
+     "expr": "federation_queue_depth",
+     "legendFormat": "queued"
+    }
+   ],
+   "fieldConfig": {
+    "defaults": {
+     "custom": {
+      "fillOpacity": 10,
+      "lineWidth": 2
+     },
+     "color": {
+      "mode": "fixed",
+      "fixedColor": "blue"
+     }
+    }
+   }
+  },
+  {
+   "title": "Federation Inbox Drops by Reason",
+   "description": "Inbound activities dropped before side effects, by reason: 'duplicate' (replay defense) and 'blocked' (instance blocklist). A spike in 'blocked' after a block confirms it is taking effect; sustained 'duplicate' can indicate a misbehaving remote redelivering.",
+   "type": "timeseries",
+   "gridPos": {
+    "h": 8,
+    "w": 24,
+    "x": 0,
+    "y": 48
+   },
+   "targets": [
+    {
+     "expr": "sum by (reason) (rate(federation_inbox_dropped_total[5m]))",
+     "legendFormat": "{{reason}}"
+    }
+   ],
+   "fieldConfig": {
+    "defaults": {
+     "custom": {
+      "fillOpacity": 20,
+      "lineWidth": 2
+     }
+    }
+   }
+  },
+  {
+   "title": "Journal Logs (Errors & Warnings)",
+   "description": "Recent error and warning logs from the journal container. Requires Promtail docker_sd_configs with Pino JSON parsing.",
+   "type": "logs",
+   "gridPos": {
+    "h": 10,
+    "w": 24,
+    "x": 0,
+    "y": 56
+   },
+   "targets": [
+    {
+     "datasource": {
+      "type": "loki",
+      "uid": "loki"
+     },
+     "expr": "{service=\"journal\"} |~ \"(error|warn|ERR|WARN|level.*(40|50|60))\"",
+     "legendFormat": ""
+    }
+   ],
+   "options": {
+    "showTime": true,
+    "sortOrder": "Descending",
+    "enableLogDetails": true,
+    "dedupStrategy": "none"
+   }
+  },
+  {
+   "title": "Caddy Logs (5xx Responses)",
+   "description": "Caddy access log entries with 5xx status codes. Shows the raw request that triggered the error.",
+   "type": "logs",
+   "gridPos": {
+    "h": 10,
+    "w": 24,
+    "x": 0,
+    "y": 66
+   },
+   "targets": [
+    {
+     "datasource": {
+      "type": "loki",
+      "uid": "loki"
+     },
+     "expr": "{service=\"caddy\"} |~ \"\\\"status\\\":\\s*5\\d\\d\"",
+     "legendFormat": ""
+    }
+   ],
+   "options": {
+    "showTime": true,
+    "sortOrder": "Descending",
+    "enableLogDetails": true,
+    "dedupStrategy": "none"
+   }
+  }
+ ]
 }

openspec/changes/federation-hardening/tasks.md

@@ -18,10 +18,10 @@
 
 ## 4. Protocol doc & observability
 
-- [ ] 4.1 Write `FEDERATION.md` (repo root): NodeInfo, actors/WebFinger, object + activity types with JSON examples, addressing, signatures + dedup expectations, retry policy, moderation semantics; link from README and docs
+- [x] 4.1 Write `FEDERATION.md` (repo root): NodeInfo, actors/WebFinger, object + activity types with JSON examples, addressing, signatures + dedup expectations, retry policy, moderation semantics; link from README and docs
-- [ ] 4.2 Add `federation_delivery_total{outcome}`, `federation_queue_depth`, `federation_inbox_dropped_total{reason}` metrics + journal dashboard row
+- [x] 4.2 Add `federation_delivery_total{outcome}`, `federation_queue_depth`, `federation_inbox_dropped_total{reason}` metrics + journal dashboard row
 
 ## 5. Verification
 
-- [ ] 5.1 Staging check: queue a fan-out, restart the journal container, confirm deliveries complete; block a test domain and verify both directions
+- [ ] 5.1 Staging check: queue a fan-out, restart the journal container, confirm deliveries complete; block a test domain and verify both directions — **post-deploy step**, run once the change is on staging (see PR description for the exact commands)
-- [ ] 5.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e`
+- [x] 5.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` — typecheck/lint/test green locally; `test:e2e` enforced by the required "E2E Tests" CI check on every PR