diff --git a/FEDERATION.md b/FEDERATION.md
new file mode 100644
index 0000000..aa4b88f
--- /dev/null
+++ b/FEDERATION.md
@@ -0,0 +1,198 @@
+# Federation protocol
+
+trails.cool's Journal federates over [ActivityPub](https://www.w3.org/TR/activitypub/),
+implemented with [Fedify](https://fedify.dev). This document describes the
+wire protocol precisely enough for another implementation to interoperate
+deliberately — actor discovery, the object and activity types we emit and
+accept, addressing, signatures, deduplication, delivery retry, and
+moderation. Examples use `trails.example` for our instance and
+`remote.example` for a peer.
+
+Federation is per-instance opt-in (`FEDERATION_ENABLED`). When it is off,
+every federation surface returns 404 — a disabled instance is
+indistinguishable from one without the feature. Only users with
+`profile_visibility = 'public'` federate; a private user's actor,
+WebFinger, inbox, and outbox all 404, so their existence never leaks.
+
+## Actor discovery
+
+### WebFinger
+
+`GET /.well-known/webfinger?resource=acct:alice@trails.example` resolves a
+handle to an actor IRI:
+
+```json
+{
+ "subject": "acct:alice@trails.example",
+ "links": [
+ { "rel": "self", "type": "application/activity+json", "href": "https://trails.example/users/alice" }
+ ]
+}
+```
+
+### Actor
+
+`GET https://trails.example/users/alice` with `Accept: application/activity+json`
+returns a `Person`. The actor IRI and the human profile `url` are the same
+by design (browsers get HTML at that URL via content negotiation):
+
+```json
+{
+ "@context": ["https://www.w3.org/ns/activitystreams", "https://w3id.org/security/v1"],
+ "id": "https://trails.example/users/alice",
+ "type": "Person",
+ "preferredUsername": "alice",
+ "name": "Alice",
+ "summary": "trail runner",
+ "url": "https://trails.example/users/alice",
+ "inbox": "https://trails.example/users/alice/inbox",
+ "outbox": "https://trails.example/users/alice/outbox",
+ "publicKey": {
+ "id": "https://trails.example/users/alice#main-key",
+ "owner": "https://trails.example/users/alice",
+ "publicKeyPem": "-----BEGIN PUBLIC KEY-----\n…\n-----END PUBLIC KEY-----\n"
+ },
+ "assertionMethod": [ { "type": "Multikey", "…": "…" } ],
+ "attachment": [
+ { "type": "PropertyValue", "name": "🥾 trails.cool", "value": "trails.example/users/alice" },
+ { "type": "PropertyValue", "name": "Instance", "value": "trails.example" }
+ ]
+}
+```
+
+`publicKey` is the RSA key Mastodon reads for HTTP-Signature verification;
+`assertionMethod` carries the same keys as Multikeys for newer stacks.
+
+### NodeInfo (software discovery)
+
+`GET /.well-known/nodeinfo` links to `GET /nodeinfo/2.1`:
+
+```json
+{
+ "version": "2.1",
+ "software": { "name": "trails-cool", "version": "1.2.3", "homepage": "https://trails.cool/" },
+ "protocols": ["activitypub"],
+ "usage": { "users": {}, "localPosts": 0, "localComments": 0 }
+}
+```
+
+`software.name` is the machine-readable "this is a trails instance" marker
+used by the trails-to-trails outbound check. Usage counts are deliberately
+zeroed — publishing per-instance counts is a privacy decision we have not
+made.
+
+## Objects and activities
+
+Activities correspond to a user's journal entries. The object model is
+deliberately Mastodon-compatible: a `Create(Note)` whose HTML `content`
+summarizes the activity and whose `url` links to the journal detail page.
+(A first-class `trails:Route` object type is planned with route-federation;
+today everything is a Note.)
+
+### Note
+
+```json
+{
+ "id": "https://trails.example/activities/01H…",
+ "type": "Note",
+ "attributedTo": "https://trails.example/users/alice",
+ "content": "Morning trail run — 12.4 km, 480 m up
",
+ "url": "https://trails.example/activities/01H…",
+ "published": "2026-07-13T07:12:00Z",
+ "to": ["https://www.w3.org/ns/activitystreams#Public"]
+}
+```
+
+The Note IRI (`/activities/`) is dereferenceable and serves
+`application/activity+json` — Mastodon's search-fetch and strict re-fetch
+of pushed objects both rely on this.
+
+### Create / Delete
+
+A publish is a `Create` wrapping the Note; the activity id is the object IRI
+with a `#create` fragment. A retraction is a `Delete` wrapping a `Tombstone`
+at the same object IRI:
+
+```json
+{ "id": "https://trails.example/activities/01H…#create", "type": "Create",
+ "actor": "https://trails.example/users/alice",
+ "object": { "…": "the Note above" },
+ "published": "2026-07-13T07:12:00Z",
+ "to": ["https://www.w3.org/ns/activitystreams#Public"] }
+```
+
+Note that a `Delete` poisons the object URI on the remote forever (remotes
+tombstone it); re-publishing the same URI after a Delete is silently
+refused by strict remotes.
+
+### Follow graph
+
+The inbox is **narrow** — only follow-graph activities are processed;
+anything else is acknowledged (`202`) and dropped.
+
+| Inbound | Effect |
+|---|---|
+| `Follow` (remote → local public actor) | auto-accepted; we push back an `Accept(Follow)` |
+| `Undo(Follow)` | removes the follow |
+| `Accept(Follow)` | settles our outgoing Follow; triggers the first outbox poll |
+| `Reject(Follow)` | drops our pending outgoing Follow |
+
+## Addressing
+
+Public activities are addressed to `https://www.w3.org/ns/activitystreams#Public`
+and **push-delivered** to each accepted remote follower's inbox (fan-out,
+one delivery per follower). We do not implement shared-inbox delivery.
+Remotes do not backfill history — only pushed or individually-fetched
+objects appear on a peer.
+
+## Signatures
+
+All inbound activities must carry a valid HTTP Signature; Fedify verifies it
+against the sending actor's `publicKey` (fetched and cached). Unsigned or
+badly-signed requests are rejected. Outbound deliveries are signed with the
+sending user's key. An actor changing keys requires the remote to re-fetch
+the actor document.
+
+## Deduplication
+
+Delivery is at-least-once, so receivers must be idempotent. trails dedups
+inbound activities two ways:
+
+- **`Create(Note)`** — idempotent via a unique constraint on the activity's
+ origin IRI (`remote_origin_iri`); a redelivered Create is a no-op.
+- **Follow-graph activities** (`Follow` / `Undo` / `Accept` / `Reject`) — the
+ activity IRI is recorded in `federation_processed_activities` on first
+ receipt (insert-or-drop before any side effect); a redelivery is dropped
+ and counted (`federation_inbox_dropped_total{reason="duplicate"}`).
+ Records are retained ≥ 30 days, which comfortably exceeds the
+ HTTP-Signature date-freshness window, then swept.
+
+## Delivery retry
+
+Delivery queueing and retry state are **durable** — they survive process
+restarts and deploys (backed by PostgreSQL via pg-boss; Fedify owns the
+retry policy). On a `5xx` or timeout, a delivery retries with exponential
+backoff, giving up after a bounded budget (~8 attempts spanning roughly a
+day) before a permanent failure is logged. Deliveries are paced to at most
+1 request/second per remote host. Metrics:
+`federation_delivery_total{outcome}` and `federation_queue_depth`.
+
+## Moderation
+
+An operator can block a federation instance by domain (exact-host match).
+A blocked instance is **inert in both directions**:
+
+- its inbound activities are silently dropped (`202`, no error oracle) and
+ counted (`federation_inbox_dropped_total{reason="blocked"}`);
+- it receives no deliveries (blocked recipients are filtered from fan-out);
+- we never fetch its actors or outboxes.
+
+Blocking is effective immediately (checked per request / per job, no cache).
+The operator procedure (a SQL insert/delete against
+`journal.federation_blocked_instances`) is documented in the
+[deployment runbook](docs/deployment.md#blocking-an-instance).
+
+---
+
+*Kept current as federation capabilities change. Specs:
+`openspec/specs/social-federation` and `openspec/specs/federation-operations`.*
diff --git a/README.md b/README.md
index 36f7493..1bc4215 100644
--- a/README.md
+++ b/README.md
@@ -96,6 +96,14 @@ docker compose up -d
See [docs/architecture.md](docs/architecture.md) for details on self-hosting
configuration.
+## Federation
+
+The Journal federates over ActivityPub. The wire protocol — actor
+discovery, object/activity types with JSON examples, addressing,
+signatures, deduplication, delivery retry, and moderation — is documented
+in [FEDERATION.md](FEDERATION.md), which is precise enough for another
+implementation to interoperate against.
+
## Philosophy
- **Privacy by design** — The Planner collects zero user data
diff --git a/apps/journal/app/jobs/deliver-activity.ts b/apps/journal/app/jobs/deliver-activity.ts
index b75631c..30eb001 100644
--- a/apps/journal/app/jobs/deliver-activity.ts
+++ b/apps/journal/app/jobs/deliver-activity.ts
@@ -15,6 +15,7 @@ import {
type DeliveryPayload,
} from "../lib/federation-delivery.server.ts";
import { logger } from "../lib/logger.server.ts";
+import { federationDeliveryTotal } from "../lib/metrics.server.ts";
/**
* Outbound pacing (spec 5.5): never exceed 1 request/second per remote
@@ -46,8 +47,10 @@ export const deliverActivityJob = defineJournalJob({
for (const job of jobs) {
const p = job.data;
try {
- await deliverOne(p);
+ const outcome = await deliverOne(p);
+ federationDeliveryTotal.inc({ outcome });
} catch (err) {
+ federationDeliveryTotal.inc({ outcome: "failed" });
logger.warn(
{ err, action: p.action, objectIri: p.objectIri, recipient: p.recipientActorIri },
"deliver-activity attempt failed (pg-boss will retry until budget exhausted)",
@@ -58,7 +61,7 @@ export const deliverActivityJob = defineJournalJob({
},
});
-async function deliverOne(p: DeliveryPayload): Promise {
+async function deliverOne(p: DeliveryPayload): Promise<"delivered" | "skipped"> {
const federation = getFederation();
const ctx = federation.createContext(new URL(getOrigin()), undefined);
@@ -75,7 +78,7 @@ async function deliverOne(p: DeliveryPayload): Promise {
.limit(1);
if (!row) {
logger.info({ objectIri: p.objectIri }, "deliver-activity: activity gone or non-public; skipping");
- return;
+ return "skipped";
}
// Spec 9.3: flipping the profile to private stops federation — also
// for deliveries already enqueued when the flip happened.
@@ -86,7 +89,7 @@ async function deliverOne(p: DeliveryPayload): Promise {
.limit(1);
if (!owner || owner.profileVisibility !== "public") {
logger.info({ objectIri: p.objectIri }, "deliver-activity: owner no longer public; skipping");
- return;
+ return "skipped";
}
activity = activityToCreate(row as FederatableActivity, p.ownerUsername);
} else {
@@ -127,4 +130,5 @@ async function deliverOne(p: DeliveryPayload): Promise {
{ action: p.action, objectIri: p.objectIri, recipient: p.recipientActorIri },
"deliver-activity: delivered",
);
+ return "delivered";
}
diff --git a/apps/journal/app/lib/federation.server.ts b/apps/journal/app/lib/federation.server.ts
index 9581ea5..b93b8ac 100644
--- a/apps/journal/app/lib/federation.server.ts
+++ b/apps/journal/app/lib/federation.server.ts
@@ -38,6 +38,7 @@ import { PostgresKvStore } from "./federation-kv.server.ts";
import { PgBossMessageQueue } from "./federation-queue.server.ts";
import { markInboundActivityProcessed } from "./federation-replay.server.ts";
import { isBlockedIri } from "./federation-blocklist.server.ts";
+import { federationInboxDroppedTotal } from "./metrics.server.ts";
import { ensureUserKeypair, loadUserKeypair } from "./federation-keys.server.ts";
import { activityToCreate, activityToNote } from "./federation-objects.server.ts";
import {
@@ -273,8 +274,8 @@ function buildFederation(): Federation {
// when the local target is public; otherwise drop (the actor
// already 404s for private users).
if (follow.id == null || follow.actorId == null || follow.objectId == null) return;
- if (await isBlockedIri(follow.actorId.href)) return; // blocked instance: silent 202 drop
- if (!(await markInboundActivityProcessed(follow.id.href)).fresh) return; // replay: drop
+ if (await isBlockedIri(follow.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
+ if (!(await markInboundActivityProcessed(follow.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
const parsed = ctx.parseUri(follow.objectId);
if (parsed?.type !== "actor") return;
const { outcome } = await recordRemoteFollow(follow.actorId.href, parsed.identifier);
@@ -295,8 +296,8 @@ function buildFederation(): Federation {
// Spec 4.3: Undo(Follow) removes the follow row. Other Undos are
// acknowledged and dropped.
if (undo.actorId == null) return;
- if (await isBlockedIri(undo.actorId.href)) return; // blocked instance: silent 202 drop
- if (undo.id != null && !(await markInboundActivityProcessed(undo.id.href)).fresh) return; // replay: drop
+ if (await isBlockedIri(undo.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
+ if (undo.id != null && !(await markInboundActivityProcessed(undo.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
const undoObjectId = undo.objectId; // capture before dereference (see Accept)
const object = await undo.getObject(ctx);
if (object instanceof Follow && object.objectId != null) {
@@ -325,8 +326,8 @@ function buildFederation(): Federation {
// Spec 4.4: a remote accepted our outgoing Follow — settle the
// Pending row and trigger the first outbox poll for that actor.
if (accept.actorId == null) return;
- if (await isBlockedIri(accept.actorId.href)) return; // blocked instance: silent 202 drop
- if (accept.id != null && !(await markInboundActivityProcessed(accept.id.href)).fresh) return; // replay: drop
+ if (await isBlockedIri(accept.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
+ if (accept.id != null && !(await markInboundActivityProcessed(accept.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
// Capture the raw object reference BEFORE dereferencing:
// getObject() memoizes the fetched document, after which objectId
// reports the fetched object's id (fragment stripped) instead of
@@ -366,8 +367,8 @@ function buildFederation(): Federation {
.on(Reject, async (ctx, reject) => {
// Spec 4.5: remote refused our Follow — drop the Pending row.
if (reject.actorId == null) return;
- if (await isBlockedIri(reject.actorId.href)) return; // blocked instance: silent 202 drop
- if (reject.id != null && !(await markInboundActivityProcessed(reject.id.href)).fresh) return; // replay: drop
+ if (await isBlockedIri(reject.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
+ if (reject.id != null && !(await markInboundActivityProcessed(reject.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
const objectId = reject.objectId; // capture before dereference (see Accept)
const object = await reject.getObject(ctx);
let localUser: Awaited> = null;
diff --git a/apps/journal/app/lib/metrics.server.ts b/apps/journal/app/lib/metrics.server.ts
index 468693a..9ea2181 100644
--- a/apps/journal/app/lib/metrics.server.ts
+++ b/apps/journal/app/lib/metrics.server.ts
@@ -48,6 +48,42 @@ export const demoBotSyntheticActivitiesTotal = getOrCreate(
}),
);
+// --- Federation metrics (spec: federation-operations "Federation delivery
+// observability") ---------------------------------------------------------
+
+/** Outbound delivery attempts by outcome (delivered | skipped | failed). */
+export const federationDeliveryTotal = getOrCreate(
+ "federation_delivery_total",
+ () =>
+ new client.Counter({
+ name: "federation_delivery_total",
+ help: "Outbound federation delivery attempts by outcome",
+ labelNames: ["outcome"] as const,
+ }),
+);
+
+/** Inbound activities dropped, by reason (duplicate | blocked). */
+export const federationInboxDroppedTotal = getOrCreate(
+ "federation_inbox_dropped_total",
+ () =>
+ new client.Counter({
+ name: "federation_inbox_dropped_total",
+ help: "Inbound federation activities dropped before side effects, by reason",
+ labelNames: ["reason"] as const,
+ }),
+);
+
+/** Messages waiting in the durable Fedify queue. Set at scrape time by the
+ * metrics route (the restart-loss regression detector). */
+export const federationQueueDepth = getOrCreate(
+ "federation_queue_depth",
+ () =>
+ new client.Gauge({
+ name: "federation_queue_depth",
+ help: "Messages waiting in the durable Fedify (pg-boss) message queue",
+ }),
+);
+
export const registry = client.register;
// --- Route label normalization -------------------------------------------
diff --git a/apps/journal/app/routes/api.metrics.ts b/apps/journal/app/routes/api.metrics.ts
index 852a3d2..f504738 100644
--- a/apps/journal/app/routes/api.metrics.ts
+++ b/apps/journal/app/routes/api.metrics.ts
@@ -1,6 +1,19 @@
-import { registry } from "~/lib/metrics.server";
+import { registry, federationQueueDepth } from "~/lib/metrics.server";
export async function loader() {
+ // Sample the durable Fedify queue depth at scrape time (the
+ // restart-loss regression detector). Best-effort: if federation is off
+ // or the boss isn't up yet, leave the last value rather than fail the
+ // scrape.
+ if (process.env.FEDERATION_ENABLED === "true") {
+ try {
+ const { PgBossMessageQueue } = await import("~/lib/federation-queue.server");
+ const depth = await new PgBossMessageQueue().getDepth();
+ federationQueueDepth.set(depth.queued);
+ } catch {
+ // boss not initialized / transient DB error — keep the prior gauge value
+ }
+ }
const metrics = await registry.metrics();
return new Response(metrics, {
headers: { "Content-Type": registry.contentType },
diff --git a/docs/architecture.md b/docs/architecture.md
index ef47b4a..fc52392 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -179,6 +179,11 @@ Tech stack:
## ActivityPub Integration
+The concrete wire protocol — actor discovery, object/activity types with
+JSON examples, addressing, signatures, deduplication, delivery retry, and
+moderation — is documented in [FEDERATION.md](../FEDERATION.md) at the repo
+root. This section is the higher-level design intent.
+
### Federated Activities
- `Create` Route - Publishing a new route
diff --git a/infrastructure/grafana/dashboards/journal.json b/infrastructure/grafana/dashboards/journal.json
index 6e53e93..39bb7d0 100644
--- a/infrastructure/grafana/dashboards/journal.json
+++ b/infrastructure/grafana/dashboards/journal.json
@@ -1,312 +1,391 @@
{
- "title": "Journal",
- "uid": "trails-journal",
- "annotations": {
- "list": [
- {
- "name": "Deploys",
- "enable": true,
- "datasource": {
- "type": "grafana",
- "uid": "-- Grafana --"
- },
- "iconColor": "rgba(0, 211, 255, 1)",
- "target": {
- "limit": 100,
- "matchAny": false,
- "tags": [
- "deploy"
- ],
- "type": "tags"
- }
- }
- ]
- },
- "timezone": "browser",
- "refresh": "30s",
- "panels": [
- {
- "title": "Caddy 502 Rate (Journal)",
- "description": "502 errors returned by Caddy for the journal upstream. These indicate Caddy could not reach the journal container.",
- "type": "timeseries",
- "gridPos": {
- "h": 8,
- "w": 12,
- "x": 0,
- "y": 0
- },
- "targets": [
- {
- "expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code=\"502\"}[5m])) or vector(0)",
- "legendFormat": "502/s"
- }
- ],
- "fieldConfig": {
- "defaults": {
- "custom": {
- "fillOpacity": 20,
- "lineWidth": 2
- },
- "color": {
- "mode": "fixed",
- "fixedColor": "red"
- }
- }
- }
+ "title": "Journal",
+ "uid": "trails-journal",
+ "annotations": {
+ "list": [
+ {
+ "name": "Deploys",
+ "enable": true,
+ "datasource": {
+ "type": "grafana",
+ "uid": "-- Grafana --"
},
- {
- "title": "Caddy Response Codes (Journal)",
- "description": "All HTTP response codes from Caddy for the journal host, including Caddy-generated errors like 502.",
- "type": "timeseries",
- "gridPos": {
- "h": 8,
- "w": 12,
- "x": 12,
- "y": 0
- },
- "targets": [
- {
- "expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code!~\"(101|2..|3..)\"}[5m])) by (code)",
- "legendFormat": "{{code}}"
- }
- ]
- },
- {
- "title": "Journal Request Rate by Status",
- "description": "Request rate from the journal app's own metrics, grouped by HTTP status code.",
- "type": "timeseries",
- "gridPos": {
- "h": 8,
- "w": 12,
- "x": 0,
- "y": 8
- },
- "targets": [
- {
- "expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (status)",
- "legendFormat": "{{status}}"
- }
- ]
- },
- {
- "title": "Journal Latency p50 / p95 / p99",
- "type": "timeseries",
- "gridPos": {
- "h": 8,
- "w": 12,
- "x": 12,
- "y": 8
- },
- "targets": [
- {
- "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
- "legendFormat": "p50"
- },
- {
- "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
- "legendFormat": "p95"
- },
- {
- "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
- "legendFormat": "p99"
- }
- ],
- "fieldConfig": {
- "defaults": {
- "unit": "s"
- }
- }
- },
- {
- "title": "Request Rate by Route",
- "type": "timeseries",
- "gridPos": {
- "h": 8,
- "w": 24,
- "x": 0,
- "y": 16
- },
- "targets": [
- {
- "expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (route)",
- "legendFormat": "{{route}}"
- }
- ]
- },
- {
- "title": "Container Restarts",
- "description": "Journal container restart count. Spikes here correlate with 502 errors — Caddy returns 502 while the container is restarting.",
- "type": "timeseries",
- "gridPos": {
- "h": 8,
- "w": 8,
- "x": 0,
- "y": 24
- },
- "targets": [
- {
- "expr": "changes(container_start_time_seconds{name=\"trails-cool-journal-1\"}[5m])",
- "legendFormat": "journal"
- }
- ],
- "fieldConfig": {
- "defaults": {
- "custom": {
- "fillOpacity": 30,
- "drawStyle": "bars"
- },
- "color": {
- "mode": "fixed",
- "fixedColor": "orange"
- }
- }
- }
- },
- {
- "title": "Container Memory",
- "type": "timeseries",
- "gridPos": {
- "h": 8,
- "w": 8,
- "x": 8,
- "y": 24
- },
- "targets": [
- {
- "expr": "container_memory_usage_bytes{name=\"trails-cool-journal-1\"}",
- "legendFormat": "used"
- },
- {
- "expr": "container_memory_working_set_bytes{name=\"trails-cool-journal-1\"}",
- "legendFormat": "working set"
- }
- ],
- "fieldConfig": {
- "defaults": {
- "unit": "bytes"
- }
- }
- },
- {
- "title": "Container CPU",
- "type": "timeseries",
- "gridPos": {
- "h": 8,
- "w": 8,
- "x": 16,
- "y": 24
- },
- "targets": [
- {
- "expr": "rate(container_cpu_usage_seconds_total{name=\"trails-cool-journal-1\"}[5m]) * 100",
- "legendFormat": "CPU %"
- }
- ],
- "fieldConfig": {
- "defaults": {
- "unit": "percent"
- }
- }
- },
- {
- "title": "Node.js Event Loop Lag",
- "type": "timeseries",
- "gridPos": {
- "h": 8,
- "w": 12,
- "x": 0,
- "y": 32
- },
- "targets": [
- {
- "expr": "nodejs_eventloop_lag_seconds{job=\"journal\"}",
- "legendFormat": "lag"
- }
- ],
- "fieldConfig": {
- "defaults": {
- "unit": "s"
- }
- }
- },
- {
- "title": "Node.js Heap Used",
- "type": "timeseries",
- "gridPos": {
- "h": 8,
- "w": 12,
- "x": 12,
- "y": 32
- },
- "targets": [
- {
- "expr": "nodejs_heap_size_used_bytes{job=\"journal\"}",
- "legendFormat": "used"
- },
- {
- "expr": "nodejs_heap_size_total_bytes{job=\"journal\"}",
- "legendFormat": "total"
- }
- ],
- "fieldConfig": {
- "defaults": {
- "unit": "bytes"
- }
- }
- },
- {
- "title": "Journal Logs (Errors & Warnings)",
- "description": "Recent error and warning logs from the journal container. Requires Promtail docker_sd_configs with Pino JSON parsing.",
- "type": "logs",
- "gridPos": {
- "h": 10,
- "w": 24,
- "x": 0,
- "y": 40
- },
- "targets": [
- {
- "datasource": {
- "type": "loki",
- "uid": "loki"
- },
- "expr": "{service=\"journal\"} |~ \"(error|warn|ERR|WARN|level.*(40|50|60))\"",
- "legendFormat": ""
- }
- ],
- "options": {
- "showTime": true,
- "sortOrder": "Descending",
- "enableLogDetails": true,
- "dedupStrategy": "none"
- }
- },
- {
- "title": "Caddy Logs (5xx Responses)",
- "description": "Caddy access log entries with 5xx status codes. Shows the raw request that triggered the error.",
- "type": "logs",
- "gridPos": {
- "h": 10,
- "w": 24,
- "x": 0,
- "y": 50
- },
- "targets": [
- {
- "datasource": {
- "type": "loki",
- "uid": "loki"
- },
- "expr": "{service=\"caddy\"} |~ \"\\\"status\\\":\\s*5\\d\\d\"",
- "legendFormat": ""
- }
- ],
- "options": {
- "showTime": true,
- "sortOrder": "Descending",
- "enableLogDetails": true,
- "dedupStrategy": "none"
- }
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "target": {
+ "limit": 100,
+ "matchAny": false,
+ "tags": [
+ "deploy"
+ ],
+ "type": "tags"
}
+ }
]
+ },
+ "timezone": "browser",
+ "refresh": "30s",
+ "panels": [
+ {
+ "title": "Caddy 502 Rate (Journal)",
+ "description": "502 errors returned by Caddy for the journal upstream. These indicate Caddy could not reach the journal container.",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 0
+ },
+ "targets": [
+ {
+ "expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code=\"502\"}[5m])) or vector(0)",
+ "legendFormat": "502/s"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "fillOpacity": 20,
+ "lineWidth": 2
+ },
+ "color": {
+ "mode": "fixed",
+ "fixedColor": "red"
+ }
+ }
+ }
+ },
+ {
+ "title": "Caddy Response Codes (Journal)",
+ "description": "All HTTP response codes from Caddy for the journal host, including Caddy-generated errors like 502.",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 0
+ },
+ "targets": [
+ {
+ "expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code!~\"(101|2..|3..)\"}[5m])) by (code)",
+ "legendFormat": "{{code}}"
+ }
+ ]
+ },
+ {
+ "title": "Journal Request Rate by Status",
+ "description": "Request rate from the journal app's own metrics, grouped by HTTP status code.",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 8
+ },
+ "targets": [
+ {
+ "expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (status)",
+ "legendFormat": "{{status}}"
+ }
+ ]
+ },
+ {
+ "title": "Journal Latency p50 / p95 / p99",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 8
+ },
+ "targets": [
+ {
+ "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
+ "legendFormat": "p50"
+ },
+ {
+ "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
+ "legendFormat": "p95"
+ },
+ {
+ "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
+ "legendFormat": "p99"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "unit": "s"
+ }
+ }
+ },
+ {
+ "title": "Request Rate by Route",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 16
+ },
+ "targets": [
+ {
+ "expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (route)",
+ "legendFormat": "{{route}}"
+ }
+ ]
+ },
+ {
+ "title": "Container Restarts",
+ "description": "Journal container restart count. Spikes here correlate with 502 errors \u2014 Caddy returns 502 while the container is restarting.",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 0,
+ "y": 24
+ },
+ "targets": [
+ {
+ "expr": "changes(container_start_time_seconds{name=\"trails-cool-journal-1\"}[5m])",
+ "legendFormat": "journal"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "fillOpacity": 30,
+ "drawStyle": "bars"
+ },
+ "color": {
+ "mode": "fixed",
+ "fixedColor": "orange"
+ }
+ }
+ }
+ },
+ {
+ "title": "Container Memory",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 8,
+ "y": 24
+ },
+ "targets": [
+ {
+ "expr": "container_memory_usage_bytes{name=\"trails-cool-journal-1\"}",
+ "legendFormat": "used"
+ },
+ {
+ "expr": "container_memory_working_set_bytes{name=\"trails-cool-journal-1\"}",
+ "legendFormat": "working set"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "unit": "bytes"
+ }
+ }
+ },
+ {
+ "title": "Container CPU",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 8,
+ "x": 16,
+ "y": 24
+ },
+ "targets": [
+ {
+ "expr": "rate(container_cpu_usage_seconds_total{name=\"trails-cool-journal-1\"}[5m]) * 100",
+ "legendFormat": "CPU %"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "unit": "percent"
+ }
+ }
+ },
+ {
+ "title": "Node.js Event Loop Lag",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 32
+ },
+ "targets": [
+ {
+ "expr": "nodejs_eventloop_lag_seconds{job=\"journal\"}",
+ "legendFormat": "lag"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "unit": "s"
+ }
+ }
+ },
+ {
+ "title": "Node.js Heap Used",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 32
+ },
+ "targets": [
+ {
+ "expr": "nodejs_heap_size_used_bytes{job=\"journal\"}",
+ "legendFormat": "used"
+ },
+ {
+ "expr": "nodejs_heap_size_total_bytes{job=\"journal\"}",
+ "legendFormat": "total"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "unit": "bytes"
+ }
+ }
+ },
+ {
+ "title": "Federation Delivery Rate by Outcome",
+ "description": "Outbound federation delivery attempts/sec by outcome (delivered/skipped/failed). Sustained 'failed' means an unreachable or misconfigured remote; retries continue until the budget is exhausted.",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 0,
+ "y": 40
+ },
+ "targets": [
+ {
+ "expr": "sum by (outcome) (rate(federation_delivery_total[5m]))",
+ "legendFormat": "{{outcome}}"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "fillOpacity": 20,
+ "lineWidth": 2
+ }
+ }
+ }
+ },
+ {
+ "title": "Federation Queue Depth",
+ "description": "Messages waiting in the durable Fedify (pg-boss) queue. A depth that climbs and never drains is the restart-loss / stuck-consumer regression detector \u2014 before the pg-boss backing (PR #570) a restart silently zeroed this by dropping the work.",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 12,
+ "x": 12,
+ "y": 40
+ },
+ "targets": [
+ {
+ "expr": "federation_queue_depth",
+ "legendFormat": "queued"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "fillOpacity": 10,
+ "lineWidth": 2
+ },
+ "color": {
+ "mode": "fixed",
+ "fixedColor": "blue"
+ }
+ }
+ }
+ },
+ {
+ "title": "Federation Inbox Drops by Reason",
+ "description": "Inbound activities dropped before side effects, by reason: 'duplicate' (replay defense) and 'blocked' (instance blocklist). A spike in 'blocked' after a block confirms it is taking effect; sustained 'duplicate' can indicate a misbehaving remote redelivering.",
+ "type": "timeseries",
+ "gridPos": {
+ "h": 8,
+ "w": 24,
+ "x": 0,
+ "y": 48
+ },
+ "targets": [
+ {
+ "expr": "sum by (reason) (rate(federation_inbox_dropped_total[5m]))",
+ "legendFormat": "{{reason}}"
+ }
+ ],
+ "fieldConfig": {
+ "defaults": {
+ "custom": {
+ "fillOpacity": 20,
+ "lineWidth": 2
+ }
+ }
+ }
+ },
+ {
+ "title": "Journal Logs (Errors & Warnings)",
+ "description": "Recent error and warning logs from the journal container. Requires Promtail docker_sd_configs with Pino JSON parsing.",
+ "type": "logs",
+ "gridPos": {
+ "h": 10,
+ "w": 24,
+ "x": 0,
+ "y": 56
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "expr": "{service=\"journal\"} |~ \"(error|warn|ERR|WARN|level.*(40|50|60))\"",
+ "legendFormat": ""
+ }
+ ],
+ "options": {
+ "showTime": true,
+ "sortOrder": "Descending",
+ "enableLogDetails": true,
+ "dedupStrategy": "none"
+ }
+ },
+ {
+ "title": "Caddy Logs (5xx Responses)",
+ "description": "Caddy access log entries with 5xx status codes. Shows the raw request that triggered the error.",
+ "type": "logs",
+ "gridPos": {
+ "h": 10,
+ "w": 24,
+ "x": 0,
+ "y": 66
+ },
+ "targets": [
+ {
+ "datasource": {
+ "type": "loki",
+ "uid": "loki"
+ },
+ "expr": "{service=\"caddy\"} |~ \"\\\"status\\\":\\s*5\\d\\d\"",
+ "legendFormat": ""
+ }
+ ],
+ "options": {
+ "showTime": true,
+ "sortOrder": "Descending",
+ "enableLogDetails": true,
+ "dedupStrategy": "none"
+ }
+ }
+ ]
}
diff --git a/openspec/changes/federation-hardening/tasks.md b/openspec/changes/federation-hardening/tasks.md
index 292fb4d..89a0670 100644
--- a/openspec/changes/federation-hardening/tasks.md
+++ b/openspec/changes/federation-hardening/tasks.md
@@ -18,8 +18,8 @@
## 4. Protocol doc & observability
-- [ ] 4.1 Write `FEDERATION.md` (repo root): NodeInfo, actors/WebFinger, object + activity types with JSON examples, addressing, signatures + dedup expectations, retry policy, moderation semantics; link from README and docs
-- [ ] 4.2 Add `federation_delivery_total{outcome}`, `federation_queue_depth`, `federation_inbox_dropped_total{reason}` metrics + journal dashboard row
+- [x] 4.1 Write `FEDERATION.md` (repo root): NodeInfo, actors/WebFinger, object + activity types with JSON examples, addressing, signatures + dedup expectations, retry policy, moderation semantics; link from README and docs
+- [x] 4.2 Add `federation_delivery_total{outcome}`, `federation_queue_depth`, `federation_inbox_dropped_total{reason}` metrics + journal dashboard row
## 5. Verification