Merge pull request #573 from trails-cool/feat/federation-hardening-docs-observability
feat(journal): federation protocol doc + delivery observability
This commit is contained in:
Ullrich Schäfer
GitHub
commit
b1f5f853c3
9 changed files with 668 additions and 324 deletions
198
FEDERATION.md
Normal file
198
FEDERATION.md
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
# Federation protocol
|
||||
|
||||
trails.cool's Journal federates over [ActivityPub](https://www.w3.org/TR/activitypub/),
|
||||
implemented with [Fedify](https://fedify.dev). This document describes the
|
||||
wire protocol precisely enough for another implementation to interoperate
|
||||
deliberately — actor discovery, the object and activity types we emit and
|
||||
accept, addressing, signatures, deduplication, delivery retry, and
|
||||
moderation. Examples use `trails.example` for our instance and
|
||||
`remote.example` for a peer.
|
||||
|
||||
Federation is per-instance opt-in (`FEDERATION_ENABLED`). When it is off,
|
||||
every federation surface returns 404 — a disabled instance is
|
||||
indistinguishable from one without the feature. Only users with
|
||||
`profile_visibility = 'public'` federate; a private user's actor,
|
||||
WebFinger, inbox, and outbox all 404, so their existence never leaks.
|
||||
|
||||
## Actor discovery
|
||||
|
||||
### WebFinger
|
||||
|
||||
`GET /.well-known/webfinger?resource=acct:alice@trails.example` resolves a
|
||||
handle to an actor IRI:
|
||||
|
||||
```json
|
||||
{
|
||||
"subject": "acct:alice@trails.example",
|
||||
"links": [
|
||||
{ "rel": "self", "type": "application/activity+json", "href": "https://trails.example/users/alice" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Actor
|
||||
|
||||
`GET https://trails.example/users/alice` with `Accept: application/activity+json`
|
||||
returns a `Person`. The actor IRI and the human profile `url` are the same
|
||||
by design (browsers get HTML at that URL via content negotiation):
|
||||
|
||||
```json
|
||||
{
|
||||
"@context": ["https://www.w3.org/ns/activitystreams", "https://w3id.org/security/v1"],
|
||||
"id": "https://trails.example/users/alice",
|
||||
"type": "Person",
|
||||
"preferredUsername": "alice",
|
||||
"name": "Alice",
|
||||
"summary": "trail runner",
|
||||
"url": "https://trails.example/users/alice",
|
||||
"inbox": "https://trails.example/users/alice/inbox",
|
||||
"outbox": "https://trails.example/users/alice/outbox",
|
||||
"publicKey": {
|
||||
"id": "https://trails.example/users/alice#main-key",
|
||||
"owner": "https://trails.example/users/alice",
|
||||
"publicKeyPem": "-----BEGIN PUBLIC KEY-----\n…\n-----END PUBLIC KEY-----\n"
|
||||
},
|
||||
"assertionMethod": [ { "type": "Multikey", "…": "…" } ],
|
||||
"attachment": [
|
||||
{ "type": "PropertyValue", "name": "🥾 trails.cool", "value": "<a href=\"https://trails.example/users/alice\" rel=\"me\">trails.example/users/alice</a>" },
|
||||
{ "type": "PropertyValue", "name": "Instance", "value": "<a href=\"https://trails.example\">trails.example</a>" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
`publicKey` is the RSA key Mastodon reads for HTTP-Signature verification;
|
||||
`assertionMethod` carries the same keys as Multikeys for newer stacks.
|
||||
|
||||
### NodeInfo (software discovery)
|
||||
|
||||
`GET /.well-known/nodeinfo` links to `GET /nodeinfo/2.1`:
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "2.1",
|
||||
"software": { "name": "trails-cool", "version": "1.2.3", "homepage": "https://trails.cool/" },
|
||||
"protocols": ["activitypub"],
|
||||
"usage": { "users": {}, "localPosts": 0, "localComments": 0 }
|
||||
}
|
||||
```
|
||||
|
||||
`software.name` is the machine-readable "this is a trails instance" marker
|
||||
used by the trails-to-trails outbound check. Usage counts are deliberately
|
||||
zeroed — publishing per-instance counts is a privacy decision we have not
|
||||
made.
|
||||
|
||||
## Objects and activities
|
||||
|
||||
Activities correspond to a user's journal entries. The object model is
|
||||
deliberately Mastodon-compatible: a `Create(Note)` whose HTML `content`
|
||||
summarizes the activity and whose `url` links to the journal detail page.
|
||||
(A first-class `trails:Route` object type is planned with route-federation;
|
||||
today everything is a Note.)
|
||||
|
||||
### Note
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "https://trails.example/activities/01H…",
|
||||
"type": "Note",
|
||||
"attributedTo": "https://trails.example/users/alice",
|
||||
"content": "<p>Morning trail run — 12.4 km, 480 m up</p>",
|
||||
"url": "https://trails.example/activities/01H…",
|
||||
"published": "2026-07-13T07:12:00Z",
|
||||
"to": ["https://www.w3.org/ns/activitystreams#Public"]
|
||||
}
|
||||
```
|
||||
|
||||
The Note IRI (`/activities/<id>`) is dereferenceable and serves
|
||||
`application/activity+json` — Mastodon's search-fetch and strict re-fetch
|
||||
of pushed objects both rely on this.
|
||||
|
||||
### Create / Delete
|
||||
|
||||
A publish is a `Create` wrapping the Note; the activity id is the object IRI
|
||||
with a `#create` fragment. A retraction is a `Delete` wrapping a `Tombstone`
|
||||
at the same object IRI:
|
||||
|
||||
```json
|
||||
{ "id": "https://trails.example/activities/01H…#create", "type": "Create",
|
||||
"actor": "https://trails.example/users/alice",
|
||||
"object": { "…": "the Note above" },
|
||||
"published": "2026-07-13T07:12:00Z",
|
||||
"to": ["https://www.w3.org/ns/activitystreams#Public"] }
|
||||
```
|
||||
|
||||
Note that a `Delete` poisons the object URI on the remote forever (remotes
|
||||
tombstone it); re-publishing the same URI after a Delete is silently
|
||||
refused by strict remotes.
|
||||
|
||||
### Follow graph
|
||||
|
||||
The inbox is **narrow** — only follow-graph activities are processed;
|
||||
anything else is acknowledged (`202`) and dropped.
|
||||
|
||||
| Inbound | Effect |
|
||||
|---|---|
|
||||
| `Follow` (remote → local public actor) | auto-accepted; we push back an `Accept(Follow)` |
|
||||
| `Undo(Follow)` | removes the follow |
|
||||
| `Accept(Follow)` | settles our outgoing Follow; triggers the first outbox poll |
|
||||
| `Reject(Follow)` | drops our pending outgoing Follow |
|
||||
|
||||
## Addressing
|
||||
|
||||
Public activities are addressed to `https://www.w3.org/ns/activitystreams#Public`
|
||||
and **push-delivered** to each accepted remote follower's inbox (fan-out,
|
||||
one delivery per follower). We do not implement shared-inbox delivery.
|
||||
Remotes do not backfill history — only pushed or individually-fetched
|
||||
objects appear on a peer.
|
||||
|
||||
## Signatures
|
||||
|
||||
All inbound activities must carry a valid HTTP Signature; Fedify verifies it
|
||||
against the sending actor's `publicKey` (fetched and cached). Unsigned or
|
||||
badly-signed requests are rejected. Outbound deliveries are signed with the
|
||||
sending user's key. An actor changing keys requires the remote to re-fetch
|
||||
the actor document.
|
||||
|
||||
## Deduplication
|
||||
|
||||
Delivery is at-least-once, so receivers must be idempotent. trails dedups
|
||||
inbound activities two ways:
|
||||
|
||||
- **`Create(Note)`** — idempotent via a unique constraint on the activity's
|
||||
origin IRI (`remote_origin_iri`); a redelivered Create is a no-op.
|
||||
- **Follow-graph activities** (`Follow` / `Undo` / `Accept` / `Reject`) — the
|
||||
activity IRI is recorded in `federation_processed_activities` on first
|
||||
receipt (insert-or-drop before any side effect); a redelivery is dropped
|
||||
and counted (`federation_inbox_dropped_total{reason="duplicate"}`).
|
||||
Records are retained ≥ 30 days, which comfortably exceeds the
|
||||
HTTP-Signature date-freshness window, then swept.
|
||||
|
||||
## Delivery retry
|
||||
|
||||
Delivery queueing and retry state are **durable** — they survive process
|
||||
restarts and deploys (backed by PostgreSQL via pg-boss; Fedify owns the
|
||||
retry policy). On a `5xx` or timeout, a delivery retries with exponential
|
||||
backoff, giving up after a bounded budget (~8 attempts spanning roughly a
|
||||
day) before a permanent failure is logged. Deliveries are paced to at most
|
||||
1 request/second per remote host. Metrics:
|
||||
`federation_delivery_total{outcome}` and `federation_queue_depth`.
|
||||
|
||||
## Moderation
|
||||
|
||||
An operator can block a federation instance by domain (exact-host match).
|
||||
A blocked instance is **inert in both directions**:
|
||||
|
||||
- its inbound activities are silently dropped (`202`, no error oracle) and
|
||||
counted (`federation_inbox_dropped_total{reason="blocked"}`);
|
||||
- it receives no deliveries (blocked recipients are filtered from fan-out);
|
||||
- we never fetch its actors or outboxes.
|
||||
|
||||
Blocking is effective immediately (checked per request / per job, no cache).
|
||||
The operator procedure (a SQL insert/delete against
|
||||
`journal.federation_blocked_instances`) is documented in the
|
||||
[deployment runbook](docs/deployment.md#blocking-an-instance).
|
||||
|
||||
---
|
||||
|
||||
*Kept current as federation capabilities change. Specs:
|
||||
`openspec/specs/social-federation` and `openspec/specs/federation-operations`.*
|
||||
|
|
@ -96,6 +96,14 @@ docker compose up -d
|
|||
See [docs/architecture.md](docs/architecture.md) for details on self-hosting
|
||||
configuration.
|
||||
|
||||
## Federation
|
||||
|
||||
The Journal federates over ActivityPub. The wire protocol — actor
|
||||
discovery, object/activity types with JSON examples, addressing,
|
||||
signatures, deduplication, delivery retry, and moderation — is documented
|
||||
in [FEDERATION.md](FEDERATION.md), which is precise enough for another
|
||||
implementation to interoperate against.
|
||||
|
||||
## Philosophy
|
||||
|
||||
- **Privacy by design** — The Planner collects zero user data
|
||||
|
|
|
|||
|
|
@ -15,6 +15,7 @@ import {
|
|||
type DeliveryPayload,
|
||||
} from "../lib/federation-delivery.server.ts";
|
||||
import { logger } from "../lib/logger.server.ts";
|
||||
import { federationDeliveryTotal } from "../lib/metrics.server.ts";
|
||||
|
||||
/**
|
||||
* Outbound pacing (spec 5.5): never exceed 1 request/second per remote
|
||||
|
|
@ -46,8 +47,10 @@ export const deliverActivityJob = defineJournalJob({
|
|||
for (const job of jobs) {
|
||||
const p = job.data;
|
||||
try {
|
||||
await deliverOne(p);
|
||||
const outcome = await deliverOne(p);
|
||||
federationDeliveryTotal.inc({ outcome });
|
||||
} catch (err) {
|
||||
federationDeliveryTotal.inc({ outcome: "failed" });
|
||||
logger.warn(
|
||||
{ err, action: p.action, objectIri: p.objectIri, recipient: p.recipientActorIri },
|
||||
"deliver-activity attempt failed (pg-boss will retry until budget exhausted)",
|
||||
|
|
@ -58,7 +61,7 @@ export const deliverActivityJob = defineJournalJob({
|
|||
},
|
||||
});
|
||||
|
||||
async function deliverOne(p: DeliveryPayload): Promise<void> {
|
||||
async function deliverOne(p: DeliveryPayload): Promise<"delivered" | "skipped"> {
|
||||
const federation = getFederation();
|
||||
const ctx = federation.createContext(new URL(getOrigin()), undefined);
|
||||
|
||||
|
|
@ -75,7 +78,7 @@ async function deliverOne(p: DeliveryPayload): Promise<void> {
|
|||
.limit(1);
|
||||
if (!row) {
|
||||
logger.info({ objectIri: p.objectIri }, "deliver-activity: activity gone or non-public; skipping");
|
||||
return;
|
||||
return "skipped";
|
||||
}
|
||||
// Spec 9.3: flipping the profile to private stops federation — also
|
||||
// for deliveries already enqueued when the flip happened.
|
||||
|
|
@ -86,7 +89,7 @@ async function deliverOne(p: DeliveryPayload): Promise<void> {
|
|||
.limit(1);
|
||||
if (!owner || owner.profileVisibility !== "public") {
|
||||
logger.info({ objectIri: p.objectIri }, "deliver-activity: owner no longer public; skipping");
|
||||
return;
|
||||
return "skipped";
|
||||
}
|
||||
activity = activityToCreate(row as FederatableActivity, p.ownerUsername);
|
||||
} else {
|
||||
|
|
@ -127,4 +130,5 @@ async function deliverOne(p: DeliveryPayload): Promise<void> {
|
|||
{ action: p.action, objectIri: p.objectIri, recipient: p.recipientActorIri },
|
||||
"deliver-activity: delivered",
|
||||
);
|
||||
return "delivered";
|
||||
}
|
||||
|
|
|
|||
|
|
@ -38,6 +38,7 @@ import { PostgresKvStore } from "./federation-kv.server.ts";
|
|||
import { PgBossMessageQueue } from "./federation-queue.server.ts";
|
||||
import { markInboundActivityProcessed } from "./federation-replay.server.ts";
|
||||
import { isBlockedIri } from "./federation-blocklist.server.ts";
|
||||
import { federationInboxDroppedTotal } from "./metrics.server.ts";
|
||||
import { ensureUserKeypair, loadUserKeypair } from "./federation-keys.server.ts";
|
||||
import { activityToCreate, activityToNote } from "./federation-objects.server.ts";
|
||||
import {
|
||||
|
|
@ -273,8 +274,8 @@ function buildFederation(): Federation<void> {
|
|||
// when the local target is public; otherwise drop (the actor
|
||||
// already 404s for private users).
|
||||
if (follow.id == null || follow.actorId == null || follow.objectId == null) return;
|
||||
if (await isBlockedIri(follow.actorId.href)) return; // blocked instance: silent 202 drop
|
||||
if (!(await markInboundActivityProcessed(follow.id.href)).fresh) return; // replay: drop
|
||||
if (await isBlockedIri(follow.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
|
||||
if (!(await markInboundActivityProcessed(follow.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
|
||||
const parsed = ctx.parseUri(follow.objectId);
|
||||
if (parsed?.type !== "actor") return;
|
||||
const { outcome } = await recordRemoteFollow(follow.actorId.href, parsed.identifier);
|
||||
|
|
@ -295,8 +296,8 @@ function buildFederation(): Federation<void> {
|
|||
// Spec 4.3: Undo(Follow) removes the follow row. Other Undos are
|
||||
// acknowledged and dropped.
|
||||
if (undo.actorId == null) return;
|
||||
if (await isBlockedIri(undo.actorId.href)) return; // blocked instance: silent 202 drop
|
||||
if (undo.id != null && !(await markInboundActivityProcessed(undo.id.href)).fresh) return; // replay: drop
|
||||
if (await isBlockedIri(undo.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
|
||||
if (undo.id != null && !(await markInboundActivityProcessed(undo.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
|
||||
const undoObjectId = undo.objectId; // capture before dereference (see Accept)
|
||||
const object = await undo.getObject(ctx);
|
||||
if (object instanceof Follow && object.objectId != null) {
|
||||
|
|
@ -325,8 +326,8 @@ function buildFederation(): Federation<void> {
|
|||
// Spec 4.4: a remote accepted our outgoing Follow — settle the
|
||||
// Pending row and trigger the first outbox poll for that actor.
|
||||
if (accept.actorId == null) return;
|
||||
if (await isBlockedIri(accept.actorId.href)) return; // blocked instance: silent 202 drop
|
||||
if (accept.id != null && !(await markInboundActivityProcessed(accept.id.href)).fresh) return; // replay: drop
|
||||
if (await isBlockedIri(accept.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
|
||||
if (accept.id != null && !(await markInboundActivityProcessed(accept.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
|
||||
// Capture the raw object reference BEFORE dereferencing:
|
||||
// getObject() memoizes the fetched document, after which objectId
|
||||
// reports the fetched object's id (fragment stripped) instead of
|
||||
|
|
@ -366,8 +367,8 @@ function buildFederation(): Federation<void> {
|
|||
.on(Reject, async (ctx, reject) => {
|
||||
// Spec 4.5: remote refused our Follow — drop the Pending row.
|
||||
if (reject.actorId == null) return;
|
||||
if (await isBlockedIri(reject.actorId.href)) return; // blocked instance: silent 202 drop
|
||||
if (reject.id != null && !(await markInboundActivityProcessed(reject.id.href)).fresh) return; // replay: drop
|
||||
if (await isBlockedIri(reject.actorId.href)) { federationInboxDroppedTotal.inc({ reason: "blocked" }); return; } // silent 202 drop
|
||||
if (reject.id != null && !(await markInboundActivityProcessed(reject.id.href)).fresh) { federationInboxDroppedTotal.inc({ reason: "duplicate" }); return; } // replay: drop
|
||||
const objectId = reject.objectId; // capture before dereference (see Accept)
|
||||
const object = await reject.getObject(ctx);
|
||||
let localUser: Awaited<ReturnType<typeof findLocalPublicUserByIri>> = null;
|
||||
|
|
|
|||
|
|
@ -48,6 +48,42 @@ export const demoBotSyntheticActivitiesTotal = getOrCreate(
|
|||
}),
|
||||
);
|
||||
|
||||
// --- Federation metrics (spec: federation-operations "Federation delivery
|
||||
// observability") ---------------------------------------------------------
|
||||
|
||||
/** Outbound delivery attempts by outcome (delivered | skipped | failed). */
|
||||
export const federationDeliveryTotal = getOrCreate(
|
||||
"federation_delivery_total",
|
||||
() =>
|
||||
new client.Counter({
|
||||
name: "federation_delivery_total",
|
||||
help: "Outbound federation delivery attempts by outcome",
|
||||
labelNames: ["outcome"] as const,
|
||||
}),
|
||||
);
|
||||
|
||||
/** Inbound activities dropped, by reason (duplicate | blocked). */
|
||||
export const federationInboxDroppedTotal = getOrCreate(
|
||||
"federation_inbox_dropped_total",
|
||||
() =>
|
||||
new client.Counter({
|
||||
name: "federation_inbox_dropped_total",
|
||||
help: "Inbound federation activities dropped before side effects, by reason",
|
||||
labelNames: ["reason"] as const,
|
||||
}),
|
||||
);
|
||||
|
||||
/** Messages waiting in the durable Fedify queue. Set at scrape time by the
|
||||
* metrics route (the restart-loss regression detector). */
|
||||
export const federationQueueDepth = getOrCreate(
|
||||
"federation_queue_depth",
|
||||
() =>
|
||||
new client.Gauge({
|
||||
name: "federation_queue_depth",
|
||||
help: "Messages waiting in the durable Fedify (pg-boss) message queue",
|
||||
}),
|
||||
);
|
||||
|
||||
export const registry = client.register;
|
||||
|
||||
// --- Route label normalization -------------------------------------------
|
||||
|
|
|
|||
|
|
@ -1,6 +1,19 @@
|
|||
import { registry } from "~/lib/metrics.server";
|
||||
import { registry, federationQueueDepth } from "~/lib/metrics.server";
|
||||
|
||||
export async function loader() {
|
||||
// Sample the durable Fedify queue depth at scrape time (the
|
||||
// restart-loss regression detector). Best-effort: if federation is off
|
||||
// or the boss isn't up yet, leave the last value rather than fail the
|
||||
// scrape.
|
||||
if (process.env.FEDERATION_ENABLED === "true") {
|
||||
try {
|
||||
const { PgBossMessageQueue } = await import("~/lib/federation-queue.server");
|
||||
const depth = await new PgBossMessageQueue().getDepth();
|
||||
federationQueueDepth.set(depth.queued);
|
||||
} catch {
|
||||
// boss not initialized / transient DB error — keep the prior gauge value
|
||||
}
|
||||
}
|
||||
const metrics = await registry.metrics();
|
||||
return new Response(metrics, {
|
||||
headers: { "Content-Type": registry.contentType },
|
||||
|
|
|
|||
|
|
@ -179,6 +179,11 @@ Tech stack:
|
|||
|
||||
## ActivityPub Integration
|
||||
|
||||
The concrete wire protocol — actor discovery, object/activity types with
|
||||
JSON examples, addressing, signatures, deduplication, delivery retry, and
|
||||
moderation — is documented in [FEDERATION.md](../FEDERATION.md) at the repo
|
||||
root. This section is the higher-level design intent.
|
||||
|
||||
### Federated Activities
|
||||
|
||||
- `Create` Route - Publishing a new route
|
||||
|
|
|
|||
|
|
@ -1,312 +1,391 @@
|
|||
{
|
||||
"title": "Journal",
|
||||
"uid": "trails-journal",
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"name": "Deploys",
|
||||
"enable": true,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"target": {
|
||||
"limit": 100,
|
||||
"matchAny": false,
|
||||
"tags": [
|
||||
"deploy"
|
||||
],
|
||||
"type": "tags"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"timezone": "browser",
|
||||
"refresh": "30s",
|
||||
"panels": [
|
||||
{
|
||||
"title": "Caddy 502 Rate (Journal)",
|
||||
"description": "502 errors returned by Caddy for the journal upstream. These indicate Caddy could not reach the journal container.",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code=\"502\"}[5m])) or vector(0)",
|
||||
"legendFormat": "502/s"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"fillOpacity": 20,
|
||||
"lineWidth": 2
|
||||
},
|
||||
"color": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "red"
|
||||
}
|
||||
}
|
||||
}
|
||||
"title": "Journal",
|
||||
"uid": "trails-journal",
|
||||
"annotations": {
|
||||
"list": [
|
||||
{
|
||||
"name": "Deploys",
|
||||
"enable": true,
|
||||
"datasource": {
|
||||
"type": "grafana",
|
||||
"uid": "-- Grafana --"
|
||||
},
|
||||
{
|
||||
"title": "Caddy Response Codes (Journal)",
|
||||
"description": "All HTTP response codes from Caddy for the journal host, including Caddy-generated errors like 502.",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code!~\"(101|2..|3..)\"}[5m])) by (code)",
|
||||
"legendFormat": "{{code}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Journal Request Rate by Status",
|
||||
"description": "Request rate from the journal app's own metrics, grouped by HTTP status code.",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (status)",
|
||||
"legendFormat": "{{status}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Journal Latency p50 / p95 / p99",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Request Rate by Route",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (route)",
|
||||
"legendFormat": "{{route}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Container Restarts",
|
||||
"description": "Journal container restart count. Spikes here correlate with 502 errors — Caddy returns 502 while the container is restarting.",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 24
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "changes(container_start_time_seconds{name=\"trails-cool-journal-1\"}[5m])",
|
||||
"legendFormat": "journal"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"fillOpacity": 30,
|
||||
"drawStyle": "bars"
|
||||
},
|
||||
"color": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "orange"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Container Memory",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 24
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "container_memory_usage_bytes{name=\"trails-cool-journal-1\"}",
|
||||
"legendFormat": "used"
|
||||
},
|
||||
{
|
||||
"expr": "container_memory_working_set_bytes{name=\"trails-cool-journal-1\"}",
|
||||
"legendFormat": "working set"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Container CPU",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 24
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(container_cpu_usage_seconds_total{name=\"trails-cool-journal-1\"}[5m]) * 100",
|
||||
"legendFormat": "CPU %"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Node.js Event Loop Lag",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 32
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "nodejs_eventloop_lag_seconds{job=\"journal\"}",
|
||||
"legendFormat": "lag"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Node.js Heap Used",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 32
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "nodejs_heap_size_used_bytes{job=\"journal\"}",
|
||||
"legendFormat": "used"
|
||||
},
|
||||
{
|
||||
"expr": "nodejs_heap_size_total_bytes{job=\"journal\"}",
|
||||
"legendFormat": "total"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Journal Logs (Errors & Warnings)",
|
||||
"description": "Recent error and warning logs from the journal container. Requires Promtail docker_sd_configs with Pino JSON parsing.",
|
||||
"type": "logs",
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 40
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{service=\"journal\"} |~ \"(error|warn|ERR|WARN|level.*(40|50|60))\"",
|
||||
"legendFormat": ""
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"enableLogDetails": true,
|
||||
"dedupStrategy": "none"
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Caddy Logs (5xx Responses)",
|
||||
"description": "Caddy access log entries with 5xx status codes. Shows the raw request that triggered the error.",
|
||||
"type": "logs",
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 50
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{service=\"caddy\"} |~ \"\\\"status\\\":\\s*5\\d\\d\"",
|
||||
"legendFormat": ""
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"enableLogDetails": true,
|
||||
"dedupStrategy": "none"
|
||||
}
|
||||
"iconColor": "rgba(0, 211, 255, 1)",
|
||||
"target": {
|
||||
"limit": 100,
|
||||
"matchAny": false,
|
||||
"tags": [
|
||||
"deploy"
|
||||
],
|
||||
"type": "tags"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"timezone": "browser",
|
||||
"refresh": "30s",
|
||||
"panels": [
|
||||
{
|
||||
"title": "Caddy 502 Rate (Journal)",
|
||||
"description": "502 errors returned by Caddy for the journal upstream. These indicate Caddy could not reach the journal container.",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code=\"502\"}[5m])) or vector(0)",
|
||||
"legendFormat": "502/s"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"fillOpacity": 20,
|
||||
"lineWidth": 2
|
||||
},
|
||||
"color": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "red"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Caddy Response Codes (Journal)",
|
||||
"description": "All HTTP response codes from Caddy for the journal host, including Caddy-generated errors like 502.",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(caddy_http_request_duration_seconds_count{server=~\".*trails.cool.*\", code!~\"(101|2..|3..)\"}[5m])) by (code)",
|
||||
"legendFormat": "{{code}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Journal Request Rate by Status",
|
||||
"description": "Request rate from the journal app's own metrics, grouped by HTTP status code.",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (status)",
|
||||
"legendFormat": "{{status}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Journal Latency p50 / p95 / p99",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{job=\"journal\"}[5m])) by (le))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Request Rate by Route",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum(rate(http_request_duration_seconds_count{job=\"journal\"}[5m])) by (route)",
|
||||
"legendFormat": "{{route}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"title": "Container Restarts",
|
||||
"description": "Journal container restart count. Spikes here correlate with 502 errors \u2014 Caddy returns 502 while the container is restarting.",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 0,
|
||||
"y": 24
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "changes(container_start_time_seconds{name=\"trails-cool-journal-1\"}[5m])",
|
||||
"legendFormat": "journal"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"fillOpacity": 30,
|
||||
"drawStyle": "bars"
|
||||
},
|
||||
"color": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "orange"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Container Memory",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 8,
|
||||
"y": 24
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "container_memory_usage_bytes{name=\"trails-cool-journal-1\"}",
|
||||
"legendFormat": "used"
|
||||
},
|
||||
{
|
||||
"expr": "container_memory_working_set_bytes{name=\"trails-cool-journal-1\"}",
|
||||
"legendFormat": "working set"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Container CPU",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 8,
|
||||
"x": 16,
|
||||
"y": 24
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "rate(container_cpu_usage_seconds_total{name=\"trails-cool-journal-1\"}[5m]) * 100",
|
||||
"legendFormat": "CPU %"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percent"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Node.js Event Loop Lag",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 32
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "nodejs_eventloop_lag_seconds{job=\"journal\"}",
|
||||
"legendFormat": "lag"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Node.js Heap Used",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 32
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "nodejs_heap_size_used_bytes{job=\"journal\"}",
|
||||
"legendFormat": "used"
|
||||
},
|
||||
{
|
||||
"expr": "nodejs_heap_size_total_bytes{job=\"journal\"}",
|
||||
"legendFormat": "total"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Federation Delivery Rate by Outcome",
|
||||
"description": "Outbound federation delivery attempts/sec by outcome (delivered/skipped/failed). Sustained 'failed' means an unreachable or misconfigured remote; retries continue until the budget is exhausted.",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 40
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (outcome) (rate(federation_delivery_total[5m]))",
|
||||
"legendFormat": "{{outcome}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"fillOpacity": 20,
|
||||
"lineWidth": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Federation Queue Depth",
|
||||
"description": "Messages waiting in the durable Fedify (pg-boss) queue. A depth that climbs and never drains is the restart-loss / stuck-consumer regression detector \u2014 before the pg-boss backing (PR #570) a restart silently zeroed this by dropping the work.",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 40
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "federation_queue_depth",
|
||||
"legendFormat": "queued"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"fillOpacity": 10,
|
||||
"lineWidth": 2
|
||||
},
|
||||
"color": {
|
||||
"mode": "fixed",
|
||||
"fixedColor": "blue"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Federation Inbox Drops by Reason",
|
||||
"description": "Inbound activities dropped before side effects, by reason: 'duplicate' (replay defense) and 'blocked' (instance blocklist). A spike in 'blocked' after a block confirms it is taking effect; sustained 'duplicate' can indicate a misbehaving remote redelivering.",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 48
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (reason) (rate(federation_inbox_dropped_total[5m]))",
|
||||
"legendFormat": "{{reason}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"custom": {
|
||||
"fillOpacity": 20,
|
||||
"lineWidth": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Journal Logs (Errors & Warnings)",
|
||||
"description": "Recent error and warning logs from the journal container. Requires Promtail docker_sd_configs with Pino JSON parsing.",
|
||||
"type": "logs",
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 56
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{service=\"journal\"} |~ \"(error|warn|ERR|WARN|level.*(40|50|60))\"",
|
||||
"legendFormat": ""
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"enableLogDetails": true,
|
||||
"dedupStrategy": "none"
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "Caddy Logs (5xx Responses)",
|
||||
"description": "Caddy access log entries with 5xx status codes. Shows the raw request that triggered the error.",
|
||||
"type": "logs",
|
||||
"gridPos": {
|
||||
"h": 10,
|
||||
"w": 24,
|
||||
"x": 0,
|
||||
"y": 66
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"expr": "{service=\"caddy\"} |~ \"\\\"status\\\":\\s*5\\d\\d\"",
|
||||
"legendFormat": ""
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"sortOrder": "Descending",
|
||||
"enableLogDetails": true,
|
||||
"dedupStrategy": "none"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -18,10 +18,10 @@
|
|||
|
||||
## 4. Protocol doc & observability
|
||||
|
||||
- [ ] 4.1 Write `FEDERATION.md` (repo root): NodeInfo, actors/WebFinger, object + activity types with JSON examples, addressing, signatures + dedup expectations, retry policy, moderation semantics; link from README and docs
|
||||
- [ ] 4.2 Add `federation_delivery_total{outcome}`, `federation_queue_depth`, `federation_inbox_dropped_total{reason}` metrics + journal dashboard row
|
||||
- [x] 4.1 Write `FEDERATION.md` (repo root): NodeInfo, actors/WebFinger, object + activity types with JSON examples, addressing, signatures + dedup expectations, retry policy, moderation semantics; link from README and docs
|
||||
- [x] 4.2 Add `federation_delivery_total{outcome}`, `federation_queue_depth`, `federation_inbox_dropped_total{reason}` metrics + journal dashboard row
|
||||
|
||||
## 5. Verification
|
||||
|
||||
- [ ] 5.1 Staging check: queue a fan-out, restart the journal container, confirm deliveries complete; block a test domain and verify both directions
|
||||
- [ ] 5.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e`
|
||||
- [ ] 5.1 Staging check: queue a fan-out, restart the journal container, confirm deliveries complete; block a test domain and verify both directions — **post-deploy step**, run once the change is on staging (see PR description for the exact commands)
|
||||
- [x] 5.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e` — typecheck/lint/test green locally; `test:e2e` enforced by the required "E2E Tests" CI check on every PR
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue