feat(journal): outbox-poll ingestion of remote trails activities (§7)

Tasks 7.1–7.3, 7.5 (+7.4 pacing; Retry-After-duration backoff noted as
remaining). Resolves the activities.owner_id open question.

Schema (design decision from the open question):
- activities.owner_id nullable + check constraint enforcing exactly
  one of (owner_id, remote_actor_iri) — the follows pattern again.
- remote_published_at carries the origin's publish time for the §8
  feed sort; index on remote_actor_iri for the feed join.
- Compiler-audited fallout: notification fan-out, the Note object
  dispatcher, and the activity detail loader explicitly skip/404
  remote rows (their canonical page is the origin; feed links there).
  Every other surface joins users on owner_id and excludes remote rows
  structurally.

Ingestion:
- federation-ingest.server.ts: parseOutboxItem targets exactly our own
  outgoing Create(Note) shape (PropertyValue stats, first-paragraph
  name, audience from to/cc); unknown items skipped, never fatal.
- ingestRemoteActivities: replay-safe via unique remote_origin_iri,
  conflict-streak early exit (outboxes are newest-first).
- pollRemoteActor: signed fetch (Authorized Fetch via a local
  follower's key), outbox resolution via remote_actors cache with
  actor-doc fallback (which refreshes the cache), 1 req/5 s per-host
  pacing, last_polled_at stamping. Network + pacing injectable.

Jobs: poll-remote-actor (per-actor; the §4 Accept listener already
enqueues it as the first poll) + poll-remote-outboxes (5-min cron
sweep over accepted remote follows not polled within the hour).

Tests: parse-shape units; integration suite for ingestion provenance,
audience→visibility mirroring, replay safety, the DB author invariant,
poll flow (actor-doc → collection → page), signer requirement, and
due-polling selection.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-06-07 12:05:32 +02:00
parent 9ada6ab7f3
commit b96bef91a9
13 changed files with 628 additions and 21 deletions

View file

@ -147,13 +147,17 @@ Inbound signature verification uses the actor's public key from their actor obje
## Open Questions
- **`activities.owner_id` is NOT NULL but remote-ingested rows have no local
owner** (surfaced during task 2.3, 2026-06-06). Options: make `owner_id`
nullable with a check constraint (`owner_id IS NOT NULL OR remote_actor_iri
IS NOT NULL`), or key remote rows purely off `remote_actor_iri` in a way
that never touches owner-joined queries. Decide in task 7.2 (ingestion)
before any remote row is written; the columns landed in 2.3 don't prejudge
either option.
- ~~**`activities.owner_id` is NOT NULL but remote-ingested rows have no local
owner**~~ — **Resolved in task 7.2 (2026-06-07):** `owner_id` is nullable
with a check constraint enforcing exactly one of (`owner_id`,
`remote_actor_iri`) — the same pattern as `follows.follower_id` /
`follower_actor_iri`. Compiler-audited fallout: notification fan-out,
the Note object dispatcher, and the activity detail loader all
explicitly 404/skip remote rows (their canonical page is the origin
instance; the feed links outward). Every other surface joins `users`
on `owner_id` and excludes remote rows structurally. A
`remote_published_at` column carries the origin's publish time for
the §8 feed sort.
- Custom AS extension for activity-type vs. plain `Create(Note)`? Mastodon will only render Notes; an extension type means non-trails clients see nothing useful. Lean toward `Create(Note)` with structured metadata in `attachment` so Mastodon shows the text + GPX link, and trails clients consuming the outbox can read the structured fields.
- Per-instance inbox vs per-user inbox? Mastodon supports a "shared inbox" optimization. Worth doing for delivery efficiency once we have any volume; v1 can use per-user inboxes only.

View file

@ -57,11 +57,12 @@
## 7. Outbox-poll ingestion
- [ ] 7.1 pg-boss recurring job `poll-remote-outboxes` (cron every 5 min): iterate distinct accepted remote actor IRIs in `follows` where `remote_actors.last_polled_at < now() - interval '1 hour'`; for each, signed GET on the outbox using one of the local accepted-followers' keys
- [ ] 7.2 Insert returned activities into `journal.activities` with `remote_origin_iri`, `remote_actor_iri`, `audience`. `ON CONFLICT (remote_origin_iri) DO NOTHING` for replay safety. Stop early on streak of conflicts
- [ ] 7.3 Refresh `remote_actors` cache row on each poll (display name, avatar, outbox URL, public key)
- [x] 7.1 pg-boss recurring job `poll-remote-outboxes` (cron every 5 min): iterate distinct accepted remote actor IRIs in `follows` where `remote_actors.last_polled_at < now() - interval '1 hour'`; for each, signed GET on the outbox using one of the local accepted-followers' keys
- [x] 7.2 Insert returned activities into `journal.activities` with `remote_origin_iri`, `remote_actor_iri`, `audience`. `ON CONFLICT (remote_origin_iri) DO NOTHING` for replay safety. Stop early on streak of conflicts
- [x] 7.3 Refresh `remote_actors` cache row on each poll (display name, avatar, outbox URL, public key)
- [ ] 7.4 Per-remote-host rate limit (1 req / 5 sec); honor `Retry-After` / `429`; back off the host
- [ ] 7.5 First-poll trigger on `accepted_at` transition (already enqueued from 4.4)
> Partially done (2026-06-07): 1 req/5 s pacing implemented; any fetch failure (incl. 429) skips the host until the next 5-min sweep. Explicit Retry-After-duration backoff remains — needs header access through Fedify's document loader errors.
- [x] 7.5 First-poll trigger on `accepted_at` transition (already enqueued from 4.4)
## 8. Feed query update