feat(journal): outbox-poll ingestion of remote trails activities (§7)
Tasks 7.1–7.3, 7.5 (+7.4 pacing; Retry-After-duration backoff noted as remaining). Resolves the activities.owner_id open question. Schema (design decision from the open question): - activities.owner_id nullable + check constraint enforcing exactly one of (owner_id, remote_actor_iri) — the follows pattern again. - remote_published_at carries the origin's publish time for the §8 feed sort; index on remote_actor_iri for the feed join. - Compiler-audited fallout: notification fan-out, the Note object dispatcher, and the activity detail loader explicitly skip/404 remote rows (their canonical page is the origin; feed links there). Every other surface joins users on owner_id and excludes remote rows structurally. Ingestion: - federation-ingest.server.ts: parseOutboxItem targets exactly our own outgoing Create(Note) shape (PropertyValue stats, first-paragraph name, audience from to/cc); unknown items skipped, never fatal. - ingestRemoteActivities: replay-safe via unique remote_origin_iri, conflict-streak early exit (outboxes are newest-first). - pollRemoteActor: signed fetch (Authorized Fetch via a local follower's key), outbox resolution via remote_actors cache with actor-doc fallback (which refreshes the cache), 1 req/5 s per-host pacing, last_polled_at stamping. Network + pacing injectable. Jobs: poll-remote-actor (per-actor; the §4 Accept listener already enqueues it as the first poll) + poll-remote-outboxes (5-min cron sweep over accepted remote follows not polled within the hour). Tests: parse-shape units; integration suite for ingestion provenance, audience→visibility mirroring, replay safety, the DB author invariant, poll flow (actor-doc → collection → page), signer requirement, and due-polling selection. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
9ada6ab7f3
commit
b96bef91a9
13 changed files with 628 additions and 21 deletions
|
|
@ -147,13 +147,17 @@ Inbound signature verification uses the actor's public key from their actor obje
|
|||
|
||||
## Open Questions
|
||||
|
||||
- **`activities.owner_id` is NOT NULL but remote-ingested rows have no local
|
||||
owner** (surfaced during task 2.3, 2026-06-06). Options: make `owner_id`
|
||||
nullable with a check constraint (`owner_id IS NOT NULL OR remote_actor_iri
|
||||
IS NOT NULL`), or key remote rows purely off `remote_actor_iri` in a way
|
||||
that never touches owner-joined queries. Decide in task 7.2 (ingestion)
|
||||
before any remote row is written; the columns landed in 2.3 don't prejudge
|
||||
either option.
|
||||
- ~~**`activities.owner_id` is NOT NULL but remote-ingested rows have no local
|
||||
owner**~~ — **Resolved in task 7.2 (2026-06-07):** `owner_id` is nullable
|
||||
with a check constraint enforcing exactly one of (`owner_id`,
|
||||
`remote_actor_iri`) — the same pattern as `follows.follower_id` /
|
||||
`follower_actor_iri`. Compiler-audited fallout: notification fan-out,
|
||||
the Note object dispatcher, and the activity detail loader all
|
||||
explicitly 404/skip remote rows (their canonical page is the origin
|
||||
instance; the feed links outward). Every other surface joins `users`
|
||||
on `owner_id` and excludes remote rows structurally. A
|
||||
`remote_published_at` column carries the origin's publish time for
|
||||
the §8 feed sort.
|
||||
|
||||
- Custom AS extension for activity-type vs. plain `Create(Note)`? Mastodon will only render Notes; an extension type means non-trails clients see nothing useful. Lean toward `Create(Note)` with structured metadata in `attachment` so Mastodon shows the text + GPX link, and trails clients consuming the outbox can read the structured fields.
|
||||
- Per-instance inbox vs per-user inbox? Mastodon supports a "shared inbox" optimization. Worth doing for delivery efficiency once we have any volume; v1 can use per-user inboxes only.
|
||||
|
|
|
|||
|
|
@ -57,11 +57,12 @@
|
|||
|
||||
## 7. Outbox-poll ingestion
|
||||
|
||||
- [ ] 7.1 pg-boss recurring job `poll-remote-outboxes` (cron every 5 min): iterate distinct accepted remote actor IRIs in `follows` where `remote_actors.last_polled_at < now() - interval '1 hour'`; for each, signed GET on the outbox using one of the local accepted-followers' keys
|
||||
- [ ] 7.2 Insert returned activities into `journal.activities` with `remote_origin_iri`, `remote_actor_iri`, `audience`. `ON CONFLICT (remote_origin_iri) DO NOTHING` for replay safety. Stop early on streak of conflicts
|
||||
- [ ] 7.3 Refresh `remote_actors` cache row on each poll (display name, avatar, outbox URL, public key)
|
||||
- [x] 7.1 pg-boss recurring job `poll-remote-outboxes` (cron every 5 min): iterate distinct accepted remote actor IRIs in `follows` where `remote_actors.last_polled_at < now() - interval '1 hour'`; for each, signed GET on the outbox using one of the local accepted-followers' keys
|
||||
- [x] 7.2 Insert returned activities into `journal.activities` with `remote_origin_iri`, `remote_actor_iri`, `audience`. `ON CONFLICT (remote_origin_iri) DO NOTHING` for replay safety. Stop early on streak of conflicts
|
||||
- [x] 7.3 Refresh `remote_actors` cache row on each poll (display name, avatar, outbox URL, public key)
|
||||
- [ ] 7.4 Per-remote-host rate limit (1 req / 5 sec); honor `Retry-After` / `429`; back off the host
|
||||
- [ ] 7.5 First-poll trigger on `accepted_at` transition (already enqueued from 4.4)
|
||||
> Partially done (2026-06-07): 1 req/5 s pacing implemented; any fetch failure (incl. 429) skips the host until the next 5-min sweep. Explicit Retry-After-duration backoff remains — needs header access through Fedify's document loader errors.
|
||||
- [x] 7.5 First-poll trigger on `accepted_at` transition (already enqueued from 4.4)
|
||||
|
||||
## 8. Feed query update
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue