docs+openspec: prior-art research (Organic Maps, Endurain, wanderer) and 15 proposals
Add docs/inspirations.md as the durable record of the 2026-07-05/06 prior-art research — per-project learnings with source paths, canonical credit lines, and the changes each spawned — and extend the acknowledgment lists in philosophy.md/architecture.md (Organic Maps, Endurain, wanderer). New OpenSpec changes (proposal/design/specs/tasks each): - Organic Maps: elevation-profile-hardening, gpx-parser-robustness, hiking-time-estimate, poi-index, hiking-foot-profile - Endurain: account-export, activity-duplicate-review, fit-parsing-hardening, activity-locations, self-hosting-guide, activity-privacy-controls - wanderer: federation-hardening, link-share-tokens - credits-page (user-visible acknowledgments) Updated in-flight changes with wanderer prior-art sections: route-federation, route-discovery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
be4f7e4ae8
commit
5dd4968626
80 changed files with 2600 additions and 2 deletions
|
|
@ -0,0 +1,2 @@
|
|||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
55
openspec/changes/activity-duplicate-review/design.md
Normal file
55
openspec/changes/activity-duplicate-review/design.md
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
## Context
|
||||
|
||||
Activity creation happens through several paths — Wahoo webhook/manual import, Komoot import, manual GPX upload, and soon Garmin backfill — all converging on the gpx-save layer. Per-provider dedup (`sync_imports` unique on provider + external id) prevents same-source re-imports; `import_batches` counts those. Nothing compares *across* sources. Activities carry `startedAt` (nullable — planned-route-style uploads may lack it) and `duration`. Feeds/stats/federation currently filter by `visibility` and `synthetic`.
|
||||
|
||||
Endurain's reference: exact same-start-time match → store with `is_hidden=true` + notification for manual review. We adopt the pattern but widen the match window (devices disagree by seconds) and tighten the quarantine semantics.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- The same physical activity arriving twice never double-counts in stats or appears twice to followers — without silently discarding data the user may want.
|
||||
- One detection point, one exclusion predicate — impossible for a new surface to forget the filter accidentally... (see risk).
|
||||
- Resolution is a two-tap user decision.
|
||||
|
||||
**Non-Goals:**
|
||||
- Merging (picking best-of fields from both copies).
|
||||
- Geometry-similarity matching (same route ≠ same activity; time overlap is the honest signal).
|
||||
- Retroactive dedup of pre-existing rows (could be a later one-shot job).
|
||||
- Route dedup.
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Heuristic: time overlap, not exact equality
|
||||
|
||||
Suspect when same owner AND both have `startedAt` AND |Δstart| ≤ 120 s AND (if both have durations) the intervals overlap ≥ 70%. Rationale: Garmin and Wahoo timestamps for one ride differ by GPS-fix seconds; exact equality (Endurain) misses those, while a bare ±2 min window without overlap check would false-positive back-to-back track repeats. Activities without `startedAt` are never suspected (nothing to compare).
|
||||
|
||||
*Alternative:* geometry proximity — rejected as both expensive and wrong (riding the same loop twice is two activities).
|
||||
|
||||
### 2. Quarantine = a self-referential FK, filtered everywhere via one predicate
|
||||
|
||||
`activities.suspected_duplicate_of_id` references the *earlier* activity. A shared query helper (e.g. `visibleActivities()` scope used by feed/listings/stats/outbox) gains `AND suspected_duplicate_of_id IS NULL`. The activity detail page remains reachable by the owner (that's the review surface); non-owners get 404 as if private. Federation: the Create is simply not emitted while quarantined; on "keep both", normal publish fires then.
|
||||
|
||||
*Alternative:* reuse `visibility='private'` + a reason column — rejected: it would destroy the user's chosen visibility (which must survive "keep both") and overload a user-facing concept with system state.
|
||||
|
||||
### 3. Resolution actions
|
||||
|
||||
Banner on the activity page (and notification deep-link): shows both copies' source (provider from `sync_imports`/upload), start time, duration, distance. **Keep both** → clear the FK; activity publishes with its original visibility (federation Create fires now). **Discard** → hard-delete the flagged copy (its `sync_imports` row remains, so the provider won't re-import it). The *original* activity is never touched.
|
||||
|
||||
### 4. Detection lives in the shared save path
|
||||
|
||||
The check runs inside the same transaction as activity insert for import-shaped creations (provider imports, uploads). Planner-saved activities don't exist (planner saves routes), so no exclusions needed. Import batches increment a new `suspectedDuplicateCount`.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [A forgotten surface still shows quarantined rows] → The exclusion lives in the shared activity-listing helper; the task list includes an audit of every query site + a regression test that a quarantined activity is absent from feed, profile, stats, API list, and outbox.
|
||||
- [False positives annoy users (e.g. two people's devices on one account)] → Both copies survive; resolution is one tap; the window is deliberately tight.
|
||||
- [Race: two imports in parallel create both copies with neither flagged] → Detection query runs `FOR UPDATE` on the owner's recent-activity window inside the insert transaction; worst case (different processes) is caught by whichever commits second.
|
||||
- [Nullable startedAt uploads bypass detection] → Accepted; without a timestamp there is no honest signal, and manual uploads are user-intentional.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Additive column + index (`owner_id, started_at` index already exists). Ship detection + exclusion + UI together. Rollback = revert; quarantined rows become visible again (harmless).
|
||||
|
||||
## Open Questions
|
||||
|
||||
- None blocking. A retroactive scan job for existing duplicates can be a follow-up once the review UI exists.
|
||||
27
openspec/changes/activity-duplicate-review/proposal.md
Normal file
27
openspec/changes/activity-duplicate-review/proposal.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
## Why
|
||||
|
||||
Dedup today is per-provider only: `sync_imports` keys on `(provider, externalWorkoutId)`, so re-delivery from the *same* provider is caught. But the same real-world ride can arrive from two sources — Wahoo webhook + Garmin backfill (pending), Komoot import + manual GPX upload — and nothing catches that: the user gets two activities, double distance in weekly stats, and a duplicated feed entry for followers. With Garmin and COROS on the roadmap this will become routine. Endurain's pattern (reviewed 2026-07-06): store the incoming activity anyway, flag it as a suspected duplicate, hide it from all surfaces, and let the user resolve — no silent auto-merge, no data loss.
|
||||
|
||||
## What Changes
|
||||
|
||||
- On every activity creation from an import path (provider sync, manual upload, Komoot), a **cross-source duplicate check** runs: same owner + overlapping time window (`started_at` within ±2 minutes and durations overlapping ≥ 70%, when both have times).
|
||||
- A suspected duplicate is stored with `suspectedDuplicateOfId` set and is **excluded from feeds, listings, profile stats, federation, and follower notifications** until resolved.
|
||||
- The owner gets a "possible duplicate import" notification and a **review UI** showing both activities side by side (source, time, distance, map) with two actions: **keep both** (clears the flag, activity becomes normal) or **discard** (deletes the flagged copy).
|
||||
- Import batches report suspected duplicates distinctly from exact-ID duplicates (`import_batches.duplicateCount` counts exact; a new counter for suspected).
|
||||
- Not in scope: auto-merging data from two sources into one activity, retroactive scanning of existing activities, and route (non-activity) dedup.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `duplicate-detection`: Cross-source duplicate detection on activity import — the heuristic, the quarantine semantics (hidden-from-everything until resolved), the notification, and the resolve flow.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
<!-- none — per-provider dedup requirements in wahoo-import stay as-is; this adds a second, cross-source layer. Feed/stats/federation specs are not contradicted: a quarantined activity is simply not yet published, which their requirements already permit for private content. -->
|
||||
|
||||
## Impact
|
||||
|
||||
- `packages/db`: `activities.suspected_duplicate_of_id` (nullable text, FK activities, on delete set null); new counter column on `import_batches`.
|
||||
- All import call sites converge on the shared save path (`gpx-save.server.ts` / activity creation helper) — the check lives once, there.
|
||||
- Query-layer exclusion: feed, listings, profile stats, weekly distance, federation outbox, follower notifications must filter `suspected_duplicate_of_id IS NULL` (single shared predicate).
|
||||
- New review UI (activity page banner + notification deep link), notification type, i18n strings.
|
||||
|
|
@ -0,0 +1,41 @@
|
|||
## ADDED Requirements
|
||||
|
||||
### Requirement: Cross-source duplicate detection on import
|
||||
When an activity is created via any import path (provider sync, manual GPX upload, Komoot), the Journal SHALL check for an existing activity of the same owner whose start time is within 120 seconds and, when both durations are known, whose time interval overlaps at least 70%. On a match, the new activity SHALL be stored with a reference to the suspected original rather than discarded. Activities without a start time SHALL never be flagged.
|
||||
|
||||
#### Scenario: Same ride from two providers
|
||||
- **WHEN** a Wahoo import created an activity and a Garmin backfill delivers the same ride starting 30 seconds apart
|
||||
- **THEN** the second activity is stored flagged as a suspected duplicate of the first
|
||||
|
||||
#### Scenario: Back-to-back repeats are not duplicates
|
||||
- **WHEN** two activities start 90 seconds apart but the first lasts only 1 minute (intervals barely overlap)
|
||||
- **THEN** neither is flagged
|
||||
|
||||
### Requirement: Quarantine until resolved
|
||||
A suspected-duplicate activity SHALL be excluded from feeds, listings, profile statistics, the public API, follower notifications, and federation until resolved. The owner SHALL still be able to open it; other users SHALL NOT.
|
||||
|
||||
#### Scenario: Stats not double-counted
|
||||
- **WHEN** a suspected duplicate exists
|
||||
- **THEN** weekly distance and profile stats count the original only
|
||||
|
||||
#### Scenario: Not federated while quarantined
|
||||
- **WHEN** an activity is flagged
|
||||
- **THEN** no ActivityPub Create is emitted for it
|
||||
|
||||
### Requirement: Owner notification and review
|
||||
The owner SHALL receive a "possible duplicate import" notification linking to a review surface that shows both activities' source, start time, duration, and distance, offering **keep both** (clears the flag; the activity publishes with its original visibility, including federation) and **discard** (deletes the flagged copy while keeping the provider import record so it is not re-imported).
|
||||
|
||||
#### Scenario: Keep both
|
||||
- **WHEN** the owner chooses keep both
|
||||
- **THEN** the flag is cleared and the activity appears in feeds/stats and federates normally
|
||||
|
||||
#### Scenario: Discard
|
||||
- **WHEN** the owner chooses discard
|
||||
- **THEN** the flagged activity is deleted, the original is untouched, and a provider re-sync does not recreate it
|
||||
|
||||
### Requirement: Import batch reporting
|
||||
Import batches SHALL count suspected duplicates separately from exact same-provider duplicates.
|
||||
|
||||
#### Scenario: Batch summary distinguishes
|
||||
- **WHEN** a backfill imports 10 activities of which 2 match existing Wahoo activities by time overlap
|
||||
- **THEN** the batch reports 2 suspected duplicates distinct from exact-ID skips
|
||||
26
openspec/changes/activity-duplicate-review/tasks.md
Normal file
26
openspec/changes/activity-duplicate-review/tasks.md
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
## 1. Schema
|
||||
|
||||
- [ ] 1.1 Add `activities.suspected_duplicate_of_id` (nullable, FK activities on delete set null) and `import_batches.suspected_duplicate_count` + migration
|
||||
|
||||
## 2. Detection
|
||||
|
||||
- [ ] 2.1 Implement the overlap heuristic (±120 s start delta, ≥70% interval overlap when durations known) as a pure, unit-tested function
|
||||
- [ ] 2.2 Wire it into the shared activity-creation path inside the insert transaction (row-locked window query); set the FK on match; increment batch counter for batch imports
|
||||
- [ ] 2.3 Unit tests: two-provider same ride flagged; back-to-back repeats not flagged; missing startedAt bypasses; concurrent insert race resolves to one flagged copy
|
||||
|
||||
## 3. Quarantine enforcement
|
||||
|
||||
- [ ] 3.1 Add `suspected_duplicate_of_id IS NULL` to the shared activity-listing scope; audit every query site (feed, listings, profile stats, weekly distance, api.v1 lists, federation outbox, follower notification trigger) to confirm they use the scope
|
||||
- [ ] 3.2 Owner can open the flagged activity; non-owners get the private-equivalent 404
|
||||
- [ ] 3.3 Regression test: a flagged activity appears in none of feed / profile stats / API list / outbox, and its follower notification never fires
|
||||
|
||||
## 4. Notification + review UI
|
||||
|
||||
- [ ] 4.1 New notification type "possible duplicate import" (payload: both activity ids) following the versioned-payload pattern; deep link to the flagged activity
|
||||
- [ ] 4.2 Review banner on the flagged activity page: side-by-side source/time/duration/distance, keep-both and discard actions (i18n en/de)
|
||||
- [ ] 4.3 Keep both → clear flag + emit federation Create + follower notification now; Discard → delete flagged copy, retain `sync_imports` row; tests for both paths
|
||||
|
||||
## 5. Verification
|
||||
|
||||
- [ ] 5.1 E2E: import the same GPX twice via two paths on a seeded account → banner appears → resolve both ways
|
||||
- [ ] 5.2 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e`
|
||||
Loading…
Add table
Add a link
Reference in a new issue