docs+openspec: prior-art research (Organic Maps, Endurain, wanderer) and 15 proposals

Add docs/inspirations.md as the durable record of the 2026-07-05/06
prior-art research — per-project learnings with source paths, canonical
credit lines, and the changes each spawned — and extend the
acknowledgment lists in philosophy.md/architecture.md (Organic Maps,
Endurain, wanderer).

New OpenSpec changes (proposal/design/specs/tasks each):
- Organic Maps: elevation-profile-hardening, gpx-parser-robustness,
  hiking-time-estimate, poi-index, hiking-foot-profile
- Endurain: account-export, activity-duplicate-review,
  fit-parsing-hardening, activity-locations, self-hosting-guide,
  activity-privacy-controls
- wanderer: federation-hardening, link-share-tokens
- credits-page (user-visible acknowledgments)

Updated in-flight changes with wanderer prior-art sections:
route-federation, route-discovery.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Ullrich Schäfer 2026-07-06 07:23:00 +02:00
parent be4f7e4ae8
commit 5dd4968626
80 changed files with 2600 additions and 2 deletions

View file

@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-07-06

View file

@ -0,0 +1,62 @@
## Context
Journal data per user (schema `packages/db/src/schema/journal.ts`): `routes` (GPX text + PostGIS geom + metadata + `route_versions` history), `activities` (GPX text + geom + stats + `photos` paths), `follows`, `notifications`, `connected_services` (encrypted tokens), `sync_imports`/`sync_pushes`, profile fields on `users`. GPX blobs are stored in-DB as text; photos on S3-compatible storage (Garage). Background jobs run via pg-boss (`@trails-cool/jobs`); notifications have an established row + rendering pattern.
Endurain's reference implementation (`export_service.py`) streams a ZIP of JSON dumps plus original uploaded files, memory-bounded, with a matching import service.
## Goals / Non-Goals
**Goals:**
- One action → one ZIP with everything the user owns, GPX as the primary format, JSON only for what GPX can't express.
- Bounded memory regardless of account size; no request-timeout coupling (async job).
- A versioned, documented archive layout a future import can rely on.
**Non-Goals:**
- Import/restore (follow-up change; the manifest version field is its hook).
- Selective/partial export, scheduled exports, or export of other users' content beyond follow handles.
- GDPR legal workflows (this is the technical capability; policy text is separate).
## Decisions
### 1. Archive layout: GPX files first, one manifest for the rest
```
export/
manifest.json — format version, generated_at, user profile, counts
routes/<id>.gpx — current version
routes/<id>.versions/<v>.gpx
routes.json — per-route metadata (visibility, tags, day breaks, timestamps)
activities/<id>.gpx
activities.json — stats, sport type, visibility, route link, provider provenance
media/<activity-id>/<file>
follows.json, notifications.json, connections.json (names + provider only)
```
Open formats carry the payload; JSON carries relations and metadata keyed by the same ids as the filenames. `manifest.json.formatVersion = 1` is the import contract.
### 2. Async job + temporary artifact, not a streaming response
Export runs as a pg-boss job that streams entities from the DB in batches into a ZIP written to the media store (Garage) under an `exports/` prefix with a 7-day expiry; completion fires a notification with a signed, owner-only download URL served through the app (no public bucket URLs). Rationale: accounts with hundreds of GPX blobs exceed sane request lifetimes, and a resumable artifact beats re-generating on a dropped connection.
*Alternative:* synchronous streaming HTTP response — simpler, but ties memory/time to the request and breaks on large accounts; rejected.
### 3. Job bookkeeping in a small table
`export_jobs` (id, user_id, status, artifact_key, error, created_at, expires_at). One active job per user (re-request returns the running job). A daily sweep (existing jobs worker) deletes expired artifacts and rows. Rationale over reusing pg-boss state: the download link must outlive job retention and be queryable by the settings page.
### 4. Exclusions are structural, not filtered
The export queries enumerate included columns explicitly (allowlist per table) rather than dumping rows and deleting secret fields — a new secret column added later can never leak by default. Connected services export as `{provider, connectedAt}` only.
## Risks / Trade-offs
- [Large accounts → big artifacts on Garage] → 7-day expiry + one-active-job cap; size is bounded by the user's own data.
- [Export contains private content in a downloadable file] → owner-only auth on the download route + unguessable artifact key + expiry; the notification link goes through app auth, not a raw bucket URL.
- [Format drifts before import lands] → `formatVersion` + a `docs/export-format.md` page written as part of this change; import will validate the version.
## Migration Plan
Additive: new table (`db:push`/migration), new job, new routes, new settings UI. No changes to existing data. Rollback = revert; sweep removes artifacts.
## Open Questions
- None blocking. Whether the archive should also include a ready-to-serve ActivityPub outbox snapshot (for migration to another instance) is deferred to the import change.

View file

@ -0,0 +1,27 @@
## Why
"All user data must be exportable in open formats" is a stated trails principle, but today it only exists as per-route/per-activity GPX downloads. There is no way to take *everything* with you — which a federated, self-hostable platform owes its users both as a data-ownership promise and as the practical instance-migration story. Endurain (reviewed 2026-07-06) ships a streaming full-account ZIP export; trails should match it with an open-format equivalent.
## What Changes
- New **"Export my data"** action in account settings: produces a ZIP containing every route and activity as its original GPX file (open format first), plus JSON manifests for entities GPX can't carry — profile, route/activity metadata (visibility, tags, sport type, day breaks), route version history, follows, connected-service *names* (never tokens), and notification history.
- Export is generated **streamed** (bounded memory) as a background job with a notification + download link when ready; the artifact is stored temporarily and expires.
- Media (activity photos) included when present.
- Explicitly excluded: OAuth tokens/credentials, sessions, other users' data (followers' profiles beyond handle), federation-internal state.
- Not in scope: **import/restore** of the archive (instance migration) — the export format is versioned and documented so a future import change can consume it.
## Capabilities
### New Capabilities
- `account-export`: The full-account export — contents, format guarantees (GPX originals + versioned JSON manifest), generation lifecycle (async job, expiry), and access control.
### Modified Capabilities
<!-- none — account-management's deletion/email requirements are untouched; export is additive -->
## Impact
- `apps/journal`: new settings surface (`/settings/account`), export job in `@trails-cool/jobs` (pg-boss), streaming ZIP assembly, temporary artifact storage (S3/Garage or local disk with expiry), download route with owner-only auth.
- `packages/db`: small `export_jobs` table (or reuse job state) for status + artifact pointer + expiry.
- Notifications: reuses existing notification pattern for "your export is ready".
- Privacy manifest: document what the export contains (it's the user's own data leaving via the user's own request).

View file

@ -0,0 +1,41 @@
## ADDED Requirements
### Requirement: Full-account export archive
The Journal SHALL let a user export all data they own as a single ZIP archive: every route (current GPX plus version-history GPX files), every activity (GPX when present), activity media, and JSON manifests for profile, route/activity metadata, follows, notification history, and connected-service names. The archive SHALL carry a `manifest.json` with a format version. OAuth tokens, credentials, sessions, and other users' content SHALL NOT be included.
#### Scenario: Archive contains originals in open formats
- **WHEN** a user with routes and activities requests an export
- **THEN** the archive contains one GPX file per route and per activity with GPX, and JSON manifests keyed by the same ids
#### Scenario: Secrets never exported
- **WHEN** a user with a connected Wahoo account exports
- **THEN** the archive lists the connection's provider and date only, with no tokens or credentials
### Requirement: Asynchronous generation with bounded memory
Export generation SHALL run as a background job that streams data in batches (memory use independent of account size), notify the user when the archive is ready, and allow only one active export job per user.
#### Scenario: Export of a large account completes
- **WHEN** a user with thousands of activities requests an export
- **THEN** the job completes without exhausting memory and a "export ready" notification links to the download
#### Scenario: Duplicate request returns the running job
- **WHEN** a user requests an export while one is already generating
- **THEN** no second job starts and the settings page shows the in-progress state
### Requirement: Owner-only, expiring download
The export artifact SHALL be downloadable only by the authenticated owner through an app route, and SHALL expire (deleted from storage) after 7 days.
#### Scenario: Another user cannot download
- **WHEN** a different authenticated user requests the export download URL
- **THEN** the request is rejected
#### Scenario: Expired artifact is gone
- **WHEN** 7 days pass after generation
- **THEN** the artifact is removed from storage and the settings page offers a fresh export
### Requirement: Documented archive format
The archive layout and manifest schema SHALL be documented in the repository (`docs/export-format.md`) and versioned via `manifest.formatVersion`, so a future import capability can consume archives across versions.
#### Scenario: Format documented
- **WHEN** the export capability ships
- **THEN** `docs/export-format.md` describes every file and manifest field for format version 1

View file

@ -0,0 +1,26 @@
## 1. Schema & job plumbing
- [ ] 1.1 Add `export_jobs` table to `packages/db` (id, user_id FK cascade, status, artifact_key, error, created_at, expires_at) + migration
- [ ] 1.2 Add `account-export` job type to `@trails-cool/jobs` and a daily sweep job that deletes expired artifacts + rows
## 2. Export generation
- [ ] 2.1 Implement the export assembler: batched queries per table (explicit column allowlists), streaming ZIP writer, layout per design (`routes/`, `activities/`, `media/`, `*.json`, `manifest.json` with formatVersion 1)
- [ ] 2.2 Stream media files from Garage into the archive; tolerate missing media gracefully
- [ ] 2.3 Upload artifact to storage under `exports/` with expiry metadata; mark job complete; fire "export ready" notification (existing notification pattern)
- [ ] 2.4 Unit tests: allowlists exclude token columns; manifest counts match; a synthetic large dataset stays within a memory budget; one-active-job constraint
## 3. UI & download
- [ ] 3.1 Settings surface: "Export my data" with request button, in-progress state, ready-with-expiry state (i18n en/de)
- [ ] 3.2 Owner-only download route streaming the artifact from storage; reject non-owners; 404 after expiry
- [ ] 3.3 E2E: request export on a seeded account → notification → download → assert archive contents (GPX count, manifest, no token fields)
## 4. Docs
- [ ] 4.1 Write `docs/export-format.md` (format version 1: layout + manifest schema)
- [ ] 4.2 Update privacy documentation: export exists, what it contains, expiry
## 5. Verification
- [ ] 5.1 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e`