docs+openspec: prior-art research (Organic Maps, Endurain, wanderer) and 15 proposals
Add docs/inspirations.md as the durable record of the 2026-07-05/06 prior-art research — per-project learnings with source paths, canonical credit lines, and the changes each spawned — and extend the acknowledgment lists in philosophy.md/architecture.md (Organic Maps, Endurain, wanderer). New OpenSpec changes (proposal/design/specs/tasks each): - Organic Maps: elevation-profile-hardening, gpx-parser-robustness, hiking-time-estimate, poi-index, hiking-foot-profile - Endurain: account-export, activity-duplicate-review, fit-parsing-hardening, activity-locations, self-hosting-guide, activity-privacy-controls - wanderer: federation-hardening, link-share-tokens - credits-page (user-visible acknowledgments) Updated in-flight changes with wanderer prior-art sections: route-federation, route-discovery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
be4f7e4ae8
commit
5dd4968626
80 changed files with 2600 additions and 2 deletions
2
openspec/changes/account-export/.openspec.yaml
Normal file
2
openspec/changes/account-export/.openspec.yaml
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
schema: spec-driven
|
||||
created: 2026-07-06
|
||||
62
openspec/changes/account-export/design.md
Normal file
62
openspec/changes/account-export/design.md
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
## Context
|
||||
|
||||
Journal data per user (schema `packages/db/src/schema/journal.ts`): `routes` (GPX text + PostGIS geom + metadata + `route_versions` history), `activities` (GPX text + geom + stats + `photos` paths), `follows`, `notifications`, `connected_services` (encrypted tokens), `sync_imports`/`sync_pushes`, profile fields on `users`. GPX blobs are stored in-DB as text; photos on S3-compatible storage (Garage). Background jobs run via pg-boss (`@trails-cool/jobs`); notifications have an established row + rendering pattern.
|
||||
|
||||
Endurain's reference implementation (`export_service.py`) streams a ZIP of JSON dumps plus original uploaded files, memory-bounded, with a matching import service.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- One action → one ZIP with everything the user owns, GPX as the primary format, JSON only for what GPX can't express.
|
||||
- Bounded memory regardless of account size; no request-timeout coupling (async job).
|
||||
- A versioned, documented archive layout a future import can rely on.
|
||||
|
||||
**Non-Goals:**
|
||||
- Import/restore (follow-up change; the manifest version field is its hook).
|
||||
- Selective/partial export, scheduled exports, or export of other users' content beyond follow handles.
|
||||
- GDPR legal workflows (this is the technical capability; policy text is separate).
|
||||
|
||||
## Decisions
|
||||
|
||||
### 1. Archive layout: GPX files first, one manifest for the rest
|
||||
|
||||
```
|
||||
export/
|
||||
manifest.json — format version, generated_at, user profile, counts
|
||||
routes/<id>.gpx — current version
|
||||
routes/<id>.versions/<v>.gpx
|
||||
routes.json — per-route metadata (visibility, tags, day breaks, timestamps)
|
||||
activities/<id>.gpx
|
||||
activities.json — stats, sport type, visibility, route link, provider provenance
|
||||
media/<activity-id>/<file>
|
||||
follows.json, notifications.json, connections.json (names + provider only)
|
||||
```
|
||||
Open formats carry the payload; JSON carries relations and metadata keyed by the same ids as the filenames. `manifest.json.formatVersion = 1` is the import contract.
|
||||
|
||||
### 2. Async job + temporary artifact, not a streaming response
|
||||
|
||||
Export runs as a pg-boss job that streams entities from the DB in batches into a ZIP written to the media store (Garage) under an `exports/` prefix with a 7-day expiry; completion fires a notification with a signed, owner-only download URL served through the app (no public bucket URLs). Rationale: accounts with hundreds of GPX blobs exceed sane request lifetimes, and a resumable artifact beats re-generating on a dropped connection.
|
||||
|
||||
*Alternative:* synchronous streaming HTTP response — simpler, but ties memory/time to the request and breaks on large accounts; rejected.
|
||||
|
||||
### 3. Job bookkeeping in a small table
|
||||
|
||||
`export_jobs` (id, user_id, status, artifact_key, error, created_at, expires_at). One active job per user (re-request returns the running job). A daily sweep (existing jobs worker) deletes expired artifacts and rows. Rationale over reusing pg-boss state: the download link must outlive job retention and be queryable by the settings page.
|
||||
|
||||
### 4. Exclusions are structural, not filtered
|
||||
|
||||
The export queries enumerate included columns explicitly (allowlist per table) rather than dumping rows and deleting secret fields — a new secret column added later can never leak by default. Connected services export as `{provider, connectedAt}` only.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Large accounts → big artifacts on Garage] → 7-day expiry + one-active-job cap; size is bounded by the user's own data.
|
||||
- [Export contains private content in a downloadable file] → owner-only auth on the download route + unguessable artifact key + expiry; the notification link goes through app auth, not a raw bucket URL.
|
||||
- [Format drifts before import lands] → `formatVersion` + a `docs/export-format.md` page written as part of this change; import will validate the version.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
Additive: new table (`db:push`/migration), new job, new routes, new settings UI. No changes to existing data. Rollback = revert; sweep removes artifacts.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- None blocking. Whether the archive should also include a ready-to-serve ActivityPub outbox snapshot (for migration to another instance) is deferred to the import change.
|
||||
27
openspec/changes/account-export/proposal.md
Normal file
27
openspec/changes/account-export/proposal.md
Normal file
|
|
@ -0,0 +1,27 @@
|
|||
## Why
|
||||
|
||||
"All user data must be exportable in open formats" is a stated trails principle, but today it only exists as per-route/per-activity GPX downloads. There is no way to take *everything* with you — which a federated, self-hostable platform owes its users both as a data-ownership promise and as the practical instance-migration story. Endurain (reviewed 2026-07-06) ships a streaming full-account ZIP export; trails should match it with an open-format equivalent.
|
||||
|
||||
## What Changes
|
||||
|
||||
- New **"Export my data"** action in account settings: produces a ZIP containing every route and activity as its original GPX file (open format first), plus JSON manifests for entities GPX can't carry — profile, route/activity metadata (visibility, tags, sport type, day breaks), route version history, follows, connected-service *names* (never tokens), and notification history.
|
||||
- Export is generated **streamed** (bounded memory) as a background job with a notification + download link when ready; the artifact is stored temporarily and expires.
|
||||
- Media (activity photos) included when present.
|
||||
- Explicitly excluded: OAuth tokens/credentials, sessions, other users' data (followers' profiles beyond handle), federation-internal state.
|
||||
- Not in scope: **import/restore** of the archive (instance migration) — the export format is versioned and documented so a future import change can consume it.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `account-export`: The full-account export — contents, format guarantees (GPX originals + versioned JSON manifest), generation lifecycle (async job, expiry), and access control.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
<!-- none — account-management's deletion/email requirements are untouched; export is additive -->
|
||||
|
||||
## Impact
|
||||
|
||||
- `apps/journal`: new settings surface (`/settings/account`), export job in `@trails-cool/jobs` (pg-boss), streaming ZIP assembly, temporary artifact storage (S3/Garage or local disk with expiry), download route with owner-only auth.
|
||||
- `packages/db`: small `export_jobs` table (or reuse job state) for status + artifact pointer + expiry.
|
||||
- Notifications: reuses existing notification pattern for "your export is ready".
|
||||
- Privacy manifest: document what the export contains (it's the user's own data leaving via the user's own request).
|
||||
41
openspec/changes/account-export/specs/account-export/spec.md
Normal file
41
openspec/changes/account-export/specs/account-export/spec.md
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
## ADDED Requirements
|
||||
|
||||
### Requirement: Full-account export archive
|
||||
The Journal SHALL let a user export all data they own as a single ZIP archive: every route (current GPX plus version-history GPX files), every activity (GPX when present), activity media, and JSON manifests for profile, route/activity metadata, follows, notification history, and connected-service names. The archive SHALL carry a `manifest.json` with a format version. OAuth tokens, credentials, sessions, and other users' content SHALL NOT be included.
|
||||
|
||||
#### Scenario: Archive contains originals in open formats
|
||||
- **WHEN** a user with routes and activities requests an export
|
||||
- **THEN** the archive contains one GPX file per route and per activity with GPX, and JSON manifests keyed by the same ids
|
||||
|
||||
#### Scenario: Secrets never exported
|
||||
- **WHEN** a user with a connected Wahoo account exports
|
||||
- **THEN** the archive lists the connection's provider and date only, with no tokens or credentials
|
||||
|
||||
### Requirement: Asynchronous generation with bounded memory
|
||||
Export generation SHALL run as a background job that streams data in batches (memory use independent of account size), notify the user when the archive is ready, and allow only one active export job per user.
|
||||
|
||||
#### Scenario: Export of a large account completes
|
||||
- **WHEN** a user with thousands of activities requests an export
|
||||
- **THEN** the job completes without exhausting memory and a "export ready" notification links to the download
|
||||
|
||||
#### Scenario: Duplicate request returns the running job
|
||||
- **WHEN** a user requests an export while one is already generating
|
||||
- **THEN** no second job starts and the settings page shows the in-progress state
|
||||
|
||||
### Requirement: Owner-only, expiring download
|
||||
The export artifact SHALL be downloadable only by the authenticated owner through an app route, and SHALL expire (deleted from storage) after 7 days.
|
||||
|
||||
#### Scenario: Another user cannot download
|
||||
- **WHEN** a different authenticated user requests the export download URL
|
||||
- **THEN** the request is rejected
|
||||
|
||||
#### Scenario: Expired artifact is gone
|
||||
- **WHEN** 7 days pass after generation
|
||||
- **THEN** the artifact is removed from storage and the settings page offers a fresh export
|
||||
|
||||
### Requirement: Documented archive format
|
||||
The archive layout and manifest schema SHALL be documented in the repository (`docs/export-format.md`) and versioned via `manifest.formatVersion`, so a future import capability can consume archives across versions.
|
||||
|
||||
#### Scenario: Format documented
|
||||
- **WHEN** the export capability ships
|
||||
- **THEN** `docs/export-format.md` describes every file and manifest field for format version 1
|
||||
26
openspec/changes/account-export/tasks.md
Normal file
26
openspec/changes/account-export/tasks.md
Normal file
|
|
@ -0,0 +1,26 @@
|
|||
## 1. Schema & job plumbing
|
||||
|
||||
- [ ] 1.1 Add `export_jobs` table to `packages/db` (id, user_id FK cascade, status, artifact_key, error, created_at, expires_at) + migration
|
||||
- [ ] 1.2 Add `account-export` job type to `@trails-cool/jobs` and a daily sweep job that deletes expired artifacts + rows
|
||||
|
||||
## 2. Export generation
|
||||
|
||||
- [ ] 2.1 Implement the export assembler: batched queries per table (explicit column allowlists), streaming ZIP writer, layout per design (`routes/`, `activities/`, `media/`, `*.json`, `manifest.json` with formatVersion 1)
|
||||
- [ ] 2.2 Stream media files from Garage into the archive; tolerate missing media gracefully
|
||||
- [ ] 2.3 Upload artifact to storage under `exports/` with expiry metadata; mark job complete; fire "export ready" notification (existing notification pattern)
|
||||
- [ ] 2.4 Unit tests: allowlists exclude token columns; manifest counts match; a synthetic large dataset stays within a memory budget; one-active-job constraint
|
||||
|
||||
## 3. UI & download
|
||||
|
||||
- [ ] 3.1 Settings surface: "Export my data" with request button, in-progress state, ready-with-expiry state (i18n en/de)
|
||||
- [ ] 3.2 Owner-only download route streaming the artifact from storage; reject non-owners; 404 after expiry
|
||||
- [ ] 3.3 E2E: request export on a seeded account → notification → download → assert archive contents (GPX count, manifest, no token fields)
|
||||
|
||||
## 4. Docs
|
||||
|
||||
- [ ] 4.1 Write `docs/export-format.md` (format version 1: layout + manifest schema)
|
||||
- [ ] 4.2 Update privacy documentation: export exists, what it contains, expiry
|
||||
|
||||
## 5. Verification
|
||||
|
||||
- [ ] 5.1 Run `pnpm typecheck && pnpm lint && pnpm test && pnpm test:e2e`
|
||||
Loading…
Add table
Add a link
Reference in a new issue