diff --git a/AGENTS.md b/AGENTS.md index 838e4ca..5d42913 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,9 +2,9 @@ ## Project Overview -Archivr is a self-hosted archival tool that captures and preserves digital content — YouTube/Twitter/Instagram/TikTok/Reddit posts, arbitrary URLs, full web pages (via SingleFile + Chromium), and local files — into self-contained, SQLite-backed archive directories with blob deduplication, hierarchical tags, collections, and role-based auth. Rust workspace + React frontend. +Archivr is a self-hosted archival tool that captures and preserves digital content — YouTube/Twitter/Instagram/TikTok/Reddit posts, arbitrary URLs, full web pages (via SingleFile + Chromium, optionally through a Freedium mirror for paywalled articles), and local files — into self-contained, SQLite-backed archive directories with blob deduplication, hierarchical tags, collections, and role-based auth. Rust workspace + React frontend. -Read `ARCHIVR-MENTAL-MODEL.md` before making structural changes; `NEXT.md` tracks the roadmap (Tracks 1–7 done, Track 8 = Collections UI is next). +Read `ARCHIVR-MENTAL-MODEL.md` before making structural changes. ## Architecture & Data Flow @@ -14,7 +14,7 @@ Three crates with a strict ownership split — **core owns truth; CLI and server - `crates/archivr-server` — Axum HTTP API + auth + static frontend serving. - `crates/archivr-cli` — clap-based CLI (`archivr` binary): `init`, `archive` subcommands. -Capture flow: locator → `determine_source()` (`crates/archivr-core/src/capture.rs`) routes by platform/shorthand (`yt:`, `x:`, `tweet:` …) → platform downloader (`downloader/ytdlp.rs`, `tweets.rs`, `singlefile.rs`, `http.rs`, `local.rs`) stages into `temp/` → SHA3-256 dedup (`hash.rs`, `downloader/store.rs`) moves blobs to `raw/A/B/HASH.EXT` → rows written to `archivr.sqlite` (runs, entries, artifacts, blobs) → served via `/api/archives/:id/...`. +Capture flow: locator → `determine_source()` (`crates/archivr-core/src/capture.rs`) routes by platform/shorthand (`yt:`, `x:`, `tweet:` …) → platform downloader (`downloader/ytdlp.rs`, `tweets.rs`, `singlefile.rs`, `http.rs`, `local.rs`) stages into `temp/` → SHA3-256 dedup (`hash.rs`, `downloader/store.rs`) moves blobs to `raw/A/B/HASH.EXT` → rows written to `archivr.sqlite` (runs, entries, artifacts, blobs) → served via `/api/archives/:id/...`. `CaptureConfig` carries per-request toggles (uBlock, reader mode, Freedium mirror, etc.); when `via_freedium` is set, the fetch URL is rewritten through `freedium-mirror.cfd` while the canonical DB URL stays the original locator. Per-archive layout (created by `archivr init`): `.archivr/` (name, store_path, `archivr.sqlite`) + sibling `store/` (`raw/`, `raw_tweets/`, `structured/`, `temp/`). Server-level auth lives in a **separate** `archivr-auth.sqlite` (users, sessions, API tokens, role bits GUEST=1/USER=2/ADMIN=4/OWNER=8). @@ -31,6 +31,7 @@ The server mounts multiple archives from a TOML registry (`crates/archivr-server | `docs/` | User docs (`README.md`), `superpowers/plans/` and `superpowers/specs/` (dated design docs — write plans there before large features) | | `modules/nixos/` | NixOS module (`services.archivr-server`) | | `vendor/twitter/` | Vendored Twitter scraper (active; the Python the server shells out to). Don't refactor casually. | +| `vendor/readability/` | Mozilla `Readability.js`, concatenated into the SingleFile reader-mode browser script by `downloader/singlefile.rs`. | | `testing/` | Legacy scraping scripts + sample data. `testing/creds.txt` holds real tokens — never read, commit, or print it. | ## Development Commands diff --git a/ARCHIVR-MENTAL-MODEL.md b/ARCHIVR-MENTAL-MODEL.md index f383ff1..ec90e05 100644 --- a/ARCHIVR-MENTAL-MODEL.md +++ b/ARCHIVR-MENTAL-MODEL.md @@ -117,6 +117,16 @@ sequenceDiagram CLI->>User: terminal result ``` +## Web Capture Pipeline + +Web pages (`Source::WebPage`) take a longer path than yt-dlp or tweets: + +1. **Fetch URL selection.** If `via_freedium` is set and the locator isn't already a Freedium URL, the downloader fetches the page through `freedium-mirror.cfd` with no forwarded cookies. The canonical DB URL stays the original locator. +2. **Browser capture.** `downloader/singlefile.rs` shells out to `single-file-cli` driving headless Chromium. Extensions (uBlock, cookie-consent) and injected browser scripts (modal closer, reader mode) attach per `CaptureConfig`. +3. **Reader mode** (optional) concatenates Mozilla's `Readability.js` from `vendor/readability/` into the SingleFile browser script and stamps absolute URLs on lazy images so the serialised DOM points to fetchable sources. +4. **Freedium cleanup** strips mirror UI (nav, footer, toaster, author header, download control) using multi-signal selectors so article-authored controls aren't hit. +5. **Rust post-processing.** After SingleFile writes the HTML, a Rust pass fetches any images the browser couldn't inline (bounded reads, same-origin cookie forwarding) and embeds them as data URIs. Title extraction runs after embedded font blocks are stripped so large fonts don't push `` beyond the read window. + ## Read Data Flow When opening the web UI: @@ -142,14 +152,17 @@ sequenceDiagram | Feature kind | Edit here | |---|---| | DB schema, inserts, archive runs, entries, tags | `crates/archivr-core/src/database.rs` | +| Capture orchestration, `Source` routing, `CaptureConfig` | `crates/archivr-core/src/capture.rs` | | Archive opening, listing entries, entry detail, runs | `crates/archivr-core/src/archive.rs` | | Download/save behavior | `crates/archivr-core/src/downloader/` | | CLI commands, argument parsing, terminal output | `crates/archivr-cli/src/main.rs` | | Server API routes | `crates/archivr-server/src/routes.rs` | +| Auth model (users, sessions, tokens, roles) | `crates/archivr-server/src/auth.rs` | | Mounted archive config model | `crates/archivr-server/src/registry.rs` | -| Browser UI behavior | `crates/archivr-server/static/app.js` | -| Browser UI layout | `crates/archivr-server/static/index.html` | -| Browser UI styling | `crates/archivr-server/static/styles.css` | +| Frontend root state + routing | `frontend/src/App.jsx` | +| Frontend API client | `frontend/src/api.js` | +| Frontend components | `frontend/src/components/` | +| Frontend styling | `frontend/src/styles.css` | ## Practical Feature Rule @@ -165,12 +178,13 @@ If a browser feature needs new data, the usual order is: 2. Expose it in `archivr-server`. 3. Render it in the static UI. -## Current Limitations +## Server Capabilities -The web server reads archive data and serves the UI. It does not yet implement capture. +The server both reads and writes archive data. Capture jobs are asynchronous: `POST /api/archives/:id/captures` inserts a job row, spawns a blocking task, and returns immediately; the frontend polls until the job completes or fails. Heavy work stays synchronous inside `archivr-core`. -Search is currently simple client-side filtering. +**Auth model.** A separate `archivr-auth.sqlite` (path derived from the server config directory) holds users, sessions, and API tokens. Role bits are `u32` flags (`GUEST`, `USER`, `ADMIN`, `OWNER`) so a single bitmask value covers assignment, checks, and visibility. The middleware stack is `setup_guard` → `login_rate_limit` → `security_headers`; route families are classified `READ / ADMIN / WRITE / STATIC` in `routes.rs`. -**Auth and session model:** The server binds to `127.0.0.1` by default and has no authentication middleware. This is intentional — Archivr is a local tool. The bind address is configurable via the TOML `bind` field or `ARCHIVR_BIND` env var; a non-loopback address triggers a startup warning. Route families are classified (READ / ADMIN / WRITE / STATIC) in `crates/archivr-server/src/routes.rs` as the decision record for when middleware is eventually added. See the "Security and Deployment" section in `docs/README.md`. +**Search** is client-side filtering over entries the frontend has already fetched. + +**Admin view** covers mounted archives, users, sessions, and API tokens. -Admin is a mounted-archives view, not a management system. diff --git a/NEXT.md b/NEXT.md deleted file mode 100644 index be3460e..0000000 --- a/NEXT.md +++ /dev/null @@ -1,225 +0,0 @@ -# Archivr Next Work Plan - -> **For future agentic workers:** Read `ARCHIVR-MENTAL-MODEL.md` first, then this file. -> All six tracks from the previous handoff are complete. This document is the current roadmap. - -## Current State - -All five product tracks from the prior session are done: - -| Feature | Status | -|---|---| -| Entry detail + artifact serving | Done | -| Server-side search with prefix filters | Done | -| Hierarchical tags (CRUD, tree, entry assignment, filter) | Done | -| Browser capture button and POST endpoint | Done | -| Local-only auth boundary, bind address config, docs | Done | - -The frontend is a React + Vite app (`frontend/`) that builds into `crates/archivr-server/static/`. - -The remaining gaps come from `docs/README.md` milestones that were never in the prior tracks. - ---- - -## Remaining Work (in order) - -### ~~1. Generic URL capture — plain file download~~ ✅ Done - -**Implemented:** `crates/archivr-core/src/downloader/http.rs` (`download` fn, extension helpers, -HTML rejection), `Source::Url` variant in `capture.rs`, `determine_source` routes all unmatched -`http://`/`https://` URLs to `Source::Url`, `perform_capture` arm calls the downloader and flows -through the existing temp → hash → raw store pipeline. `reqwest 0.12` (blocking) added to workspace. -`docs/README.md` "URLs" milestone checked off. 112 tests green. - ---- - -### ~~2. Web page archiving — single-file HTML snapshots~~ ✅ Done - -**Implemented:** `crates/archivr-core/src/downloader/singlefile.rs` (new module; shells out to -`single-file-cli` via `ARCHIVR_SINGLE_FILE` env var; launches headless Chromium via -`ARCHIVR_CHROME`; flags: `--browser-wait-until=networkidle2`, `--remove-unused-styles=false`, -`--block-scripts=false`, `--remove-alternative-medias=false` for full CSS/font fidelity). -`Source::WebPage` variant added to `capture.rs`; HEAD-first probe (`probe_url_kind` in `http.rs`) -routes `text/html` responses to `Source::WebPage` and everything else to `Source::Url`. -Font deduplication implemented as a follow-on: `font_extractor.rs` post-processes the saved HTML, -extracts embedded `@font-face` base64 blobs, stores each as a deduplicated artifact in the raw -store, and rewrites the HTML to reference them via `/api/archives/:id/blobs/:sha256`. -`hash_bytes` added to `hash.rs`; `get_blob_by_sha256` added to `database.rs`; -`GET /api/archives/:id/blobs/:sha256` route added to `routes.rs`. -`flake.nix` updated: `pkgs.single-file-cli` and (Linux-only) `pkgs.chromium` added to -`buildInputs`; `ARCHIVR_SINGLE_FILE` and `ARCHIVR_CHROME` env vars wired into both wrappers. -`docs/README.md` "Archive web pages" milestone checked off. 143 tests green. - ---- - -### 3. Async capture jobs - -**What:** Capture currently runs synchronously on the HTTP request thread. A YouTube video, -a large tweet thread, or a slow monolith page will stall the browser request. This track -adds a job queue so `POST /api/archives/:id/captures` returns immediately and the UI polls -for completion. - -**Where to edit:** - -| File | Change | -|---|---| -| `crates/archivr-core/src/database.rs` | Add `capture_jobs` table to the schema: `(id, job_uid, run_uid, archive_id, status TEXT CHECK(status IN ('pending','running','completed','failed')), error_text, created_at, updated_at)`. Add `create_capture_job`, `update_capture_job_status`, `get_capture_job` helpers. | -| `crates/archivr-core/src/archive.rs` | Add `CaptureJob` summary type and `get_capture_job` query. | -| `crates/archivr-server/src/routes.rs` | `POST /api/archives/:id/captures` → insert job row, spawn `tokio::task::spawn_blocking` for the capture call, return `{ job_uid, status: "pending" }` immediately. Add `GET /api/archives/:id/capture_jobs/:job_uid` route. | -| `frontend/src/api.js` | Add `pollCaptureJob(archiveId, jobUid)` | -| `frontend/src/components/CaptureDialog.jsx` | After submit: enter polling loop (500 ms interval), show "Running…" state, handle `completed` (close + refresh) and `failed` (show error). | - -**Acceptance criteria:** -- `POST /captures` returns within 50 ms regardless of capture duration. -- UI shows "Running…" and updates when the job finishes. -- Failed captures surface the error message in the dialog. -- A server restart while a job is `running` leaves it stalled as `running`; on next startup, - jobs stuck in `running` should be marked `failed` with a note ("interrupted by server restart"). -- Existing `cargo test` passes. - -**Key risk:** `spawn_blocking` runs on a Tokio thread pool thread. A very long capture -(15-minute YouTube video) will occupy one thread for the duration. For now that is acceptable. -A proper queue (channel + worker task) can replace it later without changing the DB schema or API. - ---- - -### ~~4. Auth foundation — session + role + setup~~ ✅ Done - -**Implemented:** `crates/archivr-server/src/auth.rs` (new; `AuthUser` extractor, Argon2id password -hashing, token generation, role bit constants). `crates/archivr-core/src/database.rs`: -`initialize_auth_schema` (roles, user_roles, sessions, api_tokens, instance_settings tables); -`open_auth_db` (separate server-level auth SQLite); `create_owner`, `compute_role_bits`, -`create_session`, `get_session`, `create_api_token`, `get_user_for_token` and related helpers. -`crates/archivr-server/src/routes.rs`: `AppState` gains `auth_db_path`; `setup_guard` middleware -returns 503 until owner created; `POST /api/auth/login`, `POST /api/auth/logout`, -`GET /api/auth/me`, `GET|POST /api/auth/setup`, `GET|POST|DELETE /api/auth/tokens` endpoints; -WRITE routes (captures, tags) guarded by `ROLE_USER`. `crates/archivr-server/src/main.rs`: -computes auth DB path from config directory; session cleanup background task. -Frontend: `LoginPage.jsx`, `SetupPage.jsx`, `AuthContext` in `App.jsx`, user menu in `Topbar.jsx`. -`docs/specs/2026-06-25-auth-foundation-design.md` has the full design. Roles use a bitmask: -guest=1, user=2, admin=4, owner=8. All tests green. - ---- - -### ~~5. User management~~ ✅ Done - -**Implemented:** `crates/archivr-core/src/database.rs`: `UserSummary` and `RoleRecord` structs; -10 new pub functions (`invalidate_user_sessions`, `get_user_id_by_uid`, `list_users`, -`get_user_by_uid`, `create_user`, `set_user_status`, `assign_role`, `remove_role`, `list_roles`, -`create_custom_role`). `crates/archivr-server/src/routes.rs`: 5 admin routes -(`GET|POST /api/admin/users`, `PATCH /api/admin/users/:uid/status`, -`POST|DELETE /api/admin/users/:uid/roles`, `GET|POST /api/admin/roles`); 7 handler functions; -request body structs. `frontend/src/api.js`: 7 admin helpers (`listAdminUsers`, -`createAdminUser`, `setUserStatus`, `assignRole`, `removeRole`, `listRoles`, `createRole`). -`frontend/src/components/AdminView.jsx`: two-tab admin panel (Users / Roles) with ban/unban, -create user form, create custom role form. Role bitmask: guest=1, user=2, admin=4, owner=8; -custom roles get bit_position≥4. Ban invalidates all sessions. Only-owner guard on role removal. -169 tests green. - ---- - -### ~~6. Permissions & visibility — collection model~~ ✅ Done - -**Implemented:** `database.rs`: `collections` + `collection_entries` tables; seed default collection ('All Entries'); -migration inserts existing entries into default collection with `visibility_bits` derived from legacy `visibility` string -(`'public'`→3, `'unlisted'`→2, `'private'`→0). 8 new pub functions (`ensure_default_collection`, `create_collection`, -`list_collections`, `get_collection_by_uid`, `add_entry_to_collection`, `update_collection_entry_visibility`, -`remove_entry_from_collection`, `get_entry_collection_memberships`). `create_archived_entry` auto-enrolls new entries -into the default collection. `archive.rs`: `list_root_entries` + `search_entries` accept `caller_bits: u32` and enforce -collection visibility (`admin/owner bypass`; others filtered by `visibility_bits & caller_bits`). `list_entries_for_collection`, -`get_entry_collections`, `EntryCollectionMembership` added. `routes.rs`: read routes extract auth and pass caller_bits; -5 collection-entry route groups (`GET|POST /api/archives/:id/collections`, `GET /api/archives/:id/collections/:uid`, -`POST|DELETE|PATCH /api/archives/:id/collections/:uid/entries(/:entry_uid)`, -`GET /api/archives/:id/entries/:uid/collections`). Frontend: `CollectionsView.jsx` (list + detail + create form); -`collections` nav item in Topbar; entry collection memberships + visibility shown in ContextRail. 169 tests green. -Depends on Track 5. - ---- - -### ~~7. Settings~~ ✅ Done - -**Implemented:** `database.rs`: `display_name TEXT` column migration (ALTER TABLE, idempotent); -`InstanceSettings` struct; 6 new pub fns (`get_instance_settings`, `update_instance_settings`, -`update_user_display_name`, `update_user_password`, `get_user_password_hash`, -`get_user_display_name`). `routes.rs`: `PATCH /api/auth/me` (display name update + current-password- -verified password change); `GET|PATCH /api/admin/instance-settings` (ROLE_ADMIN); -`auth_me` now returns `display_name`. `frontend/src/api.js`: 7 new helpers (`updateProfile`, -`changePassword`, `listTokens`, `createToken`, `deleteToken`, `getInstanceSettings`, -`updateInstanceSettings`). `frontend/src/components/SettingsView.jsx`: tabbed settings page -(Profile — display name + password change; API Tokens — create/list/revoke with one-time reveal; -Instance — public index, public content, open registration toggles + default visibility select). -`Topbar.jsx`: settings nav button; user-menu shows display_name ?? username. -`App.jsx`: renders SettingsView for `view === 'settings'`. 176 tests green. Depends on Track 5. - ---- - -### 8. Collections UI - -**What:** Full collection management — create, rename, delete, add/remove entries, -set per-entry visibility within a collection, make collections public. -Depends on Tracks 5–6. - ---- - -### 9. Cloud backup — S3-compatible - -**What:** A command or scheduled operation that syncs the archive store to an S3-compatible -bucket (AWS S3, Cloudflare R2, Backblaze B2). Incremental: only uploads blobs not already -present in the bucket. The DB is also backed up. - -**Design decisions to make before implementing:** -- CLI subcommand (`archivr backup`) vs. scheduled via the server vs. both. -- Per-archive config vs. a global backup profile. -- Whether to back up only the blob store or also the SQLite DB. - -**Recommended shape:** - -```toml -# In the archive's own config or in archivr-server.toml -[backup.s3] -bucket = "my-archivr-backup" -prefix = "personal/" -endpoint = "https://..." # optional, for R2/B2 -region = "us-east-1" -# credentials via AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars -``` - -**Where to edit (when ready):** - -| File | Change | -|---|---| -| `crates/archivr-core/src/backup.rs` | New module. Walk `store/raw/` tree, list objects in bucket, diff, upload new blobs. Also snapshot and upload `archivr.sqlite`. | -| `crates/archivr-cli/src/main.rs` | Add `archivr backup [--archive <path>]` subcommand | -| `Cargo.toml` | Add `aws-sdk-s3` or `object_store` crate dependency | -| `docs/README.md` | Document backup config and command | - -**Acceptance criteria:** -- `archivr backup` uploads all blobs not already in the bucket. -- A second run uploads nothing (idempotent). -- DB snapshot is included. -- Missing or invalid credentials return a clear error before any uploads begin. - ---- - -### 10. Cloud storage archiving (Google Drive, Dropbox, OneDrive) - -Deferred. Each requires per-service OAuth, API clients, and download logic. Implement after -Tracks 1–4 are stable. - -When approached, add a `Source::GoogleDrive`, `Source::Dropbox`, etc. variant in `capture.rs` -and a corresponding downloader module. Consider `rclone` as a shell-out strategy analogous to -`yt-dlp` and `monolith` — it handles auth and download for all three services. - ---- - -## What to Do First - -Tracks 1, 2, 3, 4, 5, and 6 are complete. Track 7 (settings) is the next priority. - -Open the next thread with: - -```text -Read ARCHIVR-MENTAL-MODEL.md and NEXT.md. I want to implement Track 7: settings — -account profile, password change, instance settings UI, API token management. -Create a task-level implementation plan first, then wait for approval. -```