mirror of
https://github.com/thegeneralist01/archivr
synced 2026-07-21 18:55:36 +02:00
feat: archive paywalled articles through Freedium (#30)
* feat(core): add via_freedium to CaptureConfig; route WebPage captures through Freedium mirror When via_freedium is true and the locator is not already a Freedium URL, perform_capture passes https://freedium-mirror.cfd/<original-url> to singlefile::save() so paywalled articles are fetched via the mirror. The original locator is kept for requested_locator and canonical_locator in the DB entry. An empty cookie map is used for the mirror fetch to prevent original-domain credentials from being sent to freedium-mirror.cfd. * feat(server): expose via_freedium in capture API (default on) CaptureBody gains via_freedium: Option<bool>; absent defaults to true. The rearchive handler sets it to false — existing entries should not be silently re-fetched through a mirror. * feat(frontend): add Freedium mirror toggle to capture advanced options - freediumEnabled state defaults to true (on by default) - via_freedium forwarded through submitCapture to the capture API - Toggle rendered last in the advanced panel, matching existing rows - Built frontend static assets included * fix(core): Freedium capture fixes — title, notifications, reader images - extract_html_title: take().read_to_end() up to 256 KiB (single read() can short-read, leaving title at byte ~106 K undiscovered); regression test added - Strip " - Freedium" suffix before storing entry title - is_freedium_fetch keys off actual fetch_url host - Remove Freedium toast overlay ([data-sonner-toaster]) before capture - Resolve lazy images (data-zoom-src etc.) before Readability in reader mode * fix(core): extract HTML title after font stripping, not before SingleFile embeds fonts as base64 data URIs in <style> blocks in the <head>, pushing the <title> tag to ~1.2 MB in the raw temp file. The 256 KiB read window in extract_html_title missed it. Font extraction rewrites the HTML in-place (2.4 MB → 1.26 MB) before hashing, so the title is accessible at byte ~106 KB in that content. Fix: add extract_html_title_str() that operates on a &str; call it on the in-memory rewritten string after font extraction (server path). CLI path (no font extraction) falls back to result.title as before. * fix(core): inline reader-mode images via Rust post-processing SingleFile cannot inline resources added to the DOM at before-capture time — only resources tracked during the page's initial load cycle get embedded. Article images in Readability output fall into this gap when the page framework has already resolved their lazy src to a CDN URL that isn't in SingleFile's resource cache for the new DOM elements. Fix in three parts: - Browser script: after body.innerHTML = article.content, stamp data-archivr-src=<absolute-proxy-url> on any image whose src is not an already-inlined large data URI. Remove loading attr. Don't touch src (let SingleFile try; Rust handles the rest). - Rust (save_with): after SingleFile writes the file and before hashing, call inline_archivr_img_srcs() to scan for data-archivr-src markers. - inline_archivr_img_srcs(): fetches each marked URL with blocking reqwest (10 s timeout, 5 redirects, image/* Content-Type guard, 20 MiB cap), base64-encodes, replaces src with the data URI, and removes the marker attr. Non-fatal — fetch failures are logged. Also: Freedium UI cleanup now removes footer, #progress, and empty data-nosnippet wrappers in addition to the existing nav/toaster removal. * fix(core): clean up Freedium chrome and drop reader-mode debug counters Two fixes: 1. freedium_cleanup: remove remaining Freedium article chrome that survived the existing nav/footer/toaster pass: - <header class="p-6 bg-gray-50 ..."> — author/metadata bar with profile pictures and byline, a Freedium wrapper around the article - <section> containing [data-slot="dropdown-menu-trigger"] or [aria-haspopup="menu"] — the "Download article" dropdown Selectors target stable Tailwind utility class prefixes (p-6, bg-gray-50, bg-zinc-800) for the header and the WAI-ARIA menu role for the button, both resilient to minor Freedium UI updates. 2. Reader-mode meta tag: strip per-capture debug counters. _archivrReaderMark now writes 'applied' instead of 'applied:pre_s=N,...:post_s=N,...' — the counters were useful during development but have no place as permanent archive content. * fix(core): prevent lazy-resolver double-processing on Freedium reader images _archivrResolveLazyImgs is called twice: before Readability (_pre) and after the post-body stamp pass (_post). The stamp pass sets data-archivr-src but previously left data-src/data-zoom-src/etc intact, so _post's 'lazySrc && _isPlaceholder' condition fired on the same images and rewrote src to the CDN URL — which SingleFile then tried and failed to fetch from the Freedium proxy context, redundantly. Fix: strip all lazy attrs (data-src, data-lazy-src, data-zoom-src, data-original, data-lazy) when stamping data-archivr-src. _post then finds no lazySrc on those images and skips them cleanly. Rust owns them via data-archivr-src. _post continues to handle any remaining placeholder images not covered by the stamp pass (non-Freedium reader captures). * fix(core): address Codex review findings in reader image post-processor P1 — Enforce size cap before buffering (singlefile.rs fetch_image_as_data_uri): resp.bytes() buffered the entire response before checking MAX_BYTES, enabling OOM on oversized or attacker-controlled images. Fix: reject via Content-Length header when present, then stream at most max_bytes+1 bytes with Read::take so the cap is enforced without materialising the full body first. P2 — Decode HTML entities in extracted image URLs (inline_archivr_img_srcs): The browser's HTML serialiser encodes & as & in attribute values, so CDN signed URLs with query parameters (e.g. ?a=1&b=2) arrived as &-escaped strings. reqwest sent the wrong URL, breaking signed or transformed CDN images. Fix: unescape & < > " before passing to the fetcher. P2 — Preserve authentication for same-origin lazy images (inline_archivr_img_srcs): The post-processor created a bare reqwest client with no cookies, causing 401/403 for reader-mode captures of auth-gated sites whose lazy images are same-origin. Fix: pass capture_url and cookies into inline_archivr_img_srcs; attach a Cookie header only when domain_from_url(img_url) == domain_from_url(capture_url) and cookies is non-empty. Third-party image hosts and Freedium fetches (which receive empty cookies in capture.rs) are unaffected. * test(core): extract helpers and add unit tests for Codex review fixes Extract two testable pure functions from inline_archivr_img_srcs: - html_attr_decode: decodes HTML character references in attribute values. Fixes decode order: & runs LAST so &lt; → < (one layer removed), not < (two layers). Previous order caused double-decoding. - same_origin_cookie_header: returns a Cookie header value only when img_url's domain matches capture_url's domain. Add 11 unit tests covering: - html_attr_decode: plain URL no-op, & in CDN query params, single-layer decode (&lt; → < not <), double-encoded amp (&amp; → &), direct </>/" decode - same_origin_cookie_header: same host attaches cookies, third-party returns None, empty cookies returns None, Freedium empty-cookies no-op - bounded_read: Read::take stops at max+1 bytes (guard fires), allows exactly-at-limit payloads (guard does not fire) * docs: refresh AGENTS.md and mental model; drop NEXT.md AGENTS.md: mention Freedium mirror in the overview and capture flow, add vendor/readability/ to Key Directories, drop the NEXT.md pointer. ARCHIVR-MENTAL-MODEL.md: fix stale references to legacy static/ paths in Where To Edit (frontend now lives in frontend/src/); add capture.rs and auth.rs rows; replace the outdated 'Current Limitations' section (which claimed no capture and no auth) with a Server Capabilities section reflecting async capture jobs, the auth model, search, and admin scope; add a Web Capture Pipeline section covering the Freedium mirror, SingleFile+Chromium, vendored Readability, cleanup, and Rust post-processing. NEXT.md: remove; the roadmap tracking is stale and unused.
This commit is contained in:
GitHub
commit
64f94c6eda
11 changed files with 542 additions and 295 deletions
|
|
@ -2,9 +2,9 @@
|
|||
|
||||
## Project Overview
|
||||
|
||||
Archivr is a self-hosted archival tool that captures and preserves digital content — YouTube/Twitter/Instagram/TikTok/Reddit posts, arbitrary URLs, full web pages (via SingleFile + Chromium), and local files — into self-contained, SQLite-backed archive directories with blob deduplication, hierarchical tags, collections, and role-based auth. Rust workspace + React frontend.
|
||||
Archivr is a self-hosted archival tool that captures and preserves digital content — YouTube/Twitter/Instagram/TikTok/Reddit posts, arbitrary URLs, full web pages (via SingleFile + Chromium, optionally through a Freedium mirror for paywalled articles), and local files — into self-contained, SQLite-backed archive directories with blob deduplication, hierarchical tags, collections, and role-based auth. Rust workspace + React frontend.
|
||||
|
||||
Read `ARCHIVR-MENTAL-MODEL.md` before making structural changes; `NEXT.md` tracks the roadmap (Tracks 1–7 done, Track 8 = Collections UI is next).
|
||||
Read `ARCHIVR-MENTAL-MODEL.md` before making structural changes.
|
||||
|
||||
## Architecture & Data Flow
|
||||
|
||||
|
|
@ -14,7 +14,7 @@ Three crates with a strict ownership split — **core owns truth; CLI and server
|
|||
- `crates/archivr-server` — Axum HTTP API + auth + static frontend serving.
|
||||
- `crates/archivr-cli` — clap-based CLI (`archivr` binary): `init`, `archive` subcommands.
|
||||
|
||||
Capture flow: locator → `determine_source()` (`crates/archivr-core/src/capture.rs`) routes by platform/shorthand (`yt:`, `x:`, `tweet:` …) → platform downloader (`downloader/ytdlp.rs`, `tweets.rs`, `singlefile.rs`, `http.rs`, `local.rs`) stages into `temp/` → SHA3-256 dedup (`hash.rs`, `downloader/store.rs`) moves blobs to `raw/A/B/HASH.EXT` → rows written to `archivr.sqlite` (runs, entries, artifacts, blobs) → served via `/api/archives/:id/...`.
|
||||
Capture flow: locator → `determine_source()` (`crates/archivr-core/src/capture.rs`) routes by platform/shorthand (`yt:`, `x:`, `tweet:` …) → platform downloader (`downloader/ytdlp.rs`, `tweets.rs`, `singlefile.rs`, `http.rs`, `local.rs`) stages into `temp/` → SHA3-256 dedup (`hash.rs`, `downloader/store.rs`) moves blobs to `raw/A/B/HASH.EXT` → rows written to `archivr.sqlite` (runs, entries, artifacts, blobs) → served via `/api/archives/:id/...`. `CaptureConfig` carries per-request toggles (uBlock, reader mode, Freedium mirror, etc.); when `via_freedium` is set, the fetch URL is rewritten through `freedium-mirror.cfd` while the canonical DB URL stays the original locator.
|
||||
|
||||
Per-archive layout (created by `archivr init`): `.archivr/` (name, store_path, `archivr.sqlite`) + sibling `store/` (`raw/`, `raw_tweets/`, `structured/`, `temp/`). Server-level auth lives in a **separate** `archivr-auth.sqlite` (users, sessions, API tokens, role bits GUEST=1/USER=2/ADMIN=4/OWNER=8).
|
||||
|
||||
|
|
@ -31,6 +31,7 @@ The server mounts multiple archives from a TOML registry (`crates/archivr-server
|
|||
| `docs/` | User docs (`README.md`), `superpowers/plans/` and `superpowers/specs/` (dated design docs — write plans there before large features) |
|
||||
| `modules/nixos/` | NixOS module (`services.archivr-server`) |
|
||||
| `vendor/twitter/` | Vendored Twitter scraper (active; the Python the server shells out to). Don't refactor casually. |
|
||||
| `vendor/readability/` | Mozilla `Readability.js`, concatenated into the SingleFile reader-mode browser script by `downloader/singlefile.rs`. |
|
||||
| `testing/` | Legacy scraping scripts + sample data. `testing/creds.txt` holds real tokens — never read, commit, or print it. |
|
||||
|
||||
## Development Commands
|
||||
|
|
|
|||
|
|
@ -117,6 +117,16 @@ sequenceDiagram
|
|||
CLI->>User: terminal result
|
||||
```
|
||||
|
||||
## Web Capture Pipeline
|
||||
|
||||
Web pages (`Source::WebPage`) take a longer path than yt-dlp or tweets:
|
||||
|
||||
1. **Fetch URL selection.** If `via_freedium` is set and the locator isn't already a Freedium URL, the downloader fetches the page through `freedium-mirror.cfd` with no forwarded cookies. The canonical DB URL stays the original locator.
|
||||
2. **Browser capture.** `downloader/singlefile.rs` shells out to `single-file-cli` driving headless Chromium. Extensions (uBlock, cookie-consent) and injected browser scripts (modal closer, reader mode) attach per `CaptureConfig`.
|
||||
3. **Reader mode** (optional) concatenates Mozilla's `Readability.js` from `vendor/readability/` into the SingleFile browser script and stamps absolute URLs on lazy images so the serialised DOM points to fetchable sources.
|
||||
4. **Freedium cleanup** strips mirror UI (nav, footer, toaster, author header, download control) using multi-signal selectors so article-authored controls aren't hit.
|
||||
5. **Rust post-processing.** After SingleFile writes the HTML, a Rust pass fetches any images the browser couldn't inline (bounded reads, same-origin cookie forwarding) and embeds them as data URIs. Title extraction runs after embedded font blocks are stripped so large fonts don't push `<title>` beyond the read window.
|
||||
|
||||
## Read Data Flow
|
||||
|
||||
When opening the web UI:
|
||||
|
|
@ -142,14 +152,17 @@ sequenceDiagram
|
|||
| Feature kind | Edit here |
|
||||
|---|---|
|
||||
| DB schema, inserts, archive runs, entries, tags | `crates/archivr-core/src/database.rs` |
|
||||
| Capture orchestration, `Source` routing, `CaptureConfig` | `crates/archivr-core/src/capture.rs` |
|
||||
| Archive opening, listing entries, entry detail, runs | `crates/archivr-core/src/archive.rs` |
|
||||
| Download/save behavior | `crates/archivr-core/src/downloader/` |
|
||||
| CLI commands, argument parsing, terminal output | `crates/archivr-cli/src/main.rs` |
|
||||
| Server API routes | `crates/archivr-server/src/routes.rs` |
|
||||
| Auth model (users, sessions, tokens, roles) | `crates/archivr-server/src/auth.rs` |
|
||||
| Mounted archive config model | `crates/archivr-server/src/registry.rs` |
|
||||
| Browser UI behavior | `crates/archivr-server/static/app.js` |
|
||||
| Browser UI layout | `crates/archivr-server/static/index.html` |
|
||||
| Browser UI styling | `crates/archivr-server/static/styles.css` |
|
||||
| Frontend root state + routing | `frontend/src/App.jsx` |
|
||||
| Frontend API client | `frontend/src/api.js` |
|
||||
| Frontend components | `frontend/src/components/` |
|
||||
| Frontend styling | `frontend/src/styles.css` |
|
||||
|
||||
## Practical Feature Rule
|
||||
|
||||
|
|
@ -165,12 +178,13 @@ If a browser feature needs new data, the usual order is:
|
|||
2. Expose it in `archivr-server`.
|
||||
3. Render it in the static UI.
|
||||
|
||||
## Current Limitations
|
||||
## Server Capabilities
|
||||
|
||||
The web server reads archive data and serves the UI. It does not yet implement capture.
|
||||
The server both reads and writes archive data. Capture jobs are asynchronous: `POST /api/archives/:id/captures` inserts a job row, spawns a blocking task, and returns immediately; the frontend polls until the job completes or fails. Heavy work stays synchronous inside `archivr-core`.
|
||||
|
||||
Search is currently simple client-side filtering.
|
||||
**Auth model.** A separate `archivr-auth.sqlite` (path derived from the server config directory) holds users, sessions, and API tokens. Role bits are `u32` flags (`GUEST`, `USER`, `ADMIN`, `OWNER`) so a single bitmask value covers assignment, checks, and visibility. The middleware stack is `setup_guard` → `login_rate_limit` → `security_headers`; route families are classified `READ / ADMIN / WRITE / STATIC` in `routes.rs`.
|
||||
|
||||
**Auth and session model:** The server binds to `127.0.0.1` by default and has no authentication middleware. This is intentional — Archivr is a local tool. The bind address is configurable via the TOML `bind` field or `ARCHIVR_BIND` env var; a non-loopback address triggers a startup warning. Route families are classified (READ / ADMIN / WRITE / STATIC) in `crates/archivr-server/src/routes.rs` as the decision record for when middleware is eventually added. See the "Security and Deployment" section in `docs/README.md`.
|
||||
**Search** is client-side filtering over entries the frontend has already fetched.
|
||||
|
||||
**Admin view** covers mounted archives, users, sessions, and API tokens.
|
||||
|
||||
Admin is a mounted-archives view, not a management system.
|
||||
|
|
|
|||
225
NEXT.md
225
NEXT.md
|
|
@ -1,225 +0,0 @@
|
|||
# Archivr Next Work Plan
|
||||
|
||||
> **For future agentic workers:** Read `ARCHIVR-MENTAL-MODEL.md` first, then this file.
|
||||
> All six tracks from the previous handoff are complete. This document is the current roadmap.
|
||||
|
||||
## Current State
|
||||
|
||||
All five product tracks from the prior session are done:
|
||||
|
||||
| Feature | Status |
|
||||
|---|---|
|
||||
| Entry detail + artifact serving | Done |
|
||||
| Server-side search with prefix filters | Done |
|
||||
| Hierarchical tags (CRUD, tree, entry assignment, filter) | Done |
|
||||
| Browser capture button and POST endpoint | Done |
|
||||
| Local-only auth boundary, bind address config, docs | Done |
|
||||
|
||||
The frontend is a React + Vite app (`frontend/`) that builds into `crates/archivr-server/static/`.
|
||||
|
||||
The remaining gaps come from `docs/README.md` milestones that were never in the prior tracks.
|
||||
|
||||
---
|
||||
|
||||
## Remaining Work (in order)
|
||||
|
||||
### ~~1. Generic URL capture — plain file download~~ ✅ Done
|
||||
|
||||
**Implemented:** `crates/archivr-core/src/downloader/http.rs` (`download` fn, extension helpers,
|
||||
HTML rejection), `Source::Url` variant in `capture.rs`, `determine_source` routes all unmatched
|
||||
`http://`/`https://` URLs to `Source::Url`, `perform_capture` arm calls the downloader and flows
|
||||
through the existing temp → hash → raw store pipeline. `reqwest 0.12` (blocking) added to workspace.
|
||||
`docs/README.md` "URLs" milestone checked off. 112 tests green.
|
||||
|
||||
---
|
||||
|
||||
### ~~2. Web page archiving — single-file HTML snapshots~~ ✅ Done
|
||||
|
||||
**Implemented:** `crates/archivr-core/src/downloader/singlefile.rs` (new module; shells out to
|
||||
`single-file-cli` via `ARCHIVR_SINGLE_FILE` env var; launches headless Chromium via
|
||||
`ARCHIVR_CHROME`; flags: `--browser-wait-until=networkidle2`, `--remove-unused-styles=false`,
|
||||
`--block-scripts=false`, `--remove-alternative-medias=false` for full CSS/font fidelity).
|
||||
`Source::WebPage` variant added to `capture.rs`; HEAD-first probe (`probe_url_kind` in `http.rs`)
|
||||
routes `text/html` responses to `Source::WebPage` and everything else to `Source::Url`.
|
||||
Font deduplication implemented as a follow-on: `font_extractor.rs` post-processes the saved HTML,
|
||||
extracts embedded `@font-face` base64 blobs, stores each as a deduplicated artifact in the raw
|
||||
store, and rewrites the HTML to reference them via `/api/archives/:id/blobs/:sha256`.
|
||||
`hash_bytes` added to `hash.rs`; `get_blob_by_sha256` added to `database.rs`;
|
||||
`GET /api/archives/:id/blobs/:sha256` route added to `routes.rs`.
|
||||
`flake.nix` updated: `pkgs.single-file-cli` and (Linux-only) `pkgs.chromium` added to
|
||||
`buildInputs`; `ARCHIVR_SINGLE_FILE` and `ARCHIVR_CHROME` env vars wired into both wrappers.
|
||||
`docs/README.md` "Archive web pages" milestone checked off. 143 tests green.
|
||||
|
||||
---
|
||||
|
||||
### 3. Async capture jobs
|
||||
|
||||
**What:** Capture currently runs synchronously on the HTTP request thread. A YouTube video,
|
||||
a large tweet thread, or a slow monolith page will stall the browser request. This track
|
||||
adds a job queue so `POST /api/archives/:id/captures` returns immediately and the UI polls
|
||||
for completion.
|
||||
|
||||
**Where to edit:**
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `crates/archivr-core/src/database.rs` | Add `capture_jobs` table to the schema: `(id, job_uid, run_uid, archive_id, status TEXT CHECK(status IN ('pending','running','completed','failed')), error_text, created_at, updated_at)`. Add `create_capture_job`, `update_capture_job_status`, `get_capture_job` helpers. |
|
||||
| `crates/archivr-core/src/archive.rs` | Add `CaptureJob` summary type and `get_capture_job` query. |
|
||||
| `crates/archivr-server/src/routes.rs` | `POST /api/archives/:id/captures` → insert job row, spawn `tokio::task::spawn_blocking` for the capture call, return `{ job_uid, status: "pending" }` immediately. Add `GET /api/archives/:id/capture_jobs/:job_uid` route. |
|
||||
| `frontend/src/api.js` | Add `pollCaptureJob(archiveId, jobUid)` |
|
||||
| `frontend/src/components/CaptureDialog.jsx` | After submit: enter polling loop (500 ms interval), show "Running…" state, handle `completed` (close + refresh) and `failed` (show error). |
|
||||
|
||||
**Acceptance criteria:**
|
||||
- `POST /captures` returns within 50 ms regardless of capture duration.
|
||||
- UI shows "Running…" and updates when the job finishes.
|
||||
- Failed captures surface the error message in the dialog.
|
||||
- A server restart while a job is `running` leaves it stalled as `running`; on next startup,
|
||||
jobs stuck in `running` should be marked `failed` with a note ("interrupted by server restart").
|
||||
- Existing `cargo test` passes.
|
||||
|
||||
**Key risk:** `spawn_blocking` runs on a Tokio thread pool thread. A very long capture
|
||||
(15-minute YouTube video) will occupy one thread for the duration. For now that is acceptable.
|
||||
A proper queue (channel + worker task) can replace it later without changing the DB schema or API.
|
||||
|
||||
---
|
||||
|
||||
### ~~4. Auth foundation — session + role + setup~~ ✅ Done
|
||||
|
||||
**Implemented:** `crates/archivr-server/src/auth.rs` (new; `AuthUser` extractor, Argon2id password
|
||||
hashing, token generation, role bit constants). `crates/archivr-core/src/database.rs`:
|
||||
`initialize_auth_schema` (roles, user_roles, sessions, api_tokens, instance_settings tables);
|
||||
`open_auth_db` (separate server-level auth SQLite); `create_owner`, `compute_role_bits`,
|
||||
`create_session`, `get_session`, `create_api_token`, `get_user_for_token` and related helpers.
|
||||
`crates/archivr-server/src/routes.rs`: `AppState` gains `auth_db_path`; `setup_guard` middleware
|
||||
returns 503 until owner created; `POST /api/auth/login`, `POST /api/auth/logout`,
|
||||
`GET /api/auth/me`, `GET|POST /api/auth/setup`, `GET|POST|DELETE /api/auth/tokens` endpoints;
|
||||
WRITE routes (captures, tags) guarded by `ROLE_USER`. `crates/archivr-server/src/main.rs`:
|
||||
computes auth DB path from config directory; session cleanup background task.
|
||||
Frontend: `LoginPage.jsx`, `SetupPage.jsx`, `AuthContext` in `App.jsx`, user menu in `Topbar.jsx`.
|
||||
`docs/specs/2026-06-25-auth-foundation-design.md` has the full design. Roles use a bitmask:
|
||||
guest=1, user=2, admin=4, owner=8. All tests green.
|
||||
|
||||
---
|
||||
|
||||
### ~~5. User management~~ ✅ Done
|
||||
|
||||
**Implemented:** `crates/archivr-core/src/database.rs`: `UserSummary` and `RoleRecord` structs;
|
||||
10 new pub functions (`invalidate_user_sessions`, `get_user_id_by_uid`, `list_users`,
|
||||
`get_user_by_uid`, `create_user`, `set_user_status`, `assign_role`, `remove_role`, `list_roles`,
|
||||
`create_custom_role`). `crates/archivr-server/src/routes.rs`: 5 admin routes
|
||||
(`GET|POST /api/admin/users`, `PATCH /api/admin/users/:uid/status`,
|
||||
`POST|DELETE /api/admin/users/:uid/roles`, `GET|POST /api/admin/roles`); 7 handler functions;
|
||||
request body structs. `frontend/src/api.js`: 7 admin helpers (`listAdminUsers`,
|
||||
`createAdminUser`, `setUserStatus`, `assignRole`, `removeRole`, `listRoles`, `createRole`).
|
||||
`frontend/src/components/AdminView.jsx`: two-tab admin panel (Users / Roles) with ban/unban,
|
||||
create user form, create custom role form. Role bitmask: guest=1, user=2, admin=4, owner=8;
|
||||
custom roles get bit_position≥4. Ban invalidates all sessions. Only-owner guard on role removal.
|
||||
169 tests green.
|
||||
|
||||
---
|
||||
|
||||
### ~~6. Permissions & visibility — collection model~~ ✅ Done
|
||||
|
||||
**Implemented:** `database.rs`: `collections` + `collection_entries` tables; seed default collection ('All Entries');
|
||||
migration inserts existing entries into default collection with `visibility_bits` derived from legacy `visibility` string
|
||||
(`'public'`→3, `'unlisted'`→2, `'private'`→0). 8 new pub functions (`ensure_default_collection`, `create_collection`,
|
||||
`list_collections`, `get_collection_by_uid`, `add_entry_to_collection`, `update_collection_entry_visibility`,
|
||||
`remove_entry_from_collection`, `get_entry_collection_memberships`). `create_archived_entry` auto-enrolls new entries
|
||||
into the default collection. `archive.rs`: `list_root_entries` + `search_entries` accept `caller_bits: u32` and enforce
|
||||
collection visibility (`admin/owner bypass`; others filtered by `visibility_bits & caller_bits`). `list_entries_for_collection`,
|
||||
`get_entry_collections`, `EntryCollectionMembership` added. `routes.rs`: read routes extract auth and pass caller_bits;
|
||||
5 collection-entry route groups (`GET|POST /api/archives/:id/collections`, `GET /api/archives/:id/collections/:uid`,
|
||||
`POST|DELETE|PATCH /api/archives/:id/collections/:uid/entries(/:entry_uid)`,
|
||||
`GET /api/archives/:id/entries/:uid/collections`). Frontend: `CollectionsView.jsx` (list + detail + create form);
|
||||
`collections` nav item in Topbar; entry collection memberships + visibility shown in ContextRail. 169 tests green.
|
||||
Depends on Track 5.
|
||||
|
||||
---
|
||||
|
||||
### ~~7. Settings~~ ✅ Done
|
||||
|
||||
**Implemented:** `database.rs`: `display_name TEXT` column migration (ALTER TABLE, idempotent);
|
||||
`InstanceSettings` struct; 6 new pub fns (`get_instance_settings`, `update_instance_settings`,
|
||||
`update_user_display_name`, `update_user_password`, `get_user_password_hash`,
|
||||
`get_user_display_name`). `routes.rs`: `PATCH /api/auth/me` (display name update + current-password-
|
||||
verified password change); `GET|PATCH /api/admin/instance-settings` (ROLE_ADMIN);
|
||||
`auth_me` now returns `display_name`. `frontend/src/api.js`: 7 new helpers (`updateProfile`,
|
||||
`changePassword`, `listTokens`, `createToken`, `deleteToken`, `getInstanceSettings`,
|
||||
`updateInstanceSettings`). `frontend/src/components/SettingsView.jsx`: tabbed settings page
|
||||
(Profile — display name + password change; API Tokens — create/list/revoke with one-time reveal;
|
||||
Instance — public index, public content, open registration toggles + default visibility select).
|
||||
`Topbar.jsx`: settings nav button; user-menu shows display_name ?? username.
|
||||
`App.jsx`: renders SettingsView for `view === 'settings'`. 176 tests green. Depends on Track 5.
|
||||
|
||||
---
|
||||
|
||||
### 8. Collections UI
|
||||
|
||||
**What:** Full collection management — create, rename, delete, add/remove entries,
|
||||
set per-entry visibility within a collection, make collections public.
|
||||
Depends on Tracks 5–6.
|
||||
|
||||
---
|
||||
|
||||
### 9. Cloud backup — S3-compatible
|
||||
|
||||
**What:** A command or scheduled operation that syncs the archive store to an S3-compatible
|
||||
bucket (AWS S3, Cloudflare R2, Backblaze B2). Incremental: only uploads blobs not already
|
||||
present in the bucket. The DB is also backed up.
|
||||
|
||||
**Design decisions to make before implementing:**
|
||||
- CLI subcommand (`archivr backup`) vs. scheduled via the server vs. both.
|
||||
- Per-archive config vs. a global backup profile.
|
||||
- Whether to back up only the blob store or also the SQLite DB.
|
||||
|
||||
**Recommended shape:**
|
||||
|
||||
```toml
|
||||
# In the archive's own config or in archivr-server.toml
|
||||
[backup.s3]
|
||||
bucket = "my-archivr-backup"
|
||||
prefix = "personal/"
|
||||
endpoint = "https://..." # optional, for R2/B2
|
||||
region = "us-east-1"
|
||||
# credentials via AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars
|
||||
```
|
||||
|
||||
**Where to edit (when ready):**
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `crates/archivr-core/src/backup.rs` | New module. Walk `store/raw/` tree, list objects in bucket, diff, upload new blobs. Also snapshot and upload `archivr.sqlite`. |
|
||||
| `crates/archivr-cli/src/main.rs` | Add `archivr backup [--archive <path>]` subcommand |
|
||||
| `Cargo.toml` | Add `aws-sdk-s3` or `object_store` crate dependency |
|
||||
| `docs/README.md` | Document backup config and command |
|
||||
|
||||
**Acceptance criteria:**
|
||||
- `archivr backup` uploads all blobs not already in the bucket.
|
||||
- A second run uploads nothing (idempotent).
|
||||
- DB snapshot is included.
|
||||
- Missing or invalid credentials return a clear error before any uploads begin.
|
||||
|
||||
---
|
||||
|
||||
### 10. Cloud storage archiving (Google Drive, Dropbox, OneDrive)
|
||||
|
||||
Deferred. Each requires per-service OAuth, API clients, and download logic. Implement after
|
||||
Tracks 1–4 are stable.
|
||||
|
||||
When approached, add a `Source::GoogleDrive`, `Source::Dropbox`, etc. variant in `capture.rs`
|
||||
and a corresponding downloader module. Consider `rclone` as a shell-out strategy analogous to
|
||||
`yt-dlp` and `monolith` — it handles auth and download for all three services.
|
||||
|
||||
---
|
||||
|
||||
## What to Do First
|
||||
|
||||
Tracks 1, 2, 3, 4, 5, and 6 are complete. Track 7 (settings) is the next priority.
|
||||
|
||||
Open the next thread with:
|
||||
|
||||
```text
|
||||
Read ARCHIVR-MENTAL-MODEL.md and NEXT.md. I want to implement Track 7: settings —
|
||||
account profile, password change, instance settings UI, API token management.
|
||||
Create a task-level implementation plan first, then wait for approval.
|
||||
```
|
||||
|
|
@ -87,6 +87,9 @@ pub struct CaptureConfig {
|
|||
pub reader_mode: bool,
|
||||
/// Override for modal-closer browser-script behavior during WebPage captures.
|
||||
pub modal_closer_enabled: Option<bool>,
|
||||
/// Route WebPage captures through the Freedium mirror to bypass paywalls.
|
||||
/// The original locator is still recorded in the DB; only the fetch URL changes.
|
||||
pub via_freedium: bool,
|
||||
}
|
||||
|
||||
/// Resolves which cookies apply to `url` by evaluating all rules in ordinal order.
|
||||
|
|
@ -1038,7 +1041,20 @@ pub fn perform_capture(
|
|||
|
||||
// Source: web page — archive as a self-contained HTML snapshot via single-file-cli
|
||||
if source == Source::WebPage {
|
||||
match downloader::singlefile::save(locator, store_path, ×tamp, &cookies, config.ublock_enabled, config.cookie_ext_enabled, config.reader_mode, config.modal_closer_enabled) {
|
||||
// When via_freedium is enabled and the URL is not already a freedium mirror,
|
||||
// fetch through the mirror to bypass paywalls. Store the original locator in DB.
|
||||
// Use an empty cookie jar for the mirror URL: cookies resolved for the original
|
||||
// domain (e.g. NYT, Medium) must not be sent to freedium-mirror.cfd.
|
||||
let (fetch_url, fetch_cookies): (String, HashMap<String, String>) =
|
||||
if config.via_freedium && !locator.starts_with("https://freedium-mirror.cfd/") {
|
||||
(format!("https://freedium-mirror.cfd/{}", locator), HashMap::new())
|
||||
} else {
|
||||
(locator.to_string(), cookies.clone())
|
||||
};
|
||||
// Key cleanup/title-stripping off the actual fetch host so that
|
||||
// user-supplied freedium-mirror.cfd URLs are also handled correctly.
|
||||
let is_freedium_fetch = fetch_url.starts_with("https://freedium-mirror.cfd/");
|
||||
match downloader::singlefile::save(&fetch_url, store_path, ×tamp, &fetch_cookies, config.ublock_enabled, config.cookie_ext_enabled, config.reader_mode, config.modal_closer_enabled, is_freedium_fetch) {
|
||||
Ok(result) => {
|
||||
let file_extension = ".html".to_string();
|
||||
let temp_html = store_path
|
||||
|
|
@ -1049,23 +1065,26 @@ pub fn perform_capture(
|
|||
// Font extraction: rewrite the HTML in-place before hashing.
|
||||
// Only runs when archive_id is known (server context). CLI passes
|
||||
// None and keeps fonts embedded — no behaviour change for CLI.
|
||||
let (html_hash, byte_size, extracted_fonts) =
|
||||
let (html_hash, byte_size, extracted_fonts, html_title) =
|
||||
if let Some(aid) = archive_id {
|
||||
let content = fs::read_to_string(&temp_html)
|
||||
.with_context(|| format!("failed to read {}", temp_html.display()))?;
|
||||
let (rewritten, fonts) =
|
||||
downloader::font_extractor::extract_and_rewrite(&content, store_path, aid)
|
||||
.unwrap_or_else(|_| (content.clone(), vec![])); // non-fatal
|
||||
// Extract title after font-stripping so the title tag is not buried
|
||||
// behind multi-MB embedded font data that would exceed the 256 KiB window.
|
||||
let title = downloader::singlefile::extract_html_title_str(&rewritten);
|
||||
fs::write(&temp_html, rewritten.as_bytes())
|
||||
.with_context(|| "failed to write rewritten HTML")?;
|
||||
let size = rewritten.len() as i64;
|
||||
let new_hash = crate::hash::hash_bytes(rewritten.as_bytes());
|
||||
(new_hash, size, fonts)
|
||||
(new_hash, size, fonts, title)
|
||||
} else {
|
||||
let size = fs::metadata(&temp_html)
|
||||
.with_context(|| format!("failed to stat {}", temp_html.display()))?
|
||||
.len() as i64;
|
||||
(result.html_hash.clone(), size, vec![])
|
||||
(result.html_hash.clone(), size, vec![], result.title.clone())
|
||||
};
|
||||
|
||||
// 1. Move HTML to raw store (if this hash hasn't been seen before).
|
||||
|
|
@ -1102,6 +1121,17 @@ pub fn perform_capture(
|
|||
let _ = fs::remove_dir_all(store_path.join("temp").join(×tamp));
|
||||
|
||||
// 4. Create the entry + primary_media artifact.
|
||||
// Strip mirror branding from title when fetched via Freedium.
|
||||
let entry_title = if is_freedium_fetch {
|
||||
html_title.as_deref().map(|t| {
|
||||
t.trim_end_matches(" - Freedium")
|
||||
.trim_end_matches(" \u{2014} Freedium")
|
||||
.trim()
|
||||
.to_string()
|
||||
})
|
||||
} else {
|
||||
html_title
|
||||
};
|
||||
let entry = record_media_entry(
|
||||
&conn,
|
||||
store_path,
|
||||
|
|
@ -1114,7 +1144,7 @@ pub fn perform_capture(
|
|||
&html_hash,
|
||||
&file_extension,
|
||||
byte_size,
|
||||
result.title,
|
||||
entry_title,
|
||||
)?;
|
||||
|
||||
// 5. Add favicon artifact if we captured one.
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
use anyhow::{Context, Result, bail};
|
||||
use regex::Regex;
|
||||
use base64::engine::general_purpose::STANDARD as B64;
|
||||
use base64::Engine as _;
|
||||
use std::{
|
||||
|
|
@ -47,12 +48,63 @@ const READER_MODE_SCRIPT: &str = concat!(
|
|||
_archivrReaderMark('failed:no-readability');
|
||||
return;
|
||||
}
|
||||
// Helper: resolve lazy-loaded images (src="data:," placeholders) to absolute URLs.
|
||||
// Called before the Readability clone so the clone has real src values, and again
|
||||
// after body replacement because Readability serialises src attributes as-is from
|
||||
// the clone — any remaining placeholders in article.content must also be fixed.
|
||||
function _archivrResolveLazyImgs(base) {
|
||||
var seen=0,lazy=0,placeholders=0,fixed=0;
|
||||
document.querySelectorAll('img').forEach(function(img) {
|
||||
seen++;
|
||||
var lazySrc = img.getAttribute('data-src') || img.getAttribute('data-lazy-src') ||
|
||||
img.getAttribute('data-zoom-src') || img.getAttribute('data-original') ||
|
||||
img.getAttribute('data-lazy');
|
||||
if(lazySrc) lazy++;
|
||||
var curSrc = img.getAttribute('src') || '';
|
||||
var _isPlaceholder = !curSrc || curSrc === 'data:,' ||
|
||||
(curSrc.startsWith('data:') && curSrc.length < 512);
|
||||
if(_isPlaceholder) placeholders++;
|
||||
if (lazySrc && _isPlaceholder) {
|
||||
try { lazySrc = new URL(lazySrc, base).href; } catch(e) {}
|
||||
img.setAttribute('src', lazySrc);
|
||||
img.removeAttribute('loading');
|
||||
fixed++;
|
||||
}
|
||||
});
|
||||
return {seen:seen,lazy:lazy,placeholders:placeholders,fixed:fixed};
|
||||
}
|
||||
var _base = document.baseURI || location.href;
|
||||
var _pre = _archivrResolveLazyImgs(_base);
|
||||
var article = new Readability(document.cloneNode(true)).parse();
|
||||
if (!article || !article.content || article.content.length < 100) {
|
||||
_archivrReaderMark('failed:no-article');
|
||||
return;
|
||||
}
|
||||
document.body.innerHTML = article.content;
|
||||
// Post-Readability pass: stamp data-archivr-src on article images so the Rust
|
||||
// post-processor can fetch and inline them after SingleFile writes the file.
|
||||
// SingleFile cannot inline resources introduced at before-capture time; don't
|
||||
// touch src here — Rust replaces it from the marker. Skip already-inlined images.
|
||||
document.querySelectorAll('img').forEach(function(img) {
|
||||
var lazySrc = img.getAttribute('data-src') || img.getAttribute('data-lazy-src') ||
|
||||
img.getAttribute('data-zoom-src') || img.getAttribute('data-original') ||
|
||||
img.getAttribute('data-lazy');
|
||||
if (!lazySrc) return;
|
||||
var curSrc = img.getAttribute('src') || '';
|
||||
if (curSrc.startsWith('data:image/') && curSrc.length > 1000) return;
|
||||
try { lazySrc = new URL(lazySrc, _base).href; } catch(e) {}
|
||||
img.setAttribute('data-archivr-src', lazySrc);
|
||||
img.removeAttribute('loading');
|
||||
// Remove lazy attrs so _archivrResolveLazyImgs (called below) cannot
|
||||
// rewrite src to a CDN URL that SingleFile cannot fetch from the proxy
|
||||
// context — Rust owns these images via data-archivr-src instead.
|
||||
img.removeAttribute('data-src');
|
||||
img.removeAttribute('data-lazy-src');
|
||||
img.removeAttribute('data-zoom-src');
|
||||
img.removeAttribute('data-original');
|
||||
img.removeAttribute('data-lazy');
|
||||
});
|
||||
var _post = _archivrResolveLazyImgs(_base);
|
||||
if (article.title) document.title = article.title;
|
||||
var hdr = document.createElement('header');
|
||||
hdr.innerHTML =
|
||||
|
|
@ -236,6 +288,7 @@ pub fn save(
|
|||
cookie_ext_enabled: Option<bool>,
|
||||
reader_mode: bool,
|
||||
modal_closer_enabled: Option<bool>,
|
||||
freedium_cleanup: bool,
|
||||
) -> Result<SaveResult> {
|
||||
let single_file =
|
||||
env::var("ARCHIVR_SINGLE_FILE").unwrap_or_else(|_| "single-file".to_string());
|
||||
|
|
@ -254,6 +307,7 @@ pub fn save(
|
|||
cookie_ext.as_deref(),
|
||||
reader_mode,
|
||||
modal_closer,
|
||||
freedium_cleanup,
|
||||
)?;
|
||||
result.ublock_skipped = ublock_skipped;
|
||||
result.cookie_ext_skipped = cookie_ext_skipped;
|
||||
|
|
@ -378,6 +432,7 @@ fn save_with(
|
|||
cookie_ext: Option<&Path>,
|
||||
reader_mode: bool,
|
||||
modal_closer: bool,
|
||||
freedium_cleanup: bool,
|
||||
) -> Result<SaveResult> {
|
||||
let temp_dir = store_path.join("temp").join(timestamp);
|
||||
std::fs::create_dir_all(&temp_dir).context("failed to create temp dir")?;
|
||||
|
|
@ -464,6 +519,36 @@ fn save_with(
|
|||
_archivr_mc_run();"
|
||||
);
|
||||
}
|
||||
if freedium_cleanup {
|
||||
strip_scripts.push_str(
|
||||
// Sonner toast overlay (data attribute is stable across Freedium updates).
|
||||
"document.querySelectorAll('[data-sonner-toaster]').forEach(function(el){\
|
||||
var s=el.closest('section')||el;s.remove();\
|
||||
});\
|
||||
document.querySelectorAll('nav#header').forEach(function(el){el.remove();});\
|
||||
document.querySelectorAll('nav').forEach(function(el){\
|
||||
if(el.querySelector('[aria-label=\"Go back\"]'))el.remove();\
|
||||
});\
|
||||
document.querySelectorAll('nav').forEach(function(el){\
|
||||
if(el.querySelector('a[href*=\"freedium-mirror.cfd/\"]'))el.remove();\
|
||||
});\
|
||||
document.querySelectorAll('footer').forEach(function(el){\
|
||||
if(el.querySelector('a[href*=\"freedium-mirror.cfd/\"]')||\
|
||||
el.textContent.toLowerCase().indexOf('freedium')>-1)el.remove();\
|
||||
});\
|
||||
var _pr=document.getElementById('progress');if(_pr)_pr.remove();\
|
||||
document.querySelectorAll('[data-nosnippet]').forEach(function(el){\
|
||||
if(!el.firstElementChild&&!el.textContent.trim())el.remove();\
|
||||
});\
|
||||
document.querySelectorAll('[data-slot=\"dropdown-menu-trigger\"]').forEach(function(btn){\
|
||||
if(!/download/i.test(btn.textContent||''))return;\
|
||||
var sec=btn.closest('section');if(!sec)return;\
|
||||
var prev=sec.previousElementSibling;\
|
||||
if(prev&&prev.tagName==='HEADER')prev.remove();\
|
||||
sec.remove();\
|
||||
});",
|
||||
);
|
||||
}
|
||||
strip_scripts.push_str("});");
|
||||
std::fs::write(&strip_scripts_path, &strip_scripts)
|
||||
.context("failed to write single-file user script")?;
|
||||
|
|
@ -584,6 +669,10 @@ fn save_with(
|
|||
stderr.trim(),
|
||||
);
|
||||
}
|
||||
// Post-process: fetch and inline images that SingleFile couldn't inline from the
|
||||
// page resource cache (resources introduced via DOM manipulation at before-capture
|
||||
// time are not tracked). Browser script stamped data-archivr-src on those images.
|
||||
inline_archivr_img_srcs(&out_file, url, cookies);
|
||||
|
||||
let title = extract_html_title(&out_file);
|
||||
let html_hash = hash_file(&out_file)?;
|
||||
|
|
@ -653,17 +742,32 @@ fn base_single_file_cmd(
|
|||
|
||||
// ── HTML helpers ──────────────────────────────────────────────────────────────
|
||||
|
||||
/// Reads the first 8 KiB of `path` and extracts the content of the first
|
||||
/// Reads up to 256 KiB of `path` and extracts the content of the first
|
||||
/// `<title>…</title>` element. Returns `None` if absent or empty.
|
||||
///
|
||||
/// Uses `to_ascii_lowercase` for case-insensitive tag matching. ASCII-only
|
||||
/// lowercasing is byte-length-preserving, so byte offsets derived from the
|
||||
/// lowercased buffer are valid indices into the original buffer.
|
||||
///
|
||||
/// Uses `take().read_to_end()` rather than a single `read()` call so the
|
||||
/// full 256 KiB is always consumed even when the OS short-reads.
|
||||
fn extract_html_title(path: &Path) -> Option<String> {
|
||||
let mut f = std::fs::File::open(path).ok()?;
|
||||
let mut buf = vec![0u8; 8192];
|
||||
let n = f.read(&mut buf).ok()?;
|
||||
let buf = &buf[..n];
|
||||
let mut buf = Vec::new();
|
||||
f.take(256 * 1024).read_to_end(&mut buf).ok()?;
|
||||
extract_html_title_from_buf(&buf)
|
||||
}
|
||||
|
||||
/// Extracts the `<title>` content from an HTML string.
|
||||
///
|
||||
/// Used in the server capture path where the font-extracted HTML is already
|
||||
/// in memory — avoids a re-read and operates on the smaller post-extraction
|
||||
/// content where the title is guaranteed to be within range.
|
||||
pub fn extract_html_title_str(html: &str) -> Option<String> {
|
||||
extract_html_title_from_buf(html.as_bytes())
|
||||
}
|
||||
|
||||
fn extract_html_title_from_buf(buf: &[u8]) -> Option<String> {
|
||||
let lower = String::from_utf8_lossy(buf).to_ascii_lowercase();
|
||||
let start = lower.find("<title>")? + "<title>".len();
|
||||
let end = lower[start..].find("</title>")? + start;
|
||||
|
|
@ -737,6 +841,177 @@ fn mime_to_favicon_ext(mime: &str) -> Option<&'static str> {
|
|||
}
|
||||
}
|
||||
|
||||
/// Decodes the minimal set of HTML character references that browsers encode
|
||||
/// in attribute values during serialisation. Covers the four characters
|
||||
/// browsers must escape (`&`, `<`, `>`, `"`) — sufficient for URL query
|
||||
/// strings and signed CDN parameters embedded in `data-*` attributes.
|
||||
fn html_attr_decode(s: &str) -> String {
|
||||
// Decode & LAST so a value like `&lt;` becomes `<` (one layer
|
||||
// removed) rather than `<` (two layers). If `&` ran first it would
|
||||
// produce `<` from the remainder, which the subsequent `<` pass
|
||||
// would then incorrectly collapse to `<`.
|
||||
s.replace("<", "<")
|
||||
.replace(">", ">")
|
||||
.replace(""", "\"")
|
||||
.replace("&", "&")
|
||||
}
|
||||
|
||||
/// Returns a `Cookie: …` header value when `img_url` is same-domain as
|
||||
/// `capture_url` and `cookies` is non-empty; `None` otherwise.
|
||||
/// Uses `domain_from_url` for consistency with the SingleFile cookie-file writer.
|
||||
fn same_origin_cookie_header(
|
||||
capture_url: &str,
|
||||
img_url: &str,
|
||||
cookies: &HashMap<String, String>,
|
||||
) -> Option<String> {
|
||||
if cookies.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let cap = domain_from_url(capture_url);
|
||||
let img = domain_from_url(img_url);
|
||||
if cap.is_empty() || img.is_empty() || cap != img {
|
||||
return None;
|
||||
}
|
||||
Some(
|
||||
cookies
|
||||
.iter()
|
||||
.map(|(k, v)| format!("{k}={v}"))
|
||||
.collect::<Vec<_>>()
|
||||
.join("; "),
|
||||
)
|
||||
}
|
||||
|
||||
/// Scans a reader-mode HTML file for `<img data-archivr-src="…">` elements whose
|
||||
/// `src` is not already a large inlined image, fetches those URLs with blocking
|
||||
/// reqwest, and replaces `src` with `data:<mime>;base64,…`. Non-fatal: any failure
|
||||
/// is logged and the image is left as-is. Guardrails: 10 s timeout, 5 redirects,
|
||||
/// `Content-Type: image/*` required, 20 MiB body cap (enforced via `Content-Length`
|
||||
/// and a bounded read). Cookies are forwarded only when the image host matches the
|
||||
/// captured page's domain; Freedium fetches pass empty cookies so this is a no-op.
|
||||
fn inline_archivr_img_srcs(path: &Path, capture_url: &str, cookies: &HashMap<String, String>) {
|
||||
const MAX_BYTES: usize = 20 * 1024 * 1024;
|
||||
|
||||
let html = match std::fs::read_to_string(path) {
|
||||
Ok(h) => h,
|
||||
Err(_) => return,
|
||||
};
|
||||
if !html.contains("data-archivr-src=") {
|
||||
return;
|
||||
}
|
||||
|
||||
let client = match reqwest::blocking::Client::builder()
|
||||
.user_agent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36")
|
||||
.redirect(reqwest::redirect::Policy::limited(5))
|
||||
.timeout(std::time::Duration::from_secs(10))
|
||||
.build()
|
||||
{
|
||||
Ok(c) => c,
|
||||
Err(_) => return,
|
||||
};
|
||||
|
||||
|
||||
let re_img = Regex::new(r"(?si)<img\b[^>]*>").unwrap();
|
||||
let re_archivr = Regex::new(r#"(?i)\bdata-archivr-src="([^"]+)""#).unwrap();
|
||||
let re_src = Regex::new(r#"(?i)\bsrc="([^"]*)""#).unwrap();
|
||||
let re_rm_archivr = Regex::new(r#"(?i)\s*data-archivr-src="[^"]*""#).unwrap();
|
||||
let re_set_src = Regex::new(r#"(?i)\bsrc="[^"]*""#).unwrap();
|
||||
|
||||
// Collect (start, end, replacement) in document order, then apply in reverse
|
||||
// so earlier byte positions are still valid when we reach them.
|
||||
let mut replacements: Vec<(usize, usize, String)> = Vec::new();
|
||||
|
||||
for m in re_img.find_iter(&html) {
|
||||
let tag = m.as_str();
|
||||
let raw_url = match re_archivr.captures(tag).map(|c| c[1].to_string()) {
|
||||
Some(u) => u,
|
||||
None => continue,
|
||||
};
|
||||
// HTML-decode entities the browser's serialiser encodes in attribute values
|
||||
// (e.g. & → & in CDN signed/query URLs).
|
||||
let img_url = html_attr_decode(&raw_url);
|
||||
|
||||
// Already a large inlined image — leave it alone.
|
||||
let cur_src = re_src.captures(tag).map(|c| c[1].to_string()).unwrap_or_default();
|
||||
if cur_src.starts_with("data:image/") && cur_src.len() > 1000 {
|
||||
continue;
|
||||
}
|
||||
|
||||
let cookie_header = same_origin_cookie_header(capture_url, &img_url, cookies);
|
||||
|
||||
let data_uri = match fetch_image_as_data_uri(&client, &img_url, MAX_BYTES, cookie_header) {
|
||||
Ok(u) => u,
|
||||
Err(e) => {
|
||||
eprintln!("warn: reader image inline ({img_url}): {e}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let new_tag = if re_src.is_match(tag) {
|
||||
re_set_src.replace(tag, format!("src=\"{}\"", data_uri)).into_owned()
|
||||
} else {
|
||||
tag.replacen("<img", &format!("<img src=\"{}\"", data_uri), 1)
|
||||
};
|
||||
let new_tag = re_rm_archivr.replace(&new_tag, "").into_owned();
|
||||
replacements.push((m.start(), m.end(), new_tag));
|
||||
}
|
||||
|
||||
if replacements.is_empty() {
|
||||
return;
|
||||
}
|
||||
let mut html = html;
|
||||
for (start, end, new_tag) in replacements.into_iter().rev() {
|
||||
html.replace_range(start..end, &new_tag);
|
||||
}
|
||||
if let Err(e) = std::fs::write(path, html.as_bytes()) {
|
||||
eprintln!("warn: reader image inline write {}: {e}", path.display());
|
||||
}
|
||||
}
|
||||
|
||||
/// Fetches `url` and returns a `data:<mime>;base64,…` URI.
|
||||
/// Requires `Content-Type: image/*`, enforces `max_bytes` cap via `Content-Length`
|
||||
/// rejection and a bounded streaming read. Attaches `cookie_header` when supplied.
|
||||
fn fetch_image_as_data_uri(
|
||||
client: &reqwest::blocking::Client,
|
||||
url: &str,
|
||||
max_bytes: usize,
|
||||
cookie_header: Option<String>,
|
||||
) -> Result<String> {
|
||||
let mut req = client.get(url);
|
||||
if let Some(cookie) = cookie_header {
|
||||
req = req.header(reqwest::header::COOKIE, cookie);
|
||||
}
|
||||
let resp = req.send().context("request failed")?;
|
||||
if !resp.status().is_success() {
|
||||
bail!("HTTP {}", resp.status());
|
||||
}
|
||||
let ct = resp
|
||||
.headers()
|
||||
.get(reqwest::header::CONTENT_TYPE)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.unwrap_or("")
|
||||
.to_string();
|
||||
let mime = ct.split(';').next().unwrap_or("").trim().to_string();
|
||||
if !mime.starts_with("image/") {
|
||||
bail!("non-image Content-Type: {mime}");
|
||||
}
|
||||
// Reject early when the server advertises a body larger than the cap.
|
||||
if let Some(cl) = resp.content_length() {
|
||||
if cl as usize > max_bytes {
|
||||
bail!("Content-Length {} exceeds {} MiB cap", cl, max_bytes / (1024 * 1024));
|
||||
}
|
||||
}
|
||||
// Stream at most max_bytes + 1 bytes so we never buffer an unbounded body.
|
||||
// Reading one byte past the limit lets us distinguish "exactly at cap" from
|
||||
// "over cap" without a separate Content-Length check.
|
||||
let mut buf = Vec::new();
|
||||
resp.take((max_bytes as u64) + 1)
|
||||
.read_to_end(&mut buf)
|
||||
.context("reading body")?;
|
||||
if buf.len() > max_bytes {
|
||||
bail!("image too large: > {} MiB", max_bytes / (1024 * 1024));
|
||||
}
|
||||
Ok(format!("data:{};base64,{}", mime, B64.encode(&buf)))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
|
@ -774,6 +1049,19 @@ mod tests {
|
|||
assert_eq!(extract_html_title(f.path()), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extract_html_title_after_100kb_preamble() {
|
||||
// Freedium / SingleFile pages can embed large CSS blocks before <title>.
|
||||
// Verify take().read_to_end() reads the full 256 KiB window.
|
||||
let mut f = NamedTempFile::new().unwrap();
|
||||
let padding = " ".repeat(100 * 1024); // 100 KiB of whitespace
|
||||
write!(f, "{}<html><head><title>Deep Title</title></head></html>", padding).unwrap();
|
||||
assert_eq!(
|
||||
extract_html_title(f.path()),
|
||||
Some("Deep Title".to_string())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn save_with_missing_binary_returns_clear_error() {
|
||||
// Calls save_with directly — no env mutation, safe in parallel test runs.
|
||||
|
|
@ -789,6 +1077,7 @@ mod tests {
|
|||
None, // no cookie ext
|
||||
false, // reader mode off
|
||||
false, // modal closer off
|
||||
false, // freedium cleanup off
|
||||
);
|
||||
let err = result.unwrap_err();
|
||||
let msg = format!("{err:#}");
|
||||
|
|
@ -830,9 +1119,125 @@ mod tests {
|
|||
enabled.eq_ignore_ascii_case("false") || enabled == "0";
|
||||
assert!(is_disabled);
|
||||
|
||||
|
||||
let enabled = "0";
|
||||
let is_disabled =
|
||||
enabled.eq_ignore_ascii_case("false") || enabled == "0";
|
||||
assert!(is_disabled);
|
||||
}
|
||||
|
||||
// ── html_attr_decode ─────────────────────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn html_attr_decode_plain_url_unchanged() {
|
||||
assert_eq!(
|
||||
html_attr_decode("https://cdn.example.com/img.jpg"),
|
||||
"https://cdn.example.com/img.jpg",
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn html_attr_decode_ampersand_in_query() {
|
||||
// CDN signed URL with & encoded as & by the browser's HTML serialiser.
|
||||
assert_eq!(
|
||||
html_attr_decode("https://cdn.example.com/img.jpg?a=1&b=2&sig=abc"),
|
||||
"https://cdn.example.com/img.jpg?a=1&b=2&sig=abc",
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn html_attr_decode_single_layer_only() {
|
||||
// &lt; → < (one browser-serialisation layer removed, not two).
|
||||
// If & ran first it would produce < then < — wrong.
|
||||
assert_eq!(html_attr_decode("&lt;"), "<");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn html_attr_decode_double_encoded_amp() {
|
||||
// &amp; → & (the attribute value is a literal &).
|
||||
assert_eq!(html_attr_decode("&amp;"), "&");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn html_attr_decode_lt_gt_quot_direct() {
|
||||
// Directly encoded entities (no surrounding &) decode normally.
|
||||
assert_eq!(html_attr_decode("<tag>"), "<tag>");
|
||||
assert_eq!(html_attr_decode("say "hi""), "say \"hi\"");
|
||||
}
|
||||
|
||||
// ── same_origin_cookie_header ─────────────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn same_origin_cookie_header_same_host_attaches_cookies() {
|
||||
let mut cookies = HashMap::new();
|
||||
cookies.insert("session".to_string(), "abc".to_string());
|
||||
cookies.insert("tok".to_string(), "xyz".to_string());
|
||||
let h = same_origin_cookie_header(
|
||||
"https://example.com/article",
|
||||
"https://example.com/img/photo.jpg",
|
||||
&cookies,
|
||||
).expect("expected Some for same host");
|
||||
assert!(h.contains("session=abc"), "missing session: {h}");
|
||||
assert!(h.contains("tok=xyz"), "missing tok: {h}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_origin_cookie_header_third_party_returns_none() {
|
||||
let mut cookies = HashMap::new();
|
||||
cookies.insert("session".to_string(), "abc".to_string());
|
||||
assert!(same_origin_cookie_header(
|
||||
"https://example.com/article",
|
||||
"https://cdn.third-party.com/img.jpg",
|
||||
&cookies,
|
||||
).is_none(), "should not forward cookies to third-party host");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_origin_cookie_header_empty_cookies_returns_none() {
|
||||
assert!(same_origin_cookie_header(
|
||||
"https://example.com/article",
|
||||
"https://example.com/img.jpg",
|
||||
&HashMap::new(),
|
||||
).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn same_origin_cookie_header_freedium_empty_cookies_noop() {
|
||||
// capture.rs passes empty cookies for Freedium fetches — confirm no-op.
|
||||
assert!(same_origin_cookie_header(
|
||||
"https://freedium-mirror.cfd/https://example.com/",
|
||||
"https://freedium-mirror.cfd/img/example.com/photo.jpg",
|
||||
&HashMap::new(),
|
||||
).is_none());
|
||||
}
|
||||
|
||||
// ── bounded read (size cap) ───────────────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn bounded_read_stops_at_limit() {
|
||||
// Exercises the same Read::take logic used in fetch_image_as_data_uri.
|
||||
let max: usize = 10;
|
||||
let data: Vec<u8> = (0u8..100).collect();
|
||||
let mut buf = Vec::new();
|
||||
std::io::Cursor::new(&data)
|
||||
.take((max as u64) + 1)
|
||||
.read_to_end(&mut buf)
|
||||
.unwrap();
|
||||
assert!(buf.len() > max, "take should read up to max+1");
|
||||
assert!(buf.len() <= max + 1, "take should not exceed max+1");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bounded_read_allows_at_limit() {
|
||||
let max: usize = 10;
|
||||
let data: Vec<u8> = vec![0u8; max];
|
||||
let mut buf = Vec::new();
|
||||
std::io::Cursor::new(&data)
|
||||
.take((max as u64) + 1)
|
||||
.read_to_end(&mut buf)
|
||||
.unwrap();
|
||||
// Exactly at cap: buf.len() == max, guard does NOT fire.
|
||||
assert_eq!(buf.len(), max);
|
||||
assert!(buf.len() <= max);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -747,6 +747,8 @@ struct CaptureBody {
|
|||
reader_mode: Option<bool>,
|
||||
cookie_ext_enabled: Option<bool>,
|
||||
modal_closer_enabled: Option<bool>,
|
||||
/// Route through Freedium mirror for WebPage captures. Absent = true (on by default).
|
||||
via_freedium: Option<bool>,
|
||||
}
|
||||
|
||||
#[derive(Debug, serde::Deserialize)]
|
||||
|
|
@ -860,6 +862,7 @@ async fn capture_handler(
|
|||
cookie_ext_enabled: Some(effective_cookie_ext),
|
||||
modal_closer_enabled: Some(effective_modal_closer),
|
||||
reader_mode: body.reader_mode.unwrap_or(false),
|
||||
via_freedium: body.via_freedium.unwrap_or(true),
|
||||
};
|
||||
|
||||
// Spawn background capture.
|
||||
|
|
@ -977,6 +980,7 @@ async fn rearchive_handler(
|
|||
cookie_ext_enabled: None,
|
||||
modal_closer_enabled: None,
|
||||
reader_mode: false,
|
||||
via_freedium: false,
|
||||
};
|
||||
|
||||
let job_uid_bg = job_uid.clone();
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
47
crates/archivr-server/static/assets/index-YmIQCrug.js
Normal file
47
crates/archivr-server/static/assets/index-YmIQCrug.js
Normal file
File diff suppressed because one or more lines are too long
|
|
@ -4,7 +4,7 @@
|
|||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Archivr</title>
|
||||
<script type="module" crossorigin src="/assets/index-BpGwC-YY.js"></script>
|
||||
<script type="module" crossorigin src="/assets/index-YmIQCrug.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-DLdY9nrw.css">
|
||||
</head>
|
||||
<body>
|
||||
|
|
|
|||
|
|
@ -142,12 +142,13 @@ export async function fetchTags(archiveId) {
|
|||
export async function submitCapture(archiveId, locator, quality = null, extensions = null) {
|
||||
const payload = { locator }
|
||||
if (quality && quality !== 'best') payload.quality = quality
|
||||
// extensions: { ublock_enabled?: bool, reader_mode?: bool, cookie_ext_enabled?: bool, modal_closer_enabled?: bool }
|
||||
// extensions: { ublock_enabled?: bool, reader_mode?: bool, cookie_ext_enabled?: bool, modal_closer_enabled?: bool, via_freedium?: bool }
|
||||
if (extensions) {
|
||||
if (typeof extensions.ublock_enabled === 'boolean') payload.ublock_enabled = extensions.ublock_enabled
|
||||
if (typeof extensions.reader_mode === 'boolean') payload.reader_mode = extensions.reader_mode
|
||||
if (typeof extensions.cookie_ext_enabled === 'boolean') payload.cookie_ext_enabled = extensions.cookie_ext_enabled
|
||||
if (typeof extensions.modal_closer_enabled === 'boolean') payload.modal_closer_enabled = extensions.modal_closer_enabled
|
||||
if (typeof extensions.via_freedium === 'boolean') payload.via_freedium = extensions.via_freedium
|
||||
}
|
||||
const res = await fetch(`/api/archives/${archiveId}/captures`, {
|
||||
method: "POST",
|
||||
|
|
|
|||
|
|
@ -123,6 +123,7 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
|
|||
// Cookie consent: session-level only, initialized from server default
|
||||
const [cookieExtEnabled, setCookieExtEnabled] = useState(true)
|
||||
const [modalCloserEnabled, setModalCloserEnabled] = useState(true)
|
||||
const [freediumEnabled, setFreediumEnabled] = useState(true)
|
||||
|
||||
// Load global settings from server once on mount
|
||||
useEffect(() => {
|
||||
|
|
@ -301,7 +302,7 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
|
|||
const qual = item.quality || 'best'
|
||||
setItems(prev => prev.map(it => it.id === item.id ? { ...it, status: 'submitting', error: null } : it))
|
||||
try {
|
||||
const extensions = { ublock_enabled: ublockEnabled, reader_mode: readerMode, cookie_ext_enabled: cookieExtEnabled, modal_closer_enabled: modalCloserEnabled }
|
||||
const extensions = { ublock_enabled: ublockEnabled, reader_mode: readerMode, cookie_ext_enabled: cookieExtEnabled, modal_closer_enabled: modalCloserEnabled, via_freedium: freediumEnabled }
|
||||
const job = await submitCapture(aid, loc, qual, extensions)
|
||||
setItems(prev => prev.map(it =>
|
||||
it.id === item.id ? { ...it, status: 'running', jobUid: job.job_uid, archiveId: aid } : it
|
||||
|
|
@ -517,6 +518,22 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
|
|||
<span className="ext-toggle-knob" />
|
||||
</button>
|
||||
</label>
|
||||
<label className="capture-ext-row" style={{ marginTop: 8 }}>
|
||||
<span className="capture-ext-label">
|
||||
<span className="capture-ext-name">Freedium mirror</span>
|
||||
<span className="capture-ext-desc">Route paywalled articles through a Freedium mirror (Medium, NYT, WaPo, etc.)</span>
|
||||
</span>
|
||||
<button
|
||||
type="button"
|
||||
role="switch"
|
||||
aria-checked={freediumEnabled}
|
||||
className={`ext-toggle ext-toggle--sm${freediumEnabled ? ' ext-toggle--on' : ''}`}
|
||||
onClick={() => setFreediumEnabled(v => !v)}
|
||||
aria-label="Toggle Freedium mirror for this capture"
|
||||
>
|
||||
<span className="ext-toggle-knob" />
|
||||
</button>
|
||||
</label>
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue