thegeneralist01/archivr
1
Fork 0
mirror of https://github.com/thegeneralist01/archivr synced 2026-07-22 03:05:32 +02:00
Code Issues Activity
200 commits 1 branch 0 tags 8.4 MiB
Commit graph

200 commits

This branch
This branch
All branches
Author SHA1 Message Date
TheGeneralist
f6d636fce2
feat(singlefile): add UA, disable-web-security, and user-data-dir 
- --user-agent: realistic Chrome UA so servers don't block headless string
- --browser-args=[--disable-web-security, --user-data-dir]: lets single-file
  inline fonts from any cross-origin CDN (e.g. fonts.gstatic.com) regardless
  of ACAO headers; user-data-dir required for --disable-web-security to take
  effect in newer Chromium builds (otherwise silently ignored)
 2026-06-25 11:39:24 +02:00
TheGeneralist
ab353f9209
fix(singlefile): improve font capture and eliminate ES module viewer errors 
Font fidelity:
- --browser-wait-delay=2000: Cloudflare Fonts injects @font-face CSS after
  HTML parse; the font hook needs extra time to see it after networkidle2
- --remove-unused-fonts=false: preserve @font-face rules even when fonts
  haven't rendered yet (font-display:swap, off-screen text)
- --remove-alternative-fonts=false: preserve unicode-range subsets instead
  of stripping them as 'alternatives'

ES module viewer error fix:
- Write a user script (sf-strip-scripts.js) that listens for
  single-file-on-before-capture-start and removes all <script> elements
  (except application/ld+json) from the live DOM before serialization.
  Scripts still execute during capture for CSS fidelity; none end up in
  the saved file, so no data:-URL base ES module resolution errors.
 2026-06-24 21:30:14 +02:00
TheGeneralist
852cc45956
fix(singlefile): disable CSS stripping and script blocking for full render fidelity 
Defaults that were destroying CSS fidelity:
- --remove-unused-styles=true: strips CSS nesting rules (site uses & selector)
  and any rule targeting JS-applied classes
- --remove-alternative-medias=true: deletes @media blocks that don't match
  the capture viewport, breaking responsive layout
- --block-scripts=true: prevents JS from applying classes before CSS snapshot

All three now set to false.
 2026-06-24 21:15:36 +02:00
TheGeneralist
fbb96fbd05
build(frontend): update bundled assets after favicon+rendering changes 2026-06-24 21:05:35 +02:00
TheGeneralist
1239ad9f34
feat(ui): show website favicon in entry list for webpage entries 2026-06-24 19:16:21 +02:00
TheGeneralist
1206d7103d
feat(server): add GET /entries/:uid/favicon endpoint 2026-06-24 19:16:21 +02:00
TheGeneralist
cc2dbf4ac2
feat(archive): add has_favicon to EntrySummary 2026-06-24 19:16:21 +02:00
TheGeneralist
653efa9705
feat(capture): store favicon artifact for WebPage entries 2026-06-24 19:16:21 +02:00
TheGeneralist
ee697625fb
feat(singlefile): add SaveResult, favicon extraction, wait for networkidle2 2026-06-24 19:16:21 +02:00
TheGeneralist
ad1c4f8ee5
fix(nix): guard pkgs.chromium behind isLinux for aarch64-darwin 2026-06-24 18:46:11 +02:00
TheGeneralist
e70c6b37cd
docs: document single-file env vars, check off web page archiving milestone 2026-06-24 18:39:12 +02:00
TheGeneralist
c9ce0ee00b
feat(nix): add single-file-cli and chromium to both wrappers 2026-06-24 18:39:05 +02:00
TheGeneralist
fe9ff2dafe
feat(capture): add Source::WebPage, route HTML URLs to singlefile downloader 2026-06-24 18:39:02 +02:00
TheGeneralist
fd06632073
feat(singlefile): add single-file-cli downloader module 2026-06-24 18:38:57 +02:00
TheGeneralist
b7ab12898e
feat(http): add UrlKind enum and probe_url_kind HEAD probe 2026-06-24 18:38:52 +02:00
TheGeneralist
143f71bc17
feat(core): auto-title for HTTP/S URL downloads 
http::download now returns (hash, extension, Option<title_hint>).
Title is derived from Content-Disposition filename header (RFC 5987
filename*= preferred over plain filename=), falling back to the last
path segment of the final URL after redirects (percent-decoded).

capture.rs Source::Url arm passes the title through to record_media_entry
instead of None, so entries like 'Facharbeit.pdf' or 'facharbeit' appear
in the Title column instead of the raw entry_uid.

Adds percent_decode(), title_from_content_disposition(), title_from_url()
helpers with 10 new unit tests (122 total, all green).
 2026-06-24 14:56:48 +02:00
TheGeneralist
03abfb4d18
feat(core): generic HTTP/S file URL capture (Track 1) 
- Add crates/archivr-core/src/downloader/http.rs
  - download(url, store_path, timestamp) -> Result<(hash, extension)>
  - Rejects text/html responses with a clear error
  - Derives extension from URL path or Content-Type header
  - Follows redirects (capped at 10), user-agent archivr/0.1
  - 10 unit tests for extension/content-type helpers
- Add Source::Url variant to capture::Source enum
- determine_source: unmatched http/https URLs route to Source::Url
- source_metadata: Source::Url => ("web", "file", "file")
- generate_entry_title: Source::Url arm -> "Downloaded File" fallback
- perform_capture: Source::Url arm calls http::download, uses existing
  temp -> hash_exists -> move_temp_to_raw -> record_media_entry pipeline
- Update test expectations: 3 plain https:// cases now expect Source::Url
- Add reqwest 0.12 (blocking) to workspace and archivr-core deps
- Mark URLs milestone done in docs/README.md
- Update NEXT.md Track 1 status
 2026-06-24 14:37:08 +02:00
TheGeneralist
a76dcf647f
fix(server): serve /assets/* from static_dir/assets/ for Vite output 2026-06-24 12:31:56 +02:00
TheGeneralist
9ee6200971
feat(ui): merge React frontend rewrite 2026-06-24 12:25:34 +02:00
TheGeneralist
4458f17b13
feat(ui): rewrite frontend in React with Vite 
- Scaffold Vite+React project in frontend/ (bun, react 18, @vitejs/plugin-react)
- vite.config.js outputs to crates/archivr-server/static/ directly
- src/utils.js: formatBytes, valueText, formatTimestamp, SOURCE_ICONS, sourceIconSvg
- src/api.js: typed fetch wrappers for all API endpoints
- App.jsx: full state management, archive switching, debounced search,
  tag filter, view routing, capture dialog orchestration
- components/Topbar.jsx: archive switcher, nav, capture button
- components/CaptureDialog.jsx: native <dialog> ref with showModal/close,
  Escape key support, locator validation
- components/EntriesView.jsx + EntryRow.jsx: flex table with source icons,
  type pills, keyboard selection
- components/ContextRail.jsx: parallel detail+tags fetch, stale-race guard
  via selectSeqRef, inline tag assign/remove
- components/RunsView.jsx: runs kept as <table> (matches original)
- components/AdminView.jsx: mounted archives list
- components/TagsView.jsx: recursive tag tree with active state
- Remove old app.js; styles.css moved to frontend/src/ (Vite bundles it)
 2026-06-24 12:24:34 +02:00
TheGeneralist
b895d6331c
refactor(ui): replace entry table with flexbox layout 2026-06-23 22:45:37 +02:00
TheGeneralist
623d2a12cc
fix(ui): handle +00:00 timezone in formatTimestamp 2026-06-23 22:40:57 +02:00
TheGeneralist
8ec369a704
feat(ui): source-type icons next to entry titles 2026-06-23 22:40:57 +02:00
TheGeneralist
80828951e2
feat: wire metadata titles through perform_capture and record_tweet_entry 
- perform_capture: fetch yt-dlp metadata (separate --dump-json call)
  before download; derive local title from file:// path filename
- Compute entry_title before record_media_entry call; pass it through
  instead of the temporary None placeholder
- record_tweet_entry: read tweet JSON before entry creation so title
  can be set on NewEntry; add tweet_metadata_from_json helper
 2026-06-23 22:12:34 +02:00
TheGeneralist
3e289383a9
feat: platform-aware entry titling for all 13 source types 
- Add PlatformMetadata struct and generate_entry_title() covering all 13
  Source variants (YouTubeVideo, YouTubePlaylist, YouTubeChannel, X, Tweet,
  TweetThread, Instagram, Facebook, TikTok, Reddit, Snapchat, Local, Other)
- Add downloader/metadata.rs: extract_from_ytdlp_json() parses yt-dlp
  --dump-json output into PlatformMetadata; extracts Reddit subreddit
  from webpage_url via regex
- Add ytdlp::fetch_metadata(): separate --dump-json invocation that does
  not interfere with the actual download call
- Extend record_media_entry() with title: Option<String> param; wire
  yt-dlp metadata fetch + local filename extraction in perform_capture
- Restructure record_tweet_entry() to read tweet JSON before entry
  creation; extract title via tweet_metadata_from_json()
- 16 title-generation unit tests + 6 metadata extraction unit tests
 2026-06-23 21:49:34 +02:00
TheGeneralist
b3f1932b62
chore: update gitignore and flake.nix 2026-06-23 20:33:41 +02:00
TheGeneralist
17d116655a
docs: remove completed planning docs and stale superpowers plan 
- Delete NEXT.md (all six tracks implemented)
- Delete docs/superpowers/plans/track1 plan (track complete)
- Trim ARCHIVR-MENTAL-MODEL.md Key Documents table to only living files
 2026-06-23 17:24:24 +02:00
TheGeneralist
2d7a4f1766
feat(server): configurable bind address with loopback default and non-loopback warning 
- Add optional `bind` field to ServerRegistry (TOML + ARCHIVR_BIND env var)
- Default bind address remains 127.0.0.1:8080; non-loopback prints a warning
- Add route security classification comment block (READ/ADMIN/WRITE/STATIC)
- Add Security and Deployment section to docs/README.md
- Replace vague auth note in ARCHIVR-MENTAL-MODEL.md with concrete model description
- Add three registry tests covering bind field round-trip and defaults
 2026-06-23 17:24:24 +02:00
TheGeneralist
10c41ef84f
feat: browser capture button 2026-06-22 22:47:51 +02:00
TheGeneralist
5ea0c396b8
fix: expand yt:/youtube: shorthands to real URLs before yt-dlp 
expand_shorthand_to_url handled instagram:/facebook:/tiktok: etc. but
not yt:/youtube:. yt-dlp doesn't know the yt: scheme, so shorthands
like yt:video/ID failed with 'Unsupported url scheme'.

Add YouTube cases for video/, short/, shorts/, playlist/, channel/,
c/, user/, and @handle. Full https:// URLs pass through unchanged.

Add tests pinning all new expansions.
 2026-06-22 22:42:41 +02:00
TheGeneralist
e3face0a1b
fix: bundle tweet scraper in archivr-server Nix wrapper 
archivr_server now includes the Python scraper and sets
ARCHIVR_TWEET_PYTHON and ARCHIVR_TWEET_SCRAPER, matching what the
archivr CLI wrapper already does. Without this, browser capture of
tweets failed with 'No such file or directory' because the server
resolved the scraper relative to cwd.
 2026-06-22 22:37:58 +02:00
TheGeneralist
8d0352c2c4
feat: browser capture button (Track 4) 
- Add archivr-core/src/capture.rs: Source enum, all capture helpers,
  fail_run (replaces process::exit fail_archive_and_exit), and
  perform_capture() as the public entry point
- Slim archivr-cli/src/main.rs to 103 lines: thin adapter over
  archivr_core::capture::perform_capture
- Move all source-classification and tweet-entry tests from CLI to
  archivr-core/src/capture.rs (they test core logic, belong in core)
- Add POST /api/archives/:archive_id/captures route in archivr-server:
  validates non-empty locator (400), resolves archive (404), delegates
  to perform_capture, returns {run_uid, status}
- Add capture dialog to browser UI: dialog markup in index.html,
  showModal/submit/cancel/loading/error wiring in app.js, dialog
  styles in styles.css
- 77 tests pass, clean build (0 warnings)
 2026-06-22 22:18:44 +02:00
TheGeneralist
b70ab4945e
feat: hierarchical tags 2026-06-22 14:48:47 +02:00
TheGeneralist
9c4f05b013
fix(server): remove unused post import from routes.rs 2026-06-22 14:48:47 +02:00
TheGeneralist
5803f2119b
feat(server+ui): tag management API routes and browser UI 
- 5 tag routes (list tree, create, assign, remove, search with tag=)
- Tags nav, tag tree view, tag filter badge
- Entry tag pills with remove, assign-tag form in context rail
 2026-06-22 14:48:47 +02:00
TheGeneralist
83f909585f
feat: server-side search with structured prefix query language 2026-06-22 14:48:46 +02:00
TheGeneralist
db953de67a
feat(core): hierarchical tag API 
- Tag/TagNode types, create_tag, list_tag_tree, assign_entry_tag,
  remove_entry_tag, entries_for_tag
- Tag-filtered search returns child entries (no root restriction)
- parent_entry_uid added to EntrySummary
- 8 new tests
 2026-06-22 14:48:46 +02:00
TheGeneralist
fa2968a21a
fix(server): remove unused Body import; keep bonus happy-path artifact test 2026-06-22 14:48:46 +02:00
TheGeneralist
dcf9e127bb
feat: entry detail view and artifact file serving 2026-06-22 14:48:46 +02:00
TheGeneralist
9337a822ad
docs: add next work handoff 2026-06-21 20:17:51 +02:00
TheGeneralist
44c17eee3d
docs: document runtime entrypoints 2026-06-21 19:40:56 +02:00
TheGeneralist
b56c969624
feat: add db and multi-archive web UI foundation (#8) 
* Add SQLite metadata database support

* Implement archive metadata database

* chore: let's guess cargoHash because there's something wrong with nixpkgs!

* Gate test-only database helpers behind cfg(test)

* Fix archive database row identity

* Use serde for archive metadata JSON

* Finalize archive runs at command level

* Handle archive command errors without panics

* Cover tweet entry metadata recording

* Document static regex invariants

* docs: add web UI design spec

* docs: add web UI implementation plan

* chore: move cli into workspace crate

* chore: track workspace crates directory

* refactor: extract archive core crate

* refactor: add core archive opening APIs

* refactor: rename taxonomy model to tags

* feat: add archive query APIs

* feat: add web server registry

* feat: expose archive server APIs

* feat: add archive table web UI

* fix: complete web UI smoke path

* docs: add architecture mental model

* docs: remove private superpowers plans

* nix: split cli and server packages

* chore: remove PLAN.md
 2026-06-14 00:27:16 +02:00
TheGeneralist
cc380ec5ba
fix: extract full tweet text from note_tweet field when available 2026-04-06 11:05:32 +02:00
TheGeneralist
5552591f4f
feat: add X Article archiving (#6) 
* Add X Article archiving
* All TOML formats changed to JSON
 2026-04-04 19:14:06 +02:00
TheGeneralist
51e986b6b2
chore: add isolate_cookies script 
Add `isolate_cookies` script for creating Twitter credentials file
 2026-04-03 15:42:20 +02:00
TheGeneralist
6bee5adfdd
feat: add Twitter tweet/thread archiving and platform shorthand support (#5) 
* Add Twitter tweet and thread archiving support

* Fix tweet scraper path resolution and error reporting

* Flatten tweet archives and rearchive tweet assets

* refactor: simplify archive source parsing

* Refactor tweet archive source handling

* Clean up some clanker-written code

Signed-off-by: TheGeneralist <180094941+thegeneralist01@users.noreply.github.com>

* Rename resolve_from_cwd to absolutize_path

Update call sites and tests to use the new API. Adjust tweet scraper
path/credentials handling and make small tweaks to local path hashing
and
raw store helpers.

Signed-off-by: TheGeneralist
<180094941+thegeneralist01@users.noreply.github.com>
Signed-off-by: TheGeneralist <180094941+thegeneralist01@users.noreply.github.com>

* Add docs for supported platforms, shorthands, and env vars

* Minor clean up

* Implement social shorthand URL expansion and tweet alias parsing

* Extract store and Twitter helpers into shared modules

---------

Signed-off-by: TheGeneralist <180094941+thegeneralist01@users.noreply.github.com>
 2026-04-03 15:34:26 +02:00
TheGeneralist
9441a9d9fb
fix: use remove_dir_all for removal of temp directories 
Signed-off-by: TheGeneralist <180094941+thegeneralist01@users.noreply.github.com>
 2026-03-31 12:49:44 +02:00
TheGeneralist
2d59ab0af5
feat: add archiving of platform media files (#1) 
* chore: specify non-ignored `.md` files

* refactor: rename youtube downloader to ytdlp

More generic name since yt-dlp supports many sites beyond YouTube.

* feat: add local file downloader

Supports file:// URLs for archiving local files.

* deps: add regex crate for URL pattern matching

* feat: expand source detection with granular YouTube types

- Split Source::YouTube into YouTubeVideo, YouTubePlaylist, YouTubeChannel
- Add Source::X for Twitter/X posts
- Add Source::Local for file:// URLs
- Add regex-based URL pattern matching for YouTube URLs
- Add shorthand schemes (yt:video/ID, youtube:playlist/ID, etc.)
- Add comprehensive tests for all URL patterns

* docs: update README milestones

Mark YouTube videos, Twitter videos, and local files as done.

* chore: update flake.lock

* feat: add shorthand schemes for X/Twitter media

* chore: move docs into docs dir

* Remove temp file using timestamp path

Delete the temp entry at store_path/temp/<timestamp> in both
the hash-exists and success paths. Stop constructing the full filename
with extension and remove the early process::exit to de-duplicate
cleanup.

* Add Nix caches and default flake package

* Add social platform source detection and update milestones

* Tighten social URL matching to avoid false positives

* Mark media archiving milestone complete
 2026-03-31 12:39:35 +02:00
TheGeneralist
553cca99ca
docs: add README and LICENSE 2025-10-15 00:43:54 +02:00
TheGeneralist
01cc7826bf
feat: finish YouTube downloading; primitive "raw" archiving 2025-10-15 00:29:08 +02:00