thegeneralist01/archivr
1
Fork 0
mirror of https://github.com/thegeneralist01/archivr synced 2026-07-21 18:55:36 +02:00
Code Issues Activity
165 commits 2 branches 0 tags 8.1 MiB
Commit graph

165 commits

This branch
This branch
All branches
Author SHA1 Message Date
TheGeneralist
a4239ba5fc
feat(auth): add auth_db_path to AppState, registry, and main.rs 2026-06-26 11:46:23 +02:00
TheGeneralist
86e8049f1b
feat(auth): session and token DB helpers 2026-06-26 11:32:32 +02:00
TheGeneralist
f482707b75
feat(auth): user and role DB helpers (create_owner, compute_role_bits) 2026-06-26 11:31:20 +02:00
TheGeneralist
696a5e1ac7
feat(auth): add initialize_auth_schema, open_auth_db, and auth record types 2026-06-26 11:30:02 +02:00
TheGeneralist
7fcc207de7
feat(auth): user menu and logout button in Topbar 2026-06-26 11:29:13 +02:00
TheGeneralist
0a5c899c5e
feat(auth): frontend login/setup pages + auth state in App.jsx 2026-06-26 11:28:58 +02:00
TheGeneralist
57fc48d73c
feat(auth): add argon2, rand, axum-extra dependencies 2026-06-26 11:28:51 +02:00
TheGeneralist
decaef389b
docs: mark Track 4 auth foundation done, add Tracks 5-8 stubs, renumber roadmap 2026-06-26 11:28:21 +02:00
TheGeneralist
7584e309de
chore: remove docs/superpowers from gitignore (not for VCS) 2026-06-26 11:19:40 +02:00
TheGeneralist
d81b8fd793
docs: add Track 4 auth foundation implementation plan 2026-06-25 17:38:30 +02:00
TheGeneralist
151835258b
docs: fix 8 reviewer findings in auth foundation spec 
- Role hierarchy: cumulative assignment + ROLE_GUEST floor; owner gets role_bits=15
- ALTER TABLE: move default_entry_visibility into CREATE TABLE DDL (idempotent)
- Session cookie: add Secure flag conditionally on HTTPS (X-Forwarded-Proto)
- API token role_bits: live query user_roles per request (vs snapshot)
- POST /api/auth/setup after complete: returns 409 already_configured
- users.role placeholder: use 'admin' in ensure_owner_exists INSERT
- Disabled users: extractor JOINs users and checks status='active'
- last_seen_at throttle: conditional update if >60s elapsed
 2026-06-25 17:27:33 +02:00
TheGeneralist
10aeebd878
docs: add Track 4 auth foundation design spec + whitelist docs/superpowers in gitignore 2026-06-25 17:20:17 +02:00
TheGeneralist
5fba6987a3
fix(capture): remove debug println! from hash_exists 2026-06-25 16:19:33 +02:00
TheGeneralist
351e74df09
feat(fonts): wire font extraction into capture pipeline + blob serving route 
- capture.rs: add archive_id: Option<&str> to perform_capture; when Some,
  call font_extractor::extract_and_rewrite before hashing HTML, register
  each font as a deduplicated blob + 'font' artifact
- main.rs: pass None as archive_id (CLI keeps fonts embedded)
- routes.rs: add GET /api/archives/:id/blobs/:sha256 (serve_blob handler),
  pass Some(&archive_id) to perform_capture in capture_handler,
  add ApiError::internal constructor
 2026-06-25 16:07:50 +02:00
TheGeneralist
22b0a55730
feat(fonts): hash_bytes, get_blob_by_sha256, font_extractor module 
- fix(hash): hash raw bytes instead of lossy UTF-8; add hash_bytes
- feat(database): add get_blob_by_sha256 lookup
- feat(font_extractor): extract embedded font data-URIs from archived HTML
 2026-06-25 16:02:57 +02:00
TheGeneralist
f6d636fce2
feat(singlefile): add UA, disable-web-security, and user-data-dir 
- --user-agent: realistic Chrome UA so servers don't block headless string
- --browser-args=[--disable-web-security, --user-data-dir]: lets single-file
  inline fonts from any cross-origin CDN (e.g. fonts.gstatic.com) regardless
  of ACAO headers; user-data-dir required for --disable-web-security to take
  effect in newer Chromium builds (otherwise silently ignored)
 2026-06-25 11:39:24 +02:00
TheGeneralist
ab353f9209
fix(singlefile): improve font capture and eliminate ES module viewer errors 
Font fidelity:
- --browser-wait-delay=2000: Cloudflare Fonts injects @font-face CSS after
  HTML parse; the font hook needs extra time to see it after networkidle2
- --remove-unused-fonts=false: preserve @font-face rules even when fonts
  haven't rendered yet (font-display:swap, off-screen text)
- --remove-alternative-fonts=false: preserve unicode-range subsets instead
  of stripping them as 'alternatives'

ES module viewer error fix:
- Write a user script (sf-strip-scripts.js) that listens for
  single-file-on-before-capture-start and removes all <script> elements
  (except application/ld+json) from the live DOM before serialization.
  Scripts still execute during capture for CSS fidelity; none end up in
  the saved file, so no data:-URL base ES module resolution errors.
 2026-06-24 21:30:14 +02:00
TheGeneralist
852cc45956
fix(singlefile): disable CSS stripping and script blocking for full render fidelity 
Defaults that were destroying CSS fidelity:
- --remove-unused-styles=true: strips CSS nesting rules (site uses & selector)
  and any rule targeting JS-applied classes
- --remove-alternative-medias=true: deletes @media blocks that don't match
  the capture viewport, breaking responsive layout
- --block-scripts=true: prevents JS from applying classes before CSS snapshot

All three now set to false.
 2026-06-24 21:15:36 +02:00
TheGeneralist
fbb96fbd05
build(frontend): update bundled assets after favicon+rendering changes 2026-06-24 21:05:35 +02:00
TheGeneralist
1239ad9f34
feat(ui): show website favicon in entry list for webpage entries 2026-06-24 19:16:21 +02:00
TheGeneralist
1206d7103d
feat(server): add GET /entries/:uid/favicon endpoint 2026-06-24 19:16:21 +02:00
TheGeneralist
cc2dbf4ac2
feat(archive): add has_favicon to EntrySummary 2026-06-24 19:16:21 +02:00
TheGeneralist
653efa9705
feat(capture): store favicon artifact for WebPage entries 2026-06-24 19:16:21 +02:00
TheGeneralist
ee697625fb
feat(singlefile): add SaveResult, favicon extraction, wait for networkidle2 2026-06-24 19:16:21 +02:00
TheGeneralist
ad1c4f8ee5
fix(nix): guard pkgs.chromium behind isLinux for aarch64-darwin 2026-06-24 18:46:11 +02:00
TheGeneralist
e70c6b37cd
docs: document single-file env vars, check off web page archiving milestone 2026-06-24 18:39:12 +02:00
TheGeneralist
c9ce0ee00b
feat(nix): add single-file-cli and chromium to both wrappers 2026-06-24 18:39:05 +02:00
TheGeneralist
fe9ff2dafe
feat(capture): add Source::WebPage, route HTML URLs to singlefile downloader 2026-06-24 18:39:02 +02:00
TheGeneralist
fd06632073
feat(singlefile): add single-file-cli downloader module 2026-06-24 18:38:57 +02:00
TheGeneralist
b7ab12898e
feat(http): add UrlKind enum and probe_url_kind HEAD probe 2026-06-24 18:38:52 +02:00
TheGeneralist
143f71bc17
feat(core): auto-title for HTTP/S URL downloads 
http::download now returns (hash, extension, Option<title_hint>).
Title is derived from Content-Disposition filename header (RFC 5987
filename*= preferred over plain filename=), falling back to the last
path segment of the final URL after redirects (percent-decoded).

capture.rs Source::Url arm passes the title through to record_media_entry
instead of None, so entries like 'Facharbeit.pdf' or 'facharbeit' appear
in the Title column instead of the raw entry_uid.

Adds percent_decode(), title_from_content_disposition(), title_from_url()
helpers with 10 new unit tests (122 total, all green).
 2026-06-24 14:56:48 +02:00
TheGeneralist
03abfb4d18
feat(core): generic HTTP/S file URL capture (Track 1) 
- Add crates/archivr-core/src/downloader/http.rs
  - download(url, store_path, timestamp) -> Result<(hash, extension)>
  - Rejects text/html responses with a clear error
  - Derives extension from URL path or Content-Type header
  - Follows redirects (capped at 10), user-agent archivr/0.1
  - 10 unit tests for extension/content-type helpers
- Add Source::Url variant to capture::Source enum
- determine_source: unmatched http/https URLs route to Source::Url
- source_metadata: Source::Url => ("web", "file", "file")
- generate_entry_title: Source::Url arm -> "Downloaded File" fallback
- perform_capture: Source::Url arm calls http::download, uses existing
  temp -> hash_exists -> move_temp_to_raw -> record_media_entry pipeline
- Update test expectations: 3 plain https:// cases now expect Source::Url
- Add reqwest 0.12 (blocking) to workspace and archivr-core deps
- Mark URLs milestone done in docs/README.md
- Update NEXT.md Track 1 status
 2026-06-24 14:37:08 +02:00
TheGeneralist
a76dcf647f
fix(server): serve /assets/* from static_dir/assets/ for Vite output 2026-06-24 12:31:56 +02:00
TheGeneralist
9ee6200971
feat(ui): merge React frontend rewrite 2026-06-24 12:25:34 +02:00
TheGeneralist
4458f17b13
feat(ui): rewrite frontend in React with Vite 
- Scaffold Vite+React project in frontend/ (bun, react 18, @vitejs/plugin-react)
- vite.config.js outputs to crates/archivr-server/static/ directly
- src/utils.js: formatBytes, valueText, formatTimestamp, SOURCE_ICONS, sourceIconSvg
- src/api.js: typed fetch wrappers for all API endpoints
- App.jsx: full state management, archive switching, debounced search,
  tag filter, view routing, capture dialog orchestration
- components/Topbar.jsx: archive switcher, nav, capture button
- components/CaptureDialog.jsx: native <dialog> ref with showModal/close,
  Escape key support, locator validation
- components/EntriesView.jsx + EntryRow.jsx: flex table with source icons,
  type pills, keyboard selection
- components/ContextRail.jsx: parallel detail+tags fetch, stale-race guard
  via selectSeqRef, inline tag assign/remove
- components/RunsView.jsx: runs kept as <table> (matches original)
- components/AdminView.jsx: mounted archives list
- components/TagsView.jsx: recursive tag tree with active state
- Remove old app.js; styles.css moved to frontend/src/ (Vite bundles it)
 2026-06-24 12:24:34 +02:00
TheGeneralist
b895d6331c
refactor(ui): replace entry table with flexbox layout 2026-06-23 22:45:37 +02:00
TheGeneralist
623d2a12cc
fix(ui): handle +00:00 timezone in formatTimestamp 2026-06-23 22:40:57 +02:00
TheGeneralist
8ec369a704
feat(ui): source-type icons next to entry titles 2026-06-23 22:40:57 +02:00
TheGeneralist
80828951e2
feat: wire metadata titles through perform_capture and record_tweet_entry 
- perform_capture: fetch yt-dlp metadata (separate --dump-json call)
  before download; derive local title from file:// path filename
- Compute entry_title before record_media_entry call; pass it through
  instead of the temporary None placeholder
- record_tweet_entry: read tweet JSON before entry creation so title
  can be set on NewEntry; add tweet_metadata_from_json helper
 2026-06-23 22:12:34 +02:00
TheGeneralist
3e289383a9
feat: platform-aware entry titling for all 13 source types 
- Add PlatformMetadata struct and generate_entry_title() covering all 13
  Source variants (YouTubeVideo, YouTubePlaylist, YouTubeChannel, X, Tweet,
  TweetThread, Instagram, Facebook, TikTok, Reddit, Snapchat, Local, Other)
- Add downloader/metadata.rs: extract_from_ytdlp_json() parses yt-dlp
  --dump-json output into PlatformMetadata; extracts Reddit subreddit
  from webpage_url via regex
- Add ytdlp::fetch_metadata(): separate --dump-json invocation that does
  not interfere with the actual download call
- Extend record_media_entry() with title: Option<String> param; wire
  yt-dlp metadata fetch + local filename extraction in perform_capture
- Restructure record_tweet_entry() to read tweet JSON before entry
  creation; extract title via tweet_metadata_from_json()
- 16 title-generation unit tests + 6 metadata extraction unit tests
 2026-06-23 21:49:34 +02:00
TheGeneralist
b3f1932b62
chore: update gitignore and flake.nix 2026-06-23 20:33:41 +02:00
TheGeneralist
17d116655a
docs: remove completed planning docs and stale superpowers plan 
- Delete NEXT.md (all six tracks implemented)
- Delete docs/superpowers/plans/track1 plan (track complete)
- Trim ARCHIVR-MENTAL-MODEL.md Key Documents table to only living files
 2026-06-23 17:24:24 +02:00
TheGeneralist
2d7a4f1766
feat(server): configurable bind address with loopback default and non-loopback warning 
- Add optional `bind` field to ServerRegistry (TOML + ARCHIVR_BIND env var)
- Default bind address remains 127.0.0.1:8080; non-loopback prints a warning
- Add route security classification comment block (READ/ADMIN/WRITE/STATIC)
- Add Security and Deployment section to docs/README.md
- Replace vague auth note in ARCHIVR-MENTAL-MODEL.md with concrete model description
- Add three registry tests covering bind field round-trip and defaults
 2026-06-23 17:24:24 +02:00
TheGeneralist
10c41ef84f
feat: browser capture button 2026-06-22 22:47:51 +02:00
TheGeneralist
5ea0c396b8
fix: expand yt:/youtube: shorthands to real URLs before yt-dlp 
expand_shorthand_to_url handled instagram:/facebook:/tiktok: etc. but
not yt:/youtube:. yt-dlp doesn't know the yt: scheme, so shorthands
like yt:video/ID failed with 'Unsupported url scheme'.

Add YouTube cases for video/, short/, shorts/, playlist/, channel/,
c/, user/, and @handle. Full https:// URLs pass through unchanged.

Add tests pinning all new expansions.
 2026-06-22 22:42:41 +02:00
TheGeneralist
e3face0a1b
fix: bundle tweet scraper in archivr-server Nix wrapper 
archivr_server now includes the Python scraper and sets
ARCHIVR_TWEET_PYTHON and ARCHIVR_TWEET_SCRAPER, matching what the
archivr CLI wrapper already does. Without this, browser capture of
tweets failed with 'No such file or directory' because the server
resolved the scraper relative to cwd.
 2026-06-22 22:37:58 +02:00
TheGeneralist
8d0352c2c4
feat: browser capture button (Track 4) 
- Add archivr-core/src/capture.rs: Source enum, all capture helpers,
  fail_run (replaces process::exit fail_archive_and_exit), and
  perform_capture() as the public entry point
- Slim archivr-cli/src/main.rs to 103 lines: thin adapter over
  archivr_core::capture::perform_capture
- Move all source-classification and tweet-entry tests from CLI to
  archivr-core/src/capture.rs (they test core logic, belong in core)
- Add POST /api/archives/:archive_id/captures route in archivr-server:
  validates non-empty locator (400), resolves archive (404), delegates
  to perform_capture, returns {run_uid, status}
- Add capture dialog to browser UI: dialog markup in index.html,
  showModal/submit/cancel/loading/error wiring in app.js, dialog
  styles in styles.css
- 77 tests pass, clean build (0 warnings)
 2026-06-22 22:18:44 +02:00
TheGeneralist
b70ab4945e
feat: hierarchical tags 2026-06-22 14:48:47 +02:00
TheGeneralist
9c4f05b013
fix(server): remove unused post import from routes.rs 2026-06-22 14:48:47 +02:00
TheGeneralist
5803f2119b
feat(server+ui): tag management API routes and browser UI 
- 5 tag routes (list tree, create, assign, remove, search with tag=)
- Tags nav, tag tree view, tag filter badge
- Entry tag pills with remove, assign-tag form in context rail
 2026-06-22 14:48:47 +02:00