1
Fork 0
mirror of https://github.com/thegeneralist01/archivr synced 2026-07-21 18:55:36 +02:00
Commit graph

165 commits

Author SHA1 Message Date
2e8820a0da
feat: uBlock Origin Lite + cookie consent extension + reader mode + ad placeholder cleanup (#21)
* feat: uBlock Origin Lite integration for ad-blocking during WebPage captures

- singlefile.rs: when ARCHIVR_UBLOCK=true and ARCHIVR_UBLOCK_EXT is set,
  archivr owns Chrome's lifecycle (--headless=new, --remote-debugging-port,
  --load-extension); single-file connects via --browser-server instead of
  launching its own Chrome. Falls back to old behaviour with ublock_skipped=true
  when the ext path is missing or invalid.
- capture.rs: thread ublock_skipped through CaptureResult
- database.rs: add notes_json TEXT column to capture_jobs (DDL + idempotent
  ALTER TABLE migration); update_capture_job_status gains notes_json param
- archive.rs: expose notes_json in CaptureJobSummary
- routes.rs: store {"ublock_skipped":true} in notes_json on completed captures
- ToastStack.jsx: warning toast variant (toast--warning) with Details expander
  and Ignore button
- CaptureDialog.jsx: fire warning toast when poll result has ublock_skipped
- App.jsx: sessionStorage-backed Ignore suppression for ublock warnings
- styles.css: .toast--warning (amber left border) + .toast-warning-detail
- flake.nix: ublockLite derivation fetches uBOLite_2026.705.2152.chromium.zip
  (pinned SHA256) from uBlockOrigin/uBOL-home; sets ARCHIVR_UBLOCK_EXT in both
  archivr and archivr-server wrappers

Env vars:
  ARCHIVR_UBLOCK=true (default) — enable uBlock during WebPage captures
  ARCHIVR_UBLOCK_EXT — path to unpacked uBOL extension dir (set by Nix)

* feat: Extensions settings tab + capture dialog redesign with Advanced options

Settings/Extensions tab (admin-only):
- New 'Extensions' tab between Cookies and Storage
- ExtensionsTab component: shows uBlock Origin Lite card with pill toggle
- Reads ublock_enabled from instance settings; patch via existing PATCH endpoint
- Shows ublock_ext_available status from server (whether ARCHIVR_UBLOCK_EXT is set)

Instance settings:
- Add ublock_enabled BOOLEAN (default true) to instance_settings auth DB table
- Idempotent ALTER TABLE migration in initialize_auth_schema()
- get/update_instance_settings include ublock_enabled
- GET /api/admin/instance-settings now also returns ublock_ext_available (computed
  from ARCHIVR_UBLOCK_EXT env var at request time)
- PATCH /api/admin/instance-settings accepts ublock_enabled

Per-capture override:
- CaptureBody gains ublock_enabled: Option<bool>
- CaptureConfig gains ublock_enabled: Option<bool>
- singlefile::save() gains ublock_enabled_override: Option<bool> param
- Capture handler resolves: body override > global instance setting > env var
- submitCapture(aid, loc, qual, extensions) in api.js passes ublock_enabled

Capture dialog redesign:
- Archive button: full-width, 13px padding, min-width 220px, primary CTA
- Cancel: full-width but text-style, below Archive
- ‹Advanced options› chevron toggle (rotates on open)
- Expanded panel shows uBlock toggle for this capture session
- Loads global ublock_enabled default from instance settings on mount

Styles:
- .ext-toggle pill switch (44×24 and 36×20 small variant)
- .ext-card for Settings Extensions tab
- .capture-advanced + .capture-advanced-panel + .capture-chevron
- .capture-ext-row / .capture-ext-label / .capture-ext-name / .capture-ext-desc
- .form-hint utility class

* fix: remove ublock_enabled from INSERT OR IGNORE in DDL batch

The INSERT ran before the ALTER TABLE migration added the column,
causing 'table instance_settings has no column named ublock_enabled'
on existing databases. The INSERT OR IGNORE for the default row only
needs the original columns; the migration's DEFAULT 1 handles the
new column for existing and new rows alike.

* feat: Reader mode via Mozilla Readability.js

Adds an opt-in 'Reader mode' advanced option to the capture dialog.
When enabled, Readability.js is injected as a browser script during
SingleFile capture; it fires on single-file-on-before-capture-start,
replaces the page body with the distilled article content, injects a
clean typographic stylesheet, and adds a header with title/byline/site.
Falls back silently if Readability fails (e.g. non-article pages).

- vendor/readability/Readability.js  Apache 2.0, Mozilla, v0.6.0
- singlefile.rs: embed READABILITY_JS + READER_MODE_WRAPPER_JS via
  include_str!; write both to temp dir when reader_mode is true;
  base_single_file_cmd now accepts &[&Path] for multiple --browser-script
- capture.rs: CaptureConfig.reader_mode: bool
- routes.rs: CaptureBody.reader_mode: Option<bool> (defaults false)
- api.js: submitCapture passes reader_mode in payload
- CaptureDialog.jsx: Reader mode toggle in Advanced options (off by default)

* fix: diagnose single-file no-output-file error + prevent stdout dumping

- Add --dump-content=false to every single-file invocation to prevent
  the Docker-detection heuristic from routing HTML to stdout instead of
  the output file (the heuristic can trigger in some macOS environments)
- Improve the no-output-file error message to include: temp dir contents,
  stderr, and first 200 chars of stdout — this gives enough context to
  diagnose any remaining cause without re-running

* fix: switch uBlock loading from --browser-server to --browser-args

The --browser-server (CDP) path caused 'Unexpected server response: 404'
on macOS Chrome because simple-cdp's WebSocket upgrade to the debugger
endpoint failed after Chrome started — likely a version-specific CDP
endpoint shape mismatch.

New approach: single-file always manages Chrome. When ARCHIVR_UBLOCK_EXT
is set, --headless=new, --load-extension, and --disable-extensions-except
are injected via --browser-args. single-file's browser.js prefix-strips
its own conflicting flags before appending ours, so --headless=new
overrides the default --headless (enabling extension support in headless).

Removes allocate_free_port, wait_for_chrome_ready, run_single_file_with_server
(all dead code now). Docblock updated to reflect actual behaviour and notes
the --single-process caveat: uBOL's declarativeNetRequest static rulesets
are expected to work (network-stack level, not service-worker), but this
has not been mechanically verified under --single-process.

Smoke tested on macOS (this machine): capture with --load-extension + all
three browser-scripts (strip, Readability, reader-mode wrapper) produces
output file correctly. Ad-blocking verification deferred to manual test
with a tracker-heavy URL.

* fix: use correct single-file hook event (single-file-on-before-capture-request)

Prior scripts listened on 'single-file-on-before-capture-start' which
does not exist in single-file-core 1.1.49.  The real hook is:

  single-file-on-before-capture-request  (dispatched by initUserScriptHandler
  after receiving single-file-user-script-init; userScriptEnabled defaults
  to true in args.js so it always fires when --browser-script is passed)

Changes:
- strip-scripts: -start -> -request (no preventDefault needed; synchronous)
- READER_MODE_SCRIPT: -start -> -request; add 'installed' meta marker at
  script-evaluation time so artifact inspection can distinguish 'script
  not injected' / 'hook never fired' / 'Readability parse failed'

* fix: correct singlefile.rs docstring (scripts.js concatenates, not isolates)

* fix: dispatch single-file-user-script-init so request hook fires

single-file's initUserScriptHandler (in single-file-bootstrap.js) listens
for 'single-file-user-script-init' and only then installs
_singleFile_waitForUserScript.  Without that dispatch our scripts'
'single-file-on-before-capture-request' listeners were never reached,
so neither strip-scripts nor reader-mode Readability applied.

Dispatch the init event at the top of strip-scripts (always present) and
redundantly in READER_MODE_SCRIPT.  Verified end-to-end: artifact for
run_b3181d6d276e4e56a1a6c356ef9bbe8f has
  meta content="applied", max-width:680px CSS, 0 script tags.

* feat: cookie consent extension support (ARCHIVR_COOKIE_EXT)

Mirrors the uBlock Origin Lite integration exactly:

Backend:
- singlefile.rs: resolve_cookie_ext_config() reads ARCHIVR_COOKIE_CONSENT
  (default true) + ARCHIVR_COOKIE_EXT path; extension paths comma-joined
  into --load-extension / --disable-extensions-except so uBlock and cookie
  ext can coexist; SaveResult.cookie_ext_skipped tracks miss
- database.rs: cookie_ext_enabled column on instance_settings (DEFAULT 1);
  idempotent ALTER TABLE migration; get/update wired through
- capture.rs: CaptureConfig.cookie_ext_enabled: Option<bool>; threaded to
  singlefile::save(); cookie_ext_skipped surfaced in CaptureResult
- routes.rs: CaptureBody + UpdateInstanceSettingsBody get cookie_ext_enabled;
  capture handler resolves effective value (body overrides global); notes_json
  only includes skipped fields that are true; GET instance-settings includes
  cookie_ext_available from env path check

Frontend:
- api.js: submitCapture forwards cookie_ext_enabled
- SettingsView.jsx: 'I Still Don't Care About Cookies' card in Extensions
  tab; always-active toggle (user can disable even when ext not installed);
  amber 'Not configured' hint + ARCHIVR_COOKIE_EXT guidance when unavailable
- CaptureDialog.jsx: 'Block cookie banners' toggle in Advanced options;
  always shown with amber hint when ext not configured; defaults from
  global setting

Operator setup: download + unzip the extension from GitHub releases, set
ARCHIVR_COOKIE_EXT=/path/to/unpacked/ext. No Node daemon needed.

* fix: surface cookie_ext_skipped warning toast in CaptureDialog

* feat: package istilldontcareaboutcookies in flake, wire ARCHIVR_COOKIE_EXT

Add isdcac derivation mirroring ublockLite:
- Fetches ISDCAC-chrome-source.zip v1.1.9 from GitHub releases
- Validates manifest.json at extension root before install (guard against
  nested-folder zip regressions in future releases)
- Sets ARCHIVR_COOKIE_EXT in both archivr and archivr_server wrappers

Verified: nix build .#archivr-server and .#archivr both succeed;
wrapper scripts export correct store paths; manifest.json present at root.

* fix: gate consent-overlay cleanup on cookie_ext; reset overflow; narrow selectors

- Strip overflow:hidden from body/html only when cookie_ext is active for
  the capture — prevents mutating legitimate pages when the feature is off
- Remove .fc-dialog (Google Funding Choices), .qc-cmp2-*, .sp-message-container,
  #sp-cc, #usercentrics-root as fallback for CMPs the extension misses
- Removed overbroad [class^="uc-"] and [id^="usercentrics"] selectors
  that could match real page content

* fix: remove ad placeholders when uBlock active; kept height causes blank gap

uBlock Origin Lite blocks ad network requests but first-party placeholder
elements (ins.adsbygoogle, #aswift_* iframe hosts) retain their computed
height (e.g. 280px for a top banner), leaving a large blank space at the
top of captured pages.

Gate cleanup on ublock_ext.is_some(): remove ins.adsbygoogle, aswift_*
iframes, and google_ads_* iframes before SingleFile serialises. Also
collapse the parent container if it becomes empty after removal.

* fix: walk up to .top-ad/.google-auto-placed ancestor before removing ad slot

Removing only the inner ins.adsbygoogle left the outer .container.top-ad
wrapper (with pb-4 padding) in the layout, preserving the blank gap.
Now walk up via closest() to the nearest ad-slot container class before
removal so the whole slot including padding collapses.
2026-07-08 23:26:48 +02:00
dae61e585d
feat: add user-configurable cookie rules (#20)
Adds per-instance cookie rules (admin-only) that are injected into
every network touchpoint during capture.

Storage:
- New cookie_rules table in the auth DB (idempotent migration)
- Rules have pattern_kind (global/wildcard/regex), optional url_pattern,
  and cookies_json (validated as string-only JSON object)

Matching (resolve_cookies_for_url):
- Global rules always apply
- Wildcard: * and ? with full metacharacter escaping; matched against
  hostname via reqwest::Url when pattern has no ://, full URL otherwise
- Regex: matched against the full URL
- Later rules in ordinal order override earlier ones per cookie name

All six network touchpoints receive resolved cookies:
- http::probe_url_kind and http::download: Cookie request header
- singlefile::save: Netscape cookie file -> --browser-cookies-file
- ytdlp::fetch_metadata and ytdlp::download: Netscape cookie file -> --cookies
- tweets::archive: semicolon credentials file -> --credentials-file
  (only when both ct0 and auth_token are present; otherwise falls back
  to ARCHIVR_TWITTER_CREDENTIALS_FILE)

Security:
- Cookie files written 0o600 (owner read/write only)
- Exact parsed hostname used as cookie domain (no PSL stripping)
- Files deleted unconditionally before any error propagates,
  including spawn failures (hold-result-then-delete pattern)
- No cookie values in process args (no --add-header exposure)

API: GET/POST /api/admin/cookie-rules, PATCH/DELETE /api/admin/cookie-rules/:uid
Frontend: Cookies tab in Settings (admin only) with rule list,
  inline edit, pattern-type selector, client-side JSON validation
CLI: CaptureConfig::default() - no behaviour change

254 tests passing (4 new cookie-rule handler tests)
2026-07-06 19:01:34 +02:00
21b11c211f
feat: YouTube Music audio capture (ytm: shorthand, Spotify detection, stalled job recovery) (#19)
* feat: add YouTube Music and Spotify source detection

- Add Source variants: YouTubeMusicTrack, YouTubeMusicPlaylist,
  SpotifyTrack, SpotifyAlbum, SpotifyPlaylist
- ytm:ID shorthand → music.youtube.com/watch?v=ID (audio-only, forced
  in core regardless of caller quality hint)
- ytm:playlist/ID and music.youtube.com/playlist URLs detected but
  fail with 'not yet implemented' via fail_run
- Spotify URLs/shorthands detected and fail fast with clear DRM error
  via fail_run (after run item created, so status is visible in /runs)
- source_metadata: youtube_music/music/audio and spotify/music/audio
  (entity_kind='music' for UI pill, representation_kind='audio' stored)
- locator_to_ytdlp_url includes YouTubeMusicTrack for probe endpoint
- generate_entry_title: 'Title — Artist' for YTM tracks
- Frontend: isVideoSource handles ytm: and music.youtube.com/watch;
  Spotify returns false (no probe, clear server error on submit)
- Placeholder updated to include ytm:ID
- SOURCE_ICONS: youtube_music (red disc) and spotify (green waves)
- 14 new tests covering all new sources (163 total, all pass)

* fix: prevent yt-dlp playlist expansion and stalled run recovery

- Add --no-playlist to ytdlp::download and fetch_metadata: URLs with a
  list= parameter (e.g. music.youtube.com/watch?v=ID&list=RDAMVM…) no
  longer cause yt-dlp to expand the full playlist and hang; both the
  metadata probe and the download are now single-item only

- Fix fail_stalled_capture_jobs to also recover archive_runs and
  archive_run_items: capture_jobs.run_uid is NULL at crash time so a
  join is unreliable; instead fail all archive_runs/items still
  in_progress directly, then recount failed_count via subquery.
  Startup recovery now makes the Runs UI reflect the correct failed
  state after a hard shutdown

- Expand fail_stalled_jobs_on_restart test to assert archive_run and
  archive_run_item rows are also marked failed, not just capture_jobs

* fix: use play triangle for youtube_music icon
2026-07-06 15:34:14 +02:00
339076e6a2
feat: add orphan blob cleanup to Settings > Storage tab (#18)
- database.rs: add has_active_capture_jobs(), list_orphaned_blob_rows(),
  all_referenced_file_relpaths(), delete_orphaned_blob_rows()
- routes.rs: GET/DELETE /api/archives/:id/blob-cleanup (ROLE_ADMIN)
  - GET returns {orphaned_blob_rows, deletable_files, total_bytes}
  - DELETE has two active-capture guards (before and after disk walk)
    to prevent deleting files mid-capture; walks raw/ and raw_tweets/
  - Referenced set = entry_artifacts.relpath ∪ live blobs' raw_relpath,
    so a file is never deleted if any artifact still points at its path
- api.js: scanOrphanBlobs(), deleteOrphanBlobs()
- App.jsx: pass archiveId to SettingsView; add 'storage' to SETTINGS_TABS
  so /settings/storage survives refresh/back navigation
- SettingsView.jsx: new Storage tab (admin-only) with idle→scanning→
  scanned→deleting→done/error state machine; shows file/record counts
  and human-readable byte sizes before a btn-danger confirm

Tests (14 new, 248 total passing):
- database.rs: has_active_capture_jobs for pending/running/completed,
  list_orphaned_blob_rows, all_referenced_file_relpaths edge cases,
  delete_orphaned_blob_rows preserves referenced rows
- routes.rs: auth (401), active-capture 409 on GET and DELETE,
  end-to-end delete preserving referenced file and removing orphan
  blob file + extra disk-only file
2026-07-06 14:01:38 +02:00
b8e496457f
feat: add video quality selection for yt-dlp captures (#17)
- ytdlp::download() accepts quality: Option<&str>; quality_format()
  maps best/1080p/720p/480p/360p to yt-dlp -f format strings
- perform_capture() threads quality through to the downloader
- CaptureBody gains optional quality field; capture_handler validates
  it against the allowlist (400 on unknown values) before spawning
- CLI passes None (preserves existing best-quality behaviour)
- Frontend: isVideoSource() mirrors determine_source() exactly —
  shows quality picker only for yt-dlp-backed sources, excludes
  playlist/channel shorthands and tweet/thread paths
- submitCapture(archiveId, locator, quality) sends quality in POST body
- CSS: .capture-quality styles the inline select to fit the capture row
- Tests: quality_format unit tests in ytdlp.rs; two new route tests
  (valid quality accepted, invalid quality rejected with 400)
- Docs: video quality section added under Supported Platforms
2026-07-05 20:42:41 +02:00
84c5730b6a
singlefile: force --window-size=1920,1080 for viewport-dependent CSS preservation
Add --window-size=1920,1080 to the Chromium flags passed via --browser-args.
This makes the existing --remove-unused-styles=false and
--remove-alternative-medias=false effective for real @media rules
and responsive styles (headless default is small).

Also document in ARCHIVR_CHROME_ARGS that users can override by
supplying their own --window-size in the env var.
2026-07-05 15:40:58 +02:00
fb1115a409
feat: non-blocking batch capture dialog, toasts on failure, fix /runs errors
CaptureDialog:
- Replace single textarea with multi-row inputs; + button adds rows
- Submit fires all pending rows in parallel, dialog stays open/usable
- Polling intervals live on a persistent ref (not cleared on close) so
  toasts fire even after the dialog is dismissed
- archiveId stored per item at submit time; page-refresh reconnect uses
  it.archiveId instead of the possibly-null prop
- Completed rows flash green then self-remove; failed rows show inline
  error + retry button
- Cancel becomes Close while jobs are in flight

ToastStack (new component):
- Fixed bottom-right overlay with spring-in animation
- Error toast: truncated locator, View error / Hide toggle expanding
  full error_text in a monospace pre block
- Auto-dismisses after 7 s; timer pauses while detail is expanded

RunsView:
- Failed rows are clickable and expand a full-width detail row showing
  error_summary in a scrollable monospace block

capture.rs (archivr-core):
- Staging dir is now "{millis}-{uuid}" — parallel captures in the same
  millisecond can no longer collide on temp paths
- create_archive_run moved before URL Content-Type probe so every
  attempt appears in /runs regardless of outcome
- Probe failures now call create_archive_run_item with source_metadata
  fallback then fail_run, recording error_text on the item and
  error_summary on the run with correct failed_count

styles.css:
- Capture dialog: header row, multi-row layout, status dots, spinner,
  add-row dashed button, per-row error text
- Toast stack: fixed overlay, error card with coloured left border,
  monospace detail expansion
- Run error rows: clickable hover tint, expand hint chevron, detail pre
2026-07-05 13:56:00 +02:00
d83faab8e6
fix: correct in_progress run status badge color and label 2026-07-04 15:15:46 +02:00
ed1f883ff1
feat: implement entry deletion
- database: cascade_cached_bytes_after_subtree_delete — one-pass set-aware
  SQL that excludes the entire subtree simultaneously, avoiding sibling-blob
  cross-counting bug
- database: delete_entry — collects subtree IDs, runs set-aware cascade,
  NULLs archive_run_items.produced_entry_id (FK blocker), deletes children
  then root; ON DELETE CASCADE handles artifacts/tags/collections
- server: DELETE /api/archives/:archive_id/entries/:entry_uid route,
  wrapped in a transaction for atomicity
- frontend: deleteEntry API call, handleEntryDeleted callback in App.jsx
  (optimistic list removal + selection clear), Delete entry button in
  ContextRail with confirm guard
- tests: 4 database tests (unknown uid, subtree removal, run_item nulling,
  cached_bytes recalculation) + 3 route tests (204+gone, 404, 401)
2026-07-04 13:46:19 +02:00
cad3ae9885
feat: truncate entry URLs with expand on hover or selection 2026-07-04 12:00:34 +02:00
f71be25a8e
feat: reorder nav tabs to archive, tags, collections, runs, admin, settings 2026-07-04 11:46:11 +02:00
98238cd1f3
fix: decode HTML entities in entry titles 2026-07-04 11:42:33 +02:00
d3a78da766
ci: weekly yt-dlp auto-update workflow 2026-07-03 18:28:18 +02:00
e181fe4c40
chore: nix flake update — yt-dlp 2026.03.17 → 2026.06.09 2026-07-03 18:17:18 +02:00
085742e934
fix: restore yt-dlp metadata title fetch in perform_capture
8082895 removed the ytdlp_metadata_json fetch, local_filename_title
derivation, and entry_title computation, replacing the title arg with
a None stub. Restores all three blocks so YouTube, Instagram, Reddit,
TikTok, Facebook, Snapchat, X, and local file entries receive proper
titles again.
2026-07-03 18:13:23 +02:00
8b2e8172c3
fix: emit yt-dlp stderr on fetch_metadata failure for debuggability 2026-07-03 18:03:22 +02:00
bda548e91a
docs: add AGENTS.md repository guide for coding agents 2026-07-03 15:40:49 +02:00
55f85134df
feat: tag delete, rename, and per-user humanize-tags display setting (#16)
* feat: add tag delete and rename (backend + frontend)

- DELETE /api/archives/:id/tags/:tag_uid — deletes tag subtree via
  recursive CTE; entry_tag_assignments cascade automatically
- PATCH  /api/archives/:id/tags/:tag_uid { name } — renames a tag
  segment (case-preserving slug), cascades full_path to all descendants
  in one transaction using hierarchy CTE (not LIKE), returns updated Tag
- database: rename_tag + delete_tag with 7 unit tests covering subtree
  cascade, collision detection, descendant path rewrite, slug stripping
- TagsView: inline rename (pencil icon / double-click → input), × delete
  with confirmation dialog (warns about child tags)
- App.jsx: handleTagRenamed rewrites tagFilter for exact + descendant
  paths; handleTagDeleted clears filter for deleted subtree
- api.js: renameTag (PATCH, returns Tag) + deleteTag (DELETE)
- ContextRail: tag pills show displayPath(full_path) (humanized) with
  raw full_path in title tooltip; humanize-tags setting coming next

* feat: per-user humanize-tags display setting

- auth DB: idempotent migration adds users.humanize_slugs INTEGER DEFAULT 0
- GET /api/auth/me: returns humanize_slugs bool
- PATCH /api/auth/me: accepts { humanize_slugs: bool }, persisted via
  database::update_user_humanize_slugs
- displayPath() helper moved to utils.js (was local to ContextRail)
- TagsView: node label shows tag.name vs tag.slug based on humanizeTags
- ContextRail: pill label applies displayPath conditionally
- App.jsx: derives humanizeTags from currentUser.humanize_slugs, passes
  to TagsView + ContextRail; filter badge label humanized when on
- Settings > Profile: Display Preferences section with checkbox toggle,
  updates backend and currentUser immediately via setCurrentUser
- api.js: patchMe(patch) for generic PATCH /api/auth/me
- Tests: auth_me default=false, PATCH persists=true (2 route tests)
2026-07-03 14:26:45 +02:00
d6b52ba06c
fix: capture popup non-persistance
- Save and restore dialog open/closed state in sessionStorage (App.jsx)
- Persist form data: locator, error, busy, jobStatus, jobUid (CaptureDialog.jsx)
- Auto-resume polling if capture job was in progress before page refresh
- Only clear form on fresh user click, not when restoring from refresh
- Clean up sessionStorage when capture completes successfully

Fixes: Capture pop-up disappears on page refresh with unsaved data
2026-07-03 13:31:46 +02:00
2502de45b6
feat: inline entry title renaming (#14)
* feat: inline entry title renaming

- database.rs: add update_entry_title(conn, entry_uid, title) -> Result<bool>
  targets archived_entries table; returns false for unknown uid
- routes.rs: PATCH /api/archives/:archive_id/entries/:entry_uid (ROLE_USER)
  maps false -> 404; 3 tests (401 unauthed, 204+persists, 404 unknown uid)
- api.js: updateEntryTitle(archiveId, entryUid, title)
- ContextRail.jsx: click title h2 -> inline input; Enter/blur commits,
  Escape cancels (cancel ref prevents double-save on blur after Escape);
  pencil icon fades in on hover as edit affordance
- App.jsx: handleEntryTitleChange mutates entries + selectedEntry in-place
- styles.css: rail-title--editable cursor/hover, pencil icon show/hide,
  rail-title-input underline style
- rebuild frontend bundle

* chore: remove + prefix from Capture button label
2026-07-03 13:01:34 +02:00
1ff91956a1
docs: verify and update milestone checklist
- Tick completed features: Archiving, Management, Web-based UI
- Add sub-items for YouTube Playlists and Channels (already supported)
- Add granular UI feature sub-items (auth, setup, browse, tags, search, runs, capture, settings, admin)
- Mark Categorization as unchecked (tags exist but not a separate system)
- Backup/Sync and Email archiving remain unimplemented
2026-07-03 12:19:02 +02:00
90d17c9d25
feat: precompute cached_bytes per entry in DB (#13)
Store how many bytes of each entry's artifacts are already on disk from
an earlier entry (content-addressed blob deduplication means shared
blobs are only stored once).

Design
------
- Add `cached_bytes INTEGER NOT NULL DEFAULT 0` to `archived_entries`
- Precompute at capture time via `database::refresh_entry_cached_bytes`
  called after all artifacts are saved for every capture path
  (web page, generic URL, tweet, yt-dlp/local)
- One-time migration in `initialize_schema`: detects missing column via
  PRAGMA table_info, ALTERs the table, then back-fills all existing rows
  with the correlated subquery
- `database::cascade_cached_bytes_after_delete` ready for when entry
  deletion is implemented; designed to run asynchronously after the
  delete is acknowledged to the user
- `cached_bytes` included in `EntrySummary` and all four SELECT paths
  (list_root_entries, search_entries, list_entries_for_collection,
  entries_for_tag) via the shared ENTRY_SELECT_COLS constant

Frontend
--------
- `EntryRow` shows a `% cached` sub-line under the size when non-zero,
  with a tooltip showing the raw cached byte count
- No separate API endpoint or extra fetch — value rides in the existing
  entries list response at zero extra query cost per read
2026-07-03 11:56:55 +02:00
c85cce579b
fix: sync URL to active view and settings sub-tab 2026-07-01 19:11:42 +02:00
44f67390be
fix: sync URL to active view and settings sub-tab 2026-07-01 19:11:33 +02:00
c4f1c7a137
fix: use supported --browser-wait-until=networkidle2 for single-file-cli (reverts invalid networkAlmostIdle from previous commit) 2026-07-01 14:10:54 +02:00
2414acf0df
chore: add Dockerfile, docker-compose, and Docker hosting docs (#12)
* chore: add Dockerfile, docker-compose, and Docker docs

- Multi-stage Dockerfile: Rust builder stage + debian:bookworm-slim runtime
  with Chromium, Node/single-file-cli, Python venv (yt-dlp + twitter-api-client)
- docker-compose.yml: wires ARCHIVR_BIND, config volume, and persistent data volume
- docker/config.example.toml: annotated TOML template for Docker deployments
- docs/README.md: add Hosting with Docker section; add ARCHIVR_BIND and
  ARCHIVR_STATIC_DIR to the Environment Variables reference

* fix: address code review issues with Docker setup

- .gitignore: whitelist Dockerfile, docker-compose.yml, docker/ so they
  are actually tracked (the * catch-all was silently dropping them)
- Dockerfile: build and ship the archivr CLI alongside archivr-server so
  users can run `archivr init` inside the container on first setup
- docker/config.example.toml: fix archive_path to point at the .archivr
  subdirectory that archivr init creates (not the parent directory), which
  is what read_archive_paths expects
- docs/README.md: replace the bare mkdir quickstart step with
  `archivr init`, explain why mkdir is insufficient; add a callout that
  auth_db_path must be set explicitly to a writable path when the config
  mount is read-only

* fix: address second round of Docker review issues

Chromium sandbox (P2):
- singlefile.rs: add ARCHIVR_CHROME_ARGS env var (space-separated flags
  appended to Chromium's --browser-args JSON array); Dockerfile sets it
  to --no-sandbox because Chromium refuses to start as root without it

Store-path outside volume (P1):
- README: pass explicit absolute store-path as the second positional arg
  to `archivr init` so the blob store lands on /data instead of the
  container layer (CLI default is ./.archivr/store, resolved from cwd,
  which is / with no WORKDIR set)

ENTRYPOINT vs CMD (P2):
- Dockerfile: switch from ENTRYPOINT to CMD so `docker compose run
  archivr archivr init …` overrides the full command instead of being
  appended to the server invocation

ffmpeg missing (P2):
- Dockerfile: add ffmpeg to the apt-get install block (required by
  yt-dlp --merge-output-format mp4 for bestvideo+bestaudio streams)

Node version (P2):
- Dockerfile: replace Debian bookworm's nodejs (18.x) with Node 20 via
  the NodeSource setup script (single-file-cli declares engines.node >=20)

Build context secrets (P2):
- Add .dockerignore excluding config/ and docker/ from the build context
  so runtime secrets (e.g. twitter-cookies.txt) are never sent to the builder
- Whitelist .dockerignore in .gitignore

docs:
- README: document ARCHIVR_CHROME_ARGS in the Environment Variables section

* fix: third round of Docker review issues

Rust toolchain (P1):
- Dockerfile: bump builder from rust:1.87 to rust:1.88; time@0.3.51,
  time-core@0.1.9, and time-macros@0.2.30 (present in Cargo.lock) all
  require MSRV 1.88, so the real cargo build --release step was failing

single-file-cli wait mode (P2):
- singlefile.rs: replace --browser-wait-until=networkidle2 with
  networkAlmostIdle; the single-file-cli option only accepts
  InteractiveTime/networkIdle/networkAlmostIdle/load/domContentLoaded
  (verified in options.js); networkidle2 is a Puppeteer concept that the
  CLI does not recognise, causing silent fallback to the earliest state
  and incomplete captures. networkAlmostIdle is the closest equivalent
  (<=2 open connections, matching Puppeteer's networkidle2 semantics)

Build context size (P3):
- .dockerignore: add target/, frontend/node_modules/, frontend/dist/;
  these can reach 1.4G+ after a local dev build and are never read by
  the Dockerfile, so sending them to the builder wastes time and memory
2026-06-30 14:35:43 +02:00
685b6cc7ea
feat: self-hosting - NixOS module, security hardening, multi-archive docs (#11)
feat: self-hosting - NixOS module, security hardening, multi-archive docs
2026-06-30 00:43:00 +02:00
120eccc032
chore: rebuild frontend bundle
JS bundle rehashed after frontend changes (App.jsx auth gating,
ContextRail full tag path, Topbar tab reorder, Capture button copy).
2026-06-30 00:39:55 +02:00
9544e707ed
fix: address 6 code-review findings
NixOS module (4 fixes):
- Remove RestrictNamespaces=true: Chromium (launched by single-file for
  web captures) needs Linux user namespaces; blocking them broke captures
- Escape TOML string values: labels/paths with quotes or backslashes
  would produce invalid archivr-server.toml and prevent startup
- Bracket IPv6 listen addresses: ::1:8080 is rejected by Rust's
  SocketAddr parser; [::1]:8080 is the correct form (RFC 2732)
- Add optional storePath per archive: covers archives initialised with
  a custom store path outside the default sibling store/ directory

Rust (1 fix):
- extract_client_ip loopback branch: take last XFF value not first.
  When a proxy appends to X-Forwarded-For an attacker controls the
  first entry; the last entry is always set by the trusted local proxy

Frontend (1 fix):
- Gate fetchArchives on authState === 'authenticated'. The empty-dep
  useEffect fired on mount (before login), hit 401, and never retried
  after login because authState was not in the dependency array
2026-06-29 21:30:45 +02:00
71e176cbd1
security: fix two Codex review findings + add port option
Fix 1 — NixOS: store/ dir missing from ReadWritePaths
  archive_path points to .archivr/; captures write artifacts to the
  sibling store/ directory. Whitelisting only .archivr/ caused POST
  /captures to fail under ProtectSystem=strict.
  Fix: use builtins.dirOf a.path to whitelist the parent, which
  contains both .archivr/ and store/.

Fix 2 — Rust: X-Forwarded-For was unconditionally trusted
  An attacker could send a different IP on every login attempt,
  bypassing LOGIN_MAX_ATTEMPTS entirely.
  Fix: use ConnectInfo<SocketAddr> (via into_make_service_with_connect_info)
  as the primary rate-limit key; XFF is trusted only when the TCP peer
  is loopback (i.e. a local reverse proxy). Tests without a real socket
  fall back to XFF unchanged.

Port configuration:
  NixOS module: split 'bind' string into separate 'listenAddress'
  (default 127.0.0.1) and 'port' (default 8080) options. openFirewall
  now uses cfg.port directly instead of parsing it from the bind string.
  README NixOS example updated accordingly.
2026-06-29 20:32:08 +02:00
7ed7cb882f
chore: update Cargo.lock after parking_lot dep addition 2026-06-29 20:13:59 +02:00
5ce68b58f3
feat(server): require auth on list_entries and search_entries_handler
Both handlers accepted AuthUser but served guests.
auth.require_auth()? added as first line closes the gap.
Existing unauthenticated tests updated with session cookies.
2026-06-29 20:13:31 +02:00
007c5fc5fb
feat(server): enforce auth on all public read endpoints
Add auth_user: AuthUser + require_auth() to entry_detail, list_runs,
serve_artifact, serve_entry_favicon, serve_blob, list_tags,
list_entry_tags, list_collections_handler, list_entry_collections_handler.
Add require_auth() to get_collection_handler (already had extractor).

Update existing tests to pass session cookies.
2026-06-29 20:12:00 +02:00
a8e52bdf71
security: add per-IP sliding-window rate limit on POST /api/auth/login
5 attempts / 15 min per IP. Excess attempts get 429 + Retry-After.
Client IP read from X-Forwarded-For with loopback fallback.
In-memory; prune inline on each check.
Refactored app() into app_with_state() + thin wrapper.
Uses parking_lot::Mutex (no unwrap, no poisoning).
2026-06-29 20:06:41 +02:00
3b3b0d7207
feat(server): add HTTP security response headers middleware
Injects X-Content-Type-Options, X-Frame-Options, Referrer-Policy,
Content-Security-Policy, and Permissions-Policy on every response.
Registered as outermost layer so it covers setup_guard 503s too.
No new dependencies.
2026-06-29 20:03:44 +02:00
f9de282b0e
feat(nix): add NixOS module for archivr-server
modules/nixos/archivr-server.nix — services.archivr-server NixOS module:
- enable/bind/archives/user/group/openFirewall options
- generates archivr-server.toml from options via pkgs.writeText
- pins auth DB to /var/lib/archivr-server/ (StateDirectory)
- dedicated archivr system user + group
- hardened systemd unit: ProtectSystem=strict, NoNewPrivileges,
  PrivateTmp, RestrictNamespaces, etc.; archive paths whitelisted
- openFirewall parses the port from the bind string automatically

flake.nix:
- add self to outputs args
- add aarch64-linux to systems (Raspberry Pi / ARM servers)
- expose nixosModules.archivr-server + nixosModules.default

.gitignore: whitelist modules/ directory

docs/README.md: add 'Hosting on NixOS' section with full example
2026-06-29 16:51:20 +02:00
4311e85f95
ui: comprehensive UI polish pass
- LoginPage/SetupPage: centered card layout with display font, styled fields and submit button
- Topbar: fix duplicate Settings nav item; add styled user-menu with username + logout button
- RunsView: format ISO timestamps to readable dates, add colored status badges (completed/failed/running)
- AdminView: view-tabs system, styled admin-table, admin-input, status-badge (active/disabled), btn-primary
- SettingsView: replace all inline styles with form-section/form-field/field-input/btn-primary/btn-danger
- CollectionsView: restructure create form with proper field labels and btn-primary
- TagsView: add Tags section heading with separator
- ContextRail: show visibility as human-readable label not raw number; fix assign-error class
- styles.css: add auth-loading, view-tabs, form utilities, btn variants, status badges,
  run-status pills, token-banner/row, checkbox-row, coll-create-form, tag-tree-header CSS
2026-06-28 22:14:08 +02:00
3ccfcce87b
feat(ui): add Storybook design system refinements 2026-06-28 21:01:50 +02:00
46ad816c4c
fix(flake): add openssl to buildInputs for Rust packages (#10)
Garnix CI was failing because openssl-sys couldn't find the OpenSSL
library. Added pkgs.openssl to buildInputs for both archivr_cli_unwrapped
and archivr_server_unwrapped to provide the necessary system dependency
and allow pkg-config to locate openssl.pc.
2026-06-27 11:54:31 +02:00
91e73d85e7
feat(collections): Track 8 — collection rename, delete, add/remove entries, per-entry visibility UI 2026-06-26 17:33:58 +02:00
ae92448fa3
build(frontend): bundle assets for collections UI 2026-06-26 17:33:50 +02:00
80dce80033
feat(collections): frontend — CollectionsView rename, delete, add/remove entries, per-entry visibility 2026-06-26 17:33:46 +02:00
c4ce2129bb
feat(collections): frontend — api.js updateCollection and deleteCollection helpers 2026-06-26 17:33:42 +02:00
383cec1f81
feat(collections): routes — PATCH|DELETE /api/archives/:id/collections/:uid, visibility_bits in GET response 2026-06-26 17:33:38 +02:00
533e9d5d2e
feat(collections): db — update_collection, delete_collection helpers 2026-06-26 17:33:35 +02:00
96eaf50609
docs(settings): mark Track 7 done in NEXT.md 2026-06-26 17:20:06 +02:00
d558ce2a3f
feat(settings): Track 7 — account profile, password change, API tokens UI, instance settings
- db: display_name column migration, InstanceSettings struct, 6 new pub helpers
  (get/update_instance_settings, update_user_display_name, update_user_password,
  get_user_password_hash, get_user_display_name)
- routes: PATCH /api/auth/me (display name + password change, current-password verify);
  GET|PATCH /api/admin/instance-settings (ROLE_ADMIN); auth/me now returns display_name
- frontend/api.js: updateProfile, changePassword, listTokens, createToken,
  deleteToken, getInstanceSettings, updateInstanceSettings helpers
- frontend: SettingsView (Profile / API Tokens / Instance tabs), settings nav in
  Topbar, App routing for view === 'settings'
- 7 new routes tests; 176 tests green
2026-06-26 17:19:43 +02:00
ea0a8f80d1
test(settings): routes tests for profile update, password change, instance settings 2026-06-26 17:19:26 +02:00
44f18f4dcc
build(frontend): bundle assets for settings UI 2026-06-26 17:18:30 +02:00
78e763c377
feat(settings): frontend — SettingsView component, settings nav, App routing 2026-06-26 17:17:45 +02:00