1
Fork 0
mirror of https://github.com/thegeneralist01/archivr synced 2026-07-22 03:05:32 +02:00

feat: uBlock Origin Lite + cookie consent extension + reader mode + ad placeholder cleanup (#21)

* feat: uBlock Origin Lite integration for ad-blocking during WebPage captures

- singlefile.rs: when ARCHIVR_UBLOCK=true and ARCHIVR_UBLOCK_EXT is set,
  archivr owns Chrome's lifecycle (--headless=new, --remote-debugging-port,
  --load-extension); single-file connects via --browser-server instead of
  launching its own Chrome. Falls back to old behaviour with ublock_skipped=true
  when the ext path is missing or invalid.
- capture.rs: thread ublock_skipped through CaptureResult
- database.rs: add notes_json TEXT column to capture_jobs (DDL + idempotent
  ALTER TABLE migration); update_capture_job_status gains notes_json param
- archive.rs: expose notes_json in CaptureJobSummary
- routes.rs: store {"ublock_skipped":true} in notes_json on completed captures
- ToastStack.jsx: warning toast variant (toast--warning) with Details expander
  and Ignore button
- CaptureDialog.jsx: fire warning toast when poll result has ublock_skipped
- App.jsx: sessionStorage-backed Ignore suppression for ublock warnings
- styles.css: .toast--warning (amber left border) + .toast-warning-detail
- flake.nix: ublockLite derivation fetches uBOLite_2026.705.2152.chromium.zip
  (pinned SHA256) from uBlockOrigin/uBOL-home; sets ARCHIVR_UBLOCK_EXT in both
  archivr and archivr-server wrappers

Env vars:
  ARCHIVR_UBLOCK=true (default) — enable uBlock during WebPage captures
  ARCHIVR_UBLOCK_EXT — path to unpacked uBOL extension dir (set by Nix)

* feat: Extensions settings tab + capture dialog redesign with Advanced options

Settings/Extensions tab (admin-only):
- New 'Extensions' tab between Cookies and Storage
- ExtensionsTab component: shows uBlock Origin Lite card with pill toggle
- Reads ublock_enabled from instance settings; patch via existing PATCH endpoint
- Shows ublock_ext_available status from server (whether ARCHIVR_UBLOCK_EXT is set)

Instance settings:
- Add ublock_enabled BOOLEAN (default true) to instance_settings auth DB table
- Idempotent ALTER TABLE migration in initialize_auth_schema()
- get/update_instance_settings include ublock_enabled
- GET /api/admin/instance-settings now also returns ublock_ext_available (computed
  from ARCHIVR_UBLOCK_EXT env var at request time)
- PATCH /api/admin/instance-settings accepts ublock_enabled

Per-capture override:
- CaptureBody gains ublock_enabled: Option<bool>
- CaptureConfig gains ublock_enabled: Option<bool>
- singlefile::save() gains ublock_enabled_override: Option<bool> param
- Capture handler resolves: body override > global instance setting > env var
- submitCapture(aid, loc, qual, extensions) in api.js passes ublock_enabled

Capture dialog redesign:
- Archive button: full-width, 13px padding, min-width 220px, primary CTA
- Cancel: full-width but text-style, below Archive
- ‹Advanced options› chevron toggle (rotates on open)
- Expanded panel shows uBlock toggle for this capture session
- Loads global ublock_enabled default from instance settings on mount

Styles:
- .ext-toggle pill switch (44×24 and 36×20 small variant)
- .ext-card for Settings Extensions tab
- .capture-advanced + .capture-advanced-panel + .capture-chevron
- .capture-ext-row / .capture-ext-label / .capture-ext-name / .capture-ext-desc
- .form-hint utility class

* fix: remove ublock_enabled from INSERT OR IGNORE in DDL batch

The INSERT ran before the ALTER TABLE migration added the column,
causing 'table instance_settings has no column named ublock_enabled'
on existing databases. The INSERT OR IGNORE for the default row only
needs the original columns; the migration's DEFAULT 1 handles the
new column for existing and new rows alike.

* feat: Reader mode via Mozilla Readability.js

Adds an opt-in 'Reader mode' advanced option to the capture dialog.
When enabled, Readability.js is injected as a browser script during
SingleFile capture; it fires on single-file-on-before-capture-start,
replaces the page body with the distilled article content, injects a
clean typographic stylesheet, and adds a header with title/byline/site.
Falls back silently if Readability fails (e.g. non-article pages).

- vendor/readability/Readability.js  Apache 2.0, Mozilla, v0.6.0
- singlefile.rs: embed READABILITY_JS + READER_MODE_WRAPPER_JS via
  include_str!; write both to temp dir when reader_mode is true;
  base_single_file_cmd now accepts &[&Path] for multiple --browser-script
- capture.rs: CaptureConfig.reader_mode: bool
- routes.rs: CaptureBody.reader_mode: Option<bool> (defaults false)
- api.js: submitCapture passes reader_mode in payload
- CaptureDialog.jsx: Reader mode toggle in Advanced options (off by default)

* fix: diagnose single-file no-output-file error + prevent stdout dumping

- Add --dump-content=false to every single-file invocation to prevent
  the Docker-detection heuristic from routing HTML to stdout instead of
  the output file (the heuristic can trigger in some macOS environments)
- Improve the no-output-file error message to include: temp dir contents,
  stderr, and first 200 chars of stdout — this gives enough context to
  diagnose any remaining cause without re-running

* fix: switch uBlock loading from --browser-server to --browser-args

The --browser-server (CDP) path caused 'Unexpected server response: 404'
on macOS Chrome because simple-cdp's WebSocket upgrade to the debugger
endpoint failed after Chrome started — likely a version-specific CDP
endpoint shape mismatch.

New approach: single-file always manages Chrome. When ARCHIVR_UBLOCK_EXT
is set, --headless=new, --load-extension, and --disable-extensions-except
are injected via --browser-args. single-file's browser.js prefix-strips
its own conflicting flags before appending ours, so --headless=new
overrides the default --headless (enabling extension support in headless).

Removes allocate_free_port, wait_for_chrome_ready, run_single_file_with_server
(all dead code now). Docblock updated to reflect actual behaviour and notes
the --single-process caveat: uBOL's declarativeNetRequest static rulesets
are expected to work (network-stack level, not service-worker), but this
has not been mechanically verified under --single-process.

Smoke tested on macOS (this machine): capture with --load-extension + all
three browser-scripts (strip, Readability, reader-mode wrapper) produces
output file correctly. Ad-blocking verification deferred to manual test
with a tracker-heavy URL.

* fix: use correct single-file hook event (single-file-on-before-capture-request)

Prior scripts listened on 'single-file-on-before-capture-start' which
does not exist in single-file-core 1.1.49.  The real hook is:

  single-file-on-before-capture-request  (dispatched by initUserScriptHandler
  after receiving single-file-user-script-init; userScriptEnabled defaults
  to true in args.js so it always fires when --browser-script is passed)

Changes:
- strip-scripts: -start -> -request (no preventDefault needed; synchronous)
- READER_MODE_SCRIPT: -start -> -request; add 'installed' meta marker at
  script-evaluation time so artifact inspection can distinguish 'script
  not injected' / 'hook never fired' / 'Readability parse failed'

* fix: correct singlefile.rs docstring (scripts.js concatenates, not isolates)

* fix: dispatch single-file-user-script-init so request hook fires

single-file's initUserScriptHandler (in single-file-bootstrap.js) listens
for 'single-file-user-script-init' and only then installs
_singleFile_waitForUserScript.  Without that dispatch our scripts'
'single-file-on-before-capture-request' listeners were never reached,
so neither strip-scripts nor reader-mode Readability applied.

Dispatch the init event at the top of strip-scripts (always present) and
redundantly in READER_MODE_SCRIPT.  Verified end-to-end: artifact for
run_b3181d6d276e4e56a1a6c356ef9bbe8f has
  meta content="applied", max-width:680px CSS, 0 script tags.

* feat: cookie consent extension support (ARCHIVR_COOKIE_EXT)

Mirrors the uBlock Origin Lite integration exactly:

Backend:
- singlefile.rs: resolve_cookie_ext_config() reads ARCHIVR_COOKIE_CONSENT
  (default true) + ARCHIVR_COOKIE_EXT path; extension paths comma-joined
  into --load-extension / --disable-extensions-except so uBlock and cookie
  ext can coexist; SaveResult.cookie_ext_skipped tracks miss
- database.rs: cookie_ext_enabled column on instance_settings (DEFAULT 1);
  idempotent ALTER TABLE migration; get/update wired through
- capture.rs: CaptureConfig.cookie_ext_enabled: Option<bool>; threaded to
  singlefile::save(); cookie_ext_skipped surfaced in CaptureResult
- routes.rs: CaptureBody + UpdateInstanceSettingsBody get cookie_ext_enabled;
  capture handler resolves effective value (body overrides global); notes_json
  only includes skipped fields that are true; GET instance-settings includes
  cookie_ext_available from env path check

Frontend:
- api.js: submitCapture forwards cookie_ext_enabled
- SettingsView.jsx: 'I Still Don't Care About Cookies' card in Extensions
  tab; always-active toggle (user can disable even when ext not installed);
  amber 'Not configured' hint + ARCHIVR_COOKIE_EXT guidance when unavailable
- CaptureDialog.jsx: 'Block cookie banners' toggle in Advanced options;
  always shown with amber hint when ext not configured; defaults from
  global setting

Operator setup: download + unzip the extension from GitHub releases, set
ARCHIVR_COOKIE_EXT=/path/to/unpacked/ext. No Node daemon needed.

* fix: surface cookie_ext_skipped warning toast in CaptureDialog

* feat: package istilldontcareaboutcookies in flake, wire ARCHIVR_COOKIE_EXT

Add isdcac derivation mirroring ublockLite:
- Fetches ISDCAC-chrome-source.zip v1.1.9 from GitHub releases
- Validates manifest.json at extension root before install (guard against
  nested-folder zip regressions in future releases)
- Sets ARCHIVR_COOKIE_EXT in both archivr and archivr_server wrappers

Verified: nix build .#archivr-server and .#archivr both succeed;
wrapper scripts export correct store paths; manifest.json present at root.

* fix: gate consent-overlay cleanup on cookie_ext; reset overflow; narrow selectors

- Strip overflow:hidden from body/html only when cookie_ext is active for
  the capture — prevents mutating legitimate pages when the feature is off
- Remove .fc-dialog (Google Funding Choices), .qc-cmp2-*, .sp-message-container,
  #sp-cc, #usercentrics-root as fallback for CMPs the extension misses
- Removed overbroad [class^="uc-"] and [id^="usercentrics"] selectors
  that could match real page content

* fix: remove ad placeholders when uBlock active; kept height causes blank gap

uBlock Origin Lite blocks ad network requests but first-party placeholder
elements (ins.adsbygoogle, #aswift_* iframe hosts) retain their computed
height (e.g. 280px for a top banner), leaving a large blank space at the
top of captured pages.

Gate cleanup on ublock_ext.is_some(): remove ins.adsbygoogle, aswift_*
iframes, and google_ads_* iframes before SingleFile serialises. Also
collapse the parent container if it becomes empty after removal.

* fix: walk up to .top-ad/.google-auto-placed ancestor before removing ad slot

Removing only the inner ins.adsbygoogle left the outer .container.top-ad
wrapper (with pb-4 padding) in the layout, preserving the blank gap.
Now walk up via closest() to the nearest ad-slot container class before
removal so the whole slot including padding collapses.
This commit is contained in:
TheGeneralist 2026-07-08 23:26:48 +02:00 committed by GitHub
parent dae61e585d
commit 2e8820a0da
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
18 changed files with 4003 additions and 266 deletions

View file

@ -650,9 +650,11 @@ async fn delete_entry_handler(
#[derive(Debug, serde::Deserialize)]
struct CaptureBody {
locator: String,
/// Optional quality cap for yt-dlp sources: `"best"` or any `"NNNp"` string
/// (e.g. `"1080p"`, `"720p"`, `"2160p"`). Absent or `"best"` → highest available.
quality: Option<String>,
ublock_enabled: Option<bool>,
/// Distil to article content via Readability before archiving. Absent = false.
reader_mode: Option<bool>,
cookie_ext_enabled: Option<bool>,
}
#[derive(Debug, serde::Deserialize)]
@ -732,15 +734,29 @@ async fn capture_handler(
let conn = database::open_or_initialize(&mounted.archive_path)?;
let job_uid = database::create_capture_job(&conn, &archive_id)?;
drop(conn);
// Load cookie rules from the auth DB to pass into the capture background task.
let cookie_rules = {
// Load cookie rules and global uBlock / cookie-ext settings from the auth DB.
let (cookie_rules, global_ublock, global_cookie_ext) = {
match database::open_auth_db(&state.auth_db_path) {
Ok(conn) => database::list_cookie_rules(&conn).unwrap_or_default(),
Err(_) => vec![],
Ok(conn) => {
let rules = database::list_cookie_rules(&conn).unwrap_or_default();
let settings = database::get_instance_settings(&conn);
let ublock = settings.as_ref().map(|s| s.ublock_enabled).unwrap_or(true);
let cookie_ext = settings.map(|s| s.cookie_ext_enabled).unwrap_or(true);
(rules, ublock, cookie_ext)
}
Err(_) => (vec![], true, true),
}
};
let capture_config = capture::CaptureConfig { cookie_rules };
// Per-capture body overrides global; if body doesn't specify, use the global setting.
// The resolved bool is then passed as Some(_) to singlefile, overriding the env var.
let effective_ublock = body.ublock_enabled.unwrap_or(global_ublock);
let effective_cookie_ext = body.cookie_ext_enabled.unwrap_or(global_cookie_ext);
let capture_config = capture::CaptureConfig {
cookie_rules,
ublock_enabled: Some(effective_ublock),
cookie_ext_enabled: Some(effective_cookie_ext),
reader_mode: body.reader_mode.unwrap_or(false),
};
// Spawn background capture.
let locator = body.locator.trim().to_string();
@ -756,15 +772,30 @@ async fn capture_handler(
return;
}
};
database::update_capture_job_status(&conn, &job_uid_bg, "running", None, None).ok();
database::update_capture_job_status(&conn, &job_uid_bg, "running", None, None, None).ok();
match capture::perform_capture(&archive_paths, &locator, Some(&archive_id_bg), quality.as_deref(), &capture_config) {
Ok(result) => {
let mut notes_map = serde_json::Map::new();
if result.ublock_skipped {
notes_map.insert("ublock_skipped".into(), serde_json::Value::Bool(true));
}
if result.cookie_ext_skipped {
notes_map.insert("cookie_ext_skipped".into(), serde_json::Value::Bool(true));
}
let notes_str;
let notes: Option<&str> = if notes_map.is_empty() {
None
} else {
notes_str = serde_json::Value::Object(notes_map).to_string();
Some(&notes_str)
};
database::update_capture_job_status(
&conn,
&job_uid_bg,
"completed",
Some(&result.run_uid),
None,
notes,
)
.ok();
}
@ -775,6 +806,7 @@ async fn capture_handler(
"failed",
None,
Some(&format!("{e:#}")),
None,
)
.ok();
}
@ -1013,10 +1045,26 @@ async fn patch_me(
async fn get_instance_settings_handler(
State(state): State<AppState>,
auth_user: AuthUser,
) -> Result<Json<database::InstanceSettings>, ApiError> {
) -> Result<Json<serde_json::Value>, ApiError> {
auth_user.require_role(ROLE_ADMIN)?;
let conn = database::open_auth_db(&state.auth_db_path)?;
Ok(Json(database::get_instance_settings(&conn)?))
let settings = database::get_instance_settings(&conn)?;
let ublock_ext_available = std::env::var("ARCHIVR_UBLOCK_EXT")
.ok()
.filter(|s| !s.is_empty())
.map(|p| std::path::Path::new(&p).is_dir())
.unwrap_or(false);
let cookie_ext_available = std::env::var("ARCHIVR_COOKIE_EXT")
.ok()
.filter(|s| !s.is_empty())
.map(|p| std::path::Path::new(&p).is_dir())
.unwrap_or(false);
let mut val = serde_json::to_value(&settings).unwrap_or_default();
if let Some(obj) = val.as_object_mut() {
obj.insert("ublock_ext_available".into(), serde_json::Value::Bool(ublock_ext_available));
obj.insert("cookie_ext_available".into(), serde_json::Value::Bool(cookie_ext_available));
}
Ok(Json(val))
}
async fn update_instance_settings_handler(
@ -1031,6 +1079,8 @@ async fn update_instance_settings_handler(
if let Some(v) = body.public_entry_content_enabled { settings.public_entry_content_enabled = v; }
if let Some(v) = body.open_registration_enabled { settings.open_registration_enabled = v; }
if let Some(v) = body.default_entry_visibility { settings.default_entry_visibility = v; }
if let Some(v) = body.ublock_enabled { settings.ublock_enabled = v; }
if let Some(v) = body.cookie_ext_enabled { settings.cookie_ext_enabled = v; }
database::update_instance_settings(&conn, &settings)?;
Ok(StatusCode::NO_CONTENT)
}
@ -1360,6 +1410,8 @@ struct UpdateInstanceSettingsBody {
public_entry_content_enabled: Option<bool>,
open_registration_enabled: Option<bool>,
default_entry_visibility: Option<u32>,
ublock_enabled: Option<bool>,
cookie_ext_enabled: Option<bool>,
}
async fn admin_list_users(