1
Fork 0
mirror of https://github.com/thegeneralist01/archivr synced 2026-07-21 18:55:36 +02:00

feat: uBlock Origin Lite + cookie consent extension + reader mode + ad placeholder cleanup (#21)

* feat: uBlock Origin Lite integration for ad-blocking during WebPage captures

- singlefile.rs: when ARCHIVR_UBLOCK=true and ARCHIVR_UBLOCK_EXT is set,
  archivr owns Chrome's lifecycle (--headless=new, --remote-debugging-port,
  --load-extension); single-file connects via --browser-server instead of
  launching its own Chrome. Falls back to old behaviour with ublock_skipped=true
  when the ext path is missing or invalid.
- capture.rs: thread ublock_skipped through CaptureResult
- database.rs: add notes_json TEXT column to capture_jobs (DDL + idempotent
  ALTER TABLE migration); update_capture_job_status gains notes_json param
- archive.rs: expose notes_json in CaptureJobSummary
- routes.rs: store {"ublock_skipped":true} in notes_json on completed captures
- ToastStack.jsx: warning toast variant (toast--warning) with Details expander
  and Ignore button
- CaptureDialog.jsx: fire warning toast when poll result has ublock_skipped
- App.jsx: sessionStorage-backed Ignore suppression for ublock warnings
- styles.css: .toast--warning (amber left border) + .toast-warning-detail
- flake.nix: ublockLite derivation fetches uBOLite_2026.705.2152.chromium.zip
  (pinned SHA256) from uBlockOrigin/uBOL-home; sets ARCHIVR_UBLOCK_EXT in both
  archivr and archivr-server wrappers

Env vars:
  ARCHIVR_UBLOCK=true (default) — enable uBlock during WebPage captures
  ARCHIVR_UBLOCK_EXT — path to unpacked uBOL extension dir (set by Nix)

* feat: Extensions settings tab + capture dialog redesign with Advanced options

Settings/Extensions tab (admin-only):
- New 'Extensions' tab between Cookies and Storage
- ExtensionsTab component: shows uBlock Origin Lite card with pill toggle
- Reads ublock_enabled from instance settings; patch via existing PATCH endpoint
- Shows ublock_ext_available status from server (whether ARCHIVR_UBLOCK_EXT is set)

Instance settings:
- Add ublock_enabled BOOLEAN (default true) to instance_settings auth DB table
- Idempotent ALTER TABLE migration in initialize_auth_schema()
- get/update_instance_settings include ublock_enabled
- GET /api/admin/instance-settings now also returns ublock_ext_available (computed
  from ARCHIVR_UBLOCK_EXT env var at request time)
- PATCH /api/admin/instance-settings accepts ublock_enabled

Per-capture override:
- CaptureBody gains ublock_enabled: Option<bool>
- CaptureConfig gains ublock_enabled: Option<bool>
- singlefile::save() gains ublock_enabled_override: Option<bool> param
- Capture handler resolves: body override > global instance setting > env var
- submitCapture(aid, loc, qual, extensions) in api.js passes ublock_enabled

Capture dialog redesign:
- Archive button: full-width, 13px padding, min-width 220px, primary CTA
- Cancel: full-width but text-style, below Archive
- ‹Advanced options› chevron toggle (rotates on open)
- Expanded panel shows uBlock toggle for this capture session
- Loads global ublock_enabled default from instance settings on mount

Styles:
- .ext-toggle pill switch (44×24 and 36×20 small variant)
- .ext-card for Settings Extensions tab
- .capture-advanced + .capture-advanced-panel + .capture-chevron
- .capture-ext-row / .capture-ext-label / .capture-ext-name / .capture-ext-desc
- .form-hint utility class

* fix: remove ublock_enabled from INSERT OR IGNORE in DDL batch

The INSERT ran before the ALTER TABLE migration added the column,
causing 'table instance_settings has no column named ublock_enabled'
on existing databases. The INSERT OR IGNORE for the default row only
needs the original columns; the migration's DEFAULT 1 handles the
new column for existing and new rows alike.

* feat: Reader mode via Mozilla Readability.js

Adds an opt-in 'Reader mode' advanced option to the capture dialog.
When enabled, Readability.js is injected as a browser script during
SingleFile capture; it fires on single-file-on-before-capture-start,
replaces the page body with the distilled article content, injects a
clean typographic stylesheet, and adds a header with title/byline/site.
Falls back silently if Readability fails (e.g. non-article pages).

- vendor/readability/Readability.js  Apache 2.0, Mozilla, v0.6.0
- singlefile.rs: embed READABILITY_JS + READER_MODE_WRAPPER_JS via
  include_str!; write both to temp dir when reader_mode is true;
  base_single_file_cmd now accepts &[&Path] for multiple --browser-script
- capture.rs: CaptureConfig.reader_mode: bool
- routes.rs: CaptureBody.reader_mode: Option<bool> (defaults false)
- api.js: submitCapture passes reader_mode in payload
- CaptureDialog.jsx: Reader mode toggle in Advanced options (off by default)

* fix: diagnose single-file no-output-file error + prevent stdout dumping

- Add --dump-content=false to every single-file invocation to prevent
  the Docker-detection heuristic from routing HTML to stdout instead of
  the output file (the heuristic can trigger in some macOS environments)
- Improve the no-output-file error message to include: temp dir contents,
  stderr, and first 200 chars of stdout — this gives enough context to
  diagnose any remaining cause without re-running

* fix: switch uBlock loading from --browser-server to --browser-args

The --browser-server (CDP) path caused 'Unexpected server response: 404'
on macOS Chrome because simple-cdp's WebSocket upgrade to the debugger
endpoint failed after Chrome started — likely a version-specific CDP
endpoint shape mismatch.

New approach: single-file always manages Chrome. When ARCHIVR_UBLOCK_EXT
is set, --headless=new, --load-extension, and --disable-extensions-except
are injected via --browser-args. single-file's browser.js prefix-strips
its own conflicting flags before appending ours, so --headless=new
overrides the default --headless (enabling extension support in headless).

Removes allocate_free_port, wait_for_chrome_ready, run_single_file_with_server
(all dead code now). Docblock updated to reflect actual behaviour and notes
the --single-process caveat: uBOL's declarativeNetRequest static rulesets
are expected to work (network-stack level, not service-worker), but this
has not been mechanically verified under --single-process.

Smoke tested on macOS (this machine): capture with --load-extension + all
three browser-scripts (strip, Readability, reader-mode wrapper) produces
output file correctly. Ad-blocking verification deferred to manual test
with a tracker-heavy URL.

* fix: use correct single-file hook event (single-file-on-before-capture-request)

Prior scripts listened on 'single-file-on-before-capture-start' which
does not exist in single-file-core 1.1.49.  The real hook is:

  single-file-on-before-capture-request  (dispatched by initUserScriptHandler
  after receiving single-file-user-script-init; userScriptEnabled defaults
  to true in args.js so it always fires when --browser-script is passed)

Changes:
- strip-scripts: -start -> -request (no preventDefault needed; synchronous)
- READER_MODE_SCRIPT: -start -> -request; add 'installed' meta marker at
  script-evaluation time so artifact inspection can distinguish 'script
  not injected' / 'hook never fired' / 'Readability parse failed'

* fix: correct singlefile.rs docstring (scripts.js concatenates, not isolates)

* fix: dispatch single-file-user-script-init so request hook fires

single-file's initUserScriptHandler (in single-file-bootstrap.js) listens
for 'single-file-user-script-init' and only then installs
_singleFile_waitForUserScript.  Without that dispatch our scripts'
'single-file-on-before-capture-request' listeners were never reached,
so neither strip-scripts nor reader-mode Readability applied.

Dispatch the init event at the top of strip-scripts (always present) and
redundantly in READER_MODE_SCRIPT.  Verified end-to-end: artifact for
run_b3181d6d276e4e56a1a6c356ef9bbe8f has
  meta content="applied", max-width:680px CSS, 0 script tags.

* feat: cookie consent extension support (ARCHIVR_COOKIE_EXT)

Mirrors the uBlock Origin Lite integration exactly:

Backend:
- singlefile.rs: resolve_cookie_ext_config() reads ARCHIVR_COOKIE_CONSENT
  (default true) + ARCHIVR_COOKIE_EXT path; extension paths comma-joined
  into --load-extension / --disable-extensions-except so uBlock and cookie
  ext can coexist; SaveResult.cookie_ext_skipped tracks miss
- database.rs: cookie_ext_enabled column on instance_settings (DEFAULT 1);
  idempotent ALTER TABLE migration; get/update wired through
- capture.rs: CaptureConfig.cookie_ext_enabled: Option<bool>; threaded to
  singlefile::save(); cookie_ext_skipped surfaced in CaptureResult
- routes.rs: CaptureBody + UpdateInstanceSettingsBody get cookie_ext_enabled;
  capture handler resolves effective value (body overrides global); notes_json
  only includes skipped fields that are true; GET instance-settings includes
  cookie_ext_available from env path check

Frontend:
- api.js: submitCapture forwards cookie_ext_enabled
- SettingsView.jsx: 'I Still Don't Care About Cookies' card in Extensions
  tab; always-active toggle (user can disable even when ext not installed);
  amber 'Not configured' hint + ARCHIVR_COOKIE_EXT guidance when unavailable
- CaptureDialog.jsx: 'Block cookie banners' toggle in Advanced options;
  always shown with amber hint when ext not configured; defaults from
  global setting

Operator setup: download + unzip the extension from GitHub releases, set
ARCHIVR_COOKIE_EXT=/path/to/unpacked/ext. No Node daemon needed.

* fix: surface cookie_ext_skipped warning toast in CaptureDialog

* feat: package istilldontcareaboutcookies in flake, wire ARCHIVR_COOKIE_EXT

Add isdcac derivation mirroring ublockLite:
- Fetches ISDCAC-chrome-source.zip v1.1.9 from GitHub releases
- Validates manifest.json at extension root before install (guard against
  nested-folder zip regressions in future releases)
- Sets ARCHIVR_COOKIE_EXT in both archivr and archivr_server wrappers

Verified: nix build .#archivr-server and .#archivr both succeed;
wrapper scripts export correct store paths; manifest.json present at root.

* fix: gate consent-overlay cleanup on cookie_ext; reset overflow; narrow selectors

- Strip overflow:hidden from body/html only when cookie_ext is active for
  the capture — prevents mutating legitimate pages when the feature is off
- Remove .fc-dialog (Google Funding Choices), .qc-cmp2-*, .sp-message-container,
  #sp-cc, #usercentrics-root as fallback for CMPs the extension misses
- Removed overbroad [class^="uc-"] and [id^="usercentrics"] selectors
  that could match real page content

* fix: remove ad placeholders when uBlock active; kept height causes blank gap

uBlock Origin Lite blocks ad network requests but first-party placeholder
elements (ins.adsbygoogle, #aswift_* iframe hosts) retain their computed
height (e.g. 280px for a top banner), leaving a large blank space at the
top of captured pages.

Gate cleanup on ublock_ext.is_some(): remove ins.adsbygoogle, aswift_*
iframes, and google_ads_* iframes before SingleFile serialises. Also
collapse the parent container if it becomes empty after removal.

* fix: walk up to .top-ad/.google-auto-placed ancestor before removing ad slot

Removing only the inner ins.adsbygoogle left the outer .container.top-ad
wrapper (with pb-4 padding) in the layout, preserving the blank gap.
Now walk up via closest() to the nearest ad-slot container class before
removal so the whole slot including padding collapses.
This commit is contained in:
TheGeneralist 2026-07-08 23:26:48 +02:00 committed by GitHub
parent dae61e585d
commit 2e8820a0da
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
18 changed files with 4003 additions and 266 deletions

View file

@ -650,9 +650,11 @@ async fn delete_entry_handler(
#[derive(Debug, serde::Deserialize)]
struct CaptureBody {
locator: String,
/// Optional quality cap for yt-dlp sources: `"best"` or any `"NNNp"` string
/// (e.g. `"1080p"`, `"720p"`, `"2160p"`). Absent or `"best"` → highest available.
quality: Option<String>,
ublock_enabled: Option<bool>,
/// Distil to article content via Readability before archiving. Absent = false.
reader_mode: Option<bool>,
cookie_ext_enabled: Option<bool>,
}
#[derive(Debug, serde::Deserialize)]
@ -732,15 +734,29 @@ async fn capture_handler(
let conn = database::open_or_initialize(&mounted.archive_path)?;
let job_uid = database::create_capture_job(&conn, &archive_id)?;
drop(conn);
// Load cookie rules from the auth DB to pass into the capture background task.
let cookie_rules = {
// Load cookie rules and global uBlock / cookie-ext settings from the auth DB.
let (cookie_rules, global_ublock, global_cookie_ext) = {
match database::open_auth_db(&state.auth_db_path) {
Ok(conn) => database::list_cookie_rules(&conn).unwrap_or_default(),
Err(_) => vec![],
Ok(conn) => {
let rules = database::list_cookie_rules(&conn).unwrap_or_default();
let settings = database::get_instance_settings(&conn);
let ublock = settings.as_ref().map(|s| s.ublock_enabled).unwrap_or(true);
let cookie_ext = settings.map(|s| s.cookie_ext_enabled).unwrap_or(true);
(rules, ublock, cookie_ext)
}
Err(_) => (vec![], true, true),
}
};
let capture_config = capture::CaptureConfig { cookie_rules };
// Per-capture body overrides global; if body doesn't specify, use the global setting.
// The resolved bool is then passed as Some(_) to singlefile, overriding the env var.
let effective_ublock = body.ublock_enabled.unwrap_or(global_ublock);
let effective_cookie_ext = body.cookie_ext_enabled.unwrap_or(global_cookie_ext);
let capture_config = capture::CaptureConfig {
cookie_rules,
ublock_enabled: Some(effective_ublock),
cookie_ext_enabled: Some(effective_cookie_ext),
reader_mode: body.reader_mode.unwrap_or(false),
};
// Spawn background capture.
let locator = body.locator.trim().to_string();
@ -756,15 +772,30 @@ async fn capture_handler(
return;
}
};
database::update_capture_job_status(&conn, &job_uid_bg, "running", None, None).ok();
database::update_capture_job_status(&conn, &job_uid_bg, "running", None, None, None).ok();
match capture::perform_capture(&archive_paths, &locator, Some(&archive_id_bg), quality.as_deref(), &capture_config) {
Ok(result) => {
let mut notes_map = serde_json::Map::new();
if result.ublock_skipped {
notes_map.insert("ublock_skipped".into(), serde_json::Value::Bool(true));
}
if result.cookie_ext_skipped {
notes_map.insert("cookie_ext_skipped".into(), serde_json::Value::Bool(true));
}
let notes_str;
let notes: Option<&str> = if notes_map.is_empty() {
None
} else {
notes_str = serde_json::Value::Object(notes_map).to_string();
Some(&notes_str)
};
database::update_capture_job_status(
&conn,
&job_uid_bg,
"completed",
Some(&result.run_uid),
None,
notes,
)
.ok();
}
@ -775,6 +806,7 @@ async fn capture_handler(
"failed",
None,
Some(&format!("{e:#}")),
None,
)
.ok();
}
@ -1013,10 +1045,26 @@ async fn patch_me(
async fn get_instance_settings_handler(
State(state): State<AppState>,
auth_user: AuthUser,
) -> Result<Json<database::InstanceSettings>, ApiError> {
) -> Result<Json<serde_json::Value>, ApiError> {
auth_user.require_role(ROLE_ADMIN)?;
let conn = database::open_auth_db(&state.auth_db_path)?;
Ok(Json(database::get_instance_settings(&conn)?))
let settings = database::get_instance_settings(&conn)?;
let ublock_ext_available = std::env::var("ARCHIVR_UBLOCK_EXT")
.ok()
.filter(|s| !s.is_empty())
.map(|p| std::path::Path::new(&p).is_dir())
.unwrap_or(false);
let cookie_ext_available = std::env::var("ARCHIVR_COOKIE_EXT")
.ok()
.filter(|s| !s.is_empty())
.map(|p| std::path::Path::new(&p).is_dir())
.unwrap_or(false);
let mut val = serde_json::to_value(&settings).unwrap_or_default();
if let Some(obj) = val.as_object_mut() {
obj.insert("ublock_ext_available".into(), serde_json::Value::Bool(ublock_ext_available));
obj.insert("cookie_ext_available".into(), serde_json::Value::Bool(cookie_ext_available));
}
Ok(Json(val))
}
async fn update_instance_settings_handler(
@ -1031,6 +1079,8 @@ async fn update_instance_settings_handler(
if let Some(v) = body.public_entry_content_enabled { settings.public_entry_content_enabled = v; }
if let Some(v) = body.open_registration_enabled { settings.open_registration_enabled = v; }
if let Some(v) = body.default_entry_visibility { settings.default_entry_visibility = v; }
if let Some(v) = body.ublock_enabled { settings.ublock_enabled = v; }
if let Some(v) = body.cookie_ext_enabled { settings.cookie_ext_enabled = v; }
database::update_instance_settings(&conn, &settings)?;
Ok(StatusCode::NO_CONTENT)
}
@ -1360,6 +1410,8 @@ struct UpdateInstanceSettingsBody {
public_entry_content_enabled: Option<bool>,
open_registration_enabled: Option<bool>,
default_entry_visibility: Option<u32>,
ublock_enabled: Option<bool>,
cookie_ext_enabled: Option<bool>,
}
async fn admin_list_users(

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View file

@ -4,8 +4,8 @@
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Archivr</title>
<script type="module" crossorigin src="/assets/index-CHDuICqH.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-BW0QKHXE.css">
<script type="module" crossorigin src="/assets/index-D6iX-c2T.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-C0BIEdow.css">
</head>
<body>
<div id="root"></div>