mirror of
https://github.com/thegeneralist01/archivr
synced 2026-07-21 18:55:36 +02:00
feat: uBlock Origin Lite + cookie consent extension + reader mode + ad placeholder cleanup (#21)
* feat: uBlock Origin Lite integration for ad-blocking during WebPage captures
- singlefile.rs: when ARCHIVR_UBLOCK=true and ARCHIVR_UBLOCK_EXT is set,
archivr owns Chrome's lifecycle (--headless=new, --remote-debugging-port,
--load-extension); single-file connects via --browser-server instead of
launching its own Chrome. Falls back to old behaviour with ublock_skipped=true
when the ext path is missing or invalid.
- capture.rs: thread ublock_skipped through CaptureResult
- database.rs: add notes_json TEXT column to capture_jobs (DDL + idempotent
ALTER TABLE migration); update_capture_job_status gains notes_json param
- archive.rs: expose notes_json in CaptureJobSummary
- routes.rs: store {"ublock_skipped":true} in notes_json on completed captures
- ToastStack.jsx: warning toast variant (toast--warning) with Details expander
and Ignore button
- CaptureDialog.jsx: fire warning toast when poll result has ublock_skipped
- App.jsx: sessionStorage-backed Ignore suppression for ublock warnings
- styles.css: .toast--warning (amber left border) + .toast-warning-detail
- flake.nix: ublockLite derivation fetches uBOLite_2026.705.2152.chromium.zip
(pinned SHA256) from uBlockOrigin/uBOL-home; sets ARCHIVR_UBLOCK_EXT in both
archivr and archivr-server wrappers
Env vars:
ARCHIVR_UBLOCK=true (default) — enable uBlock during WebPage captures
ARCHIVR_UBLOCK_EXT — path to unpacked uBOL extension dir (set by Nix)
* feat: Extensions settings tab + capture dialog redesign with Advanced options
Settings/Extensions tab (admin-only):
- New 'Extensions' tab between Cookies and Storage
- ExtensionsTab component: shows uBlock Origin Lite card with pill toggle
- Reads ublock_enabled from instance settings; patch via existing PATCH endpoint
- Shows ublock_ext_available status from server (whether ARCHIVR_UBLOCK_EXT is set)
Instance settings:
- Add ublock_enabled BOOLEAN (default true) to instance_settings auth DB table
- Idempotent ALTER TABLE migration in initialize_auth_schema()
- get/update_instance_settings include ublock_enabled
- GET /api/admin/instance-settings now also returns ublock_ext_available (computed
from ARCHIVR_UBLOCK_EXT env var at request time)
- PATCH /api/admin/instance-settings accepts ublock_enabled
Per-capture override:
- CaptureBody gains ublock_enabled: Option<bool>
- CaptureConfig gains ublock_enabled: Option<bool>
- singlefile::save() gains ublock_enabled_override: Option<bool> param
- Capture handler resolves: body override > global instance setting > env var
- submitCapture(aid, loc, qual, extensions) in api.js passes ublock_enabled
Capture dialog redesign:
- Archive button: full-width, 13px padding, min-width 220px, primary CTA
- Cancel: full-width but text-style, below Archive
- ‹Advanced options› chevron toggle (rotates on open)
- Expanded panel shows uBlock toggle for this capture session
- Loads global ublock_enabled default from instance settings on mount
Styles:
- .ext-toggle pill switch (44×24 and 36×20 small variant)
- .ext-card for Settings Extensions tab
- .capture-advanced + .capture-advanced-panel + .capture-chevron
- .capture-ext-row / .capture-ext-label / .capture-ext-name / .capture-ext-desc
- .form-hint utility class
* fix: remove ublock_enabled from INSERT OR IGNORE in DDL batch
The INSERT ran before the ALTER TABLE migration added the column,
causing 'table instance_settings has no column named ublock_enabled'
on existing databases. The INSERT OR IGNORE for the default row only
needs the original columns; the migration's DEFAULT 1 handles the
new column for existing and new rows alike.
* feat: Reader mode via Mozilla Readability.js
Adds an opt-in 'Reader mode' advanced option to the capture dialog.
When enabled, Readability.js is injected as a browser script during
SingleFile capture; it fires on single-file-on-before-capture-start,
replaces the page body with the distilled article content, injects a
clean typographic stylesheet, and adds a header with title/byline/site.
Falls back silently if Readability fails (e.g. non-article pages).
- vendor/readability/Readability.js Apache 2.0, Mozilla, v0.6.0
- singlefile.rs: embed READABILITY_JS + READER_MODE_WRAPPER_JS via
include_str!; write both to temp dir when reader_mode is true;
base_single_file_cmd now accepts &[&Path] for multiple --browser-script
- capture.rs: CaptureConfig.reader_mode: bool
- routes.rs: CaptureBody.reader_mode: Option<bool> (defaults false)
- api.js: submitCapture passes reader_mode in payload
- CaptureDialog.jsx: Reader mode toggle in Advanced options (off by default)
* fix: diagnose single-file no-output-file error + prevent stdout dumping
- Add --dump-content=false to every single-file invocation to prevent
the Docker-detection heuristic from routing HTML to stdout instead of
the output file (the heuristic can trigger in some macOS environments)
- Improve the no-output-file error message to include: temp dir contents,
stderr, and first 200 chars of stdout — this gives enough context to
diagnose any remaining cause without re-running
* fix: switch uBlock loading from --browser-server to --browser-args
The --browser-server (CDP) path caused 'Unexpected server response: 404'
on macOS Chrome because simple-cdp's WebSocket upgrade to the debugger
endpoint failed after Chrome started — likely a version-specific CDP
endpoint shape mismatch.
New approach: single-file always manages Chrome. When ARCHIVR_UBLOCK_EXT
is set, --headless=new, --load-extension, and --disable-extensions-except
are injected via --browser-args. single-file's browser.js prefix-strips
its own conflicting flags before appending ours, so --headless=new
overrides the default --headless (enabling extension support in headless).
Removes allocate_free_port, wait_for_chrome_ready, run_single_file_with_server
(all dead code now). Docblock updated to reflect actual behaviour and notes
the --single-process caveat: uBOL's declarativeNetRequest static rulesets
are expected to work (network-stack level, not service-worker), but this
has not been mechanically verified under --single-process.
Smoke tested on macOS (this machine): capture with --load-extension + all
three browser-scripts (strip, Readability, reader-mode wrapper) produces
output file correctly. Ad-blocking verification deferred to manual test
with a tracker-heavy URL.
* fix: use correct single-file hook event (single-file-on-before-capture-request)
Prior scripts listened on 'single-file-on-before-capture-start' which
does not exist in single-file-core 1.1.49. The real hook is:
single-file-on-before-capture-request (dispatched by initUserScriptHandler
after receiving single-file-user-script-init; userScriptEnabled defaults
to true in args.js so it always fires when --browser-script is passed)
Changes:
- strip-scripts: -start -> -request (no preventDefault needed; synchronous)
- READER_MODE_SCRIPT: -start -> -request; add 'installed' meta marker at
script-evaluation time so artifact inspection can distinguish 'script
not injected' / 'hook never fired' / 'Readability parse failed'
* fix: correct singlefile.rs docstring (scripts.js concatenates, not isolates)
* fix: dispatch single-file-user-script-init so request hook fires
single-file's initUserScriptHandler (in single-file-bootstrap.js) listens
for 'single-file-user-script-init' and only then installs
_singleFile_waitForUserScript. Without that dispatch our scripts'
'single-file-on-before-capture-request' listeners were never reached,
so neither strip-scripts nor reader-mode Readability applied.
Dispatch the init event at the top of strip-scripts (always present) and
redundantly in READER_MODE_SCRIPT. Verified end-to-end: artifact for
run_b3181d6d276e4e56a1a6c356ef9bbe8f has
meta content="applied", max-width:680px CSS, 0 script tags.
* feat: cookie consent extension support (ARCHIVR_COOKIE_EXT)
Mirrors the uBlock Origin Lite integration exactly:
Backend:
- singlefile.rs: resolve_cookie_ext_config() reads ARCHIVR_COOKIE_CONSENT
(default true) + ARCHIVR_COOKIE_EXT path; extension paths comma-joined
into --load-extension / --disable-extensions-except so uBlock and cookie
ext can coexist; SaveResult.cookie_ext_skipped tracks miss
- database.rs: cookie_ext_enabled column on instance_settings (DEFAULT 1);
idempotent ALTER TABLE migration; get/update wired through
- capture.rs: CaptureConfig.cookie_ext_enabled: Option<bool>; threaded to
singlefile::save(); cookie_ext_skipped surfaced in CaptureResult
- routes.rs: CaptureBody + UpdateInstanceSettingsBody get cookie_ext_enabled;
capture handler resolves effective value (body overrides global); notes_json
only includes skipped fields that are true; GET instance-settings includes
cookie_ext_available from env path check
Frontend:
- api.js: submitCapture forwards cookie_ext_enabled
- SettingsView.jsx: 'I Still Don't Care About Cookies' card in Extensions
tab; always-active toggle (user can disable even when ext not installed);
amber 'Not configured' hint + ARCHIVR_COOKIE_EXT guidance when unavailable
- CaptureDialog.jsx: 'Block cookie banners' toggle in Advanced options;
always shown with amber hint when ext not configured; defaults from
global setting
Operator setup: download + unzip the extension from GitHub releases, set
ARCHIVR_COOKIE_EXT=/path/to/unpacked/ext. No Node daemon needed.
* fix: surface cookie_ext_skipped warning toast in CaptureDialog
* feat: package istilldontcareaboutcookies in flake, wire ARCHIVR_COOKIE_EXT
Add isdcac derivation mirroring ublockLite:
- Fetches ISDCAC-chrome-source.zip v1.1.9 from GitHub releases
- Validates manifest.json at extension root before install (guard against
nested-folder zip regressions in future releases)
- Sets ARCHIVR_COOKIE_EXT in both archivr and archivr_server wrappers
Verified: nix build .#archivr-server and .#archivr both succeed;
wrapper scripts export correct store paths; manifest.json present at root.
* fix: gate consent-overlay cleanup on cookie_ext; reset overflow; narrow selectors
- Strip overflow:hidden from body/html only when cookie_ext is active for
the capture — prevents mutating legitimate pages when the feature is off
- Remove .fc-dialog (Google Funding Choices), .qc-cmp2-*, .sp-message-container,
#sp-cc, #usercentrics-root as fallback for CMPs the extension misses
- Removed overbroad [class^="uc-"] and [id^="usercentrics"] selectors
that could match real page content
* fix: remove ad placeholders when uBlock active; kept height causes blank gap
uBlock Origin Lite blocks ad network requests but first-party placeholder
elements (ins.adsbygoogle, #aswift_* iframe hosts) retain their computed
height (e.g. 280px for a top banner), leaving a large blank space at the
top of captured pages.
Gate cleanup on ublock_ext.is_some(): remove ins.adsbygoogle, aswift_*
iframes, and google_ads_* iframes before SingleFile serialises. Also
collapse the parent container if it becomes empty after removal.
* fix: walk up to .top-ad/.google-auto-placed ancestor before removing ad slot
Removing only the inner ins.adsbygoogle left the outer .container.top-ad
wrapper (with pb-4 padding) in the layout, preserving the blank gap.
Now walk up via closest() to the nearest ad-slot container class before
removal so the whole slot including padding collapses.
This commit is contained in:
parent
dae61e585d
commit
2e8820a0da
18 changed files with 4003 additions and 266 deletions
|
|
@ -650,9 +650,11 @@ async fn delete_entry_handler(
|
|||
#[derive(Debug, serde::Deserialize)]
|
||||
struct CaptureBody {
|
||||
locator: String,
|
||||
/// Optional quality cap for yt-dlp sources: `"best"` or any `"NNNp"` string
|
||||
/// (e.g. `"1080p"`, `"720p"`, `"2160p"`). Absent or `"best"` → highest available.
|
||||
quality: Option<String>,
|
||||
ublock_enabled: Option<bool>,
|
||||
/// Distil to article content via Readability before archiving. Absent = false.
|
||||
reader_mode: Option<bool>,
|
||||
cookie_ext_enabled: Option<bool>,
|
||||
}
|
||||
|
||||
#[derive(Debug, serde::Deserialize)]
|
||||
|
|
@ -732,15 +734,29 @@ async fn capture_handler(
|
|||
let conn = database::open_or_initialize(&mounted.archive_path)?;
|
||||
let job_uid = database::create_capture_job(&conn, &archive_id)?;
|
||||
drop(conn);
|
||||
|
||||
// Load cookie rules from the auth DB to pass into the capture background task.
|
||||
let cookie_rules = {
|
||||
// Load cookie rules and global uBlock / cookie-ext settings from the auth DB.
|
||||
let (cookie_rules, global_ublock, global_cookie_ext) = {
|
||||
match database::open_auth_db(&state.auth_db_path) {
|
||||
Ok(conn) => database::list_cookie_rules(&conn).unwrap_or_default(),
|
||||
Err(_) => vec![],
|
||||
Ok(conn) => {
|
||||
let rules = database::list_cookie_rules(&conn).unwrap_or_default();
|
||||
let settings = database::get_instance_settings(&conn);
|
||||
let ublock = settings.as_ref().map(|s| s.ublock_enabled).unwrap_or(true);
|
||||
let cookie_ext = settings.map(|s| s.cookie_ext_enabled).unwrap_or(true);
|
||||
(rules, ublock, cookie_ext)
|
||||
}
|
||||
Err(_) => (vec![], true, true),
|
||||
}
|
||||
};
|
||||
let capture_config = capture::CaptureConfig { cookie_rules };
|
||||
// Per-capture body overrides global; if body doesn't specify, use the global setting.
|
||||
// The resolved bool is then passed as Some(_) to singlefile, overriding the env var.
|
||||
let effective_ublock = body.ublock_enabled.unwrap_or(global_ublock);
|
||||
let effective_cookie_ext = body.cookie_ext_enabled.unwrap_or(global_cookie_ext);
|
||||
let capture_config = capture::CaptureConfig {
|
||||
cookie_rules,
|
||||
ublock_enabled: Some(effective_ublock),
|
||||
cookie_ext_enabled: Some(effective_cookie_ext),
|
||||
reader_mode: body.reader_mode.unwrap_or(false),
|
||||
};
|
||||
|
||||
// Spawn background capture.
|
||||
let locator = body.locator.trim().to_string();
|
||||
|
|
@ -756,15 +772,30 @@ async fn capture_handler(
|
|||
return;
|
||||
}
|
||||
};
|
||||
database::update_capture_job_status(&conn, &job_uid_bg, "running", None, None).ok();
|
||||
database::update_capture_job_status(&conn, &job_uid_bg, "running", None, None, None).ok();
|
||||
match capture::perform_capture(&archive_paths, &locator, Some(&archive_id_bg), quality.as_deref(), &capture_config) {
|
||||
Ok(result) => {
|
||||
let mut notes_map = serde_json::Map::new();
|
||||
if result.ublock_skipped {
|
||||
notes_map.insert("ublock_skipped".into(), serde_json::Value::Bool(true));
|
||||
}
|
||||
if result.cookie_ext_skipped {
|
||||
notes_map.insert("cookie_ext_skipped".into(), serde_json::Value::Bool(true));
|
||||
}
|
||||
let notes_str;
|
||||
let notes: Option<&str> = if notes_map.is_empty() {
|
||||
None
|
||||
} else {
|
||||
notes_str = serde_json::Value::Object(notes_map).to_string();
|
||||
Some(¬es_str)
|
||||
};
|
||||
database::update_capture_job_status(
|
||||
&conn,
|
||||
&job_uid_bg,
|
||||
"completed",
|
||||
Some(&result.run_uid),
|
||||
None,
|
||||
notes,
|
||||
)
|
||||
.ok();
|
||||
}
|
||||
|
|
@ -775,6 +806,7 @@ async fn capture_handler(
|
|||
"failed",
|
||||
None,
|
||||
Some(&format!("{e:#}")),
|
||||
None,
|
||||
)
|
||||
.ok();
|
||||
}
|
||||
|
|
@ -1013,10 +1045,26 @@ async fn patch_me(
|
|||
async fn get_instance_settings_handler(
|
||||
State(state): State<AppState>,
|
||||
auth_user: AuthUser,
|
||||
) -> Result<Json<database::InstanceSettings>, ApiError> {
|
||||
) -> Result<Json<serde_json::Value>, ApiError> {
|
||||
auth_user.require_role(ROLE_ADMIN)?;
|
||||
let conn = database::open_auth_db(&state.auth_db_path)?;
|
||||
Ok(Json(database::get_instance_settings(&conn)?))
|
||||
let settings = database::get_instance_settings(&conn)?;
|
||||
let ublock_ext_available = std::env::var("ARCHIVR_UBLOCK_EXT")
|
||||
.ok()
|
||||
.filter(|s| !s.is_empty())
|
||||
.map(|p| std::path::Path::new(&p).is_dir())
|
||||
.unwrap_or(false);
|
||||
let cookie_ext_available = std::env::var("ARCHIVR_COOKIE_EXT")
|
||||
.ok()
|
||||
.filter(|s| !s.is_empty())
|
||||
.map(|p| std::path::Path::new(&p).is_dir())
|
||||
.unwrap_or(false);
|
||||
let mut val = serde_json::to_value(&settings).unwrap_or_default();
|
||||
if let Some(obj) = val.as_object_mut() {
|
||||
obj.insert("ublock_ext_available".into(), serde_json::Value::Bool(ublock_ext_available));
|
||||
obj.insert("cookie_ext_available".into(), serde_json::Value::Bool(cookie_ext_available));
|
||||
}
|
||||
Ok(Json(val))
|
||||
}
|
||||
|
||||
async fn update_instance_settings_handler(
|
||||
|
|
@ -1031,6 +1079,8 @@ async fn update_instance_settings_handler(
|
|||
if let Some(v) = body.public_entry_content_enabled { settings.public_entry_content_enabled = v; }
|
||||
if let Some(v) = body.open_registration_enabled { settings.open_registration_enabled = v; }
|
||||
if let Some(v) = body.default_entry_visibility { settings.default_entry_visibility = v; }
|
||||
if let Some(v) = body.ublock_enabled { settings.ublock_enabled = v; }
|
||||
if let Some(v) = body.cookie_ext_enabled { settings.cookie_ext_enabled = v; }
|
||||
database::update_instance_settings(&conn, &settings)?;
|
||||
Ok(StatusCode::NO_CONTENT)
|
||||
}
|
||||
|
|
@ -1360,6 +1410,8 @@ struct UpdateInstanceSettingsBody {
|
|||
public_entry_content_enabled: Option<bool>,
|
||||
open_registration_enabled: Option<bool>,
|
||||
default_entry_visibility: Option<u32>,
|
||||
ublock_enabled: Option<bool>,
|
||||
cookie_ext_enabled: Option<bool>,
|
||||
}
|
||||
|
||||
async fn admin_list_users(
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
1
crates/archivr-server/static/assets/index-C0BIEdow.css
Normal file
1
crates/archivr-server/static/assets/index-C0BIEdow.css
Normal file
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
40
crates/archivr-server/static/assets/index-D6iX-c2T.js
Normal file
40
crates/archivr-server/static/assets/index-D6iX-c2T.js
Normal file
File diff suppressed because one or more lines are too long
|
|
@ -4,8 +4,8 @@
|
|||
<meta charset="utf-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>Archivr</title>
|
||||
<script type="module" crossorigin src="/assets/index-CHDuICqH.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-BW0QKHXE.css">
|
||||
<script type="module" crossorigin src="/assets/index-D6iX-c2T.js"></script>
|
||||
<link rel="stylesheet" crossorigin href="/assets/index-C0BIEdow.css">
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue