thegeneralist01/archivr
1
Fork 0
mirror of https://github.com/thegeneralist01/archivr synced 2026-07-21 18:55:36 +02:00
Code Issues Activity

feat: uBlock Origin Lite + cookie consent extension + reader mode + ad placeholder cleanup (#21)

Browse source 
* feat: uBlock Origin Lite integration for ad-blocking during WebPage captures

- singlefile.rs: when ARCHIVR_UBLOCK=true and ARCHIVR_UBLOCK_EXT is set,
  archivr owns Chrome's lifecycle (--headless=new, --remote-debugging-port,
  --load-extension); single-file connects via --browser-server instead of
  launching its own Chrome. Falls back to old behaviour with ublock_skipped=true
  when the ext path is missing or invalid.
- capture.rs: thread ublock_skipped through CaptureResult
- database.rs: add notes_json TEXT column to capture_jobs (DDL + idempotent
  ALTER TABLE migration); update_capture_job_status gains notes_json param
- archive.rs: expose notes_json in CaptureJobSummary
- routes.rs: store {"ublock_skipped":true} in notes_json on completed captures
- ToastStack.jsx: warning toast variant (toast--warning) with Details expander
  and Ignore button
- CaptureDialog.jsx: fire warning toast when poll result has ublock_skipped
- App.jsx: sessionStorage-backed Ignore suppression for ublock warnings
- styles.css: .toast--warning (amber left border) + .toast-warning-detail
- flake.nix: ublockLite derivation fetches uBOLite_2026.705.2152.chromium.zip
  (pinned SHA256) from uBlockOrigin/uBOL-home; sets ARCHIVR_UBLOCK_EXT in both
  archivr and archivr-server wrappers

Env vars:
  ARCHIVR_UBLOCK=true (default) — enable uBlock during WebPage captures
  ARCHIVR_UBLOCK_EXT — path to unpacked uBOL extension dir (set by Nix)

* feat: Extensions settings tab + capture dialog redesign with Advanced options

Settings/Extensions tab (admin-only):
- New 'Extensions' tab between Cookies and Storage
- ExtensionsTab component: shows uBlock Origin Lite card with pill toggle
- Reads ublock_enabled from instance settings; patch via existing PATCH endpoint
- Shows ublock_ext_available status from server (whether ARCHIVR_UBLOCK_EXT is set)

Instance settings:
- Add ublock_enabled BOOLEAN (default true) to instance_settings auth DB table
- Idempotent ALTER TABLE migration in initialize_auth_schema()
- get/update_instance_settings include ublock_enabled
- GET /api/admin/instance-settings now also returns ublock_ext_available (computed
  from ARCHIVR_UBLOCK_EXT env var at request time)
- PATCH /api/admin/instance-settings accepts ublock_enabled

Per-capture override:
- CaptureBody gains ublock_enabled: Option<bool>
- CaptureConfig gains ublock_enabled: Option<bool>
- singlefile::save() gains ublock_enabled_override: Option<bool> param
- Capture handler resolves: body override > global instance setting > env var
- submitCapture(aid, loc, qual, extensions) in api.js passes ublock_enabled

Capture dialog redesign:
- Archive button: full-width, 13px padding, min-width 220px, primary CTA
- Cancel: full-width but text-style, below Archive
- ‹Advanced options› chevron toggle (rotates on open)
- Expanded panel shows uBlock toggle for this capture session
- Loads global ublock_enabled default from instance settings on mount

Styles:
- .ext-toggle pill switch (44×24 and 36×20 small variant)
- .ext-card for Settings Extensions tab
- .capture-advanced + .capture-advanced-panel + .capture-chevron
- .capture-ext-row / .capture-ext-label / .capture-ext-name / .capture-ext-desc
- .form-hint utility class

* fix: remove ublock_enabled from INSERT OR IGNORE in DDL batch

The INSERT ran before the ALTER TABLE migration added the column,
causing 'table instance_settings has no column named ublock_enabled'
on existing databases. The INSERT OR IGNORE for the default row only
needs the original columns; the migration's DEFAULT 1 handles the
new column for existing and new rows alike.

* feat: Reader mode via Mozilla Readability.js

Adds an opt-in 'Reader mode' advanced option to the capture dialog.
When enabled, Readability.js is injected as a browser script during
SingleFile capture; it fires on single-file-on-before-capture-start,
replaces the page body with the distilled article content, injects a
clean typographic stylesheet, and adds a header with title/byline/site.
Falls back silently if Readability fails (e.g. non-article pages).

- vendor/readability/Readability.js  Apache 2.0, Mozilla, v0.6.0
- singlefile.rs: embed READABILITY_JS + READER_MODE_WRAPPER_JS via
  include_str!; write both to temp dir when reader_mode is true;
  base_single_file_cmd now accepts &[&Path] for multiple --browser-script
- capture.rs: CaptureConfig.reader_mode: bool
- routes.rs: CaptureBody.reader_mode: Option<bool> (defaults false)
- api.js: submitCapture passes reader_mode in payload
- CaptureDialog.jsx: Reader mode toggle in Advanced options (off by default)

* fix: diagnose single-file no-output-file error + prevent stdout dumping

- Add --dump-content=false to every single-file invocation to prevent
  the Docker-detection heuristic from routing HTML to stdout instead of
  the output file (the heuristic can trigger in some macOS environments)
- Improve the no-output-file error message to include: temp dir contents,
  stderr, and first 200 chars of stdout — this gives enough context to
  diagnose any remaining cause without re-running

* fix: switch uBlock loading from --browser-server to --browser-args

The --browser-server (CDP) path caused 'Unexpected server response: 404'
on macOS Chrome because simple-cdp's WebSocket upgrade to the debugger
endpoint failed after Chrome started — likely a version-specific CDP
endpoint shape mismatch.

New approach: single-file always manages Chrome. When ARCHIVR_UBLOCK_EXT
is set, --headless=new, --load-extension, and --disable-extensions-except
are injected via --browser-args. single-file's browser.js prefix-strips
its own conflicting flags before appending ours, so --headless=new
overrides the default --headless (enabling extension support in headless).

Removes allocate_free_port, wait_for_chrome_ready, run_single_file_with_server
(all dead code now). Docblock updated to reflect actual behaviour and notes
the --single-process caveat: uBOL's declarativeNetRequest static rulesets
are expected to work (network-stack level, not service-worker), but this
has not been mechanically verified under --single-process.

Smoke tested on macOS (this machine): capture with --load-extension + all
three browser-scripts (strip, Readability, reader-mode wrapper) produces
output file correctly. Ad-blocking verification deferred to manual test
with a tracker-heavy URL.

* fix: use correct single-file hook event (single-file-on-before-capture-request)

Prior scripts listened on 'single-file-on-before-capture-start' which
does not exist in single-file-core 1.1.49.  The real hook is:

  single-file-on-before-capture-request  (dispatched by initUserScriptHandler
  after receiving single-file-user-script-init; userScriptEnabled defaults
  to true in args.js so it always fires when --browser-script is passed)

Changes:
- strip-scripts: -start -> -request (no preventDefault needed; synchronous)
- READER_MODE_SCRIPT: -start -> -request; add 'installed' meta marker at
  script-evaluation time so artifact inspection can distinguish 'script
  not injected' / 'hook never fired' / 'Readability parse failed'

* fix: correct singlefile.rs docstring (scripts.js concatenates, not isolates)

* fix: dispatch single-file-user-script-init so request hook fires

single-file's initUserScriptHandler (in single-file-bootstrap.js) listens
for 'single-file-user-script-init' and only then installs
_singleFile_waitForUserScript.  Without that dispatch our scripts'
'single-file-on-before-capture-request' listeners were never reached,
so neither strip-scripts nor reader-mode Readability applied.

Dispatch the init event at the top of strip-scripts (always present) and
redundantly in READER_MODE_SCRIPT.  Verified end-to-end: artifact for
run_b3181d6d276e4e56a1a6c356ef9bbe8f has
  meta content="applied", max-width:680px CSS, 0 script tags.

* feat: cookie consent extension support (ARCHIVR_COOKIE_EXT)

Mirrors the uBlock Origin Lite integration exactly:

Backend:
- singlefile.rs: resolve_cookie_ext_config() reads ARCHIVR_COOKIE_CONSENT
  (default true) + ARCHIVR_COOKIE_EXT path; extension paths comma-joined
  into --load-extension / --disable-extensions-except so uBlock and cookie
  ext can coexist; SaveResult.cookie_ext_skipped tracks miss
- database.rs: cookie_ext_enabled column on instance_settings (DEFAULT 1);
  idempotent ALTER TABLE migration; get/update wired through
- capture.rs: CaptureConfig.cookie_ext_enabled: Option<bool>; threaded to
  singlefile::save(); cookie_ext_skipped surfaced in CaptureResult
- routes.rs: CaptureBody + UpdateInstanceSettingsBody get cookie_ext_enabled;
  capture handler resolves effective value (body overrides global); notes_json
  only includes skipped fields that are true; GET instance-settings includes
  cookie_ext_available from env path check

Frontend:
- api.js: submitCapture forwards cookie_ext_enabled
- SettingsView.jsx: 'I Still Don't Care About Cookies' card in Extensions
  tab; always-active toggle (user can disable even when ext not installed);
  amber 'Not configured' hint + ARCHIVR_COOKIE_EXT guidance when unavailable
- CaptureDialog.jsx: 'Block cookie banners' toggle in Advanced options;
  always shown with amber hint when ext not configured; defaults from
  global setting

Operator setup: download + unzip the extension from GitHub releases, set
ARCHIVR_COOKIE_EXT=/path/to/unpacked/ext. No Node daemon needed.

* fix: surface cookie_ext_skipped warning toast in CaptureDialog

* feat: package istilldontcareaboutcookies in flake, wire ARCHIVR_COOKIE_EXT

Add isdcac derivation mirroring ublockLite:
- Fetches ISDCAC-chrome-source.zip v1.1.9 from GitHub releases
- Validates manifest.json at extension root before install (guard against
  nested-folder zip regressions in future releases)
- Sets ARCHIVR_COOKIE_EXT in both archivr and archivr_server wrappers

Verified: nix build .#archivr-server and .#archivr both succeed;
wrapper scripts export correct store paths; manifest.json present at root.

* fix: gate consent-overlay cleanup on cookie_ext; reset overflow; narrow selectors

- Strip overflow:hidden from body/html only when cookie_ext is active for
  the capture — prevents mutating legitimate pages when the feature is off
- Remove .fc-dialog (Google Funding Choices), .qc-cmp2-*, .sp-message-container,
  #sp-cc, #usercentrics-root as fallback for CMPs the extension misses
- Removed overbroad [class^="uc-"] and [id^="usercentrics"] selectors
  that could match real page content

* fix: remove ad placeholders when uBlock active; kept height causes blank gap

uBlock Origin Lite blocks ad network requests but first-party placeholder
elements (ins.adsbygoogle, #aswift_* iframe hosts) retain their computed
height (e.g. 280px for a top banner), leaving a large blank space at the
top of captured pages.

Gate cleanup on ublock_ext.is_some(): remove ins.adsbygoogle, aswift_*
iframes, and google_ads_* iframes before SingleFile serialises. Also
collapse the parent container if it becomes empty after removal.

* fix: walk up to .top-ad/.google-auto-placed ancestor before removing ad slot

Removing only the inner ins.adsbygoogle left the outer .container.top-ad
wrapper (with pb-4 padding) in the layout, preserving the blank gap.
Now walk up via closest() to the nearest ad-slot container class before
removal so the whole slot including padding collapses.
This commit is contained in:
TheGeneralist 2026-07-08 23:26:48 +02:00 • committed by GitHub
parent dae61e585d
commit 2e8820a0da
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
18 changed files with 4003 additions and 266 deletions
Download patch file Download diff file Expand all files Collapse all files

66
crates/archivr-core/src/database.rs
View file

@ -105,6 +105,7 @@ pub struct CaptureJobRecord {
pub run_uid: Option<String>,
pub status: String,
pub error_text: Option<String>,
pub notes_json: Option<String>,
pub created_at: String,
pub updated_at: String,
}
@ -136,6 +137,11 @@ pub struct InstanceSettings {
pub public_entry_content_enabled: bool,
pub open_registration_enabled: bool, // maps to public_archive_submission_enabled column
pub default_entry_visibility: u32,
/// Global default for ad-blocking via uBlock Origin Lite during WebPage captures.
/// Per-capture requests can override this.
pub ublock_enabled: bool,
/// Global default for cookie-consent banner dismissal via extension during WebPage captures.
pub cookie_ext_enabled: bool,
}
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
@ -194,7 +200,8 @@ pub fn initialize_schema(conn: &Connection) -> Result<()> {
id INTEGER PRIMARY KEY CHECK (id = 1),
public_index_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_index_enabled IN (0, 1)),
public_entry_content_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_entry_content_enabled IN (0, 1)),
public_archive_submission_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_archive_submission_enabled IN (0, 1))
public_archive_submission_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_archive_submission_enabled IN (0, 1)),
cookie_ext_enabled INTEGER NOT NULL DEFAULT 1 CHECK (cookie_ext_enabled IN (0, 1))
);
INSERT OR IGNORE INTO instance_settings (
@ -308,6 +315,7 @@ pub fn initialize_schema(conn: &Connection) -> Result<()> {
run_uid TEXT,
status TEXT NOT NULL CHECK (status IN ('pending','running','completed','failed')) DEFAULT 'pending',
error_text TEXT,
notes_json TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
@ -393,6 +401,10 @@ pub fn initialize_schema(conn: &Connection) -> Result<()> {
)?;
}
// Migration: add notes_json column to existing capture_jobs tables.
// Silently ignored when the column already exists (idempotent).
let _ = conn.execute("ALTER TABLE capture_jobs ADD COLUMN notes_json TEXT", []);
Ok(())
}
@ -454,7 +466,9 @@ pub fn initialize_auth_schema(conn: &Connection) -> Result<()> {
public_index_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_index_enabled IN (0, 1)),
public_entry_content_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_entry_content_enabled IN (0, 1)),
public_archive_submission_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_archive_submission_enabled IN (0, 1)),
default_entry_visibility INTEGER NOT NULL DEFAULT 2
default_entry_visibility INTEGER NOT NULL DEFAULT 2,
ublock_enabled INTEGER NOT NULL DEFAULT 1 CHECK (ublock_enabled IN (0, 1)),
cookie_ext_enabled INTEGER NOT NULL DEFAULT 1 CHECK (cookie_ext_enabled IN (0, 1))
);
INSERT OR IGNORE INTO instance_settings
@ -494,6 +508,17 @@ pub fn initialize_auth_schema(conn: &Connection) -> Result<()> {
[],
);
// Add ublock_enabled column to instance_settings if not present (idempotent migration)
let _ = conn.execute(
"ALTER TABLE instance_settings ADD COLUMN ublock_enabled INTEGER NOT NULL DEFAULT 1",
[],
);
// Add cookie_ext_enabled column to instance_settings if not present (idempotent migration)
let _ = conn.execute(
"ALTER TABLE instance_settings ADD COLUMN cookie_ext_enabled INTEGER NOT NULL DEFAULT 1",
[],
);
Ok(())
}
@ -718,7 +743,9 @@ pub fn list_user_tokens(conn: &Connection, user_id: i64) -> Result<Vec<ApiTokenR
pub fn get_instance_settings(conn: &Connection) -> Result<InstanceSettings> {
conn.query_row(
"SELECT public_index_enabled, public_entry_content_enabled,
public_archive_submission_enabled, default_entry_visibility
public_archive_submission_enabled, default_entry_visibility,
COALESCE(ublock_enabled, 1),
COALESCE(cookie_ext_enabled, 1)
FROM instance_settings WHERE id = 1",
[],
|row| {
@ -727,6 +754,8 @@ pub fn get_instance_settings(conn: &Connection) -> Result<InstanceSettings> {
public_entry_content_enabled: row.get::<_, i64>(1)? != 0,
open_registration_enabled: row.get::<_, i64>(2)? != 0,
default_entry_visibility: row.get::<_, i64>(3)? as u32,
ublock_enabled: row.get::<_, i64>(4)? != 0,
cookie_ext_enabled: row.get::<_, i64>(5)? != 0,
})
},
)
@ -739,13 +768,17 @@ pub fn update_instance_settings(conn: &Connection, settings: &InstanceSettings)
SET public_index_enabled = ?1,
public_entry_content_enabled = ?2,
public_archive_submission_enabled = ?3,
default_entry_visibility = ?4
default_entry_visibility = ?4,
ublock_enabled = ?5,
cookie_ext_enabled = ?6
WHERE id = 1",
params![
settings.public_index_enabled as i64,
settings.public_entry_content_enabled as i64,
settings.open_registration_enabled as i64,
settings.default_entry_visibility as i64,
settings.ublock_enabled as i64,
settings.cookie_ext_enabled as i64,
],
)?;
Ok(())
@ -1142,19 +1175,21 @@ pub fn create_capture_job(conn: &Connection, archive_id: &str) -> Result<String>
Ok(job_uid)
}
/// Updates the status (and optionally run_uid / error_text) of a capture job.
/// Updates the status (and optionally run_uid / error_text / notes_json) of a capture job.
pub fn update_capture_job_status(
conn: &Connection,
job_uid: &str,
status: &str,
run_uid: Option<&str>,
error_text: Option<&str>,
notes_json: Option<&str>,
) -> Result<()> {
let now = now_timestamp();
conn.execute(
"UPDATE capture_jobs SET status = ?1, run_uid = COALESCE(?2, run_uid),
error_text = ?3, updated_at = ?4 WHERE job_uid = ?5",
rusqlite::params![status, run_uid, error_text, now, job_uid],
error_text = ?3, notes_json = COALESCE(?4, notes_json), updated_at = ?5
WHERE job_uid = ?6",
rusqlite::params![status, run_uid, error_text, notes_json, now, job_uid],
)?;
Ok(())
}
@ -1162,7 +1197,7 @@ pub fn update_capture_job_status(
/// Returns a capture job by uid.
pub fn get_capture_job(conn: &Connection, job_uid: &str) -> Result<Option<CaptureJobRecord>> {
conn.query_row(
"SELECT job_uid, archive_id, run_uid, status, error_text, created_at, updated_at
"SELECT job_uid, archive_id, run_uid, status, error_text, notes_json, created_at, updated_at
FROM capture_jobs WHERE job_uid = ?1",
[job_uid],
|row| {
@ -1172,8 +1207,9 @@ pub fn get_capture_job(conn: &Connection, job_uid: &str) -> Result<Option<Captur
run_uid: row.get(2)?,
status: row.get(3)?,
error_text: row.get(4)?,
created_at: row.get(5)?,
updated_at: row.get(6)?,
notes_json: row.get(5)?,
created_at: row.get(6)?,
updated_at: row.get(7)?,
})
},
)
@ -2806,8 +2842,8 @@ mod tests {
fn capture_job_status_transitions() {
let conn = conn();
let job_uid = create_capture_job(&conn, "test").unwrap();
update_capture_job_status(&conn, &job_uid, "running", None, None).unwrap();
update_capture_job_status(&conn, &job_uid, "completed", Some("run_abc"), None).unwrap();
update_capture_job_status(&conn, &job_uid, "running", None, None, None).unwrap();
update_capture_job_status(&conn, &job_uid, "completed", Some("run_abc"), None, None).unwrap();
let job = get_capture_job(&conn, &job_uid).unwrap().unwrap();
assert_eq!(job.status, "completed");
assert_eq!(job.run_uid.as_deref(), Some("run_abc"));
@ -2819,7 +2855,7 @@ mod tests {
// Simulate an in-progress capture_job (run_uid still NULL — common crash case).
let uid = create_capture_job(&conn, "test").unwrap();
update_capture_job_status(&conn, &uid, "running", None, None).unwrap();
update_capture_job_status(&conn, &uid, "running", None, None, None).unwrap();
// Simulate an in-progress archive_run and item with no associated capture_job
// (covers the case where run_uid was never written back before the crash).
@ -3177,7 +3213,7 @@ mod tests {
fn has_active_capture_jobs_true_for_running() {
let conn = conn();
let uid = create_capture_job(&conn, "test").unwrap();
update_capture_job_status(&conn, &uid, "running", None, None).unwrap();
update_capture_job_status(&conn, &uid, "running", None, None, None).unwrap();
assert!(has_active_capture_jobs(&conn).unwrap());
}
@ -3185,7 +3221,7 @@ mod tests {
fn has_active_capture_jobs_false_for_completed() {
let conn = conn();
let uid = create_capture_job(&conn, "test").unwrap();
update_capture_job_status(&conn, &uid, "completed", Some("run_x"), None).unwrap();
update_capture_job_status(&conn, &uid, "completed", Some("run_x"), None, None).unwrap();
assert!(!has_active_capture_jobs(&conn).unwrap());
}