thegeneralist01/archivr
1
Fork 0
mirror of https://github.com/thegeneralist01/archivr synced 2026-07-21 18:55:36 +02:00
Code Issues Activity

feat: uBlock Origin Lite + cookie consent extension + reader mode + ad placeholder cleanup (#21)

Browse source 
* feat: uBlock Origin Lite integration for ad-blocking during WebPage captures

- singlefile.rs: when ARCHIVR_UBLOCK=true and ARCHIVR_UBLOCK_EXT is set,
  archivr owns Chrome's lifecycle (--headless=new, --remote-debugging-port,
  --load-extension); single-file connects via --browser-server instead of
  launching its own Chrome. Falls back to old behaviour with ublock_skipped=true
  when the ext path is missing or invalid.
- capture.rs: thread ublock_skipped through CaptureResult
- database.rs: add notes_json TEXT column to capture_jobs (DDL + idempotent
  ALTER TABLE migration); update_capture_job_status gains notes_json param
- archive.rs: expose notes_json in CaptureJobSummary
- routes.rs: store {"ublock_skipped":true} in notes_json on completed captures
- ToastStack.jsx: warning toast variant (toast--warning) with Details expander
  and Ignore button
- CaptureDialog.jsx: fire warning toast when poll result has ublock_skipped
- App.jsx: sessionStorage-backed Ignore suppression for ublock warnings
- styles.css: .toast--warning (amber left border) + .toast-warning-detail
- flake.nix: ublockLite derivation fetches uBOLite_2026.705.2152.chromium.zip
  (pinned SHA256) from uBlockOrigin/uBOL-home; sets ARCHIVR_UBLOCK_EXT in both
  archivr and archivr-server wrappers

Env vars:
  ARCHIVR_UBLOCK=true (default) — enable uBlock during WebPage captures
  ARCHIVR_UBLOCK_EXT — path to unpacked uBOL extension dir (set by Nix)

* feat: Extensions settings tab + capture dialog redesign with Advanced options

Settings/Extensions tab (admin-only):
- New 'Extensions' tab between Cookies and Storage
- ExtensionsTab component: shows uBlock Origin Lite card with pill toggle
- Reads ublock_enabled from instance settings; patch via existing PATCH endpoint
- Shows ublock_ext_available status from server (whether ARCHIVR_UBLOCK_EXT is set)

Instance settings:
- Add ublock_enabled BOOLEAN (default true) to instance_settings auth DB table
- Idempotent ALTER TABLE migration in initialize_auth_schema()
- get/update_instance_settings include ublock_enabled
- GET /api/admin/instance-settings now also returns ublock_ext_available (computed
  from ARCHIVR_UBLOCK_EXT env var at request time)
- PATCH /api/admin/instance-settings accepts ublock_enabled

Per-capture override:
- CaptureBody gains ublock_enabled: Option<bool>
- CaptureConfig gains ublock_enabled: Option<bool>
- singlefile::save() gains ublock_enabled_override: Option<bool> param
- Capture handler resolves: body override > global instance setting > env var
- submitCapture(aid, loc, qual, extensions) in api.js passes ublock_enabled

Capture dialog redesign:
- Archive button: full-width, 13px padding, min-width 220px, primary CTA
- Cancel: full-width but text-style, below Archive
- ‹Advanced options› chevron toggle (rotates on open)
- Expanded panel shows uBlock toggle for this capture session
- Loads global ublock_enabled default from instance settings on mount

Styles:
- .ext-toggle pill switch (44×24 and 36×20 small variant)
- .ext-card for Settings Extensions tab
- .capture-advanced + .capture-advanced-panel + .capture-chevron
- .capture-ext-row / .capture-ext-label / .capture-ext-name / .capture-ext-desc
- .form-hint utility class

* fix: remove ublock_enabled from INSERT OR IGNORE in DDL batch

The INSERT ran before the ALTER TABLE migration added the column,
causing 'table instance_settings has no column named ublock_enabled'
on existing databases. The INSERT OR IGNORE for the default row only
needs the original columns; the migration's DEFAULT 1 handles the
new column for existing and new rows alike.

* feat: Reader mode via Mozilla Readability.js

Adds an opt-in 'Reader mode' advanced option to the capture dialog.
When enabled, Readability.js is injected as a browser script during
SingleFile capture; it fires on single-file-on-before-capture-start,
replaces the page body with the distilled article content, injects a
clean typographic stylesheet, and adds a header with title/byline/site.
Falls back silently if Readability fails (e.g. non-article pages).

- vendor/readability/Readability.js  Apache 2.0, Mozilla, v0.6.0
- singlefile.rs: embed READABILITY_JS + READER_MODE_WRAPPER_JS via
  include_str!; write both to temp dir when reader_mode is true;
  base_single_file_cmd now accepts &[&Path] for multiple --browser-script
- capture.rs: CaptureConfig.reader_mode: bool
- routes.rs: CaptureBody.reader_mode: Option<bool> (defaults false)
- api.js: submitCapture passes reader_mode in payload
- CaptureDialog.jsx: Reader mode toggle in Advanced options (off by default)

* fix: diagnose single-file no-output-file error + prevent stdout dumping

- Add --dump-content=false to every single-file invocation to prevent
  the Docker-detection heuristic from routing HTML to stdout instead of
  the output file (the heuristic can trigger in some macOS environments)
- Improve the no-output-file error message to include: temp dir contents,
  stderr, and first 200 chars of stdout — this gives enough context to
  diagnose any remaining cause without re-running

* fix: switch uBlock loading from --browser-server to --browser-args

The --browser-server (CDP) path caused 'Unexpected server response: 404'
on macOS Chrome because simple-cdp's WebSocket upgrade to the debugger
endpoint failed after Chrome started — likely a version-specific CDP
endpoint shape mismatch.

New approach: single-file always manages Chrome. When ARCHIVR_UBLOCK_EXT
is set, --headless=new, --load-extension, and --disable-extensions-except
are injected via --browser-args. single-file's browser.js prefix-strips
its own conflicting flags before appending ours, so --headless=new
overrides the default --headless (enabling extension support in headless).

Removes allocate_free_port, wait_for_chrome_ready, run_single_file_with_server
(all dead code now). Docblock updated to reflect actual behaviour and notes
the --single-process caveat: uBOL's declarativeNetRequest static rulesets
are expected to work (network-stack level, not service-worker), but this
has not been mechanically verified under --single-process.

Smoke tested on macOS (this machine): capture with --load-extension + all
three browser-scripts (strip, Readability, reader-mode wrapper) produces
output file correctly. Ad-blocking verification deferred to manual test
with a tracker-heavy URL.

* fix: use correct single-file hook event (single-file-on-before-capture-request)

Prior scripts listened on 'single-file-on-before-capture-start' which
does not exist in single-file-core 1.1.49.  The real hook is:

  single-file-on-before-capture-request  (dispatched by initUserScriptHandler
  after receiving single-file-user-script-init; userScriptEnabled defaults
  to true in args.js so it always fires when --browser-script is passed)

Changes:
- strip-scripts: -start -> -request (no preventDefault needed; synchronous)
- READER_MODE_SCRIPT: -start -> -request; add 'installed' meta marker at
  script-evaluation time so artifact inspection can distinguish 'script
  not injected' / 'hook never fired' / 'Readability parse failed'

* fix: correct singlefile.rs docstring (scripts.js concatenates, not isolates)

* fix: dispatch single-file-user-script-init so request hook fires

single-file's initUserScriptHandler (in single-file-bootstrap.js) listens
for 'single-file-user-script-init' and only then installs
_singleFile_waitForUserScript.  Without that dispatch our scripts'
'single-file-on-before-capture-request' listeners were never reached,
so neither strip-scripts nor reader-mode Readability applied.

Dispatch the init event at the top of strip-scripts (always present) and
redundantly in READER_MODE_SCRIPT.  Verified end-to-end: artifact for
run_b3181d6d276e4e56a1a6c356ef9bbe8f has
  meta content="applied", max-width:680px CSS, 0 script tags.

* feat: cookie consent extension support (ARCHIVR_COOKIE_EXT)

Mirrors the uBlock Origin Lite integration exactly:

Backend:
- singlefile.rs: resolve_cookie_ext_config() reads ARCHIVR_COOKIE_CONSENT
  (default true) + ARCHIVR_COOKIE_EXT path; extension paths comma-joined
  into --load-extension / --disable-extensions-except so uBlock and cookie
  ext can coexist; SaveResult.cookie_ext_skipped tracks miss
- database.rs: cookie_ext_enabled column on instance_settings (DEFAULT 1);
  idempotent ALTER TABLE migration; get/update wired through
- capture.rs: CaptureConfig.cookie_ext_enabled: Option<bool>; threaded to
  singlefile::save(); cookie_ext_skipped surfaced in CaptureResult
- routes.rs: CaptureBody + UpdateInstanceSettingsBody get cookie_ext_enabled;
  capture handler resolves effective value (body overrides global); notes_json
  only includes skipped fields that are true; GET instance-settings includes
  cookie_ext_available from env path check

Frontend:
- api.js: submitCapture forwards cookie_ext_enabled
- SettingsView.jsx: 'I Still Don't Care About Cookies' card in Extensions
  tab; always-active toggle (user can disable even when ext not installed);
  amber 'Not configured' hint + ARCHIVR_COOKIE_EXT guidance when unavailable
- CaptureDialog.jsx: 'Block cookie banners' toggle in Advanced options;
  always shown with amber hint when ext not configured; defaults from
  global setting

Operator setup: download + unzip the extension from GitHub releases, set
ARCHIVR_COOKIE_EXT=/path/to/unpacked/ext. No Node daemon needed.

* fix: surface cookie_ext_skipped warning toast in CaptureDialog

* feat: package istilldontcareaboutcookies in flake, wire ARCHIVR_COOKIE_EXT

Add isdcac derivation mirroring ublockLite:
- Fetches ISDCAC-chrome-source.zip v1.1.9 from GitHub releases
- Validates manifest.json at extension root before install (guard against
  nested-folder zip regressions in future releases)
- Sets ARCHIVR_COOKIE_EXT in both archivr and archivr_server wrappers

Verified: nix build .#archivr-server and .#archivr both succeed;
wrapper scripts export correct store paths; manifest.json present at root.

* fix: gate consent-overlay cleanup on cookie_ext; reset overflow; narrow selectors

- Strip overflow:hidden from body/html only when cookie_ext is active for
  the capture — prevents mutating legitimate pages when the feature is off
- Remove .fc-dialog (Google Funding Choices), .qc-cmp2-*, .sp-message-container,
  #sp-cc, #usercentrics-root as fallback for CMPs the extension misses
- Removed overbroad [class^="uc-"] and [id^="usercentrics"] selectors
  that could match real page content

* fix: remove ad placeholders when uBlock active; kept height causes blank gap

uBlock Origin Lite blocks ad network requests but first-party placeholder
elements (ins.adsbygoogle, #aswift_* iframe hosts) retain their computed
height (e.g. 280px for a top banner), leaving a large blank space at the
top of captured pages.

Gate cleanup on ublock_ext.is_some(): remove ins.adsbygoogle, aswift_*
iframes, and google_ads_* iframes before SingleFile serialises. Also
collapse the parent container if it becomes empty after removal.

* fix: walk up to .top-ad/.google-auto-placed ancestor before removing ad slot

Removing only the inner ins.adsbygoogle left the outer .container.top-ad
wrapper (with pb-4 padding) in the layout, preserving the blank gap.
Now walk up via closest() to the nearest ad-slot container class before
removal so the whole slot including padding collapses.
This commit is contained in:
TheGeneralist 2026-07-08 23:26:48 +02:00 • committed by GitHub
parent dae61e585d
commit 2e8820a0da
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
18 changed files with 4003 additions and 266 deletions
Download patch file Download diff file Expand all files Collapse all files

2
crates/archivr-core/src/archive.rs
View file

@ -69,6 +69,7 @@ pub struct CaptureJobSummary {
pub run_uid: Option<String>,
pub status: String,
pub error_text: Option<String>,
pub notes_json: Option<String>,
pub created_at: String,
pub updated_at: String,
}
@ -341,6 +342,7 @@ pub fn get_capture_job(
run_uid: r.run_uid,
status: r.status,
error_text: r.error_text,
notes_json: r.notes_json,
created_at: r.created_at,
updated_at: r.updated_at,
}))

21
crates/archivr-core/src/capture.rs
View file

@ -37,6 +37,11 @@ pub enum Source {
pub struct CaptureResult {
pub run_uid: String,
pub status: String,
/// `true` when uBlock was requested but the extension path was not found.
/// The capture succeeded without ad-blocking; the UI should warn the user.
pub ublock_skipped: bool,
/// `true` when cookie-consent extension was requested but the path was not found.
pub cookie_ext_skipped: bool,
}
#[derive(Debug, Clone, Default)]
@ -74,6 +79,12 @@ impl PlatformMetadata {
#[derive(Debug, Clone, Default)]
pub struct CaptureConfig {
pub cookie_rules: Vec<database::CookieRule>,
/// Override for uBlock Origin Lite during WebPage captures.
pub ublock_enabled: Option<bool>,
/// Override for cookie-consent extension during WebPage captures.
pub cookie_ext_enabled: Option<bool>,
/// Apply Mozilla Readability to distil the page to article content before archiving.
pub reader_mode: bool,
}
/// Resolves which cookies apply to `url` by evaluating all rules in ordinal order.
@ -991,6 +1002,8 @@ pub fn perform_capture(
return Ok(CaptureResult {
run_uid: run.run_uid.clone(),
status: "completed".to_string(),
ublock_skipped: false,
cookie_ext_skipped: false,
});
}
Err(e) => {
@ -1006,7 +1019,7 @@ pub fn perform_capture(
// Source: web page — archive as a self-contained HTML snapshot via single-file-cli
if source == Source::WebPage {
match downloader::singlefile::save(locator, store_path, &timestamp, &cookies) {
match downloader::singlefile::save(locator, store_path, &timestamp, &cookies, config.ublock_enabled, config.cookie_ext_enabled, config.reader_mode) {
Ok(result) => {
let file_extension = ".html".to_string();
let temp_html = store_path
@ -1133,6 +1146,8 @@ pub fn perform_capture(
return Ok(CaptureResult {
run_uid: run.run_uid.clone(),
status: "completed".to_string(),
ublock_skipped: result.ublock_skipped,
cookie_ext_skipped: result.cookie_ext_skipped,
});
}
Err(e) => {
@ -1187,6 +1202,8 @@ pub fn perform_capture(
return Ok(CaptureResult {
run_uid: run.run_uid.clone(),
status: "completed".to_string(),
ublock_skipped: false,
cookie_ext_skipped: false,
});
}
Err(e) => {
@ -1342,6 +1359,8 @@ pub fn perform_capture(
Ok(CaptureResult {
run_uid: run.run_uid.clone(),
status: "completed".to_string(),
ublock_skipped: false,
cookie_ext_skipped: false,
})
}

66
crates/archivr-core/src/database.rs
View file

@ -105,6 +105,7 @@ pub struct CaptureJobRecord {
pub run_uid: Option<String>,
pub status: String,
pub error_text: Option<String>,
pub notes_json: Option<String>,
pub created_at: String,
pub updated_at: String,
}
@ -136,6 +137,11 @@ pub struct InstanceSettings {
pub public_entry_content_enabled: bool,
pub open_registration_enabled: bool, // maps to public_archive_submission_enabled column
pub default_entry_visibility: u32,
/// Global default for ad-blocking via uBlock Origin Lite during WebPage captures.
/// Per-capture requests can override this.
pub ublock_enabled: bool,
/// Global default for cookie-consent banner dismissal via extension during WebPage captures.
pub cookie_ext_enabled: bool,
}
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
@ -194,7 +200,8 @@ pub fn initialize_schema(conn: &Connection) -> Result<()> {
id INTEGER PRIMARY KEY CHECK (id = 1),
public_index_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_index_enabled IN (0, 1)),
public_entry_content_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_entry_content_enabled IN (0, 1)),
public_archive_submission_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_archive_submission_enabled IN (0, 1))
public_archive_submission_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_archive_submission_enabled IN (0, 1)),
cookie_ext_enabled INTEGER NOT NULL DEFAULT 1 CHECK (cookie_ext_enabled IN (0, 1))
);
INSERT OR IGNORE INTO instance_settings (
@ -308,6 +315,7 @@ pub fn initialize_schema(conn: &Connection) -> Result<()> {
run_uid TEXT,
status TEXT NOT NULL CHECK (status IN ('pending','running','completed','failed')) DEFAULT 'pending',
error_text TEXT,
notes_json TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
@ -393,6 +401,10 @@ pub fn initialize_schema(conn: &Connection) -> Result<()> {
)?;
}
// Migration: add notes_json column to existing capture_jobs tables.
// Silently ignored when the column already exists (idempotent).
let _ = conn.execute("ALTER TABLE capture_jobs ADD COLUMN notes_json TEXT", []);
Ok(())
}
@ -454,7 +466,9 @@ pub fn initialize_auth_schema(conn: &Connection) -> Result<()> {
public_index_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_index_enabled IN (0, 1)),
public_entry_content_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_entry_content_enabled IN (0, 1)),
public_archive_submission_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_archive_submission_enabled IN (0, 1)),
default_entry_visibility INTEGER NOT NULL DEFAULT 2
default_entry_visibility INTEGER NOT NULL DEFAULT 2,
ublock_enabled INTEGER NOT NULL DEFAULT 1 CHECK (ublock_enabled IN (0, 1)),
cookie_ext_enabled INTEGER NOT NULL DEFAULT 1 CHECK (cookie_ext_enabled IN (0, 1))
);
INSERT OR IGNORE INTO instance_settings
@ -494,6 +508,17 @@ pub fn initialize_auth_schema(conn: &Connection) -> Result<()> {
[],
);
// Add ublock_enabled column to instance_settings if not present (idempotent migration)
let _ = conn.execute(
"ALTER TABLE instance_settings ADD COLUMN ublock_enabled INTEGER NOT NULL DEFAULT 1",
[],
);
// Add cookie_ext_enabled column to instance_settings if not present (idempotent migration)
let _ = conn.execute(
"ALTER TABLE instance_settings ADD COLUMN cookie_ext_enabled INTEGER NOT NULL DEFAULT 1",
[],
);
Ok(())
}
@ -718,7 +743,9 @@ pub fn list_user_tokens(conn: &Connection, user_id: i64) -> Result<Vec<ApiTokenR
pub fn get_instance_settings(conn: &Connection) -> Result<InstanceSettings> {
conn.query_row(
"SELECT public_index_enabled, public_entry_content_enabled,
public_archive_submission_enabled, default_entry_visibility
public_archive_submission_enabled, default_entry_visibility,
COALESCE(ublock_enabled, 1),
COALESCE(cookie_ext_enabled, 1)
FROM instance_settings WHERE id = 1",
[],
|row| {
@ -727,6 +754,8 @@ pub fn get_instance_settings(conn: &Connection) -> Result<InstanceSettings> {
public_entry_content_enabled: row.get::<_, i64>(1)? != 0,
open_registration_enabled: row.get::<_, i64>(2)? != 0,
default_entry_visibility: row.get::<_, i64>(3)? as u32,
ublock_enabled: row.get::<_, i64>(4)? != 0,
cookie_ext_enabled: row.get::<_, i64>(5)? != 0,
})
},
)
@ -739,13 +768,17 @@ pub fn update_instance_settings(conn: &Connection, settings: &InstanceSettings)
SET public_index_enabled = ?1,
public_entry_content_enabled = ?2,
public_archive_submission_enabled = ?3,
default_entry_visibility = ?4
default_entry_visibility = ?4,
ublock_enabled = ?5,
cookie_ext_enabled = ?6
WHERE id = 1",
params![
settings.public_index_enabled as i64,
settings.public_entry_content_enabled as i64,
settings.open_registration_enabled as i64,
settings.default_entry_visibility as i64,
settings.ublock_enabled as i64,
settings.cookie_ext_enabled as i64,
],
)?;
Ok(())
@ -1142,19 +1175,21 @@ pub fn create_capture_job(conn: &Connection, archive_id: &str) -> Result<String>
Ok(job_uid)
}
/// Updates the status (and optionally run_uid / error_text) of a capture job.
/// Updates the status (and optionally run_uid / error_text / notes_json) of a capture job.
pub fn update_capture_job_status(
conn: &Connection,
job_uid: &str,
status: &str,
run_uid: Option<&str>,
error_text: Option<&str>,
notes_json: Option<&str>,
) -> Result<()> {
let now = now_timestamp();
conn.execute(
"UPDATE capture_jobs SET status = ?1, run_uid = COALESCE(?2, run_uid),
error_text = ?3, updated_at = ?4 WHERE job_uid = ?5",
rusqlite::params![status, run_uid, error_text, now, job_uid],
error_text = ?3, notes_json = COALESCE(?4, notes_json), updated_at = ?5
WHERE job_uid = ?6",
rusqlite::params![status, run_uid, error_text, notes_json, now, job_uid],
)?;
Ok(())
}
@ -1162,7 +1197,7 @@ pub fn update_capture_job_status(
/// Returns a capture job by uid.
pub fn get_capture_job(conn: &Connection, job_uid: &str) -> Result<Option<CaptureJobRecord>> {
conn.query_row(
"SELECT job_uid, archive_id, run_uid, status, error_text, created_at, updated_at
"SELECT job_uid, archive_id, run_uid, status, error_text, notes_json, created_at, updated_at
FROM capture_jobs WHERE job_uid = ?1",
[job_uid],
|row| {
@ -1172,8 +1207,9 @@ pub fn get_capture_job(conn: &Connection, job_uid: &str) -> Result<Option<Captur
run_uid: row.get(2)?,
status: row.get(3)?,
error_text: row.get(4)?,
created_at: row.get(5)?,
updated_at: row.get(6)?,
notes_json: row.get(5)?,
created_at: row.get(6)?,
updated_at: row.get(7)?,
})
},
)
@ -2806,8 +2842,8 @@ mod tests {
fn capture_job_status_transitions() {
let conn = conn();
let job_uid = create_capture_job(&conn, "test").unwrap();
update_capture_job_status(&conn, &job_uid, "running", None, None).unwrap();
update_capture_job_status(&conn, &job_uid, "completed", Some("run_abc"), None).unwrap();
update_capture_job_status(&conn, &job_uid, "running", None, None, None).unwrap();
update_capture_job_status(&conn, &job_uid, "completed", Some("run_abc"), None, None).unwrap();
let job = get_capture_job(&conn, &job_uid).unwrap().unwrap();
assert_eq!(job.status, "completed");
assert_eq!(job.run_uid.as_deref(), Some("run_abc"));
@ -2819,7 +2855,7 @@ mod tests {
// Simulate an in-progress capture_job (run_uid still NULL — common crash case).
let uid = create_capture_job(&conn, "test").unwrap();
update_capture_job_status(&conn, &uid, "running", None, None).unwrap();
update_capture_job_status(&conn, &uid, "running", None, None, None).unwrap();
// Simulate an in-progress archive_run and item with no associated capture_job
// (covers the case where run_uid was never written back before the crash).
@ -3177,7 +3213,7 @@ mod tests {
fn has_active_capture_jobs_true_for_running() {
let conn = conn();
let uid = create_capture_job(&conn, "test").unwrap();
update_capture_job_status(&conn, &uid, "running", None, None).unwrap();
update_capture_job_status(&conn, &uid, "running", None, None, None).unwrap();
assert!(has_active_capture_jobs(&conn).unwrap());
}
@ -3185,7 +3221,7 @@ mod tests {
fn has_active_capture_jobs_false_for_completed() {
let conn = conn();
let uid = create_capture_job(&conn, "test").unwrap();
update_capture_job_status(&conn, &uid, "completed", Some("run_x"), None).unwrap();
update_capture_job_status(&conn, &uid, "completed", Some("run_x"), None, None).unwrap();
assert!(!has_active_capture_jobs(&conn).unwrap());
}

672
crates/archivr-core/src/downloader/singlefile.rs
View file

@ -1,11 +1,99 @@
use anyhow::{Context, Result, bail};
use base64::engine::general_purpose::STANDARD as B64;
use base64::Engine as _;
use std::{collections::HashMap, env, io::Read, path::Path, process::Command};
use std::{
collections::HashMap,
env,
io::Read,
path::{Path, PathBuf},
process::Command,
};
use crate::downloader::cookies::{domain_from_url, write_netscape_cookie_file};
use crate::hash::hash_file;
/// Combined reader-mode script: Readability.js (Apache 2.0) bundled with the
/// archivr wrapper in a single IIFE. single-file-cli concatenates all
/// `--browser-script` files into one string before injection (scripts.js:84),
/// so scope sharing is guaranteed; the combined file is kept for clarity.
///
/// Emits `<meta name="archivr-reader-mode" content="applied|failed:REASON">`
/// so the outcome is observable in the saved HTML.
const READER_MODE_SCRIPT: &str = concat!(
// Readability.js is injected verbatim first so `Readability` is in scope.
include_str!("../../../../vendor/readability/Readability.js"),
// Wrapper IIFE — runs on single-file-on-before-capture-request.
// Sets 'installed' immediately at script-evaluation time so a missing meta
// means the browser-script was never injected at all.
r#"
;(function() {
function _archivrReaderMark(content) {
try {
var m = document.querySelector('meta[name="archivr-reader-mode"]');
if (!m) {
m = document.createElement('meta');
m.name = 'archivr-reader-mode';
(document.head || document.documentElement).appendChild(m);
}
m.content = content;
} catch(_) {}
}
// Mark immediately: if this meta is absent in the artifact the script
// was never injected (separate from the hook never firing).
_archivrReaderMark('installed');
function _archivrApplyReader() {
try {
if (typeof Readability === 'undefined') {
_archivrReaderMark('failed:no-readability');
return;
}
var article = new Readability(document.cloneNode(true)).parse();
if (!article || !article.content || article.content.length < 100) {
_archivrReaderMark('failed:no-article');
return;
}
document.body.innerHTML = article.content;
if (article.title) document.title = article.title;
var hdr = document.createElement('header');
hdr.innerHTML =
'<h1 style="margin:0 0 .3em;font-family:-apple-system,sans-serif">' +
(article.title || '') + '</h1>' +
(article.byline
? '<p style="margin:0;color:#666;font-size:14px">' + article.byline + '</p>'
: '') +
(article.siteName
? '<p style="margin:.2em 0 0;color:#999;font-size:12px">' + article.siteName + '</p>'
: '');
hdr.style.cssText = 'margin-bottom:2em;padding-bottom:1em;border-bottom:1px solid #ddd';
document.body.insertBefore(hdr, document.body.firstChild);
var style = document.createElement('style');
style.textContent = [
'body{max-width:680px;margin:40px auto;padding:0 24px;',
'font-family:Georgia,"Times New Roman",serif;font-size:18px;',
'line-height:1.75;color:#1a1a1a;background:#fafaf8}',
'h1,h2,h3,h4,h5,h6{font-family:-apple-system,BlinkMacSystemFont,sans-serif;',
'line-height:1.3;margin-top:1.5em}',
'img,figure,video{max-width:100%;height:auto;display:block;margin:1em 0}',
'a{color:#0055cc}',
'pre{background:#f4f4f4;padding:1em;border-radius:4px;overflow-x:auto;font-size:14px}',
'code{background:#f4f4f4;padding:.1em .3em;border-radius:3px;font-size:14px}',
'blockquote{border-left:3px solid #ccc;margin:1em 0;padding-left:1.2em;color:#555}',
].join('');
document.head.appendChild(style);
_archivrReaderMark('applied');
} catch (e) {
_archivrReaderMark('failed:exception:' + (e && e.message ? e.message : String(e)));
}
}
// Ensure _singleFile_waitForUserScript is installed (strip-scripts also does
// this, but be explicit here in case reader-mode ever runs without it).
dispatchEvent(new CustomEvent('single-file-user-script-init'));
// Synchronous work — no preventDefault()/response dispatch needed.
addEventListener('single-file-on-before-capture-request', _archivrApplyReader);
})();
"#
);
/// Result of archiving a web page with single-file.
#[derive(Debug)]
pub struct SaveResult {
@ -17,26 +105,146 @@ pub struct SaveResult {
pub favicon_hash: Option<String>,
/// File extension for the favicon (e.g. `".ico"`, `".png"`), if present.
pub favicon_ext: Option<String>,
/// `true` when `ARCHIVR_UBLOCK=true` (the default) but the extension path
/// was missing or invalid. The capture succeeded but ran without ad-blocking.
pub ublock_skipped: bool,
/// `true` when `ARCHIVR_COOKIE_CONSENT=true` (the default) but the extension path
/// was missing or invalid. The capture succeeded but ran without cookie-consent blocking.
pub cookie_ext_skipped: bool,
}
/// Archives `url` as a self-contained HTML snapshot.
///
/// Returns `(sha256_hex, title_hint)` on success.
/// - `sha256_hex`: hash of the saved `.html` file, used as the blob key.
/// - `title_hint`: page title extracted from the `<title>` tag, if present.
///
/// Reads two env vars:
/// Env vars:
/// - `ARCHIVR_SINGLE_FILE`: path to the `single-file` binary (default: `"single-file"`).
/// - `ARCHIVR_CHROME`: path to the Chromium/Chrome binary (default: `"chromium"`).
pub fn save(url: &str, store_path: &Path, timestamp: &str, cookies: &HashMap<String, String>) -> Result<SaveResult> {
/// - `ARCHIVR_UBLOCK`: enable uBlock Origin Lite extension (default: `"true"`).
/// - `ARCHIVR_UBLOCK_EXT`: path to the unpacked uBlock Origin Lite extension directory.
/// - `ARCHIVR_CHROME_ARGS`: space-separated extra Chrome flags (e.g. `"--no-sandbox"`).
pub fn save(
url: &str,
store_path: &Path,
timestamp: &str,
cookies: &HashMap<String, String>,
ublock_enabled_override: Option<bool>,
cookie_ext_enabled: Option<bool>,
reader_mode: bool,
) -> Result<SaveResult> {
let single_file =
env::var("ARCHIVR_SINGLE_FILE").unwrap_or_else(|_| "single-file".to_string());
let chrome = env::var("ARCHIVR_CHROME").unwrap_or_else(|_| "chromium".to_string());
save_with(url, store_path, timestamp, &single_file, &chrome, cookies)
let (ublock_ext, ublock_skipped) = resolve_ublock_config(ublock_enabled_override);
let (cookie_ext, cookie_ext_skipped) = resolve_cookie_ext_config(cookie_ext_enabled);
let mut result = save_with(
url,
store_path,
timestamp,
&single_file,
&chrome,
cookies,
ublock_ext.as_deref(),
cookie_ext.as_deref(),
reader_mode,
)?;
result.ublock_skipped = ublock_skipped;
result.cookie_ext_skipped = cookie_ext_skipped;
Ok(result)
}
/// Inner implementation; takes binary paths explicitly so tests can inject them
/// without mutating process-global environment variables.
/// Resolves uBlock configuration from env vars, optionally overridden by the caller.
///
/// Returns:
/// - `(Some(path), false)` — uBlock is enabled and the extension dir is valid.
/// - `(None, true)` — uBlock is enabled but the extension dir is missing/invalid
/// (warns to stderr; the capture proceeds without ad-blocking).
/// - `(None, false)` — uBlock is disabled (`ARCHIVR_UBLOCK=false` or overridden).
fn resolve_ublock_config(enabled_override: Option<bool>) -> (Option<PathBuf>, bool) {
// The override (from instance settings or per-capture body) takes precedence over env.
let want_ublock = enabled_override.unwrap_or_else(|| {
let env_val = env::var("ARCHIVR_UBLOCK").unwrap_or_else(|_| "true".to_string());
!env_val.eq_ignore_ascii_case("false") && env_val != "0"
});
if !want_ublock {
return (None, false);
}
match env::var("ARCHIVR_UBLOCK_EXT").ok().filter(|s| !s.is_empty()) {
None => {
eprintln!(
"warn: uBlock: ARCHIVR_UBLOCK_EXT is not set; \
capturing without ad-blocking"
);
(None, true)
}
Some(ext_path_str) => {
let path = PathBuf::from(&ext_path_str);
if path.is_dir() {
(Some(path), false)
} else {
eprintln!(
"warn: uBlock: ARCHIVR_UBLOCK_EXT={ext_path_str:?} is not a directory; \
capturing without ad-blocking"
);
(None, true)
}
}
}
}
/// Resolves cookie-consent extension configuration from env vars, optionally overridden by the caller.
///
/// Returns:
/// - `(Some(path), false)` — cookie-consent ext is enabled and the extension dir is valid.
/// - `(None, true)` — cookie-consent ext is enabled but the extension dir is missing/invalid
/// (warns to stderr; the capture proceeds without cookie-consent blocking).
/// - `(None, false)` — cookie-consent ext is disabled (`ARCHIVR_COOKIE_CONSENT=false` or overridden).
fn resolve_cookie_ext_config(enabled_override: Option<bool>) -> (Option<PathBuf>, bool) {
let want_cookie_ext = enabled_override.unwrap_or_else(|| {
let env_val = env::var("ARCHIVR_COOKIE_CONSENT").unwrap_or_else(|_| "true".to_string());
!env_val.eq_ignore_ascii_case("false") && env_val != "0"
});
if !want_cookie_ext {
return (None, false);
}
match env::var("ARCHIVR_COOKIE_EXT").ok().filter(|s| !s.is_empty()) {
None => {
eprintln!(
"warn: cookie-consent: ARCHIVR_COOKIE_EXT is not set; \
capturing without cookie-consent blocking"
);
(None, true)
}
Some(ext_path_str) => {
let path = PathBuf::from(&ext_path_str);
if path.is_dir() {
(Some(path), false)
} else {
eprintln!(
"warn: cookie-consent: ARCHIVR_COOKIE_EXT={ext_path_str:?} is not a directory; \
capturing without cookie-consent blocking"
);
(None, true)
}
}
}
}
/// Inner implementation. Takes binary paths and an optional uBlock extension
/// directory explicitly so tests can inject them without touching env vars.
///
/// single-file always manages Chrome. When `ublock_ext` is `Some(path)`, the
/// extension is loaded by passing `--headless=new`, `--load-extension`, and
/// `--disable-extensions-except` inside the `--browser-args` JSON array.
/// single-file's `browser.js` prefix-strips its own conflicting flags before
/// appending ours, so `--headless=new` overrides its default `--headless`.
///
/// Note: single-file always adds `--single-process` to Chrome. uBOL's
/// `declarativeNetRequest` **static** rulesets are registered by Chrome's
/// network stack at extension load time (not by a service worker), so they are
/// expected to apply even in single-process mode. Extension service-worker
/// initialisation may fail silently; this does not affect the static filter
/// lists. Ad-blocking has not been mechanically verified under `--single-process`
/// — if a future test confirms otherwise, consider owning Chrome's lifecycle and
/// using a dedicated `--remote-debugging-port` without `--single-process`.
fn save_with(
url: &str,
store_path: &Path,
@ -44,59 +252,127 @@ fn save_with(
single_file: &str,
chrome: &str,
cookies: &HashMap<String, String>,
ublock_ext: Option<&Path>,
cookie_ext: Option<&Path>,
reader_mode: bool,
) -> Result<SaveResult> {
let temp_dir = store_path.join("temp").join(timestamp);
std::fs::create_dir_all(&temp_dir).context("failed to create temp dir")?;
let out_file = temp_dir.join(format!("{timestamp}.html"));
// Write a user script that strips <script> elements from the live DOM
// just before SingleFile serializes it. This lets scripts execute during
// capture (so JS-applied CSS is present) without leaving data:-URL ES
// modules in the saved file that would cause "base scheme isn't
// hierarchical" errors in the viewer. JSON-LD structured data is kept.
let user_script_path = temp_dir.join("sf-strip-scripts.js");
std::fs::write(
&user_script_path,
"addEventListener('single-file-on-before-capture-start',()=>{\
document.querySelectorAll('script:not([type=\"application/ld+json\"])')\
.forEach(el=>el.remove());\
});",
)
.context("failed to write single-file user script")?;
// Mandatory user script: strips <script> elements before SingleFile
// serialises so JS-applied CSS is captured without broken module imports.
// When cookie_ext is active, also resets overflow lockout and removes
// consent overlays the extension may have missed.
let strip_scripts_path = temp_dir.join("sf-strip-scripts.js");
let mut strip_scripts = String::from(
// Dispatch single-file-user-script-init so single-file installs
// _singleFile_waitForUserScript, which gates the -request hooks.
"dispatchEvent(new CustomEvent('single-file-user-script-init'));\
addEventListener('single-file-on-before-capture-request',()=>{\
document.querySelectorAll('script:not([type=\"application/ld+json\"])')\
.forEach(el=>el.remove());",
);
if cookie_ext.is_some() {
// Reset overflow:hidden that consent modals inject on body/html.
// Gate on cookie_ext so we never mutate pages where the feature is off.
strip_scripts.push_str(
"document.body&&(document.body.style.overflow='');\
document.documentElement&&(document.documentElement.style.overflow='');\
/* Remove consent overlays the extension may have missed \
* (e.g. Google Funding Choices, Quantcast, Sourcepoint). \
* Selectors are specific to consent infrastructure, not content. */\
document.querySelectorAll(\
'.fc-consent-root,.fc-dialog-overlay,.fc-dialog,\
.qc-cmp2-container,.qc-cmp2-ui,\
.sp-message-container,\
#sp-cc,\
#usercentrics-root'\
).forEach(function(el){el.remove();});",
);
}
if ublock_ext.is_some() {
// uBlock blocks ad network requests but first-party ad placeholder
// elements (ins.adsbygoogle, iframe hosts) retain their computed
// height, leaving blank space. Remove them pre-capture.
strip_scripts.push_str(
"document.querySelectorAll(\
'ins.adsbygoogle,\
[id^=\"aswift_\"],\
iframe[id^=\"google_ads_\"],\
iframe[name^=\"google_ads_frame\"],\
iframe[src*=\"googlesyndication\"],\
iframe[src*=\"doubleclick\"]'\
).forEach(function(el){\
/* Walk up to the nearest ad-slot container so padding/margin \
* on the wrapper (e.g. .top-ad, .google-auto-placed) collapses \
* too, not just the inner ins/iframe element. */\
var slot=el.closest('.top-ad,.google-auto-placed,.ad-slot,.ad-container');\
(slot||el).remove();\
});",
);
}
strip_scripts.push_str("});");
std::fs::write(&strip_scripts_path, &strip_scripts)
.context("failed to write single-file user script")?;
// Chrome's user-data-dir for this capture. Required alongside
// --disable-web-security — newer Chromium silently ignores that flag
// without a writable user-data-dir. Using a subdirectory of temp_dir
// keeps it isolated and it gets cleaned up with the rest of the temp dir.
// Optional reader-mode script: Readability.js + wrapper combined into one
// file so both run in the same execution scope. (Separate --browser-script
// files can each get their own context depending on single-file version.)
let mut extra_browser_scripts: Vec<PathBuf> = Vec::new();
if reader_mode {
let reader_path = temp_dir.join("sf-reader-mode.js");
std::fs::write(&reader_path, READER_MODE_SCRIPT)
.context("failed to write reader-mode script")?;
extra_browser_scripts.push(reader_path);
}
// Isolated Chrome profile directory; cleaned up with the rest of temp.
let chrome_data_dir = temp_dir.join("chrome-data");
// Build the browser-args JSON array. Start with the flags always required,
// then append any extra flags from ARCHIVR_CHROME_ARGS (space-separated).
// Docker containers running as root need "--no-sandbox" here because
// Chromium refuses to start as root without it.
//
// --window-size is set to a realistic desktop viewport so that
// --remove-alternative-medias=false and --remove-unused-styles=false
// actually preserve responsive @media rules and styles that only match
// at normal screen widths (headless Chromium defaults to a small viewport
// that would otherwise defeat the preservation flags).
// Build Chrome flags passed via --browser-args to single-file.
// single-file's browser.js overrides its own defaults with whatever we
// pass here (it strips conflicting flags by prefix before appending ours).
let mut chrome_flags = vec![
"--disable-web-security".to_string(),
format!("--user-data-dir={}", chrome_data_dir.display()),
"--window-size=1920,1080".to_string(),
];
if let Ok(extra) = std::env::var("ARCHIVR_CHROME_ARGS") {
chrome_flags.extend(extra.split_whitespace().filter(|s| !s.is_empty()).map(str::to_string));
// Build comma-separated extension list for Chrome flags.
// --headless=new is required for --load-extension to work.
let ext_paths: Vec<PathBuf> = [ublock_ext, cookie_ext]
.iter()
.filter_map(|p| p.map(|p| p.to_path_buf()))
.collect();
if !ext_paths.is_empty() {
let joined = ext_paths
.iter()
.map(|p| p.display().to_string())
.collect::<Vec<_>>()
.join(",");
chrome_flags.push("--headless=new".to_string());
chrome_flags.push(format!("--load-extension={joined}"));
chrome_flags.push(format!("--disable-extensions-except={joined}"));
}
// Operator extras (e.g. --no-sandbox in Docker).
let extra_chrome_args: Vec<String> = env::var("ARCHIVR_CHROME_ARGS")
.unwrap_or_default()
.split_whitespace()
.filter(|s| !s.is_empty())
.map(str::to_string)
.collect();
chrome_flags.extend(extra_chrome_args);
// single-file expects browser-args as a JSON array of strings.
let quoted: Vec<String> = chrome_flags
.iter()
.map(|f| format!("\"{}\"", f.replace('\\', "\\\\").replace('"', "\\\"")))
.collect();
let browser_args = format!("[{}]", quoted.join(","));
// Write cookie file if cookies are provided.
// Never pass cookie values in process args (ps exposure).
let cookie_file: Option<std::path::PathBuf> = if !cookies.is_empty() {
// Write cookie file (secrets must never appear in process args).
let cookie_file: Option<PathBuf> = if !cookies.is_empty() {
let cf = temp_dir.join("cookies.txt");
let domain = domain_from_url(url);
write_netscape_cookie_file(cookies, &domain, &cf)
@ -106,70 +382,126 @@ fn save_with(
None
};
let mut cmd = Command::new(single_file);
cmd.arg(url)
.arg(&out_file)
.arg(format!("--browser-executable-path={chrome}"))
.arg("--browser-headless")
.arg("--browser-wait-until=networkidle2")
// Extra delay after networkidle2: Cloudflare Fonts injects @font-face
// CSS after HTML parse, so the font hook needs more time to see it.
.arg("--browser-wait-delay=2000")
// Realistic UA: some origins block headless Chrome's default UA string.
.arg("--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36")
// Chrome-level flags: disable CORS so fonts from any CDN origin can be
// read and inlined (e.g. fonts.gstatic.com without ACAO:*).
.arg(format!("--browser-args={browser_args}"))
// Preserve all CSS: single-file's defaults strip rules it considers
// "unused" (breaks CSS nesting) and remove @media blocks that don't
// match the capture viewport (breaks responsive layout).
.arg("--remove-unused-styles=false")
.arg("--remove-alternative-medias=false")
// Allow scripts to run during capture so JS-applied classes exist in
// the DOM when CSS is evaluated. The user script above strips <script>
// elements before serialization so no broken module imports end up in
// the saved file.
.arg("--block-scripts=false")
.arg(format!("--browser-script={}", user_script_path.display()))
// Preserve fonts: defaults strip @font-face rules deemed "unused" or
// "alternative" (unicode-range subsets), losing CDN-served fonts.
.arg("--remove-unused-fonts=false")
.arg("--remove-alternative-fonts=false");
if let Some(cf) = &cookie_file {
cmd.arg(format!("--browser-cookies-file={}", cf.display()));
}
let spawn_result = cmd
.output()
.with_context(|| format!("failed to spawn {single_file} process"));
let mut scripts: Vec<&Path> = vec![strip_scripts_path.as_path()];
scripts.extend(extra_browser_scripts.iter().map(|p| p.as_path()));
// Delete cookie file unconditionally — including on spawn failure —
// so secrets are never left in store/temp when the capture fails.
let sf_output = run_single_file_standalone(
url,
&out_file,
single_file,
chrome,
&browser_args,
&scripts,
cookie_file.as_deref(),
)
.with_context(|| format!("failed to spawn single-file ({single_file})"))?;
// Delete cookie file unconditionally — including on failure — so secrets
// are never left in store/temp when the capture fails.
if let Some(cf) = &cookie_file {
let _ = std::fs::remove_file(cf);
}
let out = spawn_result?;
if !out.status.success() {
let stderr = String::from_utf8_lossy(&out.stderr);
bail!("single-file failed: {stderr}");
if !sf_output.status.success() {
let stderr = String::from_utf8_lossy(&sf_output.stderr);
bail!("single-file failed (exit {:?}): {stderr}", sf_output.status.code());
}
if !out_file.exists() {
// Collect diagnostics: stdout, stderr, and what's actually in the temp dir.
let stdout = String::from_utf8_lossy(&sf_output.stdout);
let stderr = String::from_utf8_lossy(&sf_output.stderr);
let dir_contents: String = std::fs::read_dir(&temp_dir)
.map(|rd| {
rd.filter_map(|e| e.ok())
.map(|e| e.file_name().to_string_lossy().into_owned())
.collect::<Vec<_>>()
.join(", ")
})
.unwrap_or_else(|_| "<unreadable>".to_string());
eprintln!(
"warn: single-file produced no file at {}\n temp dir contents: [{dir_contents}]\n stderr: {}\n stdout (first 200 chars): {}",
out_file.display(),
stderr.trim(),
&stdout[..stdout.len().min(200)],
);
bail!(
"single-file exited successfully but produced no output file at {}",
out_file.display()
"single-file exited successfully but produced no output file at {}; \
temp dir contains: [{dir_contents}]; \
stderr: {}",
out_file.display(),
stderr.trim(),
);
}
let title = extract_html_title(&out_file);
let html_hash = hash_file(&out_file)?;
let (favicon_hash, favicon_ext) = extract_and_save_favicon(&out_file, &temp_dir, timestamp)
.map(|(h, e)| (Some(h), Some(e)))
.unwrap_or((None, None));
Ok(SaveResult { html_hash, title, favicon_hash, favicon_ext })
let (favicon_hash, favicon_ext) =
extract_and_save_favicon(&out_file, &temp_dir, timestamp)
.map(|(h, e)| (Some(h), Some(e)))
.unwrap_or((None, None));
Ok(SaveResult {
html_hash,
title,
favicon_hash,
favicon_ext,
ublock_skipped: false, // overwritten by save() from resolve_ublock_config()
cookie_ext_skipped: false, // overwritten by save() from resolve_cookie_ext_config()
})
}
/// Runs single-file, letting it launch and manage Chrome itself.
fn run_single_file_standalone(
url: &str,
out_file: &Path,
single_file: &str,
chrome: &str,
browser_args: &str,
scripts: &[&Path],
cookie_file: Option<&Path>,
) -> std::io::Result<std::process::Output> {
let mut cmd = base_single_file_cmd(url, out_file, single_file, scripts, cookie_file);
cmd.arg(format!("--browser-executable-path={chrome}"))
.arg("--browser-headless")
.arg(format!("--browser-args={browser_args}"));
cmd.output()
}
/// Builds a `Command` with the single-file args that are the same regardless
/// of how Chrome is started. Passes each script as a separate `--browser-script` arg.
fn base_single_file_cmd(
url: &str,
out_file: &Path,
single_file: &str,
scripts: &[&Path],
cookie_file: Option<&Path>,
) -> Command {
let mut cmd = Command::new(single_file);
cmd.arg(url)
.arg(out_file)
.arg("--browser-wait-until=networkidle2")
.arg("--browser-wait-delay=2000")
.arg("--user-agent=Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36")
.arg("--remove-unused-styles=false")
.arg("--remove-alternative-medias=false")
.arg("--block-scripts=false")
.arg("--remove-unused-fonts=false")
.arg("--remove-alternative-fonts=false")
// Explicitly prevent single-file from dumping HTML to stdout instead of
// writing the file (its Docker-detection heuristic can trigger on some setups).
.arg("--dump-content=false");
for script in scripts {
cmd.arg(format!("--browser-script={}", script.display()));
}
if let Some(cf) = cookie_file {
cmd.arg(format!("--browser-cookies-file={}", cf.display()));
}
cmd
}
// ── HTML helpers ──────────────────────────────────────────────────────────────
/// Reads the first 8 KiB of `path` and extracts the content of the first
/// `<title>…</title>` element. Returns `None` if absent or empty.
///
@ -177,21 +509,17 @@ fn save_with(
/// lowercasing is byte-length-preserving, so byte offsets derived from the
/// lowercased buffer are valid indices into the original buffer.
fn extract_html_title(path: &Path) -> Option<String> {
let mut buf = [0u8; 8192];
let n = std::fs::File::open(path).ok()?.read(&mut buf).ok()?;
// Recover a valid UTF-8 prefix if the 8 KiB boundary falls mid-character.
let snippet = match std::str::from_utf8(&buf[..n]) {
Ok(s) => s,
Err(e) => std::str::from_utf8(&buf[..e.valid_up_to()]).ok()?,
};
// ASCII-only lowercase: A-Z -> a-z, all other bytes unchanged.
// Byte lengths are identical to the original, so offsets are safe to reuse.
let lower = snippet.to_ascii_lowercase();
let tag_start = lower.find("<title>")?;
let content_start = tag_start + 7; // len("<title>") == 7
let content_end = content_start + lower[content_start..].find("</title>")?;
let title = snippet[content_start..content_end].trim();
if title.is_empty() { None } else { Some(title.to_string()) }
let mut f = std::fs::File::open(path).ok()?;
let mut buf = vec![0u8; 8192];
let n = f.read(&mut buf).ok()?;
let buf = &buf[..n];
let lower = String::from_utf8_lossy(buf).to_ascii_lowercase();
let start = lower.find("<title>")? + "<title>".len();
let end = lower[start..].find("</title>")? + start;
let title = String::from_utf8_lossy(&buf[start..end])
.trim()
.to_string();
if title.is_empty() { None } else { Some(title) }
}
/// Extracts the favicon embedded in a single-file HTML archive.
@ -205,71 +533,54 @@ fn extract_and_save_favicon(
temp_dir: &Path,
timestamp: &str,
) -> Option<(String, String)> {
let html = std::fs::read_to_string(html_path).ok()?;
let lower = html.to_ascii_lowercase();
let content = std::fs::read_to_string(html_path).ok()?;
let lower = content.to_ascii_lowercase();
// Find a <link> tag that has rel="...icon..." AND href="data:image/..."
let link_start = {
let mut found = None;
let mut search = 0;
while search < lower.len() {
let off = lower[search..].find("<link")?;
let abs = search + off;
// Find end of this tag, respecting quoted attribute values so that
// a '>' inside a data URL does not terminate the tag prematurely.
let tag_slice = &lower[abs..];
let mut in_q = false;
let mut tag_end = None;
for (i, c) in tag_slice.char_indices() {
match c {
'"' => in_q = !in_q,
'>' if !in_q => { tag_end = Some(i); break; }
_ => {}
// Find the first <link …> tag that looks like a favicon with a data: href.
let mut search_pos = 0;
loop {
let tag_start = lower[search_pos..].find("<link")? + search_pos;
let tag_end = lower[tag_start..].find('>')? + tag_start;
let tag = &lower[tag_start..=tag_end];
if tag.contains("icon") {
// Look for href="data:image/...;base64,..."
if let Some(href_pos) = tag.find("href=") {
let after_href = &content[tag_start + href_pos + 5..];
let (quote, after_quote) = if after_href.starts_with('"') {
('"', &after_href[1..])
} else if after_href.starts_with('\'') {
('\'', &after_href[1..])
} else {
search_pos = tag_end + 1;
continue;
};
let value_end = after_quote.find(quote)?;
let href_value = &after_quote[..value_end];
if let Some(b64_start) = href_value.to_ascii_lowercase().find(";base64,") {
let mime_part = &href_value[5..b64_start]; // skip "data:"
let ext = mime_to_favicon_ext(mime_part)?;
let b64_data = &href_value[b64_start + 8..];
let bytes = B64.decode(b64_data).ok()?;
let out_path = temp_dir.join(format!("{timestamp}.favicon{ext}"));
std::fs::write(&out_path, &bytes).ok()?;
let hash = hash_file(&out_path).ok()?;
return Some((hash, ext.to_string()));
}
}
let tag_end = match tag_end { Some(e) => e, None => break };
let tag_s = &lower[abs..abs + tag_end];
if tag_s.contains("rel=") && tag_s.contains("icon") && tag_s.contains("href=\"data:image") {
found = Some(abs);
break;
}
search = abs + tag_end + 1;
}
found?
};
// Extract href value from the original HTML (byte positions match because
// to_ascii_lowercase is byte-length-preserving).
let tag_lower = &lower[link_start..];
let href_off = tag_lower.find("href=\"")?;
let value_start = link_start + href_off + 6; // past href="
let value_end = html[value_start..].find('"')?;
let data_url = &html[value_start..value_start + value_end];
// Parse data:<mime>;base64,<payload>
let rest = data_url.strip_prefix("data:")?;
let comma = rest.find(',')?;
let meta = &rest[..comma];
let b64 = &rest[comma + 1..];
if !meta.to_ascii_lowercase().contains("base64") {
return None;
search_pos = tag_end + 1;
}
let mime = meta.split(';').next()?.trim().to_ascii_lowercase();
let ext = mime_to_favicon_ext(&mime)?;
let bytes = B64.decode(b64.trim()).ok()?;
let out = temp_dir.join(format!("{timestamp}.favicon{ext}"));
std::fs::write(&out, &bytes).ok()?;
hash_file(&out).ok().map(|h| (h, ext.to_string()))
}
fn mime_to_favicon_ext(mime: &str) -> Option<&'static str> {
match mime {
match mime.to_ascii_lowercase().trim() {
"image/x-icon" | "image/vnd.microsoft.icon" => Some(".ico"),
"image/png" => Some(".png"),
"image/svg+xml" => Some(".svg"),
"image/png" => Some(".png"),
"image/jpeg" => Some(".jpg"),
"image/gif" => Some(".gif"),
"image/gif" => Some(".gif"),
"image/svg+xml" => Some(".svg"),
"image/webp" => Some(".webp"),
_ => None,
}
@ -323,6 +634,9 @@ mod tests {
"/nonexistent/single-file",
"chromium",
&HashMap::new(),
None, // no ublock ext
None, // no cookie ext
false, // reader mode off
);
let err = result.unwrap_err();
let msg = format!("{err:#}");
@ -331,4 +645,42 @@ mod tests {
"unexpected error: {msg}"
);
}
#[test]
fn save_with_both_extensions_uses_comma_joined_flags() {
use std::path::Path;
// We can't run single-file here, but we can exercise the flag-building
// logic by checking the path list construction directly.
let ublock = Path::new("/tmp/ublock");
let cookie = Path::new("/tmp/cookie");
let ext_paths: Vec<std::path::PathBuf> = [Some(ublock), Some(cookie)]
.iter()
.filter_map(|p| p.map(|p| p.to_path_buf()))
.collect();
let joined = ext_paths
.iter()
.map(|p| p.display().to_string())
.collect::<Vec<_>>()
.join(",");
assert_eq!(joined, "/tmp/ublock,/tmp/cookie");
let load_flag = format!("--load-extension={joined}");
let except_flag = format!("--disable-extensions-except={joined}");
assert_eq!(load_flag, "--load-extension=/tmp/ublock,/tmp/cookie");
assert_eq!(except_flag, "--disable-extensions-except=/tmp/ublock,/tmp/cookie");
}
#[test]
fn resolve_ublock_config_disabled_when_false() {
// Can't mutate env vars safely in parallel tests; test the logic directly
// by verifying the env-var parsing branch we care about.
let enabled = "false";
let is_disabled =
enabled.eq_ignore_ascii_case("false") || enabled == "0";
assert!(is_disabled);
let enabled = "0";
let is_disabled =
enabled.eq_ignore_ascii_case("false") || enabled == "0";
assert!(is_disabled);
}
}