routes.rs:
- CaptureBody gains per_item_quality (HashMap<String,String>, serde(default))
and sync (bool, serde(default)); both validated before use
- per_item_quality values validated against same quality predicate as top-level
quality field ("best"|"audio"|"NNNp") so bad per-video values are rejected
at the API boundary rather than silently falling through to quality_format
- capture_handler threads body.per_item_quality + body.sync into CaptureConfig
(replaces hardcoded empty stubs); rearchive_handler keeps empty defaults
- New POST /api/archives/:id/captures/probe-playlist: calls
probe_playlist_qualities via spawn_blocking; 400 for non-playlist locator,
502 on yt-dlp failure, returns PlaylistProbeResult as JSON
api.js:
- probePlaylist(archiveId, locator): POST probe-playlist endpoint
- submitCapture: forwards per_item_quality (non-empty) and sync:true from
extraExtensions param added to submitBgJob
CaptureDialog.jsx:
- isPlaylistSource(): detects yt:/youtube: playlist/@/channel, ytm:playlist/,
YouTube/YTM HTTP(S) URLs with list= param or channel pathnames
- makeItem(): 6 new playlist state fields
- applyPlaylistQuality(): conflict logic — videos that can reach selected
quality get it set; videos that can't and have no prior selection are left
null (conflict); videos with a prior selection keep it when quality is raised
- hasConflict(): any playlistItems entry with quality===null
- updateLocator(): isPlaylistSource branch with 800ms debounce→probePlaylist;
existing isVideoSource path unchanged
- Archive button disabled when anyConflict or any probe in flight
- Per-video expand list with individual quality selects, conflict badges,
sync toggle (appears after probe completes)
styles.css: playlist expansion, conflict, sync toggle CSS
ytdlp.rs:
- Add PlaylistItemProbe / PlaylistProbeResult (pub, serde::Serialize)
- Add private available_video_heights_from_value() helper for Value entries
- Add probe_playlist_qualities(): yt-dlp -J (full metadata, no flat flag)
returns per-video quality lists in one subprocess call
capture.rs:
- Add per_item_quality: HashMap<String,String> and sync: bool to CaptureConfig
(both Default; keyed by yt-dlp video ID, not URL)
- Add pub locator_to_playlist_url(): validates only the three playlist sources,
expands shorthands; keeps locator_to_ytdlp_url's no-playlists contract
- Playlist capture block: sync-aware container resolution
- sync + existing container → reuse it via complete_archive_run_item,
skip already-archived children (by canonical URL) before creating
run items so refresh_run_counters only counts new items
- sync + no container → create normally (first sync run)
- non-sync → always create fresh container (existing behaviour)
- Per-item quality: config.per_item_quality.get(id) falls back to child_quality
archive.rs:
- Add get_archived_playlist_child_urls(): returns HashSet of canonical URLs
of all children under any container matching the playlist canonical URL
- Add find_container_entry_id_by_canonical_url(): returns most-recent
container entry id (parent_entry_id IS NULL) for a given canonical URL
routes.rs: stub per_item_quality/sync on both CaptureConfig sites (server
agent will wire body fields in Phase 2)
server:
- add GET /api/archives/:id/entries/:uid/children → list_entry_children,
calling list_child_entries with caller_bits so visibility model is enforced
- fix capture_handler: use result.status ("completed"/"failed") to set job
status rather than always "completed", mirroring rearchive_handler; this
surfaces partial playlist failures to the polling client
frontend:
- api.js: add fetchEntryChildren(archiveId, entryUid)
- EntryRow: one outer div.entry-row-outer (display:block) keeps nth-child
striping correct; inner div.entry-row-main is the flex row with all column
cells and event handling; .child-entries sits below inside the outer wrapper
- expand chevron appears when entry.child_count > 0; clicking fetches children
lazily and renders ChildRow components reusing .col-* flex widths
- child-count badge shown next to title on container entries
- styles.css: scoped CSS with #entries-body > .entry-row-outer selectors
(higher specificity than > div) to override flex on outer wrapper; inner row
and column rules replicated at correct depth; nth-child, is-selected,
is-multi-selected, url-cell hover all handled
CaptureBody gains via_freedium: Option<bool>; absent defaults to true.
The rearchive handler sets it to false — existing entries should not be
silently re-fetched through a mirror.
Port the modalcloser abx-plugin as a SingleFile --browser-script rather
than a spawned Puppeteer daemon. No external extension download required.
Core behavior (singlefile.rs):
- MODAL_CLOSER_DIALOG_OVERRIDES: main-world <script> bridge (best-effort;
blocked by strict script-src CSP) that overrides window.alert/confirm/
prompt/print, nulls window.onbeforeunload, traps its setter, and wraps
window.addEventListener to no-op 'beforeunload' registrations.
- MODAL_CLOSER_POLLING_SETUP: defines _archivr_mc_run() which runs two
passes on every tick (immediate first run, then setInterval every 500 ms
matching MODALCLOSER_POLL_INTERVAL default):
Pass 1 (best-effort, CSP-sensitive): main-world <script> bridge calls
Bootstrap/jQuery/jQuery UI/SweetAlert teardown APIs.
Pass 2 (always, CSP-immune): isolated-world DOM mutations — Escape-key
dispatch (Radix/Headless UI/Angular Material), backdrop clicks, full
CSS selector hiding for 40+ named consent/overlay vendors, body
scroll-lock reset. Direct DOM mutations need no inline script execution.
- resolve_modal_closer_config(): reads ARCHIVR_MODAL_CLOSER env var
(default true); no external resource required.
- modal_closer_enabled: Option<bool> in CaptureConfig so Default::default()
yields None (follow env var) rather than false.
Schema (database.rs):
- modal_closer_enabled column in auth DB instance_settings (default 1).
- Idempotent ALTER TABLE migration for existing databases.
- get/update_instance_settings updated (SELECT col 6, UPDATE param ?7).
HTTP (routes.rs):
- modal_closer_enabled in CaptureBody and UpdateInstanceSettingsBody.
- Per-capture body overrides global setting which overrides env var.
Frontend:
- SettingsView ExtensionsTab: third ext-card for Modal & Dialog Closer.
- CaptureDialog: per-capture toggle in Advanced Options, state initialized
from server default, included in submitCapture payload.
- api.js: submitCapture forwards modal_closer_enabled to capture body.
Behavior vs original modalcloser daemon:
- Functionally identical on non-strict-CSP pages (the large majority).
- Gap: <script> bridge is subject to page CSP; page.evaluate() is not.
On strict-CSP pages framework teardown and dialog overrides are no-ops;
CSS selector hiding and Escape dispatch (Pass 2) always work.
- Gap: alert/confirm/prompt overridden to instant no-op vs CDP timed
dialog.accept() with 1250 ms delay; irrelevant for archival in practice.
* feat: entry previews (video, tweet, article, iframe, image, audio bar)
- Add PreviewPanel dispatch hub routing by entity_kind + artifact extension
- VideoPreview: HTML5 <video> for YouTube/Instagram/TikTok/Reddit/X posts
- TweetPreview: tweet card, thread, and X article renderer (ported from x-article-renderer)
- IframePreview: sandboxed iframe for SingleFile web pages and PDFs
- ImagePreview: image viewer with click-to-open-fullsize
- AudioBar: persistent fixed-bottom player (Spotify-style) that survives entry navigation
- Lift entryDetail to App.jsx, shared between PreviewPanel and ContextRail
- 3-column layout (workspace | 300px preview | 340px rail) when preview active
- Stale-guard fixes: seq incremented before early returns in all async effects
- handleRearchive: capture startSeq/entryUid at call time, guard every async resume
- TweetPreview: reset loading/error/tweets before early-return branches
* feat: entry previews — tweet/thread/article/video/audio/image/iframe/pdf
- PreviewModal: modal overlay with new-tab link (↗) and keyboard close
- PreviewPanel: routes by entity_kind + primary_media extension to the
correct viewer (tweet/video/audio/pdf/html/image/fallback)
- TweetPreview: full X-style tweet, thread, and article renderer with
local artifact map for archived media (CDN fallback)
- AudioBar: persistent fixed bottom player, triggered via ContextRail
Play button; body.has-audio-bar pads content above it
- VideoPreview, IframePreview, ImagePreview: inline viewers
- PreviewPage: standalone /preview/:archiveId/:entryUid route
- ContextRail: Play/Preview buttons; isAudio/isPreviewable detection
- App.jsx: preview modal state, currentAudio state, preview route guard,
has-audio-bar body class effect
- routes.rs: CSP updated (media-src self blob https; frame-ancestors self;
Google Fonts + external images/scripts whitelisted)
- styles.css: preview modal, tweet-wrap scroll (min-height:0), audio bar
body padding, newtab button, preview panel flex layout
* style: tighten article/tweet preview spacing
- aMeta padding: 14px → 10px
- article title marginBottom: 10px → 8px
- aAuthorRow marginBottom: 10px → 8px
- bH1 top margin: 20px → 16px
- bH2 top margin: 18px → 14px
- bHr margin: 20px → 14px
- .preview-tweet-wrap padding: 20px → 12px (already committed)
More content visible above the fold in both modal and standalone views.
* feat: tweet/article preview quality pass
HTML entities: decode > < & etc. on sliced segments only
(entity offsets index the stored string; decoding before slicing shifts them)
Image lightbox: click any tweet/thread/article image to open full-screen
viewer; cmd+click follows <a> to open in new tab; arrow-key + ‹ › nav;
Escape closes; 1/N counter; ↗ open-in-new-tab link
Multi-image grid: 2 photos → side-by-side (180px rows); 3 → left spans
both rows; 4 → 2×2 (140px rows); single image unchanged
Empty media grid ghost: build photos/videoItems arrays first, render
.mediaGrid div only when at least one item resolved (advisory: never gate
on raw media.length when map items can all return null)
QT indicator: ↻ QT badge on tweet.is_quote_status === true entries
Modal shrinks for short content: height: 88vh → max-height: 88vh;
.preview-modal-body gets max-height: calc(88vh - 52px) so long threads
still scroll (advisory: don't rely on flex:1 once parent has no fixed height)
Video scrollbar leak: .preview-modal-body overflow: auto → hidden; each
child (tweet-wrap, video-wrap, iframe) manages its own scroll surface
Styled thin scrollbar on .preview-tweet-wrap (matches workspace rail)
ArticleRenderer: cover image and body images are lightbox-clickable;
opts thread through renderBlocksJSX → renderBlockJSX → renderAtomicJSX
* fix: move artifact fetch to api.js; stop Escape propagation from lightbox
- Export fetchEntryArtifacts(archiveId, entryUid, indices) from api.js
using Promise.all + getJson (follows project convention: all /api calls
go through api.js, never inline fetch in components)
- TweetPreview: import fetchEntryArtifacts, replace inline Promise.all
- MediaLightbox keydown handler: stopPropagation + preventDefault for
Escape/ArrowLeft/ArrowRight so the parent PreviewModal window listener
does not also fire and close the modal behind the lightbox
* fix: iframe/page preview height chain and toolbar UX
Problem: changing .preview-modal from height to max-height broke iframe
previews - <iframe style='flex:1'> needs a concrete ancestor height, which
max-height alone doesn't supply when content is shorter than the cap.
Fix - CSS:
.preview-modal--full { height: 88vh } applied to non-tweet modals
.preview-modal--full .preview-modal-body { max-height: none }
.preview-iframe-toolbar span: remove text-transform/letter-spacing
(was uppercasing the URL/title in shouty caps)
Fix - PreviewModal: className adds --full when entity_kind is not
tweet/tweet_thread; tweet previews keep shrink-to-fit behavior.
Fix - PreviewPanel: pass title + original_url from summary to IframePreview
for both HTML and PDF; wrappers use flex:1/minHeight:0 not height:100%.
Fix - IframePreview:
- Accept title + originalUrl props; show originalUrl in toolbar (falls
back to artifact src only when original_url absent); show title above
URL when available
- flex:1 + minHeight:0 instead of height:100% on the wrap div
- Single unified layout for page + pdf (both just show the iframe)
* feat: expand t.co links; linkify bare URLs in tweet and article text
Frontend:
- resolveEntityBounds: try multiple candidate strings in order (u.url
first, since that's the t.co short URL that appears in full_text)
- normalizeUrlAnn: multi-candidate search; href = expanded > url,
display = display_url > expanded > url
- linkifyText(): regex linkifier for entity-less bare URLs; trims
trailing punctuation [.,;:!?)] before linking; used in both
renderTweetTextJSX and renderInlineJSX including their early-return
paths (anns.length === 0) that previously bypassed linkification
- renderInlineJSX: fix mention href mention.name → screen_name;
replace t.co segment text with url.display when entity covers it
Scraper (vendor/twitter/scrape_user_tweet_contents.py):
- extract_tweet_data: when note_tweet text is used, pull urls/mentions/
hashtags/symbols from note_result.entity_set (correct indices for the
note text); keep media from legacy.entities (no note media downloads)
* feat: server-side t.co resolver + frontend augmentation
Server (routes.rs):
POST /api/util/resolve-tco — unauthenticated, accepts JSON array of
https://t.co/<alphanumeric> URLs only (strict regex validation, no SSRF
via input), capped at 50 per batch, 3 s timeout, redirect(Policy::none)
so the server only ever touches t.co itself. HEAD first, GET fallback if
HEAD returns no Location. Location sanitized to http/https only —
javascript:/data:/etc. fall back to the original t.co.
api.js:
resolveTcoUrls(urls) — project-convention wrapper for the new endpoint.
Returns {} on failure (callers degrade gracefully to bare t.co links).
TweetPreview.jsx:
After tweet data loads, per-tweet range-based coverage detection:
builds covered [start,end) from existing entity fromIndex/toIndex or
indices fallback, then finds regex matches whose span is NOT covered.
Resolves unique uncovered t.co URLs via resolveTcoUrls(), synthesises
one entity per occurrence (with exact fromIndex/toIndex so normalizeUrlAnn
gets correct bounds even for duplicate t.co URLs in the same tweet).
Augments entities.urls before setTweets() so all rendering paths
see expanded URLs.
* feat: suppress rendered media attachment URLs from tweet text
renderTweetTextJSX now accepts skipSpans=[] as third param.
Skip-span boundaries are added to the pts split set so a trailing
media t.co inside a plain segment still gets isolated and suppressed—
not re-linked by linkifyText. Early return only when both anns and
skipSpans are empty.
TweetCard computes mediaSkipSpans after building photos/videoItems:
for each rawMedia item whose src resolved (photo src match; any video),
resolveEntityBounds(m, ft, m.url) gives the precise [s,e] span using
indices/fromIndex first, indexOf fallback—then the span is passed to
renderTweetTextJSX so the t.co attachment URL is silently dropped.
Orphan cleanup bug: archiving x🧵A downloaded D/C/B/A JSONs and
media, but only registered artifacts for A. D/C/B files had no
entry_artifacts rows and were deleted as orphans.
Fix (staged scraper output, precise touched set):
- tweets::archive() stages all scraper output in temp/{ts}/tweet_stage/,
validates, then renames JSONs to raw_tweets/. Return type changed from
Result<bool> to Result<Vec<String>> (store-relative relpaths of every
produced tweet JSON, i.e. the exact touched set).
- tweets::rearchive() (new): same staged approach but always runs the
scraper. On scraper failure (tweet deleted/private), errors before
touching raw_tweets/ so existing data is preserved.
- register_tweet_artifacts() (new private helper in capture.rs): registers
every JSON in the touched set as a raw_tweet_json artifact, parses each
for media blobs, registers those too. JSON read failure is a hard error
with context, not a silent skip.
- record_tweet_entry() now accepts tweet_json_relpaths: &[String] and
delegates artifact registration to register_tweet_artifacts().
- perform_capture() passes the returned vec from tweets::archive().
Re-archive feature:
- capture::perform_rearchive(): looks up entry by uid, validates
tweet/tweet_thread, runs tweets::rearchive(), atomically swaps
entry_artifacts in a DB transaction. archived_at, title, tags,
collections are untouched.
- database: add get_entry_for_rearchive() and delete_entry_artifacts().
- POST /api/archives/:id/entries/:uid/rearchive: requires ROLE_USER,
creates capture job, returns 202 + job_uid, runs perform_rearchive in
spawn_blocking.
- Frontend: re-archive button in ContextRail for tweet/tweet_thread
entries; polls job at 500ms; refreshes entry detail on success; shows
error text on failure. Poll interval cleared before early-return on
entry deselect to prevent stale updates.
* feat: uBlock Origin Lite integration for ad-blocking during WebPage captures
- singlefile.rs: when ARCHIVR_UBLOCK=true and ARCHIVR_UBLOCK_EXT is set,
archivr owns Chrome's lifecycle (--headless=new, --remote-debugging-port,
--load-extension); single-file connects via --browser-server instead of
launching its own Chrome. Falls back to old behaviour with ublock_skipped=true
when the ext path is missing or invalid.
- capture.rs: thread ublock_skipped through CaptureResult
- database.rs: add notes_json TEXT column to capture_jobs (DDL + idempotent
ALTER TABLE migration); update_capture_job_status gains notes_json param
- archive.rs: expose notes_json in CaptureJobSummary
- routes.rs: store {"ublock_skipped":true} in notes_json on completed captures
- ToastStack.jsx: warning toast variant (toast--warning) with Details expander
and Ignore button
- CaptureDialog.jsx: fire warning toast when poll result has ublock_skipped
- App.jsx: sessionStorage-backed Ignore suppression for ublock warnings
- styles.css: .toast--warning (amber left border) + .toast-warning-detail
- flake.nix: ublockLite derivation fetches uBOLite_2026.705.2152.chromium.zip
(pinned SHA256) from uBlockOrigin/uBOL-home; sets ARCHIVR_UBLOCK_EXT in both
archivr and archivr-server wrappers
Env vars:
ARCHIVR_UBLOCK=true (default) — enable uBlock during WebPage captures
ARCHIVR_UBLOCK_EXT — path to unpacked uBOL extension dir (set by Nix)
* feat: Extensions settings tab + capture dialog redesign with Advanced options
Settings/Extensions tab (admin-only):
- New 'Extensions' tab between Cookies and Storage
- ExtensionsTab component: shows uBlock Origin Lite card with pill toggle
- Reads ublock_enabled from instance settings; patch via existing PATCH endpoint
- Shows ublock_ext_available status from server (whether ARCHIVR_UBLOCK_EXT is set)
Instance settings:
- Add ublock_enabled BOOLEAN (default true) to instance_settings auth DB table
- Idempotent ALTER TABLE migration in initialize_auth_schema()
- get/update_instance_settings include ublock_enabled
- GET /api/admin/instance-settings now also returns ublock_ext_available (computed
from ARCHIVR_UBLOCK_EXT env var at request time)
- PATCH /api/admin/instance-settings accepts ublock_enabled
Per-capture override:
- CaptureBody gains ublock_enabled: Option<bool>
- CaptureConfig gains ublock_enabled: Option<bool>
- singlefile::save() gains ublock_enabled_override: Option<bool> param
- Capture handler resolves: body override > global instance setting > env var
- submitCapture(aid, loc, qual, extensions) in api.js passes ublock_enabled
Capture dialog redesign:
- Archive button: full-width, 13px padding, min-width 220px, primary CTA
- Cancel: full-width but text-style, below Archive
- ‹Advanced options› chevron toggle (rotates on open)
- Expanded panel shows uBlock toggle for this capture session
- Loads global ublock_enabled default from instance settings on mount
Styles:
- .ext-toggle pill switch (44×24 and 36×20 small variant)
- .ext-card for Settings Extensions tab
- .capture-advanced + .capture-advanced-panel + .capture-chevron
- .capture-ext-row / .capture-ext-label / .capture-ext-name / .capture-ext-desc
- .form-hint utility class
* fix: remove ublock_enabled from INSERT OR IGNORE in DDL batch
The INSERT ran before the ALTER TABLE migration added the column,
causing 'table instance_settings has no column named ublock_enabled'
on existing databases. The INSERT OR IGNORE for the default row only
needs the original columns; the migration's DEFAULT 1 handles the
new column for existing and new rows alike.
* feat: Reader mode via Mozilla Readability.js
Adds an opt-in 'Reader mode' advanced option to the capture dialog.
When enabled, Readability.js is injected as a browser script during
SingleFile capture; it fires on single-file-on-before-capture-start,
replaces the page body with the distilled article content, injects a
clean typographic stylesheet, and adds a header with title/byline/site.
Falls back silently if Readability fails (e.g. non-article pages).
- vendor/readability/Readability.js Apache 2.0, Mozilla, v0.6.0
- singlefile.rs: embed READABILITY_JS + READER_MODE_WRAPPER_JS via
include_str!; write both to temp dir when reader_mode is true;
base_single_file_cmd now accepts &[&Path] for multiple --browser-script
- capture.rs: CaptureConfig.reader_mode: bool
- routes.rs: CaptureBody.reader_mode: Option<bool> (defaults false)
- api.js: submitCapture passes reader_mode in payload
- CaptureDialog.jsx: Reader mode toggle in Advanced options (off by default)
* fix: diagnose single-file no-output-file error + prevent stdout dumping
- Add --dump-content=false to every single-file invocation to prevent
the Docker-detection heuristic from routing HTML to stdout instead of
the output file (the heuristic can trigger in some macOS environments)
- Improve the no-output-file error message to include: temp dir contents,
stderr, and first 200 chars of stdout — this gives enough context to
diagnose any remaining cause without re-running
* fix: switch uBlock loading from --browser-server to --browser-args
The --browser-server (CDP) path caused 'Unexpected server response: 404'
on macOS Chrome because simple-cdp's WebSocket upgrade to the debugger
endpoint failed after Chrome started — likely a version-specific CDP
endpoint shape mismatch.
New approach: single-file always manages Chrome. When ARCHIVR_UBLOCK_EXT
is set, --headless=new, --load-extension, and --disable-extensions-except
are injected via --browser-args. single-file's browser.js prefix-strips
its own conflicting flags before appending ours, so --headless=new
overrides the default --headless (enabling extension support in headless).
Removes allocate_free_port, wait_for_chrome_ready, run_single_file_with_server
(all dead code now). Docblock updated to reflect actual behaviour and notes
the --single-process caveat: uBOL's declarativeNetRequest static rulesets
are expected to work (network-stack level, not service-worker), but this
has not been mechanically verified under --single-process.
Smoke tested on macOS (this machine): capture with --load-extension + all
three browser-scripts (strip, Readability, reader-mode wrapper) produces
output file correctly. Ad-blocking verification deferred to manual test
with a tracker-heavy URL.
* fix: use correct single-file hook event (single-file-on-before-capture-request)
Prior scripts listened on 'single-file-on-before-capture-start' which
does not exist in single-file-core 1.1.49. The real hook is:
single-file-on-before-capture-request (dispatched by initUserScriptHandler
after receiving single-file-user-script-init; userScriptEnabled defaults
to true in args.js so it always fires when --browser-script is passed)
Changes:
- strip-scripts: -start -> -request (no preventDefault needed; synchronous)
- READER_MODE_SCRIPT: -start -> -request; add 'installed' meta marker at
script-evaluation time so artifact inspection can distinguish 'script
not injected' / 'hook never fired' / 'Readability parse failed'
* fix: correct singlefile.rs docstring (scripts.js concatenates, not isolates)
* fix: dispatch single-file-user-script-init so request hook fires
single-file's initUserScriptHandler (in single-file-bootstrap.js) listens
for 'single-file-user-script-init' and only then installs
_singleFile_waitForUserScript. Without that dispatch our scripts'
'single-file-on-before-capture-request' listeners were never reached,
so neither strip-scripts nor reader-mode Readability applied.
Dispatch the init event at the top of strip-scripts (always present) and
redundantly in READER_MODE_SCRIPT. Verified end-to-end: artifact for
run_b3181d6d276e4e56a1a6c356ef9bbe8f has
meta content="applied", max-width:680px CSS, 0 script tags.
* feat: cookie consent extension support (ARCHIVR_COOKIE_EXT)
Mirrors the uBlock Origin Lite integration exactly:
Backend:
- singlefile.rs: resolve_cookie_ext_config() reads ARCHIVR_COOKIE_CONSENT
(default true) + ARCHIVR_COOKIE_EXT path; extension paths comma-joined
into --load-extension / --disable-extensions-except so uBlock and cookie
ext can coexist; SaveResult.cookie_ext_skipped tracks miss
- database.rs: cookie_ext_enabled column on instance_settings (DEFAULT 1);
idempotent ALTER TABLE migration; get/update wired through
- capture.rs: CaptureConfig.cookie_ext_enabled: Option<bool>; threaded to
singlefile::save(); cookie_ext_skipped surfaced in CaptureResult
- routes.rs: CaptureBody + UpdateInstanceSettingsBody get cookie_ext_enabled;
capture handler resolves effective value (body overrides global); notes_json
only includes skipped fields that are true; GET instance-settings includes
cookie_ext_available from env path check
Frontend:
- api.js: submitCapture forwards cookie_ext_enabled
- SettingsView.jsx: 'I Still Don't Care About Cookies' card in Extensions
tab; always-active toggle (user can disable even when ext not installed);
amber 'Not configured' hint + ARCHIVR_COOKIE_EXT guidance when unavailable
- CaptureDialog.jsx: 'Block cookie banners' toggle in Advanced options;
always shown with amber hint when ext not configured; defaults from
global setting
Operator setup: download + unzip the extension from GitHub releases, set
ARCHIVR_COOKIE_EXT=/path/to/unpacked/ext. No Node daemon needed.
* fix: surface cookie_ext_skipped warning toast in CaptureDialog
* feat: package istilldontcareaboutcookies in flake, wire ARCHIVR_COOKIE_EXT
Add isdcac derivation mirroring ublockLite:
- Fetches ISDCAC-chrome-source.zip v1.1.9 from GitHub releases
- Validates manifest.json at extension root before install (guard against
nested-folder zip regressions in future releases)
- Sets ARCHIVR_COOKIE_EXT in both archivr and archivr_server wrappers
Verified: nix build .#archivr-server and .#archivr both succeed;
wrapper scripts export correct store paths; manifest.json present at root.
* fix: gate consent-overlay cleanup on cookie_ext; reset overflow; narrow selectors
- Strip overflow:hidden from body/html only when cookie_ext is active for
the capture — prevents mutating legitimate pages when the feature is off
- Remove .fc-dialog (Google Funding Choices), .qc-cmp2-*, .sp-message-container,
#sp-cc, #usercentrics-root as fallback for CMPs the extension misses
- Removed overbroad [class^="uc-"] and [id^="usercentrics"] selectors
that could match real page content
* fix: remove ad placeholders when uBlock active; kept height causes blank gap
uBlock Origin Lite blocks ad network requests but first-party placeholder
elements (ins.adsbygoogle, #aswift_* iframe hosts) retain their computed
height (e.g. 280px for a top banner), leaving a large blank space at the
top of captured pages.
Gate cleanup on ublock_ext.is_some(): remove ins.adsbygoogle, aswift_*
iframes, and google_ads_* iframes before SingleFile serialises. Also
collapse the parent container if it becomes empty after removal.
* fix: walk up to .top-ad/.google-auto-placed ancestor before removing ad slot
Removing only the inner ins.adsbygoogle left the outer .container.top-ad
wrapper (with pb-4 padding) in the layout, preserving the blank gap.
Now walk up via closest() to the nearest ad-slot container class before
removal so the whole slot including padding collapses.
Adds per-instance cookie rules (admin-only) that are injected into
every network touchpoint during capture.
Storage:
- New cookie_rules table in the auth DB (idempotent migration)
- Rules have pattern_kind (global/wildcard/regex), optional url_pattern,
and cookies_json (validated as string-only JSON object)
Matching (resolve_cookies_for_url):
- Global rules always apply
- Wildcard: * and ? with full metacharacter escaping; matched against
hostname via reqwest::Url when pattern has no ://, full URL otherwise
- Regex: matched against the full URL
- Later rules in ordinal order override earlier ones per cookie name
All six network touchpoints receive resolved cookies:
- http::probe_url_kind and http::download: Cookie request header
- singlefile::save: Netscape cookie file -> --browser-cookies-file
- ytdlp::fetch_metadata and ytdlp::download: Netscape cookie file -> --cookies
- tweets::archive: semicolon credentials file -> --credentials-file
(only when both ct0 and auth_token are present; otherwise falls back
to ARCHIVR_TWITTER_CREDENTIALS_FILE)
Security:
- Cookie files written 0o600 (owner read/write only)
- Exact parsed hostname used as cookie domain (no PSL stripping)
- Files deleted unconditionally before any error propagates,
including spawn failures (hold-result-then-delete pattern)
- No cookie values in process args (no --add-header exposure)
API: GET/POST /api/admin/cookie-rules, PATCH/DELETE /api/admin/cookie-rules/:uid
Frontend: Cookies tab in Settings (admin only) with rule list,
inline edit, pattern-type selector, client-side JSON validation
CLI: CaptureConfig::default() - no behaviour change
254 tests passing (4 new cookie-rule handler tests)
- database.rs: add has_active_capture_jobs(), list_orphaned_blob_rows(),
all_referenced_file_relpaths(), delete_orphaned_blob_rows()
- routes.rs: GET/DELETE /api/archives/:id/blob-cleanup (ROLE_ADMIN)
- GET returns {orphaned_blob_rows, deletable_files, total_bytes}
- DELETE has two active-capture guards (before and after disk walk)
to prevent deleting files mid-capture; walks raw/ and raw_tweets/
- Referenced set = entry_artifacts.relpath ∪ live blobs' raw_relpath,
so a file is never deleted if any artifact still points at its path
- api.js: scanOrphanBlobs(), deleteOrphanBlobs()
- App.jsx: pass archiveId to SettingsView; add 'storage' to SETTINGS_TABS
so /settings/storage survives refresh/back navigation
- SettingsView.jsx: new Storage tab (admin-only) with idle→scanning→
scanned→deleting→done/error state machine; shows file/record counts
and human-readable byte sizes before a btn-danger confirm
Tests (14 new, 248 total passing):
- database.rs: has_active_capture_jobs for pending/running/completed,
list_orphaned_blob_rows, all_referenced_file_relpaths edge cases,
delete_orphaned_blob_rows preserves referenced rows
- routes.rs: auth (401), active-capture 409 on GET and DELETE,
end-to-end delete preserving referenced file and removing orphan
blob file + extra disk-only file
- ytdlp::download() accepts quality: Option<&str>; quality_format()
maps best/1080p/720p/480p/360p to yt-dlp -f format strings
- perform_capture() threads quality through to the downloader
- CaptureBody gains optional quality field; capture_handler validates
it against the allowlist (400 on unknown values) before spawning
- CLI passes None (preserves existing best-quality behaviour)
- Frontend: isVideoSource() mirrors determine_source() exactly —
shows quality picker only for yt-dlp-backed sources, excludes
playlist/channel shorthands and tweet/thread paths
- submitCapture(archiveId, locator, quality) sends quality in POST body
- CSS: .capture-quality styles the inline select to fit the capture row
- Tests: quality_format unit tests in ytdlp.rs; two new route tests
(valid quality accepted, invalid quality rejected with 400)
- Docs: video quality section added under Supported Platforms
NixOS module (4 fixes):
- Remove RestrictNamespaces=true: Chromium (launched by single-file for
web captures) needs Linux user namespaces; blocking them broke captures
- Escape TOML string values: labels/paths with quotes or backslashes
would produce invalid archivr-server.toml and prevent startup
- Bracket IPv6 listen addresses: ::1:8080 is rejected by Rust's
SocketAddr parser; [::1]:8080 is the correct form (RFC 2732)
- Add optional storePath per archive: covers archives initialised with
a custom store path outside the default sibling store/ directory
Rust (1 fix):
- extract_client_ip loopback branch: take last XFF value not first.
When a proxy appends to X-Forwarded-For an attacker controls the
first entry; the last entry is always set by the trusted local proxy
Frontend (1 fix):
- Gate fetchArchives on authState === 'authenticated'. The empty-dep
useEffect fired on mount (before login), hit 401, and never retried
after login because authState was not in the dependency array
Fix 1 — NixOS: store/ dir missing from ReadWritePaths
archive_path points to .archivr/; captures write artifacts to the
sibling store/ directory. Whitelisting only .archivr/ caused POST
/captures to fail under ProtectSystem=strict.
Fix: use builtins.dirOf a.path to whitelist the parent, which
contains both .archivr/ and store/.
Fix 2 — Rust: X-Forwarded-For was unconditionally trusted
An attacker could send a different IP on every login attempt,
bypassing LOGIN_MAX_ATTEMPTS entirely.
Fix: use ConnectInfo<SocketAddr> (via into_make_service_with_connect_info)
as the primary rate-limit key; XFF is trusted only when the TCP peer
is loopback (i.e. a local reverse proxy). Tests without a real socket
fall back to XFF unchanged.
Port configuration:
NixOS module: split 'bind' string into separate 'listenAddress'
(default 127.0.0.1) and 'port' (default 8080) options. openFirewall
now uses cfg.port directly instead of parsing it from the bind string.
README NixOS example updated accordingly.
Both handlers accepted AuthUser but served guests.
auth.require_auth()? added as first line closes the gap.
Existing unauthenticated tests updated with session cookies.
5 attempts / 15 min per IP. Excess attempts get 429 + Retry-After.
Client IP read from X-Forwarded-For with loopback fallback.
In-memory; prune inline on each check.
Refactored app() into app_with_state() + thin wrapper.
Uses parking_lot::Mutex (no unwrap, no poisoning).
Injects X-Content-Type-Options, X-Frame-Options, Referrer-Policy,
Content-Security-Policy, and Permissions-Policy on every response.
Registered as outermost layer so it covers setup_guard 503s too.
No new dependencies.
- capture.rs: add archive_id: Option<&str> to perform_capture; when Some,
call font_extractor::extract_and_rewrite before hashing HTML, register
each font as a deduplicated blob + 'font' artifact
- main.rs: pass None as archive_id (CLI keeps fonts embedded)
- routes.rs: add GET /api/archives/:id/blobs/:sha256 (serve_blob handler),
pass Some(&archive_id) to perform_capture in capture_handler,
add ApiError::internal constructor
- Add optional `bind` field to ServerRegistry (TOML + ARCHIVR_BIND env var)
- Default bind address remains 127.0.0.1:8080; non-loopback prints a warning
- Add route security classification comment block (READ/ADMIN/WRITE/STATIC)
- Add Security and Deployment section to docs/README.md
- Replace vague auth note in ARCHIVR-MENTAL-MODEL.md with concrete model description
- Add three registry tests covering bind field round-trip and defaults
- Add archivr-core/src/capture.rs: Source enum, all capture helpers,
fail_run (replaces process::exit fail_archive_and_exit), and
perform_capture() as the public entry point
- Slim archivr-cli/src/main.rs to 103 lines: thin adapter over
archivr_core::capture::perform_capture
- Move all source-classification and tweet-entry tests from CLI to
archivr-core/src/capture.rs (they test core logic, belong in core)
- Add POST /api/archives/:archive_id/captures route in archivr-server:
validates non-empty locator (400), resolves archive (404), delegates
to perform_capture, returns {run_uid, status}
- Add capture dialog to browser UI: dialog markup in index.html,
showModal/submit/cancel/loading/error wiring in app.js, dialog
styles in styles.css
- 77 tests pass, clean build (0 warnings)
- 5 tag routes (list tree, create, assign, remove, search with tag=)
- Tags nav, tag tree view, tag filter badge
- Entry tag pills with remove, assign-tag form in context rail
