Child rows were inheriting the parent entry's stripe color, making the
expanded list look like one flat block. Apply the same odd/even palette
as root entries (var(--paper-3) / #f2ede5) so each video row is visually
distinct within the expanded group.
Moving background:#eee2d2 off .entry-row-outer onto > .entry-row-main
so that expanded child entries don't inherit the selection highlight.
Outer wrapper now explicitly unsets background (cancelling the flat-div
cascade rule) and clears outline+box-shadow. Both the selection colour
and the inset stroke live on .entry-row-main only.
The flat selector '#entries-body > div.is-selected' already applies
box-shadow to .entry-row-outer (it IS a direct child div). The outer
override rule only cleared 'outline', so the box-shadow leaked through,
wrapping the entire parent+children block with a second stroke.
Add box-shadow: none to both .is-selected and .is-multi-selected on
.entry-row-outer so the stroke sits only on .entry-row-main.
Remove the probing-state placeholder span entirely. The chevron renders
only when playlistProbeState === 'done'; all other states (idle, probing,
error) render null. The small layout shift when the chevron appears after
probe is acceptable; blank padding while there is no chevron is not.
CaptureDialog.jsx:
- Placeholder no longer appears on idle playlist rows (only during
probing); chevron shows only after probe completes — no left-padding
while the user is still typing
- Per-video delete button added to expanded playlist list; removes item
from playlistItems so its ID is absent from per_item_quality on submit
- anyEmptyPlaylist guard: Archive button disabled + handleArchive early-
return when all videos have been deleted (empty map would otherwise
silently download everything)
styles.css:
- Selection stroke switched from outline to box-shadow:inset everywhere
(entry-row-outer, child-entry-row, legacy flat-div selector) —
guaranteed inside element bounds, no layout interference
- URL cell overflow only on :hover; removed is-selected .url-cell rule
that was expanding child URLs when their parent was selected
- Playlist items redesign: separator lines instead of background fills,
amber left-border for conflicts, thin scrollbar, tighter padding
- Remove button: opacity 0.3 always-visible baseline; full opacity on
hover/:focus-visible; forced to 1 on coarse-pointer (touch) devices
archive.rs: total_artifact_bytes for root entries now adds a correlated
subquery summing children's blob bytes so playlist/channel containers
show the real download size instead of 0.
capture.rs: non-empty per_item_quality map now acts as an include-set —
items whose yt-dlp ID is absent are skipped entirely. This wires the
UI's per-video delete button to actual capture exclusion. Empty map
preserves the existing behaviour (download everything).
routes.rs + capture.rs doc: comments updated to reflect both semantics
(empty = all / non-empty = only listed IDs) accurately.
Old captureItems entries saved before the playlist fields were added
have playlistItems=undefined (missing key). The hasConflict guard
checked !== null, which undefined passes, then called .some() on
undefined → TypeError.
Two-part fix:
1. sessionStorage restore: merge each saved item over makeItem() defaults
so any missing fields (playlistItems, playlistProbeState, etc.) are
filled with their correct initial values before the item is used
2. hasConflict: use Array.isArray() instead of !== null so undefined
is also safely rejected — defence-in-depth for any future field gap
The server validates per_item_quality value shapes but does not enforce
that every playlist item has an entry. Items without an override get
the global quality as a yt-dlp cap with graceful fallback.
The 'must choose quality for unsupported videos' invariant is a UI
constraint enforced by the frontend before submission. A direct API
caller bypassing the UI accepts yt-dlp's standard cap-and-fallback
behavior. Document this scope explicitly so the gap is intentional,
not accidental.
User asked for the chevron to be on the left of the playlist input,
not tucked after the quality selector on the right.
- Remove chevron from qualityEl (it was between the quality select and
the remove button)
- Add it as the first child of capture-row-main, before the <input>,
when isPlaylistSource && playlistProbeState === 'done'
- Show a same-width placeholder span while probing/idle/error so the
input does not jump left when the chevron appears after probe completes
- Add capture-playlist-toggle--left modifier (flex-shrink:0, tighter
padding) and .capture-playlist-toggle-placeholder (fixed 22px width)
Replace maxHeight >= newHeight (cap check) with item.qualities.includes(newQ)
(exact match). A video with [2160p, 1080p] does not support 1440p; the
previous logic would mark it as supporting any quality up to 2160p and
submit '1440p' which yt-dlp silently downloads as 1080p — misrepresenting
the selected quality.
With exact match, unsupported qualities correctly fall through to the
conflict path (keep prior selection or null), enforcing the same manual-
choice requirement as any other unsupported quality.
Per-row selects are unaffected: they already render only pi.qualities
(the video's actual available formats), no maxHeight logic involved.
Previous text said 'using best quality' implying the capture would
proceed, but probe error now blocks submission. Replace with 'Probe
failed — edit URL to retry' in orange (capture-quality-hint--error)
so the disabled button and the message are consistent.
Previous guard only blocked while playlistProbeState==='probing'.
Two remaining bypass paths:
- idle: 800ms debounce not yet fired after URL typed
- error: probe failed — no per-video quality data available
Change anyProbing and handleArchive guard to:
isPlaylistSource(locator) && playlistProbeState !== 'done'
This means idle/probing/error all block submission for playlist items.
error is intentionally blocking — without quality data the per-video
requirement can't be satisfied; user must retry or remove the URL.
Backend determine_source routes yt:user/... (and youtube:user/...) to
YouTubeChannel — already covered by the yt: shorthand block for
playlist/, @, channel/, c/ but missing user/. Old-style user channel
URLs would capture correctly server-side but never show the playlist
quality/sync UI.
applyPlaylistQuality('audio'):
- Only sets quality='audio' on items where has_audio=true
- Items with has_audio=false: keep prior selection if set, else null
(conflict) — same rule as unsupported height, blocks archive until
user explicitly picks a quality for those items
Playlist-level 'Audio only' option:
- Changed hasAnyAudio → allHaveAudio (every item must have audio)
- When any item lacks audio, the option is hidden entirely so the
selector can never create immediate conflicts just by appearing
Backend determine_source playlist/channel regex only matches
(?:www\.)?youtube\.com — mobile URLs hitting m.youtube.com would be
probed as playlist in the UI but captured as Source::Url server-side.
Remove m.youtube.com from the detector to keep frontend and backend
in exact agreement. Add backend support when needed.
The Archive button is disabled when anyConflict || anyProbing, but
onKeyDown on the locator input calls onSubmit() → handleArchive()
directly, bypassing the button's disabled check entirely.
Add the same conditions as early returns inside handleArchive itself,
operating on toSubmit (the items that would actually be submitted) so
the guard is tight — items with no locator are already excluded by the
toSubmit filter.
Previous fix excluded /watch but still returned true for any youtube.com
URL with a ?list= param (e.g. /shorts/xxx?list=yyy). Backend determine_source
only routes to YouTubePlaylist on /playlist?list=... and to YouTubeChannel on
/@handle, /channel/, /c/, /user/ paths — everything else is a single item.
Rewrite the HTTP block to match:
- youtube.com: pathname==='/playlist' && list param → playlist
: /@, /channel/, /c/, /user/ → channel
: anything else (incl. /watch&list=, /shorts?list=) → false
- music.youtube.com: pathname==='/playlist' && list param → YTM playlist
: /watch → single track (falls through to false)
youtube.com/watch?v=...&list=... and music.youtube.com/watch are single
videos in the backend (Source::YouTubeVideo / YouTubeMusicTrack) regardless
of the list param. Previously isPlaylistSource returned true for these,
which would have triggered the playlist probe path while the video probe
was already running, and the render would attempt to show playlist UI on
an item whose playlistProbeState stays idle.
Guard: if pathname === '/watch', return false before the list-param check.
YouTube Music playlists force child_quality = Some("audio") because
yt-dlp can't download DRM-free audio-only tracks any other way. The
previous per_item_quality lookup could override this with e.g. "best",
defeating the invariant. Guard the lookup behind !is_audio so YTM items
are always downloaded as audio regardless of what the caller sends.
routes.rs:
- CaptureBody gains per_item_quality (HashMap<String,String>, serde(default))
and sync (bool, serde(default)); both validated before use
- per_item_quality values validated against same quality predicate as top-level
quality field ("best"|"audio"|"NNNp") so bad per-video values are rejected
at the API boundary rather than silently falling through to quality_format
- capture_handler threads body.per_item_quality + body.sync into CaptureConfig
(replaces hardcoded empty stubs); rearchive_handler keeps empty defaults
- New POST /api/archives/:id/captures/probe-playlist: calls
probe_playlist_qualities via spawn_blocking; 400 for non-playlist locator,
502 on yt-dlp failure, returns PlaylistProbeResult as JSON
api.js:
- probePlaylist(archiveId, locator): POST probe-playlist endpoint
- submitCapture: forwards per_item_quality (non-empty) and sync:true from
extraExtensions param added to submitBgJob
CaptureDialog.jsx:
- isPlaylistSource(): detects yt:/youtube: playlist/@/channel, ytm:playlist/,
YouTube/YTM HTTP(S) URLs with list= param or channel pathnames
- makeItem(): 6 new playlist state fields
- applyPlaylistQuality(): conflict logic — videos that can reach selected
quality get it set; videos that can't and have no prior selection are left
null (conflict); videos with a prior selection keep it when quality is raised
- hasConflict(): any playlistItems entry with quality===null
- updateLocator(): isPlaylistSource branch with 800ms debounce→probePlaylist;
existing isVideoSource path unchanged
- Archive button disabled when anyConflict or any probe in flight
- Per-video expand list with individual quality selects, conflict badges,
sync toggle (appears after probe completes)
styles.css: playlist expansion, conflict, sync toggle CSS
archive.rs:
- get_archived_playlist_child_urls: collect() as rusqlite::Result<HashSet<_>>
instead of filter_map(ok) so row-level errors surface rather than silently
skipping and causing duplicate downloads
capture.rs:
- match get_archived_playlist_child_urls result and fail_run on error instead
of unwrap_or_default, preventing silent re-downloads on DB failure
Tests added to capture.rs:
- locator_to_playlist_url_accepts_playlist_shorthands (yt:playlist/, ytm:playlist/, full URL)
- locator_to_playlist_url_accepts_channel_shorthands (yt:@handle)
- locator_to_playlist_url_rejects_non_playlist_sources (single video, tweet, web page)
Tests added to archive.rs (all use in-memory DB via make_tag_test_db):
- find_container_entry_id_returns_none_when_absent
- find_container_entry_id_returns_root_entry
- find_container_entry_id_ignores_child_entries (child with parent_entry_id set)
- get_archived_playlist_child_urls_empty_when_no_playlist
- get_archived_playlist_child_urls_returns_children
- get_archived_playlist_child_urls_excludes_other_playlists
ytdlp.rs:
- Add PlaylistItemProbe / PlaylistProbeResult (pub, serde::Serialize)
- Add private available_video_heights_from_value() helper for Value entries
- Add probe_playlist_qualities(): yt-dlp -J (full metadata, no flat flag)
returns per-video quality lists in one subprocess call
capture.rs:
- Add per_item_quality: HashMap<String,String> and sync: bool to CaptureConfig
(both Default; keyed by yt-dlp video ID, not URL)
- Add pub locator_to_playlist_url(): validates only the three playlist sources,
expands shorthands; keeps locator_to_ytdlp_url's no-playlists contract
- Playlist capture block: sync-aware container resolution
- sync + existing container → reuse it via complete_archive_run_item,
skip already-archived children (by canonical URL) before creating
run items so refresh_run_counters only counts new items
- sync + no container → create normally (first sync run)
- non-sync → always create fresh container (existing behaviour)
- Per-item quality: config.per_item_quality.get(id) falls back to child_quality
archive.rs:
- Add get_archived_playlist_child_urls(): returns HashSet of canonical URLs
of all children under any container matching the playlist canonical URL
- Add find_container_entry_id_by_canonical_url(): returns most-recent
container entry id (parent_entry_id IS NULL) for a given canonical URL
routes.rs: stub per_item_quality/sync on both CaptureConfig sites (server
agent will wire body fields in Phase 2)
ChildRow now receives onRowClick and selectedUids from EntryRow (which
receives selectedUids from EntriesView alongside the existing booleans).
Clicking a child row invokes onRowClick(child, e) so it flows through
handleRowClick → selectEntry → fetchEntryDetail exactly as a root entry
would. Shift-range selection gracefully degrades to single-select since
child entries are not in the root entries array.
Selected/multi-selected visual state is wired: .child-entry-row.is-selected
shows the same #eee2d2 background + accent outline as root rows; hover
restores full opacity. Frontend static assets rebuilt.
server:
- add GET /api/archives/:id/entries/:uid/children → list_entry_children,
calling list_child_entries with caller_bits so visibility model is enforced
- fix capture_handler: use result.status ("completed"/"failed") to set job
status rather than always "completed", mirroring rearchive_handler; this
surfaces partial playlist failures to the polling client
frontend:
- api.js: add fetchEntryChildren(archiveId, entryUid)
- EntryRow: one outer div.entry-row-outer (display:block) keeps nth-child
striping correct; inner div.entry-row-main is the flex row with all column
cells and event handling; .child-entries sits below inside the outer wrapper
- expand chevron appears when entry.child_count > 0; clicking fetches children
lazily and renders ChildRow components reusing .col-* flex widths
- child-count badge shown next to title on container entries
- styles.css: scoped CSS with #entries-body > .entry-row-outer selectors
(higher specificity than > div) to override flex on outer wrapper; inner row
and column rules replicated at correct depth; nth-child, is-selected,
is-multi-selected, url-cell hover all handled
- ytdlp: add fetch_playlist_info() using yt-dlp -J --flat-playlist for
reliable container title + shallow entry list; normalize item URLs via
webpage_url → absolute url → id fallback (domain inferred from container
URL so YTM stays on music.youtube.com)
- capture: add record_container_entry() (no blob, no primary_media artifact);
extend record_media_entry() with parent_entry_id/root_entry_id params (all
existing single-item call sites pass None, None)
- capture: implement YouTubePlaylist / YouTubeChannel / YouTubeMusicPlaylist
capture path replacing the two not-implemented stubs: fetch playlist info →
create container entry (reusing existing run + item) → per-child run items
(parent_item_id = container item) → download each video/track as a child
entry; per-child failures are non-fatal; perform_capture returns result.status
reflecting actual run outcome so capture_handler marks the job correctly
- archive: add child_count i64 to EntrySummary (col 12 in all listing queries);
add get_entry_summary() private helper; fix get_entry_detail() to use
get_entry_summary() so child entries are resolvable via the detail endpoint;
add list_child_entries(conn, uid, caller_bits) with the same
admin/collection visibility predicate as list_root_entries
archive_runs.requested_count stays 1 (one user locator); discovered/
completed/failed_count reflect container item + N video items via
refresh_run_counters.
* feat(frontend): add SkeletonEntryRow with shimmer animation
Adds a new SkeletonEntryRow component that renders animated shimmer
placeholder cells matching the exact column layout of EntryRow
(col-added, col-title with icon circle, col-type pill, col-size,
col-url). CSS appended to styles.css using existing design tokens
(--paper-2, --line-soft, --paper-3) for the warm-toned shimmer.
* feat(frontend): async capture UX — reset dialog on submit, show skeleton rows
When the user presses Archive, CaptureDialog now immediately resets its
form to a fresh empty state and closes. Background captures continue
polling via intervals that survive the dialog close.
Skeleton rows appear at the top of EntriesView (filtered to the active
archive) for each in-flight job, giving visual feedback that something
is being processed. On completion the skeleton is removed and the entry
list refreshes; on failure the skeleton is removed and an error toast
fires — matching the existing toast behaviour.
Architecture:
- App owns pending-capture state (pendingCaptures) as the single source
of truth, persisted to sessionStorage['pendingCaptures']. On page
refresh, App seeds the list and passes it to CaptureDialog as
activeJobs so polling reconnects without a second sessionStorage read.
- CaptureDialog emits onJobStarted({id,jobUid,locator,archiveId}) only
after submitCapture() returns a job_uid — never before, never on
failure — ensuring no orphan skeletons can persist after a refresh.
- onJobSettled(id) is called by startPolling on any terminal state
(completed, failed, or network error), removing the skeleton.
- EntriesView filters pendingCaptures by archiveId so a capture in
archive A never shows a skeleton in archive B.
Removed from CaptureDialog: submitItem, resetRow, hasActiveJobs,
anyActive, CapStatusDot. CaptureRow is simplified to idle-only display
with no disabled state, no retry button, no status dot.
* fix(frontend): skeleton replacement ordering and col-check alignment
- Await the entry list refresh before removing the skeleton so the
real row arrives before the placeholder disappears. handleCaptured
now returns its Promise.all; startPolling awaits it before calling
onJobSettled on the success path.
- Add col-check as the first child of SkeletonEntryRow to match
EntryRow's DOM structure; without it, columns misalign on touch
devices where col-check becomes display:flex.
* fix(frontend): use Promise.allSettled in handleCaptured to prevent runs-fetch failure from poisoning successful captures
Promise.all rejects on the first failure. If the /runs refresh threw,
the rejection would propagate through startPolling's await and land in
the outer catch block, firing an error toast for a capture that had
already succeeded. Promise.allSettled settles unconditionally so a
transient runs fetch error is silently absorbed while the entry list
still refreshes.
Replace the broad previousElementSibling/HEADER removal with a
precise selector that matches only the .flex.justify-end wrapper
containing the 'Download article' button. The old heuristic was
removing article headers on NYT/WaPo captures.
* feat(core): add via_freedium to CaptureConfig; route WebPage captures through Freedium mirror
When via_freedium is true and the locator is not already a Freedium URL,
perform_capture passes https://freedium-mirror.cfd/<original-url> to
singlefile::save() so paywalled articles are fetched via the mirror.
The original locator is kept for requested_locator and canonical_locator
in the DB entry. An empty cookie map is used for the mirror fetch to
prevent original-domain credentials from being sent to freedium-mirror.cfd.
* feat(server): expose via_freedium in capture API (default on)
CaptureBody gains via_freedium: Option<bool>; absent defaults to true.
The rearchive handler sets it to false — existing entries should not be
silently re-fetched through a mirror.
* feat(frontend): add Freedium mirror toggle to capture advanced options
- freediumEnabled state defaults to true (on by default)
- via_freedium forwarded through submitCapture to the capture API
- Toggle rendered last in the advanced panel, matching existing rows
- Built frontend static assets included
* fix(core): Freedium capture fixes — title, notifications, reader images
- extract_html_title: take().read_to_end() up to 256 KiB (single read() can
short-read, leaving title at byte ~106 K undiscovered); regression test added
- Strip " - Freedium" suffix before storing entry title
- is_freedium_fetch keys off actual fetch_url host
- Remove Freedium toast overlay ([data-sonner-toaster]) before capture
- Resolve lazy images (data-zoom-src etc.) before Readability in reader mode
* fix(core): extract HTML title after font stripping, not before
SingleFile embeds fonts as base64 data URIs in <style> blocks in the
<head>, pushing the <title> tag to ~1.2 MB in the raw temp file.
The 256 KiB read window in extract_html_title missed it.
Font extraction rewrites the HTML in-place (2.4 MB → 1.26 MB) before
hashing, so the title is accessible at byte ~106 KB in that content.
Fix: add extract_html_title_str() that operates on a &str; call it on
the in-memory rewritten string after font extraction (server path).
CLI path (no font extraction) falls back to result.title as before.
* fix(core): inline reader-mode images via Rust post-processing
SingleFile cannot inline resources added to the DOM at before-capture
time — only resources tracked during the page's initial load cycle get
embedded. Article images in Readability output fall into this gap when
the page framework has already resolved their lazy src to a CDN URL
that isn't in SingleFile's resource cache for the new DOM elements.
Fix in three parts:
- Browser script: after body.innerHTML = article.content, stamp
data-archivr-src=<absolute-proxy-url> on any image whose src is
not an already-inlined large data URI. Remove loading attr. Don't
touch src (let SingleFile try; Rust handles the rest).
- Rust (save_with): after SingleFile writes the file and before hashing,
call inline_archivr_img_srcs() to scan for data-archivr-src markers.
- inline_archivr_img_srcs(): fetches each marked URL with blocking
reqwest (10 s timeout, 5 redirects, image/* Content-Type guard,
20 MiB cap), base64-encodes, replaces src with the data URI, and
removes the marker attr. Non-fatal — fetch failures are logged.
Also: Freedium UI cleanup now removes footer, #progress, and empty
data-nosnippet wrappers in addition to the existing nav/toaster removal.
* fix(core): clean up Freedium chrome and drop reader-mode debug counters
Two fixes:
1. freedium_cleanup: remove remaining Freedium article chrome that
survived the existing nav/footer/toaster pass:
- <header class="p-6 bg-gray-50 ..."> — author/metadata bar with
profile pictures and byline, a Freedium wrapper around the article
- <section> containing [data-slot="dropdown-menu-trigger"] or
[aria-haspopup="menu"] — the "Download article" dropdown
Selectors target stable Tailwind utility class prefixes (p-6, bg-gray-50,
bg-zinc-800) for the header and the WAI-ARIA menu role for the button,
both resilient to minor Freedium UI updates.
2. Reader-mode meta tag: strip per-capture debug counters.
_archivrReaderMark now writes 'applied' instead of
'applied:pre_s=N,...:post_s=N,...' — the counters were useful during
development but have no place as permanent archive content.
* fix(core): prevent lazy-resolver double-processing on Freedium reader images
_archivrResolveLazyImgs is called twice: before Readability (_pre) and
after the post-body stamp pass (_post). The stamp pass sets data-archivr-src
but previously left data-src/data-zoom-src/etc intact, so _post's
'lazySrc && _isPlaceholder' condition fired on the same images and
rewrote src to the CDN URL — which SingleFile then tried and failed to
fetch from the Freedium proxy context, redundantly.
Fix: strip all lazy attrs (data-src, data-lazy-src, data-zoom-src,
data-original, data-lazy) when stamping data-archivr-src. _post then
finds no lazySrc on those images and skips them cleanly. Rust owns
them via data-archivr-src. _post continues to handle any remaining
placeholder images not covered by the stamp pass (non-Freedium reader
captures).
* fix(core): address Codex review findings in reader image post-processor
P1 — Enforce size cap before buffering (singlefile.rs fetch_image_as_data_uri):
resp.bytes() buffered the entire response before checking MAX_BYTES, enabling
OOM on oversized or attacker-controlled images. Fix: reject via Content-Length
header when present, then stream at most max_bytes+1 bytes with Read::take so
the cap is enforced without materialising the full body first.
P2 — Decode HTML entities in extracted image URLs (inline_archivr_img_srcs):
The browser's HTML serialiser encodes & as & in attribute values, so CDN
signed URLs with query parameters (e.g. ?a=1&b=2) arrived as &-escaped
strings. reqwest sent the wrong URL, breaking signed or transformed CDN
images. Fix: unescape & < > " before passing to the fetcher.
P2 — Preserve authentication for same-origin lazy images (inline_archivr_img_srcs):
The post-processor created a bare reqwest client with no cookies, causing 401/403
for reader-mode captures of auth-gated sites whose lazy images are same-origin.
Fix: pass capture_url and cookies into inline_archivr_img_srcs; attach a Cookie
header only when domain_from_url(img_url) == domain_from_url(capture_url) and
cookies is non-empty. Third-party image hosts and Freedium fetches (which receive
empty cookies in capture.rs) are unaffected.
* test(core): extract helpers and add unit tests for Codex review fixes
Extract two testable pure functions from inline_archivr_img_srcs:
- html_attr_decode: decodes HTML character references in attribute values.
Fixes decode order: & runs LAST so &lt; → < (one layer
removed), not < (two layers). Previous order caused double-decoding.
- same_origin_cookie_header: returns a Cookie header value only when
img_url's domain matches capture_url's domain.
Add 11 unit tests covering:
- html_attr_decode: plain URL no-op, & in CDN query params, single-layer
decode (&lt; → < not <), double-encoded amp (&amp; → &),
direct </>/" decode
- same_origin_cookie_header: same host attaches cookies, third-party returns
None, empty cookies returns None, Freedium empty-cookies no-op
- bounded_read: Read::take stops at max+1 bytes (guard fires), allows
exactly-at-limit payloads (guard does not fire)
* docs: refresh AGENTS.md and mental model; drop NEXT.md
AGENTS.md: mention Freedium mirror in the overview and capture flow,
add vendor/readability/ to Key Directories, drop the NEXT.md pointer.
ARCHIVR-MENTAL-MODEL.md: fix stale references to legacy static/ paths
in Where To Edit (frontend now lives in frontend/src/); add capture.rs
and auth.rs rows; replace the outdated 'Current Limitations' section
(which claimed no capture and no auth) with a Server Capabilities
section reflecting async capture jobs, the auth model, search, and
admin scope; add a Web Capture Pipeline section covering the Freedium
mirror, SingleFile+Chromium, vendored Readability, cleanup, and Rust
post-processing.
NEXT.md: remove; the roadmap tracking is stale and unused.
AGENTS.md: mention Freedium mirror in the overview and capture flow,
add vendor/readability/ to Key Directories, drop the NEXT.md pointer.
ARCHIVR-MENTAL-MODEL.md: fix stale references to legacy static/ paths
in Where To Edit (frontend now lives in frontend/src/); add capture.rs
and auth.rs rows; replace the outdated 'Current Limitations' section
(which claimed no capture and no auth) with a Server Capabilities
section reflecting async capture jobs, the auth model, search, and
admin scope; add a Web Capture Pipeline section covering the Freedium
mirror, SingleFile+Chromium, vendored Readability, cleanup, and Rust
post-processing.
NEXT.md: remove; the roadmap tracking is stale and unused.
Extract two testable pure functions from inline_archivr_img_srcs:
- html_attr_decode: decodes HTML character references in attribute values.
Fixes decode order: & runs LAST so &lt; → < (one layer
removed), not < (two layers). Previous order caused double-decoding.
- same_origin_cookie_header: returns a Cookie header value only when
img_url's domain matches capture_url's domain.
Add 11 unit tests covering:
- html_attr_decode: plain URL no-op, & in CDN query params, single-layer
decode (&lt; → < not <), double-encoded amp (&amp; → &),
direct </>/" decode
- same_origin_cookie_header: same host attaches cookies, third-party returns
None, empty cookies returns None, Freedium empty-cookies no-op
- bounded_read: Read::take stops at max+1 bytes (guard fires), allows
exactly-at-limit payloads (guard does not fire)
P1 — Enforce size cap before buffering (singlefile.rs fetch_image_as_data_uri):
resp.bytes() buffered the entire response before checking MAX_BYTES, enabling
OOM on oversized or attacker-controlled images. Fix: reject via Content-Length
header when present, then stream at most max_bytes+1 bytes with Read::take so
the cap is enforced without materialising the full body first.
P2 — Decode HTML entities in extracted image URLs (inline_archivr_img_srcs):
The browser's HTML serialiser encodes & as & in attribute values, so CDN
signed URLs with query parameters (e.g. ?a=1&b=2) arrived as &-escaped
strings. reqwest sent the wrong URL, breaking signed or transformed CDN
images. Fix: unescape & < > " before passing to the fetcher.
P2 — Preserve authentication for same-origin lazy images (inline_archivr_img_srcs):
The post-processor created a bare reqwest client with no cookies, causing 401/403
for reader-mode captures of auth-gated sites whose lazy images are same-origin.
Fix: pass capture_url and cookies into inline_archivr_img_srcs; attach a Cookie
header only when domain_from_url(img_url) == domain_from_url(capture_url) and
cookies is non-empty. Third-party image hosts and Freedium fetches (which receive
empty cookies in capture.rs) are unaffected.
_archivrResolveLazyImgs is called twice: before Readability (_pre) and
after the post-body stamp pass (_post). The stamp pass sets data-archivr-src
but previously left data-src/data-zoom-src/etc intact, so _post's
'lazySrc && _isPlaceholder' condition fired on the same images and
rewrote src to the CDN URL — which SingleFile then tried and failed to
fetch from the Freedium proxy context, redundantly.
Fix: strip all lazy attrs (data-src, data-lazy-src, data-zoom-src,
data-original, data-lazy) when stamping data-archivr-src. _post then
finds no lazySrc on those images and skips them cleanly. Rust owns
them via data-archivr-src. _post continues to handle any remaining
placeholder images not covered by the stamp pass (non-Freedium reader
captures).
Two fixes:
1. freedium_cleanup: remove remaining Freedium article chrome that
survived the existing nav/footer/toaster pass:
- <header class="p-6 bg-gray-50 ..."> — author/metadata bar with
profile pictures and byline, a Freedium wrapper around the article
- <section> containing [data-slot="dropdown-menu-trigger"] or
[aria-haspopup="menu"] — the "Download article" dropdown
Selectors target stable Tailwind utility class prefixes (p-6, bg-gray-50,
bg-zinc-800) for the header and the WAI-ARIA menu role for the button,
both resilient to minor Freedium UI updates.
2. Reader-mode meta tag: strip per-capture debug counters.
_archivrReaderMark now writes 'applied' instead of
'applied:pre_s=N,...:post_s=N,...' — the counters were useful during
development but have no place as permanent archive content.
SingleFile cannot inline resources added to the DOM at before-capture
time — only resources tracked during the page's initial load cycle get
embedded. Article images in Readability output fall into this gap when
the page framework has already resolved their lazy src to a CDN URL
that isn't in SingleFile's resource cache for the new DOM elements.
Fix in three parts:
- Browser script: after body.innerHTML = article.content, stamp
data-archivr-src=<absolute-proxy-url> on any image whose src is
not an already-inlined large data URI. Remove loading attr. Don't
touch src (let SingleFile try; Rust handles the rest).
- Rust (save_with): after SingleFile writes the file and before hashing,
call inline_archivr_img_srcs() to scan for data-archivr-src markers.
- inline_archivr_img_srcs(): fetches each marked URL with blocking
reqwest (10 s timeout, 5 redirects, image/* Content-Type guard,
20 MiB cap), base64-encodes, replaces src with the data URI, and
removes the marker attr. Non-fatal — fetch failures are logged.
Also: Freedium UI cleanup now removes footer, #progress, and empty
data-nosnippet wrappers in addition to the existing nav/toaster removal.
SingleFile embeds fonts as base64 data URIs in <style> blocks in the
<head>, pushing the <title> tag to ~1.2 MB in the raw temp file.
The 256 KiB read window in extract_html_title missed it.
Font extraction rewrites the HTML in-place (2.4 MB → 1.26 MB) before
hashing, so the title is accessible at byte ~106 KB in that content.
Fix: add extract_html_title_str() that operates on a &str; call it on
the in-memory rewritten string after font extraction (server path).
CLI path (no font extraction) falls back to result.title as before.
- extract_html_title: take().read_to_end() up to 256 KiB (single read() can
short-read, leaving title at byte ~106 K undiscovered); regression test added
- Strip " - Freedium" suffix before storing entry title
- is_freedium_fetch keys off actual fetch_url host
- Remove Freedium toast overlay ([data-sonner-toaster]) before capture
- Resolve lazy images (data-zoom-src etc.) before Readability in reader mode
- freediumEnabled state defaults to true (on by default)
- via_freedium forwarded through submitCapture to the capture API
- Toggle rendered last in the advanced panel, matching existing rows
- Built frontend static assets included
CaptureBody gains via_freedium: Option<bool>; absent defaults to true.
The rearchive handler sets it to false — existing entries should not be
silently re-fetched through a mirror.
When via_freedium is true and the locator is not already a Freedium URL,
perform_capture passes https://freedium-mirror.cfd/<original-url> to
singlefile::save() so paywalled articles are fetched via the mirror.
The original locator is kept for requested_locator and canonical_locator
in the DB entry. An empty cookie map is used for the mirror fetch to
prevent original-domain credentials from being sent to freedium-mirror.cfd.
- Apply X article typography (font-size, line-height, font-family) and
layout to the standalone preview tab (PreviewPage)
- Match scrollbar styling to x.com measurements
- Tighten body line-height to 1.5 (25.5 px at 17 px base)
- Refactor TweetPreview and PreviewPanel to share the updated styles
Port the modalcloser abx-plugin as a SingleFile --browser-script rather
than a spawned Puppeteer daemon. No external extension download required.
Core behavior (singlefile.rs):
- MODAL_CLOSER_DIALOG_OVERRIDES: main-world <script> bridge (best-effort;
blocked by strict script-src CSP) that overrides window.alert/confirm/
prompt/print, nulls window.onbeforeunload, traps its setter, and wraps
window.addEventListener to no-op 'beforeunload' registrations.
- MODAL_CLOSER_POLLING_SETUP: defines _archivr_mc_run() which runs two
passes on every tick (immediate first run, then setInterval every 500 ms
matching MODALCLOSER_POLL_INTERVAL default):
Pass 1 (best-effort, CSP-sensitive): main-world <script> bridge calls
Bootstrap/jQuery/jQuery UI/SweetAlert teardown APIs.
Pass 2 (always, CSP-immune): isolated-world DOM mutations — Escape-key
dispatch (Radix/Headless UI/Angular Material), backdrop clicks, full
CSS selector hiding for 40+ named consent/overlay vendors, body
scroll-lock reset. Direct DOM mutations need no inline script execution.
- resolve_modal_closer_config(): reads ARCHIVR_MODAL_CLOSER env var
(default true); no external resource required.
- modal_closer_enabled: Option<bool> in CaptureConfig so Default::default()
yields None (follow env var) rather than false.
Schema (database.rs):
- modal_closer_enabled column in auth DB instance_settings (default 1).
- Idempotent ALTER TABLE migration for existing databases.
- get/update_instance_settings updated (SELECT col 6, UPDATE param ?7).
HTTP (routes.rs):
- modal_closer_enabled in CaptureBody and UpdateInstanceSettingsBody.
- Per-capture body overrides global setting which overrides env var.
Frontend:
- SettingsView ExtensionsTab: third ext-card for Modal & Dialog Closer.
- CaptureDialog: per-capture toggle in Advanced Options, state initialized
from server default, included in submitCapture payload.
- api.js: submitCapture forwards modal_closer_enabled to capture body.
Behavior vs original modalcloser daemon:
- Functionally identical on non-strict-CSP pages (the large majority).
- Gap: <script> bridge is subject to page CSP; page.evaluate() is not.
On strict-CSP pages framework teardown and dialog overrides are no-ops;
CSS selector hiding and Escape dispatch (Pass 2) always work.
- Gap: alert/confirm/prompt overridden to instant no-op vs CDP timed
dialog.accept() with 1250 ms delay; irrelevant for archival in practice.
Add ⌘K search focus, URL-param'd search/tag/entry, persistent selection, extensions grid
* feat(frontend): focus search input on ⌘K/Ctrl+K
Add a global keydown handler that focuses (and selects) the search
input when Meta+K or Ctrl+K is pressed. If the current view is not
'archive', switch to it first and focus the input after the next
render frame via a pendingSearchFocus ref + requestAnimationFrame.
The ⌘K hint badge in the toolbar was already present; this wires
the behaviour behind it.
* feat(frontend): persist search query and tag filter in URL params
Extend parseLocation() to read ?q= and ?tag= search parameters.
Initialise searchQuery and tagFilter from the URL on mount so that
sharing or refreshing a filtered view restores the exact same
results.
- replaceState on every q/tag change (no extra history entries)
- pushState on view navigation now preserves existing search params
- popstate handler restores q and tag on back/forward
- firstArchiveLoad ref prevents the archive-change effect from
wiping URL-initialised filters on the first mount
* feat(frontend): persist selected entry in URL params
Extend parseLocation() to read ?entry= and initialise
selectedEntryUid from it on mount.
When entries load (or reload), a new effect checks whether
selectedEntryUid is set without a corresponding selectedEntry
object and restores it by finding the matching entry in the list.
This covers page refresh, URL sharing, and back/forward navigation.
The URL params sync effect now includes selectedEntryUid alongside
q and tag, and the popstate handler restores all three.
Also includes rebuilt frontend static assets.
* fix(frontend): add cookies and extensions to settings URL routing
SETTINGS_TABS was missing 'cookies' and 'extensions', so navigating
to /settings/cookies or /settings/extensions silently fell back to
the profile tab. Both tabs already existed in SettingsView; they
just weren't recognised by parseLocation().
Includes rebuilt frontend static assets.
* style(frontend): ext cards in auto-fill grid
* fix(frontend): URL state regressions from Codex review
Two issues:
1. Re-selection stall when popstate lands on the same entry UID.
The restoration effect only had [entries, selectedEntryUid] as
deps, so setting selectedEntry=null without changing either dep
left the rail blank. Adding selectedEntry to the dep array lets
the null→restore cycle complete.
2. tag/entry params leaking onto non-archive URLs.
pushState on view changes carried window.location.search verbatim,
so /settings?tag=foo was possible; reloading it triggered the
tagFilter effect and force-switched back to archive.
Fix: parseLocation() only extracts tag/entry when view=archive,
and the params-sync effect only emits them on archive too.
- Install unzip in runtime stage (needed to extract extension zips)
- Download uBlock Origin Lite (v2026.705.2152) and I Still Don't Care
About Cookies (v1.1.9) during build, matching flake.nix versions
- Verify each zip against its SHA-256 (converted from Nix SRI hashes)
before extraction so a changed GitHub release fails fast, same as Nix
- Check manifest.json at ISDCAC extension root post-extraction, mirroring
the flake installPhase guard
- Set ARCHIVR_UBLOCK_EXT and ARCHIVR_COOKIE_EXT env vars to the extracted
extension directories
* feat: entry previews (video, tweet, article, iframe, image, audio bar)
- Add PreviewPanel dispatch hub routing by entity_kind + artifact extension
- VideoPreview: HTML5 <video> for YouTube/Instagram/TikTok/Reddit/X posts
- TweetPreview: tweet card, thread, and X article renderer (ported from x-article-renderer)
- IframePreview: sandboxed iframe for SingleFile web pages and PDFs
- ImagePreview: image viewer with click-to-open-fullsize
- AudioBar: persistent fixed-bottom player (Spotify-style) that survives entry navigation
- Lift entryDetail to App.jsx, shared between PreviewPanel and ContextRail
- 3-column layout (workspace | 300px preview | 340px rail) when preview active
- Stale-guard fixes: seq incremented before early returns in all async effects
- handleRearchive: capture startSeq/entryUid at call time, guard every async resume
- TweetPreview: reset loading/error/tweets before early-return branches
* feat: entry previews — tweet/thread/article/video/audio/image/iframe/pdf
- PreviewModal: modal overlay with new-tab link (↗) and keyboard close
- PreviewPanel: routes by entity_kind + primary_media extension to the
correct viewer (tweet/video/audio/pdf/html/image/fallback)
- TweetPreview: full X-style tweet, thread, and article renderer with
local artifact map for archived media (CDN fallback)
- AudioBar: persistent fixed bottom player, triggered via ContextRail
Play button; body.has-audio-bar pads content above it
- VideoPreview, IframePreview, ImagePreview: inline viewers
- PreviewPage: standalone /preview/:archiveId/:entryUid route
- ContextRail: Play/Preview buttons; isAudio/isPreviewable detection
- App.jsx: preview modal state, currentAudio state, preview route guard,
has-audio-bar body class effect
- routes.rs: CSP updated (media-src self blob https; frame-ancestors self;
Google Fonts + external images/scripts whitelisted)
- styles.css: preview modal, tweet-wrap scroll (min-height:0), audio bar
body padding, newtab button, preview panel flex layout
* style: tighten article/tweet preview spacing
- aMeta padding: 14px → 10px
- article title marginBottom: 10px → 8px
- aAuthorRow marginBottom: 10px → 8px
- bH1 top margin: 20px → 16px
- bH2 top margin: 18px → 14px
- bHr margin: 20px → 14px
- .preview-tweet-wrap padding: 20px → 12px (already committed)
More content visible above the fold in both modal and standalone views.
* feat: tweet/article preview quality pass
HTML entities: decode > < & etc. on sliced segments only
(entity offsets index the stored string; decoding before slicing shifts them)
Image lightbox: click any tweet/thread/article image to open full-screen
viewer; cmd+click follows <a> to open in new tab; arrow-key + ‹ › nav;
Escape closes; 1/N counter; ↗ open-in-new-tab link
Multi-image grid: 2 photos → side-by-side (180px rows); 3 → left spans
both rows; 4 → 2×2 (140px rows); single image unchanged
Empty media grid ghost: build photos/videoItems arrays first, render
.mediaGrid div only when at least one item resolved (advisory: never gate
on raw media.length when map items can all return null)
QT indicator: ↻ QT badge on tweet.is_quote_status === true entries
Modal shrinks for short content: height: 88vh → max-height: 88vh;
.preview-modal-body gets max-height: calc(88vh - 52px) so long threads
still scroll (advisory: don't rely on flex:1 once parent has no fixed height)
Video scrollbar leak: .preview-modal-body overflow: auto → hidden; each
child (tweet-wrap, video-wrap, iframe) manages its own scroll surface
Styled thin scrollbar on .preview-tweet-wrap (matches workspace rail)
ArticleRenderer: cover image and body images are lightbox-clickable;
opts thread through renderBlocksJSX → renderBlockJSX → renderAtomicJSX
* fix: move artifact fetch to api.js; stop Escape propagation from lightbox
- Export fetchEntryArtifacts(archiveId, entryUid, indices) from api.js
using Promise.all + getJson (follows project convention: all /api calls
go through api.js, never inline fetch in components)
- TweetPreview: import fetchEntryArtifacts, replace inline Promise.all
- MediaLightbox keydown handler: stopPropagation + preventDefault for
Escape/ArrowLeft/ArrowRight so the parent PreviewModal window listener
does not also fire and close the modal behind the lightbox
* fix: iframe/page preview height chain and toolbar UX
Problem: changing .preview-modal from height to max-height broke iframe
previews - <iframe style='flex:1'> needs a concrete ancestor height, which
max-height alone doesn't supply when content is shorter than the cap.
Fix - CSS:
.preview-modal--full { height: 88vh } applied to non-tweet modals
.preview-modal--full .preview-modal-body { max-height: none }
.preview-iframe-toolbar span: remove text-transform/letter-spacing
(was uppercasing the URL/title in shouty caps)
Fix - PreviewModal: className adds --full when entity_kind is not
tweet/tweet_thread; tweet previews keep shrink-to-fit behavior.
Fix - PreviewPanel: pass title + original_url from summary to IframePreview
for both HTML and PDF; wrappers use flex:1/minHeight:0 not height:100%.
Fix - IframePreview:
- Accept title + originalUrl props; show originalUrl in toolbar (falls
back to artifact src only when original_url absent); show title above
URL when available
- flex:1 + minHeight:0 instead of height:100% on the wrap div
- Single unified layout for page + pdf (both just show the iframe)
* feat: expand t.co links; linkify bare URLs in tweet and article text
Frontend:
- resolveEntityBounds: try multiple candidate strings in order (u.url
first, since that's the t.co short URL that appears in full_text)
- normalizeUrlAnn: multi-candidate search; href = expanded > url,
display = display_url > expanded > url
- linkifyText(): regex linkifier for entity-less bare URLs; trims
trailing punctuation [.,;:!?)] before linking; used in both
renderTweetTextJSX and renderInlineJSX including their early-return
paths (anns.length === 0) that previously bypassed linkification
- renderInlineJSX: fix mention href mention.name → screen_name;
replace t.co segment text with url.display when entity covers it
Scraper (vendor/twitter/scrape_user_tweet_contents.py):
- extract_tweet_data: when note_tweet text is used, pull urls/mentions/
hashtags/symbols from note_result.entity_set (correct indices for the
note text); keep media from legacy.entities (no note media downloads)
* feat: server-side t.co resolver + frontend augmentation
Server (routes.rs):
POST /api/util/resolve-tco — unauthenticated, accepts JSON array of
https://t.co/<alphanumeric> URLs only (strict regex validation, no SSRF
via input), capped at 50 per batch, 3 s timeout, redirect(Policy::none)
so the server only ever touches t.co itself. HEAD first, GET fallback if
HEAD returns no Location. Location sanitized to http/https only —
javascript:/data:/etc. fall back to the original t.co.
api.js:
resolveTcoUrls(urls) — project-convention wrapper for the new endpoint.
Returns {} on failure (callers degrade gracefully to bare t.co links).
TweetPreview.jsx:
After tweet data loads, per-tweet range-based coverage detection:
builds covered [start,end) from existing entity fromIndex/toIndex or
indices fallback, then finds regex matches whose span is NOT covered.
Resolves unique uncovered t.co URLs via resolveTcoUrls(), synthesises
one entity per occurrence (with exact fromIndex/toIndex so normalizeUrlAnn
gets correct bounds even for duplicate t.co URLs in the same tweet).
Augments entities.urls before setTweets() so all rendering paths
see expanded URLs.
* feat: suppress rendered media attachment URLs from tweet text
renderTweetTextJSX now accepts skipSpans=[] as third param.
Skip-span boundaries are added to the pts split set so a trailing
media t.co inside a plain segment still gets isolated and suppressed—
not re-linked by linkifyText. Early return only when both anns and
skipSpans are empty.
TweetCard computes mediaSkipSpans after building photos/videoItems:
for each rawMedia item whose src resolved (photo src match; any video),
resolveEntityBounds(m, ft, m.url) gives the precise [s,e] span using
indices/fromIndex first, indexOf fallback—then the span is passed to
renderTweetTextJSX so the t.co attachment URL is silently dropped.
