1
Fork 0
mirror of https://github.com/thegeneralist01/archivr synced 2026-07-22 03:05:32 +02:00

feat: add capturing w/ parent-child entries (#32)

* feat(core): YouTube playlist/channel/YTM-playlist capture with parent–child entries

- ytdlp: add fetch_playlist_info() using yt-dlp -J --flat-playlist for
  reliable container title + shallow entry list; normalize item URLs via
  webpage_url → absolute url → id fallback (domain inferred from container
  URL so YTM stays on music.youtube.com)
- capture: add record_container_entry() (no blob, no primary_media artifact);
  extend record_media_entry() with parent_entry_id/root_entry_id params (all
  existing single-item call sites pass None, None)
- capture: implement YouTubePlaylist / YouTubeChannel / YouTubeMusicPlaylist
  capture path replacing the two not-implemented stubs: fetch playlist info →
  create container entry (reusing existing run + item) → per-child run items
  (parent_item_id = container item) → download each video/track as a child
  entry; per-child failures are non-fatal; perform_capture returns result.status
  reflecting actual run outcome so capture_handler marks the job correctly
- archive: add child_count i64 to EntrySummary (col 12 in all listing queries);
  add get_entry_summary() private helper; fix get_entry_detail() to use
  get_entry_summary() so child entries are resolvable via the detail endpoint;
  add list_child_entries(conn, uid, caller_bits) with the same
  admin/collection visibility predicate as list_root_entries

archive_runs.requested_count stays 1 (one user locator); discovered/
completed/failed_count reflect container item + N video items via
refresh_run_counters.

* feat(server,frontend): expose children endpoint + expand UI for container entries

server:
- add GET /api/archives/:id/entries/:uid/children → list_entry_children,
  calling list_child_entries with caller_bits so visibility model is enforced
- fix capture_handler: use result.status ("completed"/"failed") to set job
  status rather than always "completed", mirroring rearchive_handler; this
  surfaces partial playlist failures to the polling client

frontend:
- api.js: add fetchEntryChildren(archiveId, entryUid)
- EntryRow: one outer div.entry-row-outer (display:block) keeps nth-child
  striping correct; inner div.entry-row-main is the flex row with all column
  cells and event handling; .child-entries sits below inside the outer wrapper
- expand chevron appears when entry.child_count > 0; clicking fetches children
  lazily and renders ChildRow components reusing .col-* flex widths
- child-count badge shown next to title on container entries
- styles.css: scoped CSS with #entries-body > .entry-row-outer selectors
  (higher specificity than > div) to override flex on outer wrapper; inner row
  and column rules replicated at correct depth; nth-child, is-selected,
  is-multi-selected, url-cell hover all handled

* fix(frontend): make child entry rows interactive

ChildRow now receives onRowClick and selectedUids from EntryRow (which
receives selectedUids from EntriesView alongside the existing booleans).
Clicking a child row invokes onRowClick(child, e) so it flows through
handleRowClick → selectEntry → fetchEntryDetail exactly as a root entry
would. Shift-range selection gracefully degrades to single-select since
child entries are not in the root entries array.

Selected/multi-selected visual state is wired: .child-entry-row.is-selected
shows the same #eee2d2 background + accent outline as root rows; hover
restores full opacity. Frontend static assets rebuilt.

* feat(core): playlist per-item quality + incremental sync

ytdlp.rs:
- Add PlaylistItemProbe / PlaylistProbeResult (pub, serde::Serialize)
- Add private available_video_heights_from_value() helper for Value entries
- Add probe_playlist_qualities(): yt-dlp -J (full metadata, no flat flag)
  returns per-video quality lists in one subprocess call

capture.rs:
- Add per_item_quality: HashMap<String,String> and sync: bool to CaptureConfig
  (both Default; keyed by yt-dlp video ID, not URL)
- Add pub locator_to_playlist_url(): validates only the three playlist sources,
  expands shorthands; keeps locator_to_ytdlp_url's no-playlists contract
- Playlist capture block: sync-aware container resolution
  - sync + existing container → reuse it via complete_archive_run_item,
    skip already-archived children (by canonical URL) before creating
    run items so refresh_run_counters only counts new items
  - sync + no container → create normally (first sync run)
  - non-sync → always create fresh container (existing behaviour)
- Per-item quality: config.per_item_quality.get(id) falls back to child_quality

archive.rs:
- Add get_archived_playlist_child_urls(): returns HashSet of canonical URLs
  of all children under any container matching the playlist canonical URL
- Add find_container_entry_id_by_canonical_url(): returns most-recent
  container entry id (parent_entry_id IS NULL) for a given canonical URL

routes.rs: stub per_item_quality/sync on both CaptureConfig sites (server
agent will wire body fields in Phase 2)

* fix(core)+test: propagate sync query errors; cover new playlist/sync functions

archive.rs:
- get_archived_playlist_child_urls: collect() as rusqlite::Result<HashSet<_>>
  instead of filter_map(ok) so row-level errors surface rather than silently
  skipping and causing duplicate downloads

capture.rs:
- match get_archived_playlist_child_urls result and fail_run on error instead
  of unwrap_or_default, preventing silent re-downloads on DB failure

Tests added to capture.rs:
- locator_to_playlist_url_accepts_playlist_shorthands (yt:playlist/, ytm:playlist/, full URL)
- locator_to_playlist_url_accepts_channel_shorthands (yt:@handle)
- locator_to_playlist_url_rejects_non_playlist_sources (single video, tweet, web page)

Tests added to archive.rs (all use in-memory DB via make_tag_test_db):
- find_container_entry_id_returns_none_when_absent
- find_container_entry_id_returns_root_entry
- find_container_entry_id_ignores_child_entries (child with parent_entry_id set)
- get_archived_playlist_child_urls_empty_when_no_playlist
- get_archived_playlist_child_urls_returns_children
- get_archived_playlist_child_urls_excludes_other_playlists

* feat(server,frontend): playlist quality selector + per-video overrides + sync UI

routes.rs:
- CaptureBody gains per_item_quality (HashMap<String,String>, serde(default))
  and sync (bool, serde(default)); both validated before use
  - per_item_quality values validated against same quality predicate as top-level
    quality field ("best"|"audio"|"NNNp") so bad per-video values are rejected
    at the API boundary rather than silently falling through to quality_format
- capture_handler threads body.per_item_quality + body.sync into CaptureConfig
  (replaces hardcoded empty stubs); rearchive_handler keeps empty defaults
- New POST /api/archives/:id/captures/probe-playlist: calls
  probe_playlist_qualities via spawn_blocking; 400 for non-playlist locator,
  502 on yt-dlp failure, returns PlaylistProbeResult as JSON

api.js:
- probePlaylist(archiveId, locator): POST probe-playlist endpoint
- submitCapture: forwards per_item_quality (non-empty) and sync:true from
  extraExtensions param added to submitBgJob

CaptureDialog.jsx:
- isPlaylistSource(): detects yt:/youtube: playlist/@/channel, ytm:playlist/,
  YouTube/YTM HTTP(S) URLs with list= param or channel pathnames
- makeItem(): 6 new playlist state fields
- applyPlaylistQuality(): conflict logic — videos that can reach selected
  quality get it set; videos that can't and have no prior selection are left
  null (conflict); videos with a prior selection keep it when quality is raised
- hasConflict(): any playlistItems entry with quality===null
- updateLocator(): isPlaylistSource branch with 800ms debounce→probePlaylist;
  existing isVideoSource path unchanged
- Archive button disabled when anyConflict or any probe in flight
- Per-video expand list with individual quality selects, conflict badges,
  sync toggle (appears after probe completes)

styles.css: playlist expansion, conflict, sync toggle CSS

* fix(core): ignore per_item_quality for YTM playlist items

YouTube Music playlists force child_quality = Some("audio") because
yt-dlp can't download DRM-free audio-only tracks any other way. The
previous per_item_quality lookup could override this with e.g. "best",
defeating the invariant. Guard the lookup behind !is_audio so YTM items
are always downloaded as audio regardless of what the caller sends.

* fix(frontend): exclude /watch from isPlaylistSource

youtube.com/watch?v=...&list=... and music.youtube.com/watch are single
videos in the backend (Source::YouTubeVideo / YouTubeMusicTrack) regardless
of the list param. Previously isPlaylistSource returned true for these,
which would have triggered the playlist probe path while the video probe
was already running, and the render would attempt to show playlist UI on
an item whose playlistProbeState stays idle.

Guard: if pathname === '/watch', return false before the list-param check.

* fix(frontend): tighten isPlaylistSource to mirror backend routing exactly

Previous fix excluded /watch but still returned true for any youtube.com
URL with a ?list= param (e.g. /shorts/xxx?list=yyy). Backend determine_source
only routes to YouTubePlaylist on /playlist?list=... and to YouTubeChannel on
/@handle, /channel/, /c/, /user/ paths — everything else is a single item.

Rewrite the HTTP block to match:
- youtube.com: pathname==='/playlist' && list param → playlist
             : /@, /channel/, /c/, /user/ → channel
             : anything else (incl. /watch&list=, /shorts?list=) → false
- music.youtube.com: pathname==='/playlist' && list param → YTM playlist
                   : /watch → single track (falls through to false)

* fix(frontend): guard handleArchive against Enter-key bypass of disabled state

The Archive button is disabled when anyConflict || anyProbing, but
onKeyDown on the locator input calls onSubmit() → handleArchive()
directly, bypassing the button's disabled check entirely.

Add the same conditions as early returns inside handleArchive itself,
operating on toSubmit (the items that would actually be submitted) so
the guard is tight — items with no locator are already excluded by the
toSubmit filter.

* fix(frontend): drop m.youtube.com from isPlaylistSource

Backend determine_source playlist/channel regex only matches
(?:www\.)?youtube\.com — mobile URLs hitting m.youtube.com would be
probed as playlist in the UI but captured as Source::Url server-side.
Remove m.youtube.com from the detector to keep frontend and backend
in exact agreement. Add backend support when needed.

* fix(frontend): audio-only conflict handling in playlist quality selector

applyPlaylistQuality('audio'):
- Only sets quality='audio' on items where has_audio=true
- Items with has_audio=false: keep prior selection if set, else null
  (conflict) — same rule as unsupported height, blocks archive until
  user explicitly picks a quality for those items

Playlist-level 'Audio only' option:
- Changed hasAnyAudio → allHaveAudio (every item must have audio)
- When any item lacks audio, the option is hidden entirely so the
  selector can never create immediate conflicts just by appearing

* fix(frontend): add yt:user/ to isPlaylistSource shorthand detection

Backend determine_source routes yt:user/... (and youtube:user/...) to
YouTubeChannel — already covered by the yt: shorthand block for
playlist/, @, channel/, c/ but missing user/. Old-style user channel
URLs would capture correctly server-side but never show the playlist
quality/sync UI.

* fix(frontend): block playlist submission unless probe is done

Previous guard only blocked while playlistProbeState==='probing'.
Two remaining bypass paths:
- idle: 800ms debounce not yet fired after URL typed
- error: probe failed — no per-video quality data available

Change anyProbing and handleArchive guard to:
  isPlaylistSource(locator) && playlistProbeState !== 'done'

This means idle/probing/error all block submission for playlist items.
error is intentionally blocking — without quality data the per-video
requirement can't be satisfied; user must retry or remove the URL.

* fix(frontend): accurate error message when playlist probe fails

Previous text said 'using best quality' implying the capture would
proceed, but probe error now blocks submission. Replace with 'Probe
failed — edit URL to retry' in orange (capture-quality-hint--error)
so the disabled button and the message are consistent.

* fix(frontend): exact quality match in applyPlaylistQuality

Replace maxHeight >= newHeight (cap check) with item.qualities.includes(newQ)
(exact match). A video with [2160p, 1080p] does not support 1440p; the
previous logic would mark it as supporting any quality up to 2160p and
submit '1440p' which yt-dlp silently downloads as 1080p — misrepresenting
the selected quality.

With exact match, unsupported qualities correctly fall through to the
conflict path (keep prior selection or null), enforcing the same manual-
choice requirement as any other unsupported quality.

Per-row selects are unaffected: they already render only pi.qualities
(the video's actual available formats), no maxHeight logic involved.

* fix(frontend): move playlist expand chevron to left of input

User asked for the chevron to be on the left of the playlist input,
not tucked after the quality selector on the right.

- Remove chevron from qualityEl (it was between the quality select and
  the remove button)
- Add it as the first child of capture-row-main, before the <input>,
  when isPlaylistSource && playlistProbeState === 'done'
- Show a same-width placeholder span while probing/idle/error so the
  input does not jump left when the chevron appears after probe completes
- Add capture-playlist-toggle--left modifier (flex-shrink:0, tighter
  padding) and .capture-playlist-toggle-placeholder (fixed 22px width)

* doc(server): scope per_item_quality guarantee to the UI

The server validates per_item_quality value shapes but does not enforce
that every playlist item has an entry. Items without an override get
the global quality as a yt-dlp cap with graceful fallback.

The 'must choose quality for unsupported videos' invariant is a UI
constraint enforced by the frontend before submission. A direct API
caller bypassing the UI accepts yt-dlp's standard cap-and-fallback
behavior. Document this scope explicitly so the gap is intentional,
not accidental.

* fix(frontend): prevent 'reading some of undefined' crash from stale sessionStorage

Old captureItems entries saved before the playlist fields were added
have playlistItems=undefined (missing key). The hasConflict guard
checked !== null, which undefined passes, then called .some() on
undefined → TypeError.

Two-part fix:
1. sessionStorage restore: merge each saved item over makeItem() defaults
   so any missing fields (playlistItems, playlistProbeState, etc.) are
   filled with their correct initial values before the item is used
2. hasConflict: use Array.isArray() instead of !== null so undefined
   is also safely rejected — defence-in-depth for any future field gap

* fix(core): playlist total size includes children; per_item_quality as include-set

archive.rs: total_artifact_bytes for root entries now adds a correlated
subquery summing children's blob bytes so playlist/channel containers
show the real download size instead of 0.

capture.rs: non-empty per_item_quality map now acts as an include-set —
items whose yt-dlp ID is absent are skipped entirely. This wires the
UI's per-video delete button to actual capture exclusion. Empty map
preserves the existing behaviour (download everything).

routes.rs + capture.rs doc: comments updated to reflect both semantics
(empty = all / non-empty = only listed IDs) accurately.

* fix(frontend): QA fixes — playlist UX, selection stroke, URL expand

CaptureDialog.jsx:
- Placeholder no longer appears on idle playlist rows (only during
  probing); chevron shows only after probe completes — no left-padding
  while the user is still typing
- Per-video delete button added to expanded playlist list; removes item
  from playlistItems so its ID is absent from per_item_quality on submit
- anyEmptyPlaylist guard: Archive button disabled + handleArchive early-
  return when all videos have been deleted (empty map would otherwise
  silently download everything)

styles.css:
- Selection stroke switched from outline to box-shadow:inset everywhere
  (entry-row-outer, child-entry-row, legacy flat-div selector) —
  guaranteed inside element bounds, no layout interference
- URL cell overflow only on :hover; removed is-selected .url-cell rule
  that was expanding child URLs when their parent was selected
- Playlist items redesign: separator lines instead of background fills,
  amber left-border for conflicts, thin scrollbar, tighter padding
- Remove button: opacity 0.3 always-visible baseline; full opacity on
  hover/:focus-visible; forced to 1 on coarse-pointer (touch) devices

* fix(frontend): no left gap on playlist rows until chevron exists

Remove the probing-state placeholder span entirely. The chevron renders
only when playlistProbeState === 'done'; all other states (idle, probing,
error) render null. The small layout shift when the chevron appears after
probe is acceptable; blank padding while there is no chevron is not.

* fix(frontend): clear box-shadow on entry-row-outer to prevent double stroke

The flat selector '#entries-body > div.is-selected' already applies
box-shadow to .entry-row-outer (it IS a direct child div). The outer
override rule only cleared 'outline', so the box-shadow leaked through,
wrapping the entire parent+children block with a second stroke.

Add box-shadow: none to both .is-selected and .is-multi-selected on
.entry-row-outer so the stroke sits only on .entry-row-main.

* docs: document per-video exclude in README YouTube playlists section

* fix(frontend): scope selection background to entry-row-main only

Moving background:#eee2d2 off .entry-row-outer onto > .entry-row-main
so that expanded child entries don't inherit the selection highlight.

Outer wrapper now explicitly unsets background (cancelling the flat-div
cascade rule) and clears outline+box-shadow. Both the selection colour
and the inset stroke live on .entry-row-main only.

* fix(frontend): alternating stripe backgrounds on child entry rows

Child rows were inheriting the parent entry's stripe color, making the
expanded list look like one flat block. Apply the same odd/even palette
as root entries (var(--paper-3) / #f2ede5) so each video row is visually
distinct within the expanded group.

* fix(core): delete_entry correctly nulls FK for child entries

archive_run_items.produced_entry_id has no ON DELETE action so it must
be manually nulled before the entry row is deleted. The old query used
WHERE root_entry_id = entry_id, which finds descendants of a root but
returns nothing when entry_id IS a child (children have no sub-children,
so no row has root_entry_id = child_id). The DELETE then failed under
foreign_keys=ON.

Fix: add OR produced_entry_id = ?1 so the entry's own run_item FK is
always cleared before deletion, regardless of whether it is a root or a
child. The subtree subquery is kept for the root-deletion case where all
child run_items also need nulling.

* fix(core+frontend): child entry selection and delete correctness

database.rs — delete_entry:
- subtree_ids now uses WHERE id = ?1 OR root_entry_id = ?1 so the entry
  itself is always included; previously a child deletion passed an empty
  vec to cascade_cached_bytes_after_subtree_delete (no grandchildren
  exist), leaving cached_bytes stale on entries sharing that child's blobs

App.jsx — handleRowClick:
- Shift-range now queries DOM order (#entries-body [data-entry-uid])
  instead of entries.findIndex(); child rows are in the DOM but not in
  the root entries array, so findIndex always returned -1 for them
- Ctrl/meta branch computes next set before the state update so
  selectEntry can fire synchronously for child rows; auto-snap only
  restores root entries, so ctrl-clicking a lone child never loaded its
  detail panel — now calls selectEntry(entry) when the child ends up as
  the sole selection, selectEntry(null) on multi or deselect
- selectedUids added to handleRowClick's useCallback dep array

* fix(frontend): resolve detail entry for any remaining child after ctrl-deselect

Add entryCacheRef (uid→entry Map) populated on every row click. When
ctrl/meta-deselecting leaves exactly one other entry selected, look up
the remaining UID in the cache before falling back to the root entries
array. Without this, deselecting from a multi-selection where the
remaining entry is a child row left selectedEntry null (auto-snap only
searches root entries).

* fix(frontend): child row stripes, deleted-child visibility, selection completeness

EntryRow.jsx / ChildRow:
- Index-based light/dark classes (child-entry-row--light/dark) replace
  nth-child rules; parity comes from children.map idx so no sibling —
  including the loading div — can shift the stripe order
- Loading div moved outside .child-entries so it never affects child
  row ordering at all
- Accept deletedUids prop; filter expanded children array before render
  so deleted children disappear immediately without waiting for reload

EntriesView.jsx: thread deletedUids through to EntryRow

App.jsx:
- Add deletedUids state; handleEntryDeleted/handleBulkDeleted populate it
- isRoot/hasChildDelete computed from entries before setEntries (safe in
  StrictMode — no side-effects inside updater functions)
- Child delete triggers loadEntries to refresh stale parent child_count
  and total_artifact_bytes
- handleRowClick ctrl/meta cache-miss branch uses det.summary (not det)
  from fetchEntryDetail; archiveId added to dep array
- handleRowClick dep array includes archiveId

* fix(frontend): add inset stroke to child-entry-row.is-multi-selected

* fix(frontend): suppress mouse-click focus ring on entry expand button

* fix(frontend): index-based stripes for root and child rows

Root rows: EntriesView passes rowIndex from entries.map to EntryRow,
which applies entry-row-outer--light/dark. Retires both nth-child stripe
blocks so skeleton rows can never shift the first real entry to dark.
Skeleton rows keep a :not(.entry-row-outer) nth-child fallback.

Child rows: colours changed from the warm root palette (paper-3/#f2ede5)
to cooler near-whites (#fafaf8/#f2f0ec) so children are visually distinct
from their parent row regardless of which stripe the parent sits on.

* feat(core+frontend): bare yt:ID resolves to YouTube video

determine_source: yt:ID / youtube:ID with no prefix and exactly 11
chars [A-Za-z0-9_-] → YouTubeVideo via is_youtube_video_id helper.
Reserved prefixes (playlist/, channel/, c/, user/, @) still fire first
so they are unaffected by the fallback.

expand_shorthand_to_url: bare yt:ID expands to watch?v=ID using the
same predicate, consistent with how ytm:ID → music.youtube.com/watch.

isVideoSource (frontend): same 11-char /^[A-Za-z0-9_-]{11}$/ regex so
yt:ID triggers the quality probe and capture-row guards identically to
yt:video/ID.

Tests: test_is_youtube_video_id covers valid IDs (alphanumeric, with _
and -), too-short, too-long, and invalid-char cases using genuinely
invalid fixtures. test_youtube_sources adds bare-ID cases and confirms
reserved prefixes (playlist/, @) are not affected.

* fix(core+frontend): exclude avatar blobs from tweet % cached display

tweet/tweet_thread entries have avatar artifacts that are always
deduplicated from the first capture of each author. Counting them in
the cached-bytes percentage makes it artificially high (or incorrect
when the real media content is new but avatars are cached).

database.rs:
- refresh_entry_cached_bytes: AND ea.artifact_role != 'avatar'
- cascade_cached_bytes_after_delete: same filter
- cascade_cached_bytes_after_subtree_delete: same filter
- Initial cached_bytes migration: same filter
- Re-migration (else branch): recomputes cached_bytes for existing
  entries that have avatar artifacts, scoped to only those entries

archive.rs:
- EntrySummary gains cacheable_bytes: i64 — non-avatar total bytes,
  computed inline in every SQL query as the denominator for % cached
- ENTRY_SELECT_COLS adds cacheable_bytes at index 13 (with children
  subquery, same as total_artifact_bytes)
- list_root_entries adds same expression
- All 6 row-mapping closures include cacheable_bytes: row.get(13)?

EntryRow.jsx:
- SIZE display stays: formatBytes(entry.total_artifact_bytes)
- % cached badge uses cacheable_bytes as denominator:
  cached_bytes / cacheable_bytes * 100

* feat(frontend): j/k keyboard navigation for entries; test avatar cached_bytes

App.jsx — j/k handler:
- Fires on keydown when not focused on INPUT/TEXTAREA/SELECT/contenteditable
- Ignores meta/ctrl/alt modifier combos
- Uses DOM order (#entries-body [data-entry-uid]) so expanded child rows
  participate, matching the shift-range selection logic
- Resolves target entry via entryCacheRef → root entries array →
  fetchEntryDetail(..).summary on cache miss
- Scrolls target into view (block: nearest); updates lastAnchorIndexRef
  so subsequent shift-click ranges start from the keyboard-navigated row

archive.rs — cached_bytes_excludes_avatar_blobs test:
- Two tweet entries sharing an avatar blob (100B) and a media blob (900B)
- Asserts refresh_entry_cached_bytes sets cached_bytes = 900 (not 1000)
- Asserts list_root_entries summary: cached_bytes=900, cacheable_bytes=900,
  total_artifact_bytes=1000 — SIZE includes avatar, % cached denominator
  and numerator both exclude it

* fix(frontend): guard j/k uncached-child fetch with monotonic token

Rapid j/k over child rows that aren't in entryCacheRef triggers
fetchEntryDetail calls in parallel. Without a guard the last one to
settle wins, desyncing selectedUids (highlight) from selectedEntry
(detail panel/URL).

Fix:
- jkSeqRef (monotonic counter) incremented before each server fetch;
  the .then() guard tok === jkSeqRef.current drops results from
  superseded navigations
- handleRowClick increments jkSeqRef.current so any click also cancels
  an in-flight j/k fetch

* fix(frontend): guard ctrl/meta cache-miss fetch; add / search shortcut

App.jsx ctrl/meta branch: cache-miss fetchEntryDetail now captures
tok = ++jkSeqRef.current before the fetch and gates selectEntry on
tok === jkSeqRef.current — same pattern as the j/k handler — so a
slow response after a later click/navigate can't overwrite selection.

/ key: reuses the Cmd+K/Ctrl+K handler path (focus+select search input,
or pendingSearchFocus + archive view switch). Guards: editable targets
(INPUT/TEXTAREA/SELECT/contentEditable) and modifier keys are checked
before preventDefault() so typing / in inputs is untouched.

* feat(core): attempt SpotifyTrack via yt-dlp instead of hard-failing

Previously all Spotify sources returned an error claiming yt-dlp cannot
download DRM-protected audio. yt-dlp does have a Spotify extractor
(experimental, content-dependent), so refusing upfront is worse than
trying and letting it report the real failure.

SpotifyTrack now goes through the same yt-dlp metadata + audio-quality
download path as YouTubeMusicTrack. SpotifyAlbum and SpotifyPlaylist
still return an explicit error — fetch_playlist_info has YouTube-specific
URL fallback logic that would produce bogus child URLs for Spotify flat
entries; those sources need dedicated container handling first.

* chore: remove docs/superpowers from repo and gitignore whitelist

* feat(core+frontend): Spotify album/playlist capture via yt-dlp container path

Previously SpotifyAlbum and SpotifyPlaylist hard-failed with a DRM error.
yt-dlp does support Spotify (experimentally), so they now go through the
same probe/container/child path as YouTube playlists.

ytdlp.rs:
- Extract normalize_item_url() helper used by both fetch_playlist_info
  and probe_playlist_qualities; eliminates the duplicate URL-normalization
  blocks and the YouTube-specific fallback_host variable
- Fallback logic is now platform-aware: YouTube/YTM bare IDs → watch URL,
  Spotify → open.spotify.com/track/{id}, unknown → skip with warning

capture.rs:
- locator_to_playlist_url: accept SpotifyAlbum | SpotifyPlaylist so the
  probe-playlist endpoint accepts Spotify album/playlist URLs
- Container branch: add SpotifyAlbum | SpotifyPlaylist to the matches!
- is_audio: true for SpotifyAlbum | SpotifyPlaylist (audio-only, like YTM)
- child_source: SpotifyTrack for Spotify containers (not YouTubeMusicTrack)
- generate_entry_title and record_media_entry both use the correct child
  source so entity_kind, source_kind, and representation_kind are right

CaptureDialog.jsx:
- isPlaylistSource: recognise open.spotify.com/album/ and /playlist/,
  and spotify:album:ID / spotify:playlist:ID shorthands, so Spotify
  containers get the probe UI, per-track excludes, and sync toggle

* fix(core+server): partial playlist refresh + child visibility inheritance

database.rs:
- finish_archive_run: reverted 'partial' status (violates CHECK constraint);
  restored binary completed/failed
- get_run_completed_count(): new helper for callers that need to distinguish
  partial success without touching the DB status enum

capture.rs:
- CaptureResult gains completed_count: i64 (0 for single-item captures;
  populated from get_run_completed_count for container/playlist captures)
- All CaptureResult construction sites updated

routes.rs:
- Playlist job_status mapping: mark job 'completed' when completed_count > 0
  (even if status == 'failed'), so partially-successful playlist captures
  trigger onCaptured and show the archived entries without a manual reload.
  Truly zero-success runs (completed_count == 0, status == 'failed') stay
  failed as expected.
- probe_playlist_handler error message updated to mention Spotify

archive.rs (list_child_entries):
- Add third OR arm: children are visible when their parent container is in a
  collection visible to the caller. Fixes empty child list for non-admin
  users with access to a playlist container but not its newly-created
  children (which are all in the default 'private' collection).

* fix(server): use completed_count to distinguish partial from total failure

finish_archive_run is binary (completed/failed) — a playlist where some
tracks succeed and some fail returns status='failed'. Without this fix,
the job mapping treated any failed status as a job failure, causing
onCaptured to never fire and leaving successfully archived entries hidden
until a manual reload.

Now: job is 'failed' only when status='failed' && completed_count==0.
Partial runs (completed_count > 0) map to a completed job so the UI
refreshes and shows the entries that were successfully captured.

* fix(core+server): exclude container from child success count; rename field; tests

database.rs:
- get_run_completed_child_count(): renamed from get_run_completed_count and
  scoped to child items only (parent_item_id IS NOT NULL). The container
  run item is always completed first, so the old function inflated the count
  by at least 1 for every playlist, making all-video-failed runs appear as
  partial successes.
- New regression test: completed_child_count_excludes_container — completes
  both a root and a child item, asserts DB completed_count==2 while
  get_run_completed_child_count==1.

capture.rs:
- CaptureResult.completed_count renamed to completed_child_count to match
  the function and make the semantics unambiguous at the call site.

routes.rs:
- job_status decision now uses result.completed_child_count == 0 so that a
  playlist where every video fails (child_count==0) is correctly reported
  as a failed job, not a completed one.

archive.rs:
- New regression test: list_child_entries_inherits_parent_visibility —
  enrolls only the container in a USER-visible collection, asserts the
  child is visible to a USER caller and invisible to a GUEST caller.
This commit is contained in:
TheGeneralist 2026-07-21 21:57:29 +02:00 committed by GitHub
parent d202e177e1
commit 6377daadae
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
23 changed files with 2243 additions and 2640 deletions

View file

@ -16,6 +16,8 @@ Three crates with a strict ownership split — **core owns truth; CLI and server
Capture flow: locator → `determine_source()` (`crates/archivr-core/src/capture.rs`) routes by platform/shorthand (`yt:`, `x:`, `tweet:` …) → platform downloader (`downloader/ytdlp.rs`, `tweets.rs`, `singlefile.rs`, `http.rs`, `local.rs`) stages into `temp/` → SHA3-256 dedup (`hash.rs`, `downloader/store.rs`) moves blobs to `raw/A/B/HASH.EXT` → rows written to `archivr.sqlite` (runs, entries, artifacts, blobs) → served via `/api/archives/:id/...`. `CaptureConfig` carries per-request toggles (uBlock, reader mode, Freedium mirror, etc.); when `via_freedium` is set, the fetch URL is rewritten through `freedium-mirror.cfd` while the canonical DB URL stays the original locator. Capture flow: locator → `determine_source()` (`crates/archivr-core/src/capture.rs`) routes by platform/shorthand (`yt:`, `x:`, `tweet:` …) → platform downloader (`downloader/ytdlp.rs`, `tweets.rs`, `singlefile.rs`, `http.rs`, `local.rs`) stages into `temp/` → SHA3-256 dedup (`hash.rs`, `downloader/store.rs`) moves blobs to `raw/A/B/HASH.EXT` → rows written to `archivr.sqlite` (runs, entries, artifacts, blobs) → served via `/api/archives/:id/...`. `CaptureConfig` carries per-request toggles (uBlock, reader mode, Freedium mirror, etc.); when `via_freedium` is set, the fetch URL is rewritten through `freedium-mirror.cfd` while the canonical DB URL stays the original locator.
YouTube playlists and channels produce a **parent container entry** with each video captured as a child entry. `downloader/ytdlp.rs` handles the playlist probe (fetching per-video quality metadata before archiving), the multi-video download loop, and sync mode (skipping already-archived videos when re-archiving a playlist or channel).
Per-archive layout (created by `archivr init`): `.archivr/` (name, store_path, `archivr.sqlite`) + sibling `store/` (`raw/`, `raw_tweets/`, `structured/`, `temp/`). Server-level auth lives in a **separate** `archivr-auth.sqlite` (users, sessions, API tokens, role bits GUEST=1/USER=2/ADMIN=4/OWNER=8). Per-archive layout (created by `archivr init`): `.archivr/` (name, store_path, `archivr.sqlite`) + sibling `store/` (`raw/`, `raw_tweets/`, `structured/`, `temp/`). Server-level auth lives in a **separate** `archivr-auth.sqlite` (users, sessions, API tokens, role bits GUEST=1/USER=2/ADMIN=4/OWNER=8).
The server mounts multiple archives from a TOML registry (`crates/archivr-server/src/registry.rs`); routes are parameterized by `:archive_id`. The server mounts multiple archives from a TOML registry (`crates/archivr-server/src/registry.rs`); routes are parameterized by `:archive_id`.
@ -78,6 +80,7 @@ No CI is configured; no rustfmt.toml/clippy.toml — default `cargo fmt`/`clippy
- `crates/archivr-server/src/main.rs` — server bootstrap: config load, archive mounting, auth DB init, stalled-job recovery (running → failed on startup). - `crates/archivr-server/src/main.rs` — server bootstrap: config load, archive mounting, auth DB init, stalled-job recovery (running → failed on startup).
- `crates/archivr-server/src/routes.rs` — all HTTP handlers and the router; grep here first for API work. - `crates/archivr-server/src/routes.rs` — all HTTP handlers and the router; grep here first for API work.
- `crates/archivr-core/src/capture.rs``perform_capture()`, `Source` enum, shorthand parsing. - `crates/archivr-core/src/capture.rs``perform_capture()`, `Source` enum, shorthand parsing.
- `crates/archivr-core/src/downloader/ytdlp.rs` — yt-dlp integration; YouTube playlist/channel probe and download, sync mode logic.
- `crates/archivr-core/src/database.rs` — single source of truth for all SQLite schema and queries (both archive and auth DBs). - `crates/archivr-core/src/database.rs` — single source of truth for all SQLite schema and queries (both archive and auth DBs).
- `frontend/src/App.jsx` / `frontend/src/api.js` — frontend root state and API surface. - `frontend/src/App.jsx` / `frontend/src/api.js` — frontend root state and API surface.
- `docker/config.example.toml` — server config schema: `bind`, `auth_db_path`, repeated `[[archives]]` (`id`, `label`, `archive_path`). - `docker/config.example.toml` — server config schema: `bind`, `auth_db_path`, repeated `[[archives]]` (`id`, `label`, `archive_path`).

View file

@ -64,6 +64,17 @@ label = "Personal"
archive_path = "/path/to/archive/.archivr" archive_path = "/path/to/archive/.archivr"
``` ```
## Entry Nesting
Entries support a two-level parent/child hierarchy. A **container entry** (playlist or channel) holds zero or more child entries (individual videos). Container entries have no primary media artifact of their own; their `total_artifact_bytes` is the sum of their children's bytes.
Rules:
- Maximum nesting depth is 2 (root → child). Children cannot have children.
- The UI shows child entries collapsed under their parent, expandable with a chevron.
- Container entries are created by playlist/channel captures. Single-video and all other source types produce a standalone root entry with no children.
If a feature touches how entries are parented or how the UI groups them, start in `archivr-core` (`database.rs` for schema, `archive.rs` for listing, `capture.rs` for creation).
## How To Run It ## How To Run It
There are two user-facing binaries: There are two user-facing binaries:
@ -155,6 +166,7 @@ sequenceDiagram
| Capture orchestration, `Source` routing, `CaptureConfig` | `crates/archivr-core/src/capture.rs` | | Capture orchestration, `Source` routing, `CaptureConfig` | `crates/archivr-core/src/capture.rs` |
| Archive opening, listing entries, entry detail, runs | `crates/archivr-core/src/archive.rs` | | Archive opening, listing entries, entry detail, runs | `crates/archivr-core/src/archive.rs` |
| Download/save behavior | `crates/archivr-core/src/downloader/` | | Download/save behavior | `crates/archivr-core/src/downloader/` |
| YouTube playlist/channel download, playlist probe, sync mode | `crates/archivr-core/src/downloader/ytdlp.rs` and `capture.rs` |
| CLI commands, argument parsing, terminal output | `crates/archivr-cli/src/main.rs` | | CLI commands, argument parsing, terminal output | `crates/archivr-cli/src/main.rs` |
| Server API routes | `crates/archivr-server/src/routes.rs` | | Server API routes | `crates/archivr-server/src/routes.rs` |
| Auth model (users, sessions, tokens, roles) | `crates/archivr-server/src/auth.rs` | | Auth model (users, sessions, tokens, roles) | `crates/archivr-server/src/auth.rs` |

View file

@ -30,6 +30,10 @@ pub struct EntrySummary {
pub has_favicon: bool, pub has_favicon: bool,
/// Bytes of blobs already on disk from an earlier entry (precomputed at capture time). /// Bytes of blobs already on disk from an earlier entry (precomputed at capture time).
pub cached_bytes: i64, pub cached_bytes: i64,
/// Number of direct child entries; 0 for non-container entries.
pub child_count: i64,
/// Total non-avatar artifact bytes (query-time; used as denominator for cache-hit %).
pub cacheable_bytes: i64,
} }
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)] #[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)]
@ -212,10 +216,12 @@ pub fn list_root_entries(
e.visibility, e.visibility,
si.canonical_url, si.canonical_url,
COUNT(ea.id) AS artifact_count, COUNT(ea.id) AS artifact_count,
COALESCE(SUM(b.byte_size), 0) AS total_artifact_bytes, COALESCE(SUM(b.byte_size), 0) + COALESCE((SELECT SUM(b2.byte_size) FROM archived_entries c2 JOIN entry_artifacts ea2 ON ea2.entry_id = c2.id JOIN blobs b2 ON b2.id = ea2.blob_id WHERE c2.parent_entry_id = e.id), 0) AS total_artifact_bytes,
NULL AS parent_entry_uid, NULL AS parent_entry_uid,
EXISTS(SELECT 1 FROM entry_artifacts fav WHERE fav.entry_id = e.id AND fav.artifact_role = 'favicon') AS has_favicon, EXISTS(SELECT 1 FROM entry_artifacts fav WHERE fav.entry_id = e.id AND fav.artifact_role = 'favicon') AS has_favicon,
e.cached_bytes e.cached_bytes,
(SELECT COUNT(*) FROM archived_entries child WHERE child.parent_entry_id = e.id) AS child_count,
COALESCE(SUM(CASE WHEN ea.artifact_role != 'avatar' THEN b.byte_size ELSE 0 END), 0) + COALESCE((SELECT SUM(b2.byte_size) FROM archived_entries c2 JOIN entry_artifacts ea2 ON ea2.entry_id = c2.id JOIN blobs b2 ON b2.id = ea2.blob_id WHERE c2.parent_entry_id = e.id), 0) AS cacheable_bytes
FROM archived_entries e FROM archived_entries e
JOIN source_identities si ON si.id = e.source_identity_id JOIN source_identities si ON si.id = e.source_identity_id
LEFT JOIN entry_artifacts ea ON ea.entry_id = e.id LEFT JOIN entry_artifacts ea ON ea.entry_id = e.id
@ -248,6 +254,8 @@ pub fn list_root_entries(
parent_entry_uid: row.get(9)?, parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0, has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?, cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
}) })
})? })?
.collect::<rusqlite::Result<Vec<_>>>()?; .collect::<rusqlite::Result<Vec<_>>>()?;
@ -255,6 +263,40 @@ pub fn list_root_entries(
Ok(entries) Ok(entries)
} }
/// Fetches one `EntrySummary` for any entry (root or child) by uid.
/// Returns `None` if not found.
fn get_entry_summary(
conn: &rusqlite::Connection,
entry_uid: &str,
) -> Result<Option<EntrySummary>> {
let sql = format!(
"{} {} WHERE e.entry_uid = ?1 GROUP BY e.id",
ENTRY_SELECT_COLS, ENTRY_FROM_JOINS,
);
let mut stmt = conn.prepare(&sql)?;
let result = stmt
.query_row([entry_uid], |row| {
Ok(EntrySummary {
entry_uid: row.get(0)?,
archived_at: row.get(1)?,
source_kind: row.get(2)?,
entity_kind: row.get(3)?,
title: row.get(4)?,
visibility: row.get(5)?,
original_url: row.get(6)?,
artifact_count: row.get(7)?,
total_artifact_bytes: row.get(8)?,
parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
})
})
.optional()?;
Ok(result)
}
pub fn get_entry_detail( pub fn get_entry_detail(
conn: &rusqlite::Connection, conn: &rusqlite::Connection,
entry_uid: &str, entry_uid: &str,
@ -279,9 +321,7 @@ pub fn get_entry_detail(
return Ok(None); return Ok(None);
}; };
let summary = list_root_entries(conn, u32::MAX)? let summary = get_entry_summary(conn, entry_uid)?
.into_iter()
.find(|entry| entry.entry_uid == entry_uid)
.context("entry disappeared while loading detail")?; .context("entry disappeared while loading detail")?;
let mut stmt = conn.prepare( let mut stmt = conn.prepare(
@ -438,12 +478,121 @@ pub fn list_entries_for_collection(
parent_entry_uid: row.get(9)?, parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0, has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?, cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
}) })
})? })?
.collect::<rusqlite::Result<Vec<_>>>()?; .collect::<rusqlite::Result<Vec<_>>>()?;
Ok(entries) Ok(entries)
} }
/// Returns the direct children of the entry identified by `parent_entry_uid`,
/// ordered ascending by `archived_at, id` (preserves playlist ordinal feel).
/// Returns an empty vec if the parent has no children or does not exist.
pub fn list_child_entries(
conn: &rusqlite::Connection,
parent_entry_uid: &str,
caller_bits: u32,
) -> Result<Vec<EntrySummary>> {
let sql = format!(
"{} {} \
WHERE e.parent_entry_id = (SELECT id FROM archived_entries WHERE entry_uid = ?1) \
AND (\
CAST(?2 AS INTEGER) & 12 != 0 \
OR EXISTS (\
SELECT 1 FROM collection_entries ce \
WHERE ce.entry_id = e.id \
AND ce.visibility_bits & CAST(?2 AS INTEGER) != 0\
)\
OR EXISTS (\
SELECT 1 FROM collection_entries ce_p \
WHERE ce_p.entry_id = (SELECT id FROM archived_entries WHERE entry_uid = ?1)\
AND ce_p.visibility_bits & CAST(?2 AS INTEGER) != 0\
)\
) \
GROUP BY e.id \
ORDER BY e.archived_at ASC, e.id ASC",
ENTRY_SELECT_COLS, ENTRY_FROM_JOINS,
);
let mut stmt = conn.prepare(&sql)?;
let entries = stmt
.query_map(
rusqlite::params![parent_entry_uid, caller_bits as i64],
|row| {
Ok(EntrySummary {
entry_uid: row.get(0)?,
archived_at: row.get(1)?,
source_kind: row.get(2)?,
entity_kind: row.get(3)?,
title: row.get(4)?,
visibility: row.get(5)?,
original_url: row.get(6)?,
artifact_count: row.get(7)?,
total_artifact_bytes: row.get(8)?,
parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
})
},
)?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(entries)
}
/// Returns the set of canonical URLs for all child entries archived under
/// **any** container entry whose own canonical URL matches `playlist_canonical_url`.
///
/// Used in sync mode so the playlist capture path can skip videos that were
/// already downloaded in a previous run of the same playlist.
pub fn get_archived_playlist_child_urls(
conn: &rusqlite::Connection,
playlist_canonical_url: &str,
) -> Result<std::collections::HashSet<String>> {
let mut stmt = conn.prepare(
"SELECT si_child.canonical_url \
FROM archived_entries child \
JOIN source_identities si_child ON si_child.id = child.source_identity_id \
WHERE child.parent_entry_id IN ( \
SELECT e.id \
FROM archived_entries e \
JOIN source_identities si ON si.id = e.source_identity_id \
WHERE si.canonical_url = ?1 \
AND e.parent_entry_id IS NULL \
)",
)?;
let urls = stmt
.query_map([playlist_canonical_url], |row| row.get::<_, String>(0))?
.collect::<rusqlite::Result<std::collections::HashSet<_>>>()?;
Ok(urls)
}
/// Finds the most recent container entry (parent_entry_id IS NULL) whose
/// canonical URL matches `canonical_url`. Returns its row id, or None if
/// no such entry exists.
///
/// Used in sync mode to reuse an existing playlist/channel container instead
/// of creating a duplicate root entry on every sync run.
pub fn find_container_entry_id_by_canonical_url(
conn: &rusqlite::Connection,
canonical_url: &str,
) -> Result<Option<i64>> {
let mut stmt = conn.prepare(
"SELECT e.id \
FROM archived_entries e \
JOIN source_identities si ON si.id = e.source_identity_id \
WHERE si.canonical_url = ?1 \
AND e.parent_entry_id IS NULL \
ORDER BY e.archived_at DESC \
LIMIT 1",
)?;
let id = stmt
.query_row([canonical_url], |row| row.get::<_, i64>(0))
.optional()?;
Ok(id)
}
/// Resolves an artifact to its absolute on-disk path under `store_path`. /// Resolves an artifact to its absolute on-disk path under `store_path`.
/// ///
/// `artifact.relpath` is a store-relative path (e.g. `raw/a/b/abc.pdf`). /// `artifact.relpath` is a store-relative path (e.g. `raw/a/b/abc.pdf`).
@ -552,10 +701,12 @@ pub fn parse_search_query(raw: &str) -> Result<SearchEntriesQuery, String> {
const ENTRY_SELECT_COLS: &str = "SELECT e.entry_uid, e.archived_at, e.source_kind, e.entity_kind, e.title, \ const ENTRY_SELECT_COLS: &str = "SELECT e.entry_uid, e.archived_at, e.source_kind, e.entity_kind, e.title, \
e.visibility, si.canonical_url, COUNT(ea.id) AS artifact_count, \ e.visibility, si.canonical_url, COUNT(ea.id) AS artifact_count, \
COALESCE(SUM(b.byte_size), 0) AS total_artifact_bytes, \ COALESCE(SUM(b.byte_size), 0) + COALESCE((SELECT SUM(b2.byte_size) FROM archived_entries c2 JOIN entry_artifacts ea2 ON ea2.entry_id = c2.id JOIN blobs b2 ON b2.id = ea2.blob_id WHERE c2.parent_entry_id = e.id), 0) AS total_artifact_bytes, \
parent.entry_uid AS parent_entry_uid, \ parent.entry_uid AS parent_entry_uid, \
EXISTS(SELECT 1 FROM entry_artifacts fav WHERE fav.entry_id = e.id AND fav.artifact_role = 'favicon') AS has_favicon, \ EXISTS(SELECT 1 FROM entry_artifacts fav WHERE fav.entry_id = e.id AND fav.artifact_role = 'favicon') AS has_favicon, \
e.cached_bytes"; e.cached_bytes, \
(SELECT COUNT(*) FROM archived_entries child WHERE child.parent_entry_id = e.id) AS child_count, \
COALESCE(SUM(CASE WHEN ea.artifact_role != 'avatar' THEN b.byte_size ELSE 0 END), 0) + COALESCE((SELECT SUM(b2.byte_size) FROM archived_entries c2 JOIN entry_artifacts ea2 ON ea2.entry_id = c2.id JOIN blobs b2 ON b2.id = ea2.blob_id WHERE c2.parent_entry_id = e.id), 0) AS cacheable_bytes";
const ENTRY_FROM_JOINS: &str = "FROM archived_entries e \ const ENTRY_FROM_JOINS: &str = "FROM archived_entries e \
JOIN source_identities si ON si.id = e.source_identity_id \ JOIN source_identities si ON si.id = e.source_identity_id \
@ -663,6 +814,8 @@ pub fn search_entries(
parent_entry_uid: row.get(9)?, parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0, has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?, cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
}) })
})? })?
.collect::<rusqlite::Result<Vec<_>>>()?; .collect::<rusqlite::Result<Vec<_>>>()?;
@ -870,7 +1023,9 @@ pub fn entries_for_tag(
COALESCE(SUM(b.byte_size), 0) AS total_artifact_bytes, COALESCE(SUM(b.byte_size), 0) AS total_artifact_bytes,
parent.entry_uid AS parent_entry_uid, parent.entry_uid AS parent_entry_uid,
EXISTS(SELECT 1 FROM entry_artifacts fav WHERE fav.entry_id = e.id AND fav.artifact_role = 'favicon') AS has_favicon, EXISTS(SELECT 1 FROM entry_artifacts fav WHERE fav.entry_id = e.id AND fav.artifact_role = 'favicon') AS has_favicon,
e.cached_bytes e.cached_bytes,
(SELECT COUNT(*) FROM archived_entries child WHERE child.parent_entry_id = e.id) AS child_count,
COALESCE(SUM(CASE WHEN ea.artifact_role != 'avatar' THEN b.byte_size ELSE 0 END), 0) AS cacheable_bytes
FROM archived_entries e FROM archived_entries e
JOIN source_identities si ON si.id = e.source_identity_id JOIN source_identities si ON si.id = e.source_identity_id
LEFT JOIN entry_artifacts ea ON ea.entry_id = e.id LEFT JOIN entry_artifacts ea ON ea.entry_id = e.id
@ -896,6 +1051,8 @@ pub fn entries_for_tag(
parent_entry_uid: row.get(9)?, parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0, has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?, cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
}) })
})? })?
.collect::<rusqlite::Result<Vec<_>>>()?; .collect::<rusqlite::Result<Vec<_>>>()?;
@ -1618,4 +1775,263 @@ mod tests {
); );
assert_eq!(results[0].entry_uid, parent.entry_uid); assert_eq!(results[0].entry_uid, parent.entry_uid);
} }
// ── sync helper tests ────────────────────────────────────────────────────────
#[test]
fn find_container_entry_id_returns_none_when_absent() {
let (conn, _, _) = make_tag_test_db();
let result = find_container_entry_id_by_canonical_url(
&conn,
"https://www.youtube.com/playlist?list=PLnobody",
)
.unwrap();
assert!(result.is_none(), "should return None when no entry exists");
}
#[test]
fn find_container_entry_id_returns_root_entry() {
let (conn, user_id, run_id) = make_tag_test_db();
let playlist_url = "https://www.youtube.com/playlist?list=PLtest";
let container = make_entry_in_db(&conn, user_id, run_id, None, None, "My Playlist", playlist_url);
let result = find_container_entry_id_by_canonical_url(&conn, playlist_url)
.unwrap()
.expect("should find the container");
assert_eq!(result, container.id);
}
#[test]
fn find_container_entry_id_ignores_child_entries() {
let (conn, user_id, run_id) = make_tag_test_db();
let child_url = "https://www.youtube.com/watch?v=abc123";
// Create a parent container and a child whose canonical URL happens to be
// what we're querying — should NOT be returned since it has parent_entry_id set.
let parent = make_entry_in_db(&conn, user_id, run_id, None, None, "PL", "https://example.com/pl");
let _child = make_entry_in_db(&conn, user_id, run_id, Some(parent.id), Some(parent.id), "Vid", child_url);
let result = find_container_entry_id_by_canonical_url(&conn, child_url).unwrap();
assert!(result.is_none(), "child entry should not be returned as a container");
}
#[test]
fn get_archived_playlist_child_urls_empty_when_no_playlist() {
let (conn, _, _) = make_tag_test_db();
let urls = get_archived_playlist_child_urls(
&conn,
"https://www.youtube.com/playlist?list=PLnone",
)
.unwrap();
assert!(urls.is_empty());
}
#[test]
fn get_archived_playlist_child_urls_returns_children() {
let (conn, user_id, run_id) = make_tag_test_db();
let playlist_url = "https://www.youtube.com/playlist?list=PLchildren";
let container = make_entry_in_db(&conn, user_id, run_id, None, None, "Playlist", playlist_url);
let child1_url = "https://www.youtube.com/watch?v=vid1";
let child2_url = "https://www.youtube.com/watch?v=vid2";
make_entry_in_db(&conn, user_id, run_id, Some(container.id), Some(container.id), "Vid 1", child1_url);
make_entry_in_db(&conn, user_id, run_id, Some(container.id), Some(container.id), "Vid 2", child2_url);
let urls = get_archived_playlist_child_urls(&conn, playlist_url).unwrap();
assert_eq!(urls.len(), 2);
assert!(urls.contains(child1_url), "should contain vid1");
assert!(urls.contains(child2_url), "should contain vid2");
}
#[test]
fn get_archived_playlist_child_urls_excludes_other_playlists() {
let (conn, user_id, run_id) = make_tag_test_db();
let pl_a = "https://www.youtube.com/playlist?list=PLA";
let pl_b = "https://www.youtube.com/playlist?list=PLB";
let container_a = make_entry_in_db(&conn, user_id, run_id, None, None, "PL A", pl_a);
let container_b = make_entry_in_db(&conn, user_id, run_id, None, None, "PL B", pl_b);
let vid_a = "https://www.youtube.com/watch?v=forA";
let vid_b = "https://www.youtube.com/watch?v=forB";
make_entry_in_db(&conn, user_id, run_id, Some(container_a.id), Some(container_a.id), "A Vid", vid_a);
make_entry_in_db(&conn, user_id, run_id, Some(container_b.id), Some(container_b.id), "B Vid", vid_b);
let urls_a = get_archived_playlist_child_urls(&conn, pl_a).unwrap();
assert_eq!(urls_a.len(), 1);
assert!(urls_a.contains(vid_a));
assert!(!urls_a.contains(vid_b), "should not include children of other playlists");
}
#[test]
fn cached_bytes_excludes_avatar_blobs() {
let (conn, user_id, run_id) = make_tag_test_db();
// entry_a is older (created first, gets the lower id)
let entry_a = make_entry_in_db(
&conn,
user_id,
run_id,
None,
None,
"Tweet A",
"https://twitter.com/user/status/1",
);
// entry_b is newer (created second, higher id — tiebreak on id when archived_at is equal)
let entry_b = make_entry_in_db(
&conn,
user_id,
run_id,
None,
None,
"Tweet B",
"https://twitter.com/user/status/2",
);
// Shared avatar blob: 100 bytes
let avatar_blob_id = database::upsert_blob(
&conn,
&database::BlobRecord {
sha256: "avatar_sha256_test".to_string(),
byte_size: 100,
mime_type: Some("image/jpeg".to_string()),
extension: Some("jpg".to_string()),
raw_relpath: "raw/av/at/avatar.jpg".to_string(),
},
)
.unwrap();
// Shared media blob: 900 bytes
let media_blob_id = database::upsert_blob(
&conn,
&database::BlobRecord {
sha256: "media_sha256_test".to_string(),
byte_size: 900,
mime_type: Some("image/png".to_string()),
extension: Some("png".to_string()),
raw_relpath: "raw/me/di/media.png".to_string(),
},
)
.unwrap();
// Attach both artifacts to entry_a
database::add_entry_artifact(
&conn,
&database::NewArtifact {
entry_id: entry_a.id,
artifact_role: "avatar".to_string(),
storage_area: "raw".to_string(),
relpath: "raw/av/at/avatar.jpg".to_string(),
blob_id: Some(avatar_blob_id),
logical_path: None,
metadata_json: None,
},
)
.unwrap();
database::add_entry_artifact(
&conn,
&database::NewArtifact {
entry_id: entry_a.id,
artifact_role: "primary_media".to_string(),
storage_area: "raw".to_string(),
relpath: "raw/me/di/media.png".to_string(),
blob_id: Some(media_blob_id),
logical_path: None,
metadata_json: None,
},
)
.unwrap();
// Attach both artifacts to entry_b (same blobs — simulates shared avatar/media)
database::add_entry_artifact(
&conn,
&database::NewArtifact {
entry_id: entry_b.id,
artifact_role: "avatar".to_string(),
storage_area: "raw".to_string(),
relpath: "raw/av/at/avatar.jpg".to_string(),
blob_id: Some(avatar_blob_id),
logical_path: None,
metadata_json: None,
},
)
.unwrap();
database::add_entry_artifact(
&conn,
&database::NewArtifact {
entry_id: entry_b.id,
artifact_role: "primary_media".to_string(),
storage_area: "raw".to_string(),
relpath: "raw/me/di/media.png".to_string(),
blob_id: Some(media_blob_id),
logical_path: None,
metadata_json: None,
},
)
.unwrap();
// Recompute cached_bytes for entry_b.
// The query excludes avatar-role artifacts and counts only non-avatar blobs
// that are already held by an earlier entry. entry_a (lower id) owns the
// same media blob, so cached_bytes for entry_b should be 900, not 1000.
database::refresh_entry_cached_bytes(&conn, entry_b.id).unwrap();
// --- Assert raw DB value ---
let cached_bytes_db: i64 = conn
.query_row(
"SELECT cached_bytes FROM archived_entries WHERE id = ?1",
[entry_b.id],
|row| row.get(0),
)
.unwrap();
assert_eq!(
cached_bytes_db, 900,
"cached_bytes should be 900 (media only); avatar blob must be excluded"
);
// --- Assert list_root_entries summary for entry_b ---
let entries = list_root_entries(&conn, 12).unwrap(); // 12 = ADMIN bits
let summary_b = entries
.iter()
.find(|e| e.entry_uid == entry_b.entry_uid)
.expect("entry_b must appear in list_root_entries");
assert_eq!(
summary_b.cached_bytes, 900,
"summary cached_bytes must equal 900 (precomputed, avatar excluded)"
);
assert_eq!(
summary_b.cacheable_bytes, 900,
"cacheable_bytes (non-avatar total) must be 900 for entry_b"
);
assert_eq!(
summary_b.total_artifact_bytes, 1000,
"total_artifact_bytes must be 1000 (avatar 100 + media 900)"
);
}
#[test]
fn list_child_entries_inherits_parent_visibility() {
// Regression: non-admin users who can see a playlist container through a
// collection must also see its children. The child is auto-enrolled in the
// default collection with private visibility bits — it has no *visible*
// collection membership of its own, so it would be filtered out without
// the parent-collection OR arm in list_child_entries.
let (conn, user_id, run_id) = make_tag_test_db();
let container = make_entry_in_db(&conn, user_id, run_id, None, None,
"Playlist", "https://example.com/pl");
let child = make_entry_in_db(&conn, user_id, run_id,
Some(container.id), Some(container.id),
"Video 1", "https://example.com/pl/v1");
// Enroll the container (but NOT the child) in a USER-visible collection (bits=2).
let coll = database::create_collection(&conn, "My List", "my-list", 2).unwrap();
database::add_entry_to_collection(&conn, coll.id, container.id, 2).unwrap();
// USER caller (bits=2): child must be visible through parent's collection.
let children = list_child_entries(&conn, &container.entry_uid, 2).unwrap();
assert_eq!(children.len(), 1, "child must be visible via parent collection");
assert_eq!(children[0].entry_uid, child.entry_uid);
// GUEST caller (bits=1): parent collection has bits=2, so child must NOT be visible.
let guest_children = list_child_entries(&conn, &container.entry_uid, 1).unwrap();
assert!(guest_children.is_empty(), "guest must not see children of a USER-only collection");
}
} }

View file

@ -7,7 +7,7 @@ use std::{
fs, fs,
path::{Path, PathBuf}, path::{Path, PathBuf},
}; };
use crate::{archive::ArchivePaths, database, downloader, twitter::parse_tweet_id}; use crate::{archive::{self, ArchivePaths}, database, downloader, twitter::parse_tweet_id};
#[derive(Debug, PartialEq, Eq, Clone, Copy)] #[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub enum Source { pub enum Source {
@ -37,8 +37,11 @@ pub enum Source {
pub struct CaptureResult { pub struct CaptureResult {
pub run_uid: String, pub run_uid: String,
pub status: String, pub status: String,
/// Number of successfully completed child items (playlist/container captures only).
/// Zero for single-item captures. Used by the server to distinguish a fully-failed
/// playlist run from a partial success without needing a new DB status value.
pub completed_child_count: i64,
/// `true` when uBlock was requested but the extension path was not found. /// `true` when uBlock was requested but the extension path was not found.
/// The capture succeeded without ad-blocking; the UI should warn the user.
pub ublock_skipped: bool, pub ublock_skipped: bool,
/// `true` when cookie-consent extension was requested but the path was not found. /// `true` when cookie-consent extension was requested but the path was not found.
pub cookie_ext_skipped: bool, pub cookie_ext_skipped: bool,
@ -90,6 +93,17 @@ pub struct CaptureConfig {
/// Route WebPage captures through the Freedium mirror to bypass paywalls. /// Route WebPage captures through the Freedium mirror to bypass paywalls.
/// The original locator is still recorded in the DB; only the fetch URL changes. /// The original locator is still recorded in the DB; only the fetch URL changes.
pub via_freedium: bool, pub via_freedium: bool,
/// Per-item quality overrides and include-set for playlist captures.
/// Keys are yt-dlp video IDs (e.g. "dQw4w9WgXcQ"); values are quality strings
/// accepted by `quality_format` ("best", "audio", "1080p", etc.).
/// - Empty map: all playlist items are downloaded using the global `quality`.
/// - Non-empty map: ONLY items whose ID appears as a key are downloaded;
/// items absent from the map are skipped entirely. This is how the UI's
/// per-video delete button excludes specific videos from a capture.
pub per_item_quality: HashMap<String, String>,
/// When true, skip playlist items whose URL is already archived as a child
/// of any container entry with the same canonical playlist URL.
pub sync: bool,
} }
/// Resolves which cookies apply to `url` by evaluating all rules in ordinal order. /// Resolves which cookies apply to `url` by evaluating all rules in ordinal order.
@ -200,6 +214,12 @@ fn generate_entry_title(source: Source, meta: &PlatformMetadata) -> String {
Source::Other => "Archived Content".to_string(), Source::Other => "Archived Content".to_string(),
} }
} }
/// Returns true when `s` is a valid bare YouTube video ID:
/// exactly 11 characters from the set `[A-Za-z0-9_-]`.
fn is_youtube_video_id(s: &str) -> bool {
s.len() == 11 && s.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'_' || b == b'-')
}
fn expand_shorthand_to_url(path: &str, source: &Source) -> String { fn expand_shorthand_to_url(path: &str, source: &Source) -> String {
// YouTube shorthands: yt:video/ID, yt:playlist/ID, yt:@handle, yt:channel/ID, etc. // YouTube shorthands: yt:video/ID, yt:playlist/ID, yt:@handle, yt:channel/ID, etc.
@ -227,6 +247,10 @@ fn expand_shorthand_to_url(path: &str, source: &Source) -> String {
if let Some(handle) = after.strip_prefix("@") { if let Some(handle) = after.strip_prefix("@") {
return format!("https://www.youtube.com/@{handle}"); return format!("https://www.youtube.com/@{handle}");
} }
// bare yt:ID — validated 11-char video ID
if is_youtube_video_id(after) {
return format!("https://www.youtube.com/watch?v={after}");
}
} }
} }
@ -317,6 +341,11 @@ fn determine_source(path: &str) -> Source {
{ {
return Source::YouTubeChannel; return Source::YouTubeChannel;
} }
// bare yt:ID — exactly 11 chars [A-Za-z0-9_-], no slash/@ prefixes
if is_youtube_video_id(after_scheme) {
return Source::YouTubeVideo;
}
} }
// Shorthand scheme: ytm: // Shorthand scheme: ytm:
@ -536,6 +565,21 @@ pub fn locator_to_ytdlp_url(locator: &str) -> Option<String> {
} }
} }
/// Returns the canonical URL for playlist/channel locators that yt-dlp can
/// expand with `--flat-playlist`, or `None` for non-playlist sources.
/// Intentionally separate from `locator_to_ytdlp_url`, which excludes playlists.
pub fn locator_to_playlist_url(locator: &str) -> Option<String> {
let source = determine_source(locator);
match source {
Source::YouTubePlaylist
| Source::YouTubeChannel
| Source::YouTubeMusicPlaylist
| Source::SpotifyAlbum
| Source::SpotifyPlaylist => Some(expand_shorthand_to_url(locator, &source)),
_ => None,
}
}
fn hash_exists(hash: &str, file_extension: &str, store_path: &Path) -> Result<bool> { fn hash_exists(hash: &str, file_extension: &str, store_path: &Path) -> Result<bool> {
let path = store_path.join(raw_relative_path_from_hash(hash, file_extension)?); let path = store_path.join(raw_relative_path_from_hash(hash, file_extension)?);
Ok(path.exists()) Ok(path.exists())
@ -660,6 +704,8 @@ fn record_media_entry(
file_extension: &str, file_extension: &str,
byte_size: i64, byte_size: i64,
title: Option<String>, title: Option<String>,
parent_entry_id: Option<i64>,
root_entry_id: Option<i64>,
) -> Result<database::ArchivedEntry> { ) -> Result<database::ArchivedEntry> {
debug_assert!(run.run_uid.starts_with("run_")); debug_assert!(run.run_uid.starts_with("run_"));
debug_assert!(item.item_uid.starts_with("item_")); debug_assert!(item.item_uid.starts_with("item_"));
@ -686,8 +732,8 @@ fn record_media_entry(
&database::NewEntry { &database::NewEntry {
source_identity_id, source_identity_id,
archive_run_id: run.id, archive_run_id: run.id,
parent_entry_id: None, parent_entry_id,
root_entry_id: None, root_entry_id,
created_by_user_id: user_id, created_by_user_id: user_id,
owned_by_user_id: user_id, owned_by_user_id: user_id,
source_kind: source_kind.to_string(), source_kind: source_kind.to_string(),
@ -720,6 +766,59 @@ fn record_media_entry(
Ok(entry) Ok(entry)
} }
/// Creates a playlist/channel container entry with no blob and no primary-media artifact.
/// Calls `complete_archive_run_item` on the provided item.
fn record_container_entry(
conn: &rusqlite::Connection,
store_path: &Path,
user_id: i64,
run: &database::ArchiveRun,
item: &database::ArchiveRunItem,
requested_locator: &str,
canonical_locator: &str,
source: Source,
title: Option<String>,
playlist_id: &str,
uploader: Option<&str>,
) -> Result<database::ArchivedEntry> {
let (source_kind, entity_kind, representation_kind) = source_metadata(source);
let source_identity_id = database::upsert_source_identity(
conn,
source_kind,
entity_kind,
Some(playlist_id),
Some(canonical_locator),
canonical_locator,
)?;
let entry = database::create_archived_entry(
conn,
&database::NewEntry {
source_identity_id,
archive_run_id: run.id,
parent_entry_id: None,
root_entry_id: None,
created_by_user_id: user_id,
owned_by_user_id: user_id,
source_kind: source_kind.to_string(),
entity_kind: entity_kind.to_string(),
title,
visibility: "private".to_string(),
representation_kind: representation_kind.to_string(),
source_metadata_json: json!({
"requested_locator": requested_locator,
"canonical_locator": canonical_locator,
"playlist_id": playlist_id,
"uploader": uploader,
})
.to_string(),
display_metadata_json: None,
},
)?;
create_structured_root(store_path, &entry)?;
database::complete_archive_run_item(conn, item.id, entry.id)?;
Ok(entry)
}
/// Extracts PlatformMetadata from a tweet JSON string. /// Extracts PlatformMetadata from a tweet JSON string.
/// Returns Default on any parse failure. /// Returns Default on any parse failure.
fn tweet_metadata_from_json(json_str: &str) -> PlatformMetadata { fn tweet_metadata_from_json(json_str: &str) -> PlatformMetadata {
@ -957,26 +1056,274 @@ pub fn perform_capture(
)); ));
} }
// Sources: Spotify — not downloadable; Spotify audio is DRM-protected. // Sources: YouTube playlists, YouTube channels, YouTube Music playlists,
if matches!(source, Source::SpotifyTrack | Source::SpotifyAlbum | Source::SpotifyPlaylist) { // Spotify albums, Spotify playlists — probe via yt-dlp flat-playlist,
// create a root container entry, then download each track as a child entry.
// `run` and `item` are already created above; `item` acts as the container item.
if matches!(
source,
Source::YouTubePlaylist
| Source::YouTubeChannel
| Source::YouTubeMusicPlaylist
| Source::SpotifyAlbum
| Source::SpotifyPlaylist
) {
// `canonical_url` already holds the expanded URL (same value as `path` later).
let playlist_info = match downloader::ytdlp::fetch_playlist_info(&canonical_url, &cookies) {
Ok(info) => info,
Err(e) => {
return Err(fail_run( return Err(fail_run(
&conn, &conn,
&run, &run,
&item, &item,
"Spotify downloads are not supported: Spotify audio is DRM-protected and cannot \ &format!("Failed to fetch playlist info: {e:#}"),
be downloaded by yt-dlp. Archive the equivalent YouTube Music track instead.",
)); ));
} }
};
// Sources: YouTube Music Playlist — container expansion not yet implemented. let container_title = playlist_info
if source == Source::YouTubeMusicPlaylist { .title
.clone()
.or_else(|| playlist_info.uploader.clone().map(|u| format!("{u} (channel)")));
// Sync mode: reuse an existing container so we don't create a duplicate
// root entry on every sync run. Non-sync always creates a fresh container.
let (container_id, already_archived) = if config.sync {
match archive::find_container_entry_id_by_canonical_url(&conn, &canonical_url) {
Err(e) => {
return Err(fail_run( return Err(fail_run(
&conn, &conn, &run, &item,
&run, &format!("Failed to query existing container: {e:#}"),
&item,
"YouTube Music playlist archiving is not yet implemented.",
)); ));
} }
Ok(Some(existing_id)) => {
// Container already exists — mark the run item done pointing at it;
// don't call record_container_entry (that would create a duplicate).
if let Err(e) = database::complete_archive_run_item(&conn, item.id, existing_id) {
return Err(fail_run(
&conn, &run, &item,
&format!("Failed to complete run item for existing container: {e:#}"),
));
}
let archived = match archive::get_archived_playlist_child_urls(&conn, &canonical_url) {
Ok(set) => set,
Err(e) => return Err(fail_run(
&conn, &run, &item,
&format!("Failed to query archived playlist children: {e:#}"),
)),
};
(existing_id, archived)
}
Ok(None) => {
// First sync run for this playlist — create the container normally.
let e = match record_container_entry(
&conn, store_path, user_id, &run, &item, locator, &canonical_url,
source, container_title, &playlist_info.playlist_id,
playlist_info.uploader.as_deref(),
) {
Ok(e) => e,
Err(e) => return Err(fail_run(
&conn, &run, &item,
&format!("Failed to create container entry: {e:#}"),
)),
};
(e.id, std::collections::HashSet::new())
}
}
} else {
let e = match record_container_entry(
&conn, store_path, user_id, &run, &item, locator, &canonical_url,
source, container_title, &playlist_info.playlist_id,
playlist_info.uploader.as_deref(),
) {
Ok(e) => e,
Err(e) => return Err(fail_run(
&conn, &run, &item,
&format!("Failed to create container entry: {e:#}"),
)),
};
(e.id, std::collections::HashSet::new())
};
let is_audio = matches!(
source,
Source::YouTubeMusicPlaylist | Source::SpotifyAlbum | Source::SpotifyPlaylist
);
let child_quality: Option<&str> = if is_audio { Some("audio") } else { quality };
let child_entity_kind = if is_audio { "music" } else { "video" };
for (ordinal, playlist_item) in playlist_info.items.iter().enumerate() {
// Sync: skip items already archived — no run item created, so
// refresh_run_counters sees only newly-attempted items.
if config.sync && already_archived.contains(&playlist_item.url) {
continue;
}
// Per-item exclusion: when per_item_quality is non-empty (frontend
// probe ran and the user confirmed the item list), only download
// items whose ID appears in the map. IDs absent from a non-empty
// map were removed by the user via the delete button before submit.
if !config.per_item_quality.is_empty()
&& !config.per_item_quality.contains_key(&playlist_item.id)
{
continue;
}
let child_timestamp = format!(
"{}-{}",
Local::now().format("%Y-%m-%dT%H-%M-%S%.3f"),
Uuid::new_v4().simple(),
);
let child_item = match database::create_archive_run_item(
&conn,
run.id,
Some(item.id),
ordinal as i64,
&playlist_item.url,
Some(&playlist_item.url),
source_kind,
child_entity_kind,
) {
Ok(i) => i,
Err(e) => {
eprintln!("warn: playlist item {} create_run_item failed: {e:#}", playlist_item.url);
continue;
}
};
// Fetch metadata for the child title (best-effort).
let child_meta_json = downloader::ytdlp::fetch_metadata(&playlist_item.url, &cookies);
let child_title: Option<String> = match &child_meta_json {
Some(json) => {
let meta = downloader::metadata::extract_from_ytdlp_json(json);
Some(generate_entry_title(
match source {
Source::SpotifyAlbum | Source::SpotifyPlaylist => Source::SpotifyTrack,
_ if is_audio => Source::YouTubeMusicTrack,
_ => Source::YouTubeVideo,
},
&meta,
))
}
None => playlist_item.title.clone(),
};
// Per-item quality override keyed by yt-dlp video ID; fall back to
// playlist-level quality. YTM playlists are always audio-only —
// per_item_quality overrides are ignored so the audio-only guarantee
// cannot be bypassed by a caller sending e.g. "best".
let effective_child_quality: Option<&str> = if is_audio {
Some("audio")
} else {
config
.per_item_quality
.get(&playlist_item.id)
.map(String::as_str)
.or(child_quality)
};
// Download the media.
match downloader::ytdlp::download(
playlist_item.url.clone(),
store_path,
&child_timestamp,
effective_child_quality,
&cookies,
) {
Ok((hash, file_extension)) => {
let temp_file = store_path
.join("temp")
.join(&child_timestamp)
.join(format!("{child_timestamp}{file_extension}"));
let byte_size = match fs::metadata(&temp_file) {
Ok(m) => m.len() as i64,
Err(e) => {
eprintln!("warn: stat child temp file: {e:#}");
let _ = database::fail_archive_run_item(
&conn, child_item.id,
&format!("failed to stat downloaded file: {e:#}"),
);
continue;
}
};
if !hash_exists(&hash, &file_extension, store_path).unwrap_or(false) {
if let Err(e) = move_temp_to_raw(&temp_file, &hash, store_path) {
eprintln!("warn: move_temp_to_raw child: {e:#}");
let _ = database::fail_archive_run_item(
&conn, child_item.id,
&format!("failed to move downloaded file: {e:#}"),
);
continue;
}
}
let _ = fs::remove_dir_all(store_path.join("temp").join(&child_timestamp));
let child_source = match source {
Source::SpotifyAlbum | Source::SpotifyPlaylist => Source::SpotifyTrack,
_ if is_audio => Source::YouTubeMusicTrack,
_ => Source::YouTubeVideo,
};
match record_media_entry(
&conn,
store_path,
user_id,
&run,
&child_item,
&playlist_item.url,
&playlist_item.url,
child_source,
&hash,
&file_extension,
byte_size,
child_title,
Some(container_id),
Some(container_id),
) {
Ok(child_entry) => {
let _ = database::refresh_entry_cached_bytes(&conn, child_entry.id);
}
Err(e) => {
eprintln!("warn: record child entry: {e:#}");
let _ = database::fail_archive_run_item(
&conn, child_item.id,
&format!("failed to record entry: {e:#}"),
);
}
}
}
Err(e) => {
let _ = fs::remove_dir_all(store_path.join("temp").join(&child_timestamp));
eprintln!("warn: yt-dlp child download failed for {}: {e:#}", playlist_item.url);
let _ = database::fail_archive_run_item(
&conn, child_item.id,
&format!("yt-dlp download failed: {e:#}"),
);
}
}
}
database::refresh_entry_cached_bytes(&conn, container_id)?;
database::finish_archive_run(&conn, run.id)?;
let run_status: String = conn
.query_row(
"SELECT status FROM archive_runs WHERE id = ?1",
[run.id],
|row| row.get(0),
)
.unwrap_or_else(|_| "completed".to_string());
let completed_child_count: i64 = database::get_run_completed_child_count(&conn, run.id)
.unwrap_or(0);
return Ok(CaptureResult {
run_uid: run.run_uid.clone(),
status: run_status,
completed_child_count,
ublock_skipped: false,
cookie_ext_skipped: false,
});
}
// Source: generic HTTP/S file URL // Source: generic HTTP/S file URL
if source == Source::Url { if source == Source::Url {
@ -1018,12 +1365,15 @@ pub fn perform_capture(
&file_extension, &file_extension,
byte_size, byte_size,
title_hint, title_hint,
None,
None,
)?; )?;
database::refresh_entry_cached_bytes(&conn, entry.id)?; database::refresh_entry_cached_bytes(&conn, entry.id)?;
database::finish_archive_run(&conn, run.id)?; database::finish_archive_run(&conn, run.id)?;
return Ok(CaptureResult { return Ok(CaptureResult {
run_uid: run.run_uid.clone(), run_uid: run.run_uid.clone(),
status: "completed".to_string(), status: "completed".to_string(),
completed_child_count: 0,
ublock_skipped: false, ublock_skipped: false,
cookie_ext_skipped: false, cookie_ext_skipped: false,
}); });
@ -1145,6 +1495,8 @@ pub fn perform_capture(
&file_extension, &file_extension,
byte_size, byte_size,
entry_title, entry_title,
None,
None,
)?; )?;
// 5. Add favicon artifact if we captured one. // 5. Add favicon artifact if we captured one.
@ -1195,6 +1547,7 @@ pub fn perform_capture(
return Ok(CaptureResult { return Ok(CaptureResult {
run_uid: run.run_uid.clone(), run_uid: run.run_uid.clone(),
status: "completed".to_string(), status: "completed".to_string(),
completed_child_count: 0,
ublock_skipped: result.ublock_skipped, ublock_skipped: result.ublock_skipped,
cookie_ext_skipped: result.cookie_ext_skipped, cookie_ext_skipped: result.cookie_ext_skipped,
}); });
@ -1252,6 +1605,7 @@ pub fn perform_capture(
return Ok(CaptureResult { return Ok(CaptureResult {
run_uid: run.run_uid.clone(), run_uid: run.run_uid.clone(),
status: "completed".to_string(), status: "completed".to_string(),
completed_child_count: 0,
ublock_skipped: false, ublock_skipped: false,
cookie_ext_skipped: false, cookie_ext_skipped: false,
}); });
@ -1275,6 +1629,7 @@ pub fn perform_capture(
let ytdlp_metadata_json: Option<String> = match source { let ytdlp_metadata_json: Option<String> = match source {
Source::YouTubeVideo Source::YouTubeVideo
| Source::YouTubeMusicTrack | Source::YouTubeMusicTrack
| Source::SpotifyTrack
| Source::X | Source::X
| Source::Instagram | Source::Instagram
| Source::Facebook | Source::Facebook
@ -1315,7 +1670,7 @@ pub fn perform_capture(
} }
} }
} }
Source::YouTubeMusicTrack => { Source::YouTubeMusicTrack | Source::SpotifyTrack => {
// Music tracks are always audio-only regardless of the caller's quality hint. // Music tracks are always audio-only regardless of the caller's quality hint.
match downloader::ytdlp::download(path.clone(), store_path, &timestamp, Some("audio"), &cookies) { match downloader::ytdlp::download(path.clone(), store_path, &timestamp, Some("audio"), &cookies) {
Ok(result) => result, Ok(result) => result,
@ -1342,14 +1697,7 @@ pub fn perform_capture(
} }
} }
} }
Source::YouTubePlaylist | Source::YouTubeChannel => { Source::YouTubePlaylist | Source::YouTubeChannel => unreachable!(),
return Err(fail_run(
&conn,
&run,
&item,
"Playlist and channel container expansion are not yet implemented.",
));
}
_ => unreachable!(), _ => unreachable!(),
}; };
let temp_file = store_path let temp_file = store_path
@ -1402,6 +1750,8 @@ pub fn perform_capture(
&file_extension, &file_extension,
byte_size, byte_size,
entry_title, entry_title,
None,
None,
)?; )?;
database::refresh_entry_cached_bytes(&conn, media_entry.id)?; database::refresh_entry_cached_bytes(&conn, media_entry.id)?;
database::finish_archive_run(&conn, run.id)?; database::finish_archive_run(&conn, run.id)?;
@ -1409,6 +1759,7 @@ pub fn perform_capture(
Ok(CaptureResult { Ok(CaptureResult {
run_uid: run.run_uid.clone(), run_uid: run.run_uid.clone(),
status: "completed".to_string(), status: "completed".to_string(),
completed_child_count: 0,
ublock_skipped: false, ublock_skipped: false,
cookie_ext_skipped: false, cookie_ext_skipped: false,
}) })
@ -1790,6 +2141,15 @@ mod tests {
url: "youtube:@CoreDumpped", url: "youtube:@CoreDumpped",
expected: Source::YouTubeChannel, expected: Source::YouTubeChannel,
}, },
// Bare video ID — exactly 11 chars [A-Za-z0-9_-]
TestCase { url: "yt:dQw4w9WgXcQ", expected: Source::YouTubeVideo },
TestCase { url: "youtube:dQw4w9WgXcQ", expected: Source::YouTubeVideo },
TestCase { url: "yt:a_b-c_d-e_4", expected: Source::YouTubeVideo },
// Non-ID: wrong length (9 chars) → Other
TestCase { url: "yt:not-video", expected: Source::Other },
// Reserved prefixes still route correctly when segment looks like an ID
TestCase { url: "yt:playlist/dQw4w9WgXcQ", expected: Source::YouTubePlaylist },
TestCase { url: "yt:@dQw4w9WgXcQ", expected: Source::YouTubeChannel },
]; ];
for case in &shorthand_cases { for case in &shorthand_cases {
@ -1802,6 +2162,18 @@ mod tests {
} }
} }
#[test]
fn test_is_youtube_video_id() {
assert!(is_youtube_video_id("dQw4w9WgXcQ"), "canonical ID");
assert!(is_youtube_video_id("a_b-c_d-e_4"), "11-char ID with _ and -");
assert!(!is_youtube_video_id(""), "empty");
assert!(!is_youtube_video_id("short"), "too short");
assert!(!is_youtube_video_id("toolong12345"), "too long (12 chars)");
assert!(!is_youtube_video_id("dQw4w9WgXc!"), "invalid char (! at pos 11)");
assert!(!is_youtube_video_id("not-video"), "9 chars — too short");
assert!(!is_youtube_video_id("dQw4w9WgXC!!"), "12 chars + bad char");
}
#[test] #[test]
fn test_youtube_music_sources() { fn test_youtube_music_sources() {
// --- determine_source --- // --- determine_source ---
@ -2326,4 +2698,41 @@ mod tests {
); );
} }
#[test]
fn locator_to_playlist_url_accepts_playlist_shorthands() {
// yt: playlist shorthand
assert_eq!(
locator_to_playlist_url("yt:playlist/PLtest123"),
Some("https://www.youtube.com/playlist?list=PLtest123".to_string()),
);
// YouTube Music playlist shorthand
assert_eq!(
locator_to_playlist_url("ytm:playlist/PLmus456"),
Some("https://music.youtube.com/playlist?list=PLmus456".to_string()),
);
// Full YT playlist URL passes through (expand_shorthand_to_url is identity for full URLs)
let full = "https://www.youtube.com/playlist?list=PLabc";
assert_eq!(locator_to_playlist_url(full), Some(full.to_string()));
}
#[test]
fn locator_to_playlist_url_accepts_channel_shorthands() {
let url = locator_to_playlist_url("yt:@MyChan");
assert!(url.is_some(), "channel @ shorthand should return Some");
let u = url.unwrap();
assert!(u.contains("youtube.com"), "should be a youtube URL: {u}");
}
#[test]
fn locator_to_playlist_url_rejects_non_playlist_sources() {
// Single video
assert_eq!(locator_to_playlist_url("yt:dQw4w9WgXcQ"), None);
// Tweet
assert_eq!(locator_to_playlist_url("tweet:1234567890"), None);
// YTM track
assert_eq!(locator_to_playlist_url("ytm:MntbN1DdEP0"), None);
// Plain web URL
assert_eq!(locator_to_playlist_url("https://example.com/page"), None);
}
} }

View file

@ -390,6 +390,7 @@ pub fn initialize_schema(conn: &Connection) -> Result<()> {
JOIN blobs b ON b.id = ea.blob_id JOIN blobs b ON b.id = ea.blob_id
WHERE ea.entry_id = archived_entries.id WHERE ea.entry_id = archived_entries.id
AND ea.blob_id IS NOT NULL AND ea.blob_id IS NOT NULL
AND ea.artifact_role != 'avatar'
AND EXISTS ( AND EXISTS (
SELECT 1 SELECT 1
FROM entry_artifacts ea2 FROM entry_artifacts ea2
@ -401,6 +402,33 @@ pub fn initialize_schema(conn: &Connection) -> Result<()> {
) )
);", );",
)?; )?;
} else {
// Re-migration: strip avatar blobs from cached_bytes on entries that
// already had the column populated before this filter was introduced.
// Scoped to entries with avatar artifacts only; fast and idempotent.
conn.execute_batch(
"UPDATE archived_entries
SET cached_bytes = (
SELECT COALESCE(SUM(b.byte_size), 0)
FROM entry_artifacts ea
JOIN blobs b ON b.id = ea.blob_id
WHERE ea.entry_id = archived_entries.id
AND ea.blob_id IS NOT NULL
AND ea.artifact_role != 'avatar'
AND EXISTS (
SELECT 1
FROM entry_artifacts ea2
JOIN archived_entries e2 ON e2.id = ea2.entry_id
WHERE ea2.blob_id = ea.blob_id
AND (e2.archived_at < archived_entries.archived_at
OR (e2.archived_at = archived_entries.archived_at
AND e2.id < archived_entries.id))
)
)
WHERE id IN (
SELECT DISTINCT entry_id FROM entry_artifacts WHERE artifact_role = 'avatar'
);",
)?;
} }
// Migration: add notes_json column to existing capture_jobs tables. // Migration: add notes_json column to existing capture_jobs tables.
@ -1409,11 +1437,7 @@ pub fn finish_archive_run(conn: &Connection, run_id: i64) -> Result<()> {
[run_id], [run_id],
|row| row.get(0), |row| row.get(0),
)?; )?;
let status = if failed_count > 0 { let status = if failed_count > 0 { "failed" } else { "completed" };
"failed"
} else {
"completed"
};
conn.execute( conn.execute(
"UPDATE archive_runs SET status = ?1, finished_at = ?2 WHERE id = ?3", "UPDATE archive_runs SET status = ?1, finished_at = ?2 WHERE id = ?3",
params![status, now_timestamp(), run_id], params![status, now_timestamp(), run_id],
@ -1421,6 +1445,20 @@ pub fn finish_archive_run(conn: &Connection, run_id: i64) -> Result<()> {
Ok(()) Ok(())
} }
/// Returns the number of completed **child** archive_run_items for a run.
///
/// Excludes the container item (parent_item_id IS NULL) so that a playlist
/// where every video fails still returns 0 even though the container item
/// itself was completed before child downloads started.
pub fn get_run_completed_child_count(conn: &Connection, run_id: i64) -> Result<i64> {
Ok(conn.query_row(
"SELECT COUNT(*) FROM archive_run_items
WHERE run_id = ?1 AND status = 'completed' AND parent_item_id IS NOT NULL",
[run_id],
|row| row.get(0),
)?)
}
pub fn fail_archive_run(conn: &Connection, run_id: i64, error_summary: &str) -> Result<()> { pub fn fail_archive_run(conn: &Connection, run_id: i64, error_summary: &str) -> Result<()> {
refresh_run_counters(conn, run_id)?; refresh_run_counters(conn, run_id)?;
conn.execute( conn.execute(
@ -1482,6 +1520,7 @@ pub fn refresh_entry_cached_bytes(conn: &Connection, entry_id: i64) -> Result<()
JOIN archived_entries e ON e.id = ea.entry_id JOIN archived_entries e ON e.id = ea.entry_id
WHERE ea.entry_id = ?1 WHERE ea.entry_id = ?1
AND ea.blob_id IS NOT NULL AND ea.blob_id IS NOT NULL
AND ea.artifact_role != 'avatar'
AND EXISTS ( AND EXISTS (
SELECT 1 SELECT 1
FROM entry_artifacts ea2 FROM entry_artifacts ea2
@ -1518,6 +1557,7 @@ pub fn cascade_cached_bytes_after_delete(conn: &Connection, entry_id: i64) -> Re
JOIN blobs b ON b.id = ea.blob_id JOIN blobs b ON b.id = ea.blob_id
WHERE ea.entry_id = archived_entries.id WHERE ea.entry_id = archived_entries.id
AND ea.blob_id IS NOT NULL AND ea.blob_id IS NOT NULL
AND ea.artifact_role != 'avatar'
AND EXISTS ( AND EXISTS (
SELECT 1 SELECT 1
FROM entry_artifacts ea3 FROM entry_artifacts ea3
@ -1570,6 +1610,7 @@ fn cascade_cached_bytes_after_subtree_delete(conn: &Connection, subtree_ids: &[i
JOIN blobs b ON b.id = ea.blob_id JOIN blobs b ON b.id = ea.blob_id
WHERE ea.entry_id = archived_entries.id WHERE ea.entry_id = archived_entries.id
AND ea.blob_id IS NOT NULL AND ea.blob_id IS NOT NULL
AND ea.artifact_role != 'avatar'
AND EXISTS ( AND EXISTS (
SELECT 1 SELECT 1
FROM entry_artifacts ea3 FROM entry_artifacts ea3
@ -1627,9 +1668,14 @@ pub fn delete_entry(conn: &Connection, entry_uid: &str) -> Result<bool> {
None => return Ok(false), None => return Ok(false),
}; };
// Collect the full subtree while rows still exist. // Collect the full subtree (entry itself + any descendants) while rows still exist.
// Must include the entry itself: for a child entry root_entry_id = ?1 returns nothing
// (no grandchildren), so without `id = ?1` the set would be empty and
// cascade_cached_bytes_after_subtree_delete would not recalculate shared-blob totals.
let subtree_ids: Vec<i64> = { let subtree_ids: Vec<i64> = {
let mut stmt = conn.prepare("SELECT id FROM archived_entries WHERE root_entry_id = ?1")?; let mut stmt = conn.prepare(
"SELECT id FROM archived_entries WHERE id = ?1 OR root_entry_id = ?1",
)?;
stmt.query_map([entry_id], |row| row.get(0))? stmt.query_map([entry_id], |row| row.get(0))?
.collect::<rusqlite::Result<_>>()? .collect::<rusqlite::Result<_>>()?
}; };
@ -1638,10 +1684,13 @@ pub fn delete_entry(conn: &Connection, entry_uid: &str) -> Result<bool> {
// shared blobs with any subtree member, excluding every subtree ID simultaneously. // shared blobs with any subtree member, excluding every subtree ID simultaneously.
cascade_cached_bytes_after_subtree_delete(conn, &subtree_ids)?; cascade_cached_bytes_after_subtree_delete(conn, &subtree_ids)?;
// Null the FK that has no ON DELETE action (covers root and all descendants). // Null the FK that has no ON DELETE action. Must cover:
// - The entry itself (child entry: root_entry_id = playlist root, not self)
// - All descendants (root entry: children have root_entry_id = entry_id)
conn.execute( conn.execute(
"UPDATE archive_run_items SET produced_entry_id = NULL "UPDATE archive_run_items SET produced_entry_id = NULL
WHERE produced_entry_id IN ( WHERE produced_entry_id = ?1
OR produced_entry_id IN (
SELECT id FROM archived_entries WHERE root_entry_id = ?1 SELECT id FROM archived_entries WHERE root_entry_id = ?1
)", )",
[entry_id], [entry_id],
@ -3896,4 +3945,38 @@ mod tests {
"file must be protected because artifact.relpath references it directly" "file must be protected because artifact.relpath references it directly"
); );
} }
#[test]
fn completed_child_count_excludes_container() {
// Regression: get_run_completed_child_count must not count the root container
// item. Scenario: complete both the container item and one child item; total
// DB completed_count == 2, but completed_child_count must == 1.
let c = conn();
let user_id = ensure_default_user(&c).unwrap();
let run = create_archive_run(&c, user_id, 2).unwrap();
// Root item (parent_item_id IS NULL) — mirrors what record_container_entry does.
let root_item = create_archive_run_item(
&c, run.id, None, 0, "https://example.com/pl", None, "youtube", "playlist",
).unwrap();
// Child item (parent_item_id IS NOT NULL).
let child_item = create_archive_run_item(
&c, run.id, Some(root_item.id), 1, "https://example.com/v1", None, "youtube", "video",
).unwrap();
// Complete both — marks archive_runs.completed_count = 2.
c.execute(
"UPDATE archive_run_items SET status = 'completed' WHERE id IN (?1, ?2)",
rusqlite::params![root_item.id, child_item.id],
).unwrap();
refresh_run_counters(&c, run.id).unwrap();
let total: i64 = c.query_row(
"SELECT completed_count FROM archive_runs WHERE id = ?1", [run.id], |r| r.get(0),
).unwrap();
assert_eq!(total, 2, "both items completed: DB counter must be 2");
let child_count = get_run_completed_child_count(&c, run.id).unwrap();
assert_eq!(child_count, 1, "only the child item must be counted");
}
} }

View file

@ -6,10 +6,50 @@ use std::{
process::Command, process::Command,
}; };
use uuid::Uuid; use uuid::Uuid;
use serde_json;
use crate::downloader::cookies::{domain_from_url, write_netscape_cookie_file}; use crate::downloader::cookies::{domain_from_url, write_netscape_cookie_file};
use crate::hash::hash_file; use crate::hash::hash_file;
/// A single item in a flat playlist listing from `fetch_playlist_info`.
#[derive(Debug)]
pub struct PlaylistItem {
pub id: String,
pub url: String,
pub title: Option<String>,
pub uploader: Option<String>,
}
/// Container metadata returned by `fetch_playlist_info`.
#[derive(Debug)]
pub struct PlaylistInfo {
pub playlist_id: String,
pub title: Option<String>,
pub uploader: Option<String>,
pub items: Vec<PlaylistItem>,
}
/// Per-item quality data returned by `probe_playlist_qualities`.
#[derive(Debug, serde::Serialize)]
pub struct PlaylistItemProbe {
pub id: String,
pub url: String,
pub title: Option<String>,
/// Available video heights as strings (e.g. "1080p"), sorted highest-first.
/// Empty vec means audio-only (no video track).
pub qualities: Vec<String>,
pub has_audio: bool,
}
/// Full playlist probe result with per-item quality data.
#[derive(Debug, serde::Serialize)]
pub struct PlaylistProbeResult {
pub playlist_id: String,
pub title: Option<String>,
pub uploader: Option<String>,
pub items: Vec<PlaylistItemProbe>,
}
/// Returns the yt-dlp `-f` format selector for `quality`. /// Returns the yt-dlp `-f` format selector for `quality`.
/// ///
/// - `"audio"` → prefers native Opus/WebM (most efficient), then native /// - `"audio"` → prefers native Opus/WebM (most efficient), then native
@ -74,6 +114,23 @@ pub fn available_video_heights(json: &str) -> Vec<u32> {
heights heights
} }
/// Extracts distinct video heights from a serde_json entry Value's `formats` array,
/// sorted highest-first. Audio-only formats (vcodec == "none") are excluded.
fn available_video_heights_from_value(entry: &serde_json::Value) -> Vec<u32> {
let Some(fmts) = entry.get("formats").and_then(|v| v.as_array()) else {
return vec![];
};
let mut heights: Vec<u32> = fmts
.iter()
.filter(|f| f.get("vcodec").and_then(|c| c.as_str()).unwrap_or("none") != "none")
.filter_map(|f| f.get("height").and_then(|h| h.as_u64()).map(|h| h as u32))
.filter(|&h| h > 0)
.collect();
heights.sort_unstable_by(|a, b| b.cmp(a));
heights.dedup();
heights
}
/// Returns true when the yt-dlp `--dump-json` response contains at least one /// Returns true when the yt-dlp `--dump-json` response contains at least one
/// format with a real audio codec (i.e. `acodec != "none"`). /// format with a real audio codec (i.e. `acodec != "none"`).
pub fn has_audio_track(json: &str) -> bool { pub fn has_audio_track(json: &str) -> bool {
@ -241,6 +298,207 @@ pub fn fetch_metadata(path: &str, cookies: &HashMap<String, String>) -> Option<S
if json.trim().is_empty() { None } else { Some(json) } if json.trim().is_empty() { None } else { Some(json) }
} }
/// Resolves an absolute item URL from a flat-playlist entry JSON object.
///
/// Priority:
/// 1. `webpage_url` — yt-dlp makes this absolute when present.
/// 2. `url` when it is already an absolute HTTP(S) URL.
/// 3. Platform-specific fallback constructed from `id` + `container_url`:
/// - YouTube Music → `https://music.youtube.com/watch?v={id}`
/// - YouTube → `https://www.youtube.com/watch?v={id}`
/// - Spotify → `https://open.spotify.com/track/{id}`
/// - Other → `None` (caller should skip the item and warn).
fn normalize_item_url(
entry: &serde_json::Value,
id: &str,
container_url: &str,
) -> Option<String> {
let is_abs = |s: &str| s.starts_with("http://") || s.starts_with("https://");
if let Some(u) = entry.get("webpage_url").and_then(|v| v.as_str()).filter(|s| is_abs(s)) {
return Some(u.to_owned());
}
if let Some(u) = entry.get("url").and_then(|v| v.as_str()).filter(|s| is_abs(s)) {
return Some(u.to_owned());
}
// Bare-ID fallback keyed on the container's platform.
if container_url.contains("music.youtube.com") {
Some(format!("https://music.youtube.com/watch?v={id}"))
} else if container_url.contains("youtube.com") || container_url.contains("youtu.be") {
Some(format!("https://www.youtube.com/watch?v={id}"))
} else if container_url.contains("open.spotify.com") {
Some(format!("https://open.spotify.com/track/{id}"))
} else {
eprintln!("warn: skipping playlist item {id:?} — no absolute URL from yt-dlp");
None
}
}
/// Runs `yt-dlp -J --flat-playlist <url>` and parses the single-JSON result.
///
/// `-J` / `--dump-single-json` returns one JSON object for the whole
/// container with reliable top-level `title` / `uploader` fields plus an
/// `entries` array of shallow per-item objects.
///
/// Returns an error if yt-dlp fails, the output is not valid JSON, or
/// the root `_type` is not `"playlist"`.
pub fn fetch_playlist_info(url: &str, cookies: &HashMap<String, String>) -> Result<PlaylistInfo> {
let ytdlp = std::env::var("ARCHIVR_YT_DLP").unwrap_or_else(|_| "yt-dlp".to_string());
let cookie_file: Option<PathBuf> = if !cookies.is_empty() {
let domain = domain_from_url(url);
let p = std::env::temp_dir()
.join(format!("archivr-cookies-{}.txt", Uuid::new_v4().simple()));
write_netscape_cookie_file(cookies, &domain, &p)
.context("failed to write yt-dlp cookie file")?;
Some(p)
} else {
None
};
let mut cmd = std::process::Command::new(&ytdlp);
cmd.arg("-J").arg("--flat-playlist");
if let Some(cf) = &cookie_file {
cmd.arg("--cookies").arg(cf);
}
cmd.arg(url);
let out = cmd.output();
if let Some(cf) = &cookie_file {
let _ = std::fs::remove_file(cf);
}
let out = out.with_context(|| format!("failed to spawn {ytdlp}"))?;
if !out.status.success() {
let stderr = String::from_utf8_lossy(&out.stderr);
bail!("yt-dlp -J --flat-playlist failed for {url}: {stderr}");
}
let json: serde_json::Value = serde_json::from_slice(&out.stdout)
.context("yt-dlp -J output is not valid JSON")?;
let ty = json.get("_type").and_then(|v| v.as_str()).unwrap_or("");
if ty != "playlist" {
bail!("yt-dlp output _type is {ty:?}, expected \"playlist\"");
}
let playlist_id = json
.get("id")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
let title = json.get("title").and_then(|v| v.as_str()).map(str::to_owned);
let uploader = json.get("uploader").and_then(|v| v.as_str()).map(str::to_owned);
let raw_entries = json
.get("entries")
.and_then(|v| v.as_array())
.map(|a| a.as_slice())
.unwrap_or(&[]);
let mut items = Vec::with_capacity(raw_entries.len());
for entry in raw_entries {
if entry.is_null() {
continue; // unavailable/private item in flat listing
}
let id = match entry.get("id").and_then(|v| v.as_str()) {
Some(s) => s.to_owned(),
None => continue,
};
let item_url = match normalize_item_url(entry, &id, url) {
Some(u) => u,
None => continue,
};
let item_title = entry.get("title").and_then(|v| v.as_str()).map(str::to_owned);
let item_uploader = entry.get("uploader").and_then(|v| v.as_str()).map(str::to_owned);
items.push(PlaylistItem { id, url: item_url, title: item_title, uploader: item_uploader });
}
Ok(PlaylistInfo { playlist_id, title, uploader, items })
}
/// Runs `yt-dlp -J <url>` (full metadata, NOT --flat-playlist) and returns
/// per-item quality data for every entry in the playlist.
///
/// This makes one yt-dlp subprocess call that fetches full format data for
/// all videos — expensive for large playlists but gives accurate per-video
/// quality lists. Intended for pre-capture quality selection only.
pub fn probe_playlist_qualities(
url: &str,
cookies: &HashMap<String, String>,
) -> Result<PlaylistProbeResult> {
let ytdlp = std::env::var("ARCHIVR_YT_DLP").unwrap_or_else(|_| "yt-dlp".to_string());
let cookie_file: Option<PathBuf> = if !cookies.is_empty() {
let domain = domain_from_url(url);
let p = std::env::temp_dir()
.join(format!("archivr-cookies-{}.txt", Uuid::new_v4().simple()));
write_netscape_cookie_file(cookies, &domain, &p)
.context("failed to write yt-dlp cookie file")?;
Some(p)
} else {
None
};
let mut cmd = std::process::Command::new(&ytdlp);
cmd.arg("-J"); // full metadata — NOT --flat-playlist
if let Some(cf) = &cookie_file {
cmd.arg("--cookies").arg(cf);
}
cmd.arg(url);
let out = cmd.output();
if let Some(cf) = &cookie_file {
let _ = std::fs::remove_file(cf);
}
let out = out.with_context(|| format!("failed to spawn {ytdlp}"))?;
if !out.status.success() {
let stderr = String::from_utf8_lossy(&out.stderr);
bail!("yt-dlp -J failed for {url}: {stderr}");
}
let json: serde_json::Value = serde_json::from_slice(&out.stdout)
.context("yt-dlp -J output is not valid JSON")?;
let ty = json.get("_type").and_then(|v| v.as_str()).unwrap_or("");
if ty != "playlist" {
bail!("yt-dlp output _type is {ty:?}, expected \"playlist\"");
}
let playlist_id = json.get("id").and_then(|v| v.as_str()).unwrap_or("").to_string();
let title = json.get("title").and_then(|v| v.as_str()).map(str::to_owned);
let uploader = json.get("uploader").and_then(|v| v.as_str()).map(str::to_owned);
let raw_entries = json
.get("entries")
.and_then(|v| v.as_array())
.map(|a| a.as_slice())
.unwrap_or(&[]);
let mut items = Vec::with_capacity(raw_entries.len());
for entry in raw_entries {
if entry.is_null() { continue; }
let id = match entry.get("id").and_then(|v| v.as_str()) {
Some(s) => s.to_owned(),
None => continue,
};
let item_url = match normalize_item_url(entry, &id, url) {
Some(u) => u,
None => continue,
};
let item_title = entry.get("title").and_then(|v| v.as_str()).map(str::to_owned);
let heights = available_video_heights_from_value(entry);
let qualities: Vec<String> = heights.iter().map(|h| format!("{h}p")).collect();
let has_audio = entry
.get("formats").and_then(|v| v.as_array())
.map(|fmts| fmts.iter().any(|f| {
f.get("acodec").and_then(|c| c.as_str()).unwrap_or("none") != "none"
}))
.unwrap_or(false);
items.push(PlaylistItemProbe { id, url: item_url, title: item_title, qualities, has_audio });
}
Ok(PlaylistProbeResult { playlist_id, title, uploader, items })
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::{available_video_heights, has_audio_track, quality_format}; use super::{available_video_heights, has_audio_track, quality_format};

View file

@ -241,6 +241,10 @@ pub fn app_with_state(state: AppState) -> Router {
.patch(patch_entry_handler) .patch(patch_entry_handler)
.delete(delete_entry_handler), .delete(delete_entry_handler),
) )
.route(
"/api/archives/:archive_id/entries/:entry_uid/children",
get(list_entry_children),
)
.route( .route(
"/api/archives/:archive_id/entries/:entry_uid/artifacts/:artifact_index", "/api/archives/:archive_id/entries/:entry_uid/artifacts/:artifact_index",
get(serve_artifact), get(serve_artifact),
@ -260,6 +264,10 @@ pub fn app_with_state(state: AppState) -> Router {
"/api/archives/:archive_id/captures/probe", "/api/archives/:archive_id/captures/probe",
get(probe_handler), get(probe_handler),
) )
.route(
"/api/archives/:archive_id/captures/probe-playlist",
post(probe_playlist_handler),
)
.route( .route(
"/api/archives/:archive_id/capture_jobs/:job_uid", "/api/archives/:archive_id/capture_jobs/:job_uid",
get(get_capture_job_handler), get(get_capture_job_handler),
@ -404,6 +412,18 @@ async fn list_entries(
Ok(Json(archive::list_root_entries(&conn, caller_bits)?)) Ok(Json(archive::list_root_entries(&conn, caller_bits)?))
} }
async fn list_entry_children(
State(state): State<AppState>,
auth: AuthUser,
Path((archive_id, entry_uid)): Path<(String, String)>,
) -> Result<Json<Vec<archive::EntrySummary>>, ApiError> {
auth.require_auth()?;
let mounted = mounted_archive(&state, &archive_id)?;
let conn = database::open_or_initialize(&mounted.archive_path)?;
let caller_bits = auth_to_caller_bits(&auth);
Ok(Json(archive::list_child_entries(&conn, &entry_uid, caller_bits)?))
}
async fn search_entries_handler( async fn search_entries_handler(
State(state): State<AppState>, State(state): State<AppState>,
auth: AuthUser, auth: AuthUser,
@ -749,6 +769,13 @@ struct CaptureBody {
modal_closer_enabled: Option<bool>, modal_closer_enabled: Option<bool>,
/// Route through Freedium mirror for WebPage captures. Absent = true (on by default). /// Route through Freedium mirror for WebPage captures. Absent = true (on by default).
via_freedium: Option<bool>, via_freedium: Option<bool>,
/// Per-video quality overrides for playlist captures.
/// Keys are yt-dlp video IDs; values are quality strings ("best", "1080p", "audio", etc.).
#[serde(default)]
per_item_quality: std::collections::HashMap<String, String>,
/// When true, skip playlist items already archived under an existing container.
#[serde(default)]
sync: bool,
} }
#[derive(Debug, serde::Deserialize)] #[derive(Debug, serde::Deserialize)]
@ -756,6 +783,11 @@ struct ProbeQuery {
locator: String, locator: String,
} }
#[derive(Debug, serde::Deserialize)]
struct ProbePlaylistBody {
locator: String,
}
#[derive(Debug, serde::Deserialize)] #[derive(Debug, serde::Deserialize)]
struct LoginBody { struct LoginBody {
username: String, username: String,
@ -826,6 +858,28 @@ async fn capture_handler(
)); ));
} }
} }
{
let is_valid_quality = |q: &str| {
q == "best"
|| q == "audio"
|| q.strip_suffix('p')
.and_then(|n| n.parse::<u32>().ok())
.is_some()
};
if let Some(bad) = body.per_item_quality.values().find(|q| !is_valid_quality(q)) {
return Err(ApiError::bad_request(&format!(
"invalid per_item_quality value {bad:?}: must be \"best\", \"audio\", or a height string like \"1080p\""
)));
}
}
// per_item_quality semantics (enforced in capture.rs):
// - Absent or empty map: all playlist items are downloaded; quality is the
// global `quality` field applied as a yt-dlp cap with graceful fallback.
// - Non-empty map: ONLY items whose yt-dlp ID appears as a key are downloaded;
// absent IDs are skipped. This is how the frontend's delete-item button works.
// The "must choose quality for unsupported videos" invariant is enforced by the
// frontend before submission; a direct API caller bypassing the UI accepts
// yt-dlp's standard cap-and-fallback behavior for items it includes.
let mounted = mounted_archive(&state, &archive_id)?; let mounted = mounted_archive(&state, &archive_id)?;
let archive_paths = let archive_paths =
archive::read_archive_paths(&mounted.archive_path).map_err(ApiError::from)?; archive::read_archive_paths(&mounted.archive_path).map_err(ApiError::from)?;
@ -863,6 +917,8 @@ async fn capture_handler(
modal_closer_enabled: Some(effective_modal_closer), modal_closer_enabled: Some(effective_modal_closer),
reader_mode: body.reader_mode.unwrap_or(false), reader_mode: body.reader_mode.unwrap_or(false),
via_freedium: body.via_freedium.unwrap_or(true), via_freedium: body.via_freedium.unwrap_or(true),
per_item_quality: body.per_item_quality.clone(),
sync: body.sync,
}; };
// Spawn background capture. // Spawn background capture.
@ -902,10 +958,19 @@ async fn capture_handler(
notes_str = serde_json::Value::Object(notes_map).to_string(); notes_str = serde_json::Value::Object(notes_map).to_string();
Some(&notes_str) Some(&notes_str)
}; };
// A partial playlist (some items succeeded, some failed) has status="failed"
// but completed_child_count > 0. Treat it as a completed job so onCaptured
// fires and the archived entries appear. Only mark failed when no child
// succeeded (completed_child_count == 0).
let job_status = if result.status == "failed" && result.completed_child_count == 0 {
"failed"
} else {
"completed"
};
database::update_capture_job_status( database::update_capture_job_status(
&conn, &conn,
&job_uid_bg, &job_uid_bg,
"completed", job_status,
Some(&result.run_uid), Some(&result.run_uid),
None, None,
notes, notes,
@ -981,6 +1046,8 @@ async fn rearchive_handler(
modal_closer_enabled: None, modal_closer_enabled: None,
reader_mode: false, reader_mode: false,
via_freedium: false, via_freedium: false,
per_item_quality: std::collections::HashMap::new(),
sync: false,
}; };
let job_uid_bg = job_uid.clone(); let job_uid_bg = job_uid.clone();
@ -1109,6 +1176,41 @@ async fn probe_handler(
}))) })))
} }
async fn probe_playlist_handler(
State(state): State<AppState>,
auth_user: AuthUser,
Path(archive_id): Path<String>,
Json(body): Json<ProbePlaylistBody>,
) -> Result<Json<downloader::ytdlp::PlaylistProbeResult>, ApiError> {
auth_user.require_role(ROLE_USER)?;
let locator = body.locator.trim().to_string();
if locator.is_empty() {
return Err(ApiError::bad_request("locator must not be empty"));
}
// Validate it's a playlist/channel source and expand shorthands.
let canonical_url = capture::locator_to_playlist_url(&locator)
.ok_or_else(|| ApiError::bad_request("locator is not a YouTube playlist, channel, YTM playlist, or Spotify album/playlist"))?;
// Verify archive exists.
let _ = mounted_archive(&state, &archive_id)?;
// Resolve cookies.
let cookie_rules = match database::open_auth_db(&state.auth_db_path) {
Ok(conn) => database::list_cookie_rules(&conn).unwrap_or_default(),
Err(_) => vec![],
};
let cookies = capture::resolve_cookies_for_url(&cookie_rules, &canonical_url);
// Shell out to yt-dlp in a blocking task.
let result = tokio::task::spawn_blocking(move || {
downloader::ytdlp::probe_playlist_qualities(&canonical_url, &cookies)
})
.await
.map_err(|_| ApiError::internal("probe-playlist task panicked"))?
.map_err(|e| ApiError {
status: StatusCode::BAD_GATEWAY,
message: format!("playlist probe failed: {e:#}"),
})?;
Ok(Json(result))
}
async fn auth_setup_status( async fn auth_setup_status(
State(state): State<AppState>, State(state): State<AppState>,
) -> Result<Json<serde_json::Value>, ApiError> { ) -> Result<Json<serde_json::Value>, ApiError> {

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View file

@ -4,8 +4,8 @@
<meta charset="utf-8" /> <meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" /> <meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Archivr</title> <title>Archivr</title>
<script type="module" crossorigin src="/assets/index-YmIQCrug.js"></script> <script type="module" crossorigin src="/assets/index-De3b80Fv.js"></script>
<link rel="stylesheet" crossorigin href="/assets/index-DLdY9nrw.css"> <link rel="stylesheet" crossorigin href="/assets/index-D8ic-z4p.css">
</head> </head>
<body> <body>
<div id="root"></div> <div id="root"></div>

View file

@ -141,7 +141,7 @@ Auth and session handling will be designed when remote or public hosting becomes
### Supported Platforms ### Supported Platforms
- Local files: `file:///absolute/path/to/file.ext` - Local files: `file:///absolute/path/to/file.ext`
- YouTube media: standard video/short URLs, plus [shorthand video inputs](#supported-shorthand-inputs) - YouTube media: individual videos/shorts, playlists, and channels; standard URLs or [shorthand video inputs](#supported-shorthand-inputs). Playlists and channels archive as a container entry with each video stored as a child entry beneath it.
- X/Twitter media from Tweets: normal Tweet URLs or the `tweet:media:ID` shorthand - X/Twitter media from Tweets: normal Tweet URLs or the `tweet:media:ID` shorthand
- X/Twitter Tweet content scrape: [Tweet and Thread shorthands](#supported-shorthand-inputs). (These are saved as JSON files in `raw_tweets/`) - X/Twitter Tweet content scrape: [Tweet and Thread shorthands](#supported-shorthand-inputs). (These are saved as JSON files in `raw_tweets/`)
- Instagram, Facebook, TikTok, Reddit, Snapchat: direct URLs or platform-prefixed shorthand passed through to `yt-dlp` - Instagram, Facebook, TikTok, Reddit, Snapchat: direct URLs or platform-prefixed shorthand passed through to `yt-dlp`
@ -175,6 +175,14 @@ The probe endpoint (`GET /api/archives/:id/captures/probe?locator=…`) requires
``` ```
`has_video: false, has_audio: false` means yt-dlp found no downloadable tracks (e.g. a tweet with no media). A 502 means yt-dlp itself failed (transient network error, rate-limit, unsupported extractor) — treat as inconclusive, not "no media." `has_video: false, has_audio: false` means yt-dlp found no downloadable tracks (e.g. a tweet with no media). A 502 means yt-dlp itself failed (transient network error, rate-limit, unsupported extractor) — treat as inconclusive, not "no media."
#### YouTube playlists and channels
Capturing a YouTube playlist or channel URL creates a **container entry** for the playlist or channel, with each video archived as a child entry beneath it. Before downloading, the capture UI probes each video to fetch available quality options, letting you set quality per-video or apply a single quality to the whole playlist.
**Incremental sync:** When re-archiving a playlist or channel, enable **sync mode** in the capture dialog to skip videos that are already in the archive. Only new videos are downloaded; the existing container entry is reused.
**Excluding individual videos:** In the expanded per-video list, each video has a remove button (×) to exclude it from the current capture. Removed videos are not downloaded; the rest proceed normally.
### Hosting on NixOS ### Hosting on NixOS
The flake exposes a `nixosModules.default` output. Add it to your system flake and The flake exposes a `nixosModules.default` output. Add it to your system flake and

File diff suppressed because it is too large Load diff

View file

@ -1,403 +0,0 @@
# Auth Foundation Design
**Track:** 4 of the roadmap (inserted after Track 3: Async capture jobs)
**Date:** 2026-06-25
**Status:** Approved for implementation
---
## Context & Roadmap Position
Archivr is evolving from a local-only tool (single hard-coded user, 127.0.0.1 binding) into a
self-hosted multi-user platform — think ArchiveBox but with real accounts, roles, and
public/private visibility. This track lays the foundation. All subsequent tracks depend on it.
**Full decomposition:**
| Track | Scope | Depends on |
|---|---|---|
| 4 (this) | Auth foundation | — |
| 5 | User management — registration, custom roles, admin panel | Track 4 |
| 6 | Permissions & visibility — collection model, per-membership visibility | Track 5 |
| 7 | Settings — account profile, instance-wide toggles | Track 5 |
| 8 | Collections UI | Tracks 56 |
---
## Goals
- Password-protected login with cookie sessions and API tokens
- Role table with bitmask-based visibility (extensible to custom roles in Track 5)
- Auth middleware that protects write/admin routes
- First-run owner setup wizard
- Frontend login page and session-aware API calls
## Non-Goals (explicitly deferred)
- Custom role creation UI → Track 5
- User registration flow → Track 5
- Visibility enforcement on queries → Track 6
- Collection model (replacing `archived_entries.visibility`) → Track 6
- Account settings page → Track 7
- API token management UI → Track 7
---
## Schema
### New table: `roles`
```sql
CREATE TABLE IF NOT EXISTS roles (
id INTEGER PRIMARY KEY,
role_uid TEXT NOT NULL UNIQUE,
slug TEXT NOT NULL UNIQUE, -- 'guest', 'user', 'admin', 'owner', or custom
name TEXT NOT NULL,
level INTEGER NOT NULL, -- ordering: guest=0, user=1, admin=3, owner=4
bit_position INTEGER NOT NULL UNIQUE, -- position in visibility bitmask
is_builtin INTEGER NOT NULL DEFAULT 0 CHECK (is_builtin IN (0, 1))
);
```
**Built-in rows seeded at schema init:**
| slug | level | bit_position | bit value | is_builtin |
|---|---|---|---|---|
| guest | 0 | 0 | 1 | 1 |
| user | 1 | 1 | 2 | 1 |
| admin | 3 | 2 | 4 | 1 |
| owner | 4 | 3 | 8 | 1 |
Bit position 2 (value 4) is reserved for `admin`. Bit positions 4+ (values 16, 32, …) are assigned
to custom roles in Track 5. Level 2 is reserved for custom roles sitting between `user` and `admin`.
**role_bits computation — implicit guest floor:**
`role_bits` for any **authenticated** user is computed as:
```
role_bits = ROLE_GUEST | (OR of bit values for all rows in user_roles)
```
The `ROLE_GUEST` bit (1) is always included for authenticated users so they can access
public (guest-visible) content. Example: an owner assigned only the `owner` role gets
`role_bits = 1 | 8 = 9`, which passes `role_bits & ROLE_USER (2) = 0` — still broken.
**Therefore, role assignment is cumulative by level.** When a role is assigned, all
built-in roles at lower levels are also assigned:
- Assigning `owner` (level 4) → also assign `admin`, `user` in `user_roles`
- Assigning `admin` (level 3) → also assign `user` in `user_roles`
- Assigning `user` (level 1) → no additional rows
- `guest` is never assigned; it is the implicit unauthenticated floor
Setup creates owner with three `user_roles` rows: `user`, `admin`, `owner`.
Resulting `role_bits = ROLE_GUEST | ROLE_USER | ROLE_ADMIN | ROLE_OWNER = 1|2|4|8 = 15`.
**Visibility check:** `viewer.role_bits & content.visibility != 0` passes if the viewer
has any bit the content requires. Owner (15) can see everything. User (1|2=3) can see
guest-visible (1) and user-visible (2) content but not admin-only (4). ✓
`is_builtin = 1` rows cannot be deleted.
### New table: `sessions`
```sql
CREATE TABLE IF NOT EXISTS sessions (
id INTEGER PRIMARY KEY,
session_uid TEXT NOT NULL UNIQUE,
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
role_bits INTEGER NOT NULL, -- snapshot of bitmask at login time; role changes take effect on next login
created_at TEXT NOT NULL,
last_seen_at TEXT NOT NULL,
expires_at TEXT NOT NULL, -- 30 days from last_seen_at
user_agent TEXT
);
CREATE INDEX IF NOT EXISTS idx_sessions_user_id ON sessions(user_id);
```
### New table: `api_tokens`
```sql
CREATE TABLE IF NOT EXISTS api_tokens (
id INTEGER PRIMARY KEY,
token_uid TEXT NOT NULL UNIQUE,
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
token_hash TEXT NOT NULL UNIQUE, -- SHA-256 of raw token; raw token never stored
name TEXT NOT NULL,
created_at TEXT NOT NULL,
last_used_at TEXT,
expires_at TEXT -- NULL = never expires
);
CREATE INDEX IF NOT EXISTS idx_api_tokens_user_id ON api_tokens(user_id);
```
### `users` table — existing, minimally changed
The existing `role TEXT NOT NULL CHECK (role IN ('admin','user'))` column is **kept but inert**
auth middleware reads from `user_roles`, not this column. It will be removed in Track 5 cleanup.
`ensure_owner_exists` must supply a value for this column; use `'admin'` as the placeholder.
`ensure_default_user` is replaced by `ensure_owner_exists` which returns `false` if no owner
row exists in `user_roles` (triggers setup mode). The old local-admin stub is never created on
fresh instances. Session lookup JOINs `users` and checks `users.status = 'active'`; a session
belonging to a disabled user resolves to `AuthUser::Guest`.
### `instance_settings` — one new column
The column is added **inside** the existing `CREATE TABLE IF NOT EXISTS instance_settings` DDL,
not via `ALTER TABLE` (which is not idempotent in `initialize_schema`):
```sql
CREATE TABLE IF NOT EXISTS instance_settings (
id INTEGER PRIMARY KEY CHECK (id = 1),
public_index_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_index_enabled IN (0, 1)),
public_entry_content_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_entry_content_enabled IN (0, 1)),
public_archive_submission_enabled INTEGER NOT NULL DEFAULT 0 CHECK (public_archive_submission_enabled IN (0, 1)),
default_entry_visibility INTEGER NOT NULL DEFAULT 2 -- 2 = user-visible by default
);
```
The existing `INSERT OR IGNORE INTO instance_settings … VALUES (1, 0, 0, 0)` seed row must be
updated to include the new column: `VALUES (1, 0, 0, 0, 2)`.
### `archived_entries.visibility` — deprecated, not removed
Flagged with a `-- DEPRECATED: replaced by collection_entries.visibility in Track 6` comment in
`initialize_schema`. No data migration needed yet; Track 6 handles it.
---
## Auth Flow
### Login
```
POST /api/auth/login
Body: { username: string, password: string }
```
1. Look up user by username.
2. Verify password with Argon2id (`argon2` crate).
3. Compute `role_bits = ROLE_GUEST | (OR of bit values for all user_roles rows)` (cumulative; see Schema § role_bits computation).
4. Insert `sessions` row (`session_uid` = UUID, `expires_at` = now + 30 days).
5. Set-Cookie: `session=<session_uid>; HttpOnly; SameSite=Strict; Path=/; Max-Age=2592000`.
Add `Secure` flag when the request arrived over HTTPS (detected via `X-Forwarded-Proto: https`
header or TLS connection info). Omit `Secure` for plain HTTP to support local dev without TLS.
6. Return `200 { user_uid, username, role_bits }`.
On failure: `401 { error: "invalid_credentials" }` — same message for unknown user and wrong
password (no user enumeration).
### Logout
```
POST /api/auth/logout
```
Deletes the `sessions` row for the current session cookie. Responds with
`Set-Cookie: session=; Max-Age=0` to clear the browser cookie. Returns `204`.
### Current user
```
GET /api/auth/me
```
Returns `200 { user_uid, username, role_bits }` for an authenticated request, or `401` for a
guest. The frontend calls this once on mount to restore session state.
### First-run setup
```
GET /api/auth/setup → 200 { setup_required: bool }
POST /api/auth/setup → 201 { user_uid, username }
Body: { username: string, password: string }
```
`setup_required` is `true` when no user has the `owner` role in `user_roles`. On `POST`:
- If setup is **already complete** (an owner exists): return `409 { error: "already_configured" }`.
- Otherwise: create the user (with `users.role = 'admin'` as placeholder), assign `user_roles`
rows for `user`, `admin`, `owner` (cumulative), seed `instance_settings` row if absent.
Return `201 { user_uid, username }`. Normal login flow applies immediately after.
All non-setup API routes return `503 { error: "setup_required" }` until setup is complete.
The following routes are **exempt** from the 503 check: `GET /api/auth/setup`,
`POST /api/auth/setup`, `GET /` (static), `GET /assets/*` (static).
### API tokens
```
POST /api/auth/tokens → 201 { token_uid, raw_token, name, created_at }
GET /api/auth/tokens → 200 [{ token_uid, name, created_at, last_used_at }]
DELETE /api/auth/tokens/:token_uid → 204
```
`raw_token` is a cryptographically random 32-byte value, base64url-encoded, returned once. The
server stores only its SHA-256 hash. The management UI for these endpoints is in Track 7; the
endpoints are implemented here.
### Password hashing
Argon2id with default parameters from the `argon2` crate (memory=19 MiB, iterations=2,
parallelism=1). The current `"disabled-local-password"` sentinel in `ensure_default_user` becomes
irrelevant once setup is required on fresh instances.
### Session expiry & cleanup
`last_seen_at` is updated on every authenticated request using a **conditional update**: the
session row is already read during extraction; if `now() - last_seen_at > 60s`, issue an UPDATE.
This adds no extra query — only an extra UPDATE when the threshold is crossed.
`expires_at` = `last_seen_at + 30 days`, recalculated on each UPDATE. A background task in
`archivr-server/src/main.rs` runs `DELETE FROM sessions WHERE expires_at < now()` at startup
and every 24 hours via `tokio::time::interval`.
---
## Auth Extractor
New file: `crates/archivr-server/src/auth.rs`
```rust
pub enum AuthUser {
Guest,
Authenticated { user_id: i64, role_bits: u32 },
}
impl AuthUser {
pub fn require_auth(&self) -> Result<(i64, u32), ApiError> // 401 if Guest
pub fn require_role(&self, bit: u32) -> Result<(), ApiError> // 403 if bit not set
pub fn has_role(&self, bit: u32) -> bool
}
// Role bit constants
pub const ROLE_GUEST: u32 = 1;
pub const ROLE_USER: u32 = 2;
pub const ROLE_ADMIN: u32 = 4;
pub const ROLE_OWNER: u32 = 8;
```
Implemented as an Axum `FromRequestParts` extractor. Tries `session` cookie first, then
`Authorization: Bearer` header.
- **Cookie path**: look up `sessions` row JOIN `users` WHERE `session_uid = ?`
AND `users.status = 'active'` AND `expires_at > now()`. Use cached `role_bits` from the
session row.
- **Bearer path**: SHA-256 the token, look up `api_tokens` row JOIN `users` WHERE
`token_hash = ?` AND `users.status = 'active'` AND (`expires_at IS NULL OR expires_at > now()`).
Compute `role_bits` live: `ROLE_GUEST | (OR of user_roles bit values for that user)`.
Update `api_tokens.last_used_at`.
- Missing or invalid credential → `AuthUser::Guest` (never a hard error at extraction time).
---
## Route Protection Tiers
The existing security-boundary comment block in `routes.rs` is updated:
| Tier | Requirement | Examples |
|---|---|---|
| `STATIC` | none | `GET /`, `GET /assets/*` |
| `PUBLIC_READ` | none (visibility filtering deferred to Track 6) | `GET /api/archives/:id/entries` |
| `AUTH_READ` | `ROLE_USER` bit | authenticated entry access |
| `WRITE` | `ROLE_USER` bit | `POST /api/archives/:id/captures`, tag mutations |
| `ADMIN` | `ROLE_ADMIN` bit | `GET /api/admin/archives`, user management |
| `OWNER` | `ROLE_OWNER` bit | instance settings, ownership transfer |
**Error responses:**
- No/invalid session → `401` (frontend redirects to login)
- Valid session, insufficient role → `403`
- Private resource accessed without sufficient role → `404` (do not reveal existence)
Track 4 applies `ROLE_USER` enforcement to all existing `WRITE` routes and `ROLE_ADMIN` to
`/api/admin/*`. `PUBLIC_READ` routes return all data for now; Track 6 adds visibility filters.
---
## Frontend Changes
### New components
| Component | Purpose |
|---|---|
| `SetupPage.jsx` | First-run owner account creation wizard |
| `LoginPage.jsx` | Username/password login form |
### App.jsx changes
- On mount: call `GET /api/auth/setup`; if `setup_required`, render `<SetupPage>` and nothing else.
- Otherwise: call `GET /api/auth/me`; store result as `currentUser` state (null = guest).
- Pass `currentUser` down via React context (`AuthContext`).
- Any `401` response from any API call sets `currentUser` to null → triggers `<LoginPage>`.
### api.js changes
- Thin response interceptor: if status is `401`, dispatch a global `auth:expired` event that
`App.jsx` listens to and handles by clearing `currentUser`.
- No token storage in JS — cookies are handled entirely by the browser.
### Topbar.jsx changes
- When `currentUser` is set: show `username` and a **Log out** button.
- Log out calls `POST /api/auth/logout`, then clears `currentUser`.
### What is NOT in Track 4 frontend
- Settings page (Track 7)
- User management UI (Track 5)
- Role or visibility controls (Track 6)
- API token management UI (Track 7)
---
## New Dependencies
| Crate | Purpose |
|---|---|
| `argon2` | Password hashing (Argon2id) |
| `rand` | Cryptographically random token generation |
| `tower-cookies` | Cookie extraction in Axum (or use `axum-extra`) |
Add to `archivr-server/Cargo.toml` and workspace `Cargo.toml` as needed.
---
## Files Changed
| File | Change |
|---|---|
| `crates/archivr-core/src/database.rs` | Add `roles`, `user_roles`, `sessions`, `api_tokens` tables; seed built-in roles; add `instance_settings.default_entry_visibility`; replace `ensure_default_user` with `ensure_owner_exists`; add session/token CRUD helpers |
| `crates/archivr-server/src/auth.rs` | New: `AuthUser` extractor, role bit constants, session/token lookup |
| `crates/archivr-server/src/routes.rs` | Add auth endpoints (`/api/auth/*`); apply `AuthUser` extractor to WRITE/ADMIN routes; update security-boundary comment |
| `crates/archivr-server/src/main.rs` | Session cleanup background task |
| `frontend/src/App.jsx` | Setup check, auth state, `AuthContext` |
| `frontend/src/api.js` | 401 interceptor |
| `frontend/src/components/LoginPage.jsx` | New |
| `frontend/src/components/SetupPage.jsx` | New |
| `frontend/src/components/Topbar.jsx` | User menu + logout |
| `Cargo.toml` | Add `argon2`, `rand`, `tower-cookies` (or `axum-extra`) |
---
## Test Coverage
- `database.rs`: role seeding, `ensure_owner_exists`, session CRUD, token hash round-trip
- `auth.rs`: extractor resolves cookie → session → user; extractor resolves Bearer → token → user;
missing credential → Guest; expired session → Guest
- `routes.rs`: login happy path; login wrong password returns 401; logout clears session;
setup endpoint returns 503 after setup complete; WRITE route returns 401 for Guest;
WRITE route returns 403 for insufficient role; setup flow end-to-end
---
## Track Numbering Update for NEXT.md
Original tracks 4 and 5 shift to 8 and 9. Collections is a named future track (no number until
scoped):
| # | Track |
|---|---|
| 3 | Async capture jobs |
| **4** | **Auth foundation (this spec)** |
| **5** | **User management** |
| **6** | **Permissions & visibility (collection model)** |
| **7** | **Settings** |
| **8** | **Collections UI** |
| 9 | Cloud backup (was 4) |
| 10 | Cloud storage (was 5) |

View file

@ -86,6 +86,7 @@ export default function App() {
const [archives, setArchives] = useState([]) const [archives, setArchives] = useState([])
const [archiveId, setArchiveId] = useState(null) const [archiveId, setArchiveId] = useState(null)
const [entries, setEntries] = useState([]) const [entries, setEntries] = useState([])
const [deletedUids, setDeletedUids] = useState(() => new Set())
const [selectedEntryUid, setSelectedEntryUid] = useState(() => parseLocation().entry) const [selectedEntryUid, setSelectedEntryUid] = useState(() => parseLocation().entry)
const [selectedEntry, setSelectedEntry] = useState(null) const [selectedEntry, setSelectedEntry] = useState(null)
const [selectedUids, setSelectedUids] = useState(() => { const [selectedUids, setSelectedUids] = useState(() => {
@ -131,6 +132,11 @@ export default function App() {
const pendingSearchFocus = useRef(false) const pendingSearchFocus = useRef(false)
const firstArchiveLoad = useRef(true) const firstArchiveLoad = useRef(true)
const lastAnchorIndexRef = useRef(null) const lastAnchorIndexRef = useRef(null)
// Monotonic token for j/k keyboard navigation; cancels stale fetchEntryDetail calls.
const jkSeqRef = useRef(0)
// uid entry object cache; populated on every row click so ctrl/shift
// selections can resolve child entries that aren't in the root entries array.
const entryCacheRef = useRef(new Map())
const humanizeTags = currentUser?.humanize_slugs ?? false; const humanizeTags = currentUser?.humanize_slugs ?? false;
@ -260,12 +266,20 @@ export default function App() {
}, []) }, [])
const handleRowClick = useCallback((entry, e) => { const handleRowClick = useCallback((entry, e) => {
// Cache every clicked entry so shift/ctrl can resolve child entries
// that are not present in the root `entries` array.
entryCacheRef.current.set(entry.entry_uid, entry)
// Invalidate any in-flight j/k uncached-child fetch.
++jkSeqRef.current
if (e.shiftKey && lastAnchorIndexRef.current !== null) { if (e.shiftKey && lastAnchorIndexRef.current !== null) {
e.preventDefault() e.preventDefault()
const anchorIdx = entries.findIndex(x => x.entry_uid === lastAnchorIndexRef.current) // Use DOM order so child rows (not in the `entries` array) participate
const clickIdx = entries.findIndex(x => x.entry_uid === entry.entry_uid) // in range selection. Every rendered row has data-entry-uid.
const allNodes = [...document.querySelectorAll('#entries-body [data-entry-uid]')]
const anchorIdx = allNodes.findIndex(n => n.dataset.entryUid === lastAnchorIndexRef.current)
const clickIdx = allNodes.findIndex(n => n.dataset.entryUid === entry.entry_uid)
if (anchorIdx === -1 || clickIdx === -1) { if (anchorIdx === -1 || clickIdx === -1) {
// anchor evicted by search/filter/delete fall back to single select
lastAnchorIndexRef.current = entry.entry_uid lastAnchorIndexRef.current = entry.entry_uid
setSelectedUids(new Set([entry.entry_uid])) setSelectedUids(new Set([entry.entry_uid]))
selectEntry(entry) selectEntry(entry)
@ -273,24 +287,41 @@ export default function App() {
} }
const lo = Math.min(anchorIdx, clickIdx) const lo = Math.min(anchorIdx, clickIdx)
const hi = Math.max(anchorIdx, clickIdx) const hi = Math.max(anchorIdx, clickIdx)
const range = entries.slice(lo, hi + 1) const uids = new Set(allNodes.slice(lo, hi + 1).map(n => n.dataset.entryUid))
const uids = new Set(range.map(x => x.entry_uid))
setSelectedUids(uids) setSelectedUids(uids)
if (uids.size === 1) selectEntry(range[0]) if (uids.size === 1) selectEntry(entry)
} else if (e.ctrlKey || e.metaKey) { } else if (e.ctrlKey || e.metaKey) {
lastAnchorIndexRef.current = entry.entry_uid lastAnchorIndexRef.current = entry.entry_uid
setSelectedUids(prev => { const next = new Set(selectedUids)
const next = new Set(prev)
if (next.has(entry.entry_uid)) next.delete(entry.entry_uid) if (next.has(entry.entry_uid)) next.delete(entry.entry_uid)
else next.add(entry.entry_uid) else next.add(entry.entry_uid)
return next setSelectedUids(next)
}) if (next.size === 0) {
selectEntry(null)
} else if (next.size === 1) {
// Resolve the remaining UID may be a child not in the root entries array
// and not yet cached (e.g. picked up via shift-range without a direct click).
const [remainingUid] = next
const cached = entryCacheRef.current.get(remainingUid)
?? entries.find(x => x.entry_uid === remainingUid)
?? null
if (cached) {
selectEntry(cached)
} else {
const tok = ++jkSeqRef.current
fetchEntryDetail(archiveId, remainingUid)
.then(det => { if (tok === jkSeqRef.current && det?.summary) selectEntry(det.summary) })
.catch(() => {})
}
} else {
selectEntry(null)
}
} else { } else {
lastAnchorIndexRef.current = entry.entry_uid lastAnchorIndexRef.current = entry.entry_uid
setSelectedUids(new Set([entry.entry_uid])) setSelectedUids(new Set([entry.entry_uid]))
selectEntry(entry) selectEntry(entry)
} }
}, [entries, selectEntry]) }, [entries, selectedUids, selectEntry, archiveId])
const handleTagFilterSet = useCallback((fullPath) => { const handleTagFilterSet = useCallback((fullPath) => {
setTagFilter(fullPath) setTagFilter(fullPath)
@ -342,18 +373,26 @@ export default function App() {
}, [archiveId, selectedEntry]) }, [archiveId, selectedEntry])
const handleEntryDeleted = useCallback((entryUid) => { const handleEntryDeleted = useCallback((entryUid) => {
const isRoot = entries.some(e => e.entry_uid === entryUid)
setDeletedUids(prev => { const n = new Set(prev); n.add(entryUid); return n })
setEntries(prev => prev.filter(e => e.entry_uid !== entryUid)) setEntries(prev => prev.filter(e => e.entry_uid !== entryUid))
setSelectedEntry(prev => prev?.entry_uid === entryUid ? null : prev) setSelectedEntry(prev => prev?.entry_uid === entryUid ? null : prev)
setSelectedEntryUid(prev => prev === entryUid ? null : prev) setSelectedEntryUid(prev => prev === entryUid ? null : prev)
setSelectedUids(prev => { const n = new Set(prev); n.delete(entryUid); return n }) setSelectedUids(prev => { const n = new Set(prev); n.delete(entryUid); return n })
}, []) // Child delete: parent row's child_count/size are stale reload after state updates.
if (!isRoot) loadEntries(archiveId, searchQuery, tagFilter)
}, [entries, archiveId, searchQuery, tagFilter, loadEntries])
const handleBulkDeleted = useCallback((uids) => { const handleBulkDeleted = useCallback((uids) => {
const rootUids = new Set(entries.map(e => e.entry_uid))
const hasChildDelete = [...uids].some(u => !rootUids.has(u))
setDeletedUids(prev => { const n = new Set(prev); uids.forEach(u => n.add(u)); return n })
setEntries(prev => prev.filter(e => !uids.has(e.entry_uid))) setEntries(prev => prev.filter(e => !uids.has(e.entry_uid)))
setSelectedUids(new Set()) setSelectedUids(new Set())
setSelectedEntry(null) setSelectedEntry(null)
setSelectedEntryUid(null) setSelectedEntryUid(null)
}, []) if (hasChildDelete) loadEntries(archiveId, searchQuery, tagFilter)
}, [entries, archiveId, searchQuery, tagFilter, loadEntries])
// Auto-snap: drive selectedEntryUid from selectedUids so URL sync and detail // Auto-snap: drive selectedEntryUid from selectedUids so URL sync and detail
// panel stay correct. size >= 2 clears single-entry state (bulk panel takes over). // panel stay correct. size >= 2 clears single-entry state (bulk panel takes over).
@ -397,10 +436,16 @@ export default function App() {
if (current !== url) history.replaceState(null, '', url) if (current !== url) history.replaceState(null, '', url)
}, [searchQuery, tagFilter, selectedEntryUid, view]) }, [searchQuery, tagFilter, selectedEntryUid, view])
// K / Ctrl+K: focus the search input, switching to archive view first if needed. // K / Ctrl+K / /: focus the search input, switching to archive view first if needed.
useEffect(() => { useEffect(() => {
const handler = (e) => { const handler = (e) => {
if ((e.metaKey || e.ctrlKey) && e.key === 'k') { const isSlash = e.key === '/' && !e.metaKey && !e.ctrlKey && !e.altKey
const isCmdK = (e.metaKey || e.ctrlKey) && e.key === 'k'
if (!isSlash && !isCmdK) return
// Don't intercept / when already typing in an input
const tag = document.activeElement?.tagName
if (isSlash && (tag === 'INPUT' || tag === 'TEXTAREA' || tag === 'SELECT')) return
if (isSlash && document.activeElement?.isContentEditable) return
e.preventDefault() e.preventDefault()
if (view === 'archive') { if (view === 'archive') {
searchInputRef.current?.focus() searchInputRef.current?.focus()
@ -410,11 +455,62 @@ export default function App() {
setView('archive') setView('archive')
} }
} }
}
document.addEventListener('keydown', handler) document.addEventListener('keydown', handler)
return () => document.removeEventListener('keydown', handler) return () => document.removeEventListener('keydown', handler)
}, [view]) }, [view])
// j/k: navigate entries down/up when not focused on an input element.
useEffect(() => {
const handler = (e) => {
// Ignore when a modifier is held (don't steal browser/app shortcuts)
if (e.metaKey || e.ctrlKey || e.altKey) return
if (e.key !== 'j' && e.key !== 'k') return
// Ignore when focus is on any editable target
const tag = document.activeElement?.tagName
if (tag === 'INPUT' || tag === 'TEXTAREA' || tag === 'SELECT') return
if (document.activeElement?.isContentEditable) return
const allNodes = [...document.querySelectorAll('#entries-body [data-entry-uid]')]
if (allNodes.length === 0) return
const [currentUid] = selectedUids.size === 1 ? selectedUids : [null]
const currentIdx = currentUid
? allNodes.findIndex(n => n.dataset.entryUid === currentUid)
: -1
const nextIdx = e.key === 'j'
? Math.min(currentIdx + 1, allNodes.length - 1)
: Math.max(currentIdx - 1, 0)
if (nextIdx === currentIdx && currentIdx !== -1) return
const nextNode = allNodes[nextIdx < 0 ? 0 : nextIdx]
const nextUid = nextNode.dataset.entryUid
lastAnchorIndexRef.current = nextUid
const tok = ++jkSeqRef.current
setSelectedUids(new Set([nextUid]))
// Resolve entry object: cache root entries array server fetch.
// Token guards against a slow fetch settling after the user has moved on.
const cached = entryCacheRef.current.get(nextUid)
?? entries.find(x => x.entry_uid === nextUid)
?? null
if (cached) {
selectEntry(cached)
} else {
fetchEntryDetail(archiveId, nextUid)
.then(det => { if (tok === jkSeqRef.current && det?.summary) selectEntry(det.summary) })
.catch(() => {})
}
nextNode.scrollIntoView({ block: 'nearest' })
e.preventDefault()
}
document.addEventListener('keydown', handler)
return () => document.removeEventListener('keydown', handler)
}, [selectedUids, entries, selectEntry, archiveId])
// After switching to archive view via K, focus the search input once rendered. // After switching to archive view via K, focus the search input once rendered.
useEffect(() => { useEffect(() => {
if (view === 'archive' && pendingSearchFocus.current) { if (view === 'archive' && pendingSearchFocus.current) {
@ -563,6 +659,7 @@ export default function App() {
onRowClick={handleRowClick} onRowClick={handleRowClick}
archiveId={archiveId} archiveId={archiveId}
pendingCaptures={pendingCaptures} pendingCaptures={pendingCaptures}
deletedUids={deletedUids}
/> />
)} )}
{view === 'runs' && <RunsView runs={runs} />} {view === 'runs' && <RunsView runs={runs} />}

View file

@ -25,6 +25,10 @@ export async function fetchEntryDetail(archiveId, entryUid) {
return getJson(`/api/archives/${archiveId}/entries/${entryUid}`); return getJson(`/api/archives/${archiveId}/entries/${entryUid}`);
} }
export async function fetchEntryChildren(archiveId, entryUid) {
return getJson(`/api/archives/${archiveId}/entries/${entryUid}/children`);
}
// Fetch multiple artifact JSON payloads for an entry in parallel. // Fetch multiple artifact JSON payloads for an entry in parallel.
// Returns a Promise<Array> preserving index order. // Returns a Promise<Array> preserving index order.
export function fetchEntryArtifacts(archiveId, entryUid, indices) { export function fetchEntryArtifacts(archiveId, entryUid, indices) {
@ -149,6 +153,8 @@ export async function submitCapture(archiveId, locator, quality = null, extensio
if (typeof extensions.cookie_ext_enabled === 'boolean') payload.cookie_ext_enabled = extensions.cookie_ext_enabled if (typeof extensions.cookie_ext_enabled === 'boolean') payload.cookie_ext_enabled = extensions.cookie_ext_enabled
if (typeof extensions.modal_closer_enabled === 'boolean') payload.modal_closer_enabled = extensions.modal_closer_enabled if (typeof extensions.modal_closer_enabled === 'boolean') payload.modal_closer_enabled = extensions.modal_closer_enabled
if (typeof extensions.via_freedium === 'boolean') payload.via_freedium = extensions.via_freedium if (typeof extensions.via_freedium === 'boolean') payload.via_freedium = extensions.via_freedium
if (extensions.per_item_quality && typeof extensions.per_item_quality === 'object' && Object.keys(extensions.per_item_quality).length > 0) payload.per_item_quality = extensions.per_item_quality
if (extensions.sync === true) payload.sync = true
} }
const res = await fetch(`/api/archives/${archiveId}/captures`, { const res = await fetch(`/api/archives/${archiveId}/captures`, {
method: "POST", method: "POST",
@ -168,6 +174,19 @@ export async function probeCapture(archiveId, locator) {
return getJson(`/api/archives/${archiveId}/captures/probe?locator=${encodeURIComponent(locator)}`); return getJson(`/api/archives/${archiveId}/captures/probe?locator=${encodeURIComponent(locator)}`);
} }
export async function probePlaylist(archiveId, locator) {
const res = await fetch(`/api/archives/${archiveId}/captures/probe-playlist`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ locator }),
})
if (!res.ok) {
const body = await res.json().catch(() => ({}))
throw new Error(body.error || `HTTP ${res.status}`)
}
return res.json()
}
export async function pollCaptureJob(archiveId, jobUid) { export async function pollCaptureJob(archiveId, jobUid) {
return getJson(`/api/archives/${archiveId}/capture_jobs/${jobUid}`); return getJson(`/api/archives/${archiveId}/capture_jobs/${jobUid}`);
} }

View file

@ -1,5 +1,5 @@
import { useRef, useEffect, useState, useCallback } from 'react' import { useRef, useEffect, useState, useCallback } from 'react'
import { submitCapture, pollCaptureJob, probeCapture, getInstanceSettings } from '../api' import { submitCapture, pollCaptureJob, probeCapture, probePlaylist, getInstanceSettings } from '../api'
let nextItemId = 1 let nextItemId = 1
@ -14,7 +14,11 @@ function isVideoSource(locator) {
for (const scheme of ['yt:', 'youtube:']) { for (const scheme of ['yt:', 'youtube:']) {
if (ll.startsWith(scheme)) { if (ll.startsWith(scheme)) {
const after = ll.slice(scheme.length) const after = ll.slice(scheme.length)
return after.startsWith('video/') || after.startsWith('short/') || after.startsWith('shorts/') if (after.startsWith('video/') || after.startsWith('short/') || after.startsWith('shorts/'))
return true
// bare yt:ID exactly 11 chars [A-Za-z0-9_-], same predicate as is_youtube_video_id in core
if (/^[a-z0-9_-]{11}$/i.test(after)) return true
return false
} }
} }
@ -63,6 +67,52 @@ function isVideoSource(locator) {
return false return false
} }
function isPlaylistSource(locator) {
const l = locator.trim()
const ll = l.toLowerCase()
// yt: / youtube: shorthands playlist, channel, @ handles
for (const scheme of ['yt:', 'youtube:']) {
if (ll.startsWith(scheme)) {
const after = ll.slice(scheme.length)
return after.startsWith('playlist/') || after.startsWith('@') ||
after.startsWith('channel/') || after.startsWith('c/') || after.startsWith('user/')
}
}
// ytm: shorthand playlist
if (ll.startsWith('ytm:')) {
return ll.slice(4).startsWith('playlist/')
}
// spotify: shorthands album and playlist (not track)
if (ll.startsWith('spotify:')) {
const after = ll.slice(8)
return after.startsWith('album:') || after.startsWith('playlist:')
}
// HTTP/HTTPS URLs
if (ll.startsWith('http://') || ll.startsWith('https://')) {
try {
const url = new URL(l)
const host = url.hostname
if (host === 'youtube.com' || host === 'www.youtube.com') {
if (url.pathname === '/playlist' && url.searchParams.has('list')) return true
if (url.pathname.startsWith('/@') || url.pathname.startsWith('/channel/') ||
url.pathname.startsWith('/c/') || url.pathname.startsWith('/user/')) return true
}
if (host === 'music.youtube.com') {
if (url.pathname === '/playlist' && url.searchParams.has('list')) return true
}
if (host === 'open.spotify.com') {
if (url.pathname.startsWith('/album/') || url.pathname.startsWith('/playlist/')) return true
}
} catch {}
}
return false
}
function makeItem(locator = '') { function makeItem(locator = '') {
return { return {
id: nextItemId++, locator, quality: 'best', id: nextItemId++, locator, quality: 'best',
@ -71,9 +121,47 @@ function makeItem(locator = '') {
probeQualities: null, // null | string[] when done, e.g. ["1080p","720p","480p"] probeQualities: null, // null | string[] when done, e.g. ["1080p","720p","480p"]
probeHasAudio: false, // true when probe confirms at least one audio track probeHasAudio: false, // true when probe confirms at least one audio track
status: 'idle', error: null, jobUid: null, archiveId: null, status: 'idle', error: null, jobUid: null, archiveId: null,
// playlist probe state
playlistProbeState: 'idle', // 'idle' | 'probing' | 'done' | 'error'
playlistInfo: null, // raw API response or null
playlistItems: null, // [{id, url, title, qualities, has_audio, quality}] or null
playlistQuality: null, // selected playlist-level quality string or null
playlistExpanded: false, // whether per-video list is expanded
syncEnabled: false, // sync mode toggle
} }
} }
function applyPlaylistQuality(newQ, currentItems) {
if (newQ === 'best') {
return currentItems.map(item => ({ ...item, quality: 'best' }))
}
if (newQ === 'audio') {
return currentItems.map(item => {
if (item.has_audio) return { ...item, quality: 'audio' }
// No audio track same conflict rule as unsupported height:
// keep a prior selection if one exists, otherwise leave null (blocks archive).
if (item.quality !== null) return item
return { ...item, quality: null }
})
}
return currentItems.map(item => {
// Exact match only: the video must list this quality in its available formats.
// maxHeight >= newHeight would be a cap (yt-dlp silent fallback), not what
// the user selected; a video with [2160p, 1080p] does NOT support 1440p.
if (item.qualities.includes(newQ)) {
return { ...item, quality: newQ }
}
// Quality not available for this item keep prior selection if one exists,
// otherwise null (conflict, blocks archive until user picks manually).
if (item.quality !== null) return item
return { ...item, quality: null }
})
}
function hasConflict(item) {
return Array.isArray(item.playlistItems) && item.playlistItems.some(pi => pi.quality === null)
}
export default function CaptureDialog({ open, archiveId, onClose, onCaptured, onToast, onJobStarted, onJobSettled, activeJobs = [] }) { export default function CaptureDialog({ open, archiveId, onClose, onCaptured, onToast, onJobStarted, onJobSettled, activeJobs = [] }) {
const dialogRef = useRef(null) const dialogRef = useRef(null)
const isFirstRenderRef = useRef(true) const isFirstRenderRef = useRef(true)
@ -105,7 +193,9 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
const idle = saved.filter(it => !it.status || it.status === 'idle') const idle = saved.filter(it => !it.status || it.status === 'idle')
if (idle.length > 0) { if (idle.length > 0) {
idle.forEach(it => { if (it.id >= nextItemId) nextItemId = it.id + 1 }) idle.forEach(it => { if (it.id >= nextItemId) nextItemId = it.id + 1 })
return idle // Merge with makeItem() defaults so items saved before the playlist
// fields were added don't have undefined where null/false is expected.
return idle.map(it => ({ ...makeItem(it.locator), ...it }))
} }
} }
} catch {} } catch {}
@ -283,7 +373,7 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
onToastRef.current(text, null, type, headline) onToastRef.current(text, null, type, headline)
} }
async function submitBgJob(locator, quality, batchId) { async function submitBgJob(locator, quality, batchId, extraExtensions = {}) {
const aid = archiveIdRef.current const aid = archiveIdRef.current
const id = crypto.randomUUID?.() ?? `job-${Date.now()}-${Math.random()}` const id = crypto.randomUUID?.() ?? `job-${Date.now()}-${Math.random()}`
// Capture session options at call time (synchronous before first await) // Capture session options at call time (synchronous before first await)
@ -293,6 +383,7 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
cookie_ext_enabled: cookieExtEnabled, cookie_ext_enabled: cookieExtEnabled,
modal_closer_enabled: modalCloserEnabled, modal_closer_enabled: modalCloserEnabled,
via_freedium: freediumEnabled, via_freedium: freediumEnabled,
...extraExtensions,
} }
try { try {
const job = await submitCapture(aid, locator, quality, extensions) const job = await submitCapture(aid, locator, quality, extensions)
@ -309,19 +400,31 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
function handleArchive() { function handleArchive() {
const toSubmit = items.filter(it => it.locator.trim()) const toSubmit = items.filter(it => it.locator.trim())
if (toSubmit.length === 0) return if (toSubmit.length === 0) return
if (toSubmit.some(it => hasConflict(it))) return
if (toSubmit.some(it => it.probeState === 'probing' ||
(isPlaylistSource(it.locator) && it.playlistProbeState !== 'done'))) return
if (toSubmit.some(it => Array.isArray(it.playlistItems) && it.playlistItems.length === 0)) return
const batchId = toSubmit.length > 1 const batchId = toSubmit.length > 1
? (crypto.randomUUID?.() ?? `batch-${Date.now()}`) ? (crypto.randomUUID?.() ?? `batch-${Date.now()}`)
: null : null
if (batchId) { if (batchId) {
batchRef.current.set(batchId, { total: toSubmit.length, archived: 0, warnings: 0, failed: 0, failedLocators: [], warningLocators: [] }) batchRef.current.set(batchId, { total: toSubmit.length, archived: 0, warnings: 0, failed: 0, failedLocators: [], warningLocators: [] })
} }
// Capture options at call time before any state changes // Capture all submission data before any state changes
const capturedQuality = toSubmit.map(it => it.quality || 'best') const submissions = toSubmit.map(it => ({
locator: it.locator.trim(),
quality: it.playlistItems !== null ? null : (it.quality || 'best'),
extraExtensions: it.playlistItems !== null
? { per_item_quality: Object.fromEntries(it.playlistItems.map(pi => [pi.id, pi.quality])), sync: it.syncEnabled }
: {},
}))
// Reset form and close dialog immediately // Reset form and close dialog immediately
setItems([makeItem()]) setItems([makeItem()])
dialogRef.current?.close() dialogRef.current?.close()
// Submit each in background // Submit each in background
toSubmit.forEach((it, i) => submitBgJob(it.locator.trim(), capturedQuality[i], batchId)) submissions.forEach(({ locator, quality, extraExtensions }) =>
submitBgJob(locator, quality, batchId, extraExtensions)
)
} }
function addRow() { function addRow() {
@ -340,16 +443,16 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
function updateLocator(id, val) { function updateLocator(id, val) {
// Cancel any in-flight debounce and immediately clear stale probe results. // Cancel any in-flight debounce and immediately clear stale probe results.
// This prevents old qualities from being visible (and submittable) while // This prevents old qualities from being visible (and submittable) while
// the 600ms debounce is pending for the new URL. // the debounce is pending for the new URL.
clearTimeout(probeTimers.current.get(id)) clearTimeout(probeTimers.current.get(id))
setItems(prev => prev.map(it => setItems(prev => prev.map(it =>
it.id === id it.id === id
? { ...it, locator: val, probeState: 'idle', probeQualities: null, probeHasAudio: false, quality: 'best' } ? { ...it, locator: val, probeState: 'idle', probeQualities: null, probeHasAudio: false, quality: 'best',
playlistProbeState: 'idle', playlistInfo: null, playlistItems: null, playlistQuality: null, playlistExpanded: false }
: it : it
)) ))
if (!isVideoSource(val)) return if (isVideoSource(val)) {
// Schedule a fresh probe after the user stops typing // Schedule a fresh probe after the user stops typing
const timer = setTimeout(async () => { const timer = setTimeout(async () => {
probeTimers.current.delete(id) probeTimers.current.delete(id)
@ -372,13 +475,76 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
} }
}, 600) }, 600)
probeTimers.current.set(id, timer) probeTimers.current.set(id, timer)
} else if (isPlaylistSource(val)) {
// Schedule a playlist probe (playlists are slower 800ms debounce)
const timer = setTimeout(async () => {
probeTimers.current.delete(id)
setItems(prev => prev.map(it => it.id === id ? { ...it, playlistProbeState: 'probing' } : it))
try {
const result = await probePlaylist(archiveIdRef.current, val.trim())
setItems(prev => prev.map(it => {
if (it.id !== id || it.locator !== val) return it // stale locator changed again
return {
...it,
playlistProbeState: 'done',
playlistInfo: result,
playlistItems: result.items.map(pi => ({ ...pi, quality: null })),
playlistQuality: null,
}
}))
} catch {
setItems(prev => prev.map(it =>
it.id === id ? { ...it, playlistProbeState: 'error' } : it
))
}
}, 800)
probeTimers.current.set(id, timer)
}
} }
function updateQuality(id, val) { function updateQuality(id, val) {
setItems(prev => prev.map(it => it.id === id ? { ...it, quality: val } : it)) setItems(prev => prev.map(it => it.id === id ? { ...it, quality: val } : it))
} }
function updatePlaylistQuality(id, q) {
setItems(prev => prev.map(it => {
if (it.id !== id) return it
const newItems = applyPlaylistQuality(q, it.playlistItems)
return { ...it, playlistQuality: q, playlistItems: newItems }
}))
}
function updatePlaylistItemQuality(id, videoId, q) {
setItems(prev => prev.map(it => {
if (it.id !== id) return it
return { ...it, playlistItems: it.playlistItems.map(pi => pi.id === videoId ? { ...pi, quality: q } : pi) }
}))
}
function togglePlaylistExpanded(id) {
setItems(prev => prev.map(it => it.id === id ? { ...it, playlistExpanded: !it.playlistExpanded } : it))
}
function updateSync(id, val) {
setItems(prev => prev.map(it => it.id === id ? { ...it, syncEnabled: val } : it))
}
function deletePlaylistItem(itemId, videoId) {
setItems(prev => prev.map(it =>
it.id !== itemId ? it :
{ ...it, playlistItems: it.playlistItems.filter(pi => pi.id !== videoId) }
))
}
const pendingCount = items.filter(it => it.locator.trim()).length const pendingCount = items.filter(it => it.locator.trim()).length
const anyConflict = items.some(it => hasConflict(it))
// True if any playlist row has had all its videos deleted archive would be a no-op.
const anyEmptyPlaylist = items.some(it =>
Array.isArray(it.playlistItems) && it.playlistItems.length === 0
)
const anyProbing = items.some(it =>
it.probeState === 'probing' ||
// For playlist sources block unless probe completed successfully:
// idle = debounce not yet fired; probing = in flight; error = no quality data.
(isPlaylistSource(it.locator) && it.playlistProbeState !== 'done')
)
return ( return (
<dialog ref={dialogRef} className="capture-dialog"> <dialog ref={dialogRef} className="capture-dialog">
@ -407,6 +573,11 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
onQualityChange={val => updateQuality(item.id, val)} onQualityChange={val => updateQuality(item.id, val)}
onRemove={() => removeRow(item.id)} onRemove={() => removeRow(item.id)}
onSubmit={handleArchive} onSubmit={handleArchive}
onPlaylistQualityChange={q => updatePlaylistQuality(item.id, q)}
onPlaylistItemQualityChange={(vid, q) => updatePlaylistItemQuality(item.id, vid, q)}
onPlaylistToggle={() => togglePlaylistExpanded(item.id)}
onSyncChange={val => updateSync(item.id, val)}
onPlaylistItemDelete={(vid) => deletePlaylistItem(item.id, vid)}
/> />
))} ))}
</div> </div>
@ -530,7 +701,7 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
type="button" type="button"
className="capture-submit" className="capture-submit"
onClick={handleArchive} onClick={handleArchive}
disabled={pendingCount === 0} disabled={pendingCount === 0 || anyConflict || anyProbing || anyEmptyPlaylist}
> >
{pendingCount > 1 ? `Archive ${pendingCount}` : 'Archive'} {pendingCount > 1 ? `Archive ${pendingCount}` : 'Archive'}
</button> </button>
@ -543,7 +714,8 @@ export default function CaptureDialog({ open, archiveId, onClose, onCaptured, on
) )
} }
function CaptureRow({ item, autoFocus, onLocatorChange, onQualityChange, onRemove, onSubmit }) { function CaptureRow({ item, autoFocus, onLocatorChange, onQualityChange, onRemove, onSubmit,
onPlaylistQualityChange, onPlaylistItemQualityChange, onPlaylistToggle, onSyncChange, onPlaylistItemDelete }) {
const inputRef = useRef(null) const inputRef = useRef(null)
useEffect(() => { useEffect(() => {
@ -554,6 +726,47 @@ function CaptureRow({ item, autoFocus, onLocatorChange, onQualityChange, onRemov
// Quality control shown right of the input // Quality control shown right of the input
const qualityEl = (() => { const qualityEl = (() => {
// Playlist source handling
if (isPlaylistSource(item.locator)) {
if (item.playlistProbeState === 'probing') {
return <span className="capture-quality-probing" aria-label="Probing playlist…"></span>
}
if (item.playlistProbeState === 'done') {
const allHeights = [...new Set(
item.playlistItems.flatMap(pi => pi.qualities.map(q => parseInt(q)))
)].sort((a, b) => b - a)
const allHaveAudio = item.playlistItems.every(pi => pi.has_audio)
const conflictCount = item.playlistItems.filter(pi => pi.quality === null).length
return (
<>
<select
className="capture-quality"
value={item.playlistQuality ?? ''}
onChange={e => onPlaylistQualityChange(e.target.value)}
aria-label="Playlist quality"
>
{!item.playlistQuality && <option value="" disabled>Select quality</option>}
<option value="best">Best quality</option>
{allHeights.map(h => <option key={h} value={`${h}p`}>{h}p</option>)}
{allHaveAudio && <option value="audio">Audio only</option>}
</select>
{conflictCount > 0 && (
<span className="capture-conflict-badge">{conflictCount} need selection</span>
)}
</>
)
}
if (item.playlistProbeState === 'error') {
return (
<span className="capture-quality-hint capture-quality-hint--error">
Probe failed edit URL to retry
</span>
)
}
return null
}
// Video source handling (unchanged)
if (!isVideoSource(item.locator)) return null if (!isVideoSource(item.locator)) return null
if (item.probeState === 'probing') { if (item.probeState === 'probing') {
return <span className="capture-quality-probing" aria-label="Checking available qualities"></span> return <span className="capture-quality-probing" aria-label="Checking available qualities"></span>
@ -565,8 +778,6 @@ function CaptureRow({ item, autoFocus, onLocatorChange, onQualityChange, onRemov
return <span className="capture-quality-hint">No media detected</span> return <span className="capture-quality-hint">No media detected</span>
} }
if (qualities.length === 0 && hasAudio) { if (qualities.length === 0 && hasAudio) {
// Audio-only source: no video tracks, only audio available.
// Don't offer "Best quality" it would request a video format and fail.
return ( return (
<select <select
className="capture-quality" className="capture-quality"
@ -594,9 +805,29 @@ function CaptureRow({ item, autoFocus, onLocatorChange, onQualityChange, onRemov
return null // probeState === 'idle', debounce not yet fired return null // probeState === 'idle', debounce not yet fired
})() })()
const syncToggle = isPlaylistSource(item.locator) && item.playlistProbeState === 'done' ? (
<label className="capture-sync-row">
<input type="checkbox" checked={item.syncEnabled} onChange={e => onSyncChange(e.target.checked)} />
<span>Sync skip already-archived videos</span>
</label>
) : null
return ( return (
<div className="capture-row"> <div className="capture-row">
<div className="capture-row-main"> <div className="capture-row-main">
{isPlaylistSource(item.locator) ? (
item.playlistProbeState === 'done' ? (
<button
type="button"
className="capture-playlist-toggle capture-playlist-toggle--left"
onClick={onPlaylistToggle}
aria-label={item.playlistExpanded ? 'Collapse video list' : 'Expand video list'}
aria-expanded={item.playlistExpanded}
>
{item.playlistExpanded ? '▲' : '▼'}
</button>
) : null
) : null}
<input <input
ref={inputRef} ref={inputRef}
className="capture-input" className="capture-input"
@ -618,6 +849,41 @@ function CaptureRow({ item, autoFocus, onLocatorChange, onQualityChange, onRemov
{item.error && ( {item.error && (
<p className="capture-row-error">{item.error}</p> <p className="capture-row-error">{item.error}</p>
)} )}
{isPlaylistSource(item.locator) && item.playlistProbeState === 'done' && item.playlistExpanded ? (
<div className="capture-playlist-items">
{item.playlistItems.map(pi => (
<div
key={pi.id}
className={`capture-playlist-item${pi.quality === null ? ' capture-playlist-item--conflict' : ''}`}
>
<span className="capture-playlist-item-title">{pi.title || pi.url}</span>
<select
className="capture-item-quality"
value={pi.quality ?? ''}
onChange={e => onPlaylistItemQualityChange(pi.id, e.target.value)}
aria-label={`Quality for ${pi.title || pi.url}`}
>
{pi.quality === null && <option value="" disabled>Choose</option>}
<option value="best">Best quality</option>
{pi.qualities.map(q => <option key={q} value={q}>{q}</option>)}
{pi.has_audio && <option value="audio">Audio only</option>}
</select>
{pi.quality === null && (
<span className="capture-playlist-conflict-badge">Choose quality</span>
)}
<button
type="button"
className="capture-playlist-item-remove"
aria-label={`Remove ${pi.title || pi.url}`}
onClick={() => onPlaylistItemDelete(pi.id)}
>
&times;
</button>
</div>
))}
{syncToggle}
</div>
) : syncToggle}
</div> </div>
) )
} }

View file

@ -2,7 +2,7 @@ import SkeletonEntryRow from './SkeletonEntryRow';
import EntryRow from './EntryRow'; import EntryRow from './EntryRow';
export default function EntriesView({ entries, selectedUids, onRowClick, archiveId, pendingCaptures = [] }) { export default function EntriesView({ entries, selectedUids, onRowClick, archiveId, pendingCaptures = [], deletedUids }) {
return ( return (
<section id="archive-view" className="view is-active"> <section id="archive-view" className="view is-active">
<div className="entry-table"> <div className="entry-table">
@ -18,14 +18,17 @@ export default function EntriesView({ entries, selectedUids, onRowClick, archive
{pendingCaptures.filter(c => c.archiveId === archiveId).reverse().map(cap => ( {pendingCaptures.filter(c => c.archiveId === archiveId).reverse().map(cap => (
<SkeletonEntryRow key={cap.id} /> <SkeletonEntryRow key={cap.id} />
))} ))}
{entries.map(entry => ( {entries.map((entry, idx) => (
<EntryRow <EntryRow
key={entry.entry_uid} key={entry.entry_uid}
entry={entry} entry={entry}
rowIndex={idx}
archiveId={archiveId} archiveId={archiveId}
isSelected={selectedUids.size === 1 && selectedUids.has(entry.entry_uid)} isSelected={selectedUids.size === 1 && selectedUids.has(entry.entry_uid)}
isMultiSelected={selectedUids.size >= 2 && selectedUids.has(entry.entry_uid)} isMultiSelected={selectedUids.size >= 2 && selectedUids.has(entry.entry_uid)}
onRowClick={onRowClick} onRowClick={onRowClick}
selectedUids={selectedUids}
deletedUids={deletedUids}
/> />
))} ))}
</div> </div>

View file

@ -1,8 +1,51 @@
import { useState } from 'react'; import { useState } from 'react';
import { formatTimestamp, formatBytes, valueText, sourceIconSvg } from '../utils'; import { formatTimestamp, formatBytes, valueText, sourceIconSvg } from '../utils';
import { fetchEntryChildren } from '../api';
export default function EntryRow({ entry, archiveId, isSelected, isMultiSelected, onRowClick }) { function ChildRow({ entry, index, onRowClick, selectedUids }) {
const isSelected = (selectedUids?.size === 1) && selectedUids.has(entry.entry_uid);
const isMultiSelected = (selectedUids?.size >= 2) && selectedUids.has(entry.entry_uid);
const cls = ['child-entry-row',
index % 2 === 0 ? 'child-entry-row--light' : 'child-entry-row--dark',
isSelected && 'is-selected',
isMultiSelected && 'is-multi-selected',
].filter(Boolean).join(' ');
return (
<div
className={cls}
tabIndex={0}
data-entry-uid={entry.entry_uid}
onMouseDown={e => { if (e.shiftKey) e.preventDefault(); }}
onClick={e => onRowClick(entry, e)}
onKeyDown={e => { if (e.key === 'Enter') onRowClick(entry, e); }}
>
<div className="col-check" aria-hidden="true" />
<div className="col-added">{formatTimestamp(entry.archived_at)}</div>
<div className="col-title">
<span className="source-icon">
<span dangerouslySetInnerHTML={{ __html: sourceIconSvg(entry.source_kind) }} />
</span>
<span className="entry-title">{valueText(entry.title) || valueText(entry.entry_uid)}</span>
</div>
<div className="col-type">
<span className="type-pill">{valueText(entry.entity_kind)}</span>
</div>
<div className="col-size">
<span className="size-total">{formatBytes(entry.total_artifact_bytes)}</span>
</div>
<div className="url-cell col-url">{valueText(entry.original_url)}</div>
</div>
);
}
export default function EntryRow({ entry, archiveId, rowIndex, isSelected, isMultiSelected, onRowClick, selectedUids, deletedUids }) {
const [favFailed, setFavFailed] = useState(false); const [favFailed, setFavFailed] = useState(false);
const [expanded, setExpanded] = useState(false);
const [children, setChildren] = useState(null);
const [childrenLoading, setChildrenLoading] = useState(false);
const showFavicon = const showFavicon =
entry.source_kind === 'web' && entry.source_kind === 'web' &&
entry.entity_kind === 'page' && entry.entity_kind === 'page' &&
@ -24,21 +67,48 @@ export default function EntryRow({ entry, archiveId, isSelected, isMultiSelected
); );
const checked = isSelected || isMultiSelected; const checked = isSelected || isMultiSelected;
const hasChildren = entry.child_count > 0;
function handleCheckboxClick(e) { function handleCheckboxClick(e) {
e.stopPropagation(); e.stopPropagation();
// treat checkbox tap as ctrl+click: toggle this entry without clearing others
onRowClick(entry, { ctrlKey: true, metaKey: false, shiftKey: false, preventDefault() {} }); onRowClick(entry, { ctrlKey: true, metaKey: false, shiftKey: false, preventDefault() {} });
} }
async function handleExpandClick(e) {
e.stopPropagation();
if (expanded) {
setExpanded(false);
return;
}
setExpanded(true);
if (children === null && !childrenLoading) {
setChildrenLoading(true);
try {
const result = await fetchEntryChildren(archiveId, entry.entry_uid);
setChildren(result);
} catch (_) {
setChildren([]);
} finally {
setChildrenLoading(false);
}
}
}
const outerClass = [
'entry-row-outer',
rowIndex % 2 === 0 ? 'entry-row-outer--light' : 'entry-row-outer--dark',
isSelected && 'is-selected',
isMultiSelected && 'is-multi-selected',
].filter(Boolean).join(' ');
return ( return (
<div className={outerClass} data-entry-uid={entry.entry_uid}>
<div <div
className={[isSelected && 'is-selected', isMultiSelected && 'is-multi-selected'].filter(Boolean).join(' ') || undefined} className="entry-row-main"
tabIndex={0} tabIndex={0}
data-entry-uid={entry.entry_uid} onMouseDown={e => { if (e.shiftKey) e.preventDefault(); }}
onMouseDown={e => { if (e.shiftKey) e.preventDefault() }}
onClick={e => onRowClick(entry, e)} onClick={e => onRowClick(entry, e)}
onKeyDown={e => { if (e.key === 'Enter') onRowClick(entry, e) }} onKeyDown={e => { if (e.key === 'Enter') onRowClick(entry, e); }}
> >
<div className="col-check"> <div className="col-check">
<button <button
@ -52,21 +122,53 @@ export default function EntryRow({ entry, archiveId, isSelected, isMultiSelected
</div> </div>
<div className="col-added">{formatTimestamp(entry.archived_at)}</div> <div className="col-added">{formatTimestamp(entry.archived_at)}</div>
<div className="col-title"> <div className="col-title">
{hasChildren && (
<button
type="button"
className={`entry-expand-btn${expanded ? ' is-expanded' : ''}`}
aria-label={expanded ? 'Collapse children' : `Expand ${entry.child_count} items`}
aria-expanded={expanded}
onClick={handleExpandClick}
onKeyDown={e => e.stopPropagation()}
/>
)}
<span className="source-icon">{icon}</span> <span className="source-icon">{icon}</span>
<span className="entry-title">{valueText(entry.title) || valueText(entry.entry_uid)}</span> <span className="entry-title">{valueText(entry.title) || valueText(entry.entry_uid)}</span>
{hasChildren && (
<span className="child-count-badge" aria-hidden="true">{entry.child_count}</span>
)}
</div> </div>
<div className="col-type"> <div className="col-type">
<span className="type-pill">{valueText(entry.entity_kind)}</span> <span className="type-pill">{valueText(entry.entity_kind)}</span>
</div> </div>
<div className="col-size"> <div className="col-size">
<span className="size-total">{formatBytes(entry.total_artifact_bytes)}</span> <span className="size-total">{formatBytes(entry.total_artifact_bytes)}</span>
{entry.cached_bytes > 0 && entry.total_artifact_bytes > 0 && ( {entry.cached_bytes > 0 && entry.cacheable_bytes > 0 && (
<span className="size-cached-pct" title={`${formatBytes(entry.cached_bytes)} already on disk from an earlier entry`}> <span className="size-cached-pct" title={`${formatBytes(entry.cached_bytes)} already on disk from an earlier entry`}>
{Math.round(entry.cached_bytes / entry.total_artifact_bytes * 100)}% cached {Math.round(entry.cached_bytes / entry.cacheable_bytes * 100)}% cached
</span> </span>
)} )}
</div> </div>
<div className="url-cell col-url">{valueText(entry.original_url)}</div> <div className="url-cell col-url">{valueText(entry.original_url)}</div>
</div> </div>
{expanded && (
<>
{childrenLoading && <div className="child-entries-loading">Loading</div>}
<div className="child-entries" aria-label={`${entry.child_count} child entries`}>
{children && children
.filter(c => !deletedUids?.has(c.entry_uid))
.map((child, idx) => (
<ChildRow
key={child.entry_uid}
entry={child}
index={idx}
onRowClick={onRowClick}
selectedUids={selectedUids}
/>
))}
</div>
</>
)}
</div>
); );
} }

View file

@ -299,17 +299,19 @@ select {
border-bottom: 1px solid var(--line-soft); border-bottom: 1px solid var(--line-soft);
} }
#entries-body > div > div { padding: 7px 10px; flex-shrink: 0; overflow: hidden; } #entries-body > div > div { padding: 7px 10px; flex-shrink: 0; overflow: hidden; }
#entries-body > div:nth-child(even) { background: #f2ede5; } /* Skeleton rows (no index class) fall back to nth-child; real rows use explicit classes. */
#entries-body > div:nth-child(odd) { background: var(--paper-3); } #entries-body > div:not(.entry-row-outer):nth-child(even) { background: #f2ede5; }
#entries-body > div:not(.entry-row-outer):nth-child(odd) { background: var(--paper-3); }
/* Index-based stripes for real entry rows — immune to skeleton sibling count. */
#entries-body > .entry-row-outer--light { background: var(--paper-3); }
#entries-body > .entry-row-outer--dark { background: #f2ede5; }
#entries-body > div.is-selected { #entries-body > div.is-selected {
background: #eee2d2; background: #eee2d2;
outline: 2px solid var(--accent); box-shadow: inset 0 0 0 2px var(--accent);
outline-offset: -2px;
} }
#entries-body > div.is-multi-selected { #entries-body > div.is-multi-selected {
background: #eee2d2; background: #eee2d2;
outline: 2px solid var(--accent); box-shadow: inset 0 0 0 2px var(--accent);
outline-offset: -2px;
} }
.col-added { width: 162px; color: var(--muted); } .col-added { width: 162px; color: var(--muted); }
@ -339,8 +341,7 @@ select {
.source-icon > * { width: 100%; height: 100%; display: flex; align-items: center; justify-content: center; } .source-icon > * { width: 100%; height: 100%; display: flex; align-items: center; justify-content: center; }
.source-icon svg { width: 100%; height: 100%; } .source-icon svg { width: 100%; height: 100%; }
.url-cell { color: #555b55; white-space: nowrap; text-overflow: ellipsis; word-break: break-all; } .url-cell { color: #555b55; white-space: nowrap; text-overflow: ellipsis; word-break: break-all; }
#entries-body .url-cell:hover, #entries-body .url-cell:hover { overflow: visible; white-space: normal; }
#entries-body .is-selected .url-cell { overflow: visible; white-space: normal; }
.type-pill { display: inline-block; padding: 2px 6px; background: #d8e3df; color: #275a5f; border: 1px solid #bfd0ca; border-radius: var(--r); } .type-pill { display: inline-block; padding: 2px 6px; background: #d8e3df; color: #275a5f; border: 1px solid #bfd0ca; border-radius: var(--r); }
/* ── Multi-select checkbox column ───────────────────────────────────────── */ /* ── Multi-select checkbox column ───────────────────────────────────────── */
@ -890,6 +891,7 @@ select {
color: var(--muted); color: var(--muted);
font-style: italic; font-style: italic;
} }
.capture-quality-hint--error { color: #c05000; font-style: normal; }
/* Status dot */ /* Status dot */
.cap-dot { .cap-dot {
@ -2726,3 +2728,254 @@ body.has-audio-bar { padding-bottom: 56px; }
@media (pointer: coarse) { @media (pointer: coarse) {
.skeleton-row .col-added { padding-left: 10px; } .skeleton-row .col-added { padding-left: 10px; }
} }
/* ── Child entry expansion ───────────────────────────────────────────────── */
/* Outer wrapper: one block per entry-group so nth-child stays correct.
#entries-body > .entry-row-outer beats #entries-body > div on specificity
(id + class > id + element) so display:flex is safely overridden. */
#entries-body > .entry-row-outer {
display: block;
border-bottom: none;
}
/* Inner flex row: replicates the #entries-body > div row behaviour. */
#entries-body > .entry-row-outer > .entry-row-main {
display: flex;
align-items: center;
cursor: default;
border-bottom: 1px solid var(--line-soft);
/* Reset padding that #entries-body > div > div would otherwise apply. */
padding: 0;
flex-shrink: unset;
overflow: visible;
}
/* Column cells inside the inner row. */
#entries-body > .entry-row-outer > .entry-row-main > div {
padding: 7px 10px;
flex-shrink: 0;
overflow: hidden;
}
#entries-body > .entry-row-outer > .entry-row-main > div:last-child { padding-right: 22px; }
#entries-body > .entry-row-outer > .entry-row-main .col-added { padding-left: 22px; }
/* Selection: class lives on outer wrapper; background + stroke scoped to inner row
so expanded child entries don't inherit the selection highlight. */
#entries-body > .entry-row-outer.is-selected { background: unset; outline: none; box-shadow: none; }
#entries-body > .entry-row-outer.is-multi-selected { background: unset; outline: none; box-shadow: none; }
#entries-body > .entry-row-outer.is-selected > .entry-row-main {
background: #eee2d2;
box-shadow: inset 0 0 0 2px var(--accent);
}
#entries-body > .entry-row-outer.is-multi-selected > .entry-row-main {
background: #eee2d2;
box-shadow: inset 0 0 0 2px var(--accent);
}
/* URL overflow on hover only — explicit pointer, not selection. */
#entries-body > .entry-row-outer .url-cell:hover { overflow: visible; white-space: normal; }
/* child-entries container: reset the padding that #entries-body > div > div applies. */
#entries-body > .entry-row-outer > .child-entries {
display: block;
padding: 0;
flex-shrink: unset;
overflow: visible;
border-left: 2px solid var(--line-soft);
margin-left: 32px;
}
/* Each child row reuses the same .col-* flex widths as normal rows. */
.child-entry-row {
display: flex;
align-items: center;
cursor: default;
border-bottom: 1px solid var(--line-soft);
opacity: 0.88;
}
.child-entry-row:hover { opacity: 1; }
.child-entry-row:last-child { border-bottom: none; }
.child-entry-row--light { background: #fafaf8; }
.child-entry-row--dark { background: #f2f0ec; }
.child-entry-row.is-selected {
background: #eee2d2;
box-shadow: inset 0 0 0 2px var(--accent);
opacity: 1;
}
.child-entry-row.is-multi-selected {
background: #eee2d2;
box-shadow: inset 0 0 0 2px var(--accent);
opacity: 1;
}
.child-entry-row > div {
padding: 6px 10px;
flex-shrink: 0;
overflow: hidden;
}
.child-entry-row .col-added { padding-left: 22px; }
.child-entry-row > div:last-child { padding-right: 22px; }
/* Expand chevron button */
.entry-expand-btn {
display: inline-flex;
align-items: center;
justify-content: center;
width: 16px;
height: 16px;
flex-shrink: 0;
padding: 0;
margin-right: 2px;
background: none;
border: none;
cursor: pointer;
opacity: 0.45;
color: inherit;
transition: opacity 0.15s;
}
.entry-expand-btn:focus { outline: none; }
.entry-expand-btn:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; border-radius: 2px; }
.entry-expand-btn::before {
content: '';
display: block;
width: 0;
height: 0;
border-top: 4px solid transparent;
border-bottom: 4px solid transparent;
border-left: 6px solid currentColor;
transition: transform 0.15s;
}
.entry-expand-btn:hover { opacity: 1; }
.entry-expand-btn.is-expanded::before { transform: rotate(90deg); }
/* Child count badge next to title */
.child-count-badge {
display: inline-block;
margin-left: 5px;
padding: 0 5px;
font-size: 0.72em;
font-weight: 600;
background: color-mix(in srgb, var(--line-soft) 60%, transparent);
border-radius: 10px;
opacity: 0.75;
vertical-align: middle;
line-height: 1.6;
}
/* Loading placeholder inside child-entries */
.child-entries-loading {
padding: 8px 12px;
font-size: 0.85em;
opacity: 0.55;
}
/* ── Playlist quality expansion ──────────────────────────── */
.capture-playlist-toggle {
display: inline-flex;
align-items: center;
gap: 4px;
font-size: 0.75rem;
color: var(--muted);
background: none;
border: none;
cursor: pointer;
padding: 2px 6px;
}
.capture-playlist-toggle:hover { color: var(--text); }
.capture-playlist-toggle--left {
flex-shrink: 0;
padding: 2px 4px;
font-size: 0.7rem;
}
.capture-conflict-badge {
font-size: 0.7rem;
color: #c07000;
background: #fff3cd;
border: 1px solid #e0a000;
border-radius: 4px;
padding: 1px 6px;
margin-left: 6px;
}
.capture-playlist-items {
border-top: 1px solid var(--line-soft);
max-height: 320px;
overflow-y: auto;
scrollbar-width: thin;
scrollbar-color: var(--line) transparent;
}
.capture-playlist-item {
display: flex;
align-items: center;
gap: 8px;
padding: 5px 10px 5px 14px;
font-size: 0.8rem;
border-bottom: 1px solid var(--line-soft);
border-left: 3px solid transparent;
}
.capture-playlist-item:last-of-type { border-bottom: none; }
.capture-playlist-item--conflict {
border-left-color: #c07000;
background: color-mix(in srgb, #c07000 6%, transparent);
}
.capture-playlist-item-title {
flex: 1;
min-width: 0;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
color: var(--text);
opacity: 0.9;
}
.capture-item-quality {
font-size: 0.73rem;
padding: 2px 4px;
border: 1px solid var(--line);
border-radius: 3px;
background: var(--paper-3);
color: var(--text);
flex-shrink: 0;
}
.capture-playlist-conflict-badge {
font-size: 0.68rem;
font-weight: 600;
color: #c07000;
white-space: nowrap;
flex-shrink: 0;
}
.capture-playlist-item-remove {
flex-shrink: 0;
display: inline-flex;
align-items: center;
justify-content: center;
width: 18px;
height: 18px;
padding: 0;
border: none;
background: none;
color: var(--muted);
cursor: pointer;
border-radius: 3px;
opacity: 0.3;
transition: opacity 0.1s, color 0.1s;
}
.capture-playlist-item:hover .capture-playlist-item-remove,
.capture-playlist-item-remove:focus-visible { opacity: 1; }
.capture-playlist-item-remove:hover { color: #c04000; opacity: 1; }
@media (pointer: coarse) { .capture-playlist-item-remove { opacity: 1; } }
.capture-playlist-item-remove { font-size: 0.75rem; line-height: 1; }
.capture-sync-row {
display: flex;
align-items: center;
gap: 6px;
padding: 6px 12px;
font-size: 0.78rem;
color: var(--muted);
border-top: 1px solid var(--line-soft);
cursor: pointer;
}
.capture-sync-row input[type=checkbox] { cursor: pointer; }