thegeneralist01/archivr
1
Fork 0
mirror of https://github.com/thegeneralist01/archivr synced 2026-07-22 11:15:41 +02:00
Code Issues Activity

feat: add capturing w/ parent-child entries (#32)

Browse source 
* feat(core): YouTube playlist/channel/YTM-playlist capture with parent–child entries

- ytdlp: add fetch_playlist_info() using yt-dlp -J --flat-playlist for
  reliable container title + shallow entry list; normalize item URLs via
  webpage_url → absolute url → id fallback (domain inferred from container
  URL so YTM stays on music.youtube.com)
- capture: add record_container_entry() (no blob, no primary_media artifact);
  extend record_media_entry() with parent_entry_id/root_entry_id params (all
  existing single-item call sites pass None, None)
- capture: implement YouTubePlaylist / YouTubeChannel / YouTubeMusicPlaylist
  capture path replacing the two not-implemented stubs: fetch playlist info →
  create container entry (reusing existing run + item) → per-child run items
  (parent_item_id = container item) → download each video/track as a child
  entry; per-child failures are non-fatal; perform_capture returns result.status
  reflecting actual run outcome so capture_handler marks the job correctly
- archive: add child_count i64 to EntrySummary (col 12 in all listing queries);
  add get_entry_summary() private helper; fix get_entry_detail() to use
  get_entry_summary() so child entries are resolvable via the detail endpoint;
  add list_child_entries(conn, uid, caller_bits) with the same
  admin/collection visibility predicate as list_root_entries

archive_runs.requested_count stays 1 (one user locator); discovered/
completed/failed_count reflect container item + N video items via
refresh_run_counters.

* feat(server,frontend): expose children endpoint + expand UI for container entries

server:
- add GET /api/archives/:id/entries/:uid/children → list_entry_children,
  calling list_child_entries with caller_bits so visibility model is enforced
- fix capture_handler: use result.status ("completed"/"failed") to set job
  status rather than always "completed", mirroring rearchive_handler; this
  surfaces partial playlist failures to the polling client

frontend:
- api.js: add fetchEntryChildren(archiveId, entryUid)
- EntryRow: one outer div.entry-row-outer (display:block) keeps nth-child
  striping correct; inner div.entry-row-main is the flex row with all column
  cells and event handling; .child-entries sits below inside the outer wrapper
- expand chevron appears when entry.child_count > 0; clicking fetches children
  lazily and renders ChildRow components reusing .col-* flex widths
- child-count badge shown next to title on container entries
- styles.css: scoped CSS with #entries-body > .entry-row-outer selectors
  (higher specificity than > div) to override flex on outer wrapper; inner row
  and column rules replicated at correct depth; nth-child, is-selected,
  is-multi-selected, url-cell hover all handled

* fix(frontend): make child entry rows interactive

ChildRow now receives onRowClick and selectedUids from EntryRow (which
receives selectedUids from EntriesView alongside the existing booleans).
Clicking a child row invokes onRowClick(child, e) so it flows through
handleRowClick → selectEntry → fetchEntryDetail exactly as a root entry
would. Shift-range selection gracefully degrades to single-select since
child entries are not in the root entries array.

Selected/multi-selected visual state is wired: .child-entry-row.is-selected
shows the same #eee2d2 background + accent outline as root rows; hover
restores full opacity. Frontend static assets rebuilt.

* feat(core): playlist per-item quality + incremental sync

ytdlp.rs:
- Add PlaylistItemProbe / PlaylistProbeResult (pub, serde::Serialize)
- Add private available_video_heights_from_value() helper for Value entries
- Add probe_playlist_qualities(): yt-dlp -J (full metadata, no flat flag)
  returns per-video quality lists in one subprocess call

capture.rs:
- Add per_item_quality: HashMap<String,String> and sync: bool to CaptureConfig
  (both Default; keyed by yt-dlp video ID, not URL)
- Add pub locator_to_playlist_url(): validates only the three playlist sources,
  expands shorthands; keeps locator_to_ytdlp_url's no-playlists contract
- Playlist capture block: sync-aware container resolution
  - sync + existing container → reuse it via complete_archive_run_item,
    skip already-archived children (by canonical URL) before creating
    run items so refresh_run_counters only counts new items
  - sync + no container → create normally (first sync run)
  - non-sync → always create fresh container (existing behaviour)
- Per-item quality: config.per_item_quality.get(id) falls back to child_quality

archive.rs:
- Add get_archived_playlist_child_urls(): returns HashSet of canonical URLs
  of all children under any container matching the playlist canonical URL
- Add find_container_entry_id_by_canonical_url(): returns most-recent
  container entry id (parent_entry_id IS NULL) for a given canonical URL

routes.rs: stub per_item_quality/sync on both CaptureConfig sites (server
agent will wire body fields in Phase 2)

* fix(core)+test: propagate sync query errors; cover new playlist/sync functions

archive.rs:
- get_archived_playlist_child_urls: collect() as rusqlite::Result<HashSet<_>>
  instead of filter_map(ok) so row-level errors surface rather than silently
  skipping and causing duplicate downloads

capture.rs:
- match get_archived_playlist_child_urls result and fail_run on error instead
  of unwrap_or_default, preventing silent re-downloads on DB failure

Tests added to capture.rs:
- locator_to_playlist_url_accepts_playlist_shorthands (yt:playlist/, ytm:playlist/, full URL)
- locator_to_playlist_url_accepts_channel_shorthands (yt:@handle)
- locator_to_playlist_url_rejects_non_playlist_sources (single video, tweet, web page)

Tests added to archive.rs (all use in-memory DB via make_tag_test_db):
- find_container_entry_id_returns_none_when_absent
- find_container_entry_id_returns_root_entry
- find_container_entry_id_ignores_child_entries (child with parent_entry_id set)
- get_archived_playlist_child_urls_empty_when_no_playlist
- get_archived_playlist_child_urls_returns_children
- get_archived_playlist_child_urls_excludes_other_playlists

* feat(server,frontend): playlist quality selector + per-video overrides + sync UI

routes.rs:
- CaptureBody gains per_item_quality (HashMap<String,String>, serde(default))
  and sync (bool, serde(default)); both validated before use
  - per_item_quality values validated against same quality predicate as top-level
    quality field ("best"|"audio"|"NNNp") so bad per-video values are rejected
    at the API boundary rather than silently falling through to quality_format
- capture_handler threads body.per_item_quality + body.sync into CaptureConfig
  (replaces hardcoded empty stubs); rearchive_handler keeps empty defaults
- New POST /api/archives/:id/captures/probe-playlist: calls
  probe_playlist_qualities via spawn_blocking; 400 for non-playlist locator,
  502 on yt-dlp failure, returns PlaylistProbeResult as JSON

api.js:
- probePlaylist(archiveId, locator): POST probe-playlist endpoint
- submitCapture: forwards per_item_quality (non-empty) and sync:true from
  extraExtensions param added to submitBgJob

CaptureDialog.jsx:
- isPlaylistSource(): detects yt:/youtube: playlist/@/channel, ytm:playlist/,
  YouTube/YTM HTTP(S) URLs with list= param or channel pathnames
- makeItem(): 6 new playlist state fields
- applyPlaylistQuality(): conflict logic — videos that can reach selected
  quality get it set; videos that can't and have no prior selection are left
  null (conflict); videos with a prior selection keep it when quality is raised
- hasConflict(): any playlistItems entry with quality===null
- updateLocator(): isPlaylistSource branch with 800ms debounce→probePlaylist;
  existing isVideoSource path unchanged
- Archive button disabled when anyConflict or any probe in flight
- Per-video expand list with individual quality selects, conflict badges,
  sync toggle (appears after probe completes)

styles.css: playlist expansion, conflict, sync toggle CSS

* fix(core): ignore per_item_quality for YTM playlist items

YouTube Music playlists force child_quality = Some("audio") because
yt-dlp can't download DRM-free audio-only tracks any other way. The
previous per_item_quality lookup could override this with e.g. "best",
defeating the invariant. Guard the lookup behind !is_audio so YTM items
are always downloaded as audio regardless of what the caller sends.

* fix(frontend): exclude /watch from isPlaylistSource

youtube.com/watch?v=...&list=... and music.youtube.com/watch are single
videos in the backend (Source::YouTubeVideo / YouTubeMusicTrack) regardless
of the list param. Previously isPlaylistSource returned true for these,
which would have triggered the playlist probe path while the video probe
was already running, and the render would attempt to show playlist UI on
an item whose playlistProbeState stays idle.

Guard: if pathname === '/watch', return false before the list-param check.

* fix(frontend): tighten isPlaylistSource to mirror backend routing exactly

Previous fix excluded /watch but still returned true for any youtube.com
URL with a ?list= param (e.g. /shorts/xxx?list=yyy). Backend determine_source
only routes to YouTubePlaylist on /playlist?list=... and to YouTubeChannel on
/@handle, /channel/, /c/, /user/ paths — everything else is a single item.

Rewrite the HTTP block to match:
- youtube.com: pathname==='/playlist' && list param → playlist
             : /@, /channel/, /c/, /user/ → channel
             : anything else (incl. /watch&list=, /shorts?list=) → false
- music.youtube.com: pathname==='/playlist' && list param → YTM playlist
                   : /watch → single track (falls through to false)

* fix(frontend): guard handleArchive against Enter-key bypass of disabled state

The Archive button is disabled when anyConflict || anyProbing, but
onKeyDown on the locator input calls onSubmit() → handleArchive()
directly, bypassing the button's disabled check entirely.

Add the same conditions as early returns inside handleArchive itself,
operating on toSubmit (the items that would actually be submitted) so
the guard is tight — items with no locator are already excluded by the
toSubmit filter.

* fix(frontend): drop m.youtube.com from isPlaylistSource

Backend determine_source playlist/channel regex only matches
(?:www\.)?youtube\.com — mobile URLs hitting m.youtube.com would be
probed as playlist in the UI but captured as Source::Url server-side.
Remove m.youtube.com from the detector to keep frontend and backend
in exact agreement. Add backend support when needed.

* fix(frontend): audio-only conflict handling in playlist quality selector

applyPlaylistQuality('audio'):
- Only sets quality='audio' on items where has_audio=true
- Items with has_audio=false: keep prior selection if set, else null
  (conflict) — same rule as unsupported height, blocks archive until
  user explicitly picks a quality for those items

Playlist-level 'Audio only' option:
- Changed hasAnyAudio → allHaveAudio (every item must have audio)
- When any item lacks audio, the option is hidden entirely so the
  selector can never create immediate conflicts just by appearing

* fix(frontend): add yt:user/ to isPlaylistSource shorthand detection

Backend determine_source routes yt:user/... (and youtube:user/...) to
YouTubeChannel — already covered by the yt: shorthand block for
playlist/, @, channel/, c/ but missing user/. Old-style user channel
URLs would capture correctly server-side but never show the playlist
quality/sync UI.

* fix(frontend): block playlist submission unless probe is done

Previous guard only blocked while playlistProbeState==='probing'.
Two remaining bypass paths:
- idle: 800ms debounce not yet fired after URL typed
- error: probe failed — no per-video quality data available

Change anyProbing and handleArchive guard to:
  isPlaylistSource(locator) && playlistProbeState !== 'done'

This means idle/probing/error all block submission for playlist items.
error is intentionally blocking — without quality data the per-video
requirement can't be satisfied; user must retry or remove the URL.

* fix(frontend): accurate error message when playlist probe fails

Previous text said 'using best quality' implying the capture would
proceed, but probe error now blocks submission. Replace with 'Probe
failed — edit URL to retry' in orange (capture-quality-hint--error)
so the disabled button and the message are consistent.

* fix(frontend): exact quality match in applyPlaylistQuality

Replace maxHeight >= newHeight (cap check) with item.qualities.includes(newQ)
(exact match). A video with [2160p, 1080p] does not support 1440p; the
previous logic would mark it as supporting any quality up to 2160p and
submit '1440p' which yt-dlp silently downloads as 1080p — misrepresenting
the selected quality.

With exact match, unsupported qualities correctly fall through to the
conflict path (keep prior selection or null), enforcing the same manual-
choice requirement as any other unsupported quality.

Per-row selects are unaffected: they already render only pi.qualities
(the video's actual available formats), no maxHeight logic involved.

* fix(frontend): move playlist expand chevron to left of input

User asked for the chevron to be on the left of the playlist input,
not tucked after the quality selector on the right.

- Remove chevron from qualityEl (it was between the quality select and
  the remove button)
- Add it as the first child of capture-row-main, before the <input>,
  when isPlaylistSource && playlistProbeState === 'done'
- Show a same-width placeholder span while probing/idle/error so the
  input does not jump left when the chevron appears after probe completes
- Add capture-playlist-toggle--left modifier (flex-shrink:0, tighter
  padding) and .capture-playlist-toggle-placeholder (fixed 22px width)

* doc(server): scope per_item_quality guarantee to the UI

The server validates per_item_quality value shapes but does not enforce
that every playlist item has an entry. Items without an override get
the global quality as a yt-dlp cap with graceful fallback.

The 'must choose quality for unsupported videos' invariant is a UI
constraint enforced by the frontend before submission. A direct API
caller bypassing the UI accepts yt-dlp's standard cap-and-fallback
behavior. Document this scope explicitly so the gap is intentional,
not accidental.

* fix(frontend): prevent 'reading some of undefined' crash from stale sessionStorage

Old captureItems entries saved before the playlist fields were added
have playlistItems=undefined (missing key). The hasConflict guard
checked !== null, which undefined passes, then called .some() on
undefined → TypeError.

Two-part fix:
1. sessionStorage restore: merge each saved item over makeItem() defaults
   so any missing fields (playlistItems, playlistProbeState, etc.) are
   filled with their correct initial values before the item is used
2. hasConflict: use Array.isArray() instead of !== null so undefined
   is also safely rejected — defence-in-depth for any future field gap

* fix(core): playlist total size includes children; per_item_quality as include-set

archive.rs: total_artifact_bytes for root entries now adds a correlated
subquery summing children's blob bytes so playlist/channel containers
show the real download size instead of 0.

capture.rs: non-empty per_item_quality map now acts as an include-set —
items whose yt-dlp ID is absent are skipped entirely. This wires the
UI's per-video delete button to actual capture exclusion. Empty map
preserves the existing behaviour (download everything).

routes.rs + capture.rs doc: comments updated to reflect both semantics
(empty = all / non-empty = only listed IDs) accurately.

* fix(frontend): QA fixes — playlist UX, selection stroke, URL expand

CaptureDialog.jsx:
- Placeholder no longer appears on idle playlist rows (only during
  probing); chevron shows only after probe completes — no left-padding
  while the user is still typing
- Per-video delete button added to expanded playlist list; removes item
  from playlistItems so its ID is absent from per_item_quality on submit
- anyEmptyPlaylist guard: Archive button disabled + handleArchive early-
  return when all videos have been deleted (empty map would otherwise
  silently download everything)

styles.css:
- Selection stroke switched from outline to box-shadow:inset everywhere
  (entry-row-outer, child-entry-row, legacy flat-div selector) —
  guaranteed inside element bounds, no layout interference
- URL cell overflow only on :hover; removed is-selected .url-cell rule
  that was expanding child URLs when their parent was selected
- Playlist items redesign: separator lines instead of background fills,
  amber left-border for conflicts, thin scrollbar, tighter padding
- Remove button: opacity 0.3 always-visible baseline; full opacity on
  hover/:focus-visible; forced to 1 on coarse-pointer (touch) devices

* fix(frontend): no left gap on playlist rows until chevron exists

Remove the probing-state placeholder span entirely. The chevron renders
only when playlistProbeState === 'done'; all other states (idle, probing,
error) render null. The small layout shift when the chevron appears after
probe is acceptable; blank padding while there is no chevron is not.

* fix(frontend): clear box-shadow on entry-row-outer to prevent double stroke

The flat selector '#entries-body > div.is-selected' already applies
box-shadow to .entry-row-outer (it IS a direct child div). The outer
override rule only cleared 'outline', so the box-shadow leaked through,
wrapping the entire parent+children block with a second stroke.

Add box-shadow: none to both .is-selected and .is-multi-selected on
.entry-row-outer so the stroke sits only on .entry-row-main.

* docs: document per-video exclude in README YouTube playlists section

* fix(frontend): scope selection background to entry-row-main only

Moving background:#eee2d2 off .entry-row-outer onto > .entry-row-main
so that expanded child entries don't inherit the selection highlight.

Outer wrapper now explicitly unsets background (cancelling the flat-div
cascade rule) and clears outline+box-shadow. Both the selection colour
and the inset stroke live on .entry-row-main only.

* fix(frontend): alternating stripe backgrounds on child entry rows

Child rows were inheriting the parent entry's stripe color, making the
expanded list look like one flat block. Apply the same odd/even palette
as root entries (var(--paper-3) / #f2ede5) so each video row is visually
distinct within the expanded group.

* fix(core): delete_entry correctly nulls FK for child entries

archive_run_items.produced_entry_id has no ON DELETE action so it must
be manually nulled before the entry row is deleted. The old query used
WHERE root_entry_id = entry_id, which finds descendants of a root but
returns nothing when entry_id IS a child (children have no sub-children,
so no row has root_entry_id = child_id). The DELETE then failed under
foreign_keys=ON.

Fix: add OR produced_entry_id = ?1 so the entry's own run_item FK is
always cleared before deletion, regardless of whether it is a root or a
child. The subtree subquery is kept for the root-deletion case where all
child run_items also need nulling.

* fix(core+frontend): child entry selection and delete correctness

database.rs — delete_entry:
- subtree_ids now uses WHERE id = ?1 OR root_entry_id = ?1 so the entry
  itself is always included; previously a child deletion passed an empty
  vec to cascade_cached_bytes_after_subtree_delete (no grandchildren
  exist), leaving cached_bytes stale on entries sharing that child's blobs

App.jsx — handleRowClick:
- Shift-range now queries DOM order (#entries-body [data-entry-uid])
  instead of entries.findIndex(); child rows are in the DOM but not in
  the root entries array, so findIndex always returned -1 for them
- Ctrl/meta branch computes next set before the state update so
  selectEntry can fire synchronously for child rows; auto-snap only
  restores root entries, so ctrl-clicking a lone child never loaded its
  detail panel — now calls selectEntry(entry) when the child ends up as
  the sole selection, selectEntry(null) on multi or deselect
- selectedUids added to handleRowClick's useCallback dep array

* fix(frontend): resolve detail entry for any remaining child after ctrl-deselect

Add entryCacheRef (uid→entry Map) populated on every row click. When
ctrl/meta-deselecting leaves exactly one other entry selected, look up
the remaining UID in the cache before falling back to the root entries
array. Without this, deselecting from a multi-selection where the
remaining entry is a child row left selectedEntry null (auto-snap only
searches root entries).

* fix(frontend): child row stripes, deleted-child visibility, selection completeness

EntryRow.jsx / ChildRow:
- Index-based light/dark classes (child-entry-row--light/dark) replace
  nth-child rules; parity comes from children.map idx so no sibling —
  including the loading div — can shift the stripe order
- Loading div moved outside .child-entries so it never affects child
  row ordering at all
- Accept deletedUids prop; filter expanded children array before render
  so deleted children disappear immediately without waiting for reload

EntriesView.jsx: thread deletedUids through to EntryRow

App.jsx:
- Add deletedUids state; handleEntryDeleted/handleBulkDeleted populate it
- isRoot/hasChildDelete computed from entries before setEntries (safe in
  StrictMode — no side-effects inside updater functions)
- Child delete triggers loadEntries to refresh stale parent child_count
  and total_artifact_bytes
- handleRowClick ctrl/meta cache-miss branch uses det.summary (not det)
  from fetchEntryDetail; archiveId added to dep array
- handleRowClick dep array includes archiveId

* fix(frontend): add inset stroke to child-entry-row.is-multi-selected

* fix(frontend): suppress mouse-click focus ring on entry expand button

* fix(frontend): index-based stripes for root and child rows

Root rows: EntriesView passes rowIndex from entries.map to EntryRow,
which applies entry-row-outer--light/dark. Retires both nth-child stripe
blocks so skeleton rows can never shift the first real entry to dark.
Skeleton rows keep a :not(.entry-row-outer) nth-child fallback.

Child rows: colours changed from the warm root palette (paper-3/#f2ede5)
to cooler near-whites (#fafaf8/#f2f0ec) so children are visually distinct
from their parent row regardless of which stripe the parent sits on.

* feat(core+frontend): bare yt:ID resolves to YouTube video

determine_source: yt:ID / youtube:ID with no prefix and exactly 11
chars [A-Za-z0-9_-] → YouTubeVideo via is_youtube_video_id helper.
Reserved prefixes (playlist/, channel/, c/, user/, @) still fire first
so they are unaffected by the fallback.

expand_shorthand_to_url: bare yt:ID expands to watch?v=ID using the
same predicate, consistent with how ytm:ID → music.youtube.com/watch.

isVideoSource (frontend): same 11-char /^[A-Za-z0-9_-]{11}$/ regex so
yt:ID triggers the quality probe and capture-row guards identically to
yt:video/ID.

Tests: test_is_youtube_video_id covers valid IDs (alphanumeric, with _
and -), too-short, too-long, and invalid-char cases using genuinely
invalid fixtures. test_youtube_sources adds bare-ID cases and confirms
reserved prefixes (playlist/, @) are not affected.

* fix(core+frontend): exclude avatar blobs from tweet % cached display

tweet/tweet_thread entries have avatar artifacts that are always
deduplicated from the first capture of each author. Counting them in
the cached-bytes percentage makes it artificially high (or incorrect
when the real media content is new but avatars are cached).

database.rs:
- refresh_entry_cached_bytes: AND ea.artifact_role != 'avatar'
- cascade_cached_bytes_after_delete: same filter
- cascade_cached_bytes_after_subtree_delete: same filter
- Initial cached_bytes migration: same filter
- Re-migration (else branch): recomputes cached_bytes for existing
  entries that have avatar artifacts, scoped to only those entries

archive.rs:
- EntrySummary gains cacheable_bytes: i64 — non-avatar total bytes,
  computed inline in every SQL query as the denominator for % cached
- ENTRY_SELECT_COLS adds cacheable_bytes at index 13 (with children
  subquery, same as total_artifact_bytes)
- list_root_entries adds same expression
- All 6 row-mapping closures include cacheable_bytes: row.get(13)?

EntryRow.jsx:
- SIZE display stays: formatBytes(entry.total_artifact_bytes)
- % cached badge uses cacheable_bytes as denominator:
  cached_bytes / cacheable_bytes * 100

* feat(frontend): j/k keyboard navigation for entries; test avatar cached_bytes

App.jsx — j/k handler:
- Fires on keydown when not focused on INPUT/TEXTAREA/SELECT/contenteditable
- Ignores meta/ctrl/alt modifier combos
- Uses DOM order (#entries-body [data-entry-uid]) so expanded child rows
  participate, matching the shift-range selection logic
- Resolves target entry via entryCacheRef → root entries array →
  fetchEntryDetail(..).summary on cache miss
- Scrolls target into view (block: nearest); updates lastAnchorIndexRef
  so subsequent shift-click ranges start from the keyboard-navigated row

archive.rs — cached_bytes_excludes_avatar_blobs test:
- Two tweet entries sharing an avatar blob (100B) and a media blob (900B)
- Asserts refresh_entry_cached_bytes sets cached_bytes = 900 (not 1000)
- Asserts list_root_entries summary: cached_bytes=900, cacheable_bytes=900,
  total_artifact_bytes=1000 — SIZE includes avatar, % cached denominator
  and numerator both exclude it

* fix(frontend): guard j/k uncached-child fetch with monotonic token

Rapid j/k over child rows that aren't in entryCacheRef triggers
fetchEntryDetail calls in parallel. Without a guard the last one to
settle wins, desyncing selectedUids (highlight) from selectedEntry
(detail panel/URL).

Fix:
- jkSeqRef (monotonic counter) incremented before each server fetch;
  the .then() guard tok === jkSeqRef.current drops results from
  superseded navigations
- handleRowClick increments jkSeqRef.current so any click also cancels
  an in-flight j/k fetch

* fix(frontend): guard ctrl/meta cache-miss fetch; add / search shortcut

App.jsx ctrl/meta branch: cache-miss fetchEntryDetail now captures
tok = ++jkSeqRef.current before the fetch and gates selectEntry on
tok === jkSeqRef.current — same pattern as the j/k handler — so a
slow response after a later click/navigate can't overwrite selection.

/ key: reuses the Cmd+K/Ctrl+K handler path (focus+select search input,
or pendingSearchFocus + archive view switch). Guards: editable targets
(INPUT/TEXTAREA/SELECT/contentEditable) and modifier keys are checked
before preventDefault() so typing / in inputs is untouched.

* feat(core): attempt SpotifyTrack via yt-dlp instead of hard-failing

Previously all Spotify sources returned an error claiming yt-dlp cannot
download DRM-protected audio. yt-dlp does have a Spotify extractor
(experimental, content-dependent), so refusing upfront is worse than
trying and letting it report the real failure.

SpotifyTrack now goes through the same yt-dlp metadata + audio-quality
download path as YouTubeMusicTrack. SpotifyAlbum and SpotifyPlaylist
still return an explicit error — fetch_playlist_info has YouTube-specific
URL fallback logic that would produce bogus child URLs for Spotify flat
entries; those sources need dedicated container handling first.

* chore: remove docs/superpowers from repo and gitignore whitelist

* feat(core+frontend): Spotify album/playlist capture via yt-dlp container path

Previously SpotifyAlbum and SpotifyPlaylist hard-failed with a DRM error.
yt-dlp does support Spotify (experimentally), so they now go through the
same probe/container/child path as YouTube playlists.

ytdlp.rs:
- Extract normalize_item_url() helper used by both fetch_playlist_info
  and probe_playlist_qualities; eliminates the duplicate URL-normalization
  blocks and the YouTube-specific fallback_host variable
- Fallback logic is now platform-aware: YouTube/YTM bare IDs → watch URL,
  Spotify → open.spotify.com/track/{id}, unknown → skip with warning

capture.rs:
- locator_to_playlist_url: accept SpotifyAlbum | SpotifyPlaylist so the
  probe-playlist endpoint accepts Spotify album/playlist URLs
- Container branch: add SpotifyAlbum | SpotifyPlaylist to the matches!
- is_audio: true for SpotifyAlbum | SpotifyPlaylist (audio-only, like YTM)
- child_source: SpotifyTrack for Spotify containers (not YouTubeMusicTrack)
- generate_entry_title and record_media_entry both use the correct child
  source so entity_kind, source_kind, and representation_kind are right

CaptureDialog.jsx:
- isPlaylistSource: recognise open.spotify.com/album/ and /playlist/,
  and spotify:album:ID / spotify:playlist:ID shorthands, so Spotify
  containers get the probe UI, per-track excludes, and sync toggle

* fix(core+server): partial playlist refresh + child visibility inheritance

database.rs:
- finish_archive_run: reverted 'partial' status (violates CHECK constraint);
  restored binary completed/failed
- get_run_completed_count(): new helper for callers that need to distinguish
  partial success without touching the DB status enum

capture.rs:
- CaptureResult gains completed_count: i64 (0 for single-item captures;
  populated from get_run_completed_count for container/playlist captures)
- All CaptureResult construction sites updated

routes.rs:
- Playlist job_status mapping: mark job 'completed' when completed_count > 0
  (even if status == 'failed'), so partially-successful playlist captures
  trigger onCaptured and show the archived entries without a manual reload.
  Truly zero-success runs (completed_count == 0, status == 'failed') stay
  failed as expected.
- probe_playlist_handler error message updated to mention Spotify

archive.rs (list_child_entries):
- Add third OR arm: children are visible when their parent container is in a
  collection visible to the caller. Fixes empty child list for non-admin
  users with access to a playlist container but not its newly-created
  children (which are all in the default 'private' collection).

* fix(server): use completed_count to distinguish partial from total failure

finish_archive_run is binary (completed/failed) — a playlist where some
tracks succeed and some fail returns status='failed'. Without this fix,
the job mapping treated any failed status as a job failure, causing
onCaptured to never fire and leaving successfully archived entries hidden
until a manual reload.

Now: job is 'failed' only when status='failed' && completed_count==0.
Partial runs (completed_count > 0) map to a completed job so the UI
refreshes and shows the entries that were successfully captured.

* fix(core+server): exclude container from child success count; rename field; tests

database.rs:
- get_run_completed_child_count(): renamed from get_run_completed_count and
  scoped to child items only (parent_item_id IS NOT NULL). The container
  run item is always completed first, so the old function inflated the count
  by at least 1 for every playlist, making all-video-failed runs appear as
  partial successes.
- New regression test: completed_child_count_excludes_container — completes
  both a root and a child item, asserts DB completed_count==2 while
  get_run_completed_child_count==1.

capture.rs:
- CaptureResult.completed_count renamed to completed_child_count to match
  the function and make the semantics unambiguous at the call site.

routes.rs:
- job_status decision now uses result.completed_child_count == 0 so that a
  playlist where every video fails (child_count==0) is correctly reported
  as a failed job, not a completed one.

archive.rs:
- New regression test: list_child_entries_inherits_parent_visibility —
  enrolls only the container in a USER-visible collection, asserts the
  child is visible to a USER caller and invisible to a GUEST caller.
This commit is contained in:
TheGeneralist 2026-07-21 21:57:29 +02:00 • committed by GitHub
parent d202e177e1
commit 6377daadae
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
23 changed files with 2243 additions and 2640 deletions
Download patch file Download diff file Expand all files Collapse all files

432
crates/archivr-core/src/archive.rs
View file

@ -30,6 +30,10 @@ pub struct EntrySummary {
pub has_favicon: bool,
/// Bytes of blobs already on disk from an earlier entry (precomputed at capture time).
pub cached_bytes: i64,
/// Number of direct child entries; 0 for non-container entries.
pub child_count: i64,
/// Total non-avatar artifact bytes (query-time; used as denominator for cache-hit %).
pub cacheable_bytes: i64,
}
#[derive(Debug, Clone, PartialEq, Eq, serde::Serialize)]
@ -212,10 +216,12 @@ pub fn list_root_entries(
e.visibility,
si.canonical_url,
COUNT(ea.id) AS artifact_count,
COALESCE(SUM(b.byte_size), 0) AS total_artifact_bytes,
COALESCE(SUM(b.byte_size), 0) + COALESCE((SELECT SUM(b2.byte_size) FROM archived_entries c2 JOIN entry_artifacts ea2 ON ea2.entry_id = c2.id JOIN blobs b2 ON b2.id = ea2.blob_id WHERE c2.parent_entry_id = e.id), 0) AS total_artifact_bytes,
NULL AS parent_entry_uid,
EXISTS(SELECT 1 FROM entry_artifacts fav WHERE fav.entry_id = e.id AND fav.artifact_role = 'favicon') AS has_favicon,
e.cached_bytes
e.cached_bytes,
(SELECT COUNT(*) FROM archived_entries child WHERE child.parent_entry_id = e.id) AS child_count,
COALESCE(SUM(CASE WHEN ea.artifact_role != 'avatar' THEN b.byte_size ELSE 0 END), 0) + COALESCE((SELECT SUM(b2.byte_size) FROM archived_entries c2 JOIN entry_artifacts ea2 ON ea2.entry_id = c2.id JOIN blobs b2 ON b2.id = ea2.blob_id WHERE c2.parent_entry_id = e.id), 0) AS cacheable_bytes
FROM archived_entries e
JOIN source_identities si ON si.id = e.source_identity_id
LEFT JOIN entry_artifacts ea ON ea.entry_id = e.id
@ -248,6 +254,8 @@ pub fn list_root_entries(
parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
@ -255,6 +263,40 @@ pub fn list_root_entries(
Ok(entries)
}
/// Fetches one `EntrySummary` for any entry (root or child) by uid.
/// Returns `None` if not found.
fn get_entry_summary(
conn: &rusqlite::Connection,
entry_uid: &str,
) -> Result<Option<EntrySummary>> {
let sql = format!(
"{} {} WHERE e.entry_uid = ?1 GROUP BY e.id",
ENTRY_SELECT_COLS, ENTRY_FROM_JOINS,
);
let mut stmt = conn.prepare(&sql)?;
let result = stmt
.query_row([entry_uid], |row| {
Ok(EntrySummary {
entry_uid: row.get(0)?,
archived_at: row.get(1)?,
source_kind: row.get(2)?,
entity_kind: row.get(3)?,
title: row.get(4)?,
visibility: row.get(5)?,
original_url: row.get(6)?,
artifact_count: row.get(7)?,
total_artifact_bytes: row.get(8)?,
parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
})
})
.optional()?;
Ok(result)
}
pub fn get_entry_detail(
conn: &rusqlite::Connection,
entry_uid: &str,
@ -279,9 +321,7 @@ pub fn get_entry_detail(
return Ok(None);
};
let summary = list_root_entries(conn, u32::MAX)?
.into_iter()
.find(|entry| entry.entry_uid == entry_uid)
let summary = get_entry_summary(conn, entry_uid)?
.context("entry disappeared while loading detail")?;
let mut stmt = conn.prepare(
@ -438,12 +478,121 @@ pub fn list_entries_for_collection(
parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(entries)
}
/// Returns the direct children of the entry identified by `parent_entry_uid`,
/// ordered ascending by `archived_at, id` (preserves playlist ordinal feel).
/// Returns an empty vec if the parent has no children or does not exist.
pub fn list_child_entries(
conn: &rusqlite::Connection,
parent_entry_uid: &str,
caller_bits: u32,
) -> Result<Vec<EntrySummary>> {
let sql = format!(
"{} {} \
WHERE e.parent_entry_id = (SELECT id FROM archived_entries WHERE entry_uid = ?1) \
AND (\
CAST(?2 AS INTEGER) & 12 != 0 \
OR EXISTS (\
SELECT 1 FROM collection_entries ce \
WHERE ce.entry_id = e.id \
AND ce.visibility_bits & CAST(?2 AS INTEGER) != 0\
)\
OR EXISTS (\
SELECT 1 FROM collection_entries ce_p \
WHERE ce_p.entry_id = (SELECT id FROM archived_entries WHERE entry_uid = ?1)\
AND ce_p.visibility_bits & CAST(?2 AS INTEGER) != 0\
)\
) \
GROUP BY e.id \
ORDER BY e.archived_at ASC, e.id ASC",
ENTRY_SELECT_COLS, ENTRY_FROM_JOINS,
);
let mut stmt = conn.prepare(&sql)?;
let entries = stmt
.query_map(
rusqlite::params![parent_entry_uid, caller_bits as i64],
|row| {
Ok(EntrySummary {
entry_uid: row.get(0)?,
archived_at: row.get(1)?,
source_kind: row.get(2)?,
entity_kind: row.get(3)?,
title: row.get(4)?,
visibility: row.get(5)?,
original_url: row.get(6)?,
artifact_count: row.get(7)?,
total_artifact_bytes: row.get(8)?,
parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
})
},
)?
.collect::<rusqlite::Result<Vec<_>>>()?;
Ok(entries)
}
/// Returns the set of canonical URLs for all child entries archived under
/// **any** container entry whose own canonical URL matches `playlist_canonical_url`.
///
/// Used in sync mode so the playlist capture path can skip videos that were
/// already downloaded in a previous run of the same playlist.
pub fn get_archived_playlist_child_urls(
conn: &rusqlite::Connection,
playlist_canonical_url: &str,
) -> Result<std::collections::HashSet<String>> {
let mut stmt = conn.prepare(
"SELECT si_child.canonical_url \
FROM archived_entries child \
JOIN source_identities si_child ON si_child.id = child.source_identity_id \
WHERE child.parent_entry_id IN ( \
SELECT e.id \
FROM archived_entries e \
JOIN source_identities si ON si.id = e.source_identity_id \
WHERE si.canonical_url = ?1 \
AND e.parent_entry_id IS NULL \
)",
)?;
let urls = stmt
.query_map([playlist_canonical_url], |row| row.get::<_, String>(0))?
.collect::<rusqlite::Result<std::collections::HashSet<_>>>()?;
Ok(urls)
}
/// Finds the most recent container entry (parent_entry_id IS NULL) whose
/// canonical URL matches `canonical_url`. Returns its row id, or None if
/// no such entry exists.
///
/// Used in sync mode to reuse an existing playlist/channel container instead
/// of creating a duplicate root entry on every sync run.
pub fn find_container_entry_id_by_canonical_url(
conn: &rusqlite::Connection,
canonical_url: &str,
) -> Result<Option<i64>> {
let mut stmt = conn.prepare(
"SELECT e.id \
FROM archived_entries e \
JOIN source_identities si ON si.id = e.source_identity_id \
WHERE si.canonical_url = ?1 \
AND e.parent_entry_id IS NULL \
ORDER BY e.archived_at DESC \
LIMIT 1",
)?;
let id = stmt
.query_row([canonical_url], |row| row.get::<_, i64>(0))
.optional()?;
Ok(id)
}
/// Resolves an artifact to its absolute on-disk path under `store_path`.
///
/// `artifact.relpath` is a store-relative path (e.g. `raw/a/b/abc.pdf`).
@ -552,10 +701,12 @@ pub fn parse_search_query(raw: &str) -> Result<SearchEntriesQuery, String> {
const ENTRY_SELECT_COLS: &str = "SELECT e.entry_uid, e.archived_at, e.source_kind, e.entity_kind, e.title, \
e.visibility, si.canonical_url, COUNT(ea.id) AS artifact_count, \
COALESCE(SUM(b.byte_size), 0) AS total_artifact_bytes, \
COALESCE(SUM(b.byte_size), 0) + COALESCE((SELECT SUM(b2.byte_size) FROM archived_entries c2 JOIN entry_artifacts ea2 ON ea2.entry_id = c2.id JOIN blobs b2 ON b2.id = ea2.blob_id WHERE c2.parent_entry_id = e.id), 0) AS total_artifact_bytes, \
parent.entry_uid AS parent_entry_uid, \
EXISTS(SELECT 1 FROM entry_artifacts fav WHERE fav.entry_id = e.id AND fav.artifact_role = 'favicon') AS has_favicon, \
e.cached_bytes";
e.cached_bytes, \
(SELECT COUNT(*) FROM archived_entries child WHERE child.parent_entry_id = e.id) AS child_count, \
COALESCE(SUM(CASE WHEN ea.artifact_role != 'avatar' THEN b.byte_size ELSE 0 END), 0) + COALESCE((SELECT SUM(b2.byte_size) FROM archived_entries c2 JOIN entry_artifacts ea2 ON ea2.entry_id = c2.id JOIN blobs b2 ON b2.id = ea2.blob_id WHERE c2.parent_entry_id = e.id), 0) AS cacheable_bytes";
const ENTRY_FROM_JOINS: &str = "FROM archived_entries e \
JOIN source_identities si ON si.id = e.source_identity_id \
@ -663,6 +814,8 @@ pub fn search_entries(
parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
@ -870,7 +1023,9 @@ pub fn entries_for_tag(
COALESCE(SUM(b.byte_size), 0) AS total_artifact_bytes,
parent.entry_uid AS parent_entry_uid,
EXISTS(SELECT 1 FROM entry_artifacts fav WHERE fav.entry_id = e.id AND fav.artifact_role = 'favicon') AS has_favicon,
e.cached_bytes
e.cached_bytes,
(SELECT COUNT(*) FROM archived_entries child WHERE child.parent_entry_id = e.id) AS child_count,
COALESCE(SUM(CASE WHEN ea.artifact_role != 'avatar' THEN b.byte_size ELSE 0 END), 0) AS cacheable_bytes
FROM archived_entries e
JOIN source_identities si ON si.id = e.source_identity_id
LEFT JOIN entry_artifacts ea ON ea.entry_id = e.id
@ -896,6 +1051,8 @@ pub fn entries_for_tag(
parent_entry_uid: row.get(9)?,
has_favicon: row.get::<_, i64>(10)? != 0,
cached_bytes: row.get(11)?,
child_count: row.get(12)?,
cacheable_bytes: row.get(13)?,
})
})?
.collect::<rusqlite::Result<Vec<_>>>()?;
@ -1618,4 +1775,263 @@ mod tests {
);
assert_eq!(results[0].entry_uid, parent.entry_uid);
}
// ── sync helper tests ────────────────────────────────────────────────────────
#[test]
fn find_container_entry_id_returns_none_when_absent() {
let (conn, _, _) = make_tag_test_db();
let result = find_container_entry_id_by_canonical_url(
&conn,
"https://www.youtube.com/playlist?list=PLnobody",
)
.unwrap();
assert!(result.is_none(), "should return None when no entry exists");
}
#[test]
fn find_container_entry_id_returns_root_entry() {
let (conn, user_id, run_id) = make_tag_test_db();
let playlist_url = "https://www.youtube.com/playlist?list=PLtest";
let container = make_entry_in_db(&conn, user_id, run_id, None, None, "My Playlist", playlist_url);
let result = find_container_entry_id_by_canonical_url(&conn, playlist_url)
.unwrap()
.expect("should find the container");
assert_eq!(result, container.id);
}
#[test]
fn find_container_entry_id_ignores_child_entries() {
let (conn, user_id, run_id) = make_tag_test_db();
let child_url = "https://www.youtube.com/watch?v=abc123";
// Create a parent container and a child whose canonical URL happens to be
// what we're querying — should NOT be returned since it has parent_entry_id set.
let parent = make_entry_in_db(&conn, user_id, run_id, None, None, "PL", "https://example.com/pl");
let _child = make_entry_in_db(&conn, user_id, run_id, Some(parent.id), Some(parent.id), "Vid", child_url);
let result = find_container_entry_id_by_canonical_url(&conn, child_url).unwrap();
assert!(result.is_none(), "child entry should not be returned as a container");
}
#[test]
fn get_archived_playlist_child_urls_empty_when_no_playlist() {
let (conn, _, _) = make_tag_test_db();
let urls = get_archived_playlist_child_urls(
&conn,
"https://www.youtube.com/playlist?list=PLnone",
)
.unwrap();
assert!(urls.is_empty());
}
#[test]
fn get_archived_playlist_child_urls_returns_children() {
let (conn, user_id, run_id) = make_tag_test_db();
let playlist_url = "https://www.youtube.com/playlist?list=PLchildren";
let container = make_entry_in_db(&conn, user_id, run_id, None, None, "Playlist", playlist_url);
let child1_url = "https://www.youtube.com/watch?v=vid1";
let child2_url = "https://www.youtube.com/watch?v=vid2";
make_entry_in_db(&conn, user_id, run_id, Some(container.id), Some(container.id), "Vid 1", child1_url);
make_entry_in_db(&conn, user_id, run_id, Some(container.id), Some(container.id), "Vid 2", child2_url);
let urls = get_archived_playlist_child_urls(&conn, playlist_url).unwrap();
assert_eq!(urls.len(), 2);
assert!(urls.contains(child1_url), "should contain vid1");
assert!(urls.contains(child2_url), "should contain vid2");
}
#[test]
fn get_archived_playlist_child_urls_excludes_other_playlists() {
let (conn, user_id, run_id) = make_tag_test_db();
let pl_a = "https://www.youtube.com/playlist?list=PLA";
let pl_b = "https://www.youtube.com/playlist?list=PLB";
let container_a = make_entry_in_db(&conn, user_id, run_id, None, None, "PL A", pl_a);
let container_b = make_entry_in_db(&conn, user_id, run_id, None, None, "PL B", pl_b);
let vid_a = "https://www.youtube.com/watch?v=forA";
let vid_b = "https://www.youtube.com/watch?v=forB";
make_entry_in_db(&conn, user_id, run_id, Some(container_a.id), Some(container_a.id), "A Vid", vid_a);
make_entry_in_db(&conn, user_id, run_id, Some(container_b.id), Some(container_b.id), "B Vid", vid_b);
let urls_a = get_archived_playlist_child_urls(&conn, pl_a).unwrap();
assert_eq!(urls_a.len(), 1);
assert!(urls_a.contains(vid_a));
assert!(!urls_a.contains(vid_b), "should not include children of other playlists");
}
#[test]
fn cached_bytes_excludes_avatar_blobs() {
let (conn, user_id, run_id) = make_tag_test_db();
// entry_a is older (created first, gets the lower id)
let entry_a = make_entry_in_db(
&conn,
user_id,
run_id,
None,
None,
"Tweet A",
"https://twitter.com/user/status/1",
);
// entry_b is newer (created second, higher id — tiebreak on id when archived_at is equal)
let entry_b = make_entry_in_db(
&conn,
user_id,
run_id,
None,
None,
"Tweet B",
"https://twitter.com/user/status/2",
);
// Shared avatar blob: 100 bytes
let avatar_blob_id = database::upsert_blob(
&conn,
&database::BlobRecord {
sha256: "avatar_sha256_test".to_string(),
byte_size: 100,
mime_type: Some("image/jpeg".to_string()),
extension: Some("jpg".to_string()),
raw_relpath: "raw/av/at/avatar.jpg".to_string(),
},
)
.unwrap();
// Shared media blob: 900 bytes
let media_blob_id = database::upsert_blob(
&conn,
&database::BlobRecord {
sha256: "media_sha256_test".to_string(),
byte_size: 900,
mime_type: Some("image/png".to_string()),
extension: Some("png".to_string()),
raw_relpath: "raw/me/di/media.png".to_string(),
},
)
.unwrap();
// Attach both artifacts to entry_a
database::add_entry_artifact(
&conn,
&database::NewArtifact {
entry_id: entry_a.id,
artifact_role: "avatar".to_string(),
storage_area: "raw".to_string(),
relpath: "raw/av/at/avatar.jpg".to_string(),
blob_id: Some(avatar_blob_id),
logical_path: None,
metadata_json: None,
},
)
.unwrap();
database::add_entry_artifact(
&conn,
&database::NewArtifact {
entry_id: entry_a.id,
artifact_role: "primary_media".to_string(),
storage_area: "raw".to_string(),
relpath: "raw/me/di/media.png".to_string(),
blob_id: Some(media_blob_id),
logical_path: None,
metadata_json: None,
},
)
.unwrap();
// Attach both artifacts to entry_b (same blobs — simulates shared avatar/media)
database::add_entry_artifact(
&conn,
&database::NewArtifact {
entry_id: entry_b.id,
artifact_role: "avatar".to_string(),
storage_area: "raw".to_string(),
relpath: "raw/av/at/avatar.jpg".to_string(),
blob_id: Some(avatar_blob_id),
logical_path: None,
metadata_json: None,
},
)
.unwrap();
database::add_entry_artifact(
&conn,
&database::NewArtifact {
entry_id: entry_b.id,
artifact_role: "primary_media".to_string(),
storage_area: "raw".to_string(),
relpath: "raw/me/di/media.png".to_string(),
blob_id: Some(media_blob_id),
logical_path: None,
metadata_json: None,
},
)
.unwrap();
// Recompute cached_bytes for entry_b.
// The query excludes avatar-role artifacts and counts only non-avatar blobs
// that are already held by an earlier entry. entry_a (lower id) owns the
// same media blob, so cached_bytes for entry_b should be 900, not 1000.
database::refresh_entry_cached_bytes(&conn, entry_b.id).unwrap();
// --- Assert raw DB value ---
let cached_bytes_db: i64 = conn
.query_row(
"SELECT cached_bytes FROM archived_entries WHERE id = ?1",
[entry_b.id],
|row| row.get(0),
)
.unwrap();
assert_eq!(
cached_bytes_db, 900,
"cached_bytes should be 900 (media only); avatar blob must be excluded"
);
// --- Assert list_root_entries summary for entry_b ---
let entries = list_root_entries(&conn, 12).unwrap(); // 12 = ADMIN bits
let summary_b = entries
.iter()
.find(|e| e.entry_uid == entry_b.entry_uid)
.expect("entry_b must appear in list_root_entries");
assert_eq!(
summary_b.cached_bytes, 900,
"summary cached_bytes must equal 900 (precomputed, avatar excluded)"
);
assert_eq!(
summary_b.cacheable_bytes, 900,
"cacheable_bytes (non-avatar total) must be 900 for entry_b"
);
assert_eq!(
summary_b.total_artifact_bytes, 1000,
"total_artifact_bytes must be 1000 (avatar 100 + media 900)"
);
}
#[test]
fn list_child_entries_inherits_parent_visibility() {
// Regression: non-admin users who can see a playlist container through a
// collection must also see its children. The child is auto-enrolled in the
// default collection with private visibility bits — it has no *visible*
// collection membership of its own, so it would be filtered out without
// the parent-collection OR arm in list_child_entries.
let (conn, user_id, run_id) = make_tag_test_db();
let container = make_entry_in_db(&conn, user_id, run_id, None, None,
"Playlist", "https://example.com/pl");
let child = make_entry_in_db(&conn, user_id, run_id,
Some(container.id), Some(container.id),
"Video 1", "https://example.com/pl/v1");
// Enroll the container (but NOT the child) in a USER-visible collection (bits=2).
let coll = database::create_collection(&conn, "My List", "my-list", 2).unwrap();
database::add_entry_to_collection(&conn, coll.id, container.id, 2).unwrap();
// USER caller (bits=2): child must be visible through parent's collection.
let children = list_child_entries(&conn, &container.entry_uid, 2).unwrap();
assert_eq!(children.len(), 1, "child must be visible via parent collection");
assert_eq!(children[0].entry_uid, child.entry_uid);
// GUEST caller (bits=1): parent collection has bits=2, so child must NOT be visible.
let guest_children = list_child_entries(&conn, &container.entry_uid, 1).unwrap();
assert!(guest_children.is_empty(), "guest must not see children of a USER-only collection");
}
}

475
crates/archivr-core/src/capture.rs
View file

@ -7,7 +7,7 @@ use std::{
fs,
path::{Path, PathBuf},
};
use crate::{archive::ArchivePaths, database, downloader, twitter::parse_tweet_id};
use crate::{archive::{self, ArchivePaths}, database, downloader, twitter::parse_tweet_id};
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub enum Source {
@ -37,8 +37,11 @@ pub enum Source {
pub struct CaptureResult {
pub run_uid: String,
pub status: String,
/// Number of successfully completed child items (playlist/container captures only).
/// Zero for single-item captures. Used by the server to distinguish a fully-failed
/// playlist run from a partial success without needing a new DB status value.
pub completed_child_count: i64,
/// `true` when uBlock was requested but the extension path was not found.
/// The capture succeeded without ad-blocking; the UI should warn the user.
pub ublock_skipped: bool,
/// `true` when cookie-consent extension was requested but the path was not found.
pub cookie_ext_skipped: bool,
@ -90,6 +93,17 @@ pub struct CaptureConfig {
/// Route WebPage captures through the Freedium mirror to bypass paywalls.
/// The original locator is still recorded in the DB; only the fetch URL changes.
pub via_freedium: bool,
/// Per-item quality overrides and include-set for playlist captures.
/// Keys are yt-dlp video IDs (e.g. "dQw4w9WgXcQ"); values are quality strings
/// accepted by `quality_format` ("best", "audio", "1080p", etc.).
/// - Empty map: all playlist items are downloaded using the global `quality`.
/// - Non-empty map: ONLY items whose ID appears as a key are downloaded;
/// items absent from the map are skipped entirely. This is how the UI's
/// per-video delete button excludes specific videos from a capture.
pub per_item_quality: HashMap<String, String>,
/// When true, skip playlist items whose URL is already archived as a child
/// of any container entry with the same canonical playlist URL.
pub sync: bool,
}
/// Resolves which cookies apply to `url` by evaluating all rules in ordinal order.
@ -200,6 +214,12 @@ fn generate_entry_title(source: Source, meta: &PlatformMetadata) -> String {
Source::Other => "Archived Content".to_string(),
}
}
/// Returns true when `s` is a valid bare YouTube video ID:
/// exactly 11 characters from the set `[A-Za-z0-9_-]`.
fn is_youtube_video_id(s: &str) -> bool {
s.len() == 11 && s.bytes().all(|b| b.is_ascii_alphanumeric() || b == b'_' || b == b'-')
}
fn expand_shorthand_to_url(path: &str, source: &Source) -> String {
// YouTube shorthands: yt:video/ID, yt:playlist/ID, yt:@handle, yt:channel/ID, etc.
@ -227,6 +247,10 @@ fn expand_shorthand_to_url(path: &str, source: &Source) -> String {
if let Some(handle) = after.strip_prefix("@") {
return format!("https://www.youtube.com/@{handle}");
}
// bare yt:ID — validated 11-char video ID
if is_youtube_video_id(after) {
return format!("https://www.youtube.com/watch?v={after}");
}
}
}
@ -317,6 +341,11 @@ fn determine_source(path: &str) -> Source {
{
return Source::YouTubeChannel;
}
// bare yt:ID — exactly 11 chars [A-Za-z0-9_-], no slash/@ prefixes
if is_youtube_video_id(after_scheme) {
return Source::YouTubeVideo;
}
}
// Shorthand scheme: ytm:
@ -536,6 +565,21 @@ pub fn locator_to_ytdlp_url(locator: &str) -> Option<String> {
}
}
/// Returns the canonical URL for playlist/channel locators that yt-dlp can
/// expand with `--flat-playlist`, or `None` for non-playlist sources.
/// Intentionally separate from `locator_to_ytdlp_url`, which excludes playlists.
pub fn locator_to_playlist_url(locator: &str) -> Option<String> {
let source = determine_source(locator);
match source {
Source::YouTubePlaylist
| Source::YouTubeChannel
| Source::YouTubeMusicPlaylist
| Source::SpotifyAlbum
| Source::SpotifyPlaylist => Some(expand_shorthand_to_url(locator, &source)),
_ => None,
}
}
fn hash_exists(hash: &str, file_extension: &str, store_path: &Path) -> Result<bool> {
let path = store_path.join(raw_relative_path_from_hash(hash, file_extension)?);
Ok(path.exists())
@ -660,6 +704,8 @@ fn record_media_entry(
file_extension: &str,
byte_size: i64,
title: Option<String>,
parent_entry_id: Option<i64>,
root_entry_id: Option<i64>,
) -> Result<database::ArchivedEntry> {
debug_assert!(run.run_uid.starts_with("run_"));
debug_assert!(item.item_uid.starts_with("item_"));
@ -686,8 +732,8 @@ fn record_media_entry(
&database::NewEntry {
source_identity_id,
archive_run_id: run.id,
parent_entry_id: None,
root_entry_id: None,
parent_entry_id,
root_entry_id,
created_by_user_id: user_id,
owned_by_user_id: user_id,
source_kind: source_kind.to_string(),
@ -720,6 +766,59 @@ fn record_media_entry(
Ok(entry)
}
/// Creates a playlist/channel container entry with no blob and no primary-media artifact.
/// Calls `complete_archive_run_item` on the provided item.
fn record_container_entry(
conn: &rusqlite::Connection,
store_path: &Path,
user_id: i64,
run: &database::ArchiveRun,
item: &database::ArchiveRunItem,
requested_locator: &str,
canonical_locator: &str,
source: Source,
title: Option<String>,
playlist_id: &str,
uploader: Option<&str>,
) -> Result<database::ArchivedEntry> {
let (source_kind, entity_kind, representation_kind) = source_metadata(source);
let source_identity_id = database::upsert_source_identity(
conn,
source_kind,
entity_kind,
Some(playlist_id),
Some(canonical_locator),
canonical_locator,
)?;
let entry = database::create_archived_entry(
conn,
&database::NewEntry {
source_identity_id,
archive_run_id: run.id,
parent_entry_id: None,
root_entry_id: None,
created_by_user_id: user_id,
owned_by_user_id: user_id,
source_kind: source_kind.to_string(),
entity_kind: entity_kind.to_string(),
title,
visibility: "private".to_string(),
representation_kind: representation_kind.to_string(),
source_metadata_json: json!({
"requested_locator": requested_locator,
"canonical_locator": canonical_locator,
"playlist_id": playlist_id,
"uploader": uploader,
})
.to_string(),
display_metadata_json: None,
},
)?;
create_structured_root(store_path, &entry)?;
database::complete_archive_run_item(conn, item.id, entry.id)?;
Ok(entry)
}
/// Extracts PlatformMetadata from a tweet JSON string.
/// Returns Default on any parse failure.
fn tweet_metadata_from_json(json_str: &str) -> PlatformMetadata {
@ -957,25 +1056,273 @@ pub fn perform_capture(
));
}
// Sources: Spotify — not downloadable; Spotify audio is DRM-protected.
if matches!(source, Source::SpotifyTrack | Source::SpotifyAlbum | Source::SpotifyPlaylist) {
return Err(fail_run(
&conn,
&run,
&item,
"Spotify downloads are not supported: Spotify audio is DRM-protected and cannot \
be downloaded by yt-dlp. Archive the equivalent YouTube Music track instead.",
));
}
// Sources: YouTube playlists, YouTube channels, YouTube Music playlists,
// Spotify albums, Spotify playlists — probe via yt-dlp flat-playlist,
// create a root container entry, then download each track as a child entry.
// `run` and `item` are already created above; `item` acts as the container item.
if matches!(
source,
Source::YouTubePlaylist
| Source::YouTubeChannel
| Source::YouTubeMusicPlaylist
| Source::SpotifyAlbum
| Source::SpotifyPlaylist
) {
// `canonical_url` already holds the expanded URL (same value as `path` later).
let playlist_info = match downloader::ytdlp::fetch_playlist_info(&canonical_url, &cookies) {
Ok(info) => info,
Err(e) => {
return Err(fail_run(
&conn,
&run,
&item,
&format!("Failed to fetch playlist info: {e:#}"),
));
}
};
// Sources: YouTube Music Playlist — container expansion not yet implemented.
if source == Source::YouTubeMusicPlaylist {
return Err(fail_run(
&conn,
&run,
&item,
"YouTube Music playlist archiving is not yet implemented.",
));
let container_title = playlist_info
.title
.clone()
.or_else(|| playlist_info.uploader.clone().map(|u| format!("{u} (channel)")));
// Sync mode: reuse an existing container so we don't create a duplicate
// root entry on every sync run. Non-sync always creates a fresh container.
let (container_id, already_archived) = if config.sync {
match archive::find_container_entry_id_by_canonical_url(&conn, &canonical_url) {
Err(e) => {
return Err(fail_run(
&conn, &run, &item,
&format!("Failed to query existing container: {e:#}"),
));
}
Ok(Some(existing_id)) => {
// Container already exists — mark the run item done pointing at it;
// don't call record_container_entry (that would create a duplicate).
if let Err(e) = database::complete_archive_run_item(&conn, item.id, existing_id) {
return Err(fail_run(
&conn, &run, &item,
&format!("Failed to complete run item for existing container: {e:#}"),
));
}
let archived = match archive::get_archived_playlist_child_urls(&conn, &canonical_url) {
Ok(set) => set,
Err(e) => return Err(fail_run(
&conn, &run, &item,
&format!("Failed to query archived playlist children: {e:#}"),
)),
};
(existing_id, archived)
}
Ok(None) => {
// First sync run for this playlist — create the container normally.
let e = match record_container_entry(
&conn, store_path, user_id, &run, &item, locator, &canonical_url,
source, container_title, &playlist_info.playlist_id,
playlist_info.uploader.as_deref(),
) {
Ok(e) => e,
Err(e) => return Err(fail_run(
&conn, &run, &item,
&format!("Failed to create container entry: {e:#}"),
)),
};
(e.id, std::collections::HashSet::new())
}
}
} else {
let e = match record_container_entry(
&conn, store_path, user_id, &run, &item, locator, &canonical_url,
source, container_title, &playlist_info.playlist_id,
playlist_info.uploader.as_deref(),
) {
Ok(e) => e,
Err(e) => return Err(fail_run(
&conn, &run, &item,
&format!("Failed to create container entry: {e:#}"),
)),
};
(e.id, std::collections::HashSet::new())
};
let is_audio = matches!(
source,
Source::YouTubeMusicPlaylist | Source::SpotifyAlbum | Source::SpotifyPlaylist
);
let child_quality: Option<&str> = if is_audio { Some("audio") } else { quality };
let child_entity_kind = if is_audio { "music" } else { "video" };
for (ordinal, playlist_item) in playlist_info.items.iter().enumerate() {
// Sync: skip items already archived — no run item created, so
// refresh_run_counters sees only newly-attempted items.
if config.sync && already_archived.contains(&playlist_item.url) {
continue;
}
// Per-item exclusion: when per_item_quality is non-empty (frontend
// probe ran and the user confirmed the item list), only download
// items whose ID appears in the map. IDs absent from a non-empty
// map were removed by the user via the delete button before submit.
if !config.per_item_quality.is_empty()
&& !config.per_item_quality.contains_key(&playlist_item.id)
{
continue;
}
let child_timestamp = format!(
"{}-{}",
Local::now().format("%Y-%m-%dT%H-%M-%S%.3f"),
Uuid::new_v4().simple(),
);
let child_item = match database::create_archive_run_item(
&conn,
run.id,
Some(item.id),
ordinal as i64,
&playlist_item.url,
Some(&playlist_item.url),
source_kind,
child_entity_kind,
) {
Ok(i) => i,
Err(e) => {
eprintln!("warn: playlist item {} create_run_item failed: {e:#}", playlist_item.url);
continue;
}
};
// Fetch metadata for the child title (best-effort).
let child_meta_json = downloader::ytdlp::fetch_metadata(&playlist_item.url, &cookies);
let child_title: Option<String> = match &child_meta_json {
Some(json) => {
let meta = downloader::metadata::extract_from_ytdlp_json(json);
Some(generate_entry_title(
match source {
Source::SpotifyAlbum | Source::SpotifyPlaylist => Source::SpotifyTrack,
_ if is_audio => Source::YouTubeMusicTrack,
_ => Source::YouTubeVideo,
},
&meta,
))
}
None => playlist_item.title.clone(),
};
// Per-item quality override keyed by yt-dlp video ID; fall back to
// playlist-level quality. YTM playlists are always audio-only —
// per_item_quality overrides are ignored so the audio-only guarantee
// cannot be bypassed by a caller sending e.g. "best".
let effective_child_quality: Option<&str> = if is_audio {
Some("audio")
} else {
config
.per_item_quality
.get(&playlist_item.id)
.map(String::as_str)
.or(child_quality)
};
// Download the media.
match downloader::ytdlp::download(
playlist_item.url.clone(),
store_path,
&child_timestamp,
effective_child_quality,
&cookies,
) {
Ok((hash, file_extension)) => {
let temp_file = store_path
.join("temp")
.join(&child_timestamp)
.join(format!("{child_timestamp}{file_extension}"));
let byte_size = match fs::metadata(&temp_file) {
Ok(m) => m.len() as i64,
Err(e) => {
eprintln!("warn: stat child temp file: {e:#}");
let _ = database::fail_archive_run_item(
&conn, child_item.id,
&format!("failed to stat downloaded file: {e:#}"),
);
continue;
}
};
if !hash_exists(&hash, &file_extension, store_path).unwrap_or(false) {
if let Err(e) = move_temp_to_raw(&temp_file, &hash, store_path) {
eprintln!("warn: move_temp_to_raw child: {e:#}");
let _ = database::fail_archive_run_item(
&conn, child_item.id,
&format!("failed to move downloaded file: {e:#}"),
);
continue;
}
}
let _ = fs::remove_dir_all(store_path.join("temp").join(&child_timestamp));
let child_source = match source {
Source::SpotifyAlbum | Source::SpotifyPlaylist => Source::SpotifyTrack,
_ if is_audio => Source::YouTubeMusicTrack,
_ => Source::YouTubeVideo,
};
match record_media_entry(
&conn,
store_path,
user_id,
&run,
&child_item,
&playlist_item.url,
&playlist_item.url,
child_source,
&hash,
&file_extension,
byte_size,
child_title,
Some(container_id),
Some(container_id),
) {
Ok(child_entry) => {
let _ = database::refresh_entry_cached_bytes(&conn, child_entry.id);
}
Err(e) => {
eprintln!("warn: record child entry: {e:#}");
let _ = database::fail_archive_run_item(
&conn, child_item.id,
&format!("failed to record entry: {e:#}"),
);
}
}
}
Err(e) => {
let _ = fs::remove_dir_all(store_path.join("temp").join(&child_timestamp));
eprintln!("warn: yt-dlp child download failed for {}: {e:#}", playlist_item.url);
let _ = database::fail_archive_run_item(
&conn, child_item.id,
&format!("yt-dlp download failed: {e:#}"),
);
}
}
}
database::refresh_entry_cached_bytes(&conn, container_id)?;
database::finish_archive_run(&conn, run.id)?;
let run_status: String = conn
.query_row(
"SELECT status FROM archive_runs WHERE id = ?1",
[run.id],
|row| row.get(0),
)
.unwrap_or_else(|_| "completed".to_string());
let completed_child_count: i64 = database::get_run_completed_child_count(&conn, run.id)
.unwrap_or(0);
return Ok(CaptureResult {
run_uid: run.run_uid.clone(),
status: run_status,
completed_child_count,
ublock_skipped: false,
cookie_ext_skipped: false,
});
}
// Source: generic HTTP/S file URL
@ -1018,12 +1365,15 @@ pub fn perform_capture(
&file_extension,
byte_size,
title_hint,
None,
None,
)?;
database::refresh_entry_cached_bytes(&conn, entry.id)?;
database::finish_archive_run(&conn, run.id)?;
return Ok(CaptureResult {
run_uid: run.run_uid.clone(),
status: "completed".to_string(),
completed_child_count: 0,
ublock_skipped: false,
cookie_ext_skipped: false,
});
@ -1144,8 +1494,10 @@ pub fn perform_capture(
&html_hash,
&file_extension,
byte_size,
entry_title,
)?;
entry_title,
None,
None,
)?;
// 5. Add favicon artifact if we captured one.
if let Some((fav_relpath, fav_blob_id)) = favicon_info {
@ -1195,6 +1547,7 @@ pub fn perform_capture(
return Ok(CaptureResult {
run_uid: run.run_uid.clone(),
status: "completed".to_string(),
completed_child_count: 0,
ublock_skipped: result.ublock_skipped,
cookie_ext_skipped: result.cookie_ext_skipped,
});
@ -1252,6 +1605,7 @@ pub fn perform_capture(
return Ok(CaptureResult {
run_uid: run.run_uid.clone(),
status: "completed".to_string(),
completed_child_count: 0,
ublock_skipped: false,
cookie_ext_skipped: false,
});
@ -1275,6 +1629,7 @@ pub fn perform_capture(
let ytdlp_metadata_json: Option<String> = match source {
Source::YouTubeVideo
| Source::YouTubeMusicTrack
| Source::SpotifyTrack
| Source::X
| Source::Instagram
| Source::Facebook
@ -1315,7 +1670,7 @@ pub fn perform_capture(
}
}
}
Source::YouTubeMusicTrack => {
Source::YouTubeMusicTrack | Source::SpotifyTrack => {
// Music tracks are always audio-only regardless of the caller's quality hint.
match downloader::ytdlp::download(path.clone(), store_path, &timestamp, Some("audio"), &cookies) {
Ok(result) => result,
@ -1342,14 +1697,7 @@ pub fn perform_capture(
}
}
}
Source::YouTubePlaylist | Source::YouTubeChannel => {
return Err(fail_run(
&conn,
&run,
&item,
"Playlist and channel container expansion are not yet implemented.",
));
}
Source::YouTubePlaylist | Source::YouTubeChannel => unreachable!(),
_ => unreachable!(),
};
let temp_file = store_path
@ -1402,6 +1750,8 @@ pub fn perform_capture(
&file_extension,
byte_size,
entry_title,
None,
None,
)?;
database::refresh_entry_cached_bytes(&conn, media_entry.id)?;
database::finish_archive_run(&conn, run.id)?;
@ -1409,6 +1759,7 @@ pub fn perform_capture(
Ok(CaptureResult {
run_uid: run.run_uid.clone(),
status: "completed".to_string(),
completed_child_count: 0,
ublock_skipped: false,
cookie_ext_skipped: false,
})
@ -1790,6 +2141,15 @@ mod tests {
url: "youtube:@CoreDumpped",
expected: Source::YouTubeChannel,
},
// Bare video ID — exactly 11 chars [A-Za-z0-9_-]
TestCase { url: "yt:dQw4w9WgXcQ", expected: Source::YouTubeVideo },
TestCase { url: "youtube:dQw4w9WgXcQ", expected: Source::YouTubeVideo },
TestCase { url: "yt:a_b-c_d-e_4", expected: Source::YouTubeVideo },
// Non-ID: wrong length (9 chars) → Other
TestCase { url: "yt:not-video", expected: Source::Other },
// Reserved prefixes still route correctly when segment looks like an ID
TestCase { url: "yt:playlist/dQw4w9WgXcQ", expected: Source::YouTubePlaylist },
TestCase { url: "yt:@dQw4w9WgXcQ", expected: Source::YouTubeChannel },
];
for case in &shorthand_cases {
@ -1802,6 +2162,18 @@ mod tests {
}
}
#[test]
fn test_is_youtube_video_id() {
assert!(is_youtube_video_id("dQw4w9WgXcQ"), "canonical ID");
assert!(is_youtube_video_id("a_b-c_d-e_4"), "11-char ID with _ and -");
assert!(!is_youtube_video_id(""), "empty");
assert!(!is_youtube_video_id("short"), "too short");
assert!(!is_youtube_video_id("toolong12345"), "too long (12 chars)");
assert!(!is_youtube_video_id("dQw4w9WgXc!"), "invalid char (! at pos 11)");
assert!(!is_youtube_video_id("not-video"), "9 chars — too short");
assert!(!is_youtube_video_id("dQw4w9WgXC!!"), "12 chars + bad char");
}
#[test]
fn test_youtube_music_sources() {
// --- determine_source ---
@ -2326,4 +2698,41 @@ mod tests {
);
}
#[test]
fn locator_to_playlist_url_accepts_playlist_shorthands() {
// yt: playlist shorthand
assert_eq!(
locator_to_playlist_url("yt:playlist/PLtest123"),
Some("https://www.youtube.com/playlist?list=PLtest123".to_string()),
);
// YouTube Music playlist shorthand
assert_eq!(
locator_to_playlist_url("ytm:playlist/PLmus456"),
Some("https://music.youtube.com/playlist?list=PLmus456".to_string()),
);
// Full YT playlist URL passes through (expand_shorthand_to_url is identity for full URLs)
let full = "https://www.youtube.com/playlist?list=PLabc";
assert_eq!(locator_to_playlist_url(full), Some(full.to_string()));
}
#[test]
fn locator_to_playlist_url_accepts_channel_shorthands() {
let url = locator_to_playlist_url("yt:@MyChan");
assert!(url.is_some(), "channel @ shorthand should return Some");
let u = url.unwrap();
assert!(u.contains("youtube.com"), "should be a youtube URL: {u}");
}
#[test]
fn locator_to_playlist_url_rejects_non_playlist_sources() {
// Single video
assert_eq!(locator_to_playlist_url("yt:dQw4w9WgXcQ"), None);
// Tweet
assert_eq!(locator_to_playlist_url("tweet:1234567890"), None);
// YTM track
assert_eq!(locator_to_playlist_url("ytm:MntbN1DdEP0"), None);
// Plain web URL
assert_eq!(locator_to_playlist_url("https://example.com/page"), None);
}
}

105
crates/archivr-core/src/database.rs
View file

@ -390,6 +390,7 @@ pub fn initialize_schema(conn: &Connection) -> Result<()> {
JOIN blobs b ON b.id = ea.blob_id
WHERE ea.entry_id = archived_entries.id
AND ea.blob_id IS NOT NULL
AND ea.artifact_role != 'avatar'
AND EXISTS (
SELECT 1
FROM entry_artifacts ea2
@ -401,6 +402,33 @@ pub fn initialize_schema(conn: &Connection) -> Result<()> {
)
);",
)?;
} else {
// Re-migration: strip avatar blobs from cached_bytes on entries that
// already had the column populated before this filter was introduced.
// Scoped to entries with avatar artifacts only; fast and idempotent.
conn.execute_batch(
"UPDATE archived_entries
SET cached_bytes = (
SELECT COALESCE(SUM(b.byte_size), 0)
FROM entry_artifacts ea
JOIN blobs b ON b.id = ea.blob_id
WHERE ea.entry_id = archived_entries.id
AND ea.blob_id IS NOT NULL
AND ea.artifact_role != 'avatar'
AND EXISTS (
SELECT 1
FROM entry_artifacts ea2
JOIN archived_entries e2 ON e2.id = ea2.entry_id
WHERE ea2.blob_id = ea.blob_id
AND (e2.archived_at < archived_entries.archived_at
OR (e2.archived_at = archived_entries.archived_at
AND e2.id < archived_entries.id))
)
)
WHERE id IN (
SELECT DISTINCT entry_id FROM entry_artifacts WHERE artifact_role = 'avatar'
);",
)?;
}
// Migration: add notes_json column to existing capture_jobs tables.
@ -1409,11 +1437,7 @@ pub fn finish_archive_run(conn: &Connection, run_id: i64) -> Result<()> {
[run_id],
|row| row.get(0),
)?;
let status = if failed_count > 0 {
"failed"
} else {
"completed"
};
let status = if failed_count > 0 { "failed" } else { "completed" };
conn.execute(
"UPDATE archive_runs SET status = ?1, finished_at = ?2 WHERE id = ?3",
params![status, now_timestamp(), run_id],
@ -1421,6 +1445,20 @@ pub fn finish_archive_run(conn: &Connection, run_id: i64) -> Result<()> {
Ok(())
}
/// Returns the number of completed **child** archive_run_items for a run.
///
/// Excludes the container item (parent_item_id IS NULL) so that a playlist
/// where every video fails still returns 0 even though the container item
/// itself was completed before child downloads started.
pub fn get_run_completed_child_count(conn: &Connection, run_id: i64) -> Result<i64> {
Ok(conn.query_row(
"SELECT COUNT(*) FROM archive_run_items
WHERE run_id = ?1 AND status = 'completed' AND parent_item_id IS NOT NULL",
[run_id],
|row| row.get(0),
)?)
}
pub fn fail_archive_run(conn: &Connection, run_id: i64, error_summary: &str) -> Result<()> {
refresh_run_counters(conn, run_id)?;
conn.execute(
@ -1482,6 +1520,7 @@ pub fn refresh_entry_cached_bytes(conn: &Connection, entry_id: i64) -> Result<()
JOIN archived_entries e ON e.id = ea.entry_id
WHERE ea.entry_id = ?1
AND ea.blob_id IS NOT NULL
AND ea.artifact_role != 'avatar'
AND EXISTS (
SELECT 1
FROM entry_artifacts ea2
@ -1518,6 +1557,7 @@ pub fn cascade_cached_bytes_after_delete(conn: &Connection, entry_id: i64) -> Re
JOIN blobs b ON b.id = ea.blob_id
WHERE ea.entry_id = archived_entries.id
AND ea.blob_id IS NOT NULL
AND ea.artifact_role != 'avatar'
AND EXISTS (
SELECT 1
FROM entry_artifacts ea3
@ -1570,6 +1610,7 @@ fn cascade_cached_bytes_after_subtree_delete(conn: &Connection, subtree_ids: &[i
JOIN blobs b ON b.id = ea.blob_id
WHERE ea.entry_id = archived_entries.id
AND ea.blob_id IS NOT NULL
AND ea.artifact_role != 'avatar'
AND EXISTS (
SELECT 1
FROM entry_artifacts ea3
@ -1627,9 +1668,14 @@ pub fn delete_entry(conn: &Connection, entry_uid: &str) -> Result<bool> {
None => return Ok(false),
};
// Collect the full subtree while rows still exist.
// Collect the full subtree (entry itself + any descendants) while rows still exist.
// Must include the entry itself: for a child entry root_entry_id = ?1 returns nothing
// (no grandchildren), so without `id = ?1` the set would be empty and
// cascade_cached_bytes_after_subtree_delete would not recalculate shared-blob totals.
let subtree_ids: Vec<i64> = {
let mut stmt = conn.prepare("SELECT id FROM archived_entries WHERE root_entry_id = ?1")?;
let mut stmt = conn.prepare(
"SELECT id FROM archived_entries WHERE id = ?1 OR root_entry_id = ?1",
)?;
stmt.query_map([entry_id], |row| row.get(0))?
.collect::<rusqlite::Result<_>>()?
};
@ -1638,12 +1684,15 @@ pub fn delete_entry(conn: &Connection, entry_uid: &str) -> Result<bool> {
// shared blobs with any subtree member, excluding every subtree ID simultaneously.
cascade_cached_bytes_after_subtree_delete(conn, &subtree_ids)?;
// Null the FK that has no ON DELETE action (covers root and all descendants).
// Null the FK that has no ON DELETE action. Must cover:
// - The entry itself (child entry: root_entry_id = playlist root, not self)
// - All descendants (root entry: children have root_entry_id = entry_id)
conn.execute(
"UPDATE archive_run_items SET produced_entry_id = NULL
WHERE produced_entry_id IN (
SELECT id FROM archived_entries WHERE root_entry_id = ?1
)",
WHERE produced_entry_id = ?1
OR produced_entry_id IN (
SELECT id FROM archived_entries WHERE root_entry_id = ?1
)",
[entry_id],
)?;
@ -3896,4 +3945,38 @@ mod tests {
"file must be protected because artifact.relpath references it directly"
);
}
#[test]
fn completed_child_count_excludes_container() {
// Regression: get_run_completed_child_count must not count the root container
// item. Scenario: complete both the container item and one child item; total
// DB completed_count == 2, but completed_child_count must == 1.
let c = conn();
let user_id = ensure_default_user(&c).unwrap();
let run = create_archive_run(&c, user_id, 2).unwrap();
// Root item (parent_item_id IS NULL) — mirrors what record_container_entry does.
let root_item = create_archive_run_item(
&c, run.id, None, 0, "https://example.com/pl", None, "youtube", "playlist",
).unwrap();
// Child item (parent_item_id IS NOT NULL).
let child_item = create_archive_run_item(
&c, run.id, Some(root_item.id), 1, "https://example.com/v1", None, "youtube", "video",
).unwrap();
// Complete both — marks archive_runs.completed_count = 2.
c.execute(
"UPDATE archive_run_items SET status = 'completed' WHERE id IN (?1, ?2)",
rusqlite::params![root_item.id, child_item.id],
).unwrap();
refresh_run_counters(&c, run.id).unwrap();
let total: i64 = c.query_row(
"SELECT completed_count FROM archive_runs WHERE id = ?1", [run.id], |r| r.get(0),
).unwrap();
assert_eq!(total, 2, "both items completed: DB counter must be 2");
let child_count = get_run_completed_child_count(&c, run.id).unwrap();
assert_eq!(child_count, 1, "only the child item must be counted");
}
}

258
crates/archivr-core/src/downloader/ytdlp.rs
View file

@ -6,10 +6,50 @@ use std::{
process::Command,
};
use uuid::Uuid;
use serde_json;
use crate::downloader::cookies::{domain_from_url, write_netscape_cookie_file};
use crate::hash::hash_file;
/// A single item in a flat playlist listing from `fetch_playlist_info`.
#[derive(Debug)]
pub struct PlaylistItem {
pub id: String,
pub url: String,
pub title: Option<String>,
pub uploader: Option<String>,
}
/// Container metadata returned by `fetch_playlist_info`.
#[derive(Debug)]
pub struct PlaylistInfo {
pub playlist_id: String,
pub title: Option<String>,
pub uploader: Option<String>,
pub items: Vec<PlaylistItem>,
}
/// Per-item quality data returned by `probe_playlist_qualities`.
#[derive(Debug, serde::Serialize)]
pub struct PlaylistItemProbe {
pub id: String,
pub url: String,
pub title: Option<String>,
/// Available video heights as strings (e.g. "1080p"), sorted highest-first.
/// Empty vec means audio-only (no video track).
pub qualities: Vec<String>,
pub has_audio: bool,
}
/// Full playlist probe result with per-item quality data.
#[derive(Debug, serde::Serialize)]
pub struct PlaylistProbeResult {
pub playlist_id: String,
pub title: Option<String>,
pub uploader: Option<String>,
pub items: Vec<PlaylistItemProbe>,
}
/// Returns the yt-dlp `-f` format selector for `quality`.
///
/// - `"audio"` → prefers native Opus/WebM (most efficient), then native
@ -74,6 +114,23 @@ pub fn available_video_heights(json: &str) -> Vec<u32> {
heights
}
/// Extracts distinct video heights from a serde_json entry Value's `formats` array,
/// sorted highest-first. Audio-only formats (vcodec == "none") are excluded.
fn available_video_heights_from_value(entry: &serde_json::Value) -> Vec<u32> {
let Some(fmts) = entry.get("formats").and_then(|v| v.as_array()) else {
return vec![];
};
let mut heights: Vec<u32> = fmts
.iter()
.filter(|f| f.get("vcodec").and_then(|c| c.as_str()).unwrap_or("none") != "none")
.filter_map(|f| f.get("height").and_then(|h| h.as_u64()).map(|h| h as u32))
.filter(|&h| h > 0)
.collect();
heights.sort_unstable_by(|a, b| b.cmp(a));
heights.dedup();
heights
}
/// Returns true when the yt-dlp `--dump-json` response contains at least one
/// format with a real audio codec (i.e. `acodec != "none"`).
pub fn has_audio_track(json: &str) -> bool {
@ -241,6 +298,207 @@ pub fn fetch_metadata(path: &str, cookies: &HashMap<String, String>) -> Option<S
if json.trim().is_empty() { None } else { Some(json) }
}
/// Resolves an absolute item URL from a flat-playlist entry JSON object.
///
/// Priority:
/// 1. `webpage_url` — yt-dlp makes this absolute when present.
/// 2. `url` when it is already an absolute HTTP(S) URL.
/// 3. Platform-specific fallback constructed from `id` + `container_url`:
/// - YouTube Music → `https://music.youtube.com/watch?v={id}`
/// - YouTube → `https://www.youtube.com/watch?v={id}`
/// - Spotify → `https://open.spotify.com/track/{id}`
/// - Other → `None` (caller should skip the item and warn).
fn normalize_item_url(
entry: &serde_json::Value,
id: &str,
container_url: &str,
) -> Option<String> {
let is_abs = |s: &str| s.starts_with("http://") || s.starts_with("https://");
if let Some(u) = entry.get("webpage_url").and_then(|v| v.as_str()).filter(|s| is_abs(s)) {
return Some(u.to_owned());
}
if let Some(u) = entry.get("url").and_then(|v| v.as_str()).filter(|s| is_abs(s)) {
return Some(u.to_owned());
}
// Bare-ID fallback keyed on the container's platform.
if container_url.contains("music.youtube.com") {
Some(format!("https://music.youtube.com/watch?v={id}"))
} else if container_url.contains("youtube.com") || container_url.contains("youtu.be") {
Some(format!("https://www.youtube.com/watch?v={id}"))
} else if container_url.contains("open.spotify.com") {
Some(format!("https://open.spotify.com/track/{id}"))
} else {
eprintln!("warn: skipping playlist item {id:?} — no absolute URL from yt-dlp");
None
}
}
/// Runs `yt-dlp -J --flat-playlist <url>` and parses the single-JSON result.
///
/// `-J` / `--dump-single-json` returns one JSON object for the whole
/// container with reliable top-level `title` / `uploader` fields plus an
/// `entries` array of shallow per-item objects.
///
/// Returns an error if yt-dlp fails, the output is not valid JSON, or
/// the root `_type` is not `"playlist"`.
pub fn fetch_playlist_info(url: &str, cookies: &HashMap<String, String>) -> Result<PlaylistInfo> {
let ytdlp = std::env::var("ARCHIVR_YT_DLP").unwrap_or_else(|_| "yt-dlp".to_string());
let cookie_file: Option<PathBuf> = if !cookies.is_empty() {
let domain = domain_from_url(url);
let p = std::env::temp_dir()
.join(format!("archivr-cookies-{}.txt", Uuid::new_v4().simple()));
write_netscape_cookie_file(cookies, &domain, &p)
.context("failed to write yt-dlp cookie file")?;
Some(p)
} else {
None
};
let mut cmd = std::process::Command::new(&ytdlp);
cmd.arg("-J").arg("--flat-playlist");
if let Some(cf) = &cookie_file {
cmd.arg("--cookies").arg(cf);
}
cmd.arg(url);
let out = cmd.output();
if let Some(cf) = &cookie_file {
let _ = std::fs::remove_file(cf);
}
let out = out.with_context(|| format!("failed to spawn {ytdlp}"))?;
if !out.status.success() {
let stderr = String::from_utf8_lossy(&out.stderr);
bail!("yt-dlp -J --flat-playlist failed for {url}: {stderr}");
}
let json: serde_json::Value = serde_json::from_slice(&out.stdout)
.context("yt-dlp -J output is not valid JSON")?;
let ty = json.get("_type").and_then(|v| v.as_str()).unwrap_or("");
if ty != "playlist" {
bail!("yt-dlp output _type is {ty:?}, expected \"playlist\"");
}
let playlist_id = json
.get("id")
.and_then(|v| v.as_str())
.unwrap_or("")
.to_string();
let title = json.get("title").and_then(|v| v.as_str()).map(str::to_owned);
let uploader = json.get("uploader").and_then(|v| v.as_str()).map(str::to_owned);
let raw_entries = json
.get("entries")
.and_then(|v| v.as_array())
.map(|a| a.as_slice())
.unwrap_or(&[]);
let mut items = Vec::with_capacity(raw_entries.len());
for entry in raw_entries {
if entry.is_null() {
continue; // unavailable/private item in flat listing
}
let id = match entry.get("id").and_then(|v| v.as_str()) {
Some(s) => s.to_owned(),
None => continue,
};
let item_url = match normalize_item_url(entry, &id, url) {
Some(u) => u,
None => continue,
};
let item_title = entry.get("title").and_then(|v| v.as_str()).map(str::to_owned);
let item_uploader = entry.get("uploader").and_then(|v| v.as_str()).map(str::to_owned);
items.push(PlaylistItem { id, url: item_url, title: item_title, uploader: item_uploader });
}
Ok(PlaylistInfo { playlist_id, title, uploader, items })
}
/// Runs `yt-dlp -J <url>` (full metadata, NOT --flat-playlist) and returns
/// per-item quality data for every entry in the playlist.
///
/// This makes one yt-dlp subprocess call that fetches full format data for
/// all videos — expensive for large playlists but gives accurate per-video
/// quality lists. Intended for pre-capture quality selection only.
pub fn probe_playlist_qualities(
url: &str,
cookies: &HashMap<String, String>,
) -> Result<PlaylistProbeResult> {
let ytdlp = std::env::var("ARCHIVR_YT_DLP").unwrap_or_else(|_| "yt-dlp".to_string());
let cookie_file: Option<PathBuf> = if !cookies.is_empty() {
let domain = domain_from_url(url);
let p = std::env::temp_dir()
.join(format!("archivr-cookies-{}.txt", Uuid::new_v4().simple()));
write_netscape_cookie_file(cookies, &domain, &p)
.context("failed to write yt-dlp cookie file")?;
Some(p)
} else {
None
};
let mut cmd = std::process::Command::new(&ytdlp);
cmd.arg("-J"); // full metadata — NOT --flat-playlist
if let Some(cf) = &cookie_file {
cmd.arg("--cookies").arg(cf);
}
cmd.arg(url);
let out = cmd.output();
if let Some(cf) = &cookie_file {
let _ = std::fs::remove_file(cf);
}
let out = out.with_context(|| format!("failed to spawn {ytdlp}"))?;
if !out.status.success() {
let stderr = String::from_utf8_lossy(&out.stderr);
bail!("yt-dlp -J failed for {url}: {stderr}");
}
let json: serde_json::Value = serde_json::from_slice(&out.stdout)
.context("yt-dlp -J output is not valid JSON")?;
let ty = json.get("_type").and_then(|v| v.as_str()).unwrap_or("");
if ty != "playlist" {
bail!("yt-dlp output _type is {ty:?}, expected \"playlist\"");
}
let playlist_id = json.get("id").and_then(|v| v.as_str()).unwrap_or("").to_string();
let title = json.get("title").and_then(|v| v.as_str()).map(str::to_owned);
let uploader = json.get("uploader").and_then(|v| v.as_str()).map(str::to_owned);
let raw_entries = json
.get("entries")
.and_then(|v| v.as_array())
.map(|a| a.as_slice())
.unwrap_or(&[]);
let mut items = Vec::with_capacity(raw_entries.len());
for entry in raw_entries {
if entry.is_null() { continue; }
let id = match entry.get("id").and_then(|v| v.as_str()) {
Some(s) => s.to_owned(),
None => continue,
};
let item_url = match normalize_item_url(entry, &id, url) {
Some(u) => u,
None => continue,
};
let item_title = entry.get("title").and_then(|v| v.as_str()).map(str::to_owned);
let heights = available_video_heights_from_value(entry);
let qualities: Vec<String> = heights.iter().map(|h| format!("{h}p")).collect();
let has_audio = entry
.get("formats").and_then(|v| v.as_array())
.map(|fmts| fmts.iter().any(|f| {
f.get("acodec").and_then(|c| c.as_str()).unwrap_or("none") != "none"
}))
.unwrap_or(false);
items.push(PlaylistItemProbe { id, url: item_url, title: item_title, qualities, has_audio });
}
Ok(PlaylistProbeResult { playlist_id, title, uploader, items })
}
#[cfg(test)]
mod tests {
use super::{available_video_heights, has_audio_track, quality_format};