1
Fork 0
mirror of https://github.com/thegeneralist01/archivr synced 2026-07-22 11:15:41 +02:00
archivr/docs/superpowers/plans/2026-07-20-playlist-children.md
TheGeneralist ccdacfd582
feat(core): YouTube playlist/channel/YTM-playlist capture with parent–child entries
- ytdlp: add fetch_playlist_info() using yt-dlp -J --flat-playlist for
  reliable container title + shallow entry list; normalize item URLs via
  webpage_url → absolute url → id fallback (domain inferred from container
  URL so YTM stays on music.youtube.com)
- capture: add record_container_entry() (no blob, no primary_media artifact);
  extend record_media_entry() with parent_entry_id/root_entry_id params (all
  existing single-item call sites pass None, None)
- capture: implement YouTubePlaylist / YouTubeChannel / YouTubeMusicPlaylist
  capture path replacing the two not-implemented stubs: fetch playlist info →
  create container entry (reusing existing run + item) → per-child run items
  (parent_item_id = container item) → download each video/track as a child
  entry; per-child failures are non-fatal; perform_capture returns result.status
  reflecting actual run outcome so capture_handler marks the job correctly
- archive: add child_count i64 to EntrySummary (col 12 in all listing queries);
  add get_entry_summary() private helper; fix get_entry_detail() to use
  get_entry_summary() so child entries are resolvable via the detail endpoint;
  add list_child_entries(conn, uid, caller_bits) with the same
  admin/collection visibility predicate as list_root_entries

archive_runs.requested_count stays 1 (one user locator); discovered/
completed/failed_count reflect container item + N video items via
refresh_run_counters.
2026-07-20 16:07:00 +02:00

8.9 KiB
Raw Blame History

Playlist / Channel Captures + ParentChild Entry Display

Date: 2026-07-20 Branch: feat-captures-with-children


Goal

  1. YouTube (Music) playlists and channels — when the user submits a playlist or channel URL/shorthand, archive every video in it as a tree of entries: one root "container" entry for the playlist/channel, one child entry per video.
  2. Parentchild display — the entry list and detail view can show children of a container entry in a collapsible inline section, replacing the current flat/single-entry-only model.

Schema Contract (no migrations needed — columns already exist)

Column Meaning
archived_entries.parent_entry_id NULL for root entries; points to container id for children
archived_entries.root_entry_id Same as parent_entry_id for depth-1 children (direct children of a container)
archive_run_items.parent_item_id NULL for the container item; points to container item id for video items
archive_runs.requested_count Always 1 (user submitted one locator)
archive_runs.discovered_count N + 1 — container item + N video items (from refresh_run_counters)
archive_runs.completed_count Items completed (container + each successful video)
archive_runs.failed_count Items that failed to download

Counter note: refresh_run_counters counts every archive_run_items row, so discovered_count = 1 + N when a playlist has N videos. This is intentional and accurate — the container item itself counts as a discovered/completed item.


yt-dlp Probe Strategy

For container sources (playlist, channel, YouTube Music playlist), use:

yt-dlp -J --flat-playlist <url>

-J (--dump-single-json) returns a single JSON object for the whole container — reliable title and uploader fields at the top level, plus an entries array (shallow per-item objects with at minimum id, url, title). This avoids the per-item playlist_title reliability issue that --dump-json (multi-line) has.

Individual child video downloads stay single-item (--no-playlist), same as current video capture.


File Map

File Action What changes
crates/archivr-core/src/downloader/ytdlp.rs Modify Add PlaylistItem, PlaylistInfo structs; fetch_playlist_info()
crates/archivr-core/src/capture.rs Modify record_media_entry gains parent_entry_id/root_entry_id params; add record_container_entry(); implement playlist/channel/YTM-playlist capture paths
crates/archivr-core/src/archive.rs Modify Add child_count: i64 to EntrySummary; add get_entry_summary(); fix get_entry_detail() to handle child entries; add list_child_entries()
crates/archivr-server/src/routes.rs Modify Add GET /api/archives/:id/entries/:uid/children route + handler
frontend/src/api.js Modify Add fetchEntryChildren()
frontend/src/components/EntryRow.jsx Modify Expand toggle when child_count > 0; inline child sub-rows on expand
frontend/src/styles.css Modify Child row indentation + expand toggle styling

Detailed Design

ytdlp.rs additions

pub struct PlaylistItem {
    pub id: String,
    pub url: String,
    pub title: Option<String>,
    pub uploader: Option<String>,
}

pub struct PlaylistInfo {
    pub playlist_id: String,
    pub title: Option<String>,
    pub uploader: Option<String>,
    pub items: Vec<PlaylistItem>,
}

/// Runs `yt-dlp -J --flat-playlist <url>` and parses the result.
/// Returns Err if yt-dlp fails or the output isn't a playlist object.
pub fn fetch_playlist_info(url: &str, cookies: &HashMap<String, String>) -> Result<PlaylistInfo>

capture.rs changes

record_media_entry signature extension:

fn record_media_entry(
    ...,
    title: Option<String>,
    parent_entry_id: Option<i64>,  // NEW
    root_entry_id: Option<i64>,    // NEW
) -> Result<database::ArchivedEntry>

All existing call sites pass None, None. Playlist child calls pass the container entry's ids.

New record_container_entry():

  • Creates an entry with no blob and no primary_media artifact.
  • source_kind/entity_kind from source_metadata(source).
  • parent_entry_id: None, root_entry_id: None.
  • Stores playlist metadata in source_metadata_json.
  • Returns the ArchivedEntry (needed for child parent_entry_id).
  • Calls database::complete_archive_run_item() on the container run item.

Playlist/channel capture path (replaces return Err(...) stubs):

1. fetch_playlist_info(url, cookies)            → PlaylistInfo
2. create_archive_run(conn, user_id, 1)         → run (requested_count=1)
3. create_archive_run_item(run, None, 0, ..., "playlist"/"channel", "container")
4. record_container_entry(...)                  → container_entry
5. complete_archive_run_item(container_item, container_entry.id)
6. for (ordinal, item) in playlist_info.items:
   a. create_archive_run_item(run, Some(container_item.id), ordinal, item.url, ...)
   b. fetch_metadata(item.url, cookies)          → metadata_json (for title)
   c. ytdlp::download(item.url, ...)            → (hash, ext)
   d. record_media_entry(..., Some(container.id), Some(container.id))
      OR: fail_archive_run_item(child_item, error) and continue
7. finish_archive_run(conn, run.id)

Error handling: if a child video fails, call fail_archive_run_item and continue — partial success is correct for playlists.

archive.rs changes

EntrySummary new field:

pub child_count: i64,  // number of direct children; 0 for non-container entries

ENTRY_SELECT_COLS extension (adds col 12):

(SELECT COUNT(*) FROM archived_entries child WHERE child.parent_entry_id = e.id) AS child_count

list_root_entries inline SQL also extended the same way (col 12).

All query_map closures that build EntrySummary get child_count: row.get(12)?.

get_entry_summary(conn, entry_uid) — new private helper:

  • Fetches one entry by uid without the parent_entry_id IS NULL constraint.
  • Used by the fixed get_entry_detail.

get_entry_detail fix:

// BEFORE (broken for child entries):
let summary = list_root_entries(conn, u32::MAX)?
    .into_iter()
    .find(|entry| entry.entry_uid == entry_uid)
    .context("entry disappeared")?;

// AFTER:
let summary = get_entry_summary(conn, entry_uid)?
    .context("entry disappeared")?;

New list_child_entries(conn, parent_uid) -> Result<Vec<EntrySummary>>:

ENTRY_SELECT_COLS ENTRY_FROM_JOINS
WHERE e.parent_entry_id = (SELECT id FROM archived_entries WHERE entry_uid = ?1)
GROUP BY e.id
ORDER BY e.archived_at ASC, e.id ASC

(ascending order — preserves playlist ordinal feel)

routes.rs addition

.route(
    "/api/archives/:archive_id/entries/:entry_uid/children",
    get(list_entry_children),
)
async fn list_entry_children(
    State(state): State<AppState>,
    auth: AuthUser,
    Path((archive_id, entry_uid)): Path<(String, String)>,
) -> Result<Json<Vec<archive::EntrySummary>>, ApiError> {
    auth.require_auth()?;
    let mounted = mounted_archive(&state, &archive_id)?;
    let conn = database::open_or_initialize(&mounted.archive_path)?;
    Ok(Json(archive::list_child_entries(&conn, &entry_uid)?))
}

Frontend changes

api.js:

export async function fetchEntryChildren(archiveId, entryUid) {
  return getJson(`/api/archives/${archiveId}/entries/${entryUid}/children`);
}

EntryRow.jsx:

  • Accept optional archiveId for child fetching (already passed).
  • When entry.child_count > 0: render a chevron expand button in col-title area.
  • Local state: expanded (bool), children (array | null), loading (bool).
  • On chevron click: toggle; if expanding and children === null, call fetchEntryChildren and store result.
  • Render children as <div className="child-entries"> containing simplified <ChildEntryRow> elements (or reuse EntryRow without nesting).

styles.css:

  • .child-entries — slight left indent, separator line.
  • .entry-expand-btn — minimal chevron button.

Acceptance Criteria

  • cargo test -p archivr-core green.
  • cargo check -p archivr-server clean.
  • Submitting yt:playlist/PLxxx archives the playlist as a container entry with N child video entries in the DB.
  • Submitting yt:@handle or yt:channel/UC... does the same for channel uploads.
  • Submitting ytm:playlist/PLxxx does the same for YouTube Music playlists.
  • Individual video/track captures (yt:video/ID, ytm:ID) unchanged.
  • If one video in a playlist fails, the run status is failed but other children are still archived.
  • GET /api/archives/:id/entries returns root entries only, each with correct child_count.
  • GET /api/archives/:id/entries/:uid works for both root and child entries.
  • GET /api/archives/:id/entries/:uid/children returns the child entries for a container.
  • Frontend expand button appears on container entries; clicking it fetches and shows children inline.