archivr/docs/PLAN.md

# Archivr Database Design Plan

## Summary
Design the first database as a `SQLite` metadata/index layer for the existing file-based archive store, while making the schema multi-user and public-archive ready from day one. The filesystem remains the source of truth for bytes and rendered archive output; the database becomes the source of truth for users, roles, archive runs, archived entries, visibility, hierarchy, blob reuse, and organization.

Each successfully archived thing becomes its own archived entry. Re-archiving the same source creates a new archived entry row, while deduplicated raw files continue to reuse the same blob rows underneath.

## Key Changes
### Identity, access, and visibility
- `users`
  - Columns: stable public `user_uid`, `username`, `email` nullable, `password_hash`, `status`, `role`, `created_at`, `last_login_at` nullable.
  - Roles: `admin`, `user`.
- `instance_settings`
  - Global booleans for `public_index_enabled`, `public_entry_content_enabled`, `public_archive_submission_enabled`.
  - Defaults all `false`.
- `archived_entries`
  - Add `created_by_user_id`, `owned_by_user_id`, `visibility`.
  - `visibility` values: `private`, `unlisted`, `public`.
- `archive_runs`
  - Add `created_by_user_id`.
- Do not add groups or per-entry ACL tables in v1; keep the schema portable enough to add them later.

### Core archive model
- `archive_runs`
  - One user-started archive operation.
  - Columns: stable public `run_uid`, `created_by_user_id`, `started_at`, `finished_at`, `status`, `requested_count`, `discovered_count`, `completed_count`, `failed_count`, `error_summary`.
- `archive_run_items`
  - One requested or discovered work item inside an archive run.
  - Columns: `run_id`, stable `item_uid`, `parent_item_id` nullable, `ordinal`, `requested_locator`, `canonical_locator` nullable, `source_kind`, `entity_kind`, `status`, `error_text`, `produced_entry_id` nullable.
  - Supports batch requests and container expansion with progress like `0/14`.
- `source_identities`
  - Canonical identity of the thing being archived across re-archives.
  - Columns: `source_kind`, `entity_kind`, `external_id` nullable, `canonical_url` nullable, `normalized_locator`, `identity_key`.
  - Unique constraint on `identity_key`.
- `archived_entries`
  - One archived thing shown in the archive.
  - Columns: stable public `entry_uid`, `source_identity_id`, `archive_run_id`, `parent_entry_id` nullable, `root_entry_id`, `created_by_user_id`, `owned_by_user_id`, `source_kind`, `entity_kind`, `title` nullable, `visibility`, `archived_at`, `original_published_at` nullable, `structured_root_relpath`, `representation_kind`, `source_metadata_json`, `display_metadata_json` nullable.
  - `structured_root_relpath` is required and points to one root under `structured/<entry_uid>/`.
  - Main archive view queries only rows with `parent_entry_id IS NULL`.
  - Child entries remain first-class rows but are nested under the parent in the main view.
- `blobs`
  - One deduplicated raw file in `raw/`.
  - Columns: `sha256`, `byte_size`, `mime_type` nullable, `extension` nullable, `raw_relpath`, `created_at`.
- `entry_artifacts`
  - Selective file pointers attached to an archived entry.
  - Columns: `entry_id`, `artifact_role`, `storage_area`, `relpath`, `blob_id` nullable, `logical_path` nullable, `metadata_json` nullable.
  - `storage_area`: `raw`, `raw_tweets`, `structured`.
  - Store important files only: primary media, raw tweet JSON, avatar, subtitle, thumbnail, manifest, cover image.

### Organization and extensibility
- `taxonomy_nodes`
  - Hierarchical organization tree.
  - Columns: stable `node_uid`, `parent_id` nullable, `name`, `slug`, `full_path`.
  - `full_path` unique, example `/sciences/computer-science/compilers`.
- `entry_taxonomy_assignments`
  - Many-to-many link between archived entries and taxonomy nodes.
  - Assign the most specific node; ancestor membership is derived via recursive queries.
- Keep shared fields relational and source-specific details in `source_metadata_json`.
  - YouTube examples: `video_id`, `channel_id`, duration, playlist membership.
  - Tweet examples: `tweet_id`, `author_handle`, conversation ID, text summary fields.
  - Do not create per-source tables in v1.

### Public/archive access behavior implied by schema
- Public archive browsing is controlled by both instance settings and entry visibility.
- `public` entries are eligible for anonymous listing/viewing only when instance-level public settings allow it.
- `unlisted` entries are not shown in public indexes but can be directly served later by URL/token design.
- `private` entries are visible only to authorized users.
- Ownership is recorded now even if the first UI only exposes simple admin/user behavior.

## Public APIs / Interfaces
- `archivr init`
  - Create the SQLite database and schema alongside the existing archive metadata directory.
  - Keep existing store directories.
- `archivr archive`
  - Start one `archive_run` owned by a user.
  - Insert one or more `archive_run_items`.
  - On success, create one or more `archived_entries`.
  - Link reused raw files through `blobs` and `entry_artifacts`.
  - Record the entry’s `structured_root_relpath`, visibility, and source metadata JSON.
- New persisted domain types
  - `User`
  - `ArchiveRun`
  - `ArchiveRunItem`
  - `ArchivedEntry`
  - `SourceIdentity`
  - `Blob`
  - `EntryArtifact`
  - `TaxonomyNode`
  - `InstanceSettings`

## Test Plan
- Re-archiving the same YouTube video creates two `archived_entries`, one shared `source_identity`, and one shared primary `blob`.
- Archiving a tweet/thread creates one archived entry, records the raw tweet JSON as an `entry_artifact` in `raw_tweets`, and links downloaded media/avatar blobs correctly.
- Archiving a playlist/channel creates one top-level parent entry plus child entries; the main archive query returns only the parent.
- A single archive run with multiple requested locators records multiple run items and correct progress counters.
- A normal user can create entries but cannot manage other users or instance-wide public settings.
- An admin can manage users and instance-wide public settings.
- A `public` entry is still hidden from anonymous users when `public_index_enabled` or `public_entry_content_enabled` is disabled at the instance level.
- A `private` entry never appears in anonymous/public queries.
- Assigning `/sciences/computer-science/compilers` makes the item discoverable through ancestor queries for `sciences` and `computer-science`.
- A website-style entry can be represented as one archived entry with one structured root and no per-asset DB explosion.

## Assumptions
- SQLite is the only target for the first implementation, but the schema should avoid SQLite-only modeling that would block a later Postgres migration.
- The database indexes archive metadata; archive bytes stay on disk.
- Every archived entry gets a stable public ID used for `structured/<entry_uid>/`; timestamps are metadata, not identity.
- `raw_tweets/` remains a valid sibling storage area and is referenced through `entry_artifacts`.
- Titles are optional and nullable.
- Search, FTS, subtitles, transcript indexing, groups, and per-entry ACL sharing are deferred.
- Organization uses hierarchical taxonomy only for now; free-form tags are out of scope.
- The first permissions model matches the simpler ArchiveBox-style shape: admins, normal users, and optional public visibility, without custom group policy in v1.