Add SQLite metadata database support

docs/PLAN.md

@@ -0,0 +1,111 @@
+# Archivr Database Design Plan
+
+## Summary
+Design the first database as a `SQLite` metadata/index layer for the existing file-based archive store, while making the schema multi-user and public-archive ready from day one. The filesystem remains the source of truth for bytes and rendered archive output; the database becomes the source of truth for users, roles, archive runs, archived entries, visibility, hierarchy, blob reuse, and organization.
+
+Each successfully archived thing becomes its own archived entry. Re-archiving the same source creates a new archived entry row, while deduplicated raw files continue to reuse the same blob rows underneath.
+
+## Key Changes
+### Identity, access, and visibility
+- `users`
+  - Columns: stable public `user_uid`, `username`, `email` nullable, `password_hash`, `status`, `role`, `created_at`, `last_login_at` nullable.
+  - Roles: `admin`, `user`.
+- `instance_settings`
+  - Global booleans for `public_index_enabled`, `public_entry_content_enabled`, `public_archive_submission_enabled`.
+  - Defaults all `false`.
+- `archived_entries`
+  - Add `created_by_user_id`, `owned_by_user_id`, `visibility`.
+  - `visibility` values: `private`, `unlisted`, `public`.
+- `archive_runs`
+  - Add `created_by_user_id`.
+- Do not add groups or per-entry ACL tables in v1; keep the schema portable enough to add them later.
+
+### Core archive model
+- `archive_runs`
+  - One user-started archive operation.
+  - Columns: stable public `run_uid`, `created_by_user_id`, `started_at`, `finished_at`, `status`, `requested_count`, `discovered_count`, `completed_count`, `failed_count`, `error_summary`.
+- `archive_run_items`
+  - One requested or discovered work item inside an archive run.
+  - Columns: `run_id`, stable `item_uid`, `parent_item_id` nullable, `ordinal`, `requested_locator`, `canonical_locator` nullable, `source_kind`, `entity_kind`, `status`, `error_text`, `produced_entry_id` nullable.
+  - Supports batch requests and container expansion with progress like `0/14`.
+- `source_identities`
+  - Canonical identity of the thing being archived across re-archives.
+  - Columns: `source_kind`, `entity_kind`, `external_id` nullable, `canonical_url` nullable, `normalized_locator`, `identity_key`.
+  - Unique constraint on `identity_key`.
+- `archived_entries`
+  - One archived thing shown in the archive.
+  - Columns: stable public `entry_uid`, `source_identity_id`, `archive_run_id`, `parent_entry_id` nullable, `root_entry_id`, `created_by_user_id`, `owned_by_user_id`, `source_kind`, `entity_kind`, `title` nullable, `visibility`, `archived_at`, `original_published_at` nullable, `structured_root_relpath`, `representation_kind`, `source_metadata_json`, `display_metadata_json` nullable.
+  - `structured_root_relpath` is required and points to one root under `structured/<entry_uid>/`.
+  - Main archive view queries only rows with `parent_entry_id IS NULL`.
+  - Child entries remain first-class rows but are nested under the parent in the main view.
+- `blobs`
+  - One deduplicated raw file in `raw/`.
+  - Columns: `sha256`, `byte_size`, `mime_type` nullable, `extension` nullable, `raw_relpath`, `created_at`.
+- `entry_artifacts`
+  - Selective file pointers attached to an archived entry.
+  - Columns: `entry_id`, `artifact_role`, `storage_area`, `relpath`, `blob_id` nullable, `logical_path` nullable, `metadata_json` nullable.
+  - `storage_area`: `raw`, `raw_tweets`, `structured`.
+  - Store important files only: primary media, raw tweet JSON, avatar, subtitle, thumbnail, manifest, cover image.
+
+### Organization and extensibility
+- `taxonomy_nodes`
+  - Hierarchical organization tree.
+  - Columns: stable `node_uid`, `parent_id` nullable, `name`, `slug`, `full_path`.
+  - `full_path` unique, example `/sciences/computer-science/compilers`.
+- `entry_taxonomy_assignments`
+  - Many-to-many link between archived entries and taxonomy nodes.
+  - Assign the most specific node; ancestor membership is derived via recursive queries.
+- Keep shared fields relational and source-specific details in `source_metadata_json`.
+  - YouTube examples: `video_id`, `channel_id`, duration, playlist membership.
+  - Tweet examples: `tweet_id`, `author_handle`, conversation ID, text summary fields.
+  - Do not create per-source tables in v1.
+
+### Public/archive access behavior implied by schema
+- Public archive browsing is controlled by both instance settings and entry visibility.
+- `public` entries are eligible for anonymous listing/viewing only when instance-level public settings allow it.
+- `unlisted` entries are not shown in public indexes but can be directly served later by URL/token design.
+- `private` entries are visible only to authorized users.
+- Ownership is recorded now even if the first UI only exposes simple admin/user behavior.
+
+## Public APIs / Interfaces
+- `archivr init`
+  - Create the SQLite database and schema alongside the existing archive metadata directory.
+  - Keep existing store directories.
+- `archivr archive`
+  - Start one `archive_run` owned by a user.
+  - Insert one or more `archive_run_items`.
+  - On success, create one or more `archived_entries`.
+  - Link reused raw files through `blobs` and `entry_artifacts`.
+  - Record the entry’s `structured_root_relpath`, visibility, and source metadata JSON.
+- New persisted domain types
+  - `User`
+  - `ArchiveRun`
+  - `ArchiveRunItem`
+  - `ArchivedEntry`
+  - `SourceIdentity`
+  - `Blob`
+  - `EntryArtifact`
+  - `TaxonomyNode`
+  - `InstanceSettings`
+
+## Test Plan
+- Re-archiving the same YouTube video creates two `archived_entries`, one shared `source_identity`, and one shared primary `blob`.
+- Archiving a tweet/thread creates one archived entry, records the raw tweet JSON as an `entry_artifact` in `raw_tweets`, and links downloaded media/avatar blobs correctly.
+- Archiving a playlist/channel creates one top-level parent entry plus child entries; the main archive query returns only the parent.
+- A single archive run with multiple requested locators records multiple run items and correct progress counters.
+- A normal user can create entries but cannot manage other users or instance-wide public settings.
+- An admin can manage users and instance-wide public settings.
+- A `public` entry is still hidden from anonymous users when `public_index_enabled` or `public_entry_content_enabled` is disabled at the instance level.
+- A `private` entry never appears in anonymous/public queries.
+- Assigning `/sciences/computer-science/compilers` makes the item discoverable through ancestor queries for `sciences` and `computer-science`.
+- A website-style entry can be represented as one archived entry with one structured root and no per-asset DB explosion.
+
+## Assumptions
+- SQLite is the only target for the first implementation, but the schema should avoid SQLite-only modeling that would block a later Postgres migration.
+- The database indexes archive metadata; archive bytes stay on disk.
+- Every archived entry gets a stable public ID used for `structured/<entry_uid>/`; timestamps are metadata, not identity.
+- `raw_tweets/` remains a valid sibling storage area and is referenced through `entry_artifacts`.
+- Titles are optional and nullable.
+- Search, FTS, subtitles, transcript indexing, groups, and per-entry ACL sharing are deferred.
+- Organization uses hierarchical taxonomy only for now; free-form tags are out of scope.
+- The first permissions model matches the simpler ArchiveBox-style shape: admins, normal users, and optional public visibility, without custom group policy in v1.