mirror of
https://github.com/thegeneralist01/archivr
synced 2026-10-09 12:55:00 +02:00
TheGeneralist
fd3f6903e2
Per-entry LLM summaries as a regenerable child record, not a column on archived_entries and not an on-disk artifact: an entry may carry several summaries (one per provider/model/prompt version), any of which can be discarded and recomputed. Generation is manual-only — nothing in capture.rs calls into this module. - database.rs: entry_summaries table + index, EntrySummaryRecord, and upsert/update/find/latest helpers mirroring the capture_jobs style. provider_model is stored as '' rather than NULL because SQLite treats NULLs as distinct inside a UNIQUE index, which would stop the CLI providers (no model) from ever deduping on the cache key. - summarizer.rs: SummaryProvider trait with four implementations — Anthropic Messages API, OpenAI-compatible chat completions, `claude -p` and `codex exec -`. Configuration comes from env vars only (never TOML), matching how yt-dlp / single-file / tweet-scraper are resolved, which also keeps API keys out of anything the archive persists. - archive.rs: EntryDetail gains latest_summary, populated by one extra LIMIT 1 query in get_entry_detail. EntrySummaryView aliases the DB row rather than duplicating it. Implementation notes: - No tokio in core. CLI timeouts are enforced structurally: stdout is drained on its own thread and handed back over a channel so the calling thread can recv_timeout and kill an overrunning child; stdin is written on a third thread so a 48 KB prompt cannot deadlock against a child waiting for us to read. - HTML is reduced with regex rather than a parser: html5ever is not in the tree, and a model tolerates imperfect whitespace. Paired tags are spelled out per tag because Rust's regex engine has no backreferences by design. - reqwest is declared with only the `blocking` feature here, so bodies are serialized via .body(value.to_string()) instead of widening the workspace dependency for .json(). - input_sha256 holds a SHA3-256 digest via hash::hash_bytes, the tree's one hashing primitive; the content is truncated to 48 KB *before* hashing so the cache key describes exactly the bytes the model saw. Tests: no mockito/wiremock in dev-deps, and adding a mock HTTP server for one JSON shape is a poor trade, so the two halves that can actually break are tested directly — request-body builders and response parsers — leaving only reqwest's own transport uncovered. Plus schema idempotency, cache-key dedupe, cascade-on-delete, provider_from_env happy/missing-var paths, HTML and tweet extraction, output normalization, and the CLI runner's stdin round-trip, timeout kill, and nonzero-exit paths. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| archivr-cli | ||
| archivr-core | ||
| archivr-server | ||