1
Fork 0
mirror of https://github.com/thegeneralist01/archivr synced 2026-10-09 21:03:17 +02:00
Commit graph

266 commits

Author SHA1 Message Date
archivr-qa
fce3bb147e
Merge branch 'sol-ytdlp' into integration-all-three 2026-08-24 16:30:24 +02:00
archivr-qa
bc66649cc8
Merge branch 'sol-text-backend' into integration-all-three 2026-08-24 16:30:24 +02:00
archivr-qa
26df7dd2e9
Merge branch 'sol-summary-core' into integration-all-three
# Conflicts:
#	crates/archivr-core/src/database.rs
#	crates/archivr-core/src/summarizer.rs
2026-08-24 16:29:49 +02:00
archivr-qa
53fab8c76c
Merge branch 'sol-summary-lifecycle' into integration-all-three 2026-08-24 16:24:00 +02:00
archivr-qa
0b5ac0a516
test: cover tweet image summary selection 2026-08-24 16:22:13 +02:00
archivr-qa
43acfd3c58
fix: preserve missing summary entry status 2026-08-24 16:22:11 +02:00
archivr-qa
e371b37290
test: cover terminal newline in text capture 2026-08-24 16:13:05 +02:00
archivr-qa
f0cbdc3175
fix: protect public summary diagnostics 2026-08-24 15:31:45 +02:00
archivr-qa
d56fd8b41b
fix: bound codex fallback and record resolved model 2026-08-24 15:31:10 +02:00
archivr-qa
e5b9fc8ae0
fix: recover and preserve summary attempts 2026-08-24 15:31:04 +02:00
archivr-qa
771c64a811
fix: preserve text capture bytes and hide synthetic URL 2026-08-24 15:30:56 +02:00
archivr-qa
b4f6b9fc69
fix: infer trusted MIME for tweet images 2026-08-24 15:29:39 +02:00
archivr-qa
6dbde86575
fix: show forced yt-dlp candidate in status 2026-08-24 15:29:25 +02:00
archivr-qa
492b1a3168
chore(static): rebuild bundle for civilized summary errors 2026-08-24 14:41:15 +02:00
archivr-qa
8f36617c5d
Merge branch 'fix/civil-summary-errors' into integration-all-three 2026-08-24 14:39:18 +02:00
archivr-qa
b38cd19c42
fix: show civilized unsupported summary errors 2026-08-24 14:37:04 +02:00
archivr-qa
e2e998482e
chore(static): rebuild bundle for summary image consent 2026-08-23 22:05:25 +02:00
archivr-qa
c63fc7903e
Merge branch 'docs-xarticle-vision' into integration-all-three 2026-08-23 21:59:43 +02:00
archivr-qa
af6e473bde
docs: explain X Article, image summaries and summary search 2026-08-23 21:57:04 +02:00
archivr-qa
ecc25af05a
Merge branch 'summary-images-ui' into integration-all-three 2026-08-23 21:48:12 +02:00
archivr-qa
7051d459db
feat: add summary image consent control 2026-08-23 21:46:45 +02:00
archivr-qa
ed3bc56e14
Merge branch 'summary-images-core' into integration-all-three 2026-08-23 21:33:58 +02:00
archivr-qa
2c79c83a9e
feat: accept image summary requests 2026-08-23 21:32:26 +02:00
archivr-qa
a107b849b7
feat: attach opted-in images to summaries 2026-08-23 21:19:15 +02:00
archivr-qa
3a6de87e10
feat: model opt-in summary images 2026-08-23 21:05:53 +02:00
archivr-qa
6c6de347d1
Merge branch 'x-article-summary' into integration-all-three 2026-08-23 20:53:15 +02:00
archivr-qa
55a74ea3c2
fix: preserve X Article block order 2026-08-23 20:51:19 +02:00
archivr-qa
55f96c66e9
Merge branch 'summary-search' into integration-all-three 2026-08-23 20:44:33 +02:00
archivr-qa
3a48d2bfca
feat: search completed summary tags 2026-08-23 20:42:01 +02:00
archivr-qa
b17db8d04a
fix: summarize X Article text 2026-08-23 20:42:00 +02:00
archivr-qa
7a10fedaf2
docs(plan): specify X Article, image summaries, summary search 2026-08-23 20:26:45 +02:00
archivr-qa
ccb4d6f8a4
docs(readme): document LLM summaries, text notes, yt-dlp resolver + bump paths
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 20:02:29 +02:00
archivr-qa
44137a8eb1
docs(maintainer): document summarizer, text capture, and yt-dlp lifecycle
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 19:47:48 +02:00
archivr-qa
953512cbff
Merge branch 'ci/update-ytdlp-from-github-releases' into integration-all-three 2026-08-23 19:23:50 +02:00
archivr-qa
d8cd1cf9d4
Merge branch 'feat/yt-dlp-self-update' into integration-all-three 2026-08-23 19:23:50 +02:00
archivr-qa
585d0a049b
feat(cli): add archivr yt-dlp update|status subcommand
`update` fetches the latest release tag from the GitHub API (or takes
--version), downloads the cross-platform python zipapp, and installs it
into archivr's state dir. The install is atomic — staged as yt-dlp.new,
chmod +x'd, then renamed over the target — so a concurrently running
capture never sees a half-written binary. A sibling .version file makes a
repeat update a no-op instead of a 3MB re-download.

The download is checked for the python3 shebang before install, which
catches the usual failure mode of getting an HTML error page back. python3
itself is only warned about, not required: the server may run under a nix
wrapper with its own PATH.

`status` prints all three candidates (env / state-dir / PATH fallback) with
their versions and stars whichever the resolver picks, so it is obvious
which yt-dlp a capture will actually use.

reqwest is pulled from the existing workspace dependency; the GitHub JSON is
parsed with serde_json so the "json" feature is not needed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 19:22:45 +02:00
archivr-qa
d0e8e8beef
ci: bump yt-dlp from upstream releases, not nixpkgs
The flake no longer takes yt-dlp from nixpkgs; a dedicated `ytDlp`
derivation fetches the upstream release binary directly and pins both
`version` and an SRI `hash`. That makes the previous workflow inert: it
ran `nix flake update nixpkgs` and compared `nixpkgs#yt-dlp.version`
before and after, so it could churn the lockfile forever without ever
moving the version we actually ship.

The workflow now reads the pinned version straight out of the `ytDlp`
block in flake.nix, asks the GitHub API for yt-dlp's latest release tag,
short-circuits when they already match, downloads the new release to
recompute its SRI hash (required — the hash is part of the derivation's
identity, so the URL cannot be changed alone), and rewrites the three
pinned fields under a sed range address scoped to that block so sibling
pins like ublockLite and isdcac are untouched. It asserts only flake.nix
changed and that the new version appears exactly twice before opening
the PR.
2026-08-23 19:21:06 +02:00
archivr-qa
4e7bd086ea
feat(core): resolve_yt_dlp picks the newer of pinned vs state-dir
The nix flake wrapper pins a yt-dlp via ARCHIVR_YT_DLP, but yt-dlp rots
fast — extractors break within weeks of a pin. Add a resolver that probes
`--version` on both the pinned binary and a user-installed copy under the
mutable state dir, and runs whichever is newer.

Version strings are YYYY.MM.DD, so plain string ordering is chronological.
Ties resolve toward the state dir: a user who installed it there did so
deliberately. ARCHIVR_YT_DLP_FORCE bypasses the comparison entirely, and
with no candidate at all we fall back to bare `yt-dlp` on PATH — exactly
the previous behaviour.

Resolution is cached in a OnceLock so `--version` costs one subprocess per
process, and all four inline env::var lookups now go through it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 19:20:58 +02:00
archivr-qa
a5d422b68b
chore(nix): pin python 3.12 for yt-dlp zipapp (avoid py3.14 libffi crash on darwin/arm64) 2026-08-23 19:05:23 +02:00
archivr-qa
43ddf9d6b1
chore(static): rebuild bundle for round-3 fixes 2026-08-23 19:03:31 +02:00
archivr-qa
bb2ab4f8db
Merge branch 'fix/nix-flake-yt-dlp-pin' into integration-all-three 2026-08-23 19:03:25 +02:00
archivr-qa
73b0010d60
Merge branch 'feat/archive-text-with-title' into integration-all-three
# Conflicts:
#	crates/archivr-server/static/assets/index-C4rpqY7g.js
#	crates/archivr-server/static/index.html
2026-08-23 19:03:25 +02:00
archivr-qa
e1dcec5d53
Merge branch 'feat/llm-summaries' into integration-all-three 2026-08-23 19:03:25 +02:00
archivr-qa
23ed13d821
fix(nix): pin yt-dlp from its own release + wire into server wrapper
Two independent problems, one commit:

1. Stale binary. nixpkgs-provided `pkgs.yt-dlp` on the pinned
   nixos-unstable rev is 2026.03.17 (Mar 2026). yt-dlp itself
   releases days-to-weeks, and YouTube frequently rotates the
   player-signature / client surfaces the older builds request
   (`android_vr` is the current casualty), which returns HTTP 403
   mid-download for the format specs archivr passes (`-f
   bestvideo+bestaudio/best`). Even bumping the nixpkgs input would
   leave us dependent on that channel's yt-dlp cadence.

   Fetch the upstream zipapp directly instead
   (github.com/yt-dlp/yt-dlp/releases/download/<ver>/yt-dlp), wrap so
   `python3` and `ffmpeg` are on PATH, and pin version+hash in one
   place. Bumping is: change version, replace hash from
   `nix hash file <url>`.

2. Missing pin in server wrapper. `archivr-cli` was already wrapped
   with `--set ARCHIVR_YT_DLP` + a PATH prefix; `archivr-server`
   was NOT — it only pinned single-file, chrome, and the tweet
   scraper, silently falling back to whatever `yt-dlp` the user
   happened to have on PATH. Server captures therefore inherited
   the user's (often stale) system yt-dlp regardless of the flake
   pin. Same wrapper flags now apply to both binaries.

devShell keeps `pkgs.yt-dlp` for now: the dev shell is a
convenience, not a release surface, and matching wouldn't fit in this
commit without duplicating the derivation across let-scopes.
2026-08-23 19:03:08 +02:00
archivr-qa
e8bb03e08f
feat(frontend): preview text-capture entries (.md / .txt)
Text captures land as `.md` (Markdown) or `.txt` (plain) blobs, but
PreviewPanel only dispatched on video/audio/image/pdf/html extensions,
so opening a text entry hit the 'No preview available' fallback with
the raw artifact path exposed.

- New `TextPreview` component fetches the primary artifact as text,
  renders it in a monospace `<pre>` with word-wrap, and shows the
  entry title on top and the MIME as a small trailing tag. Handles
  loading/error states.
- `PreviewPanel` gains a `TEXT_EXTS` set + a branch that dispatches
  to `TextPreview` for `md` / `markdown` / `txt`.
- CSS is padded and centered to ~780px so a text note reads like a
  document rather than an edge-to-edge terminal dump.

v1 intentionally does NOT parse Markdown: keeping frontend deps at
react+react-dom only. Bump to a real Markdown renderer if we start
capturing Markdown-authored notes.
2026-08-23 19:02:46 +02:00
archivr-qa
189fe2d392
fix(core): summarize tweets + walk all tweets in a thread
Tweet and tweet_thread entries store their payload under artifact_role
`raw_tweet_json`, not `primary_media`. `build_summary_input` filtered
strictly for `primary_media LIMIT 1`, so both cases silently failed
with 'entry X has no primary_media artifact to summarize'.

Threads compound the problem: the tweet scraper writes ONE json file
per status, so even a fixed lookup that took the first row would
summarize only the initial tweet and lose the rest of the conversation.

Fixes:
- New `load_summary_artifacts` helper returns every artifact for a
  role in insertion order.
- For entity_kind `tweet` / `tweet_thread`, load all
  `raw_tweet_json` artifacts (falling back to `primary_media` for
  archives predating that role convention).
- Iterate artifacts, extract text per file with the existing
  markdown/html/json branches, then join thread pieces with a
  `---` separator so the model sees a real paragraph break between
  statuses instead of one flowing document.

Single-tweet entries produce one piece and the separator never
renders. Non-tweet entries behave exactly as before.
2026-08-23 19:02:29 +02:00
archivr-qa
a5c2e347e6
chore(static): rebuild bundle after text-row CSS merge 2026-08-23 18:49:35 +02:00
archivr-qa
f9b50b3ae9
Merge branch 'feat/archive-text-with-title' into integration-all-three
# Conflicts:
#	crates/archivr-server/static/assets/index-CQcpPA_I.js
#	crates/archivr-server/static/index.html
2026-08-23 18:49:31 +02:00
archivr-qa
067d766819
fix(frontend): give text-capture row real CSS
The text row shipped with semantic classnames (`capture-text-inputs`,
`capture-text-title`, `capture-text-body`, `capture-text-mime`,
`capture-text-icon`) but no CSS rules. Falling through to the parent
`.capture-row-main` flex-row (`display: flex; align-items: center`)
meant the title, textarea, and mime-select stacked as intrinsic-width
boxes centered on the tall body, producing a layout where the body
floated to the top-right, the title box appeared BELOW it, and the mime
selector rendered as an unstyled OS dropdown.

Fix:
- `.capture-text-row .capture-row-main` uses `align-items: flex-start`
  so the leading icon and trailing × pin to the top of the block.
- `.capture-text-inputs` is now a full-width column-flex container with
  proper gaps.
- `.capture-text-title` reuses the 44px input height and typography of
  `.capture-input`; `.capture-text-body` gets a 140px min-height,
  vertical resize, and matching border/focus treatment.
- `.capture-text-mime` is styled as a small chip with a custom caret
  so it matches `.capture-quality` and stops looking like a raw
  `<select>`. Sits in a right-aligned footer under the body.
- `.capture-text-icon` gets a 44px column so it aligns with the title
  input; remove button gets a small top-margin for the same reason.

Rebuilt static bundle bumped as well (`index-BLxoi9rt.css`,
`index-CQcpPA_I.js`).
2026-08-23 18:49:15 +02:00
archivr-qa
121ed15714
Merge branch 'feat/llm-summaries' into integration-all-three 2026-08-23 18:42:07 +02:00