AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary
Python ai-promptsai-skillblueskyclaudeclaude-codeclawhubdeep-researchhackernewsinstagramopenclawpolymarketrecencyredditresearchsocial-mediatiktoktrendstwitterweb-searchyoutube
fix(youtube): SSH transcript routing and surface SSH search errors (#422)
Trevin Chow91987294cda7
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 8ca3f967..84b2d8ce 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,8 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added-### Added-+- **`LAST30DAYS_YOUTUBE_SSH_HOST` transcript routing** — yt-dlp transcript fetch runs on the remote SSH host via a mktemp + cat pipeline ([#422](https://github.com/mvanhorn/last30days-skill/pull/422)).- Browser-cookie auth for X/Twitter now covers the full Chromium family on macOS - Brave, Microsoft Edge, Vivaldi, Opera, Arc, and Chromium - alongside the existing Chrome, Firefox, and Safari. They all share Chrome's v10 AES-128-CBC decryption, differing only in profile path and Keychain service name, so they run through one shared decryption core. The profile finder probes both the modern `Default/Network/Cookies` layout (Chromium >= 96) and the legacy flat `Default/Cookies`, and Chrome now resolves through that same finder so it picks up the modern layout too. Set `FROM_BROWSER=auto` to try every browser, or `FROM_BROWSER=<name>` (e.g. `brave`, `edge`, `arc`) to target one. Verified end-to-end on real Brave and Edge installs ([#572](https://github.com/mvanhorn/last30days-skill/pull/572)).- **First-party positioning research + pitch-vs-pulse synthesis (company / product / service topics).** A new mandatory research step captures each entity's current stated positioning from first-party sources (homepage, docs, pricing) rather than from memory. The fetched pitch grounds `What it is` descriptions (entities described as they pitch themselves today), helps reject unrelated brand-name noise, and feeds an evidence-triggered prose beat: when the month's conversation directly supports a specific claim, cuts against one, or is squarely about the pitched ground, the synthesis says so anchored to the top thread — and stays silent when the pulse is orthogonal to the pitch, because a manufactured connection is worse than omission. Claims are tested at matched altitude (specific claims against specific threads; broad taglines are never graded against individual items), and statements stay windowed to the 30 days — no trend verdicts. Scoped to entities with an identifiable first party: people are always excluded (even founders whose companies qualify), as are events, abstract concepts, and ownerless topics like Bitcoin; the beat requires positioning fetched during the run, never from memory.@@ -20,6 +19,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **SSH routing failures no longer present as "0 results"** — `search_youtube` surfaces non-zero SSH exit codes as an explicit `error` field ([#422](https://github.com/mvanhorn/last30days-skill/pull/422)).- `extract_browser_credentials()` silently ignored Brave even though the lower-level `cookie_extract` layer already supported it: `FROM_BROWSER=brave` fell back to Firefox/Safari and `FROM_BROWSER=auto` never tried Brave. The env wiring now passes Brave - and the rest of the Chromium family - through to the extractor ([#572](https://github.com/mvanhorn/last30days-skill/pull/572)).- Chromium cookie extraction now fetches the macOS Keychain key lazily - only when an encrypted cookie actually needs decrypting. Previously the key was fetched as soon as the cookie DB existed, so `FROM_BROWSER=auto` could trigger a Keychain prompt for every installed Chromium browser. Now only the browser that actually holds the requested cookie prompts ([#572](https://github.com/mvanhorn/last30days-skill/pull/572)).- YouTube transcript budget prioritises recent videos (by a combination of views and recency) instead of views alone, preventing transcript slots from being consumed by old high-view-count videos that would be discarded by strict_recent freshness pruning ([#531](https://github.com/mvanhorn/last30days-skill/issues/531))diff --git a/skills/last30days/scripts/lib/youtube_yt.py b/skills/last30days/scripts/lib/youtube_yt.pyindex 33742d6f..05624ef4 100644--- a/skills/last30days/scripts/lib/youtube_yt.py+++ b/skills/last30days/scripts/lib/youtube_yt.py@@ -313,6 +313,7 @@ def search_youtube("--no-download",]cmd = _wrap_ytdlp_cmd(cmd)+ ssh_host = _ytdlp_ssh_host()try:result = subproc.run_with_timeout(cmd, timeout=120)@@ -323,6 +324,17 @@ def search_youtube(return {"items": [], "error": "yt-dlp not found"}stdout = result.stdout+ if ssh_host and result.returncode != 0 and not stdout.strip():+ stderr_first = (result.stderr or "").strip().splitlines()+ first_line = stderr_first[0] if stderr_first else "(no stderr)"+ _log(+ f"YouTube search via SSH host {ssh_host!r} failed "+ f"(rc={result.returncode}): {first_line}"+ )+ return {+ "items": [],+ "error": f"SSH routing to {ssh_host!r} failed: {first_line}",+ }if not stdout.strip():_log("YouTube search returned 0 results")return {"items": []}@@ -511,6 +523,45 @@ def _fetch_transcript_direct(return vtt_text+def _fetch_transcript_ytdlp_via_ssh(video_id: str, ssh_host: str) -> Optional[str]:+ """Fetch transcript via yt-dlp on a remote SSH host (mktemp + cat pipeline)."""+ if not _SSH_HOST_ALIAS_RE.match(ssh_host):+ return None+ url = f"https://www.youtube.com/watch?v={video_id}"+ quoted_url = shlex.quote(url)+ sub_langs = shlex.quote(_ytdlp_sub_langs())+ remote_script = (+ "set -e; "+ "TMPD=$(mktemp -d); "+ "yt-dlp --ignore-config --no-cookies-from-browser "+ f"--write-auto-subs --sub-lang {sub_langs} --sub-format vtt "+ "--skip-download --no-warnings "+ f'-o "$TMPD/%(id)s" {quoted_url} >/dev/null 2>&1 || true; '+ 'VTT=$(find "$TMPD" -maxdepth 1 -name "*.vtt" 2>/dev/null | head -1); '+ '[ -n "$VTT" ] && cat "$VTT"; '+ 'rm -rf "$TMPD"'+ )+ cmd = ["ssh", "-o", "BatchMode=yes", "--", ssh_host, remote_script]+ try:+ result = subproc.run_with_timeout(cmd, timeout=45)+ except subproc.SubprocTimeout:+ _log(f"SSH yt-dlp transcript timed out for {video_id} via {ssh_host!r}")+ return None+ except FileNotFoundError:+ _log("ssh executable not found; cannot route transcript fetch")+ return None+ out = result.stdout or ""+ if not out.strip().startswith("WEBVTT"):+ if result.returncode != 0 and result.stderr:+ first_line = result.stderr.strip().splitlines()[0]+ _log(+ f"SSH yt-dlp transcript via {ssh_host!r} failed for "+ f"{video_id} (rc={result.returncode}): {first_line}"+ )+ return None+ return out++def _ytdlp_sub_langs() -> str:"""Caption languages to try, from LAST30DAYS_YT_SUB_LANGS (default en,es,pt)."""raw = os.environ.get("LAST30DAYS_YT_SUB_LANGS", "").strip()@@ -663,30 +714,23 @@ def fetch_transcript(Plaintext transcript string, or None if no captions available."""raw_vtt = None- # When SSH-routing is on, the yt-dlp transcript path would write a VTT- # file on the remote host that we can't easily read back. Skip it and- # use the HTTP transcript fallback (different YouTube endpoint, less- # bot-walled, works fine from datacenter IPs).ssh_host = _ytdlp_ssh_host()- use_ytdlp = is_ytdlp_installed() and not ssh_host- if use_ytdlp:+ if ssh_host and is_ytdlp_installed():+ raw_vtt = _fetch_transcript_ytdlp_via_ssh(video_id, ssh_host)+ if not raw_vtt:+ _log(f"SSH yt-dlp transcript failed for {video_id}, trying direct HTTP fallback")+ raw_vtt = _fetch_transcript_direct(video_id, status=status)+ elif is_ytdlp_installed():raw_vtt = _fetch_transcript_ytdlp(video_id, temp_dir, status=status)if not raw_vtt:ytdlp_error = (status or {}).get("ytdlp_error")if ytdlp_error:- # yt-dlp hit a real error (rate-limit / bot-check / timeout) —- # captions likely exist. The direct-HTTP fallback hits the same- # wall and returns an empty body that masquerades as "no- # captions", so skip it and surface the real reason instead._log(f"Transcript fetch failed for {video_id}: {ytdlp_error}")return None_log(f"yt-dlp found no captions for {video_id}, trying direct HTTP fallback")raw_vtt = _fetch_transcript_direct(video_id, status=status)else:- if ssh_host:- _log("SSH-routing active, using direct HTTP transcript fetch")- else:- _log("yt-dlp not installed, using direct HTTP transcript fetch")+ _log("yt-dlp not installed, using direct HTTP transcript fetch")raw_vtt = _fetch_transcript_direct(video_id, status=status)if not raw_vtt:diff --git a/tests/test_youtube_yt.py b/tests/test_youtube_yt.pyindex ab03d62d..9735473f 100644--- a/tests/test_youtube_yt.py+++ b/tests/test_youtube_yt.py@@ -774,5 +774,60 @@ class TestYtdlpSSHRouting(unittest.TestCase):self.assertIn("--ignore-config", cmd[5])self.assertIn("--no-cookies-from-browser", cmd[5])+ def test_search_surfaces_ssh_failure_as_error(self):+ """SSH connection failures surface as an error, not silent '0 results'."""+ os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = "macmini"+ from lib.subproc import SubprocResult+ fake_result = SubprocResult(+ returncode=255,+ stdout="",+ stderr="ssh: connect to host macmini port 22: Connection refused\n",+ )+ with mock.patch.object(youtube_yt.subproc, "run_with_timeout",+ return_value=fake_result):+ out = youtube_yt.search_youtube("test", "2026-02-01", "2026-03-01")+ self.assertIn("error", out)+ self.assertIn("Connection refused", out["error"])+++class TestTranscriptSSHRouting(unittest.TestCase):+ """LAST30DAYS_YOUTUBE_SSH_HOST routes yt-dlp transcript fetches through SSH."""++ def setUp(self):+ self._saved_env = os.environ.pop("LAST30DAYS_YOUTUBE_SSH_HOST", None)++ def tearDown(self):+ os.environ.pop("LAST30DAYS_YOUTUBE_SSH_HOST", None)+ if self._saved_env is not None:+ os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = self._saved_env++ def test_ssh_helper_invokes_remote_mktemp_pipeline(self):+ os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = "macmini"+ from lib.subproc import SubprocResult+ fake_vtt = "WEBVTT\nKind: captions\nLanguage: en\n\n00:00:00.000 --> 00:00:02.000\nhi\n"+ fake_result = SubprocResult(returncode=0, stdout=fake_vtt, stderr="")+ with mock.patch.object(youtube_yt.subproc, "run_with_timeout",+ return_value=fake_result) as run_mock:+ out = youtube_yt._fetch_transcript_ytdlp_via_ssh("vid1", "macmini")+ self.assertEqual(out, fake_vtt)+ remote_script = run_mock.call_args.args[0][5]+ self.assertIn("mktemp -d", remote_script)+ self.assertIn("find ", remote_script)++ def test_fetch_transcript_uses_ssh_helper_when_routing_on(self):+ os.environ["LAST30DAYS_YOUTUBE_SSH_HOST"] = "macmini"+ fake_vtt = "WEBVTT\n\n00:00:00.000 --> 00:00:02.000\nhello there friends\n"+ with mock.patch.object(youtube_yt, "is_ytdlp_installed", return_value=True), \+ mock.patch.object(youtube_yt, "_fetch_transcript_ytdlp_via_ssh",+ return_value=fake_vtt) as ssh_mock, \+ mock.patch.object(youtube_yt, "_fetch_transcript_ytdlp") as local_mock, \+ mock.patch.object(youtube_yt, "_fetch_transcript_direct") as direct_mock:+ result = youtube_yt.fetch_transcript("vidX", "/tmp/test")+ ssh_mock.assert_called_once_with("vidX", "macmini")+ local_mock.assert_not_called()+ direct_mock.assert_not_called()+ self.assertIn("hello there friends", result)++if __name__ == "__main__":unittest.main()