AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary
Python ai-promptsai-skillblueskyclaudeclaude-codeclawhubdeep-researchhackernewsinstagramopenclawpolymarketrecencyredditresearchsocial-mediatiktoktrendstwitterweb-searchyoutube
fix: backfill thin YouTube search results (#1248)
Ilia Alshanetsky5ba637cee328
diff --git a/CONFIGURATION.md b/CONFIGURATION.mdindex 14dba1e4..625c0a0d 100644--- a/CONFIGURATION.md+++ b/CONFIGURATION.md@@ -91,7 +91,7 @@ The Modal and Non-Modal flows share the same consent points:1. **Browser cookies** - the model asks before reading anything and explains that permission covers later research runs. On yes it runs `setup --allow-browser-cookies`. On macOS, the scan checks Chrome and other Chromium browsers first, then Firefox and Safari, unless you already configured `FROM_BROWSER`. On Windows and Linux, this native cookie scan supports Firefox only; under WSL2, it also falls back to Windows Firefox. Unless you already configured `FROM_BROWSER`, setup saves the browser names that supplied complete cookies in `FROM_BROWSER` so later research can read those signed-in sessions again; it never saves the cookie values. If different services use different browsers, it saves their comma-separated names. Chromium reads on macOS may prompt for Keychain access again on later runs; choose **Always Allow** if you want to grant continuing OS access. Setup also installs yt-dlp + the keyless Digg CLI. On no it runs setup without `--allow-browser-cookies` (or with `FROM_BROWSER=off`), which skips all cookie reads and still installs the tools. The engine saves the decision as `BROWSER_CONSENT=true` or `false`; refusal blocks native browser and CDP reads on later runs, even if an older `FROM_BROWSER` selection remains. To disable these reads later, set `BROWSER_CONSENT=false` or `FROM_BROWSER=off` in `~/.config/last30days/.env`. A later consented setup can replace the refusal. An explicit `FROM_BROWSER=off` remains a separate denial until you change it. The `agentcookie` sidecar has its own `AGENTCOOKIE=off` control.2. **Full Disk Access (macOS)** - if a cookie read is permission-denied, the model surfaces the System Settings > Privacy & Security > Full Disk Access fix and offers one retry.-3. **ScrapeCreators GitHub signup** - offered on every first run (10,000 free calls). On consent it runs `setup --github`, which opens a browser for GitHub device-auth (or registers instantly via the `gh` CLI when installed) and, on success, **persists `SCRAPECREATORS_API_KEY` automatically** (0o600, masked in output) so TikTok, Instagram, empty-path Reddit search backup, and the YouTube transcript fallback activate on the next run. Decline anytime; you can run it later by asking to set up ScrapeCreators. The Step 5 opt-in has two tiers, both comment-enabled: **Recommended** (TikTok + Instagram posts AND top comments, plus YouTube comments — `INCLUDE_SOURCES=tiktok,instagram,youtube_comments,tiktok_comments,instagram_comments`) and **Everything**, which also adds Threads + Pinterest. Comments are on by default; Threads and Pinterest are the only opt-in extras.+3. **ScrapeCreators GitHub signup** - offered on every first run (10,000 free calls). On consent it runs `setup --github`, which opens a browser for GitHub device-auth (or registers instantly via the `gh` CLI when installed) and, on success, **persists `SCRAPECREATORS_API_KEY` automatically** (0o600, masked in output). The key enables TikTok and Instagram, Reddit search backfill below 5 free results, YouTube search backfill below 3 yt-dlp videos, and YouTube transcript rescue; these spend credits. Set `LAST30DAYS_YT_SC_MIN_ITEMS=0` for empty-only YouTube search backfill. Decline anytime; you can run it later by asking to set up ScrapeCreators. The Step 5 opt-in has two tiers, both comment-enabled: **Recommended** (TikTok + Instagram posts AND top comments, plus YouTube comments — `INCLUDE_SOURCES=tiktok,instagram,youtube_comments,tiktok_comments,instagram_comments`) and **Everything**, which also adds Threads + Pinterest. Comments are on by default; Threads and Pinterest are the only opt-in extras.To revisit setup, ask the skill to set up a source; deleting the global `.env` alone does not force onboarding when another credential source remains active. The mechanical work lives in `scripts/lib/setup_wizard.py`; the consent conversation and the host flows are specified in Step 0 of `skills/last30days/references/setup-wizard.md`, which the root `SKILL.md` routes to when needed. The original v3.0.0 wizard is captured at `docs/reference/old-nux-wizard-v3.0.0.md`.@@ -158,7 +158,7 @@ python3 skills/last30days/scripts/last30days.py "MCP servers" \| StockTwits | none | auto-on for ticker/crypto topics only (gated by symbol detection); never registered for non-financial topics | yes (public API, ~200 req/hr per IP) || DripStack | none | opt-in only: per run with `--search dripstack`, or persistently with `INCLUDE_SOURCES=dripstack` in `.env`. Searches premium financial newsletters and analyst writeups via a free, public search API — no key needed. Never active without the opt-in. | yes when opted in (public API, no auth) || GitHub | `gh` CLI installed (uses your GitHub auth) | always on if `gh` present | yes |-| YouTube | `yt-dlp` CLI installed; `SCRAPECREATORS_API_KEY` adds a server-side transcript fallback used only when yt-dlp fails (429 / bot-gate) | always on if `yt-dlp` present; SC transcript fallback default-on when key set (no credit spent unless yt-dlp fails) | yes |+| YouTube | `yt-dlp` CLI installed; `SCRAPECREATORS_API_KEY` adds transcript rescue and a search backfill when yt-dlp returns fewer than 3 videos by default | always on if `yt-dlp` present; keyed search backfill and transcript rescue spend ScrapeCreators credits | yes || YouTube comments | `yt-dlp` CLI installed — **free and keyless, no API key and no opt-in needed**. Falls back to `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` containing `youtube_comments` only when yt-dlp is absent. Suppress with `EXCLUDE_SOURCES=youtube_comments`. | top comments (by likes) on the top ~3 videos by engagement | yes — free via yt-dlp (no credits spent) || TikTok comments | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `tiktok_comments` (**on by default** — Step 5 Recommended tier) | top comments (by `digg_count`) on the top ~3 TikTok posts | ~3 calls/run; 10K free calls || Instagram comments | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `instagram_comments` (**on by default** — Step 5 Recommended tier) | top comments (by `comment_like_count`) on the top ~3 Instagram posts, via `/v2/instagram/post/comments` | ~3 calls/run; 10K free calls |@@ -188,6 +188,8 @@ python3 skills/last30days/scripts/last30days.py "MCP servers" \**YouTube transcript tuning.** `LAST30DAYS_YT_SUB_LANGS` controls the comma-separated caption-language priority passed to yt-dlp and defaults to `en,es,pt`. The ScrapeCreators transcript fallback tries at most the first three distinct languages in that order, with a shared 30-second deadline per video; it stops after the first usable transcript. `LAST30DAYS_YT_PLAYER_CLIENT` defaults to `android` so yt-dlp can pass YouTube's web bot-gate without cookies (search, transcripts, and comments); set it empty to disable. When `SCRAPECREATORS_API_KEY` is available, yt-dlp uses one fast attempt before the paid fallback; set `LAST30DAYS_YT_TRANSCRIPT_FAST_TIMEOUT` to the number of seconds allowed for that attempt when a throttled host needs longer than the 12-second default. A VTT completed before the timeout is reused rather than discarded. `LAST30DAYS_YT_SEARCH_TIMEOUT` sets the per-search yt-dlp deadline (default 120s). Comparison-mode fan-out also caps concurrent yt-dlp processes process-wide and caches identical searches within a run so redundant `ytsearch` calls do not self-throttle the same IP.+**YouTube search backfill.** With `SCRAPECREATORS_API_KEY` set, `LAST30DAYS_YT_SC_MIN_ITEMS` defaults to `3`: a yt-dlp search returning 0, 1, or 2 videos makes one ScrapeCreators search per stream attempt. Set `0` for the previous empty-only behavior, or another positive integer to change the floor. An invalid value behaves like `0`, so a typo does not increase paid usage. Each backfill can also fetch transcripts for up to 6 videos at default depth (up to 3 language requests per video); endpoint retries and optional comment enrichment can add calls. A later thin-source retry may run another backfill. Free results and their transcripts remain in the merged result. A clean thin result records backfill detail; actual yt-dlp or ScrapeCreators failures mark partial coverage if retry does not recover.+**X backend priority (bird first).** The default X backend chain is bird (browser cookies) → xai (API key) → xurl (OAuth2 CLI) → xquik (API key). Cookies beat `XAI_API_KEY` when both are present. A leftover grok login never steals the X lane; see below. `xapi` (the official X API v2 with `X_BEARER_TOKEN`) is opt-in on these hosts (`LAST30DAYS_X_BACKEND=xapi`), so an ambient bearer never spends X API credits when the free path comes back empty. **Grok Bot exception.** On a Grok Bot host (`LAST30DAYS_HOST=grok-bot`) the unpinned chain is the official chain instead: xapi (`X_BEARER_TOKEN`) → xai (`XAI_API_KEY`) → xurl (the X API through X's CLI). When the bot fetched X itself (its built-in X tools, or the X connector), those results come first and the chain is not called at all; see [Grok Bot](#grok-bot) under Per-client patterns.**xAI backend errors.** For an `xai:` authentication or model failure, check `XAI_API_KEY` and its chat/model permissions in console.x.ai. Set `LAST30DAYS_X_MODEL` to a model available to that key if the configured model is unavailable. For HTTP 402, check xAI billing and credits; for HTTP 429, wait for the xAI rate limit to reset; for a timeout, retry later. Logging into x.com or replacing `X_BEARER_TOKEN` does not repair the xAI API backend.@@ -575,6 +577,7 @@ Every live run writes its JSON result to `~/.config/last30days/doctor-cache.json| `BROWSER_CDP_URL` | Explicit Chrome DevTools endpoint (e.g. `http://127.0.0.1:18800`) for the extra-host CDP cookie lookup. Preferred over the `18800` / `9222`+`$DISPLAY` defaults. Extra hosts only; setting an endpoint does not grant browser-cookie consent. || `LAST30DAYS_REDDIT_BACKEND` | `scrapecreators` makes ScrapeCreators the primary Reddit backend; doctor renders Reddit's conditional routing with the pin applied. || `LAST30DAYS_REDDIT_SC_MIN_ITEMS` | Integer thinness floor for ScrapeCreators Reddit **search** backfill. Default `5`: backfill when the free path returns fewer than 5 items, spending ScrapeCreators credits. Set another positive count to move the floor, or `0` to backfill only when the free path returns nothing. Unset or blank means `5`; a malformed value behaves like `0` (spends nothing extra). A run makes at most one ScrapeCreators Reddit backfill per distinct query, date window, and subreddit set, and each backfill is several API calls (searches plus comment fetches, roughly 10 at default depth). A failed backfill after free items only adds a detail note. Merged results dedupe by post id. Requires `SCRAPECREATORS_API_KEY`. Ignored when `LAST30DAYS_REDDIT_BACKEND=scrapecreators` (SC is already primary). |+| `LAST30DAYS_YT_SC_MIN_ITEMS` | Keyed YouTube search backfill floor. Default `3`; `0` restores empty-only search fallback. Invalid values behave like `0`. See [YouTube search backfill](#youtube-search-backfill) for per-stream credit usage and retry behavior. |Web search has **no** env pin — pin it per-run with `--web-backend=<name>` only (see [Web search backend priority](#web-search-backend-priority)).diff --git a/changelog.d/977.fixed.md b/changelog.d/977.fixed.mdnew file mode 100644index 00000000..02080afe--- /dev/null+++ b/changelog.d/977.fixed.md@@ -0,0 +1 @@+Backfill thin YouTube search results from ScrapeCreators when a key is set, retain yt-dlp videos and transcripts, and surface backfill and failure details. Set `LAST30DAYS_YT_SC_MIN_ITEMS=0` for empty-only fallback.diff --git a/skills/last30days/SKILL.md b/skills/last30days/SKILL.mdindex ff6ed26c..90e398e7 100644--- a/skills/last30days/SKILL.md+++ b/skills/last30days/SKILL.md@@ -234,7 +234,7 @@ After the response and required citations, stop and wait unless governing instru- Sends search queries to Algolia HN Search API (`hn.algolia.com`) for Hacker News story and comment discovery (free, no auth)- Sends search queries to Polymarket Gamma API (`gamma-api.polymarket.com`) for prediction market discovery (free, no auth)- Runs `yt-dlp` locally for YouTube search and transcript extraction (no API key, public data)-- Sends search queries to ScrapeCreators API (`api.scrapecreators.com`) for TikTok and Instagram search, transcript/caption extraction (10,000 free calls, then PAYG)+- Sends TikTok/Instagram search, YouTube search backfill below the configured floor (default 3; `LAST30DAYS_YT_SC_MIN_ITEMS=0` means empty-only), and transcripts to ScrapeCreators (`api.scrapecreators.com`); keyed calls spend credits (10,000 free, then PAYG).- Optionally sends search queries to Brave Search API, Parallel AI API, Perplexity API (`api.perplexity.ai`), or OpenRouter API for web search / synthesis- Fetches public Reddit thread data from `reddit.com` for engagement metrics- Stores research findings in local SQLite database (watchlist mode only)diff --git a/skills/last30days/references/setup-wizard.md b/skills/last30days/references/setup-wizard.mdindex 7a265d3e..969cc491 100644--- a/skills/last30days/references/setup-wizard.md+++ b/skills/last30days/references/setup-wizard.md@@ -73,12 +73,12 @@ Setup also best-effort installs yt-dlp (YouTube), the free keyless Digg CLI (`di**Step 4: ScrapeCreators offer (every first run unless the Research Continuation Override already started a waiting topic).** Show this as plain text, then a modal:-ScrapeCreators adds TikTok and Instagram - posts AND top comments - plus YouTube comments, all on by default. 10,000 free calls, no credit card. Your key also backfills Reddit **search** when the free path returns fewer than 5 items (set `LAST30DAYS_REDDIT_SC_MIN_ITEMS=0` to backfill only when it finds nothing; Reddit comments already come free via shreddit), and backstops YouTube transcripts if yt-dlp gets throttled. (We don't get a cut.) You can widen coverage even further in the next step.+ScrapeCreators adds TikTok/Instagram posts and top comments, plus YouTube comments. The key grants 10,000 free calls (then PAYG, no card or cut) and spends credits on Reddit search when free results are fewer than 5 items, YouTube search below 3 yt-dlp videos, and YouTube transcript rescue. Set `LAST30DAYS_REDDIT_SC_MIN_ITEMS=0` or `LAST30DAYS_YT_SC_MIN_ITEMS=0` for empty-only search fallback. Each YouTube backfill can fetch transcripts; a retry can spend again. Reddit comments remain free via shreddit.Before the modal, run `which gh` via Bash silently; store as gh_available.**Call AskUserQuestion:**-Question: "Want to add TikTok and Instagram? Your key also backfills empty Reddit search and backs up YouTube when yt-dlp is throttled. (We don't get a cut.)"+Question: "Want to add TikTok and Instagram? The key also spends credits on Reddit search below 5, YouTube search below 3, and transcript rescue. Set LAST30DAYS_YT_SC_MIN_ITEMS=0 for empty-only YouTube search."Options:- "ScrapeCreators via GitHub (recommended - most free calls)" - description: "Opens GitHub - we copy your code to your clipboard automatically, so you just paste it (Cmd+V), ~20-30s. Grants the full 10,000 free calls - more than the web signup." (Recommend this over the web option because the GitHub path grants more free calls.) This is a **two-command flow** - `--github-start` returns the code fast (foreground), then `--github-poll` waits for you to authorize. The code comes back in the command output, so it can't be missed:1. **Run `--github-start` in the FOREGROUND** (it returns in ~1-2s, it does NOT block-poll): `"${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" setup --github-start`. It submits the device flow, copies the code to the clipboard, opens the browser, and returns a JSON blob plus a plain `Your GitHub code: XXXX-XXXX` line on stdout.@@ -86,7 +86,7 @@ Options:- If `status == "error"`: show the message and offer the web option below.2. **SHOW THE CODE.** Read the `user_code` from the output and output ONE chat message: "Enter this code on the GitHub page: **XXXX-XXXX** - it's already on your clipboard, so just paste (Cmd+V) and click Continue." (If the output said the clipboard copy failed, tell them to type it instead.) The code is right there in step 1's output - surfacing it is the whole point.3. **Run `--github-poll`** (background with a 5-minute timeout, or foreground): `"${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" setup --github-poll`. Parse the **LAST** JSON line of its stdout for the final status:- - `status == "success"`: the engine persisted the key (`"persisted": true`, MASKED `api_key` - never ask for or echo the raw key); confirm "You're in! 10,000 free calls. TikTok, Instagram, empty-path Reddit search backup, and YouTube transcript fallback are now active."+ - `status == "success"`: the engine persisted the key (`"persisted": true`, MASKED `api_key` - never echo it); confirm "You're in! 10,000 free calls. TikTok/Instagram, Reddit search below 5, YouTube search below 3, and transcript rescue are active. LAST30DAYS_YT_SC_MIN_ITEMS=0 makes YouTube search empty-only."- `status == "success"` but `"persisted": false` (key write failed): do NOT claim sources are active - tell the user signup worked but saving the key failed, and have them add `SCRAPECREATORS_API_KEY=<key>` to `~/.config/last30days/.env` manually.- `status == "error"` **with `message == "Authorized but failed to fetch API key"`** (exact match; often also `"reason": "no_api_key"`): GitHub authorized fine - do NOT say auth failed. This usually means your GitHub is **already linked** to a ScrapeCreators account. Tell the user: "GitHub authorized, but I couldn't auto-grab your ScrapeCreators key - your GitHub is probably already linked to an account. Get your key at scrapecreators.com and paste it here, or Skip." Then accept a pasted key (write `SCRAPECREATORS_API_KEY` to `.env`) or offer the web/skip options.- `status == "error"` **with `message` containing `ScrapeCreators profile failed`** (or `"reason": "upstream_error"`): GitHub authorized fine - do NOT say auth failed, and do **NOT** say the GitHub is already linked. ScrapeCreators returned a server error after auth. Tell the user: "GitHub authorized, but ScrapeCreators had a server error minting your key - this is on their side, not yours. Sign up or log in at scrapecreators.com, grab a key there, and paste it, or Skip / retry later." Accept a pasted key or offer web/skip. Do not imply they already have a working linked account.@@ -94,7 +94,7 @@ Options:- **One-shot fallback:** hosts that prefer a single call can still run `setup --github` (foreground), which chains start+poll; tell the user first that a code will appear on their clipboard to paste.- "Open scrapecreators.com (Google sign-in)" - run `open https://scrapecreators.com` via Bash, then ask them to paste the API key. Write `SCRAPECREATORS_API_KEY={key}` to `~/.config/last30days/.env`.- "I have a key" - accept the key, write to `.env`.-- "Skip for now" - proceed without ScrapeCreators. No TikTok/Instagram, no empty-path Reddit search backup, and no YouTube transcript fallback when yt-dlp is throttled (your free sources still work, including keyless Reddit comments via shreddit).+- "Skip for now" - proceed without ScrapeCreators. TikTok/Instagram, paid Reddit/YouTube backfill, and transcript rescue stay off; free sources and Reddit comments still work.**Step 5: Source opt-in (only if a ScrapeCreators key was saved, not if skipped).** Comments are the DEFAULT, never an opt-in - there is no posts-only tier. Plain text then modal:@@ -136,7 +136,7 @@ For hosts without interactive modal prompts (OpenClaw, Codex, Cursor, Gemini CLI**3. Full Disk Access remediation (macOS only).** After `setup`, inspect stderr. If it contains `Permission denied reading Cookies.binarycookies` on macOS, surface: `macOS blocked the cookie read. To enable X/Twitter: System Settings > Privacy & Security > Full Disk Access > enable your terminal (or the Claude app), then I can retry.` Offer ONE retry only when no research topic is waiting. If a topic is already waiting or the user skips, continue immediately with available sources.-**4. ScrapeCreators signup offer (every first run, consent BEFORE launching the browser).** Explain it grants 10,000 free calls that add TikTok and Instagram, plus optional backups: Reddit search backfill when the free path returns fewer than 5 items (`LAST30DAYS_REDDIT_SC_MIN_ITEMS=0` limits it to empty runs; SC-primary is the opt-in `LAST30DAYS_REDDIT_BACKEND=scrapecreators` env knob), and a YouTube transcript fallback when yt-dlp is rate-limited or bot-gated. GitHub signup grants the full 10,000 free calls (more than the web form), and it opens a GitHub authorization page where you enter a short code. Ask, e.g.: `Want to unlock TikTok, Instagram, and more? I can sign you up for ScrapeCreators with GitHub (10,000 free calls, ~20-30s) - it opens a browser and you enter a short code. (yes / no)` **Wait for the answer.**+**4. ScrapeCreators offer (every first run; consent before browser launch).** Explain: the key grants 10,000 free calls for TikTok/Instagram and spends credits on Reddit search with fewer than 5 free items, YouTube search with fewer than 3 yt-dlp videos, and transcript rescue. `LAST30DAYS_REDDIT_SC_MIN_ITEMS=0` / `LAST30DAYS_YT_SC_MIN_ITEMS=0` make backfills empty-only; `LAST30DAYS_REDDIT_BACKEND=scrapecreators` pins Reddit to SC. A YouTube backfill may fetch transcripts and retry. GitHub signup grants more free calls than web signup and opens a page for a short code. Ask, e.g.: `Want to add TikTok and Instagram? Your key also spends credits on Reddit search below 5, YouTube search below 3, and transcripts. LAST30DAYS_YT_SC_MIN_ITEMS=0 makes YouTube search empty-only. GitHub signup grants 10,000 free calls and opens a browser for a code. (yes / no)` **Wait for the answer.**- On **yes** → two commands. FIRST run `"${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" setup --github-start` in the FOREGROUND - it returns in ~1-2s with a `Your GitHub code: XXXX-XXXX` line plus a JSON blob, copies the code to the clipboard, and opens the browser. Read the `user_code` from that output and immediately tell the user: the code, that it's on their clipboard so they can just paste it (Cmd+V) on the GitHub page - do not make them hunt for it. (If `status == "already_registered"`, stop here - their existing key is active. If the output said the clipboard copy failed, tell them to type the code.) THEN run `"${LAST30DAYS_PYTHON}" "${SKILL_DIR}/scripts/last30days.py" setup --github-poll` (background with a 5-min timeout, or foreground) and parse the **LAST** JSON line of its stdout for the final status. On success the engine persists the key automatically and returns `"persisted": true` with a MASKED `api_key` (never ask for or echo the raw key). Confirm the paid sources are active.- On **success but `"persisted": false`** (auth completed yet the key write failed) → do NOT claim sources are active. Tell the user signup worked but saving failed, and have them add `SCRAPECREATORS_API_KEY=<key>` to `~/.config/last30days/.env` manually (the raw key is masked in output, so re-run `setup --github` or retrieve it from scrapecreators.com to get the value).- On **`status == "error"` with `message == "Authorized but failed to fetch API key"`** (exact match; often `"reason": "no_api_key"`) → GitHub authorized fine, so do NOT say auth failed. This usually means the GitHub account is already linked to a ScrapeCreators account. Tell the user: "GitHub authorized, but I couldn't auto-grab your ScrapeCreators key - your GitHub is probably already linked to an account. Get your key at scrapecreators.com and paste it, or Skip." Accept a pasted key or offer web/skip.@@ -144,7 +144,7 @@ For hosts without interactive modal prompts (OpenClaw, Codex, Cursor, Gemini CLI- On **timeout, or any other error** → tell the user it didn't complete and offer to retry or the web signup at scrapecreators.com.- On **no** → note they can run it later by asking to set up ScrapeCreators, then continue.-**4b. Source tier (only if a key was saved).** Comments are the default, never opt-in. Your key runs TikTok + Instagram posts AND top comments, plus YouTube comments. Reddit stays on the free keyless path (ScrapeCreators search backfill when it returns fewer than 5 items; comments via shreddit). Ask whether they want the widest net, e.g.: `Recommended is TikTok + Instagram + all comments (posts and top comments for TikTok/Instagram plus YouTube comments). Or Everything - also Threads + Pinterest (more credits). (recommended / everything)` **Wait for the answer.**+**4b. Source tier (after saving a key).** Comments default on: TikTok/Instagram posts and top comments, plus YouTube comments. Backfills run regardless of tier: Reddit with fewer than 5 free items, YouTube below 3 yt-dlp videos. Ask: `Recommended adds TikTok + Instagram posts and top comments, plus YouTube comments. Everything also adds Threads + Pinterest (more credits). (recommended / everything)` **Wait for the answer.**- On **recommended** → append `INCLUDE_SOURCES=tiktok,instagram,youtube_comments,tiktok_comments,instagram_comments` to `~/.config/last30days/.env` (include `tiktok,instagram` so they are not treated as excluded). Confirm posts + top comments for TikTok/Instagram/YouTube are on.- On **everything** → append `INCLUDE_SOURCES=tiktok,instagram,youtube_comments,tiktok_comments,instagram_comments,threads,pinterest`. Confirm Threads and Pinterest are on too.@@ -220,7 +220,7 @@ The magic of /last30days is Reddit comments + X posts together - and both are fr**YouTube (free, open source):**- Run `brew install yt-dlp` (or `pip install yt-dlp`) - enables YouTube search + transcripts.-- `SCRAPECREATORS_API_KEY=xxx` - optional server-side transcript fallback, used only when yt-dlp is rate-limited/bot-gated.+- `SCRAPECREATORS_API_KEY=xxx` - paid search backfill below 3 yt-dlp videos and transcript rescue. `LAST30DAYS_YT_SC_MIN_ITEMS=0` makes search empty-only; a retry may spend again.**Digg (free, keyless):**- Run `npx @mvanhorn/printing-press-library install digg --cli-only` - installs the Digg CLI for trending news, GitHub stars, and pipeline feeds. Activates when `digg-pp-cli` is on your PATH (typically `$HOME/.local/bin`).diff --git a/skills/last30days/scripts/lib/env.py b/skills/last30days/scripts/lib/env.pyindex 8030d694..5d04e675 100644--- a/skills/last30days/scripts/lib/env.py+++ b/skills/last30days/scripts/lib/env.py@@ -623,6 +623,7 @@ def get_config(policy: ConfigLoadPolicy | None = None) -> dict[str, Any]:# Per-source deadline (seconds) for doctor --probe live checks.('LAST30DAYS_DOCTOR_PROBE_TIMEOUT', None),('LAST30DAYS_REDDIT_SC_MIN_ITEMS', None),+ ('LAST30DAYS_YT_SC_MIN_ITEMS', None),('LAST30DAYS_STORE', None),# Discovery topic queue (podcast/X-article pipeline memory). Default# ON; the literal value "off" disables queue writes and annotations.@@ -1326,6 +1327,8 @@ REDDIT_SC_MIN_ITEMS_VAR = 'LAST30DAYS_REDDIT_SC_MIN_ITEMS'# fewer than this many items. Thin topics yield 2-3 free results; healthy# topics many more, so 5 spends credits only where it adds coverage.REDDIT_SC_MIN_ITEMS_DEFAULT = 5+YOUTUBE_SC_MIN_ITEMS_VAR = 'LAST30DAYS_YT_SC_MIN_ITEMS'+YOUTUBE_SC_MIN_ITEMS_DEFAULT = 3def reddit_sc_min_items(config: dict[str, Any]) -> int:@@ -1344,6 +1347,17 @@ def reddit_sc_min_items(config: dict[str, Any]) -> int:return 0+def youtube_sc_min_items(config: dict[str, Any]) -> int:+ """Minimum yt-dlp result count before keyed YouTube search backfill is skipped."""+ raw = config.get(YOUTUBE_SC_MIN_ITEMS_VAR)+ if raw is None or (isinstance(raw, str) and not raw.strip()):+ return YOUTUBE_SC_MIN_ITEMS_DEFAULT+ try:+ return max(int(raw), 0)+ except (TypeError, ValueError):+ return 0++@dataclass(frozen=True)class XPolicy:"""The host-conditional X routing rule, resolved once per config.diff --git a/skills/last30days/scripts/lib/pipeline.py b/skills/last30days/scripts/lib/pipeline.pyindex 2502ed4c..678cff2d 100644--- a/skills/last30days/scripts/lib/pipeline.py+++ b/skills/last30days/scripts/lib/pipeline.py@@ -2803,6 +2803,11 @@ def run(# Do not re-fan-out against a host still inside its window.with rate_limit_lock:rate_limited_sources.add(source)+ if isinstance(artifact, dict) and artifact.get("_source_outcome_deferred"):+ artifact = dict(artifact)+ deferred_retryable_failures.setdefault(+ source, artifact.pop("_source_outcome_deferred")+ )if isinstance(artifact, dict):artifact.pop("_x_backup_not_rate_limited", None)normalized = _normalize_score_dedupe(@@ -4106,6 +4111,13 @@ def _retryable_youtube_outcome(outcome: dict[str, Any]) -> bool:)+def _retryable_youtube_error(message: str, state: str | None = None) -> bool:+ return _retryable_youtube_outcome({+ "state": state or youtube_yt.classify_run_failure(message),+ "detail": message,+ })++def _topic_handle_mentions(topic: str) -> set[str]:"""@mentions in the topic, which are real X handles.@@ -4804,6 +4816,8 @@ def _retry_thin_sources(normalized = _apply_reddit_stream_keepers(source, normalized, settings["per_stream_limit"], topic)+ if isinstance(artifact, dict) and artifact.get("_source_outcome_deferred"):+ outcome_note = outcome_note or artifact["_source_outcome_deferred"]if not normalized and isinstance(artifact, dict):outcome_note = outcome_note or artifact.get("_source_outcome_if_empty")return source, normalized, outcome_note, (detail_note, detail_state)@@ -4971,6 +4985,23 @@ def _merge_reddit_items(free: list[dict], sc: list[dict]) -> list[dict]:return merged+def _merge_youtube_items(free: list[dict], sc: list[dict]) -> list[dict]:+ merged = copy.deepcopy(free)+ by_id = {item.get("video_id"): item for item in merged if item.get("video_id")}+ for item in sc:+ video_id = item.get("video_id")+ if not video_id:+ continue+ existing = by_id.get(video_id)+ if existing is None:+ merged.append(item)+ by_id[video_id] = item+ elif not str(existing.get("transcript_snippet") or "").strip() and item.get("transcript_snippet"):+ existing["transcript_snippet"] = item["transcript_snippet"]+ existing["transcript_highlights"] = item.get("transcript_highlights", [])+ return merged++def _detail_rate_limits_source(source: str, artifact: dict) -> bool:if artifact.get("_source_outcome_detail_state") != health.RATE_LIMITED:return False@@ -5656,20 +5687,79 @@ def _retrieve_stream_impl(except Exception as exc:youtube_failure = str(exc)result = None- # Fall back to SC YouTube search if yt-dlp failed or isn't installed.- if (result is None or not result.get("items")) and sc_token:+ free_items = list((result or {}).get("items") or [])+ min_items = env.youtube_sc_min_items(config)+ backfill_thin = bool(free_items) and len(free_items) < min_items+ thin_detail: str | None = None+ thin_error = False+ thin_retryable = False+ if (not free_items or backfill_thin) and sc_token:+ backfill_scope = (+ http.capture_failures() if free_items else contextlib.nullcontext([])+ )try:- result = youtube_yt.search_youtube_sc(- yt_query, from_date, to_date, depth=depth, token=sc_token,- )- if result.get("error"):- youtube_failure = str(result["error"])- elif result.get("items") and youtube_failure:- recovered_failure = youtube_failure+ with backfill_scope as sc_failures:+ transcribed_free_ids = {+ item["video_id"] for item in free_items+ if item.get("video_id")+ and str(item.get("transcript_snippet") or "").strip()+ }+ sc_result = youtube_yt.search_youtube_sc(+ yt_query, from_date, to_date, depth=depth, token=sc_token,+ skip_transcript_ids=transcribed_free_ids,+ )+ sc_error = str(sc_result["error"]) if sc_result.get("error") else None+ if backfill_thin:+ merged = _merge_youtube_items(free_items, sc_result.get("items") or [])+ result = {"items": merged}+ thin_detail = (+ f"yt-dlp returned {len(free_items)} videos below the "+ f"{min_items}-video backfill floor; ScrapeCreators added "+ f"{len(merged) - len(free_items)} videos"+ )+ if youtube_failure:+ thin_detail += f"; yt-dlp search: {youtube_failure}"+ if sc_error:+ thin_detail += f"; ScrapeCreators search: {sc_error}"+ if sc_failures:+ thin_detail += f"; ScrapeCreators: {_summarize_lane_failures(sc_failures, 'youtube')}"+ thin_error = bool(youtube_failure or sc_error or sc_failures)+ thin_retryable = thin_error and (+ len(merged) < min_items+ and (not youtube_failure or _retryable_youtube_error(youtube_failure))+ and (not sc_error or _retryable_youtube_error(sc_error))+ and all(+ _retryable_youtube_error(str(f), f.outcome_state)+ for f in sc_failures+ )+ )youtube_failure = None+ recovered_failure = None+ else:+ result = sc_result+ if sc_error:+ youtube_failure = sc_error+ elif result.get("items") and youtube_failure:+ recovered_failure = youtube_failure+ youtube_failure = Noneexcept Exception as exc:+ prior_youtube_failure = youtube_failureyoutube_failure = str(exc)- result = None+ result = {"items": free_items} if free_items else None+ if backfill_thin:+ thin_error = True+ thin_detail = (+ f"yt-dlp returned {len(free_items)} videos below the "+ f"{min_items}-video backfill floor; ScrapeCreators failed: {exc}"+ )+ if prior_youtube_failure:+ thin_detail += f"; yt-dlp search: {prior_youtube_failure}"+ thin_retryable = (+ _retryable_youtube_error(+ str(exc), getattr(exc, "outcome_state", None)+ )+ and (not prior_youtube_failure or _retryable_youtube_error(prior_youtube_failure))+ )if result is None:result = {"items": []}# Enrich top videos with comments (default-on when a key is present).@@ -5678,6 +5768,16 @@ def _retrieve_stream_impl(youtube_yt.enrich_with_comments(items, token=config.get("SCRAPECREATORS_API_KEY", ""),)+ if thin_detail:+ if not thin_error:+ return items, {"_source_outcome_detail": thin_detail}+ outcome = _outcome_artifact(health.PARTIAL, thin_detail)+ if thin_retryable:+ return items, {+ "_source_outcome_detail": thin_detail,+ "_source_outcome_deferred": outcome["_source_outcome"],+ }+ return items, outcomeif youtube_failure:state = youtube_yt.classify_run_failure(youtube_failure)attempted = state != schema.SKIPPED_UNCONFIGUREDdiff --git a/skills/last30days/scripts/lib/youtube_yt.py b/skills/last30days/scripts/lib/youtube_yt.pyindex ea44be13..f12adec2 100644--- a/skills/last30days/scripts/lib/youtube_yt.py+++ b/skills/last30days/scripts/lib/youtube_yt.py@@ -1557,11 +1557,12 @@ def search_youtube_sc(to_date: str,depth: str = "default",token: str = None,+ skip_transcript_ids: Optional[Set[str]] = None,) -> Dict[str, Any]:- """Search YouTube via ScrapeCreators API (fallback when yt-dlp is unavailable).+ """Search YouTube via ScrapeCreators when yt-dlp is absent, empty, or thin.Uses SC keyword search to find videos and SC transcript endpoint to- fetch transcripts. Called by pipeline.py when yt-dlp fails.+ fetch transcripts. The pipeline uses the configured thin-result floor.Args:topic: Search topic@@ -1569,6 +1570,7 @@ def search_youtube_sc(to_date: End date (YYYY-MM-DD)depth: 'quick', 'default', or 'deep'token: ScrapeCreators API key+ skip_transcript_ids: Video IDs already transcribed by the free searchReturns:Dict with 'items' list of video metadata dicts.@@ -1641,14 +1643,19 @@ def search_youtube_sc(# Step 2: Fetch transcripts for top videostranscript_limit = TRANSCRIPT_LIMITS.get(depth, TRANSCRIPT_LIMITS["default"])if transcript_limit > 0 and items:- attempt_count = min(len(items), transcript_limit * 3)# Same in-window-first ordering as search_and_transcribe(): don't let# an out-of-window back-catalog (kept by the soft date filter above)# consume the transcript budget of videos the freshness scorer keeps.in_window = [i for i in items if i.get("date") and i["date"] >= from_date]out_of_window = [i for i in items if not (i.get("date") and i["date"] >= from_date)]+ excluded_ids = skip_transcript_ids or set()+ transcript_candidates = [+ item for item in in_window + out_of_window+ if item["video_id"] not in excluded_ids+ ]+ attempt_count = min(len(transcript_candidates), transcript_limit * 3)_log(f"Fetching SC transcripts for up to {attempt_count} videos (target: {transcript_limit})")- for item in (in_window + out_of_window)[:attempt_count]:+ for item in transcript_candidates[:attempt_count]:vid = item["video_id"]if not vid:continuediff --git a/tests/test_doc_flag_contract.py b/tests/test_doc_flag_contract.pyindex 83ff54a7..ab36e994 100644--- a/tests/test_doc_flag_contract.py+++ b/tests/test_doc_flag_contract.py@@ -77,7 +77,8 @@ def test_reddit_backend_env_var_is_documented_for_users_and_runtime_skill():assert "LAST30DAYS_REDDIT_SC_MIN_ITEMS=0" in skill_textassert "LAST30DAYS_REDDIT_SC_MIN_ITEMS=0" in config_textfor text in (skill_text, config_text):- assert "empty-only" not in text+ assert "LAST30DAYS_YT_SC_MIN_ITEMS=0" in text+ assert "YouTube search" in textassert "returns no items" not in textassert "returns **no items**" not in textdiff --git a/tests/test_onboarding_contract.py b/tests/test_onboarding_contract.pyindex a6a55f65..ab2e5c1e 100644--- a/tests/test_onboarding_contract.py+++ b/tests/test_onboarding_contract.py@@ -314,8 +314,11 @@ class TestOnboardingContract(unittest.TestCase):# transport/rate-limit escalation, and no longer empty-only).self.assertIn("fewer than 5 items", before)self.assertIn("LAST30DAYS_REDDIT_SC_MIN_ITEMS=0", before)+ self.assertIn("YouTube search below 3", before)+ self.assertIn("LAST30DAYS_YT_SC_MIN_ITEMS=0", before)+ self.assertIn("spends credits", before)self.assertNotIn("returns no items", before)- self.assertNotIn("empty-only", before)+ self.assertNotIn("empty-path Reddit", before)self.assertNotIn("when they hit rate limits", before)# Free-path comments are shreddit; do not claim SC comment preference.self.assertNotIn("prefers ScrapeCreators for Reddit", before)diff --git a/tests/test_youtube_backstop.py b/tests/test_youtube_backstop.pynew file mode 100644index 00000000..2e79b37a--- /dev/null+++ b/tests/test_youtube_backstop.py@@ -0,0 +1,382 @@+from unittest import mock++from lib import health, http, pipeline, schema, youtube_yt+++def _youtube_stream(config, *, wrapped=False):+ retrieve = pipeline._retrieve_stream if wrapped else pipeline._retrieve_stream_impl+ return retrieve(+ topic="Vuori",+ subquery=schema.SubQuery(+ label="q", search_query="Vuori", ranking_query="Vuori", sources=["youtube"],+ ),+ source="youtube",+ config=config,+ depth="default",+ date_range=("2026-06-01", "2026-07-01"),+ runtime=schema.ProviderRuntime(+ reasoning_provider="mock", planner_model="mock", rerank_model="mock",+ ),+ mock=False,+ )+++def test_one_free_video_triggers_backstop_and_preserves_its_transcript():+ free = {"video_id": "free", "transcript_snippet": "free transcript"}+ paid = {"video_id": "paid", "transcript_snippet": "paid transcript"}+ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(youtube_yt, "search_and_transcribe", return_value={"items": [free]}),+ mock.patch.object(+ youtube_yt, "search_youtube_sc",+ return_value={"items": [{"video_id": "free", "transcript_snippet": ""}, paid]},+ ) as sc_search,+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ items, artifact = _youtube_stream({"SCRAPECREATORS_API_KEY": "dummy-key"})++ assert sc_search.call_count == 1+ assert [item["video_id"] for item in items] == ["free", "paid"]+ assert items[0]["transcript_snippet"] == "free transcript"+ assert "added 1" in artifact["_source_outcome_detail"]+ assert "_source_outcome" not in artifact+++def test_two_free_videos_merge_duplicate_transcript_without_replacing_free_metadata():+ free = [+ {"video_id": "a", "title": "free title", "transcript_snippet": ""},+ {"video_id": "b", "title": "other", "transcript_snippet": "free transcript"},+ ]+ sc = [+ {"video_id": "a", "title": "paid title", "transcript_snippet": "rescued"},+ {"video_id": "c", "title": "new", "transcript_snippet": ""},+ ]+ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(youtube_yt, "search_and_transcribe", return_value={"items": free}),+ mock.patch.object(youtube_yt, "search_youtube_sc", return_value={"items": sc}) as search,+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ items, artifact = _youtube_stream({"SCRAPECREATORS_API_KEY": "dummy-key"})++ assert search.call_count == 1+ assert [item["video_id"] for item in items] == ["a", "b", "c"]+ assert items[0]["title"] == "free title"+ assert items[0]["transcript_snippet"] == "rescued"+ assert items[1]["transcript_snippet"] == "free transcript"+ assert free[0]["transcript_snippet"] == ""+ assert "added 1" in artifact["_source_outcome_detail"]+++def test_backfill_spends_transcript_calls_only_on_missing_or_new_videos():+ free = [+ {"video_id": "ready", "title": "free ready", "transcript_snippet": "free transcript"},+ {"video_id": "missing", "title": "free missing", "transcript_snippet": " "},+ ]+ video_ids = ["ready", "missing", *(f"new{i}" for i in range(5))]+ raw_search = [+ {+ "id": video_id, "title": f"SC {video_id}", "date": "2026-06-15",+ "views": 100 - index,+ }+ for index, video_id in enumerate(video_ids)+ ]+ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(youtube_yt, "search_and_transcribe", return_value={"items": free}),+ mock.patch.object(youtube_yt, "_sc_youtube_search", return_value=raw_search) as search,+ mock.patch.object(+ youtube_yt, "_sc_fetch_transcript",+ side_effect=lambda video_id, token: f"paid transcript for {video_id}",+ ) as transcripts,+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ items, _ = _youtube_stream({"SCRAPECREATORS_API_KEY": "dummy-key"})++ search.assert_called_once()+ assert [call.args[0] for call in transcripts.call_args_list] == video_ids[1:]+ assert len(transcripts.call_args_list) == 6+ assert [item["video_id"] for item in items] == video_ids+ assert items[0]["title"] == "free ready"+ assert items[0]["transcript_snippet"] == "free transcript"+ assert items[1]["title"] == "free missing"+ assert items[1]["transcript_snippet"] == "paid transcript for missing"+ assert items[-1]["title"] == "SC new4"+ assert items[-1]["transcript_snippet"] == "paid transcript for new4"+++def test_keyless_and_disabled_floor_keep_free_results_without_paid_search():+ free = {"video_id": "a", "transcript_snippet": ""}+ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(youtube_yt, "search_and_transcribe", return_value={"items": [free]}),+ mock.patch.object(youtube_yt, "search_youtube_sc") as search,+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ for config in (+ {},+ {"SCRAPECREATORS_API_KEY": "dummy-key", "LAST30DAYS_YT_SC_MIN_ITEMS": "0"},+ {"SCRAPECREATORS_API_KEY": "dummy-key", "LAST30DAYS_YT_SC_MIN_ITEMS": "bad"},+ ):+ items, artifact = _youtube_stream(config)+ assert items == [free]+ assert artifact == {}+ search.assert_not_called()+++def test_at_floor_does_not_backfill():+ free = [{"video_id": str(i)} for i in range(3)]+ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(youtube_yt, "search_and_transcribe", return_value={"items": free}),+ mock.patch.object(youtube_yt, "search_youtube_sc") as search,+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ items, artifact = _youtube_stream({"SCRAPECREATORS_API_KEY": "dummy-key"})+ assert items == free+ assert artifact == {}+ search.assert_not_called()+++def test_custom_floor_backfills_three_free_videos():+ free = [{"video_id": str(i)} for i in range(3)]+ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(youtube_yt, "search_and_transcribe", return_value={"items": free}),+ mock.patch.object(+ youtube_yt, "search_youtube_sc", return_value={"items": [{"video_id": "new"}]},+ ) as search,+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ items, artifact = _youtube_stream({+ "SCRAPECREATORS_API_KEY": "dummy-key", "LAST30DAYS_YT_SC_MIN_ITEMS": "4",+ })+ assert search.call_count == 1+ assert [item["video_id"] for item in items] == ["0", "1", "2", "new"]+ assert "4-video backfill floor" in artifact["_source_outcome_detail"]+++def test_paid_search_failure_keeps_free_video_and_reports_partial():+ free = {"video_id": "a", "transcript_snippet": "free transcript"}+ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(youtube_yt, "search_and_transcribe", return_value={"items": [free]}),+ mock.patch.object(youtube_yt, "search_youtube_sc", side_effect=RuntimeError("SC HTTP 429")) as search,+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ items, artifact = _youtube_stream({"SCRAPECREATORS_API_KEY": "dummy-key"})+ assert search.call_count == 1+ assert items == [free]+ assert artifact["_source_outcome"]["state"] == health.PARTIAL+ assert "SC HTTP 429" in artifact["_source_outcome"]["detail"]+++def test_paid_http_429_with_no_free_items_stays_rate_limited():+ def failed_sc(*args, **kwargs):+ http._record_failure(http.HTTPError("SC HTTP 429", status_code=429))+ return {"items": []}++ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(youtube_yt, "search_and_transcribe", return_value={"items": []}),+ mock.patch.object(youtube_yt, "search_youtube_sc", side_effect=failed_sc),+ ):+ items, artifact = _youtube_stream(+ {"SCRAPECREATORS_API_KEY": "dummy-key"}, wrapped=True,+ )+ assert items == []+ assert artifact["_source_outcome"]["state"] == health.RATE_LIMITED+++def test_paid_http_429_after_free_item_is_partial_not_source_rate_limit():+ def failed_sc(*args, **kwargs):+ http._record_failure(http.HTTPError("SC HTTP 429", status_code=429))+ return {"items": []}++ free = {"video_id": "a"}+ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(youtube_yt, "search_and_transcribe", return_value={"items": [free]}),+ mock.patch.object(youtube_yt, "search_youtube_sc", side_effect=failed_sc),+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ items, artifact = _youtube_stream(+ {"SCRAPECREATORS_API_KEY": "dummy-key"}, wrapped=True,+ )+ assert items == [free]+ assert artifact["_source_outcome"]["state"] == health.PARTIAL+ assert "rate" in artifact["_source_outcome"]["detail"].lower()+++def test_partial_ytdlp_search_error_stays_visible_after_backfill():+ free = {"video_id": "a"}+ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(+ youtube_yt, "search_and_transcribe",+ return_value={"items": [free], "error": "yt-dlp bot gate"},+ ),+ mock.patch.object(youtube_yt, "search_youtube_sc", return_value={"items": []}),+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ items, artifact = _youtube_stream({"SCRAPECREATORS_API_KEY": "dummy-key"})+ assert items == [free]+ assert artifact["_source_outcome"]["state"] == health.PARTIAL+ assert "yt-dlp bot gate" in artifact["_source_outcome"]["detail"]+++def test_full_pipeline_reports_clean_backfill_and_searches_paid_once_per_subquery():+ free = {+ "video_id": "a", "title": "Vuori review", "date": "2026-06-15",+ "url": "https://www.youtube.com/watch?v=a", "transcript_snippet": "free transcript",+ }+ paid = [+ {+ "video_id": str(i), "title": f"Vuori review {i}", "date": "2026-06-15",+ "url": f"https://www.youtube.com/watch?v={i}", "transcript_snippet": "",+ }+ for i in range(3)+ ]+ plan = {+ "intent": "general", "freshness_mode": "balanced_recent", "cluster_mode": "story",+ "subqueries": [+ {"label": "one", "search_query": "Vuori", "ranking_query": "Vuori", "sources": ["youtube"]},+ {"label": "two", "search_query": "Vuori reviews", "ranking_query": "Vuori", "sources": ["youtube"]},+ ],+ "source_weights": {"youtube": 1.0},+ }+ retrieve = pipeline._retrieve_stream++ def live_youtube(*args, **kwargs):+ return retrieve(*args, **{**kwargs, "mock": False})++ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(pipeline, "_retrieve_stream", side_effect=live_youtube),+ mock.patch.object(youtube_yt, "search_and_transcribe", return_value={"items": [free]}),+ mock.patch.object(youtube_yt, "search_youtube_sc", return_value={"items": paid}) as search,+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ report = pipeline.run(+ topic="Vuori", config={"SCRAPECREATORS_API_KEY": "dummy-key"},+ depth="default", requested_sources=["youtube"], mock=True,+ external_plan=plan, as_of_date="2026-07-01",+ )++ assert search.call_count == 2+ assert report.source_status["youtube"].state == health.OK+ assert "ScrapeCreators added 3" in report.source_status["youtube"].detail+++def _run_thin_youtube_report(sc_responses):+ free = {+ "video_id": "free", "title": "Vuori review", "date": "2026-06-15",+ "url": "https://www.youtube.com/watch?v=free", "transcript_snippet": "free transcript",+ }+ plan = {+ "intent": "general", "freshness_mode": "balanced_recent", "cluster_mode": "story",+ "subqueries": [{+ "label": "primary", "search_query": "Vuori", "ranking_query": "Vuori",+ "sources": ["youtube"],+ }],+ "source_weights": {"youtube": 1.0},+ }+ retrieve = pipeline._retrieve_stream++ def live_youtube(*args, **kwargs):+ return retrieve(*args, **{**kwargs, "mock": False})++ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(pipeline, "_retrieve_stream", side_effect=live_youtube),+ mock.patch.object(youtube_yt, "search_and_transcribe", return_value={"items": [free]}),+ mock.patch.object(youtube_yt, "search_youtube_sc", side_effect=sc_responses) as search,+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ report = pipeline.run(+ topic="Vuori", config={"SCRAPECREATORS_API_KEY": "dummy-key"},+ depth="default", requested_sources=["youtube"], mock=True,+ external_plan=plan, as_of_date="2026-07-01",+ )+ return report, search.call_count+++def test_empty_clean_backfill_can_retry_and_recover():+ recovered = [+ {+ "video_id": str(i), "title": f"Vuori review {i}", "date": "2026-06-15",+ "url": f"https://www.youtube.com/watch?v={i}",+ }+ for i in range(2)+ ]+ report, calls = _run_thin_youtube_report([+ {"items": []}, {"items": recovered},+ ])+ assert calls == 2+ assert report.source_status["youtube"].state == health.OK+ assert {item.item_id for item in report.items_by_source["youtube"]} >= {+ "free", "0",+ }+++def test_clean_sparse_topic_stays_ok_after_empty_backfill_and_retry():+ report, calls = _run_thin_youtube_report([{"items": []}, {"items": []}])+ assert calls == 2+ assert report.source_status["youtube"].state == health.OK+ assert [item.item_id for item in report.items_by_source["youtube"]] == ["free"]+ assert "added 0" in report.source_status["youtube"].detail+++def test_paid_429_after_free_item_stops_paid_thin_retry():+ report, calls = _run_thin_youtube_report([+ {"items": [], "error": "SC HTTP 429"},+ ])+ assert calls == 1+ assert report.source_status["youtube"].state == health.PARTIAL+ assert "429" in report.source_status["youtube"].detail+++def test_transient_paid_failure_stays_partial_if_retry_finds_no_new_video():+ report, calls = _run_thin_youtube_report([+ {"items": [], "error": "SC connection reset"},+ {"items": [], "error": "SC connection reset"},+ ])+ assert calls == 2+ assert report.source_status["youtube"].state == health.PARTIAL+ assert "connection reset" in report.source_status["youtube"].detail+++def test_retry_error_stays_partial_even_when_retry_adds_a_video():+ new = {+ "video_id": "new", "title": "Vuori review new", "date": "2026-06-15",+ "url": "https://www.youtube.com/watch?v=new",+ }+ report, calls = _run_thin_youtube_report([+ {"items": [], "error": "SC connection reset"},+ {"items": [new], "error": "SC connection reset"},+ ])+ assert calls == 2+ assert "new" in {item.item_id for item in report.items_by_source["youtube"]}+ assert report.source_status["youtube"].state == health.PARTIAL+ assert "connection reset" in report.source_status["youtube"].detail+++def test_ytdlp_bot_gate_and_paid_exception_do_not_retry_paid_search():+ free = {"video_id": "free"}+ with (+ mock.patch.object(pipeline, "which", return_value="/usr/bin/yt-dlp"),+ mock.patch.object(+ youtube_yt, "search_and_transcribe",+ return_value={"items": [free], "error": "Sign in to confirm you're not a bot"},+ ),+ mock.patch.object(+ youtube_yt, "search_youtube_sc", side_effect=RuntimeError("SC disconnected"),+ ) as search,+ mock.patch.object(pipeline.env, "is_youtube_comments_available", return_value=False),+ ):+ items, artifact = _youtube_stream({"SCRAPECREATORS_API_KEY": "dummy-key"})+ assert items == [free]+ assert search.call_count == 1+ assert artifact["_source_outcome"]["state"] == health.PARTIAL+ assert "not a bot" in artifact["_source_outcome"]["detail"]+ assert "SC disconnected" in artifact["_source_outcome"]["detail"]