AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary
Python ai-promptsai-skillblueskyclaudeclaude-codeclawhubdeep-researchhackernewsinstagramopenclawpolymarketrecencyredditresearchsocial-mediatiktoktrendstwitterweb-searchyoutube
fix(sources): honor INCLUDE_SOURCES for xiaohongshu and dripstack (#812)
Matt Van Horn22e095fa0adb
diff --git a/CONFIGURATION.md b/CONFIGURATION.mdindex 85fcaeb1..f419e07d 100644--- a/CONFIGURATION.md+++ b/CONFIGURATION.md@@ -130,7 +130,7 @@ python3 skills/last30days/scripts/last30days.py "MCP servers" \| Hacker News | none | always on | yes || Polymarket | none | always on | yes || StockTwits | none | auto-on for ticker/crypto topics only (gated by symbol detection); never registered for non-financial topics | yes (public API, ~200 req/hr per IP) |-| DripStack | none | requested-only: include it with `--search dripstack` (or LAST30DAYS_DEFAULT_SEARCH). Searches premium financial newsletters and analyst writeups via a free, public search API — no key needed. Never active on default runs. | yes when requested (public API, no auth) |+| DripStack | none | opt-in only: per run with `--search dripstack`, or persistently with `INCLUDE_SOURCES=dripstack` in `.env`. Searches premium financial newsletters and analyst writeups via a free, public search API — no key needed. Never active without the opt-in. | yes when opted in (public API, no auth) || GitHub | `gh` CLI installed (uses your GitHub auth) | always on if `gh` present | yes || YouTube | `yt-dlp` CLI installed; `SCRAPECREATORS_API_KEY` adds a server-side transcript fallback used only when yt-dlp fails (429 / bot-gate) | always on if `yt-dlp` present; SC transcript fallback default-on when key set (no credit spent unless yt-dlp fails) | yes || YouTube comments | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `youtube_comments` (**on by default** — written by the Step 5 Recommended tier) | top comments (by likes) on the top ~3 videos by engagement | ~3 calls/run; 10K free calls |diff --git a/README.md b/README.mdindex 0522cb78..9db4cb58 100644--- a/README.md+++ b/README.md@@ -290,8 +290,8 @@ These platforms don't have relationships with each other. X doesn't know what Re| YouTube | `brew install yt-dlp` | Free || Bluesky | App password from bsky.app | Free || TikTok + Instagram + Threads + Pinterest + LinkedIn + YouTube comments | ScrapeCreators key | 10,000 free calls, then PAYG |-| Xiaohongshu (RED) | Run a logged-in x-mcp browser plugin or `xiaohongshu-mcp` service and request it with `--search xhs`; last30days auto-probes `http://localhost:18060` then `http://host.docker.internal:18060`, or use `XIAOHONGSHU_API_BASE` for a custom URL | No last30days API key; depends on your local browser-session service |-| DripStack (premium financial newsletters) | Requested-only: include it with `--search dripstack` | No key; free public search API |+| Xiaohongshu (RED) | Run a logged-in x-mcp browser plugin or `xiaohongshu-mcp` service and opt in with `--search xhs` per run or `INCLUDE_SOURCES=xiaohongshu` in `.env`; last30days auto-probes `http://localhost:18060` then `http://host.docker.internal:18060`, or use `XIAOHONGSHU_API_BASE` for a custom URL | No last30days API key; depends on your local browser-session service |+| DripStack (premium financial newsletters) | Opt-in: `--search dripstack` per run, or `INCLUDE_SOURCES=dripstack` in `.env` | No key; free public search API || Perplexity Sonar / Search API / Deep Research | Perplexity key, or OpenRouter key as Sonar fallback | Pay as you go || Web search | Brave Search key | 2,000 free queries/month |diff --git a/skills/last30days/SKILL.md b/skills/last30days/SKILL.mdindex e24b31e8..06915403 100644--- a/skills/last30days/SKILL.md+++ b/skills/last30days/SKILL.md@@ -591,8 +591,8 @@ The magic of /last30days is Reddit comments + X posts together - and both are fr**Other optional sources (add anytime):**- `PERPLEXITY_API_KEY=xxx` (or `OPENROUTER_API_KEY=xxx`) - AI-synthesized research with citations; set `INCLUDE_SOURCES=perplexity`.-- `XIAOHONGSHU_API_BASE=http://localhost:18060` - Xiaohongshu/RED via a logged-in x-mcp browser plugin or `xiaohongshu-mcp` service; optional unless the local service runs on a custom URL. Request it per run with `--search xhs`.-- DripStack (premium financial newsletter search) is requested-only: include it per run with `--search dripstack`. Free public search API, no key; never active unless requested.+- `XIAOHONGSHU_API_BASE=http://localhost:18060` - Xiaohongshu/RED via a logged-in x-mcp browser plugin or `xiaohongshu-mcp` service; optional unless the local service runs on a custom URL. Opt in per run with `--search xhs`, or persistently via `INCLUDE_SOURCES=xiaohongshu`.+- DripStack (premium financial newsletter search) is opt-in only: per run with `--search dripstack`, or persistently via `INCLUDE_SOURCES=dripstack`. Free public search API, no key; never active without the opt-in.- `BSKY_HANDLE=you.bsky.social` + `BSKY_APP_PASSWORD=xxx` - Bluesky (free app password).- `BRAVE_API_KEY=xxx` or `EXA_API_KEY=xxx` - web search backends.diff --git a/skills/last30days/scripts/lib/pipeline.py b/skills/last30days/scripts/lib/pipeline.pyindex 2386beb1..4447e112 100644--- a/skills/last30days/scripts/lib/pipeline.py+++ b/skills/last30days/scripts/lib/pipeline.py@@ -188,11 +188,19 @@ def available_sources(# GitHub is reachable via the unauthenticated REST tier too, so it is# available even without a token/gh CLI (a token only raises rate limits).available.append("github")- # DripStack is requested-only (owner decision, #791): a commercial- # third-party API must never receive default-run traffic. Request it per- # run (--search dripstack) or via LAST30DAYS_DEFAULT_SEARCH; the search- # API is free and public (no key), so the request itself is the gate.- if requested_sources and "dripstack" in requested_sources:+ # DripStack is opt-in only (owner decision, #791): a commercial+ # third-party API must never receive default-run traffic. Opt in per run+ # (--search dripstack) or persistently (INCLUDE_SOURCES=dripstack in+ # .env, the LinkedIn/Perplexity pattern); the search API is free and+ # public (no key), so the opt-in itself is the gate.+ include_sources = {+ token.strip()+ for token in (config.get("INCLUDE_SOURCES") or "").lower().split(",")+ if token.strip()+ }+ if "dripstack" in include_sources or (+ requested_sources and "dripstack" in requested_sources+ ):available.append("dripstack")if which("digg-pp-cli"):available.append("digg")@@ -219,7 +227,6 @@ def available_sources(if requested_sources and "jobs" in requested_sources:available.append("jobs")# Perplexity Sonar: opt-in additive source via INCLUDE_SOURCES=perplexity- include_sources = (config.get("INCLUDE_SOURCES") or "").lower().split(",")if _has_perplexity_provider(config) and ("perplexity" in include_sources or (requested_sources and "perplexity" in requested_sources)):@@ -241,7 +248,10 @@ def available_sources("trustpilot" in include_sources or (requested_sources and "trustpilot" in requested_sources)):available.append("trustpilot")- if requested_sources and "xiaohongshu" in requested_sources and env.is_xiaohongshu_available(config):+ if (+ "xiaohongshu" in include_sources+ or (requested_sources and "xiaohongshu" in requested_sources)+ ) and env.is_xiaohongshu_available(config):available.append("xiaohongshu")# Threads: opt-in via INCLUDE_SOURCES (same pattern as perplexity/linkedin).# Was auto-on with the key; gated so the onboarding "Everything" tier is adiff --git a/tests/test_dripstack.py b/tests/test_dripstack.pyindex fdd03f86..7f3806d3 100644--- a/tests/test_dripstack.py+++ b/tests/test_dripstack.py@@ -106,3 +106,26 @@ def test_parse_emits_body_and_normalizer_prefers_it():"dripstack", parsed[0], 0, "2026-06-12", "2026-07-12")assert normalized.body == "Capital, offtake and datacenters."+++def test_dripstack_activates_via_persisted_include_sources():+ """INCLUDE_SOURCES is the .env checkbox for persistent opt-ins (the+ LinkedIn/Perplexity pattern); dripstack must honor it, not only --search."""+ available = pipeline.available_sources(+ {"INCLUDE_SOURCES": "dripstack"}, None, x_pending=False+ )+ assert "dripstack" in available+++def test_unrelated_include_sources_keeps_dripstack_off():+ available = pipeline.available_sources(+ {"INCLUDE_SOURCES": "linkedin,tiktok"}, None, x_pending=False+ )+ assert "dripstack" not in available+++def test_include_sources_tolerates_whitespace_around_commas():+ available = pipeline.available_sources(+ {"INCLUDE_SOURCES": "linkedin, dripstack"}, None, x_pending=False+ )+ assert "dripstack" in availablediff --git a/tests/test_xiaohongshu_api.py b/tests/test_xiaohongshu_api.pyindex bce56a54..13e69fd9 100644--- a/tests/test_xiaohongshu_api.py+++ b/tests/test_xiaohongshu_api.py@@ -170,3 +170,31 @@ def test_search_feeds_requires_logged_in_xiaohongshu_session():"matcha latte", "2026-06-01", "2026-07-01","http://localhost:18060",)+++def test_xiaohongshu_activates_via_persisted_include_sources(monkeypatch):+ from unittest import mock as _mock+ from lib import env as _env, pipeline as _pipeline++ probe = _mock.Mock(return_value=True)+ monkeypatch.setattr(_env, "is_xiaohongshu_available", probe)++ available = _pipeline.available_sources(+ {"INCLUDE_SOURCES": "xiaohongshu"}, None, x_pending=False+ )++ assert "xiaohongshu" in available+ probe.assert_called_once()+++def test_xiaohongshu_probe_never_fires_without_any_opt_in(monkeypatch):+ from unittest import mock as _mock+ from lib import env as _env, pipeline as _pipeline++ probe = _mock.Mock(return_value=True)+ monkeypatch.setattr(_env, "is_xiaohongshu_available", probe)++ available = _pipeline.available_sources({}, None, x_pending=False)++ assert "xiaohongshu" not in available+ probe.assert_not_called()