Skip to content

AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary

fix(sources): honor INCLUDE_SOURCES for xiaohongshu and dripstack (#812)

Matt Van Horn22e095fa0adb
diff --git a/CONFIGURATION.md b/CONFIGURATION.md
index 85fcaeb1..f419e07d 100644
--- a/CONFIGURATION.md
+++ b/CONFIGURATION.md
@@ -130,7 +130,7 @@ python3 skills/last30days/scripts/last30days.py "MCP servers" \
| Hacker News | none | always on | yes |
| Polymarket | none | always on | yes |
| StockTwits | none | auto-on for ticker/crypto topics only (gated by symbol detection); never registered for non-financial topics | yes (public API, ~200 req/hr per IP) |
-| DripStack | none | requested-only: include it with `--search dripstack` (or LAST30DAYS_DEFAULT_SEARCH). Searches premium financial newsletters and analyst writeups via a free, public search API — no key needed. Never active on default runs. | yes when requested (public API, no auth) |
+| DripStack | none | opt-in only: per run with `--search dripstack`, or persistently with `INCLUDE_SOURCES=dripstack` in `.env`. Searches premium financial newsletters and analyst writeups via a free, public search API — no key needed. Never active without the opt-in. | yes when opted in (public API, no auth) |
| GitHub | `gh` CLI installed (uses your GitHub auth) | always on if `gh` present | yes |
| YouTube | `yt-dlp` CLI installed; `SCRAPECREATORS_API_KEY` adds a server-side transcript fallback used only when yt-dlp fails (429 / bot-gate) | always on if `yt-dlp` present; SC transcript fallback default-on when key set (no credit spent unless yt-dlp fails) | yes |
| YouTube comments | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `youtube_comments` (**on by default** — written by the Step 5 Recommended tier) | top comments (by likes) on the top ~3 videos by engagement | ~3 calls/run; 10K free calls |
diff --git a/README.md b/README.md
index 0522cb78..9db4cb58 100644
--- a/README.md
+++ b/README.md
@@ -290,8 +290,8 @@ These platforms don't have relationships with each other. X doesn't know what Re
| YouTube | `brew install yt-dlp` | Free |
| Bluesky | App password from bsky.app | Free |
| TikTok + Instagram + Threads + Pinterest + LinkedIn + YouTube comments | ScrapeCreators key | 10,000 free calls, then PAYG |
-| Xiaohongshu (RED) | Run a logged-in x-mcp browser plugin or `xiaohongshu-mcp` service and request it with `--search xhs`; last30days auto-probes `http://localhost:18060` then `http://host.docker.internal:18060`, or use `XIAOHONGSHU_API_BASE` for a custom URL | No last30days API key; depends on your local browser-session service |
-| DripStack (premium financial newsletters) | Requested-only: include it with `--search dripstack` | No key; free public search API |
+| Xiaohongshu (RED) | Run a logged-in x-mcp browser plugin or `xiaohongshu-mcp` service and opt in with `--search xhs` per run or `INCLUDE_SOURCES=xiaohongshu` in `.env`; last30days auto-probes `http://localhost:18060` then `http://host.docker.internal:18060`, or use `XIAOHONGSHU_API_BASE` for a custom URL | No last30days API key; depends on your local browser-session service |
+| DripStack (premium financial newsletters) | Opt-in: `--search dripstack` per run, or `INCLUDE_SOURCES=dripstack` in `.env` | No key; free public search API |
| Perplexity Sonar / Search API / Deep Research | Perplexity key, or OpenRouter key as Sonar fallback | Pay as you go |
| Web search | Brave Search key | 2,000 free queries/month |
diff --git a/skills/last30days/SKILL.md b/skills/last30days/SKILL.md
index e24b31e8..06915403 100644
--- a/skills/last30days/SKILL.md
+++ b/skills/last30days/SKILL.md
@@ -591,8 +591,8 @@ The magic of /last30days is Reddit comments + X posts together - and both are fr
**Other optional sources (add anytime):**
- `PERPLEXITY_API_KEY=xxx` (or `OPENROUTER_API_KEY=xxx`) - AI-synthesized research with citations; set `INCLUDE_SOURCES=perplexity`.
-- `XIAOHONGSHU_API_BASE=http://localhost:18060` - Xiaohongshu/RED via a logged-in x-mcp browser plugin or `xiaohongshu-mcp` service; optional unless the local service runs on a custom URL. Request it per run with `--search xhs`.
-- DripStack (premium financial newsletter search) is requested-only: include it per run with `--search dripstack`. Free public search API, no key; never active unless requested.
+- `XIAOHONGSHU_API_BASE=http://localhost:18060` - Xiaohongshu/RED via a logged-in x-mcp browser plugin or `xiaohongshu-mcp` service; optional unless the local service runs on a custom URL. Opt in per run with `--search xhs`, or persistently via `INCLUDE_SOURCES=xiaohongshu`.
+- DripStack (premium financial newsletter search) is opt-in only: per run with `--search dripstack`, or persistently via `INCLUDE_SOURCES=dripstack`. Free public search API, no key; never active without the opt-in.
- `BSKY_HANDLE=you.bsky.social` + `BSKY_APP_PASSWORD=xxx` - Bluesky (free app password).
- `BRAVE_API_KEY=xxx` or `EXA_API_KEY=xxx` - web search backends.
diff --git a/skills/last30days/scripts/lib/pipeline.py b/skills/last30days/scripts/lib/pipeline.py
index 2386beb1..4447e112 100644
--- a/skills/last30days/scripts/lib/pipeline.py
+++ b/skills/last30days/scripts/lib/pipeline.py
@@ -188,11 +188,19 @@ def available_sources(
# GitHub is reachable via the unauthenticated REST tier too, so it is
# available even without a token/gh CLI (a token only raises rate limits).
available.append("github")
- # DripStack is requested-only (owner decision, #791): a commercial
- # third-party API must never receive default-run traffic. Request it per
- # run (--search dripstack) or via LAST30DAYS_DEFAULT_SEARCH; the search
- # API is free and public (no key), so the request itself is the gate.
- if requested_sources and "dripstack" in requested_sources:
+ # DripStack is opt-in only (owner decision, #791): a commercial
+ # third-party API must never receive default-run traffic. Opt in per run
+ # (--search dripstack) or persistently (INCLUDE_SOURCES=dripstack in
+ # .env, the LinkedIn/Perplexity pattern); the search API is free and
+ # public (no key), so the opt-in itself is the gate.
+ include_sources = {
+ token.strip()
+ for token in (config.get("INCLUDE_SOURCES") or "").lower().split(",")
+ if token.strip()
+ }
+ if "dripstack" in include_sources or (
+ requested_sources and "dripstack" in requested_sources
+ ):
available.append("dripstack")
if which("digg-pp-cli"):
available.append("digg")
@@ -219,7 +227,6 @@ def available_sources(
if requested_sources and "jobs" in requested_sources:
available.append("jobs")
# Perplexity Sonar: opt-in additive source via INCLUDE_SOURCES=perplexity
- include_sources = (config.get("INCLUDE_SOURCES") or "").lower().split(",")
if _has_perplexity_provider(config) and (
"perplexity" in include_sources or (requested_sources and "perplexity" in requested_sources)
):
@@ -241,7 +248,10 @@ def available_sources(
"trustpilot" in include_sources or (requested_sources and "trustpilot" in requested_sources)
):
available.append("trustpilot")
- if requested_sources and "xiaohongshu" in requested_sources and env.is_xiaohongshu_available(config):
+ if (
+ "xiaohongshu" in include_sources
+ or (requested_sources and "xiaohongshu" in requested_sources)
+ ) and env.is_xiaohongshu_available(config):
available.append("xiaohongshu")
# Threads: opt-in via INCLUDE_SOURCES (same pattern as perplexity/linkedin).
# Was auto-on with the key; gated so the onboarding "Everything" tier is a
diff --git a/tests/test_dripstack.py b/tests/test_dripstack.py
index fdd03f86..7f3806d3 100644
--- a/tests/test_dripstack.py
+++ b/tests/test_dripstack.py
@@ -106,3 +106,26 @@ def test_parse_emits_body_and_normalizer_prefers_it():
"dripstack", parsed[0], 0, "2026-06-12", "2026-07-12"
)
assert normalized.body == "Capital, offtake and datacenters."
+
+
+def test_dripstack_activates_via_persisted_include_sources():
+ """INCLUDE_SOURCES is the .env checkbox for persistent opt-ins (the
+ LinkedIn/Perplexity pattern); dripstack must honor it, not only --search."""
+ available = pipeline.available_sources(
+ {"INCLUDE_SOURCES": "dripstack"}, None, x_pending=False
+ )
+ assert "dripstack" in available
+
+
+def test_unrelated_include_sources_keeps_dripstack_off():
+ available = pipeline.available_sources(
+ {"INCLUDE_SOURCES": "linkedin,tiktok"}, None, x_pending=False
+ )
+ assert "dripstack" not in available
+
+
+def test_include_sources_tolerates_whitespace_around_commas():
+ available = pipeline.available_sources(
+ {"INCLUDE_SOURCES": "linkedin, dripstack"}, None, x_pending=False
+ )
+ assert "dripstack" in available
diff --git a/tests/test_xiaohongshu_api.py b/tests/test_xiaohongshu_api.py
index bce56a54..13e69fd9 100644
--- a/tests/test_xiaohongshu_api.py
+++ b/tests/test_xiaohongshu_api.py
@@ -170,3 +170,31 @@ def test_search_feeds_requires_logged_in_xiaohongshu_session():
"matcha latte", "2026-06-01", "2026-07-01",
"http://localhost:18060",
)
+
+
+def test_xiaohongshu_activates_via_persisted_include_sources(monkeypatch):
+ from unittest import mock as _mock
+ from lib import env as _env, pipeline as _pipeline
+
+ probe = _mock.Mock(return_value=True)
+ monkeypatch.setattr(_env, "is_xiaohongshu_available", probe)
+
+ available = _pipeline.available_sources(
+ {"INCLUDE_SOURCES": "xiaohongshu"}, None, x_pending=False
+ )
+
+ assert "xiaohongshu" in available
+ probe.assert_called_once()
+
+
+def test_xiaohongshu_probe_never_fires_without_any_opt_in(monkeypatch):
+ from unittest import mock as _mock
+ from lib import env as _env, pipeline as _pipeline
+
+ probe = _mock.Mock(return_value=True)
+ monkeypatch.setattr(_env, "is_xiaohongshu_available", probe)
+
+ available = _pipeline.available_sources({}, None, x_pending=False)
+
+ assert "xiaohongshu" not in available
+ probe.assert_not_called()