Skip to content

AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary

feat(reddit): free discovery (dedicated subs + RSS) + arctic-shift scores + SC thinness floor (#696)

Matt Van Horn4bd20f3f4beb
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7ef3b04d..933af6c6 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
+### Added
+
+- Free Reddit gets dedicated-subreddit lanes: entity-home subs (e.g. r/Kanye for "Kanye West", via the new `--dedicated-subreddits` flag) are pulled in full from top+hot+new listings and exempt from the relevance floor, since the whole sub is the topic. Fixes the over-aggressive floor that dropped on-topic posts whose titles lacked the entity name.
+- `reddit_arctic` resolves upvote counts for threads found only via RSS search (which carries no score) using the free, keyless arctic-shift archive — batched, paced, cached, and graceful-degrading. Reddit now gets headlines-with-points and best-comments-with-points entirely for free, at parity with ScrapeCreators.
+- `LAST30DAYS_REDDIT_SC_MIN_ITEMS` (default 0 = unchanged empty-only behavior): set above 0 to let the ScrapeCreators backup backfill a thin free Reddit run instead of sitting idle. Backfilled items merge deduped by post id.
+
+### Removed
+
+- The permanently-403 `search.json` Tier 0 is gone from the keyless Reddit path; discovery is RSS breadth + shreddit listing partials (real scores) + the dedicated-sub lanes, with no wasted 403 calls.
+
## [3.8.2] - 2026-06-25
### Added
diff --git a/skills/last30days/SKILL.md b/skills/last30days/SKILL.md
index 535b33ae..3d2dcaf9 100644
--- a/skills/last30days/SKILL.md
+++ b/skills/last30days/SKILL.md
@@ -987,6 +987,11 @@ The first search finds subreddits. The second gives you current events context (
Extract 3-5 subreddit names from the results. Store as `RESOLVED_SUBREDDITS` (comma-separated, no r/ prefix).
+**Dedicated vs broad subreddits.** Split the resolved subs into two buckets:
+- **Dedicated** = subreddits whose entire purpose IS the topic (the entity's home: `r/Kanye` / `r/WestSubEver` / `r/GoodAssSub` for "Kanye West", `r/OpenClaw` for OpenClaw). Every post there is on-topic. Store as `RESOLVED_DEDICATED_SUBREDDITS` and pass via `--dedicated-subreddits`. The engine pulls these in full (top+hot+new) and skips the relevance floor for them, so an on-topic post whose title lacks the entity name (a "BULLY Deluxe" thread in r/Kanye) is not dropped.
+- **Broad** = mixed-content communities where the topic is only sometimes discussed (`r/hiphopheads`, `r/Music`, category peers from 2a). Store as `RESOLVED_SUBREDDITS` and pass via `--subreddits`. These stay relevance-floored.
+Label conservatively: only a sub clearly named for / dedicated to the entity goes in the dedicated bucket. Most topics have 0-3 dedicated subs (people and products often have one; generic concepts have none). When unsure, treat it as broad.
+
**2a. Category-peer expansion (MANDATORY for product topics).** If the topic is a product in a recognizable category (AI image generation, AI video generation, AI coding agents, AI music, AI chat models, SaaS screen recording, prediction markets, etc.), the brand-specific subreddits that WebSearch returned are INSUFFICIENT. Add 2-3 peer subreddits from the category. Peer subs are where cross-product technique discussion actually lives. Missing them is the 2026-04-22 `GPT Image 2` failure mode: the model resolved `r/OpenAI, r/ChatGPT, r/singularity, r/ChatGPTpromptengineering` (all OpenAI-brand) and missed `r/StableDiffusion, r/midjourney, r/dalle2, r/aiArt` where prompting techniques are actually shared. The user had to manually prompt "check image generation reddits too" to get a usable run.
Canonical category peers (single source of truth; `scripts/lib/categories.py` mirrors this for the `--auto-resolve` engine path):
@@ -1252,7 +1257,8 @@ Then add to the engine command:
- `--plan "$QUERY_PLAN_FILE"` (path to the file you just wrote)
- `--x-handle={RESOLVED_HANDLE}` (from Step 0.5)
-- `--subreddits={RESOLVED_SUBREDDITS}` (from Step 0.55)
+- `--subreddits={RESOLVED_SUBREDDITS}` (broad/category subs, from Step 0.55)
+- `--dedicated-subreddits={RESOLVED_DEDICATED_SUBREDDITS}` (entity-home subs, from Step 0.55; pulled in full + floor-exempt)
- `--tiktok-hashtags={RESOLVED_HASHTAGS}` (from Step 0.55)
- `--tiktok-creators={RESOLVED_TIKTOK_CREATORS}` (from Step 0.55)
- `--ig-creators={RESOLVED_IG_CREATORS}` (from Step 0.55)
diff --git a/skills/last30days/scripts/last30days.py b/skills/last30days/scripts/last30days.py
index 0e57f0fb..3a05b448 100644
--- a/skills/last30days/scripts/last30days.py
+++ b/skills/last30days/scripts/last30days.py
@@ -356,7 +356,8 @@ def build_parser() -> argparse.ArgumentParser:
help="Analyze public jobs/careers postings as evidence-backed company focus signals.")
parser.add_argument("--plan", help="JSON query plan (skips internal LLM planner). Can be a JSON string or a file path.")
parser.add_argument("--save-suffix", help="Suffix for saved output filename (e.g., 'gemini' → kanye-west-raw-gemini.md)")
- parser.add_argument("--subreddits", help="Comma-separated subreddit names to search (e.g., SaaS,Entrepreneur)")
+ parser.add_argument("--subreddits", help="Comma-separated broad/category subreddit names to search (e.g., SaaS,Entrepreneur)")
+ parser.add_argument("--dedicated-subreddits", help="Comma-separated entity-home subreddit names (e.g., Kanye,WestSubEver). Pulled in full (top+hot+new) and exempt from the relevance floor since the whole sub is the topic.")
parser.add_argument("--tiktok-hashtags", help="Comma-separated TikTok hashtags without # (e.g., tella,screenrecording)")
parser.add_argument("--tiktok-creators", help="Comma-separated TikTok creator handles (e.g., TellaHQ,taborplace)")
parser.add_argument("--ig-creators", help="Comma-separated Instagram creator handles (e.g., tella.tv,laborstories)")
@@ -1080,6 +1081,7 @@ def main() -> int:
try:
x_related = [h.strip() for h in args.x_related.split(",") if h.strip()] if args.x_related else None
subreddits = [s.strip().removeprefix("r/") for s in args.subreddits.split(",") if s.strip()] if args.subreddits else None
+ dedicated_subreddits = [s.strip().removeprefix("r/") for s in args.dedicated_subreddits.split(",") if s.strip()] if args.dedicated_subreddits else None
tiktok_hashtags = [h.strip().lstrip("#") for h in args.tiktok_hashtags.split(",") if h.strip()] if args.tiktok_hashtags else None
tiktok_creators = [c.strip().lstrip("@") for c in args.tiktok_creators.split(",") if c.strip()] if args.tiktok_creators else None
ig_creators = [c.strip().lstrip("@") for c in args.ig_creators.split(",") if c.strip()] if args.ig_creators else None
@@ -1193,6 +1195,12 @@ def main() -> int:
f"{' vs '.join(vs_entities)}\n"
)
+ # Dedicated subs ride the config dict (already threaded to every source
+ # fetch) so the keyless Reddit path can pull them floor-exempt without
+ # widening pipeline.run / _retrieve_stream signatures.
+ if dedicated_subreddits:
+ config["_dedicated_subreddits"] = dedicated_subreddits
+
def _main_runner() -> schema.Report:
r = pipeline.run(
topic=topic,
diff --git a/skills/last30days/scripts/lib/env.py b/skills/last30days/scripts/lib/env.py
index 7f208e2f..c19b2de3 100644
--- a/skills/last30days/scripts/lib/env.py
+++ b/skills/last30days/scripts/lib/env.py
@@ -507,6 +507,7 @@ def get_config(policy: ConfigLoadPolicy | None = None) -> dict[str, Any]:
('LAST30DAYS_X_MODEL', None),
('LAST30DAYS_X_BACKEND', None),
('LAST30DAYS_REDDIT_BACKEND', None),
+ ('LAST30DAYS_REDDIT_SC_MIN_ITEMS', None),
('LAST30DAYS_STORE', None),
('LAST30DAYS_MEMORY_DIR', None),
('OPENAI_MODEL_PIN', None),
diff --git a/skills/last30days/scripts/lib/pipeline.py b/skills/last30days/scripts/lib/pipeline.py
index 5c925cfe..5704378a 100644
--- a/skills/last30days/scripts/lib/pipeline.py
+++ b/skills/last30days/scripts/lib/pipeline.py
@@ -1155,6 +1155,29 @@ def _fetch_x_backend(backend, subquery, from_date, to_date, depth, config):
return items, (err or "")
+def _reddit_post_key(item: dict) -> str:
+ """Stable per-thread dedupe key (base36 post id from the url/permalink)."""
+ url = item.get("url") or item.get("permalink") or ""
+ m = re.search(r"/comments/([A-Za-z0-9]+)", url)
+ return m.group(1) if m else url
+
+
+def _merge_reddit_items(free: list[dict], sc: list[dict]) -> list[dict]:
+ """Merge free + ScrapeCreators Reddit items, free first, deduped by post id.
+
+ Used when the thinness-floor trigger backfills a thin free run with SC, so a
+ thread present in both is never double-listed.
+ """
+ merged = list(free)
+ seen = {_reddit_post_key(it) for it in free}
+ for it in sc:
+ key = _reddit_post_key(it)
+ if key and key not in seen:
+ seen.add(key)
+ merged.append(it)
+ return merged
+
+
def _retrieve_stream(
*,
topic: str,
@@ -1196,6 +1219,7 @@ def _retrieve_stream(
# Use raw_topic so expand_reddit_queries() generates diverse variants
# from the original user topic, not the planner's narrowed search_query.
reddit_query = raw_topic or subquery.search_query
+ dedicated_subreddits = config.get("_dedicated_subreddits") or None
has_sc_key = bool(config.get("SCRAPECREATORS_API_KEY"))
sc_first = (
has_sc_key
@@ -1240,14 +1264,20 @@ def _retrieve_stream(
)
return [], {}
- # Default: public Reddit first (free, gets comments); SC as backup
+ # Default: public Reddit first (free). ScrapeCreators backfills when the
+ # free path is empty OR returns fewer than the configured thinness floor
+ # (LAST30DAYS_REDDIT_SC_MIN_ITEMS, default 0 = empty-only — today's
+ # behavior, no extra credit spend unless the user opts in).
+ try:
+ min_items = int(config.get("LAST30DAYS_REDDIT_SC_MIN_ITEMS") or 0)
+ except (TypeError, ValueError):
+ min_items = 0
+ public_results: list[dict] = []
try:
public_results = reddit_public.search_reddit_public(
reddit_query, from_date, to_date, depth=depth,
- subreddits=subreddits,
- )
- if public_results:
- return public_results, {}
+ subreddits=subreddits, dedicated_subreddits=dedicated_subreddits,
+ ) or []
except Exception as exc:
sys.stderr.write(
f"[Reddit] Public search failed ({type(exc).__name__}: {exc})"
@@ -1256,20 +1286,30 @@ def _retrieve_stream(
sys.stderr.write("\n")
return [], {}
sys.stderr.write(", using ScrapeCreators backup\n")
- if has_sc_key:
- try:
- result = reddit.search_and_enrich(
- reddit_query, from_date, to_date, depth=depth,
- token=config.get("SCRAPECREATORS_API_KEY"),
- subreddits=subreddits,
- )
- return reddit.parse_reddit_response(result), {}
- except Exception as exc:
- sys.stderr.write(
- f"[Reddit] ScrapeCreators backup also failed "
- f"({type(exc).__name__}: {exc})\n"
- )
- return [], {}
+ # Enough free results, or no key to backfill with -> done. max(min_items,
+ # 1) keeps the default (min_items=0) as empty-only AND treats exactly
+ # `min_items` results as acceptable (no backfill) for min_items > 0.
+ if len(public_results) >= max(min_items, 1) or not has_sc_key:
+ return public_results, {}
+ if public_results:
+ sys.stderr.write(
+ f"[Reddit] Free path returned {len(public_results)} "
+ f"(below the {min_items}-item floor); backfilling with ScrapeCreators\n"
+ )
+ try:
+ result = reddit.search_and_enrich(
+ reddit_query, from_date, to_date, depth=depth,
+ token=config.get("SCRAPECREATORS_API_KEY"),
+ subreddits=subreddits,
+ )
+ sc_items = reddit.parse_reddit_response(result)
+ except Exception as exc:
+ sys.stderr.write(
+ f"[Reddit] ScrapeCreators backup also failed "
+ f"({type(exc).__name__}: {exc})\n"
+ )
+ return public_results, {}
+ return _merge_reddit_items(public_results, sc_items), {}
if source == "x":
# One X source, an ordered chain of interchangeable backends. Try the
# primary; fall through to the next only if it returns nothing or errors.
diff --git a/skills/last30days/scripts/lib/reddit_arctic.py b/skills/last30days/scripts/lib/reddit_arctic.py
new file mode 100644
index 00000000..ebd8ee3c
--- /dev/null
+++ b/skills/last30days/scripts/lib/reddit_arctic.py
@@ -0,0 +1,92 @@
+"""Arctic-shift score resolver — post upvote counts by id, keyless and free.
+
+``search.json`` and ``/comments/{id}.json`` are 403 keyless, and ``search.rss``
+(used for discovery) carries titles but NO score; the shreddit listing partials
+score only posts that appear in a pulled listing. For a thread found only via
+global RSS search in a broad sub, the free score comes from arctic-shift
+(https://arctic-shift.photon-reddit.com), a public Reddit archive whose
+``/api/posts/ids`` returns the post object (score, num_comments, title) for a
+batch of base36 post ids. Scores are point-in-time snapshots — slightly stale vs
+live, which is fine for ranking and display.
+
+Best-effort, never raises. On rate-limit (HTTP 422 "slow down"), error, or an
+unreachable host it returns ``{}`` so the caller shows the thread without a point
+count rather than failing the Reddit source.
+"""
+
+import sys
+import time
+from typing import Dict, List
+
+from . import http
+
+API = "https://arctic-shift.photon-reddit.com/api/posts/ids"
+BATCH = 50 # ids per request
+TIMEOUT = 15
+MAX_BATCHES = 3 # cap total requests per run (bounds latency + rate-limit risk)
+PACE_SECONDS = 0.4 # gap between batches; arctic-shift answers 422 "slow down"
+CACHE_MAX = 4096 # hard size bound so the in-run memo can never grow unbounded
+# In-run memo: base36 id -> {score, num_comments}. Module-level so repeated
+# fetch_scores calls within one `/last30days` run (e.g. across subqueries) reuse
+# results, but capped at CACHE_MAX entries (never reached in a normal CLI run).
+# Tests clear it via reddit_arctic._cache.clear().
+_cache: Dict[str, Dict[str, int]] = {}
+
+
+def _log(msg: str) -> None:
+ sys.stderr.write(f"[ArcticShift] {msg}\n")
+ sys.stderr.flush()
+
+
+def fetch_scores(post_ids: List[str]) -> Dict[str, Dict[str, int]]:
+ """Return ``{base36_post_id: {"score", "num_comments"}}`` for the given ids.
+
+ Batched, paced, in-run cached, and never raises. Ids that fail or are absent
+ from the archive are simply missing from the result (caller degrades to no
+ point count for those threads).
+ """
+ out: Dict[str, Dict[str, int]] = {}
+ todo: List[str] = []
+ for pid in post_ids:
+ if not pid:
+ continue
+ if pid in _cache:
+ out[pid] = _cache[pid]
+ elif pid not in todo:
+ todo.append(pid)
+
+ batches = [todo[i:i + BATCH] for i in range(0, len(todo), BATCH)][:MAX_BATCHES]
+ for n, batch in enumerate(batches):
+ if n:
+ time.sleep(PACE_SECONDS)
+ try:
+ data = http.get(
+ f"{API}?ids={','.join(batch)}",
+ headers={"User-Agent": http.BROWSER_USER_AGENT},
+ timeout=TIMEOUT,
+ )
+ except Exception as e: # network error / non-200 — degrade, never raise
+ _log(f"lookup failed ({e}); {len(batch)} ids left unscored")
+ break
+ rows = (data or {}).get("data")
+ if not isinstance(rows, list):
+ # arctic-shift returns {"error": "..."} on rate-limit / bad request.
+ _log(f"unexpected response (rate-limited?): {str(data)[:80]}")
+ break
+ for row in rows:
+ if not isinstance(row, dict):
+ continue
+ rid = str(row.get("id") or "").removeprefix("t3_")
+ if not rid:
+ continue
+ try:
+ entry = {
+ "score": int(row.get("score") or 0),
+ "num_comments": int(row.get("num_comments") or 0),
+ }
+ except (TypeError, ValueError):
+ continue
+ if len(_cache) < CACHE_MAX:
+ _cache[rid] = entry
+ out[rid] = entry
+ return out
diff --git a/skills/last30days/scripts/lib/reddit_keyless.py b/skills/last30days/scripts/lib/reddit_keyless.py
index 7050b165..a377f573 100644
--- a/skills/last30days/scripts/lib/reddit_keyless.py
+++ b/skills/last30days/scripts/lib/reddit_keyless.py
@@ -1,17 +1,19 @@
-"""Keyless Reddit pipeline: tiered free search + comment enrichment.
+"""Keyless Reddit pipeline: free discovery + comment enrichment.
-Replaces the dead ``.json`` free path. Discovery tiers, cheapest/most-likely
-first; enrichment then runs on whatever was discovered:
+``search.json`` is permanently 403/429 keyless, so it is not used. Discovery
+runs on the surfaces that still serve data without a key, then enrichment runs
+on whatever was discovered:
- Tier 0 one-shot legacy ``.json`` search — demoted. Datacenter IPs get 403,
- but a residential machine (where the skill usually runs) may still
- get 200, so it is worth one cheap try. Honors the "brute-force .json"
- intent without depending on it.
- Tier 1 RSS discovery (reddit_rss) — keyless, robust, the load-bearing path.
- Tier 2 shreddit comment + count enrichment (reddit_shreddit) for top posts.
+ Dedicated lane entity-home subreddits (e.g. r/Kanye) pulled in full via the
+ shreddit listing partials (top+hot+new, real scores), kept
+ whole — floor-exempt — because the sub IS the topic.
+ RSS lane reddit_rss breadth (incl. global keyword search) + broad-sub
+ listing partials for real upvote scores. Relevance-floored.
+ Enrichment shreddit comment + count enrichment (reddit_shreddit) for the
+ top-ranked posts (author + score + text + permalink).
Returns ``[]`` (never raises) so ``pipeline.py`` can fall through to the
-ScrapeCreators backup when every keyless tier comes up empty.
+ScrapeCreators backup when every keyless lane comes up empty.
"""
import concurrent.futures
@@ -22,7 +24,7 @@ from typing import Any, Dict, List, Optional
from collections import Counter
-from . import reddit_rss, reddit_shreddit, reddit_listing
+from . import reddit_rss, reddit_shreddit, reddit_listing, reddit_arctic
# Scores are backfilled from popular derived subreddits, so an engagement-first
# final sort buries on-topic RSS hits under viral off-topic posts. A relevance
# floor + relevance-first final ranking keeps the section on-topic. Thresholds
@@ -33,6 +35,10 @@ ENRICH_LIMITS = reddit_shreddit.ENRICH_LIMITS
ENRICH_BUDGET = 45 # seconds total across all enrichment threads
MAX_ENRICH_WORKERS = 4
MAX_DERIVED_SUBS = 5 # subreddits derived from RSS results for score backfill
+# Dedicated subreddits (the entity's home, e.g. r/Kanye for "Kanye West") are
+# wholly on-topic, so pull top+hot+new — the top-of-month listing alone misses
+# fresh threads — and keep every item (floor-exempt).
+DEDICATED_SORTS = ["top", "hot", "new"]
def _relevance_rank_key(post: Dict[str, Any]) -> float:
@@ -52,16 +58,6 @@ def _log(msg: str) -> None:
sys.stderr.flush()
-def _tier0_json(topic: str, depth: str) -> List[Dict[str, Any]]:
- """One cheap global ``.json`` discovery attempt. Returns [] on the 403 wall."""
- try:
- from . import reddit_public
- return reddit_public.search(topic, depth=depth) or []
- except Exception as e: # never let the demoted tier sink the run
- _log(f"Tier 0 (.json) unavailable: {e}")
- return []
-
-
def _top_subreddits(posts: List[Dict[str, Any]], limit: int = MAX_DERIVED_SUBS) -> List[str]:
"""Most frequent subreddits across discovered posts (for score backfill)."""
counts = Counter(p.get("subreddit", "") for p in posts if p.get("subreddit"))
@@ -75,15 +71,27 @@ def _apply_scores(post: Dict[str, Any], scored: Dict[str, int]) -> None:
post["engagement"]["num_comments"] = scored["num_comments"]
-def _discover(topic: str, depth: str, subreddits: Optional[List[str]]) -> List[Dict[str, Any]]:
- # Tier 0: demoted one-shot .json (dead for normal users too, but free to try).
- posts = _tier0_json(topic, depth)
- if posts:
- _log(f"Tier 0 (.json) returned {len(posts)} posts")
- return posts
-
- # Tier 1: keyless discovery. RSS gives breadth (incl. global keyword search);
- # the listing partials give real upvote scores.
+def _discover(
+ topic: str,
+ depth: str,
+ subreddits: Optional[List[str]],
+ dedicated_subreddits: Optional[List[str]] = None,
+) -> List[Dict[str, Any]]:
+ # Dedicated lane: the entity's home subs are wholly on-topic. Pull
+ # top+hot+new (real scores from the listing) and mark them floor-exempt so
+ # an on-topic post whose title lacks the entity name is never dropped.
+ dedicated_posts: List[Dict[str, Any]] = []
+ if dedicated_subreddits:
+ dedicated_posts = reddit_listing.fetch_listings(
+ dedicated_subreddits, depth=depth, query=topic, sorts=DEDICATED_SORTS
+ )
+ for p in dedicated_posts:
+ p["dedicated"] = True
+ _log(f"Dedicated lane: {len(dedicated_posts)} posts from {dedicated_subreddits}")
+
+ # search.json is permanently 403/429 keyless (no Tier 0). Discovery is RSS
+ # breadth (incl. global keyword search) + broad-sub listing partials for
+ # real upvote scores.
rss_posts = reddit_rss.search_rss(topic, depth=depth, subreddits=subreddits)
if subreddits:
@@ -112,11 +120,13 @@ def _discover(topic: str, depth: str, subreddits: Optional[List[str]]) -> List[D
if pid:
score_map[pid] = {"score": p["score"], "num_comments": p["num_comments"]}
- # Merge: scored listing posts first (targeted only), then RSS breadth,
- # backfilled with real scores where the post appears in a listing.
+ # Merge: dedicated-sub posts first (floor-exempt), then scored broad listing
+ # posts (targeted only), then RSS breadth backfilled with real scores where
+ # the post appears in a listing. First writer wins the dedupe, so a thread
+ # in both the dedicated lane and a listing keeps its floor-exempt status.
merged: List[Dict[str, Any]] = []
seen: set = set()
- for p in listing_posts:
+ for p in dedicated_posts + listing_posts:
if p["url"] not in seen:
seen.add(p["url"])
merged.append(p)
@@ -128,6 +138,25 @@ def _discover(topic: str, depth: str, subreddits: Optional[List[str]]) -> List[D
_apply_scores(p, score_map[pid])
seen.add(p["url"])
merged.append(p)
+
+ # Backfill scores for RSS-only posts (no listing card scored them) from the
+ # free arctic-shift archive. Posts already scored by a listing keep that
+ # live score; arctic only fills the gap, and is best-effort (never raises).
+ need = [pid for p in merged
+ if not (p.get("engagement", {}).get("score"))
+ for pid in [reddit_listing._post_id(p["url"])] if pid]
+ if need:
+ scores = reddit_arctic.fetch_scores(need)
+ filled = 0
+ for p in merged:
+ if p.get("engagement", {}).get("score"):
+ continue
+ pid = reddit_listing._post_id(p["url"])
+ if pid in scores:
+ _apply_scores(p, scores[pid])
+ filled += 1
+ if filled:
+ _log(f"arctic-shift backfilled {filled} post scores")
return merged
@@ -228,22 +257,25 @@ def search_and_enrich(
to_date: str,
depth: str = "default",
subreddits: Optional[List[str]] = None,
+ dedicated_subreddits: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
- """Full keyless Reddit pipeline: discover (Tier 0/1) then enrich (Tier 2).
+ """Full keyless Reddit pipeline: discover then enrich.
Args:
topic: Search topic
from_date: Start date (YYYY-MM-DD)
to_date: End date (YYYY-MM-DD)
depth: 'quick', 'default', or 'deep'
- subreddits: Optional pre-resolved subreddit names (without r/)
+ subreddits: Optional pre-resolved broad/category subreddit names (no r/)
+ dedicated_subreddits: Optional entity-home subreddit names (no r/) pulled
+ in full (top+hot+new) and exempt from the relevance floor.
Returns:
List of normalized item dicts matching the reddit_public output shape,
with top_comments/comment_insights attached on enriched posts.
Empty list when all keyless tiers fail (so SC backup can engage).
"""
- posts = _discover(topic, depth, subreddits)
+ posts = _discover(topic, depth, subreddits, dedicated_subreddits)
if not posts:
return []
@@ -258,11 +290,13 @@ def search_and_enrich(
# backfilled high-upvote posts from popular subs can't bury on-topic RSS
# hits. Keep all only when nothing scored above zero.
before = len(posts)
- on_topic = [p for p in posts if (p.get("relevance") or 0) >= RELEVANCE_FLOOR]
+ # Dedicated-sub posts are floor-exempt: their whole subreddit is the topic,
+ # so an on-topic post whose title lacks the entity name must not be dropped.
+ on_topic = [p for p in posts if p.get("dedicated") or (p.get("relevance") or 0) >= RELEVANCE_FLOOR]
if len(on_topic) >= MIN_ON_TOPIC:
posts = on_topic
else:
- nonzero = [p for p in posts if (p.get("relevance") or 0) > 0]
+ nonzero = [p for p in posts if p.get("dedicated") or (p.get("relevance") or 0) > 0]
if nonzero:
posts = nonzero
if len(posts) < before:
diff --git a/skills/last30days/scripts/lib/reddit_listing.py b/skills/last30days/scripts/lib/reddit_listing.py
index f3b92d2b..5cda23a0 100644
--- a/skills/last30days/scripts/lib/reddit_listing.py
+++ b/skills/last30days/scripts/lib/reddit_listing.py
@@ -140,15 +140,20 @@ def fetch_listings(
subreddits: List[str],
depth: str = "default",
query: str = "",
+ sorts: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
- """Fetch scored post cards across subreddits × depth-appropriate sorts.
+ """Fetch scored post cards across subreddits × sorts.
Returns deduped normalized posts (with real scores), unranked/unsliced —
the caller merges these with other sources, ranks, and slices.
+
+ ``sorts`` overrides the depth-derived sort set. Dedicated-subreddit lanes
+ pass ``["top", "hot", "new"]`` so fresh threads (which the top-of-month
+ listing misses) are caught with their scores regardless of depth.
"""
if not subreddits:
return []
- sorts = LISTING_SORTS.get(depth, LISTING_SORTS["default"])
+ sorts = sorts or LISTING_SORTS.get(depth, LISTING_SORTS["default"])
jobs = [(sub, sort) for sub in subreddits for sort in sorts]
all_posts: List[Dict[str, Any]] = []
with ThreadPoolExecutor(max_workers=min(MAX_WORKERS, len(jobs)) or 1) as executor:
diff --git a/skills/last30days/scripts/lib/reddit_public.py b/skills/last30days/scripts/lib/reddit_public.py
index b6789b06..ec8b1db2 100644
--- a/skills/last30days/scripts/lib/reddit_public.py
+++ b/skills/last30days/scripts/lib/reddit_public.py
@@ -242,18 +242,20 @@ def search_reddit_public(
to_date: str,
depth: str = "default",
subreddits: Optional[List[str]] = None,
+ dedicated_subreddits: Optional[List[str]] = None,
) -> List[Dict[str, Any]]:
"""High-level free Reddit search + enrichment (keyless).
- Thin compatibility shim over the tiered keyless pipeline: the legacy
- ``.json`` search/enrichment endpoints now return HTTP 403, so this delegates
- to ``reddit_keyless.search_and_enrich`` (Tier 0 one-shot ``.json`` →
- Tier 1 RSS discovery → Tier 2 shreddit comment enrichment). The name and
- signature are preserved so ``pipeline.py`` and other callers need no change
- and the ScrapeCreators backup still engages when this returns empty.
+ Thin compatibility shim over the keyless pipeline: the legacy ``.json``
+ search/enrichment endpoints now return HTTP 403, so this delegates to
+ ``reddit_keyless.search_and_enrich`` (dedicated-sub listings + RSS discovery
+ → shreddit comment enrichment; no ``.json`` search). The name and signature
+ are preserved so ``pipeline.py`` and other callers need no change and the
+ ScrapeCreators backup still engages when this returns empty.
- The module-level ``search`` / ``_parse_posts`` helpers remain in use as the
- keyless pipeline's demoted Tier 0 ``.json`` attempt.
+ The module-level ``search`` / ``_parse_posts`` helpers remain as a
+ standalone ``.json`` search utility (own test coverage), no longer wired
+ into the keyless production path.
Args:
topic: Search topic
@@ -268,5 +270,6 @@ def search_reddit_public(
"""
from . import reddit_keyless
return reddit_keyless.search_and_enrich(
- topic, from_date, to_date, depth=depth, subreddits=subreddits
+ topic, from_date, to_date, depth=depth, subreddits=subreddits,
+ dedicated_subreddits=dedicated_subreddits,
)
diff --git a/tests/conftest.py b/tests/conftest.py
index 14195d18..9dc63d7d 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,4 +1,16 @@
import sys
from pathlib import Path
+from unittest import mock
+
+import pytest
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "skills" / "last30days" / "scripts"))
+
+
[email protected](autouse=True)
+def _no_arctic_network():
+ """Default the arctic-shift score lookup to a no-op so the suite never makes
+ a real network call. test_reddit_arctic overrides this fixture (same name) to
+ exercise the real lookup with http.get mocked."""
+ with mock.patch("lib.reddit_arctic.fetch_scores", return_value={}):
+ yield
diff --git a/tests/test_codex_host_contract.py b/tests/test_codex_host_contract.py
index 4542500e..bfccdf29 100644
--- a/tests/test_codex_host_contract.py
+++ b/tests/test_codex_host_contract.py
@@ -114,3 +114,13 @@ def test_plan_invocation_warns_against_bash_lc_apostrophe_wrapper():
text = SKILL_MD.read_text(encoding="utf-8")
assert "bash -lc '...'" in text
assert "unmatched" in text
+
+
+def test_step055_documents_dedicated_vs_broad_subreddits():
+ # Step 0.55 must instruct the model to split entity-home (dedicated) subs from
+ # broad subs and pass them via --dedicated-subreddits, which the engine pulls
+ # in full and exempts from the relevance floor.
+ text = SKILL_MD.read_text(encoding="utf-8")
+ assert "RESOLVED_DEDICATED_SUBREDDITS" in text
+ assert "--dedicated-subreddits" in text
+ assert "relevance floor" in text
diff --git a/tests/test_reddit_arctic.py b/tests/test_reddit_arctic.py
new file mode 100644
index 00000000..c4d6bdac
--- /dev/null
+++ b/tests/test_reddit_arctic.py
@@ -0,0 +1,87 @@
+"""Tests for reddit_arctic — keyless post-score lookup via arctic-shift (U6)."""
+
+from unittest import mock
+
+import pytest
+
+from lib import reddit_arctic
+
+
[email protected](autouse=True)
+def _no_arctic_network():
+ # Override conftest's stub so this module exercises the real fetch_scores;
+ # network is mocked per-test at http.get. Clear the in-run cache each time.
+ reddit_arctic._cache.clear()
+ yield
+ reddit_arctic._cache.clear()
+
+
+def _resp(rows):
+ return {"data": rows}
+
+
+class TestFetchScores:
+ def test_returns_scores_by_id(self):
+ with mock.patch.object(reddit_arctic.http, "get",
+ return_value=_resp([{"id": "abc", "score": 1531, "num_comments": 336}])):
+ out = reddit_arctic.fetch_scores(["abc"])
+ assert out == {"abc": {"score": 1531, "num_comments": 336}}
+
+ def test_strips_t3_prefix(self):
+ with mock.patch.object(reddit_arctic.http, "get",
+ return_value=_resp([{"id": "t3_xyz", "score": 5, "num_comments": 2}])):
+ out = reddit_arctic.fetch_scores(["xyz"])
+ assert out["xyz"]["score"] == 5
+
+ def test_rate_limit_response_degrades(self):
+ # arctic-shift answers {"error": "...slow down"} with no data list.
+ with mock.patch.object(reddit_arctic.http, "get",
+ return_value={"error": "Timeout. Maybe slow down a bit"}):
+ out = reddit_arctic.fetch_scores(["abc"])
+ assert out == {}
+
+ def test_network_error_degrades(self):
+ with mock.patch.object(reddit_arctic.http, "get", side_effect=Exception("boom")):
+ out = reddit_arctic.fetch_scores(["abc"])
+ assert out == {}
+
+ def test_in_run_cache_avoids_refetch(self):
+ with mock.patch.object(reddit_arctic.http, "get",
+ return_value=_resp([{"id": "abc", "score": 9, "num_comments": 1}])) as g:
+ reddit_arctic.fetch_scores(["abc"])
+ reddit_arctic.fetch_scores(["abc"]) # served from cache, no 2nd call
+ assert g.call_count == 1
+
+ def test_dedupes_into_single_batch(self):
+ with mock.patch.object(reddit_arctic.http, "get",
+ return_value=_resp([{"id": "a", "score": 1, "num_comments": 0},
+ {"id": "b", "score": 2, "num_comments": 0}])) as g:
+ out = reddit_arctic.fetch_scores(["a", "b", "a", ""])
+ assert g.call_count == 1
+ assert set(out) == {"a", "b"}
+
+ def test_cache_is_size_bounded(self):
+ # The in-run memo never grows past CACHE_MAX; scores are still returned
+ # for the current call once the cache is full.
+ with mock.patch.object(reddit_arctic, "CACHE_MAX", 1), \
+ mock.patch.object(reddit_arctic.http, "get",
+ return_value=_resp([{"id": "a", "score": 1, "num_comments": 0},
+ {"id": "b", "score": 2, "num_comments": 0}])):
+ out = reddit_arctic.fetch_scores(["a", "b"])
+ assert set(out) == {"a", "b"} # both returned
+ assert len(reddit_arctic._cache) <= 1 # cache stayed bounded
+
+ def test_empty_input_makes_no_call(self):
+ with mock.patch.object(reddit_arctic.http, "get") as g:
+ out = reddit_arctic.fetch_scores([])
+ assert out == {}
+ g.assert_not_called()
+
+ def test_malformed_rows_skipped(self):
+ rows = [{"id": "ok", "score": 7, "num_comments": 3},
+ {"id": "", "score": 1}, # no id
+ "not-a-dict", # junk
+ {"id": "bad", "score": "x"}] # unparseable score
+ with mock.patch.object(reddit_arctic.http, "get", return_value=_resp(rows)):
+ out = reddit_arctic.fetch_scores(["ok", "bad"])
+ assert out == {"ok": {"score": 7, "num_comments": 3}}
diff --git a/tests/test_reddit_dedicated.py b/tests/test_reddit_dedicated.py
new file mode 100644
index 00000000..56fc20cf
--- /dev/null
+++ b/tests/test_reddit_dedicated.py
@@ -0,0 +1,99 @@
+"""Tests for the dedicated-subreddit lane in reddit_keyless (U1).
+
+A dedicated subreddit (the entity's home, e.g. r/Kanye for "Kanye West") is
+wholly on-topic: pulled in full via top+hot+new listings and exempt from the
+relevance floor, so an on-topic post whose title lacks the entity name is kept.
+"""
+
+from unittest import mock
+
+from lib import reddit_keyless
+
+
+def _post(i, date="2026-05-20", rel=0.0):
+ url = f"https://www.reddit.com/r/test/comments/{i:06d}/post_{i}/"
+ return {
+ "id": "", "title": f"Post {i}", "url": url, "score": 0, "num_comments": 0,
+ "subreddit": "test", "created_utc": None, "author": "u", "selftext": "",
+ "date": date, "engagement": {"score": 0, "num_comments": 0, "upvote_ratio": None},
+ "relevance": rel, "why_relevant": "Reddit RSS", "metadata": {},
+ }
+
+
+def _listing_post(i, score, title, rel=0.0):
+ p = _post(i, rel=rel)
+ p["title"] = title
+ p["score"] = score
+ p["engagement"]["score"] = score
+ p["why_relevant"] = "Reddit listing"
+ p["metadata"] = {"post_id": f"{i:06d}"}
+ return p
+
+
+def _no_enrich():
+ return mock.patch.object(
+ reddit_keyless.reddit_shreddit, "fetch_comments",
+ return_value={"top_comments": [], "comment_insights": [], "num_comments": None},
+ )
+
+
+class TestDedicatedLane:
+ def test_dedicated_listings_pulled_with_top_hot_new_and_marked(self):
+ ded = _listing_post(1, 2643, "What the actual fuck is this ye?")
+ captured = {}
+
+ def fake_fetch(subs, depth="default", query="", sorts=None):
+ if sorts == reddit_keyless.DEDICATED_SORTS:
+ captured["sorts"] = sorts
+ captured["subs"] = subs
+ return [ded]
+ return []
+
+ with mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings",
+ side_effect=fake_fetch), \
+ mock.patch.object(reddit_keyless.reddit_rss, "search_rss", return_value=[]):
+ out = reddit_keyless._discover("Kanye West", "default", None,
+ dedicated_subreddits=["Kanye"])
+ assert captured["sorts"] == ["top", "hot", "new"]
+ assert captured["subs"] == ["Kanye"]
+ assert len(out) == 1
+ assert out[0]["dedicated"] is True
+
+ def test_dedicated_post_survives_floor_without_entity_name(self):
+ # On-topic dedicated post whose title lacks "Kanye" (relevance 0) must be
+ # kept; off-topic non-dedicated posts (relevance 0) are floored out.
+ ded = _listing_post(1, 2284, "There's 2 types of people", rel=0.0)
+ ded["dedicated"] = True
+ offtopic = [_post(10 + i, rel=0.0) for i in range(5)]
+ with mock.patch.object(reddit_keyless, "_discover", return_value=[ded] + offtopic), \
+ _no_enrich():
+ out = reddit_keyless.search_and_enrich("Kanye West", "2026-05-01", "2026-05-31")
+ urls = {p["url"] for p in out}
+ assert ded["url"] in urls # dedicated kept despite relevance 0
+ assert all(o["url"] not in urls for o in offtopic) # off-topic floored
+
+ def test_dedicated_dedup_keeps_floor_exempt_status(self):
+ # A thread present in both the dedicated lane and a broad listing keeps
+ # its dedicated (floor-exempt) flag — dedicated is merged first.
+ shared_ded = _listing_post(1, 500, "fresh thread", rel=0.0)
+ shared_broad = _listing_post(1, 500, "fresh thread", rel=0.0) # same url/id
+
+ def fake_fetch(subs, depth="default", query="", sorts=None):
+ return [shared_ded] if sorts == reddit_keyless.DEDICATED_SORTS else [shared_broad]
+
+ with mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings",
+ side_effect=fake_fetch), \
+ mock.patch.object(reddit_keyless.reddit_rss, "search_rss", return_value=[]):
+ out = reddit_keyless._discover("Kanye West", "default", ["hiphopheads"],
+ dedicated_subreddits=["Kanye"])
+ same = [p for p in out if p["url"] == shared_ded["url"]]
+ assert len(same) == 1
+ assert same[0].get("dedicated") is True
+
+ def test_no_dedicated_subs_is_noop(self):
+ with mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
+ return_value=[_post(1)]), \
+ mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings",
+ return_value=[]):
+ out = reddit_keyless._discover("topic", "default", ["test"])
+ assert all(not p.get("dedicated") for p in out)
diff --git a/tests/test_reddit_dispatch.py b/tests/test_reddit_dispatch.py
new file mode 100644
index 00000000..073d2336
--- /dev/null
+++ b/tests/test_reddit_dispatch.py
@@ -0,0 +1,109 @@
+"""Pipeline Reddit dispatch: free-first, SC thinness-floor backfill (U7), and
+that the keyless path never calls search.json (U2)."""
+
+from unittest import mock
+
+from lib import pipeline, reddit_keyless, schema
+
+
+def _subquery():
+ return schema.SubQuery(label="t", search_query="kanye", ranking_query="kanye",
+ sources=["reddit"])
+
+
+def _runtime():
+ return schema.ProviderRuntime(reasoning_provider="mock", planner_model="mock",
+ rerank_model="mock")
+
+
+def _item(rid):
+ return {"url": f"https://www.reddit.com/r/x/comments/{rid}/t/", "title": rid}
+
+
+def _ids(items):
+ return [pipeline._reddit_post_key(i) for i in items]
+
+
+class TestThinnessFloor:
+ KEY = {"SCRAPECREATORS_API_KEY": "k"}
+
+ def _run(self, config, public, sc_parsed):
+ with mock.patch("lib.reddit_public.search_reddit_public", return_value=public), \
+ mock.patch("lib.reddit.search_and_enrich", return_value={"raw": 1}) as sc, \
+ mock.patch("lib.reddit.parse_reddit_response", return_value=sc_parsed):
+ items, _ = pipeline._retrieve_stream(
+ topic="kanye", subquery=_subquery(), source="reddit", config=config,
+ depth="quick", date_range=("2026-05-26", "2026-06-25"),
+ runtime=_runtime(), mock=False,
+ )
+ return items, sc
+
+ def test_default_zero_does_not_call_sc_when_free_has_items(self):
+ # min_items unset (0): today's empty-only behavior — free wins, no SC.
+ items, sc = self._run(self.KEY, [_item("a"), _item("b"), _item("c")], [_item("z")])
+ assert len(items) == 3
+ sc.assert_not_called()
+
+ def test_default_empty_free_falls_to_sc(self):
+ items, sc = self._run(self.KEY, [], [_item("z")])
+ assert _ids(items) == ["z"]
+ sc.assert_called_once()
+
+ def test_threshold_fires_on_thin_run_and_merges_deduped(self):
+ cfg = {**self.KEY, "LAST30DAYS_REDDIT_SC_MIN_ITEMS": "5"}
+ free = [_item("a"), _item("b")] # 2 < 5 -> backfill
+ sc_parsed = [_item("b"), _item("c")] # overlaps "b"
+ items, sc = self._run(cfg, free, sc_parsed)
+ sc.assert_called_once()
+ assert _ids(items) == ["a", "b", "c"] # free first, dedup b, append c
+
+ def test_threshold_not_fired_when_free_above_floor(self):
+ cfg = {**self.KEY, "LAST30DAYS_REDDIT_SC_MIN_ITEMS": "2"}
+ items, sc = self._run(cfg, [_item("a"), _item("b"), _item("c")], [_item("z")])
+ sc.assert_not_called()
+ assert len(items) == 3
+
+ def test_exactly_floor_is_acceptable_no_backfill(self):
+ # MIN_ITEMS=N means N results are acceptable; only fewer than N backfills.
+ cfg = {**self.KEY, "LAST30DAYS_REDDIT_SC_MIN_ITEMS": "3"}
+ items, sc = self._run(cfg, [_item("a"), _item("b"), _item("c")], [_item("z")])
+ sc.assert_not_called()
+ assert len(items) == 3
+
+ def test_no_key_never_calls_sc(self):
+ items, sc = self._run({"LAST30DAYS_REDDIT_SC_MIN_ITEMS": "5"}, [_item("a")], [_item("z")])
+ sc.assert_not_called()
+ assert len(items) == 1
+
+ def test_bad_threshold_value_defaults_to_empty_only(self):
+ cfg = {**self.KEY, "LAST30DAYS_REDDIT_SC_MIN_ITEMS": "not-an-int"}
+ items, sc = self._run(cfg, [_item("a")], [_item("z")])
+ sc.assert_not_called() # falls back to 0 -> free (1 item) wins
+ assert len(items) == 1
+
+ def test_sc_failure_degrades_to_free(self):
+ cfg = {**self.KEY, "LAST30DAYS_REDDIT_SC_MIN_ITEMS": "5"}
+ with mock.patch("lib.reddit_public.search_reddit_public", return_value=[_item("a")]), \
+ mock.patch("lib.reddit.search_and_enrich", side_effect=Exception("down")):
+ items, _ = pipeline._retrieve_stream(
+ topic="kanye", subquery=_subquery(), source="reddit", config=cfg,
+ depth="quick", date_range=("2026-05-26", "2026-06-25"),
+ runtime=_runtime(), mock=False,
+ )
+ assert _ids(items) == ["a"] # backup failed -> keep the free items
+
+
+class TestMergeHelper:
+ def test_dedup_by_post_id_free_first(self):
+ out = pipeline._merge_reddit_items([_item("a"), _item("b")], [_item("b"), _item("c")])
+ assert _ids(out) == ["a", "b", "c"]
+
+
+class TestNoSearchJson:
+ def test_keyless_discovery_never_calls_searchjson(self):
+ # reddit_public.search (the .json caller) must never run in the keyless flow.
+ with mock.patch("lib.reddit_public.search") as json_search, \
+ mock.patch("lib.reddit_keyless.reddit_rss.search_rss", return_value=[]), \
+ mock.patch("lib.reddit_keyless.reddit_listing.fetch_listings", return_value=[]):
+ reddit_keyless._discover("topic", "default", ["test"])
+ json_search.assert_not_called()
diff --git a/tests/test_reddit_keyless.py b/tests/test_reddit_keyless.py
index 1ffb52b0..161d8b6c 100644
--- a/tests/test_reddit_keyless.py
+++ b/tests/test_reddit_keyless.py
@@ -26,22 +26,11 @@ def _scored(i, score, ncmt=0):
return p
-class TestDiscoveryTierOrder:
- """Tier 0 (.json) is tried first; RSS + scored listings are the keyless path."""
+class TestDiscovery:
+ """RSS breadth + scored listings are the keyless discovery path (no .json)."""
- def test_tier0_success_skips_keyless(self):
- with mock.patch.object(reddit_keyless, "_tier0_json", return_value=[_post(1)]) as t0, \
- mock.patch.object(reddit_keyless.reddit_rss, "search_rss") as rss, \
- mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings") as lst:
- out = reddit_keyless._discover("topic", "default", None)
- assert len(out) == 1
- t0.assert_called_once()
- rss.assert_not_called()
- lst.assert_not_called()
-
- def test_tier0_empty_falls_to_keyless(self):
- with mock.patch.object(reddit_keyless, "_tier0_json", return_value=[]), \
- mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
+ def test_keyless_path_runs_rss_and_listings(self):
+ with mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
return_value=[_post(1), _post(2)]) as rss, \
mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings",
return_value=[]):
@@ -53,8 +42,7 @@ class TestDiscoveryTierOrder:
# RSS finds post 1 (no score); listing card for post 1 carries the score.
rss_post = _post(1)
listing_post = _scored(1, score=52692, ncmt=1743)
- with mock.patch.object(reddit_keyless, "_tier0_json", return_value=[]), \
- mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
+ with mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
return_value=[rss_post]), \
mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings",
return_value=[listing_post]):
@@ -69,8 +57,7 @@ class TestDiscoveryTierOrder:
rss_post = _post(7) # url .../000007/...
listing_post = _scored(7, score=999)
listing_post["url"] = "https://www.reddit.com/r/test/comments/zzzzzz/other/"
- with mock.patch.object(reddit_keyless, "_tier0_json", return_value=[]), \
- mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
+ with mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
return_value=[rss_post]), \
mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings",
return_value=[listing_post]):
@@ -85,8 +72,7 @@ class TestDiscoveryTierOrder:
rss_post = _post(1) # on-topic keyword match
offtopic_listing = _scored(99, score=88888) # high score, unrelated sub
offtopic_listing["url"] = "https://www.reddit.com/r/random/comments/zzz999/x/"
- with mock.patch.object(reddit_keyless, "_tier0_json", return_value=[]), \
- mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
+ with mock.patch.object(reddit_keyless.reddit_rss, "search_rss",
return_value=[rss_post]), \
mock.patch.object(reddit_keyless, "_top_subreddits", return_value=["random"]), \
mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings",
@@ -96,9 +82,8 @@ class TestDiscoveryTierOrder:
assert rss_post["url"] in urls
assert offtopic_listing["url"] not in urls # not merged as discovery
- def test_tier0_never_raises(self):
- with mock.patch("lib.reddit_public.search", side_effect=Exception("boom")), \
- mock.patch.object(reddit_keyless.reddit_rss, "search_rss", return_value=[]), \
+ def test_discover_never_raises_returns_empty(self):
+ with mock.patch.object(reddit_keyless.reddit_rss, "search_rss", return_value=[]), \
mock.patch.object(reddit_keyless.reddit_listing, "fetch_listings", return_value=[]):
assert reddit_keyless._discover("t", "default", None) == []
diff --git a/tests/test_reddit_public.py b/tests/test_reddit_public.py
index 10b91940..96154fe6 100644
--- a/tests/test_reddit_public.py
+++ b/tests/test_reddit_public.py
@@ -278,25 +278,23 @@ class TestDepthLimits:
class TestSearchRedditPublicHighLevel:
"""Test the high-level search_reddit_public interface."""
- @mock.patch("lib.reddit_public.urllib.request.urlopen")
- def test_date_filtering(self, mock_urlopen):
- listing = _make_reddit_listing([
- {
- "title": "In range",
- "permalink": "/r/test/comments/aaa/in_range/",
- "created_utc": 1711670400, # 2024-03-29
- },
- {
- "title": "Out of range",
- "permalink": "/r/test/comments/bbb/out_of_range/",
- "created_utc": 1609459200, # 2021-01-01
- },
- ])
- mock_urlopen.return_value = _mock_urlopen_ok(listing)
-
- results = reddit_public.search_reddit_public(
- "test", "2024-03-01", "2024-03-31"
- )
+ def test_date_filtering(self):
+ # search_reddit_public delegates to the keyless pipeline, which filters
+ # discovered posts to the requested date window. (search.json is gone, so
+ # the filter is verified at the keyless discovery seam, not via .json.)
+ def _p(title, slug, date):
+ return {
+ "id": "", "title": title, "score": 0, "num_comments": 0,
+ "url": f"https://www.reddit.com/r/test/comments/{slug}/x/",
+ "subreddit": "test", "author": "u", "selftext": "", "date": date,
+ "engagement": {"score": 0, "num_comments": 0}, "relevance": 0.0,
+ "metadata": {},
+ }
+ posts = [_p("In range", "aaa", "2024-03-29"), _p("Out of range", "bbb", "2021-01-01")]
+ with mock.patch("lib.reddit_keyless._discover", return_value=posts), \
+ mock.patch("lib.reddit_keyless.reddit_shreddit.fetch_comments",
+ return_value={"top_comments": [], "comment_insights": [], "num_comments": None}):
+ results = reddit_public.search_reddit_public("test", "2024-03-01", "2024-03-31")
titles = [r["title"] for r in results]
assert "In range" in titles
@@ -345,5 +343,5 @@ class TestSearchRedditPublicDelegatesToKeyless:
assert results == [{"id": "R1", "title": "x"}]
mock_keyless.assert_called_once_with(
"test", "2024-03-01", "2024-03-31",
- depth="quick", subreddits=["ClaudeAI"],
+ depth="quick", subreddits=["ClaudeAI"], dedicated_subreddits=None,
)