Shell
Reddit Research Agent - Architecture
Overview
The Reddit Research Agent is an autonomous system that discovers market research opportunities from Reddit discussions. It uses the hidden .json API endpoint to extract complete thread data, analyzes content with Claude, and stores structured insights in SQLite.
System Architecture
┌─────────────────────────────────────────────────────────────────┐
│ Reddit Research Agent │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Scanner │───▶│ Analyzer │───▶│ Storage │ │
│ │ │ │ (Claude) │ │ (SQLite) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Reddit JSON │ │ Insights │ │ Reports │ │
│ │ API │ │ Extraction │ │ & Alerts │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
Core Components
1. Reddit Fetcher (src/reddit/fetcher.ts)
Handles all Reddit API interactions using the hidden JSON endpoint.
Key Functions:
fetchThread(url)- Extracts complete thread with nested commentssearchSubreddit(subreddit, query)- Searches for relevant threadsfindGoldmineThreads(subreddit)- Runs multiple searches to find high-value content
The Trick:
https://reddit.com/r/SaaS/comments/xyz/title/
↓ Add .json
https://reddit.com/r/SaaS/comments/xyz/title/.json
2. Thread Analyzer (src/agent/analyzer.ts)
Analyzes thread content to extract market insights.
Two Modes:
- Claude Analysis - Full LLM analysis for deep insights
- Quick Analysis - Pattern matching for fast, free scanning
Extracted Insights:
- Pain Points (with frequency and sentiment)
- Product Opportunities (with confidence levels)
- Competitor Mentions (positive/negative/neutral)
- Buying Signals (looking, considering, ready, switched)
- Key Quotes (high-engagement insights)
3. Database Layer (src/db/)
SQLite database for persistent storage.
Tables:
threads- Scanned thread metadatainsights- Extracted market insightsagent_runs- Run history and statstracked_sources- Configured subreddits/queries
4. Agent Runner (src/agent/runner.ts)
Orchestrates the full pipeline.
Modes:
- Single Run - One scan cycle
- Scheduled - Cron-based continuous monitoring
Data Flow
1. SCAN
└─▶ Search subreddits for goldmine queries
└─▶ Calculate goldmine scores
└─▶ Save threads to database
2. ANALYZE
└─▶ Fetch full thread content
└─▶ Send to Claude for analysis
└─▶ Parse structured insights
3. STORE
└─▶ Save insights by type
└─▶ Mark thread as analyzed
└─▶ Update run statistics
4. REPORT (future)
└─▶ Generate summaries
└─▶ Send alerts
└─▶ Export to Notion
Goldmine Score Algorithm
Threads are scored to prioritize high-value content:
score = 0
// Engagement signals
score += min(numComments / 10, 20) // Up to 20 pts
score += upvoteRatio * 10 // Up to 10 pts
score += min(postScore / 50, 10) // Up to 10 pts
// Content signals (keywords in title)
goldmineKeywords = ['wish', 'hate', 'frustrating', 'recommend', ...]
for (keyword in goldmineKeywords) {
if (title.includes(keyword)) score += 3
}
Configuration
Environment Variables
ANTHROPIC_API_KEY=sk-ant-... # Required for Claude analysis
Tracked Sources
// Add via CLI
npx tsx src/agent/index.ts add SaaS "wish existed"
npx tsx src/agent/index.ts add entrepreneur
// Or programmatically
addTrackedSource('smallbusiness', 'software');
Schedule
// Default: Daily at 9 AM
startScheduledAgent('0 9 * * *');
// Every 6 hours
startScheduledAgent('0 */6 * * *');
// Every hour
startScheduledAgent('0 * * * *');
Cost Estimation
| Operation | Est. Cost |
|---|---|
| Reddit fetch | Free |
| Quick analysis | Free |
| Claude analysis | ~$0.01/thread |
| 100 threads/day | ~$1/day |
Scaling Considerations
Current Limits
- Reddit rate limits (need delays between requests)
- SQLite single-writer (fine for single agent)
- Claude API costs scale linearly
Future Optimizations
- Batch Reddit requests
- PostgreSQL for multi-agent
- Caching for frequently-accessed threads
- Priority queue for analysis