Skip to content
Shell
5.8 KB
Raw Blame

Reddit Research Agent - Architecture

Overview

The Reddit Research Agent is an autonomous system that discovers market research opportunities from Reddit discussions. It uses the hidden .json API endpoint to extract complete thread data, analyzes content with Claude, and stores structured insights in SQLite.

System Architecture

┌─────────────────────────────────────────────────────────────────┐
│                     Reddit Research Agent                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐       │
│  │   Scanner    │───▶│   Analyzer   │───▶│   Storage    │       │
│  │              │    │   (Claude)   │    │   (SQLite)   │       │
│  └──────────────┘    └──────────────┘    └──────────────┘       │
│         │                   │                   │                │
│         ▼                   ▼                   ▼                │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐       │
│  │ Reddit JSON  │    │   Insights   │    │   Reports    │       │
│  │     API      │    │  Extraction  │    │   & Alerts   │       │
│  └──────────────┘    └──────────────┘    └──────────────┘       │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Core Components

1. Reddit Fetcher (src/reddit/fetcher.ts)

Handles all Reddit API interactions using the hidden JSON endpoint.

Key Functions:

  • fetchThread(url) - Extracts complete thread with nested comments
  • searchSubreddit(subreddit, query) - Searches for relevant threads
  • findGoldmineThreads(subreddit) - Runs multiple searches to find high-value content

The Trick:

https://reddit.com/r/SaaS/comments/xyz/title/
                    ↓ Add .json
https://reddit.com/r/SaaS/comments/xyz/title/.json

2. Thread Analyzer (src/agent/analyzer.ts)

Analyzes thread content to extract market insights.

Two Modes:

  1. Claude Analysis - Full LLM analysis for deep insights
  2. Quick Analysis - Pattern matching for fast, free scanning

Extracted Insights:

  • Pain Points (with frequency and sentiment)
  • Product Opportunities (with confidence levels)
  • Competitor Mentions (positive/negative/neutral)
  • Buying Signals (looking, considering, ready, switched)
  • Key Quotes (high-engagement insights)

3. Database Layer (src/db/)

SQLite database for persistent storage.

Tables:

  • threads - Scanned thread metadata
  • insights - Extracted market insights
  • agent_runs - Run history and stats
  • tracked_sources - Configured subreddits/queries

4. Agent Runner (src/agent/runner.ts)

Orchestrates the full pipeline.

Modes:

  • Single Run - One scan cycle
  • Scheduled - Cron-based continuous monitoring

Data Flow

1. SCAN
   └─▶ Search subreddits for goldmine queries
   └─▶ Calculate goldmine scores
   └─▶ Save threads to database

2. ANALYZE
   └─▶ Fetch full thread content
   └─▶ Send to Claude for analysis
   └─▶ Parse structured insights

3. STORE
   └─▶ Save insights by type
   └─▶ Mark thread as analyzed
   └─▶ Update run statistics

4. REPORT (future)
   └─▶ Generate summaries
   └─▶ Send alerts
   └─▶ Export to Notion

Goldmine Score Algorithm

Threads are scored to prioritize high-value content:

score = 0

// Engagement signals
score += min(numComments / 10, 20)  // Up to 20 pts
score += upvoteRatio * 10           // Up to 10 pts
score += min(postScore / 50, 10)    // Up to 10 pts

// Content signals (keywords in title)
goldmineKeywords = ['wish', 'hate', 'frustrating', 'recommend', ...]
for (keyword in goldmineKeywords) {
  if (title.includes(keyword)) score += 3
}

Configuration

Environment Variables

ANTHROPIC_API_KEY=sk-ant-...  # Required for Claude analysis

Tracked Sources

// Add via CLI
npx tsx src/agent/index.ts add SaaS "wish existed"
npx tsx src/agent/index.ts add entrepreneur

// Or programmatically
addTrackedSource('smallbusiness', 'software');

Schedule

// Default: Daily at 9 AM
startScheduledAgent('0 9 * * *');

// Every 6 hours
startScheduledAgent('0 */6 * * *');

// Every hour
startScheduledAgent('0 * * * *');

Cost Estimation

Operation Est. Cost
Reddit fetch Free
Quick analysis Free
Claude analysis ~$0.01/thread
100 threads/day ~$1/day

Scaling Considerations

Current Limits

  • Reddit rate limits (need delays between requests)
  • SQLite single-writer (fine for single agent)
  • Claude API costs scale linearly

Future Optimizations

  • Batch Reddit requests
  • PostgreSQL for multi-agent
  • Caching for frequently-accessed threads
  • Priority queue for analysis