Skip to content

Autonomous AI sales agent powered by Claude Code or Codex. Finds prospects, writes cold emails, sends campaigns, handles replies, books meetings. No API keys needed — runs on your Claude or Codex subscriptions.

Upgrade trainer with Cloudflare /crawl for deep website crawling

Ethan48215bc2ba1f
diff --git a/.env.example b/.env.example
index 5e53c2c..ca2de95 100644
--- a/.env.example
+++ b/.env.example
@@ -5,6 +5,12 @@ INSTANTLY_API_KEY=your_instantly_api_key_here
LINKEDIN_EMAIL=your_linkedin_email
LINKEDIN_PASSWORD=your_linkedin_password
+# Cloudflare Browser Rendering (for deep website crawling during training)
+# $5/mo paid plan = ~12,000 pages/month. Free plan available with limits.
+# Get credentials at: https://dash.cloudflare.com → Workers & Pages → Browser Rendering
+CLOUDFLARE_ACCOUNT_ID=your_account_id_here
+CLOUDFLARE_API_TOKEN=your_api_token_here
+
# Hunter.io (optional, free tier = 50 lookups/mo)
HUNTER_API_KEY=your_hunter_api_key_here
diff --git a/README.md b/README.md
index 6447e1a..c420067 100644
--- a/README.md
+++ b/README.md
@@ -151,13 +151,7 @@ Point Harvey at your website and it learns everything automatically:
python -m harvey.trainer https://yourcompany.com
```
-Harvey will:
-1. Crawl up to 15 key pages (homepage, pricing, features, about, etc.)
-2. Extract your product info, benefits, pricing, and value proposition
-3. Identify your ideal customer profile (industries, titles, company size)
-4. Generate objection handling responses specific to your product
-5. Create a `product_knowledge.md` skill file with everything it learned
-6. Write a complete `harvey.yaml` config ready to use
+Harvey will deep-crawl your entire website, analyze every page, and generate a complete configuration:
```
============================================================
@@ -165,27 +159,74 @@ Harvey will:
Learning from: https://yourcompany.com
============================================================
-[1/5] Crawling website...
- Scraped 12 pages.
+[1/6] Deep crawling with Cloudflare (up to 100 pages)...
+ Crawling... 47/47 pages (34s)
+ Done! Crawled 47 pages.
-[2/5] Analyzing product information...
+[2/6] Analyzing product information...
Found: AcmeWidget
-[3/5] Identifying ideal customer profile...
+[3/6] Identifying ideal customer profile...
Target: E-commerce, DTC Brands
-[4/5] Generating objection handling playbook...
- Prepared 7 objection responses.
+[4/6] Analyzing competitive landscape...
+ Identified 4 competitors.
+
+[5/6] Generating objection handling playbook...
+ Prepared 10 objection responses.
-[5/5] Building configuration...
+[6/6] Building configuration and product knowledge...
+ Generated: skills/product_knowledge.md
+ Generated: skills/competitive_intel.md
============================================================
Training complete!
- Config written to: harvey.yaml
============================================================
+
+ Product: AcmeWidget
+ Company: Acme Corp
+ ICP titles: VP Marketing, Head of Growth, E-commerce Director
+ Industries: E-commerce, DTC Brands
+ Pages crawled: 47
+
+ Files generated:
+ - harvey.yaml
+ - skills/product_knowledge.md
+ - skills/competitive_intel.md
+```
+
+#### Deep Crawling with Cloudflare
+
+Harvey uses [Cloudflare's Browser Rendering /crawl API](https://developers.cloudflare.com/browser-rendering/rest-api/crawl-endpoint/) to crawl websites. This means:
+
+- **JavaScript rendering** — works on SPAs, React sites, dynamic content
+- **Automatic page discovery** — follows sitemaps and internal links
+- **Clean markdown output** — no HTML parsing needed
+- **Up to 100,000 pages** — crawl the entire site, not just key pages
+- **~$5/month** for ~12,000 pages on Cloudflare's paid Workers plan
+
+Add your Cloudflare credentials to `.env`:
+```
+CLOUDFLARE_ACCOUNT_ID=your_account_id
+CLOUDFLARE_API_TOKEN=your_api_token
+```
+
+Without Cloudflare credentials, Harvey falls back to basic HTTP scraping (still works, but no JS rendering and limited discovery).
+
+#### What the Trainer Generates
+
+| File | What's in it |
+|------|-------------|
+| `harvey.yaml` | Complete config — persona, product, ICP, channels, usage limits |
+| `skills/product_knowledge.md` | Everything about your product — features, benefits, use cases, pricing, social proof, pain points, buying triggers |
+| `skills/competitive_intel.md` | Battle cards for every competitor — how you win, their weaknesses, migration angles, ready-to-use responses |
+
+You can also crawl more pages for larger sites:
+```bash
+python -m harvey.trainer https://yourcompany.com 500
```
-Review the generated `harvey.yaml` and tweak anything that needs adjusting. The trainer gets you 90% of the way there.
+Review the generated files and tweak anything that needs adjusting. The trainer gets you 90% of the way there.
### 3. Add Credentials
diff --git a/harvey/brain.py b/harvey/brain.py
index 2e8632d..f435e49 100644
--- a/harvey/brain.py
+++ b/harvey/brain.py
@@ -125,12 +125,12 @@ class Brain:
Returns concatenated skill content that should be injected
into the agent's prompts as foundational knowledge.
"""
- # product_knowledge is auto-generated by the trainer
+ # product_knowledge and competitive_intel are auto-generated by the trainer
# and loaded by every agent that needs product context
skill_map = {
"scout": ["prospecting_tactics", "lead_qualification", "product_knowledge"],
- "writer": ["email_frameworks", "sales_methodology", "product_knowledge"],
- "handler": ["objection_handling", "sales_methodology", "product_knowledge"],
+ "writer": ["email_frameworks", "sales_methodology", "product_knowledge", "competitive_intel"],
+ "handler": ["objection_handling", "sales_methodology", "product_knowledge", "competitive_intel"],
"sender": ["email_frameworks", "product_knowledge"],
"linkedin": ["linkedin_outreach", "prospecting_tactics", "product_knowledge"],
}
diff --git a/harvey/trainer.py b/harvey/trainer.py
index b37bddb..b3c3805 100644
--- a/harvey/trainer.py
+++ b/harvey/trainer.py
@@ -1,128 +1,196 @@
-"""Trainer — give Harvey a URL and it learns everything about the product."""
+"""Trainer — give Harvey a URL and it learns everything about the product.
+
+Uses Cloudflare Browser Rendering /crawl API for deep site crawling.
+Falls back to manual scraping if no Cloudflare credentials are configured.
+"""
import asyncio
import json
import logging
+import os
import re
+import time
from pathlib import Path
from urllib.parse import urljoin, urlparse
import httpx
import yaml
from bs4 import BeautifulSoup
+from dotenv import load_dotenv
from harvey.brain import Brain
from harvey.state import StateManager
logger = logging.getLogger("harvey.trainer")
-# Pages most likely to contain useful product/company info
-PRIORITY_PATHS = [
- "/",
- "/about",
- "/about-us",
- "/pricing",
- "/features",
- "/product",
- "/products",
- "/solutions",
- "/services",
- "/how-it-works",
- "/why-us",
- "/case-studies",
- "/customers",
- "/testimonials",
- "/faq",
- "/contact",
- "/team",
- "/our-team",
+# Fallback: pages to try if Cloudflare crawl is unavailable
+FALLBACK_PATHS = [
+ "/", "/about", "/about-us", "/pricing", "/features", "/product",
+ "/products", "/solutions", "/services", "/how-it-works", "/why-us",
+ "/case-studies", "/customers", "/testimonials", "/faq", "/contact",
+ "/team", "/our-team", "/blog", "/resources", "/integrations",
]
-MAX_PAGES = 15
-MAX_CONTENT_PER_PAGE = 5000 # chars
+# Cloudflare API
+CF_API_BASE = "https://api.cloudflare.com/client/v4/accounts"
+# How long to poll before giving up (seconds)
+CRAWL_TIMEOUT = 600 # 10 minutes
+POLL_INTERVAL = 10 # check every 10 seconds
-class Trainer:
- def __init__(self):
- self.state = StateManager()
- self.brain = Brain(self.state)
- self.scraped_pages: dict[str, str] = {}
- async def train(self, url: str, output_path: str = "harvey.yaml"):
- """Train Harvey on a product website.
+class CloudflareCrawler:
+ """Cloudflare Browser Rendering /crawl API client."""
+
+ def __init__(self, account_id: str, api_token: str):
+ self.account_id = account_id
+ self.api_token = api_token
+ self.base_url = f"{CF_API_BASE}/{account_id}/browser-rendering/crawl"
+ self.headers = {
+ "Authorization": f"Bearer {api_token}",
+ "Content-Type": "application/json",
+ }
- 1. Crawl the site (key pages)
- 2. Extract all text content
- 3. Ask Claude to analyze and extract product info
- 4. Generate a complete harvey.yaml config
+ async def crawl(
+ self,
+ url: str,
+ max_pages: int = 100,
+ depth: int = 5,
+ render: bool = True,
+ ) -> dict[str, str]:
+ """Crawl a website and return {url: markdown_content} for all pages.
+
+ Uses Cloudflare's /crawl endpoint which:
+ - Renders JavaScript (SPAs, dynamic content)
+ - Discovers pages via sitemaps + links
+ - Returns content as clean markdown
+ - Respects robots.txt
+ - Can crawl up to 100,000 pages
"""
- await self.state.init_db()
+ async with httpx.AsyncClient(timeout=60, headers=self.headers) as client:
+ # 1. Start the crawl job
+ payload = {
+ "url": url,
+ "limit": max_pages,
+ "depth": depth,
+ "formats": ["markdown"],
+ "render": render,
+ "source": "all",
+ "options": {
+ "includeSubdomains": False,
+ "includeExternalLinks": False,
+ },
+ "rejectResourceTypes": ["image", "media", "font", "stylesheet"],
+ }
- domain = urlparse(url).netloc
- base_url = f"{urlparse(url).scheme}://{domain}"
+ logger.info(f"Starting Cloudflare crawl: {url} (max {max_pages} pages)")
+ resp = await client.post(self.base_url, json=payload)
- logger.info(f"Training Harvey on {base_url}...")
- print(f"\n{'='*60}")
- print(f" Harvey Training Mode")
- print(f" Learning from: {base_url}")
- print(f"{'='*60}\n")
+ if resp.status_code != 200:
+ logger.error(f"Cloudflare crawl start failed: {resp.status_code} {resp.text}")
+ return {}
- # Step 1: Crawl the site
- print("[1/5] Crawling website...")
- await self._crawl_site(base_url)
+ result = resp.json()
+ if not result.get("success"):
+ logger.error(f"Cloudflare crawl error: {result}")
+ return {}
- if not self.scraped_pages:
- print("ERROR: Could not scrape any pages. Check the URL.")
- return
+ job_id = result.get("result")
+ if not job_id:
+ logger.error("No job ID returned from Cloudflare crawl")
+ return {}
- print(f" Scraped {len(self.scraped_pages)} pages.\n")
+ logger.info(f"Crawl job started: {job_id}")
- # Step 2: Extract product information
- print("[2/5] Analyzing product information...")
- product_info = await self._extract_product_info()
- if not product_info:
- print("ERROR: Could not extract product information.")
- return
- print(f" Found: {product_info.get('product_name', 'Unknown Product')}\n")
+ # 2. Poll for completion
+ pages = await self._poll_job(client, job_id)
+ return pages
- # Step 3: Identify ICP
- print("[3/5] Identifying ideal customer profile...")
- icp_info = await self._extract_icp()
- print(f" Target: {', '.join(icp_info.get('industries', ['Unknown']))}\n")
+ async def _poll_job(self, client: httpx.AsyncClient, job_id: str) -> dict[str, str]:
+ """Poll a crawl job until complete, then collect all pages."""
+ start_time = time.time()
+ pages: dict[str, str] = {}
- # Step 4: Generate objection responses
- print("[4/5] Generating objection handling playbook...")
- objections = await self._generate_objections(product_info)
- print(f" Prepared {len(objections)} objection responses.\n")
+ while time.time() - start_time < CRAWL_TIMEOUT:
+ # Check status with minimal data
+ resp = await client.get(
+ f"{self.base_url}/{job_id}",
+ params={"limit": 1},
+ )
- # Step 5: Generate config
- print("[5/5] Building configuration...")
- config = self._build_config(product_info, icp_info, objections, domain)
+ if resp.status_code != 200:
+ logger.error(f"Poll failed: {resp.status_code}")
+ break
+
+ data = resp.json().get("result", {})
+ status = data.get("status", "unknown")
+ total = data.get("total", 0)
+ finished = data.get("finished", 0)
+
+ elapsed = int(time.time() - start_time)
+ print(f"\r Crawling... {finished}/{total} pages ({elapsed}s)", end="", flush=True)
+
+ if status in ("completed", "cancelled_due_to_timeout", "cancelled_due_to_limits"):
+ print() # newline after progress
+ if status != "completed":
+ logger.warning(f"Crawl ended with status: {status}")
+ break
+
+ if status == "errored":
+ logger.error("Crawl job errored")
+ print()
+ break
+
+ await asyncio.sleep(POLL_INTERVAL)
+
+ # 3. Collect all results with pagination
+ pages = await self._collect_results(client, job_id)
+ return pages
+
+ async def _collect_results(
+ self, client: httpx.AsyncClient, job_id: str
+ ) -> dict[str, str]:
+ """Paginate through all crawl results and collect markdown content."""
+ pages: dict[str, str] = {}
+ cursor = None
+
+ while True:
+ params = {"limit": 50, "status": "completed"}
+ if cursor:
+ params["cursor"] = cursor
+
+ resp = await client.get(
+ f"{self.base_url}/{job_id}",
+ params=params,
+ )
- # Write the config
- config_path = Path(output_path)
- with open(config_path, "w") as f:
- yaml.dump(config, f, default_flow_style=False, sort_keys=False)
+ if resp.status_code != 200:
+ break
- print(f"\n{'='*60}")
- print(f" Training complete!")
- print(f" Config written to: {config_path}")
- print(f"{'='*60}")
- print(f"\n Product: {config['product']['name']}")
- print(f" Company: {config['persona']['company']}")
- print(f" ICP: {', '.join(config['icp']['titles'])}")
- print(f" Industries: {', '.join(config['icp']['industries'])}")
- print(f"\n Review {config_path} and adjust as needed.")
- print(f" Then run: python -m harvey\n")
+ data = resp.json().get("result", {})
+ records = data.get("records", [])
- # Also generate custom prompts based on the product
- await self._generate_custom_prompts(product_info, icp_info)
+ for record in records:
+ url = record.get("url", "")
+ markdown = record.get("markdown", "")
+ if url and markdown:
+ pages[url] = markdown
- return config
+ # Check for next page
+ next_cursor = data.get("cursor")
+ if not next_cursor or next_cursor == cursor:
+ break
+ cursor = next_cursor
+
+ return pages
- async def _crawl_site(self, base_url: str):
- """Crawl key pages of the website."""
- crawled = set()
+
+class FallbackCrawler:
+ """Simple HTTP crawler for when Cloudflare is not configured."""
+
+ async def crawl(self, base_url: str, max_pages: int = 20) -> dict[str, str]:
+ """Crawl key pages using plain HTTP requests."""
+ pages: dict[str, str] = {}
async with httpx.AsyncClient(
timeout=15,
@@ -135,40 +203,35 @@ class Trainer:
)
},
) as client:
- # First, try priority paths
- for path in PRIORITY_PATHS:
- if len(self.scraped_pages) >= MAX_PAGES:
+ for path in FALLBACK_PATHS:
+ if len(pages) >= max_pages:
break
url = urljoin(base_url, path)
- if url in crawled:
- continue
- crawled.add(url)
-
content = await self._fetch_page(client, url)
if content:
- self.scraped_pages[path] = content
- logger.debug(f"Scraped {url} ({len(content)} chars)")
+ pages[url] = content
+ print(f"\r Scraped {len(pages)} pages...", end="", flush=True)
- # Then discover linked pages from homepage
- if "/" in self.scraped_pages:
- discovered = self._find_internal_links(
- self.scraped_pages["/"], base_url
- )
+ # Discover more pages from homepage links
+ homepage_url = urljoin(base_url, "/")
+ if homepage_url in pages:
+ discovered = await self._discover_links(client, base_url)
for link_url in discovered:
- if len(self.scraped_pages) >= MAX_PAGES:
+ if len(pages) >= max_pages:
break
- if link_url in crawled:
+ if link_url in pages:
continue
- crawled.add(link_url)
-
- path = urlparse(link_url).path
content = await self._fetch_page(client, link_url)
if content:
- self.scraped_pages[path] = content
+ pages[link_url] = content
+ print(f"\r Scraped {len(pages)} pages...", end="", flush=True)
+
+ print() # newline after progress
+ return pages
async def _fetch_page(self, client: httpx.AsyncClient, url: str) -> str:
- """Fetch a page and extract text content."""
+ """Fetch a page and convert to clean text."""
try:
resp = await client.get(url)
if resp.status_code != 200:
@@ -180,47 +243,176 @@ class Trainer:
soup = BeautifulSoup(resp.text, "html.parser")
- # Remove noise
- for tag in soup(["script", "style", "nav", "footer", "header", "noscript"]):
+ # Remove noise elements
+ for tag in soup(["script", "style", "nav", "footer", "noscript", "iframe"]):
tag.decompose()
text = soup.get_text(separator="\n", strip=True)
-
- # Clean up whitespace
lines = [line.strip() for line in text.splitlines() if line.strip()]
- text = "\n".join(lines)
-
- # Truncate if too long
- if len(text) > MAX_CONTENT_PER_PAGE:
- text = text[:MAX_CONTENT_PER_PAGE]
-
- return text
+ return "\n".join(lines)
except Exception as e:
logger.debug(f"Failed to fetch {url}: {e}")
return ""
- def _find_internal_links(self, html_content: str, base_url: str) -> list[str]:
- """Find internal links from page content (re-parse from raw isn't available,
- so we use the base_url to filter discovered links)."""
- # Since we only have text content at this point, we'd need the raw HTML.
- # For simplicity, we rely on PRIORITY_PATHS for discovery.
- return []
+ async def _discover_links(self, client: httpx.AsyncClient, base_url: str) -> list[str]:
+ """Find internal links from the homepage."""
+ try:
+ resp = await client.get(base_url)
+ if resp.status_code != 200:
+ return []
+
+ soup = BeautifulSoup(resp.text, "html.parser")
+ domain = urlparse(base_url).netloc
+ links = []
- async def _extract_product_info(self) -> dict:
- """Ask Claude to extract product information from scraped content."""
- # Combine all page content
+ for a in soup.find_all("a", href=True):
+ href = a["href"]
+ full_url = urljoin(base_url, href)
+ parsed = urlparse(full_url)
+
+ # Only internal links, no anchors/mailto/tel
+ if parsed.netloc == domain and parsed.scheme in ("http", "https"):
+ clean = f"{parsed.scheme}://{parsed.netloc}{parsed.path}"
+ if clean not in links:
+ links.append(clean)
+
+ return links[:30] # cap discovery
+
+ except Exception:
+ return []
+
+
+class Trainer:
+ def __init__(self):
+ self.state = StateManager()
+ self.brain = Brain(self.state)
+ self.scraped_pages: dict[str, str] = {}
+
+ async def train(
+ self,
+ url: str,
+ output_path: str = "harvey.yaml",
+ max_pages: int = 100,
+ ):
+ """Train Harvey on a product website.
+
+ 1. Deep crawl the site (Cloudflare /crawl or fallback)
+ 2. Analyze all content with Claude
+ 3. Extract product info, ICP, objections, competitive intel
+ 4. Generate complete config + product knowledge skill
+ """
+ load_dotenv()
+ await self.state.init_db()
+
+ parsed = urlparse(url)
+ domain = parsed.netloc
+ base_url = f"{parsed.scheme}://{domain}"
+
+ logger.info(f"Training Harvey on {base_url}...")
+ print(f"\n{'='*60}")
+ print(f" Harvey Training Mode")
+ print(f" Learning from: {base_url}")
+ print(f"{'='*60}\n")
+
+ # Step 1: Deep crawl
+ cf_account_id = os.getenv("CLOUDFLARE_ACCOUNT_ID", "")
+ cf_api_token = os.getenv("CLOUDFLARE_API_TOKEN", "")
+
+ if cf_account_id and cf_api_token:
+ print(f"[1/6] Deep crawling with Cloudflare (up to {max_pages} pages)...")
+ crawler = CloudflareCrawler(cf_account_id, cf_api_token)
+ self.scraped_pages = await crawler.crawl(
+ base_url, max_pages=max_pages, depth=10
+ )
+ else:
+ print("[1/6] Crawling website (add CLOUDFLARE_ACCOUNT_ID and")
+ print(" CLOUDFLARE_API_TOKEN to .env for deep JS-rendered crawling)...")
+ crawler = FallbackCrawler()
+ self.scraped_pages = await crawler.crawl(base_url, max_pages=min(max_pages, 30))
+
+ if not self.scraped_pages:
+ print("\nERROR: Could not scrape any pages. Check the URL.")
+ return None
+
+ print(f" Done! Crawled {len(self.scraped_pages)} pages.\n")
+
+ # Step 2: Extract product information
+ print("[2/6] Analyzing product information...")
+ product_info = await self._extract_product_info()
+ if not product_info:
+ print("ERROR: Could not extract product information.")
+ return None
+ print(f" Found: {product_info.get('product_name', 'Unknown Product')}\n")
+
+ # Step 3: Identify ICP
+ print("[3/6] Identifying ideal customer profile...")
+ icp_info = await self._extract_icp()
+ print(f" Target: {', '.join(icp_info.get('industries', ['Unknown']))}\n")
+
+ # Step 4: Competitive analysis
+ print("[4/6] Analyzing competitive landscape...")
+ competitive_intel = await self._extract_competitive_intel(product_info)
+ print(f" Identified {len(competitive_intel.get('competitors', []))} competitors.\n")
+
+ # Step 5: Generate objection responses
+ print("[5/6] Generating objection handling playbook...")
+ objections = await self._generate_objections(product_info, competitive_intel)
+ print(f" Prepared {len(objections)} objection responses.\n")
+
+ # Step 6: Generate config + skills
+ print("[6/6] Building configuration and product knowledge...")
+ config = self._build_config(product_info, icp_info, objections, domain)
+
+ # Write config
+ config_path = Path(output_path)
+ with open(config_path, "w") as f:
+ yaml.dump(config, f, default_flow_style=False, sort_keys=False)
+
+ # Generate product knowledge skill
+ await self._generate_product_knowledge(product_info, icp_info, competitive_intel)
+
+ # Generate competitive battle cards skill
+ if competitive_intel.get("competitors"):
+ await self._generate_battle_cards(product_info, competitive_intel)
+
+ print(f"\n{'='*60}")
+ print(f" Training complete!")
+ print(f"{'='*60}")
+ print(f"\n Product: {config['product']['name']}")
+ print(f" Company: {config['persona']['company']}")
+ print(f" ICP titles: {', '.join(config['icp']['titles'])}")
+ print(f" Industries: {', '.join(config['icp']['industries'])}")
+ print(f" Pages crawled: {len(self.scraped_pages)}")
+ print(f"\n Files generated:")
+ print(f" - {config_path}")
+ print(f" - skills/product_knowledge.md")
+ if competitive_intel.get("competitors"):
+ print(f" - skills/competitive_intel.md")
+ print(f"\n Review {config_path} and adjust as needed.")
+ print(f" Then run: python -m harvey\n")
+
+ return config
+
+ def _get_all_content(self, max_chars: int = 60000) -> str:
+ """Combine all scraped pages into one string for analysis."""
all_content = ""
- for path, content in self.scraped_pages.items():
- all_content += f"\n\n--- PAGE: {path} ---\n{content}"
+ for url, content in self.scraped_pages.items():
+ path = urlparse(url).path or "/"
+ # Truncate individual pages to keep total manageable
+ page_content = content[:8000] if len(content) > 8000 else content
+ all_content += f"\n\n--- PAGE: {path} ---\n{page_content}"
+ if len(all_content) > max_chars:
+ break
+ return all_content
- # Truncate to fit in a reasonable prompt
- if len(all_content) > 30000:
- all_content = all_content[:30000]
+ async def _extract_product_info(self) -> dict:
+ """Ask Claude to extract product information from all crawled content."""
+ all_content = self._get_all_content()
- prompt = f"""Analyze this website content and extract product/company information.
+ prompt = f"""Analyze this entire website and extract comprehensive product/company information.
-WEBSITE CONTENT:
+WEBSITE CONTENT ({len(self.scraped_pages)} pages):
{all_content}
Extract the following and return as JSON:
@@ -228,41 +420,48 @@ Extract the following and return as JSON:
"company_name": "the company name",
"product_name": "the main product or service name",
"product_description": "2-3 sentence description of what the product does and who it's for",
- "pricing": "pricing info if available, otherwise 'Contact for pricing'",
+ "pricing": "pricing info if available (tiers, starting price, etc.), otherwise 'Contact for pricing'",
"key_benefits": ["benefit 1", "benefit 2", "benefit 3", "benefit 4", "benefit 5"],
- "features": ["feature 1", "feature 2", "feature 3"],
- "target_audience": "who this product is for",
+ "features": ["feature 1", "feature 2", "feature 3", "feature 4", "feature 5", "feature 6"],
+ "target_audience": "who this product is for — be specific about roles and company types",
"value_proposition": "the core value proposition in one sentence",
- "competitors": ["competitor 1", "competitor 2"],
- "social_proof": ["any case studies, testimonials, or notable customers mentioned"],
+ "differentiators": ["what makes this different from alternatives — point 1", "point 2"],
+ "competitors": ["competitor 1", "competitor 2", "competitor 3"],
+ "social_proof": ["any case studies, testimonials, notable customers, or stats mentioned"],
+ "integrations": ["tools and platforms this integrates with"],
+ "use_cases": ["specific use case 1", "use case 2", "use case 3"],
"tone": "describe the brand's communication tone in 3-4 words"
-}}"""
+}}
+
+Be thorough — you have access to the full website. Extract everything relevant."""
result = await self.brain.think_json(prompt, session_id="harvey-trainer")
return result if isinstance(result, dict) else {}
async def _extract_icp(self) -> dict:
- """Ask Claude to determine the ideal customer profile."""
- all_content = "\n".join(
- f"[{path}]: {content[:2000]}"
- for path, content in list(self.scraped_pages.items())[:8]
- )
+ """Ask Claude to determine the ideal customer profile from all content."""
+ all_content = self._get_all_content(max_chars=40000)
- prompt = f"""Based on this website content, determine the Ideal Customer Profile (ICP).
+ prompt = f"""Based on this entire website, determine the Ideal Customer Profile (ICP).
+Look at case studies, testimonials, pricing tiers, feature descriptions, and target messaging
+to understand exactly who this product is built for.
-WEBSITE CONTENT:
+WEBSITE CONTENT ({len(self.scraped_pages)} pages):
{all_content}
-Determine who would buy this product and return as JSON:
+Return as JSON:
{{
"industries": ["industry 1", "industry 2", "industry 3"],
- "company_size": "employee range like '10-200 employees' or '50-500 employees'",
+ "company_size": "employee range like '10-200 employees'",
"titles": ["decision maker title 1", "title 2", "title 3", "title 4"],
- "geography": ["country or region 1"],
- "buyer_persona": "A brief description of the ideal buyer — their role, challenges, and goals",
- "pain_points": ["pain point 1", "pain point 2", "pain point 3"],
- "buying_triggers": ["trigger event 1", "trigger event 2", "trigger event 3"]
-}}"""
+ "geography": ["country or region 1", "region 2"],
+ "buyer_persona": "A detailed description of the ideal buyer — their role, daily challenges, goals, and what success looks like for them",
+ "pain_points": ["specific pain point 1", "pain point 2", "pain point 3", "pain point 4"],
+ "buying_triggers": ["trigger event that makes them ready to buy 1", "trigger 2", "trigger 3"],
+ "disqualifiers": ["signals that someone is NOT a good fit 1", "disqualifier 2"]
+}}
+
+Be specific — don't guess generically. Base this on what the website actually says about their customers."""
result = await self.brain.think_json(prompt, session_id="harvey-trainer")
return result if isinstance(result, dict) else {
@@ -272,36 +471,77 @@ Determine who would buy this product and return as JSON:
"geography": ["United States"],
}
- async def _generate_objections(self, product_info: dict) -> dict:
- """Generate objection handling responses based on the product."""
- prompt = f"""You are a sales expert. Based on this product, generate common objections
-and ideal responses.
+ async def _extract_competitive_intel(self, product_info: dict) -> dict:
+ """Extract competitive positioning and differentiators."""
+ all_content = self._get_all_content(max_chars=40000)
+
+ prompt = f"""Analyze this website for competitive intelligence.
+The product is: {product_info.get('product_name', '')}
+Known competitors: {', '.join(product_info.get('competitors', []))}
+
+WEBSITE CONTENT:
+{all_content}
+
+Look for comparison pages, "why us" messaging, feature differentiation,
+and any mentions of competitors or alternatives.
+
+Return as JSON:
+{{
+ "competitors": [
+ {{
+ "name": "Competitor Name",
+ "how_we_win": "What makes our product better than this competitor",
+ "their_weakness": "Known limitations or complaints about this competitor",
+ "migration_angle": "Why someone would switch from them to us"
+ }}
+ ],
+ "positioning": "How this product positions itself in the market — one paragraph",
+ "unique_strengths": ["strength that no competitor has 1", "strength 2"],
+ "common_alternatives": ["what prospects do instead of buying this — DIY, spreadsheets, etc."]
+}}
+
+If the website doesn't mention specific competitors, infer the most likely ones based on the product category."""
+
+ result = await self.brain.think_json(prompt, session_id="harvey-trainer")
+ return result if isinstance(result, dict) else {"competitors": []}
+
+ async def _generate_objections(self, product_info: dict, competitive_intel: dict) -> dict:
+ """Generate objection handling responses based on product + competitive intel."""
+ competitors = competitive_intel.get("competitors", [])
+ competitor_names = [c.get("name", "") for c in competitors if isinstance(c, dict)]
+
+ prompt = f"""You are a world-class B2B sales trainer. Based on this product and its competitive
+landscape, generate the most common objections a prospect would raise, with killer responses.
Product: {product_info.get('product_name', '')}
Description: {product_info.get('product_description', '')}
Pricing: {product_info.get('pricing', '')}
-Competitors: {', '.join(product_info.get('competitors', []))}
+Key Benefits: {', '.join(product_info.get('key_benefits', []))}
+Competitors: {', '.join(competitor_names)}
+Differentiators: {', '.join(product_info.get('differentiators', []))}
-Generate 6-8 common objections a prospect would raise, with concise responses (1-2 sentences).
-Return as a JSON object where keys are the objection and values are the response.
+Generate 8-12 objections covering:
+- Price/budget concerns
+- Existing solutions / competitor loyalty
+- Timing / priority
+- Implementation / switching costs
+- Trust / credibility
+- Need / relevance
-Example:
-{{
- "too expensive": "Most clients see ROI within 30 days. We can walk through the numbers together.",
- "we already use X": "Makes sense. The teams that switched from X tell us the biggest difference is..."
-}}"""
+Each response should be 1-2 sentences, consultative (not defensive), and end with
+a question or redirect that keeps the conversation going.
+
+Return as a JSON object where keys are the objection and values are the response."""
result = await self.brain.think_json(prompt, session_id="harvey-trainer")
return result if isinstance(result, dict) else {}
- async def _generate_custom_prompts(self, product_info: dict, icp_info: dict):
- """Generate product-specific prompt enhancements."""
+ async def _generate_product_knowledge(
+ self, product_info: dict, icp_info: dict, competitive_intel: dict
+ ):
+ """Generate comprehensive product knowledge skill file."""
product_name = product_info.get("product_name", "the product")
- social_proof = product_info.get("social_proof", [])
- pain_points = icp_info.get("pain_points", [])
- buying_triggers = icp_info.get("buying_triggers", [])
- # Create a product knowledge file in skills/
knowledge = f"""# Product Knowledge: {product_name}
## What We Sell
@@ -316,24 +556,40 @@ Example:
## Features
{chr(10).join('- ' + f for f in product_info.get('features', []))}
+## Use Cases
+{chr(10).join('- ' + u for u in product_info.get('use_cases', []))}
+
## Pricing
{product_info.get('pricing', 'Contact for pricing')}
## Target Audience
{product_info.get('target_audience', '')}
-## Competitors
-{chr(10).join('- ' + c for c in product_info.get('competitors', []))}
+## Ideal Buyer Persona
+{icp_info.get('buyer_persona', '')}
+
+## Integrations
+{chr(10).join('- ' + i for i in product_info.get('integrations', []))}
+
+## What Makes Us Different
+{chr(10).join('- ' + d for d in product_info.get('differentiators', []))}
## Social Proof & Case Studies
-{chr(10).join('- ' + s for s in social_proof)}
+{chr(10).join('- ' + s for s in product_info.get('social_proof', []))}
## Prospect Pain Points
-{chr(10).join('- ' + p for p in pain_points)}
+{chr(10).join('- ' + p for p in icp_info.get('pain_points', []))}
## Buying Triggers
When these events happen, prospects are most likely to buy:
-{chr(10).join('- ' + t for t in buying_triggers)}
+{chr(10).join('- ' + t for t in icp_info.get('buying_triggers', []))}
+
+## Disqualifiers
+Do NOT pursue prospects who:
+{chr(10).join('- ' + d for d in icp_info.get('disqualifiers', []))}
+
+## Market Positioning
+{competitive_intel.get('positioning', '')}
## Tone & Voice
{product_info.get('tone', 'professional, consultative, confident')}
@@ -341,8 +597,55 @@ When these events happen, prospects are most likely to buy:
skills_path = Path(__file__).parent.parent / "skills" / "product_knowledge.md"
skills_path.write_text(knowledge)
- print(f" Generated product knowledge file: {skills_path}")
- logger.info(f"Product knowledge written to {skills_path}")
+ print(f" Generated: skills/product_knowledge.md")
+
+ async def _generate_battle_cards(self, product_info: dict, competitive_intel: dict):
+ """Generate competitive battle cards as a skill file."""
+ competitors = competitive_intel.get("competitors", [])
+ if not competitors:
+ return
+
+ product_name = product_info.get("product_name", "Our Product")
+
+ cards = f"""# Competitive Battle Cards: {product_name}
+
+Use these when a prospect mentions a competitor or asks "how are you different?"
+
+## Market Positioning
+{competitive_intel.get('positioning', '')}
+
+## Our Unique Strengths
+{chr(10).join('- ' + s for s in competitive_intel.get('unique_strengths', []))}
+
+## Common Alternatives (Non-Competitors)
+What prospects do instead of buying a solution like ours:
+{chr(10).join('- ' + a for a in competitive_intel.get('common_alternatives', []))}
+
+---
+"""
+
+ for comp in competitors:
+ if not isinstance(comp, dict):
+ continue
+ name = comp.get("name", "Unknown")
+ cards += f"""
+## vs. {name}
+
+**How we win:** {comp.get('how_we_win', 'N/A')}
+
+**Their weakness:** {comp.get('their_weakness', 'N/A')}
+
+**Migration angle:** {comp.get('migration_angle', 'N/A')}
+
+**When a prospect says "We use {name}":**
+"That's a solid tool. The teams that've moved to {product_name} from {name} tell us the biggest difference is {comp.get('how_we_win', 'the experience')}. Worth a quick comparison?"
+
+---
+"""
+
+ skills_path = Path(__file__).parent.parent / "skills" / "competitive_intel.md"
+ skills_path.write_text(cards)
+ print(f" Generated: skills/competitive_intel.md")
def _build_config(
self,
@@ -402,10 +705,10 @@ When these events happen, prospects are most likely to buy:
}
-async def run_training(url: str, output: str = "harvey.yaml"):
+async def run_training(url: str, output: str = "harvey.yaml", max_pages: int = 100):
"""Entry point for training."""
trainer = Trainer()
- await trainer.train(url, output)
+ await trainer.train(url, output, max_pages=max_pages)
def main():
@@ -413,14 +716,24 @@ def main():
import sys
if len(sys.argv) < 2:
- print("Usage: python -m harvey.trainer <website-url>")
- print("Example: python -m harvey.trainer https://acmecorp.com")
+ print("Usage: python -m harvey.trainer <website-url> [max-pages]")
+ print("")
+ print("Examples:")
+ print(" python -m harvey.trainer https://acmecorp.com")
+ print(" python -m harvey.trainer https://acmecorp.com 200")
+ print("")
+ print("For deep JS-rendered crawling, add to .env:")
+ print(" CLOUDFLARE_ACCOUNT_ID=your_account_id")
+ print(" CLOUDFLARE_API_TOKEN=your_api_token")
+ print("")
+ print("Without Cloudflare credentials, Harvey will use basic HTTP")
+ print("scraping (no JavaScript rendering, limited page discovery).")
sys.exit(1)
url = sys.argv[1]
- output = sys.argv[2] if len(sys.argv) > 2 else "harvey.yaml"
+ max_pages = int(sys.argv[2]) if len(sys.argv) > 2 else 100
- asyncio.run(run_training(url, output))
+ asyncio.run(run_training(url, max_pages=max_pages))
if __name__ == "__main__":