Autonomous AI sales agent powered by Claude Code or Codex. Finds prospects, writes cold emails, sends campaigns, handles replies, books meetings. No API keys needed — runs on your Claude or Codex subscriptions.
Upgrade trainer with Cloudflare /crawl for deep website crawling
Ethan48215bc2ba1f
diff --git a/.env.example b/.env.exampleindex 5e53c2c..ca2de95 100644--- a/.env.example+++ b/.env.example@@ -5,6 +5,12 @@ INSTANTLY_API_KEY=your_instantly_api_key_hereLINKEDIN_EMAIL=your_linkedin_emailLINKEDIN_PASSWORD=your_linkedin_password+# Cloudflare Browser Rendering (for deep website crawling during training)+# $5/mo paid plan = ~12,000 pages/month. Free plan available with limits.+# Get credentials at: https://dash.cloudflare.com → Workers & Pages → Browser Rendering+CLOUDFLARE_ACCOUNT_ID=your_account_id_here+CLOUDFLARE_API_TOKEN=your_api_token_here+# Hunter.io (optional, free tier = 50 lookups/mo)HUNTER_API_KEY=your_hunter_api_key_herediff --git a/README.md b/README.mdindex 6447e1a..c420067 100644--- a/README.md+++ b/README.md@@ -151,13 +151,7 @@ Point Harvey at your website and it learns everything automatically:python -m harvey.trainer https://yourcompany.com```-Harvey will:-1. Crawl up to 15 key pages (homepage, pricing, features, about, etc.)-2. Extract your product info, benefits, pricing, and value proposition-3. Identify your ideal customer profile (industries, titles, company size)-4. Generate objection handling responses specific to your product-5. Create a `product_knowledge.md` skill file with everything it learned-6. Write a complete `harvey.yaml` config ready to use+Harvey will deep-crawl your entire website, analyze every page, and generate a complete configuration:```============================================================@@ -165,27 +159,74 @@ Harvey will:Learning from: https://yourcompany.com============================================================-[1/5] Crawling website...- Scraped 12 pages.+[1/6] Deep crawling with Cloudflare (up to 100 pages)...+ Crawling... 47/47 pages (34s)+ Done! Crawled 47 pages.-[2/5] Analyzing product information...+[2/6] Analyzing product information...Found: AcmeWidget-[3/5] Identifying ideal customer profile...+[3/6] Identifying ideal customer profile...Target: E-commerce, DTC Brands-[4/5] Generating objection handling playbook...- Prepared 7 objection responses.+[4/6] Analyzing competitive landscape...+ Identified 4 competitors.++[5/6] Generating objection handling playbook...+ Prepared 10 objection responses.-[5/5] Building configuration...+[6/6] Building configuration and product knowledge...+ Generated: skills/product_knowledge.md+ Generated: skills/competitive_intel.md============================================================Training complete!- Config written to: harvey.yaml============================================================++ Product: AcmeWidget+ Company: Acme Corp+ ICP titles: VP Marketing, Head of Growth, E-commerce Director+ Industries: E-commerce, DTC Brands+ Pages crawled: 47++ Files generated:+ - harvey.yaml+ - skills/product_knowledge.md+ - skills/competitive_intel.md+```++#### Deep Crawling with Cloudflare++Harvey uses [Cloudflare's Browser Rendering /crawl API](https://developers.cloudflare.com/browser-rendering/rest-api/crawl-endpoint/) to crawl websites. This means:++- **JavaScript rendering** — works on SPAs, React sites, dynamic content+- **Automatic page discovery** — follows sitemaps and internal links+- **Clean markdown output** — no HTML parsing needed+- **Up to 100,000 pages** — crawl the entire site, not just key pages+- **~$5/month** for ~12,000 pages on Cloudflare's paid Workers plan++Add your Cloudflare credentials to `.env`:+```+CLOUDFLARE_ACCOUNT_ID=your_account_id+CLOUDFLARE_API_TOKEN=your_api_token+```++Without Cloudflare credentials, Harvey falls back to basic HTTP scraping (still works, but no JS rendering and limited discovery).++#### What the Trainer Generates++| File | What's in it |+|------|-------------|+| `harvey.yaml` | Complete config — persona, product, ICP, channels, usage limits |+| `skills/product_knowledge.md` | Everything about your product — features, benefits, use cases, pricing, social proof, pain points, buying triggers |+| `skills/competitive_intel.md` | Battle cards for every competitor — how you win, their weaknesses, migration angles, ready-to-use responses |++You can also crawl more pages for larger sites:+```bash+python -m harvey.trainer https://yourcompany.com 500```-Review the generated `harvey.yaml` and tweak anything that needs adjusting. The trainer gets you 90% of the way there.+Review the generated files and tweak anything that needs adjusting. The trainer gets you 90% of the way there.### 3. Add Credentialsdiff --git a/harvey/brain.py b/harvey/brain.pyindex 2e8632d..f435e49 100644--- a/harvey/brain.py+++ b/harvey/brain.py@@ -125,12 +125,12 @@ class Brain:Returns concatenated skill content that should be injectedinto the agent's prompts as foundational knowledge."""- # product_knowledge is auto-generated by the trainer+ # product_knowledge and competitive_intel are auto-generated by the trainer# and loaded by every agent that needs product contextskill_map = {"scout": ["prospecting_tactics", "lead_qualification", "product_knowledge"],- "writer": ["email_frameworks", "sales_methodology", "product_knowledge"],- "handler": ["objection_handling", "sales_methodology", "product_knowledge"],+ "writer": ["email_frameworks", "sales_methodology", "product_knowledge", "competitive_intel"],+ "handler": ["objection_handling", "sales_methodology", "product_knowledge", "competitive_intel"],"sender": ["email_frameworks", "product_knowledge"],"linkedin": ["linkedin_outreach", "prospecting_tactics", "product_knowledge"],}diff --git a/harvey/trainer.py b/harvey/trainer.pyindex b37bddb..b3c3805 100644--- a/harvey/trainer.py+++ b/harvey/trainer.py@@ -1,128 +1,196 @@-"""Trainer — give Harvey a URL and it learns everything about the product."""+"""Trainer — give Harvey a URL and it learns everything about the product.++Uses Cloudflare Browser Rendering /crawl API for deep site crawling.+Falls back to manual scraping if no Cloudflare credentials are configured.+"""import asyncioimport jsonimport logging+import osimport re+import timefrom pathlib import Pathfrom urllib.parse import urljoin, urlparseimport httpximport yamlfrom bs4 import BeautifulSoup+from dotenv import load_dotenvfrom harvey.brain import Brainfrom harvey.state import StateManagerlogger = logging.getLogger("harvey.trainer")-# Pages most likely to contain useful product/company info-PRIORITY_PATHS = [- "/",- "/about",- "/about-us",- "/pricing",- "/features",- "/product",- "/products",- "/solutions",- "/services",- "/how-it-works",- "/why-us",- "/case-studies",- "/customers",- "/testimonials",- "/faq",- "/contact",- "/team",- "/our-team",+# Fallback: pages to try if Cloudflare crawl is unavailable+FALLBACK_PATHS = [+ "/", "/about", "/about-us", "/pricing", "/features", "/product",+ "/products", "/solutions", "/services", "/how-it-works", "/why-us",+ "/case-studies", "/customers", "/testimonials", "/faq", "/contact",+ "/team", "/our-team", "/blog", "/resources", "/integrations",]-MAX_PAGES = 15-MAX_CONTENT_PER_PAGE = 5000 # chars+# Cloudflare API+CF_API_BASE = "https://api.cloudflare.com/client/v4/accounts"+# How long to poll before giving up (seconds)+CRAWL_TIMEOUT = 600 # 10 minutes+POLL_INTERVAL = 10 # check every 10 seconds-class Trainer:- def __init__(self):- self.state = StateManager()- self.brain = Brain(self.state)- self.scraped_pages: dict[str, str] = {}- async def train(self, url: str, output_path: str = "harvey.yaml"):- """Train Harvey on a product website.+class CloudflareCrawler:+ """Cloudflare Browser Rendering /crawl API client."""++ def __init__(self, account_id: str, api_token: str):+ self.account_id = account_id+ self.api_token = api_token+ self.base_url = f"{CF_API_BASE}/{account_id}/browser-rendering/crawl"+ self.headers = {+ "Authorization": f"Bearer {api_token}",+ "Content-Type": "application/json",+ }- 1. Crawl the site (key pages)- 2. Extract all text content- 3. Ask Claude to analyze and extract product info- 4. Generate a complete harvey.yaml config+ async def crawl(+ self,+ url: str,+ max_pages: int = 100,+ depth: int = 5,+ render: bool = True,+ ) -> dict[str, str]:+ """Crawl a website and return {url: markdown_content} for all pages.++ Uses Cloudflare's /crawl endpoint which:+ - Renders JavaScript (SPAs, dynamic content)+ - Discovers pages via sitemaps + links+ - Returns content as clean markdown+ - Respects robots.txt+ - Can crawl up to 100,000 pages"""- await self.state.init_db()+ async with httpx.AsyncClient(timeout=60, headers=self.headers) as client:+ # 1. Start the crawl job+ payload = {+ "url": url,+ "limit": max_pages,+ "depth": depth,+ "formats": ["markdown"],+ "render": render,+ "source": "all",+ "options": {+ "includeSubdomains": False,+ "includeExternalLinks": False,+ },+ "rejectResourceTypes": ["image", "media", "font", "stylesheet"],+ }- domain = urlparse(url).netloc- base_url = f"{urlparse(url).scheme}://{domain}"+ logger.info(f"Starting Cloudflare crawl: {url} (max {max_pages} pages)")+ resp = await client.post(self.base_url, json=payload)- logger.info(f"Training Harvey on {base_url}...")- print(f"\n{'='*60}")- print(f" Harvey Training Mode")- print(f" Learning from: {base_url}")- print(f"{'='*60}\n")+ if resp.status_code != 200:+ logger.error(f"Cloudflare crawl start failed: {resp.status_code} {resp.text}")+ return {}- # Step 1: Crawl the site- print("[1/5] Crawling website...")- await self._crawl_site(base_url)+ result = resp.json()+ if not result.get("success"):+ logger.error(f"Cloudflare crawl error: {result}")+ return {}- if not self.scraped_pages:- print("ERROR: Could not scrape any pages. Check the URL.")- return+ job_id = result.get("result")+ if not job_id:+ logger.error("No job ID returned from Cloudflare crawl")+ return {}- print(f" Scraped {len(self.scraped_pages)} pages.\n")+ logger.info(f"Crawl job started: {job_id}")- # Step 2: Extract product information- print("[2/5] Analyzing product information...")- product_info = await self._extract_product_info()- if not product_info:- print("ERROR: Could not extract product information.")- return- print(f" Found: {product_info.get('product_name', 'Unknown Product')}\n")+ # 2. Poll for completion+ pages = await self._poll_job(client, job_id)+ return pages- # Step 3: Identify ICP- print("[3/5] Identifying ideal customer profile...")- icp_info = await self._extract_icp()- print(f" Target: {', '.join(icp_info.get('industries', ['Unknown']))}\n")+ async def _poll_job(self, client: httpx.AsyncClient, job_id: str) -> dict[str, str]:+ """Poll a crawl job until complete, then collect all pages."""+ start_time = time.time()+ pages: dict[str, str] = {}- # Step 4: Generate objection responses- print("[4/5] Generating objection handling playbook...")- objections = await self._generate_objections(product_info)- print(f" Prepared {len(objections)} objection responses.\n")+ while time.time() - start_time < CRAWL_TIMEOUT:+ # Check status with minimal data+ resp = await client.get(+ f"{self.base_url}/{job_id}",+ params={"limit": 1},+ )- # Step 5: Generate config- print("[5/5] Building configuration...")- config = self._build_config(product_info, icp_info, objections, domain)+ if resp.status_code != 200:+ logger.error(f"Poll failed: {resp.status_code}")+ break++ data = resp.json().get("result", {})+ status = data.get("status", "unknown")+ total = data.get("total", 0)+ finished = data.get("finished", 0)++ elapsed = int(time.time() - start_time)+ print(f"\r Crawling... {finished}/{total} pages ({elapsed}s)", end="", flush=True)++ if status in ("completed", "cancelled_due_to_timeout", "cancelled_due_to_limits"):+ print() # newline after progress+ if status != "completed":+ logger.warning(f"Crawl ended with status: {status}")+ break++ if status == "errored":+ logger.error("Crawl job errored")+ print()+ break++ await asyncio.sleep(POLL_INTERVAL)++ # 3. Collect all results with pagination+ pages = await self._collect_results(client, job_id)+ return pages++ async def _collect_results(+ self, client: httpx.AsyncClient, job_id: str+ ) -> dict[str, str]:+ """Paginate through all crawl results and collect markdown content."""+ pages: dict[str, str] = {}+ cursor = None++ while True:+ params = {"limit": 50, "status": "completed"}+ if cursor:+ params["cursor"] = cursor++ resp = await client.get(+ f"{self.base_url}/{job_id}",+ params=params,+ )- # Write the config- config_path = Path(output_path)- with open(config_path, "w") as f:- yaml.dump(config, f, default_flow_style=False, sort_keys=False)+ if resp.status_code != 200:+ break- print(f"\n{'='*60}")- print(f" Training complete!")- print(f" Config written to: {config_path}")- print(f"{'='*60}")- print(f"\n Product: {config['product']['name']}")- print(f" Company: {config['persona']['company']}")- print(f" ICP: {', '.join(config['icp']['titles'])}")- print(f" Industries: {', '.join(config['icp']['industries'])}")- print(f"\n Review {config_path} and adjust as needed.")- print(f" Then run: python -m harvey\n")+ data = resp.json().get("result", {})+ records = data.get("records", [])- # Also generate custom prompts based on the product- await self._generate_custom_prompts(product_info, icp_info)+ for record in records:+ url = record.get("url", "")+ markdown = record.get("markdown", "")+ if url and markdown:+ pages[url] = markdown- return config+ # Check for next page+ next_cursor = data.get("cursor")+ if not next_cursor or next_cursor == cursor:+ break+ cursor = next_cursor++ return pages- async def _crawl_site(self, base_url: str):- """Crawl key pages of the website."""- crawled = set()++class FallbackCrawler:+ """Simple HTTP crawler for when Cloudflare is not configured."""++ async def crawl(self, base_url: str, max_pages: int = 20) -> dict[str, str]:+ """Crawl key pages using plain HTTP requests."""+ pages: dict[str, str] = {}async with httpx.AsyncClient(timeout=15,@@ -135,40 +203,35 @@ class Trainer:)},) as client:- # First, try priority paths- for path in PRIORITY_PATHS:- if len(self.scraped_pages) >= MAX_PAGES:+ for path in FALLBACK_PATHS:+ if len(pages) >= max_pages:breakurl = urljoin(base_url, path)- if url in crawled:- continue- crawled.add(url)-content = await self._fetch_page(client, url)if content:- self.scraped_pages[path] = content- logger.debug(f"Scraped {url} ({len(content)} chars)")+ pages[url] = content+ print(f"\r Scraped {len(pages)} pages...", end="", flush=True)- # Then discover linked pages from homepage- if "/" in self.scraped_pages:- discovered = self._find_internal_links(- self.scraped_pages["/"], base_url- )+ # Discover more pages from homepage links+ homepage_url = urljoin(base_url, "/")+ if homepage_url in pages:+ discovered = await self._discover_links(client, base_url)for link_url in discovered:- if len(self.scraped_pages) >= MAX_PAGES:+ if len(pages) >= max_pages:break- if link_url in crawled:+ if link_url in pages:continue- crawled.add(link_url)-- path = urlparse(link_url).pathcontent = await self._fetch_page(client, link_url)if content:- self.scraped_pages[path] = content+ pages[link_url] = content+ print(f"\r Scraped {len(pages)} pages...", end="", flush=True)++ print() # newline after progress+ return pagesasync def _fetch_page(self, client: httpx.AsyncClient, url: str) -> str:- """Fetch a page and extract text content."""+ """Fetch a page and convert to clean text."""try:resp = await client.get(url)if resp.status_code != 200:@@ -180,47 +243,176 @@ class Trainer:soup = BeautifulSoup(resp.text, "html.parser")- # Remove noise- for tag in soup(["script", "style", "nav", "footer", "header", "noscript"]):+ # Remove noise elements+ for tag in soup(["script", "style", "nav", "footer", "noscript", "iframe"]):tag.decompose()text = soup.get_text(separator="\n", strip=True)-- # Clean up whitespacelines = [line.strip() for line in text.splitlines() if line.strip()]- text = "\n".join(lines)-- # Truncate if too long- if len(text) > MAX_CONTENT_PER_PAGE:- text = text[:MAX_CONTENT_PER_PAGE]-- return text+ return "\n".join(lines)except Exception as e:logger.debug(f"Failed to fetch {url}: {e}")return ""- def _find_internal_links(self, html_content: str, base_url: str) -> list[str]:- """Find internal links from page content (re-parse from raw isn't available,- so we use the base_url to filter discovered links)."""- # Since we only have text content at this point, we'd need the raw HTML.- # For simplicity, we rely on PRIORITY_PATHS for discovery.- return []+ async def _discover_links(self, client: httpx.AsyncClient, base_url: str) -> list[str]:+ """Find internal links from the homepage."""+ try:+ resp = await client.get(base_url)+ if resp.status_code != 200:+ return []++ soup = BeautifulSoup(resp.text, "html.parser")+ domain = urlparse(base_url).netloc+ links = []- async def _extract_product_info(self) -> dict:- """Ask Claude to extract product information from scraped content."""- # Combine all page content+ for a in soup.find_all("a", href=True):+ href = a["href"]+ full_url = urljoin(base_url, href)+ parsed = urlparse(full_url)++ # Only internal links, no anchors/mailto/tel+ if parsed.netloc == domain and parsed.scheme in ("http", "https"):+ clean = f"{parsed.scheme}://{parsed.netloc}{parsed.path}"+ if clean not in links:+ links.append(clean)++ return links[:30] # cap discovery++ except Exception:+ return []+++class Trainer:+ def __init__(self):+ self.state = StateManager()+ self.brain = Brain(self.state)+ self.scraped_pages: dict[str, str] = {}++ async def train(+ self,+ url: str,+ output_path: str = "harvey.yaml",+ max_pages: int = 100,+ ):+ """Train Harvey on a product website.++ 1. Deep crawl the site (Cloudflare /crawl or fallback)+ 2. Analyze all content with Claude+ 3. Extract product info, ICP, objections, competitive intel+ 4. Generate complete config + product knowledge skill+ """+ load_dotenv()+ await self.state.init_db()++ parsed = urlparse(url)+ domain = parsed.netloc+ base_url = f"{parsed.scheme}://{domain}"++ logger.info(f"Training Harvey on {base_url}...")+ print(f"\n{'='*60}")+ print(f" Harvey Training Mode")+ print(f" Learning from: {base_url}")+ print(f"{'='*60}\n")++ # Step 1: Deep crawl+ cf_account_id = os.getenv("CLOUDFLARE_ACCOUNT_ID", "")+ cf_api_token = os.getenv("CLOUDFLARE_API_TOKEN", "")++ if cf_account_id and cf_api_token:+ print(f"[1/6] Deep crawling with Cloudflare (up to {max_pages} pages)...")+ crawler = CloudflareCrawler(cf_account_id, cf_api_token)+ self.scraped_pages = await crawler.crawl(+ base_url, max_pages=max_pages, depth=10+ )+ else:+ print("[1/6] Crawling website (add CLOUDFLARE_ACCOUNT_ID and")+ print(" CLOUDFLARE_API_TOKEN to .env for deep JS-rendered crawling)...")+ crawler = FallbackCrawler()+ self.scraped_pages = await crawler.crawl(base_url, max_pages=min(max_pages, 30))++ if not self.scraped_pages:+ print("\nERROR: Could not scrape any pages. Check the URL.")+ return None++ print(f" Done! Crawled {len(self.scraped_pages)} pages.\n")++ # Step 2: Extract product information+ print("[2/6] Analyzing product information...")+ product_info = await self._extract_product_info()+ if not product_info:+ print("ERROR: Could not extract product information.")+ return None+ print(f" Found: {product_info.get('product_name', 'Unknown Product')}\n")++ # Step 3: Identify ICP+ print("[3/6] Identifying ideal customer profile...")+ icp_info = await self._extract_icp()+ print(f" Target: {', '.join(icp_info.get('industries', ['Unknown']))}\n")++ # Step 4: Competitive analysis+ print("[4/6] Analyzing competitive landscape...")+ competitive_intel = await self._extract_competitive_intel(product_info)+ print(f" Identified {len(competitive_intel.get('competitors', []))} competitors.\n")++ # Step 5: Generate objection responses+ print("[5/6] Generating objection handling playbook...")+ objections = await self._generate_objections(product_info, competitive_intel)+ print(f" Prepared {len(objections)} objection responses.\n")++ # Step 6: Generate config + skills+ print("[6/6] Building configuration and product knowledge...")+ config = self._build_config(product_info, icp_info, objections, domain)++ # Write config+ config_path = Path(output_path)+ with open(config_path, "w") as f:+ yaml.dump(config, f, default_flow_style=False, sort_keys=False)++ # Generate product knowledge skill+ await self._generate_product_knowledge(product_info, icp_info, competitive_intel)++ # Generate competitive battle cards skill+ if competitive_intel.get("competitors"):+ await self._generate_battle_cards(product_info, competitive_intel)++ print(f"\n{'='*60}")+ print(f" Training complete!")+ print(f"{'='*60}")+ print(f"\n Product: {config['product']['name']}")+ print(f" Company: {config['persona']['company']}")+ print(f" ICP titles: {', '.join(config['icp']['titles'])}")+ print(f" Industries: {', '.join(config['icp']['industries'])}")+ print(f" Pages crawled: {len(self.scraped_pages)}")+ print(f"\n Files generated:")+ print(f" - {config_path}")+ print(f" - skills/product_knowledge.md")+ if competitive_intel.get("competitors"):+ print(f" - skills/competitive_intel.md")+ print(f"\n Review {config_path} and adjust as needed.")+ print(f" Then run: python -m harvey\n")++ return config++ def _get_all_content(self, max_chars: int = 60000) -> str:+ """Combine all scraped pages into one string for analysis."""all_content = ""- for path, content in self.scraped_pages.items():- all_content += f"\n\n--- PAGE: {path} ---\n{content}"+ for url, content in self.scraped_pages.items():+ path = urlparse(url).path or "/"+ # Truncate individual pages to keep total manageable+ page_content = content[:8000] if len(content) > 8000 else content+ all_content += f"\n\n--- PAGE: {path} ---\n{page_content}"+ if len(all_content) > max_chars:+ break+ return all_content- # Truncate to fit in a reasonable prompt- if len(all_content) > 30000:- all_content = all_content[:30000]+ async def _extract_product_info(self) -> dict:+ """Ask Claude to extract product information from all crawled content."""+ all_content = self._get_all_content()- prompt = f"""Analyze this website content and extract product/company information.+ prompt = f"""Analyze this entire website and extract comprehensive product/company information.-WEBSITE CONTENT:+WEBSITE CONTENT ({len(self.scraped_pages)} pages):{all_content}Extract the following and return as JSON:@@ -228,41 +420,48 @@ Extract the following and return as JSON:"company_name": "the company name","product_name": "the main product or service name","product_description": "2-3 sentence description of what the product does and who it's for",- "pricing": "pricing info if available, otherwise 'Contact for pricing'",+ "pricing": "pricing info if available (tiers, starting price, etc.), otherwise 'Contact for pricing'","key_benefits": ["benefit 1", "benefit 2", "benefit 3", "benefit 4", "benefit 5"],- "features": ["feature 1", "feature 2", "feature 3"],- "target_audience": "who this product is for",+ "features": ["feature 1", "feature 2", "feature 3", "feature 4", "feature 5", "feature 6"],+ "target_audience": "who this product is for — be specific about roles and company types","value_proposition": "the core value proposition in one sentence",- "competitors": ["competitor 1", "competitor 2"],- "social_proof": ["any case studies, testimonials, or notable customers mentioned"],+ "differentiators": ["what makes this different from alternatives — point 1", "point 2"],+ "competitors": ["competitor 1", "competitor 2", "competitor 3"],+ "social_proof": ["any case studies, testimonials, notable customers, or stats mentioned"],+ "integrations": ["tools and platforms this integrates with"],+ "use_cases": ["specific use case 1", "use case 2", "use case 3"],"tone": "describe the brand's communication tone in 3-4 words"-}}"""+}}++Be thorough — you have access to the full website. Extract everything relevant."""result = await self.brain.think_json(prompt, session_id="harvey-trainer")return result if isinstance(result, dict) else {}async def _extract_icp(self) -> dict:- """Ask Claude to determine the ideal customer profile."""- all_content = "\n".join(- f"[{path}]: {content[:2000]}"- for path, content in list(self.scraped_pages.items())[:8]- )+ """Ask Claude to determine the ideal customer profile from all content."""+ all_content = self._get_all_content(max_chars=40000)- prompt = f"""Based on this website content, determine the Ideal Customer Profile (ICP).+ prompt = f"""Based on this entire website, determine the Ideal Customer Profile (ICP).+Look at case studies, testimonials, pricing tiers, feature descriptions, and target messaging+to understand exactly who this product is built for.-WEBSITE CONTENT:+WEBSITE CONTENT ({len(self.scraped_pages)} pages):{all_content}-Determine who would buy this product and return as JSON:+Return as JSON:{{"industries": ["industry 1", "industry 2", "industry 3"],- "company_size": "employee range like '10-200 employees' or '50-500 employees'",+ "company_size": "employee range like '10-200 employees'","titles": ["decision maker title 1", "title 2", "title 3", "title 4"],- "geography": ["country or region 1"],- "buyer_persona": "A brief description of the ideal buyer — their role, challenges, and goals",- "pain_points": ["pain point 1", "pain point 2", "pain point 3"],- "buying_triggers": ["trigger event 1", "trigger event 2", "trigger event 3"]-}}"""+ "geography": ["country or region 1", "region 2"],+ "buyer_persona": "A detailed description of the ideal buyer — their role, daily challenges, goals, and what success looks like for them",+ "pain_points": ["specific pain point 1", "pain point 2", "pain point 3", "pain point 4"],+ "buying_triggers": ["trigger event that makes them ready to buy 1", "trigger 2", "trigger 3"],+ "disqualifiers": ["signals that someone is NOT a good fit 1", "disqualifier 2"]+}}++Be specific — don't guess generically. Base this on what the website actually says about their customers."""result = await self.brain.think_json(prompt, session_id="harvey-trainer")return result if isinstance(result, dict) else {@@ -272,36 +471,77 @@ Determine who would buy this product and return as JSON:"geography": ["United States"],}- async def _generate_objections(self, product_info: dict) -> dict:- """Generate objection handling responses based on the product."""- prompt = f"""You are a sales expert. Based on this product, generate common objections-and ideal responses.+ async def _extract_competitive_intel(self, product_info: dict) -> dict:+ """Extract competitive positioning and differentiators."""+ all_content = self._get_all_content(max_chars=40000)++ prompt = f"""Analyze this website for competitive intelligence.+The product is: {product_info.get('product_name', '')}+Known competitors: {', '.join(product_info.get('competitors', []))}++WEBSITE CONTENT:+{all_content}++Look for comparison pages, "why us" messaging, feature differentiation,+and any mentions of competitors or alternatives.++Return as JSON:+{{+ "competitors": [+ {{+ "name": "Competitor Name",+ "how_we_win": "What makes our product better than this competitor",+ "their_weakness": "Known limitations or complaints about this competitor",+ "migration_angle": "Why someone would switch from them to us"+ }}+ ],+ "positioning": "How this product positions itself in the market — one paragraph",+ "unique_strengths": ["strength that no competitor has 1", "strength 2"],+ "common_alternatives": ["what prospects do instead of buying this — DIY, spreadsheets, etc."]+}}++If the website doesn't mention specific competitors, infer the most likely ones based on the product category."""++ result = await self.brain.think_json(prompt, session_id="harvey-trainer")+ return result if isinstance(result, dict) else {"competitors": []}++ async def _generate_objections(self, product_info: dict, competitive_intel: dict) -> dict:+ """Generate objection handling responses based on product + competitive intel."""+ competitors = competitive_intel.get("competitors", [])+ competitor_names = [c.get("name", "") for c in competitors if isinstance(c, dict)]++ prompt = f"""You are a world-class B2B sales trainer. Based on this product and its competitive+landscape, generate the most common objections a prospect would raise, with killer responses.Product: {product_info.get('product_name', '')}Description: {product_info.get('product_description', '')}Pricing: {product_info.get('pricing', '')}-Competitors: {', '.join(product_info.get('competitors', []))}+Key Benefits: {', '.join(product_info.get('key_benefits', []))}+Competitors: {', '.join(competitor_names)}+Differentiators: {', '.join(product_info.get('differentiators', []))}-Generate 6-8 common objections a prospect would raise, with concise responses (1-2 sentences).-Return as a JSON object where keys are the objection and values are the response.+Generate 8-12 objections covering:+- Price/budget concerns+- Existing solutions / competitor loyalty+- Timing / priority+- Implementation / switching costs+- Trust / credibility+- Need / relevance-Example:-{{- "too expensive": "Most clients see ROI within 30 days. We can walk through the numbers together.",- "we already use X": "Makes sense. The teams that switched from X tell us the biggest difference is..."-}}"""+Each response should be 1-2 sentences, consultative (not defensive), and end with+a question or redirect that keeps the conversation going.++Return as a JSON object where keys are the objection and values are the response."""result = await self.brain.think_json(prompt, session_id="harvey-trainer")return result if isinstance(result, dict) else {}- async def _generate_custom_prompts(self, product_info: dict, icp_info: dict):- """Generate product-specific prompt enhancements."""+ async def _generate_product_knowledge(+ self, product_info: dict, icp_info: dict, competitive_intel: dict+ ):+ """Generate comprehensive product knowledge skill file."""product_name = product_info.get("product_name", "the product")- social_proof = product_info.get("social_proof", [])- pain_points = icp_info.get("pain_points", [])- buying_triggers = icp_info.get("buying_triggers", [])- # Create a product knowledge file in skills/knowledge = f"""# Product Knowledge: {product_name}## What We Sell@@ -316,24 +556,40 @@ Example:## Features{chr(10).join('- ' + f for f in product_info.get('features', []))}+## Use Cases+{chr(10).join('- ' + u for u in product_info.get('use_cases', []))}+## Pricing{product_info.get('pricing', 'Contact for pricing')}## Target Audience{product_info.get('target_audience', '')}-## Competitors-{chr(10).join('- ' + c for c in product_info.get('competitors', []))}+## Ideal Buyer Persona+{icp_info.get('buyer_persona', '')}++## Integrations+{chr(10).join('- ' + i for i in product_info.get('integrations', []))}++## What Makes Us Different+{chr(10).join('- ' + d for d in product_info.get('differentiators', []))}## Social Proof & Case Studies-{chr(10).join('- ' + s for s in social_proof)}+{chr(10).join('- ' + s for s in product_info.get('social_proof', []))}## Prospect Pain Points-{chr(10).join('- ' + p for p in pain_points)}+{chr(10).join('- ' + p for p in icp_info.get('pain_points', []))}## Buying TriggersWhen these events happen, prospects are most likely to buy:-{chr(10).join('- ' + t for t in buying_triggers)}+{chr(10).join('- ' + t for t in icp_info.get('buying_triggers', []))}++## Disqualifiers+Do NOT pursue prospects who:+{chr(10).join('- ' + d for d in icp_info.get('disqualifiers', []))}++## Market Positioning+{competitive_intel.get('positioning', '')}## Tone & Voice{product_info.get('tone', 'professional, consultative, confident')}@@ -341,8 +597,55 @@ When these events happen, prospects are most likely to buy:skills_path = Path(__file__).parent.parent / "skills" / "product_knowledge.md"skills_path.write_text(knowledge)- print(f" Generated product knowledge file: {skills_path}")- logger.info(f"Product knowledge written to {skills_path}")+ print(f" Generated: skills/product_knowledge.md")++ async def _generate_battle_cards(self, product_info: dict, competitive_intel: dict):+ """Generate competitive battle cards as a skill file."""+ competitors = competitive_intel.get("competitors", [])+ if not competitors:+ return++ product_name = product_info.get("product_name", "Our Product")++ cards = f"""# Competitive Battle Cards: {product_name}++Use these when a prospect mentions a competitor or asks "how are you different?"++## Market Positioning+{competitive_intel.get('positioning', '')}++## Our Unique Strengths+{chr(10).join('- ' + s for s in competitive_intel.get('unique_strengths', []))}++## Common Alternatives (Non-Competitors)+What prospects do instead of buying a solution like ours:+{chr(10).join('- ' + a for a in competitive_intel.get('common_alternatives', []))}++---+"""++ for comp in competitors:+ if not isinstance(comp, dict):+ continue+ name = comp.get("name", "Unknown")+ cards += f"""+## vs. {name}++**How we win:** {comp.get('how_we_win', 'N/A')}++**Their weakness:** {comp.get('their_weakness', 'N/A')}++**Migration angle:** {comp.get('migration_angle', 'N/A')}++**When a prospect says "We use {name}":**+"That's a solid tool. The teams that've moved to {product_name} from {name} tell us the biggest difference is {comp.get('how_we_win', 'the experience')}. Worth a quick comparison?"++---+"""++ skills_path = Path(__file__).parent.parent / "skills" / "competitive_intel.md"+ skills_path.write_text(cards)+ print(f" Generated: skills/competitive_intel.md")def _build_config(self,@@ -402,10 +705,10 @@ When these events happen, prospects are most likely to buy:}-async def run_training(url: str, output: str = "harvey.yaml"):+async def run_training(url: str, output: str = "harvey.yaml", max_pages: int = 100):"""Entry point for training."""trainer = Trainer()- await trainer.train(url, output)+ await trainer.train(url, output, max_pages=max_pages)def main():@@ -413,14 +716,24 @@ def main():import sysif len(sys.argv) < 2:- print("Usage: python -m harvey.trainer <website-url>")- print("Example: python -m harvey.trainer https://acmecorp.com")+ print("Usage: python -m harvey.trainer <website-url> [max-pages]")+ print("")+ print("Examples:")+ print(" python -m harvey.trainer https://acmecorp.com")+ print(" python -m harvey.trainer https://acmecorp.com 200")+ print("")+ print("For deep JS-rendered crawling, add to .env:")+ print(" CLOUDFLARE_ACCOUNT_ID=your_account_id")+ print(" CLOUDFLARE_API_TOKEN=your_api_token")+ print("")+ print("Without Cloudflare credentials, Harvey will use basic HTTP")+ print("scraping (no JavaScript rendering, limited page discovery).")sys.exit(1)url = sys.argv[1]- output = sys.argv[2] if len(sys.argv) > 2 else "harvey.yaml"+ max_pages = int(sys.argv[2]) if len(sys.argv) > 2 else 100- asyncio.run(run_training(url, output))+ asyncio.run(run_training(url, max_pages=max_pages))if __name__ == "__main__":