Autonomous AI sales agent powered by Claude Code or Codex. Finds prospects, writes cold emails, sends campaigns, handles replies, books meetings. No API keys needed — runs on your Claude or Codex subscriptions.
DISCOVER: a provider menu, a cost estimate, and a free default
Ethan6ca7e551f89d
diff --git a/.env.example b/.env.exampleindex cd00ed3..3c5b462 100644--- a/.env.example+++ b/.env.example@@ -82,11 +82,29 @@ ZEROBOUNCE_API_KEY=HUNTER_API_KEY=# -------------------------------------------------------------# Serper.dev — Google Search API for prospecting [OPTIONAL, recommended]+# DISCOVERY — where Harvey finds businesses. [ALL OPTIONAL]+# The default source (OpenStreetMap) needs no account at all:+# harvey discover+# See every option, with real prices, via:+# harvey discover --providers (or the dashboard's Discover tab)+# ------------------------------------------------------------++# DataForSEO — cheapest local-business and SERP data anywhere.+# Where to get it: https://app.dataforseo.com/api-access+# NOTE: the API password is auto-generated there. It is NOT your account+# password. $1 free credit on signup; $50 minimum deposit to go live.+# Cost: $0.372 per 1,000 businesses; $0.0018 per depth-30 search.+DATAFORSEO_LOGIN=+DATAFORSEO_PASSWORD=+# Set to any value to route every call to DataForSEO's free sandbox. Same+# response shape, dummy data, never charged — the right way to try it.+DATAFORSEO_SANDBOX=++# Serper.dev — Google Search API, used for discovery and for Scout's searches.# Where to get it: https://serper.dev → sign up → Dashboard → API Key-# Cost: ~$5 for 2,500 searches (no subscription). The most reliable search-# backend; without it Harvey falls back to DuckDuckGo/Bing scraping,-# which can be rate-limited.+# Cost: 2,500 queries free with no card, then a $50 prepaid pack (credits+# expire after 6 months). Without it Harvey falls back to DuckDuckGo/Bing+# scraping, which is heavily rate-limited.SERPER_API_KEY=# ------------------------------------------------------------diff --git a/CLAUDE.md b/CLAUDE.mdindex 2c9c7d6..d6258ee 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -86,15 +86,19 @@ REOON_API_KEY= # 600 free/mo — best free tierZEROBOUNCE_API_KEY= # 100 free/mo — best for M365/Workspace catch-allsHUNTER_API_KEY= # 50 free/mo + pattern lookup+# --- Discovery (optional — the free OpenStreetMap source needs no key) ---+DATAFORSEO_LOGIN= DATAFORSEO_PASSWORD= # cheapest local-business + SERP data+DATAFORSEO_SANDBOX= # any value routes to the free sandbox+# --- Optional ---LINKEDIN_EMAIL= LINKEDIN_PASSWORD= # LinkedIn prospectingCLOUDFLARE_ACCOUNT_ID= CLOUDFLARE_API_TOKEN= # deep JS crawling for training-SERPER_API_KEY= # reliable web search ($5/mo at serper.dev)+SERPER_API_KEY= # web search + discovery (2,500 free, then a $50 prepaid pack)```**Recommended provider — Gmail:** for <50 cold emails/day, a real Google Workspace mailbox on a *dedicated secondary domain* (never the main one) is the most deliverable, cheapest (~$7/mo) option. Set `channels.email.provider: gmail` in harvey.yaml, put the OAuth client id/secret in `.env`, then run `harvey gmail auth` (one-time browser login). SMTP works with any mailbox (AgentMail, Fastmail). Instantly still works as a legacy option.-**Email verification matters:** Harvey learns each company's email *pattern* and verifies ONE candidate rather than guessing (raw SMTP probing no longer works against Google Workspace / Microsoft 365). Without a verifier key, found emails are marked `guess` and are **never sent**.+**Email verification matters:** Harvey learns each company's email *pattern* and verifies ONE candidate rather than guessing (raw SMTP probing is not viable from a locally-running agent — outbound port 25 is usually blocked, and Google Workspace / Microsoft 365 accept everything from an unknown IP). Without a verifier key, found emails are marked `guess` and are **never sent**.**Approval by default:** with a native provider, every outgoing email waits in the **Outbox** for your approval (dashboard Outbox tab, or `harvey outbox`). Once you trust the output, set `channels.email.require_approval: false` in harvey.yaml for full autopilot.@@ -159,7 +163,37 @@ Once signals are confirmed, the **cohort builder** at the bottom of the Signalstab turns them into a target list: pick what a good prospect must have (and whatdisqualifies them) and it counts the matches live.-### Step 5: Behavior Settings+### Step 5: Find Businesses++Discovery is the only stage that spends money, so it always estimates first.++```bash+harvey discover --providers # the menu, with real prices+harvey discover --estimate # projected spend, then exits+harvey discover # free OpenStreetMap source+harvey discover --provider dataforseo_listings --max-spend 2.00+```++The dashboard's **Discover** tab is the same thing with the tradeoffs laid out+side by side: what each source does, what it costs, its free tier, and which+`.env` keys it needs. Nothing is called until the user presses Run.++| Source | Cost | Free tier | Best for |+|---|---|---|---|+| **OpenStreetMap** (default) | free | unlimited | Trying the whole pipeline with no account. Thin coverage — under 2,000 roofers in the entire US — so it's a trial source, not a complete list. |+| **DataForSEO Business Listings** | $0.372/1k businesses | $1 credit | Local trades, clinics, contractors. Phone + domain + rating + whether the listing is claimed, and it can filter for businesses with **no website at all**. |+| **DataForSEO SERP** | $0.0018 a search at depth 30 | $1 credit + free sandbox | Rank as the buying signal. Positions 11-30 are the sweet spot. |+| **Serper** | ~$0.30-$1.00/1k | 2,500 free, no card | The easiest paid one to try. |++Cities are geocoded automatically (cached forever, one lookup per city), or set+`icp.geo_coordinates` in harvey.yaml to control the radius:+`"Denver, CO": "39.7392,-104.9903,50"`.++**Depth 20-30, not 100.** Google removed 100-results-per-page in September 2025,+so depth 100 is now billed as ten pages at nearly every provider. Below rank 30+it is mostly directories anyway.++### Step 6: Behavior SettingsThese go in `harvey.yaml` under `usage:`. Use sensible defaults unless they want to customize:- `max_daily_claude_percent`: 80 (how much of daily Claude quota to use)@@ -167,7 +201,7 @@ These go in `harvey.yaml` under `usage:`. Use sensible defaults unless they want- `quiet_hours`: 22:00-07:00 in their timezone- `max_daily_sends`: 50 (email send limit)-### Step 6: Start Harvey+### Step 7: Start Harvey```bashsource .venv/bin/activate && harvey run@@ -204,6 +238,7 @@ Users will come back with questions and tasks. Common ones:- **Skills** (`skills/`): Markdown knowledge files injected into agent prompts- **Signals** (`signals.py`, `state.py`): every fact Harvey learns is an OBSERVATION — a row in `observations`, never a column. A new signal needs no migration, re-observing over time is a free time series, and confidence + provenance travel with the fact. The vocabulary in `signal_codes` is governed (a trigger rejects any code not in it) and gated: only `status = 'confirmed'` signals are collected, and only a human sets that. `state.cohort(require, exclude)` does the set intersection in SQL.- **Dashboard** (`dashboard.py` + `harvey/web/`): FastAPI JSON API plus plain HTML/CSS/JS served from disk — no build step. Edit `harvey/web/app.css` or `app.js` and reload the page.+- **DISCOVER** (`collectors/discover.py`): the only stage that spends money, so cost discipline lives in the collector, not the database — every run estimates first, checks a spend cap between batches, reads a kill switch, and flushes observations per batch. Providers are adapters behind one interface (`DiscoveryProvider`); adding one is a small class. Junk filtering matches *patterns*, not a literal blocklist. Entity resolution is the normalised domain, falling back to the provider's `external_id` for businesses with no website.### Sub-Agents- **Scout**: Python does all web searching (DuckDuckGo → Bing → Google → Serper API) and email resolution (pattern-first: cache → scraped mailto → Hunter domain search → default, then verify one candidate via Reoon/ZeroBounce/Hunter/SMTP; catch-alls flagged `risky`). Claude only scores/personalizes found data. Scout also collects **buying signals**: tech stack detected on each company's site (HubSpot, Shopify, Intercom, ~35 tools — zero extra requests) and hiring signals from careers pages. With `pip install python-jobspy` (optional), a job-board strategy discovers companies actively hiring for roles in `icp.hiring_signals` (falls back to `icp.titles`) — the strongest in-market signal. Signals land in prospects' personalization notes and boost their score.@@ -233,6 +268,7 @@ harvey gmail auth # One-time Gmail OAuth (when provider: gmail)harvey gmail test # Verify the Gmail connectionharvey outbox # Review queued emails; --approve <id> / --approve-all / --reject <id>harvey signals # The signal vocabulary; --confirm / --reject CODES (or 'free' / 'all')+harvey discover # Find businesses; --providers / --estimate / --provider <key>harvey sending pause|resume # Kill switch for all outbound```diff --git a/harvey/cli.py b/harvey/cli.pyindex 20dfd3a..01c342c 100644--- a/harvey/cli.py+++ b/harvey/cli.py@@ -320,6 +320,79 @@ def cmd_outbox(args):asyncio.run(_outbox())+def cmd_discover(args):+ """Find businesses. The only stage that spends money, so it estimates first."""+ from harvey.state import StateManager+ from harvey.config import load_config+ from harvey.collectors.discover import (+ PROVIDERS, build_queries, provider_menu, run_discovery,+ )++ async def _discover():+ if args.providers:+ print("\n Discovery providers")+ print(" " + "=" * 68)+ for m in provider_menu():+ mark = "ready" if m["configured"] else "needs setup"+ print(f"\n {m['label']} [{m['key']}] ({mark})")+ print(f" {m['blurb']}")+ print(f" Cost: {m['cost_note']}")+ print(f" Free: {m['free_tier']}")+ if m["env_keys"]:+ print(f" Needs: {', '.join(m['env_keys'])} in .env — {m['signup_url']}")+ if m["caveat"]:+ print(f" Note: {m['caveat']}")+ print("\n Run one with: harvey discover --provider <key> --estimate\n")+ return++ if args.provider not in PROVIDERS:+ print(f"\n Unknown provider {args.provider!r}. "+ f"See: harvey discover --providers\n")+ return++ config = load_config()+ state = StateManager()+ await state.init_db()++ # Semicolons, not commas: "Denver, CO" is one city, not two.+ cities = [c.strip() for c in args.city.split(";") if c.strip()] or None+ queries = build_queries(config, cities=cities,+ depth=args.depth, limit=args.limit)++ report = await run_discovery(state, config, args.provider, queries,+ max_spend=args.max_spend, dry_run=True)+ print(f"\n {len(queries)} queries via {PROVIDERS[args.provider].label}")+ for q in queries[:8]:+ print(f" - {q.keyword()}" + ("" if q.coordinate else+ " (no coordinates — add icp.geo_coordinates for radius search)"))+ if len(queries) > 8:+ print(f" ... and {len(queries) - 8} more")+ print(f"\n Estimated cost: ${report.estimated_cost:.4f}"+ f" (cap: ${args.max_spend:.2f})")++ if args.estimate:+ print("\n Estimate only. Re-run without --estimate to collect.\n")+ return++ print("\n Running...\n")+ report = await run_discovery(state, config, args.provider, queries,+ max_spend=args.max_spend)++ r = report+ print(f" Found {r.found} results across {r.queries} queries")+ print(f" {r.new_companies} new companies, {r.known_companies} already known")+ print(f" {r.junk} filtered out as directories/aggregators")+ print(f" {r.observations} observations recorded")+ print(f" actual cost: ${r.actual_cost:.4f}")+ if r.stopped:+ print(f"\n Stopped early: {r.stopped}")+ for err in r.errors[:5]:+ print(f" ! {err}")+ print("\n Next: harvey dashboard → Signals → cohort builder\n")++ asyncio.run(_discover())++def cmd_signals(args):"""Review the signal vocabulary — Harvey proposes, you confirm.@@ -476,6 +549,24 @@ def main():sub.set_defaults(func=cmd_outbox)# harvey sending pause|resume|status+ sub = subparsers.add_parser("discover", help="Find businesses matching your ICP")+ sub.add_argument("--provider", default="osm",+ help="Discovery provider (default: osm — free, no account)")+ sub.add_argument("--providers", action="store_true",+ help="List every provider with cost and setup, then exit")+ sub.add_argument("--estimate", action="store_true",+ help="Print projected spend and exit without calling anything")+ sub.add_argument("--city", default="",+ help='Semicolon-separated cities, e.g. "Denver, CO;Dallas, TX" '+ '(default: icp.geography)')+ sub.add_argument("--depth", type=int, default=30,+ help="SERP depth (default: 30 — depth 100 costs 10x since Sept 2025)")+ sub.add_argument("--limit", type=int, default=100,+ help="Max records per query (default: 100)")+ sub.add_argument("--max-spend", type=float, default=1.0,+ help="Hard cap in dollars (default: 1.00)")+ sub.set_defaults(func=cmd_discover)+sub = subparsers.add_parser("signals", help="Review/confirm which signals Harvey prospects against")sub.add_argument("--confirm", metavar="CODES", default="",diff --git a/harvey/collectors/discover.py b/harvey/collectors/discover.pynew file mode 100644index 0000000..3531e8f--- /dev/null+++ b/harvey/collectors/discover.py@@ -0,0 +1,864 @@+"""DISCOVER — who exists, and how visible they are.++The only stage that costs money, so it is the one with a provider menu, a+mandatory cost estimate, and a budget check inside the loop.++Two shapes of discovery, and the difference matters:++* **Listings** ask "which businesses of this type are in this place?" and come+ back with a phone, a category, a rating and — crucially — whether there is a+ website at all. For local SMB verticals this is the primary path.+* **SERP** asks "who ranks for this search?" and comes back with a rank+ position. You are buying the *signal*, not the list: positions 11-30 are+ businesses visible enough to be trying and not winning, which is the most+ useful prospecting signal there is.++Cost discipline lives here, not in the database. A table can only record spend+after the fact; only the collector can decline to spend it. Every run+estimates first, checks a budget between batches, reads the kill switch, and+flushes observations per batch — a long run that dies must not lose what it+already learned.+"""++import asyncio+import base64+import logging+import re+from dataclasses import dataclass, field+from urllib.parse import urlparse++import httpx++from harvey.models import Company++logger = logging.getLogger("harvey.collectors.discover")++TIMEOUT = httpx.Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)+++# ── Junk filtering ────────────────────────────────────────────────────+# Discovery fills a database with directories, hospital systems, media and+# lead-gen middlemen unless you filter hard. Match PATTERNS, not a literal+# list: a literal list is exactly how `whatclinic.com` and `ufhealth.org` get+# through on the first run.++# Anchored on a dot boundary, not the string start: SERP results arrive as+# `en.wikipedia.org` and `m.facebook.com` at least as often as bare domains,+# and `^(www\.)?` catches neither.+JUNK_DOMAIN_PATTERNS = tuple(re.compile(p, re.I) for p in (+ # directories & marketplaces+ r"(^|\.)(yelp|yellowpages|superpages|manta|bbb|angi|angieslist|homeadvisor)\.",+ r"(^|\.)(thumbtack|houzz|porch|buildzoom|networx|modernize|nextdoor)\.",+ r"(^|\.)(facebook|instagram|twitter|x|linkedin|youtube|tiktok|pinterest)\.",+ r"(^|\.)(indeed|glassdoor|ziprecruiter|craigslist|zillow|realtor)\.",+ r"(^|\.)(mapquest|foursquare|tripadvisor|trustpilot|birdeye|expertise)\.",+ r"(directory|listings?|find-?a-|top\d+|best-?of)\.",+ # media & reference+ r"(^|\.)(wikipedia|reddit|quora|medium|substack|forbes|inc|entrepreneur)\.",+ r"(news|magazine|journal|gazette|tribune|herald)\.(com|org|net)$",+ # institutional+ r"\.(edu|gov|mil)$",+ r"(hospital|healthsystem|medicalcenter)\.",+ r"health\.org$",+ # infrastructure that is never a prospect+ r"(^|\.)(google|bing|yahoo|duckduckgo|amazon|apple|microsoft)\.",+ r"(wordpress|wix|squarespace|shopify|godaddy|weebly|webflow)\.com$",+))++# Names that mean the row is a category page or an aggregator, not a business.+JUNK_NAME_PATTERNS = tuple(re.compile(p, re.I) for p in (+ r"^\s*(top|best|\d+)\s+\d*\s*(best\s+)?\w+.*\b(in|near|of)\b",+ r"\b(directory|listings?|reviews? of|compare|find a)\b",+))+++def normalize_domain(url_or_domain: str) -> str:+ """A bare, lowercase, www-less hostname — the entity-resolution key.++ Entity resolution decides whether any of this is real. Get it wrong and+ every trend built on top is garbage.+ """+ if not url_or_domain:+ return ""+ raw = url_or_domain.strip().lower()+ if "://" not in raw:+ raw = "http://" + raw+ host = (urlparse(raw).hostname or "").strip(".")+ return host[4:] if host.startswith("www.") else host+++def is_junk(domain: str, name: str = "") -> bool:+ """True when this row would pollute the database rather than fill it."""+ if domain:+ if any(p.search(domain) for p in JUNK_DOMAIN_PATTERNS):+ return True+ # A bare TLD or an IP address is never a business we can profile.+ if "." not in domain or re.fullmatch(r"[\d.]+", domain):+ return True+ if name and any(p.search(name) for p in JUNK_NAME_PATTERNS):+ return True+ return False+++# ── Geocoding ─────────────────────────────────────────────────────────+# Listings providers search a radius, not a place name, so "Denver, CO" has to+# become coordinates. Nominatim allows exactly this (single place lookups at+# <=1 req/s) and forbids bulk enumeration, so every result is cached forever in+# the settings table: a given city is looked up once, ever.++NOMINATIM = "https://nominatim.openstreetmap.org/search"+GEOCODE_UA = "Harvey/0.1 (open-source sales agent; github.com/ethanplusai/harvey)"+++async def geocode_city(state, city: str, radius_km: int = 40,+ client: httpx.AsyncClient | None = None) -> str:+ """"Denver, CO" -> "39.7392,-104.9903,40". Cached permanently."""+ if not city:+ return ""+ cache_key = f"geo:{city.strip().lower()}"+ cached = await state.get_setting(cache_key)+ if cached:+ # A previous miss is cached as "-" so we don't re-ask every run.+ return "" if cached == "-" else f"{cached},{radius_km}"++ owned = client is None+ client = client or httpx.AsyncClient(timeout=TIMEOUT)+ try:+ r = await client.get(+ NOMINATIM,+ params={"q": city, "format": "json", "limit": 1},+ headers={"User-Agent": GEOCODE_UA},+ )+ r.raise_for_status()+ hits = r.json()+ except Exception as e:+ logger.warning("geocode failed for %r: %s", city, e)+ return ""+ finally:+ if owned:+ await client.aclose()++ if not hits:+ await state.set_setting(cache_key, "-")+ logger.warning("geocode: no match for %r", city)+ return ""++ point = f"{float(hits[0]['lat']):.4f},{float(hits[0]['lon']):.4f}"+ await state.set_setting(cache_key, point)+ return f"{point},{radius_km}"+++async def resolve_coordinates(state, queries: list["DiscoveryQuery"],+ radius_km: int = 40,+ client: httpx.AsyncClient | None = None+ ) -> list["DiscoveryQuery"]:+ """Fill in coordinates for any query that lacks them.++ Configured ``icp.geo_coordinates`` always wins; this only covers what the+ user did not spell out. Distinct cities are geocoded once each, however+ many queries share them.+ """+ needed = {q.location for q in queries if q.location and not q.coordinate}+ points: dict[str, str] = {}+ for i, city in enumerate(sorted(needed)):+ if i:+ await asyncio.sleep(1.1) # Nominatim: max 1 request/second+ points[city] = await geocode_city(state, city, radius_km, client)++ for query in queries:+ if not query.coordinate:+ query.coordinate = points.get(query.location, "")+ return queries+++# ── The record every provider returns ─────────────────────────────────+++@dataclass+class Business:+ """One discovered business, normalised across providers."""++ name: str = ""+ domain: str = ""+ website: str = ""+ phone: str = ""+ address: str = ""+ location: str = ""+ category: str = ""+ rating: float | None = None+ review_count: int | None = None+ is_claimed: bool | None = None+ has_website: bool | None = None+ rank: int | None = None # organic rank, when the provider gives one+ external_id: str = "" # "<provider>:<stable id>"+ source_url: str = ""+ provider: str = ""++ def identity(self) -> str:+ return self.domain or self.external_id+++@dataclass+class DiscoveryQuery:+ """What to look for, and where."""++ term: str # "roofing contractor"+ location: str = "" # "Denver, CO"+ category: str = "" # provider-native category slug, optional+ coordinate: str = "" # "lat,lng,radius_km" for listings providers+ depth: int = 30 # SERP depth; ignored by listings providers+ limit: int = 100 # max records to take from one query++ def keyword(self) -> str:+ return f"{self.term} {self.location}".strip()+++# ── Provider adapters ─────────────────────────────────────────────────+++class DiscoveryProvider:+ """One way of answering "who exists".++ Subclasses implement ``fetch``. Everything else — the menu metadata the+ dashboard renders, cost estimation, configuration checks — lives here so a+ new provider is a small class rather than a wiring exercise.+ """++ key: str = ""+ label: str = ""+ kind: str = "listings" # listings | serp+ blurb: str = ""+ cost_note: str = ""+ free_tier: str = ""+ signup_url: str = ""+ env_keys: tuple[str, ...] = ()+ caveat: str = ""++ def configured(self, env: dict) -> bool:+ return all(env.get(k) for k in self.env_keys)++ def estimate(self, queries: list[DiscoveryQuery]) -> float:+ """Projected spend, in dollars, before anything is called."""+ return 0.0++ async def fetch(self, client: httpx.AsyncClient, env: dict,+ query: DiscoveryQuery) -> list[Business]:+ raise NotImplementedError++ # Shared helper: providers report their own per-query spend so the run log+ # records what actually happened, not what was projected.+ last_cost: float = 0.0+++class DataForSEOListings(DiscoveryProvider):+ """Google Business Profile data as a queryable database, not a scrape.++ The cheapest way to enumerate local businesses anywhere, and the only one+ with a filter for "has no website" — which for anyone selling websites is+ the highest-intent query available.+ """++ key = "dataforseo_listings"+ label = "DataForSEO Business Listings"+ kind = "listings"+ blurb = ("Enumerate every business of a type within a radius, with phone, "+ "domain, rating and whether the listing is claimed. Best choice "+ "for local trades, clinics and contractors.")+ cost_note = "$0.372 per 1,000 businesses ($0.012/query + $0.00036 each)"+ free_tier = "$1 signup credit; $50 minimum deposit to go live"+ signup_url = "https://app.dataforseo.com/api-access"+ env_keys = ("dataforseo_login", "dataforseo_password")+ caveat = "No email addresses — Harvey resolves those in a later stage."++ TASK_COST = 0.012+ ITEM_COST = 0.00036++ def estimate(self, queries):+ return sum(self.TASK_COST + self.ITEM_COST * q.limit for q in queries)++ def _auth(self, env: dict) -> str:+ raw = f"{env['dataforseo_login']}:{env['dataforseo_password']}"+ return base64.b64encode(raw.encode()).decode()++ def base_url(self, env: dict) -> str:+ # The sandbox mirrors the response schema exactly and is never+ # charged — the right target for a first run and for the test suite.+ return ("https://sandbox.dataforseo.com/v3"+ if env.get("dataforseo_sandbox") else+ "https://api.dataforseo.com/v3")++ async def fetch(self, client, env, query):+ payload = {+ "limit": min(query.limit, 1000),+ "order_by": ["rating.votes_count,desc"],+ }+ if query.category:+ payload["categories"] = [query.category]+ else:+ payload["title"] = query.term+ if query.coordinate:+ payload["location_coordinate"] = query.coordinate+ elif query.location:+ # Without coordinates the API can only match on the title, so say+ # so rather than silently returning nationwide results.+ payload["description"] = query.location++ r = await client.post(+ f"{self.base_url(env)}/business_data/business_listings/search/live",+ headers={"Authorization": f"Basic {self._auth(env)}",+ "Content-Type": "application/json"},+ json=[payload],+ )+ r.raise_for_status()+ data = r.json()+ self.last_cost = float(data.get("cost") or 0.0)+ return [self._to_business(item) for item in _dfs_items(data)]++ def _to_business(self, item: dict) -> Business:+ url = item.get("url") or ""+ rating = (item.get("rating") or {})+ addr = item.get("address_info") or {}+ city = ", ".join(x for x in (addr.get("city"), addr.get("region")) if x)+ return Business(+ name=item.get("title") or "",+ domain=normalize_domain(item.get("domain") or url),+ website=url,+ phone=item.get("phone") or "",+ address=item.get("address") or "",+ location=city,+ category=item.get("category") or "",+ rating=rating.get("value"),+ review_count=rating.get("votes_count"),+ is_claimed=item.get("is_claimed"),+ has_website=bool(url),+ external_id=f"dataforseo:{item['place_id']}" if item.get("place_id") else "",+ source_url=item.get("check_url") or "",+ provider=self.key,+ )+++class DataForSEOSerp(DiscoveryProvider):+ """Organic rank as the buying signal.++ Buy the rank, not the list. A business sitting at 11-30 for its own core+ search is visible enough to have tried and is losing — the single most+ useful thing you can know before writing to them.+ """++ key = "dataforseo_serp"+ label = "DataForSEO SERP (organic rank)"+ kind = "serp"+ blurb = ("Search '<service> <city>' and record where each business ranks. "+ "Discovers businesses you didn't have AND scores the ones you did.")+ cost_note = "$0.0006 per 10 results — depth 30 costs $0.0018 a search"+ free_tier = "$1 signup credit; free sandbox; $50 minimum deposit to go live"+ signup_url = "https://app.dataforseo.com/api-access"+ env_keys = ("dataforseo_login", "dataforseo_password")+ caveat = ("Google removed 100-results-per-page in September 2025, so depth "+ "100 now costs 10x depth 10. Depth 20-30 is the sweet spot; "+ "below rank 30 it is mostly directories anyway.")++ PAGE_COST = 0.0006++ def estimate(self, queries):+ # Billing is base price x number of pages, 10 results to a page.+ return sum(self.PAGE_COST * max(1, -(-q.depth // 10)) for q in queries)++ _auth = DataForSEOListings._auth+ base_url = DataForSEOListings.base_url++ async def fetch(self, client, env, query):+ r = await client.post(+ f"{self.base_url(env)}/serp/google/organic/live/advanced",+ headers={"Authorization": f"Basic {self._auth(env)}",+ "Content-Type": "application/json"},+ json=[{"keyword": query.keyword(), "language_code": "en",+ "depth": query.depth, "device": "desktop"}],+ )+ r.raise_for_status()+ data = r.json()+ self.last_cost = float(data.get("cost") or 0.0)++ out = []+ for item in _dfs_items(data):+ if item.get("type") != "organic":+ continue+ domain = normalize_domain(item.get("domain") or item.get("url") or "")+ if not domain:+ continue+ out.append(Business(+ name=item.get("title") or domain,+ domain=domain,+ website=item.get("url") or "",+ location=query.location,+ # rank_group is position within organic results.+ # `position` in this payload means left/right column, NOT rank.+ rank=item.get("rank_group"),+ has_website=True,+ source_url=item.get("url") or "",+ provider=self.key,+ ))+ return out+++class Serper(DiscoveryProvider):+ """Google search with the friendliest free tier and the best depth pricing.++ The one provider that did not go to 10x for depth 100 when Google removed+ 100-results-per-page, and 2,500 free queries with no card.+ """++ key = "serper"+ label = "Serper"+ kind = "serp"+ blurb = ("Fast Google search with rank positions. The easiest one to try: "+ "2,500 free queries, no credit card, one API key.")+ cost_note = "~$0.30-$1.00 per 1,000 searches depending on plan"+ free_tier = "2,500 queries free, no card required"+ signup_url = "https://serper.dev"+ env_keys = ("serper_api_key",)+ caveat = "Paid plans start at a $50 prepaid pack; credits expire after 6 months."++ def estimate(self, queries):+ # One credit per 10 results on the standard plan.+ return sum(0.001 * max(1, -(-q.depth // 10)) for q in queries)++ async def fetch(self, client, env, query):+ r = await client.post(+ "https://google.serper.dev/search",+ headers={"X-API-KEY": env["serper_api_key"],+ "Content-Type": "application/json"},+ json={"q": query.keyword(), "num": min(query.depth, 100)},+ )+ r.raise_for_status()+ data = r.json()+ out = []+ for item in data.get("organic") or []:+ domain = normalize_domain(item.get("link") or "")+ if not domain:+ continue+ out.append(Business(+ name=item.get("title") or domain,+ domain=domain,+ website=item.get("link") or "",+ location=query.location,+ rank=item.get("position"),+ has_website=True,+ source_url=item.get("link") or "",+ provider=self.key,+ ))+ return out+++class OpenStreetMap(DiscoveryProvider):+ """Genuinely free, no account, no key. Honest about what that costs you.++ Good enough to try the whole pipeline end to end without signing up for+ anything, and a useful cross-reference layer afterwards. Not good enough+ to be your only source: OSM maps physical premises, so service-area+ businesses (roofers, HVAC, plumbers) are heavily under-represented.+ """++ key = "osm"+ label = "OpenStreetMap (Overpass)"+ kind = "listings"+ blurb = ("Free public map data. No signup, no key, no card. Try the whole "+ "pipeline for nothing before you pay anyone.")+ cost_note = "free"+ free_tier = "unlimited, within community rate etiquette"+ signup_url = "https://www.openstreetmap.org"+ env_keys = ()+ caveat = ("Coverage is thin for businesses without a storefront — OSM has "+ "under 2,000 roofers for the entire US. Great for a free trial "+ "run or as a cross-reference; not a complete list.")++ ENDPOINT = "https://overpass-api.de/api/interpreter"+ # Overpass etiquette for a shipped tool: modest volume, no parallelism.+ MIN_INTERVAL = 2.0+ BACKOFF = 30.0++ # A friendly term maps to the OSM tags that actually carry those businesses.+ TAG_MAP = {+ "roof": ['craft=roofer', 'shop=roofing'],+ "plumb": ['craft=plumber'],+ "hvac": ['craft=hvac'],+ "electric": ['craft=electrician'],+ "dentist": ['amenity=dentist'],+ "lawyer": ['office=lawyer'],+ "attorney": ['office=lawyer'],+ "contractor": ['craft=builder', 'office=construction_company'],+ "landscap": ['craft=gardener', 'shop=garden_centre'],+ }++ def _tags(self, term: str) -> list[str]:+ """The OSM tags that actually carry this kind of business, or none.++ There is deliberately no catch-all fallback. An earlier version fell+ back to `office=company`, which answers "roofers in Denver" with every+ registered office in Denver — coffee roasters, churches, photography+ studios. A search this source cannot answer has to say so; filling the+ database with plausible-looking junk is far worse than returning+ nothing.+ """+ low = term.lower()+ for needle, tags in self.TAG_MAP.items():+ if needle in low:+ return tags+ return []++ async def fetch(self, client, env, query):+ if not query.coordinate:+ logger.warning("osm: no coordinate for %r — Overpass needs a "+ "lat,lng,radius to search", query.keyword())+ return []+ tags = self._tags(query.term)+ if not tags:+ raise RuntimeError(+ f"OpenStreetMap has no category for {query.term!r}. It can only "+ f"search mapped trades ({', '.join(sorted(self.TAG_MAP))}). Use a "+ f"listings or SERP provider for this one.")+ lat, lng, radius_km = (query.coordinate.split(",") + ["", "", ""])[:3]+ radius_m = int(float(radius_km or 25) * 1000)+ clauses = "".join(+ f'node[{tag.split("=")[0]}={tag.split("=")[1]}](around:{radius_m},{lat},{lng});'+ f'way[{tag.split("=")[0]}={tag.split("=")[1]}](around:{radius_m},{lat},{lng});'+ for tag in tags+ )+ body = f"[out:json][timeout:40];({clauses});out center {query.limit};"++ # Overpass is a free volunteer service and 504s readily under load —+ # a cold first request very often does. Retry once with a real+ # backoff, then RAISE: returning [] would make a throttled run look+ # exactly like a market with no businesses in it.+ headers = {"User-Agent": "Harvey/0.1 (open-source sales agent; "+ "github.com/ethanplusai/harvey)"}+ for attempt in range(2):+ r = await client.post(self.ENDPOINT, content=body.encode(),+ headers=headers)+ if r.status_code not in (429, 504):+ break+ if attempt == 0:+ logger.warning("osm: throttled (%s) — backing off %ss",+ r.status_code, self.BACKOFF)+ await asyncio.sleep(self.BACKOFF)+ else:+ raise RuntimeError(+ f"Overpass is throttling (HTTP {r.status_code}). This is a free "+ f"community service — wait a minute and re-run, or use a paid "+ f"provider for bulk work.")+ r.raise_for_status()++ out = []+ for el in r.json().get("elements") or []:+ tags = el.get("tags") or {}+ name = tags.get("name") or ""+ if not name:+ continue+ site = tags.get("website") or tags.get("contact:website") or ""+ city = ", ".join(x for x in (tags.get("addr:city"),+ tags.get("addr:state")) if x)+ out.append(Business(+ name=name,+ domain=normalize_domain(site),+ website=site,+ phone=tags.get("phone") or tags.get("contact:phone") or "",+ address=" ".join(x for x in (tags.get("addr:housenumber"),+ tags.get("addr:street")) if x),+ location=city or query.location,+ category=tags.get("craft") or tags.get("shop") or tags.get("office") or "",+ has_website=bool(site),+ external_id=f"osm:{el.get('type')}/{el.get('id')}",+ source_url=f"https://www.openstreetmap.org/{el.get('type')}/{el.get('id')}",+ provider=self.key,+ ))+ return out+++PROVIDERS: dict[str, DiscoveryProvider] = {+ p.key: p for p in (+ OpenStreetMap(),+ DataForSEOListings(),+ DataForSEOSerp(),+ Serper(),+ )+}++DEFAULT_PROVIDER = "osm"+++def _dfs_items(payload: dict) -> list[dict]:+ """Pull result items out of a DataForSEO envelope, surfacing errors.++ A bare status code costs three wrong guesses; the message in the body+ usually makes the problem obvious immediately.+ """+ if payload.get("status_code") not in (20000, None):+ raise RuntimeError(+ f"DataForSEO {payload.get('status_code')}: "+ f"{payload.get('status_message')}")+ tasks = payload.get("tasks") or []+ if not tasks:+ return []+ task = tasks[0]+ if task.get("status_code") not in (20000, None):+ raise RuntimeError(+ f"DataForSEO task {task.get('status_code')}: "+ f"{task.get('status_message')}")+ results = task.get("result") or []+ if not results or not results[0]:+ return []+ return results[0].get("items") or []+++# ── The run ───────────────────────────────────────────────────────────+++@dataclass+class DiscoveryReport:+ """What a run did — the honest version, including what it skipped."""++ provider: str = ""+ queries: int = 0+ estimated_cost: float = 0.0+ actual_cost: float = 0.0+ found: int = 0+ junk: int = 0+ new_companies: int = 0+ known_companies: int = 0+ observations: int = 0+ run_id: str = ""+ errors: list[str] = field(default_factory=list)+ stopped: str = ""++ def as_dict(self) -> dict:+ return {+ "provider": self.provider, "queries": self.queries,+ "estimated_cost": round(self.estimated_cost, 4),+ "actual_cost": round(self.actual_cost, 4),+ "found": self.found, "junk": self.junk,+ "new_companies": self.new_companies,+ "known_companies": self.known_companies,+ "observations": self.observations,+ "run_id": self.run_id, "errors": self.errors,+ "stopped": self.stopped,+ }+++def build_queries(config, cities: list[str] | None = None,+ depth: int = 30, limit: int = 100) -> list[DiscoveryQuery]:+ """Turn the configured ICP into a concrete query list.++ One query per (industry x city). Coordinates come from+ ``icp.geo_coordinates`` when present, because listings providers need a+ radius, not a place name.+ """+ icp = config.icp+ terms = list(icp.industries) or [config.product.name]+ places = cities if cities is not None else list(icp.geography) or [""]+ coords = getattr(icp, "geo_coordinates", {}) or {}++ return [+ DiscoveryQuery(term=term, location=place, depth=depth, limit=limit,+ coordinate=coords.get(place, ""))+ for term in terms+ for place in places+ ]+++def estimate_cost(provider_key: str, queries: list[DiscoveryQuery]) -> float:+ """Projected spend. Always call this before a run — and print it."""+ provider = PROVIDERS.get(provider_key)+ return provider.estimate(queries) if provider else 0.0+++async def run_discovery(+ state,+ config,+ provider_key: str = DEFAULT_PROVIDER,+ queries: list[DiscoveryQuery] | None = None,+ max_spend: float = 1.0,+ dry_run: bool = False,+) -> DiscoveryReport:+ """Discover businesses and record what was learned about them.++ ``dry_run`` estimates and returns without calling anything. Run it first,+ every time: the first small trial is what exposes the data-quality+ problems that would otherwise poison every market you scale to.+ """+ from harvey.config import load_env+ from harvey.signals import seed_signal_catalog++ provider = PROVIDERS.get(provider_key)+ if provider is None:+ raise ValueError(f"unknown discovery provider: {provider_key!r}")++ queries = queries or build_queries(config)+ if provider.kind == "listings":+ queries = await resolve_coordinates(state, queries)+ report = DiscoveryReport(provider=provider_key, queries=len(queries))+ report.estimated_cost = provider.estimate(queries)++ if dry_run:+ return report++ env = load_env().model_dump()+ if not provider.configured(env):+ missing = ", ".join(k.upper() for k in provider.env_keys+ if not env.get(k))+ report.errors.append(f"{provider.label} needs {missing} in .env")+ report.stopped = "not configured"+ return report++ if report.estimated_cost > max_spend:+ report.errors.append(+ f"estimated ${report.estimated_cost:.2f} exceeds the ${max_spend:.2f} "+ f"cap — raise the cap or run fewer queries")+ report.stopped = "over budget"+ return report++ await seed_signal_catalog(state)+ confirmed = await state.confirmed_signal_codes()+ if not confirmed:+ report.errors.append(+ "no signals confirmed — confirm signals first (dashboard → Signals, "+ "or `harvey signals --confirm free`) or discovery has nothing to record")+ report.stopped = "no confirmed signals"+ return report++ report.run_id = await state.start_run(+ "discover", provider=provider_key,+ params={"queries": len(queries), "estimate": report.estimated_cost},+ )++ seen: set[str] = set()+ try:+ async with httpx.AsyncClient(timeout=TIMEOUT, follow_redirects=True) as client:+ for i, query in enumerate(queries):+ # The kill switch is read between batches, not once at the+ # top: a long run has to be stoppable while it is running.+ if await state.get_setting("discovery_paused"):+ report.stopped = "paused"+ break+ if report.actual_cost > max_spend:+ report.stopped = "budget reached"+ break+ if i and provider.key == "osm":+ await asyncio.sleep(OpenStreetMap.MIN_INTERVAL)++ try:+ provider.last_cost = 0.0+ found = await provider.fetch(client, env, query)+ report.actual_cost += provider.last_cost+ except Exception as e:+ # A failure is an observation about the run, not a reason+ # to lose the queries that already succeeded.+ logger.warning("discover: %s failed on %r: %s",+ provider_key, query.keyword(), e)+ report.errors.append(f"{query.keyword()}: {e}")+ continue++ report.found += len(found)+ batch = []+ for biz in found:+ if is_junk(biz.domain, biz.name):+ report.junk += 1+ continue+ ident = biz.identity()+ if not ident or ident in seen:+ continue+ seen.add(ident)++ company_id, is_new = await _upsert(state, biz, query)+ if is_new:+ report.new_companies += 1+ else:+ report.known_companies += 1+ batch.extend(_observations(biz, company_id, confirmed,+ report.run_id))++ # Flush every batch. The observations are the asset; buffering+ # them until some threshold means a run that dies loses them.+ if batch:+ report.observations += await state.add_observations(+ batch, run_id=report.run_id)++ await state.finish_run(report.run_id, status="completed",+ records=report.new_companies + report.known_companies,+ cost_usd=report.actual_cost)+ except Exception as e:+ logger.exception("discover run failed")+ report.errors.append(str(e))+ await state.finish_run(report.run_id, status="failed",+ records=report.new_companies,+ cost_usd=report.actual_cost, error=str(e))+ return report+++async def _upsert(state, biz: Business, query: DiscoveryQuery) -> tuple[str, bool]:+ """Record the business, returning (id, was_new)."""+ # Look before inserting, so "already knew this one" is a fact we report+ # rather than something inferred from an insert's return value.+ existing = None+ if biz.domain:+ existing = await state.get_company_by_domain(biz.domain)+ if existing is None and biz.external_id:+ existing = await state.get_company_by_external_id(biz.external_id)+ if existing:+ return existing.id, False++ company = Company(+ name=biz.name,+ domain=biz.domain,+ website=biz.website,+ phone=biz.phone,+ industry=biz.category or query.term,+ location=biz.location or query.location,+ source=f"discover:{biz.provider}",+ source_url=biz.source_url,+ external_id=biz.external_id,+ )+ return await state.add_company(company), True+++def _observations(biz: Business, company_id: str, confirmed: set[str],+ run_id: str) -> list[dict]:+ """Every fact this business gave us, as rows — and only the confirmed ones."""+ rows: list[dict] = []++ def add(code, *, num=None, text="", conf=1.0):+ if code in confirmed:+ rows.append({+ "company_id": company_id, "signal_code": code,+ "collector": "discover", "value_num": num, "value_text": text,+ "confidence": conf, "evidence_url": biz.source_url,+ "run_id": run_id,+ })++ add("FOUND_IN_SERP", text=biz.provider)+ if biz.rank is not None:+ add("SERP_RANK", num=float(biz.rank))+ if biz.has_website is False:+ add("NO_WEBSITE", num=1.0)+ if biz.is_claimed is False:+ add("UNCLAIMED_LISTING", num=1.0)+ if biz.rating is not None:+ add("REVIEW_RATING", num=float(biz.rating))+ if biz.review_count is not None:+ add("REVIEW_COUNT", num=float(biz.review_count))+ return rows+++def provider_menu(env: dict | None = None) -> list[dict]:+ """The menu the dashboard renders: what each provider does, and its cost."""+ env = env or {}+ return [+ {+ "key": p.key, "label": p.label, "kind": p.kind, "blurb": p.blurb,+ "cost_note": p.cost_note, "free_tier": p.free_tier,+ "signup_url": p.signup_url, "caveat": p.caveat,+ "env_keys": [k.upper() for k in p.env_keys],+ "configured": p.configured(env),+ "needs_key": bool(p.env_keys),+ }+ for p in PROVIDERS.values()+ ]diff --git a/harvey/config.py b/harvey/config.pyindex f8c319c..882cc0b 100644--- a/harvey/config.py+++ b/harvey/config.py@@ -60,6 +60,11 @@ class ICPConfig(BaseModel):# hiring a "Head of Growth" is buying growth tooling). Empty → falls back# to `titles`. Used for careers-page scanning and job-board discovery.hiring_signals: list[str] = []+ # Discovery needs a radius, not a place name. Maps each entry in+ # `geography` to "lat,lng,radius_km" — e.g.+ # "Denver, CO": "39.7392,-104.9903,50"+ # Without one, listings providers can only match on the business name.+ geo_coordinates: dict[str, str] = {}class EmailChannelConfig(BaseModel):@@ -157,6 +162,10 @@ class HarveyConfig(BaseModel):class EnvConfig(BaseModel):instantly_api_key: str = ""+ # Discovery providers+ dataforseo_login: str = ""+ dataforseo_password: str = ""+ dataforseo_sandbox: str = "" # any truthy value routes to the free sandboxlinkedin_email: str = ""linkedin_password: str = ""hunter_api_key: str = ""@@ -231,6 +240,9 @@ def load_env() -> EnvConfig:load_dotenv()env = EnvConfig(instantly_api_key=os.getenv("INSTANTLY_API_KEY", "").strip(),+ dataforseo_login=os.getenv("DATAFORSEO_LOGIN", "").strip(),+ dataforseo_password=os.getenv("DATAFORSEO_PASSWORD", "").strip(),+ dataforseo_sandbox=os.getenv("DATAFORSEO_SANDBOX", "").strip(),linkedin_email=os.getenv("LINKEDIN_EMAIL", "").strip(),linkedin_password=os.getenv("LINKEDIN_PASSWORD", "").strip(),hunter_api_key=os.getenv("HUNTER_API_KEY", "").strip(),diff --git a/harvey/dashboard.py b/harvey/dashboard.pyindex 60eb4a6..9ac889b 100644--- a/harvey/dashboard.py+++ b/harvey/dashboard.py@@ -761,6 +761,134 @@ async def get_activity():# ── Dashboard UI ──+# ── Discovery: the provider menu, an estimate, and a run ──+#+# This is the only stage that spends money, so the UI never starts one without+# showing what it will cost first.++_discovery_task: asyncio.Task | None = None+_discovery_report: dict | None = None+++def _discovery_queries(body: dict, config):+ from harvey.collectors.discover import build_queries++ cities = [c.strip() for c in (body.get("cities") or []) if c.strip()] or None+ return build_queries(+ config, cities=cities,+ depth=int(body.get("depth") or 30),+ limit=int(body.get("limit") or 100),+ )++[email protected]("/api/discover/providers")+async def get_discovery_providers():+ """What each source does, what it costs, and whether it's ready to use."""+ try:+ from harvey.collectors.discover import DEFAULT_PROVIDER, provider_menu+ from harvey.config import load_env++ state = _state()+ await state.init_db()+ return {+ "providers": provider_menu(load_env().model_dump()),+ "default": DEFAULT_PROVIDER,+ "selected": await state.get_setting("discovery_provider") or DEFAULT_PROVIDER,+ "paused": await state.get_setting("discovery_paused"),+ "running": bool(_discovery_task and not _discovery_task.done()),+ "last_report": _discovery_report,+ }+ except Exception as e:+ logger.exception("discovery providers failed")+ return {"providers": [], "error": str(e)}++[email protected]("/api/discover/estimate")+async def estimate_discovery(request: Request):+ """Projected spend and the exact query list, before anything is called."""+ try:+ from harvey.collectors.discover import PROVIDERS, estimate_cost+ from harvey.config import load_config++ body = await request.json()+ provider = body.get("provider") or ""+ if provider not in PROVIDERS:+ return JSONResponse({"error": f"unknown provider {provider!r}"},+ status_code=400)++ queries = _discovery_queries(body, load_config())+ return {+ "provider": provider,+ "queries": [q.keyword() for q in queries],+ "query_count": len(queries),+ "estimated_cost": round(estimate_cost(provider, queries), 4),+ "free": PROVIDERS[provider].estimate(queries) == 0,+ }+ except Exception as e:+ return JSONResponse({"error": str(e)}, status_code=500)++[email protected]("/api/discover/run")+async def start_discovery(request: Request):+ """Kick off a run in the background and hand back immediately.++ Discovery takes minutes, not milliseconds — holding the request open+ would just time out. Progress shows up in the run log.+ """+ global _discovery_task, _discovery_report++ if _discovery_task and not _discovery_task.done():+ return JSONResponse({"success": False, "message": "a run is already going"},+ status_code=409)+ try:+ from harvey.collectors.discover import PROVIDERS, run_discovery+ from harvey.config import load_config++ body = await request.json()+ provider = body.get("provider") or ""+ if provider not in PROVIDERS:+ return JSONResponse({"success": False,+ "message": f"unknown provider {provider!r}"},+ status_code=400)++ config = load_config()+ queries = _discovery_queries(body, config)+ max_spend = float(body.get("max_spend") or 1.0)++ state = _state()+ await state.init_db()+ await state.set_setting("discovery_provider", provider)+ await state.set_setting("discovery_paused", "")++ async def _go():+ global _discovery_report+ try:+ report = await run_discovery(state, config, provider, queries,+ max_spend=max_spend)+ _discovery_report = report.as_dict()+ except Exception as exc:+ logger.exception("discovery run failed")+ _discovery_report = {"errors": [str(exc)], "stopped": "failed"}++ _discovery_report = None+ _discovery_task = asyncio.create_task(_go())+ return {"success": True, "queries": len(queries)}+ except Exception as e:+ return JSONResponse({"success": False, "message": str(e)}, status_code=500)++[email protected]("/api/discover/stop")+async def stop_discovery():+ """Kill switch. Read between batches, so an in-flight run stops cleanly."""+ try:+ state = _state()+ await state.init_db()+ await state.set_setting("discovery_paused", "stopped from dashboard")+ return {"success": True}+ except Exception as e:+ return JSONResponse({"success": False, "message": str(e)}, status_code=500)++@app.get("/api/today")async def get_today():"""What needs a human, right now.@@ -840,6 +968,15 @@ async def get_today():"action": "Review signals", "tab": "signals",})+ if not n_companies and confirmed:+ items.append({+ "key": "discover", "tone": "good",+ "title": "No companies yet",+ "detail": ("Signals are confirmed but nothing has been collected. "+ "Discovery is free to try — no account needed."),+ "action": "Find businesses", "tab": "discover",+ })+if pending:items.append({"key": "outbox", "tone": "warn",diff --git a/harvey/models/company.py b/harvey/models/company.pyindex c3f29d6..0c261d0 100644--- a/harvey/models/company.py+++ b/harvey/models/company.py@@ -19,8 +19,13 @@ class Company(BaseModel):industry: str = ""company_size: str = ""location: str = ""+ phone: str = ""source: str = "" # how we found them: google_dork, linkedin, company_scrapesource_url: str = "" # specific URL where we found the info+ # Provider-stable identity ("dataforseo:ChIJ...", "osm:node/123"). The+ # dedup key for businesses that have no website — which are precisely the+ # best prospects for anyone selling one.+ external_id: str = ""notes: str = ""# Buying signals: detected tools on their site, and timely triggers# (hiring, funding, ...) as [{"type": ..., "detail": ..., "found_at": ...}]diff --git a/harvey/signals.py b/harvey/signals.pyindex 1b62eae..525d785 100644--- a/harvey/signals.py+++ b/harvey/signals.py@@ -46,6 +46,62 @@ SIGNAL_CATALOG: list[dict] = ["confidence_floor": 0.0,},+ {+ "code": "NO_WEBSITE",+ "label": "No website at all",+ "description": (+ "The business has a listing and a phone number but no site. If you "+ "sell websites this is the highest-intent signal that exists — and "+ "it is a filter you can query for directly, not something you infer."+ ),+ "category": "discovery",+ "value_type": "bool",+ "collector": "discover",+ "cost_note": "included in the business listing lookup",+ "confidence_floor": 0.0,+ },+ {+ "code": "UNCLAIMED_LISTING",+ "label": "Unclaimed Google listing",+ "description": (+ "Nobody has claimed their Google Business Profile, so they are not "+ "managing how they show up locally. Strong signal for anyone "+ "selling local marketing."+ ),+ "category": "discovery",+ "value_type": "bool",+ "collector": "discover",+ "cost_note": "included in the business listing lookup",+ "confidence_floor": 0.0,+ },+ {+ "code": "REVIEW_RATING",+ "label": "Review rating",+ "description": (+ "Their average star rating. Useful in both directions: a low "+ "rating is a reputation problem to solve, a high one with few "+ "reviews is proof they are good but invisible."+ ),+ "category": "discovery",+ "value_type": "num",+ "collector": "discover",+ "cost_note": "included in the business listing lookup",+ "confidence_floor": 0.0,+ },+ {+ "code": "REVIEW_COUNT",+ "label": "Number of reviews",+ "description": (+ "How many reviews they have. A rough proxy for size and for how "+ "much attention they pay to their online presence."+ ),+ "category": "discovery",+ "value_type": "num",+ "collector": "discover",+ "cost_note": "included in the business listing lookup",+ "confidence_floor": 0.0,+ },+# ── Profile: what they are (free — HTML Harvey already fetches) ──{"code": "INCUMBENT_AGENCY",diff --git a/harvey/state.py b/harvey/state.pyindex 9f3c6ed..5f495c7 100644--- a/harvey/state.py+++ b/harvey/state.py@@ -362,6 +362,18 @@ MIGRATIONS: list[str] = [CREATE INDEX IF NOT EXISTS idx_runs_stage_started ON runs(stage, started_at);""",+ # ── v8: discovery — entity resolution for businesses without a website ──+ """+ -- The best prospect for anyone selling websites is a business that has+ -- none, so `domain` cannot be the only identity key. `external_id` holds+ -- the provider's stable id ("dataforseo:ChIJ...", "osm:node/123") and is+ -- what dedups a re-run for those records.+ ALTER TABLE companies ADD COLUMN external_id TEXT DEFAULT '';+ ALTER TABLE companies ADD COLUMN phone TEXT DEFAULT '';++ CREATE UNIQUE INDEX IF NOT EXISTS uq_companies_external+ ON companies(external_id) WHERE external_id != '';+ """,]# Column whitelists for dynamic UPDATEs (prevents SQL injection via kwargs).@@ -425,7 +437,13 @@ class StateManager:return Company(**d)async def add_company(self, company: Company) -> str:- """Insert a company. If one with the same domain exists, return its id."""+ """Insert a company, or return the id of the one already recorded.++ Identity is the normalised domain when there is one, and the+ provider's ``external_id`` when there isn't — a business with no+ website still has to dedup across re-runs, and for anyone selling+ websites those are the best prospects on the list.+ """if not company.id:company.id = _new_id()company.domain = _norm(company.domain)@@ -433,28 +451,34 @@ class StateManager:cursor = await db.execute("""INSERT OR IGNORE INTO companies(id, name, domain, website, description, industry,- company_size, location, source, source_url, notes,- tech_stack_json, signals_json, created_at, updated_at)- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",+ company_size, location, phone, source, source_url,+ external_id, notes, tech_stack_json, signals_json,+ created_at, updated_at)+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",(company.id, company.name, company.domain, company.website,company.description, company.industry, company.company_size,- company.location, company.source, company.source_url,- company.notes,+ company.location, company.phone, company.source,+ company.source_url, company.external_id, company.notes,json.dumps(company.tech_stack), json.dumps(company.signals),company.created_at.isoformat(),company.updated_at.isoformat(),),)await db.commit()- if cursor.rowcount == 0 and company.domain:- # Unique-domain conflict: hand back the existing record's id.- async with db.execute(- "SELECT id FROM companies WHERE domain = ?", (company.domain,)- ) as cur:- row = await cur.fetchone()- if row:- company.id = row[0]+ if cursor.rowcount == 0:+ # Uniqueness conflict: hand back the existing record's id.+ for column, value in (("domain", company.domain),+ ("external_id", company.external_id)):+ if not value:+ continue+ async with db.execute(+ f"SELECT id FROM companies WHERE {column} = ?", (value,)+ ) as cur:+ row = await cur.fetchone()+ if row:+ company.id = row[0]+ breakreturn company.idasync def update_company_signals(@@ -526,6 +550,22 @@ class StateManager:row = await cursor.fetchone()return self._company_from_row(row) if row else None+ async def get_company_by_external_id(self, external_id: str) -> Company | None:+ """Look a company up by its provider-stable id.++ The identity path for businesses with no website — which is exactly+ the cohort worth the most to anyone selling one.+ """+ if not external_id:+ return None+ async with self._connect() as db:+ db.row_factory = aiosqlite.Row+ async with db.execute(+ "SELECT * FROM companies WHERE external_id = ?", (external_id,)+ ) as cursor:+ row = await cursor.fetchone()+ return self._company_from_row(row) if row else None+async def get_contacts_for_company(self, company_id: str) -> list[Prospect]:async with self._connect() as db:db.row_factory = aiosqlite.Rowdiff --git a/harvey/web/app.css b/harvey/web/app.cssindex c27e3f5..0ff87de 100644--- a/harvey/web/app.css+++ b/harvey/web/app.css@@ -546,3 +546,31 @@.badge.t-bad { background: rgba(224,108,117,0.1); color: var(--red); border-color: rgba(224,108,117,0.22); }.badge.t-idle { background: rgba(255,255,255,0.04); color: var(--text-3); border-color: var(--border); }.badge.t-note { background: rgba(180,140,232,0.12); color: var(--purple); border-color: rgba(180,140,232,0.25); }++ /* ── Discovery provider menu ── */+ .prov-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); gap: 12px; margin-bottom: 24px; }+ .prov-card {+ background: var(--panel); border: 1px solid var(--border); border-radius: 12px;+ padding: 18px; cursor: pointer; display: flex; flex-direction: column; gap: 8px;+ }+ .prov-card:hover { border-color: var(--border-strong); }+ .prov-card.selected { border-color: var(--accent); background: linear-gradient(180deg, var(--accent-soft), transparent 60%); }+ .prov-card .prov-head { display: flex; align-items: baseline; justify-content: space-between; gap: 10px; }+ .prov-card h3 { font-size: 14.5px; font-weight: 700; letter-spacing: -0.2px; }+ .prov-card .blurb { font-size: 12.5px; color: var(--text-2); line-height: 1.6; }+ .prov-card .row { display: flex; gap: 8px; align-items: baseline; font-size: 12px; }+ .prov-card .row .k { color: var(--text-3); min-width: 42px; }+ .prov-card .row .v { color: var(--text-2); }+ .prov-card .caveat {+ font-size: 11.5px; color: var(--amber); background: rgba(229,181,103,0.07);+ border-left: 2px solid rgba(229,181,103,0.4); padding: 7px 10px; border-radius: 0 6px 6px 0;+ line-height: 1.55;+ }+ .estimate-box {+ border: 1px solid var(--border); border-radius: 10px; padding: 16px 18px;+ background: rgba(255,255,255,0.015); margin-bottom: 16px;+ }+ .estimate-box .amount { font-size: 26px; font-weight: 700; letter-spacing: -0.6px; font-variant-numeric: tabular-nums; }+ .estimate-box .amount.free { color: var(--accent); }+ .estimate-box .amount.paid { color: var(--amber); }+ .query-list { font-family: var(--mono); font-size: 11.5px; color: var(--text-3); margin-top: 12px; line-height: 1.8; max-height: 160px; overflow-y: auto; }diff --git a/harvey/web/app.js b/harvey/web/app.jsindex 1860719..2135f95 100644--- a/harvey/web/app.js+++ b/harvey/web/app.js@@ -135,6 +135,7 @@ function loadCurrentTab() {switch (currentTab) {case 'today': loadToday(); loadStats(); loadSetupStatus(); loadRuns(); break;case 'signals': loadSignals(); break;+ case 'discover': loadDiscoverProviders(); break;case 'companies': if (!companyDrill) loadCompanies(); break;case 'prospects': loadProspects(); break;case 'campaigns': loadCampaigns(); break;@@ -350,6 +351,165 @@ async function setSignals(codes, status) {loadSignals();}+// ── Discover: pick a source, price it, then run it ──++let _provider = null;+let _discoverPoll = null;++async function loadDiscoverProviders() {+ const el = document.getElementById('discover-providers');+ const data = await api('/api/discover/providers');+ if (!data || !data.providers) { el.innerHTML = offlineState(); return; }++ _provider = _provider || data.selected || data.default;+ el.innerHTML = '<div class="prov-grid">' + data.providers.map(p => {+ const ready = p.configured+ ? '<span class="badge t-good">ready</span>'+ : '<span class="badge t-waiting">needs a key</span>';+ return '<div class="prov-card ' + (p.key === _provider ? 'selected' : '') ++ '" onclick="pickProvider(\'' + p.key + '\')">' ++ '<div class="prov-head"><h3>' + escHtml(p.label) + '</h3>' + ready + '</div>' ++ '<div class="blurb">' + escHtml(p.blurb) + '</div>' ++ '<div class="row"><span class="k">Cost</span><span class="v">' ++ escHtml(p.cost_note) + '</span></div>' ++ '<div class="row"><span class="k">Free</span><span class="v">' ++ escHtml(p.free_tier) + '</span></div>' ++ (p.needs_key+ ? '<div class="row"><span class="k">Setup</span><span class="v">' ++ escHtml(p.env_keys.join(', ')) + ' in .env · ' ++ '<a href="' + escHtml(p.signup_url) + '" target="_blank" rel="noopener">get a key</a>' ++ '</span></div>'+ : '') ++ (p.caveat ? '<div class="caveat">' + escHtml(p.caveat) + '</div>' : '') ++ '</div>';+ }).join('') + '</div>';++ const running = data.running;+ document.getElementById('disc-stop').style.display = running ? '' : 'none';+ if (running && !_discoverPoll) {+ _discoverPoll = setInterval(loadDiscoverProviders, 4000);+ } else if (!running && _discoverPoll) {+ clearInterval(_discoverPoll);+ _discoverPoll = null;+ // A finished run leaves the button stuck on "Running…" otherwise.+ const btn = document.getElementById('disc-run');+ btn.disabled = true;+ btn.textContent = 'Estimate first';+ showToast('Discovery finished.', 'success');+ }+ renderDiscoverResult(data.last_report, running);+}++function pickProvider(key) {+ _provider = key;+ document.getElementById('disc-run').disabled = true;+ document.getElementById('disc-run').textContent = 'Estimate first';+ document.getElementById('discover-estimate').innerHTML = '';+ loadDiscoverProviders();+}++function discoverBody() {+ const raw = document.getElementById('disc-cities').value.trim();+ return {+ provider: _provider,+ // Semicolons or newlines, never commas: "Denver, CO" is one city.+ cities: raw ? raw.split(/[;\n]/).map(c => c.trim()).filter(Boolean) : [],+ depth: Number(document.getElementById('disc-depth').value) || 30,+ limit: Number(document.getElementById('disc-limit').value) || 100,+ max_spend: Number(document.getElementById('disc-cap').value) || 1,+ };+}++async function estimateDiscovery() {+ const el = document.getElementById('discover-estimate');+ el.innerHTML = '<p class="muted" style="font-size:13px">Pricing it…</p>';+ const data = await api('/api/discover/estimate', {+ method: 'POST', headers: {'Content-Type': 'application/json'},+ body: JSON.stringify(discoverBody()),+ });+ if (!data || data.error) {+ el.innerHTML = '<div class="test-result error">' ++ escHtml((data && data.error) || 'Could not estimate.') + '</div>';+ return;+ }++ const cap = discoverBody().max_spend;+ const over = data.estimated_cost > cap;+ el.innerHTML = '<div class="estimate-box">' ++ '<div class="amount ' + (data.free ? 'free' : 'paid') + '">' ++ (data.free ? 'Free' : '$' + data.estimated_cost.toFixed(4)) + '</div>' ++ '<div class="muted" style="font-size:13px;margin-top:2px">' ++ data.query_count + ' quer' + (data.query_count === 1 ? 'y' : 'ies') ++ (data.free ? '' : ' · cap is $' + cap.toFixed(2)) + '</div>' ++ (over ? '<div class="test-result error" style="margin-top:12px">' ++ 'Over your cap. Raise the cap or narrow the search.</div>' : '') ++ '<div class="query-list">' ++ data.queries.slice(0, 40).map(escHtml).join('<br>') ++ (data.queries.length > 40 ? '<br>… and ' + (data.queries.length - 40) + ' more' : '') ++ '</div></div>';++ const btn = document.getElementById('disc-run');+ btn.disabled = over;+ btn.textContent = over ? 'Over cap'+ : (data.free ? 'Run — free' : 'Run — spend up to $' + data.estimated_cost.toFixed(2));+}++async function runDiscovery() {+ const btn = document.getElementById('disc-run');+ btn.disabled = true;+ btn.textContent = 'Running…';+ const data = await api('/api/discover/run', {+ method: 'POST', headers: {'Content-Type': 'application/json'},+ body: JSON.stringify(discoverBody()),+ });+ if (data && data.success) {+ showToast('Discovery started — ' + data.queries + ' queries.', 'success');+ } else {+ showToast((data && data.message) || 'Could not start.', 'error');+ btn.disabled = false;+ }+ loadDiscoverProviders();+}++async function stopDiscovery() {+ const data = await api('/api/discover/stop', {method: 'POST'});+ showToast(data && data.success ? 'Stopping after the current query.'+ : 'Could not stop.', data ? 'success' : 'error');+ loadDiscoverProviders();+}++function renderDiscoverResult(report, running) {+ const el = document.getElementById('discover-result');+ if (running) {+ el.innerHTML = '<div class="card"><h2>Running…</h2>' ++ '<p class="muted" style="font-size:13px">Harvey is working through the ' ++ 'queries. Results land in Companies, and the run log is on Today.</p></div>';+ return;+ }+ if (!report) { el.innerHTML = ''; return; }++ const stat = (label, value, tone) =>+ '<div class="stat-card"><div class="label">' + label + '</div>' ++ '<div class="value"' + (tone ? ' style="color:var(--' + tone + ')"' : '') + '>' ++ value + '</div></div>';++ el.innerHTML = '<div class="subhead">Last run</div>' ++ '<div class="stats-grid">' ++ stat('New companies', report.new_companies || 0, 'accent') ++ stat('Already known', report.known_companies || 0) ++ stat('Observations', report.observations || 0) ++ stat('Filtered as junk', report.junk || 0) ++ stat('Actual cost', report.actual_cost ? '$' + report.actual_cost.toFixed(4) : 'free') ++ '</div>' ++ (report.stopped ? '<div class="test-result error" style="margin-top:14px">' ++ 'Stopped early: ' + escHtml(report.stopped) + '</div>' : '') ++ ((report.errors || []).length+ ? '<div class="card" style="margin-top:14px"><h2>Problems</h2>' ++ (report.errors || []).slice(0, 8).map(e =>+ '<div class="check-help">' + escHtml(e) + '</div>').join('') + '</div>'+ : '');+}+// ── Cohort builder: a prospect list is a query ──let _cohort = { require: new Set(), exclude: new Set() };diff --git a/harvey/web/index.html b/harvey/web/index.htmlindex dbef5af..5d1aa9c 100644--- a/harvey/web/index.html+++ b/harvey/web/index.html@@ -28,6 +28,7 @@<nav><button class="active" onclick="showTab('today', this)">Today<span class="nav-count" id="nav-today"></span></button><button onclick="showTab('signals', this)">Signals<span class="nav-count" id="nav-signals"></span></button>+ <button onclick="showTab('discover', this)">Discover</button><button onclick="showTab('companies', this)">Companies</button><button onclick="showTab('prospects', this)">Contacts</button><button onclick="showTab('campaigns', this)">Campaigns</button>@@ -75,6 +76,47 @@</div></div>+<!-- Discover: the only stage that spends money -->+<div id="discover" class="section">+ <div class="section-head">+ <h2>Discover</h2>+ <p>Find businesses matching your ICP. Pick a source, see what it will cost,+ then run it — nothing is called until you press the button.</p>+ </div>+ <div id="discover-providers"></div>++ <div class="subhead">Run</div>+ <div class="card">+ <div class="form-row" style="margin-bottom:14px">+ <div class="form-group" style="flex:2">+ <label class="form-label">Cities</label>+ <input class="form-input" id="disc-cities" placeholder="from your ICP — or override, one per line">+ </div>+ <div class="form-group">+ <label class="form-label">Results per query</label>+ <input class="form-input" id="disc-limit" type="number" value="100" min="1" max="1000">+ </div>+ <div class="form-group">+ <label class="form-label">Search depth</label>+ <input class="form-input" id="disc-depth" type="number" value="30" min="10" max="100" step="10">+ </div>+ <div class="form-group">+ <label class="form-label">Spend cap ($)</label>+ <input class="form-input" id="disc-cap" type="number" value="1.00" min="0" step="0.25">+ </div>+ </div>+ <div id="discover-estimate"></div>+ <div class="btn-group">+ <button class="btn btn-secondary" onclick="estimateDiscovery()">Estimate cost</button>+ <button class="btn btn-primary" id="disc-run" onclick="runDiscovery()" disabled>+ Estimate first</button>+ <button class="btn btn-danger" id="disc-stop" onclick="stopDiscovery()" style="display:none">+ Stop</button>+ </div>+ </div>+ <div id="discover-result"></div>+</div>+<!-- Companies --><div id="companies" class="section"><div class="section-head"><h2>Companies</h2><p>Organizations Harvey has researched. Click a row to see its contacts.</p></div>diff --git a/tests/test_dashboard_signals.py b/tests/test_dashboard_signals.pyindex 9785bd8..76b1508 100644--- a/tests/test_dashboard_signals.py+++ b/tests/test_dashboard_signals.py@@ -167,7 +167,7 @@ def test_today_leads_with_unconfirmed_signals(client):assert "signals" in keysitem = next(i for i in data["items"] if i["key"] == "signals")assert item["tab"] == "signals"- assert "19" in item["title"]+ assert str(len(SIGNAL_CATALOG)) in item["title"]def test_today_flags_a_fully_rejected_vocabulary(client):diff --git a/tests/test_discover.py b/tests/test_discover.pynew file mode 100644index 0000000..616bfb7--- /dev/null+++ b/tests/test_discover.py@@ -0,0 +1,660 @@+"""DISCOVER — junk filtering, entity resolution, cost discipline, adapters.++The expensive stage, so the tests care most about the things that cost money+or corrupt the database: that an estimate is honest, that a budget is+enforced before spending rather than after, that a re-run doesn't duplicate+anyone, and that directories never make it in.+"""++import asyncio+import os+import tempfile++import httpx+import pytest+import pytest_asyncio++from harvey.collectors import discover as D+from harvey.config import load_config+from harvey.signals import SIGNAL_CATALOG, seed_signal_catalog+from harvey.state import StateManager+++@pytest_asyncio.fixture+async def state():+ with tempfile.TemporaryDirectory() as tmp:+ sm = StateManager(os.path.join(tmp, "test.db"))+ await sm.init_db()+ await seed_signal_catalog(sm)+ for sig in SIGNAL_CATALOG:+ await sm.set_signal_status(sig["code"], "confirmed")+ yield sm+++def config():+ return load_config()+++# The real implementations, captured before the autouse fixture stubs them,+# so the geocoding tests can exercise them directly.+REAL_GEOCODE = D.geocode_city+REAL_RESOLVE = D.resolve_coordinates+REAL_SLEEP = asyncio.sleep++[email protected](autouse=True)+def offline(monkeypatch):+ """No test reaches the network, and none of them wait a real second.++ The geocoding tests call REAL_GEOCODE / REAL_RESOLVE with their own+ mock-transport client.+ """+ async def no_geocode(state, city, radius_km=40, client=None):+ return ""++ async def no_sleep(_seconds):+ return None++ monkeypatch.setattr(D, "geocode_city", no_geocode)+ monkeypatch.setattr(D.asyncio, "sleep", no_sleep)+++class FakeProvider(D.DiscoveryProvider):+ """A provider that returns what the test hands it, and spends nothing."""++ key = "fake"+ label = "Fake"+ cost_note = "free"++ def __init__(self, batches, per_query_cost=0.0):+ self.batches = list(batches)+ self.per_query_cost = per_query_cost+ self.calls = 0++ def configured(self, env):+ return True++ def estimate(self, queries):+ return self.per_query_cost * len(queries)++ async def fetch(self, client, env, query):+ self.calls += 1+ self.last_cost = self.per_query_cost+ return self.batches.pop(0) if self.batches else []+++def register(monkeypatch):+ """Install a fake provider into the registry for one test."""+ def _install(provider):+ monkeypatch.setitem(D.PROVIDERS, provider.key, provider)+ return provider+ return _install+++def biz(**kw):+ kw.setdefault("provider", "fake")+ return D.Business(**kw)+++# ── Junk filtering ──++[email protected]("domain", [+ "yelp.com", "www.yellowpages.com", "angi.com", "homeadvisor.com",+ "facebook.com", "linkedin.com", "thumbtack.com", "houzz.com",+ "en.wikipedia.org", "reddit.com", "indeed.com",+ "miami.edu", "ufhealth.org", "umiamihealth.org",+ "somehospital.com", "mysite.wordpress.com", "shop.shopify.com",+ "192.168.1.1", "localhost",+])+def test_directories_and_institutions_are_junk(domain):+ assert D.is_junk(domain), f"{domain} should have been filtered out"++[email protected]("domain", [+ "kettlemanroofing.com", "northvaleroofing.com", "sandpiperexteriors.com",+ "quarrylaneroofing.com", "smith-and-sons.co.uk", "acme.dental",+])+def test_real_businesses_survive_the_filter(domain):+ assert not D.is_junk(domain)+++def test_listicle_titles_are_junk_even_with_a_clean_domain():+ assert D.is_junk("", "10 Best Roofers in Denver")+ assert D.is_junk("example.com", "Top Roofing Contractors in Dallas")+ assert D.is_junk("example.com", "Roofer Directory")+ assert not D.is_junk("example.com", "Northvale Roofing Company")+++def test_normalize_domain_is_the_entity_key():+ for raw in ("https://WWW.Example.com/path?q=1", "www.example.com",+ "http://example.com.", "EXAMPLE.COM"):+ assert D.normalize_domain(raw) == "example.com"+ assert D.normalize_domain("") == ""+++# ── Cost discipline ──+++def test_estimates_match_published_pricing():+ q = [D.DiscoveryQuery(term="roofer", location="Denver", limit=1000, depth=30)]+ # DataForSEO's own worked example: 1,000 items = $0.012 + $0.36 = $0.372+ assert D.estimate_cost("dataforseo_listings", q) == pytest.approx(0.372)+ # SERP billing is base price x pages; depth 30 is three pages.+ assert D.estimate_cost("dataforseo_serp", q) == pytest.approx(0.0018)+ assert D.estimate_cost("osm", q) == 0.0+++async def test_dry_run_estimates_without_calling_anything(state, config, register):+ fake = register(FakeProvider([[biz(domain="a.com")]], per_query_cost=0.5))+ q = [D.DiscoveryQuery(term="roofer", location="Denver")]+ report = await D.run_discovery(state, config, "fake", q, dry_run=True)+ assert report.estimated_cost == 0.5+ assert fake.calls == 0+ assert report.found == 0+ assert not await state.get_runs() # a dry run isn't a run+++async def test_budget_cap_refuses_before_spending(state, config, register):+ fake = register(FakeProvider([[biz(domain="a.com")]] * 10, per_query_cost=1.0))+ q = [D.DiscoveryQuery(term="roofer", location=c) for c in "abcdefghij"]+ report = await D.run_discovery(state, config, "fake", q, max_spend=2.0)+ assert report.stopped == "over budget"+ assert fake.calls == 0, "the cap must be checked before the first call, not after"+ assert report.actual_cost == 0.0+++async def test_budget_stops_mid_run_when_actual_exceeds_estimate(state, config, register):+ # Cheap to estimate, expensive in reality — the loop has to notice.+ fake = FakeProvider([[biz(domain=f"{c}.com")] for c in "abcdefghij"],+ per_query_cost=1.0)+ fake.estimate = lambda queries: 0.0+ register(fake)+ q = [D.DiscoveryQuery(term="roofer", location=c) for c in "abcdefghij"]+ report = await D.run_discovery(state, config, "fake", q, max_spend=3.0)+ assert report.stopped == "budget reached"+ assert fake.calls < 10+++async def test_kill_switch_stops_between_batches(state, config, register):+ await state.set_setting("discovery_paused", "stopped by hand")+ fake = register(FakeProvider([[biz(domain="a.com")]], per_query_cost=0.0))+ q = [D.DiscoveryQuery(term="roofer", location="Denver")]+ report = await D.run_discovery(state, config, "fake", q)+ assert report.stopped == "paused"+ assert fake.calls == 0+++async def test_nothing_runs_until_signals_are_confirmed(state, config, register):+ for sig in SIGNAL_CATALOG:+ await state.set_signal_status(sig["code"], "proposed")+ fake = register(FakeProvider([[biz(domain="a.com")]]))+ report = await D.run_discovery(state, config, "fake",+ [D.DiscoveryQuery(term="roofer")])+ assert report.stopped == "no confirmed signals"+ assert fake.calls == 0+ assert "confirm signals" in report.errors[0].lower()+++# ── Entity resolution ──+++async def test_rerun_finds_everyone_already_known(state, config, register):+ batch = [+ biz(name="Northvale Roofing", domain="northvaleroofing.com"),+ biz(name="No Site Roofing", external_id="osm:node/1", has_website=False),+ ]+ register(FakeProvider([list(batch), list(batch)]))+ q = [D.DiscoveryQuery(term="roofer", location="Denver")]++ first = await D.run_discovery(state, config, "fake", q)+ assert (first.new_companies, first.known_companies) == (2, 0)++ second = await D.run_discovery(state, config, "fake", q)+ assert (second.new_companies, second.known_companies) == (0, 2), (+ "a re-run must recognise both the domain-keyed and the "+ "external-id-keyed record")+++async def test_websiteless_businesses_are_kept_and_deduped(state, config, register):+ """The best prospect for someone selling websites has no website, so+ `domain` can't be the only identity key."""+ register(FakeProvider([+ [biz(name="A", external_id="osm:node/1", has_website=False),+ biz(name="B", external_id="osm:node/2", has_website=False),+ biz(name="A again", external_id="osm:node/1", has_website=False)],+ ]))+ report = await D.run_discovery(state, config, "fake",+ [D.DiscoveryQuery(term="roofer")])+ assert report.new_companies == 2+ assert len(await state.cohort(["NO_WEBSITE"])) == 2+++async def test_duplicates_within_one_run_are_collapsed(state, config, register):+ register(FakeProvider([[+ biz(domain="northvale.com"), biz(domain="northvale.com"), biz(domain="other.com"),+ ]]))+ report = await D.run_discovery(state, config, "fake",+ [D.DiscoveryQuery(term="roofer")])+ assert report.found == 3+ assert report.new_companies == 2+++async def test_junk_is_counted_not_silently_dropped(state, config, register):+ register(FakeProvider([[+ biz(domain="yelp.com"), biz(domain="angi.com"), biz(domain="real.com"),+ ]]))+ report = await D.run_discovery(state, config, "fake",+ [D.DiscoveryQuery(term="roofer")])+ assert report.junk == 2+ assert report.new_companies == 1+++# ── Observations ──+++async def test_every_fact_becomes_an_observation(state, config, register):+ register(FakeProvider([[biz(+ name="Northvale", domain="northvale.com", rank=14, rating=4.6,+ review_count=88, is_claimed=False, has_website=True,+ )]]))+ await D.run_discovery(state, config, "fake", [D.DiscoveryQuery(term="roofer")])++ rows = {o["signal_code"]: o for o in await state.get_observations()}+ assert rows["SERP_RANK"]["value_num"] == 14+ assert rows["REVIEW_RATING"]["value_num"] == pytest.approx(4.6)+ assert rows["REVIEW_COUNT"]["value_num"] == 88+ assert "UNCLAIMED_LISTING" in rows+ assert "NO_WEBSITE" not in rows+ assert rows["FOUND_IN_SERP"]["collector"] == "discover"+++async def test_unconfirmed_signals_are_not_recorded(state, config, register):+ """The confirmation gate has to hold inside the collector, not just in+ the UI that sets it."""+ await state.set_signal_status("REVIEW_RATING", "rejected")+ register(FakeProvider([[biz(domain="northvale.com", rating=4.6, review_count=88)]]))+ await D.run_discovery(state, config, "fake", [D.DiscoveryQuery(term="roofer")])++ codes = {o["signal_code"] for o in await state.get_observations()}+ assert "REVIEW_RATING" not in codes+ assert "REVIEW_COUNT" in codes+++async def test_a_failed_query_does_not_lose_the_successful_ones(state, config, register):+ class Flaky(FakeProvider):+ async def fetch(self, client, env, query):+ self.calls += 1+ if self.calls == 2:+ raise httpx.ConnectError("boom")+ return [biz(domain=f"co{self.calls}.com")]++ register(Flaky([], per_query_cost=0.0))+ q = [D.DiscoveryQuery(term="roofer", location=c) for c in ("a", "b", "c")]+ report = await D.run_discovery(state, config, "fake", q)++ assert report.new_companies == 2+ assert len(report.errors) == 1 and "boom" in report.errors[0]+ runs = await state.get_runs()+ assert runs[0]["status"] == "completed"+++async def test_run_is_logged_with_what_it_actually_cost(state, config, register):+ register(FakeProvider([[biz(domain="a.com")], [biz(domain="b.com")]],+ per_query_cost=0.25))+ q = [D.DiscoveryQuery(term="roofer", location=c) for c in ("a", "b")]+ report = await D.run_discovery(state, config, "fake", q, max_spend=5.0)++ runs = await state.get_runs()+ assert runs[0]["stage"] == "discover"+ assert runs[0]["records"] == 2+ assert runs[0]["cost_usd"] == pytest.approx(0.5)+ assert report.actual_cost == pytest.approx(0.5)+++# ── Provider adapters (parsing, against real response shapes) ──+++DFS_LISTINGS = {+ "status_code": 20000,+ "cost": 0.0132,+ "tasks": [{"status_code": 20000, "result": [{"items": [+ {+ "title": "Northvale Roofing Company", "category": "Roofing contractor",+ "place_id": "ChIJabc123", "phone": "+15550100199",+ "url": "https://www.northvaleroofing.com/", "domain": "northvaleroofing.com",+ "address": "1400 Example Row, Raleigh, NC",+ "address_info": {"city": "Raleigh", "region": "North Carolina"},+ "is_claimed": True,+ "rating": {"value": 4.4, "votes_count": 512, "rating_max": 5},+ "check_url": "https://google.com/maps?cid=1",+ },+ {+ "title": "No Website Roofing", "category": "Roofing contractor",+ "place_id": "ChIJxyz789", "phone": "+15550100377",+ "url": None, "domain": None,+ "address_info": {"city": "Denver", "region": "Colorado"},+ "is_claimed": False,+ "rating": {"value": 4.9, "votes_count": 31, "rating_max": 5},+ },+ ]}]}],+}++DFS_SERP = {+ "status_code": 20000,+ "cost": 0.0018,+ "tasks": [{"status_code": 20000, "result": [{"items": [+ {"type": "organic", "rank_group": 1, "rank_absolute": 3,+ "position": "left", "domain": "yelp.com",+ "url": "https://yelp.com/denver", "title": "Best Roofers"},+ {"type": "organic", "rank_group": 14, "rank_absolute": 18,+ "position": "left", "domain": "kettlemanroofing.com",+ "url": "https://kettlemanroofing.com/", "title": "Kettleman Roofing"},+ {"type": "paid", "rank_group": 1, "domain": "ads.example.com"},+ {"type": "people_also_ask", "rank_group": 2},+ ]}]}],+}++SERPER_RESPONSE = {"organic": [+ {"title": "Kettleman Roofing", "link": "https://www.kettlemanroofing.com/",+ "position": 3},+ {"title": "Angi", "link": "https://www.angi.com/companylist/denver/roofing",+ "position": 4},+]}+++def _client(handler):+ return httpx.AsyncClient(transport=httpx.MockTransport(handler))+++async def test_dataforseo_listings_adapter():+ provider = D.DataForSEOListings()+ env = {"dataforseo_login": "u", "dataforseo_password": "p"}+ seen = {}++ def handler(request):+ seen["url"] = str(request.url)+ seen["auth"] = request.headers.get("authorization")+ return httpx.Response(200, json=DFS_LISTINGS)++ async with _client(handler) as client:+ out = await provider.fetch(client, env, D.DiscoveryQuery(+ term="roofing contractor", location="Denver, CO",+ coordinate="39.7,-104.9,50"))++ assert seen["auth"].startswith("Basic ")+ assert provider.last_cost == pytest.approx(0.0132)++ northvale, nosite = out+ assert northvale.domain == "northvaleroofing.com"+ assert northvale.phone == "+15550100199"+ assert northvale.location == "Raleigh, North Carolina"+ assert northvale.rating == 4.4 and northvale.review_count == 512+ assert northvale.has_website is True+ assert northvale.external_id == "dataforseo:ChIJabc123"++ # The websiteless record is the valuable one — it must survive intact.+ assert nosite.domain == "" and nosite.has_website is False+ assert nosite.is_claimed is False+ assert nosite.identity() == "dataforseo:ChIJxyz789"+++async def test_dataforseo_serp_adapter_reads_rank_group_not_position():+ """`position` in this payload means left/right column, not rank. Reading+ it as rank would score every result identically."""+ provider = D.DataForSEOSerp()+ async with _client(lambda r: httpx.Response(200, json=DFS_SERP)) as client:+ out = await provider.fetch(+ client, {"dataforseo_login": "u", "dataforseo_password": "p"},+ D.DiscoveryQuery(term="roofer", location="Denver", depth=30))++ assert [b.rank for b in out] == [1, 14], "paid and PAA rows must be dropped"+ assert out[1].domain == "kettlemanroofing.com"+++async def test_serper_adapter():+ provider = D.Serper()+ async with _client(lambda r: httpx.Response(200, json=SERPER_RESPONSE)) as client:+ out = await provider.fetch(client, {"serper_api_key": "k"},+ D.DiscoveryQuery(term="roofer", location="Denver"))+ assert out[0].domain == "kettlemanroofing.com" and out[0].rank == 3+ # Junk filtering happens in the run, so the adapter still returns Angi.+ assert out[1].domain == "angi.com"+++async def test_dataforseo_errors_surface_the_message_not_just_the_code():+ """A bare status code costs several wrong guesses; the body usually makes+ the problem obvious."""+ payload = {"status_code": 40501, "status_message": "Invalid Field: 'categories'"}+ provider = D.DataForSEOListings()+ with pytest.raises(RuntimeError, match="Invalid Field"):+ async with _client(lambda r: httpx.Response(200, json=payload)) as client:+ await provider.fetch(client, {"dataforseo_login": "u",+ "dataforseo_password": "p"},+ D.DiscoveryQuery(term="roofer"))+++def test_dataforseo_sandbox_is_never_the_live_host():+ provider = D.DataForSEOListings()+ assert "sandbox" in provider.base_url({"dataforseo_sandbox": "1"})+ assert "sandbox" not in provider.base_url({})+++# ── The provider menu the dashboard renders ──+++def test_provider_menu_explains_cost_and_configuration():+ menu = {m["key"]: m for m in D.provider_menu({"serper_api_key": "k"})}+ assert menu["osm"]["configured"] is True and not menu["osm"]["needs_key"]+ assert menu["serper"]["configured"] is True+ assert menu["dataforseo_listings"]["configured"] is False+ assert menu["dataforseo_listings"]["env_keys"] == [+ "DATAFORSEO_LOGIN", "DATAFORSEO_PASSWORD"]+ for entry in menu.values():+ assert entry["blurb"] and entry["cost_note"] and entry["signup_url"]+++def test_the_default_provider_needs_no_account():+ """Someone should be able to run the whole pipeline before paying anyone."""+ default = D.PROVIDERS[D.DEFAULT_PROVIDER]+ assert default.env_keys == ()+ assert default.caveat, "a free source with thin coverage must say so"+++async def test_unconfigured_provider_says_which_key_is_missing(state, config):+ report = await D.run_discovery(state, config, "dataforseo_listings",+ [D.DiscoveryQuery(term="roofer")])+ assert report.stopped == "not configured"+ assert "DATAFORSEO_LOGIN" in report.errors[0]+++def test_unknown_provider_fails_loudly():+ with pytest.raises(ValueError, match="unknown discovery provider"):+ import asyncio+ asyncio.run(D.run_discovery(None, None, "nope"))+++async def test_overpass_throttling_is_an_error_not_an_empty_market(monkeypatch):+ """A silent [] on a 504 is indistinguishable from 'no businesses here',+ which is how a throttled run gets mistaken for a clean one."""+ monkeypatch.setattr(D.OpenStreetMap, "BACKOFF", 0.0)+ calls = {"n": 0}++ def handler(request):+ calls["n"] += 1+ return httpx.Response(504, text="")++ provider = D.OpenStreetMap()+ with pytest.raises(RuntimeError, match="throttling"):+ async with _client(handler) as client:+ await provider.fetch(client, {}, D.DiscoveryQuery(+ term="roofer", coordinate="39.7,-104.9,40"))+ assert calls["n"] == 2, "it should retry once before giving up"+++async def test_overpass_recovers_on_the_retry(monkeypatch):+ monkeypatch.setattr(D.OpenStreetMap, "BACKOFF", 0.0)+ calls = {"n": 0}++ def handler(request):+ calls["n"] += 1+ if calls["n"] == 1:+ return httpx.Response(504, text="")+ return httpx.Response(200, json={"elements": [{+ "type": "node", "id": 42,+ "tags": {"name": "Sandpiper Exteriors", "craft": "roofer",+ "phone": "+15550100288", "addr:city": "Denver"},+ }]})++ provider = D.OpenStreetMap()+ async with _client(handler) as client:+ out = await provider.fetch(client, {}, D.DiscoveryQuery(+ term="roofer", coordinate="39.7,-104.9,40"))+ assert len(out) == 1+ assert out[0].name == "Sandpiper Exteriors"+ assert out[0].external_id == "osm:node/42"+ assert out[0].has_website is False+++async def test_overpass_needs_a_coordinate_and_says_so(caplog):+ provider = D.OpenStreetMap()+ async with _client(lambda r: httpx.Response(200, json={})) as client:+ out = await provider.fetch(client, {}, D.DiscoveryQuery(+ term="roofer", location="Denver, CO"))+ assert out == []+ assert "coordinate" in caplog.text.lower()+++# ── Geocoding ──+++def _geo_client(handler):+ return httpx.AsyncClient(transport=httpx.MockTransport(handler))+++async def test_geocode_is_cached_so_a_city_is_looked_up_once(state):+ """Nominatim permits single place lookups and forbids bulk enumeration.+ Caching forever is what keeps us on the right side of that."""+ calls = {"n": 0}++ def handler(request):+ calls["n"] += 1+ assert "Harvey" in request.headers["user-agent"]+ return httpx.Response(200, json=[{"lat": "39.7392", "lon": "-104.9903"}])++ async with _geo_client(handler) as client:+ assert await REAL_GEOCODE(state, "Denver, CO", client=client) \+ == "39.7392,-104.9903,40"+ assert await REAL_GEOCODE(state, "Denver, CO", client=client) \+ == "39.7392,-104.9903,40"+ assert await REAL_GEOCODE(state, "DENVER, co", client=client) \+ == "39.7392,-104.9903,40"+ assert calls["n"] == 1+++async def test_a_geocode_miss_is_cached_too(state):+ calls = {"n": 0}++ def handler(request):+ calls["n"] += 1+ return httpx.Response(200, json=[])++ async with _geo_client(handler) as client:+ assert await REAL_GEOCODE(state, "Nowhere XYZ", client=client) == ""+ assert await REAL_GEOCODE(state, "Nowhere XYZ", client=client) == ""+ assert calls["n"] == 1, "a miss must not be re-asked on every run"+++async def test_configured_coordinates_win_over_geocoding(state, monkeypatch):+ async def boom(*a, **kw):+ raise AssertionError("should not geocode a query that already has a point")+ monkeypatch.setattr(D, "geocode_city", boom)++ queries = [D.DiscoveryQuery(term="roofer", location="Denver, CO",+ coordinate="1.0,2.0,25")]+ out = await REAL_RESOLVE(state, queries)+ assert out[0].coordinate == "1.0,2.0,25"+++async def test_one_geocode_per_city_not_per_query(state, monkeypatch):+ """Twelve queries across four cities is four lookups, not twelve."""+ monkeypatch.setattr(D, "geocode_city", REAL_GEOCODE)+ calls = {"n": 0}++ def handler(request):+ calls["n"] += 1+ return httpx.Response(200, json=[{"lat": "1.0", "lon": "2.0"}])++ queries = [D.DiscoveryQuery(term=t, location=c)+ for t in ("roofing", "gutters", "siding")+ for c in ("Denver, CO", "Dallas, TX", "Tampa, FL", "Nashville, TN")]+ async with _geo_client(handler) as client:+ await REAL_RESOLVE(state, queries, client=client)+ assert calls["n"] == 4+ assert all(q.coordinate == "1.0000,2.0000,40" for q in queries)+++async def test_osm_refuses_a_term_it_cannot_map_instead_of_guessing():+ """An earlier fallback to `office=company` answered "roofers in Denver"+ with every registered office in Denver. Returning nothing is correct;+ returning plausible junk is not."""+ provider = D.OpenStreetMap()+ with pytest.raises(RuntimeError, match="no category"):+ async with _client(lambda r: httpx.Response(200, json={})) as client:+ await provider.fetch(client, {}, D.DiscoveryQuery(+ term="storm damage restoration", coordinate="39.7,-104.9,40"))+++async def test_osm_maps_the_trades_it_does_know():+ provider = D.OpenStreetMap()+ assert provider._tags("Roofing contractors") == ["craft=roofer", "shop=roofing"]+ assert provider._tags("Residential roofing") == ["craft=roofer", "shop=roofing"]+ assert provider._tags("Emergency plumber") == ["craft=plumber"]+ assert provider._tags("cosmetic dentist") == ["amenity=dentist"]+ assert provider._tags("B2B SaaS") == []+++async def test_an_unmappable_term_is_reported_not_silently_skipped(+ state, config, register, monkeypatch):+ """The run must surface why a query produced nothing."""+ monkeypatch.setitem(D.PROVIDERS, "osm", D.OpenStreetMap())++ async def fake_fetch(self, client, env, query):+ raise RuntimeError("OpenStreetMap has no category for 'widgets'")+ monkeypatch.setattr(D.OpenStreetMap, "fetch", fake_fetch)++ report = await D.run_discovery(state, config, "osm",+ [D.DiscoveryQuery(term="widgets",+ coordinate="1,2,40")])+ assert report.found == 0+ assert report.errors and "no category" in report.errors[0]