Skip to content

Autonomous AI sales agent powered by Claude Code or Codex. Finds prospects, writes cold emails, sends campaigns, handles replies, books meetings. No API keys needed — runs on your Claude or Codex subscriptions.

DISCOVER: a provider menu, a cost estimate, and a free default

Ethan6ca7e551f89d
diff --git a/.env.example b/.env.example
index cd00ed3..3c5b462 100644
--- a/.env.example
+++ b/.env.example
@@ -82,11 +82,29 @@ ZEROBOUNCE_API_KEY=
HUNTER_API_KEY=
# ------------------------------------------------------------
-# Serper.dev — Google Search API for prospecting [OPTIONAL, recommended]
+# DISCOVERY — where Harvey finds businesses. [ALL OPTIONAL]
+# The default source (OpenStreetMap) needs no account at all:
+# harvey discover
+# See every option, with real prices, via:
+# harvey discover --providers (or the dashboard's Discover tab)
+# ------------------------------------------------------------
+
+# DataForSEO — cheapest local-business and SERP data anywhere.
+# Where to get it: https://app.dataforseo.com/api-access
+# NOTE: the API password is auto-generated there. It is NOT your account
+# password. $1 free credit on signup; $50 minimum deposit to go live.
+# Cost: $0.372 per 1,000 businesses; $0.0018 per depth-30 search.
+DATAFORSEO_LOGIN=
+DATAFORSEO_PASSWORD=
+# Set to any value to route every call to DataForSEO's free sandbox. Same
+# response shape, dummy data, never charged — the right way to try it.
+DATAFORSEO_SANDBOX=
+
+# Serper.dev — Google Search API, used for discovery and for Scout's searches.
# Where to get it: https://serper.dev → sign up → Dashboard → API Key
-# Cost: ~$5 for 2,500 searches (no subscription). The most reliable search
-# backend; without it Harvey falls back to DuckDuckGo/Bing scraping,
-# which can be rate-limited.
+# Cost: 2,500 queries free with no card, then a $50 prepaid pack (credits
+# expire after 6 months). Without it Harvey falls back to DuckDuckGo/Bing
+# scraping, which is heavily rate-limited.
SERPER_API_KEY=
# ------------------------------------------------------------
diff --git a/CLAUDE.md b/CLAUDE.md
index 2c9c7d6..d6258ee 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -86,15 +86,19 @@ REOON_API_KEY= # 600 free/mo — best free tier
ZEROBOUNCE_API_KEY= # 100 free/mo — best for M365/Workspace catch-alls
HUNTER_API_KEY= # 50 free/mo + pattern lookup
+# --- Discovery (optional — the free OpenStreetMap source needs no key) ---
+DATAFORSEO_LOGIN= DATAFORSEO_PASSWORD= # cheapest local-business + SERP data
+DATAFORSEO_SANDBOX= # any value routes to the free sandbox
+
# --- Optional ---
LINKEDIN_EMAIL= LINKEDIN_PASSWORD= # LinkedIn prospecting
CLOUDFLARE_ACCOUNT_ID= CLOUDFLARE_API_TOKEN= # deep JS crawling for training
-SERPER_API_KEY= # reliable web search ($5/mo at serper.dev)
+SERPER_API_KEY= # web search + discovery (2,500 free, then a $50 prepaid pack)
```
**Recommended provider — Gmail:** for <50 cold emails/day, a real Google Workspace mailbox on a *dedicated secondary domain* (never the main one) is the most deliverable, cheapest (~$7/mo) option. Set `channels.email.provider: gmail` in harvey.yaml, put the OAuth client id/secret in `.env`, then run `harvey gmail auth` (one-time browser login). SMTP works with any mailbox (AgentMail, Fastmail). Instantly still works as a legacy option.
-**Email verification matters:** Harvey learns each company's email *pattern* and verifies ONE candidate rather than guessing (raw SMTP probing no longer works against Google Workspace / Microsoft 365). Without a verifier key, found emails are marked `guess` and are **never sent**.
+**Email verification matters:** Harvey learns each company's email *pattern* and verifies ONE candidate rather than guessing (raw SMTP probing is not viable from a locally-running agent — outbound port 25 is usually blocked, and Google Workspace / Microsoft 365 accept everything from an unknown IP). Without a verifier key, found emails are marked `guess` and are **never sent**.
**Approval by default:** with a native provider, every outgoing email waits in the **Outbox** for your approval (dashboard Outbox tab, or `harvey outbox`). Once you trust the output, set `channels.email.require_approval: false` in harvey.yaml for full autopilot.
@@ -159,7 +163,37 @@ Once signals are confirmed, the **cohort builder** at the bottom of the Signals
tab turns them into a target list: pick what a good prospect must have (and what
disqualifies them) and it counts the matches live.
-### Step 5: Behavior Settings
+### Step 5: Find Businesses
+
+Discovery is the only stage that spends money, so it always estimates first.
+
+```bash
+harvey discover --providers # the menu, with real prices
+harvey discover --estimate # projected spend, then exits
+harvey discover # free OpenStreetMap source
+harvey discover --provider dataforseo_listings --max-spend 2.00
+```
+
+The dashboard's **Discover** tab is the same thing with the tradeoffs laid out
+side by side: what each source does, what it costs, its free tier, and which
+`.env` keys it needs. Nothing is called until the user presses Run.
+
+| Source | Cost | Free tier | Best for |
+|---|---|---|---|
+| **OpenStreetMap** (default) | free | unlimited | Trying the whole pipeline with no account. Thin coverage — under 2,000 roofers in the entire US — so it's a trial source, not a complete list. |
+| **DataForSEO Business Listings** | $0.372/1k businesses | $1 credit | Local trades, clinics, contractors. Phone + domain + rating + whether the listing is claimed, and it can filter for businesses with **no website at all**. |
+| **DataForSEO SERP** | $0.0018 a search at depth 30 | $1 credit + free sandbox | Rank as the buying signal. Positions 11-30 are the sweet spot. |
+| **Serper** | ~$0.30-$1.00/1k | 2,500 free, no card | The easiest paid one to try. |
+
+Cities are geocoded automatically (cached forever, one lookup per city), or set
+`icp.geo_coordinates` in harvey.yaml to control the radius:
+`"Denver, CO": "39.7392,-104.9903,50"`.
+
+**Depth 20-30, not 100.** Google removed 100-results-per-page in September 2025,
+so depth 100 is now billed as ten pages at nearly every provider. Below rank 30
+it is mostly directories anyway.
+
+### Step 6: Behavior Settings
These go in `harvey.yaml` under `usage:`. Use sensible defaults unless they want to customize:
- `max_daily_claude_percent`: 80 (how much of daily Claude quota to use)
@@ -167,7 +201,7 @@ These go in `harvey.yaml` under `usage:`. Use sensible defaults unless they want
- `quiet_hours`: 22:00-07:00 in their timezone
- `max_daily_sends`: 50 (email send limit)
-### Step 6: Start Harvey
+### Step 7: Start Harvey
```bash
source .venv/bin/activate && harvey run
@@ -204,6 +238,7 @@ Users will come back with questions and tasks. Common ones:
- **Skills** (`skills/`): Markdown knowledge files injected into agent prompts
- **Signals** (`signals.py`, `state.py`): every fact Harvey learns is an OBSERVATION — a row in `observations`, never a column. A new signal needs no migration, re-observing over time is a free time series, and confidence + provenance travel with the fact. The vocabulary in `signal_codes` is governed (a trigger rejects any code not in it) and gated: only `status = 'confirmed'` signals are collected, and only a human sets that. `state.cohort(require, exclude)` does the set intersection in SQL.
- **Dashboard** (`dashboard.py` + `harvey/web/`): FastAPI JSON API plus plain HTML/CSS/JS served from disk — no build step. Edit `harvey/web/app.css` or `app.js` and reload the page.
+- **DISCOVER** (`collectors/discover.py`): the only stage that spends money, so cost discipline lives in the collector, not the database — every run estimates first, checks a spend cap between batches, reads a kill switch, and flushes observations per batch. Providers are adapters behind one interface (`DiscoveryProvider`); adding one is a small class. Junk filtering matches *patterns*, not a literal blocklist. Entity resolution is the normalised domain, falling back to the provider's `external_id` for businesses with no website.
### Sub-Agents
- **Scout**: Python does all web searching (DuckDuckGo → Bing → Google → Serper API) and email resolution (pattern-first: cache → scraped mailto → Hunter domain search → default, then verify one candidate via Reoon/ZeroBounce/Hunter/SMTP; catch-alls flagged `risky`). Claude only scores/personalizes found data. Scout also collects **buying signals**: tech stack detected on each company's site (HubSpot, Shopify, Intercom, ~35 tools — zero extra requests) and hiring signals from careers pages. With `pip install python-jobspy` (optional), a job-board strategy discovers companies actively hiring for roles in `icp.hiring_signals` (falls back to `icp.titles`) — the strongest in-market signal. Signals land in prospects' personalization notes and boost their score.
@@ -233,6 +268,7 @@ harvey gmail auth # One-time Gmail OAuth (when provider: gmail)
harvey gmail test # Verify the Gmail connection
harvey outbox # Review queued emails; --approve <id> / --approve-all / --reject <id>
harvey signals # The signal vocabulary; --confirm / --reject CODES (or 'free' / 'all')
+harvey discover # Find businesses; --providers / --estimate / --provider <key>
harvey sending pause|resume # Kill switch for all outbound
```
diff --git a/harvey/cli.py b/harvey/cli.py
index 20dfd3a..01c342c 100644
--- a/harvey/cli.py
+++ b/harvey/cli.py
@@ -320,6 +320,79 @@ def cmd_outbox(args):
asyncio.run(_outbox())
+def cmd_discover(args):
+ """Find businesses. The only stage that spends money, so it estimates first."""
+ from harvey.state import StateManager
+ from harvey.config import load_config
+ from harvey.collectors.discover import (
+ PROVIDERS, build_queries, provider_menu, run_discovery,
+ )
+
+ async def _discover():
+ if args.providers:
+ print("\n Discovery providers")
+ print(" " + "=" * 68)
+ for m in provider_menu():
+ mark = "ready" if m["configured"] else "needs setup"
+ print(f"\n {m['label']} [{m['key']}] ({mark})")
+ print(f" {m['blurb']}")
+ print(f" Cost: {m['cost_note']}")
+ print(f" Free: {m['free_tier']}")
+ if m["env_keys"]:
+ print(f" Needs: {', '.join(m['env_keys'])} in .env — {m['signup_url']}")
+ if m["caveat"]:
+ print(f" Note: {m['caveat']}")
+ print("\n Run one with: harvey discover --provider <key> --estimate\n")
+ return
+
+ if args.provider not in PROVIDERS:
+ print(f"\n Unknown provider {args.provider!r}. "
+ f"See: harvey discover --providers\n")
+ return
+
+ config = load_config()
+ state = StateManager()
+ await state.init_db()
+
+ # Semicolons, not commas: "Denver, CO" is one city, not two.
+ cities = [c.strip() for c in args.city.split(";") if c.strip()] or None
+ queries = build_queries(config, cities=cities,
+ depth=args.depth, limit=args.limit)
+
+ report = await run_discovery(state, config, args.provider, queries,
+ max_spend=args.max_spend, dry_run=True)
+ print(f"\n {len(queries)} queries via {PROVIDERS[args.provider].label}")
+ for q in queries[:8]:
+ print(f" - {q.keyword()}" + ("" if q.coordinate else
+ " (no coordinates — add icp.geo_coordinates for radius search)"))
+ if len(queries) > 8:
+ print(f" ... and {len(queries) - 8} more")
+ print(f"\n Estimated cost: ${report.estimated_cost:.4f}"
+ f" (cap: ${args.max_spend:.2f})")
+
+ if args.estimate:
+ print("\n Estimate only. Re-run without --estimate to collect.\n")
+ return
+
+ print("\n Running...\n")
+ report = await run_discovery(state, config, args.provider, queries,
+ max_spend=args.max_spend)
+
+ r = report
+ print(f" Found {r.found} results across {r.queries} queries")
+ print(f" {r.new_companies} new companies, {r.known_companies} already known")
+ print(f" {r.junk} filtered out as directories/aggregators")
+ print(f" {r.observations} observations recorded")
+ print(f" actual cost: ${r.actual_cost:.4f}")
+ if r.stopped:
+ print(f"\n Stopped early: {r.stopped}")
+ for err in r.errors[:5]:
+ print(f" ! {err}")
+ print("\n Next: harvey dashboard → Signals → cohort builder\n")
+
+ asyncio.run(_discover())
+
+
def cmd_signals(args):
"""Review the signal vocabulary — Harvey proposes, you confirm.
@@ -476,6 +549,24 @@ def main():
sub.set_defaults(func=cmd_outbox)
# harvey sending pause|resume|status
+ sub = subparsers.add_parser("discover", help="Find businesses matching your ICP")
+ sub.add_argument("--provider", default="osm",
+ help="Discovery provider (default: osm — free, no account)")
+ sub.add_argument("--providers", action="store_true",
+ help="List every provider with cost and setup, then exit")
+ sub.add_argument("--estimate", action="store_true",
+ help="Print projected spend and exit without calling anything")
+ sub.add_argument("--city", default="",
+ help='Semicolon-separated cities, e.g. "Denver, CO;Dallas, TX" '
+ '(default: icp.geography)')
+ sub.add_argument("--depth", type=int, default=30,
+ help="SERP depth (default: 30 — depth 100 costs 10x since Sept 2025)")
+ sub.add_argument("--limit", type=int, default=100,
+ help="Max records per query (default: 100)")
+ sub.add_argument("--max-spend", type=float, default=1.0,
+ help="Hard cap in dollars (default: 1.00)")
+ sub.set_defaults(func=cmd_discover)
+
sub = subparsers.add_parser(
"signals", help="Review/confirm which signals Harvey prospects against")
sub.add_argument("--confirm", metavar="CODES", default="",
diff --git a/harvey/collectors/discover.py b/harvey/collectors/discover.py
new file mode 100644
index 0000000..3531e8f
--- /dev/null
+++ b/harvey/collectors/discover.py
@@ -0,0 +1,864 @@
+"""DISCOVER — who exists, and how visible they are.
+
+The only stage that costs money, so it is the one with a provider menu, a
+mandatory cost estimate, and a budget check inside the loop.
+
+Two shapes of discovery, and the difference matters:
+
+* **Listings** ask "which businesses of this type are in this place?" and come
+ back with a phone, a category, a rating and — crucially — whether there is a
+ website at all. For local SMB verticals this is the primary path.
+* **SERP** asks "who ranks for this search?" and comes back with a rank
+ position. You are buying the *signal*, not the list: positions 11-30 are
+ businesses visible enough to be trying and not winning, which is the most
+ useful prospecting signal there is.
+
+Cost discipline lives here, not in the database. A table can only record spend
+after the fact; only the collector can decline to spend it. Every run
+estimates first, checks a budget between batches, reads the kill switch, and
+flushes observations per batch — a long run that dies must not lose what it
+already learned.
+"""
+
+import asyncio
+import base64
+import logging
+import re
+from dataclasses import dataclass, field
+from urllib.parse import urlparse
+
+import httpx
+
+from harvey.models import Company
+
+logger = logging.getLogger("harvey.collectors.discover")
+
+TIMEOUT = httpx.Timeout(connect=10.0, read=45.0, write=10.0, pool=10.0)
+
+
+# ── Junk filtering ────────────────────────────────────────────────────
+# Discovery fills a database with directories, hospital systems, media and
+# lead-gen middlemen unless you filter hard. Match PATTERNS, not a literal
+# list: a literal list is exactly how `whatclinic.com` and `ufhealth.org` get
+# through on the first run.
+
+# Anchored on a dot boundary, not the string start: SERP results arrive as
+# `en.wikipedia.org` and `m.facebook.com` at least as often as bare domains,
+# and `^(www\.)?` catches neither.
+JUNK_DOMAIN_PATTERNS = tuple(re.compile(p, re.I) for p in (
+ # directories & marketplaces
+ r"(^|\.)(yelp|yellowpages|superpages|manta|bbb|angi|angieslist|homeadvisor)\.",
+ r"(^|\.)(thumbtack|houzz|porch|buildzoom|networx|modernize|nextdoor)\.",
+ r"(^|\.)(facebook|instagram|twitter|x|linkedin|youtube|tiktok|pinterest)\.",
+ r"(^|\.)(indeed|glassdoor|ziprecruiter|craigslist|zillow|realtor)\.",
+ r"(^|\.)(mapquest|foursquare|tripadvisor|trustpilot|birdeye|expertise)\.",
+ r"(directory|listings?|find-?a-|top\d+|best-?of)\.",
+ # media & reference
+ r"(^|\.)(wikipedia|reddit|quora|medium|substack|forbes|inc|entrepreneur)\.",
+ r"(news|magazine|journal|gazette|tribune|herald)\.(com|org|net)$",
+ # institutional
+ r"\.(edu|gov|mil)$",
+ r"(hospital|healthsystem|medicalcenter)\.",
+ r"health\.org$",
+ # infrastructure that is never a prospect
+ r"(^|\.)(google|bing|yahoo|duckduckgo|amazon|apple|microsoft)\.",
+ r"(wordpress|wix|squarespace|shopify|godaddy|weebly|webflow)\.com$",
+))
+
+# Names that mean the row is a category page or an aggregator, not a business.
+JUNK_NAME_PATTERNS = tuple(re.compile(p, re.I) for p in (
+ r"^\s*(top|best|\d+)\s+\d*\s*(best\s+)?\w+.*\b(in|near|of)\b",
+ r"\b(directory|listings?|reviews? of|compare|find a)\b",
+))
+
+
+def normalize_domain(url_or_domain: str) -> str:
+ """A bare, lowercase, www-less hostname — the entity-resolution key.
+
+ Entity resolution decides whether any of this is real. Get it wrong and
+ every trend built on top is garbage.
+ """
+ if not url_or_domain:
+ return ""
+ raw = url_or_domain.strip().lower()
+ if "://" not in raw:
+ raw = "http://" + raw
+ host = (urlparse(raw).hostname or "").strip(".")
+ return host[4:] if host.startswith("www.") else host
+
+
+def is_junk(domain: str, name: str = "") -> bool:
+ """True when this row would pollute the database rather than fill it."""
+ if domain:
+ if any(p.search(domain) for p in JUNK_DOMAIN_PATTERNS):
+ return True
+ # A bare TLD or an IP address is never a business we can profile.
+ if "." not in domain or re.fullmatch(r"[\d.]+", domain):
+ return True
+ if name and any(p.search(name) for p in JUNK_NAME_PATTERNS):
+ return True
+ return False
+
+
+# ── Geocoding ─────────────────────────────────────────────────────────
+# Listings providers search a radius, not a place name, so "Denver, CO" has to
+# become coordinates. Nominatim allows exactly this (single place lookups at
+# <=1 req/s) and forbids bulk enumeration, so every result is cached forever in
+# the settings table: a given city is looked up once, ever.
+
+NOMINATIM = "https://nominatim.openstreetmap.org/search"
+GEOCODE_UA = "Harvey/0.1 (open-source sales agent; github.com/ethanplusai/harvey)"
+
+
+async def geocode_city(state, city: str, radius_km: int = 40,
+ client: httpx.AsyncClient | None = None) -> str:
+ """"Denver, CO" -> "39.7392,-104.9903,40". Cached permanently."""
+ if not city:
+ return ""
+ cache_key = f"geo:{city.strip().lower()}"
+ cached = await state.get_setting(cache_key)
+ if cached:
+ # A previous miss is cached as "-" so we don't re-ask every run.
+ return "" if cached == "-" else f"{cached},{radius_km}"
+
+ owned = client is None
+ client = client or httpx.AsyncClient(timeout=TIMEOUT)
+ try:
+ r = await client.get(
+ NOMINATIM,
+ params={"q": city, "format": "json", "limit": 1},
+ headers={"User-Agent": GEOCODE_UA},
+ )
+ r.raise_for_status()
+ hits = r.json()
+ except Exception as e:
+ logger.warning("geocode failed for %r: %s", city, e)
+ return ""
+ finally:
+ if owned:
+ await client.aclose()
+
+ if not hits:
+ await state.set_setting(cache_key, "-")
+ logger.warning("geocode: no match for %r", city)
+ return ""
+
+ point = f"{float(hits[0]['lat']):.4f},{float(hits[0]['lon']):.4f}"
+ await state.set_setting(cache_key, point)
+ return f"{point},{radius_km}"
+
+
+async def resolve_coordinates(state, queries: list["DiscoveryQuery"],
+ radius_km: int = 40,
+ client: httpx.AsyncClient | None = None
+ ) -> list["DiscoveryQuery"]:
+ """Fill in coordinates for any query that lacks them.
+
+ Configured ``icp.geo_coordinates`` always wins; this only covers what the
+ user did not spell out. Distinct cities are geocoded once each, however
+ many queries share them.
+ """
+ needed = {q.location for q in queries if q.location and not q.coordinate}
+ points: dict[str, str] = {}
+ for i, city in enumerate(sorted(needed)):
+ if i:
+ await asyncio.sleep(1.1) # Nominatim: max 1 request/second
+ points[city] = await geocode_city(state, city, radius_km, client)
+
+ for query in queries:
+ if not query.coordinate:
+ query.coordinate = points.get(query.location, "")
+ return queries
+
+
+# ── The record every provider returns ─────────────────────────────────
+
+
+@dataclass
+class Business:
+ """One discovered business, normalised across providers."""
+
+ name: str = ""
+ domain: str = ""
+ website: str = ""
+ phone: str = ""
+ address: str = ""
+ location: str = ""
+ category: str = ""
+ rating: float | None = None
+ review_count: int | None = None
+ is_claimed: bool | None = None
+ has_website: bool | None = None
+ rank: int | None = None # organic rank, when the provider gives one
+ external_id: str = "" # "<provider>:<stable id>"
+ source_url: str = ""
+ provider: str = ""
+
+ def identity(self) -> str:
+ return self.domain or self.external_id
+
+
+@dataclass
+class DiscoveryQuery:
+ """What to look for, and where."""
+
+ term: str # "roofing contractor"
+ location: str = "" # "Denver, CO"
+ category: str = "" # provider-native category slug, optional
+ coordinate: str = "" # "lat,lng,radius_km" for listings providers
+ depth: int = 30 # SERP depth; ignored by listings providers
+ limit: int = 100 # max records to take from one query
+
+ def keyword(self) -> str:
+ return f"{self.term} {self.location}".strip()
+
+
+# ── Provider adapters ─────────────────────────────────────────────────
+
+
+class DiscoveryProvider:
+ """One way of answering "who exists".
+
+ Subclasses implement ``fetch``. Everything else — the menu metadata the
+ dashboard renders, cost estimation, configuration checks — lives here so a
+ new provider is a small class rather than a wiring exercise.
+ """
+
+ key: str = ""
+ label: str = ""
+ kind: str = "listings" # listings | serp
+ blurb: str = ""
+ cost_note: str = ""
+ free_tier: str = ""
+ signup_url: str = ""
+ env_keys: tuple[str, ...] = ()
+ caveat: str = ""
+
+ def configured(self, env: dict) -> bool:
+ return all(env.get(k) for k in self.env_keys)
+
+ def estimate(self, queries: list[DiscoveryQuery]) -> float:
+ """Projected spend, in dollars, before anything is called."""
+ return 0.0
+
+ async def fetch(self, client: httpx.AsyncClient, env: dict,
+ query: DiscoveryQuery) -> list[Business]:
+ raise NotImplementedError
+
+ # Shared helper: providers report their own per-query spend so the run log
+ # records what actually happened, not what was projected.
+ last_cost: float = 0.0
+
+
+class DataForSEOListings(DiscoveryProvider):
+ """Google Business Profile data as a queryable database, not a scrape.
+
+ The cheapest way to enumerate local businesses anywhere, and the only one
+ with a filter for "has no website" — which for anyone selling websites is
+ the highest-intent query available.
+ """
+
+ key = "dataforseo_listings"
+ label = "DataForSEO Business Listings"
+ kind = "listings"
+ blurb = ("Enumerate every business of a type within a radius, with phone, "
+ "domain, rating and whether the listing is claimed. Best choice "
+ "for local trades, clinics and contractors.")
+ cost_note = "$0.372 per 1,000 businesses ($0.012/query + $0.00036 each)"
+ free_tier = "$1 signup credit; $50 minimum deposit to go live"
+ signup_url = "https://app.dataforseo.com/api-access"
+ env_keys = ("dataforseo_login", "dataforseo_password")
+ caveat = "No email addresses — Harvey resolves those in a later stage."
+
+ TASK_COST = 0.012
+ ITEM_COST = 0.00036
+
+ def estimate(self, queries):
+ return sum(self.TASK_COST + self.ITEM_COST * q.limit for q in queries)
+
+ def _auth(self, env: dict) -> str:
+ raw = f"{env['dataforseo_login']}:{env['dataforseo_password']}"
+ return base64.b64encode(raw.encode()).decode()
+
+ def base_url(self, env: dict) -> str:
+ # The sandbox mirrors the response schema exactly and is never
+ # charged — the right target for a first run and for the test suite.
+ return ("https://sandbox.dataforseo.com/v3"
+ if env.get("dataforseo_sandbox") else
+ "https://api.dataforseo.com/v3")
+
+ async def fetch(self, client, env, query):
+ payload = {
+ "limit": min(query.limit, 1000),
+ "order_by": ["rating.votes_count,desc"],
+ }
+ if query.category:
+ payload["categories"] = [query.category]
+ else:
+ payload["title"] = query.term
+ if query.coordinate:
+ payload["location_coordinate"] = query.coordinate
+ elif query.location:
+ # Without coordinates the API can only match on the title, so say
+ # so rather than silently returning nationwide results.
+ payload["description"] = query.location
+
+ r = await client.post(
+ f"{self.base_url(env)}/business_data/business_listings/search/live",
+ headers={"Authorization": f"Basic {self._auth(env)}",
+ "Content-Type": "application/json"},
+ json=[payload],
+ )
+ r.raise_for_status()
+ data = r.json()
+ self.last_cost = float(data.get("cost") or 0.0)
+ return [self._to_business(item) for item in _dfs_items(data)]
+
+ def _to_business(self, item: dict) -> Business:
+ url = item.get("url") or ""
+ rating = (item.get("rating") or {})
+ addr = item.get("address_info") or {}
+ city = ", ".join(x for x in (addr.get("city"), addr.get("region")) if x)
+ return Business(
+ name=item.get("title") or "",
+ domain=normalize_domain(item.get("domain") or url),
+ website=url,
+ phone=item.get("phone") or "",
+ address=item.get("address") or "",
+ location=city,
+ category=item.get("category") or "",
+ rating=rating.get("value"),
+ review_count=rating.get("votes_count"),
+ is_claimed=item.get("is_claimed"),
+ has_website=bool(url),
+ external_id=f"dataforseo:{item['place_id']}" if item.get("place_id") else "",
+ source_url=item.get("check_url") or "",
+ provider=self.key,
+ )
+
+
+class DataForSEOSerp(DiscoveryProvider):
+ """Organic rank as the buying signal.
+
+ Buy the rank, not the list. A business sitting at 11-30 for its own core
+ search is visible enough to have tried and is losing — the single most
+ useful thing you can know before writing to them.
+ """
+
+ key = "dataforseo_serp"
+ label = "DataForSEO SERP (organic rank)"
+ kind = "serp"
+ blurb = ("Search '<service> <city>' and record where each business ranks. "
+ "Discovers businesses you didn't have AND scores the ones you did.")
+ cost_note = "$0.0006 per 10 results — depth 30 costs $0.0018 a search"
+ free_tier = "$1 signup credit; free sandbox; $50 minimum deposit to go live"
+ signup_url = "https://app.dataforseo.com/api-access"
+ env_keys = ("dataforseo_login", "dataforseo_password")
+ caveat = ("Google removed 100-results-per-page in September 2025, so depth "
+ "100 now costs 10x depth 10. Depth 20-30 is the sweet spot; "
+ "below rank 30 it is mostly directories anyway.")
+
+ PAGE_COST = 0.0006
+
+ def estimate(self, queries):
+ # Billing is base price x number of pages, 10 results to a page.
+ return sum(self.PAGE_COST * max(1, -(-q.depth // 10)) for q in queries)
+
+ _auth = DataForSEOListings._auth
+ base_url = DataForSEOListings.base_url
+
+ async def fetch(self, client, env, query):
+ r = await client.post(
+ f"{self.base_url(env)}/serp/google/organic/live/advanced",
+ headers={"Authorization": f"Basic {self._auth(env)}",
+ "Content-Type": "application/json"},
+ json=[{"keyword": query.keyword(), "language_code": "en",
+ "depth": query.depth, "device": "desktop"}],
+ )
+ r.raise_for_status()
+ data = r.json()
+ self.last_cost = float(data.get("cost") or 0.0)
+
+ out = []
+ for item in _dfs_items(data):
+ if item.get("type") != "organic":
+ continue
+ domain = normalize_domain(item.get("domain") or item.get("url") or "")
+ if not domain:
+ continue
+ out.append(Business(
+ name=item.get("title") or domain,
+ domain=domain,
+ website=item.get("url") or "",
+ location=query.location,
+ # rank_group is position within organic results.
+ # `position` in this payload means left/right column, NOT rank.
+ rank=item.get("rank_group"),
+ has_website=True,
+ source_url=item.get("url") or "",
+ provider=self.key,
+ ))
+ return out
+
+
+class Serper(DiscoveryProvider):
+ """Google search with the friendliest free tier and the best depth pricing.
+
+ The one provider that did not go to 10x for depth 100 when Google removed
+ 100-results-per-page, and 2,500 free queries with no card.
+ """
+
+ key = "serper"
+ label = "Serper"
+ kind = "serp"
+ blurb = ("Fast Google search with rank positions. The easiest one to try: "
+ "2,500 free queries, no credit card, one API key.")
+ cost_note = "~$0.30-$1.00 per 1,000 searches depending on plan"
+ free_tier = "2,500 queries free, no card required"
+ signup_url = "https://serper.dev"
+ env_keys = ("serper_api_key",)
+ caveat = "Paid plans start at a $50 prepaid pack; credits expire after 6 months."
+
+ def estimate(self, queries):
+ # One credit per 10 results on the standard plan.
+ return sum(0.001 * max(1, -(-q.depth // 10)) for q in queries)
+
+ async def fetch(self, client, env, query):
+ r = await client.post(
+ "https://google.serper.dev/search",
+ headers={"X-API-KEY": env["serper_api_key"],
+ "Content-Type": "application/json"},
+ json={"q": query.keyword(), "num": min(query.depth, 100)},
+ )
+ r.raise_for_status()
+ data = r.json()
+ out = []
+ for item in data.get("organic") or []:
+ domain = normalize_domain(item.get("link") or "")
+ if not domain:
+ continue
+ out.append(Business(
+ name=item.get("title") or domain,
+ domain=domain,
+ website=item.get("link") or "",
+ location=query.location,
+ rank=item.get("position"),
+ has_website=True,
+ source_url=item.get("link") or "",
+ provider=self.key,
+ ))
+ return out
+
+
+class OpenStreetMap(DiscoveryProvider):
+ """Genuinely free, no account, no key. Honest about what that costs you.
+
+ Good enough to try the whole pipeline end to end without signing up for
+ anything, and a useful cross-reference layer afterwards. Not good enough
+ to be your only source: OSM maps physical premises, so service-area
+ businesses (roofers, HVAC, plumbers) are heavily under-represented.
+ """
+
+ key = "osm"
+ label = "OpenStreetMap (Overpass)"
+ kind = "listings"
+ blurb = ("Free public map data. No signup, no key, no card. Try the whole "
+ "pipeline for nothing before you pay anyone.")
+ cost_note = "free"
+ free_tier = "unlimited, within community rate etiquette"
+ signup_url = "https://www.openstreetmap.org"
+ env_keys = ()
+ caveat = ("Coverage is thin for businesses without a storefront — OSM has "
+ "under 2,000 roofers for the entire US. Great for a free trial "
+ "run or as a cross-reference; not a complete list.")
+
+ ENDPOINT = "https://overpass-api.de/api/interpreter"
+ # Overpass etiquette for a shipped tool: modest volume, no parallelism.
+ MIN_INTERVAL = 2.0
+ BACKOFF = 30.0
+
+ # A friendly term maps to the OSM tags that actually carry those businesses.
+ TAG_MAP = {
+ "roof": ['craft=roofer', 'shop=roofing'],
+ "plumb": ['craft=plumber'],
+ "hvac": ['craft=hvac'],
+ "electric": ['craft=electrician'],
+ "dentist": ['amenity=dentist'],
+ "lawyer": ['office=lawyer'],
+ "attorney": ['office=lawyer'],
+ "contractor": ['craft=builder', 'office=construction_company'],
+ "landscap": ['craft=gardener', 'shop=garden_centre'],
+ }
+
+ def _tags(self, term: str) -> list[str]:
+ """The OSM tags that actually carry this kind of business, or none.
+
+ There is deliberately no catch-all fallback. An earlier version fell
+ back to `office=company`, which answers "roofers in Denver" with every
+ registered office in Denver — coffee roasters, churches, photography
+ studios. A search this source cannot answer has to say so; filling the
+ database with plausible-looking junk is far worse than returning
+ nothing.
+ """
+ low = term.lower()
+ for needle, tags in self.TAG_MAP.items():
+ if needle in low:
+ return tags
+ return []
+
+ async def fetch(self, client, env, query):
+ if not query.coordinate:
+ logger.warning("osm: no coordinate for %r — Overpass needs a "
+ "lat,lng,radius to search", query.keyword())
+ return []
+ tags = self._tags(query.term)
+ if not tags:
+ raise RuntimeError(
+ f"OpenStreetMap has no category for {query.term!r}. It can only "
+ f"search mapped trades ({', '.join(sorted(self.TAG_MAP))}). Use a "
+ f"listings or SERP provider for this one.")
+ lat, lng, radius_km = (query.coordinate.split(",") + ["", "", ""])[:3]
+ radius_m = int(float(radius_km or 25) * 1000)
+ clauses = "".join(
+ f'node[{tag.split("=")[0]}={tag.split("=")[1]}](around:{radius_m},{lat},{lng});'
+ f'way[{tag.split("=")[0]}={tag.split("=")[1]}](around:{radius_m},{lat},{lng});'
+ for tag in tags
+ )
+ body = f"[out:json][timeout:40];({clauses});out center {query.limit};"
+
+ # Overpass is a free volunteer service and 504s readily under load —
+ # a cold first request very often does. Retry once with a real
+ # backoff, then RAISE: returning [] would make a throttled run look
+ # exactly like a market with no businesses in it.
+ headers = {"User-Agent": "Harvey/0.1 (open-source sales agent; "
+ "github.com/ethanplusai/harvey)"}
+ for attempt in range(2):
+ r = await client.post(self.ENDPOINT, content=body.encode(),
+ headers=headers)
+ if r.status_code not in (429, 504):
+ break
+ if attempt == 0:
+ logger.warning("osm: throttled (%s) — backing off %ss",
+ r.status_code, self.BACKOFF)
+ await asyncio.sleep(self.BACKOFF)
+ else:
+ raise RuntimeError(
+ f"Overpass is throttling (HTTP {r.status_code}). This is a free "
+ f"community service — wait a minute and re-run, or use a paid "
+ f"provider for bulk work.")
+ r.raise_for_status()
+
+ out = []
+ for el in r.json().get("elements") or []:
+ tags = el.get("tags") or {}
+ name = tags.get("name") or ""
+ if not name:
+ continue
+ site = tags.get("website") or tags.get("contact:website") or ""
+ city = ", ".join(x for x in (tags.get("addr:city"),
+ tags.get("addr:state")) if x)
+ out.append(Business(
+ name=name,
+ domain=normalize_domain(site),
+ website=site,
+ phone=tags.get("phone") or tags.get("contact:phone") or "",
+ address=" ".join(x for x in (tags.get("addr:housenumber"),
+ tags.get("addr:street")) if x),
+ location=city or query.location,
+ category=tags.get("craft") or tags.get("shop") or tags.get("office") or "",
+ has_website=bool(site),
+ external_id=f"osm:{el.get('type')}/{el.get('id')}",
+ source_url=f"https://www.openstreetmap.org/{el.get('type')}/{el.get('id')}",
+ provider=self.key,
+ ))
+ return out
+
+
+PROVIDERS: dict[str, DiscoveryProvider] = {
+ p.key: p for p in (
+ OpenStreetMap(),
+ DataForSEOListings(),
+ DataForSEOSerp(),
+ Serper(),
+ )
+}
+
+DEFAULT_PROVIDER = "osm"
+
+
+def _dfs_items(payload: dict) -> list[dict]:
+ """Pull result items out of a DataForSEO envelope, surfacing errors.
+
+ A bare status code costs three wrong guesses; the message in the body
+ usually makes the problem obvious immediately.
+ """
+ if payload.get("status_code") not in (20000, None):
+ raise RuntimeError(
+ f"DataForSEO {payload.get('status_code')}: "
+ f"{payload.get('status_message')}")
+ tasks = payload.get("tasks") or []
+ if not tasks:
+ return []
+ task = tasks[0]
+ if task.get("status_code") not in (20000, None):
+ raise RuntimeError(
+ f"DataForSEO task {task.get('status_code')}: "
+ f"{task.get('status_message')}")
+ results = task.get("result") or []
+ if not results or not results[0]:
+ return []
+ return results[0].get("items") or []
+
+
+# ── The run ───────────────────────────────────────────────────────────
+
+
+@dataclass
+class DiscoveryReport:
+ """What a run did — the honest version, including what it skipped."""
+
+ provider: str = ""
+ queries: int = 0
+ estimated_cost: float = 0.0
+ actual_cost: float = 0.0
+ found: int = 0
+ junk: int = 0
+ new_companies: int = 0
+ known_companies: int = 0
+ observations: int = 0
+ run_id: str = ""
+ errors: list[str] = field(default_factory=list)
+ stopped: str = ""
+
+ def as_dict(self) -> dict:
+ return {
+ "provider": self.provider, "queries": self.queries,
+ "estimated_cost": round(self.estimated_cost, 4),
+ "actual_cost": round(self.actual_cost, 4),
+ "found": self.found, "junk": self.junk,
+ "new_companies": self.new_companies,
+ "known_companies": self.known_companies,
+ "observations": self.observations,
+ "run_id": self.run_id, "errors": self.errors,
+ "stopped": self.stopped,
+ }
+
+
+def build_queries(config, cities: list[str] | None = None,
+ depth: int = 30, limit: int = 100) -> list[DiscoveryQuery]:
+ """Turn the configured ICP into a concrete query list.
+
+ One query per (industry x city). Coordinates come from
+ ``icp.geo_coordinates`` when present, because listings providers need a
+ radius, not a place name.
+ """
+ icp = config.icp
+ terms = list(icp.industries) or [config.product.name]
+ places = cities if cities is not None else list(icp.geography) or [""]
+ coords = getattr(icp, "geo_coordinates", {}) or {}
+
+ return [
+ DiscoveryQuery(term=term, location=place, depth=depth, limit=limit,
+ coordinate=coords.get(place, ""))
+ for term in terms
+ for place in places
+ ]
+
+
+def estimate_cost(provider_key: str, queries: list[DiscoveryQuery]) -> float:
+ """Projected spend. Always call this before a run — and print it."""
+ provider = PROVIDERS.get(provider_key)
+ return provider.estimate(queries) if provider else 0.0
+
+
+async def run_discovery(
+ state,
+ config,
+ provider_key: str = DEFAULT_PROVIDER,
+ queries: list[DiscoveryQuery] | None = None,
+ max_spend: float = 1.0,
+ dry_run: bool = False,
+) -> DiscoveryReport:
+ """Discover businesses and record what was learned about them.
+
+ ``dry_run`` estimates and returns without calling anything. Run it first,
+ every time: the first small trial is what exposes the data-quality
+ problems that would otherwise poison every market you scale to.
+ """
+ from harvey.config import load_env
+ from harvey.signals import seed_signal_catalog
+
+ provider = PROVIDERS.get(provider_key)
+ if provider is None:
+ raise ValueError(f"unknown discovery provider: {provider_key!r}")
+
+ queries = queries or build_queries(config)
+ if provider.kind == "listings":
+ queries = await resolve_coordinates(state, queries)
+ report = DiscoveryReport(provider=provider_key, queries=len(queries))
+ report.estimated_cost = provider.estimate(queries)
+
+ if dry_run:
+ return report
+
+ env = load_env().model_dump()
+ if not provider.configured(env):
+ missing = ", ".join(k.upper() for k in provider.env_keys
+ if not env.get(k))
+ report.errors.append(f"{provider.label} needs {missing} in .env")
+ report.stopped = "not configured"
+ return report
+
+ if report.estimated_cost > max_spend:
+ report.errors.append(
+ f"estimated ${report.estimated_cost:.2f} exceeds the ${max_spend:.2f} "
+ f"cap — raise the cap or run fewer queries")
+ report.stopped = "over budget"
+ return report
+
+ await seed_signal_catalog(state)
+ confirmed = await state.confirmed_signal_codes()
+ if not confirmed:
+ report.errors.append(
+ "no signals confirmed — confirm signals first (dashboard → Signals, "
+ "or `harvey signals --confirm free`) or discovery has nothing to record")
+ report.stopped = "no confirmed signals"
+ return report
+
+ report.run_id = await state.start_run(
+ "discover", provider=provider_key,
+ params={"queries": len(queries), "estimate": report.estimated_cost},
+ )
+
+ seen: set[str] = set()
+ try:
+ async with httpx.AsyncClient(timeout=TIMEOUT, follow_redirects=True) as client:
+ for i, query in enumerate(queries):
+ # The kill switch is read between batches, not once at the
+ # top: a long run has to be stoppable while it is running.
+ if await state.get_setting("discovery_paused"):
+ report.stopped = "paused"
+ break
+ if report.actual_cost > max_spend:
+ report.stopped = "budget reached"
+ break
+ if i and provider.key == "osm":
+ await asyncio.sleep(OpenStreetMap.MIN_INTERVAL)
+
+ try:
+ provider.last_cost = 0.0
+ found = await provider.fetch(client, env, query)
+ report.actual_cost += provider.last_cost
+ except Exception as e:
+ # A failure is an observation about the run, not a reason
+ # to lose the queries that already succeeded.
+ logger.warning("discover: %s failed on %r: %s",
+ provider_key, query.keyword(), e)
+ report.errors.append(f"{query.keyword()}: {e}")
+ continue
+
+ report.found += len(found)
+ batch = []
+ for biz in found:
+ if is_junk(biz.domain, biz.name):
+ report.junk += 1
+ continue
+ ident = biz.identity()
+ if not ident or ident in seen:
+ continue
+ seen.add(ident)
+
+ company_id, is_new = await _upsert(state, biz, query)
+ if is_new:
+ report.new_companies += 1
+ else:
+ report.known_companies += 1
+ batch.extend(_observations(biz, company_id, confirmed,
+ report.run_id))
+
+ # Flush every batch. The observations are the asset; buffering
+ # them until some threshold means a run that dies loses them.
+ if batch:
+ report.observations += await state.add_observations(
+ batch, run_id=report.run_id)
+
+ await state.finish_run(report.run_id, status="completed",
+ records=report.new_companies + report.known_companies,
+ cost_usd=report.actual_cost)
+ except Exception as e:
+ logger.exception("discover run failed")
+ report.errors.append(str(e))
+ await state.finish_run(report.run_id, status="failed",
+ records=report.new_companies,
+ cost_usd=report.actual_cost, error=str(e))
+ return report
+
+
+async def _upsert(state, biz: Business, query: DiscoveryQuery) -> tuple[str, bool]:
+ """Record the business, returning (id, was_new)."""
+ # Look before inserting, so "already knew this one" is a fact we report
+ # rather than something inferred from an insert's return value.
+ existing = None
+ if biz.domain:
+ existing = await state.get_company_by_domain(biz.domain)
+ if existing is None and biz.external_id:
+ existing = await state.get_company_by_external_id(biz.external_id)
+ if existing:
+ return existing.id, False
+
+ company = Company(
+ name=biz.name,
+ domain=biz.domain,
+ website=biz.website,
+ phone=biz.phone,
+ industry=biz.category or query.term,
+ location=biz.location or query.location,
+ source=f"discover:{biz.provider}",
+ source_url=biz.source_url,
+ external_id=biz.external_id,
+ )
+ return await state.add_company(company), True
+
+
+def _observations(biz: Business, company_id: str, confirmed: set[str],
+ run_id: str) -> list[dict]:
+ """Every fact this business gave us, as rows — and only the confirmed ones."""
+ rows: list[dict] = []
+
+ def add(code, *, num=None, text="", conf=1.0):
+ if code in confirmed:
+ rows.append({
+ "company_id": company_id, "signal_code": code,
+ "collector": "discover", "value_num": num, "value_text": text,
+ "confidence": conf, "evidence_url": biz.source_url,
+ "run_id": run_id,
+ })
+
+ add("FOUND_IN_SERP", text=biz.provider)
+ if biz.rank is not None:
+ add("SERP_RANK", num=float(biz.rank))
+ if biz.has_website is False:
+ add("NO_WEBSITE", num=1.0)
+ if biz.is_claimed is False:
+ add("UNCLAIMED_LISTING", num=1.0)
+ if biz.rating is not None:
+ add("REVIEW_RATING", num=float(biz.rating))
+ if biz.review_count is not None:
+ add("REVIEW_COUNT", num=float(biz.review_count))
+ return rows
+
+
+def provider_menu(env: dict | None = None) -> list[dict]:
+ """The menu the dashboard renders: what each provider does, and its cost."""
+ env = env or {}
+ return [
+ {
+ "key": p.key, "label": p.label, "kind": p.kind, "blurb": p.blurb,
+ "cost_note": p.cost_note, "free_tier": p.free_tier,
+ "signup_url": p.signup_url, "caveat": p.caveat,
+ "env_keys": [k.upper() for k in p.env_keys],
+ "configured": p.configured(env),
+ "needs_key": bool(p.env_keys),
+ }
+ for p in PROVIDERS.values()
+ ]
diff --git a/harvey/config.py b/harvey/config.py
index f8c319c..882cc0b 100644
--- a/harvey/config.py
+++ b/harvey/config.py
@@ -60,6 +60,11 @@ class ICPConfig(BaseModel):
# hiring a "Head of Growth" is buying growth tooling). Empty → falls back
# to `titles`. Used for careers-page scanning and job-board discovery.
hiring_signals: list[str] = []
+ # Discovery needs a radius, not a place name. Maps each entry in
+ # `geography` to "lat,lng,radius_km" — e.g.
+ # "Denver, CO": "39.7392,-104.9903,50"
+ # Without one, listings providers can only match on the business name.
+ geo_coordinates: dict[str, str] = {}
class EmailChannelConfig(BaseModel):
@@ -157,6 +162,10 @@ class HarveyConfig(BaseModel):
class EnvConfig(BaseModel):
instantly_api_key: str = ""
+ # Discovery providers
+ dataforseo_login: str = ""
+ dataforseo_password: str = ""
+ dataforseo_sandbox: str = "" # any truthy value routes to the free sandbox
linkedin_email: str = ""
linkedin_password: str = ""
hunter_api_key: str = ""
@@ -231,6 +240,9 @@ def load_env() -> EnvConfig:
load_dotenv()
env = EnvConfig(
instantly_api_key=os.getenv("INSTANTLY_API_KEY", "").strip(),
+ dataforseo_login=os.getenv("DATAFORSEO_LOGIN", "").strip(),
+ dataforseo_password=os.getenv("DATAFORSEO_PASSWORD", "").strip(),
+ dataforseo_sandbox=os.getenv("DATAFORSEO_SANDBOX", "").strip(),
linkedin_email=os.getenv("LINKEDIN_EMAIL", "").strip(),
linkedin_password=os.getenv("LINKEDIN_PASSWORD", "").strip(),
hunter_api_key=os.getenv("HUNTER_API_KEY", "").strip(),
diff --git a/harvey/dashboard.py b/harvey/dashboard.py
index 60eb4a6..9ac889b 100644
--- a/harvey/dashboard.py
+++ b/harvey/dashboard.py
@@ -761,6 +761,134 @@ async def get_activity():
# ── Dashboard UI ──
+# ── Discovery: the provider menu, an estimate, and a run ──
+#
+# This is the only stage that spends money, so the UI never starts one without
+# showing what it will cost first.
+
+_discovery_task: asyncio.Task | None = None
+_discovery_report: dict | None = None
+
+
+def _discovery_queries(body: dict, config):
+ from harvey.collectors.discover import build_queries
+
+ cities = [c.strip() for c in (body.get("cities") or []) if c.strip()] or None
+ return build_queries(
+ config, cities=cities,
+ depth=int(body.get("depth") or 30),
+ limit=int(body.get("limit") or 100),
+ )
+
+
[email protected]("/api/discover/providers")
+async def get_discovery_providers():
+ """What each source does, what it costs, and whether it's ready to use."""
+ try:
+ from harvey.collectors.discover import DEFAULT_PROVIDER, provider_menu
+ from harvey.config import load_env
+
+ state = _state()
+ await state.init_db()
+ return {
+ "providers": provider_menu(load_env().model_dump()),
+ "default": DEFAULT_PROVIDER,
+ "selected": await state.get_setting("discovery_provider") or DEFAULT_PROVIDER,
+ "paused": await state.get_setting("discovery_paused"),
+ "running": bool(_discovery_task and not _discovery_task.done()),
+ "last_report": _discovery_report,
+ }
+ except Exception as e:
+ logger.exception("discovery providers failed")
+ return {"providers": [], "error": str(e)}
+
+
[email protected]("/api/discover/estimate")
+async def estimate_discovery(request: Request):
+ """Projected spend and the exact query list, before anything is called."""
+ try:
+ from harvey.collectors.discover import PROVIDERS, estimate_cost
+ from harvey.config import load_config
+
+ body = await request.json()
+ provider = body.get("provider") or ""
+ if provider not in PROVIDERS:
+ return JSONResponse({"error": f"unknown provider {provider!r}"},
+ status_code=400)
+
+ queries = _discovery_queries(body, load_config())
+ return {
+ "provider": provider,
+ "queries": [q.keyword() for q in queries],
+ "query_count": len(queries),
+ "estimated_cost": round(estimate_cost(provider, queries), 4),
+ "free": PROVIDERS[provider].estimate(queries) == 0,
+ }
+ except Exception as e:
+ return JSONResponse({"error": str(e)}, status_code=500)
+
+
[email protected]("/api/discover/run")
+async def start_discovery(request: Request):
+ """Kick off a run in the background and hand back immediately.
+
+ Discovery takes minutes, not milliseconds — holding the request open
+ would just time out. Progress shows up in the run log.
+ """
+ global _discovery_task, _discovery_report
+
+ if _discovery_task and not _discovery_task.done():
+ return JSONResponse({"success": False, "message": "a run is already going"},
+ status_code=409)
+ try:
+ from harvey.collectors.discover import PROVIDERS, run_discovery
+ from harvey.config import load_config
+
+ body = await request.json()
+ provider = body.get("provider") or ""
+ if provider not in PROVIDERS:
+ return JSONResponse({"success": False,
+ "message": f"unknown provider {provider!r}"},
+ status_code=400)
+
+ config = load_config()
+ queries = _discovery_queries(body, config)
+ max_spend = float(body.get("max_spend") or 1.0)
+
+ state = _state()
+ await state.init_db()
+ await state.set_setting("discovery_provider", provider)
+ await state.set_setting("discovery_paused", "")
+
+ async def _go():
+ global _discovery_report
+ try:
+ report = await run_discovery(state, config, provider, queries,
+ max_spend=max_spend)
+ _discovery_report = report.as_dict()
+ except Exception as exc:
+ logger.exception("discovery run failed")
+ _discovery_report = {"errors": [str(exc)], "stopped": "failed"}
+
+ _discovery_report = None
+ _discovery_task = asyncio.create_task(_go())
+ return {"success": True, "queries": len(queries)}
+ except Exception as e:
+ return JSONResponse({"success": False, "message": str(e)}, status_code=500)
+
+
[email protected]("/api/discover/stop")
+async def stop_discovery():
+ """Kill switch. Read between batches, so an in-flight run stops cleanly."""
+ try:
+ state = _state()
+ await state.init_db()
+ await state.set_setting("discovery_paused", "stopped from dashboard")
+ return {"success": True}
+ except Exception as e:
+ return JSONResponse({"success": False, "message": str(e)}, status_code=500)
+
+
@app.get("/api/today")
async def get_today():
"""What needs a human, right now.
@@ -840,6 +968,15 @@ async def get_today():
"action": "Review signals", "tab": "signals",
})
+ if not n_companies and confirmed:
+ items.append({
+ "key": "discover", "tone": "good",
+ "title": "No companies yet",
+ "detail": ("Signals are confirmed but nothing has been collected. "
+ "Discovery is free to try — no account needed."),
+ "action": "Find businesses", "tab": "discover",
+ })
+
if pending:
items.append({
"key": "outbox", "tone": "warn",
diff --git a/harvey/models/company.py b/harvey/models/company.py
index c3f29d6..0c261d0 100644
--- a/harvey/models/company.py
+++ b/harvey/models/company.py
@@ -19,8 +19,13 @@ class Company(BaseModel):
industry: str = ""
company_size: str = ""
location: str = ""
+ phone: str = ""
source: str = "" # how we found them: google_dork, linkedin, company_scrape
source_url: str = "" # specific URL where we found the info
+ # Provider-stable identity ("dataforseo:ChIJ...", "osm:node/123"). The
+ # dedup key for businesses that have no website — which are precisely the
+ # best prospects for anyone selling one.
+ external_id: str = ""
notes: str = ""
# Buying signals: detected tools on their site, and timely triggers
# (hiring, funding, ...) as [{"type": ..., "detail": ..., "found_at": ...}]
diff --git a/harvey/signals.py b/harvey/signals.py
index 1b62eae..525d785 100644
--- a/harvey/signals.py
+++ b/harvey/signals.py
@@ -46,6 +46,62 @@ SIGNAL_CATALOG: list[dict] = [
"confidence_floor": 0.0,
},
+ {
+ "code": "NO_WEBSITE",
+ "label": "No website at all",
+ "description": (
+ "The business has a listing and a phone number but no site. If you "
+ "sell websites this is the highest-intent signal that exists — and "
+ "it is a filter you can query for directly, not something you infer."
+ ),
+ "category": "discovery",
+ "value_type": "bool",
+ "collector": "discover",
+ "cost_note": "included in the business listing lookup",
+ "confidence_floor": 0.0,
+ },
+ {
+ "code": "UNCLAIMED_LISTING",
+ "label": "Unclaimed Google listing",
+ "description": (
+ "Nobody has claimed their Google Business Profile, so they are not "
+ "managing how they show up locally. Strong signal for anyone "
+ "selling local marketing."
+ ),
+ "category": "discovery",
+ "value_type": "bool",
+ "collector": "discover",
+ "cost_note": "included in the business listing lookup",
+ "confidence_floor": 0.0,
+ },
+ {
+ "code": "REVIEW_RATING",
+ "label": "Review rating",
+ "description": (
+ "Their average star rating. Useful in both directions: a low "
+ "rating is a reputation problem to solve, a high one with few "
+ "reviews is proof they are good but invisible."
+ ),
+ "category": "discovery",
+ "value_type": "num",
+ "collector": "discover",
+ "cost_note": "included in the business listing lookup",
+ "confidence_floor": 0.0,
+ },
+ {
+ "code": "REVIEW_COUNT",
+ "label": "Number of reviews",
+ "description": (
+ "How many reviews they have. A rough proxy for size and for how "
+ "much attention they pay to their online presence."
+ ),
+ "category": "discovery",
+ "value_type": "num",
+ "collector": "discover",
+ "cost_note": "included in the business listing lookup",
+ "confidence_floor": 0.0,
+ },
+
# ── Profile: what they are (free — HTML Harvey already fetches) ──
{
"code": "INCUMBENT_AGENCY",
diff --git a/harvey/state.py b/harvey/state.py
index 9f3c6ed..5f495c7 100644
--- a/harvey/state.py
+++ b/harvey/state.py
@@ -362,6 +362,18 @@ MIGRATIONS: list[str] = [
CREATE INDEX IF NOT EXISTS idx_runs_stage_started ON runs(stage, started_at);
""",
+ # ── v8: discovery — entity resolution for businesses without a website ──
+ """
+ -- The best prospect for anyone selling websites is a business that has
+ -- none, so `domain` cannot be the only identity key. `external_id` holds
+ -- the provider's stable id ("dataforseo:ChIJ...", "osm:node/123") and is
+ -- what dedups a re-run for those records.
+ ALTER TABLE companies ADD COLUMN external_id TEXT DEFAULT '';
+ ALTER TABLE companies ADD COLUMN phone TEXT DEFAULT '';
+
+ CREATE UNIQUE INDEX IF NOT EXISTS uq_companies_external
+ ON companies(external_id) WHERE external_id != '';
+ """,
]
# Column whitelists for dynamic UPDATEs (prevents SQL injection via kwargs).
@@ -425,7 +437,13 @@ class StateManager:
return Company(**d)
async def add_company(self, company: Company) -> str:
- """Insert a company. If one with the same domain exists, return its id."""
+ """Insert a company, or return the id of the one already recorded.
+
+ Identity is the normalised domain when there is one, and the
+ provider's ``external_id`` when there isn't — a business with no
+ website still has to dedup across re-runs, and for anyone selling
+ websites those are the best prospects on the list.
+ """
if not company.id:
company.id = _new_id()
company.domain = _norm(company.domain)
@@ -433,28 +451,34 @@ class StateManager:
cursor = await db.execute(
"""INSERT OR IGNORE INTO companies
(id, name, domain, website, description, industry,
- company_size, location, source, source_url, notes,
- tech_stack_json, signals_json, created_at, updated_at)
- VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
+ company_size, location, phone, source, source_url,
+ external_id, notes, tech_stack_json, signals_json,
+ created_at, updated_at)
+ VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
(
company.id, company.name, company.domain, company.website,
company.description, company.industry, company.company_size,
- company.location, company.source, company.source_url,
- company.notes,
+ company.location, company.phone, company.source,
+ company.source_url, company.external_id, company.notes,
json.dumps(company.tech_stack), json.dumps(company.signals),
company.created_at.isoformat(),
company.updated_at.isoformat(),
),
)
await db.commit()
- if cursor.rowcount == 0 and company.domain:
- # Unique-domain conflict: hand back the existing record's id.
- async with db.execute(
- "SELECT id FROM companies WHERE domain = ?", (company.domain,)
- ) as cur:
- row = await cur.fetchone()
- if row:
- company.id = row[0]
+ if cursor.rowcount == 0:
+ # Uniqueness conflict: hand back the existing record's id.
+ for column, value in (("domain", company.domain),
+ ("external_id", company.external_id)):
+ if not value:
+ continue
+ async with db.execute(
+ f"SELECT id FROM companies WHERE {column} = ?", (value,)
+ ) as cur:
+ row = await cur.fetchone()
+ if row:
+ company.id = row[0]
+ break
return company.id
async def update_company_signals(
@@ -526,6 +550,22 @@ class StateManager:
row = await cursor.fetchone()
return self._company_from_row(row) if row else None
+ async def get_company_by_external_id(self, external_id: str) -> Company | None:
+ """Look a company up by its provider-stable id.
+
+ The identity path for businesses with no website — which is exactly
+ the cohort worth the most to anyone selling one.
+ """
+ if not external_id:
+ return None
+ async with self._connect() as db:
+ db.row_factory = aiosqlite.Row
+ async with db.execute(
+ "SELECT * FROM companies WHERE external_id = ?", (external_id,)
+ ) as cursor:
+ row = await cursor.fetchone()
+ return self._company_from_row(row) if row else None
+
async def get_contacts_for_company(self, company_id: str) -> list[Prospect]:
async with self._connect() as db:
db.row_factory = aiosqlite.Row
diff --git a/harvey/web/app.css b/harvey/web/app.css
index c27e3f5..0ff87de 100644
--- a/harvey/web/app.css
+++ b/harvey/web/app.css
@@ -546,3 +546,31 @@
.badge.t-bad { background: rgba(224,108,117,0.1); color: var(--red); border-color: rgba(224,108,117,0.22); }
.badge.t-idle { background: rgba(255,255,255,0.04); color: var(--text-3); border-color: var(--border); }
.badge.t-note { background: rgba(180,140,232,0.12); color: var(--purple); border-color: rgba(180,140,232,0.25); }
+
+ /* ── Discovery provider menu ── */
+ .prov-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); gap: 12px; margin-bottom: 24px; }
+ .prov-card {
+ background: var(--panel); border: 1px solid var(--border); border-radius: 12px;
+ padding: 18px; cursor: pointer; display: flex; flex-direction: column; gap: 8px;
+ }
+ .prov-card:hover { border-color: var(--border-strong); }
+ .prov-card.selected { border-color: var(--accent); background: linear-gradient(180deg, var(--accent-soft), transparent 60%); }
+ .prov-card .prov-head { display: flex; align-items: baseline; justify-content: space-between; gap: 10px; }
+ .prov-card h3 { font-size: 14.5px; font-weight: 700; letter-spacing: -0.2px; }
+ .prov-card .blurb { font-size: 12.5px; color: var(--text-2); line-height: 1.6; }
+ .prov-card .row { display: flex; gap: 8px; align-items: baseline; font-size: 12px; }
+ .prov-card .row .k { color: var(--text-3); min-width: 42px; }
+ .prov-card .row .v { color: var(--text-2); }
+ .prov-card .caveat {
+ font-size: 11.5px; color: var(--amber); background: rgba(229,181,103,0.07);
+ border-left: 2px solid rgba(229,181,103,0.4); padding: 7px 10px; border-radius: 0 6px 6px 0;
+ line-height: 1.55;
+ }
+ .estimate-box {
+ border: 1px solid var(--border); border-radius: 10px; padding: 16px 18px;
+ background: rgba(255,255,255,0.015); margin-bottom: 16px;
+ }
+ .estimate-box .amount { font-size: 26px; font-weight: 700; letter-spacing: -0.6px; font-variant-numeric: tabular-nums; }
+ .estimate-box .amount.free { color: var(--accent); }
+ .estimate-box .amount.paid { color: var(--amber); }
+ .query-list { font-family: var(--mono); font-size: 11.5px; color: var(--text-3); margin-top: 12px; line-height: 1.8; max-height: 160px; overflow-y: auto; }
diff --git a/harvey/web/app.js b/harvey/web/app.js
index 1860719..2135f95 100644
--- a/harvey/web/app.js
+++ b/harvey/web/app.js
@@ -135,6 +135,7 @@ function loadCurrentTab() {
switch (currentTab) {
case 'today': loadToday(); loadStats(); loadSetupStatus(); loadRuns(); break;
case 'signals': loadSignals(); break;
+ case 'discover': loadDiscoverProviders(); break;
case 'companies': if (!companyDrill) loadCompanies(); break;
case 'prospects': loadProspects(); break;
case 'campaigns': loadCampaigns(); break;
@@ -350,6 +351,165 @@ async function setSignals(codes, status) {
loadSignals();
}
+// ── Discover: pick a source, price it, then run it ──
+
+let _provider = null;
+let _discoverPoll = null;
+
+async function loadDiscoverProviders() {
+ const el = document.getElementById('discover-providers');
+ const data = await api('/api/discover/providers');
+ if (!data || !data.providers) { el.innerHTML = offlineState(); return; }
+
+ _provider = _provider || data.selected || data.default;
+ el.innerHTML = '<div class="prov-grid">' + data.providers.map(p => {
+ const ready = p.configured
+ ? '<span class="badge t-good">ready</span>'
+ : '<span class="badge t-waiting">needs a key</span>';
+ return '<div class="prov-card ' + (p.key === _provider ? 'selected' : '') +
+ '" onclick="pickProvider(\'' + p.key + '\')">' +
+ '<div class="prov-head"><h3>' + escHtml(p.label) + '</h3>' + ready + '</div>' +
+ '<div class="blurb">' + escHtml(p.blurb) + '</div>' +
+ '<div class="row"><span class="k">Cost</span><span class="v">' +
+ escHtml(p.cost_note) + '</span></div>' +
+ '<div class="row"><span class="k">Free</span><span class="v">' +
+ escHtml(p.free_tier) + '</span></div>' +
+ (p.needs_key
+ ? '<div class="row"><span class="k">Setup</span><span class="v">' +
+ escHtml(p.env_keys.join(', ')) + ' in .env &middot; ' +
+ '<a href="' + escHtml(p.signup_url) + '" target="_blank" rel="noopener">get a key</a>' +
+ '</span></div>'
+ : '') +
+ (p.caveat ? '<div class="caveat">' + escHtml(p.caveat) + '</div>' : '') +
+ '</div>';
+ }).join('') + '</div>';
+
+ const running = data.running;
+ document.getElementById('disc-stop').style.display = running ? '' : 'none';
+ if (running && !_discoverPoll) {
+ _discoverPoll = setInterval(loadDiscoverProviders, 4000);
+ } else if (!running && _discoverPoll) {
+ clearInterval(_discoverPoll);
+ _discoverPoll = null;
+ // A finished run leaves the button stuck on "Running…" otherwise.
+ const btn = document.getElementById('disc-run');
+ btn.disabled = true;
+ btn.textContent = 'Estimate first';
+ showToast('Discovery finished.', 'success');
+ }
+ renderDiscoverResult(data.last_report, running);
+}
+
+function pickProvider(key) {
+ _provider = key;
+ document.getElementById('disc-run').disabled = true;
+ document.getElementById('disc-run').textContent = 'Estimate first';
+ document.getElementById('discover-estimate').innerHTML = '';
+ loadDiscoverProviders();
+}
+
+function discoverBody() {
+ const raw = document.getElementById('disc-cities').value.trim();
+ return {
+ provider: _provider,
+ // Semicolons or newlines, never commas: "Denver, CO" is one city.
+ cities: raw ? raw.split(/[;\n]/).map(c => c.trim()).filter(Boolean) : [],
+ depth: Number(document.getElementById('disc-depth').value) || 30,
+ limit: Number(document.getElementById('disc-limit').value) || 100,
+ max_spend: Number(document.getElementById('disc-cap').value) || 1,
+ };
+}
+
+async function estimateDiscovery() {
+ const el = document.getElementById('discover-estimate');
+ el.innerHTML = '<p class="muted" style="font-size:13px">Pricing it…</p>';
+ const data = await api('/api/discover/estimate', {
+ method: 'POST', headers: {'Content-Type': 'application/json'},
+ body: JSON.stringify(discoverBody()),
+ });
+ if (!data || data.error) {
+ el.innerHTML = '<div class="test-result error">' +
+ escHtml((data && data.error) || 'Could not estimate.') + '</div>';
+ return;
+ }
+
+ const cap = discoverBody().max_spend;
+ const over = data.estimated_cost > cap;
+ el.innerHTML = '<div class="estimate-box">' +
+ '<div class="amount ' + (data.free ? 'free' : 'paid') + '">' +
+ (data.free ? 'Free' : '$' + data.estimated_cost.toFixed(4)) + '</div>' +
+ '<div class="muted" style="font-size:13px;margin-top:2px">' +
+ data.query_count + ' quer' + (data.query_count === 1 ? 'y' : 'ies') +
+ (data.free ? '' : ' &middot; cap is $' + cap.toFixed(2)) + '</div>' +
+ (over ? '<div class="test-result error" style="margin-top:12px">' +
+ 'Over your cap. Raise the cap or narrow the search.</div>' : '') +
+ '<div class="query-list">' +
+ data.queries.slice(0, 40).map(escHtml).join('<br>') +
+ (data.queries.length > 40 ? '<br>… and ' + (data.queries.length - 40) + ' more' : '') +
+ '</div></div>';
+
+ const btn = document.getElementById('disc-run');
+ btn.disabled = over;
+ btn.textContent = over ? 'Over cap'
+ : (data.free ? 'Run — free' : 'Run — spend up to $' + data.estimated_cost.toFixed(2));
+}
+
+async function runDiscovery() {
+ const btn = document.getElementById('disc-run');
+ btn.disabled = true;
+ btn.textContent = 'Running…';
+ const data = await api('/api/discover/run', {
+ method: 'POST', headers: {'Content-Type': 'application/json'},
+ body: JSON.stringify(discoverBody()),
+ });
+ if (data && data.success) {
+ showToast('Discovery started — ' + data.queries + ' queries.', 'success');
+ } else {
+ showToast((data && data.message) || 'Could not start.', 'error');
+ btn.disabled = false;
+ }
+ loadDiscoverProviders();
+}
+
+async function stopDiscovery() {
+ const data = await api('/api/discover/stop', {method: 'POST'});
+ showToast(data && data.success ? 'Stopping after the current query.'
+ : 'Could not stop.', data ? 'success' : 'error');
+ loadDiscoverProviders();
+}
+
+function renderDiscoverResult(report, running) {
+ const el = document.getElementById('discover-result');
+ if (running) {
+ el.innerHTML = '<div class="card"><h2>Running…</h2>' +
+ '<p class="muted" style="font-size:13px">Harvey is working through the ' +
+ 'queries. Results land in Companies, and the run log is on Today.</p></div>';
+ return;
+ }
+ if (!report) { el.innerHTML = ''; return; }
+
+ const stat = (label, value, tone) =>
+ '<div class="stat-card"><div class="label">' + label + '</div>' +
+ '<div class="value"' + (tone ? ' style="color:var(--' + tone + ')"' : '') + '>' +
+ value + '</div></div>';
+
+ el.innerHTML = '<div class="subhead">Last run</div>' +
+ '<div class="stats-grid">' +
+ stat('New companies', report.new_companies || 0, 'accent') +
+ stat('Already known', report.known_companies || 0) +
+ stat('Observations', report.observations || 0) +
+ stat('Filtered as junk', report.junk || 0) +
+ stat('Actual cost', report.actual_cost ? '$' + report.actual_cost.toFixed(4) : 'free') +
+ '</div>' +
+ (report.stopped ? '<div class="test-result error" style="margin-top:14px">' +
+ 'Stopped early: ' + escHtml(report.stopped) + '</div>' : '') +
+ ((report.errors || []).length
+ ? '<div class="card" style="margin-top:14px"><h2>Problems</h2>' +
+ (report.errors || []).slice(0, 8).map(e =>
+ '<div class="check-help">' + escHtml(e) + '</div>').join('') + '</div>'
+ : '');
+}
+
// ── Cohort builder: a prospect list is a query ──
let _cohort = { require: new Set(), exclude: new Set() };
diff --git a/harvey/web/index.html b/harvey/web/index.html
index dbef5af..5d1aa9c 100644
--- a/harvey/web/index.html
+++ b/harvey/web/index.html
@@ -28,6 +28,7 @@
<nav>
<button class="active" onclick="showTab('today', this)">Today<span class="nav-count" id="nav-today"></span></button>
<button onclick="showTab('signals', this)">Signals<span class="nav-count" id="nav-signals"></span></button>
+ <button onclick="showTab('discover', this)">Discover</button>
<button onclick="showTab('companies', this)">Companies</button>
<button onclick="showTab('prospects', this)">Contacts</button>
<button onclick="showTab('campaigns', this)">Campaigns</button>
@@ -75,6 +76,47 @@
</div>
</div>
+<!-- Discover: the only stage that spends money -->
+<div id="discover" class="section">
+ <div class="section-head">
+ <h2>Discover</h2>
+ <p>Find businesses matching your ICP. Pick a source, see what it will cost,
+ then run it &mdash; nothing is called until you press the button.</p>
+ </div>
+ <div id="discover-providers"></div>
+
+ <div class="subhead">Run</div>
+ <div class="card">
+ <div class="form-row" style="margin-bottom:14px">
+ <div class="form-group" style="flex:2">
+ <label class="form-label">Cities</label>
+ <input class="form-input" id="disc-cities" placeholder="from your ICP — or override, one per line">
+ </div>
+ <div class="form-group">
+ <label class="form-label">Results per query</label>
+ <input class="form-input" id="disc-limit" type="number" value="100" min="1" max="1000">
+ </div>
+ <div class="form-group">
+ <label class="form-label">Search depth</label>
+ <input class="form-input" id="disc-depth" type="number" value="30" min="10" max="100" step="10">
+ </div>
+ <div class="form-group">
+ <label class="form-label">Spend cap ($)</label>
+ <input class="form-input" id="disc-cap" type="number" value="1.00" min="0" step="0.25">
+ </div>
+ </div>
+ <div id="discover-estimate"></div>
+ <div class="btn-group">
+ <button class="btn btn-secondary" onclick="estimateDiscovery()">Estimate cost</button>
+ <button class="btn btn-primary" id="disc-run" onclick="runDiscovery()" disabled>
+ Estimate first</button>
+ <button class="btn btn-danger" id="disc-stop" onclick="stopDiscovery()" style="display:none">
+ Stop</button>
+ </div>
+ </div>
+ <div id="discover-result"></div>
+</div>
+
<!-- Companies -->
<div id="companies" class="section">
<div class="section-head"><h2>Companies</h2><p>Organizations Harvey has researched. Click a row to see its contacts.</p></div>
diff --git a/tests/test_dashboard_signals.py b/tests/test_dashboard_signals.py
index 9785bd8..76b1508 100644
--- a/tests/test_dashboard_signals.py
+++ b/tests/test_dashboard_signals.py
@@ -167,7 +167,7 @@ def test_today_leads_with_unconfirmed_signals(client):
assert "signals" in keys
item = next(i for i in data["items"] if i["key"] == "signals")
assert item["tab"] == "signals"
- assert "19" in item["title"]
+ assert str(len(SIGNAL_CATALOG)) in item["title"]
def test_today_flags_a_fully_rejected_vocabulary(client):
diff --git a/tests/test_discover.py b/tests/test_discover.py
new file mode 100644
index 0000000..616bfb7
--- /dev/null
+++ b/tests/test_discover.py
@@ -0,0 +1,660 @@
+"""DISCOVER — junk filtering, entity resolution, cost discipline, adapters.
+
+The expensive stage, so the tests care most about the things that cost money
+or corrupt the database: that an estimate is honest, that a budget is
+enforced before spending rather than after, that a re-run doesn't duplicate
+anyone, and that directories never make it in.
+"""
+
+import asyncio
+import os
+import tempfile
+
+import httpx
+import pytest
+import pytest_asyncio
+
+from harvey.collectors import discover as D
+from harvey.config import load_config
+from harvey.signals import SIGNAL_CATALOG, seed_signal_catalog
+from harvey.state import StateManager
+
+
+@pytest_asyncio.fixture
+async def state():
+ with tempfile.TemporaryDirectory() as tmp:
+ sm = StateManager(os.path.join(tmp, "test.db"))
+ await sm.init_db()
+ await seed_signal_catalog(sm)
+ for sig in SIGNAL_CATALOG:
+ await sm.set_signal_status(sig["code"], "confirmed")
+ yield sm
+
+
+def config():
+ return load_config()
+
+
+# The real implementations, captured before the autouse fixture stubs them,
+# so the geocoding tests can exercise them directly.
+REAL_GEOCODE = D.geocode_city
+REAL_RESOLVE = D.resolve_coordinates
+REAL_SLEEP = asyncio.sleep
+
+
[email protected](autouse=True)
+def offline(monkeypatch):
+ """No test reaches the network, and none of them wait a real second.
+
+ The geocoding tests call REAL_GEOCODE / REAL_RESOLVE with their own
+ mock-transport client.
+ """
+ async def no_geocode(state, city, radius_km=40, client=None):
+ return ""
+
+ async def no_sleep(_seconds):
+ return None
+
+ monkeypatch.setattr(D, "geocode_city", no_geocode)
+ monkeypatch.setattr(D.asyncio, "sleep", no_sleep)
+
+
+class FakeProvider(D.DiscoveryProvider):
+ """A provider that returns what the test hands it, and spends nothing."""
+
+ key = "fake"
+ label = "Fake"
+ cost_note = "free"
+
+ def __init__(self, batches, per_query_cost=0.0):
+ self.batches = list(batches)
+ self.per_query_cost = per_query_cost
+ self.calls = 0
+
+ def configured(self, env):
+ return True
+
+ def estimate(self, queries):
+ return self.per_query_cost * len(queries)
+
+ async def fetch(self, client, env, query):
+ self.calls += 1
+ self.last_cost = self.per_query_cost
+ return self.batches.pop(0) if self.batches else []
+
+
+def register(monkeypatch):
+ """Install a fake provider into the registry for one test."""
+ def _install(provider):
+ monkeypatch.setitem(D.PROVIDERS, provider.key, provider)
+ return provider
+ return _install
+
+
+def biz(**kw):
+ kw.setdefault("provider", "fake")
+ return D.Business(**kw)
+
+
+# ── Junk filtering ──
+
+
[email protected]("domain", [
+ "yelp.com", "www.yellowpages.com", "angi.com", "homeadvisor.com",
+ "facebook.com", "linkedin.com", "thumbtack.com", "houzz.com",
+ "en.wikipedia.org", "reddit.com", "indeed.com",
+ "miami.edu", "ufhealth.org", "umiamihealth.org",
+ "somehospital.com", "mysite.wordpress.com", "shop.shopify.com",
+ "192.168.1.1", "localhost",
+])
+def test_directories_and_institutions_are_junk(domain):
+ assert D.is_junk(domain), f"{domain} should have been filtered out"
+
+
[email protected]("domain", [
+ "kettlemanroofing.com", "northvaleroofing.com", "sandpiperexteriors.com",
+ "quarrylaneroofing.com", "smith-and-sons.co.uk", "acme.dental",
+])
+def test_real_businesses_survive_the_filter(domain):
+ assert not D.is_junk(domain)
+
+
+def test_listicle_titles_are_junk_even_with_a_clean_domain():
+ assert D.is_junk("", "10 Best Roofers in Denver")
+ assert D.is_junk("example.com", "Top Roofing Contractors in Dallas")
+ assert D.is_junk("example.com", "Roofer Directory")
+ assert not D.is_junk("example.com", "Northvale Roofing Company")
+
+
+def test_normalize_domain_is_the_entity_key():
+ for raw in ("https://WWW.Example.com/path?q=1", "www.example.com",
+ "http://example.com.", "EXAMPLE.COM"):
+ assert D.normalize_domain(raw) == "example.com"
+ assert D.normalize_domain("") == ""
+
+
+# ── Cost discipline ──
+
+
+def test_estimates_match_published_pricing():
+ q = [D.DiscoveryQuery(term="roofer", location="Denver", limit=1000, depth=30)]
+ # DataForSEO's own worked example: 1,000 items = $0.012 + $0.36 = $0.372
+ assert D.estimate_cost("dataforseo_listings", q) == pytest.approx(0.372)
+ # SERP billing is base price x pages; depth 30 is three pages.
+ assert D.estimate_cost("dataforseo_serp", q) == pytest.approx(0.0018)
+ assert D.estimate_cost("osm", q) == 0.0
+
+
+async def test_dry_run_estimates_without_calling_anything(state, config, register):
+ fake = register(FakeProvider([[biz(domain="a.com")]], per_query_cost=0.5))
+ q = [D.DiscoveryQuery(term="roofer", location="Denver")]
+ report = await D.run_discovery(state, config, "fake", q, dry_run=True)
+ assert report.estimated_cost == 0.5
+ assert fake.calls == 0
+ assert report.found == 0
+ assert not await state.get_runs() # a dry run isn't a run
+
+
+async def test_budget_cap_refuses_before_spending(state, config, register):
+ fake = register(FakeProvider([[biz(domain="a.com")]] * 10, per_query_cost=1.0))
+ q = [D.DiscoveryQuery(term="roofer", location=c) for c in "abcdefghij"]
+ report = await D.run_discovery(state, config, "fake", q, max_spend=2.0)
+ assert report.stopped == "over budget"
+ assert fake.calls == 0, "the cap must be checked before the first call, not after"
+ assert report.actual_cost == 0.0
+
+
+async def test_budget_stops_mid_run_when_actual_exceeds_estimate(state, config, register):
+ # Cheap to estimate, expensive in reality — the loop has to notice.
+ fake = FakeProvider([[biz(domain=f"{c}.com")] for c in "abcdefghij"],
+ per_query_cost=1.0)
+ fake.estimate = lambda queries: 0.0
+ register(fake)
+ q = [D.DiscoveryQuery(term="roofer", location=c) for c in "abcdefghij"]
+ report = await D.run_discovery(state, config, "fake", q, max_spend=3.0)
+ assert report.stopped == "budget reached"
+ assert fake.calls < 10
+
+
+async def test_kill_switch_stops_between_batches(state, config, register):
+ await state.set_setting("discovery_paused", "stopped by hand")
+ fake = register(FakeProvider([[biz(domain="a.com")]], per_query_cost=0.0))
+ q = [D.DiscoveryQuery(term="roofer", location="Denver")]
+ report = await D.run_discovery(state, config, "fake", q)
+ assert report.stopped == "paused"
+ assert fake.calls == 0
+
+
+async def test_nothing_runs_until_signals_are_confirmed(state, config, register):
+ for sig in SIGNAL_CATALOG:
+ await state.set_signal_status(sig["code"], "proposed")
+ fake = register(FakeProvider([[biz(domain="a.com")]]))
+ report = await D.run_discovery(state, config, "fake",
+ [D.DiscoveryQuery(term="roofer")])
+ assert report.stopped == "no confirmed signals"
+ assert fake.calls == 0
+ assert "confirm signals" in report.errors[0].lower()
+
+
+# ── Entity resolution ──
+
+
+async def test_rerun_finds_everyone_already_known(state, config, register):
+ batch = [
+ biz(name="Northvale Roofing", domain="northvaleroofing.com"),
+ biz(name="No Site Roofing", external_id="osm:node/1", has_website=False),
+ ]
+ register(FakeProvider([list(batch), list(batch)]))
+ q = [D.DiscoveryQuery(term="roofer", location="Denver")]
+
+ first = await D.run_discovery(state, config, "fake", q)
+ assert (first.new_companies, first.known_companies) == (2, 0)
+
+ second = await D.run_discovery(state, config, "fake", q)
+ assert (second.new_companies, second.known_companies) == (0, 2), (
+ "a re-run must recognise both the domain-keyed and the "
+ "external-id-keyed record")
+
+
+async def test_websiteless_businesses_are_kept_and_deduped(state, config, register):
+ """The best prospect for someone selling websites has no website, so
+ `domain` can't be the only identity key."""
+ register(FakeProvider([
+ [biz(name="A", external_id="osm:node/1", has_website=False),
+ biz(name="B", external_id="osm:node/2", has_website=False),
+ biz(name="A again", external_id="osm:node/1", has_website=False)],
+ ]))
+ report = await D.run_discovery(state, config, "fake",
+ [D.DiscoveryQuery(term="roofer")])
+ assert report.new_companies == 2
+ assert len(await state.cohort(["NO_WEBSITE"])) == 2
+
+
+async def test_duplicates_within_one_run_are_collapsed(state, config, register):
+ register(FakeProvider([[
+ biz(domain="northvale.com"), biz(domain="northvale.com"), biz(domain="other.com"),
+ ]]))
+ report = await D.run_discovery(state, config, "fake",
+ [D.DiscoveryQuery(term="roofer")])
+ assert report.found == 3
+ assert report.new_companies == 2
+
+
+async def test_junk_is_counted_not_silently_dropped(state, config, register):
+ register(FakeProvider([[
+ biz(domain="yelp.com"), biz(domain="angi.com"), biz(domain="real.com"),
+ ]]))
+ report = await D.run_discovery(state, config, "fake",
+ [D.DiscoveryQuery(term="roofer")])
+ assert report.junk == 2
+ assert report.new_companies == 1
+
+
+# ── Observations ──
+
+
+async def test_every_fact_becomes_an_observation(state, config, register):
+ register(FakeProvider([[biz(
+ name="Northvale", domain="northvale.com", rank=14, rating=4.6,
+ review_count=88, is_claimed=False, has_website=True,
+ )]]))
+ await D.run_discovery(state, config, "fake", [D.DiscoveryQuery(term="roofer")])
+
+ rows = {o["signal_code"]: o for o in await state.get_observations()}
+ assert rows["SERP_RANK"]["value_num"] == 14
+ assert rows["REVIEW_RATING"]["value_num"] == pytest.approx(4.6)
+ assert rows["REVIEW_COUNT"]["value_num"] == 88
+ assert "UNCLAIMED_LISTING" in rows
+ assert "NO_WEBSITE" not in rows
+ assert rows["FOUND_IN_SERP"]["collector"] == "discover"
+
+
+async def test_unconfirmed_signals_are_not_recorded(state, config, register):
+ """The confirmation gate has to hold inside the collector, not just in
+ the UI that sets it."""
+ await state.set_signal_status("REVIEW_RATING", "rejected")
+ register(FakeProvider([[biz(domain="northvale.com", rating=4.6, review_count=88)]]))
+ await D.run_discovery(state, config, "fake", [D.DiscoveryQuery(term="roofer")])
+
+ codes = {o["signal_code"] for o in await state.get_observations()}
+ assert "REVIEW_RATING" not in codes
+ assert "REVIEW_COUNT" in codes
+
+
+async def test_a_failed_query_does_not_lose_the_successful_ones(state, config, register):
+ class Flaky(FakeProvider):
+ async def fetch(self, client, env, query):
+ self.calls += 1
+ if self.calls == 2:
+ raise httpx.ConnectError("boom")
+ return [biz(domain=f"co{self.calls}.com")]
+
+ register(Flaky([], per_query_cost=0.0))
+ q = [D.DiscoveryQuery(term="roofer", location=c) for c in ("a", "b", "c")]
+ report = await D.run_discovery(state, config, "fake", q)
+
+ assert report.new_companies == 2
+ assert len(report.errors) == 1 and "boom" in report.errors[0]
+ runs = await state.get_runs()
+ assert runs[0]["status"] == "completed"
+
+
+async def test_run_is_logged_with_what_it_actually_cost(state, config, register):
+ register(FakeProvider([[biz(domain="a.com")], [biz(domain="b.com")]],
+ per_query_cost=0.25))
+ q = [D.DiscoveryQuery(term="roofer", location=c) for c in ("a", "b")]
+ report = await D.run_discovery(state, config, "fake", q, max_spend=5.0)
+
+ runs = await state.get_runs()
+ assert runs[0]["stage"] == "discover"
+ assert runs[0]["records"] == 2
+ assert runs[0]["cost_usd"] == pytest.approx(0.5)
+ assert report.actual_cost == pytest.approx(0.5)
+
+
+# ── Provider adapters (parsing, against real response shapes) ──
+
+
+DFS_LISTINGS = {
+ "status_code": 20000,
+ "cost": 0.0132,
+ "tasks": [{"status_code": 20000, "result": [{"items": [
+ {
+ "title": "Northvale Roofing Company", "category": "Roofing contractor",
+ "place_id": "ChIJabc123", "phone": "+15550100199",
+ "url": "https://www.northvaleroofing.com/", "domain": "northvaleroofing.com",
+ "address": "1400 Example Row, Raleigh, NC",
+ "address_info": {"city": "Raleigh", "region": "North Carolina"},
+ "is_claimed": True,
+ "rating": {"value": 4.4, "votes_count": 512, "rating_max": 5},
+ "check_url": "https://google.com/maps?cid=1",
+ },
+ {
+ "title": "No Website Roofing", "category": "Roofing contractor",
+ "place_id": "ChIJxyz789", "phone": "+15550100377",
+ "url": None, "domain": None,
+ "address_info": {"city": "Denver", "region": "Colorado"},
+ "is_claimed": False,
+ "rating": {"value": 4.9, "votes_count": 31, "rating_max": 5},
+ },
+ ]}]}],
+}
+
+DFS_SERP = {
+ "status_code": 20000,
+ "cost": 0.0018,
+ "tasks": [{"status_code": 20000, "result": [{"items": [
+ {"type": "organic", "rank_group": 1, "rank_absolute": 3,
+ "position": "left", "domain": "yelp.com",
+ "url": "https://yelp.com/denver", "title": "Best Roofers"},
+ {"type": "organic", "rank_group": 14, "rank_absolute": 18,
+ "position": "left", "domain": "kettlemanroofing.com",
+ "url": "https://kettlemanroofing.com/", "title": "Kettleman Roofing"},
+ {"type": "paid", "rank_group": 1, "domain": "ads.example.com"},
+ {"type": "people_also_ask", "rank_group": 2},
+ ]}]}],
+}
+
+SERPER_RESPONSE = {"organic": [
+ {"title": "Kettleman Roofing", "link": "https://www.kettlemanroofing.com/",
+ "position": 3},
+ {"title": "Angi", "link": "https://www.angi.com/companylist/denver/roofing",
+ "position": 4},
+]}
+
+
+def _client(handler):
+ return httpx.AsyncClient(transport=httpx.MockTransport(handler))
+
+
+async def test_dataforseo_listings_adapter():
+ provider = D.DataForSEOListings()
+ env = {"dataforseo_login": "u", "dataforseo_password": "p"}
+ seen = {}
+
+ def handler(request):
+ seen["url"] = str(request.url)
+ seen["auth"] = request.headers.get("authorization")
+ return httpx.Response(200, json=DFS_LISTINGS)
+
+ async with _client(handler) as client:
+ out = await provider.fetch(client, env, D.DiscoveryQuery(
+ term="roofing contractor", location="Denver, CO",
+ coordinate="39.7,-104.9,50"))
+
+ assert seen["auth"].startswith("Basic ")
+ assert provider.last_cost == pytest.approx(0.0132)
+
+ northvale, nosite = out
+ assert northvale.domain == "northvaleroofing.com"
+ assert northvale.phone == "+15550100199"
+ assert northvale.location == "Raleigh, North Carolina"
+ assert northvale.rating == 4.4 and northvale.review_count == 512
+ assert northvale.has_website is True
+ assert northvale.external_id == "dataforseo:ChIJabc123"
+
+ # The websiteless record is the valuable one — it must survive intact.
+ assert nosite.domain == "" and nosite.has_website is False
+ assert nosite.is_claimed is False
+ assert nosite.identity() == "dataforseo:ChIJxyz789"
+
+
+async def test_dataforseo_serp_adapter_reads_rank_group_not_position():
+ """`position` in this payload means left/right column, not rank. Reading
+ it as rank would score every result identically."""
+ provider = D.DataForSEOSerp()
+ async with _client(lambda r: httpx.Response(200, json=DFS_SERP)) as client:
+ out = await provider.fetch(
+ client, {"dataforseo_login": "u", "dataforseo_password": "p"},
+ D.DiscoveryQuery(term="roofer", location="Denver", depth=30))
+
+ assert [b.rank for b in out] == [1, 14], "paid and PAA rows must be dropped"
+ assert out[1].domain == "kettlemanroofing.com"
+
+
+async def test_serper_adapter():
+ provider = D.Serper()
+ async with _client(lambda r: httpx.Response(200, json=SERPER_RESPONSE)) as client:
+ out = await provider.fetch(client, {"serper_api_key": "k"},
+ D.DiscoveryQuery(term="roofer", location="Denver"))
+ assert out[0].domain == "kettlemanroofing.com" and out[0].rank == 3
+ # Junk filtering happens in the run, so the adapter still returns Angi.
+ assert out[1].domain == "angi.com"
+
+
+async def test_dataforseo_errors_surface_the_message_not_just_the_code():
+ """A bare status code costs several wrong guesses; the body usually makes
+ the problem obvious."""
+ payload = {"status_code": 40501, "status_message": "Invalid Field: 'categories'"}
+ provider = D.DataForSEOListings()
+ with pytest.raises(RuntimeError, match="Invalid Field"):
+ async with _client(lambda r: httpx.Response(200, json=payload)) as client:
+ await provider.fetch(client, {"dataforseo_login": "u",
+ "dataforseo_password": "p"},
+ D.DiscoveryQuery(term="roofer"))
+
+
+def test_dataforseo_sandbox_is_never_the_live_host():
+ provider = D.DataForSEOListings()
+ assert "sandbox" in provider.base_url({"dataforseo_sandbox": "1"})
+ assert "sandbox" not in provider.base_url({})
+
+
+# ── The provider menu the dashboard renders ──
+
+
+def test_provider_menu_explains_cost_and_configuration():
+ menu = {m["key"]: m for m in D.provider_menu({"serper_api_key": "k"})}
+ assert menu["osm"]["configured"] is True and not menu["osm"]["needs_key"]
+ assert menu["serper"]["configured"] is True
+ assert menu["dataforseo_listings"]["configured"] is False
+ assert menu["dataforseo_listings"]["env_keys"] == [
+ "DATAFORSEO_LOGIN", "DATAFORSEO_PASSWORD"]
+ for entry in menu.values():
+ assert entry["blurb"] and entry["cost_note"] and entry["signup_url"]
+
+
+def test_the_default_provider_needs_no_account():
+ """Someone should be able to run the whole pipeline before paying anyone."""
+ default = D.PROVIDERS[D.DEFAULT_PROVIDER]
+ assert default.env_keys == ()
+ assert default.caveat, "a free source with thin coverage must say so"
+
+
+async def test_unconfigured_provider_says_which_key_is_missing(state, config):
+ report = await D.run_discovery(state, config, "dataforseo_listings",
+ [D.DiscoveryQuery(term="roofer")])
+ assert report.stopped == "not configured"
+ assert "DATAFORSEO_LOGIN" in report.errors[0]
+
+
+def test_unknown_provider_fails_loudly():
+ with pytest.raises(ValueError, match="unknown discovery provider"):
+ import asyncio
+ asyncio.run(D.run_discovery(None, None, "nope"))
+
+
+async def test_overpass_throttling_is_an_error_not_an_empty_market(monkeypatch):
+ """A silent [] on a 504 is indistinguishable from 'no businesses here',
+ which is how a throttled run gets mistaken for a clean one."""
+ monkeypatch.setattr(D.OpenStreetMap, "BACKOFF", 0.0)
+ calls = {"n": 0}
+
+ def handler(request):
+ calls["n"] += 1
+ return httpx.Response(504, text="")
+
+ provider = D.OpenStreetMap()
+ with pytest.raises(RuntimeError, match="throttling"):
+ async with _client(handler) as client:
+ await provider.fetch(client, {}, D.DiscoveryQuery(
+ term="roofer", coordinate="39.7,-104.9,40"))
+ assert calls["n"] == 2, "it should retry once before giving up"
+
+
+async def test_overpass_recovers_on_the_retry(monkeypatch):
+ monkeypatch.setattr(D.OpenStreetMap, "BACKOFF", 0.0)
+ calls = {"n": 0}
+
+ def handler(request):
+ calls["n"] += 1
+ if calls["n"] == 1:
+ return httpx.Response(504, text="")
+ return httpx.Response(200, json={"elements": [{
+ "type": "node", "id": 42,
+ "tags": {"name": "Sandpiper Exteriors", "craft": "roofer",
+ "phone": "+15550100288", "addr:city": "Denver"},
+ }]})
+
+ provider = D.OpenStreetMap()
+ async with _client(handler) as client:
+ out = await provider.fetch(client, {}, D.DiscoveryQuery(
+ term="roofer", coordinate="39.7,-104.9,40"))
+ assert len(out) == 1
+ assert out[0].name == "Sandpiper Exteriors"
+ assert out[0].external_id == "osm:node/42"
+ assert out[0].has_website is False
+
+
+async def test_overpass_needs_a_coordinate_and_says_so(caplog):
+ provider = D.OpenStreetMap()
+ async with _client(lambda r: httpx.Response(200, json={})) as client:
+ out = await provider.fetch(client, {}, D.DiscoveryQuery(
+ term="roofer", location="Denver, CO"))
+ assert out == []
+ assert "coordinate" in caplog.text.lower()
+
+
+# ── Geocoding ──
+
+
+def _geo_client(handler):
+ return httpx.AsyncClient(transport=httpx.MockTransport(handler))
+
+
+async def test_geocode_is_cached_so_a_city_is_looked_up_once(state):
+ """Nominatim permits single place lookups and forbids bulk enumeration.
+ Caching forever is what keeps us on the right side of that."""
+ calls = {"n": 0}
+
+ def handler(request):
+ calls["n"] += 1
+ assert "Harvey" in request.headers["user-agent"]
+ return httpx.Response(200, json=[{"lat": "39.7392", "lon": "-104.9903"}])
+
+ async with _geo_client(handler) as client:
+ assert await REAL_GEOCODE(state, "Denver, CO", client=client) \
+ == "39.7392,-104.9903,40"
+ assert await REAL_GEOCODE(state, "Denver, CO", client=client) \
+ == "39.7392,-104.9903,40"
+ assert await REAL_GEOCODE(state, "DENVER, co", client=client) \
+ == "39.7392,-104.9903,40"
+ assert calls["n"] == 1
+
+
+async def test_a_geocode_miss_is_cached_too(state):
+ calls = {"n": 0}
+
+ def handler(request):
+ calls["n"] += 1
+ return httpx.Response(200, json=[])
+
+ async with _geo_client(handler) as client:
+ assert await REAL_GEOCODE(state, "Nowhere XYZ", client=client) == ""
+ assert await REAL_GEOCODE(state, "Nowhere XYZ", client=client) == ""
+ assert calls["n"] == 1, "a miss must not be re-asked on every run"
+
+
+async def test_configured_coordinates_win_over_geocoding(state, monkeypatch):
+ async def boom(*a, **kw):
+ raise AssertionError("should not geocode a query that already has a point")
+ monkeypatch.setattr(D, "geocode_city", boom)
+
+ queries = [D.DiscoveryQuery(term="roofer", location="Denver, CO",
+ coordinate="1.0,2.0,25")]
+ out = await REAL_RESOLVE(state, queries)
+ assert out[0].coordinate == "1.0,2.0,25"
+
+
+async def test_one_geocode_per_city_not_per_query(state, monkeypatch):
+ """Twelve queries across four cities is four lookups, not twelve."""
+ monkeypatch.setattr(D, "geocode_city", REAL_GEOCODE)
+ calls = {"n": 0}
+
+ def handler(request):
+ calls["n"] += 1
+ return httpx.Response(200, json=[{"lat": "1.0", "lon": "2.0"}])
+
+ queries = [D.DiscoveryQuery(term=t, location=c)
+ for t in ("roofing", "gutters", "siding")
+ for c in ("Denver, CO", "Dallas, TX", "Tampa, FL", "Nashville, TN")]
+ async with _geo_client(handler) as client:
+ await REAL_RESOLVE(state, queries, client=client)
+ assert calls["n"] == 4
+ assert all(q.coordinate == "1.0000,2.0000,40" for q in queries)
+
+
+async def test_osm_refuses_a_term_it_cannot_map_instead_of_guessing():
+ """An earlier fallback to `office=company` answered "roofers in Denver"
+ with every registered office in Denver. Returning nothing is correct;
+ returning plausible junk is not."""
+ provider = D.OpenStreetMap()
+ with pytest.raises(RuntimeError, match="no category"):
+ async with _client(lambda r: httpx.Response(200, json={})) as client:
+ await provider.fetch(client, {}, D.DiscoveryQuery(
+ term="storm damage restoration", coordinate="39.7,-104.9,40"))
+
+
+async def test_osm_maps_the_trades_it_does_know():
+ provider = D.OpenStreetMap()
+ assert provider._tags("Roofing contractors") == ["craft=roofer", "shop=roofing"]
+ assert provider._tags("Residential roofing") == ["craft=roofer", "shop=roofing"]
+ assert provider._tags("Emergency plumber") == ["craft=plumber"]
+ assert provider._tags("cosmetic dentist") == ["amenity=dentist"]
+ assert provider._tags("B2B SaaS") == []
+
+
+async def test_an_unmappable_term_is_reported_not_silently_skipped(
+ state, config, register, monkeypatch):
+ """The run must surface why a query produced nothing."""
+ monkeypatch.setitem(D.PROVIDERS, "osm", D.OpenStreetMap())
+
+ async def fake_fetch(self, client, env, query):
+ raise RuntimeError("OpenStreetMap has no category for 'widgets'")
+ monkeypatch.setattr(D.OpenStreetMap, "fetch", fake_fetch)
+
+ report = await D.run_discovery(state, config, "osm",
+ [D.DiscoveryQuery(term="widgets",
+ coordinate="1,2,40")])
+ assert report.found == 0
+ assert report.errors and "no category" in report.errors[0]