Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
feat(scrape): --grid breaks the 120-per-search cap, deep email pass on by default
Mahanaicoachb111ce7faf9c
diff --git a/.claude/commands/scrape.md b/.claude/commands/scrape.mdindex 1a9225b..107d06f 100644--- a/.claude/commands/scrape.md+++ b/.claude/commands/scrape.md@@ -16,6 +16,8 @@ Use the `google-maps-scraper` skill. Execute:5. **Poll to completion in the BACKGROUND** (the Bash tool blocks foreground `sleep` — use `run_in_background: true` and read the output file when notified). Then download the CSV.6. **Save a CSV file** (the default — `scripts/scrape.py` writes `results-<id>.csv`; if you use the raw API, save the parsed lead fields to a `.csv` yourself). Then **present ONLY money-useful lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count` (+ `instagram, facebook, linkedin` if socials were requested). **Drop everything else** — especially `latitude`/`longitude`, IDs, hours, images, review blobs. Do NOT dump the full 34-column CSV into chat; show a few sample rows and tell the user the CSV path. (Include dropped fields only if the user explicitly asks.)+**All / every / entire city:** add `--grid 3` (3 x 3 map tiles, one job at a time, deduped) to get past Google's ~120 results per search. Warn that it is slower and more block-prone, then proceed.+**Emails:** on by default in this kit — the scraper visits each business website to find them, which is a bit slower. Some businesses simply don't publish an email, so coverage is partial. For a deliberately fast run with no emails, pass `--no-email`.**Socials:** when the user says yes, `scripts/scrape.py … --socials` pulls Instagram/Facebook/LinkedIn. Extraction is **0 LLM tokens** (runs in code); it only adds ~40–50 tokens/business if you load rows into chat. **Never** WebFetch each website yourself to find socials — that costs thousands of tokens.diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdindex 211c874..69ba9ea 100644--- a/.claude/skills/google-maps-scraper/SKILL.md+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -113,6 +113,13 @@ said). If yes → **use the script:** `python3 scripts/scrape.py … --socials`.The API takes a `keywords` array, so put all terms in a single job rather than firing many jobs.Do the manual curl flow only for custom job bodies (e.g. setting `proxies`).+**Whole-city runs:** when the user says all / every / entire city, or a run comes back with about 120+rows (Google's per-search cap), use `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --grid 3`.+It splits the city into 3 x 3 tiles and dedupes on `place_id`. Cells run one at a time, so it is slower+and more block-prone: warn once, don't block.+**Deep emails** are on by default whenever emails are on: rows with a website but no email get their+contact/about pages checked, and the script prints the found rate. `--no-deep-emails` skips that pass.+## Other endpoints- `GET /api/v1/jobs` — list jobs. `DELETE /api/v1/jobs/{id}` — delete a job + free disk.- Browser UI + OpenAPI docs: `http://localhost:8080` and `http://localhost:8080/api/docs`.diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex c96ecce..c0c7cdf 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -4,7 +4,7 @@ Thanks for helping improve the Google Maps Scraper Kit.- **Bugs:** open an issue with the bug report template. Include the exact command and the full error.- **Pull requests:** keep them small and focused on one change. Test with a `depth 1` scrape before you- open the PR.+ open the PR. Run `python3 scripts/test_scrape.py` before opening a PR.- **Scripts:** `scripts/scrape.py` uses the Python standard library only. Do not add pip dependencies.`scripts/scrape.sh` must run on macOS's default bash 3.2.- **Scraping engine:** changes to the engine itself belong upstream indiff --git a/README.md b/README.mdindex 15585a2..24c3b52 100644--- a/README.md+++ b/README.md@@ -49,6 +49,12 @@ No coordinates handy? Let the Python CLI geocode the city:python3 scripts/scrape.py "dentists in Denver CO" --city "Denver, CO" --depth 5 --socials```+Need the whole city, not just the first 120? Add `--grid 3`:++```bash+python3 scripts/scrape.py "dentists in Denver CO" --city "Denver, CO" --grid 3+```+### Example output`coffee_shops_austin.csv` (illustrative rows):@@ -108,7 +114,8 @@ scrapes Google Maps business listings only. It does not scrape social platforms| | ||---|---|| **8 lead fields by default** | name, phone, email, website, category, address, rating, review count. The engine captures ~34 raw fields; `--full` keeps them all. |-| **Email extraction, on by default** | Visits each business website and pulls published contact emails. |+| **Email extraction, on by default** | Visits each business website and pulls published contact emails. A second pass checks the contact and about pages of rows still missing one, then prints your found rate. |+| **Beyond the 120 cap** | `--grid 3` splits the city into 3 x 3 map tiles, runs them one at a time and dedupes on `place_id`. || **Social enrichment** | `--socials` finds Instagram, Facebook and LinkedIn profiles in code (HTTP + regex). Zero AI tokens. || **Auto-geocoding** | Type a city name. Coordinates are resolved for you via OpenStreetMap. || **Batch jobs** | Many keywords, one job, one CSV. |@@ -131,12 +138,14 @@ One keyword per line. One job. One CSV.## Honest limits-- **About 120 listings per keyword.** Google Maps stops scrolling at roughly 120 results per search.- That is Google's cap, not the kit's. To go past it, batch more specific keywords ("vegan cafes",- "espresso bars") or neighbourhoods ("coffee shops in South Congress Austin") in one job.-- **Emails only when a business publishes one.** The extractor reads the business website. Listings- with no site, or a site with no visible email, return an empty `emails` cell. Expect emails on a- fraction of rows, not all of them.+- **About 120 listings per search.** Google Maps stops scrolling at roughly 120 results per search.+ That is Google's cap, not the kit's. `--grid` is the fix: it splits the city into N x N map tiles,+ runs one search per tile in a single batch, and dedupes the results on `place_id`. Tiles run one at a+ time, so a grid is slower and more likely to trip rate limits than a single search.+- **Emails only when a business publishes one.** The extractor reads the business website, then a+ second pass checks its contact and about pages. Listings with no site, or a site with no visible+ email, return an empty `emails` cell. Instagram bios are not read. Expect emails on a fraction of+ rows, not all of them.- **Rate limits are real.** Heavy use without proxies can get your IP temporarily rate-limited byGoogle for minutes to hours. Your Google account is not affected. See [Responsible use](#responsible-use).- **Apple Silicon runs the engine under amd64 emulation.** It works. It is slower than native.@@ -151,7 +160,7 @@ One keyword per line. One job. One CSV.| Cost | Free (MIT) | Free | Per-result credits or subscription | Free || API key or account | None | Account | Required | None || Where data goes | Stays on your machine | Provider's servers | Provider's servers | Your machine |-| Rows per query | Up to ~120 per keyword, unlimited keywords | Close to zero | Up to ~120 per keyword, unlimited keywords | A handful |+| Rows per query | Whole city with --grid, deduped | Close to zero | Up to ~120 per keyword, unlimited keywords | A handful || Emails and socials | Built in | No | Often a paid add-on | Manual || Output | Clean CSV, 8 lead fields | Chat text | CSV or JSON | Spreadsheet || Driven by an AI agent | Yes, skill included | n/a | No | No |@@ -191,8 +200,9 @@ Name, phone, email, website, category, address, rating and review count by defaufor Instagram, Facebook and LinkedIn, or `--full` for all ~34 raw fields.**How many leads can I scrape?**-Up to about 120 per keyword, and as many keywords as you like in one batch job. For large or-repeated runs, add proxies (see [Responsible use](#responsible-use)).+Up to about 120 per search. Add `--grid 3` to split the city into 9 map tiles and get past that cap,+with duplicates removed. Batch as many keywords as you like in one job. For large or repeated runs,+add proxies (see [Responsible use](#responsible-use)).**Does it work on Windows, Linux and Apple Silicon Macs?**Yes. Anywhere Docker runs. On Windows, use `scripts/scrape.py`, or run `scrape.sh` under WSL2 or Gitdiff --git a/SETUP.md b/SETUP.mdindex 8f23d14..79652cc 100644--- a/SETUP.md+++ b/SETUP.md@@ -100,6 +100,12 @@ python3 scripts/scrape.py "cafes in Austin TX" --city "Austin, TX" --depth 5python3 scripts/scrape.py --keywords-file examples/queries.example.txt --city "Denver, CO"```+**Go past Google's ~120 results per search** by splitting the city into a 3 x 3 grid of map tiles+(9 jobs, run one at a time, duplicates removed):+```bash+python3 scripts/scrape.py "plumbers in Phoenix AZ" --city "Phoenix, AZ" --grid 3+```+---## 4. Use it with Claude (the autopilot way) 🤖diff --git a/scripts/scrape.py b/scripts/scrape.pyindex a3219aa..f6c9d38 100755--- a/scripts/scrape.py+++ b/scripts/scrape.py@@ -10,12 +10,18 @@ Auto-geocode (no coordinates needed):Batch (one job, many keywords) from a file (one keyword per line):python3 scripts/scrape.py --keywords-file examples/queries.example.txt --city "Denver, CO"+Whole city past Google's ~120 cap (3x3 map tiles, run one at a time, deduped on place_id):+ python3 scripts/scrape.py "plumbers in Phoenix AZ" --city "Phoenix, AZ" --grid 3++Skip the contact-page email pass (on by default whenever emails are on):+ python3 scripts/scrape.py "gyms in Miami FL" --city "Miami, FL" --no-deep-emails+Notes:- Required by the API: keywords, lat/lon (strings), max_time (SECONDS). This script fills them in.- Geocoding uses OpenStreetMap Nominatim (free, no key). Please be gentle: it allows ~1 request/secand requires a descriptive User-Agent (set below). Don't loop it aggressively."""-import argparse, csv, io, json, os, re, sys, time, urllib.request, urllib.parse, urllib.error+import argparse, csv, functools, io, json, math, os, re, sys, time, urllib.request, urllib.parse, urllib.errorfrom concurrent.futures import ThreadPoolExecutorENV_KEYS = ("SCRAPER_BASE_URL", "SCRAPER_PORT", "SCRAPER_API_KEY")@@ -41,7 +47,7 @@ KEY = os.environ.get("SCRAPER_API_KEY", "")# Money-useful LEAD fields only — what you actually use to contact/qualify a lead.# Everything else (geo coordinates, IDs, hours, images, reviews blobs…) is dropped by default.LEAD = ["title", "phone", "emails", "website", "category", "address", "review_rating", "review_count"]-UA = "google-maps-scraper-kit/1.0 (https://github.com/Mahanaicoach/google-maps-scraper-kit)"+UA = "google-maps-scraper-kit/1.1 (https://github.com/Mahanaicoach/google-maps-scraper-kit)"def req(method, path, body=None):@@ -54,8 +60,9 @@ def req(method, path, body=None):return resp.status, resp.read()-def geocode(place):- """City/place name -> ('lat','lon') strings via Nominatim, or None."""+def geocode(place, with_bbox=False):+ """City/place name -> ('lat','lon') strings via Nominatim, or None.+ with_bbox=True returns ('lat','lon', [south, north, west, east]) instead."""q = urllib.parse.urlencode({"format": "json", "limit": 1, "q": place})url = f"https://nominatim.openstreetmap.org/search?{q}"r = urllib.request.Request(url, headers={"User-Agent": UA})@@ -64,12 +71,87 @@ def geocode(place):hits = json.loads(resp.read())time.sleep(1) # respect Nominatim's ~1 req/sec policyif hits:- return str(hits[0]["lat"]), str(hits[0]["lon"])+ ll = (str(hits[0]["lat"]), str(hits[0]["lon"]))+ return ll + (hits[0].get("boundingbox"),) if with_bbox else llexcept Exception as e:print(f" (geocoding failed: {e})", file=sys.stderr)return None+def grid_cells(bbox, n):+ """Split a Nominatim bbox [south, north, west, east] into n x n cells -> [{lat, lon, radius, zoom}].+ radius (m) = 1.15 x half the cell diagonal (equirectangular), at least 1000.+ zoom = the Maps zoom whose ~1000px viewport is about as wide as the cell (11..17), so each+ search is launched looking at its own tile instead of a city-wide or block-wide view."""+ s, nn, w, e = (float(x) for x in bbox)+ dlat, dlon = (nn - s) / n, (e - w) / n+ m_per_deg = 6371000 * math.pi / 180+ cells = []+ for i in range(n):+ for j in range(n):+ lat, lon = s + (i + 0.5) * dlat, w + (j + 0.5) * dlon+ cos = math.cos(math.radians(lat))+ width = max(1.0, dlon * m_per_deg * cos)+ diag = math.hypot(dlat * m_per_deg, width)+ zoom = round(math.log2(156543.03 * cos * 1000 / width)) # 156543 m/px at zoom 0+ cells.append({"lat": f"{lat:.6f}", "lon": f"{lon:.6f}",+ "radius": max(1000, math.ceil(1.15 * diag / 2)),+ "zoom": min(17, max(11, zoom))})+ return cells+++def in_bbox(row, bbox, margin=0.10):+ """True if the row's latitude/longitude sit inside bbox [south, north, west, east] grown by+ `margin` of its size on each side. Rows with no usable coordinates are kept (True)."""+ try:+ lat, lon = float(row.get("latitude")), float(row.get("longitude"))+ except (TypeError, ValueError):+ return True+ s, n, w, e = (float(x) for x in bbox)+ my, mx = (n - s) * margin, (e - w) * margin+ return s - my <= lat <= n + my and w - mx <= lon <= e + mx+++def dedupe(rows):+ """Keep the first row per place_id, else cid, else link, else title+address."""+ seen, out = set(), []+ for r in rows:+ key = next((f"{k}:{r[k]}" for k in ("place_id", "cid", "link") if r.get(k)),+ f"ta:{r.get('title', '')}|{r.get('address', '')}")+ if key not in seen:+ seen.add(key); out.append(r)+ return out+++def run_job(body):+ """Create one engine job, poll it, download it -> (job_id, raw CSV rows). Raises RuntimeError."""+ try:+ _, raw = req("POST", "/api/v1/jobs", body)+ except urllib.error.HTTPError as e:+ raise RuntimeError(f"✗ Create failed: HTTP {e.code} — {e.read().decode()[:200]}")+ job_id = json.loads(raw).get("id")+ if not job_id:+ raise RuntimeError("✗ No job id returned.")+ print(f" job id: {job_id}")++ print("▶ Polling…")+ status = None+ for i in range(120):+ _, raw = req("GET", f"/api/v1/jobs/{job_id}")+ status = json.loads(raw).get("Status")+ print(f"\r status: {str(status):<10} (attempt {i + 1})", end="", flush=True)+ if status == "ok":+ print(); break+ if status == "failed":+ raise RuntimeError("\n✗ Job failed. If this keeps happening you may be rate-limited — wait or add proxies.")+ time.sleep(8)+ else:+ raise RuntimeError("\n✗ Timed out.")++ _, raw = req("GET", f"/api/v1/jobs/{job_id}/download")+ return job_id, list(csv.DictReader(io.StringIO(raw.decode("utf-8", "replace"))))++def collect_keywords(a):kws = []if a.keyword:@@ -112,6 +194,11 @@ def _fetch_html(url, timeout=10):except Exception:return ""[email protected]_cache(maxsize=512) # ponytail: homepages only; evicted pages just refetch+def _fetch_cached(url):+ """Homepage fetch shared by the deep email pass and --socials (one fetch per site)."""+ return _fetch_html(url, timeout=8)+def _find_socials(html):out = {"instagram": "", "facebook": "", "linkedin": ""}for plat, rx in SOCIAL_RE.items():@@ -132,7 +219,7 @@ def enrich_socials(results, workers=8):todo = [r for r in results if r.get("website")]done = 0def work(r):- r.update(_find_socials(_fetch_html(r["website"])))+ r.update(_find_socials(_fetch_cached(r["website"])))with ThreadPoolExecutor(max_workers=workers) as ex:for _ in ex.map(work, todo):done += 1@@ -140,6 +227,71 @@ def enrich_socials(results, workers=8):print()+# ── Deep email pass: contact/about pages for rows the engine left without an email ──+EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")+MAILTO_RE = re.compile(r"mailto:([^\"'>\s?]+)", re.I)+HREF_RE = re.compile(r"<a\s[^>]*?href\s*=\s*[\"']([^\"']+)[\"'][^>]*>(.*?)</a>", re.I | re.S)+_JUNK_EXT = (".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp", ".css", ".js")+_JUNK_DOMAIN = ("example", "sentry", "wixpress", "wix.com", "godaddy", "squarespace", "wordpress",+ "domain.com", "email.com", "yourdomain", "schema.org", "w3.org", "sentry.io")+_JUNK_LOCAL = {"name", "email", "user", "you", "someone"}++def _host(url):+ if not url.startswith(("http://", "https://")):+ url = "http://" + url+ h = (urllib.parse.urlparse(url).hostname or "").lower()+ return h[4:] if h.startswith("www.") else h++def extract_emails(html, site_host):+ """Emails from mailto: links and plain text, junk dropped, site-domain first, max 3."""+ site = _host(site_host or "")+ found = []+ for e in MAILTO_RE.findall(html or "") + EMAIL_RE.findall(html or ""):+ e = urllib.parse.unquote(e).strip().lower()+ local, _, dom = e.rpartition("@")+ if (not EMAIL_RE.fullmatch(e) or e.endswith(_JUNK_EXT) or local in _JUNK_LOCAL+ or any(j in dom for j in _JUNK_DOMAIN) or e in found):+ continue+ found.append(e)+ on_site = lambda e: site and (e.endswith("@" + site) or e.endswith("." + site))+ return sorted(found, key=lambda e: not on_site(e))[:3]++def contact_links(html, base_url):+ """Up to 3 same-site contact/about URLs: discovered links first, then fixed fallbacks."""+ if not base_url.startswith(("http://", "https://")):+ base_url = "http://" + base_url+ host, out = _host(base_url), []+ for href, text in HREF_RE.findall(html or ""):+ if not re.search(r"contact|about", href + " " + text, re.I):+ continue+ url = urllib.parse.urldefrag(urllib.parse.urljoin(base_url, href.strip()))[0]+ if url.startswith(("http://", "https://")) and _host(url) == host and url not in out:+ out.append(url)+ for path in ("/contact", "/contact-us", "/about", "/about-us"):+ url = urllib.parse.urljoin(base_url, path)+ if len(out) < 3 and url not in out:+ out.append(url)+ return out[:3]++def enrich_emails(results, workers=8):+ """Fill empty `emails` from the homepage, then up to 3 contact/about pages. Zero LLM tokens."""+ todo = [r for r in results if r.get("website") and not r.get("emails")]+ done = 0+ def work(r):+ site = r["website"]+ home = _fetch_cached(site)+ found = extract_emails(home, site)+ if home and not found:+ pages = "".join(_fetch_html(u, timeout=8) for u in contact_links(home, site))+ found = extract_emails(pages, site)+ r["emails"] = ", ".join(found)+ with ThreadPoolExecutor(max_workers=workers) as ex:+ for _ in ex.map(work, todo):+ done += 1+ print(f"\r emails: {done}/{len(todo)} sites checked", end="", flush=True)+ print()++def main():ap = argparse.ArgumentParser(description="Scrape Google Maps business listings.")ap.add_argument("keyword", nargs="?", help='e.g. "coffee shops in Austin TX"')@@ -154,6 +306,11 @@ def main():ap.add_argument("--email", dest="email", action="store_true", default=True, help=argparse.SUPPRESS)ap.add_argument("--no-email", dest="email", action="store_false",help="skip email extraction for a faster run (emails are on by default)")+ ap.add_argument("--no-deep-emails", dest="deep_emails", action="store_false",+ help="skip the second email pass over contact/about pages (on whenever emails are on)")+ ap.add_argument("--grid", type=int, metavar="N",+ help="split the --city area into N x N map tiles, one job each, deduped "+ "(breaks Google's ~120 per search cap; max 10)")ap.add_argument("--max-time", type=int, default=600, help="job time limit in SECONDS")ap.add_argument("--out", default=None, help="output file path (default: results-<id>.csv). ""The extension picks the format: .csv (default) or .json")@@ -168,9 +325,24 @@ def main():if not keywords:sys.exit("✗ No keywords. Pass a keyword, --keyword, or --keywords-file.")+ if a.grid is not None:+ if not a.city:+ sys.exit("✗ --grid needs --city: the city's bounding box is what gets split into tiles.")+ if not 1 <= a.grid <= 10:+ sys.exit("✗ --grid must be between 1 and 10 (10 x 10 = 100 jobs is the ceiling).")+# Resolve coordinates: explicit lat/lon, else geocode --city, else geocode first keyword.+ # --grid always geocodes --city, because it needs the city's bounding box.lat, lon = a.lat, a.lon- if not (lat and lon):+ if a.grid:+ print(f"▶ Geocoding \"{a.city}\"…")+ geo = geocode(a.city, with_bbox=True)+ if not geo or not geo[2]:+ sys.exit("✗ Could not resolve a bounding box. Try a clearer --city.")+ lat, lon, bbox = geo+ cells = grid_cells(bbox, a.grid)+ print(f" → bbox S{bbox[0]} N{bbox[1]} W{bbox[2]} E{bbox[3]}, {len(cells)} tiles")+ elif not (lat and lon):place = a.city or keywords[0]print(f"▶ Geocoding \"{place}\"…")coords = geocode(place)@@ -184,8 +356,8 @@ def main():"Use --no-email for a fast run.")# --- WARN, don't block: flag high-volume jobs (proceed anyway) ---- if a.depth >= 15 or len(keywords) >= 10:- print("⚠️ Heads-up: this is a large job (high depth or many keywords).")+ if a.depth >= 15 or len(keywords) >= 10 or (a.grid or 1) ** 2 >= 4:+ print("⚠️ Heads-up: this is a large job (high depth, many keywords or a grid).")print(" Running it back-to-back without proxies can get your IP temporarily rate-limited by")print(" Google (clears in minutes–hours; returns empty/failed jobs meanwhile). For big or")print(" repeated runs, add proxies. Proceeding…\n")@@ -199,34 +371,36 @@ def main():body = {"name": "scrape-py", "keywords": keywords, "lang": "en", "zoom": 15,"lat": str(lat), "lon": str(lon), "fast_mode": False, "radius": 10000,"depth": a.depth, "email": a.email, "max_time": a.max_time}- print(f"▶ Creating job: {len(keywords)} keyword(s) @ {lat},{lon} depth={a.depth} email={a.email}")- for k in keywords:- print(f" • {k}")- try:- _, raw = req("POST", "/api/v1/jobs", body)- except urllib.error.HTTPError as e:- sys.exit(f"✗ Create failed: HTTP {e.code} — {e.read().decode()[:200]}")- job_id = json.loads(raw).get("id")- if not job_id:- sys.exit("✗ No job id returned.")- print(f" job id: {job_id}")-- print("▶ Polling…")- status = None- for i in range(120):- _, raw = req("GET", f"/api/v1/jobs/{job_id}")- status = json.loads(raw).get("Status")- print(f"\r status: {str(status):<10} (attempt {i + 1})", end="", flush=True)- if status == "ok":- print(); break- if status == "failed":- sys.exit("\n✗ Job failed. If this keeps happening you may be rate-limited — wait or add proxies.")- time.sleep(8)+ if not a.grid:+ print(f"▶ Creating job: {len(keywords)} keyword(s) @ {lat},{lon} depth={a.depth} email={a.email}")+ for k in keywords:+ print(f" • {k}")+ try:+ job_id, rows = run_job(body)+ except RuntimeError as e:+ sys.exit(str(e))else:- sys.exit("\n✗ Timed out.")-- _, raw = req("GET", f"/api/v1/jobs/{job_id}/download")- rows = list(csv.DictReader(io.StringIO(raw.decode("utf-8", "replace"))))+ print(f"▶ Grid {a.grid}x{a.grid}: {len(cells)} jobs, one at a time, "+ f"{len(keywords)} keyword(s) depth={a.depth} email={a.email}")+ for k in keywords:+ print(f" • {k}")+ job_id, rows = None, []+ for i, cell in enumerate(cells, 1): # sequential on purpose: never parallel jobs against Google+ print(f"▶ Cell {i}/{len(cells)} @ {cell['lat']},{cell['lon']} radius={cell['radius']}m")+ try:+ jid, cell_rows = run_job(dict(body, **cell))+ except Exception as e:+ print(f"\n ⚠ cell {i}/{len(cells)} skipped: {str(e).strip()}")+ continue+ job_id = job_id or jid+ rows = dedupe(rows + cell_rows)+ print(f" cell {i}/{len(cells)}: {len(cell_rows)} rows, {len(rows)} unique so far")+ if job_id is None:+ sys.exit("✗ Every grid cell failed. You may be rate-limited — wait or add proxies.")+ # radius does not clip results in normal mode, so tiles leak outside the city+ kept = [r for r in rows if in_bbox(r, bbox)]+ print(f" dropped {len(rows) - len(kept)} rows outside the city box")+ rows = keptif a.full:fields = list(rows[0].keys()) if rows else LEADelif a.fields:@@ -236,6 +410,15 @@ def main():results = [{k: r.get(k, "") for k in fields} for r in rows]print(f"✓ Done — {len(results)} businesses (fields: {', '.join(fields)}).")+ if a.email and a.deep_emails and "emails" in fields:+ print("▶ Deep email pass: checking contact/about pages for rows with no email…")+ engine = sum(1 for r in results if r.get("emails"))+ enrich_emails(results)+ total = sum(1 for r in results if r.get("emails"))+ n = len(results)+ print(f" emails: {engine}/{n} from the engine, +{total - engine} from contact pages "+ f"= {total}/{n} ({round(100 * total / n) if n else 0}%)")+if a.socials:print("▶ Finding socials (Instagram / Facebook / LinkedIn) on each website…")print(" ℹ Cost: this runs in CODE = 0 LLM tokens. It only adds ~40–50 tokens per business")diff --git a/scripts/test_scrape.py b/scripts/test_scrape.pynew file mode 100644index 0000000..0588f1d--- /dev/null+++ b/scripts/test_scrape.py@@ -0,0 +1,183 @@+#!/usr/bin/env python3+"""Offline tests for scripts/scrape.py (standard library only, no network).+Run: python3 scripts/test_scrape.py"""+import csv, importlib.util, io, json, os, sys, tempfile, threading, unittest+from contextlib import redirect_stdout+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from unittest import mock++_spec = importlib.util.spec_from_file_location(+ "scrape", os.path.join(os.path.dirname(os.path.abspath(__file__)), "scrape.py"))+scrape = importlib.util.module_from_spec(_spec)+_spec.loader.exec_module(scrape)++BBOX = ["30.0", "30.5", "-98.0", "-97.5"] # Nominatim order: south, north, west, east+++class GridCells(unittest.TestCase):+ def test_count_centers_radius(self):+ cells = scrape.grid_cells(BBOX, 3)+ self.assertEqual(len(cells), 9)+ for c in cells:+ self.assertTrue(30.0 < float(c["lat"]) < 30.5)+ self.assertTrue(-98.0 < float(c["lon"]) < -97.5)+ self.assertGreaterEqual(c["radius"], 1000)+ self.assertEqual(len(c["lat"].split(".")[1]), 6)+ self.assertEqual(c["zoom"], 13) # ~16 km tiles: city-district view, not a 3 km block++ def test_zoom_follows_tile_size(self):+ self.assertEqual(scrape.grid_cells(BBOX, 1)[0]["zoom"], 11) # ~48 km: clamped+ self.assertEqual(scrape.grid_cells(["0", "0.001", "0", "0.001"], 1)[0]["zoom"], 17) # 110 m: clamped++ def test_one_cell_is_bbox_center(self):+ (c,) = scrape.grid_cells(BBOX, 1)+ self.assertEqual((c["lat"], c["lon"]), ("30.250000", "-97.750000"))+ self.assertGreater(c["radius"], 30000) # ~0.5 x 0.5 degree box++ def test_tiny_box_floors_at_1000(self):+ self.assertEqual(scrape.grid_cells(["0", "0.001", "0", "0.001"], 2)[0]["radius"], 1000)+++class Dedupe(unittest.TestCase):+ def test_key_order(self):+ rows = [+ {"place_id": "P1", "cid": "C1", "title": "a"},+ {"place_id": "P1", "cid": "C9", "title": "dup place_id"},+ {"cid": "C2", "link": "L1"},+ {"cid": "C2", "link": "L9", "title": "dup cid"},+ {"link": "L3"},+ {"link": "L3", "title": "dup link"},+ {"title": "Joe's", "address": "1 Main"},+ {"title": "Joe's", "address": "1 Main", "phone": "dup title+address"},+ {"title": "Joe's", "address": "2 Main"},+ ]+ out = scrape.dedupe(rows)+ self.assertEqual(len(out), 5)+ self.assertEqual(out[0]["title"], "a") # first occurrence wins+ self.assertNotIn("dup", json.dumps(out))+++class InBbox(unittest.TestCase):+ def test_margin_and_missing(self):+ self.assertTrue(scrape.in_bbox({"latitude": "30.2", "longitude": "-97.7"}, BBOX))+ self.assertTrue(scrape.in_bbox({"latitude": "30.54", "longitude": "-97.7"}, BBOX)) # inside 10%+ self.assertFalse(scrape.in_bbox({"latitude": "30.56", "longitude": "-97.7"}, BBOX)) # past 10%+ self.assertFalse(scrape.in_bbox({"latitude": "30.2", "longitude": "-98.1"}, BBOX))+ self.assertTrue(scrape.in_bbox({"latitude": "", "longitude": ""}, BBOX))+ self.assertTrue(scrape.in_bbox({"latitude": "n/a"}, BBOX))+ self.assertTrue(scrape.in_bbox({}, BBOX))+++class ExtractEmails(unittest.TestCase):+ def test_mailto_regex_junk_preference_cap(self):+ html = """+ <a href="mailto:[email protected]?subject=Hi">Mail</a>+ <a href="mailto:[email protected]">info</a>+ Write to [email protected] or [email protected].+ """+ out = scrape.extract_emails(html, "www.acmeplumbing.com")++ def test_empty(self):+ self.assertEqual(scrape.extract_emails("", "x.com"), [])+ self.assertEqual(scrape.extract_emails("no emails here", "x.com"), [])+++class ContactLinks(unittest.TestCase):+ def test_discovers_same_site_only(self):+ html = """<a href="/contact-us#form">Get in touch</a>+ <a href="https://www.acme.com/team">About us</a>+ <a href="https://facebook.com/acme/about">FB</a>+ <a href="mailto:[email protected]">contact</a>+ <a href="/services">Services</a>"""+ out = scrape.contact_links(html, "https://acme.com")+ self.assertEqual(out[:2], ["https://acme.com/contact-us", "https://www.acme.com/team"])+ self.assertEqual(len(out), 3)+ self.assertEqual(out[2], "https://acme.com/contact") # first fixed fallback+ self.assertFalse(any("facebook" in u or "mailto" in u for u in out))++ def test_fallbacks_and_max(self):+ self.assertEqual(scrape.contact_links("", "acme.com"),+ ["http://acme.com/contact", "http://acme.com/contact-us", "http://acme.com/about"])+ many = "".join(f'<a href="/contact{i}">x</a>' for i in range(6))+ self.assertEqual(len(scrape.contact_links(many, "http://acme.com")), 3)+++# ── End to end against a mock engine ──+FIELDS = ["input_id", "link", "title", "category", "address", "website", "phone", "review_count",+ "review_rating", "latitude", "longitude", "cid", "place_id", "emails"]++def _row(pid, lat="30.2"):+ return {"title": f"Biz {pid}", "place_id": pid, "cid": "c" + pid, "latitude": lat,+ "longitude": "-97.7", "phone": "+1 555", "emails": f"{pid.lower()}@biz.net"}++CELL_ROWS = {"job-1": [_row("A"), _row("B"), _row("C")],+ "job-2": [_row("C"), _row("D")],+ "job-3": None, # this cell fails+ "job-4": [_row("A"), _row("E"), _row("F", lat="40.0")]} # F is outside the city box+++class MockEngine(BaseHTTPRequestHandler):+ created, polls = 0, {}++ def log_message(self, *a):+ pass++ def _send(self, code, body, ctype="application/json"):+ data = body.encode()+ self.send_response(code)+ self.send_header("Content-Type", ctype)+ self.send_header("Content-Length", str(len(data)))+ self.end_headers()+ self.wfile.write(data)++ def do_POST(self):+ self.rfile.read(int(self.headers.get("Content-Length", 0)))+ MockEngine.created += 1+ self._send(201, json.dumps({"id": f"job-{MockEngine.created}"}))++ def do_GET(self):+ parts = self.path.strip("/").split("/") # api v1 jobs [id [download]]+ if len(parts) == 3:+ return self._send(200, "[]")+ jid = parts[3]+ if len(parts) == 5:+ buf = io.StringIO()+ w = csv.DictWriter(buf, fieldnames=FIELDS, restval="")+ w.writeheader(); w.writerows(CELL_ROWS[jid])+ return self._send(200, buf.getvalue(), "text/csv")+ n = MockEngine.polls[jid] = MockEngine.polls.get(jid, 0) + 1+ status = "failed" if CELL_ROWS[jid] is None else ("ok" if n >= 2 else "working")+ self._send(200, json.dumps({"Status": status}))+++class EndToEnd(unittest.TestCase):+ def test_grid_run(self):+ srv = ThreadingHTTPServer(("127.0.0.1", 0), MockEngine)+ threading.Thread(target=srv.serve_forever, daemon=True).start()+ out = os.path.join(tempfile.mkdtemp(), "leads.csv")+ argv = ["scrape.py", "plumbers in Austin TX", "--city", "Austin, TX",+ "--grid", "2", "--no-deep-emails", "--out", out]+ log = io.StringIO()+ try:+ with mock.patch.object(scrape, "BASE", f"http://127.0.0.1:{srv.server_port}"), \+ mock.patch.object(scrape, "geocode", return_value=("30.25", "-97.75", BBOX)), \+ mock.patch.object(scrape.time, "sleep"), \+ mock.patch.object(sys, "argv", argv), redirect_stdout(log):+ scrape.main()+ finally:+ srv.shutdown(); srv.server_close()+ text = log.getvalue()+ self.assertIn("cell 4/4: 3 rows, 6 unique so far", text)+ self.assertIn("cell 3/4 skipped", text)+ self.assertIn("dropped 1 rows outside the city box", text)+ with open(out, newline="") as f:+ rows = list(csv.reader(f))+ self.assertEqual(rows[0], scrape.LEAD)+ self.assertEqual([r[0] for r in rows[1:]], ["Biz A", "Biz B", "Biz C", "Biz D", "Biz E"])+++if __name__ == "__main__":+ unittest.main(verbosity=2)