Skip to content

Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.

feat(scrape): --grid breaks the 120-per-search cap, deep email pass on by default

Mahanaicoachb111ce7faf9c
diff --git a/.claude/commands/scrape.md b/.claude/commands/scrape.md
index 1a9225b..107d06f 100644
--- a/.claude/commands/scrape.md
+++ b/.claude/commands/scrape.md
@@ -16,6 +16,8 @@ Use the `google-maps-scraper` skill. Execute:
5. **Poll to completion in the BACKGROUND** (the Bash tool blocks foreground `sleep` — use `run_in_background: true` and read the output file when notified). Then download the CSV.
6. **Save a CSV file** (the default — `scripts/scrape.py` writes `results-<id>.csv`; if you use the raw API, save the parsed lead fields to a `.csv` yourself). Then **present ONLY money-useful lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count` (+ `instagram, facebook, linkedin` if socials were requested). **Drop everything else** — especially `latitude`/`longitude`, IDs, hours, images, review blobs. Do NOT dump the full 34-column CSV into chat; show a few sample rows and tell the user the CSV path. (Include dropped fields only if the user explicitly asks.)
+**All / every / entire city:** add `--grid 3` (3 x 3 map tiles, one job at a time, deduped) to get past Google's ~120 results per search. Warn that it is slower and more block-prone, then proceed.
+
**Emails:** on by default in this kit — the scraper visits each business website to find them, which is a bit slower. Some businesses simply don't publish an email, so coverage is partial. For a deliberately fast run with no emails, pass `--no-email`.
**Socials:** when the user says yes, `scripts/scrape.py … --socials` pulls Instagram/Facebook/LinkedIn. Extraction is **0 LLM tokens** (runs in code); it only adds ~40–50 tokens/business if you load rows into chat. **Never** WebFetch each website yourself to find socials — that costs thousands of tokens.
diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.md
index 211c874..69ba9ea 100644
--- a/.claude/skills/google-maps-scraper/SKILL.md
+++ b/.claude/skills/google-maps-scraper/SKILL.md
@@ -113,6 +113,13 @@ said). If yes → **use the script:** `python3 scripts/scrape.py … --socials`.
The API takes a `keywords` array, so put all terms in a single job rather than firing many jobs.
Do the manual curl flow only for custom job bodies (e.g. setting `proxies`).
+**Whole-city runs:** when the user says all / every / entire city, or a run comes back with about 120
+rows (Google's per-search cap), use `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --grid 3`.
+It splits the city into 3 x 3 tiles and dedupes on `place_id`. Cells run one at a time, so it is slower
+and more block-prone: warn once, don't block.
+**Deep emails** are on by default whenever emails are on: rows with a website but no email get their
+contact/about pages checked, and the script prints the found rate. `--no-deep-emails` skips that pass.
+
## Other endpoints
- `GET /api/v1/jobs` — list jobs. `DELETE /api/v1/jobs/{id}` — delete a job + free disk.
- Browser UI + OpenAPI docs: `http://localhost:8080` and `http://localhost:8080/api/docs`.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index c96ecce..c0c7cdf 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -4,7 +4,7 @@ Thanks for helping improve the Google Maps Scraper Kit.
- **Bugs:** open an issue with the bug report template. Include the exact command and the full error.
- **Pull requests:** keep them small and focused on one change. Test with a `depth 1` scrape before you
- open the PR.
+ open the PR. Run `python3 scripts/test_scrape.py` before opening a PR.
- **Scripts:** `scripts/scrape.py` uses the Python standard library only. Do not add pip dependencies.
`scripts/scrape.sh` must run on macOS's default bash 3.2.
- **Scraping engine:** changes to the engine itself belong upstream in
diff --git a/README.md b/README.md
index 15585a2..24c3b52 100644
--- a/README.md
+++ b/README.md
@@ -49,6 +49,12 @@ No coordinates handy? Let the Python CLI geocode the city:
python3 scripts/scrape.py "dentists in Denver CO" --city "Denver, CO" --depth 5 --socials
```
+Need the whole city, not just the first 120? Add `--grid 3`:
+
+```bash
+python3 scripts/scrape.py "dentists in Denver CO" --city "Denver, CO" --grid 3
+```
+
### Example output
`coffee_shops_austin.csv` (illustrative rows):
@@ -108,7 +114,8 @@ scrapes Google Maps business listings only. It does not scrape social platforms
| | |
|---|---|
| **8 lead fields by default** | name, phone, email, website, category, address, rating, review count. The engine captures ~34 raw fields; `--full` keeps them all. |
-| **Email extraction, on by default** | Visits each business website and pulls published contact emails. |
+| **Email extraction, on by default** | Visits each business website and pulls published contact emails. A second pass checks the contact and about pages of rows still missing one, then prints your found rate. |
+| **Beyond the 120 cap** | `--grid 3` splits the city into 3 x 3 map tiles, runs them one at a time and dedupes on `place_id`. |
| **Social enrichment** | `--socials` finds Instagram, Facebook and LinkedIn profiles in code (HTTP + regex). Zero AI tokens. |
| **Auto-geocoding** | Type a city name. Coordinates are resolved for you via OpenStreetMap. |
| **Batch jobs** | Many keywords, one job, one CSV. |
@@ -131,12 +138,14 @@ One keyword per line. One job. One CSV.
## Honest limits
-- **About 120 listings per keyword.** Google Maps stops scrolling at roughly 120 results per search.
- That is Google's cap, not the kit's. To go past it, batch more specific keywords ("vegan cafes",
- "espresso bars") or neighbourhoods ("coffee shops in South Congress Austin") in one job.
-- **Emails only when a business publishes one.** The extractor reads the business website. Listings
- with no site, or a site with no visible email, return an empty `emails` cell. Expect emails on a
- fraction of rows, not all of them.
+- **About 120 listings per search.** Google Maps stops scrolling at roughly 120 results per search.
+ That is Google's cap, not the kit's. `--grid` is the fix: it splits the city into N x N map tiles,
+ runs one search per tile in a single batch, and dedupes the results on `place_id`. Tiles run one at a
+ time, so a grid is slower and more likely to trip rate limits than a single search.
+- **Emails only when a business publishes one.** The extractor reads the business website, then a
+ second pass checks its contact and about pages. Listings with no site, or a site with no visible
+ email, return an empty `emails` cell. Instagram bios are not read. Expect emails on a fraction of
+ rows, not all of them.
- **Rate limits are real.** Heavy use without proxies can get your IP temporarily rate-limited by
Google for minutes to hours. Your Google account is not affected. See [Responsible use](#responsible-use).
- **Apple Silicon runs the engine under amd64 emulation.** It works. It is slower than native.
@@ -151,7 +160,7 @@ One keyword per line. One job. One CSV.
| Cost | Free (MIT) | Free | Per-result credits or subscription | Free |
| API key or account | None | Account | Required | None |
| Where data goes | Stays on your machine | Provider's servers | Provider's servers | Your machine |
-| Rows per query | Up to ~120 per keyword, unlimited keywords | Close to zero | Up to ~120 per keyword, unlimited keywords | A handful |
+| Rows per query | Whole city with --grid, deduped | Close to zero | Up to ~120 per keyword, unlimited keywords | A handful |
| Emails and socials | Built in | No | Often a paid add-on | Manual |
| Output | Clean CSV, 8 lead fields | Chat text | CSV or JSON | Spreadsheet |
| Driven by an AI agent | Yes, skill included | n/a | No | No |
@@ -191,8 +200,9 @@ Name, phone, email, website, category, address, rating and review count by defau
for Instagram, Facebook and LinkedIn, or `--full` for all ~34 raw fields.
**How many leads can I scrape?**
-Up to about 120 per keyword, and as many keywords as you like in one batch job. For large or
-repeated runs, add proxies (see [Responsible use](#responsible-use)).
+Up to about 120 per search. Add `--grid 3` to split the city into 9 map tiles and get past that cap,
+with duplicates removed. Batch as many keywords as you like in one job. For large or repeated runs,
+add proxies (see [Responsible use](#responsible-use)).
**Does it work on Windows, Linux and Apple Silicon Macs?**
Yes. Anywhere Docker runs. On Windows, use `scripts/scrape.py`, or run `scrape.sh` under WSL2 or Git
diff --git a/SETUP.md b/SETUP.md
index 8f23d14..79652cc 100644
--- a/SETUP.md
+++ b/SETUP.md
@@ -100,6 +100,12 @@ python3 scripts/scrape.py "cafes in Austin TX" --city "Austin, TX" --depth 5
python3 scripts/scrape.py --keywords-file examples/queries.example.txt --city "Denver, CO"
```
+**Go past Google's ~120 results per search** by splitting the city into a 3 x 3 grid of map tiles
+(9 jobs, run one at a time, duplicates removed):
+```bash
+python3 scripts/scrape.py "plumbers in Phoenix AZ" --city "Phoenix, AZ" --grid 3
+```
+
---
## 4. Use it with Claude (the autopilot way) 🤖
diff --git a/scripts/scrape.py b/scripts/scrape.py
index a3219aa..f6c9d38 100755
--- a/scripts/scrape.py
+++ b/scripts/scrape.py
@@ -10,12 +10,18 @@ Auto-geocode (no coordinates needed):
Batch (one job, many keywords) from a file (one keyword per line):
python3 scripts/scrape.py --keywords-file examples/queries.example.txt --city "Denver, CO"
+Whole city past Google's ~120 cap (3x3 map tiles, run one at a time, deduped on place_id):
+ python3 scripts/scrape.py "plumbers in Phoenix AZ" --city "Phoenix, AZ" --grid 3
+
+Skip the contact-page email pass (on by default whenever emails are on):
+ python3 scripts/scrape.py "gyms in Miami FL" --city "Miami, FL" --no-deep-emails
+
Notes:
- Required by the API: keywords, lat/lon (strings), max_time (SECONDS). This script fills them in.
- Geocoding uses OpenStreetMap Nominatim (free, no key). Please be gentle: it allows ~1 request/sec
and requires a descriptive User-Agent (set below). Don't loop it aggressively.
"""
-import argparse, csv, io, json, os, re, sys, time, urllib.request, urllib.parse, urllib.error
+import argparse, csv, functools, io, json, math, os, re, sys, time, urllib.request, urllib.parse, urllib.error
from concurrent.futures import ThreadPoolExecutor
ENV_KEYS = ("SCRAPER_BASE_URL", "SCRAPER_PORT", "SCRAPER_API_KEY")
@@ -41,7 +47,7 @@ KEY = os.environ.get("SCRAPER_API_KEY", "")
# Money-useful LEAD fields only — what you actually use to contact/qualify a lead.
# Everything else (geo coordinates, IDs, hours, images, reviews blobs…) is dropped by default.
LEAD = ["title", "phone", "emails", "website", "category", "address", "review_rating", "review_count"]
-UA = "google-maps-scraper-kit/1.0 (https://github.com/Mahanaicoach/google-maps-scraper-kit)"
+UA = "google-maps-scraper-kit/1.1 (https://github.com/Mahanaicoach/google-maps-scraper-kit)"
def req(method, path, body=None):
@@ -54,8 +60,9 @@ def req(method, path, body=None):
return resp.status, resp.read()
-def geocode(place):
- """City/place name -> ('lat','lon') strings via Nominatim, or None."""
+def geocode(place, with_bbox=False):
+ """City/place name -> ('lat','lon') strings via Nominatim, or None.
+ with_bbox=True returns ('lat','lon', [south, north, west, east]) instead."""
q = urllib.parse.urlencode({"format": "json", "limit": 1, "q": place})
url = f"https://nominatim.openstreetmap.org/search?{q}"
r = urllib.request.Request(url, headers={"User-Agent": UA})
@@ -64,12 +71,87 @@ def geocode(place):
hits = json.loads(resp.read())
time.sleep(1) # respect Nominatim's ~1 req/sec policy
if hits:
- return str(hits[0]["lat"]), str(hits[0]["lon"])
+ ll = (str(hits[0]["lat"]), str(hits[0]["lon"]))
+ return ll + (hits[0].get("boundingbox"),) if with_bbox else ll
except Exception as e:
print(f" (geocoding failed: {e})", file=sys.stderr)
return None
+def grid_cells(bbox, n):
+ """Split a Nominatim bbox [south, north, west, east] into n x n cells -> [{lat, lon, radius, zoom}].
+ radius (m) = 1.15 x half the cell diagonal (equirectangular), at least 1000.
+ zoom = the Maps zoom whose ~1000px viewport is about as wide as the cell (11..17), so each
+ search is launched looking at its own tile instead of a city-wide or block-wide view."""
+ s, nn, w, e = (float(x) for x in bbox)
+ dlat, dlon = (nn - s) / n, (e - w) / n
+ m_per_deg = 6371000 * math.pi / 180
+ cells = []
+ for i in range(n):
+ for j in range(n):
+ lat, lon = s + (i + 0.5) * dlat, w + (j + 0.5) * dlon
+ cos = math.cos(math.radians(lat))
+ width = max(1.0, dlon * m_per_deg * cos)
+ diag = math.hypot(dlat * m_per_deg, width)
+ zoom = round(math.log2(156543.03 * cos * 1000 / width)) # 156543 m/px at zoom 0
+ cells.append({"lat": f"{lat:.6f}", "lon": f"{lon:.6f}",
+ "radius": max(1000, math.ceil(1.15 * diag / 2)),
+ "zoom": min(17, max(11, zoom))})
+ return cells
+
+
+def in_bbox(row, bbox, margin=0.10):
+ """True if the row's latitude/longitude sit inside bbox [south, north, west, east] grown by
+ `margin` of its size on each side. Rows with no usable coordinates are kept (True)."""
+ try:
+ lat, lon = float(row.get("latitude")), float(row.get("longitude"))
+ except (TypeError, ValueError):
+ return True
+ s, n, w, e = (float(x) for x in bbox)
+ my, mx = (n - s) * margin, (e - w) * margin
+ return s - my <= lat <= n + my and w - mx <= lon <= e + mx
+
+
+def dedupe(rows):
+ """Keep the first row per place_id, else cid, else link, else title+address."""
+ seen, out = set(), []
+ for r in rows:
+ key = next((f"{k}:{r[k]}" for k in ("place_id", "cid", "link") if r.get(k)),
+ f"ta:{r.get('title', '')}|{r.get('address', '')}")
+ if key not in seen:
+ seen.add(key); out.append(r)
+ return out
+
+
+def run_job(body):
+ """Create one engine job, poll it, download it -> (job_id, raw CSV rows). Raises RuntimeError."""
+ try:
+ _, raw = req("POST", "/api/v1/jobs", body)
+ except urllib.error.HTTPError as e:
+ raise RuntimeError(f"✗ Create failed: HTTP {e.code} — {e.read().decode()[:200]}")
+ job_id = json.loads(raw).get("id")
+ if not job_id:
+ raise RuntimeError("✗ No job id returned.")
+ print(f" job id: {job_id}")
+
+ print("▶ Polling…")
+ status = None
+ for i in range(120):
+ _, raw = req("GET", f"/api/v1/jobs/{job_id}")
+ status = json.loads(raw).get("Status")
+ print(f"\r status: {str(status):<10} (attempt {i + 1})", end="", flush=True)
+ if status == "ok":
+ print(); break
+ if status == "failed":
+ raise RuntimeError("\n✗ Job failed. If this keeps happening you may be rate-limited — wait or add proxies.")
+ time.sleep(8)
+ else:
+ raise RuntimeError("\n✗ Timed out.")
+
+ _, raw = req("GET", f"/api/v1/jobs/{job_id}/download")
+ return job_id, list(csv.DictReader(io.StringIO(raw.decode("utf-8", "replace"))))
+
+
def collect_keywords(a):
kws = []
if a.keyword:
@@ -112,6 +194,11 @@ def _fetch_html(url, timeout=10):
except Exception:
return ""
[email protected]_cache(maxsize=512) # ponytail: homepages only; evicted pages just refetch
+def _fetch_cached(url):
+ """Homepage fetch shared by the deep email pass and --socials (one fetch per site)."""
+ return _fetch_html(url, timeout=8)
+
def _find_socials(html):
out = {"instagram": "", "facebook": "", "linkedin": ""}
for plat, rx in SOCIAL_RE.items():
@@ -132,7 +219,7 @@ def enrich_socials(results, workers=8):
todo = [r for r in results if r.get("website")]
done = 0
def work(r):
- r.update(_find_socials(_fetch_html(r["website"])))
+ r.update(_find_socials(_fetch_cached(r["website"])))
with ThreadPoolExecutor(max_workers=workers) as ex:
for _ in ex.map(work, todo):
done += 1
@@ -140,6 +227,71 @@ def enrich_socials(results, workers=8):
print()
+# ── Deep email pass: contact/about pages for rows the engine left without an email ──
+EMAIL_RE = re.compile(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
+MAILTO_RE = re.compile(r"mailto:([^\"'>\s?]+)", re.I)
+HREF_RE = re.compile(r"<a\s[^>]*?href\s*=\s*[\"']([^\"']+)[\"'][^>]*>(.*?)</a>", re.I | re.S)
+_JUNK_EXT = (".png", ".jpg", ".jpeg", ".gif", ".svg", ".webp", ".css", ".js")
+_JUNK_DOMAIN = ("example", "sentry", "wixpress", "wix.com", "godaddy", "squarespace", "wordpress",
+ "domain.com", "email.com", "yourdomain", "schema.org", "w3.org", "sentry.io")
+_JUNK_LOCAL = {"name", "email", "user", "you", "someone"}
+
+def _host(url):
+ if not url.startswith(("http://", "https://")):
+ url = "http://" + url
+ h = (urllib.parse.urlparse(url).hostname or "").lower()
+ return h[4:] if h.startswith("www.") else h
+
+def extract_emails(html, site_host):
+ """Emails from mailto: links and plain text, junk dropped, site-domain first, max 3."""
+ site = _host(site_host or "")
+ found = []
+ for e in MAILTO_RE.findall(html or "") + EMAIL_RE.findall(html or ""):
+ e = urllib.parse.unquote(e).strip().lower()
+ local, _, dom = e.rpartition("@")
+ if (not EMAIL_RE.fullmatch(e) or e.endswith(_JUNK_EXT) or local in _JUNK_LOCAL
+ or any(j in dom for j in _JUNK_DOMAIN) or e in found):
+ continue
+ found.append(e)
+ on_site = lambda e: site and (e.endswith("@" + site) or e.endswith("." + site))
+ return sorted(found, key=lambda e: not on_site(e))[:3]
+
+def contact_links(html, base_url):
+ """Up to 3 same-site contact/about URLs: discovered links first, then fixed fallbacks."""
+ if not base_url.startswith(("http://", "https://")):
+ base_url = "http://" + base_url
+ host, out = _host(base_url), []
+ for href, text in HREF_RE.findall(html or ""):
+ if not re.search(r"contact|about", href + " " + text, re.I):
+ continue
+ url = urllib.parse.urldefrag(urllib.parse.urljoin(base_url, href.strip()))[0]
+ if url.startswith(("http://", "https://")) and _host(url) == host and url not in out:
+ out.append(url)
+ for path in ("/contact", "/contact-us", "/about", "/about-us"):
+ url = urllib.parse.urljoin(base_url, path)
+ if len(out) < 3 and url not in out:
+ out.append(url)
+ return out[:3]
+
+def enrich_emails(results, workers=8):
+ """Fill empty `emails` from the homepage, then up to 3 contact/about pages. Zero LLM tokens."""
+ todo = [r for r in results if r.get("website") and not r.get("emails")]
+ done = 0
+ def work(r):
+ site = r["website"]
+ home = _fetch_cached(site)
+ found = extract_emails(home, site)
+ if home and not found:
+ pages = "".join(_fetch_html(u, timeout=8) for u in contact_links(home, site))
+ found = extract_emails(pages, site)
+ r["emails"] = ", ".join(found)
+ with ThreadPoolExecutor(max_workers=workers) as ex:
+ for _ in ex.map(work, todo):
+ done += 1
+ print(f"\r emails: {done}/{len(todo)} sites checked", end="", flush=True)
+ print()
+
+
def main():
ap = argparse.ArgumentParser(description="Scrape Google Maps business listings.")
ap.add_argument("keyword", nargs="?", help='e.g. "coffee shops in Austin TX"')
@@ -154,6 +306,11 @@ def main():
ap.add_argument("--email", dest="email", action="store_true", default=True, help=argparse.SUPPRESS)
ap.add_argument("--no-email", dest="email", action="store_false",
help="skip email extraction for a faster run (emails are on by default)")
+ ap.add_argument("--no-deep-emails", dest="deep_emails", action="store_false",
+ help="skip the second email pass over contact/about pages (on whenever emails are on)")
+ ap.add_argument("--grid", type=int, metavar="N",
+ help="split the --city area into N x N map tiles, one job each, deduped "
+ "(breaks Google's ~120 per search cap; max 10)")
ap.add_argument("--max-time", type=int, default=600, help="job time limit in SECONDS")
ap.add_argument("--out", default=None, help="output file path (default: results-<id>.csv). "
"The extension picks the format: .csv (default) or .json")
@@ -168,9 +325,24 @@ def main():
if not keywords:
sys.exit("✗ No keywords. Pass a keyword, --keyword, or --keywords-file.")
+ if a.grid is not None:
+ if not a.city:
+ sys.exit("✗ --grid needs --city: the city's bounding box is what gets split into tiles.")
+ if not 1 <= a.grid <= 10:
+ sys.exit("✗ --grid must be between 1 and 10 (10 x 10 = 100 jobs is the ceiling).")
+
# Resolve coordinates: explicit lat/lon, else geocode --city, else geocode first keyword.
+ # --grid always geocodes --city, because it needs the city's bounding box.
lat, lon = a.lat, a.lon
- if not (lat and lon):
+ if a.grid:
+ print(f"▶ Geocoding \"{a.city}\"…")
+ geo = geocode(a.city, with_bbox=True)
+ if not geo or not geo[2]:
+ sys.exit("✗ Could not resolve a bounding box. Try a clearer --city.")
+ lat, lon, bbox = geo
+ cells = grid_cells(bbox, a.grid)
+ print(f" → bbox S{bbox[0]} N{bbox[1]} W{bbox[2]} E{bbox[3]}, {len(cells)} tiles")
+ elif not (lat and lon):
place = a.city or keywords[0]
print(f"▶ Geocoding \"{place}\"…")
coords = geocode(place)
@@ -184,8 +356,8 @@ def main():
"Use --no-email for a fast run.")
# --- WARN, don't block: flag high-volume jobs (proceed anyway) ---
- if a.depth >= 15 or len(keywords) >= 10:
- print("⚠️ Heads-up: this is a large job (high depth or many keywords).")
+ if a.depth >= 15 or len(keywords) >= 10 or (a.grid or 1) ** 2 >= 4:
+ print("⚠️ Heads-up: this is a large job (high depth, many keywords or a grid).")
print(" Running it back-to-back without proxies can get your IP temporarily rate-limited by")
print(" Google (clears in minutes–hours; returns empty/failed jobs meanwhile). For big or")
print(" repeated runs, add proxies. Proceeding…\n")
@@ -199,34 +371,36 @@ def main():
body = {"name": "scrape-py", "keywords": keywords, "lang": "en", "zoom": 15,
"lat": str(lat), "lon": str(lon), "fast_mode": False, "radius": 10000,
"depth": a.depth, "email": a.email, "max_time": a.max_time}
- print(f"▶ Creating job: {len(keywords)} keyword(s) @ {lat},{lon} depth={a.depth} email={a.email}")
- for k in keywords:
- print(f" • {k}")
- try:
- _, raw = req("POST", "/api/v1/jobs", body)
- except urllib.error.HTTPError as e:
- sys.exit(f"✗ Create failed: HTTP {e.code} — {e.read().decode()[:200]}")
- job_id = json.loads(raw).get("id")
- if not job_id:
- sys.exit("✗ No job id returned.")
- print(f" job id: {job_id}")
-
- print("▶ Polling…")
- status = None
- for i in range(120):
- _, raw = req("GET", f"/api/v1/jobs/{job_id}")
- status = json.loads(raw).get("Status")
- print(f"\r status: {str(status):<10} (attempt {i + 1})", end="", flush=True)
- if status == "ok":
- print(); break
- if status == "failed":
- sys.exit("\n✗ Job failed. If this keeps happening you may be rate-limited — wait or add proxies.")
- time.sleep(8)
+ if not a.grid:
+ print(f"▶ Creating job: {len(keywords)} keyword(s) @ {lat},{lon} depth={a.depth} email={a.email}")
+ for k in keywords:
+ print(f" • {k}")
+ try:
+ job_id, rows = run_job(body)
+ except RuntimeError as e:
+ sys.exit(str(e))
else:
- sys.exit("\n✗ Timed out.")
-
- _, raw = req("GET", f"/api/v1/jobs/{job_id}/download")
- rows = list(csv.DictReader(io.StringIO(raw.decode("utf-8", "replace"))))
+ print(f"▶ Grid {a.grid}x{a.grid}: {len(cells)} jobs, one at a time, "
+ f"{len(keywords)} keyword(s) depth={a.depth} email={a.email}")
+ for k in keywords:
+ print(f" • {k}")
+ job_id, rows = None, []
+ for i, cell in enumerate(cells, 1): # sequential on purpose: never parallel jobs against Google
+ print(f"▶ Cell {i}/{len(cells)} @ {cell['lat']},{cell['lon']} radius={cell['radius']}m")
+ try:
+ jid, cell_rows = run_job(dict(body, **cell))
+ except Exception as e:
+ print(f"\n ⚠ cell {i}/{len(cells)} skipped: {str(e).strip()}")
+ continue
+ job_id = job_id or jid
+ rows = dedupe(rows + cell_rows)
+ print(f" cell {i}/{len(cells)}: {len(cell_rows)} rows, {len(rows)} unique so far")
+ if job_id is None:
+ sys.exit("✗ Every grid cell failed. You may be rate-limited — wait or add proxies.")
+ # radius does not clip results in normal mode, so tiles leak outside the city
+ kept = [r for r in rows if in_bbox(r, bbox)]
+ print(f" dropped {len(rows) - len(kept)} rows outside the city box")
+ rows = kept
if a.full:
fields = list(rows[0].keys()) if rows else LEAD
elif a.fields:
@@ -236,6 +410,15 @@ def main():
results = [{k: r.get(k, "") for k in fields} for r in rows]
print(f"✓ Done — {len(results)} businesses (fields: {', '.join(fields)}).")
+ if a.email and a.deep_emails and "emails" in fields:
+ print("▶ Deep email pass: checking contact/about pages for rows with no email…")
+ engine = sum(1 for r in results if r.get("emails"))
+ enrich_emails(results)
+ total = sum(1 for r in results if r.get("emails"))
+ n = len(results)
+ print(f" emails: {engine}/{n} from the engine, +{total - engine} from contact pages "
+ f"= {total}/{n} ({round(100 * total / n) if n else 0}%)")
+
if a.socials:
print("▶ Finding socials (Instagram / Facebook / LinkedIn) on each website…")
print(" ℹ Cost: this runs in CODE = 0 LLM tokens. It only adds ~40–50 tokens per business")
diff --git a/scripts/test_scrape.py b/scripts/test_scrape.py
new file mode 100644
index 0000000..0588f1d
--- /dev/null
+++ b/scripts/test_scrape.py
@@ -0,0 +1,183 @@
+#!/usr/bin/env python3
+"""Offline tests for scripts/scrape.py (standard library only, no network).
+Run: python3 scripts/test_scrape.py"""
+import csv, importlib.util, io, json, os, sys, tempfile, threading, unittest
+from contextlib import redirect_stdout
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from unittest import mock
+
+_spec = importlib.util.spec_from_file_location(
+ "scrape", os.path.join(os.path.dirname(os.path.abspath(__file__)), "scrape.py"))
+scrape = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(scrape)
+
+BBOX = ["30.0", "30.5", "-98.0", "-97.5"] # Nominatim order: south, north, west, east
+
+
+class GridCells(unittest.TestCase):
+ def test_count_centers_radius(self):
+ cells = scrape.grid_cells(BBOX, 3)
+ self.assertEqual(len(cells), 9)
+ for c in cells:
+ self.assertTrue(30.0 < float(c["lat"]) < 30.5)
+ self.assertTrue(-98.0 < float(c["lon"]) < -97.5)
+ self.assertGreaterEqual(c["radius"], 1000)
+ self.assertEqual(len(c["lat"].split(".")[1]), 6)
+ self.assertEqual(c["zoom"], 13) # ~16 km tiles: city-district view, not a 3 km block
+
+ def test_zoom_follows_tile_size(self):
+ self.assertEqual(scrape.grid_cells(BBOX, 1)[0]["zoom"], 11) # ~48 km: clamped
+ self.assertEqual(scrape.grid_cells(["0", "0.001", "0", "0.001"], 1)[0]["zoom"], 17) # 110 m: clamped
+
+ def test_one_cell_is_bbox_center(self):
+ (c,) = scrape.grid_cells(BBOX, 1)
+ self.assertEqual((c["lat"], c["lon"]), ("30.250000", "-97.750000"))
+ self.assertGreater(c["radius"], 30000) # ~0.5 x 0.5 degree box
+
+ def test_tiny_box_floors_at_1000(self):
+ self.assertEqual(scrape.grid_cells(["0", "0.001", "0", "0.001"], 2)[0]["radius"], 1000)
+
+
+class Dedupe(unittest.TestCase):
+ def test_key_order(self):
+ rows = [
+ {"place_id": "P1", "cid": "C1", "title": "a"},
+ {"place_id": "P1", "cid": "C9", "title": "dup place_id"},
+ {"cid": "C2", "link": "L1"},
+ {"cid": "C2", "link": "L9", "title": "dup cid"},
+ {"link": "L3"},
+ {"link": "L3", "title": "dup link"},
+ {"title": "Joe's", "address": "1 Main"},
+ {"title": "Joe's", "address": "1 Main", "phone": "dup title+address"},
+ {"title": "Joe's", "address": "2 Main"},
+ ]
+ out = scrape.dedupe(rows)
+ self.assertEqual(len(out), 5)
+ self.assertEqual(out[0]["title"], "a") # first occurrence wins
+ self.assertNotIn("dup", json.dumps(out))
+
+
+class InBbox(unittest.TestCase):
+ def test_margin_and_missing(self):
+ self.assertTrue(scrape.in_bbox({"latitude": "30.2", "longitude": "-97.7"}, BBOX))
+ self.assertTrue(scrape.in_bbox({"latitude": "30.54", "longitude": "-97.7"}, BBOX)) # inside 10%
+ self.assertFalse(scrape.in_bbox({"latitude": "30.56", "longitude": "-97.7"}, BBOX)) # past 10%
+ self.assertFalse(scrape.in_bbox({"latitude": "30.2", "longitude": "-98.1"}, BBOX))
+ self.assertTrue(scrape.in_bbox({"latitude": "", "longitude": ""}, BBOX))
+ self.assertTrue(scrape.in_bbox({"latitude": "n/a"}, BBOX))
+ self.assertTrue(scrape.in_bbox({}, BBOX))
+
+
+class ExtractEmails(unittest.TestCase):
+ def test_mailto_regex_junk_preference_cap(self):
+ html = """
+ <a href="mailto:[email protected]?subject=Hi">Mail</a>
+ <a href="mailto:[email protected]">info</a>
+ """
+ out = scrape.extract_emails(html, "www.acmeplumbing.com")
+ self.assertEqual(out, ["[email protected]", "[email protected]", "[email protected]"])
+
+ def test_empty(self):
+ self.assertEqual(scrape.extract_emails("", "x.com"), [])
+ self.assertEqual(scrape.extract_emails("no emails here", "x.com"), [])
+
+
+class ContactLinks(unittest.TestCase):
+ def test_discovers_same_site_only(self):
+ html = """<a href="/contact-us#form">Get in touch</a>
+ <a href="https://www.acme.com/team">About us</a>
+ <a href="https://facebook.com/acme/about">FB</a>
+ <a href="mailto:[email protected]">contact</a>
+ <a href="/services">Services</a>"""
+ out = scrape.contact_links(html, "https://acme.com")
+ self.assertEqual(out[:2], ["https://acme.com/contact-us", "https://www.acme.com/team"])
+ self.assertEqual(len(out), 3)
+ self.assertEqual(out[2], "https://acme.com/contact") # first fixed fallback
+ self.assertFalse(any("facebook" in u or "mailto" in u for u in out))
+
+ def test_fallbacks_and_max(self):
+ self.assertEqual(scrape.contact_links("", "acme.com"),
+ ["http://acme.com/contact", "http://acme.com/contact-us", "http://acme.com/about"])
+ many = "".join(f'<a href="/contact{i}">x</a>' for i in range(6))
+ self.assertEqual(len(scrape.contact_links(many, "http://acme.com")), 3)
+
+
+# ── End to end against a mock engine ──
+FIELDS = ["input_id", "link", "title", "category", "address", "website", "phone", "review_count",
+ "review_rating", "latitude", "longitude", "cid", "place_id", "emails"]
+
+def _row(pid, lat="30.2"):
+ return {"title": f"Biz {pid}", "place_id": pid, "cid": "c" + pid, "latitude": lat,
+ "longitude": "-97.7", "phone": "+1 555", "emails": f"{pid.lower()}@biz.net"}
+
+CELL_ROWS = {"job-1": [_row("A"), _row("B"), _row("C")],
+ "job-2": [_row("C"), _row("D")],
+ "job-3": None, # this cell fails
+ "job-4": [_row("A"), _row("E"), _row("F", lat="40.0")]} # F is outside the city box
+
+
+class MockEngine(BaseHTTPRequestHandler):
+ created, polls = 0, {}
+
+ def log_message(self, *a):
+ pass
+
+ def _send(self, code, body, ctype="application/json"):
+ data = body.encode()
+ self.send_response(code)
+ self.send_header("Content-Type", ctype)
+ self.send_header("Content-Length", str(len(data)))
+ self.end_headers()
+ self.wfile.write(data)
+
+ def do_POST(self):
+ self.rfile.read(int(self.headers.get("Content-Length", 0)))
+ MockEngine.created += 1
+ self._send(201, json.dumps({"id": f"job-{MockEngine.created}"}))
+
+ def do_GET(self):
+ parts = self.path.strip("/").split("/") # api v1 jobs [id [download]]
+ if len(parts) == 3:
+ return self._send(200, "[]")
+ jid = parts[3]
+ if len(parts) == 5:
+ buf = io.StringIO()
+ w = csv.DictWriter(buf, fieldnames=FIELDS, restval="")
+ w.writeheader(); w.writerows(CELL_ROWS[jid])
+ return self._send(200, buf.getvalue(), "text/csv")
+ n = MockEngine.polls[jid] = MockEngine.polls.get(jid, 0) + 1
+ status = "failed" if CELL_ROWS[jid] is None else ("ok" if n >= 2 else "working")
+ self._send(200, json.dumps({"Status": status}))
+
+
+class EndToEnd(unittest.TestCase):
+ def test_grid_run(self):
+ srv = ThreadingHTTPServer(("127.0.0.1", 0), MockEngine)
+ threading.Thread(target=srv.serve_forever, daemon=True).start()
+ out = os.path.join(tempfile.mkdtemp(), "leads.csv")
+ argv = ["scrape.py", "plumbers in Austin TX", "--city", "Austin, TX",
+ "--grid", "2", "--no-deep-emails", "--out", out]
+ log = io.StringIO()
+ try:
+ with mock.patch.object(scrape, "BASE", f"http://127.0.0.1:{srv.server_port}"), \
+ mock.patch.object(scrape, "geocode", return_value=("30.25", "-97.75", BBOX)), \
+ mock.patch.object(scrape.time, "sleep"), \
+ mock.patch.object(sys, "argv", argv), redirect_stdout(log):
+ scrape.main()
+ finally:
+ srv.shutdown(); srv.server_close()
+ text = log.getvalue()
+ self.assertIn("cell 4/4: 3 rows, 6 unique so far", text)
+ self.assertIn("cell 3/4 skipped", text)
+ self.assertIn("dropped 1 rows outside the city box", text)
+ with open(out, newline="") as f:
+ rows = list(csv.reader(f))
+ self.assertEqual(rows[0], scrape.LEAD)
+ self.assertEqual([r[0] for r in rows[1:]], ["Biz A", "Biz B", "Biz C", "Biz D", "Biz E"])
+
+
+if __name__ == "__main__":
+ unittest.main(verbosity=2)