Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
fix: first-run polish for newcomers: wait for the engine, Excel-friendly CSV, clearer README and SETUP, wider --exclude matching
Mahanaicoach96555c211fc0
diff --git a/README.md b/README.mdindex 1759c52..ce3d5b3 100644--- a/README.md+++ b/README.md@@ -61,16 +61,22 @@ Maps API, delivered as clean, structured JSON and Markdown.## Quick start-Requires [Docker Desktop](https://www.docker.com/products/docker-desktop) and Python 3. The scripts-use the standard library only, so there is nothing to `pip install`.+Requires [Docker Desktop](https://www.docker.com/products/docker-desktop) (installed **and open**) and+Python 3. The scripts use the standard library only, so there is nothing to `pip install`.++Open a terminal (macOS: Terminal; Windows: PowerShell) and run these four lines. The first+`docker compose up -d` downloads the engine once, about 600 MB, and the script waits for it to start:```bashgit clone https://github.com/Mahanaicoach/google-maps-scraper-kit.gitcd google-maps-scraper-kitdocker compose up -d-./scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5+bash scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5```+On Windows, skip the `bash` line and use the Python CLI below with `py` in place of `python3`.+Prefer clicking? http://localhost:8080 has a form; it returns the engine's raw 36-column CSV.+No coordinates handy? Let the Python CLI geocode the city:```bashdiff --git a/SETUP.md b/SETUP.mdindex 84dfade..d47681c 100644--- a/SETUP.md+++ b/SETUP.md@@ -25,8 +25,10 @@ docker ps # should NOT error (means the daemon is up)> **Platform notes:**> - **Apple Silicon (M1/M2/M3):** from v1.18.0 the image publishes an arm64 build, so it runs> natively and no `platform` line is needed. Only older tags are amd64 and run under emulation.-> - **Windows:** use `scripts/scrape.py` (run with `py` or `python3`). The bash script `scrape.sh` needs-> **WSL2** or **Git Bash**.+> - **Windows:** when you install Python, tick **Add python.exe to PATH**. Then use `scripts/scrape.py`+> (run with `py`). The bash script `scrape.sh` needs **WSL2** or **Git Bash**.+> - **macOS:** the first `git` or `python3` command may ask to install the Command Line Tools. Click+> Install, wait, then run the command again.---@@ -82,7 +84,7 @@ If `curl` prints `null` or a JSON list, the engine is live.Use the bash script. Its arguments are `"<keyword>" <lat> <lon> [depth]`:```bash-./scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5+bash scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5```Or use the Python CLI, which needs no dependencies:@@ -187,6 +189,9 @@ curl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>| Symptom | Fix ||---|---|| `curl: connection refused` on :8080 | Container not up. `docker compose up -d`, then `docker ps`. |+| `Cannot connect to the Docker daemon` / `docker: command not found` | Docker Desktop is not running (or not installed). Open Docker Desktop, wait until its whale icon stops animating, then run `docker compose up -d` again. |+| `502 Bad Gateway` right after `docker compose up -d` | The engine is still starting, about 10 seconds. The scripts wait for it; in the browser, reload the page. |+| `Permission denied` running `./scripts/scrape.sh` | The file lost its execute bit (a ZIP download does that). Run `bash scripts/scrape.sh …` instead. || `port is already allocated` / something else already uses 8080 | Set `SCRAPER_PORT=8081` in `.env`, then `docker compose up -d`. Compose publishes the scraper on that host port and `scripts/scrape.sh` and `scripts/scrape.py` read the same file, so nothing else needs editing. The port inside the container stays 8080 (do not change the right-hand side of the mapping). If your `.env` still has a `SCRAPER_BASE_URL` line, delete it or update its port: it overrides `SCRAPER_PORT`. The slash commands and the skill show `localhost:8080` in their examples, so tell Claude the port you picked. || `422 missing max time` | Job body needs `max_time` (in **seconds**, e.g. `300`). || `422 missing geo coordinates` | Job body needs `lat` and `lon` as **strings**, e.g. `"30.2672"`. |diff --git a/scripts/scrape.py b/scripts/scrape.pyindex b462132..4d184e8 100755--- a/scripts/scrape.py+++ b/scripts/scrape.py@@ -152,8 +152,10 @@ def _keys(r):"""Every identity a row carries: place_id, cid, link and title+address. A lean 8-column CSVfrom an earlier run has only the last one, so matching on any key lets --exclude use it."""ks = [f"{k}:{r[k]}" for k in ("place_id", "cid", "link") if r.get(k)]- if r.get("title"):- ks.append(f"ta:{r['title']}|{r.get('address', '')}")+ t = r.get("title")+ if t: # name plus address, and name plus phone or website: a --fields file may lack the address+ ks.append(f"ta:{t}|{r.get('address', '')}")+ ks += [f"t{c[0]}:{t}|{r[c]}" for c in ("phone", "website") if r.get(c)]return ks@@ -169,7 +171,7 @@ def dedupe(rows, exclude=()):def load_rows(path):"""Rows from an earlier results file, .csv or .json."""- with open(path, encoding="utf-8", newline="") as f:+ with open(path, encoding="utf-8-sig", newline="") as f: # -sig: also reads the kit's own CSVsreturn json.load(f) if path.lower().endswith(".json") else list(csv.DictReader(f))@@ -799,11 +801,21 @@ def main():print(" Google (clears in minutes–hours; returns empty/failed jobs meanwhile). For big or")print(" repeated runs, add proxies. Proceeding…\n")- # Health check- try:- req("GET", "/api/v1/jobs")- except Exception as e:- sys.exit(f"✗ Scraper not reachable at {BASE} — run 'docker compose up -d' first.\n ({e})")+ # Health check. Right after `docker compose up -d` the proxy answers 502 for a few seconds while+ # the engine starts, and the README's next line runs at once, so wait up to a minute for it.+ for i in range(30):+ try:+ req("GET", "/api/v1/jobs")+ break+ except Exception as e:+ err = e+ print("\r waiting for the engine to start" + "." * (i + 1), end="", flush=True)+ time.sleep(2)+ else:+ sys.exit(f"\n✗ Scraper not reachable at {BASE}. Is Docker Desktop open? Start the engine with "+ f"'docker compose up -d' in the kit folder, wait 10 seconds, then try again.\n ({err})")+ if i:+ print()body = {"name": "scrape-py", "keywords": keywords, "lang": a.lang, "zoom": 15,"fast_mode": False, "radius": 10000,@@ -882,11 +894,12 @@ def main():with open(out, "w", encoding="utf-8") as f:json.dump(results, f, indent=2, ensure_ascii=False)else:- with open(out, "w", newline="", encoding="utf-8") as f:+ # utf-8-sig: the BOM makes Excel on Windows read accented names as UTF-8 instead of cp1252.+ with open(out, "w", newline="", encoding="utf-8-sig") as f:w = csv.DictWriter(f, fieldnames=fields)w.writeheader()w.writerows(results)- print(f" saved → {out}")+ print(f" saved → {os.path.abspath(out)} (opens in Excel or Google Sheets)")for r in results[:5]:tail = f" | IG:{r.get('instagram','') or '—'}" if a.socials else f" | {r.get('website','')}"print(f" • {r.get('title','')} | {r.get('phone','')} | {r.get('emails','')}{tail}")diff --git a/scripts/test_scrape.py b/scripts/test_scrape.pyindex 164f6d1..1552da7 100644--- a/scripts/test_scrape.py+++ b/scripts/test_scrape.py@@ -449,7 +449,7 @@ class EndToEnd(unittest.TestCase):self.assertIn("cell 3/4 skipped", text)self.assertIn("Incomplete: 1 of 4 tiles failed", text)self.assertIn("dropped 1 rows outside the city box", text)- with open(out, newline="") as f:+ with open(out, newline="", encoding="utf-8-sig") as f: # the kit writes a BOM for Excelrows = list(csv.reader(f))self.assertEqual(rows[0], scrape.LEAD)self.assertEqual([r[0] for r in rows[1:]], ["Biz A", "Biz B", "Biz C", "Biz D", "Biz E"])@@ -478,7 +478,7 @@ class EndToEnd(unittest.TestCase):srv.shutdown(); srv.server_close()self.assertEqual(geo.call_count, 2)self.assertIn("kept 3 of 4 rows", log.getvalue()) # A, B, C + C, D -> 4 unique, minus excluded A- with open(out, newline="") as f:+ with open(out, newline="", encoding="utf-8-sig") as f: # the kit writes a BOM for Excelrows = list(csv.DictReader(f))self.assertEqual([r["title"] for r in rows], ["Biz B", "Biz C"]) # --limit 2@@ -499,13 +499,18 @@ class Filters(unittest.TestCase):new = [{"place_id": "P1", "title": "Joe's", "address": "1 Main"}, {"place_id": "P2", "title": "Ann's"}]self.assertEqual(scrape.dedupe(new, exclude=[{"title": "Joe's", "address": "1 Main"}]), new[1:])self.assertEqual(scrape.dedupe(new, exclude=[{"place_id": "P2"}]), new[:1])+ # A --fields file without address still matches on name plus phone or website.+ new2 = [{"place_id": "P3", "title": "Ann's", "address": "2 Main", "phone": "+1 555", "website": "http://ann.com"}]+ self.assertEqual(scrape.dedupe(new2, exclude=[{"title": "Ann's", "phone": "+1 555"}]), [])+ self.assertEqual(scrape.dedupe(new2, exclude=[{"title": "Ann's", "website": "http://ann.com"}]), [])+ self.assertEqual(scrape.dedupe(new2, exclude=[{"title": "Ann's"}]), new2) # name alone is not enoughdef test_load_rows_csv_and_json(self):d = tempfile.mkdtemp()j = os.path.join(d, "a.json"); open(j, "w").write('[{"title": "A"}]')- c = os.path.join(d, "a.csv"); open(c, "w").write("title,phone\nB,1\n")+ c = os.path.join(d, "a.csv"); open(c, "w", encoding="utf-8-sig").write("title,phone\nB,1\n")self.assertEqual(scrape.load_rows(j), [{"title": "A"}])- self.assertEqual(scrape.load_rows(c), [{"title": "B", "phone": "1"}])+ self.assertEqual(scrape.load_rows(c), [{"title": "B", "phone": "1"}]) # BOM stripped, key is cleanclass VerifyEmails(unittest.TestCase):