Skip to content

Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.

fix: first-run polish for newcomers: wait for the engine, Excel-friendly CSV, clearer README and SETUP, wider --exclude matching

Mahanaicoach96555c211fc0
diff --git a/README.md b/README.md
index 1759c52..ce3d5b3 100644
--- a/README.md
+++ b/README.md
@@ -61,16 +61,22 @@ Maps API, delivered as clean, structured JSON and Markdown.
## Quick start
-Requires [Docker Desktop](https://www.docker.com/products/docker-desktop) and Python 3. The scripts
-use the standard library only, so there is nothing to `pip install`.
+Requires [Docker Desktop](https://www.docker.com/products/docker-desktop) (installed **and open**) and
+Python 3. The scripts use the standard library only, so there is nothing to `pip install`.
+
+Open a terminal (macOS: Terminal; Windows: PowerShell) and run these four lines. The first
+`docker compose up -d` downloads the engine once, about 600 MB, and the script waits for it to start:
```bash
git clone https://github.com/Mahanaicoach/google-maps-scraper-kit.git
cd google-maps-scraper-kit
docker compose up -d
-./scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5
+bash scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5
```
+On Windows, skip the `bash` line and use the Python CLI below with `py` in place of `python3`.
+Prefer clicking? http://localhost:8080 has a form; it returns the engine's raw 36-column CSV.
+
No coordinates handy? Let the Python CLI geocode the city:
```bash
diff --git a/SETUP.md b/SETUP.md
index 84dfade..d47681c 100644
--- a/SETUP.md
+++ b/SETUP.md
@@ -25,8 +25,10 @@ docker ps # should NOT error (means the daemon is up)
> **Platform notes:**
> - **Apple Silicon (M1/M2/M3):** from v1.18.0 the image publishes an arm64 build, so it runs
> natively and no `platform` line is needed. Only older tags are amd64 and run under emulation.
-> - **Windows:** use `scripts/scrape.py` (run with `py` or `python3`). The bash script `scrape.sh` needs
-> **WSL2** or **Git Bash**.
+> - **Windows:** when you install Python, tick **Add python.exe to PATH**. Then use `scripts/scrape.py`
+> (run with `py`). The bash script `scrape.sh` needs **WSL2** or **Git Bash**.
+> - **macOS:** the first `git` or `python3` command may ask to install the Command Line Tools. Click
+> Install, wait, then run the command again.
---
@@ -82,7 +84,7 @@ If `curl` prints `null` or a JSON list, the engine is live.
Use the bash script. Its arguments are `"<keyword>" <lat> <lon> [depth]`:
```bash
-./scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5
+bash scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5
```
Or use the Python CLI, which needs no dependencies:
@@ -187,6 +189,9 @@ curl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>
| Symptom | Fix |
|---|---|
| `curl: connection refused` on :8080 | Container not up. `docker compose up -d`, then `docker ps`. |
+| `Cannot connect to the Docker daemon` / `docker: command not found` | Docker Desktop is not running (or not installed). Open Docker Desktop, wait until its whale icon stops animating, then run `docker compose up -d` again. |
+| `502 Bad Gateway` right after `docker compose up -d` | The engine is still starting, about 10 seconds. The scripts wait for it; in the browser, reload the page. |
+| `Permission denied` running `./scripts/scrape.sh` | The file lost its execute bit (a ZIP download does that). Run `bash scripts/scrape.sh …` instead. |
| `port is already allocated` / something else already uses 8080 | Set `SCRAPER_PORT=8081` in `.env`, then `docker compose up -d`. Compose publishes the scraper on that host port and `scripts/scrape.sh` and `scripts/scrape.py` read the same file, so nothing else needs editing. The port inside the container stays 8080 (do not change the right-hand side of the mapping). If your `.env` still has a `SCRAPER_BASE_URL` line, delete it or update its port: it overrides `SCRAPER_PORT`. The slash commands and the skill show `localhost:8080` in their examples, so tell Claude the port you picked. |
| `422 missing max time` | Job body needs `max_time` (in **seconds**, e.g. `300`). |
| `422 missing geo coordinates` | Job body needs `lat` and `lon` as **strings**, e.g. `"30.2672"`. |
diff --git a/scripts/scrape.py b/scripts/scrape.py
index b462132..4d184e8 100755
--- a/scripts/scrape.py
+++ b/scripts/scrape.py
@@ -152,8 +152,10 @@ def _keys(r):
"""Every identity a row carries: place_id, cid, link and title+address. A lean 8-column CSV
from an earlier run has only the last one, so matching on any key lets --exclude use it."""
ks = [f"{k}:{r[k]}" for k in ("place_id", "cid", "link") if r.get(k)]
- if r.get("title"):
- ks.append(f"ta:{r['title']}|{r.get('address', '')}")
+ t = r.get("title")
+ if t: # name plus address, and name plus phone or website: a --fields file may lack the address
+ ks.append(f"ta:{t}|{r.get('address', '')}")
+ ks += [f"t{c[0]}:{t}|{r[c]}" for c in ("phone", "website") if r.get(c)]
return ks
@@ -169,7 +171,7 @@ def dedupe(rows, exclude=()):
def load_rows(path):
"""Rows from an earlier results file, .csv or .json."""
- with open(path, encoding="utf-8", newline="") as f:
+ with open(path, encoding="utf-8-sig", newline="") as f: # -sig: also reads the kit's own CSVs
return json.load(f) if path.lower().endswith(".json") else list(csv.DictReader(f))
@@ -799,11 +801,21 @@ def main():
print(" Google (clears in minutes–hours; returns empty/failed jobs meanwhile). For big or")
print(" repeated runs, add proxies. Proceeding…\n")
- # Health check
- try:
- req("GET", "/api/v1/jobs")
- except Exception as e:
- sys.exit(f"✗ Scraper not reachable at {BASE} — run 'docker compose up -d' first.\n ({e})")
+ # Health check. Right after `docker compose up -d` the proxy answers 502 for a few seconds while
+ # the engine starts, and the README's next line runs at once, so wait up to a minute for it.
+ for i in range(30):
+ try:
+ req("GET", "/api/v1/jobs")
+ break
+ except Exception as e:
+ err = e
+ print("\r waiting for the engine to start" + "." * (i + 1), end="", flush=True)
+ time.sleep(2)
+ else:
+ sys.exit(f"\n✗ Scraper not reachable at {BASE}. Is Docker Desktop open? Start the engine with "
+ f"'docker compose up -d' in the kit folder, wait 10 seconds, then try again.\n ({err})")
+ if i:
+ print()
body = {"name": "scrape-py", "keywords": keywords, "lang": a.lang, "zoom": 15,
"fast_mode": False, "radius": 10000,
@@ -882,11 +894,12 @@ def main():
with open(out, "w", encoding="utf-8") as f:
json.dump(results, f, indent=2, ensure_ascii=False)
else:
- with open(out, "w", newline="", encoding="utf-8") as f:
+ # utf-8-sig: the BOM makes Excel on Windows read accented names as UTF-8 instead of cp1252.
+ with open(out, "w", newline="", encoding="utf-8-sig") as f:
w = csv.DictWriter(f, fieldnames=fields)
w.writeheader()
w.writerows(results)
- print(f" saved → {out}")
+ print(f" saved → {os.path.abspath(out)} (opens in Excel or Google Sheets)")
for r in results[:5]:
tail = f" | IG:{r.get('instagram','') or '—'}" if a.socials else f" | {r.get('website','')}"
print(f" • {r.get('title','')} | {r.get('phone','')} | {r.get('emails','')}{tail}")
diff --git a/scripts/test_scrape.py b/scripts/test_scrape.py
index 164f6d1..1552da7 100644
--- a/scripts/test_scrape.py
+++ b/scripts/test_scrape.py
@@ -449,7 +449,7 @@ class EndToEnd(unittest.TestCase):
self.assertIn("cell 3/4 skipped", text)
self.assertIn("Incomplete: 1 of 4 tiles failed", text)
self.assertIn("dropped 1 rows outside the city box", text)
- with open(out, newline="") as f:
+ with open(out, newline="", encoding="utf-8-sig") as f: # the kit writes a BOM for Excel
rows = list(csv.reader(f))
self.assertEqual(rows[0], scrape.LEAD)
self.assertEqual([r[0] for r in rows[1:]], ["Biz A", "Biz B", "Biz C", "Biz D", "Biz E"])
@@ -478,7 +478,7 @@ class EndToEnd(unittest.TestCase):
srv.shutdown(); srv.server_close()
self.assertEqual(geo.call_count, 2)
self.assertIn("kept 3 of 4 rows", log.getvalue()) # A, B, C + C, D -> 4 unique, minus excluded A
- with open(out, newline="") as f:
+ with open(out, newline="", encoding="utf-8-sig") as f: # the kit writes a BOM for Excel
rows = list(csv.DictReader(f))
self.assertEqual([r["title"] for r in rows], ["Biz B", "Biz C"]) # --limit 2
@@ -499,13 +499,18 @@ class Filters(unittest.TestCase):
new = [{"place_id": "P1", "title": "Joe's", "address": "1 Main"}, {"place_id": "P2", "title": "Ann's"}]
self.assertEqual(scrape.dedupe(new, exclude=[{"title": "Joe's", "address": "1 Main"}]), new[1:])
self.assertEqual(scrape.dedupe(new, exclude=[{"place_id": "P2"}]), new[:1])
+ # A --fields file without address still matches on name plus phone or website.
+ new2 = [{"place_id": "P3", "title": "Ann's", "address": "2 Main", "phone": "+1 555", "website": "http://ann.com"}]
+ self.assertEqual(scrape.dedupe(new2, exclude=[{"title": "Ann's", "phone": "+1 555"}]), [])
+ self.assertEqual(scrape.dedupe(new2, exclude=[{"title": "Ann's", "website": "http://ann.com"}]), [])
+ self.assertEqual(scrape.dedupe(new2, exclude=[{"title": "Ann's"}]), new2) # name alone is not enough
def test_load_rows_csv_and_json(self):
d = tempfile.mkdtemp()
j = os.path.join(d, "a.json"); open(j, "w").write('[{"title": "A"}]')
- c = os.path.join(d, "a.csv"); open(c, "w").write("title,phone\nB,1\n")
+ c = os.path.join(d, "a.csv"); open(c, "w", encoding="utf-8-sig").write("title,phone\nB,1\n")
self.assertEqual(scrape.load_rows(j), [{"title": "A"}])
- self.assertEqual(scrape.load_rows(c), [{"title": "B", "phone": "1"}])
+ self.assertEqual(scrape.load_rows(c), [{"title": "B", "phone": "1"}]) # BOM stripped, key is clean
class VerifyEmails(unittest.TestCase):