Skip to content

Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.

fix(kit): P2 audit loop 1: proxy errors, Google blocks, upgrade notes, no download wording in the skill

Mahan Jafarif5c85bf15291
diff --git a/.claude/commands/scrape-setup.md b/.claude/commands/scrape-setup.md
index 40b27a3..dc73298 100644
--- a/.claude/commands/scrape-setup.md
+++ b/.claude/commands/scrape-setup.md
@@ -3,8 +3,8 @@ description: Start the local Google Maps scraper engine and run a health check.
---
Set up the kit's engine on this computer:
-1. Pick how the engine runs. By default, without Docker: from the kit root, `python3 scripts/engine.py start` (Windows: `py scripts/engine.py start`). It needs Python 3.9 or newer and an installed Chrome, Chromium, Edge or Brave. On a machine with none, it downloads Chrome for Testing once (about 100 MB, pinned by SHA-256): tell the user before you run it. If the user runs Docker instead, confirm it is up with `docker ps` (if it errors, point them to **Docker Desktop**, link in `SETUP.md` §0, and stop here), then run `docker compose up -d`; the first run builds the image, about a minute.
-2. `python3 scripts/engine.py status` prints the browser the engine uses (skip it for Docker).
+1. Pick how the engine runs. By default, without Docker: from the kit root, run `python3 scripts/engine.py status` (Windows: `py scripts/engine.py status`). It needs Python 3.9 or newer. If it says the engine is running, go to step 3. It names the browser the engine will use; if it finds none, `start` would download Chrome for Testing once (about 100 MB, pinned by SHA-256), so ask the user first, or suggest installing Google Chrome or using Docker. If the user runs Docker instead, confirm it is up with `docker ps` (if it errors, point them to **Docker Desktop**, link in `SETUP.md` §0, and stop here), then run `docker compose up -d`; the first run builds the image, about a minute.
+2. Without Docker, start the engine: `python3 scripts/engine.py start` (Windows: `py scripts/engine.py start`).
3. Verify it is live: `curl http://localhost:8080/api/v1/jobs` should return `null` (no jobs yet) or a JSON array of jobs.
4. Report success. Tell the user they can now run `/scrape <business> in <city>`, or ask in plain language (e.g. "scrape gyms in Miami").
5. **Show this warning once setup succeeds:**
diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.md
index 3e40321..c6b8a14 100644
--- a/.claude/skills/google-maps-scraper/SKILL.md
+++ b/.claude/skills/google-maps-scraper/SKILL.md
@@ -20,9 +20,9 @@ curl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOW
```
If `DOWN` and the working folder has `scripts/engine.py` (the full kit), start the engine there:
`python3 scripts/engine.py start` (`py` on Windows), or `docker compose up -d` if the user runs it in
-Docker. Then retry. `engine.py start` uses the installed Chrome, Chromium, Edge or Brave; on a machine
-with none it downloads Chrome for Testing once (about 100 MB, pinned by SHA-256), so tell the user
-before you run it there. Setup help:
+Docker. Then retry. The engine drives the installed Chrome, Chromium, Edge or Brave:
+`python3 scripts/engine.py status` names the one it will use. If it finds none, do not start it: ask
+the user to install Google Chrome, or to use Docker. Setup help:
[SETUP.md](https://github.com/Mahanaicoach/google-maps-scraper-kit/blob/master/SETUP.md).
**Skill installed on its own?** `npx skills add` copies this file only. The engine ships with the
diff --git a/CLAUDE.md b/CLAUDE.md
index e595e73..c95cd7c 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -45,9 +45,10 @@ clearly abusive use (surveilling individuals, spam). The skill has the full guid
## Setup
If the engine is not running: `python3 scripts/engine.py start` (or `docker compose up -d` when the user
-runs Docker), then verify with `curl http://localhost:8080/api/v1/jobs`. On a machine with no Chrome,
-Chromium, Edge or Brave, `engine.py start` downloads Chrome for Testing once (about 100 MB, pinned by
-SHA-256): tell the user before you run it. `SETUP.md` has the full setup.
+runs Docker), then verify with `curl http://localhost:8080/api/v1/jobs`. `python3 scripts/engine.py status`
+names the browser the engine will use. If it finds none, `start` would download Chrome for Testing
+(about 100 MB, pinned by SHA-256): ask the user first, or suggest installing Google Chrome or using
+Docker. `SETUP.md` has the full setup.
## Changing the kit
`python3 scripts/test_scrape.py`, `python3 engine/test_engine.py` and `python3 engine/test_cdp.py` run the
diff --git a/SETUP.md b/SETUP.md
index 44cf483..fbabebf 100644
--- a/SETUP.md
+++ b/SETUP.md
@@ -240,7 +240,7 @@ curl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>
| `Cannot connect to the Docker daemon` / `docker: command not found` | Docker Desktop is not running (or not installed). Open Docker Desktop, wait until its whale icon stops animating, then run `docker compose up -d` again. Or use the engine without Docker (§2 A). |
| `Empty reply from server` or `Connection reset` right after `docker compose up -d` | The engine is still starting, a few seconds. The scripts wait for it; in the browser, reload the page. |
| `Permission denied` running `./scripts/scrape.sh` | The file lost its execute bit (a ZIP download does that). Run `bash scripts/scrape.sh …` instead. |
-| `port is already allocated` / something else already uses 8080 | Set `SCRAPER_PORT=8081` in `.env`, then start the engine again (`python3 scripts/engine.py restart` or `docker compose up -d`). The engine, Compose and both scripts read the same file, so nothing else needs editing. With Docker the port inside the container stays 8080 (do not change the right-hand side of the mapping). If your `.env` still has a `SCRAPER_BASE_URL` line, delete it or update its port: it overrides `SCRAPER_PORT`. The slash commands and the skill show `localhost:8080` in their examples, so tell Claude the port you picked; its pre-approved `curl` rules are pinned to 8080 too, so add `Bash(curl http://localhost:8081/:*)` to `.claude/settings.json` if you want it to stop asking. **A second copy of the kit on the same machine** (a fork next to the original, a test clone): change `SCRAPER_PORT` in its `.env`. Without Docker that is all, because each copy keeps its own `.engine/` folder. With Docker, also set `COMPOSE_PROJECT_NAME` to a name nobody else uses (`docker compose ls` lists them), as `.env.example` shows; with the default name, `docker compose up -d` takes over the other copy's container. |
+| `port is already allocated` / something else already uses 8080 | Just updated from kit 1.x? Its `ui` container still holds the port: run `docker compose up -d --remove-orphans`. Otherwise set `SCRAPER_PORT=8081` in `.env`, then start the engine again (`python3 scripts/engine.py restart` or `docker compose up -d`). The engine, Compose and both scripts read the same file, so nothing else needs editing. With Docker the port inside the container stays 8080 (do not change the right-hand side of the mapping). If your `.env` still has a `SCRAPER_BASE_URL` line, delete it or update its port: it overrides `SCRAPER_PORT`. The slash commands and the skill show `localhost:8080` in their examples, so tell Claude the port you picked; its pre-approved `curl` rules are pinned to 8080 too, so add `Bash(curl http://localhost:8081/:*)` to `.claude/settings.json` if you want it to stop asking. **A second copy of the kit on the same machine** (a fork next to the original, a test clone): change `SCRAPER_PORT` in its `.env`. Without Docker that is all, because each copy keeps its own `.engine/` folder. With Docker, also set `COMPOSE_PROJECT_NAME` to a name nobody else uses (`docker compose ls` lists them), as `.env.example` shows; with the default name, `docker compose up -d` takes over the other copy's container. |
| `422 missing max time` | Job body needs `max_time` (in **seconds**, e.g. `300`). |
| `422 missing geo coordinates` | Job body needs `lat` and `lon` as **strings**, e.g. `"30.2672"`. |
| `422 missing name` / `422 missing lang` | Job body needs `name` (any label) and `lang` (`"en"`). All six required fields: `name`, `keywords`, `lang`, `lat`, `lon`, `max_time`. |
@@ -256,12 +256,13 @@ curl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>
### Upgrading from kit 1.x (the gosom engine)
-`bash scripts/update.sh` pulls the new kit and, if the old Docker engine is running, rebuilds and
-restarts it with `docker compose up -d --build --remove-orphans`. By hand, run that same command in the
-kit folder. `--remove-orphans` removes 1.x's `ui` container, which would otherwise keep port 8080.
-Jobs from 1.x are not moved over: they stay in the old `gmaps_data` volume, which `docker volume ls`
-lists with your project's name in front. Download what you need from it before you remove it with
-`docker volume rm`.
+The new engine cannot read 1.x jobs, so first download the CSVs you still need from the old web UI
+at http://localhost:8080. Then `bash scripts/update.sh` pulls the new kit and, if the old Docker
+engine is running, rebuilds and restarts it with `docker compose up -d --build --remove-orphans`. By
+hand, run that same command in the kit folder. `--remove-orphans` removes 1.x's `ui` container, which
+would otherwise keep port 8080. The old jobs stay in the old `gmaps_data` volume, which
+`docker volume ls` lists with your project's name in front; remove it with `docker volume rm` when
+you no longer need it.
---
diff --git a/scripts/engine.py b/scripts/engine.py
index 9ae4d6a..56ce5d6 100644
--- a/scripts/engine.py
+++ b/scripts/engine.py
@@ -271,7 +271,9 @@ def status():
print(f"ℹ Port {port} answers, but not as this kit's engine (the kit's older Docker engine, probably).")
sys.exit(1)
else:
- print(f"✗ Not running. Start it with: {PY} scripts/engine.py start")
+ found = browser.find_chrome()
+ print(f"✗ Not running. Start it with: {PY} scripts/engine.py start\n Browser: " + (found or
+ "none found, so start would download Chrome for Testing (about 100 MB, pinned by SHA-256)"))
sys.exit(1)
diff --git a/scripts/scrape.py b/scripts/scrape.py
index dc7f77d..9b978c8 100755
--- a/scripts/scrape.py
+++ b/scripts/scrape.py
@@ -84,6 +84,9 @@ RESTART = f"{PY} scripts/engine.py restart (Docker: docker compose restart)"
CHROME_START = ("No Chrome", "Chrome cannot start", "Chrome exited at once", "Could not start Chrome",
"Chrome did not open its DevTools port", "Could not connect to Chrome's DevTools port")
BLOCKED = "unusual-traffic page" # the engine's Error when Google refused this IP
+# Chrome's network errors for a proxy that failed. Matched as written: the engine's message also
+# holds the search URL, so a keyword such as "tunnel car wash" must not read as a proxy error.
+PROXY_ERROR = re.compile(r"\bERR_(?:PROXY|SOCKS|TUNNEL)_")
def poll_attempts(max_time):
@@ -197,6 +200,14 @@ class JobFailed(RuntimeError):
pass
+class Blocked(RuntimeError):
+ """Google answered with its unusual-traffic page. More searches from this IP only extend the
+ block, so the grid and city loops stop. Carries the rows found before it (job_id, rows)."""
+ def __init__(self, message, job_id=None, rows=()):
+ super().__init__(message)
+ self.job_id, self.rows = job_id, list(rows)
+
+
def run_job(body):
"""Create one engine job, poll it, download it -> (job_id, raw CSV rows). A job the engine marks
"failed" is retried once after a pause, unless its Error says a retry cannot help (Chrome would
@@ -211,12 +222,12 @@ def run_job(body):
raise RuntimeError(f"\n✗ Job failed: the engine could not start Chrome.{said}\n"
f" Fix that and scrape again. If it fails the same way, restart the "
f"engine: {RESTART}")
- if any(k in reason.upper() for k in ("PROXY", "SOCKS", "TUNNEL")):
+ if PROXY_ERROR.search(reason):
raise RuntimeError(f"\n✗ Job failed: the proxy did not work.{said}\n"
" Check its host and port; this engine takes no user:pass yet, so the "
"proxy must allow your IP address.")
- if BLOCKED in reason: # more searches from a rate-limited IP only extend the block
- raise RuntimeError(f"\n✗ Job failed.{said}")
+ if BLOCKED in reason:
+ raise Blocked(f"\n✗ Job failed.{said}")
if attempt == 2:
raise RuntimeError(f"\n✗ Job failed twice.{said}\n The engine's log has more: "
".engine/engine.log (Docker: docker compose logs)")
@@ -265,6 +276,8 @@ def _run_job_once(body):
print(f"\r status: {str(status):<10} (attempt {i + 1}{progress})", end="", flush=True)
if status == "ok":
print()
+ if BLOCKED in error: # the rows before the block are kept; searching on would extend it
+ raise Blocked(f"⚠ The engine says: {error}", job_id, download_rows(job_id))
if error:
print(f"ℹ The engine says: {error}")
return job_id, download_rows(job_id)
@@ -698,6 +711,8 @@ def scrape_place(a, body, city, keywords):
print(f"▶ Creating job: {len(keywords)} keyword(s) @ {lat},{lon} depth={a.depth}")
try:
job_id, rows = run_job(body)
+ except Blocked:
+ raise
except RuntimeError as e:
print(str(e))
return None, [], 0, 0
@@ -708,6 +723,11 @@ def scrape_place(a, body, city, keywords):
print(f"▶ Cell {i}/{len(cells)} @ {cell['lat']},{cell['lon']} radius={cell['radius']}m")
try:
jid, cell_rows = run_job(dict(body, **cell))
+ except Blocked as e: # keep what the tiles so far found, inside the city box, and stop
+ e.job_id = job_id or e.job_id
+ e.rows = [r for r in dedupe(rows + e.rows) if in_bbox(r, bbox)]
+ print(f"\n ⚠ Stopped at cell {i}/{len(cells)}: the cells after it were not searched.")
+ raise
except Exception as e:
print(f"\n ⚠ cell {i}/{len(cells)} skipped: {str(e).strip()}")
skipped += 1
@@ -857,8 +877,16 @@ def main():
if a.proxy:
body["proxies"] = a.proxy
job_id, rows, skipped, ncells, failed_places = None, [], 0, 0, 0
- for city in cities: # one place at a time, like the grid tiles
- jid, place_rows, s, n = scrape_place(a, body, city, keywords)
+ for i, city in enumerate(cities): # one place at a time, like the grid tiles
+ try:
+ jid, place_rows, s, n = scrape_place(a, body, city, keywords)
+ except Blocked as e:
+ print(str(e))
+ print("⚠ Google is rate-limiting this IP, so the scrape stopped here; the rows found so far "
+ "are kept. Wait (minutes to hours), or add proxies, then rerun with --exclude on this file.")
+ job_id, rows = job_id or e.job_id, dedupe(rows + e.rows)
+ failed_places += len(cities) - i - (1 if e.rows else 0)
+ break
job_id, skipped, ncells = job_id or jid, skipped + s, ncells + n
failed_places += jid is None
rows = dedupe(rows + place_rows)
diff --git a/scripts/test_scrape.py b/scripts/test_scrape.py
index 356b995..505e6e1 100644
--- a/scripts/test_scrape.py
+++ b/scripts/test_scrape.py
@@ -159,13 +159,27 @@ class JobOutcomes(unittest.TestCase):
self.assertIn("proxy did not work", str(err))
self.assertIn("ERR_PROXY_CONNECTION_FAILED", str(err))
+ def test_keyword_naming_a_tunnel_is_not_a_proxy_error(self):
+ # The engine's message holds the search URL, so the keyword is in it.
+ reason = "navigate https://www.google.com/maps/search/tunnel%20car%20wash%20socks: net::ERR_INTERNET_DISCONNECTED"
+ err, posts, _, _ = run_jobs([[("failed", reason)], [("failed", reason)]])
+ self.assertEqual(posts, 2) # retried like any other failure
+ self.assertNotIn("proxy", str(err))
+
def test_google_block_is_not_retried(self):
reason = ("Google answered with its unusual-traffic page: this IP is rate-limited for now. "
"Wait, or add proxies")
err, posts, slept, _ = run_jobs([[("failed", reason)]])
+ self.assertIsInstance(err, scrape.Blocked)
self.assertEqual(posts, 1)
slept.assert_not_called()
self.assertIn("unusual-traffic page", str(err))
+ # An ok job that stopped on the block keeps its rows, and still says stop.
+ note = "Google answered with its unusual-traffic page: this IP is rate-limited for now. " \
+ "Wait, or add proxies. The rows found before that are kept"
+ err, _, _, _ = run_jobs([[("ok", note)]], downloads={"j1": self.CSV})
+ self.assertIsInstance(err, scrape.Blocked)
+ self.assertEqual((err.job_id, len(err.rows)), ("j1", 2))
def test_failed_part_way_keeps_its_rows(self):
# The engine was stopped while the job ran: it marks the job failed and keeps the rows.
@@ -187,6 +201,32 @@ class JobOutcomes(unittest.TestCase):
self.assertIn("Gave up waiting", str(err))
+class StopsWhenBlocked(unittest.TestCase):
+ """After Google's unusual-traffic page, no more tiles or cities are searched."""
+ def run_main(self, argv, results):
+ out = os.path.join(tempfile.mkdtemp(), "leads.csv")
+ log = io.StringIO()
+ with mock.patch.object(scrape, "req", return_value=(200, b"null")), \
+ mock.patch.object(scrape, "run_job", side_effect=results) as run, \
+ mock.patch.object(scrape, "geocode", side_effect=lambda place, with_bbox=False: ("30.25", "-97.75") + ((BBOX,) if with_bbox else ())), \
+ mock.patch.object(sys, "argv", ["scrape.py", *argv, "--no-email", "--out", out]), redirect_stdout(log):
+ scrape.main()
+ with open(out, newline="", encoding="utf-8-sig") as f:
+ return run.call_count, [r["title"] for r in csv.DictReader(f)], log.getvalue()
+
+ def test_grid_stops_at_the_blocked_cell(self):
+ calls, titles, text = self.run_main(["plumbers", "--city", "Austin, TX", "--grid", "2"],
+ [("job-1", [_row("A"), _row("B")]), scrape.Blocked("✗ Job failed.")])
+ self.assertEqual((calls, titles), (2, ["Biz A", "Biz B"])) # cells 3 and 4 never searched
+ self.assertIn("Stopped at cell 2/4", text)
+ self.assertIn("rate-limiting this IP", text)
+
+ def test_cities_stop_at_the_blocked_one(self):
+ calls, titles, _ = self.run_main(["plumbers", "--city", "Austin, TX", "--city", "Waco, TX"],
+ [scrape.Blocked("⚠ blocked", "job-1", [_row("A")]), ("job-2", [_row("B")])])
+ self.assertEqual((calls, titles), (1, ["Biz A"]))
+
+
class NoDockerPaths(unittest.TestCase):
def test_script_never_reads_or_restarts_a_container(self):
# The old engine's workarounds read `docker logs` and ran `docker restart`, which could restart