Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
fix(kit): P2 audit loop 1: proxy errors, Google blocks, upgrade notes, no download wording in the skill
Mahan Jafarif5c85bf15291
diff --git a/.claude/commands/scrape-setup.md b/.claude/commands/scrape-setup.mdindex 40b27a3..dc73298 100644--- a/.claude/commands/scrape-setup.md+++ b/.claude/commands/scrape-setup.md@@ -3,8 +3,8 @@ description: Start the local Google Maps scraper engine and run a health check.---Set up the kit's engine on this computer:-1. Pick how the engine runs. By default, without Docker: from the kit root, `python3 scripts/engine.py start` (Windows: `py scripts/engine.py start`). It needs Python 3.9 or newer and an installed Chrome, Chromium, Edge or Brave. On a machine with none, it downloads Chrome for Testing once (about 100 MB, pinned by SHA-256): tell the user before you run it. If the user runs Docker instead, confirm it is up with `docker ps` (if it errors, point them to **Docker Desktop**, link in `SETUP.md` §0, and stop here), then run `docker compose up -d`; the first run builds the image, about a minute.-2. `python3 scripts/engine.py status` prints the browser the engine uses (skip it for Docker).+1. Pick how the engine runs. By default, without Docker: from the kit root, run `python3 scripts/engine.py status` (Windows: `py scripts/engine.py status`). It needs Python 3.9 or newer. If it says the engine is running, go to step 3. It names the browser the engine will use; if it finds none, `start` would download Chrome for Testing once (about 100 MB, pinned by SHA-256), so ask the user first, or suggest installing Google Chrome or using Docker. If the user runs Docker instead, confirm it is up with `docker ps` (if it errors, point them to **Docker Desktop**, link in `SETUP.md` §0, and stop here), then run `docker compose up -d`; the first run builds the image, about a minute.+2. Without Docker, start the engine: `python3 scripts/engine.py start` (Windows: `py scripts/engine.py start`).3. Verify it is live: `curl http://localhost:8080/api/v1/jobs` should return `null` (no jobs yet) or a JSON array of jobs.4. Report success. Tell the user they can now run `/scrape <business> in <city>`, or ask in plain language (e.g. "scrape gyms in Miami").5. **Show this warning once setup succeeds:**diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdindex 3e40321..c6b8a14 100644--- a/.claude/skills/google-maps-scraper/SKILL.md+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -20,9 +20,9 @@ curl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOW```If `DOWN` and the working folder has `scripts/engine.py` (the full kit), start the engine there:`python3 scripts/engine.py start` (`py` on Windows), or `docker compose up -d` if the user runs it in-Docker. Then retry. `engine.py start` uses the installed Chrome, Chromium, Edge or Brave; on a machine-with none it downloads Chrome for Testing once (about 100 MB, pinned by SHA-256), so tell the user-before you run it there. Setup help:+Docker. Then retry. The engine drives the installed Chrome, Chromium, Edge or Brave:+`python3 scripts/engine.py status` names the one it will use. If it finds none, do not start it: ask+the user to install Google Chrome, or to use Docker. Setup help:[SETUP.md](https://github.com/Mahanaicoach/google-maps-scraper-kit/blob/master/SETUP.md).**Skill installed on its own?** `npx skills add` copies this file only. The engine ships with thediff --git a/CLAUDE.md b/CLAUDE.mdindex e595e73..c95cd7c 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -45,9 +45,10 @@ clearly abusive use (surveilling individuals, spam). The skill has the full guid## SetupIf the engine is not running: `python3 scripts/engine.py start` (or `docker compose up -d` when the user-runs Docker), then verify with `curl http://localhost:8080/api/v1/jobs`. On a machine with no Chrome,-Chromium, Edge or Brave, `engine.py start` downloads Chrome for Testing once (about 100 MB, pinned by-SHA-256): tell the user before you run it. `SETUP.md` has the full setup.+runs Docker), then verify with `curl http://localhost:8080/api/v1/jobs`. `python3 scripts/engine.py status`+names the browser the engine will use. If it finds none, `start` would download Chrome for Testing+(about 100 MB, pinned by SHA-256): ask the user first, or suggest installing Google Chrome or using+Docker. `SETUP.md` has the full setup.## Changing the kit`python3 scripts/test_scrape.py`, `python3 engine/test_engine.py` and `python3 engine/test_cdp.py` run thediff --git a/SETUP.md b/SETUP.mdindex 44cf483..fbabebf 100644--- a/SETUP.md+++ b/SETUP.md@@ -240,7 +240,7 @@ curl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>| `Cannot connect to the Docker daemon` / `docker: command not found` | Docker Desktop is not running (or not installed). Open Docker Desktop, wait until its whale icon stops animating, then run `docker compose up -d` again. Or use the engine without Docker (§2 A). || `Empty reply from server` or `Connection reset` right after `docker compose up -d` | The engine is still starting, a few seconds. The scripts wait for it; in the browser, reload the page. || `Permission denied` running `./scripts/scrape.sh` | The file lost its execute bit (a ZIP download does that). Run `bash scripts/scrape.sh …` instead. |-| `port is already allocated` / something else already uses 8080 | Set `SCRAPER_PORT=8081` in `.env`, then start the engine again (`python3 scripts/engine.py restart` or `docker compose up -d`). The engine, Compose and both scripts read the same file, so nothing else needs editing. With Docker the port inside the container stays 8080 (do not change the right-hand side of the mapping). If your `.env` still has a `SCRAPER_BASE_URL` line, delete it or update its port: it overrides `SCRAPER_PORT`. The slash commands and the skill show `localhost:8080` in their examples, so tell Claude the port you picked; its pre-approved `curl` rules are pinned to 8080 too, so add `Bash(curl http://localhost:8081/:*)` to `.claude/settings.json` if you want it to stop asking. **A second copy of the kit on the same machine** (a fork next to the original, a test clone): change `SCRAPER_PORT` in its `.env`. Without Docker that is all, because each copy keeps its own `.engine/` folder. With Docker, also set `COMPOSE_PROJECT_NAME` to a name nobody else uses (`docker compose ls` lists them), as `.env.example` shows; with the default name, `docker compose up -d` takes over the other copy's container. |+| `port is already allocated` / something else already uses 8080 | Just updated from kit 1.x? Its `ui` container still holds the port: run `docker compose up -d --remove-orphans`. Otherwise set `SCRAPER_PORT=8081` in `.env`, then start the engine again (`python3 scripts/engine.py restart` or `docker compose up -d`). The engine, Compose and both scripts read the same file, so nothing else needs editing. With Docker the port inside the container stays 8080 (do not change the right-hand side of the mapping). If your `.env` still has a `SCRAPER_BASE_URL` line, delete it or update its port: it overrides `SCRAPER_PORT`. The slash commands and the skill show `localhost:8080` in their examples, so tell Claude the port you picked; its pre-approved `curl` rules are pinned to 8080 too, so add `Bash(curl http://localhost:8081/:*)` to `.claude/settings.json` if you want it to stop asking. **A second copy of the kit on the same machine** (a fork next to the original, a test clone): change `SCRAPER_PORT` in its `.env`. Without Docker that is all, because each copy keeps its own `.engine/` folder. With Docker, also set `COMPOSE_PROJECT_NAME` to a name nobody else uses (`docker compose ls` lists them), as `.env.example` shows; with the default name, `docker compose up -d` takes over the other copy's container. || `422 missing max time` | Job body needs `max_time` (in **seconds**, e.g. `300`). || `422 missing geo coordinates` | Job body needs `lat` and `lon` as **strings**, e.g. `"30.2672"`. || `422 missing name` / `422 missing lang` | Job body needs `name` (any label) and `lang` (`"en"`). All six required fields: `name`, `keywords`, `lang`, `lat`, `lon`, `max_time`. |@@ -256,12 +256,13 @@ curl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>### Upgrading from kit 1.x (the gosom engine)-`bash scripts/update.sh` pulls the new kit and, if the old Docker engine is running, rebuilds and-restarts it with `docker compose up -d --build --remove-orphans`. By hand, run that same command in the-kit folder. `--remove-orphans` removes 1.x's `ui` container, which would otherwise keep port 8080.-Jobs from 1.x are not moved over: they stay in the old `gmaps_data` volume, which `docker volume ls`-lists with your project's name in front. Download what you need from it before you remove it with-`docker volume rm`.+The new engine cannot read 1.x jobs, so first download the CSVs you still need from the old web UI+at http://localhost:8080. Then `bash scripts/update.sh` pulls the new kit and, if the old Docker+engine is running, rebuilds and restarts it with `docker compose up -d --build --remove-orphans`. By+hand, run that same command in the kit folder. `--remove-orphans` removes 1.x's `ui` container, which+would otherwise keep port 8080. The old jobs stay in the old `gmaps_data` volume, which+`docker volume ls` lists with your project's name in front; remove it with `docker volume rm` when+you no longer need it.---diff --git a/scripts/engine.py b/scripts/engine.pyindex 9ae4d6a..56ce5d6 100644--- a/scripts/engine.py+++ b/scripts/engine.py@@ -271,7 +271,9 @@ def status():print(f"ℹ Port {port} answers, but not as this kit's engine (the kit's older Docker engine, probably).")sys.exit(1)else:- print(f"✗ Not running. Start it with: {PY} scripts/engine.py start")+ found = browser.find_chrome()+ print(f"✗ Not running. Start it with: {PY} scripts/engine.py start\n Browser: " + (found or+ "none found, so start would download Chrome for Testing (about 100 MB, pinned by SHA-256)"))sys.exit(1)diff --git a/scripts/scrape.py b/scripts/scrape.pyindex dc7f77d..9b978c8 100755--- a/scripts/scrape.py+++ b/scripts/scrape.py@@ -84,6 +84,9 @@ RESTART = f"{PY} scripts/engine.py restart (Docker: docker compose restart)"CHROME_START = ("No Chrome", "Chrome cannot start", "Chrome exited at once", "Could not start Chrome","Chrome did not open its DevTools port", "Could not connect to Chrome's DevTools port")BLOCKED = "unusual-traffic page" # the engine's Error when Google refused this IP+# Chrome's network errors for a proxy that failed. Matched as written: the engine's message also+# holds the search URL, so a keyword such as "tunnel car wash" must not read as a proxy error.+PROXY_ERROR = re.compile(r"\bERR_(?:PROXY|SOCKS|TUNNEL)_")def poll_attempts(max_time):@@ -197,6 +200,14 @@ class JobFailed(RuntimeError):pass+class Blocked(RuntimeError):+ """Google answered with its unusual-traffic page. More searches from this IP only extend the+ block, so the grid and city loops stop. Carries the rows found before it (job_id, rows)."""+ def __init__(self, message, job_id=None, rows=()):+ super().__init__(message)+ self.job_id, self.rows = job_id, list(rows)++def run_job(body):"""Create one engine job, poll it, download it -> (job_id, raw CSV rows). A job the engine marks"failed" is retried once after a pause, unless its Error says a retry cannot help (Chrome would@@ -211,12 +222,12 @@ def run_job(body):raise RuntimeError(f"\n✗ Job failed: the engine could not start Chrome.{said}\n"f" Fix that and scrape again. If it fails the same way, restart the "f"engine: {RESTART}")- if any(k in reason.upper() for k in ("PROXY", "SOCKS", "TUNNEL")):+ if PROXY_ERROR.search(reason):raise RuntimeError(f"\n✗ Job failed: the proxy did not work.{said}\n"" Check its host and port; this engine takes no user:pass yet, so the ""proxy must allow your IP address.")- if BLOCKED in reason: # more searches from a rate-limited IP only extend the block- raise RuntimeError(f"\n✗ Job failed.{said}")+ if BLOCKED in reason:+ raise Blocked(f"\n✗ Job failed.{said}")if attempt == 2:raise RuntimeError(f"\n✗ Job failed twice.{said}\n The engine's log has more: "".engine/engine.log (Docker: docker compose logs)")@@ -265,6 +276,8 @@ def _run_job_once(body):print(f"\r status: {str(status):<10} (attempt {i + 1}{progress})", end="", flush=True)if status == "ok":print()+ if BLOCKED in error: # the rows before the block are kept; searching on would extend it+ raise Blocked(f"⚠ The engine says: {error}", job_id, download_rows(job_id))if error:print(f"ℹ The engine says: {error}")return job_id, download_rows(job_id)@@ -698,6 +711,8 @@ def scrape_place(a, body, city, keywords):print(f"▶ Creating job: {len(keywords)} keyword(s) @ {lat},{lon} depth={a.depth}")try:job_id, rows = run_job(body)+ except Blocked:+ raiseexcept RuntimeError as e:print(str(e))return None, [], 0, 0@@ -708,6 +723,11 @@ def scrape_place(a, body, city, keywords):print(f"▶ Cell {i}/{len(cells)} @ {cell['lat']},{cell['lon']} radius={cell['radius']}m")try:jid, cell_rows = run_job(dict(body, **cell))+ except Blocked as e: # keep what the tiles so far found, inside the city box, and stop+ e.job_id = job_id or e.job_id+ e.rows = [r for r in dedupe(rows + e.rows) if in_bbox(r, bbox)]+ print(f"\n ⚠ Stopped at cell {i}/{len(cells)}: the cells after it were not searched.")+ raiseexcept Exception as e:print(f"\n ⚠ cell {i}/{len(cells)} skipped: {str(e).strip()}")skipped += 1@@ -857,8 +877,16 @@ def main():if a.proxy:body["proxies"] = a.proxyjob_id, rows, skipped, ncells, failed_places = None, [], 0, 0, 0- for city in cities: # one place at a time, like the grid tiles- jid, place_rows, s, n = scrape_place(a, body, city, keywords)+ for i, city in enumerate(cities): # one place at a time, like the grid tiles+ try:+ jid, place_rows, s, n = scrape_place(a, body, city, keywords)+ except Blocked as e:+ print(str(e))+ print("⚠ Google is rate-limiting this IP, so the scrape stopped here; the rows found so far "+ "are kept. Wait (minutes to hours), or add proxies, then rerun with --exclude on this file.")+ job_id, rows = job_id or e.job_id, dedupe(rows + e.rows)+ failed_places += len(cities) - i - (1 if e.rows else 0)+ breakjob_id, skipped, ncells = job_id or jid, skipped + s, ncells + nfailed_places += jid is Nonerows = dedupe(rows + place_rows)diff --git a/scripts/test_scrape.py b/scripts/test_scrape.pyindex 356b995..505e6e1 100644--- a/scripts/test_scrape.py+++ b/scripts/test_scrape.py@@ -159,13 +159,27 @@ class JobOutcomes(unittest.TestCase):self.assertIn("proxy did not work", str(err))self.assertIn("ERR_PROXY_CONNECTION_FAILED", str(err))+ def test_keyword_naming_a_tunnel_is_not_a_proxy_error(self):+ # The engine's message holds the search URL, so the keyword is in it.+ reason = "navigate https://www.google.com/maps/search/tunnel%20car%20wash%20socks: net::ERR_INTERNET_DISCONNECTED"+ err, posts, _, _ = run_jobs([[("failed", reason)], [("failed", reason)]])+ self.assertEqual(posts, 2) # retried like any other failure+ self.assertNotIn("proxy", str(err))+def test_google_block_is_not_retried(self):reason = ("Google answered with its unusual-traffic page: this IP is rate-limited for now. ""Wait, or add proxies")err, posts, slept, _ = run_jobs([[("failed", reason)]])+ self.assertIsInstance(err, scrape.Blocked)self.assertEqual(posts, 1)slept.assert_not_called()self.assertIn("unusual-traffic page", str(err))+ # An ok job that stopped on the block keeps its rows, and still says stop.+ note = "Google answered with its unusual-traffic page: this IP is rate-limited for now. " \+ "Wait, or add proxies. The rows found before that are kept"+ err, _, _, _ = run_jobs([[("ok", note)]], downloads={"j1": self.CSV})+ self.assertIsInstance(err, scrape.Blocked)+ self.assertEqual((err.job_id, len(err.rows)), ("j1", 2))def test_failed_part_way_keeps_its_rows(self):# The engine was stopped while the job ran: it marks the job failed and keeps the rows.@@ -187,6 +201,32 @@ class JobOutcomes(unittest.TestCase):self.assertIn("Gave up waiting", str(err))+class StopsWhenBlocked(unittest.TestCase):+ """After Google's unusual-traffic page, no more tiles or cities are searched."""+ def run_main(self, argv, results):+ out = os.path.join(tempfile.mkdtemp(), "leads.csv")+ log = io.StringIO()+ with mock.patch.object(scrape, "req", return_value=(200, b"null")), \+ mock.patch.object(scrape, "run_job", side_effect=results) as run, \+ mock.patch.object(scrape, "geocode", side_effect=lambda place, with_bbox=False: ("30.25", "-97.75") + ((BBOX,) if with_bbox else ())), \+ mock.patch.object(sys, "argv", ["scrape.py", *argv, "--no-email", "--out", out]), redirect_stdout(log):+ scrape.main()+ with open(out, newline="", encoding="utf-8-sig") as f:+ return run.call_count, [r["title"] for r in csv.DictReader(f)], log.getvalue()++ def test_grid_stops_at_the_blocked_cell(self):+ calls, titles, text = self.run_main(["plumbers", "--city", "Austin, TX", "--grid", "2"],+ [("job-1", [_row("A"), _row("B")]), scrape.Blocked("✗ Job failed.")])+ self.assertEqual((calls, titles), (2, ["Biz A", "Biz B"])) # cells 3 and 4 never searched+ self.assertIn("Stopped at cell 2/4", text)+ self.assertIn("rate-limiting this IP", text)++ def test_cities_stop_at_the_blocked_one(self):+ calls, titles, _ = self.run_main(["plumbers", "--city", "Austin, TX", "--city", "Waco, TX"],+ [scrape.Blocked("⚠ blocked", "job-1", [_row("A")]), ("job-2", [_row("B")])])+ self.assertEqual((calls, titles), (1, ["Biz A"]))++class NoDockerPaths(unittest.TestCase):def test_script_never_reads_or_restarts_a_container(self):# The old engine's workarounds read `docker logs` and ran `docker restart`, which could restart