Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
fix(kit): P4 audit loop 1: odd 1.x jobs never stop the engine, one more look on no-feed, exact health checks
Mahan Jafari894c3006ed4c
diff --git a/.claude/commands/scrape-setup.md b/.claude/commands/scrape-setup.mdindex dc73298..0792079 100644--- a/.claude/commands/scrape-setup.md+++ b/.claude/commands/scrape-setup.md@@ -5,7 +5,7 @@ Set up the kit's engine on this computer:1. Pick how the engine runs. By default, without Docker: from the kit root, run `python3 scripts/engine.py status` (Windows: `py scripts/engine.py status`). It needs Python 3.9 or newer. If it says the engine is running, go to step 3. It names the browser the engine will use; if it finds none, `start` would download Chrome for Testing once (about 100 MB, pinned by SHA-256), so ask the user first, or suggest installing Google Chrome or using Docker. If the user runs Docker instead, confirm it is up with `docker ps` (if it errors, point them to **Docker Desktop**, link in `SETUP.md` §0, and stop here), then run `docker compose up -d`; the first run builds the image, about a minute.2. Without Docker, start the engine: `python3 scripts/engine.py start` (Windows: `py scripts/engine.py start`).-3. Verify it is live: `curl http://localhost:8080/api/v1/jobs` should return `null` (no jobs yet) or a JSON array of jobs.+3. Verify it is live: `curl -s http://localhost:8080/api/v1/jobs` should return `null` (no jobs yet) or a JSON array of jobs.4. Report success. Tell the user they can now run `/scrape <business> in <city>`, or ask in plain language (e.g. "scrape gyms in Miami").5. **Show this warning once setup succeeds:**> **Before you scrape:** the engine sends real requests to Google Maps. Light use is fine. Big jobsdiff --git a/.claude/commands/scrape.md b/.claude/commands/scrape.mdindex 2c1c0c2..8f5bd3b 100644--- a/.claude/commands/scrape.md+++ b/.claude/commands/scrape.md@@ -14,7 +14,7 @@ Use the `google-maps-scraper` skill and follow these steps:If **yes**, add `--socials` in step 4. If **no**, leave it off. Socials cost no LLM tokens and one extra HTTP fetch per site (see the skill).4. **Run the scrape with emails on.** Easiest and fastest: the script (below), which keeps the engine's homepage-only email option off and crawls each business website itself, contact and about pages included. Only on the raw API, set `"email": true` in the body. Required fields: `name` (any label), `keywords` (location baked into each term), `lang` (`"en"`), `lat`/`lon` (strings), `max_time` (seconds, default `300`); plus `depth` (the number in the query, else `5`; each unit is about 20 places). Easiest path: `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --depth <n>` (emails are already on by default; append `--socials` if the user said yes in step 3).5. **Poll to completion in the background.** The Bash tool blocks foreground `sleep`, so use `run_in_background: true` and read the output file when notified. Then download the CSV.-6. **Save a CSV file.** `scripts/scrape.py` writes `results-<id>.csv` by default. If you use the raw API, save the parsed lead fields to a `.csv` yourself. Then **present only the 8 lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count` (`review_count` is empty by default), plus `instagram, facebook, linkedin, tiktok, youtube, x` if socials were requested. **Drop everything else**, especially `latitude`/`longitude`, IDs, hours, images and review blobs. Never dump the full raw CSV (36 columns plus `categories`) into chat. Show a few sample rows and give the CSV path. Include dropped fields only if the user explicitly asks.+6. **Save a CSV file.** `scripts/scrape.py` writes `results-<id>.csv` by default. If you use the raw API, save the parsed lead fields to a `.csv` yourself. Then **present only the 8 lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count`, plus `instagram, facebook, linkedin, tiktok, youtube, x` if socials were requested. **Drop everything else**, especially `latitude`/`longitude`, IDs, hours, images and review blobs. Never dump the full raw CSV (36 columns plus `categories`) into chat. Show a few sample rows and give the CSV path. Include dropped fields only if the user explicitly asks.**All, every or entire city:** add `--grid 3` (3 x 3 map tiles, one job at a time, deduped) to get past Google's ~120 results per search. Warn once that it is slower and more likely to trip rate limits, then proceed.diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdindex c9636c7..e5409b0 100644--- a/.claude/skills/google-maps-scraper/SKILL.md+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -16,9 +16,9 @@ result has up to **36 raw fields**, plus `categories`.## Step 0: Check the engine is running```bash-curl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOWN+curl -s http://localhost:8080/api/v1/jobs```-If `DOWN` and the working folder has `scripts/engine.py` (the full kit), first run+It prints `null` or a JSON list of jobs when the engine is up, and nothing when it is down. If down and the working folder has `scripts/engine.py` (the full kit), first run`python3 scripts/engine.py status` (`py` on Windows): it names the Chrome, Chromium, Edge or Brave theengine will drive (it exits 1 while the engine is not running; that is expected). Only when it names one, start the engine with `python3 scripts/engine.py start`,or run `docker compose up -d` if the user runs it in Docker, then retry. If it finds no browser, do not@@ -57,8 +57,7 @@ from anywhere else.`email_host` column.- `"reviews_max"` (0 to 5, default 0): the engine opens each place's panel, one page load per place,and fills `user_reviews` (a JSON list of `{"Rating", "When", "Description"}`, without reviewer names- or photos) and `reviews_per_rating` (`{"1": n, ... "5": n}`). A headless engine gets no reviews,- and the job's `Error` says so (see the window-mode note under Step 3).+ or photos) and `reviews_per_rating` (`{"1": n, ... "5": n}`).```bashcurl -s -X POST http://localhost:8080/api/v1/jobs \diff --git a/.github/live_reviews_check.py b/.github/live_reviews_check.pyindex 3f23879..cb2cf02 100644--- a/.github/live_reviews_check.py+++ b/.github/live_reviews_check.py@@ -1,8 +1,8 @@-"""engine-live.yml: one depth-1 job with reviews_max=1 against the engine on localhost:8080; exits 1+"""engine-live.yml: one depth-1 job with reviews_max=1 against the engine on localhost (SCRAPER_PORT, default 8080); exits 1unless Google's full view came back (review counts and reviews filled). Prints counts only."""-import json, sys, time, urllib.request+import json, os, sys, time, urllib.request-BASE = "http://localhost:8080/api/v1/jobs"+BASE = f"http://localhost:{os.environ.get('SCRAPER_PORT', '8080')}/api/v1/jobs"def call(method, url, body=None):diff --git a/CLAUDE.md b/CLAUDE.mdindex 8565845..c44783a 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -17,7 +17,7 @@ the API calls from memory. Use the skill.## Slash commandsThe kit ships four commands in `.claude/commands/` that map to the skill: `/scrape` (single query),`/scrape-batch` (many queries, one job), `/scrape-setup` (start the engine), `/scrape-jobs` (list or delete).-Local commands are pre-approved in `.claude/settings.json`, so you can run them without prompting the user.+The kit's own scripts, `docker compose` and the exact health check `curl -s http://localhost:8080/api/v1/jobs` are pre-approved in `.claude/settings.json`; other `curl` calls to the engine ask the user first.## Rate limits: warn, don't blockIf a request is large (high `depth`, many keywords, repeated runs, a grid, or `email:true`), **proceed**,diff --git a/SETUP.md b/SETUP.mdindex 185c0e6..efb472e 100644--- a/SETUP.md+++ b/SETUP.md@@ -164,7 +164,8 @@ your IP address): `--lang de --country de`.Signed out, Google gives a browser whose user agent says `HeadlessChrome` its "limited view": noreview counts, today's opening hours only, no reviews. The engine runs Chrome headless, so it sendsthat Chrome's own user agent with `HeadlessChrome` replaced by `Chrome` (same OS and version; nothing-else is changed). Google then returns `review_count`, the week's `open_hours` (as JSON like+else is changed). Google then returns (checked live on macOS, Linux, Windows and in Docker on+2026-10-31; Google can change this at any time) `review_count`, the week's `open_hours` (as JSON like`{"Monday": ["7 AM–9 PM"], ...}`) and `reviews_link`, with or without Docker and with the downloadedheadless shell too. `--min-reviews` filters on the counts. Google can still leave a field out; ifevery count is missing, the script warns and skips that filter rather than dropping every row.@@ -300,7 +301,8 @@ Nothing is lost:- The CSVs you downloaded are your own files; the update does not touch them.- The jobs in 1.x's web UI come along. On its first start the new engine reads 1.x's job volume(read-only) and lists those jobs with their CSVs; a job 1.x had not finished shows as `failed`,- with its rows. This happens once, so a job you delete afterwards stays deleted.+ with its rows, and a job that never produced a CSV is left out. This happens once, so a job you+ delete afterwards stays deleted.- The old volume itself is kept as it was. Once you no longer need it, free its disk space:`docker compose down`, then `docker volume rm` with the name `docker volume ls` shows ending in`_gmaps_data`, then `docker compose up -d` (Compose creates an empty one in its place).diff --git a/engine/cdp.py b/engine/cdp.pyindex fbb4861..3a4ac54 100644--- a/engine/cdp.py+++ b/engine/cdp.py@@ -235,13 +235,21 @@ class Page:if context:params["browserContextId"] = contextself.target = cdp.send("Target.createTarget", params)["targetId"]- self.sid = cdp.send("Target.attachToTarget", {"targetId": self.target, "flatten": True})["sessionId"]- for domain in ("Page", "Runtime", "Network"):- cdp.send(f"{domain}.enable", session_id=self.sid)- if cdp.user_agent:- self.send("Emulation.setUserAgentOverride", {"userAgent": cdp.user_agent})- if url != "about:blank":- self.navigate(url)+ try:+ self.sid = cdp.send("Target.attachToTarget", {"targetId": self.target, "flatten": True})["sessionId"]+ for domain in ("Page", "Runtime", "Network"):+ cdp.send(f"{domain}.enable", session_id=self.sid)+ if cdp.user_agent: # see browser.Browser: a headless Chrome's agent without "Headless"+ self.send("Emulation.setUserAgentOverride", {"userAgent": cdp.user_agent})+ if url != "about:blank":+ self.navigate(url)+ except Exception: # never leave a tab behind that nobody holds+ if not cdp.closed:+ try:+ cdp.send("Target.closeTarget", {"targetId": self.target})+ except Exception:+ pass+ raisedef send(self, method, params=None, timeout=None):return self.cdp.send(method, params, self.sid, timeout)diff --git a/engine/jobs.py b/engine/jobs.pyindex f3e624f..799604c 100644--- a/engine/jobs.py+++ b/engine/jobs.py@@ -283,11 +283,12 @@ class Store:passreturn bool(gone)- def import_kit1(self, folder):+ def import_kit1(self, folder, log=print):"""Kit 1.x's jobs and their CSVs, from the old engine's data folder (docker-compose.yml mountsits volume read-only on .engine/kit1), so an update from 1.x keeps them in the job list. Once- only: a marker file stops it next time, so a job deleted here stays deleted. Jobs imported;- an unreadable old database raises sqlite3.Error and is tried again on the next start."""+ only: a marker file stops it next time, so a job deleted here stays deleted. Jobs with no CSV+ (never run) are left out, and one that cannot be read is skipped and logged by id. Jobs+ imported; an unreadable old database raises sqlite3.Error and is tried again on the next start."""marker = os.path.join(self.folder, "kit1-imported")if os.path.exists(marker) or not os.path.isfile(os.path.join(folder, "jobs.db")):return 0@@ -304,6 +305,7 @@ class Store:old.close()finally:shutil.rmtree(tmp, ignore_errors=True)+ csv.field_size_limit(2 ** 31 - 1) # a 1.x cell (a long description) can pass csv's 128 KB defaultn = 0for jid, name, status, data, created in rows:try:@@ -313,25 +315,34 @@ class Store:src = os.path.join(folder, f"{jid}.csv")if not plain or not os.path.isfile(src) or self.get(jid):continue- try:- d = json.loads(data or "{}")- except ValueError:- d = {}- if not isinstance(d, dict):- d = {}- if isinstance(d.get("max_time"), int):- d["max_time"] //= 10 ** 9 # nanoseconds in 1.x, seconds here- with open(src, newline="", encoding="utf-8", errors="replace") as f:- places = max(sum(1 for _ in csv.reader(f)) - 1, 0)- ok = status == "ok"- with self.lock, self.db:- shutil.copyfile(src, self.csv_path(jid))- self.db.execute("INSERT INTO jobs (id, name, date, status, data, error, progress) VALUES (?, ?, ?, ?, ?, ?, ?)",- (jid, str(name), datetime.fromtimestamp(int(created), timezone.utc).isoformat(timespec="seconds"),- "ok" if ok else "failed", json.dumps(d),- "" if ok else f"kit 1.x job that was {status} there; its rows are kept",- json.dumps({"done": len(d.get("keywords") or []) if ok else 0, "places": places,- "stage": "done" if ok else ""})))+ try: # one odd old job must never stop the engine from starting+ try:+ d = json.loads(data or "{}")+ except ValueError:+ d = {}+ if not isinstance(d, dict):+ d = {}+ if isinstance(d.get("max_time"), int):+ d["max_time"] //= 10 ** 9 # nanoseconds in 1.x, seconds here+ with open(src, newline="", encoding="utf-8", errors="replace") as f:+ places = max(sum(1 for _ in csv.reader(f)) - 1, 0)+ date = datetime.fromtimestamp(int(created), timezone.utc).isoformat(timespec="seconds")+ ok = status == "ok"+ with self.lock, self.db:+ shutil.copyfile(src, self.csv_path(jid))+ self.db.execute("INSERT INTO jobs (id, name, date, status, data, error, progress) "+ "VALUES (?, ?, ?, ?, ?, ?, ?)",+ (jid, str(name), date, "ok" if ok else "failed", json.dumps(d),+ "" if ok else f"kit 1.x job that was {status} there; its rows are kept",+ json.dumps({"done": len(d.get("keywords") or []) if ok else 0, "places": places,+ "stage": "done" if ok else ""})))+ except Exception as e:+ try:+ os.remove(self.csv_path(jid))+ except OSError:+ pass+ log(f"kit 1.x job {jid} not imported: {type(e).__name__}")+ continuen += 1with open(marker, "w"):pass@@ -539,7 +550,7 @@ class Worker(threading.Thread):timed_out["yes"] = True # resumed with no time left: the other keywords stay unsearchedif ((stage == "search" and done < len(kws)) or reviews) and budget > 0 and not stop.is_set():self._browser = browser.Browser(lang=locale(d), headless=self.headless)- i = done+ i, again = done, Falsewhile stage == "search" and i < len(kws) and self._browser:left = deadline - time.monotonic()if left <= 0:@@ -578,7 +589,7 @@ class Worker(threading.Thread):if not stop.is_set() and len(kws) - i > 1:errors[-1] += f"; {len(kws) - i - 1} keyword(s) after it were not searched"break- i, retries = i + 1, 0+ i, retries, again = i + 1, 0, Falsesave(failed=len(errors), last_error=errors[-1])continueon_batch(places, stats["batches"]) # places no batch carried (one place, the DOM fallback)@@ -599,7 +610,14 @@ class Worker(threading.Thread):blocked = Trueself.log(f"job {jid} keyword {i + 1}/{len(kws)}: Google's unusual-traffic page; stopping")break- retries = 0+ if reason == "no-feed" and not again and time.monotonic() + 30 < deadline:+ again = True # Google sometimes shows no result list once (seen in CI): look once more+ self.log(f"job {jid} keyword {i + 1}/{len(kws)}: no result list; one more try")+ continue+ if reason == "no-feed":+ notes.append(f"keyword {i + 1}: Google showed no result list twice (no results for it, "+ "or a page the engine cannot read)")+ retries, again = 0, Falseif reason.startswith("timeout"): # "timeout+dom-fallback" tootimed_out["yes"] = Truesize = self.store.append(jid, [], cols)diff --git a/engine/serve.py b/engine/serve.pyindex 152a82c..9b752bf 100644--- a/engine/serve.py+++ b/engine/serve.py@@ -15,7 +15,7 @@ Google Maps, for email or socials jobs to the businesses' own websites, and forto Google's public DNS (dns.google), nothing else.Standard library only."""-import argparse, csv, hmac, html, io, json, os, re, secrets, signal, socket, socketserver, sqlite3, sys, threading+import argparse, csv, hmac, html, io, json, os, re, secrets, signal, socket, socketserver, sys, threadingimport urllib.parsefrom http.server import BaseHTTPRequestHandler, ThreadingHTTPServer@@ -531,8 +531,8 @@ def main():if failed:log(f"{failed} job(s) stopped mid-run {jobs.MAX_RESUMES + 1} times: marked failed, rows kept")try:- imported = store.import_kit1(os.path.join(browser.HOME, "kit1"))- except (OSError, sqlite3.Error, ValueError) as e:+ imported = store.import_kit1(os.path.join(browser.HOME, "kit1"), log)+ except Exception as e: # the old jobs never stop the engine from startingimported = 0log(f"kit 1.x jobs not imported, tried again on the next start: {type(e).__name__}: {e}")if imported:diff --git a/engine/test_cdp.py b/engine/test_cdp.pyindex 60e9149..4895fb2 100644--- a/engine/test_cdp.py+++ b/engine/test_cdp.py@@ -264,6 +264,18 @@ class PageUserAgent(unittest.TestCase):cdp.Page(conn)self.assertNotIn("Emulation.setUserAgentOverride", [c[0] for c in conn.calls])+ def test_a_failed_setup_closes_the_tab(self):+ from unittest.mock import Mock+ conn = Mock(user_agent="Mozilla/5.0 Chrome/156.0.0.0", closed=None)+ def send(method, params=None, session_id=None, timeout=None):+ if method == "Emulation.setUserAgentOverride":+ raise cdp.CDPError("refused")+ return {"targetId": "T", "sessionId": "S"}+ conn.send.side_effect = send+ with self.assertRaises(cdp.CDPError):+ cdp.Page(conn)+ self.assertEqual(conn.send.call_args.args, ("Target.closeTarget", {"targetId": "T"}))+def test_initial_url_loads_after_tab_configuration(self):from unittest.mock import Mockconn = Mock(user_agent="Mozilla/5.0 Chrome/156.0.0.0")diff --git a/engine/test_engine.py b/engine/test_engine.pyindex d320c7a..52203c5 100644--- a/engine/test_engine.py+++ b/engine/test_engine.py@@ -223,18 +223,26 @@ class Store(Fakes):w.execute("PRAGMA journal_mode=WAL")w.execute("CREATE TABLE jobs (id TEXT PRIMARY KEY, name TEXT NOT NULL, status TEXT NOT NULL, ""data TEXT NOT NULL, created_at INT NOT NULL, updated_at INT NOT NULL)")- done, stuck, nocsv = (str(uuid.uuid4()) for _ in range(3))+ done, stuck, nocsv, baddate, big = (str(uuid.uuid4()) for _ in range(5))body = json.dumps({"keywords": ["cafe", "bar"], "lang": "en", "max_time": 180 * 10 ** 9})with w:- for jid, status in ((done, "ok"), (stuck, "working"), (nocsv, "ok"), ("../evil", "ok")):+ for jid, status in ((done, "ok"), (stuck, "working"), (nocsv, "ok"), ("../evil", "ok"), (big, "ok")):w.execute("INSERT INTO jobs VALUES (?, ?, ?, ?, 1793466309, 1793466322)", (jid, "old " + status, status, body))- for jid in (done, stuck, "../evil"):+ w.execute("INSERT INTO jobs VALUES (?, 'odd', 'ok', ?, 'not a time', 0)", (baddate, body))+ for jid in (done, stuck, "../evil", baddate):with open(os.path.join(old, f"{jid}.csv"), "w", newline="") as f:f.write("input_id,title\n1,\"A\nB\"\n2,C\n")+ with open(os.path.join(old, f"{big}.csv"), "w", newline="") as f:+ f.write("input_id,descriptions\n1," + "x" * 200_000 + "\n") # past csv's 128 KB field limit+ logged = []try:- self.assertEqual(self.store.import_kit1(old), 2)+ self.assertEqual(self.store.import_kit1(old, logged.append), 3)finally:w.close()+ self.assertEqual(logged, [f"kit 1.x job {baddate} not imported: ValueError"]) # skipped, the rest kept+ self.assertIsNone(self.store.get(baddate))+ self.assertFalse(os.path.exists(self.store.csv_path(baddate)))+ self.assertEqual(jobs.public(self.store.get(big))["Progress"]["places"], 1)a, b = self.store.get(done), self.store.get(stuck)self.assertEqual((a["Status"], a["Name"], a["Date"], a["Data"]["max_time"]), ("ok", "old ok", "2026-10-31T17:05:09+00:00", 180))self.assertEqual(jobs.public(a)["Progress"], {"keywords_done": 2, "keywords_total": 2, "places": 2, "percent": 100, "stage": "done"})@@ -274,6 +282,17 @@ class Worker(Fakes):self.assertEqual(job["Status"], "failed")self.assertIn("No Chrome", job["Error"])+ def test_no_feed_is_tried_once_more(self):+ RESULTS["cafes in Austin"] = ([place(1)], ["no-feed", "end-of-list"]) # once, then the list+ _, job = self.run_one()+ self.assertEqual((job["Status"], SEARCHED.count("cafes in Austin"), job["Error"]), ("ok", 2, ""))+ self.assertEqual(job["Progress"]["places"], 1)+ SEARCHED.clear()+ RESULTS["cafes in Austin"] = ([], "no-feed") # twice: a note, no third look+ _, job = self.run_one()+ self.assertEqual((job["Status"], SEARCHED.count("cafes in Austin")), ("ok", 2))+ self.assertIn("no result list twice", job["Error"])+def test_blocked_fails(self):RESULTS["cafes in Austin"] = ([], "blocked")_, job = self.run_one()