Skip to content

Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.

fix(kit): P4 audit loop 1: odd 1.x jobs never stop the engine, one more look on no-feed, exact health checks

Mahan Jafari894c3006ed4c
diff --git a/.claude/commands/scrape-setup.md b/.claude/commands/scrape-setup.md
index dc73298..0792079 100644
--- a/.claude/commands/scrape-setup.md
+++ b/.claude/commands/scrape-setup.md
@@ -5,7 +5,7 @@ Set up the kit's engine on this computer:
1. Pick how the engine runs. By default, without Docker: from the kit root, run `python3 scripts/engine.py status` (Windows: `py scripts/engine.py status`). It needs Python 3.9 or newer. If it says the engine is running, go to step 3. It names the browser the engine will use; if it finds none, `start` would download Chrome for Testing once (about 100 MB, pinned by SHA-256), so ask the user first, or suggest installing Google Chrome or using Docker. If the user runs Docker instead, confirm it is up with `docker ps` (if it errors, point them to **Docker Desktop**, link in `SETUP.md` §0, and stop here), then run `docker compose up -d`; the first run builds the image, about a minute.
2. Without Docker, start the engine: `python3 scripts/engine.py start` (Windows: `py scripts/engine.py start`).
-3. Verify it is live: `curl http://localhost:8080/api/v1/jobs` should return `null` (no jobs yet) or a JSON array of jobs.
+3. Verify it is live: `curl -s http://localhost:8080/api/v1/jobs` should return `null` (no jobs yet) or a JSON array of jobs.
4. Report success. Tell the user they can now run `/scrape <business> in <city>`, or ask in plain language (e.g. "scrape gyms in Miami").
5. **Show this warning once setup succeeds:**
> **Before you scrape:** the engine sends real requests to Google Maps. Light use is fine. Big jobs
diff --git a/.claude/commands/scrape.md b/.claude/commands/scrape.md
index 2c1c0c2..8f5bd3b 100644
--- a/.claude/commands/scrape.md
+++ b/.claude/commands/scrape.md
@@ -14,7 +14,7 @@ Use the `google-maps-scraper` skill and follow these steps:
If **yes**, add `--socials` in step 4. If **no**, leave it off. Socials cost no LLM tokens and one extra HTTP fetch per site (see the skill).
4. **Run the scrape with emails on.** Easiest and fastest: the script (below), which keeps the engine's homepage-only email option off and crawls each business website itself, contact and about pages included. Only on the raw API, set `"email": true` in the body. Required fields: `name` (any label), `keywords` (location baked into each term), `lang` (`"en"`), `lat`/`lon` (strings), `max_time` (seconds, default `300`); plus `depth` (the number in the query, else `5`; each unit is about 20 places). Easiest path: `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --depth <n>` (emails are already on by default; append `--socials` if the user said yes in step 3).
5. **Poll to completion in the background.** The Bash tool blocks foreground `sleep`, so use `run_in_background: true` and read the output file when notified. Then download the CSV.
-6. **Save a CSV file.** `scripts/scrape.py` writes `results-<id>.csv` by default. If you use the raw API, save the parsed lead fields to a `.csv` yourself. Then **present only the 8 lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count` (`review_count` is empty by default), plus `instagram, facebook, linkedin, tiktok, youtube, x` if socials were requested. **Drop everything else**, especially `latitude`/`longitude`, IDs, hours, images and review blobs. Never dump the full raw CSV (36 columns plus `categories`) into chat. Show a few sample rows and give the CSV path. Include dropped fields only if the user explicitly asks.
+6. **Save a CSV file.** `scripts/scrape.py` writes `results-<id>.csv` by default. If you use the raw API, save the parsed lead fields to a `.csv` yourself. Then **present only the 8 lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count`, plus `instagram, facebook, linkedin, tiktok, youtube, x` if socials were requested. **Drop everything else**, especially `latitude`/`longitude`, IDs, hours, images and review blobs. Never dump the full raw CSV (36 columns plus `categories`) into chat. Show a few sample rows and give the CSV path. Include dropped fields only if the user explicitly asks.
**All, every or entire city:** add `--grid 3` (3 x 3 map tiles, one job at a time, deduped) to get past Google's ~120 results per search. Warn once that it is slower and more likely to trip rate limits, then proceed.
diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.md
index c9636c7..e5409b0 100644
--- a/.claude/skills/google-maps-scraper/SKILL.md
+++ b/.claude/skills/google-maps-scraper/SKILL.md
@@ -16,9 +16,9 @@ result has up to **36 raw fields**, plus `categories`.
## Step 0: Check the engine is running
```bash
-curl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOWN
+curl -s http://localhost:8080/api/v1/jobs
```
-If `DOWN` and the working folder has `scripts/engine.py` (the full kit), first run
+It prints `null` or a JSON list of jobs when the engine is up, and nothing when it is down. If down and the working folder has `scripts/engine.py` (the full kit), first run
`python3 scripts/engine.py status` (`py` on Windows): it names the Chrome, Chromium, Edge or Brave the
engine will drive (it exits 1 while the engine is not running; that is expected). Only when it names one, start the engine with `python3 scripts/engine.py start`,
or run `docker compose up -d` if the user runs it in Docker, then retry. If it finds no browser, do not
@@ -57,8 +57,7 @@ from anywhere else.
`email_host` column.
- `"reviews_max"` (0 to 5, default 0): the engine opens each place's panel, one page load per place,
and fills `user_reviews` (a JSON list of `{"Rating", "When", "Description"}`, without reviewer names
- or photos) and `reviews_per_rating` (`{"1": n, ... "5": n}`). A headless engine gets no reviews,
- and the job's `Error` says so (see the window-mode note under Step 3).
+ or photos) and `reviews_per_rating` (`{"1": n, ... "5": n}`).
```bash
curl -s -X POST http://localhost:8080/api/v1/jobs \
diff --git a/.github/live_reviews_check.py b/.github/live_reviews_check.py
index 3f23879..cb2cf02 100644
--- a/.github/live_reviews_check.py
+++ b/.github/live_reviews_check.py
@@ -1,8 +1,8 @@
-"""engine-live.yml: one depth-1 job with reviews_max=1 against the engine on localhost:8080; exits 1
+"""engine-live.yml: one depth-1 job with reviews_max=1 against the engine on localhost (SCRAPER_PORT, default 8080); exits 1
unless Google's full view came back (review counts and reviews filled). Prints counts only."""
-import json, sys, time, urllib.request
+import json, os, sys, time, urllib.request
-BASE = "http://localhost:8080/api/v1/jobs"
+BASE = f"http://localhost:{os.environ.get('SCRAPER_PORT', '8080')}/api/v1/jobs"
def call(method, url, body=None):
diff --git a/CLAUDE.md b/CLAUDE.md
index 8565845..c44783a 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -17,7 +17,7 @@ the API calls from memory. Use the skill.
## Slash commands
The kit ships four commands in `.claude/commands/` that map to the skill: `/scrape` (single query),
`/scrape-batch` (many queries, one job), `/scrape-setup` (start the engine), `/scrape-jobs` (list or delete).
-Local commands are pre-approved in `.claude/settings.json`, so you can run them without prompting the user.
+The kit's own scripts, `docker compose` and the exact health check `curl -s http://localhost:8080/api/v1/jobs` are pre-approved in `.claude/settings.json`; other `curl` calls to the engine ask the user first.
## Rate limits: warn, don't block
If a request is large (high `depth`, many keywords, repeated runs, a grid, or `email:true`), **proceed**,
diff --git a/SETUP.md b/SETUP.md
index 185c0e6..efb472e 100644
--- a/SETUP.md
+++ b/SETUP.md
@@ -164,7 +164,8 @@ your IP address): `--lang de --country de`.
Signed out, Google gives a browser whose user agent says `HeadlessChrome` its "limited view": no
review counts, today's opening hours only, no reviews. The engine runs Chrome headless, so it sends
that Chrome's own user agent with `HeadlessChrome` replaced by `Chrome` (same OS and version; nothing
-else is changed). Google then returns `review_count`, the week's `open_hours` (as JSON like
+else is changed). Google then returns (checked live on macOS, Linux, Windows and in Docker on
+2026-10-31; Google can change this at any time) `review_count`, the week's `open_hours` (as JSON like
`{"Monday": ["7 AM–9 PM"], ...}`) and `reviews_link`, with or without Docker and with the downloaded
headless shell too. `--min-reviews` filters on the counts. Google can still leave a field out; if
every count is missing, the script warns and skips that filter rather than dropping every row.
@@ -300,7 +301,8 @@ Nothing is lost:
- The CSVs you downloaded are your own files; the update does not touch them.
- The jobs in 1.x's web UI come along. On its first start the new engine reads 1.x's job volume
(read-only) and lists those jobs with their CSVs; a job 1.x had not finished shows as `failed`,
- with its rows. This happens once, so a job you delete afterwards stays deleted.
+ with its rows, and a job that never produced a CSV is left out. This happens once, so a job you
+ delete afterwards stays deleted.
- The old volume itself is kept as it was. Once you no longer need it, free its disk space:
`docker compose down`, then `docker volume rm` with the name `docker volume ls` shows ending in
`_gmaps_data`, then `docker compose up -d` (Compose creates an empty one in its place).
diff --git a/engine/cdp.py b/engine/cdp.py
index fbb4861..3a4ac54 100644
--- a/engine/cdp.py
+++ b/engine/cdp.py
@@ -235,13 +235,21 @@ class Page:
if context:
params["browserContextId"] = context
self.target = cdp.send("Target.createTarget", params)["targetId"]
- self.sid = cdp.send("Target.attachToTarget", {"targetId": self.target, "flatten": True})["sessionId"]
- for domain in ("Page", "Runtime", "Network"):
- cdp.send(f"{domain}.enable", session_id=self.sid)
- if cdp.user_agent:
- self.send("Emulation.setUserAgentOverride", {"userAgent": cdp.user_agent})
- if url != "about:blank":
- self.navigate(url)
+ try:
+ self.sid = cdp.send("Target.attachToTarget", {"targetId": self.target, "flatten": True})["sessionId"]
+ for domain in ("Page", "Runtime", "Network"):
+ cdp.send(f"{domain}.enable", session_id=self.sid)
+ if cdp.user_agent: # see browser.Browser: a headless Chrome's agent without "Headless"
+ self.send("Emulation.setUserAgentOverride", {"userAgent": cdp.user_agent})
+ if url != "about:blank":
+ self.navigate(url)
+ except Exception: # never leave a tab behind that nobody holds
+ if not cdp.closed:
+ try:
+ cdp.send("Target.closeTarget", {"targetId": self.target})
+ except Exception:
+ pass
+ raise
def send(self, method, params=None, timeout=None):
return self.cdp.send(method, params, self.sid, timeout)
diff --git a/engine/jobs.py b/engine/jobs.py
index f3e624f..799604c 100644
--- a/engine/jobs.py
+++ b/engine/jobs.py
@@ -283,11 +283,12 @@ class Store:
pass
return bool(gone)
- def import_kit1(self, folder):
+ def import_kit1(self, folder, log=print):
"""Kit 1.x's jobs and their CSVs, from the old engine's data folder (docker-compose.yml mounts
its volume read-only on .engine/kit1), so an update from 1.x keeps them in the job list. Once
- only: a marker file stops it next time, so a job deleted here stays deleted. Jobs imported;
- an unreadable old database raises sqlite3.Error and is tried again on the next start."""
+ only: a marker file stops it next time, so a job deleted here stays deleted. Jobs with no CSV
+ (never run) are left out, and one that cannot be read is skipped and logged by id. Jobs
+ imported; an unreadable old database raises sqlite3.Error and is tried again on the next start."""
marker = os.path.join(self.folder, "kit1-imported")
if os.path.exists(marker) or not os.path.isfile(os.path.join(folder, "jobs.db")):
return 0
@@ -304,6 +305,7 @@ class Store:
old.close()
finally:
shutil.rmtree(tmp, ignore_errors=True)
+ csv.field_size_limit(2 ** 31 - 1) # a 1.x cell (a long description) can pass csv's 128 KB default
n = 0
for jid, name, status, data, created in rows:
try:
@@ -313,25 +315,34 @@ class Store:
src = os.path.join(folder, f"{jid}.csv")
if not plain or not os.path.isfile(src) or self.get(jid):
continue
- try:
- d = json.loads(data or "{}")
- except ValueError:
- d = {}
- if not isinstance(d, dict):
- d = {}
- if isinstance(d.get("max_time"), int):
- d["max_time"] //= 10 ** 9 # nanoseconds in 1.x, seconds here
- with open(src, newline="", encoding="utf-8", errors="replace") as f:
- places = max(sum(1 for _ in csv.reader(f)) - 1, 0)
- ok = status == "ok"
- with self.lock, self.db:
- shutil.copyfile(src, self.csv_path(jid))
- self.db.execute("INSERT INTO jobs (id, name, date, status, data, error, progress) VALUES (?, ?, ?, ?, ?, ?, ?)",
- (jid, str(name), datetime.fromtimestamp(int(created), timezone.utc).isoformat(timespec="seconds"),
- "ok" if ok else "failed", json.dumps(d),
- "" if ok else f"kit 1.x job that was {status} there; its rows are kept",
- json.dumps({"done": len(d.get("keywords") or []) if ok else 0, "places": places,
- "stage": "done" if ok else ""})))
+ try: # one odd old job must never stop the engine from starting
+ try:
+ d = json.loads(data or "{}")
+ except ValueError:
+ d = {}
+ if not isinstance(d, dict):
+ d = {}
+ if isinstance(d.get("max_time"), int):
+ d["max_time"] //= 10 ** 9 # nanoseconds in 1.x, seconds here
+ with open(src, newline="", encoding="utf-8", errors="replace") as f:
+ places = max(sum(1 for _ in csv.reader(f)) - 1, 0)
+ date = datetime.fromtimestamp(int(created), timezone.utc).isoformat(timespec="seconds")
+ ok = status == "ok"
+ with self.lock, self.db:
+ shutil.copyfile(src, self.csv_path(jid))
+ self.db.execute("INSERT INTO jobs (id, name, date, status, data, error, progress) "
+ "VALUES (?, ?, ?, ?, ?, ?, ?)",
+ (jid, str(name), date, "ok" if ok else "failed", json.dumps(d),
+ "" if ok else f"kit 1.x job that was {status} there; its rows are kept",
+ json.dumps({"done": len(d.get("keywords") or []) if ok else 0, "places": places,
+ "stage": "done" if ok else ""})))
+ except Exception as e:
+ try:
+ os.remove(self.csv_path(jid))
+ except OSError:
+ pass
+ log(f"kit 1.x job {jid} not imported: {type(e).__name__}")
+ continue
n += 1
with open(marker, "w"):
pass
@@ -539,7 +550,7 @@ class Worker(threading.Thread):
timed_out["yes"] = True # resumed with no time left: the other keywords stay unsearched
if ((stage == "search" and done < len(kws)) or reviews) and budget > 0 and not stop.is_set():
self._browser = browser.Browser(lang=locale(d), headless=self.headless)
- i = done
+ i, again = done, False
while stage == "search" and i < len(kws) and self._browser:
left = deadline - time.monotonic()
if left <= 0:
@@ -578,7 +589,7 @@ class Worker(threading.Thread):
if not stop.is_set() and len(kws) - i > 1:
errors[-1] += f"; {len(kws) - i - 1} keyword(s) after it were not searched"
break
- i, retries = i + 1, 0
+ i, retries, again = i + 1, 0, False
save(failed=len(errors), last_error=errors[-1])
continue
on_batch(places, stats["batches"]) # places no batch carried (one place, the DOM fallback)
@@ -599,7 +610,14 @@ class Worker(threading.Thread):
blocked = True
self.log(f"job {jid} keyword {i + 1}/{len(kws)}: Google's unusual-traffic page; stopping")
break
- retries = 0
+ if reason == "no-feed" and not again and time.monotonic() + 30 < deadline:
+ again = True # Google sometimes shows no result list once (seen in CI): look once more
+ self.log(f"job {jid} keyword {i + 1}/{len(kws)}: no result list; one more try")
+ continue
+ if reason == "no-feed":
+ notes.append(f"keyword {i + 1}: Google showed no result list twice (no results for it, "
+ "or a page the engine cannot read)")
+ retries, again = 0, False
if reason.startswith("timeout"): # "timeout+dom-fallback" too
timed_out["yes"] = True
size = self.store.append(jid, [], cols)
diff --git a/engine/serve.py b/engine/serve.py
index 152a82c..9b752bf 100644
--- a/engine/serve.py
+++ b/engine/serve.py
@@ -15,7 +15,7 @@ Google Maps, for email or socials jobs to the businesses' own websites, and for
to Google's public DNS (dns.google), nothing else.
Standard library only.
"""
-import argparse, csv, hmac, html, io, json, os, re, secrets, signal, socket, socketserver, sqlite3, sys, threading
+import argparse, csv, hmac, html, io, json, os, re, secrets, signal, socket, socketserver, sys, threading
import urllib.parse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
@@ -531,8 +531,8 @@ def main():
if failed:
log(f"{failed} job(s) stopped mid-run {jobs.MAX_RESUMES + 1} times: marked failed, rows kept")
try:
- imported = store.import_kit1(os.path.join(browser.HOME, "kit1"))
- except (OSError, sqlite3.Error, ValueError) as e:
+ imported = store.import_kit1(os.path.join(browser.HOME, "kit1"), log)
+ except Exception as e: # the old jobs never stop the engine from starting
imported = 0
log(f"kit 1.x jobs not imported, tried again on the next start: {type(e).__name__}: {e}")
if imported:
diff --git a/engine/test_cdp.py b/engine/test_cdp.py
index 60e9149..4895fb2 100644
--- a/engine/test_cdp.py
+++ b/engine/test_cdp.py
@@ -264,6 +264,18 @@ class PageUserAgent(unittest.TestCase):
cdp.Page(conn)
self.assertNotIn("Emulation.setUserAgentOverride", [c[0] for c in conn.calls])
+ def test_a_failed_setup_closes_the_tab(self):
+ from unittest.mock import Mock
+ conn = Mock(user_agent="Mozilla/5.0 Chrome/156.0.0.0", closed=None)
+ def send(method, params=None, session_id=None, timeout=None):
+ if method == "Emulation.setUserAgentOverride":
+ raise cdp.CDPError("refused")
+ return {"targetId": "T", "sessionId": "S"}
+ conn.send.side_effect = send
+ with self.assertRaises(cdp.CDPError):
+ cdp.Page(conn)
+ self.assertEqual(conn.send.call_args.args, ("Target.closeTarget", {"targetId": "T"}))
+
def test_initial_url_loads_after_tab_configuration(self):
from unittest.mock import Mock
conn = Mock(user_agent="Mozilla/5.0 Chrome/156.0.0.0")
diff --git a/engine/test_engine.py b/engine/test_engine.py
index d320c7a..52203c5 100644
--- a/engine/test_engine.py
+++ b/engine/test_engine.py
@@ -223,18 +223,26 @@ class Store(Fakes):
w.execute("PRAGMA journal_mode=WAL")
w.execute("CREATE TABLE jobs (id TEXT PRIMARY KEY, name TEXT NOT NULL, status TEXT NOT NULL, "
"data TEXT NOT NULL, created_at INT NOT NULL, updated_at INT NOT NULL)")
- done, stuck, nocsv = (str(uuid.uuid4()) for _ in range(3))
+ done, stuck, nocsv, baddate, big = (str(uuid.uuid4()) for _ in range(5))
body = json.dumps({"keywords": ["cafe", "bar"], "lang": "en", "max_time": 180 * 10 ** 9})
with w:
- for jid, status in ((done, "ok"), (stuck, "working"), (nocsv, "ok"), ("../evil", "ok")):
+ for jid, status in ((done, "ok"), (stuck, "working"), (nocsv, "ok"), ("../evil", "ok"), (big, "ok")):
w.execute("INSERT INTO jobs VALUES (?, ?, ?, ?, 1793466309, 1793466322)", (jid, "old " + status, status, body))
- for jid in (done, stuck, "../evil"):
+ w.execute("INSERT INTO jobs VALUES (?, 'odd', 'ok', ?, 'not a time', 0)", (baddate, body))
+ for jid in (done, stuck, "../evil", baddate):
with open(os.path.join(old, f"{jid}.csv"), "w", newline="") as f:
f.write("input_id,title\n1,\"A\nB\"\n2,C\n")
+ with open(os.path.join(old, f"{big}.csv"), "w", newline="") as f:
+ f.write("input_id,descriptions\n1," + "x" * 200_000 + "\n") # past csv's 128 KB field limit
+ logged = []
try:
- self.assertEqual(self.store.import_kit1(old), 2)
+ self.assertEqual(self.store.import_kit1(old, logged.append), 3)
finally:
w.close()
+ self.assertEqual(logged, [f"kit 1.x job {baddate} not imported: ValueError"]) # skipped, the rest kept
+ self.assertIsNone(self.store.get(baddate))
+ self.assertFalse(os.path.exists(self.store.csv_path(baddate)))
+ self.assertEqual(jobs.public(self.store.get(big))["Progress"]["places"], 1)
a, b = self.store.get(done), self.store.get(stuck)
self.assertEqual((a["Status"], a["Name"], a["Date"], a["Data"]["max_time"]), ("ok", "old ok", "2026-10-31T17:05:09+00:00", 180))
self.assertEqual(jobs.public(a)["Progress"], {"keywords_done": 2, "keywords_total": 2, "places": 2, "percent": 100, "stage": "done"})
@@ -274,6 +282,17 @@ class Worker(Fakes):
self.assertEqual(job["Status"], "failed")
self.assertIn("No Chrome", job["Error"])
+ def test_no_feed_is_tried_once_more(self):
+ RESULTS["cafes in Austin"] = ([place(1)], ["no-feed", "end-of-list"]) # once, then the list
+ _, job = self.run_one()
+ self.assertEqual((job["Status"], SEARCHED.count("cafes in Austin"), job["Error"]), ("ok", 2, ""))
+ self.assertEqual(job["Progress"]["places"], 1)
+ SEARCHED.clear()
+ RESULTS["cafes in Austin"] = ([], "no-feed") # twice: a note, no third look
+ _, job = self.run_one()
+ self.assertEqual((job["Status"], SEARCHED.count("cafes in Austin")), ("ok", 2))
+ self.assertIn("no result list twice", job["Error"])
+
def test_blocked_fails(self):
RESULTS["cafes in Austin"] = ([], "blocked")
_, job = self.run_one()