Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
feat: tell users when a newer kit is released (web UI line and engine.py status)
Mahan1e389e7308ac
diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdindex 488f778..29f31e7 100644--- a/.claude/skills/google-maps-scraper/SKILL.md+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -32,7 +32,8 @@ python3 scripts/engine.py statusIt prints `✓ Running on http://localhost:<port>` only when the kit's own engine answers (it checks theengine's `X-Engine` header), so another program on that port cannot fool it. When the engine is notrunning it prints `✗ Not running` and exits 1; that is expected. It also names the Chrome, Chromium,-Edge or Brave the engine will drive. Only when it names one, start the engine with+Edge or Brave the engine will drive. If it prints a `↑ Kit vX.Y.Z is out` line, tell the user once and carry+on; run `bash scripts/update.sh` only when the user asks. Only when it names one, start the engine with`python3 scripts/engine.py start`, or run `docker compose up -d` if the user runs it in Docker, then run`status` again. If it finds no browser, do not start it: ask the user to install Google Chrome, or touse Docker. Setup help: `SETUP.md` in the kit folder.diff --git a/.env.example b/.env.exampleindex a7f8854..37cc967 100644--- a/.env.example+++ b/.env.example@@ -32,3 +32,9 @@ SCRAPER_API_KEY=# on macOS a second Chrome icon shows in the Dock and the window may take focus when a job starts.# Not used in Docker, where Chrome is always headless.# SCRAPER_HEADLESS=0++# Update notice. Default: on. `python3 scripts/engine.py status` and the web UI ask GitHub's public+# releases endpoint once a day whether a newer kit is out (nothing about you in the request) and say+# so; nothing updates itself. Set 1 to turn the check off (the web UI's check is in your browser and+# is not affected by this file).+# SCRAPER_NO_UPDATE_CHECK=1diff --git a/CLAUDE.md b/CLAUDE.mdindex 4b68749..51e78ef 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -54,7 +54,8 @@ individuals, spam). The skill has the full guidance.## SetupCheck the engine first with `python3 scripts/engine.py status`. It prints `✓ Running on http://localhost:<port>`only when the kit's own engine answers (it checks the engine's `X-Engine` header, so another program on-the port cannot fool it), and that URL is the base to use. It also names the browser the engine will use.+the port cannot fool it), and that URL is the base to use. It also names the browser the engine will use, and prints `↑ Kit vX.Y.Z is out` when a newer+release exists: mention it once; never run `scripts/update.sh` unasked.If the engine is not running, run `python3 scripts/engine.py start` (or `docker compose up -d` when theuser runs Docker), then run `status` again. If `status` finds no browser, `start` would download Chrome forTesting (about 100 MB, pinned by SHA-256): ask the user first, or suggest installing Google Chrome ordiff --git a/README.md b/README.mdindex 843526e..52879d1 100644--- a/README.md+++ b/README.md@@ -200,7 +200,7 @@ scrapes Google Maps business listings only. It does not scrape social platforms| **Batch jobs** | Many keywords, one job, one CSV. The engine lists a place found by several keywords once. || **Proxies** | For large or repeated runs: `--proxy URL` with `http`, `https`, `socks5` or `socks5h`, and `user:pass@` on `http` and `https`. Repeat it and the keywords take turns; a keyword Google refuses is retried through the next proxy. || **Agent-native** | A skill plus four slash commands. The agent runs the script and presents the lead fields; it never hand-builds API calls. |-| **Safe defaults** | API bound to `127.0.0.1`, and the engine refuses requests from other websites in your browser (DNS rebinding, cross-site posts); an optional API key. No telemetry: the engine talks to Google Maps, and to the businesses' websites and `dns.google` only for jobs that ask. Proxy passwords show as `***`. A cell that a spreadsheet would run as a formula gets a leading `'` in the engine's CSV download and in the script's file. Results git-ignored; website fetches skip private addresses; rate-limit and data-law guardrails built into the skill. |+| **Safe defaults** | API bound to `127.0.0.1`, and the engine refuses requests from other websites in your browser (DNS rebinding, cross-site posts); an optional API key. No telemetry: the engine talks to Google Maps, and to the businesses' websites and `dns.google` only for jobs that ask. The web UI and `engine.py status` ask GitHub once a day whether a newer release is out, with nothing about you in the request (`SCRAPER_NO_UPDATE_CHECK=1` in `.env` turns it off); nothing updates itself. Proxy passwords show as `***`. A cell that a spreadsheet would run as a formula gets a leading `'` in the engine's CSV download and in the script's file. Results git-ignored; website fetches skip private addresses; rate-limit and data-law guardrails built into the skill. |### Measured, not estimated@@ -451,7 +451,9 @@ google-maps-scraper-kit/## Updating-Pull the latest fixes with one command. It works in a clone of this repo or of your fork, and it+The web UI shows a line under its title when a newer release is out, and `python3 scripts/engine.py status`+prints one (both check GitHub's releases once a day; `SCRAPER_NO_UPDATE_CHECK=1` in `.env` turns it off).+Nothing updates by itself. Pull the latest fixes with one command. It works in a clone of this repo or of your fork, and itkeeps your own commits:```bashdiff --git a/SETUP.md b/SETUP.mdindex 1c11a42..44be288 100644--- a/SETUP.md+++ b/SETUP.md@@ -101,6 +101,10 @@ separate job lists, so jobs started in one do not show in the other.python3 scripts/engine.py status # should print ✓ Running on http://localhost:8080```+When a newer kit is released, `status` adds a line (`↑ Kit v2.3.0 is out ... bash scripts/update.sh`) and the+web UI shows the same under its title. Both ask GitHub's releases endpoint once a day, with nothing about+you in the request; `SCRAPER_NO_UPDATE_CHECK=1` in `.env` turns the check off. Nothing updates itself.+It prints `✓ Running on http://localhost:<port>` only when the kit's own engine answers (it checks theengine's `X-Engine` header, so another program on the port cannot fool it). The engine also serves aweb UI and API docs in the browser:diff --git a/engine/serve.py b/engine/serve.pyindex 95fa3e6..70448f5 100644--- a/engine/serve.py+++ b/engine/serve.py@@ -34,6 +34,10 @@ REPO = "https://github.com/Mahanaicoach/google-maps-scraper-kit"# The web UI's sponsor box: the browser reads this static file from GitHub (cached a day, no# tracking), so the sponsors it lists change without anyone updating their copy of the kit.SPONSORS = "https://raw.githubusercontent.com/Mahanaicoach/google-maps-scraper-kit/master/assets/sponsors/sponsors.json"+# The web UI's update notice: the browser asks GitHub for the kit's latest release once a day (a+# public JSON endpoint, nothing about the user in the request) and shows a line when it is newer+# than this copy. Nothing updates itself: the line says to run scripts/update.sh.+RELEASES = "https://api.github.com/repos/Mahanaicoach/google-maps-scraper-kit/releases/latest"# The web UI's What and Where boxes offer only these, so nobody searches for a business type or a# place that does not exist: presets/business-types.json (hand-picked; add a niche there) and# presets/cities.tsv.gz (GeoNames cities1000, CC BY 4.0: name, region, country code, lat, lon, biggest first).@@ -259,6 +263,8 @@ a {{ color: inherit; text-underline-offset: 3px; }} a:hover {{ color: var(--go);h1 {{ margin: 0; font: 600 1.6rem/1.2 var(--serif); letter-spacing: -.01em; }}.top nav {{ display: flex; gap: 16px; font-size: .9rem; color: var(--soft); }}.mono {{ font-family: var(--mono); font-size: .85em; }} .soft {{ color: var(--soft); font-size: .875rem; }}+.update {{ margin-top: 14px; padding: 10px 14px; border: 1px solid var(--rule); border-radius: 10px; background: var(--sheet); font-size: .9rem; }}+.update code {{ font-size: .85rem; }}.stars {{ display: flex; flex-wrap: wrap; align-items: center; gap: 8px 14px; padding-top: 16px; color: var(--soft); font-size: .9rem; }}.gh {{ display: inline-flex; align-items: center; gap: 8px; min-height: 40px; padding: 8px 14px; border-radius: 8px;color: var(--paper); background: var(--ink); font-size: .9rem; font-weight: 600; text-decoration: none; transition: filter .15s; }}@@ -368,6 +374,9 @@ progress {{ display: block; width: 100%; max-width: 200px; height: 6px; margin-t</style></head><body><main><header class="top"><h1>Google Maps Scraper Kit</h1><nav><span class="mono">v{version}</span><a href="/api/docs">API docs</a></nav></header>+<div class="update" id="update" role="status" hidden><b>Kit v<span id="up-v"></span> is out</b> (this copy is v{version}).+ <a id="up-link" href="{repo}/releases/latest" target="_blank" rel="noopener">What changed</a> ·+ update with <code>bash scripts/update.sh</code>, or <code>docker compose up -d --build</code> in Docker.</div><div class="stars"><a class="gh" href="{repo}" target="_blank" rel="noopener"><span class="star"><svg width="16" height="16" viewBox="0 0 24 24" aria-hidden="true"><path fill="currentColor" d="M12 2.5l2.9 6 6.6.8-4.9 4.6 1.3 6.6L12 17.2l-5.9 3.3 1.3-6.6L2.5 9.3l6.6-.8z"/></svg></span>Star on GitHub</a><span>Useful? A star helps other people find this free kit.</span></div><aside class="sponsors" id="sponsors" aria-label="Sponsors"><button type="button" class="sp-close" id="sp-close"@@ -616,6 +625,27 @@ $('f').addEventListener('submit', async ev => {{finally {{ go.disabled = false; go.textContent = 'Start scraping'; load(); }}}});load(); setInterval(load, 3000);+(function () {{ // update notice: GitHub's latest release once a day, shown only when it is newer than this copy+ const key = 'gmskit-update', day = 86400000, mine = '{version}';+ const num = v => (String(v).replace(/^v/, '').split('.').slice(0, 3).map(x => parseInt(x, 10) || 0));+ const newer = (a, b) => {{ const x = num(a), y = num(b); for (let i = 0; i < 3; i++) if (x[i] !== y[i]) return x[i] > y[i]; return false; }};+ function show(tag) {{+ if (!/^v?\d+\.\d+\.\d+/.test(tag || '') || !newer(tag, mine)) return;+ $('up-v').textContent = String(tag).replace(/^v/, '').slice(0, 20);+ $('up-link').href = '{repo}/releases/tag/' + encodeURIComponent(tag);+ $('update').hidden = false;+ }}+ try {{+ const c = JSON.parse(localStorage.getItem(key) || 'null');+ if (c && Date.now() - c.t < day) return show(c.tag);+ }} catch (e) {{}}+ fetch('{releases}', {{cache: 'no-store', credentials: 'omit', headers: {{Accept: 'application/vnd.github+json'}}}})+ .then(r => r.ok ? r.json() : null).then(d => {{+ if (!d || typeof d.tag_name !== 'string') return;+ show(d.tag_name);+ try {{ localStorage.setItem(key, JSON.stringify({{t: Date.now(), tag: d.tag_name}})); }} catch (e) {{}}+ }}).catch(() => {{}});+}})();(function () {{ // sponsor tiles: only the entries Mahan marks "ui": true in sponsors.json, as text, https links onlyconst box = $('sponsors'), grid = $('sp-grid'), key = 'gmskit-sponsors', day = 86400000;$('sp-close').addEventListener('click', () => {{ box.hidden = true; }}); // as gosom: hidden until the page reloads@@ -774,10 +804,10 @@ class Handler(BaseHTTPRequestHandler):if path == "/":nonce = secrets.token_urlsafe(16)csp = (f"default-src 'none'; script-src 'nonce-{nonce}'; style-src 'nonce-{nonce}'; "- "connect-src 'self' https://raw.githubusercontent.com; img-src 'self' data:; "+ "connect-src 'self' https://raw.githubusercontent.com https://api.github.com; img-src 'self' data:; ""form-action 'self'; base-uri 'none'; ""frame-ancestors 'none'")- page = PAGE.format(nonce=nonce, version=VERSION, repo=REPO, sponsors=SPONSORS, types=TYPES_JS,+ page = PAGE.format(nonce=nonce, version=VERSION, repo=REPO, sponsors=SPONSORS, releases=RELEASES, types=TYPES_JS,categories=CATEGORIES).encode()return self.reply(200, page, "text/html; charset=utf-8", [("Content-Security-Policy", csp)])self.fail(404, "not found")diff --git a/engine/test_engine.py b/engine/test_engine.pyindex 3d11595..c12c001 100644--- a/engine/test_engine.py+++ b/engine/test_engine.py@@ -505,8 +505,10 @@ class Api(Server):self.assertIn(f'<script nonce="{nonce}">', body.decode())self.assertIn("github.com/Mahanaicoach/google-maps-scraper-kit", body.decode())self.assertIn("frame-ancestors 'none'", csp)- self.assertIn("connect-src 'self' https://raw.githubusercontent.com;", csp) # sponsors.json, nothing else+ # sponsors.json from raw.githubusercontent.com and the latest release from api.github.com, nothing else+ self.assertIn("connect-src 'self' https://raw.githubusercontent.com https://api.github.com;", csp)self.assertIn("assets/sponsors/sponsors.json", body.decode())+ self.assertIn("/releases/latest", body.decode())self.assertNotRegex(body.decode(), r"\{[a-z_]+\}") # every placeholder filledcode, body, _ = self.call("GET", "/api/openapi.json")self.assertEqual(json.loads(body)["paths"]["/api/v1/jobs"]["post"]["requestBody"]["content"]["application/json"]@@ -1286,6 +1288,11 @@ class ApiP3(Server):for rule in ("s.ui === true", "s.url.startsWith('https://')", ".slice(0, 4)", "noopener noreferrer sponsored"):self.assertIn(rule, ui)self.assertNotIn(".innerHTML", ui) # sponsor text is set as text, never parsed as HTML+ # update notice: hidden by default, filled from GitHub's latest release once a day, shown only when newer+ self.assertIn('<div class="update" id="update" role="status" hidden>', ui)+ for rule in ("gmskit-update", "api.github.com/repos/Mahanaicoach/google-maps-scraper-kit/releases/latest",+ "newer(tag, mine)", "bash scripts/update.sh"):+ self.assertIn(rule, ui)script = re.search(r"<script nonce=[^>]+>(.*)</script>", ui, re.S).group(1)self.assertNotIn("'\n'", script) # PAGE must say \\n: a bare newline in a JS string breaks the pagediff --git a/scripts/engine.py b/scripts/engine.pyindex 416a3e1..ee6d49a 100644--- a/scripts/engine.py+++ b/scripts/engine.py@@ -35,6 +35,11 @@ LOG_KEEP = 2_000_000 # bytes of engine.log kept across startsMARK = "gmskit-engine/"_NO_PROXY = urllib.request.build_opener(urllib.request.ProxyHandler({})) # HTTP_PROXY must not catch localhostPY = "py" if os.name == "nt" else "python3" # what the user types; the README says `py` on Windows+# `status` tells the user when a newer kit is released: one request a day to GitHub's public releases+# endpoint (nothing about the user in it), cached here. It only prints; scripts/update.sh updates.+RELEASES = "https://api.github.com/repos/Mahanaicoach/google-maps-scraper-kit/releases/latest"+RELEASE_PAGE = "https://github.com/Mahanaicoach/google-maps-scraper-kit/releases/latest"+UPDATE_CACHE = os.path.join(HOME, "update-check.json")def public_port():@@ -320,6 +325,7 @@ def status():if seen == "ours":st = read_state() or {}print(f"✓ Running on http://localhost:{port}" + (f" (browser: {st['chrome']})" if st.get("chrome") else ""))+ update_hint()elif seen == "other":print(f"ℹ Port {port} answers, but not as this kit's engine (the kit's older Docker engine, probably).")sys.exit(1)@@ -327,9 +333,67 @@ def status():found = browser.find_chrome()print(f"✗ Not running. Start it with: {PY} scripts/engine.py start\n Browser: " + (found or"none found, so start would download Chrome for Testing (about 100 MB, pinned by SHA-256)"))+ update_hint()sys.exit(1)+def local_version():+ """This copy's engine version, read from engine/serve.py without importing it."""+ path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "engine", "serve.py")+ try:+ with open(path, encoding="utf-8") as f:+ m = re.search(r'^VERSION = "([^"]+)"', f.read(), re.M)+ return m.group(1) if m else ""+ except OSError:+ return ""+++def newer(latest, current):+ """True when `latest` (a tag like v2.3.0) is above `current` on its first three numbers."""+ def num(v):+ out = []+ for x in re.sub(r"^v", "", str(v)).split(".")[:3]:+ m = re.match(r"\d+", x)+ out.append(int(m.group()) if m else 0)+ return out + [0] * (3 - len(out))+ return num(latest) > num(current)+++def latest_release(now=None):+ """The latest release tag, from the day's cache or one 3 s request to GitHub; "" when unknown."""+ now = time.time() if now is None else now+ try:+ with open(UPDATE_CACHE, encoding="utf-8") as f:+ c = json.load(f)+ if now - float(c.get("t", 0)) < 86400:+ return str(c.get("tag", ""))+ except (OSError, ValueError, TypeError):+ pass+ try:+ req = urllib.request.Request(RELEASES, headers={"Accept": "application/vnd.github+json",+ "User-Agent": "google-maps-scraper-kit"})+ with urllib.request.urlopen(req, timeout=3) as r:+ tag = str(json.load(r).get("tag_name", ""))+ except Exception: # noqa: BLE001 - offline, rate-limited, odd answer: no notice this time+ return ""+ try:+ os.makedirs(HOME, exist_ok=True)+ with open(UPDATE_CACHE, "w", encoding="utf-8") as f:+ json.dump({"t": now, "tag": tag}, f)+ except OSError:+ pass+ return tag+++def update_hint():+ """Print one line when a newer kit is released. Off with SCRAPER_NO_UPDATE_CHECK=1 (environment or .env)."""+ if os.environ.get("SCRAPER_NO_UPDATE_CHECK", "").strip().lower() in ("1", "true", "yes"):+ return+ mine, tag = local_version(), latest_release()+ if mine and re.match(r"^v?\d+\.\d+\.\d+", tag) and newer(tag, mine):+ print(f"↑ Kit {tag} is out (this copy is v{mine}). Update: bash scripts/update.sh · {RELEASE_PAGE}")++JOB_ID = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")diff --git a/scripts/scrape.py b/scripts/scrape.pyindex d8da515..83cefc0 100755--- a/scripts/scrape.py+++ b/scripts/scrape.py@@ -42,7 +42,7 @@ for _stream in (sys.stdout, sys.stderr):except (AttributeError, ValueError): # not a reconfigurable stream (pytest capture, a pipe wrapper)pass-ENV_KEYS = ("SCRAPER_BASE_URL", "SCRAPER_PORT", "SCRAPER_API_KEY")+ENV_KEYS = ("SCRAPER_BASE_URL", "SCRAPER_PORT", "SCRAPER_API_KEY", "SCRAPER_NO_UPDATE_CHECK")def _env_value(v):diff --git a/scripts/test_scrape.py b/scripts/test_scrape.pyindex 351639e..bd38661 100644--- a/scripts/test_scrape.py+++ b/scripts/test_scrape.py@@ -1217,5 +1217,66 @@ class BrowserSites(unittest.TestCase):self.assertEqual(rows[0]["website_fetch_status"], "http")+_espec = importlib.util.spec_from_file_location(+ "enginepy", os.path.join(os.path.dirname(os.path.abspath(__file__)), "engine.py"))+enginepy = importlib.util.module_from_spec(_espec)+_espec.loader.exec_module(enginepy)+++class UpdateNotice(unittest.TestCase):+ """`engine.py status` says when a newer kit is released: once a day, cached, silent on any failure."""++ def test_newer_compares_the_first_three_numbers(self):+ self.assertTrue(enginepy.newer("v2.3.0", "2.2.0"))+ self.assertTrue(enginepy.newer("v3.0.0", "2.9.9"))+ self.assertFalse(enginepy.newer("v2.2.0", "2.2.0"))+ self.assertFalse(enginepy.newer("v2.1.9", "2.2.0"))+ self.assertFalse(enginepy.newer("v2.2.0", "2.2.0-dev.4"))+ self.assertTrue(enginepy.newer("v2.2.1", "2.2.0-dev.4"))+ self.assertFalse(enginepy.newer("garbage", "2.2.0"))++ def test_hint_prints_once_a_day_from_one_request_and_never_raises(self):+ with tempfile.TemporaryDirectory() as d:+ cache = os.path.join(d, "update-check.json")+ answer = io.BytesIO(json.dumps({"tag_name": "v9.9.9"}).encode())+ answer.__enter__, answer.__exit__ = lambda: answer, lambda *a: None+ with mock.patch.object(enginepy, "UPDATE_CACHE", cache), mock.patch.object(enginepy, "HOME", d), \+ mock.patch.object(enginepy, "local_version", return_value="2.2.0"), \+ mock.patch.object(enginepy.urllib.request, "urlopen", return_value=answer) as urlopen, \+ mock.patch.dict(os.environ, {"SCRAPER_NO_UPDATE_CHECK": ""}):+ out = io.StringIO()+ with redirect_stdout(out):+ enginepy.update_hint()+ enginepy.update_hint() # the second call reads the cache+ self.assertEqual(urlopen.call_count, 1)+ self.assertEqual(out.getvalue().count("↑ Kit v9.9.9 is out (this copy is v2.2.0). Update: bash scripts/update.sh"), 2)+ self.assertEqual(json.load(open(cache))["tag"], "v9.9.9")+ # nothing newer: silent+ with mock.patch.object(enginepy, "UPDATE_CACHE", cache), mock.patch.object(enginepy, "local_version", return_value="9.9.9"):+ out = io.StringIO()+ with redirect_stdout(out):+ enginepy.update_hint()+ self.assertEqual(out.getvalue(), "")+ # offline: silent, and nothing cached+ os.remove(cache)+ with mock.patch.object(enginepy, "UPDATE_CACHE", cache), mock.patch.object(enginepy, "HOME", d), \+ mock.patch.object(enginepy, "local_version", return_value="2.2.0"), \+ mock.patch.object(enginepy.urllib.request, "urlopen", side_effect=OSError("offline")):+ out = io.StringIO()+ with redirect_stdout(out):+ enginepy.update_hint()+ self.assertEqual(out.getvalue(), "")+ self.assertFalse(os.path.exists(cache))+ # opted out: no request at all+ with mock.patch.object(enginepy, "UPDATE_CACHE", cache), \+ mock.patch.object(enginepy.urllib.request, "urlopen") as urlopen, \+ mock.patch.dict(os.environ, {"SCRAPER_NO_UPDATE_CHECK": "1"}):+ enginepy.update_hint()+ urlopen.assert_not_called()++ def test_local_version_reads_serve_py(self):+ self.assertRegex(enginepy.local_version(), r"^\d+\.\d+\.\d+")++if __name__ == "__main__":unittest.main(verbosity=2)