Skip to content

Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.

feat: tell users when a newer kit is released (web UI line and engine.py status)

Mahan1e389e7308ac
diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.md
index 488f778..29f31e7 100644
--- a/.claude/skills/google-maps-scraper/SKILL.md
+++ b/.claude/skills/google-maps-scraper/SKILL.md
@@ -32,7 +32,8 @@ python3 scripts/engine.py status
It prints `✓ Running on http://localhost:<port>` only when the kit's own engine answers (it checks the
engine's `X-Engine` header), so another program on that port cannot fool it. When the engine is not
running it prints `✗ Not running` and exits 1; that is expected. It also names the Chrome, Chromium,
-Edge or Brave the engine will drive. Only when it names one, start the engine with
+Edge or Brave the engine will drive. If it prints a `↑ Kit vX.Y.Z is out` line, tell the user once and carry
+on; run `bash scripts/update.sh` only when the user asks. Only when it names one, start the engine with
`python3 scripts/engine.py start`, or run `docker compose up -d` if the user runs it in Docker, then run
`status` again. If it finds no browser, do not start it: ask the user to install Google Chrome, or to
use Docker. Setup help: `SETUP.md` in the kit folder.
diff --git a/.env.example b/.env.example
index a7f8854..37cc967 100644
--- a/.env.example
+++ b/.env.example
@@ -32,3 +32,9 @@ SCRAPER_API_KEY=
# on macOS a second Chrome icon shows in the Dock and the window may take focus when a job starts.
# Not used in Docker, where Chrome is always headless.
# SCRAPER_HEADLESS=0
+
+# Update notice. Default: on. `python3 scripts/engine.py status` and the web UI ask GitHub's public
+# releases endpoint once a day whether a newer kit is out (nothing about you in the request) and say
+# so; nothing updates itself. Set 1 to turn the check off (the web UI's check is in your browser and
+# is not affected by this file).
+# SCRAPER_NO_UPDATE_CHECK=1
diff --git a/CLAUDE.md b/CLAUDE.md
index 4b68749..51e78ef 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -54,7 +54,8 @@ individuals, spam). The skill has the full guidance.
## Setup
Check the engine first with `python3 scripts/engine.py status`. It prints `✓ Running on http://localhost:<port>`
only when the kit's own engine answers (it checks the engine's `X-Engine` header, so another program on
-the port cannot fool it), and that URL is the base to use. It also names the browser the engine will use.
+the port cannot fool it), and that URL is the base to use. It also names the browser the engine will use, and prints `↑ Kit vX.Y.Z is out` when a newer
+release exists: mention it once; never run `scripts/update.sh` unasked.
If the engine is not running, run `python3 scripts/engine.py start` (or `docker compose up -d` when the
user runs Docker), then run `status` again. If `status` finds no browser, `start` would download Chrome for
Testing (about 100 MB, pinned by SHA-256): ask the user first, or suggest installing Google Chrome or
diff --git a/README.md b/README.md
index 843526e..52879d1 100644
--- a/README.md
+++ b/README.md
@@ -200,7 +200,7 @@ scrapes Google Maps business listings only. It does not scrape social platforms
| **Batch jobs** | Many keywords, one job, one CSV. The engine lists a place found by several keywords once. |
| **Proxies** | For large or repeated runs: `--proxy URL` with `http`, `https`, `socks5` or `socks5h`, and `user:pass@` on `http` and `https`. Repeat it and the keywords take turns; a keyword Google refuses is retried through the next proxy. |
| **Agent-native** | A skill plus four slash commands. The agent runs the script and presents the lead fields; it never hand-builds API calls. |
-| **Safe defaults** | API bound to `127.0.0.1`, and the engine refuses requests from other websites in your browser (DNS rebinding, cross-site posts); an optional API key. No telemetry: the engine talks to Google Maps, and to the businesses' websites and `dns.google` only for jobs that ask. Proxy passwords show as `***`. A cell that a spreadsheet would run as a formula gets a leading `'` in the engine's CSV download and in the script's file. Results git-ignored; website fetches skip private addresses; rate-limit and data-law guardrails built into the skill. |
+| **Safe defaults** | API bound to `127.0.0.1`, and the engine refuses requests from other websites in your browser (DNS rebinding, cross-site posts); an optional API key. No telemetry: the engine talks to Google Maps, and to the businesses' websites and `dns.google` only for jobs that ask. The web UI and `engine.py status` ask GitHub once a day whether a newer release is out, with nothing about you in the request (`SCRAPER_NO_UPDATE_CHECK=1` in `.env` turns it off); nothing updates itself. Proxy passwords show as `***`. A cell that a spreadsheet would run as a formula gets a leading `'` in the engine's CSV download and in the script's file. Results git-ignored; website fetches skip private addresses; rate-limit and data-law guardrails built into the skill. |
### Measured, not estimated
@@ -451,7 +451,9 @@ google-maps-scraper-kit/
## Updating
-Pull the latest fixes with one command. It works in a clone of this repo or of your fork, and it
+The web UI shows a line under its title when a newer release is out, and `python3 scripts/engine.py status`
+prints one (both check GitHub's releases once a day; `SCRAPER_NO_UPDATE_CHECK=1` in `.env` turns it off).
+Nothing updates by itself. Pull the latest fixes with one command. It works in a clone of this repo or of your fork, and it
keeps your own commits:
```bash
diff --git a/SETUP.md b/SETUP.md
index 1c11a42..44be288 100644
--- a/SETUP.md
+++ b/SETUP.md
@@ -101,6 +101,10 @@ separate job lists, so jobs started in one do not show in the other.
python3 scripts/engine.py status # should print ✓ Running on http://localhost:8080
```
+When a newer kit is released, `status` adds a line (`↑ Kit v2.3.0 is out ... bash scripts/update.sh`) and the
+web UI shows the same under its title. Both ask GitHub's releases endpoint once a day, with nothing about
+you in the request; `SCRAPER_NO_UPDATE_CHECK=1` in `.env` turns the check off. Nothing updates itself.
+
It prints `✓ Running on http://localhost:<port>` only when the kit's own engine answers (it checks the
engine's `X-Engine` header, so another program on the port cannot fool it). The engine also serves a
web UI and API docs in the browser:
diff --git a/engine/serve.py b/engine/serve.py
index 95fa3e6..70448f5 100644
--- a/engine/serve.py
+++ b/engine/serve.py
@@ -34,6 +34,10 @@ REPO = "https://github.com/Mahanaicoach/google-maps-scraper-kit"
# The web UI's sponsor box: the browser reads this static file from GitHub (cached a day, no
# tracking), so the sponsors it lists change without anyone updating their copy of the kit.
SPONSORS = "https://raw.githubusercontent.com/Mahanaicoach/google-maps-scraper-kit/master/assets/sponsors/sponsors.json"
+# The web UI's update notice: the browser asks GitHub for the kit's latest release once a day (a
+# public JSON endpoint, nothing about the user in the request) and shows a line when it is newer
+# than this copy. Nothing updates itself: the line says to run scripts/update.sh.
+RELEASES = "https://api.github.com/repos/Mahanaicoach/google-maps-scraper-kit/releases/latest"
# The web UI's What and Where boxes offer only these, so nobody searches for a business type or a
# place that does not exist: presets/business-types.json (hand-picked; add a niche there) and
# presets/cities.tsv.gz (GeoNames cities1000, CC BY 4.0: name, region, country code, lat, lon, biggest first).
@@ -259,6 +263,8 @@ a {{ color: inherit; text-underline-offset: 3px; }} a:hover {{ color: var(--go);
h1 {{ margin: 0; font: 600 1.6rem/1.2 var(--serif); letter-spacing: -.01em; }}
.top nav {{ display: flex; gap: 16px; font-size: .9rem; color: var(--soft); }}
.mono {{ font-family: var(--mono); font-size: .85em; }} .soft {{ color: var(--soft); font-size: .875rem; }}
+.update {{ margin-top: 14px; padding: 10px 14px; border: 1px solid var(--rule); border-radius: 10px; background: var(--sheet); font-size: .9rem; }}
+.update code {{ font-size: .85rem; }}
.stars {{ display: flex; flex-wrap: wrap; align-items: center; gap: 8px 14px; padding-top: 16px; color: var(--soft); font-size: .9rem; }}
.gh {{ display: inline-flex; align-items: center; gap: 8px; min-height: 40px; padding: 8px 14px; border-radius: 8px;
color: var(--paper); background: var(--ink); font-size: .9rem; font-weight: 600; text-decoration: none; transition: filter .15s; }}
@@ -368,6 +374,9 @@ progress {{ display: block; width: 100%; max-width: 200px; height: 6px; margin-t
</style></head><body><main>
<header class="top"><h1>Google Maps Scraper Kit</h1>
<nav><span class="mono">v{version}</span><a href="/api/docs">API docs</a></nav></header>
+<div class="update" id="update" role="status" hidden><b>Kit v<span id="up-v"></span> is out</b> (this copy is v{version}).
+ <a id="up-link" href="{repo}/releases/latest" target="_blank" rel="noopener">What changed</a> &middot;
+ update with <code>bash scripts/update.sh</code>, or <code>docker compose up -d --build</code> in Docker.</div>
<div class="stars"><a class="gh" href="{repo}" target="_blank" rel="noopener"><span class="star"><svg width="16" height="16" viewBox="0 0 24 24" aria-hidden="true"><path fill="currentColor" d="M12 2.5l2.9 6 6.6.8-4.9 4.6 1.3 6.6L12 17.2l-5.9 3.3 1.3-6.6L2.5 9.3l6.6-.8z"/></svg></span>Star on GitHub</a>
<span>Useful? A star helps other people find this free kit.</span></div>
<aside class="sponsors" id="sponsors" aria-label="Sponsors"><button type="button" class="sp-close" id="sp-close"
@@ -616,6 +625,27 @@ $('f').addEventListener('submit', async ev => {{
finally {{ go.disabled = false; go.textContent = 'Start scraping'; load(); }}
}});
load(); setInterval(load, 3000);
+(function () {{ // update notice: GitHub's latest release once a day, shown only when it is newer than this copy
+ const key = 'gmskit-update', day = 86400000, mine = '{version}';
+ const num = v => (String(v).replace(/^v/, '').split('.').slice(0, 3).map(x => parseInt(x, 10) || 0));
+ const newer = (a, b) => {{ const x = num(a), y = num(b); for (let i = 0; i < 3; i++) if (x[i] !== y[i]) return x[i] > y[i]; return false; }};
+ function show(tag) {{
+ if (!/^v?\d+\.\d+\.\d+/.test(tag || '') || !newer(tag, mine)) return;
+ $('up-v').textContent = String(tag).replace(/^v/, '').slice(0, 20);
+ $('up-link').href = '{repo}/releases/tag/' + encodeURIComponent(tag);
+ $('update').hidden = false;
+ }}
+ try {{
+ const c = JSON.parse(localStorage.getItem(key) || 'null');
+ if (c && Date.now() - c.t < day) return show(c.tag);
+ }} catch (e) {{}}
+ fetch('{releases}', {{cache: 'no-store', credentials: 'omit', headers: {{Accept: 'application/vnd.github+json'}}}})
+ .then(r => r.ok ? r.json() : null).then(d => {{
+ if (!d || typeof d.tag_name !== 'string') return;
+ show(d.tag_name);
+ try {{ localStorage.setItem(key, JSON.stringify({{t: Date.now(), tag: d.tag_name}})); }} catch (e) {{}}
+ }}).catch(() => {{}});
+}})();
(function () {{ // sponsor tiles: only the entries Mahan marks "ui": true in sponsors.json, as text, https links only
const box = $('sponsors'), grid = $('sp-grid'), key = 'gmskit-sponsors', day = 86400000;
$('sp-close').addEventListener('click', () => {{ box.hidden = true; }}); // as gosom: hidden until the page reloads
@@ -774,10 +804,10 @@ class Handler(BaseHTTPRequestHandler):
if path == "/":
nonce = secrets.token_urlsafe(16)
csp = (f"default-src 'none'; script-src 'nonce-{nonce}'; style-src 'nonce-{nonce}'; "
- "connect-src 'self' https://raw.githubusercontent.com; img-src 'self' data:; "
+ "connect-src 'self' https://raw.githubusercontent.com https://api.github.com; img-src 'self' data:; "
"form-action 'self'; base-uri 'none'; "
"frame-ancestors 'none'")
- page = PAGE.format(nonce=nonce, version=VERSION, repo=REPO, sponsors=SPONSORS, types=TYPES_JS,
+ page = PAGE.format(nonce=nonce, version=VERSION, repo=REPO, sponsors=SPONSORS, releases=RELEASES, types=TYPES_JS,
categories=CATEGORIES).encode()
return self.reply(200, page, "text/html; charset=utf-8", [("Content-Security-Policy", csp)])
self.fail(404, "not found")
diff --git a/engine/test_engine.py b/engine/test_engine.py
index 3d11595..c12c001 100644
--- a/engine/test_engine.py
+++ b/engine/test_engine.py
@@ -505,8 +505,10 @@ class Api(Server):
self.assertIn(f'<script nonce="{nonce}">', body.decode())
self.assertIn("github.com/Mahanaicoach/google-maps-scraper-kit", body.decode())
self.assertIn("frame-ancestors 'none'", csp)
- self.assertIn("connect-src 'self' https://raw.githubusercontent.com;", csp) # sponsors.json, nothing else
+ # sponsors.json from raw.githubusercontent.com and the latest release from api.github.com, nothing else
+ self.assertIn("connect-src 'self' https://raw.githubusercontent.com https://api.github.com;", csp)
self.assertIn("assets/sponsors/sponsors.json", body.decode())
+ self.assertIn("/releases/latest", body.decode())
self.assertNotRegex(body.decode(), r"\{[a-z_]+\}") # every placeholder filled
code, body, _ = self.call("GET", "/api/openapi.json")
self.assertEqual(json.loads(body)["paths"]["/api/v1/jobs"]["post"]["requestBody"]["content"]["application/json"]
@@ -1286,6 +1288,11 @@ class ApiP3(Server):
for rule in ("s.ui === true", "s.url.startsWith('https://')", ".slice(0, 4)", "noopener noreferrer sponsored"):
self.assertIn(rule, ui)
self.assertNotIn(".innerHTML", ui) # sponsor text is set as text, never parsed as HTML
+ # update notice: hidden by default, filled from GitHub's latest release once a day, shown only when newer
+ self.assertIn('<div class="update" id="update" role="status" hidden>', ui)
+ for rule in ("gmskit-update", "api.github.com/repos/Mahanaicoach/google-maps-scraper-kit/releases/latest",
+ "newer(tag, mine)", "bash scripts/update.sh"):
+ self.assertIn(rule, ui)
script = re.search(r"<script nonce=[^>]+>(.*)</script>", ui, re.S).group(1)
self.assertNotIn("'\n'", script) # PAGE must say \\n: a bare newline in a JS string breaks the page
diff --git a/scripts/engine.py b/scripts/engine.py
index 416a3e1..ee6d49a 100644
--- a/scripts/engine.py
+++ b/scripts/engine.py
@@ -35,6 +35,11 @@ LOG_KEEP = 2_000_000 # bytes of engine.log kept across starts
MARK = "gmskit-engine/"
_NO_PROXY = urllib.request.build_opener(urllib.request.ProxyHandler({})) # HTTP_PROXY must not catch localhost
PY = "py" if os.name == "nt" else "python3" # what the user types; the README says `py` on Windows
+# `status` tells the user when a newer kit is released: one request a day to GitHub's public releases
+# endpoint (nothing about the user in it), cached here. It only prints; scripts/update.sh updates.
+RELEASES = "https://api.github.com/repos/Mahanaicoach/google-maps-scraper-kit/releases/latest"
+RELEASE_PAGE = "https://github.com/Mahanaicoach/google-maps-scraper-kit/releases/latest"
+UPDATE_CACHE = os.path.join(HOME, "update-check.json")
def public_port():
@@ -320,6 +325,7 @@ def status():
if seen == "ours":
st = read_state() or {}
print(f"✓ Running on http://localhost:{port}" + (f" (browser: {st['chrome']})" if st.get("chrome") else ""))
+ update_hint()
elif seen == "other":
print(f"ℹ Port {port} answers, but not as this kit's engine (the kit's older Docker engine, probably).")
sys.exit(1)
@@ -327,9 +333,67 @@ def status():
found = browser.find_chrome()
print(f"✗ Not running. Start it with: {PY} scripts/engine.py start\n Browser: " + (found or
"none found, so start would download Chrome for Testing (about 100 MB, pinned by SHA-256)"))
+ update_hint()
sys.exit(1)
+def local_version():
+ """This copy's engine version, read from engine/serve.py without importing it."""
+ path = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "engine", "serve.py")
+ try:
+ with open(path, encoding="utf-8") as f:
+ m = re.search(r'^VERSION = "([^"]+)"', f.read(), re.M)
+ return m.group(1) if m else ""
+ except OSError:
+ return ""
+
+
+def newer(latest, current):
+ """True when `latest` (a tag like v2.3.0) is above `current` on its first three numbers."""
+ def num(v):
+ out = []
+ for x in re.sub(r"^v", "", str(v)).split(".")[:3]:
+ m = re.match(r"\d+", x)
+ out.append(int(m.group()) if m else 0)
+ return out + [0] * (3 - len(out))
+ return num(latest) > num(current)
+
+
+def latest_release(now=None):
+ """The latest release tag, from the day's cache or one 3 s request to GitHub; "" when unknown."""
+ now = time.time() if now is None else now
+ try:
+ with open(UPDATE_CACHE, encoding="utf-8") as f:
+ c = json.load(f)
+ if now - float(c.get("t", 0)) < 86400:
+ return str(c.get("tag", ""))
+ except (OSError, ValueError, TypeError):
+ pass
+ try:
+ req = urllib.request.Request(RELEASES, headers={"Accept": "application/vnd.github+json",
+ "User-Agent": "google-maps-scraper-kit"})
+ with urllib.request.urlopen(req, timeout=3) as r:
+ tag = str(json.load(r).get("tag_name", ""))
+ except Exception: # noqa: BLE001 - offline, rate-limited, odd answer: no notice this time
+ return ""
+ try:
+ os.makedirs(HOME, exist_ok=True)
+ with open(UPDATE_CACHE, "w", encoding="utf-8") as f:
+ json.dump({"t": now, "tag": tag}, f)
+ except OSError:
+ pass
+ return tag
+
+
+def update_hint():
+ """Print one line when a newer kit is released. Off with SCRAPER_NO_UPDATE_CHECK=1 (environment or .env)."""
+ if os.environ.get("SCRAPER_NO_UPDATE_CHECK", "").strip().lower() in ("1", "true", "yes"):
+ return
+ mine, tag = local_version(), latest_release()
+ if mine and re.match(r"^v?\d+\.\d+\.\d+", tag) and newer(tag, mine):
+ print(f"↑ Kit {tag} is out (this copy is v{mine}). Update: bash scripts/update.sh · {RELEASE_PAGE}")
+
+
JOB_ID = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
diff --git a/scripts/scrape.py b/scripts/scrape.py
index d8da515..83cefc0 100755
--- a/scripts/scrape.py
+++ b/scripts/scrape.py
@@ -42,7 +42,7 @@ for _stream in (sys.stdout, sys.stderr):
except (AttributeError, ValueError): # not a reconfigurable stream (pytest capture, a pipe wrapper)
pass
-ENV_KEYS = ("SCRAPER_BASE_URL", "SCRAPER_PORT", "SCRAPER_API_KEY")
+ENV_KEYS = ("SCRAPER_BASE_URL", "SCRAPER_PORT", "SCRAPER_API_KEY", "SCRAPER_NO_UPDATE_CHECK")
def _env_value(v):
diff --git a/scripts/test_scrape.py b/scripts/test_scrape.py
index 351639e..bd38661 100644
--- a/scripts/test_scrape.py
+++ b/scripts/test_scrape.py
@@ -1217,5 +1217,66 @@ class BrowserSites(unittest.TestCase):
self.assertEqual(rows[0]["website_fetch_status"], "http")
+_espec = importlib.util.spec_from_file_location(
+ "enginepy", os.path.join(os.path.dirname(os.path.abspath(__file__)), "engine.py"))
+enginepy = importlib.util.module_from_spec(_espec)
+_espec.loader.exec_module(enginepy)
+
+
+class UpdateNotice(unittest.TestCase):
+ """`engine.py status` says when a newer kit is released: once a day, cached, silent on any failure."""
+
+ def test_newer_compares_the_first_three_numbers(self):
+ self.assertTrue(enginepy.newer("v2.3.0", "2.2.0"))
+ self.assertTrue(enginepy.newer("v3.0.0", "2.9.9"))
+ self.assertFalse(enginepy.newer("v2.2.0", "2.2.0"))
+ self.assertFalse(enginepy.newer("v2.1.9", "2.2.0"))
+ self.assertFalse(enginepy.newer("v2.2.0", "2.2.0-dev.4"))
+ self.assertTrue(enginepy.newer("v2.2.1", "2.2.0-dev.4"))
+ self.assertFalse(enginepy.newer("garbage", "2.2.0"))
+
+ def test_hint_prints_once_a_day_from_one_request_and_never_raises(self):
+ with tempfile.TemporaryDirectory() as d:
+ cache = os.path.join(d, "update-check.json")
+ answer = io.BytesIO(json.dumps({"tag_name": "v9.9.9"}).encode())
+ answer.__enter__, answer.__exit__ = lambda: answer, lambda *a: None
+ with mock.patch.object(enginepy, "UPDATE_CACHE", cache), mock.patch.object(enginepy, "HOME", d), \
+ mock.patch.object(enginepy, "local_version", return_value="2.2.0"), \
+ mock.patch.object(enginepy.urllib.request, "urlopen", return_value=answer) as urlopen, \
+ mock.patch.dict(os.environ, {"SCRAPER_NO_UPDATE_CHECK": ""}):
+ out = io.StringIO()
+ with redirect_stdout(out):
+ enginepy.update_hint()
+ enginepy.update_hint() # the second call reads the cache
+ self.assertEqual(urlopen.call_count, 1)
+ self.assertEqual(out.getvalue().count("↑ Kit v9.9.9 is out (this copy is v2.2.0). Update: bash scripts/update.sh"), 2)
+ self.assertEqual(json.load(open(cache))["tag"], "v9.9.9")
+ # nothing newer: silent
+ with mock.patch.object(enginepy, "UPDATE_CACHE", cache), mock.patch.object(enginepy, "local_version", return_value="9.9.9"):
+ out = io.StringIO()
+ with redirect_stdout(out):
+ enginepy.update_hint()
+ self.assertEqual(out.getvalue(), "")
+ # offline: silent, and nothing cached
+ os.remove(cache)
+ with mock.patch.object(enginepy, "UPDATE_CACHE", cache), mock.patch.object(enginepy, "HOME", d), \
+ mock.patch.object(enginepy, "local_version", return_value="2.2.0"), \
+ mock.patch.object(enginepy.urllib.request, "urlopen", side_effect=OSError("offline")):
+ out = io.StringIO()
+ with redirect_stdout(out):
+ enginepy.update_hint()
+ self.assertEqual(out.getvalue(), "")
+ self.assertFalse(os.path.exists(cache))
+ # opted out: no request at all
+ with mock.patch.object(enginepy, "UPDATE_CACHE", cache), \
+ mock.patch.object(enginepy.urllib.request, "urlopen") as urlopen, \
+ mock.patch.dict(os.environ, {"SCRAPER_NO_UPDATE_CHECK": "1"}):
+ enginepy.update_hint()
+ urlopen.assert_not_called()
+
+ def test_local_version_reads_serve_py(self):
+ self.assertRegex(enginepy.local_version(), r"^\d+\.\d+\.\d+")
+
+
if __name__ == "__main__":
unittest.main(verbosity=2)