Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
fix(security): block DNS rebinding and cross-site posts to the local engine (#29)
Mahan Ai35de8a3a9e2c
diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdindex cbb7c34..4b90411 100644--- a/.claude/skills/google-maps-scraper/SKILL.md+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -29,6 +29,8 @@ docker run -d --name gmaps-scraper --restart unless-stopped -p 127.0.0.1:8080:80-v gmaps_data:/gmapsdata gosom/google-maps-scraper:v1.18.1 -web -data-folder /gmapsdata -c 4```Without `scripts/scrape.py`, run Steps 1 to 3 below with `curl`. They need nothing but Docker.+This bare engine lacks the kit proxy's Host and Origin checks, so a web page in the user's browser+could reach it. Tell the user once, and offer `docker stop gmaps-scraper` when the scrape is done.## Step 1: Create a job (`POST /api/v1/jobs`)**Required fields. The API returns `422` without them:**diff --git a/README.md b/README.mdindex f6fad02..2bd4ff6 100644--- a/README.md+++ b/README.md@@ -151,7 +151,7 @@ scrapes Google Maps business listings only. It does not scrape social platforms| **Batch jobs** | Many keywords, one job, one CSV. The engine lists a place found by several keywords once. || **Proxy rotation** | Built into the engine for large or repeated runs. `socks5`, `socks5h`, `http`, `https`. || **Agent-native** | A skill plus four slash commands. The agent handles create, poll, download, clean and present. |-| **Safe defaults** | API bound to `127.0.0.1`, results git-ignored, rate-limit and data-law guardrails built into the skill. |+| **Safe defaults** | API bound to `127.0.0.1`, and the proxy refuses requests from other websites in your browser (DNS rebinding, cross-site posts). Results git-ignored; website fetches skip private addresses; rate-limit and data-law guardrails built into the skill. |### Batch jobs and the Python CLIdiff --git a/SECURITY.md b/SECURITY.mdindex 373394e..1001303 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -11,6 +11,12 @@ command in `.claude/settings.json` that grants more than local access.The kit binds the engine's API to `127.0.0.1` because the engine has no built-in authentication.Do not expose port 8080 to a network without an authenticating proxy in front of it.+The `ui` proxy in `docker-compose.yml` also answers only `localhost`, `127.0.0.1` and `[::1]` Host+headers and refuses requests whose `Origin` is another site, so a web page in your browser cannot+reach the engine through DNS rebinding or a cross-site post. `scripts/scrape.py` fetches business+websites only when they resolve, redirect and connect to a public address. A way around either+check is in scope.+## Where to reportUse GitHub's privatediff --git a/SETUP.md b/SETUP.mdindex 835e414..848426a 100644--- a/SETUP.md+++ b/SETUP.md@@ -199,7 +199,9 @@ curl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>This command runs the engine alone, with the localhost binding. It skips the `ui` proxy from`docker-compose.yml`, so the web UI shows upstream's links and keeps the engine's job-list bug (StartScraping does nothing, [gosom/google-maps-scraper#337](https://github.com/gosom/google-maps-scraper/issues/337)).-The API and both scripts work the same:+It also skips the proxy's Host and Origin checks, so a web page open in your browser could reach the+engine through DNS rebinding or a cross-site post. Stop it when you are not scraping. The API and both+scripts work the same:```bashdocker run -d --name gmaps-scraper --restart unless-stopped \diff --git a/scripts/scrape.py b/scripts/scrape.pyindex 9f70559..17bccef 100755--- a/scripts/scrape.py+++ b/scripts/scrape.py@@ -24,7 +24,7 @@ Notes:descriptive User-Agent (set below). Do not loop it aggressively."""import argparse, csv, functools, io, ipaddress, json, math, os, re, socket, sys, time-import html as _html, urllib.request, urllib.parse, urllib.error+import html as _html, http.client, urllib.request, urllib.parse, urllib.errorfrom concurrent.futures import ThreadPoolExecutor# The status lines below use ▶ ✓ ℹ •, and every file this script writes can hold accented business@@ -60,7 +60,7 @@ KEY = os.environ.get("SCRAPER_API_KEY", "")# The 8 lead fields: what you use to contact and qualify a lead.# Everything else (geo coordinates, IDs, hours, images, review blobs) is dropped by default.LEAD = ["title", "phone", "emails", "website", "category", "address", "review_rating", "review_count"]-UA = "google-maps-scraper-kit/1.3 (https://github.com/Mahanaicoach/google-maps-scraper-kit)"+UA = "google-maps-scraper-kit/1.4 (https://github.com/Mahanaicoach/google-maps-scraper-kit)"POLL_EVERY = 8 # seconds between status checksPOLL_GRACE = 120 # the engine still has to flush results and flip the status@@ -229,6 +229,12 @@ _SKIP_HANDLE = {"", "home", "pages", "people", "help", "about", "policies", "pol"p", "reel", "reels", "explore", "stories", "tv", "watch", "events","groups", "marketplace", "gaming", "photo", "hashtag", "search"}+def _is_public_ip(ip):+ try:+ return ipaddress.ip_address(ip.split("%")[0]).is_global+ except ValueError:+ return False+@functools.lru_cache(maxsize=1024)def _is_public_host(host):"""True only if every address the host resolves to is public. Websites come from third-party@@ -236,9 +242,39 @@ def _is_public_host(host):localhost:8080, a router admin page, a cloud metadata endpoint)."""try:infos = socket.getaddrinfo(host, None)- return bool(infos) and all(ipaddress.ip_address(i[4][0].split("%")[0]).is_global for i in infos)- except (OSError, UnicodeError, ValueError):+ except (OSError, UnicodeError):return False+ return bool(infos) and all(_is_public_ip(i[4][0]) for i in infos)++def _public_connection(address, *args, **kwargs):+ """socket.create_connection that refuses a private peer. The connect resolves the host again,+ so a DNS answer that changed after _is_public_host (rebinding) is caught here, before a byte+ of the request is sent."""+ sock = socket.create_connection(address, *args, **kwargs)+ peer = sock.getpeername()[0]+ if not _is_public_ip(peer):+ sock.close()+ raise OSError(f"refused non-public address {peer}")+ return sock++class _PublicPeer:+ def __init__(self, *args, **kwargs):+ super().__init__(*args, **kwargs)+ self._create_connection = _public_connection++class _PublicHTTPConnection(_PublicPeer, http.client.HTTPConnection):+ pass++class _PublicHTTPSConnection(_PublicPeer, http.client.HTTPSConnection):+ pass++class _PublicHTTPHandler(urllib.request.HTTPHandler):+ def http_open(self, req):+ return self.do_open(_PublicHTTPConnection, req)++class _PublicHTTPSHandler(urllib.request.HTTPSHandler):+ def https_open(self, req):+ return self.do_open(_PublicHTTPSConnection, req, context=self._context)class _PublicRedirectsOnly(urllib.request.HTTPRedirectHandler):"""A public site may not redirect the fetch to a private address either."""@@ -247,15 +283,18 @@ class _PublicRedirectsOnly(urllib.request.HTTPRedirectHandler):return None # urllib then raises HTTPError, which _fetch_html turns into ""return super().redirect_request(req, fp, code, msg, headers, newurl)-_OPENER = urllib.request.build_opener(_PublicRedirectsOnly)+# Behind a proxy from the environment the socket peer is the proxy, often on a private address,+# so only the resolve-time check applies there.+_HANDLERS = [_PublicRedirectsOnly]+if not urllib.request.getproxies():+ _HANDLERS += [_PublicHTTPHandler, _PublicHTTPSHandler]+_OPENER = urllib.request.build_opener(*_HANDLERS)def _fetch_html(url, timeout=10):if not url:return ""if not url.startswith(("http://", "https://")):url = "http://" + url- # ponytail: the host is resolved again by the actual connect, so DNS rebinding can still slip- # through; pin the checked IP in a custom connection if this ever runs as a shared service.if not _is_public_host(urllib.parse.urlparse(url).hostname or ""):return ""try:diff --git a/scripts/test_scrape.py b/scripts/test_scrape.pyindex b074a6d..3fb94e3 100644--- a/scripts/test_scrape.py+++ b/scripts/test_scrape.py@@ -152,6 +152,28 @@ class PublicHostGuard(unittest.TestCase):req = scrape.urllib.request.Request("http://8.8.8.8/")self.assertIsNone(handler.redirect_request(req, None, 302, "Found", {}, "http://127.0.0.1/admin"))+ @unittest.skipIf(scrape.urllib.request.getproxies(), "a proxy in the environment owns the connect")+ def test_rebinding_caught_at_connect(self):+ # DNS rebinding: the name check saw a public address, the connect lands on 127.0.0.1.+ # No request may reach the local server.+ hits = []+ class Local(BaseHTTPRequestHandler):+ def do_GET(self):+ hits.append(self.path)+ self.send_response(200); self.end_headers(); self.wfile.write(b"secret")+ def log_message(self, *a):+ pass+ srv = ThreadingHTTPServer(("127.0.0.1", 0), Local)+ threading.Thread(target=srv.serve_forever, daemon=True).start()+ try:+ with mock.patch.object(scrape, "_is_public_host", return_value=True):+ self.assertEqual(scrape._fetch_html(f"http://127.0.0.1:{srv.server_port}/"), "")+ with self.assertRaises(OSError):+ scrape._public_connection(("127.0.0.1", srv.server_port))+ finally:+ srv.shutdown(); srv.server_close()+ self.assertEqual(hits, [])+class TechSignals(unittest.TestCase):def test_detects_pixel_platform_viewport(self):diff --git a/ui-branding.conf b/ui-branding.confindex f2f1a70..0ea7155 100644--- a/ui-branding.conf+++ b/ui-branding.conf@@ -8,6 +8,18 @@ server {resolver 127.0.0.11 valid=10s;set $engine http://google-maps-scraper:8080;+ # The engine has no auth, and binding to 127.0.0.1 does not stop the user's own browser. A web+ # page can rebind its domain to 127.0.0.1 and read every job and lead (blocked by the Host+ # check), or post a job from another site without reading the answer (blocked by the Origin+ # check: browsers send Origin on every cross-site POST). Scripts and curl send neither header+ # value, so they pass. Fronting the kit with your own proxy on another hostname? Add it here.+ if ($host !~* ^(localhost|127\.0\.0\.1|\[::1\])$) {+ return 421;+ }+ if ($http_origin !~* ^(https?://(localhost|127\.0\.0\.1|\[::1\])(:\d+)?)?$) {+ return 403;+ }+location / {proxy_pass $engine;proxy_set_header Host $http_host;