Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
fix: a second copy of the kit no longer collides with the first; faster engine-down error; named jobs; doc fixes from a fresh-user run
Mahanaicoach7960e244f0a2
diff --git a/.claude/settings.json b/.claude/settings.jsonindex 14a507a..16fd343 100644--- a/.claude/settings.json+++ b/.claude/settings.json@@ -14,13 +14,13 @@"Bash(curl -s -X POST http://localhost:8080/:*)","Bash(curl -s -X DELETE http://localhost:8080/:*)","Bash(curl -X DELETE http://localhost:8080/:*)",- "Bash(docker restart gmaps-scraper)","Bash(python3 scripts/test_scrape.py)","Bash(./scripts/scrape.sh:*)","Bash(bash scripts/scrape.sh:*)","Bash(python3 scripts/scrape.py:*)","Bash(python scripts/scrape.py:*)",- "Bash(py scripts/scrape.py:*)"+ "Bash(py scripts/scrape.py:*)",+ "Bash(docker restart:*)"],"deny": []}diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdindex 5ad4c40..cadf904 100644--- a/.claude/skills/google-maps-scraper/SKILL.md+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -112,7 +112,7 @@ outreach), `link`, `plus_code`, `cid`, `data_id`, `place_id`, `open_hours`, `pop`user_reviews`, `user_reviews_extended`.`scripts/scrape.py` already returns exactly the 8 lead fields. Use `--full` to keep all columns, or-`--fields "a,b,c"` to choose your own. It **saves a CSV file by default** (`results-<id>.csv`); pass+`--fields "a,b,c"` to choose your own. It **saves a CSV file by default** (`results-<short job id>.csv`); pass`--json` for JSON. If you call the API directly, **strip to the lead fields yourself** beforepresenting. Never dump the full 36-column row at the user.diff --git a/.env.example b/.env.exampleindex 2a07e3c..0c81023 100644--- a/.env.example+++ b/.env.example@@ -12,10 +12,13 @@ SCRAPER_PORT=8080# Set it only if the engine runs somewhere else. When set, the scripts ignore SCRAPER_PORT.# SCRAPER_BASE_URL=http://localhost:8080-# Name of the engine container, for the stuck-job recovery (the script reads its log and restarts it).-# Default: gmaps-scraper, the container_name in docker-compose.yml. Set it only if you renamed the-# container or run a second stack.-# SCRAPER_CONTAINER=gmaps-scraper+# A second copy of the kit on the same machine (your fork next to the original, a test clone):+# Compose names the project after the folder and the containers gmaps-scraper and gmaps-scraper-ui,+# so two copies would fight over them. Set all three, then "docker compose up -d" as usual:+# COMPOSE_PROJECT_NAME=gmaps-kit-2+# SCRAPER_PORT=8081+# SCRAPER_CONTAINER=gmaps-scraper-2+# SCRAPER_CONTAINER also tells the scripts which container to read and restart on a stuck job.# Sent as an X-API-Key header on every request. Default: blank, which is right for local use.# Set it only if you expose the engine beyond localhost behind an auth proxy (e.g. Caddy ordiff --git a/README.md b/README.mdindex c127b9c..1ef0eee 100644--- a/README.md+++ b/README.md@@ -93,7 +93,7 @@ python3 scripts/scrape.py "dentists in Denver CO" --city "Denver, CO" --grid 3### Example output`results-coffee_shops_in_Austin_TX.csv` from the first command above (`scrape.py` names its file-`results-<job id>.csv` unless you pass `--out`). Illustrative rows:+`results-<short job id>.csv` unless you pass `--out`). Illustrative rows:| title | phone | emails | website | category | address | review_rating | review_count ||---|---|---|---|---|---|---|---|@@ -162,7 +162,7 @@ scrapes Google Maps business listings only. It does not scrape social platforms| **Social enrichment** | `--socials` finds Instagram, Facebook, LinkedIn, TikTok, YouTube and X profiles in code (HTTP + regex). Zero AI tokens. || **Who already advertises** | `--tech` spots a Meta Pixel, Google Ads tag, analytics, Bing Ads, LinkedIn Insight and TikTok pixels on each website, plus its platform (WordPress, Shopify, Wix…) and mobile support, then ranks leads by an `intent_score` from 0 to 100. Zero AI tokens. || **Filters** | `--no-website` (businesses that still need a site), `--min-rating`, `--min-reviews`, `--has-phone`, `--has-email`, `--limit`. `--exclude earlier.csv` skips the places you already have, so a weekly re-run returns only new leads. |-| **Email checks** | `--verify-emails` looks up each domain's MX record over DNS (no key), drops dead domains and labels the mail host (`google_workspace`, `microsoft_365`, `other`). |+| **Email checks** | `--verify-emails` looks up each domain's MX record over DNS (no key), drops dead domains and labels the mail host (`google_workspace`, `microsoft_365`, `other`, `none`; blank when the row has no email). || **Auto-geocoding** | Type a city name. Coordinates are resolved for you via OpenStreetMap. Repeat `--city` to run several cities into one file. || **Batch jobs** | Many keywords, one job, one CSV. The engine lists a place found by several keywords once. || **Proxy rotation** | Built into the engine for large or repeated runs: `--proxy URL` (repeatable). `socks5`, `socks5h`, `http`, `https`. |@@ -196,7 +196,7 @@ One keyword per line. One job. One CSV.- **Rate limits are real.** Heavy use without proxies can get your IP temporarily rate-limited byGoogle for minutes to hours. Your Google account is not affected. See [Responsible use](#responsible-use).- **The web UI is the engine's own page.** It downloads the raw 36-column CSV and uses the engine's- slower email option, whose `emails` column keeps junk such as `sentry.wixpress.com` and image+ slower email option, whose `emails` column can keep junk such as `sentry.wixpress.com` and imagenames. For a clean outreach list, use the scripts. The page also loads its scripts,fonts and map tiles from public CDNs and OpenStreetMap, so it needs internet. Scraped data stays onyour machine.@@ -333,6 +333,7 @@ google-maps-scraper-kit/├── scripts/│ ├── scrape.py Python CLI: single, batch, geocoding, socials, CSV│ ├── scrape.sh bash CLI: single query+│ ├── update.sh one-command update of a clone or fork│ └── test_scrape.py offline tests, no network: python3 scripts/test_scrape.py├── docs/ guides and recipes├── examples/ sample job body and batch keyword listdiff --git a/SETUP.md b/SETUP.mdindex d47681c..705444b 100644--- a/SETUP.md+++ b/SETUP.md@@ -14,7 +14,7 @@ Install these once:| **Docker Desktop** | runs the scraper container | https://www.docker.com/products/docker-desktop || **Python 3** | runs `scrape.py` and `scrape.sh` (standard library only, no `pip install`) | https://www.python.org/downloads || **Claude Code** *(optional but recommended)* | lets Claude drive the scraper for you | https://claude.com/claude-code |-| **git** *(optional)* | only if you'll push this to GitHub | https://git-scm.com |+| **git** | clones the kit and runs `scripts/update.sh` later | https://git-scm.com |Check that Docker is running:```bash@@ -56,7 +56,7 @@ docker compose up -dThe first run pulls the `gosom/google-maps-scraper` image once: about 600 MB to download, 2.4 GB on disk. When it finishes, check it:```bash-docker ps | grep gmaps-scraper # should show it "Up"+docker compose ps # both services should show "Up" or "running"curl http://localhost:8080/api/v1/jobs # should print null (no jobs yet) or a JSON list of jobs```@@ -192,7 +192,7 @@ curl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>| `Cannot connect to the Docker daemon` / `docker: command not found` | Docker Desktop is not running (or not installed). Open Docker Desktop, wait until its whale icon stops animating, then run `docker compose up -d` again. || `502 Bad Gateway` right after `docker compose up -d` | The engine is still starting, about 10 seconds. The scripts wait for it; in the browser, reload the page. || `Permission denied` running `./scripts/scrape.sh` | The file lost its execute bit (a ZIP download does that). Run `bash scripts/scrape.sh …` instead. |-| `port is already allocated` / something else already uses 8080 | Set `SCRAPER_PORT=8081` in `.env`, then `docker compose up -d`. Compose publishes the scraper on that host port and `scripts/scrape.sh` and `scripts/scrape.py` read the same file, so nothing else needs editing. The port inside the container stays 8080 (do not change the right-hand side of the mapping). If your `.env` still has a `SCRAPER_BASE_URL` line, delete it or update its port: it overrides `SCRAPER_PORT`. The slash commands and the skill show `localhost:8080` in their examples, so tell Claude the port you picked. |+| `port is already allocated` / something else already uses 8080 | Set `SCRAPER_PORT=8081` in `.env`, then `docker compose up -d`. Compose publishes the scraper on that host port and `scripts/scrape.sh` and `scripts/scrape.py` read the same file, so nothing else needs editing. The port inside the container stays 8080 (do not change the right-hand side of the mapping). If your `.env` still has a `SCRAPER_BASE_URL` line, delete it or update its port: it overrides `SCRAPER_PORT`. The slash commands and the skill show `localhost:8080` in their examples, so tell Claude the port you picked; its pre-approved `curl` rules are pinned to 8080 too, so add `Bash(curl http://localhost:8081/:*)` to `.claude/settings.json` if you want it to stop asking. **A second copy of the kit on the same machine** (a fork next to the original, a test clone): also set `COMPOSE_PROJECT_NAME` and `SCRAPER_CONTAINER` in `.env`, as `.env.example` shows, or `docker compose up -d` adopts the other copy's containers. || `422 missing max time` | Job body needs `max_time` (in **seconds**, e.g. `300`). || `422 missing geo coordinates` | Job body needs `lat` and `lon` as **strings**, e.g. `"30.2672"`. || `422 missing name` / `422 missing lang` | Job body needs `name` (any label) and `lang` (`"en"`). All six required fields: `name`, `keywords`, `lang`, `lat`, `lon`, `max_time`. |@@ -264,3 +264,4 @@ The skill holds the full rules. These are the essentials:faster run without them.- Add proxies for big jobs. The engine rotates them for you.- Never commit result files. They are git-ignored for you.+- Changed something? `python3 scripts/test_scrape.py` runs the offline checks in about a second, no engine needed.diff --git a/docker-compose.yml b/docker-compose.ymlindex e8bd395..a52a56d 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -14,7 +14,9 @@ services:# Only older tags are amd64 and need the 'platform' line below. Default: commented out.image: gosom/google-maps-scraper:v1.18.1# platform: linux/amd64- container_name: gmaps-scraper+ # SCRAPER_CONTAINER in .env renames both containers, for a second copy of the kit on one machine.+ # The scripts read the same variable, so the stuck-job recovery restarts the right engine.+ container_name: ${SCRAPER_CONTAINER:-gmaps-scraper}restart: unless-stopped# -c: parallel workers. The engine's own default resolves to 1 in this image, so a depth 5 job# with emails visits every listing and website one at a time (~8 min). Most of that is business@@ -32,7 +34,7 @@ services:# Serves the engine's UI and API with the kit's own GitHub and sponsor links (ui-branding.conf).ui:image: nginx:1.27-alpine- container_name: gmaps-scraper-ui+ container_name: ${SCRAPER_CONTAINER:-gmaps-scraper}-uirestart: unless-stoppeddepends_on:- google-maps-scraperdiff --git a/docs/google-maps-email-scraper.md b/docs/google-maps-email-scraper.mdindex d338c2b..c581499 100644--- a/docs/google-maps-email-scraper.md+++ b/docs/google-maps-email-scraper.md@@ -32,7 +32,7 @@ python3 scripts/scrape.py "dentists in Denver CO" --city "Denver, CO" --no-deep-python3 scripts/scrape.py "dentists in Denver CO" --city "Denver, CO" --no-email```-The CSV lands in `results-<id>.csv` unless you pass `--out`.+The CSV lands in `results-<short job id>.csv` unless you pass `--out`.## Example outputdiff --git a/docs/google-maps-scraper-python.md b/docs/google-maps-scraper-python.mdindex 7b58dc0..25eaba1 100644--- a/docs/google-maps-scraper-python.md+++ b/docs/google-maps-scraper-python.md@@ -28,7 +28,7 @@ On Windows, run the same commands with `py` or `python3`. Python is only the cli| `--no-deep-emails` | Checks homepages only for emails, not contact and about pages. || `--grid N` | Splits the `--city` area into N x N map tiles, one job each, deduped. Needs `--city`. Max 10. || `--max-time` | Job time limit in seconds. Default `600`. |-| `--out` | Output path. Default `results-<id>.csv`. A `.json` extension writes JSON. |+| `--out` | Output path. Default `results-<short job id>.csv`. A `.json` extension writes JSON. || `--json` | Writes JSON instead of CSV. || `--full` | Keeps all 36 raw columns instead of the 8 lead fields. || `--fields` | Comma-separated columns to keep, overriding the lead fields. |@@ -43,7 +43,7 @@ On Windows, run the same commands with `py` or `python3`. Python is only the cli| `--has-phone`, `--has-email` | Keep only rows with a phone, or rows where an email was found. || `--exclude FILE` | Drop places already in an earlier results file (`.csv` or `.json`), repeatable. Matches on `place_id`, `cid`, link, or name plus address, so the lean 8-column CSV works. || `--limit N` | Stop at N rows after filtering. |-| `--verify-emails` | Checks each email domain's MX record over DNS (dns.google, no key). Drops addresses on domains that do not exist and adds `email_host`: `google_workspace`, `microsoft_365`, `other` or `none`. |+| `--verify-emails` | Checks each email domain's MX record over DNS (dns.google, no key). Drops addresses on domains that do not exist and adds `email_host`: `google_workspace`, `microsoft_365`, `other` or `none` (blank when the row has no email). |The script creates the job, polls until it finishes, downloads the CSV and keeps the 8 lead fields. It prints a short preview and saves the full results to a file.diff --git a/scripts/scrape.py b/scripts/scrape.pyindex f45359a..7e19cde 100755--- a/scripts/scrape.py+++ b/scripts/scrape.py@@ -778,7 +778,8 @@ def scrape_place(a, body, city, keywords):return None, [], 0, 0lat, lon = coordsprint(f" → {lat}, {lon}")- body = dict(body, lat=str(lat), lon=str(lon))+ # The job name is what the web UI lists, so make it say what the job was.+ body = dict(body, lat=str(lat), lon=str(lon), name=(f"{keywords[0]} @ {city}" if city else keywords[0])[:80])for k in keywords:print(f" • {k}")if not cells:@@ -907,11 +908,14 @@ def main():print(" Google (clears in minutes–hours; returns empty/failed jobs meanwhile). For big or")print(" repeated runs, add proxies. Proceeding…\n")- # Health check. Right after `docker compose up -d` the proxy answers 502 for a few seconds while- # the engine starts, and the README's next line runs at once, so wait up to a minute for it.+ # Health check. Right after `docker compose up -d` the proxy answers 502 while the engine starts+ # (up to a minute), and the README's next line runs at once, so wait for that. Nothing listening at+ # all (connection refused) means the stack is down: say so after 10 s, not after a minute of dots.+ refused, err = 0, Nonefor i in range(30):try:req("GET", "/api/v1/jobs")+ err = Nonebreakexcept urllib.error.HTTPError as e:if e.code < 500: # the engine is up but refuses us (wrong API key, wrong URL): say so at once@@ -919,15 +923,18 @@ def main():err = eexcept Exception as e:err = e+ refused += isinstance(getattr(e, "reason", e), ConnectionRefusedError)+ if refused >= 5:+ breakprint("\r waiting for the engine to start" + "." * (i + 1), end="", flush=True)time.sleep(2)- else:+ if err is not None:sys.exit(f"\n✗ Scraper not reachable at {BASE}. Is Docker Desktop open? Start the engine with "f"'docker compose up -d' in the kit folder, wait 10 seconds, then try again.\n ({err})")if i:print()- body = {"name": "scrape-py", "keywords": keywords, "lang": a.lang, "zoom": 15,+ body = {"keywords": keywords, "lang": a.lang, "zoom": 15,"fast_mode": False, "radius": 10000,# The engine's own email pass visits every website in a browser, one per worker: it took# most of each job and found fewer emails than the crawl below (OpenLeadFinder's@@ -965,7 +972,7 @@ def main():if skipped:print(f"⚠ Incomplete: {skipped} of {ncells} tiles failed, so parts of the area are missing. Rerun later.")- if a.email and "emails" in fields: # on the raw rows, so --fields without website still works+ if a.email and "emails" in fields and not a.no_website: # no website, nothing to crawlpages = "homepage, then contact and about pages" if a.deep_emails else "homepage only"print(f"▶ Emails: checking each business website ({pages}, {SITE_WORKERS} at a time)…")enrich_emails(rows, deep=a.deep_emails)@@ -984,8 +991,6 @@ def main():# Enrichment runs on the raw rows, which always carry the website, whatever --fields keeps.if a.socials:print("▶ Finding socials (Instagram / Facebook / LinkedIn / TikTok / YouTube / X) on each website…")- print(" ℹ Cost: this runs in CODE = 0 LLM tokens. It only adds ~40–50 tokens per business")- print(" IF you load the results into an AI's context (negligible if you keep the file on disk).")enrich_sites(rows, _find_socials, SOCIAL_FIELDS, "socials")fields = fields + SOCIAL_FIELDSfound = sum(1 for r in rows if any(r.get(k) for k in SOCIAL_FIELDS))diff --git a/scripts/test_scrape.py b/scripts/test_scrape.pyindex 7804d43..481634c 100644--- a/scripts/test_scrape.py+++ b/scripts/test_scrape.py@@ -368,6 +368,7 @@ class EmailCrawl(unittest.TestCase):mock.patch.object(sys, "argv", argv), redirect_stdout(io.StringIO()):scrape.main()self.assertIs(sent[0]["email"], False)+ self.assertEqual(sent[0]["name"], "dentists in Austin TX") # the web UI lists this nameclass IsRealEmail(unittest.TestCase):@@ -703,6 +704,12 @@ class ArgumentBounds(unittest.TestCase):scrape.main()return str(cm.exception)+ def test_nothing_listening_fails_in_ten_seconds_not_sixty(self):+ refused = scrape.urllib.error.URLError(ConnectionRefusedError(61, "Connection refused"))+ with mock.patch.object(scrape, "req", side_effect=refused), mock.patch.object(scrape.time, "sleep") as slept:+ self.assertIn("Docker Desktop", self.run_main())+ self.assertEqual(slept.call_count, 4) # five tries, two seconds apart+def test_has_email_needs_the_crawl(self):self.assertIn("--has-email", self.run_main("--has-email", "--no-email"))