Skip to content

Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.

fix: a second copy of the kit no longer collides with the first; faster engine-down error; named jobs; doc fixes from a fresh-user run

Mahanaicoach7960e244f0a2
diff --git a/.claude/settings.json b/.claude/settings.json
index 14a507a..16fd343 100644
--- a/.claude/settings.json
+++ b/.claude/settings.json
@@ -14,13 +14,13 @@
"Bash(curl -s -X POST http://localhost:8080/:*)",
"Bash(curl -s -X DELETE http://localhost:8080/:*)",
"Bash(curl -X DELETE http://localhost:8080/:*)",
- "Bash(docker restart gmaps-scraper)",
"Bash(python3 scripts/test_scrape.py)",
"Bash(./scripts/scrape.sh:*)",
"Bash(bash scripts/scrape.sh:*)",
"Bash(python3 scripts/scrape.py:*)",
"Bash(python scripts/scrape.py:*)",
- "Bash(py scripts/scrape.py:*)"
+ "Bash(py scripts/scrape.py:*)",
+ "Bash(docker restart:*)"
],
"deny": []
}
diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.md
index 5ad4c40..cadf904 100644
--- a/.claude/skills/google-maps-scraper/SKILL.md
+++ b/.claude/skills/google-maps-scraper/SKILL.md
@@ -112,7 +112,7 @@ outreach), `link`, `plus_code`, `cid`, `data_id`, `place_id`, `open_hours`, `pop
`user_reviews`, `user_reviews_extended`.
`scripts/scrape.py` already returns exactly the 8 lead fields. Use `--full` to keep all columns, or
-`--fields "a,b,c"` to choose your own. It **saves a CSV file by default** (`results-<id>.csv`); pass
+`--fields "a,b,c"` to choose your own. It **saves a CSV file by default** (`results-<short job id>.csv`); pass
`--json` for JSON. If you call the API directly, **strip to the lead fields yourself** before
presenting. Never dump the full 36-column row at the user.
diff --git a/.env.example b/.env.example
index 2a07e3c..0c81023 100644
--- a/.env.example
+++ b/.env.example
@@ -12,10 +12,13 @@ SCRAPER_PORT=8080
# Set it only if the engine runs somewhere else. When set, the scripts ignore SCRAPER_PORT.
# SCRAPER_BASE_URL=http://localhost:8080
-# Name of the engine container, for the stuck-job recovery (the script reads its log and restarts it).
-# Default: gmaps-scraper, the container_name in docker-compose.yml. Set it only if you renamed the
-# container or run a second stack.
-# SCRAPER_CONTAINER=gmaps-scraper
+# A second copy of the kit on the same machine (your fork next to the original, a test clone):
+# Compose names the project after the folder and the containers gmaps-scraper and gmaps-scraper-ui,
+# so two copies would fight over them. Set all three, then "docker compose up -d" as usual:
+# COMPOSE_PROJECT_NAME=gmaps-kit-2
+# SCRAPER_PORT=8081
+# SCRAPER_CONTAINER=gmaps-scraper-2
+# SCRAPER_CONTAINER also tells the scripts which container to read and restart on a stuck job.
# Sent as an X-API-Key header on every request. Default: blank, which is right for local use.
# Set it only if you expose the engine beyond localhost behind an auth proxy (e.g. Caddy or
diff --git a/README.md b/README.md
index c127b9c..1ef0eee 100644
--- a/README.md
+++ b/README.md
@@ -93,7 +93,7 @@ python3 scripts/scrape.py "dentists in Denver CO" --city "Denver, CO" --grid 3
### Example output
`results-coffee_shops_in_Austin_TX.csv` from the first command above (`scrape.py` names its file
-`results-<job id>.csv` unless you pass `--out`). Illustrative rows:
+`results-<short job id>.csv` unless you pass `--out`). Illustrative rows:
| title | phone | emails | website | category | address | review_rating | review_count |
|---|---|---|---|---|---|---|---|
@@ -162,7 +162,7 @@ scrapes Google Maps business listings only. It does not scrape social platforms
| **Social enrichment** | `--socials` finds Instagram, Facebook, LinkedIn, TikTok, YouTube and X profiles in code (HTTP + regex). Zero AI tokens. |
| **Who already advertises** | `--tech` spots a Meta Pixel, Google Ads tag, analytics, Bing Ads, LinkedIn Insight and TikTok pixels on each website, plus its platform (WordPress, Shopify, Wix…) and mobile support, then ranks leads by an `intent_score` from 0 to 100. Zero AI tokens. |
| **Filters** | `--no-website` (businesses that still need a site), `--min-rating`, `--min-reviews`, `--has-phone`, `--has-email`, `--limit`. `--exclude earlier.csv` skips the places you already have, so a weekly re-run returns only new leads. |
-| **Email checks** | `--verify-emails` looks up each domain's MX record over DNS (no key), drops dead domains and labels the mail host (`google_workspace`, `microsoft_365`, `other`). |
+| **Email checks** | `--verify-emails` looks up each domain's MX record over DNS (no key), drops dead domains and labels the mail host (`google_workspace`, `microsoft_365`, `other`, `none`; blank when the row has no email). |
| **Auto-geocoding** | Type a city name. Coordinates are resolved for you via OpenStreetMap. Repeat `--city` to run several cities into one file. |
| **Batch jobs** | Many keywords, one job, one CSV. The engine lists a place found by several keywords once. |
| **Proxy rotation** | Built into the engine for large or repeated runs: `--proxy URL` (repeatable). `socks5`, `socks5h`, `http`, `https`. |
@@ -196,7 +196,7 @@ One keyword per line. One job. One CSV.
- **Rate limits are real.** Heavy use without proxies can get your IP temporarily rate-limited by
Google for minutes to hours. Your Google account is not affected. See [Responsible use](#responsible-use).
- **The web UI is the engine's own page.** It downloads the raw 36-column CSV and uses the engine's
- slower email option, whose `emails` column keeps junk such as `sentry.wixpress.com` and image
+ slower email option, whose `emails` column can keep junk such as `sentry.wixpress.com` and image
names. For a clean outreach list, use the scripts. The page also loads its scripts,
fonts and map tiles from public CDNs and OpenStreetMap, so it needs internet. Scraped data stays on
your machine.
@@ -333,6 +333,7 @@ google-maps-scraper-kit/
├── scripts/
│ ├── scrape.py Python CLI: single, batch, geocoding, socials, CSV
│ ├── scrape.sh bash CLI: single query
+│ ├── update.sh one-command update of a clone or fork
│ └── test_scrape.py offline tests, no network: python3 scripts/test_scrape.py
├── docs/ guides and recipes
├── examples/ sample job body and batch keyword list
diff --git a/SETUP.md b/SETUP.md
index d47681c..705444b 100644
--- a/SETUP.md
+++ b/SETUP.md
@@ -14,7 +14,7 @@ Install these once:
| **Docker Desktop** | runs the scraper container | https://www.docker.com/products/docker-desktop |
| **Python 3** | runs `scrape.py` and `scrape.sh` (standard library only, no `pip install`) | https://www.python.org/downloads |
| **Claude Code** *(optional but recommended)* | lets Claude drive the scraper for you | https://claude.com/claude-code |
-| **git** *(optional)* | only if you'll push this to GitHub | https://git-scm.com |
+| **git** | clones the kit and runs `scripts/update.sh` later | https://git-scm.com |
Check that Docker is running:
```bash
@@ -56,7 +56,7 @@ docker compose up -d
The first run pulls the `gosom/google-maps-scraper` image once: about 600 MB to download, 2.4 GB on disk. When it finishes, check it:
```bash
-docker ps | grep gmaps-scraper # should show it "Up"
+docker compose ps # both services should show "Up" or "running"
curl http://localhost:8080/api/v1/jobs # should print null (no jobs yet) or a JSON list of jobs
```
@@ -192,7 +192,7 @@ curl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>
| `Cannot connect to the Docker daemon` / `docker: command not found` | Docker Desktop is not running (or not installed). Open Docker Desktop, wait until its whale icon stops animating, then run `docker compose up -d` again. |
| `502 Bad Gateway` right after `docker compose up -d` | The engine is still starting, about 10 seconds. The scripts wait for it; in the browser, reload the page. |
| `Permission denied` running `./scripts/scrape.sh` | The file lost its execute bit (a ZIP download does that). Run `bash scripts/scrape.sh …` instead. |
-| `port is already allocated` / something else already uses 8080 | Set `SCRAPER_PORT=8081` in `.env`, then `docker compose up -d`. Compose publishes the scraper on that host port and `scripts/scrape.sh` and `scripts/scrape.py` read the same file, so nothing else needs editing. The port inside the container stays 8080 (do not change the right-hand side of the mapping). If your `.env` still has a `SCRAPER_BASE_URL` line, delete it or update its port: it overrides `SCRAPER_PORT`. The slash commands and the skill show `localhost:8080` in their examples, so tell Claude the port you picked. |
+| `port is already allocated` / something else already uses 8080 | Set `SCRAPER_PORT=8081` in `.env`, then `docker compose up -d`. Compose publishes the scraper on that host port and `scripts/scrape.sh` and `scripts/scrape.py` read the same file, so nothing else needs editing. The port inside the container stays 8080 (do not change the right-hand side of the mapping). If your `.env` still has a `SCRAPER_BASE_URL` line, delete it or update its port: it overrides `SCRAPER_PORT`. The slash commands and the skill show `localhost:8080` in their examples, so tell Claude the port you picked; its pre-approved `curl` rules are pinned to 8080 too, so add `Bash(curl http://localhost:8081/:*)` to `.claude/settings.json` if you want it to stop asking. **A second copy of the kit on the same machine** (a fork next to the original, a test clone): also set `COMPOSE_PROJECT_NAME` and `SCRAPER_CONTAINER` in `.env`, as `.env.example` shows, or `docker compose up -d` adopts the other copy's containers. |
| `422 missing max time` | Job body needs `max_time` (in **seconds**, e.g. `300`). |
| `422 missing geo coordinates` | Job body needs `lat` and `lon` as **strings**, e.g. `"30.2672"`. |
| `422 missing name` / `422 missing lang` | Job body needs `name` (any label) and `lang` (`"en"`). All six required fields: `name`, `keywords`, `lang`, `lat`, `lon`, `max_time`. |
@@ -264,3 +264,4 @@ The skill holds the full rules. These are the essentials:
faster run without them.
- Add proxies for big jobs. The engine rotates them for you.
- Never commit result files. They are git-ignored for you.
+- Changed something? `python3 scripts/test_scrape.py` runs the offline checks in about a second, no engine needed.
diff --git a/docker-compose.yml b/docker-compose.yml
index e8bd395..a52a56d 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -14,7 +14,9 @@ services:
# Only older tags are amd64 and need the 'platform' line below. Default: commented out.
image: gosom/google-maps-scraper:v1.18.1
# platform: linux/amd64
- container_name: gmaps-scraper
+ # SCRAPER_CONTAINER in .env renames both containers, for a second copy of the kit on one machine.
+ # The scripts read the same variable, so the stuck-job recovery restarts the right engine.
+ container_name: ${SCRAPER_CONTAINER:-gmaps-scraper}
restart: unless-stopped
# -c: parallel workers. The engine's own default resolves to 1 in this image, so a depth 5 job
# with emails visits every listing and website one at a time (~8 min). Most of that is business
@@ -32,7 +34,7 @@ services:
# Serves the engine's UI and API with the kit's own GitHub and sponsor links (ui-branding.conf).
ui:
image: nginx:1.27-alpine
- container_name: gmaps-scraper-ui
+ container_name: ${SCRAPER_CONTAINER:-gmaps-scraper}-ui
restart: unless-stopped
depends_on:
- google-maps-scraper
diff --git a/docs/google-maps-email-scraper.md b/docs/google-maps-email-scraper.md
index d338c2b..c581499 100644
--- a/docs/google-maps-email-scraper.md
+++ b/docs/google-maps-email-scraper.md
@@ -32,7 +32,7 @@ python3 scripts/scrape.py "dentists in Denver CO" --city "Denver, CO" --no-deep-
python3 scripts/scrape.py "dentists in Denver CO" --city "Denver, CO" --no-email
```
-The CSV lands in `results-<id>.csv` unless you pass `--out`.
+The CSV lands in `results-<short job id>.csv` unless you pass `--out`.
## Example output
diff --git a/docs/google-maps-scraper-python.md b/docs/google-maps-scraper-python.md
index 7b58dc0..25eaba1 100644
--- a/docs/google-maps-scraper-python.md
+++ b/docs/google-maps-scraper-python.md
@@ -28,7 +28,7 @@ On Windows, run the same commands with `py` or `python3`. Python is only the cli
| `--no-deep-emails` | Checks homepages only for emails, not contact and about pages. |
| `--grid N` | Splits the `--city` area into N x N map tiles, one job each, deduped. Needs `--city`. Max 10. |
| `--max-time` | Job time limit in seconds. Default `600`. |
-| `--out` | Output path. Default `results-<id>.csv`. A `.json` extension writes JSON. |
+| `--out` | Output path. Default `results-<short job id>.csv`. A `.json` extension writes JSON. |
| `--json` | Writes JSON instead of CSV. |
| `--full` | Keeps all 36 raw columns instead of the 8 lead fields. |
| `--fields` | Comma-separated columns to keep, overriding the lead fields. |
@@ -43,7 +43,7 @@ On Windows, run the same commands with `py` or `python3`. Python is only the cli
| `--has-phone`, `--has-email` | Keep only rows with a phone, or rows where an email was found. |
| `--exclude FILE` | Drop places already in an earlier results file (`.csv` or `.json`), repeatable. Matches on `place_id`, `cid`, link, or name plus address, so the lean 8-column CSV works. |
| `--limit N` | Stop at N rows after filtering. |
-| `--verify-emails` | Checks each email domain's MX record over DNS (dns.google, no key). Drops addresses on domains that do not exist and adds `email_host`: `google_workspace`, `microsoft_365`, `other` or `none`. |
+| `--verify-emails` | Checks each email domain's MX record over DNS (dns.google, no key). Drops addresses on domains that do not exist and adds `email_host`: `google_workspace`, `microsoft_365`, `other` or `none` (blank when the row has no email). |
The script creates the job, polls until it finishes, downloads the CSV and keeps the 8 lead fields. It prints a short preview and saves the full results to a file.
diff --git a/scripts/scrape.py b/scripts/scrape.py
index f45359a..7e19cde 100755
--- a/scripts/scrape.py
+++ b/scripts/scrape.py
@@ -778,7 +778,8 @@ def scrape_place(a, body, city, keywords):
return None, [], 0, 0
lat, lon = coords
print(f" → {lat}, {lon}")
- body = dict(body, lat=str(lat), lon=str(lon))
+ # The job name is what the web UI lists, so make it say what the job was.
+ body = dict(body, lat=str(lat), lon=str(lon), name=(f"{keywords[0]} @ {city}" if city else keywords[0])[:80])
for k in keywords:
print(f" • {k}")
if not cells:
@@ -907,11 +908,14 @@ def main():
print(" Google (clears in minutes–hours; returns empty/failed jobs meanwhile). For big or")
print(" repeated runs, add proxies. Proceeding…\n")
- # Health check. Right after `docker compose up -d` the proxy answers 502 for a few seconds while
- # the engine starts, and the README's next line runs at once, so wait up to a minute for it.
+ # Health check. Right after `docker compose up -d` the proxy answers 502 while the engine starts
+ # (up to a minute), and the README's next line runs at once, so wait for that. Nothing listening at
+ # all (connection refused) means the stack is down: say so after 10 s, not after a minute of dots.
+ refused, err = 0, None
for i in range(30):
try:
req("GET", "/api/v1/jobs")
+ err = None
break
except urllib.error.HTTPError as e:
if e.code < 500: # the engine is up but refuses us (wrong API key, wrong URL): say so at once
@@ -919,15 +923,18 @@ def main():
err = e
except Exception as e:
err = e
+ refused += isinstance(getattr(e, "reason", e), ConnectionRefusedError)
+ if refused >= 5:
+ break
print("\r waiting for the engine to start" + "." * (i + 1), end="", flush=True)
time.sleep(2)
- else:
+ if err is not None:
sys.exit(f"\n✗ Scraper not reachable at {BASE}. Is Docker Desktop open? Start the engine with "
f"'docker compose up -d' in the kit folder, wait 10 seconds, then try again.\n ({err})")
if i:
print()
- body = {"name": "scrape-py", "keywords": keywords, "lang": a.lang, "zoom": 15,
+ body = {"keywords": keywords, "lang": a.lang, "zoom": 15,
"fast_mode": False, "radius": 10000,
# The engine's own email pass visits every website in a browser, one per worker: it took
# most of each job and found fewer emails than the crawl below (OpenLeadFinder's
@@ -965,7 +972,7 @@ def main():
if skipped:
print(f"⚠ Incomplete: {skipped} of {ncells} tiles failed, so parts of the area are missing. Rerun later.")
- if a.email and "emails" in fields: # on the raw rows, so --fields without website still works
+ if a.email and "emails" in fields and not a.no_website: # no website, nothing to crawl
pages = "homepage, then contact and about pages" if a.deep_emails else "homepage only"
print(f"▶ Emails: checking each business website ({pages}, {SITE_WORKERS} at a time)…")
enrich_emails(rows, deep=a.deep_emails)
@@ -984,8 +991,6 @@ def main():
# Enrichment runs on the raw rows, which always carry the website, whatever --fields keeps.
if a.socials:
print("▶ Finding socials (Instagram / Facebook / LinkedIn / TikTok / YouTube / X) on each website…")
- print(" ℹ Cost: this runs in CODE = 0 LLM tokens. It only adds ~40–50 tokens per business")
- print(" IF you load the results into an AI's context (negligible if you keep the file on disk).")
enrich_sites(rows, _find_socials, SOCIAL_FIELDS, "socials")
fields = fields + SOCIAL_FIELDS
found = sum(1 for r in rows if any(r.get(k) for k in SOCIAL_FIELDS))
diff --git a/scripts/test_scrape.py b/scripts/test_scrape.py
index 7804d43..481634c 100644
--- a/scripts/test_scrape.py
+++ b/scripts/test_scrape.py
@@ -368,6 +368,7 @@ class EmailCrawl(unittest.TestCase):
mock.patch.object(sys, "argv", argv), redirect_stdout(io.StringIO()):
scrape.main()
self.assertIs(sent[0]["email"], False)
+ self.assertEqual(sent[0]["name"], "dentists in Austin TX") # the web UI lists this name
class IsRealEmail(unittest.TestCase):
@@ -703,6 +704,12 @@ class ArgumentBounds(unittest.TestCase):
scrape.main()
return str(cm.exception)
+ def test_nothing_listening_fails_in_ten_seconds_not_sixty(self):
+ refused = scrape.urllib.error.URLError(ConnectionRefusedError(61, "Connection refused"))
+ with mock.patch.object(scrape, "req", side_effect=refused), mock.patch.object(scrape.time, "sleep") as slept:
+ self.assertIn("Docker Desktop", self.run_main())
+ self.assertEqual(slept.call_count, 4) # five tries, two seconds apart
+
def test_has_email_needs_the_crawl(self):
self.assertIn("--has-email", self.run_main("--has-email", "--no-email"))