Skip to content

Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.

feat(scrape): emails on by default, ask before socials, CSV output default

Mahanaicoacha6a39d1c6479
diff --git a/.claude/commands/scrape-batch.md b/.claude/commands/scrape-batch.md
index 6d4b0a1..41aaf25 100644
--- a/.claude/commands/scrape-batch.md
+++ b/.claude/commands/scrape-batch.md
@@ -10,10 +10,14 @@ so batch = one job with all terms.
1. Ensure the scraper is up (`curl -s http://localhost:8080/api/v1/jobs`; `docker compose up -d` if not).
2. Read the keyword list (one per line; ignore blank / `#` lines). If a `--city` was given, geocode it once
for `lat`/`lon`; otherwise expect the location baked into each keyword (geocode the first one).
-3. **Warn, don't block:** a batch with many keywords is high-volume — print ONE short warning that this can
+3. **ASK about social profiles first** (skip only if the user already said yes/no): *"Also grab social profiles
+ (Instagram / Facebook / LinkedIn) for these? Emails are already included. (yes / no)"* — if yes, add `--socials`.
+4. **Warn, don't block:** a batch with many keywords is high-volume — print ONE short warning that this can
get the IP temporarily rate-limited and suggest proxies for large/repeated runs, then **proceed**.
-4. POST a single job with the full `keywords` array + required fields (`max_time` seconds, string `lat`/`lon`).
- Use a generous `max_time` (e.g. `900`) since many keywords take longer. Poll in the BACKGROUND, download.
-5. Present results grouped/clean and save the full file to disk.
+5. POST a single job with the full `keywords` array + required fields (`max_time` seconds, string `lat`/`lon`)
+ and **`"email": true`** (emails on — the key lead field). Use a generous `max_time` (e.g. `900`) since many
+ keywords + email extraction take longer. Poll in the BACKGROUND, download.
+6. Present results grouped/clean and save the full file to disk.
-Tip: `scripts/scrape.py --keywords-file <file> --city "<City, ST>"` already does all of this.
+Tip: `scripts/scrape.py --keywords-file <file> --city "<City, ST>"` already does all of this — emails are on by
+default; append `--socials` if the user said yes in step 3.
diff --git a/.claude/commands/scrape.md b/.claude/commands/scrape.md
index d515019..1a9225b 100644
--- a/.claude/commands/scrape.md
+++ b/.claude/commands/scrape.md
@@ -8,13 +8,17 @@ Use the `google-maps-scraper` skill. Execute:
1. **Health-check** the API: `curl -s http://localhost:8080/api/v1/jobs`. If it's down, run `docker compose up -d` from the kit root, wait ~10s, and retry.
2. **Parse** the business type and city from the query. Look up the city's **latitude/longitude** (as strings).
-3. **Create the job** (`POST /api/v1/jobs`) with the required fields — `keywords` (location baked into each term), `lat`/`lon` (strings), `max_time` (seconds, default `300`), and `depth` (use the number in the query, else `5`).
-4. **Poll to completion in the BACKGROUND** (the Bash tool blocks foreground `sleep` — use `run_in_background: true` and read the output file when notified). Then download the CSV.
-5. **Present ONLY money-useful lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count`. **Drop everything else** — especially `latitude`/`longitude`, IDs, hours, images, review blobs. Save to a file on disk; do NOT dump the full 34-column CSV into chat. (Include dropped fields only if the user explicitly asks.)
+3. **ASK about social profiles — always, before scraping.** Ask the user ONE short question:
+ *"Want me to also grab their social profiles (Instagram / Facebook / LinkedIn)? Emails are already included by default. (yes / no)"*
+ Wait for the answer. **Skip the question only if** the user already said yes/no to socials in their request.
+ If **yes** → add `--socials` in step 4. If **no** → leave it off. (Socials add no LLM-token cost and one extra HTTP fetch per site — see the skill.)
+4. **Create the job with EMAIL EXTRACTION ON** (`"email": true` in the body — emails are the #1 lead field). Required fields — `keywords` (location baked into each term), `lat`/`lon` (strings), `max_time` (seconds, default `300`), `depth` (the number in the query, else `5`). Easiest path: `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --depth <n>` (emails are already on by default; append `--socials` if the user said yes in step 3).
+5. **Poll to completion in the BACKGROUND** (the Bash tool blocks foreground `sleep` — use `run_in_background: true` and read the output file when notified). Then download the CSV.
+6. **Save a CSV file** (the default — `scripts/scrape.py` writes `results-<id>.csv`; if you use the raw API, save the parsed lead fields to a `.csv` yourself). Then **present ONLY money-useful lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count` (+ `instagram, facebook, linkedin` if socials were requested). **Drop everything else** — especially `latitude`/`longitude`, IDs, hours, images, review blobs. Do NOT dump the full 34-column CSV into chat; show a few sample rows and tell the user the CSV path. (Include dropped fields only if the user explicitly asks.)
-**Socials (if asked):** if the user wants Instagram/Facebook/LinkedIn, run `scripts/scrape.py … --socials`.
-Tell them the cost: extraction is **0 LLM tokens** (runs in code); it only adds ~40–50 tokens/business if you
-load rows into chat. **Never** WebFetch each website yourself to find socials — that costs thousands of tokens.
+**Emails:** on by default in this kit — the scraper visits each business website to find them, which is a bit slower. Some businesses simply don't publish an email, so coverage is partial. For a deliberately fast run with no emails, pass `--no-email`.
+
+**Socials:** when the user says yes, `scripts/scrape.py … --socials` pulls Instagram/Facebook/LinkedIn. Extraction is **0 LLM tokens** (runs in code); it only adds ~40–50 tokens/business if you load rows into chat. **Never** WebFetch each website yourself to find socials — that costs thousands of tokens.
**Warn, don't block:** if the request is high-volume (high `depth`, many keywords, or repeated runs),
print ONE short warning that over-use can get their IP temporarily rate-limited by Google and suggest
diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.md
index a72def9..211c874 100644
--- a/.claude/skills/google-maps-scraper/SKILL.md
+++ b/.claude/skills/google-maps-scraper/SKILL.md
@@ -27,17 +27,24 @@ the user to `SETUP.md`.
**Recommended fields:**
- `depth` (default 10) — how far to scroll → roughly how many listings per keyword. Start at `5`.
-- `lang` `"en"`, `zoom` `15` (city level), `radius` `10000` (meters), `fast_mode` `false`, `email` `false`.
+- `lang` `"en"`, `zoom` `15` (city level), `radius` `10000` (meters), `fast_mode` `false`.
+- **`email` `true` — ON by default in this kit.** Emails are the #1 lead field; the scraper visits each
+ business website to find them (a bit slower). Only set `false` for a deliberately fast, no-email run.
```bash
curl -s -X POST http://localhost:8080/api/v1/jobs \
-H "Content-Type: application/json" \
-d '{"name":"job","keywords":["coffee shops in Austin TX"],"lang":"en","zoom":15,
"lat":"30.2672","lon":"-97.7431","fast_mode":false,"radius":10000,
- "depth":5,"email":false,"max_time":300}'
+ "depth":5,"email":true,"max_time":300}'
# → {"id":"<uuid>"} (HTTP 201; note: lowercase "id")
```
+> **Two things to handle on every scrape:**
+> 1. **Emails: on by default** (`email:true`). Don't turn them off unless the user wants a fast run.
+> 2. **Socials: ASK first.** Before creating the job, ask the user once whether they also want
+> Instagram/Facebook/LinkedIn (see "Social profiles" below). Don't silently skip it.
+
## Step 2 — Poll until done (`GET /api/v1/jobs/{id}`)
The response field is `"Status"` (capital S): `working` → `ok` (success) or `failed`.
@@ -79,12 +86,14 @@ outreach), `link`, `plus_code`, `cid`, `data_id`, `place_id`, `open_hours`, `pop
`user_reviews`, `user_reviews_extended`.
`scripts/scrape.py` already returns exactly this lead set (use `--full` to keep all columns, or
-`--fields "a,b,c"` to customize). If you call the API directly, **strip to the lead fields yourself** before
-presenting — never dump the full 34-column row at the user.
+`--fields "a,b,c"` to customize) and **saves a CSV file by default** (`results-<id>.csv`; pass `--json` for
+JSON). If you call the API directly, **strip to the lead fields yourself** before presenting — never dump the
+full 34-column row at the user.
-### Social profiles — optional (Instagram / Facebook / LinkedIn)
+### Social profiles — ALWAYS ASK the user (Instagram / Facebook / LinkedIn)
Google Maps has no social links, so this is an enrichment: visit each business's `website` and regex out its
-IG/FB/LinkedIn URLs. **Use the script:** `python3 scripts/scrape.py … --socials`.
+IG/FB/LinkedIn URLs. **Before scraping, ask the user once** whether they want socials too (unless they already
+said). If yes → **use the script:** `python3 scripts/scrape.py … --socials`.
- **Token cost — say this to the user when they ask for socials:** the extraction itself is **0 LLM tokens**
(pure HTTP + regex in the script). It only adds **~40–50 tokens per business** to *your* context **if** you
@@ -99,6 +108,7 @@ IG/FB/LinkedIn URLs. **Use the script:** `python3 scripts/scrape.py … --social
- One keyword: `scripts/scrape.sh "<keyword>" <lat> <lon> [depth]` (bash) — create→poll→download in one go.
- Auto-geocode (no coords): `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N]` — resolves
lat/lon via OpenStreetMap Nominatim. You can also geocode the city yourself and pass coords.
+ **Emails come back by default** (use `--no-email` to skip); add `--socials` if the user asked for socials.
- **Batch (many keywords, ONE job):** `python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>"`.
The API takes a `keywords` array, so put all terms in a single job rather than firing many jobs.
Do the manual curl flow only for custom job bodies (e.g. setting `proxies`).
@@ -110,7 +120,8 @@ Do the manual curl flow only for custom job bodies (e.g. setting `proxies`).
## Best practices (from the upstream docs)
- **Depth:** higher `depth` = more results but slower and more block-prone. Start low (5), raise as needed.
- **One job at a time** locally. Many concurrent jobs without proxies → throttling/blocks by Google.
-- **Email extraction (`email:true`)** visits each business's website → *significantly* slower. Only when needed.
+- **Email extraction (`email:true`)** visits each business's website → slower, but it's **on by default** here
+ because emails are the key lead field. Pass `--no-email` (script) or `email:false` (raw API) for a fast run.
- **`fast_mode:true`** returns reduced data, up to ~21 results/query, faster — good for quick lookups.
- **Proxies:** for large/repeated jobs, set `"proxies"` (array of `socks5://`/`http://`/`https://` URLs,
auth supported). The scraper has built-in rotation. This is the main defense against rate-limiting.
diff --git a/scripts/scrape.py b/scripts/scrape.py
index 5978687..97652b7 100755
--- a/scripts/scrape.py
+++ b/scripts/scrape.py
@@ -131,9 +131,15 @@ def main():
ap.add_argument("--keywords-file", help="file with one keyword per line (batch)")
ap.add_argument("--city", help="city/place to auto-geocode for lat/lon")
ap.add_argument("--depth", type=int, default=5)
- ap.add_argument("--email", action="store_true", help="also extract emails (much slower)")
+ # Emails are ON by default (they're the most valuable lead field). Visits each business
+ # website to find an address — a bit slower. Use --no-email to skip for a fast run.
+ ap.add_argument("--email", dest="email", action="store_true", default=True, help=argparse.SUPPRESS)
+ ap.add_argument("--no-email", dest="email", action="store_false",
+ help="skip email extraction for a faster run (emails are on by default)")
ap.add_argument("--max-time", type=int, default=600, help="job time limit in SECONDS")
- ap.add_argument("--out", default=None, help="write parsed JSON results here")
+ ap.add_argument("--out", default=None, help="output file path (default: results-<id>.csv). "
+ "The extension picks the format: .csv (default) or .json")
+ ap.add_argument("--json", action="store_true", help="write JSON instead of the default CSV")
ap.add_argument("--full", action="store_true", help="keep ALL raw columns (default: lean lead fields)")
ap.add_argument("--fields", help="comma-separated columns to keep (overrides the default lead set)")
ap.add_argument("--socials", action="store_true",
@@ -155,9 +161,13 @@ def main():
lat, lon = coords
print(f" → {lat}, {lon}")
+ if a.email:
+ print("ℹ Email extraction is ON (visits each business website; a bit slower). "
+ "Use --no-email for a fast run.")
+
# --- WARN, don't block: flag high-volume jobs (proceed anyway) ---
- if a.depth >= 15 or len(keywords) >= 10 or a.email:
- print("⚠️ Heads-up: this is a large/slow job (high depth, many keywords, or email extraction).")
+ if a.depth >= 15 or len(keywords) >= 10:
+ print("⚠️ Heads-up: this is a large job (high depth or many keywords).")
print(" Running it back-to-back without proxies can get your IP temporarily rate-limited by")
print(" Google (clears in minutes–hours; returns empty/failed jobs meanwhile). For big or")
print(" repeated runs, add proxies. Proceeding…\n")
@@ -217,9 +227,17 @@ def main():
found = sum(1 for r in results if r.get("instagram") or r.get("facebook") or r.get("linkedin"))
print(f" socials found for {found}/{len(results)} businesses")
- out = a.out or f"results-{job_id[:8]}.json"
- with open(out, "w") as f:
- json.dump(results, f, indent=2, ensure_ascii=False)
+ # Default output is a CSV file (opens in Excel / Google Sheets). Use --json (or a .json --out path) for JSON.
+ as_json = a.json or (a.out and a.out.lower().endswith(".json"))
+ out = a.out or f"results-{job_id[:8]}.{'json' if as_json else 'csv'}"
+ if as_json:
+ with open(out, "w") as f:
+ json.dump(results, f, indent=2, ensure_ascii=False)
+ else:
+ with open(out, "w", newline="") as f:
+ w = csv.DictWriter(f, fieldnames=fields)
+ w.writeheader()
+ w.writerows(results)
print(f" saved → {out}")
for r in results[:5]:
tail = f" | IG:{r.get('instagram','') or '—'}" if a.socials else f" | {r.get('website','')}"
diff --git a/scripts/scrape.sh b/scripts/scrape.sh
index a1e633b..88ac7b8 100755
--- a/scripts/scrape.sh
+++ b/scripts/scrape.sh
@@ -21,8 +21,9 @@ if ! curl -s -m 5 "${AUTH[@]}" "$BASE/api/v1/jobs" >/dev/null 2>&1; then
exit 1
fi
-echo "▶ Creating job: \"$KEYWORD\" @ $LAT,$LON depth=$DEPTH"
-BODY=$(printf '{"name":"scrape-cli","keywords":["%s"],"lang":"en","zoom":15,"lat":"%s","lon":"%s","fast_mode":false,"radius":10000,"depth":%s,"email":false,"max_time":600}' \
+echo "▶ Creating job: \"$KEYWORD\" @ $LAT,$LON depth=$DEPTH (email extraction ON)"
+# email:true → the scraper visits each business website to pull emails (the key lead field). A bit slower.
+BODY=$(printf '{"name":"scrape-cli","keywords":["%s"],"lang":"en","zoom":15,"lat":"%s","lon":"%s","fast_mode":false,"radius":10000,"depth":%s,"email":true,"max_time":600}' \
"$KEYWORD" "$LAT" "$LON" "$DEPTH")
ID=$(curl -s -X POST "$BASE/api/v1/jobs" "${AUTH[@]}" -H "Content-Type: application/json" -d "$BODY" \
| grep -oE '[a-f0-9-]{36}' | head -1 || true)