Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
feat(scrape): emails on by default, ask before socials, CSV output default
Mahanaicoacha6a39d1c6479
diff --git a/.claude/commands/scrape-batch.md b/.claude/commands/scrape-batch.mdindex 6d4b0a1..41aaf25 100644--- a/.claude/commands/scrape-batch.md+++ b/.claude/commands/scrape-batch.md@@ -10,10 +10,14 @@ so batch = one job with all terms.1. Ensure the scraper is up (`curl -s http://localhost:8080/api/v1/jobs`; `docker compose up -d` if not).2. Read the keyword list (one per line; ignore blank / `#` lines). If a `--city` was given, geocode it oncefor `lat`/`lon`; otherwise expect the location baked into each keyword (geocode the first one).-3. **Warn, don't block:** a batch with many keywords is high-volume — print ONE short warning that this can+3. **ASK about social profiles first** (skip only if the user already said yes/no): *"Also grab social profiles+ (Instagram / Facebook / LinkedIn) for these? Emails are already included. (yes / no)"* — if yes, add `--socials`.+4. **Warn, don't block:** a batch with many keywords is high-volume — print ONE short warning that this canget the IP temporarily rate-limited and suggest proxies for large/repeated runs, then **proceed**.-4. POST a single job with the full `keywords` array + required fields (`max_time` seconds, string `lat`/`lon`).- Use a generous `max_time` (e.g. `900`) since many keywords take longer. Poll in the BACKGROUND, download.-5. Present results grouped/clean and save the full file to disk.+5. POST a single job with the full `keywords` array + required fields (`max_time` seconds, string `lat`/`lon`)+ and **`"email": true`** (emails on — the key lead field). Use a generous `max_time` (e.g. `900`) since many+ keywords + email extraction take longer. Poll in the BACKGROUND, download.+6. Present results grouped/clean and save the full file to disk.-Tip: `scripts/scrape.py --keywords-file <file> --city "<City, ST>"` already does all of this.+Tip: `scripts/scrape.py --keywords-file <file> --city "<City, ST>"` already does all of this — emails are on by+default; append `--socials` if the user said yes in step 3.diff --git a/.claude/commands/scrape.md b/.claude/commands/scrape.mdindex d515019..1a9225b 100644--- a/.claude/commands/scrape.md+++ b/.claude/commands/scrape.md@@ -8,13 +8,17 @@ Use the `google-maps-scraper` skill. Execute:1. **Health-check** the API: `curl -s http://localhost:8080/api/v1/jobs`. If it's down, run `docker compose up -d` from the kit root, wait ~10s, and retry.2. **Parse** the business type and city from the query. Look up the city's **latitude/longitude** (as strings).-3. **Create the job** (`POST /api/v1/jobs`) with the required fields — `keywords` (location baked into each term), `lat`/`lon` (strings), `max_time` (seconds, default `300`), and `depth` (use the number in the query, else `5`).-4. **Poll to completion in the BACKGROUND** (the Bash tool blocks foreground `sleep` — use `run_in_background: true` and read the output file when notified). Then download the CSV.-5. **Present ONLY money-useful lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count`. **Drop everything else** — especially `latitude`/`longitude`, IDs, hours, images, review blobs. Save to a file on disk; do NOT dump the full 34-column CSV into chat. (Include dropped fields only if the user explicitly asks.)+3. **ASK about social profiles — always, before scraping.** Ask the user ONE short question:+ *"Want me to also grab their social profiles (Instagram / Facebook / LinkedIn)? Emails are already included by default. (yes / no)"*+ Wait for the answer. **Skip the question only if** the user already said yes/no to socials in their request.+ If **yes** → add `--socials` in step 4. If **no** → leave it off. (Socials add no LLM-token cost and one extra HTTP fetch per site — see the skill.)+4. **Create the job with EMAIL EXTRACTION ON** (`"email": true` in the body — emails are the #1 lead field). Required fields — `keywords` (location baked into each term), `lat`/`lon` (strings), `max_time` (seconds, default `300`), `depth` (the number in the query, else `5`). Easiest path: `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --depth <n>` (emails are already on by default; append `--socials` if the user said yes in step 3).+5. **Poll to completion in the BACKGROUND** (the Bash tool blocks foreground `sleep` — use `run_in_background: true` and read the output file when notified). Then download the CSV.+6. **Save a CSV file** (the default — `scripts/scrape.py` writes `results-<id>.csv`; if you use the raw API, save the parsed lead fields to a `.csv` yourself). Then **present ONLY money-useful lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count` (+ `instagram, facebook, linkedin` if socials were requested). **Drop everything else** — especially `latitude`/`longitude`, IDs, hours, images, review blobs. Do NOT dump the full 34-column CSV into chat; show a few sample rows and tell the user the CSV path. (Include dropped fields only if the user explicitly asks.)-**Socials (if asked):** if the user wants Instagram/Facebook/LinkedIn, run `scripts/scrape.py … --socials`.-Tell them the cost: extraction is **0 LLM tokens** (runs in code); it only adds ~40–50 tokens/business if you-load rows into chat. **Never** WebFetch each website yourself to find socials — that costs thousands of tokens.+**Emails:** on by default in this kit — the scraper visits each business website to find them, which is a bit slower. Some businesses simply don't publish an email, so coverage is partial. For a deliberately fast run with no emails, pass `--no-email`.++**Socials:** when the user says yes, `scripts/scrape.py … --socials` pulls Instagram/Facebook/LinkedIn. Extraction is **0 LLM tokens** (runs in code); it only adds ~40–50 tokens/business if you load rows into chat. **Never** WebFetch each website yourself to find socials — that costs thousands of tokens.**Warn, don't block:** if the request is high-volume (high `depth`, many keywords, or repeated runs),print ONE short warning that over-use can get their IP temporarily rate-limited by Google and suggestdiff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdindex a72def9..211c874 100644--- a/.claude/skills/google-maps-scraper/SKILL.md+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -27,17 +27,24 @@ the user to `SETUP.md`.**Recommended fields:**- `depth` (default 10) — how far to scroll → roughly how many listings per keyword. Start at `5`.-- `lang` `"en"`, `zoom` `15` (city level), `radius` `10000` (meters), `fast_mode` `false`, `email` `false`.+- `lang` `"en"`, `zoom` `15` (city level), `radius` `10000` (meters), `fast_mode` `false`.+- **`email` `true` — ON by default in this kit.** Emails are the #1 lead field; the scraper visits each+ business website to find them (a bit slower). Only set `false` for a deliberately fast, no-email run.```bashcurl -s -X POST http://localhost:8080/api/v1/jobs \-H "Content-Type: application/json" \-d '{"name":"job","keywords":["coffee shops in Austin TX"],"lang":"en","zoom":15,"lat":"30.2672","lon":"-97.7431","fast_mode":false,"radius":10000,- "depth":5,"email":false,"max_time":300}'+ "depth":5,"email":true,"max_time":300}'# → {"id":"<uuid>"} (HTTP 201; note: lowercase "id")```+> **Two things to handle on every scrape:**+> 1. **Emails: on by default** (`email:true`). Don't turn them off unless the user wants a fast run.+> 2. **Socials: ASK first.** Before creating the job, ask the user once whether they also want+> Instagram/Facebook/LinkedIn (see "Social profiles" below). Don't silently skip it.+## Step 2 — Poll until done (`GET /api/v1/jobs/{id}`)The response field is `"Status"` (capital S): `working` → `ok` (success) or `failed`.@@ -79,12 +86,14 @@ outreach), `link`, `plus_code`, `cid`, `data_id`, `place_id`, `open_hours`, `pop`user_reviews`, `user_reviews_extended`.`scripts/scrape.py` already returns exactly this lead set (use `--full` to keep all columns, or-`--fields "a,b,c"` to customize). If you call the API directly, **strip to the lead fields yourself** before-presenting — never dump the full 34-column row at the user.+`--fields "a,b,c"` to customize) and **saves a CSV file by default** (`results-<id>.csv`; pass `--json` for+JSON). If you call the API directly, **strip to the lead fields yourself** before presenting — never dump the+full 34-column row at the user.-### Social profiles — optional (Instagram / Facebook / LinkedIn)+### Social profiles — ALWAYS ASK the user (Instagram / Facebook / LinkedIn)Google Maps has no social links, so this is an enrichment: visit each business's `website` and regex out its-IG/FB/LinkedIn URLs. **Use the script:** `python3 scripts/scrape.py … --socials`.+IG/FB/LinkedIn URLs. **Before scraping, ask the user once** whether they want socials too (unless they already+said). If yes → **use the script:** `python3 scripts/scrape.py … --socials`.- **Token cost — say this to the user when they ask for socials:** the extraction itself is **0 LLM tokens**(pure HTTP + regex in the script). It only adds **~40–50 tokens per business** to *your* context **if** you@@ -99,6 +108,7 @@ IG/FB/LinkedIn URLs. **Use the script:** `python3 scripts/scrape.py … --social- One keyword: `scripts/scrape.sh "<keyword>" <lat> <lon> [depth]` (bash) — create→poll→download in one go.- Auto-geocode (no coords): `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N]` — resolveslat/lon via OpenStreetMap Nominatim. You can also geocode the city yourself and pass coords.+ **Emails come back by default** (use `--no-email` to skip); add `--socials` if the user asked for socials.- **Batch (many keywords, ONE job):** `python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>"`.The API takes a `keywords` array, so put all terms in a single job rather than firing many jobs.Do the manual curl flow only for custom job bodies (e.g. setting `proxies`).@@ -110,7 +120,8 @@ Do the manual curl flow only for custom job bodies (e.g. setting `proxies`).## Best practices (from the upstream docs)- **Depth:** higher `depth` = more results but slower and more block-prone. Start low (5), raise as needed.- **One job at a time** locally. Many concurrent jobs without proxies → throttling/blocks by Google.-- **Email extraction (`email:true`)** visits each business's website → *significantly* slower. Only when needed.+- **Email extraction (`email:true`)** visits each business's website → slower, but it's **on by default** here+ because emails are the key lead field. Pass `--no-email` (script) or `email:false` (raw API) for a fast run.- **`fast_mode:true`** returns reduced data, up to ~21 results/query, faster — good for quick lookups.- **Proxies:** for large/repeated jobs, set `"proxies"` (array of `socks5://`/`http://`/`https://` URLs,auth supported). The scraper has built-in rotation. This is the main defense against rate-limiting.diff --git a/scripts/scrape.py b/scripts/scrape.pyindex 5978687..97652b7 100755--- a/scripts/scrape.py+++ b/scripts/scrape.py@@ -131,9 +131,15 @@ def main():ap.add_argument("--keywords-file", help="file with one keyword per line (batch)")ap.add_argument("--city", help="city/place to auto-geocode for lat/lon")ap.add_argument("--depth", type=int, default=5)- ap.add_argument("--email", action="store_true", help="also extract emails (much slower)")+ # Emails are ON by default (they're the most valuable lead field). Visits each business+ # website to find an address — a bit slower. Use --no-email to skip for a fast run.+ ap.add_argument("--email", dest="email", action="store_true", default=True, help=argparse.SUPPRESS)+ ap.add_argument("--no-email", dest="email", action="store_false",+ help="skip email extraction for a faster run (emails are on by default)")ap.add_argument("--max-time", type=int, default=600, help="job time limit in SECONDS")- ap.add_argument("--out", default=None, help="write parsed JSON results here")+ ap.add_argument("--out", default=None, help="output file path (default: results-<id>.csv). "+ "The extension picks the format: .csv (default) or .json")+ ap.add_argument("--json", action="store_true", help="write JSON instead of the default CSV")ap.add_argument("--full", action="store_true", help="keep ALL raw columns (default: lean lead fields)")ap.add_argument("--fields", help="comma-separated columns to keep (overrides the default lead set)")ap.add_argument("--socials", action="store_true",@@ -155,9 +161,13 @@ def main():lat, lon = coordsprint(f" → {lat}, {lon}")+ if a.email:+ print("ℹ Email extraction is ON (visits each business website; a bit slower). "+ "Use --no-email for a fast run.")+# --- WARN, don't block: flag high-volume jobs (proceed anyway) ---- if a.depth >= 15 or len(keywords) >= 10 or a.email:- print("⚠️ Heads-up: this is a large/slow job (high depth, many keywords, or email extraction).")+ if a.depth >= 15 or len(keywords) >= 10:+ print("⚠️ Heads-up: this is a large job (high depth or many keywords).")print(" Running it back-to-back without proxies can get your IP temporarily rate-limited by")print(" Google (clears in minutes–hours; returns empty/failed jobs meanwhile). For big or")print(" repeated runs, add proxies. Proceeding…\n")@@ -217,9 +227,17 @@ def main():found = sum(1 for r in results if r.get("instagram") or r.get("facebook") or r.get("linkedin"))print(f" socials found for {found}/{len(results)} businesses")- out = a.out or f"results-{job_id[:8]}.json"- with open(out, "w") as f:- json.dump(results, f, indent=2, ensure_ascii=False)+ # Default output is a CSV file (opens in Excel / Google Sheets). Use --json (or a .json --out path) for JSON.+ as_json = a.json or (a.out and a.out.lower().endswith(".json"))+ out = a.out or f"results-{job_id[:8]}.{'json' if as_json else 'csv'}"+ if as_json:+ with open(out, "w") as f:+ json.dump(results, f, indent=2, ensure_ascii=False)+ else:+ with open(out, "w", newline="") as f:+ w = csv.DictWriter(f, fieldnames=fields)+ w.writeheader()+ w.writerows(results)print(f" saved → {out}")for r in results[:5]:tail = f" | IG:{r.get('instagram','') or '—'}" if a.socials else f" | {r.get('website','')}"diff --git a/scripts/scrape.sh b/scripts/scrape.shindex a1e633b..88ac7b8 100755--- a/scripts/scrape.sh+++ b/scripts/scrape.sh@@ -21,8 +21,9 @@ if ! curl -s -m 5 "${AUTH[@]}" "$BASE/api/v1/jobs" >/dev/null 2>&1; thenexit 1fi-echo "▶ Creating job: \"$KEYWORD\" @ $LAT,$LON depth=$DEPTH"-BODY=$(printf '{"name":"scrape-cli","keywords":["%s"],"lang":"en","zoom":15,"lat":"%s","lon":"%s","fast_mode":false,"radius":10000,"depth":%s,"email":false,"max_time":600}' \+echo "▶ Creating job: \"$KEYWORD\" @ $LAT,$LON depth=$DEPTH (email extraction ON)"+# email:true → the scraper visits each business website to pull emails (the key lead field). A bit slower.+BODY=$(printf '{"name":"scrape-cli","keywords":["%s"],"lang":"en","zoom":15,"lat":"%s","lon":"%s","fast_mode":false,"radius":10000,"depth":%s,"email":true,"max_time":600}' \"$KEYWORD" "$LAT" "$LON" "$DEPTH")ID=$(curl -s -X POST "$BASE/api/v1/jobs" "${AUTH[@]}" -H "Content-Type: application/json" -d "$BODY" \| grep -oE '[a-f0-9-]{36}' | head -1 || true)