Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
feat: optional social profile extraction (Instagram/Facebook/LinkedIn)
Mahanaicoachb3a444f0f8ec
diff --git a/.claude/commands/scrape.md b/.claude/commands/scrape.mdindex a945f50..d515019 100644--- a/.claude/commands/scrape.md+++ b/.claude/commands/scrape.md@@ -12,6 +12,10 @@ Use the `google-maps-scraper` skill. Execute:4. **Poll to completion in the BACKGROUND** (the Bash tool blocks foreground `sleep` — use `run_in_background: true` and read the output file when notified). Then download the CSV.5. **Present ONLY money-useful lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count`. **Drop everything else** — especially `latitude`/`longitude`, IDs, hours, images, review blobs. Save to a file on disk; do NOT dump the full 34-column CSV into chat. (Include dropped fields only if the user explicitly asks.)+**Socials (if asked):** if the user wants Instagram/Facebook/LinkedIn, run `scripts/scrape.py … --socials`.+Tell them the cost: extraction is **0 LLM tokens** (runs in code); it only adds ~40–50 tokens/business if you+load rows into chat. **Never** WebFetch each website yourself to find socials — that costs thousands of tokens.+**Warn, don't block:** if the request is high-volume (high `depth`, many keywords, or repeated runs),print ONE short warning that over-use can get their IP temporarily rate-limited by Google and suggestadding proxies — then **proceed with the scrape anyway**. Do not gate or demand confirmation just for adiff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdindex 8dd9500..a72def9 100644--- a/.claude/skills/google-maps-scraper/SKILL.md+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -82,6 +82,19 @@ outreach), `link`, `plus_code`, `cid`, `data_id`, `place_id`, `open_hours`, `pop`--fields "a,b,c"` to customize). If you call the API directly, **strip to the lead fields yourself** beforepresenting — never dump the full 34-column row at the user.+### Social profiles — optional (Instagram / Facebook / LinkedIn)+Google Maps has no social links, so this is an enrichment: visit each business's `website` and regex out its+IG/FB/LinkedIn URLs. **Use the script:** `python3 scripts/scrape.py … --socials`.++- **Token cost — say this to the user when they ask for socials:** the extraction itself is **0 LLM tokens**+ (pure HTTP + regex in the script). It only adds **~40–50 tokens per business** to *your* context **if** you+ load the rows into chat — e.g. ~+2k tokens for 50 leads. Negligible if you keep the file on disk and show a+ sample. Save the file; show a few rows.+- **DO NOT** fetch each website yourself with WebFetch to find socials — that reads every page into your+ context and costs **thousands of tokens**. The script does it for free. Always prefer `--socials`.+- It's **opt-in/slower** (one HTTP fetch per business) and coverage is partial (~40–70%: only businesses that+ link socials on their site; no website → no socials). Mention this if the user expects 100%.+**Shortcuts (prefer these for common cases):**- One keyword: `scripts/scrape.sh "<keyword>" <lat> <lon> [depth]` (bash) — create→poll→download in one go.- Auto-geocode (no coords): `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N]` — resolvesdiff --git a/README.md b/README.mdindex e25e929..9c18d9d 100644--- a/README.md+++ b/README.md@@ -32,6 +32,7 @@ ready-to-run scripts, and a **Claude skill** so Claude knows exactly how to use## What it scrapes (and what it doesn't)- ✅ **Google Maps business listings** → a clean **lead list** by default: `name, phone, email, website, category, address, rating, review count`. The scraper captures ~34 raw fields, but the kit **strips the noise** (geo coordinates, IDs, hours, images, review blobs) so you only get data you can actually use for outreach. Want everything? `scrape.py --full`.+- ➕ **Optional socials** (`--socials`): also pulls each business's **Instagram / Facebook / LinkedIn** by scanning its website. **Token cost: 0** — it runs in code (HTTP + regex), no AI. It only adds ~40–50 tokens *per business* if you later load the rows into an AI chat (≈+2k for 50 leads; nothing if you keep the file on disk). It's slower (one fetch per site) and coverage is partial (only businesses that link socials on their site).- ❌ **Not** a social-media scraper. It cannot touch Instagram / TikTok / YouTube.## Why use this instead of asking an AI to "just scrape Google Maps"diff --git a/scripts/scrape.py b/scripts/scrape.pyindex 10bd964..5978687 100755--- a/scripts/scrape.py+++ b/scripts/scrape.py@@ -15,7 +15,8 @@ Notes:- Geocoding uses OpenStreetMap Nominatim (free, no key). Please be gentle: it allows ~1 request/secand requires a descriptive User-Agent (set below). Don't loop it aggressively."""-import argparse, csv, io, json, os, sys, time, urllib.request, urllib.parse, urllib.error+import argparse, csv, io, json, os, re, sys, time, urllib.request, urllib.parse, urllib.error+from concurrent.futures import ThreadPoolExecutorBASE = os.environ.get("SCRAPER_BASE_URL", "http://localhost:8080")KEY = os.environ.get("SCRAPER_API_KEY", "")@@ -67,6 +68,60 @@ def collect_keywords(a):return out+# ── Optional social enrichment (Instagram / Facebook / LinkedIn) ──────────────+# Pure code (HTTP + regex) → ZERO LLM tokens. Visits each business website once.+SOCIAL_RE = {+ "instagram": re.compile(r"https?://(?:www\.)?instagram\.com/([A-Za-z0-9_.]+)", re.I),+ "facebook": re.compile(r"https?://(?:www\.|m\.|web\.)?facebook\.com/([A-Za-z0-9_.\-]+)", re.I),+ "linkedin": re.compile(r"https?://(?:[a-z]{2,3}\.)?linkedin\.com/(?:company|in|school)/([A-Za-z0-9_.\-%]+)", re.I),+}+_SKIP_HANDLE = {"", "home", "pages", "people", "help", "about", "policies", "policy",+ "legal", "tos", "privacy", "settings", "sharer", "tr", "profile.php",+ "plugins", "dialog", "intent", "login", "share.php", "permalink.php",+ "p", "reel", "reels", "explore", "stories", "tv", "watch", "events",+ "groups", "marketplace", "gaming", "photo", "hashtag", "search"}++def _fetch_html(url, timeout=10):+ if not url:+ return ""+ if not url.startswith(("http://", "https://")):+ url = "http://" + url+ try:+ r = urllib.request.Request(url, headers={+ "User-Agent": "Mozilla/5.0 (compatible; " + UA + ")", "Accept": "text/html"})+ with urllib.request.urlopen(r, timeout=timeout) as resp:+ return resp.read(400_000).decode("utf-8", "replace")+ except Exception:+ return ""++def _find_socials(html):+ out = {"instagram": "", "facebook": "", "linkedin": ""}+ for plat, rx in SOCIAL_RE.items():+ for m in rx.finditer(html or ""):+ h = m.group(1).lower()+ if h in _SKIP_HANDLE:+ continue+ if plat == "facebook" and (h.isdigit() or len(h) < 3): # skip junk like /2008+ continue+ out[plat] = m.group(0).rstrip('"\'/').replace("\\", "")+ break+ return out++def enrich_socials(results, workers=8):+ """Add instagram/facebook/linkedin to each lead by scanning its website. Zero LLM tokens."""+ for r in results: # make sure every row has the keys+ r.setdefault("instagram", ""); r.setdefault("facebook", ""); r.setdefault("linkedin", "")+ todo = [r for r in results if r.get("website")]+ done = 0+ def work(r):+ r.update(_find_socials(_fetch_html(r["website"])))+ with ThreadPoolExecutor(max_workers=workers) as ex:+ for _ in ex.map(work, todo):+ done += 1+ print(f"\r socials: {done}/{len(todo)} sites checked", end="", flush=True)+ print()++def main():ap = argparse.ArgumentParser(description="Scrape Google Maps business listings.")ap.add_argument("keyword", nargs="?", help='e.g. "coffee shops in Austin TX"')@@ -81,6 +136,8 @@ def main():ap.add_argument("--out", default=None, help="write parsed JSON results here")ap.add_argument("--full", action="store_true", help="keep ALL raw columns (default: lean lead fields)")ap.add_argument("--fields", help="comma-separated columns to keep (overrides the default lead set)")+ ap.add_argument("--socials", action="store_true",+ help="also find Instagram/Facebook/LinkedIn from each website (0 LLM tokens; slower)")a = ap.parse_args()keywords = collect_keywords(a)@@ -151,12 +208,22 @@ def main():results = [{k: r.get(k, "") for k in fields} for r in rows]print(f"✓ Done — {len(results)} businesses (fields: {', '.join(fields)}).")+ if a.socials:+ print("▶ Finding socials (Instagram / Facebook / LinkedIn) on each website…")+ print(" ℹ Cost: this runs in CODE = 0 LLM tokens. It only adds ~40–50 tokens per business")+ print(" IF you load the results into an AI's context (negligible if you keep the file on disk).")+ enrich_socials(results)+ fields = fields + ["instagram", "facebook", "linkedin"]+ found = sum(1 for r in results if r.get("instagram") or r.get("facebook") or r.get("linkedin"))+ print(f" socials found for {found}/{len(results)} businesses")+out = a.out or f"results-{job_id[:8]}.json"with open(out, "w") as f:json.dump(results, f, indent=2, ensure_ascii=False)print(f" saved → {out}")for r in results[:5]:- print(f" • {r.get('title','')} | {r.get('phone','')} | {r.get('emails','')} | {r.get('website','')}")+ tail = f" | IG:{r.get('instagram','') or '—'}" if a.socials else f" | {r.get('website','')}"+ print(f" • {r.get('title','')} | {r.get('phone','')} | {r.get('emails','')}{tail}")if __name__ == "__main__":