Skip to content

Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.

feat: optional social profile extraction (Instagram/Facebook/LinkedIn)

Mahanaicoachb3a444f0f8ec
diff --git a/.claude/commands/scrape.md b/.claude/commands/scrape.md
index a945f50..d515019 100644
--- a/.claude/commands/scrape.md
+++ b/.claude/commands/scrape.md
@@ -12,6 +12,10 @@ Use the `google-maps-scraper` skill. Execute:
4. **Poll to completion in the BACKGROUND** (the Bash tool blocks foreground `sleep` — use `run_in_background: true` and read the output file when notified). Then download the CSV.
5. **Present ONLY money-useful lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count`. **Drop everything else** — especially `latitude`/`longitude`, IDs, hours, images, review blobs. Save to a file on disk; do NOT dump the full 34-column CSV into chat. (Include dropped fields only if the user explicitly asks.)
+**Socials (if asked):** if the user wants Instagram/Facebook/LinkedIn, run `scripts/scrape.py … --socials`.
+Tell them the cost: extraction is **0 LLM tokens** (runs in code); it only adds ~40–50 tokens/business if you
+load rows into chat. **Never** WebFetch each website yourself to find socials — that costs thousands of tokens.
+
**Warn, don't block:** if the request is high-volume (high `depth`, many keywords, or repeated runs),
print ONE short warning that over-use can get their IP temporarily rate-limited by Google and suggest
adding proxies — then **proceed with the scrape anyway**. Do not gate or demand confirmation just for a
diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.md
index 8dd9500..a72def9 100644
--- a/.claude/skills/google-maps-scraper/SKILL.md
+++ b/.claude/skills/google-maps-scraper/SKILL.md
@@ -82,6 +82,19 @@ outreach), `link`, `plus_code`, `cid`, `data_id`, `place_id`, `open_hours`, `pop
`--fields "a,b,c"` to customize). If you call the API directly, **strip to the lead fields yourself** before
presenting — never dump the full 34-column row at the user.
+### Social profiles — optional (Instagram / Facebook / LinkedIn)
+Google Maps has no social links, so this is an enrichment: visit each business's `website` and regex out its
+IG/FB/LinkedIn URLs. **Use the script:** `python3 scripts/scrape.py … --socials`.
+
+- **Token cost — say this to the user when they ask for socials:** the extraction itself is **0 LLM tokens**
+ (pure HTTP + regex in the script). It only adds **~40–50 tokens per business** to *your* context **if** you
+ load the rows into chat — e.g. ~+2k tokens for 50 leads. Negligible if you keep the file on disk and show a
+ sample. Save the file; show a few rows.
+- **DO NOT** fetch each website yourself with WebFetch to find socials — that reads every page into your
+ context and costs **thousands of tokens**. The script does it for free. Always prefer `--socials`.
+- It's **opt-in/slower** (one HTTP fetch per business) and coverage is partial (~40–70%: only businesses that
+ link socials on their site; no website → no socials). Mention this if the user expects 100%.
+
**Shortcuts (prefer these for common cases):**
- One keyword: `scripts/scrape.sh "<keyword>" <lat> <lon> [depth]` (bash) — create→poll→download in one go.
- Auto-geocode (no coords): `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N]` — resolves
diff --git a/README.md b/README.md
index e25e929..9c18d9d 100644
--- a/README.md
+++ b/README.md
@@ -32,6 +32,7 @@ ready-to-run scripts, and a **Claude skill** so Claude knows exactly how to use
## What it scrapes (and what it doesn't)
- ✅ **Google Maps business listings** → a clean **lead list** by default: `name, phone, email, website, category, address, rating, review count`. The scraper captures ~34 raw fields, but the kit **strips the noise** (geo coordinates, IDs, hours, images, review blobs) so you only get data you can actually use for outreach. Want everything? `scrape.py --full`.
+- ➕ **Optional socials** (`--socials`): also pulls each business's **Instagram / Facebook / LinkedIn** by scanning its website. **Token cost: 0** — it runs in code (HTTP + regex), no AI. It only adds ~40–50 tokens *per business* if you later load the rows into an AI chat (≈+2k for 50 leads; nothing if you keep the file on disk). It's slower (one fetch per site) and coverage is partial (only businesses that link socials on their site).
- ❌ **Not** a social-media scraper. It cannot touch Instagram / TikTok / YouTube.
## Why use this instead of asking an AI to "just scrape Google Maps"
diff --git a/scripts/scrape.py b/scripts/scrape.py
index 10bd964..5978687 100755
--- a/scripts/scrape.py
+++ b/scripts/scrape.py
@@ -15,7 +15,8 @@ Notes:
- Geocoding uses OpenStreetMap Nominatim (free, no key). Please be gentle: it allows ~1 request/sec
and requires a descriptive User-Agent (set below). Don't loop it aggressively.
"""
-import argparse, csv, io, json, os, sys, time, urllib.request, urllib.parse, urllib.error
+import argparse, csv, io, json, os, re, sys, time, urllib.request, urllib.parse, urllib.error
+from concurrent.futures import ThreadPoolExecutor
BASE = os.environ.get("SCRAPER_BASE_URL", "http://localhost:8080")
KEY = os.environ.get("SCRAPER_API_KEY", "")
@@ -67,6 +68,60 @@ def collect_keywords(a):
return out
+# ── Optional social enrichment (Instagram / Facebook / LinkedIn) ──────────────
+# Pure code (HTTP + regex) → ZERO LLM tokens. Visits each business website once.
+SOCIAL_RE = {
+ "instagram": re.compile(r"https?://(?:www\.)?instagram\.com/([A-Za-z0-9_.]+)", re.I),
+ "facebook": re.compile(r"https?://(?:www\.|m\.|web\.)?facebook\.com/([A-Za-z0-9_.\-]+)", re.I),
+ "linkedin": re.compile(r"https?://(?:[a-z]{2,3}\.)?linkedin\.com/(?:company|in|school)/([A-Za-z0-9_.\-%]+)", re.I),
+}
+_SKIP_HANDLE = {"", "home", "pages", "people", "help", "about", "policies", "policy",
+ "legal", "tos", "privacy", "settings", "sharer", "tr", "profile.php",
+ "plugins", "dialog", "intent", "login", "share.php", "permalink.php",
+ "p", "reel", "reels", "explore", "stories", "tv", "watch", "events",
+ "groups", "marketplace", "gaming", "photo", "hashtag", "search"}
+
+def _fetch_html(url, timeout=10):
+ if not url:
+ return ""
+ if not url.startswith(("http://", "https://")):
+ url = "http://" + url
+ try:
+ r = urllib.request.Request(url, headers={
+ "User-Agent": "Mozilla/5.0 (compatible; " + UA + ")", "Accept": "text/html"})
+ with urllib.request.urlopen(r, timeout=timeout) as resp:
+ return resp.read(400_000).decode("utf-8", "replace")
+ except Exception:
+ return ""
+
+def _find_socials(html):
+ out = {"instagram": "", "facebook": "", "linkedin": ""}
+ for plat, rx in SOCIAL_RE.items():
+ for m in rx.finditer(html or ""):
+ h = m.group(1).lower()
+ if h in _SKIP_HANDLE:
+ continue
+ if plat == "facebook" and (h.isdigit() or len(h) < 3): # skip junk like /2008
+ continue
+ out[plat] = m.group(0).rstrip('"\'/').replace("\\", "")
+ break
+ return out
+
+def enrich_socials(results, workers=8):
+ """Add instagram/facebook/linkedin to each lead by scanning its website. Zero LLM tokens."""
+ for r in results: # make sure every row has the keys
+ r.setdefault("instagram", ""); r.setdefault("facebook", ""); r.setdefault("linkedin", "")
+ todo = [r for r in results if r.get("website")]
+ done = 0
+ def work(r):
+ r.update(_find_socials(_fetch_html(r["website"])))
+ with ThreadPoolExecutor(max_workers=workers) as ex:
+ for _ in ex.map(work, todo):
+ done += 1
+ print(f"\r socials: {done}/{len(todo)} sites checked", end="", flush=True)
+ print()
+
+
def main():
ap = argparse.ArgumentParser(description="Scrape Google Maps business listings.")
ap.add_argument("keyword", nargs="?", help='e.g. "coffee shops in Austin TX"')
@@ -81,6 +136,8 @@ def main():
ap.add_argument("--out", default=None, help="write parsed JSON results here")
ap.add_argument("--full", action="store_true", help="keep ALL raw columns (default: lean lead fields)")
ap.add_argument("--fields", help="comma-separated columns to keep (overrides the default lead set)")
+ ap.add_argument("--socials", action="store_true",
+ help="also find Instagram/Facebook/LinkedIn from each website (0 LLM tokens; slower)")
a = ap.parse_args()
keywords = collect_keywords(a)
@@ -151,12 +208,22 @@ def main():
results = [{k: r.get(k, "") for k in fields} for r in rows]
print(f"✓ Done — {len(results)} businesses (fields: {', '.join(fields)}).")
+ if a.socials:
+ print("▶ Finding socials (Instagram / Facebook / LinkedIn) on each website…")
+ print(" ℹ Cost: this runs in CODE = 0 LLM tokens. It only adds ~40–50 tokens per business")
+ print(" IF you load the results into an AI's context (negligible if you keep the file on disk).")
+ enrich_socials(results)
+ fields = fields + ["instagram", "facebook", "linkedin"]
+ found = sum(1 for r in results if r.get("instagram") or r.get("facebook") or r.get("linkedin"))
+ print(f" socials found for {found}/{len(results)} businesses")
+
out = a.out or f"results-{job_id[:8]}.json"
with open(out, "w") as f:
json.dump(results, f, indent=2, ensure_ascii=False)
print(f" saved → {out}")
for r in results[:5]:
- print(f" • {r.get('title','')} | {r.get('phone','')} | {r.get('emails','')} | {r.get('website','')}")
+ tail = f" | IG:{r.get('instagram','') or '—'}" if a.socials else f" | {r.get('website','')}"
+ print(f" • {r.get('title','')} | {r.get('phone','')} | {r.get('emails','')}{tail}")
if __name__ == "__main__":