Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
Google Maps Scraper Kit — free local Google Maps scraper, driven by Claude
Mahanaicoachfa805c6973ed
diff --git a/.claude/commands/scrape-batch.md b/.claude/commands/scrape-batch.mdnew file mode 100644index 0000000..6d4b0a1--- /dev/null+++ b/.claude/commands/scrape-batch.md@@ -0,0 +1,19 @@+---+description: Scrape many Google Maps queries at once (batch) from a list of keywords+argument-hint: <keywords-file> [--city "City, ST"]+---+The user wants a **batch** Google Maps scrape. Input: **$ARGUMENTS** (a keywords file, optionally a city).++Use the `google-maps-scraper` skill. The API scrapes **many keywords in ONE job** (the `keywords` array),+so batch = one job with all terms.++1. Ensure the scraper is up (`curl -s http://localhost:8080/api/v1/jobs`; `docker compose up -d` if not).+2. Read the keyword list (one per line; ignore blank / `#` lines). If a `--city` was given, geocode it once+ for `lat`/`lon`; otherwise expect the location baked into each keyword (geocode the first one).+3. **Warn, don't block:** a batch with many keywords is high-volume — print ONE short warning that this can+ get the IP temporarily rate-limited and suggest proxies for large/repeated runs, then **proceed**.+4. POST a single job with the full `keywords` array + required fields (`max_time` seconds, string `lat`/`lon`).+ Use a generous `max_time` (e.g. `900`) since many keywords take longer. Poll in the BACKGROUND, download.+5. Present results grouped/clean and save the full file to disk.++Tip: `scripts/scrape.py --keywords-file <file> --city "<City, ST>"` already does all of this.diff --git a/.claude/commands/scrape-jobs.md b/.claude/commands/scrape-jobs.mdnew file mode 100644index 0000000..d4142cc--- /dev/null+++ b/.claude/commands/scrape-jobs.md@@ -0,0 +1,11 @@+---+description: List or clean up Google Maps scrape jobs+argument-hint: "[list | delete <job-id>]"+---+Manage scrape jobs via the API. Request: **$ARGUMENTS** (default to "list" if empty).++- **list** → `GET http://localhost:8080/api/v1/jobs`. Show a tidy table of `ID`, `Name`, `Status`, `Date` (newest first).+- **delete <job-id>** → `DELETE http://localhost:8080/api/v1/jobs/<job-id>` to free disk space, then confirm it returned HTTP 200.+- **download <job-id>** → `GET .../jobs/<job-id>/download` → save CSV and summarize row count.++See the `google-maps-scraper` skill for the full API reference.diff --git a/.claude/commands/scrape-setup.md b/.claude/commands/scrape-setup.mdnew file mode 100644index 0000000..66f78ba--- /dev/null+++ b/.claude/commands/scrape-setup.md@@ -0,0 +1,18 @@+---+description: Set up and start the local Google Maps scraper (Docker)+---+Set up the Google Maps scraper on this computer:++1. Confirm Docker is installed and running: `docker ps`. If it errors, tell the user to install/start **Docker Desktop** (link in `SETUP.md` §0) and stop here.+2. From the kit root, start the scraper: `docker compose up -d`.+3. Wait ~10s, then verify it's live: `curl http://localhost:8080/api/v1/jobs` should return a JSON array (often `[]`).+4. Report success and tell the user they can now run `/scrape <business> in <city>` — or just ask in natural language (e.g. "scrape gyms in Miami").+5. **Show this over-use warning** (always print it once setup succeeds):+ > ⚠️ **Before you scrape:** this hits Google Maps for real. Light use is fine, but running big jobs+ > back-to-back, very high `depth`, many keywords, or scheduled runs **without proxies** can get your+ > IP **temporarily rate-limited/blocked by Google** (usually clears in minutes–hours; jobs start+ > returning empty/failed meanwhile). It won't ban your Google account. Stay safe: one job at a time,+ > start at `depth 5`, and add **proxies** for large or repeated runs. Scraping Maps is also against+ > Google's ToS — use responsibly and follow data laws (GDPR/CCPA) for any contact info.++If a step fails, consult `SETUP.md` §6 (Troubleshooting).diff --git a/.claude/commands/scrape.md b/.claude/commands/scrape.mdnew file mode 100644index 0000000..a945f50--- /dev/null+++ b/.claude/commands/scrape.md@@ -0,0 +1,19 @@+---+description: Scrape Google Maps business listings for a query + city (drives the google-maps-scraper skill)+argument-hint: <business type> in <city, ST> [depth]+---+The user wants to scrape Google Maps business listings. Query: **$ARGUMENTS**++Use the `google-maps-scraper` skill. Execute:++1. **Health-check** the API: `curl -s http://localhost:8080/api/v1/jobs`. If it's down, run `docker compose up -d` from the kit root, wait ~10s, and retry.+2. **Parse** the business type and city from the query. Look up the city's **latitude/longitude** (as strings).+3. **Create the job** (`POST /api/v1/jobs`) with the required fields — `keywords` (location baked into each term), `lat`/`lon` (strings), `max_time` (seconds, default `300`), and `depth` (use the number in the query, else `5`).+4. **Poll to completion in the BACKGROUND** (the Bash tool blocks foreground `sleep` — use `run_in_background: true` and read the output file when notified). Then download the CSV.+5. **Present ONLY money-useful lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count`. **Drop everything else** — especially `latitude`/`longitude`, IDs, hours, images, review blobs. Save to a file on disk; do NOT dump the full 34-column CSV into chat. (Include dropped fields only if the user explicitly asks.)++**Warn, don't block:** if the request is high-volume (high `depth`, many keywords, or repeated runs),+print ONE short warning that over-use can get their IP temporarily rate-limited by Google and suggest+adding proxies — then **proceed with the scrape anyway**. Do not gate or demand confirmation just for a+big job. Only refuse outright for clearly abusive use (surveilling individuals, spam/harassment). Also+remind the user about PII / Google ToS if they'll store or contact the data.diff --git a/.claude/settings.json b/.claude/settings.jsonnew file mode 100644index 0000000..15e01bd--- /dev/null+++ b/.claude/settings.json@@ -0,0 +1,21 @@+{+ "$comment": "Pre-approves the safe, local commands this kit uses so Claude can scrape on autopilot without a permission prompt each time. All entries are local-only (localhost / this folder's scripts / docker). Nothing here can reach the public internet except the scraper's own outbound calls.",+ "permissions": {+ "allow": [+ "Bash(docker compose up:*)",+ "Bash(docker compose down:*)",+ "Bash(docker compose stop:*)",+ "Bash(docker compose start:*)",+ "Bash(docker compose ps:*)",+ "Bash(docker ps:*)",+ "Bash(curl http://localhost:8080:*)",+ "Bash(curl -s http://localhost:8080:*)",+ "Bash(curl http://127.0.0.1:8080:*)",+ "Bash(./scripts/scrape.sh:*)",+ "Bash(bash scripts/scrape.sh:*)",+ "Bash(python3 scripts/scrape.py:*)",+ "Bash(python scripts/scrape.py:*)"+ ],+ "deny": []+ }+}diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdnew file mode 100644index 0000000..8dd9500--- /dev/null+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -0,0 +1,144 @@+---+name: google-maps-scraper+description: Scrape Google Maps business listings (name, address, phone, website, rating, reviews, lat/lng, hours, emails) via the local gosom google-maps-scraper REST API. Use when the user wants local-business / lead-gen data, "a list of [businesses] in [place]", or to enrich places with contact info. NOT for Instagram/TikTok/YouTube or any social-media scraping.+---++# Google Maps Scraper++Drive the local Google Maps scraper API to turn a business-type + location into clean, structured rows.++## Mental model+The scraper runs as a local Docker container exposing a REST API at `http://localhost:8080` (no auth —+localhost only). A "scrape" is an **async job**: you create it, poll until it's done, then download a CSV.+One job can run many keywords. Each result has up to **34 fields**.++## Step 0 — Make sure it's running+```bash+curl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOWN+```+If `DOWN`: `docker compose up -d` (from the kit root), wait ~10s, retry. If Docker isn't installed, point+the user to `SETUP.md`.++## Step 1 — Create a job (`POST /api/v1/jobs`)+**Required fields — the API returns `422` without them:**+- `keywords` — array of search strings. **Bake the location into each term**: `"plumbers in Denver CO"`.+- `lat`, `lon` — **strings**, the city's coordinates: `"39.7392"`, `"-104.9903"`.+- `max_time` — integer **seconds** (max wall-clock for the job), e.g. `300`. (Sent as seconds; the API stores it as nanoseconds internally — just send seconds.)++**Recommended fields:**+- `depth` (default 10) — how far to scroll → roughly how many listings per keyword. Start at `5`.+- `lang` `"en"`, `zoom` `15` (city level), `radius` `10000` (meters), `fast_mode` `false`, `email` `false`.++```bash+curl -s -X POST http://localhost:8080/api/v1/jobs \+ -H "Content-Type: application/json" \+ -d '{"name":"job","keywords":["coffee shops in Austin TX"],"lang":"en","zoom":15,+ "lat":"30.2672","lon":"-97.7431","fast_mode":false,"radius":10000,+ "depth":5,"email":false,"max_time":300}'+# → {"id":"<uuid>"} (HTTP 201; note: lowercase "id")+```++## Step 2 — Poll until done (`GET /api/v1/jobs/{id}`)+The response field is `"Status"` (capital S): `working` → `ok` (success) or `failed`.++> ⚠️ **Claude harness rule:** the Bash tool **blocks foreground `sleep`**. Run the poll loop as a+> **background** Bash command (`run_in_background: true`) and read its output file when notified.+> Do NOT poll with a foreground `sleep`.++Background poll snippet (parses the value safely — match up to the closing quote, don't anchor on `$`):+```bash+ID="<uuid>"+for i in $(seq 1 40); do+ S=$(curl -s "http://localhost:8080/api/v1/jobs/$ID" | grep -oE '"Status":"[^"]*"' | head -1 | cut -d'"' -f4)+ echo "status=$S"+ [ "$S" = ok ] && { echo DONE; break; }+ [ "$S" = failed ] && { echo FAILED; break; }+ sleep 15+done+```++## Step 3 — Download + parse (`GET /api/v1/jobs/{id}/download`)+```bash+curl -s "http://localhost:8080/api/v1/jobs/$ID/download" -o results.csv+```+CSV columns (34): `input_id, link, title, category, address, open_hours, popular_times, website, phone,+plus_code, review_count, review_rating, reviews_per_rating, latitude, longitude, cid, status,+descriptions, reviews_link, thumbnail, timezone, price_range, data_id, place_id, images, reservations,+order_online, menu, owner, complete_address, about, user_reviews, user_reviews_extended, emails`.++### ⭐ Output ONLY money-useful lead fields (default)+The raw CSV has 34 columns and most are noise. **By default, return ONLY these lead fields** — the data you+actually use to contact and qualify a lead — and **drop everything else**:++> `title` (name), `phone`, `emails`, `website`, `category`, `address`, `review_rating`, `review_count`++**DROP by default** (do not show these unless the user explicitly asks): `latitude`/`longitude` (no use for+outreach), `link`, `plus_code`, `cid`, `data_id`, `place_id`, `open_hours`, `popular_times`,+`reviews_per_rating`, `reviews_link`, `thumbnail`, `images`, `timezone`, `price_range`, `status`,+`input_id`, `complete_address`, `reservations`, `order_online`, `menu`, `owner`, `about`, `descriptions`,+`user_reviews`, `user_reviews_extended`.++`scripts/scrape.py` already returns exactly this lead set (use `--full` to keep all columns, or+`--fields "a,b,c"` to customize). If you call the API directly, **strip to the lead fields yourself** before+presenting — never dump the full 34-column row at the user.++**Shortcuts (prefer these for common cases):**+- One keyword: `scripts/scrape.sh "<keyword>" <lat> <lon> [depth]` (bash) — create→poll→download in one go.+- Auto-geocode (no coords): `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N]` — resolves+ lat/lon via OpenStreetMap Nominatim. You can also geocode the city yourself and pass coords.+- **Batch (many keywords, ONE job):** `python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>"`.+ The API takes a `keywords` array, so put all terms in a single job rather than firing many jobs.+Do the manual curl flow only for custom job bodies (e.g. setting `proxies`).++## Other endpoints+- `GET /api/v1/jobs` — list jobs. `DELETE /api/v1/jobs/{id}` — delete a job + free disk.+- Browser UI + OpenAPI docs: `http://localhost:8080` and `http://localhost:8080/api/docs`.++## Best practices (from the upstream docs)+- **Depth:** higher `depth` = more results but slower and more block-prone. Start low (5), raise as needed.+- **One job at a time** locally. Many concurrent jobs without proxies → throttling/blocks by Google.+- **Email extraction (`email:true`)** visits each business's website → *significantly* slower. Only when needed.+- **`fast_mode:true`** returns reduced data, up to ~21 results/query, faster — good for quick lookups.+- **Proxies:** for large/repeated jobs, set `"proxies"` (array of `socks5://`/`http://`/`https://` URLs,+ auth supported). The scraper has built-in rotation. This is the main defense against rate-limiting.+- **`zoom`/`radius`** control the search area around `lat`/`lon`. Widen `radius` if results are too few.+- **Extended reviews** are available but heavy — don't enable unless the user asks for review text.++## Rate limits, bans & proxies (read before large jobs)+This hits Google Maps for real. The upstream project's only formal note is a disclaimer:+> "Please use this scraper responsibly and in accordance with applicable laws and regulations. Unauthorized scraping may violate terms of service."++There is **no published hard threshold** for bans, so be conservative:+- Google may **temporarily rate-limit / block your IP** if you scrape too fast or too much. It clears in+ minutes–hours and does **not** ban your Google account — but jobs start failing meanwhile.+- **Block signals to watch:** jobs returning `failed`, empty or unusually short results, or a sudden drop+ in row counts vs. a prior identical run. If you see these, **back off** (pause, lower depth) or add proxies.+- **Concurrency ↔ blocking** (upstream): *"Higher concurrency … can increase blocking or failures,+ especially without proxies. Start with the default for a first run."* Reference throughput ≈ **120+ places/min** at `-c 8 -depth 1`. Locally, run **one job at a time** and start at `depth 5`.++**When to add proxies** (upstream: *"For larger scraping jobs, proxies help avoid rate limiting"*):+large jobs, many keywords, repeated/scheduled runs, or after you see block signals.+- Set the `"proxies"` array in the job body. Types: `socks5`, `socks5h`, `http`, `https`.+- Format: `protocol://user:pass@host:port` (auth optional), e.g.+ `"proxies": ["socks5://user:pass@host:port", "http://host2:port2"]`. The scraper rotates them automatically.++## Safety & guardrails+- **WARN, don't block.** When a request is large/high-volume (high `depth`, many keywords, repeated runs,+ or `email:true`), **proceed with it** but first print ONE short warning about temporary IP-block risk and+ suggest proxies. Do **not** gate, force-stop, or demand confirmation just because a normal scrape is big.+ Refuse outright **only** for clearly abusive/illegal use (surveilling individuals, spam/harassment).+- **Never expose** the API beyond `localhost` without an auth proxy; never print any API key if one exists.+- **PII:** scraped emails/phones are personal data. If the user will store or contact them, remind them to+ comply with GDPR/CCPA/CAN-SPAM (lawful basis, opt-outs, suppression). Don't help with spam/harassment.+- **Google ToS:** scraping Maps is against Google's Terms — keep volume modest, treat output as leads to+ verify, don't resell raw Google data. Refuse uses aimed at surveilling individuals.+- **Output hygiene:** don't dump huge CSVs into chat — save the file to disk, summarize counts, show a few+ sample rows. Dedupe by `place_id`/`cid` before storing.+- **Disk:** results pile up in the Docker volume; offer to `DELETE` old jobs periodically.++## Troubleshooting+- `422 missing max time` → add `max_time` (seconds). `422 missing geo coordinates` → add string `lat`/`lon`.+- Stuck `working` → lower `depth` / raise `max_time` / IP throttled (add proxies or wait).+- Empty CSV → keyword too narrow or geo wrong → widen `radius`, fix coordinates.+- Connection refused → container down → `docker compose up -d`.diff --git a/.env.example b/.env.examplenew file mode 100644index 0000000..9b4b0bd--- /dev/null+++ b/.env.example@@ -0,0 +1,10 @@+# Copy this to ".env" and adjust if needed: cp .env.example .env+#+# This local setup binds the scraper to 127.0.0.1 only, so NO API key is required.+# (The scraper image has no built-in authentication — localhost binding is the boundary.)++SCRAPER_BASE_URL=http://localhost:8080++# Only needed if you later expose the scraper beyond localhost behind an auth proxy+# (e.g. Caddy/nginx that checks an X-API-Key header). Leave blank for local use.+SCRAPER_API_KEY=diff --git a/.gitignore b/.gitignorenew file mode 100644index 0000000..d76c569--- /dev/null+++ b/.gitignore@@ -0,0 +1,18 @@+# Secrets+.env++# Scrape outputs (may contain personal data — never commit)+*.csv+results-*.json+out/+output/+gmaps-output/+data/++# Python build cruft+__pycache__/+*.pyc++# OS / editor noise+.DS_Store+Thumbs.dbdiff --git a/CLAUDE.md b/CLAUDE.mdnew file mode 100644index 0000000..d501cf7--- /dev/null+++ b/CLAUDE.md@@ -0,0 +1,42 @@+# CLAUDE.md — Google Maps Scraper Kit++You are working inside the **Google Maps Scraper Kit**. This folder runs a local Google Maps scraper+and you drive it for the user.++## What this project is+A Docker-based local deployment of the open-source `gosom/google-maps-scraper`, exposed as a REST API+at **`http://localhost:8080`**, plus a skill that teaches you to use it. It scrapes **Google Maps+business listings only** (not social media).++## When the user asks to scrape businesses / get a list of places / lead-gen data+**Load and follow the `google-maps-scraper` skill** (`.claude/skills/google-maps-scraper/SKILL.md`).+It has the exact API flow, the required fields, and the best-practice + safety rules. Don't improvise+the API calls from memory — use the skill.++## Slash commands+This kit ships commands in `.claude/commands/` that map to the skill: `/scrape` (single query),+`/scrape-batch` (many queries, one job), `/scrape-setup` (start the scraper), `/scrape-jobs` (list/delete).+Local commands are pre-approved in `.claude/settings.json` so you can run them without prompting the user.++## Rate limits — warn, don't block+If a request is large/high-volume (high `depth`, many keywords, repeated runs, or `email:true`), **proceed**+but first print ONE short warning that over-use can get the user's IP temporarily rate-limited by Google,+and suggest proxies. Do NOT gate or demand confirmation for a normal big scrape. Refuse only clearly+abusive use (surveilling individuals, spam). Full guidance is in the skill.++## Hard facts you must not get wrong+- API base: `http://localhost:8080` — **no auth** (it's localhost-only). Never expose it publicly.+- Create a job: `POST /api/v1/jobs`. **Required** body fields or it 422s:+ `max_time` (in **seconds**, e.g. `300`), `lat` & `lon` (as **strings**), plus `keywords` (array).+- The create response returns `{"id":"<uuid>"}` (lowercase `id`). Status checks use `GET /api/v1/jobs/{id}` whose field is `"Status"` and becomes `"ok"` when done.+- **Poll in the background.** The Bash tool blocks foreground `sleep`; run the poll loop as a+ background command (`run_in_background: true`) and read its output file when notified.+- Before a big job (high depth / many keywords), **confirm with the user** — cost is time + IP-block risk.+- Scraped emails/phones are **personal data**; don't dump huge CSVs into chat — summarize + save to disk.++## Setup+If the container isn't running: `docker compose up -d`, then verify `curl http://localhost:8080/api/v1/jobs`.+Full setup is in `SETUP.md`.++## Attribution+This wraps `gosom/google-maps-scraper` (MIT, © Georgios Komninos). Keep `CREDITS.md` + `LICENSE` intact.diff --git a/CREDITS.md b/CREDITS.mdnew file mode 100644index 0000000..c4fe689--- /dev/null+++ b/CREDITS.md@@ -0,0 +1,45 @@+# Credits & Attribution++This kit is a **wrapper** around an existing open-source project. The actual scraping engine is **not**+written by this kit's author — full credit goes to the original author.++## Upstream project++- **Project:** google-maps-scraper+- **Author:** **Georgios Komninos** ([@gosom](https://github.com/gosom))+- **Repository:** https://github.com/gosom/google-maps-scraper+- **License:** MIT License — Copyright (c) 2023 Georgios Komninos+- **Docker image used:** `gosom/google-maps-scraper`++This kit pulls and runs the official published Docker image. It does **not** modify or redistribute the+upstream source code — it only adds a Docker Compose setup, helper scripts, documentation, and a Claude+skill around it. All scraping capability and field extraction come from the upstream project.++## Upstream license (reproduced as required by MIT)++```+MIT License++Copyright (c) 2023 Georgios Komninos++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.+```++If you found this useful, please **⭐ star the upstream repo** to support the original author:+https://github.com/gosom/google-maps-scraperdiff --git a/LICENSE b/LICENSEnew file mode 100644index 0000000..9ec7d22--- /dev/null+++ b/LICENSE@@ -0,0 +1,26 @@+MIT License++Copyright (c) 2026 Mahan (github.com/mahanjafari-dev)++This license covers the "Google Maps Scraper Kit" wrapper (Docker Compose setup,+scripts, documentation, and Claude skill) ONLY. The underlying scraping engine,+"google-maps-scraper", is a separate work by Georgios Komninos, licensed under the+MIT License (Copyright (c) 2023 Georgios Komninos) — see CREDITS.md.++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.diff --git a/README.md b/README.mdnew file mode 100644index 0000000..e25e929--- /dev/null+++ b/README.md@@ -0,0 +1,151 @@+# Google Maps Scraper Kit 🗺️++<p align="center">+ <img src="assets/banner.svg" alt="Google Maps Scraper Kit — run a free Google Maps scraper locally, let Claude drive it on autopilot" width="100%">+</p>++A **plug-and-play kit** that runs a free, open-source Google Maps scraper **on your own computer**+and lets **Claude** drive it on autopilot — give Claude a city + a business type, get back a clean+list of businesses (name, address, phone, website, rating, reviews, lat/lng, emails).++Built as a thin wrapper around [**`gosom/google-maps-scraper`**](https://github.com/gosom/google-maps-scraper)+by **Georgios Komninos** (MIT). See [CREDITS.md](CREDITS.md). This kit adds: a one-command Docker setup,+ready-to-run scripts, and a **Claude skill** so Claude knows exactly how to use the API the right way.++> ⚠️ **Before you scrape — over-use can get your IP temporarily blocked.**+> This runs a real scraper against Google Maps. Light use is fine, but big jobs back-to-back, very high+> `depth`, many keywords, or scheduled runs **without proxies** can make Google **temporarily rate-limit or+> block your IP** (usually clears in minutes–hours; jobs return empty/failed meanwhile). It does **not** ban+> your Google account. Stay safe: **one job at a time, start at `depth 5`, add proxies for large/repeated+> runs.** Scraping Maps is against Google's ToS — use responsibly and follow data law (GDPR/CCPA) for any+> contact data. Full guidance: [Rate limits, bans & responsible use](#️-rate-limits-bans--responsible-use).++---++## What you get++- 🐳 **One-command local setup** — `docker compose up -d` runs the scraper at `http://localhost:8080`.+- 🤖 **A Claude skill** — open this folder in Claude Code and just say *"scrape coffee shops in Austin"*. Claude does create → poll → download → clean results, following best practices automatically.+- 🛠️ **Standalone scripts** — `scripts/scrape.sh` (bash) and `scripts/scrape.py` (Python, no dependencies) if you'd rather not use Claude.+- 🔒 **Safe by default** — the scraper binds to `127.0.0.1` only, secrets are git-ignored, and the skill enforces rate-limit / legal / data-handling guardrails.++## What it scrapes (and what it doesn't)++- ✅ **Google Maps business listings** → a clean **lead list** by default: `name, phone, email, website, category, address, rating, review count`. The scraper captures ~34 raw fields, but the kit **strips the noise** (geo coordinates, IDs, hours, images, review blobs) so you only get data you can actually use for outreach. Want everything? `scrape.py --full`.+- ❌ **Not** a social-media scraper. It cannot touch Instagram / TikTok / YouTube.++## Why use this instead of asking an AI to "just scrape Google Maps"++Tested head-to-head: a chat AI fetching `maps.google.com` directly hits a **consent wall** and+**JavaScript-rendered** content, so it returns **~0 structured results**. This kit reliably returns+**dozens–hundreds of clean rows per query**. For Google Maps, it's the difference between *can't* and *can*.++---++## Quick start (3 steps)++```bash+# 1. Start the scraper (needs Docker Desktop)+docker compose up -d++# 2. Check it's alive — should print a JSON list (probably empty [])+curl http://localhost:8080/api/v1/jobs++# 3a. Scrape with the script…+./scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5++# 3b. …or open this folder in Claude Code and say:+# "Scrape gyms in Miami and give me phones + websites."+```++Full instructions: **[SETUP.md](SETUP.md)**. How Claude uses it: **[.claude/skills/google-maps-scraper/SKILL.md](.claude/skills/google-maps-scraper/SKILL.md)**.++---++## Invoking it in Claude Code++### Slash commands+| Command | What it does |+|---|---|+| `/scrape <business> in <city, ST> [depth]` | Run a scrape and get a clean table |+| `/scrape-batch <keywords-file> [--city "City, ST"]` | Scrape many queries in one job |+| `/scrape-setup` | Start the local scraper + health check |+| `/scrape-jobs [list \| delete <id>]` | List or delete jobs |++Example: `/scrape dentists in Denver CO 10`++### Or just talk to it — the skill auto-triggers on phrases like:+- "scrape coffee shops in Austin"+- "find all gyms in Miami with phone numbers"+- "build me a lead list of dentists in Denver, CO"+- "pull Google Maps listings for plumbers in Phoenix"++It intentionally **does not** trigger for social media (Instagram/TikTok) — that's not what it does.++---++## ⚠️ Rate limits, bans & responsible use++The upstream project's guidance (and ours): *"Please use this scraper responsibly and in accordance with+applicable laws and regulations. Unauthorized scraping may violate terms of service."* There's **no+published ban threshold**, so be conservative.++**What "over-use" looks like** and the risk:+- Big jobs back-to-back, very high `depth`, many keywords at once, or scheduled runs **without proxies**.+- Risk: Google **temporarily rate-limits/blocks your IP** (minutes–hours, then clears). Your Google+ **account is not banned**. While blocked, jobs come back `failed` or with empty/short results.++**Block signals to watch:** jobs returning `failed`, suddenly empty results, or far fewer rows than an+identical earlier run. If you see these, **slow down** — pause, lower `depth`, or add proxies.++**Stay safe:**+- One job at a time; start at `depth 5` and raise only as needed.+- Turn on `email` extraction only when you need emails (it's much slower).+- Treat results as **leads to verify**, not a redistributable dataset. Don't resell raw Google data.+- Scraped phones/emails are **personal data** → follow GDPR/CCPA/CAN-SPAM if you store or contact them.++**When to add proxies** (*"For larger scraping jobs, proxies help avoid rate limiting"*): large jobs, many+keywords, repeated/scheduled runs, or after you hit block signals. Set the `proxies` array in the job body:+```json+"proxies": ["socks5://user:pass@host:port", "http://host2:port2"]+```+Types: `socks5`, `socks5h`, `http`, `https` (auth optional). The scraper rotates them automatically.+For reference, upstream measures ~120 places/min at `-c 8 -depth 1`.++> Claude's rule here is **warn, don't block**: if you ask for a large scrape, it'll flag the risk and+> suggest proxies, then run it anyway. It only refuses clearly abusive use.++---++## What's inside++```+google-maps-scraper-kit/+├── README.md ← you are here+├── SETUP.md ← detailed, copy-paste setup (start here)+├── CLAUDE.md ← auto-loaded instructions for Claude Code+├── CREDITS.md ← attribution to the upstream author (gosom)+├── LICENSE ← MIT (this kit)+├── docker-compose.yml ← runs the scraper locally on :8080+├── .env.example ← config template+├── .gitignore+├── scripts/+│ ├── scrape.sh ← one-shot bash scraper (single keyword)+│ └── scrape.py ← Python scraper: single, batch, + auto-geocoding (stdlib only)+├── examples/+│ ├── queries.example.json ← reference job body + coordinate cheatsheet+│ └── queries.example.txt ← batch keyword list (one per line)+└── .claude/+ ├── settings.json ← pre-approves local commands so Claude runs on autopilot+ ├── commands/ ← /scrape, /scrape-batch, /scrape-setup, /scrape-jobs+ └── skills/+ └── google-maps-scraper/+ └── SKILL.md ← THE skill that teaches Claude to use the API well+```++## Credits & license++This kit orchestrates **[google-maps-scraper](https://github.com/gosom/google-maps-scraper)** by+**Georgios Komninos**, used under the MIT License. The kit's own code is MIT licensed too. Full+attribution in **[CREDITS.md](CREDITS.md)**. Please respect Google's Terms of Service and applicable+data-protection law (GDPR/CCPA) when using scraped data — see the Safety section in the skill.diff --git a/SETUP.md b/SETUP.mdnew file mode 100644index 0000000..1a43dd8--- /dev/null+++ b/SETUP.md@@ -0,0 +1,187 @@+# SETUP — Google Maps Scraper Kit++This guide sets up the whole system **on your computer** from scratch. It's written so that a person+*or* Claude can follow it exactly. Total time: ~5 minutes (most of it is Docker pulling the image).++---++## 0. Prerequisites++Install these first (one-time):++| Tool | Why | Get it |+|---|---|---|+| **Docker Desktop** | runs the scraper container | https://www.docker.com/products/docker-desktop |+| **Claude Code** *(optional but recommended)* | lets Claude drive the scraper for you | https://claude.com/claude-code |+| **git** *(optional)* | only if you'll push this to GitHub | https://git-scm.com |++Verify Docker is running:+```bash+docker --version # should print a version+docker ps # should NOT error (means the daemon is up)+```++> **Platform notes:**+> - **Apple Silicon (M1/M2/M3):** the scraper image is `linux/amd64` and runs under emulation. If the+> container won't start, uncomment the `platform: linux/amd64` line in `docker-compose.yml`.+> - **Windows:** use `scripts/scrape.py` (run with `py` or `python3`). The bash script `scrape.sh` needs+> **WSL2** or **Git Bash**.++---++## 1. Get the kit++If someone shared this folder with you, just `cd` into it:+```bash+cd google-maps-scraper-kit+```+(Or clone it from GitHub if it's hosted: `git clone <repo-url> && cd google-maps-scraper-kit`.)++Create your local config (optional — defaults work as-is):+```bash+cp .env.example .env+```++---++## 2. Start the scraper++```bash+docker compose up -d+```++First run pulls the `gosom/google-maps-scraper` image (~once, a few hundred MB). When it finishes:++```bash+docker ps | grep gmaps-scraper # should show it "Up"+curl http://localhost:8080/api/v1/jobs # should print [] (empty list) or existing jobs+```++You can also open the built-in UI and API docs in a browser:+- Web UI: **http://localhost:8080**+- OpenAPI docs: **http://localhost:8080/api/docs**++✅ If `curl` returns a JSON array, the system is live.++> ⚠️ **Now that it's running — read this once.** This hits Google Maps for real. Light use is fine, but+> big jobs back-to-back, very high `depth`, many keywords, or scheduled runs **without proxies** can get+> your **IP temporarily rate-limited/blocked by Google** (clears in minutes–hours; jobs return empty/failed+> meanwhile). It does **not** ban your Google account. **Stay safe:** one job at a time, start at `depth 5`,+> and add **proxies** for large or repeated runs. Scraping Maps is against Google's ToS — use responsibly+> and follow data law (GDPR/CCPA) for any contact data. (Full guidance in README → *Rate limits, bans &+> responsible use*.)++---++## 3. Run your first scrape (without Claude)++Use the included script. Arguments: `"<keyword>" <lat> <lon> [depth]`++```bash+./scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5+```++or the Python version (no dependencies needed):++```bash+python3 scripts/scrape.py "gyms in Miami FL" 25.7617 -80.1918 --depth 5+```++The script will **create the job → poll until done → download the CSV → print a summary** and save+the full results to a file.++**Don't want to look up coordinates?** Let it geocode the city for you:+```bash+python3 scripts/scrape.py "cafes in Austin TX" --city "Austin, TX" --depth 5+```++**Scrape many queries at once** (one job, many keywords) from a file:+```bash+python3 scripts/scrape.py --keywords-file examples/queries.example.txt --city "Denver, CO"+```++---++## 4. Use it with Claude (the autopilot way) 🤖++This is the point of the kit. Two parts make it work:++1. **[CLAUDE.md](CLAUDE.md)** — auto-loaded when you open this folder in Claude Code. It tells Claude+ what this project is and the key rules.+2. **[.claude/skills/google-maps-scraper/SKILL.md](.claude/skills/google-maps-scraper/SKILL.md)** — a+ **skill** Claude loads on demand. It teaches Claude the exact API flow + best practices.++### How to connect it to Claude+There's nothing to "connect" — the scraper is a local HTTP API at `http://localhost:8080`, and Claude+calls it with its normal `curl`/Bash ability. Just:++```bash+cd google-maps-scraper-kit+claude # start Claude Code inside this folder+```++Then talk to it naturally:+> "Make sure the scraper is running, then scrape **dentists in Denver CO**, depth 10, and give me a table of name, phone, website."++Claude will: check the container is up (start it if not) → build the correct job (with the required+`max_time`, `lat`, `lon`) → poll in the background → download → hand you clean rows. It follows the+safety/best-practice rules in the skill automatically.++> ℹ️ **Advanced (optional):** prefer a Model Context Protocol server instead of curl? You can wrap the+> API as an MCP tool, but it's unnecessary for a local single-user setup — the skill + local API is+> simpler and just as capable.++---++### Slash-command shortcuts+Inside Claude Code you can also use these commands (type `/` to see them):++| Command | What it does |+|---|---|+| `/scrape <business> in <city, ST> [depth]` | Run a scrape, get a clean table |+| `/scrape-batch <keywords-file> [--city "City, ST"]` | Scrape many queries in one job |+| `/scrape-setup` | Start the scraper + health check |+| `/scrape-jobs [list \| delete <id>]` | List or delete jobs |++…or just ask in plain English: *"scrape gyms in Miami with phone numbers."* (The scraper's commands and+`localhost` calls are pre-approved in `.claude/settings.json`, so Claude won't prompt for each one.)++---++## 5. Stop / clean up++```bash+docker compose stop # pause (keeps data + image)+docker compose down # stop + remove container (keeps named volumes/data)+docker compose down -v # ALSO delete scraped data volumes (full reset)+```++Delete old jobs to free disk (results accumulate in the data volume):+```bash+curl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>+```++---++## 6. Troubleshooting++| Symptom | Fix |+|---|---|+| `curl: connection refused` on :8080 | Container not up. `docker compose up -d`, then `docker ps`. |+| `422 missing max time` | Job body needs `max_time` (in **seconds**, e.g. `300`). |+| `422 missing geo coordinates` | Job body needs `lat` and `lon` as **strings**, e.g. `"30.2672"`. |+| Job stuck `working` forever | Lower `depth`, or the IP is being throttled by Google — wait, or add proxies (see skill). |+| Empty CSV | Keyword too narrow or wrong geo — widen `radius` or fix `lat`/`lon`. |+| Docker pull is slow | Normal on first run; it caches after that. |++---++## 7. Best practices & safety (summary)++The full rules live in the skill, but the essentials:+- **Always** send `max_time` (seconds) and `lat`/`lon` (strings).+- Start with **low `depth`** (5) and **one job at a time**. Raise only when needed.+- Turn on **email extraction** only when you need emails — it's much slower.+- For big jobs, add **proxies** (the scraper has built-in rotation) to avoid IP blocks.+- Scraped **phones/emails are personal data** — comply with GDPR/CCPA/CAN-SPAM, and respect Google's ToS.+- Never commit result files (they're git-ignored for you).diff --git a/assets/banner.svg b/assets/banner.svgnew file mode 100644index 0000000..fd01448--- /dev/null+++ b/assets/banner.svg@@ -0,0 +1,83 @@+<svg width="1280" height="360" viewBox="0 0 1280 360" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Google Maps Scraper Kit">+ <defs>+ <linearGradient id="bg" x1="0" y1="0" x2="1" y2="1">+ <stop offset="0" stop-color="#0b1220"/>+ <stop offset="1" stop-color="#111c33"/>+ </linearGradient>+ <linearGradient id="pin" x1="0" y1="0" x2="0" y2="1">+ <stop offset="0" stop-color="#34d399"/>+ <stop offset="1" stop-color="#059669"/>+ </linearGradient>+ <filter id="shadow" x="-40%" y="-40%" width="180%" height="180%">+ <feDropShadow dx="0" dy="8" stdDeviation="12" flood-color="#000" flood-opacity="0.40"/>+ </filter>+ <style>+ .t { font-family: -apple-system,Segoe UI,Roboto,Helvetica,Arial,sans-serif; }+ .mono { font-family: ui-monospace,SFMono-Regular,Menlo,Consolas,monospace; }+ </style>+ </defs>++ <!-- background -->+ <rect width="1280" height="360" fill="url(#bg)"/>++ <!-- faint map grid + roads -->+ <g stroke="#1e293b" stroke-width="1" opacity="0.5">+ <path d="M0 72 H1280 M0 144 H1280 M0 216 H1280 M0 288 H1280"/>+ <path d="M160 0 V360 M360 0 V360 M560 0 V360 M760 0 V360 M960 0 V360 M1160 0 V360"/>+ </g>+ <g stroke="#10b981" stroke-width="3" opacity="0.16" fill="none" stroke-linecap="round">+ <path d="M-20 270 C 200 250, 280 110, 520 135 S 980 110, 1300 50"/>+ <path d="M-20 80 C 260 100, 420 225, 720 215 S 1100 270, 1300 230"/>+ </g>++ <!-- map pin with magnifier -->+ <g transform="translate(140,182) scale(0.82)" filter="url(#shadow)">+ <path d="M0,-95 C 52,-95 96,-54 96,-2 C 96,58 18,112 0,150 C -18,112 -96,58 -96,-2 C -96,-54 -52,-95 0,-95 Z" fill="url(#pin)"/>+ <circle cx="0" cy="-6" r="40" fill="#0b1220"/>+ <circle cx="-6" cy="-13" r="19" fill="none" stroke="#34d399" stroke-width="6"/>+ <line x1="9" y1="6" x2="27" y2="24" stroke="#34d399" stroke-width="8" stroke-linecap="round"/>+ </g>++ <!-- title + subtitle (kept clear of the card on the right) -->+ <text x="300" y="128" class="t" font-weight="800" font-size="44" fill="#f8fafc">Google Maps Scraper Kit</text>+ <text x="302" y="170" class="t" font-weight="400" font-size="21" fill="#94a3b8">Run a free Google Maps scraper locally, driven by Claude.</text>++ <!-- chips -->+ <g transform="translate(300,200)">+ <g>+ <rect x="0" y="0" rx="22" ry="22" width="170" height="42" fill="#0f213b" stroke="#1f3a5f"/>+ <circle cx="24" cy="21" r="6" fill="#38bdf8"/>+ <text x="40" y="28" class="t" font-weight="600" font-size="19" fill="#cbd5e1">Local Docker</text>+ </g>+ <g transform="translate(186,0)">+ <rect x="0" y="0" rx="22" ry="22" width="158" height="42" fill="#0f213b" stroke="#1f3a5f"/>+ <circle cx="24" cy="21" r="6" fill="#a78bfa"/>+ <text x="40" y="28" class="t" font-weight="600" font-size="19" fill="#cbd5e1">Claude skill</text>+ </g>+ <g transform="translate(360,0)">+ <rect x="0" y="0" rx="22" ry="22" width="158" height="42" fill="#0f213b" stroke="#1f3a5f"/>+ <circle cx="24" cy="21" r="6" fill="#34d399"/>+ <text x="40" y="28" class="t" font-weight="600" font-size="19" fill="#cbd5e1">Clean leads</text>+ </g>+ </g>++ <!-- footer url -->+ <text x="302" y="296" class="mono" font-size="17" fill="#64748b">github.com/Mahanaicoach/google-maps-scraper-kit</text>++ <!-- lead-list card (right side, no overlap with text) -->+ <g transform="translate(972,86)" filter="url(#shadow)">+ <rect x="0" y="0" rx="16" ry="16" width="268" height="188" fill="#0f1a30" stroke="#24324d"/>+ <text x="20" y="34" class="t" font-size="13" letter-spacing="2" fill="#64748b">LEAD LIST</text>+ <g>+ <circle cx="28" cy="72" r="7" fill="#34d399"/>+ <rect x="46" y="64" rx="5" width="150" height="14" fill="#334155"/>+ <rect x="46" y="84" rx="5" width="104" height="9" fill="#1f2c44"/>+ <circle cx="28" cy="116" r="7" fill="#34d399"/>+ <rect x="46" y="108" rx="5" width="120" height="14" fill="#334155"/>+ <rect x="46" y="128" rx="5" width="140" height="9" fill="#1f2c44"/>+ <circle cx="28" cy="160" r="7" fill="#34d399"/>+ <rect x="46" y="152" rx="5" width="160" height="14" fill="#334155"/>+ <rect x="46" y="172" rx="5" width="96" height="9" fill="#1f2c44"/>+ </g>+ </g>+</svg>diff --git a/docker-compose.yml b/docker-compose.ymlnew file mode 100644index 0000000..f090f2d--- /dev/null+++ b/docker-compose.yml@@ -0,0 +1,24 @@+# Runs the open-source Google Maps scraper (gosom/google-maps-scraper) locally.+# Web/API mode on http://localhost:8080. The image has NO built-in auth, so we bind+# to 127.0.0.1 ONLY — that localhost binding is the security boundary. Do not change+# the port mapping to 0.0.0.0 / a public interface without putting an auth proxy in front.++services:+ google-maps-scraper:+ # Pinned to a known-good release for reproducibility (bump deliberately).+ # The published image is linux/amd64. On Apple Silicon (M1/M2/M3) it runs under+ # emulation; if it won't start, UNCOMMENT the 'platform' line below.+ image: gosom/google-maps-scraper:v1.15.0+ # platform: linux/amd64+ container_name: gmaps-scraper+ restart: unless-stopped+ command: ["-web", "-data-folder", "/gmapsdata"]+ ports:+ - "127.0.0.1:8080:8080" # localhost only — see warning above+ volumes:+ - gmaps_data:/gmapsdata # job results persist here+ - gmaps_cache:/opt # Playwright browser cache (faster restarts)++volumes:+ gmaps_data:+ gmaps_cache:diff --git a/examples/queries.example.json b/examples/queries.example.jsonnew file mode 100644index 0000000..4103d40--- /dev/null+++ b/examples/queries.example.json@@ -0,0 +1,31 @@+{+ "_comment": "Reference request body for POST /api/v1/jobs. Copy the 'job' object and edit. Required fields: keywords, lat, lon (strings), max_time (seconds).",+ "job": {+ "name": "my scrape",+ "keywords": ["coffee shops in Austin TX"],+ "lang": "en",+ "zoom": 15,+ "lat": "30.2672",+ "lon": "-97.7431",+ "fast_mode": false,+ "radius": 10000,+ "depth": 5,+ "email": false,+ "max_time": 300,+ "proxies": null+ },+ "keyword_ideas": [+ "dentists in Denver CO",+ "plumbers in Phoenix AZ",+ "yoga studios in Portland OR",+ "barbershops in Brooklyn NY",+ "wedding photographers in Nashville TN"+ ],+ "city_coordinates_cheatsheet": {+ "Austin, TX": ["30.2672", "-97.7431"],+ "Miami, FL": ["25.7617", "-80.1918"],+ "Denver, CO": ["39.7392", "-104.9903"],+ "New York, NY": ["40.7128", "-74.0060"],+ "Los Angeles, CA": ["34.0522", "-118.2437"]+ }+}diff --git a/examples/queries.example.txt b/examples/queries.example.txtnew file mode 100644index 0000000..2ba52c5--- /dev/null+++ b/examples/queries.example.txt@@ -0,0 +1,7 @@+# Batch keywords — one per line. Lines starting with # are ignored.+# Bake the location into each line, OR pass --city to geocode one place for all of them.+# Run: python3 scripts/scrape.py --keywords-file examples/queries.example.txt --city "Denver, CO"+dentists in Denver CO+orthodontists in Denver CO+dental hygienists in Denver CO+pediatric dentists in Denver COdiff --git a/scripts/scrape.py b/scripts/scrape.pynew file mode 100755index 0000000..10bd964--- /dev/null+++ b/scripts/scrape.py@@ -0,0 +1,163 @@+#!/usr/bin/env python3+"""One-shot Google Maps scrape using only the Python standard library (no pip installs).++Single keyword:+ python3 scripts/scrape.py "gyms in Miami FL" 25.7617 -80.1918 --depth 5++Auto-geocode (no coordinates needed):+ python3 scripts/scrape.py "cafes in Austin TX" --city "Austin, TX" --depth 5++Batch (one job, many keywords) from a file (one keyword per line):+ python3 scripts/scrape.py --keywords-file examples/queries.example.txt --city "Denver, CO"++Notes:+- Required by the API: keywords, lat/lon (strings), max_time (SECONDS). This script fills them in.+- Geocoding uses OpenStreetMap Nominatim (free, no key). Please be gentle: it allows ~1 request/sec+ and requires a descriptive User-Agent (set below). Don't loop it aggressively.+"""+import argparse, csv, io, json, os, sys, time, urllib.request, urllib.parse, urllib.error++BASE = os.environ.get("SCRAPER_BASE_URL", "http://localhost:8080")+KEY = os.environ.get("SCRAPER_API_KEY", "")+# Money-useful LEAD fields only — what you actually use to contact/qualify a lead.+# Everything else (geo coordinates, IDs, hours, images, reviews blobs…) is dropped by default.+LEAD = ["title", "phone", "emails", "website", "category", "address", "review_rating", "review_count"]+UA = "google-maps-scraper-kit/1.0 (https://github.com/Mahanaicoach/google-maps-scraper-kit)"+++def req(method, path, body=None):+ headers = {"Content-Type": "application/json", "User-Agent": UA}+ if KEY:+ headers["X-API-Key"] = KEY+ data = json.dumps(body).encode() if body is not None else None+ r = urllib.request.Request(BASE + path, data=data, headers=headers, method=method)+ with urllib.request.urlopen(r, timeout=60) as resp:+ return resp.status, resp.read()+++def geocode(place):+ """City/place name -> ('lat','lon') strings via Nominatim, or None."""+ q = urllib.parse.urlencode({"format": "json", "limit": 1, "q": place})+ url = f"https://nominatim.openstreetmap.org/search?{q}"+ r = urllib.request.Request(url, headers={"User-Agent": UA})+ try:+ with urllib.request.urlopen(r, timeout=30) as resp:+ hits = json.loads(resp.read())+ time.sleep(1) # respect Nominatim's ~1 req/sec policy+ if hits:+ return str(hits[0]["lat"]), str(hits[0]["lon"])+ except Exception as e:+ print(f" (geocoding failed: {e})", file=sys.stderr)+ return None+++def collect_keywords(a):+ kws = []+ if a.keyword:+ kws.append(a.keyword)+ kws.extend(a.also or [])+ if a.keywords_file:+ with open(a.keywords_file) as f:+ kws.extend(line.strip() for line in f if line.strip() and not line.startswith("#"))+ # de-dupe, keep order+ seen, out = set(), []+ for k in kws:+ if k not in seen:+ seen.add(k); out.append(k)+ return out+++def main():+ ap = argparse.ArgumentParser(description="Scrape Google Maps business listings.")+ ap.add_argument("keyword", nargs="?", help='e.g. "coffee shops in Austin TX"')+ ap.add_argument("lat", nargs="?", help="latitude (optional if --city given)")+ ap.add_argument("lon", nargs="?", help="longitude (optional if --city given)")+ ap.add_argument("--keyword", dest="also", action="append", help="extra keyword (repeatable)")+ ap.add_argument("--keywords-file", help="file with one keyword per line (batch)")+ ap.add_argument("--city", help="city/place to auto-geocode for lat/lon")+ ap.add_argument("--depth", type=int, default=5)+ ap.add_argument("--email", action="store_true", help="also extract emails (much slower)")+ ap.add_argument("--max-time", type=int, default=600, help="job time limit in SECONDS")+ ap.add_argument("--out", default=None, help="write parsed JSON results here")+ ap.add_argument("--full", action="store_true", help="keep ALL raw columns (default: lean lead fields)")+ ap.add_argument("--fields", help="comma-separated columns to keep (overrides the default lead set)")+ a = ap.parse_args()++ keywords = collect_keywords(a)+ if not keywords:+ sys.exit("✗ No keywords. Pass a keyword, --keyword, or --keywords-file.")++ # Resolve coordinates: explicit lat/lon, else geocode --city, else geocode first keyword.+ lat, lon = a.lat, a.lon+ if not (lat and lon):+ place = a.city or keywords[0]+ print(f"▶ Geocoding \"{place}\"…")+ coords = geocode(place)+ if not coords:+ sys.exit("✗ Could not resolve coordinates. Pass lat/lon or a clearer --city.")+ lat, lon = coords+ print(f" → {lat}, {lon}")++ # --- WARN, don't block: flag high-volume jobs (proceed anyway) ---+ if a.depth >= 15 or len(keywords) >= 10 or a.email:+ print("⚠️ Heads-up: this is a large/slow job (high depth, many keywords, or email extraction).")+ print(" Running it back-to-back without proxies can get your IP temporarily rate-limited by")+ print(" Google (clears in minutes–hours; returns empty/failed jobs meanwhile). For big or")+ print(" repeated runs, add proxies. Proceeding…\n")++ # Health check+ try:+ req("GET", "/api/v1/jobs")+ except Exception as e:+ sys.exit(f"✗ Scraper not reachable at {BASE} — run 'docker compose up -d' first.\n ({e})")++ body = {"name": "scrape-py", "keywords": keywords, "lang": "en", "zoom": 15,+ "lat": str(lat), "lon": str(lon), "fast_mode": False, "radius": 10000,+ "depth": a.depth, "email": a.email, "max_time": a.max_time}+ print(f"▶ Creating job: {len(keywords)} keyword(s) @ {lat},{lon} depth={a.depth} email={a.email}")+ for k in keywords:+ print(f" • {k}")+ try:+ _, raw = req("POST", "/api/v1/jobs", body)+ except urllib.error.HTTPError as e:+ sys.exit(f"✗ Create failed: HTTP {e.code} — {e.read().decode()[:200]}")+ job_id = json.loads(raw).get("id")+ if not job_id:+ sys.exit("✗ No job id returned.")+ print(f" job id: {job_id}")++ print("▶ Polling…")+ status = None+ for i in range(120):+ _, raw = req("GET", f"/api/v1/jobs/{job_id}")+ status = json.loads(raw).get("Status")+ print(f"\r status: {str(status):<10} (attempt {i + 1})", end="", flush=True)+ if status == "ok":+ print(); break+ if status == "failed":+ sys.exit("\n✗ Job failed. If this keeps happening you may be rate-limited — wait or add proxies.")+ time.sleep(8)+ else:+ sys.exit("\n✗ Timed out.")++ _, raw = req("GET", f"/api/v1/jobs/{job_id}/download")+ rows = list(csv.DictReader(io.StringIO(raw.decode("utf-8", "replace"))))+ if a.full:+ fields = list(rows[0].keys()) if rows else LEAD+ elif a.fields:+ fields = [c.strip() for c in a.fields.split(",") if c.strip()]+ else:+ fields = LEAD+ results = [{k: r.get(k, "") for k in fields} for r in rows]+ print(f"✓ Done — {len(results)} businesses (fields: {', '.join(fields)}).")++ out = a.out or f"results-{job_id[:8]}.json"+ with open(out, "w") as f:+ json.dump(results, f, indent=2, ensure_ascii=False)+ print(f" saved → {out}")+ for r in results[:5]:+ print(f" • {r.get('title','')} | {r.get('phone','')} | {r.get('emails','')} | {r.get('website','')}")+++if __name__ == "__main__":+ main()diff --git a/scripts/scrape.sh b/scripts/scrape.shnew file mode 100755index 0000000..a1e633b--- /dev/null+++ b/scripts/scrape.sh@@ -0,0 +1,69 @@+#!/usr/bin/env bash+# One-shot Google Maps scrape: create job -> poll -> download -> summarize.+# Usage: ./scripts/scrape.sh "<keyword>" <lat> <lon> [depth]+# Example: ./scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5+set -euo pipefail++BASE="${SCRAPER_BASE_URL:-http://localhost:8080}"+KEY="${SCRAPER_API_KEY:-}"+AUTH=(); [ -n "$KEY" ] && AUTH=(-H "X-API-Key: $KEY")++KEYWORD="${1:-}"; LAT="${2:-}"; LON="${3:-}"; DEPTH="${4:-5}"+if [ -z "$KEYWORD" ] || [ -z "$LAT" ] || [ -z "$LON" ]; then+ echo "Usage: $0 \"<keyword>\" <lat> <lon> [depth]" >&2+ echo "Example: $0 \"coffee shops in Austin TX\" 30.2672 -97.7431 5" >&2+ exit 1+fi++# Is the scraper up?+if ! curl -s -m 5 "${AUTH[@]}" "$BASE/api/v1/jobs" >/dev/null 2>&1; then+ echo "✗ Scraper not reachable at $BASE — run 'docker compose up -d' first." >&2+ exit 1+fi++echo "▶ Creating job: \"$KEYWORD\" @ $LAT,$LON depth=$DEPTH"+BODY=$(printf '{"name":"scrape-cli","keywords":["%s"],"lang":"en","zoom":15,"lat":"%s","lon":"%s","fast_mode":false,"radius":10000,"depth":%s,"email":false,"max_time":600}' \+ "$KEYWORD" "$LAT" "$LON" "$DEPTH")+ID=$(curl -s -X POST "$BASE/api/v1/jobs" "${AUTH[@]}" -H "Content-Type: application/json" -d "$BODY" \+ | grep -oE '[a-f0-9-]{36}' | head -1 || true)+[ -n "$ID" ] || { echo "✗ Failed to create job (check required fields)." >&2; exit 1; }+echo " job id: $ID"++echo "▶ Polling (up to ~10 min)…"+for i in $(seq 1 60); do+ # '|| true' keeps the script alive under 'set -euo pipefail' when grep finds no match yet+ S=$(curl -s "${AUTH[@]}" "$BASE/api/v1/jobs/$ID" | grep -oE '"Status":"[^"]*"' | head -1 | cut -d'"' -f4 || true)+ printf '\r status: %-10s (attempt %d)' "${S:-?}" "$i"+ [ "$S" = ok ] && { echo; break; }+ [ "$S" = failed ] && { echo; echo "✗ Job failed." >&2; exit 1; }+ sleep 10+done+[ "$S" = ok ] || { echo; echo "✗ Timed out waiting for job." >&2; exit 1; }++OUT="results-$(echo "$KEYWORD" | tr ' /' '__' | tr -cd 'A-Za-z0-9_-').csv"+RAW="$(mktemp)"+echo "▶ Downloading…"+curl -s "${AUTH[@]}" "$BASE/api/v1/jobs/$ID/download" -o "$RAW"++# Trim the raw 34-column dump down to money-useful LEAD fields only (drops geo coords, IDs, hours, images…).+if command -v python3 >/dev/null 2>&1; then+ python3 - "$RAW" "$OUT" <<'PY'+import csv, sys+LEAD = ["title", "phone", "emails", "website", "category", "address", "review_rating", "review_count"]+with open(sys.argv[1], newline='') as f:+ rows = list(csv.DictReader(f))+with open(sys.argv[2], "w", newline='') as f:+ w = csv.DictWriter(f, fieldnames=LEAD); w.writeheader()+ for r in rows:+ w.writerow({k: r.get(k, "") for k in LEAD})+print(f"✓ Done — {len(rows)} leads saved to {sys.argv[2]}")+print(" (name, phone, email, website, category, address, rating, reviews)\n")+print("Preview (name | phone | email | website):")+for r in rows[:6]:+ print(f" • {r.get('title','')} | {r.get('phone','')} | {r.get('emails','')} | {r.get('website','')}")+PY+ rm -f "$RAW"+else+ mv "$RAW" "$OUT"+ echo "✓ Done — raw CSV saved to $OUT (install python3 to auto-trim to lead fields only)"+fi