Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
docs: one voice across setup, skill, commands, policies and config comments
Mahanaicoachc836464841bc
diff --git a/.claude/commands/scrape-batch.md b/.claude/commands/scrape-batch.mdindex 41aaf25..ea5525b 100644--- a/.claude/commands/scrape-batch.md+++ b/.claude/commands/scrape-batch.md@@ -1,23 +1,23 @@----description: Scrape many Google Maps queries at once (batch) from a list of keywords+description: Scrape many Google Maps keywords from a file in one job and return one lead CSV.argument-hint: <keywords-file> [--city "City, ST"]----The user wants a **batch** Google Maps scrape. Input: **$ARGUMENTS** (a keywords file, optionally a city).+The user wants a **batch** Google Maps scrape. Input: **$ARGUMENTS** (a keywords file, and optionally a city).-Use the `google-maps-scraper` skill. The API scrapes **many keywords in ONE job** (the `keywords` array),-so batch = one job with all terms.+Use the `google-maps-scraper` skill. The API scrapes **many keywords in ONE job** through the `keywords`+array, so a batch is one job with all the terms.-1. Ensure the scraper is up (`curl -s http://localhost:8080/api/v1/jobs`; `docker compose up -d` if not).-2. Read the keyword list (one per line; ignore blank / `#` lines). If a `--city` was given, geocode it once- for `lat`/`lon`; otherwise expect the location baked into each keyword (geocode the first one).-3. **ASK about social profiles first** (skip only if the user already said yes/no): *"Also grab social profiles- (Instagram / Facebook / LinkedIn) for these? Emails are already included. (yes / no)"* — if yes, add `--socials`.-4. **Warn, don't block:** a batch with many keywords is high-volume — print ONE short warning that this can- get the IP temporarily rate-limited and suggest proxies for large/repeated runs, then **proceed**.-5. POST a single job with the full `keywords` array + required fields (`max_time` seconds, string `lat`/`lon`)- and **`"email": true`** (emails on — the key lead field). Use a generous `max_time` (e.g. `900`) since many- keywords + email extraction take longer. Poll in the BACKGROUND, download.-6. Present results grouped/clean and save the full file to disk.+1. Check the engine is up (`curl -s http://localhost:8080/api/v1/jobs`; `docker compose up -d` if not).+2. Read the keyword list (one per line; ignore blank and `#` lines). If a `--city` was given, geocode it once+ for `lat`/`lon`. Otherwise expect the location baked into each keyword, and geocode the first one.+3. **Ask about social profiles first.** Skip only if the user already said yes or no: *"Also grab social profiles+ (Instagram / Facebook / LinkedIn) for these? Emails are already included. (yes / no)"*. If yes, add `--socials`.+4. **Warn, don't block:** a batch with many keywords is high-volume. Print ONE short warning that it can+ get the IP temporarily rate-limited and suggest proxies for large or repeated runs, then **proceed**.+5. POST a single job with the full `keywords` array, the required fields (`max_time` seconds, string `lat`/`lon`)+ and **`"email": true`** (emails are the key lead field). Use a generous `max_time` (e.g. `900`), because many+ keywords plus email extraction take longer. Poll in the background, then download.+6. Present the 8 lead fields in a clean table and save the full file to disk.-Tip: `scripts/scrape.py --keywords-file <file> --city "<City, ST>"` already does all of this — emails are on by-default; append `--socials` if the user said yes in step 3.+Tip: `scripts/scrape.py --keywords-file <file> --city "<City, ST>"` already does all of this, with emails on by+default. Append `--socials` if the user said yes in step 3.diff --git a/.claude/commands/scrape-jobs.md b/.claude/commands/scrape-jobs.mdindex d4142cc..f0357c2 100644--- a/.claude/commands/scrape-jobs.md+++ b/.claude/commands/scrape-jobs.md@@ -1,11 +1,11 @@----description: List or clean up Google Maps scrape jobs+description: List, download or delete Google Maps scrape jobs on the local engine.argument-hint: "[list | delete <job-id>]"----Manage scrape jobs via the API. Request: **$ARGUMENTS** (default to "list" if empty).+Manage scrape jobs through the engine's API. Request: **$ARGUMENTS** (default to "list" if empty).-- **list** → `GET http://localhost:8080/api/v1/jobs`. Show a tidy table of `ID`, `Name`, `Status`, `Date` (newest first).-- **delete <job-id>** → `DELETE http://localhost:8080/api/v1/jobs/<job-id>` to free disk space, then confirm it returned HTTP 200.-- **download <job-id>** → `GET .../jobs/<job-id>/download` → save CSV and summarize row count.+- **list**: `GET http://localhost:8080/api/v1/jobs`. Show a tidy table of `ID`, `Name`, `Status`, `Date` (newest first).+- **delete <job-id>**: `DELETE http://localhost:8080/api/v1/jobs/<job-id>` to free disk space, then confirm it returned HTTP 200.+- **download <job-id>**: `GET .../jobs/<job-id>/download`, then save the CSV and report the row count.-See the `google-maps-scraper` skill for the full API reference.+The `google-maps-scraper` skill has the full API reference.diff --git a/.claude/commands/scrape-setup.md b/.claude/commands/scrape-setup.mdindex 66f78ba..71d47dd 100644--- a/.claude/commands/scrape-setup.md+++ b/.claude/commands/scrape-setup.md@@ -1,18 +1,18 @@----description: Set up and start the local Google Maps scraper (Docker)+description: Start the local Google Maps scraper engine in Docker and run a health check.----Set up the Google Maps scraper on this computer:+Set up the kit's engine on this computer:-1. Confirm Docker is installed and running: `docker ps`. If it errors, tell the user to install/start **Docker Desktop** (link in `SETUP.md` §0) and stop here.-2. From the kit root, start the scraper: `docker compose up -d`.-3. Wait ~10s, then verify it's live: `curl http://localhost:8080/api/v1/jobs` should return a JSON array (often `[]`).-4. Report success and tell the user they can now run `/scrape <business> in <city>` — or just ask in natural language (e.g. "scrape gyms in Miami").-5. **Show this over-use warning** (always print it once setup succeeds):- > ⚠️ **Before you scrape:** this hits Google Maps for real. Light use is fine, but running big jobs- > back-to-back, very high `depth`, many keywords, or scheduled runs **without proxies** can get your- > IP **temporarily rate-limited/blocked by Google** (usually clears in minutes–hours; jobs start- > returning empty/failed meanwhile). It won't ban your Google account. Stay safe: one job at a time,- > start at `depth 5`, and add **proxies** for large or repeated runs. Scraping Maps is also against- > Google's ToS — use responsibly and follow data laws (GDPR/CCPA) for any contact info.+1. Confirm Docker is installed and running: `docker ps`. If it errors, tell the user to install or start **Docker Desktop** (link in `SETUP.md` §0) and stop here.+2. From the kit root, start the engine: `docker compose up -d`.+3. Wait ~10s, then verify it is live: `curl http://localhost:8080/api/v1/jobs` should return a JSON array (often `[]`).+4. Report success. Tell the user they can now run `/scrape <business> in <city>`, or ask in plain language (e.g. "scrape gyms in Miami").+5. **Show this warning once setup succeeds:**+ > **Before you scrape:** the engine sends real requests to Google Maps. Light use is fine. Big jobs+ > back to back, high `depth`, many keywords or scheduled runs without proxies can get your IP+ > temporarily rate-limited by Google, usually for minutes to hours. Your Google account is not+ > affected. Run one job at a time, start at `depth 5`, and add proxies for large or repeated runs.+ > Scraping Google Maps is against Google's Terms of Service, and contact details are personal data,+ > so follow GDPR, CCPA and CAN-SPAM.-If a step fails, consult `SETUP.md` §6 (Troubleshooting).+If a step fails, see `SETUP.md` §6 (Troubleshooting).diff --git a/.claude/commands/scrape.md b/.claude/commands/scrape.mdindex 107d06f..3bbd475 100644--- a/.claude/commands/scrape.md+++ b/.claude/commands/scrape.md@@ -1,29 +1,25 @@----description: Scrape Google Maps business listings for a query + city (drives the google-maps-scraper skill)+description: Scrape Google Maps business listings for one business type in one city and return the 8 lead fields as a table and CSV.argument-hint: <business type> in <city, ST> [depth]----The user wants to scrape Google Maps business listings. Query: **$ARGUMENTS**+The user wants a Google Maps lead list. Query: **$ARGUMENTS**-Use the `google-maps-scraper` skill. Execute:+Use the `google-maps-scraper` skill and follow these steps:-1. **Health-check** the API: `curl -s http://localhost:8080/api/v1/jobs`. If it's down, run `docker compose up -d` from the kit root, wait ~10s, and retry.+1. **Check the engine:** `curl -s http://localhost:8080/api/v1/jobs`. If it is down, run `docker compose up -d` from the kit root, wait ~10s, and retry.2. **Parse** the business type and city from the query. Look up the city's **latitude/longitude** (as strings).-3. **ASK about social profiles — always, before scraping.** Ask the user ONE short question:+3. **Ask about social profiles before scraping, every time.** Ask the user ONE short question:*"Want me to also grab their social profiles (Instagram / Facebook / LinkedIn)? Emails are already included by default. (yes / no)"*- Wait for the answer. **Skip the question only if** the user already said yes/no to socials in their request.- If **yes** → add `--socials` in step 4. If **no** → leave it off. (Socials add no LLM-token cost and one extra HTTP fetch per site — see the skill.)-4. **Create the job with EMAIL EXTRACTION ON** (`"email": true` in the body — emails are the #1 lead field). Required fields — `keywords` (location baked into each term), `lat`/`lon` (strings), `max_time` (seconds, default `300`), `depth` (the number in the query, else `5`). Easiest path: `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --depth <n>` (emails are already on by default; append `--socials` if the user said yes in step 3).-5. **Poll to completion in the BACKGROUND** (the Bash tool blocks foreground `sleep` — use `run_in_background: true` and read the output file when notified). Then download the CSV.-6. **Save a CSV file** (the default — `scripts/scrape.py` writes `results-<id>.csv`; if you use the raw API, save the parsed lead fields to a `.csv` yourself). Then **present ONLY money-useful lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count` (+ `instagram, facebook, linkedin` if socials were requested). **Drop everything else** — especially `latitude`/`longitude`, IDs, hours, images, review blobs. Do NOT dump the full 34-column CSV into chat; show a few sample rows and tell the user the CSV path. (Include dropped fields only if the user explicitly asks.)+ Wait for the answer. **Skip the question only if** the user already said yes or no to socials in the request.+ If **yes**, add `--socials` in step 4. If **no**, leave it off. Socials cost no LLM tokens and one extra HTTP fetch per site (see the skill).+4. **Create the job with email extraction on** (`"email": true` in the body; emails are the most valuable lead field). Required fields: `keywords` (location baked into each term), `lat`/`lon` (strings), `max_time` (seconds, default `300`), `depth` (the number in the query, else `5`). Easiest path: `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --depth <n>` (emails are already on by default; append `--socials` if the user said yes in step 3).+5. **Poll to completion in the background.** The Bash tool blocks foreground `sleep`, so use `run_in_background: true` and read the output file when notified. Then download the CSV.+6. **Save a CSV file.** `scripts/scrape.py` writes `results-<id>.csv` by default. If you use the raw API, save the parsed lead fields to a `.csv` yourself. Then **present only the 8 lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count`, plus `instagram, facebook, linkedin` if socials were requested. **Drop everything else**, especially `latitude`/`longitude`, IDs, hours, images and review blobs. Never dump the full 34-column CSV into chat. Show a few sample rows and give the CSV path. Include dropped fields only if the user explicitly asks.-**All / every / entire city:** add `--grid 3` (3 x 3 map tiles, one job at a time, deduped) to get past Google's ~120 results per search. Warn that it is slower and more block-prone, then proceed.+**All, every or entire city:** add `--grid 3` (3 x 3 map tiles, one job at a time, deduped) to get past Google's ~120 results per search. Warn once that it is slower and more likely to trip rate limits, then proceed.-**Emails:** on by default in this kit — the scraper visits each business website to find them, which is a bit slower. Some businesses simply don't publish an email, so coverage is partial. For a deliberately fast run with no emails, pass `--no-email`.+**Emails:** on by default. The engine visits each business website to find them, which is a little slower. Some businesses publish no email, so coverage is partial. For a deliberately fast run without emails, pass `--no-email`.-**Socials:** when the user says yes, `scripts/scrape.py … --socials` pulls Instagram/Facebook/LinkedIn. Extraction is **0 LLM tokens** (runs in code); it only adds ~40–50 tokens/business if you load rows into chat. **Never** WebFetch each website yourself to find socials — that costs thousands of tokens.+**Socials:** when the user says yes, `scripts/scrape.py … --socials` adds Instagram, Facebook and LinkedIn. Extraction costs **0 LLM tokens** because it runs in code. It adds ~40–50 tokens per business only if you load the rows into chat. **Never** WebFetch each website yourself to find socials; that costs thousands of tokens.-**Warn, don't block:** if the request is high-volume (high `depth`, many keywords, or repeated runs),-print ONE short warning that over-use can get their IP temporarily rate-limited by Google and suggest-adding proxies — then **proceed with the scrape anyway**. Do not gate or demand confirmation just for a-big job. Only refuse outright for clearly abusive use (surveilling individuals, spam/harassment). Also-remind the user about PII / Google ToS if they'll store or contact the data.+**Warn, don't block:** if the request is high-volume (high `depth`, many keywords or repeated runs), print ONE short warning that heavy use can get the user's IP temporarily rate-limited by Google, and suggest proxies. Then **proceed with the scrape**. Do not gate or demand confirmation for a big job. Refuse outright only for clearly abusive use (surveilling individuals, spam, harassment). If the user will store or contact the data, remind them once that it is personal data and that scraping is against Google's Terms of Service.diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdindex 69ba9ea..05891da 100644--- a/.claude/skills/google-maps-scraper/SKILL.md+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -1,35 +1,38 @@---name: google-maps-scraper-description: Scrape Google Maps business listings (name, address, phone, website, rating, reviews, lat/lng, hours, emails) via the local gosom google-maps-scraper REST API. Use when the user wants local-business / lead-gen data, "a list of [businesses] in [place]", or to enrich places with contact info. NOT for Instagram/TikTok/YouTube or any social-media scraping.+description: Google Maps scraper for lead generation. Turns a business type and a place into a clean CSV of local businesses by driving the gosom/google-maps-scraper engine, running locally in Docker as a REST API on localhost:8080. Returns 8 lead fields by default (name, phone, email, website, category, address, rating and review count), plus optional Instagram, Facebook and LinkedIn profiles or all ~34 raw fields. Use when the user wants a lead list, "a list of [businesses] in [place]", local business data, contact enrichment for places (emails, phones, socials), or a whole-city run past Google's ~120 results per search. Do not use for scraping Instagram, TikTok, YouTube, X or any other social media platform directly.---# Google Maps Scraper-Drive the local Google Maps scraper API to turn a business-type + location into clean, structured rows.+Turn a business type and a location into clean lead rows by driving the local engine's job API.## Mental model-The scraper runs as a local Docker container exposing a REST API at `http://localhost:8080` (no auth —-localhost only). A "scrape" is an **async job**: you create it, poll until it's done, then download a CSV.-One job can run many keywords. Each result has up to **34 fields**.+The engine (gosom/google-maps-scraper) runs as a local Docker container with a REST API at+`http://localhost:8080`. It has no auth, so it stays bound to localhost. A scrape is an **async job**:+create it, poll until it is done, then download a CSV. One job can run many keywords. Each result has+up to **34 raw fields**.-## Step 0 — Make sure it's running+## Step 0: Check the engine is running```bashcurl -s http://localhost:8080/api/v1/jobs >/dev/null 2>&1 && echo UP || echo DOWN```-If `DOWN`: `docker compose up -d` (from the kit root), wait ~10s, retry. If Docker isn't installed, point-the user to `SETUP.md`.+If `DOWN`: run `docker compose up -d` from the kit root, wait ~10s, and retry. If Docker is not+installed, point the user to `SETUP.md`.-## Step 1 — Create a job (`POST /api/v1/jobs`)-**Required fields — the API returns `422` without them:**-- `keywords` — array of search strings. **Bake the location into each term**: `"plumbers in Denver CO"`.-- `lat`, `lon` — **strings**, the city's coordinates: `"39.7392"`, `"-104.9903"`.-- `max_time` — integer **seconds** (max wall-clock for the job), e.g. `300`. (Sent as seconds; the API stores it as nanoseconds internally — just send seconds.)+## Step 1: Create a job (`POST /api/v1/jobs`)+**Required fields. The API returns `422` without them:**+- `keywords`: array of search strings. **Bake the location into each term**: `"plumbers in Denver CO"`.+- `lat`, `lon`: **strings**, the city's coordinates: `"39.7392"`, `"-104.9903"`.+- `max_time`: integer **seconds**, the job's wall-clock limit, e.g. `300`. The API stores it as+ nanoseconds internally, but you always send seconds.**Recommended fields:**-- `depth` (default 10) — how far to scroll → roughly how many listings per keyword. Start at `5`.+- `depth` (default 10): how far to scroll, so roughly how many listings per keyword. Start at `5`.- `lang` `"en"`, `zoom` `15` (city level), `radius` `10000` (meters), `fast_mode` `false`.-- **`email` `true` — ON by default in this kit.** Emails are the #1 lead field; the scraper visits each- business website to find them (a bit slower). Only set `false` for a deliberately fast, no-email run.+- **`email` `true`: on by default in the kit.** Emails are the most valuable lead field. The engine+ visits each business website to find them, which is a little slower. Set `false` only for a+ deliberately fast run without emails.```bashcurl -s -X POST http://localhost:8080/api/v1/jobs \@@ -40,19 +43,19 @@ curl -s -X POST http://localhost:8080/api/v1/jobs \# → {"id":"<uuid>"} (HTTP 201; note: lowercase "id")```-> **Two things to handle on every scrape:**-> 1. **Emails: on by default** (`email:true`). Don't turn them off unless the user wants a fast run.-> 2. **Socials: ASK first.** Before creating the job, ask the user once whether they also want-> Instagram/Facebook/LinkedIn (see "Social profiles" below). Don't silently skip it.+> **Handle two things on every scrape:**+> 1. **Emails: on by default** (`email:true`). Turn them off only if the user wants a fast run.+> 2. **Socials: ask first.** Before creating the job, ask the user once whether they also want+> Instagram, Facebook and LinkedIn profiles (see "Social profiles" below). Never skip the question silently.-## Step 2 — Poll until done (`GET /api/v1/jobs/{id}`)-The response field is `"Status"` (capital S): `working` → `ok` (success) or `failed`.+## Step 2: Poll until done (`GET /api/v1/jobs/{id}`)+The response field is `"Status"` (capital S). It moves from `working` to `ok` (success) or `failed`.-> ⚠️ **Claude harness rule:** the Bash tool **blocks foreground `sleep`**. Run the poll loop as a+> **Harness rule:** the Bash tool **blocks foreground `sleep`**. Run the poll loop as a> **background** Bash command (`run_in_background: true`) and read its output file when notified.-> Do NOT poll with a foreground `sleep`.+> Never poll with a foreground `sleep`.-Background poll snippet (parses the value safely — match up to the closing quote, don't anchor on `$`):+Background poll snippet. It parses the value safely: match up to the closing quote, do not anchor on `$`.```bashID="<uuid>"for i in $(seq 1 40); do@@ -64,7 +67,7 @@ for i in $(seq 1 40); dodone```-## Step 3 — Download + parse (`GET /api/v1/jobs/{id}/download`)+## Step 3: Download and parse (`GET /api/v1/jobs/{id}/download`)```bashcurl -s "http://localhost:8080/api/v1/jobs/$ID/download" -o results.csv```@@ -73,103 +76,108 @@ plus_code, review_count, review_rating, reviews_per_rating, latitude, longitude,descriptions, reviews_link, thumbnail, timezone, price_range, data_id, place_id, images, reservations,order_online, menu, owner, complete_address, about, user_reviews, user_reviews_extended, emails`.-### ⭐ Output ONLY money-useful lead fields (default)-The raw CSV has 34 columns and most are noise. **By default, return ONLY these lead fields** — the data you-actually use to contact and qualify a lead — and **drop everything else**:+### Return only the 8 lead fields (default)+The raw CSV has 34 columns and most are noise for outreach. **By default, return only the 8 lead+fields**, the data used to contact and qualify a lead, and **drop everything else**:> `title` (name), `phone`, `emails`, `website`, `category`, `address`, `review_rating`, `review_count`-**DROP by default** (do not show these unless the user explicitly asks): `latitude`/`longitude` (no use for+**Drop by default.** Show these only if the user explicitly asks: `latitude`/`longitude` (no use foroutreach), `link`, `plus_code`, `cid`, `data_id`, `place_id`, `open_hours`, `popular_times`,`reviews_per_rating`, `reviews_link`, `thumbnail`, `images`, `timezone`, `price_range`, `status`,`input_id`, `complete_address`, `reservations`, `order_online`, `menu`, `owner`, `about`, `descriptions`,`user_reviews`, `user_reviews_extended`.-`scripts/scrape.py` already returns exactly this lead set (use `--full` to keep all columns, or-`--fields "a,b,c"` to customize) and **saves a CSV file by default** (`results-<id>.csv`; pass `--json` for-JSON). If you call the API directly, **strip to the lead fields yourself** before presenting — never dump the-full 34-column row at the user.--### Social profiles — ALWAYS ASK the user (Instagram / Facebook / LinkedIn)-Google Maps has no social links, so this is an enrichment: visit each business's `website` and regex out its-IG/FB/LinkedIn URLs. **Before scraping, ask the user once** whether they want socials too (unless they already-said). If yes → **use the script:** `python3 scripts/scrape.py … --socials`.--- **Token cost — say this to the user when they ask for socials:** the extraction itself is **0 LLM tokens**- (pure HTTP + regex in the script). It only adds **~40–50 tokens per business** to *your* context **if** you- load the rows into chat — e.g. ~+2k tokens for 50 leads. Negligible if you keep the file on disk and show a- sample. Save the file; show a few rows.-- **DO NOT** fetch each website yourself with WebFetch to find socials — that reads every page into your- context and costs **thousands of tokens**. The script does it for free. Always prefer `--socials`.-- It's **opt-in/slower** (one HTTP fetch per business) and coverage is partial (~40–70%: only businesses that- link socials on their site; no website → no socials). Mention this if the user expects 100%.--**Shortcuts (prefer these for common cases):**-- One keyword: `scripts/scrape.sh "<keyword>" <lat> <lon> [depth]` (bash) — create→poll→download in one go.-- Auto-geocode (no coords): `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N]` — resolves- lat/lon via OpenStreetMap Nominatim. You can also geocode the city yourself and pass coords.- **Emails come back by default** (use `--no-email` to skip); add `--socials` if the user asked for socials.-- **Batch (many keywords, ONE job):** `python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>"`.+`scripts/scrape.py` already returns exactly the 8 lead fields. Use `--full` to keep all columns, or+`--fields "a,b,c"` to choose your own. It **saves a CSV file by default** (`results-<id>.csv`); pass+`--json` for JSON. If you call the API directly, **strip to the lead fields yourself** before+presenting. Never dump the full 34-column row at the user.++### Social profiles: always ask the user (Instagram, Facebook, LinkedIn)+Google Maps has no social links, so socials are an enrichment step: visit each business `website` and+regex out its Instagram, Facebook and LinkedIn URLs. **Before scraping, ask the user once** whether they+want socials too, unless they already said. If yes, **use the script:** `python3 scripts/scrape.py … --socials`.++- **Token cost. Tell the user this when they ask for socials:** the extraction itself costs **0 LLM+ tokens**, because it is plain HTTP and regex in the script. It adds **~40–50 tokens per business** to+ *your* context only **if** you load the rows into chat, e.g. ~+2k tokens for 50 leads. That is+ negligible if you keep the file on disk and show a sample. Save the file and show a few rows.+- **Do not** fetch each website yourself with WebFetch to find socials. That reads every page into your+ context and costs **thousands of tokens**. The script does it for free, so always prefer `--socials`.+- Socials are **opt-in and slower** (one HTTP fetch per business), and coverage is partial (~40–70%).+ Only businesses that link their socials on their site have them, and no website means no socials.+ Say so if the user expects 100%.++**Shortcuts. Prefer these for common cases:**+- One keyword: `scripts/scrape.sh "<keyword>" <lat> <lon> [depth]` (bash) creates, polls and downloads in one go.+- Auto-geocode (no coordinates): `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" [--depth N]`+ resolves lat/lon via OpenStreetMap Nominatim. You can also geocode the city yourself and pass coordinates.+ **Emails come back by default** (`--no-email` skips them). Add `--socials` if the user asked for socials.+- **Batch (many keywords, one job):** `python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>"`.The API takes a `keywords` array, so put all terms in a single job rather than firing many jobs.-Do the manual curl flow only for custom job bodies (e.g. setting `proxies`).+Use the manual curl flow only for custom job bodies, e.g. to set `proxies`.-**Whole-city runs:** when the user says all / every / entire city, or a run comes back with about 120-rows (Google's per-search cap), use `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --grid 3`.-It splits the city into 3 x 3 tiles and dedupes on `place_id`. Cells run one at a time, so it is slower-and more block-prone: warn once, don't block.-**Deep emails** are on by default whenever emails are on: rows with a website but no email get their-contact/about pages checked, and the script prints the found rate. `--no-deep-emails` skips that pass.+**Whole-city runs:** when the user says all, every or entire city, or a run returns about 120 rows+(Google's per-search cap), use `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --grid 3`.+It splits the city into 3 x 3 tiles (9 jobs) and dedupes on `place_id`. Tiles run one at a time, so a+grid is slower and more likely to trip rate limits: warn once, don't block.++**Deep email pass:** on by default whenever emails are on. Rows with a website but no email get their+contact and about pages checked, and the script prints the found rate. `--no-deep-emails` skips it.## Other endpoints-- `GET /api/v1/jobs` — list jobs. `DELETE /api/v1/jobs/{id}` — delete a job + free disk.-- Browser UI + OpenAPI docs: `http://localhost:8080` and `http://localhost:8080/api/docs`.+- `GET /api/v1/jobs` lists jobs. `DELETE /api/v1/jobs/{id}` deletes a job and frees disk.+- Browser UI and OpenAPI docs: `http://localhost:8080` and `http://localhost:8080/api/docs`.## Best practices (from the upstream docs)-- **Depth:** higher `depth` = more results but slower and more block-prone. Start low (5), raise as needed.-- **One job at a time** locally. Many concurrent jobs without proxies → throttling/blocks by Google.-- **Email extraction (`email:true`)** visits each business's website → slower, but it's **on by default** here- because emails are the key lead field. Pass `--no-email` (script) or `email:false` (raw API) for a fast run.-- **`fast_mode:true`** returns reduced data, up to ~21 results/query, faster — good for quick lookups.-- **Proxies:** for large/repeated jobs, set `"proxies"` (array of `socks5://`/`http://`/`https://` URLs,- auth supported). The scraper has built-in rotation. This is the main defense against rate-limiting.-- **`zoom`/`radius`** control the search area around `lat`/`lon`. Widen `radius` if results are too few.-- **Extended reviews** are available but heavy — don't enable unless the user asks for review text.--## Rate limits, bans & proxies (read before large jobs)-This hits Google Maps for real. The upstream project's only formal note is a disclaimer:+- **Depth:** higher `depth` returns more results but is slower and more likely to be blocked. Start at 5+ and raise it as needed.+- **One job at a time** locally. Many concurrent jobs without proxies lead to throttling by Google.+- **Email extraction (`email:true`)** visits each business website, so it is slower. It stays **on by+ default** because emails are the key lead field. Pass `--no-email` (script) or `email:false` (raw API)+ for a fast run.+- **`fast_mode:true`** returns reduced data, up to ~21 results per query, faster. Use it for quick lookups.+- **Proxies:** for large or repeated jobs, set `"proxies"` (an array of `socks5://`, `http://` or+ `https://` URLs, auth supported). The engine rotates them. This is the main defense against rate limits.+- **`zoom` and `radius`** set the search area around `lat`/`lon`. Widen `radius` if results are too few.+- **Extended reviews** are available but heavy. Enable them only if the user asks for review text.++## Rate limits, blocks and proxies (read before large jobs)+The engine sends real requests to Google Maps. The upstream project's only formal note is a disclaimer:> "Please use this scraper responsibly and in accordance with applicable laws and regulations. Unauthorized scraping may violate terms of service."-There is **no published hard threshold** for bans, so be conservative:-- Google may **temporarily rate-limit / block your IP** if you scrape too fast or too much. It clears in- minutes–hours and does **not** ban your Google account — but jobs start failing meanwhile.-- **Block signals to watch:** jobs returning `failed`, empty or unusually short results, or a sudden drop- in row counts vs. a prior identical run. If you see these, **back off** (pause, lower depth) or add proxies.-- **Concurrency ↔ blocking** (upstream): *"Higher concurrency … can increase blocking or failures,- especially without proxies. Start with the default for a first run."* Reference throughput ≈ **120- places/min** at `-c 8 -depth 1`. Locally, run **one job at a time** and start at `depth 5`.+There is **no published hard threshold** for blocks, so stay conservative:+- Google may **temporarily rate-limit your IP** if you scrape too fast or too much. It clears in+ minutes to hours and does **not** affect the user's Google account, but jobs fail in the meantime.+- **Block signals:** jobs returning `failed`, empty or unusually short results, or a sudden drop in+ row count against an identical earlier run. When you see them, **back off** (pause, lower depth) or add proxies.+- **Concurrency and blocking** (upstream): *"Higher concurrency … can increase blocking or failures,+ especially without proxies. Start with the default for a first run."* Reference throughput is about+ **120 places/min** at `-c 8 -depth 1`. Locally, run **one job at a time** and start at `depth 5`.**When to add proxies** (upstream: *"For larger scraping jobs, proxies help avoid rate limiting"*):-large jobs, many keywords, repeated/scheduled runs, or after you see block signals.+large jobs, many keywords, repeated or scheduled runs, or after you see block signals.- Set the `"proxies"` array in the job body. Types: `socks5`, `socks5h`, `http`, `https`.- Format: `protocol://user:pass@host:port` (auth optional), e.g.- `"proxies": ["socks5://user:pass@host:port", "http://host2:port2"]`. The scraper rotates them automatically.--## Safety & guardrails-- **WARN, don't block.** When a request is large/high-volume (high `depth`, many keywords, repeated runs,- or `email:true`), **proceed with it** but first print ONE short warning about temporary IP-block risk and- suggest proxies. Do **not** gate, force-stop, or demand confirmation just because a normal scrape is big.- Refuse outright **only** for clearly abusive/illegal use (surveilling individuals, spam/harassment).-- **Never expose** the API beyond `localhost` without an auth proxy; never print any API key if one exists.-- **PII:** scraped emails/phones are personal data. If the user will store or contact them, remind them to- comply with GDPR/CCPA/CAN-SPAM (lawful basis, opt-outs, suppression). Don't help with spam/harassment.-- **Google ToS:** scraping Maps is against Google's Terms — keep volume modest, treat output as leads to- verify, don't resell raw Google data. Refuse uses aimed at surveilling individuals.-- **Output hygiene:** don't dump huge CSVs into chat — save the file to disk, summarize counts, show a few- sample rows. Dedupe by `place_id`/`cid` before storing.-- **Disk:** results pile up in the Docker volume; offer to `DELETE` old jobs periodically.+ `"proxies": ["socks5://user:pass@host:port", "http://host2:port2"]`. The engine rotates them automatically.++## Safety and guardrails+- **Warn, don't block.** When a request is large (high `depth`, many keywords, repeated runs, a grid, or+ `email:true`), **proceed with it**, but first print ONE short warning about temporary IP rate limits and+ suggest proxies. Do **not** gate, force-stop or demand confirmation because a normal scrape is big.+ Refuse outright **only** for clearly abusive or illegal use (surveilling individuals, spam, harassment).+- **Never expose** the API beyond `localhost` without an auth proxy. Never print an API key if one exists.+- **Personal data:** scraped emails and phones are personal data. If the user will store or contact them,+ remind them to follow GDPR, CCPA and CAN-SPAM (lawful basis, opt-outs, suppression). Do not help with+ spam or harassment.+- **Google's Terms of Service:** scraping Google Maps is against them. Keep volume modest, treat output as+ leads to verify, and do not resell raw Google data. Refuse uses aimed at surveilling individuals.+- **Output hygiene:** do not dump large CSVs into chat. Save the file to disk, summarize the counts and+ show a few sample rows. Dedupe by `place_id`/`cid` before storing.+- **Disk:** results pile up in the Docker volume. Offer to `DELETE` old jobs from time to time.## Troubleshooting-- `422 missing max time` → add `max_time` (seconds). `422 missing geo coordinates` → add string `lat`/`lon`.-- Stuck `working` → lower `depth` / raise `max_time` / IP throttled (add proxies or wait).-- Empty CSV → keyword too narrow or geo wrong → widen `radius`, fix coordinates.-- Connection refused → container down → `docker compose up -d`.+- `422 missing max time`: add `max_time` (seconds). `422 missing geo coordinates`: add string `lat`/`lon`.+- Stuck on `working`: lower `depth`, raise `max_time`, or the IP is throttled (add proxies or wait).+- Empty CSV: the keyword is too narrow or the coordinates are wrong. Widen `radius` or fix the coordinates.+- Connection refused: the container is down. Run `docker compose up -d`.diff --git a/.env.example b/.env.exampleindex bbac372..2777ebd 100644--- a/.env.example+++ b/.env.example@@ -1,17 +1,18 @@-# Copy this to ".env" and adjust if needed: cp .env.example .env+# Local config for the kit. Copy it to ".env" and edit only what you need: cp .env.example .env#-# This local setup binds the scraper to 127.0.0.1 only, so NO API key is required.-# (The scraper image has no built-in authentication — localhost binding is the boundary.)+# No API key is required. The engine is bound to 127.0.0.1 only, and that localhost+# binding is the security boundary, because the engine image has no built-in authentication.-# Host port for the scraper. Change it if 8080 is already taken on your machine, then run-# "docker compose up -d" again. Compose and both scripts read it from this file.+# Host port the engine listens on. Default: 8080. Change it only if 8080 is already taken,+# then run "docker compose up -d" again. Compose and both scripts read it from this file.# The port inside the container stays 8080.SCRAPER_PORT=8080-# Only set this if the scraper runs somewhere other than http://localhost:<SCRAPER_PORT>.-# When set, the scripts use it and ignore SCRAPER_PORT.+# Full URL of the engine's API. Default: unset, so the scripts use http://localhost:<SCRAPER_PORT>.+# Set it only if the engine runs somewhere else. When set, the scripts ignore SCRAPER_PORT.# SCRAPER_BASE_URL=http://localhost:8080-# Only needed if you later expose the scraper beyond localhost behind an auth proxy-# (e.g. Caddy/nginx that checks an X-API-Key header). Leave blank for local use.+# Sent as an X-API-Key header on every request. Default: blank, which is right for local use.+# Set it only if you expose the engine beyond localhost behind an auth proxy (e.g. Caddy or+# nginx checking that header).SCRAPER_API_KEY=diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.ymlindex ed6aba4..729be75 100644--- a/.github/ISSUE_TEMPLATE/bug_report.yml+++ b/.github/ISSUE_TEMPLATE/bug_report.yml@@ -1,5 +1,5 @@name: Bug report-description: Something in the kit fails or behaves wrongly.+description: Report something in the kit that fails or behaves wrongly.labels: [bug]body:- type: textarea@@ -13,7 +13,7 @@ body:id: commandattributes:label: Command or prompt- description: The exact command you ran (or what you asked Claude), and the full error output.+ description: The exact command you ran, or what you asked the agent, and the full error output.render: shellvalidations:required: true@@ -21,7 +21,7 @@ body:id: envattributes:label: Environment- description: OS, chip (Intel or Apple Silicon), Docker version, and the scraper image tag.+ description: OS, chip (Intel or Apple Silicon), Docker version, and the engine image tag.placeholder: macOS 15, Apple Silicon, Docker 28.1, gosom/google-maps-scraper:v1.18.1validations:required: truediff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.ymlindex f86eb80..3916c7f 100644--- a/.github/ISSUE_TEMPLATE/config.yml+++ b/.github/ISSUE_TEMPLATE/config.yml@@ -2,4 +2,4 @@ blank_issues_enabled: falsecontact_links:- name: Run a scrape yourselfurl: https://github.com/Mahanaicoach/google-maps-scraper-kit#quick-start- about: This repo is a tool you run on your own machine. Issues are not a scraping request service.+ about: The kit runs on your own machine. Follow the quick start to run your own scrape. Issues are for problems with the kit, not scrape requests.diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.ymlindex ef01f42..1bc1e4c 100644--- a/.github/ISSUE_TEMPLATE/feature_request.yml+++ b/.github/ISSUE_TEMPLATE/feature_request.yml@@ -1,11 +1,11 @@name: Feature request-description: Suggest an improvement to the kit.+description: Suggest a change that would make the kit more useful.labels: [enhancement]body:- type: textareaid: ideaattributes:label: What should the kit do?- description: Describe the problem it solves for you.+ description: Describe the problem it would solve for you.validations:required: truediff --git a/.github/pull_request_template.md b/.github/pull_request_template.mdindex 5d5a2d0..13bdad6 100644--- a/.github/pull_request_template.md+++ b/.github/pull_request_template.md@@ -1,6 +1,6 @@## What this changes-<!-- One or two sentences. Keep the PR to one focused change. -->+<!-- One or two sentences. Keep the PR to one change. -->## Why@@ -16,4 +16,5 @@- [ ] `scripts/scrape.py` still uses the Python standard library only (no pip dependencies)- [ ] `scripts/scrape.sh` still runs on macOS's default bash 3.2- [ ] No scrape output (CSV or result files) committed+- [ ] `python3 scripts/test_scrape.py` passes- [ ] Docs updated (README, SETUP or the skill) if behavior changeddiff --git a/ACCESSIBILITY.md b/ACCESSIBILITY.mdindex 65e695f..617f81b 100644--- a/ACCESSIBILITY.md+++ b/ACCESSIBILITY.md@@ -1,11 +1,12 @@# Accessibility-The Google Maps Scraper Kit aims to be usable by everyone, including people who use screen readers,-keyboard-only input, or high-contrast and magnified displays.+This page tells you how the Google Maps Scraper Kit works with assistive technology, and how to report a barrier.+The kit aims to be usable by everyone, including people who use screen readers, keyboard-only input,+or high-contrast and magnified displays.## Commitment-- Every step works from the keyboard. The kit is driven by text: Claude Code commands, a terminal,+- Every step works from the keyboard. The kit is driven by text: agent commands, a terminal,and plain files. There is no mouse-only interface.- Output is plain text and CSV, which opens in any spreadsheet app or screen reader.- Documentation is Markdown with real headings, lists, and tables, so it reads in order with@@ -28,7 +29,7 @@ keyboard-only input, or high-contrast and magnified displays.## Report a barrier-If something in the kit or its docs is hard to use with your setup, please+If something in the kit or its docs is hard to use with your setup,[open an issue](https://github.com/Mahanaicoach/google-maps-scraper-kit/issues/new/choose).Describe what you were trying to do, the assistive technology you use, and where you got stuck.Accessibility reports are treated as bugs.diff --git a/CLAUDE.md b/CLAUDE.mdindex 9f7aca5..235257b 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -1,42 +1,42 @@-# CLAUDE.md — Google Maps Scraper Kit+# CLAUDE.md: Google Maps Scraper Kit-You are working inside the **Google Maps Scraper Kit**. This folder runs a local Google Maps scraper-and you drive it for the user.+You are working inside the **Google Maps Scraper Kit**. The kit runs a local Google Maps scraper for+lead generation, and you drive it for the user.-## What this project is-A Docker-based local deployment of the open-source `gosom/google-maps-scraper`, exposed as a REST API+## What the kit is+A local Docker deployment of the open-source engine `gosom/google-maps-scraper`, exposed as a REST APIat **`http://localhost:8080`**, plus a skill that teaches you to use it. It scrapes **Google Maps-business listings only** (not social media).+business listings only**, not social media.-## When the user asks to scrape businesses / get a list of places / lead-gen data+## When the user asks for businesses, a list of places or lead data**Load and follow the `google-maps-scraper` skill** (`.claude/skills/google-maps-scraper/SKILL.md`).-It has the exact API flow, the required fields, and the best-practice + safety rules. Don't improvise-the API calls from memory — use the skill.+It has the exact API flow, the required fields, and the best-practice and safety rules. Do not write+the API calls from memory. Use the skill.## Slash commands-This kit ships commands in `.claude/commands/` that map to the skill: `/scrape` (single query),-`/scrape-batch` (many queries, one job), `/scrape-setup` (start the scraper), `/scrape-jobs` (list/delete).-Local commands are pre-approved in `.claude/settings.json` so you can run them without prompting the user.+The kit ships four commands in `.claude/commands/` that map to the skill: `/scrape` (single query),+`/scrape-batch` (many queries, one job), `/scrape-setup` (start the engine), `/scrape-jobs` (list or delete).+Local commands are pre-approved in `.claude/settings.json`, so you can run them without prompting the user.-## Rate limits — warn, don't block-If a request is large/high-volume (high `depth`, many keywords, repeated runs, or `email:true`), **proceed**-but first print ONE short warning that over-use can get the user's IP temporarily rate-limited by Google,-and suggest proxies. Do NOT gate or demand confirmation for a normal big scrape. Refuse only clearly-abusive use (surveilling individuals, spam). Full guidance is in the skill.+## Rate limits: warn, don't block+If a request is large (high `depth`, many keywords, repeated runs, a grid, or `email:true`), **proceed**,+but first print ONE short warning that heavy use can get the user's IP temporarily rate-limited by+Google, and suggest proxies. Do NOT gate or demand confirmation for a normal big scrape. Refuse only+clearly abusive use (surveilling individuals, spam). The skill has the full guidance.## Hard facts you must not get wrong-- API base: `http://localhost:8080` — **no auth** (it's localhost-only). Never expose it publicly.-- Create a job: `POST /api/v1/jobs`. **Required** body fields or it 422s:- `max_time` (in **seconds**, e.g. `300`), `lat` & `lon` (as **strings**), plus `keywords` (array).-- The create response returns `{"id":"<uuid>"}` (lowercase `id`). Status checks use `GET /api/v1/jobs/{id}` whose field is `"Status"` and becomes `"ok"` when done.-- **Poll in the background.** The Bash tool blocks foreground `sleep`; run the poll loop as a+- API base: `http://localhost:8080`, with **no auth** because it is bound to localhost. Never expose it publicly.+- Create a job with `POST /api/v1/jobs`. These body fields are **required**, or it returns 422:+ `max_time` (in **seconds**, e.g. `300`), `lat` and `lon` (as **strings**), and `keywords` (array).+- The create response is `{"id":"<uuid>"}` (lowercase `id`). Check status with `GET /api/v1/jobs/{id}`; its field is `"Status"`, and it becomes `"ok"` when done.+- **Poll in the background.** The Bash tool blocks foreground `sleep`. Run the poll loop as abackground command (`run_in_background: true`) and read its output file when notified.-- Before a big job (high depth / many keywords), **confirm with the user** — cost is time + IP-block risk.-- Scraped emails/phones are **personal data**; don't dump huge CSVs into chat — summarize + save to disk.+- Before a big job (high depth, many keywords), tell the user the cost once: time and IP rate-limit risk. Then proceed.+- Scraped emails and phones are **personal data**. Do not dump large CSVs into chat: summarize and save to disk.## Setup-If the container isn't running: `docker compose up -d`, then verify `curl http://localhost:8080/api/v1/jobs`.-Full setup is in `SETUP.md`.+If the container is not running: `docker compose up -d`, then verify with `curl http://localhost:8080/api/v1/jobs`.+`SETUP.md` has the full setup.## Attribution-The scraping engine is `gosom/google-maps-scraper` (MIT, © Georgios Komninos), run as its official Docker image. Keep `CREDITS.md` and `LICENSE` intact.+The engine is `gosom/google-maps-scraper` (MIT, © Georgios Komninos), run as its official Docker image. Keep `CREDITS.md` and `LICENSE` intact.diff --git a/CONTRIBUTING.md b/CONTRIBUTING.mdindex c0c7cdf..e5e14a5 100644--- a/CONTRIBUTING.md+++ b/CONTRIBUTING.md@@ -1,13 +1,12 @@# Contributing-Thanks for helping improve the Google Maps Scraper Kit.+This page tells you how to get a bug fixed or a change merged into the Google Maps Scraper Kit.-- **Bugs:** open an issue with the bug report template. Include the exact command and the full error.-- **Pull requests:** keep them small and focused on one change. Test with a `depth 1` scrape before you- open the PR. Run `python3 scripts/test_scrape.py` before opening a PR.-- **Scripts:** `scripts/scrape.py` uses the Python standard library only. Do not add pip dependencies.- `scripts/scrape.sh` must run on macOS's default bash 3.2.-- **Scraping engine:** changes to the engine itself belong upstream in+- **Report bugs** with the bug report issue template. Include the exact command and the full error.+- **Keep pull requests small**, one change each. Test with a `depth 1` scrape and run+ `python3 scripts/test_scrape.py` before you open the PR.+- **Keep the scripts portable.** `scripts/scrape.py` uses the Python standard library only, so do not+ add pip dependencies. `scripts/scrape.sh` must run on macOS's default bash 3.2.+- **Send engine changes upstream.** Changes to the engine itself belong in[gosom/google-maps-scraper](https://github.com/gosom/google-maps-scraper).-- **Data:** never commit scrape output. CSV and result files are git-ignored because they hold- personal data.+- **Never commit scrape output.** CSV and result files are git-ignored because they hold personal data.diff --git a/CREDITS.md b/CREDITS.mdindex 573bee7..16ee377 100644--- a/CREDITS.md+++ b/CREDITS.md@@ -1,6 +1,8 @@-# Credits & Attribution+# Credits and attribution-Google Maps Scraper Kit is designed and built by Mahan ([@Mahanaicoach](https://github.com/Mahanaicoach)).+This page lists who built what in the Google Maps Scraper Kit, and the license that covers the engine.++The kit is designed and built by Mahan ([@Mahanaicoach](https://github.com/Mahanaicoach)).One component, the Google Maps scraping engine, is open-source work by Georgios Komninos, credited below.## Scraping engine@@ -8,12 +10,12 @@ One component, the Google Maps scraping engine, is open-source work by Georgios- **Project:** google-maps-scraper- **Author:** **Georgios Komninos** ([@gosom](https://github.com/gosom))- **Repository:** https://github.com/gosom/google-maps-scraper-- **License:** MIT License — Copyright (c) 2023 Georgios Komninos+- **License:** MIT License, Copyright (c) 2023 Georgios Komninos- **Docker image used:** `gosom/google-maps-scraper`The kit runs the engine's official Docker image, unmodified. Everything else in this repository isoriginal to the kit: local deployment, lead-field cleaning, CSV output, auto-geocoding, batch jobs,-social profile enrichment, and the Claude skill and commands.+social profile enrichment, and the Claude Code skill and slash commands.## Engine license (reproduced as required by MIT)@@ -41,4 +43,4 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THESOFTWARE.```-Support the engine's author by starring the repository: https://github.com/gosom/google-maps-scraper+To support the engine's author, star the repository: https://github.com/gosom/google-maps-scraperdiff --git a/SECURITY.md b/SECURITY.mdindex 5de31b5..373394e 100644--- a/SECURITY.md+++ b/SECURITY.md@@ -1,8 +1,25 @@# Security policy-The kit binds the scraper API to `127.0.0.1` because the engine has no built-in authentication.+This page tells you what to report, where to report it, and what belongs elsewhere.++## What to report++Report any way the kit could expose the engine, leak data or run code it should not. Examples: a+change that binds the API beyond `127.0.0.1`, a script that leaks `SCRAPER_API_KEY`, or a+command in `.claude/settings.json` that grants more than local access.++The kit binds the engine's API to `127.0.0.1` because the engine has no built-in authentication.Do not expose port 8080 to a network without an authenticating proxy in front of it.-To report a vulnerability, use GitHub's private+## Where to report++Use GitHub's private[security advisory form](https://github.com/Mahanaicoach/google-maps-scraper-kit/security/advisories/new).Do not open a public issue for security problems.++## Out of scope++- Bugs in the engine itself. Report those upstream to+ [gosom/google-maps-scraper](https://github.com/gosom/google-maps-scraper).+- Problems that only occur after you expose the API to a network without an auth proxy.+- Google rate-limiting your IP. That is expected behavior, covered in [SETUP.md](SETUP.md).diff --git a/SETUP.md b/SETUP.mdindex 79652cc..4b15b16 100644--- a/SETUP.md+++ b/SETUP.md@@ -1,13 +1,13 @@-# SETUP — Google Maps Scraper Kit+# Setup guide: Google Maps Scraper Kit-This guide sets up the whole system **on your computer** from scratch. It's written so that a person-*or* Claude can follow it exactly. Total time: ~5 minutes (most of it is Docker pulling the image).+This guide takes you from a fresh machine to a finished Google Maps lead list in about 5 minutes.+Most of that time is Docker pulling the engine image. A person or an agent can follow it step by step.----## 0. Prerequisites+## 0. Install the prerequisites-Install these first (one-time):+Install these once:| Tool | Why | Get it ||---|---|---|@@ -15,14 +15,14 @@ Install these first (one-time):| **Claude Code** *(optional but recommended)* | lets Claude drive the scraper for you | https://claude.com/claude-code || **git** *(optional)* | only if you'll push this to GitHub | https://git-scm.com |-Verify Docker is running:+Check that Docker is running:```bashdocker --version # should print a versiondocker ps # should NOT error (means the daemon is up)```> **Platform notes:**-> - **Apple Silicon (M1/M2/M3):** the scraper image is `linux/amd64` and runs under emulation. If the+> - **Apple Silicon (M1/M2/M3):** the engine image is `linux/amd64` and runs under emulation. If the> container won't start, uncomment the `platform: linux/amd64` line in `docker-compose.yml`.> - **Windows:** use `scripts/scrape.py` (run with `py` or `python3`). The bash script `scrape.sh` needs> **WSL2** or **Git Bash**.@@ -31,116 +31,122 @@ docker ps # should NOT error (means the daemon is up)## 1. Get the kit-If someone shared this folder with you, just `cd` into it:+If someone shared the folder with you, `cd` into it:```bashcd google-maps-scraper-kit```-(Or clone it from GitHub: `git clone https://github.com/Mahanaicoach/google-maps-scraper-kit.git && cd google-maps-scraper-kit`.)+To clone it from GitHub instead: `git clone https://github.com/Mahanaicoach/google-maps-scraper-kit.git && cd google-maps-scraper-kit`.-Create your local config (optional — defaults work as-is):+Create your local config. This step is optional, because the defaults work as they are:```bashcp .env.example .env```----## 2. Start the scraper+## 2. Start the engine```bashdocker compose up -d```-First run pulls the `gosom/google-maps-scraper` image (~once, a few hundred MB). When it finishes:+The first run pulls the `gosom/google-maps-scraper` image once, a few hundred MB. When it finishes, check it:```bashdocker ps | grep gmaps-scraper # should show it "Up"curl http://localhost:8080/api/v1/jobs # should print [] (empty list) or existing jobs```-You can also open the built-in UI and API docs in a browser:+The engine also serves a web UI and API docs in the browser:- Web UI: **http://localhost:8080**- OpenAPI docs: **http://localhost:8080/api/docs**-✅ If `curl` returns a JSON array, the system is live.+If `curl` returns a JSON array, the engine is live.-> ⚠️ **Now that it's running — read this once.** This hits Google Maps for real. Light use is fine, but-> big jobs back-to-back, very high `depth`, many keywords, or scheduled runs **without proxies** can get-> your **IP temporarily rate-limited/blocked by Google** (clears in minutes–hours; jobs return empty/failed-> meanwhile). It does **not** ban your Google account. **Stay safe:** one job at a time, start at `depth 5`,-> and add **proxies** for large or repeated runs. Scraping Maps is against Google's ToS — use responsibly-> and follow data law (GDPR/CCPA) for any contact data. (Full guidance in README → *Rate limits, bans &-> responsible use*.)+> **Rate limits and the law.** The engine sends real requests to Google Maps. Light use is fine. Big+> jobs back to back, high `depth`, many keywords or scheduled runs without proxies can get your IP+> temporarily rate-limited by Google, for minutes to hours. Your Google account is not affected. Run+> one job at a time, start at `depth 5`, and add proxies for large or repeated runs. Scraping Google+> Maps is against Google's Terms of Service, and phones and emails are personal data, so follow GDPR,+> CCPA and CAN-SPAM. Details: [Responsible use](README.md#responsible-use).----## 3. Run your first scrape (without Claude)+## 3. Run your first scrape from the terminal-Use the included script. Arguments: `"<keyword>" <lat> <lon> [depth]`+Use the bash script. Its arguments are `"<keyword>" <lat> <lon> [depth]`:```bash./scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5```-or the Python version (no dependencies needed):+Or use the Python CLI, which needs no dependencies:```bashpython3 scripts/scrape.py "gyms in Miami FL" 25.7617 -80.1918 --depth 5```-The script will **create the job → poll until done → download the CSV → print a summary** and save-the full results to a file.+Both scripts create the job, poll until it is done, download the CSV, keep the 8 lead fields and+print a short preview. The full results go to a file.-**Don't want to look up coordinates?** Let it geocode the city for you:+**Skip the coordinates.** Pass `--city` and the Python CLI geocodes the city for you:```bashpython3 scripts/scrape.py "cafes in Austin TX" --city "Austin, TX" --depth 5```-**Scrape many queries at once** (one job, many keywords) from a file:+**Run many keywords in one job** from a file, one keyword per line:```bashpython3 scripts/scrape.py --keywords-file examples/queries.example.txt --city "Denver, CO"```-**Go past Google's ~120 results per search** by splitting the city into a 3 x 3 grid of map tiles-(9 jobs, run one at a time, duplicates removed):+**Go past Google's ~120 results per search.** `--grid 3` splits the city into a 3 x 3 grid of map+tiles. That is 9 jobs, run one at a time, with duplicates removed:```bashpython3 scripts/scrape.py "plumbers in Phoenix AZ" --city "Phoenix, AZ" --grid 3```----## 4. Use it with Claude (the autopilot way) 🤖+## 4. Run it from Claude Code or another agent-This is the point of the kit. Two parts make it work:+The kit ships a Claude Code skill that teaches the agent the job API, the required fields, polling+and the guardrails. It works with Claude Code, Codex, Cursor and any other+[Agent Skills](https://agentskills.io)-compatible tool. Two files do the work:-1. **[CLAUDE.md](CLAUDE.md)** — auto-loaded when you open this folder in Claude Code. It tells Claude- what this project is and the key rules.-2. **[.claude/skills/google-maps-scraper/SKILL.md](.claude/skills/google-maps-scraper/SKILL.md)** — a- **skill** Claude loads on demand. It teaches Claude the exact API flow + best practices.+1. **[CLAUDE.md](CLAUDE.md)** loads automatically when you open this folder in Claude Code. It tells+ the agent what the kit is and its key rules.+2. **[.claude/skills/google-maps-scraper/SKILL.md](.claude/skills/google-maps-scraper/SKILL.md)** is+ the skill. The agent loads it on demand for the exact API flow and best practices.-### How to connect it to Claude-There's nothing to "connect" — the scraper is a local HTTP API at `http://localhost:8080`, and Claude-calls it with its normal `curl`/Bash ability. Just:+There is nothing to connect. The engine is a local HTTP API at `http://localhost:8080`, and the agent+calls it with `curl` and its normal shell access.++**Option A: open the folder in Claude Code.** The skill and four slash commands load automatically.```bashcd google-maps-scraper-kitclaude # start Claude Code inside this folder```-Then talk to it naturally:+**Option B: install the skill into an agent you already use.**++```bash+npx skills add mahanaicoach/google-maps-scraper-kit+```++Then ask in plain language:> "Make sure the scraper is running, then scrape **dentists in Denver CO**, depth 10, and give me a table of name, phone, website."-Claude will: check the container is up (start it if not) → build the correct job (with the required-`max_time`, `lat`, `lon`) → poll in the background → download → hand you clean rows. It follows the-safety/best-practice rules in the skill automatically.+The agent checks the engine is up and starts it if not. It then builds the job with the required+`max_time`, `lat` and `lon`, polls in the background, downloads the CSV and hands you clean rows. It+follows the skill's safety rules on every run.-> ℹ️ **Advanced (optional):** prefer a Model Context Protocol server instead of curl? You can wrap the-> API as an MCP tool, but it's unnecessary for a local single-user setup — the skill + local API is-> simpler and just as capable.+> **MCP is optional.** You can wrap the API as a Model Context Protocol server, but a local,+> single-user setup does not need one. The skill and the local API do the same job with less setup.----+### Slash commands-### Slash-command shortcuts-Inside Claude Code you can also use these commands (type `/` to see them):+Claude Code also offers these commands. Type `/` to see them:| Command | What it does ||---|---|@@ -149,12 +155,13 @@ Inside Claude Code you can also use these commands (type `/` to see them):| `/scrape-setup` | Start the scraper + health check || `/scrape-jobs [list \| delete <id>]` | List or delete jobs |-…or just ask in plain English: *"scrape gyms in Miami with phone numbers."* (The scraper's commands and-`localhost` calls are pre-approved in `.claude/settings.json`, so Claude won't prompt for each one.)+Plain requests work too: *"scrape gyms in Miami with phone numbers."* The kit's local commands and+`localhost` calls are pre-approved in `.claude/settings.json`, so Claude Code does not prompt for+each one.----## 5. Stop / clean up+## 5. Stop and clean up```bashdocker compose stop # pause (keeps data + image)@@ -162,7 +169,7 @@ docker compose down # stop + remove container (keeps named volumes/data)docker compose down -v # ALSO delete scraped data volumes (full reset)```-Delete old jobs to free disk (results accumulate in the data volume):+Results pile up in the data volume. Delete old jobs to free disk:```bashcurl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>```@@ -187,7 +194,7 @@ curl -X DELETE http://localhost:8080/api/v1/jobs/<job-id>### No Compose? Run the same container with `docker run`-`docker-compose.yml` is thin — this is the exact equivalent, localhost binding and all:+`docker-compose.yml` is thin. This command is the exact equivalent, including the localhost binding:```bashdocker run -d --name gmaps-scraper --restart unless-stopped \@@ -200,22 +207,23 @@ Add `--platform linux/amd64` on Apple Silicon. Stop it with `docker stop gmaps-s### Running without Docker (fallback)-If Docker is unavailable — or the image's amd64 emulation is too slow on Apple Silicon — run the-scraper natively. It's a single Go binary, and it serves the **same API on the same port**, so every-script and slash command in this kit works unchanged.+Use this when Docker is unavailable, or when amd64 emulation is too slow on Apple Silicon. The engine+is a single Go binary. It serves the **same API on the same port**, so every script and slash command+in the kit works unchanged.```bashgo install github.com/gosom/[email protected]google-maps-scraper -web -addr 127.0.0.1:8080 -data-folder ./gmapsdata```-> Keep `-addr 127.0.0.1:8080`. The binary defaults to `:8080` — **all interfaces** — and the scraper-> has no authentication. In Docker the localhost binding comes from the port mapping; here it's on you.+> Keep `-addr 127.0.0.1:8080`. The binary defaults to `:8080`, which means **all interfaces**, and the+> engine has no authentication. In Docker the port mapping provides the localhost binding. Here you+> provide it.**If the first job dies with `could not install driver: … 404`:** `playwright-go` fetches its driver-from `playwright.azureedge.net`, which Microsoft retired; the replacement hosts currently answer `400`-for driver zips (browser downloads still work). That driver is just the `playwright-core` npm package-plus a `node` binary, so you can assemble it by hand — match the version the error names:+from `playwright.azureedge.net`, which Microsoft retired. The replacement hosts currently answer `400`+for driver zips, though browser downloads still work. The driver is the `playwright-core` npm package+plus a `node` binary, so you can assemble it by hand. Match the version the error names:```bashV=1.57.0 # whatever version the error asks for@@ -225,16 +233,16 @@ mkdir -p "$D" && cp -R package "$D/package" && cp "$(command -v node)" "$D/node""$D/node" "$D/package/cli.js" --version # should print: Version <V>```-Restart the scraper; it finds the driver, downloads Chromium normally, and runs.+Restart the engine. It finds the driver, downloads Chromium normally, and runs.----## 7. Best practices & safety (summary)+## 7. Best practices-The full rules live in the skill, but the essentials:-- **Always** send `max_time` (seconds) and `lat`/`lon` (strings).-- Start with **low `depth`** (5) and **one job at a time**. Raise only when needed.-- Turn on **email extraction** only when you need emails — it's much slower.-- For big jobs, add **proxies** (the scraper has built-in rotation) to avoid IP blocks.-- Scraped **phones/emails are personal data** — comply with GDPR/CCPA/CAN-SPAM, and respect Google's ToS.-- Never commit result files (they're git-ignored for you).+The skill holds the full rules. These are the essentials:+- Always send `max_time` (seconds) and `lat`/`lon` (strings).+- Start at `depth 5` and run one job at a time. Raise depth only when needed.+- Keep email extraction on, the default, because emails are the key lead field. Use `--no-email` for a+ faster run without them.+- Add proxies for big jobs. The engine rotates them for you.+- Never commit result files. They are git-ignored for you.diff --git a/docker-compose.yml b/docker-compose.ymlindex 8bd1e06..2c8b39b 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -1,27 +1,28 @@-# Runs the open-source Google Maps scraper (gosom/google-maps-scraper) locally.-# Web/API mode on http://localhost:8080 (host port: SCRAPER_PORT in .env, default 8080). The image has NO built-in auth, so we bind-# to 127.0.0.1 ONLY — that localhost binding is the security boundary. Do not change-# the port mapping to 0.0.0.0 / a public interface without putting an auth proxy in front.+# Runs the kit's engine, the open-source Google Maps scraper (gosom/google-maps-scraper), locally.+# Serves the web UI and API on http://localhost:8080. Host port: SCRAPER_PORT in .env, default 8080.+# The image has no built-in auth, so the port is bound to 127.0.0.1 only. That localhost binding is+# the security boundary. Do not change the mapping to 0.0.0.0 or a public interface without an auth+# proxy in front.services:google-maps-scraper:- # Pinned to a known-good release for reproducibility (bump deliberately).- # The published image is linux/amd64. On Apple Silicon (M1/M2/M3) it runs under- # emulation; if it won't start, UNCOMMENT the 'platform' line below.- # Bumped from v1.15.0 on 2026-09-23: that build's embedded Playwright driver- # (1.57.0) 404s on Microsoft's CDN (old driver builds get purged). v1.18.1 ships- # a driver version still hosted.+ # Engine image, pinned to a known-good release so runs are reproducible. Bump it deliberately.+ # Bumped from v1.15.0 on 2026-09-23: that build's embedded Playwright driver (1.57.0)+ # returns 404 from Microsoft's CDN, which purges old driver builds. v1.18.1 ships a driver+ # version that is still hosted.+ # The published image is linux/amd64. On Apple Silicon (M1/M2/M3) it runs under emulation.+ # If the container won't start there, uncomment the 'platform' line below. Default: commented out.image: gosom/google-maps-scraper:v1.18.1# platform: linux/amd64container_name: gmaps-scraperrestart: unless-stoppedcommand: ["-web", "-data-folder", "/gmapsdata"]ports:- # Host port comes from SCRAPER_PORT in .env. The container side stays 8080.- - "127.0.0.1:${SCRAPER_PORT:-8080}:8080" # localhost only — see warning above+ # Host port comes from SCRAPER_PORT in .env (default 8080). The container side stays 8080.+ - "127.0.0.1:${SCRAPER_PORT:-8080}:8080" # localhost only, see the warning abovevolumes:- - gmaps_data:/gmapsdata # job results persist here- - gmaps_cache:/opt # Playwright browser cache (faster restarts)+ - gmaps_data:/gmapsdata # job results; they persist until you delete the job or the volume+ - gmaps_cache:/opt # Playwright browser cache, so restarts skip the downloadvolumes:gmaps_data:diff --git a/examples/queries.example.json b/examples/queries.example.jsonindex 4103d40..f18dd6d 100644--- a/examples/queries.example.json+++ b/examples/queries.example.json@@ -1,5 +1,5 @@{- "_comment": "Reference request body for POST /api/v1/jobs. Copy the 'job' object and edit. Required fields: keywords, lat, lon (strings), max_time (seconds).",+ "_comment": "Reference request body for POST /api/v1/jobs. Copy the 'job' object and edit it. Required fields: keywords, lat and lon (strings), max_time (seconds). This sample sets email to false for a fast run; the kit's scripts send email true by default. Set proxies to an array of proxy URLs for large or repeated runs.","job": {"name": "my scrape","keywords": ["coffee shops in Austin TX"],diff --git a/examples/queries.example.txt b/examples/queries.example.txtindex 2ba52c5..4c48ac9 100644--- a/examples/queries.example.txt+++ b/examples/queries.example.txt@@ -1,5 +1,6 @@-# Batch keywords — one per line. Lines starting with # are ignored.-# Bake the location into each line, OR pass --city to geocode one place for all of them.+# Batch keyword list: every line becomes one keyword in a single job and one CSV.+# One keyword per line. Blank lines and lines starting with # are ignored.+# Bake the location into each line, or pass --city to geocode one place for all of them.# Run: python3 scripts/scrape.py --keywords-file examples/queries.example.txt --city "Denver, CO"dentists in Denver COorthodontists in Denver COdiff --git a/scripts/scrape.py b/scripts/scrape.pyindex f6c9d38..a669000 100755--- a/scripts/scrape.py+++ b/scripts/scrape.py@@ -1,5 +1,7 @@#!/usr/bin/env python3-"""One-shot Google Maps scrape using only the Python standard library (no pip installs).+"""Google Maps scraper CLI for lead generation: turns a keyword and a place into a CSV of the 8 lead fields.++Uses only the Python standard library (no pip installs) and drives the kit's local engine.Single keyword:python3 scripts/scrape.py "gyms in Miami FL" 25.7617 -80.1918 --depth 5@@ -10,16 +12,16 @@ Auto-geocode (no coordinates needed):Batch (one job, many keywords) from a file (one keyword per line):python3 scripts/scrape.py --keywords-file examples/queries.example.txt --city "Denver, CO"-Whole city past Google's ~120 cap (3x3 map tiles, run one at a time, deduped on place_id):+Whole city past Google's ~120 results per search (3 x 3 map tiles, run one at a time, deduped on place_id):python3 scripts/scrape.py "plumbers in Phoenix AZ" --city "Phoenix, AZ" --grid 3-Skip the contact-page email pass (on by default whenever emails are on):+Skip the deep email pass over contact and about pages (on by default whenever emails are on):python3 scripts/scrape.py "gyms in Miami FL" --city "Miami, FL" --no-deep-emailsNotes:- Required by the API: keywords, lat/lon (strings), max_time (SECONDS). This script fills them in.-- Geocoding uses OpenStreetMap Nominatim (free, no key). Please be gentle: it allows ~1 request/sec- and requires a descriptive User-Agent (set below). Don't loop it aggressively.+- Geocoding uses OpenStreetMap Nominatim (free, no key). It allows ~1 request/sec and requires a+ descriptive User-Agent (set below). Do not loop it aggressively."""import argparse, csv, functools, io, json, math, os, re, sys, time, urllib.request, urllib.parse, urllib.errorfrom concurrent.futures import ThreadPoolExecutor@@ -44,8 +46,8 @@ def load_env_file():load_env_file()BASE = os.environ.get("SCRAPER_BASE_URL") or f"http://localhost:{os.environ.get('SCRAPER_PORT') or '8080'}"KEY = os.environ.get("SCRAPER_API_KEY", "")-# Money-useful LEAD fields only — what you actually use to contact/qualify a lead.-# Everything else (geo coordinates, IDs, hours, images, reviews blobs…) is dropped by default.+# The 8 lead fields: what you use to contact and qualify a lead.+# Everything else (geo coordinates, IDs, hours, images, review blobs) is dropped by default.LEAD = ["title", "phone", "emails", "website", "category", "address", "review_rating", "review_count"]UA = "google-maps-scraper-kit/1.1 (https://github.com/Mahanaicoach/google-maps-scraper-kit)"diff --git a/scripts/scrape.sh b/scripts/scrape.shindex 5be0f5c..90d6fed 100755--- a/scripts/scrape.sh+++ b/scripts/scrape.sh@@ -1,5 +1,5 @@#!/usr/bin/env bash-# One-shot Google Maps scrape: create job -> poll -> download -> summarize.+# Google Maps scrape for one keyword: creates the job, polls, downloads, and saves the 8 lead fields to a CSV.# Usage: ./scripts/scrape.sh "<keyword>" <lat> <lon> [depth]# Example: ./scripts/scrape.sh "coffee shops in Austin TX" 30.2672 -97.7431 5set -euo pipefail@@ -36,7 +36,7 @@ if ! curl -s -m 5 ${AUTH[@]+"${AUTH[@]}"} "$BASE/api/v1/jobs" >/dev/null 2>&1; tfiecho "▶ Creating job: \"$KEYWORD\" @ $LAT,$LON depth=$DEPTH (email extraction ON)"-# email:true → the scraper visits each business website to pull emails (the key lead field). A bit slower.+# email:true makes the engine visit each business website for emails (the key lead field). A little slower.BODY=$(printf '{"name":"scrape-cli","keywords":["%s"],"lang":"en","zoom":15,"lat":"%s","lon":"%s","fast_mode":false,"radius":10000,"depth":%s,"email":true,"max_time":600}' \"$KEYWORD" "$LAT" "$LON" "$DEPTH")ID=$(curl -s -X POST "$BASE/api/v1/jobs" ${AUTH[@]+"${AUTH[@]}"} -H "Content-Type: application/json" -d "$BODY" \@@ -60,7 +60,7 @@ RAW="$(mktemp)"echo "▶ Downloading…"curl -s ${AUTH[@]+"${AUTH[@]}"} "$BASE/api/v1/jobs/$ID/download" -o "$RAW"-# Trim the raw 34-column dump down to money-useful LEAD fields only (drops geo coords, IDs, hours, images…).+# Trim the raw 34-column CSV to the 8 lead fields (drops coordinates, IDs, hours, images).if command -v python3 >/dev/null 2>&1; thenpython3 - "$RAW" "$OUT" <<'PY'import csv, sys