Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
feat: row filters, --exclude, --limit, --proxy, --verify-emails, more socials and pixels, several cities
Mahanaicoachc027fe9ad02c
diff --git a/.claude/commands/scrape.md b/.claude/commands/scrape.mdindex 005354b..3e6c072 100644--- a/.claude/commands/scrape.md+++ b/.claude/commands/scrape.md@@ -9,17 +9,19 @@ Use the `google-maps-scraper` skill and follow these steps:1. **Check the engine:** `curl -s http://localhost:8080/api/v1/jobs`. If it is down, run `docker compose up -d` from the kit root, wait ~10s, and retry.2. **Parse** the business type and city from the query. Look up the city's **latitude/longitude** (as strings).3. **Ask about social profiles before scraping, every time.** Ask the user ONE short question:- *"Want me to also grab their social profiles (Instagram / Facebook / LinkedIn)? Emails are already included by default. (yes / no)"*+ *"Want me to also grab their social profiles (Instagram / Facebook / LinkedIn / TikTok / YouTube / X)? Emails are already included by default. (yes / no)"*Wait for the answer. **Skip the question only if** the user already said yes or no to socials in the request.If **yes**, add `--socials` in step 4. If **no**, leave it off. Socials cost no LLM tokens and one extra HTTP fetch per site (see the skill).4. **Run the scrape with emails on.** Easiest and fastest: the script (below), which keeps the engine's email option off and crawls each business website itself. Only on the raw API, set `"email": true` in the body. Required fields: `name` (any label), `keywords` (location baked into each term), `lang` (`"en"`), `lat`/`lon` (strings), `max_time` (seconds, default `300`); plus `depth` (the number in the query, else `5`). Easiest path: `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --depth <n>` (emails are already on by default; append `--socials` if the user said yes in step 3).5. **Poll to completion in the background.** The Bash tool blocks foreground `sleep`, so use `run_in_background: true` and read the output file when notified. Then download the CSV.-6. **Save a CSV file.** `scripts/scrape.py` writes `results-<id>.csv` by default. If you use the raw API, save the parsed lead fields to a `.csv` yourself. Then **present only the 8 lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count`, plus `instagram, facebook, linkedin` if socials were requested. **Drop everything else**, especially `latitude`/`longitude`, IDs, hours, images and review blobs. Never dump the full 36-column CSV into chat. Show a few sample rows and give the CSV path. Include dropped fields only if the user explicitly asks.+6. **Save a CSV file.** `scripts/scrape.py` writes `results-<id>.csv` by default. If you use the raw API, save the parsed lead fields to a `.csv` yourself. Then **present only the 8 lead fields** in a clean markdown table: `name (title), phone, emails, website, category, address, review_rating, review_count`, plus `instagram, facebook, linkedin, tiktok, youtube, x` if socials were requested. **Drop everything else**, especially `latitude`/`longitude`, IDs, hours, images and review blobs. Never dump the full 36-column CSV into chat. Show a few sample rows and give the CSV path. Include dropped fields only if the user explicitly asks.**All, every or entire city:** add `--grid 3` (3 x 3 map tiles, one job at a time, deduped) to get past Google's ~120 results per search. Warn once that it is slower and more likely to trip rate limits, then proceed.**Emails:** on by default. The script crawls each business website after the Maps job, 16 at a time. Some businesses publish no email, so coverage is partial. For a deliberately fast run without emails, pass `--no-email`.+**Filters:** when the user wants only businesses without a website, a minimum rating or review count, or only rows with a phone or email, add `--no-website`, `--min-rating X`, `--min-reviews N`, `--has-phone`, `--has-email`. `--exclude <earlier.csv>` drops places from a previous run; `--limit N` caps the rows. Several cities: repeat `--city`.+**Socials:** when the user says yes, `scripts/scrape.py … --socials` adds Instagram, Facebook and LinkedIn. Extraction costs **0 LLM tokens** because it runs in code. It adds ~40–50 tokens per business only if you load the rows into chat. **Never** WebFetch each website yourself to find socials; that costs thousands of tokens.**Warn, don't block:** if the request is high-volume (high `depth`, many keywords or repeated runs), print ONE short warning that heavy use can get the user's IP temporarily rate-limited by Google, and suggest proxies. Then **proceed with the scrape**. Do not gate or demand confirmation for a big job. Refuse outright only for clearly abusive use (surveilling individuals, spam, harassment). If the user will store or contact the data, remind them once that it is personal data and that scraping is against Google's Terms of Service.diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdindex 20feb5d..787562c 100644--- a/.claude/skills/google-maps-scraper/SKILL.md+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -1,6 +1,6 @@---name: google-maps-scraper-description: Google Maps scraper for lead generation. Turns a business type and a place into a clean CSV of local businesses by driving the gosom/google-maps-scraper engine, running locally in Docker as a REST API on localhost:8080. Returns 8 lead fields by default (name, phone, email, website, category, address, rating and review count), plus optional Instagram, Facebook and LinkedIn profiles or all 36 raw fields. Use when the user wants a lead list, "a list of [businesses] in [place]", local business data, contact enrichment for places (emails, phones, socials), or a whole-city run past Google's ~120 results per search. Do not use for scraping Instagram, TikTok, YouTube, X or any other social media platform directly.+description: Google Maps scraper for lead generation. Turns a business type and a place into a clean CSV of local businesses by driving the gosom/google-maps-scraper engine, running locally in Docker as a REST API on localhost:8080. Returns 8 lead fields by default (name, phone, email, website, category, address, rating and review count), plus optional social profiles (Instagram, Facebook, LinkedIn, TikTok, YouTube, X), row filters such as businesses without a website, or all 36 raw fields. Use when the user wants a lead list, "a list of [businesses] in [place]", local business data, contact enrichment for places (emails, phones, socials), or a whole-city run past Google's ~120 results per search. Do not use for scraping Instagram, TikTok, YouTube, X or any other social media platform directly.---# Google Maps Scraper@@ -116,9 +116,9 @@ outreach), `link`, `plus_code`, `cid`, `data_id`, `place_id`, `open_hours`, `pop`--json` for JSON. If you call the API directly, **strip to the lead fields yourself** beforepresenting. Never dump the full 36-column row at the user.-### Social profiles: always ask the user (Instagram, Facebook, LinkedIn)+### Social profiles: always ask the user (Instagram, Facebook, LinkedIn, TikTok, YouTube, X)Google Maps has no social links, so socials are an enrichment step: visit each business `website` and-regex out its Instagram, Facebook and LinkedIn URLs. **Before scraping, ask the user once** whether they+regex out its Instagram, Facebook, LinkedIn, TikTok, YouTube and X URLs. **Before scraping, ask the user once** whether theywant socials too, unless they already said. If yes, **use the script:** `python3 scripts/scrape.py … --socials`.No `scripts/` (skill installed on its own)? Socials need the full kit: say so once, skip the question,and do not fetch the websites yourself.@@ -141,7 +141,9 @@ the curl flow above):**Emails come back by default** (`--no-email` skips them). Add `--socials` if the user asked for socials.- **Batch (many keywords, one job):** `python3 scripts/scrape.py --keywords-file <file> --city "<City, ST>"`.The API takes a `keywords` array, so put all terms in a single job rather than firing many jobs.-Use the manual curl flow only for custom job bodies, e.g. to set `proxies`.+- **Several cities:** repeat `--city`; they run one after another into one deduped file.+- **Proxies:** `--proxy socks5://user:pass@host:port` (repeatable; `socks5h`, `http`, `https` too).+Use the manual curl flow only for custom job bodies.**Whole-city runs:** when the user says all, every or entire city, or a run returns about 120 rows(Google's per-search cap), use `python3 scripts/scrape.py "<keyword>" --city "<City, ST>" --grid 3`.@@ -150,11 +152,19 @@ grid is slower and more likely to trip rate limits: warn once, don't block.**Email crawl:** on by default. After the Maps job the script fetches every business homepage, 16 ata time, then contact and about pages where the homepage shows no email, and prints the found rate.-`--no-deep-emails` checks homepages only; `--no-email` skips the crawl.+`--no-deep-emails` checks homepages only; `--no-email` skips the crawl. `--verify-emails` looks up each+address's domain over DNS (dns.google, no key), drops domains that do not exist and adds `email_host`+(`google_workspace`, `microsoft_365`, `other`, `none`).++**Filters and follow-up runs:** when the user wants only businesses without a website (the classic+web-design prospect list), a minimum rating or review count, or only rows with a phone or an email,+pass `--no-website`, `--min-rating 4.5`, `--min-reviews 20`, `--has-phone`, `--has-email`. `--limit N`+caps the rows. `--exclude earlier.csv` (repeatable, `.csv` or `.json`) drops places from an earlier+run, so a weekly re-run returns only new leads. Filters run in code, before the website crawl.**Who already advertises:** when the user wants warm leads, businesses that run ads, or to know a-site's platform, add `--tech`. It reads each homepage for a Meta Pixel, Google Ads tag and analytics,-the platform (WordPress, Shopify, Wix, Squarespace, Webflow) and mobile support, scores each lead 0-100+site's platform, add `--tech`. It reads each homepage for a Meta Pixel, Google Ads tag, analytics, Bing Ads,+LinkedIn Insight and TikTok pixels, the platform (WordPress, Shopify, Wix, Squarespace, Webflow) and mobile support, scores each lead 0-100(`intent_score`: pixels dominate, reviews, rating and an email add a little) and sorts highest first.It finds installed tags, not live ad campaigns. Zero LLM tokens.@@ -196,7 +206,7 @@ There is **no published hard threshold** for blocks, so stay conservative:**When to add proxies** (upstream: *"For larger scraping jobs, proxies help avoid rate limiting"*):large jobs, many keywords, repeated or scheduled runs, or after you see block signals.-- Set the `"proxies"` array in the job body. Types: `socks5`, `socks5h`, `http`, `https`.+- `--proxy URL` on the script (repeatable), or the `"proxies"` array in a raw job body. Types: `socks5`, `socks5h`, `http`, `https`.- Format: `protocol://user:pass@host:port` (auth optional), e.g.`"proxies": ["socks5://user:pass@host:port", "http://host2:port2"]`. The engine rotates them automatically.diff --git a/.env.example b/.env.exampleindex 2777ebd..2a07e3c 100644--- a/.env.example+++ b/.env.example@@ -12,6 +12,11 @@ SCRAPER_PORT=8080# Set it only if the engine runs somewhere else. When set, the scripts ignore SCRAPER_PORT.# SCRAPER_BASE_URL=http://localhost:8080+# Name of the engine container, for the stuck-job recovery (the script reads its log and restarts it).+# Default: gmaps-scraper, the container_name in docker-compose.yml. Set it only if you renamed the+# container or run a second stack.+# SCRAPER_CONTAINER=gmaps-scraper+# Sent as an X-API-Key header on every request. Default: blank, which is right for local use.# Set it only if you expose the engine beyond localhost behind an auth proxy (e.g. Caddy or# nginx checking that header).diff --git a/README.md b/README.mdindex d680e93..8fffc2f 100644--- a/README.md+++ b/README.md@@ -17,7 +17,8 @@**A free, open-source Google Maps scraper for lead generation that runs on your own machine.**Name a business type and a city. Get back an outreach-ready CSV with name, phone, email, website,-category, address, rating and review count. Add `--socials` for Instagram, Facebook and LinkedIn.+category, address, rating and review count. Add `--socials` for Instagram, Facebook, LinkedIn, TikTok,+YouTube and X, or `--no-website` for the businesses that still need a site.No Google Maps API key, no SaaS credits, no data leaving your computer. One Docker command starts theengine. Then run it from the terminal, or let Claude Code, Codex or Cursor drive it for you in plain@@ -93,9 +94,10 @@ python3 scripts/scrape.py "dentists in Denver CO" --city "Denver, CO" --grid 3| Example Roasters | +1 512-555-0101 | [email protected] | example.com | Coffee shop | 100 Congress Ave, Austin, TX | 4.7 | 812 || Sample Espresso Bar | +1 512-555-0102 | [email protected] | example.org | Cafe | 200 S Lamar Blvd, Austin, TX | 4.5 | 356 |-`--socials` adds `instagram`, `facebook` and `linkedin` columns. `--tech` adds `fb_pixel`,-`google_ads_tag`, `analytics`, `cms`, `mobile_friendly` and `intent_score`, and sorts the hottest-leads first. `--full` keeps all 36 raw fields, including opening hours, coordinates, price range and+`--socials` adds `instagram`, `facebook`, `linkedin`, `tiktok`, `youtube` and `x` columns. `--tech` adds+`fb_pixel`, `google_ads_tag`, `analytics`, `bing_ads`, `linkedin_insight`, `tiktok_pixel`, `cms`,+`mobile_friendly` and `intent_score`, and sorts the hottest leads first. `--verify-emails` adds+`email_host`. `--full` keeps all 36 raw fields, including opening hours, coordinates, price range andreviews.Full walkthrough, platform notes and troubleshooting: [SETUP.md](SETUP.md).@@ -151,11 +153,13 @@ scrapes Google Maps business listings only. It does not scrape social platforms| **8 lead fields by default** | name, phone, email, website, category, address, rating, review count. The engine captures 36 raw fields; `--full` keeps them all. || **Email extraction, on by default** | After the Maps job, the script crawls every business website, 16 at a time: the homepage, then its contact and about pages if the homepage shows no email. It prints your found rate. || **Beyond the 120 cap** | `--grid 3` splits the city into 3 x 3 map tiles, runs them one at a time and dedupes on `place_id`. |-| **Social enrichment** | `--socials` finds Instagram, Facebook and LinkedIn profiles in code (HTTP + regex). Zero AI tokens. |-| **Who already advertises** | `--tech` spots a Meta Pixel, Google Ads tag and analytics on each website, plus its platform (WordPress, Shopify, Wix…) and mobile support, then ranks leads by an `intent_score` from 0 to 100. Zero AI tokens. |-| **Auto-geocoding** | Type a city name. Coordinates are resolved for you via OpenStreetMap. |+| **Social enrichment** | `--socials` finds Instagram, Facebook, LinkedIn, TikTok, YouTube and X profiles in code (HTTP + regex). Zero AI tokens. |+| **Who already advertises** | `--tech` spots a Meta Pixel, Google Ads tag, analytics, Bing Ads, LinkedIn Insight and TikTok pixels on each website, plus its platform (WordPress, Shopify, Wix…) and mobile support, then ranks leads by an `intent_score` from 0 to 100. Zero AI tokens. |+| **Filters** | `--no-website` (businesses that still need a site), `--min-rating`, `--min-reviews`, `--has-phone`, `--has-email`, `--limit`. `--exclude earlier.csv` skips the places you already have, so a weekly re-run returns only new leads. |+| **Email checks** | `--verify-emails` looks up each domain's MX record over DNS (no key), drops dead domains and labels the mail host (`google_workspace`, `microsoft_365`, `other`). |+| **Auto-geocoding** | Type a city name. Coordinates are resolved for you via OpenStreetMap. Repeat `--city` to run several cities into one file. || **Batch jobs** | Many keywords, one job, one CSV. The engine lists a place found by several keywords once. |-| **Proxy rotation** | Built into the engine for large or repeated runs. `socks5`, `socks5h`, `http`, `https`. |+| **Proxy rotation** | Built into the engine for large or repeated runs: `--proxy URL` (repeatable). `socks5`, `socks5h`, `http`, `https`. || **Agent-native** | A skill plus four slash commands. The agent handles create, poll, download, clean and present. || **Safe defaults** | API bound to `127.0.0.1`, and the proxy refuses requests from other websites in your browser (DNS rebinding, cross-site posts). Results git-ignored; website fetches skip private addresses; rate-limit and data-law guardrails built into the skill. |@@ -259,7 +263,7 @@ as untrusted, and a re-audit is requested in**What data do I get for each business?**Name, phone, email, website, category, address, rating and review count by default. Add `--socials`-for Instagram, Facebook and LinkedIn, or `--full` for all 36 raw fields.+for Instagram, Facebook, LinkedIn, TikTok, YouTube and X, or `--full` for all 36 raw fields.**How many leads can I scrape?**Up to about 120 per search. Add `--grid 3` to split the city into 9 map tiles and get past that cap,@@ -284,7 +288,8 @@ your IP **temporarily rate-limited** (minutes to hours). Your Google account is- Run one job at a time. Start at `depth 5` and raise only when needed.- Watch for block signals: `failed` jobs, empty results, or far fewer rows than an identical earlier run.-- Add proxies for large or repeated runs. The engine rotates them automatically. Sponsor+- Add proxies for large or repeated runs: `--proxy socks5://user:pass@host:port` on the script+ (repeatable), or the `"proxies"` array on the raw API. The engine rotates them automatically. Sponsor[Swiftproxy](https://www.swiftproxy.net/?ref=googlemapsscraperkit) takes 10% off with code `PROXY90`:```json"proxies": ["socks5://user:pass@host:port", "http://host2:port2"]diff --git a/docs/google-maps-email-scraper.md b/docs/google-maps-email-scraper.mdindex 422061e..d338c2b 100644--- a/docs/google-maps-email-scraper.md+++ b/docs/google-maps-email-scraper.md@@ -49,7 +49,8 @@ The empty `emails` cell on the last row is normal. That site published no email## What to expect- Expect emails on a fraction of rows, not all of them. A business with no website, or a site with no visible email, returns an empty cell.-- Instagram bios are not read. Add `--socials` to get Instagram, Facebook and LinkedIn profile links from each website instead.+- Instagram bios are not read. Add `--socials` to get Instagram, Facebook, LinkedIn, TikTok, YouTube and X profile links from each website instead.+- Add `--verify-emails` to drop addresses whose domain does not exist and to see who hosts each domain's mail (`email_host`: Google Workspace, Microsoft 365, other). Add `--has-email` to keep only rows with an address.- Email extraction visits every website, so a run with emails is slower than one with `--no-email`.- Google Maps returns about 120 listings per search. Add `--grid 3` to split the city into 9 map tiles and get past that cap, with duplicates removed.- Heavy use without proxies can get your IP temporarily rate-limited by Google for minutes to hours. Run one job at a time and add proxies for large or repeated runs.diff --git a/docs/google-maps-scraper-python.md b/docs/google-maps-scraper-python.mdindex 35b1e3d..5e98a41 100644--- a/docs/google-maps-scraper-python.md+++ b/docs/google-maps-scraper-python.md@@ -22,7 +22,7 @@ On Windows, run the same commands with `py` or `python3`. Python is only the cli|---|---|| `keyword` | The search, with the place baked in, e.g. `"coffee shops in Austin TX"`. || `lat`, `lon` | Coordinates. Optional when you pass `--city`. |-| `--city` | Geocodes a city name to coordinates via OpenStreetMap. |+| `--city` | Geocodes a city name to coordinates via OpenStreetMap. Repeat it to run several cities, one after another, into one deduped file. || `--depth` | How far to scroll each search. Default `5`. Higher is slower and more likely to be blocked. || `--no-email` | Skips email extraction for a faster run. Emails are on by default. || `--no-deep-emails` | Checks homepages only for emails, not contact and about pages. |@@ -32,11 +32,18 @@ On Windows, run the same commands with `py` or `python3`. Python is only the cli| `--json` | Writes JSON instead of CSV. || `--full` | Keeps all 36 raw columns instead of the 8 lead fields. || `--fields` | Comma-separated columns to keep, overriding the lead fields. |-| `--socials` | Adds Instagram, Facebook and LinkedIn from each website. Slower. Zero AI tokens. |-| `--tech` | Adds `fb_pixel`, `google_ads_tag`, `analytics` (yes/no, blank if the site did not load), `cms`, `mobile_friendly` and a 0-100 `intent_score`, and sorts by that score. Shares the homepage fetch with `--socials`. Zero AI tokens. |+| `--socials` | Adds Instagram, Facebook, LinkedIn, TikTok, YouTube and X from each website. Slower. Zero AI tokens. |+| `--tech` | Adds `fb_pixel`, `google_ads_tag`, `analytics`, `bing_ads`, `linkedin_insight`, `tiktok_pixel` (yes/no, blank if the site did not load), `cms`, `mobile_friendly` and a 0-100 `intent_score`, and sorts by that score. Shares the homepage fetch with `--socials`. Zero AI tokens. || `--keyword` | Adds an extra keyword to the same job. Repeatable. || `--lang` | Two-letter language the engine asks Google for, e.g. `pt`, `es`, `de`. Default `en`. Use the market's language so category names come back in it. || `--keywords-file` | Reads one keyword per line from a file. One job, one CSV. |+| `--proxy` | Proxy URL for the engine, repeatable: `socks5://`, `socks5h://`, `http://` or `https://`, `user:pass@` allowed. The engine rotates them. |+| `--min-rating`, `--min-reviews` | Keep rows rated X or higher, or with N or more reviews. Unrated rows drop. |+| `--no-website` | Keep only businesses without a website: the classic web-design prospect list. |+| `--has-phone`, `--has-email` | Keep only rows with a phone, or rows where an email was found. |+| `--exclude FILE` | Drop places already in an earlier results file (`.csv` or `.json`), repeatable. Matches on `place_id`, `cid`, link, or name plus address, so the lean 8-column CSV works. |+| `--limit N` | Stop at N rows after filtering. |+| `--verify-emails` | Checks each email domain's MX record over DNS (dns.google, no key). Drops addresses on domains that do not exist and adds `email_host`: `google_workspace`, `microsoft_365`, `other` or `none`. |The script creates the job, polls until it finishes, downloads the CSV and keeps the 8 lead fields. It prints a short preview and saves the full results to a file.diff --git a/scripts/scrape.py b/scripts/scrape.pyindex 34f643a..b462132 100755--- a/scripts/scrape.py+++ b/scripts/scrape.py@@ -37,7 +37,7 @@ for _stream in (sys.stdout, sys.stderr):except (AttributeError, ValueError): # not a reconfigurable stream (pytest capture, a pipe wrapper)pass-ENV_KEYS = ("SCRAPER_BASE_URL", "SCRAPER_PORT", "SCRAPER_API_KEY")+ENV_KEYS = ("SCRAPER_BASE_URL", "SCRAPER_PORT", "SCRAPER_API_KEY", "SCRAPER_CONTAINER")def load_env_file():@@ -69,7 +69,9 @@ POLL_EVERY = 8 # seconds between status checksPOLL_GRACE = 120 # the engine still has to flush results and flip the statusPOLL_CEILING = 7200 # never wait longer than 2h, whatever --max-time asked forENGINE_MIN_RUNTIME = 180 # the engine floors anything below this (runner/webrunner)-ENGINE_CONTAINER = "gmaps-scraper" # container_name in docker-compose.yml and SETUP's docker run+# container_name in docker-compose.yml and SETUP's docker run; the one the script reads the log of+# and restarts. SCRAPER_CONTAINER in .env for a renamed container or a second stack.+ENGINE_CONTAINER = os.environ.get("SCRAPER_CONTAINER") or "gmaps-scraper"def poll_attempts(max_time):@@ -146,17 +148,31 @@ def in_bbox(row, bbox, margin=0.10):return s - my <= lat <= n + my and w - mx <= lon <= e + mx-def dedupe(rows):- """Keep the first row per place_id, else cid, else link, else title+address."""- seen, out = set(), []+def _keys(r):+ """Every identity a row carries: place_id, cid, link and title+address. A lean 8-column CSV+ from an earlier run has only the last one, so matching on any key lets --exclude use it."""+ ks = [f"{k}:{r[k]}" for k in ("place_id", "cid", "link") if r.get(k)]+ if r.get("title"):+ ks.append(f"ta:{r['title']}|{r.get('address', '')}")+ return ks+++def dedupe(rows, exclude=()):+ """Keep the first row per place; drop rows already present in `exclude` (an earlier run)."""+ seen, out = {k for r in exclude for k in _keys(r)}, []for r in rows:- key = next((f"{k}:{r[k]}" for k in ("place_id", "cid", "link") if r.get(k)),- f"ta:{r.get('title', '')}|{r.get('address', '')}")- if key not in seen:- seen.add(key); out.append(r)+ ks = _keys(r)+ if not any(k in seen for k in ks):+ seen.update(ks); out.append(r)return out+def load_rows(path):+ """Rows from an earlier results file, .csv or .json."""+ with open(path, encoding="utf-8", newline="") as f:+ return json.load(f) if path.lower().endswith(".json") else list(csv.DictReader(f))++def _local_docker():return urllib.parse.urlparse(BASE).hostname in ("localhost", "127.0.0.1", "::1") \and shutil.which("docker") is not None@@ -334,13 +350,16 @@ SOCIAL_RE = {"instagram": re.compile(r"https?://(?:www\.)?instagram\.com/([A-Za-z0-9_.]+)", re.I),"facebook": re.compile(r"https?://(?:www\.|m\.|web\.)?facebook\.com/([A-Za-z0-9_.\-]+)", re.I),"linkedin": re.compile(r"https?://(?:[a-z]{2,3}\.)?linkedin\.com/(?:company|in|school)/([A-Za-z0-9_.\-%]+)", re.I),+ "tiktok": re.compile(r"https?://(?:www\.)?tiktok\.com/@([A-Za-z0-9_.]+)", re.I),+ "youtube": re.compile(r"https?://(?:www\.)?youtube\.com/(?:@|channel/|c/|user/)([A-Za-z0-9_.\-]+)", re.I),+ "x": re.compile(r"https?://(?:www\.)?(?:twitter|x)\.com/([A-Za-z0-9_]{1,15})\b", re.I),}SOCIAL_FIELDS = list(SOCIAL_RE)_SKIP_HANDLE = {"", "home", "pages", "people", "help", "about", "policies", "policy","legal", "tos", "privacy", "settings", "sharer", "tr", "profile.php","plugins", "dialog", "intent", "login", "share.php", "permalink.php","p", "reel", "reels", "explore", "stories", "tv", "watch", "events",- "groups", "marketplace", "gaming", "photo", "hashtag", "search"}+ "groups", "marketplace", "gaming", "photo", "hashtag", "search", "share", "i"}def _is_public_ip(ip):try:@@ -442,7 +461,7 @@ def _each_site(todo, work, label, workers=None):with ThreadPoolExecutor(max_workers=workers or SITE_WORKERS) as ex:for _ in ex.map(work, todo):done += 1- print(f"\r {label}: {done}/{len(todo)} sites checked", end="", flush=True)+ print(f"\r {label}: {done}/{len(todo)} checked", end="", flush=True)print()def enrich_sites(results, find, fields, label, workers=None):@@ -460,6 +479,9 @@ TECH_RE = {"fb_pixel": re.compile(r"connect\.facebook\.net/[^\"']*fbevents\.js|fbq\(", re.I),"google_ads_tag": re.compile(r"googleadservices\.com|AW-\d{6,}|googleads\.g\.doubleclick\.net", re.I),"analytics": re.compile(r"googletagmanager\.com/gtag|gtag\(|UA-\d{4,}|G-[A-Z0-9]{6,}", re.I),+ "bing_ads": re.compile(r"bat\.bing\.com|window\.uetq", re.I),+ "linkedin_insight": re.compile(r"snap\.licdn\.com|_linkedin_partner_id", re.I),+ "tiktok_pixel": re.compile(r"analytics\.tiktok\.com|ttq\.load\(", re.I),}CMS_RE = [("wordpress", re.compile(r"/wp-content/|/wp-includes/", re.I)),@@ -574,7 +596,7 @@ def contact_links(html, base_url):base_url = "http://" + base_urlhost, out = _host(base_url), []for href, text in HREF_RE.findall(html or ""):- if not re.search(r"contact|about", href + " " + text, re.I):+ if not re.search(r"contact|about|team|staff|support", href + " " + text, re.I):continueurl = urllib.parse.urldefrag(urllib.parse.urljoin(base_url, href.strip()))[0]if url.startswith(("http://", "https://")) and _host(url) == host and url not in out:@@ -598,6 +620,101 @@ def enrich_emails(results, deep=True, workers=None):_each_site([r for r in results if r.get("website") and not r.get("emails")], work, "emails", workers)+# ── Optional email domain check (--verify-emails): can the domain receive mail, and who hosts it ──+def _doh_mx(domain):+ """MX lookup over DNS-over-HTTPS (Google's public resolver, no key) -> the JSON answer."""+ url = "https://dns.google/resolve?" + urllib.parse.urlencode({"name": domain, "type": "MX"})+ with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": UA}), timeout=8) as r:+ return json.load(r)+[email protected]_cache(maxsize=1024)+def mx_host(domain):+ """'nxdomain' when the domain does not exist, 'none' when it has no MX record, else who runs+ its mail: 'google_workspace', 'microsoft_365' or 'other'. '' when the lookup itself failed."""+ try:+ d = _doh_mx(domain)+ except Exception:+ return ""+ if d.get("Status") == 3:+ return "nxdomain"+ mx = " ".join(a.get("data", "") for a in d.get("Answer") or [] if a.get("type") == 15).lower()+ if not mx:+ return "none" if d.get("Status") == 0 else ""+ if "google.com" in mx or "googlemail.com" in mx:+ return "google_workspace"+ return "microsoft_365" if "outlook.com" in mx else "other"++def verify_emails(rows):+ """Drop addresses on domains that do not exist; label the rest with email_host."""+ def work(r):+ kept = [(e, mx_host(e.rpartition("@")[2])) for e in r["emails"].split(", ") if e]+ kept = [(e, h) for e, h in kept if h != "nxdomain"]+ r["emails"] = ", ".join(e for e, _ in kept)+ r["email_host"] = kept[0][1] if kept else ""+ _each_site([r for r in rows if r.get("emails")], work, "email domains", workers=8)+++def keep_row(r, a):+ """The row filters, minus --has-email (applied after the crawl). A missing rating or review+ count reads as 0, so --min-rating and --min-reviews drop unrated listings."""+ return not ((a.min_rating and _num(r.get("review_rating")) < a.min_rating)+ or (a.min_reviews and _num(r.get("review_count"), int) < a.min_reviews)+ or (a.no_website and r.get("website"))+ or (a.has_phone and not r.get("phone")))+++def scrape_place(a, body, city, keywords):+ """All the engine jobs for one place: a single job, or --grid tiles one at a time clipped to the+ city box -> (first job id or None, rows, failed tile count, tile count)."""+ lat, lon, bbox, cells = a.lat, a.lon, None, None+ if a.grid: # always geocodes, because it needs the bounding box+ print(f"▶ Geocoding \"{city}\"…")+ geo = geocode(city, with_bbox=True)+ if not geo or not geo[2]:+ print("✗ Could not resolve a bounding box. Try a clearer --city.")+ return None, [], 0, 0+ lat, lon, bbox = geo+ cells = grid_cells(bbox, a.grid)+ print(f" → bbox S{bbox[0]} N{bbox[1]} W{bbox[2]} E{bbox[3]}, {len(cells)} tiles")+ elif city or not (lat and lon): # explicit lat/lon, else geocode --city, else the first keyword+ place = city or keywords[0]+ print(f"▶ Geocoding \"{place}\"…")+ coords = geocode(place)+ if not coords:+ print("✗ Could not resolve coordinates. Pass lat/lon or a clearer --city.")+ return None, [], 0, 0+ lat, lon = coords+ print(f" → {lat}, {lon}")+ body = dict(body, lat=str(lat), lon=str(lon))+ for k in keywords:+ print(f" • {k}")+ if not cells:+ print(f"▶ Creating job: {len(keywords)} keyword(s) @ {lat},{lon} depth={a.depth}")+ try:+ job_id, rows = run_job(body)+ except RuntimeError as e:+ print(str(e))+ return None, [], 0, 0+ return job_id, rows, 0, 0+ print(f"▶ Grid {a.grid}x{a.grid}: {len(cells)} jobs, one at a time, {len(keywords)} keyword(s) depth={a.depth}")+ job_id, rows, skipped = None, [], 0+ for i, cell in enumerate(cells, 1): # sequential on purpose: never parallel jobs against Google+ print(f"▶ Cell {i}/{len(cells)} @ {cell['lat']},{cell['lon']} radius={cell['radius']}m")+ try:+ jid, cell_rows = run_job(dict(body, **cell))+ except Exception as e:+ print(f"\n ⚠ cell {i}/{len(cells)} skipped: {str(e).strip()}")+ skipped += 1+ continue+ job_id = job_id or jid+ rows = dedupe(rows + cell_rows)+ print(f" cell {i}/{len(cells)}: {len(cell_rows)} rows, {len(rows)} unique so far")+ # radius does not clip results in normal mode, so tiles leak outside the city+ kept = [r for r in rows if in_bbox(r, bbox)]+ print(f" dropped {len(rows) - len(kept)} rows outside the city box")+ return job_id, kept, skipped, len(cells)++def main():ap = argparse.ArgumentParser(description="Scrape Google Maps business listings.")ap.add_argument("keyword", nargs="?", help='e.g. "coffee shops in Austin TX"')@@ -605,7 +722,11 @@ def main():ap.add_argument("lon", nargs="?", help="longitude (optional if --city given)")ap.add_argument("--keyword", dest="also", action="append", help="extra keyword (repeatable)")ap.add_argument("--keywords-file", help="file with one keyword per line (batch)")- ap.add_argument("--city", help="city/place to auto-geocode for lat/lon")+ ap.add_argument("--city", action="append",+ help="city/place to auto-geocode for lat/lon; repeatable, one run per city into one file")+ ap.add_argument("--proxy", action="append", metavar="URL",+ help="proxy for the engine, repeatable (socks5://, socks5h://, http://, https://, "+ "user:pass@ allowed); the engine rotates them")ap.add_argument("--depth", type=int, default=5)# The engine passes this to Google as the interface language, which decides the language of# the category names it returns (and nudges which listings come back). It has to match the@@ -634,8 +755,22 @@ def main():ap.add_argument("--socials", action="store_true",help="also find Instagram/Facebook/LinkedIn from each website (0 LLM tokens; slower)")ap.add_argument("--tech", action="store_true",- help="also detect Meta Pixel, Google Ads tag, analytics, site platform and mobile "- "support on each website, and sort by an intent_score (0 LLM tokens; slower)")+ help="also detect Meta Pixel, Google Ads tag, analytics, Bing, LinkedIn and TikTok "+ "pixels, site platform and mobile support on each website, and sort by an "+ "intent_score (0 LLM tokens; slower)")+ ap.add_argument("--verify-emails", action="store_true",+ help="check each email domain's MX record over DNS (dns.google, no key): drops "+ "addresses on domains that do not exist and adds an email_host column "+ "(google_workspace, microsoft_365, other, none)")+ f = ap.add_argument_group("filters", "applied before the email crawl, except --has-email")+ f.add_argument("--min-rating", type=float, metavar="X", help="keep rows rated X or higher")+ f.add_argument("--min-reviews", type=int, metavar="N", help="keep rows with N or more reviews")+ f.add_argument("--no-website", action="store_true", help="keep only businesses without a website")+ f.add_argument("--has-phone", action="store_true", help="keep only rows with a phone number")+ f.add_argument("--has-email", action="store_true", help="keep only rows where an email was found")+ f.add_argument("--exclude", action="append", metavar="FILE",+ help="drop places already in this earlier results file (.csv or .json); repeatable")+ f.add_argument("--limit", type=int, metavar="N", help="stop at N rows after filtering")a = ap.parse_args()keywords = collect_keywords(a)@@ -651,28 +786,14 @@ def main():if not 1 <= a.grid <= 10:sys.exit("✗ --grid must be between 1 and 10 (10 x 10 = 100 jobs is the ceiling).")- # Resolve coordinates: explicit lat/lon, else geocode --city, else geocode first keyword.- # --grid always geocodes --city, because it needs the city's bounding box.- lat, lon = a.lat, a.lon- if a.grid:- print(f"▶ Geocoding \"{a.city}\"…")- geo = geocode(a.city, with_bbox=True)- if not geo or not geo[2]:- sys.exit("✗ Could not resolve a bounding box. Try a clearer --city.")- lat, lon, bbox = geo- cells = grid_cells(bbox, a.grid)- print(f" → bbox S{bbox[0]} N{bbox[1]} W{bbox[2]} E{bbox[3]}, {len(cells)} tiles")- elif not (lat and lon):- place = a.city or keywords[0]- print(f"▶ Geocoding \"{place}\"…")- coords = geocode(place)- if not coords:- sys.exit("✗ Could not resolve coordinates. Pass lat/lon or a clearer --city.")- lat, lon = coords- print(f" → {lat}, {lon}")+ cities = a.city or [None]+ try:+ excluded = [r for p in a.exclude or [] for r in load_rows(p)]+ except (OSError, ValueError) as e:+ sys.exit(f"✗ Cannot read --exclude file: {e}")# --- WARN, don't block: flag high-volume jobs (proceed anyway) ---- if a.depth >= 15 or len(keywords) >= 10 or (a.grid or 1) ** 2 >= 4:+ if a.depth >= 15 or len(keywords) >= 10 or (a.grid or 1) ** 2 >= 4 or len(cities) >= 3:print("⚠️ Heads-up: this is a large job (high depth, many keywords or a grid).")print(" Running it back-to-back without proxies can get your IP temporarily rate-limited by")print(" Google (clears in minutes–hours; returns empty/failed jobs meanwhile). For big or")@@ -685,46 +806,28 @@ def main():sys.exit(f"✗ Scraper not reachable at {BASE} — run 'docker compose up -d' first.\n ({e})")body = {"name": "scrape-py", "keywords": keywords, "lang": a.lang, "zoom": 15,- "lat": str(lat), "lon": str(lon), "fast_mode": False, "radius": 10000,+ "fast_mode": False, "radius": 10000,# The engine's own email pass visits every website in a browser, one per worker: it took# most of each job and found fewer emails than the crawl below (OpenLeadFinder's# approach), so the engine only reads Google Maps."depth": a.depth, "email": False, "max_time": a.max_time}- skipped = [] # grid tiles that failed- if not a.grid:- print(f"▶ Creating job: {len(keywords)} keyword(s) @ {lat},{lon} depth={a.depth}")- for k in keywords:- print(f" • {k}")- try:- job_id, rows = run_job(body)- except RuntimeError as e:- sys.exit(str(e))- else:- print(f"▶ Grid {a.grid}x{a.grid}: {len(cells)} jobs, one at a time, "- f"{len(keywords)} keyword(s) depth={a.depth}")- for k in keywords:- print(f" • {k}")- job_id, rows = None, []- for i, cell in enumerate(cells, 1): # sequential on purpose: never parallel jobs against Google- print(f"▶ Cell {i}/{len(cells)} @ {cell['lat']},{cell['lon']} radius={cell['radius']}m")- try:- jid, cell_rows = run_job(dict(body, **cell))- except Exception as e:- print(f"\n ⚠ cell {i}/{len(cells)} skipped: {str(e).strip()}")- skipped.append(i)- continue- job_id = job_id or jid- rows = dedupe(rows + cell_rows)- print(f" cell {i}/{len(cells)}: {len(cell_rows)} rows, {len(rows)} unique so far")- if job_id is None:- sys.exit("✗ Every grid cell failed. You may be rate-limited — wait or add proxies.")- # radius does not clip results in normal mode, so tiles leak outside the city- kept = [r for r in rows if in_bbox(r, bbox)]- print(f" dropped {len(rows) - len(kept)} rows outside the city box")- rows = kept+ if a.proxy:+ body["proxies"] = a.proxy+ job_id, rows, skipped, ncells = None, [], 0, 0+ for city in cities: # one place at a time, like the grid tiles+ jid, place_rows, s, n = scrape_place(a, body, city, keywords)+ job_id, skipped, ncells = job_id or jid, skipped + s, ncells + n+ rows = dedupe(rows + place_rows)+ if job_id is None:+ sys.exit("✗ No job finished. You may be rate-limited — wait or add proxies.")for r in rows:r["website"] = clean_website(r.get("website", ""))- rows = dedupe(rows) # the engine lists a place once per job; this also covers repeats across runs of --keyword+ before = len(rows)+ rows = [r for r in dedupe(rows, exclude=excluded) if keep_row(r, a)]+ if len(rows) != before:+ print(f" filters and --exclude kept {len(rows)} of {before} rows")+ if a.limit and not a.has_email:+ rows = rows[:a.limit]if a.full:fields = list(rows[0].keys()) if rows else LEADelif a.fields:@@ -733,8 +836,7 @@ def main():fields = LEADprint(f"✓ Done — {len(rows)} businesses (fields: {', '.join(fields)}).")if skipped:- print(f"⚠ Incomplete: {len(skipped)} of {len(cells)} tiles failed (cells "- f"{', '.join(map(str, skipped))}), so those parts of the city are missing. Rerun later.")+ print(f"⚠ Incomplete: {skipped} of {ncells} tiles failed, so parts of the area are missing. Rerun later.")if a.email and "emails" in fields: # on the raw rows, so --fields without website still workspages = "homepage, then contact and about pages" if a.deep_emails else "homepage only"@@ -742,10 +844,19 @@ def main():enrich_emails(rows, deep=a.deep_emails)total, n = sum(1 for r in rows if r.get("emails")), len(rows)print(f" emails found for {total}/{n} businesses ({round(100 * total / n) if n else 0}%)")+ if a.verify_emails:+ print("▶ Checking each email domain's MX record over DNS (dns.google)…")+ verify_emails(rows)+ fields = fields + ["email_host"]+ dead = total - sum(1 for r in rows if r.get("emails"))+ print(f" dropped {dead} addresses on domains that do not exist")+ if a.has_email:+ rows = [r for r in rows if r.get("emails")][:a.limit or None]+ print(f" --has-email kept {len(rows)} rows")results = [{k: r.get(k, "") for k in fields} for r in rows]if a.socials:- print("▶ Finding socials (Instagram / Facebook / LinkedIn) on each website…")+ print("▶ Finding socials (Instagram / Facebook / LinkedIn / TikTok / YouTube / X) on each website…")print(" ℹ Cost: this runs in CODE = 0 LLM tokens. It only adds ~40–50 tokens per business")print(" IF you load the results into an AI's context (negligible if you keep the file on disk).")enrich_sites(results, _find_socials, SOCIAL_FIELDS, "socials")diff --git a/scripts/test_scrape.py b/scripts/test_scrape.pyindex 71f9956..164f6d1 100644--- a/scripts/test_scrape.py+++ b/scripts/test_scrape.py@@ -270,7 +270,19 @@ class TechSignals(unittest.TestCase):'<meta name="viewport" content="width=device-width"><link href="/wp-content/a.css">'"<script>gtag('config','AW-123456789')</script>")self.assertEqual(scrape._find_tech(html), {"fb_pixel": "yes", "google_ads_tag": "yes",- "analytics": "yes", "cms": "wordpress", "mobile_friendly": "yes"})+ "analytics": "yes", "bing_ads": "no", "linkedin_insight": "no",+ "tiktok_pixel": "no", "cms": "wordpress", "mobile_friendly": "yes"})+ more = '<script src="https://bat.bing.com/bat.js"></script><script>ttq.load("X"); _linkedin_partner_id="1"</script>'+ self.assertEqual({k: v for k, v in scrape._find_tech(more).items() if v == "yes"},+ {"bing_ads": "yes", "linkedin_insight": "yes", "tiktok_pixel": "yes"})++ def test_more_socials(self):+ html = ('<a href="https://www.tiktok.com/@acme.shop">t</a> <a href="https://youtube.com/@AcmeTV">y</a>'+ ' <a href="https://x.com/acme_inc">x</a> <a href="https://twitter.com/intent/tweet">no</a>')+ out = scrape._find_socials(html)+ self.assertEqual((out["tiktok"], out["youtube"], out["x"]),+ ("https://www.tiktok.com/@acme.shop", "https://youtube.com/@AcmeTV", "https://x.com/acme_inc"))+ self.assertEqual(scrape._find_socials('<a href="https://twitter.com/intent/tweet">no</a>')["x"], "")def test_unreachable_site_is_unknown_not_no(self):self.assertEqual(set(scrape._find_tech("").values()), {""})@@ -435,13 +447,89 @@ class EndToEnd(unittest.TestCase):text = log.getvalue()self.assertIn("cell 4/4: 3 rows, 6 unique so far", text)self.assertIn("cell 3/4 skipped", text)- self.assertIn("Incomplete: 1 of 4 tiles failed (cells 3)", text)+ self.assertIn("Incomplete: 1 of 4 tiles failed", text)self.assertIn("dropped 1 rows outside the city box", text)with open(out, newline="") as f:rows = list(csv.reader(f))self.assertEqual(rows[0], scrape.LEAD)self.assertEqual([r[0] for r in rows[1:]], ["Biz A", "Biz B", "Biz C", "Biz D", "Biz E"])+ def test_two_cities_filters_exclude_limit(self):+ # --city twice = two jobs into one deduped file; --exclude drops a place from an earlier lean+ # CSV (title+address only); --min-reviews and --limit trim the rest.+ MockEngine.created, MockEngine.polls = 0, {}+ srv = ThreadingHTTPServer(("127.0.0.1", 0), MockEngine)+ threading.Thread(target=srv.serve_forever, daemon=True).start()+ d = tempfile.mkdtemp()+ out, earlier = os.path.join(d, "leads.csv"), os.path.join(d, "earlier.csv")+ with open(earlier, "w", newline="") as f:+ w = csv.DictWriter(f, fieldnames=scrape.LEAD); w.writeheader()+ w.writerow({"title": "Biz A", "address": ""})+ argv = ["scrape.py", "plumbers", "--city", "Austin, TX", "--city", "Round Rock, TX", "--no-email",+ "--exclude", earlier, "--limit", "2", "--out", out]+ log = io.StringIO()+ try:+ with mock.patch.object(scrape, "BASE", f"http://127.0.0.1:{srv.server_port}"), \+ mock.patch.object(scrape, "geocode", return_value=("30.25", "-97.75")) as geo, \+ mock.patch.object(scrape.time, "sleep"), \+ mock.patch.object(sys, "argv", argv), redirect_stdout(log):+ scrape.main()+ finally:+ srv.shutdown(); srv.server_close()+ self.assertEqual(geo.call_count, 2)+ self.assertIn("kept 3 of 4 rows", log.getvalue()) # A, B, C + C, D -> 4 unique, minus excluded A+ with open(out, newline="") as f:+ rows = list(csv.DictReader(f))+ self.assertEqual([r["title"] for r in rows], ["Biz B", "Biz C"]) # --limit 2+++class Filters(unittest.TestCase):+ def test_keep_row(self):+ a = mock.Mock(min_rating=4.5, min_reviews=10, no_website=False, has_phone=True)+ self.assertTrue(scrape.keep_row({"review_rating": "4.6", "review_count": "1,204", "phone": "+1"}, a))+ self.assertFalse(scrape.keep_row({"review_rating": "4.4", "review_count": "50", "phone": "+1"}, a))+ self.assertFalse(scrape.keep_row({"review_rating": "4.9", "review_count": "9", "phone": "+1"}, a))+ self.assertFalse(scrape.keep_row({"review_rating": "4.9", "review_count": "99", "phone": ""}, a))+ self.assertFalse(scrape.keep_row({"review_rating": "", "review_count": "99", "phone": "+1"}, a)) # unrated+ nw = mock.Mock(min_rating=None, min_reviews=None, no_website=True, has_phone=False)+ self.assertTrue(scrape.keep_row({"website": ""}, nw))+ self.assertFalse(scrape.keep_row({"website": "http://x.com"}, nw))++ def test_exclude_matches_lean_rows(self):+ new = [{"place_id": "P1", "title": "Joe's", "address": "1 Main"}, {"place_id": "P2", "title": "Ann's"}]+ self.assertEqual(scrape.dedupe(new, exclude=[{"title": "Joe's", "address": "1 Main"}]), new[1:])+ self.assertEqual(scrape.dedupe(new, exclude=[{"place_id": "P2"}]), new[:1])++ def test_load_rows_csv_and_json(self):+ d = tempfile.mkdtemp()+ j = os.path.join(d, "a.json"); open(j, "w").write('[{"title": "A"}]')+ c = os.path.join(d, "a.csv"); open(c, "w").write("title,phone\nB,1\n")+ self.assertEqual(scrape.load_rows(j), [{"title": "A"}])+ self.assertEqual(scrape.load_rows(c), [{"title": "B", "phone": "1"}])+++class VerifyEmails(unittest.TestCase):+ def test_mx_host_labels_and_drops_dead_domains(self):+ answers = {+ "gws.com": {"Status": 0, "Answer": [{"type": 15, "data": "10 aspmx.l.google.com."}]},+ "m365.com": {"Status": 0, "Answer": [{"type": 15, "data": "0 m365-com.mail.protection.outlook.com."}]},+ "own.com": {"Status": 0, "Answer": [{"type": 15, "data": "10 mail.own.com."}]},+ "nomx.com": {"Status": 0},+ "dead.com": {"Status": 3},+ }+ scrape.mx_host.cache_clear()+ rows = [{"emails": "[email protected], [email protected]"}, {"emails": "[email protected]"}, {"emails": "[email protected]"},+ {"emails": "[email protected]"}, {"emails": "[email protected]"}, {"emails": ""}]+ with mock.patch.object(scrape, "_doh_mx", side_effect=lambda dom: answers[dom]), \+ redirect_stdout(io.StringIO()):+ scrape.verify_emails(rows)+ self.assertEqual([(r["emails"], r.get("email_host", "-")) for r in rows],+ [("[email protected]", "google_workspace"), ("[email protected]", "microsoft_365"),+ ("[email protected]", "other"), ("[email protected]", "none"), ("", ""), ("", "-")])+ scrape.mx_host.cache_clear()+ with mock.patch.object(scrape, "_doh_mx", side_effect=OSError("offline")):+ self.assertEqual(scrape.mx_host("x.com"), "") # lookup failed: keep the address, no label+if __name__ == "__main__":unittest.main(verbosity=2)