Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
feat(ui): What and Where boxes take only preset business types and GeoNames cities
Mahan Jafari1ecf5a88f73a
diff --git a/CREDITS.md b/CREDITS.mdindex 92d55b2..2d8ff2c 100644--- a/CREDITS.md+++ b/CREDITS.md@@ -26,13 +26,14 @@ straight into `.engine/` on your machine, under Google's terms for Chrome (its `open-source licenses listed in its `LICENSE.headless_shell` file. The Docker image installs Debian's`chromium` package instead, under Chromium's BSD-style license and the notices Debian ships with it.-## Business category names: lobstrio/google-business-categories+## City list: GeoNames-The web UI's "Add a business type" box suggests Google's business category names (4,053, US-English) from [lobstrio/google-business-categories](https://github.com/lobstrio/google-business-categories)-by lobstr.io. Your browser loads the list from GitHub at runtime, pinned to one commit in-`engine/serve.py`, and caches it for 30 days. This repository does not contain or redistribute it,-because that repository has no license file.+The web UI's Where box offers only real places: about 168,000 cities and towns of more than 1,000+people (name, region, country, coordinates), from the [GeoNames](https://www.geonames.org/)+`cities1000` dump, licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).+`engine/presets/cities.tsv.gz` is that data, cut to those five fields, one row per city label.++The What box's business types (`engine/presets/business-types.json`) are a hand-picked list for this kit.## Website signals adapted from OpenLeadFinderdiff --git a/README.md b/README.mdindex 65cdf74..3a12dd8 100644--- a/README.md+++ b/README.md@@ -227,22 +227,22 @@ One keyword per line. One job. One CSV.Google for minutes to hours. When Google answers with its unusual-traffic page, the engine waitsand retries the keyword up to 2 times, then stops the job and keeps the rows found before that.Your Google account is not affected. See [Responsible use](#responsible-use).-- **The web UI is a simple form for beginners.** Up front are "What to search" (one keyword per line,- with the city named in each), an optional business-type box, and two options that are on by- default: "Emails" and "Social profiles", under "Collect". Pick a business type and the page adds- `<Type> in ` to "What to search" (for example `Bakery in `) and puts the cursor there, so you type- the city. Enter in the business-type box adds the line and never starts a search. A search line- with no place (no " in ", " near " or comma) and no map centre gets a warning first; a second press- of "Start anyway" runs it. Three collapsed sections hold the rest: "Location+- **The web UI is a simple form for beginners.** Up front are "What" (a business type) and "Where"+ (a city), and two options that are on by default: "Emails" and "Social profiles", under "Collect".+ Both boxes take only picks from their lists, and each pick becomes a tag you can remove: type a+ few letters, then pick. Text that is not in a list is refused, and Enter never starts a search.+ Three collapsed sections hold the rest: "Locationand language", "Advanced options" and "Proxies". The job list shows each job's status with aprogress bar (a review stage reads "reading reviews 7 of 20"), its times in your local time, andCSV, JSON and Delete buttons. Downloads are named after the search, for example`bakery-in-portland-or-c063a815.csv`. It downloads the raw CSV or JSON, and its email andsocial options read homepages only, so the scripts find more emails and return the clean lead- fields. The page loads- two static files from GitHub (`raw.githubusercontent.com`), with no tracking: the sponsor box's- `assets/sponsors/sponsors.json` from this repo (cached for a day), and the business category list- behind its "Add a business type" box (cached for 30 days). Scraped data stays on your machine.+ fields. Its What and Where boxes take only picks from their lists: about 200 business types+ (`engine/presets/business-types.json`, add a niche there) and about 168,000 cities from GeoNames,+ so a search never names a place that does not exist. Each type is searched in each city, one+ job per city, with the map centred on that city. The page loads one static file from GitHub+ (`raw.githubusercontent.com`), with no tracking: the sponsor box's+ `assets/sponsors/sponsors.json` from this repo (cached for a day). Scraped data stays on your machine.- **Service-area businesses often have no address.** Plumbers, cleaners and other mobile servicesoften hide their street address on Google, so `address` is empty for them. The city and the phoneusually remain.diff --git a/SETUP.md b/SETUP.mdindex 0106fd4..2add5f1 100644--- a/SETUP.md+++ b/SETUP.md@@ -109,15 +109,14 @@ web UI and API docs in the browser:- API docs: **http://localhost:8080/api/docs** (a plain page; the OpenAPI JSON is athttp://localhost:8080/api/openapi.json). Neither needs the API key.-The page has a form. Up front is "What to search": one keyword per line, with the city named in each-("coffee shops in Austin TX"). Below it are an optional "Add a business type" box and two checkboxes-that are on by default, "Emails" and "Social profiles", under "Collect". Both read each business's homepage.-Pick a business type and the page adds `<Type> in ` to "What to search" (for example `Bakery in `) and-puts the cursor there, so you type the city. Pressing Enter in the business type box adds the line and-never starts a search. A search line with no place (no " in ", " near " or comma) and no map centre-gets a warning first; a second press of "Start anyway" runs it. Three collapsed sections hold the rest:-- **Location and language:** an optional map centre pasted as `30.2672, -97.7431`, zoom, language and- country.+The page has a form. Up front are "What" and "Where": type a few letters, then pick a business type+(about 200, such as "Dentists") and a city (about 168,000 from GeoNames, such as "Austin, Texas, US").+Both boxes take only picks from their lists, so a search never names a place or a type that does not+exist; add as many of each as you like. Each type is searched in each city, one job per city, with the+map centred on that city ("Dentists in Austin, Texas, US"). Pressing Enter in either box adds the pick+and never starts a search. Below them are two checkboxes that are on by default, "Emails" and "Social+profiles", under "Collect". Both read each business's homepage. Three collapsed sections hold the rest:+- **Location and language:** zoom, language and country (default: the city's country).- **Advanced options:** depth, time limit, reviews per place (0 to 5) and an MX check of email domains.- **Proxies:** one per line. `http://` and `https://` work with or without `user:pass`; `socks5://`works without a login.diff --git a/engine/presets/business-types.json b/engine/presets/business-types.jsonnew file mode 100644index 0000000..b1f494b--- /dev/null+++ b/engine/presets/business-types.json@@ -0,0 +1,203 @@+[+ "Dentists",+ "Orthodontists",+ "Cosmetic Dentists",+ "Pediatric Dentists",+ "Dental Clinics",+ "Chiropractors",+ "Physical Therapists",+ "Massage Therapists",+ "Acupuncturists",+ "Optometrists",+ "Dermatologists",+ "Plastic Surgeons",+ "Med Spas",+ "Medical Spas",+ "Weight Loss Clinics",+ "Urgent Care Clinics",+ "Veterinarians",+ "Animal Hospitals",+ "Pet Groomers",+ "Pet Boarding",+ "Dog Trainers",+ "Plumbers",+ "Electricians",+ "HVAC Contractors",+ "Roofing Contractors",+ "General Contractors",+ "Remodeling Contractors",+ "Kitchen Remodelers",+ "Bathroom Remodelers",+ "Home Builders",+ "Painters",+ "Painting Contractors",+ "Flooring Contractors",+ "Landscapers",+ "Landscaping Companies",+ "Lawn Care Services",+ "Tree Services",+ "Pest Control",+ "Fencing Contractors",+ "Concrete Contractors",+ "Masonry Contractors",+ "Handyman Services",+ "Window Installers",+ "Garage Door Repair",+ "Solar Installers",+ "Pool Builders",+ "Pool Cleaning Services",+ "Septic Services",+ "Foundation Repair",+ "Insulation Contractors",+ "Gutter Installers",+ "Locksmiths",+ "Appliance Repair",+ "Junk Removal",+ "Dumpster Rental",+ "Moving Companies",+ "Cleaning Services",+ "House Cleaning Services",+ "Commercial Cleaning",+ "Carpet Cleaning",+ "Window Cleaning",+ "Pressure Washing",+ "Restoration Services",+ "Water Damage Restoration",+ "Auto Repair Shops",+ "Auto Body Shops",+ "Car Dealers",+ "Used Car Dealers",+ "RV Dealers",+ "Motorcycle Dealers",+ "Auto Detailing",+ "Car Wash",+ "Tire Shops",+ "Towing Services",+ "Windshield Repair",+ "Transmission Repair",+ "Oil Change Services",+ "Real Estate Agents",+ "Real Estate Agencies",+ "Property Management",+ "Mortgage Brokers",+ "Home Inspectors",+ "Appraisers",+ "Title Companies",+ "Insurance Agencies",+ "Insurance Brokers",+ "Financial Advisors",+ "Accountants",+ "Bookkeepers",+ "Tax Preparers",+ "CPA Firms",+ "Attorneys",+ "Personal Injury Lawyers",+ "Divorce Lawyers",+ "Family Lawyers",+ "Criminal Defense Lawyers",+ "Immigration Lawyers",+ "Bankruptcy Lawyers",+ "Estate Planning Lawyers",+ "Business Consultants",+ "Marketing Agencies",+ "Digital Marketing Agencies",+ "SEO Agencies",+ "Web Design Agencies",+ "Advertising Agencies",+ "Graphic Designers",+ "Photographers",+ "Wedding Photographers",+ "Videographers",+ "Printing Services",+ "Sign Companies",+ "Restaurants",+ "Pizza Restaurants",+ "Mexican Restaurants",+ "Italian Restaurants",+ "Chinese Restaurants",+ "Sushi Restaurants",+ "Steakhouses",+ "Seafood Restaurants",+ "Cafes",+ "Coffee Shops",+ "Bakeries",+ "Bars",+ "Breweries",+ "Wineries",+ "Food Trucks",+ "Caterers",+ "Ice Cream Shops",+ "Juice Bars",+ "Gyms",+ "Fitness Centers",+ "Yoga Studios",+ "Pilates Studios",+ "CrossFit Gyms",+ "Personal Trainers",+ "Martial Arts Schools",+ "Dance Studios",+ "Nutritionists",+ "Hair Salons",+ "Barber Shops",+ "Nail Salons",+ "Beauty Salons",+ "Spas",+ "Day Spas",+ "Tanning Salons",+ "Tattoo Shops",+ "Lash Studios",+ "Waxing Salons",+ "Wedding Planners",+ "Event Planners",+ "Event Venues",+ "Florists",+ "Party Rentals",+ "DJs",+ "Daycares",+ "Preschools",+ "Tutoring Services",+ "Music Schools",+ "Driving Schools",+ "Private Schools",+ "Childcare Centers",+ "Clothing Stores",+ "Boutiques",+ "Jewelry Stores",+ "Furniture Stores",+ "Mattress Stores",+ "Hardware Stores",+ "Garden Centers",+ "Pet Stores",+ "Bike Shops",+ "Sporting Goods Stores",+ "Gift Shops",+ "Consignment Shops",+ "Vape Shops",+ "Liquor Stores",+ "Convenience Stores",+ "Grocery Stores",+ "Hotels",+ "Motels",+ "Bed and Breakfasts",+ "Travel Agencies",+ "Storage Facilities",+ "Self Storage",+ "Security Companies",+ "Staffing Agencies",+ "Recruiting Agencies",+ "IT Services",+ "Managed IT Services",+ "Computer Repair",+ "Home Health Care",+ "Assisted Living Facilities",+ "Nursing Homes",+ "Funeral Homes",+ "Therapists",+ "Counselors",+ "Psychologists",+ "Psychiatrists",+ "Interior Designers",+ "Architects",+ "Engineering Firms",+ "Surveyors"+]diff --git a/engine/presets/cities.tsv.gz b/engine/presets/cities.tsv.gznew file mode 100644index 0000000..cc2fe8eBinary files /dev/null and b/engine/presets/cities.tsv.gz differdiff --git a/engine/serve.py b/engine/serve.pyindex 97dae4b..1540485 100644--- a/engine/serve.py+++ b/engine/serve.py@@ -15,7 +15,7 @@ Google Maps, for email or socials jobs to the businesses' own websites, and forto Google's public DNS (dns.google), nothing else.Standard library only."""-import argparse, csv, hmac, html, io, json, os, re, secrets, signal, socket, socketserver, sys, threading, unicodedata+import argparse, csv, gzip, hmac, html, io, json, os, re, secrets, signal, socket, socketserver, sys, threading, unicodedataimport urllib.parsefrom http.server import BaseHTTPRequestHandler, ThreadingHTTPServer@@ -34,12 +34,14 @@ REPO = "https://github.com/Mahanaicoach/google-maps-scraper-kit"# The web UI's sponsor box: the browser reads this static file from GitHub (cached a day, no# tracking), so the sponsors it lists change without anyone updating their copy of the kit.SPONSORS = "https://raw.githubusercontent.com/Mahanaicoach/google-maps-scraper-kit/master/assets/sponsors/sponsors.json"-# Keyword ideas for the web UI: Google Business Profile category names (4,053, US English) from-# lobstrio/google-business-categories, pinned to one commit. The browser loads it from GitHub, like-# the sponsor box; the kit ships no copy, because that repository has no licence file.-CATEGORIES = ("https://raw.githubusercontent.com/lobstrio/google-business-categories/"- "4ee1c4788d07c812a32986177c2a51c28957062f/data/google-business-categories.txt")-CATEGORIES_REPO = "https://github.com/lobstrio/google-business-categories"+# The web UI's What and Where boxes offer only these, so nobody searches for a business type or a+# place that does not exist: presets/business-types.json (hand-picked; add a niche there) and+# presets/cities.tsv.gz (GeoNames cities1000, CC BY 4.0: name, region, country code, lat, lon, biggest first).+PRESETS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "presets")+with open(os.path.join(PRESETS, "business-types.json"), encoding="utf-8") as _f:+ TYPES = json.load(_f)+TYPES_JS = json.dumps(TYPES).replace("</", "<\\/") # the What list, inlined in the page script+_cities, _cities_lock = None, threading.Lock()ENV_KEYS = ("SCRAPER_PORT", "SCRAPER_API_KEY", "SCRAPER_HEADLESS")MAX_BODY = 1 << 20LOCAL = r"(localhost|127\.0\.0\.1|\[::1\])"@@ -59,6 +61,44 @@ def file_name(job, jid):return (re.sub(r"[^a-z0-9]+", "-", s).strip("-")[:60] or "results") + "-" + jid[:8]+FOLD = str.maketrans({"ł": "l", "ø": "o", "đ": "d", "ı": "i", "ß": "ss", "æ": "ae", "œ": "oe", "þ": "th"})+++def fold(s):+ """"sao paulo sp br" for "São Paulo, SP, BR": accents dropped (also ł, ø, ı...), lower case, punctuation+ as single spaces."""+ s = "".join(c for c in unicodedata.normalize("NFKD", s.lower().translate(FOLD)) if not unicodedata.combining(c))+ return " ".join(re.sub(r"[\W_]+", " ", s).split())+++def cities():+ """Every city as one "folded label<TAB>label<TAB>lat<TAB>lon<TAB>country" line, biggest first. Read on the+ first lookup, then kept (about 35 MB)."""+ global _cities+ with _cities_lock:+ if _cities is None:+ rows = []+ with gzip.open(os.path.join(PRESETS, "cities.tsv.gz"), "rt", encoding="utf-8") as f:+ for line in f:+ name, admin, cc, lat, lon = line.rstrip("\n").split("\t")+ label = ", ".join([name] + ([admin] if admin and admin != name else []) + [cc])+ rows.append("\t".join((fold(label), label, lat, lon, cc)))+ _cities = rows+ return _cities+++def find_cities(q, limit=20):+ """The biggest cities whose "name, region, country" starts with q; with no q, the biggest in the US."""+ q, out = fold(q), []+ for row in cities(): # ponytail: a linear scan, about 30 ms; a prefix index if lookups ever lag+ if row.startswith(q) if q else row.endswith("\tUS"):+ _, label, lat, lon, cc = row.split("\t")+ out.append({"label": label, "lat": lat, "lon": lon, "country": cc})+ if len(out) == limit:+ break+ return out++def log(msg):"""One line on stderr with a UTC timestamp and its offset. Counts only, never scraped values."""with _log_lock:@@ -133,6 +173,9 @@ OPENAPI = {"delete": {"summary": "Delete a job and its results (stops it if it is running)"}},"/api/v1/jobs/{id}/download": {"get": {"summary": "The job's CSV, the rows so far while it works; ""?format=json for a JSON array of the same rows"}},+ "/api/v1/cities": {"get": {"summary": "Up to 20 cities for the web UI's Where box, each {label, lat, lon, "+ "country}: ?q=aus gives the biggest whose \"name, region, country\" "+ "starts with it, no q the biggest US cities (GeoNames, CC BY 4.0)"}},"/api/openapi.json": {"get": {"summary": "This description as OpenAPI JSON (the page at /api/docs shows it)"}},},}@@ -216,18 +259,28 @@ h2 {{ margin: 0 0 12px; font: 600 1.15rem/1.3 var(--serif); }}section {{ padding: 28px 0 0; }}form {{ display: grid; gap: 14px; }}.sr {{ position: absolute; width: 1px; height: 1px; overflow: hidden; clip: rect(0 0 0 0); white-space: nowrap; }}-.search {{ display: flex; align-items: flex-start; gap: 10px; padding: 12px 14px; background: var(--sheet);+.ww {{ display: grid; gap: 14px; grid-template-columns: repeat(auto-fit, minmax(240px, 1fr)); }}+.pick {{ display: grid; gap: 6px; align-content: start; }}+.search {{ display: flex; align-items: center; gap: 10px; padding: 6px 14px; background: var(--sheet);border: 1.5px solid var(--ink); border-radius: 14px; box-shadow: 0 1px 0 var(--rule); }}.search:focus-within {{ outline: 3px solid var(--ring); outline-offset: 2px; }}-.search svg {{ flex: none; margin-top: 3px; color: var(--go); }}-.search textarea {{ flex: 1; min-height: 52px; padding: 0; border: 0; outline: 0; background: transparent; color: var(--ink);- font: 1.05rem/1.55 var(--sans); resize: vertical; }}+.search svg {{ flex: none; color: var(--go); }}+.search input {{ flex: 1; padding: 0; border: 0; outline: 0; background: transparent; font-size: 1.05rem; }}+.box {{ position: relative; }}+.opts {{ position: absolute; z-index: 5; left: 0; right: 0; top: calc(100% + 4px); max-height: 280px; overflow-y: auto;+ margin: 0; padding: 4px; list-style: none; background: var(--sheet); border: 1px solid var(--rule); border-radius: 10px;+ box-shadow: 0 8px 24px rgb(0 0 0 / .14); }}+.opts li {{ padding: 9px 10px; border-radius: 6px; cursor: pointer; }}+.opts li:hover, .opts li[aria-selected=true] {{ background: color-mix(in srgb, var(--go) 12%, var(--sheet)); }}+.tags:empty {{ display: none; }}+.tag {{ min-height: 36px; padding: 6px 12px; border-color: var(--go); border-radius: 999px;+ background: color-mix(in srgb, var(--go) 10%, var(--sheet)); }}.hint {{ margin: -6px 2px 0; }}label {{ display: grid; gap: 6px; font-size: .9rem; font-weight: 600; }}input {{ font: inherit; font-weight: 400; min-height: 44px; width: 100%; padding: 9px 12px; color: var(--ink);background: var(--sheet); border: 1px solid var(--rule); border-radius: 8px; }}input:hover {{ border-color: var(--soft); }}-input[type=number], #coords {{ font-family: var(--mono); font-size: .95rem; }}+input[type=number] {{ font-family: var(--mono); font-size: .95rem; }}textarea {{ font: inherit; }}.collect {{ display: flex; flex-wrap: wrap; gap: 8px; margin: 0; padding: 0; border: 0; }}.collect legend {{ float: left; margin-right: 6px; padding: 12px 0; font-size: .9rem; color: var(--soft); }}@@ -287,27 +340,33 @@ progress {{ display: block; width: 100%; max-width: 200px; height: 6px; margin-t</aside><section aria-labelledby="h-new"><h2 id="h-new">New search</h2><form id="f">- <label class="sr" for="keywords">What to search, one per line</label>- <div class="search"><svg width="22" height="22" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"- stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M12 21s-7-6.2-7-11.5a7 7 0 0 1 14 0C19 14.8 12 21 12 21z"/>- <circle cx="12" cy="9.5" r="2.5"/></svg>- <textarea id="keywords" rows="2" required placeholder="e.g. coffee shops in Austin TX"></textarea></div>- <p class="hint soft">One search per line. Name the city, as you would in Google Maps.</p>- <label class="sr" for="kwpick">Add a business type</label>- <input id="kwpick" list="kwlist" autocomplete="off" placeholder="+ Add a business type, e.g. dentist or roofing contractor">- <datalist id="kwlist"></datalist>+ <div class="ww">+ <div class="pick"><label for="what">What</label><div class="box">+ <div class="search"><svg width="22" height="22" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"+ stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><circle cx="11" cy="11" r="7"/><path d="M20 20l-4-4"/></svg>+ <input id="what" role="combobox" aria-autocomplete="list" aria-expanded="false" aria-controls="what-opts"+ autocomplete="off" placeholder="e.g. Dentists"></div>+ <ul class="opts" id="what-opts" role="listbox" aria-label="Business types" hidden></ul></div>+ <div class="row tags" id="whats"></div></div>+ <div class="pick"><label for="where">Where</label><div class="box">+ <div class="search"><svg width="22" height="22" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"+ stroke-linecap="round" stroke-linejoin="round" aria-hidden="true"><path d="M12 21s-7-6.2-7-11.5a7 7 0 0 1 14 0C19 14.8 12 21 12 21z"/>+ <circle cx="12" cy="9.5" r="2.5"/></svg>+ <input id="where" role="combobox" aria-autocomplete="list" aria-expanded="false" aria-controls="where-opts"+ autocomplete="off" placeholder="e.g. Austin"></div>+ <ul class="opts" id="where-opts" role="listbox" aria-label="Cities" hidden></ul></div>+ <div class="row tags" id="wheres"></div></div>+ </div>+ <p class="hint soft">Pick from the lists; add as many as you like. Each business type is searched in each city, one job per city.</p><fieldset class="collect"><legend>Collect</legend><label class="chip"><input type="checkbox" id="email" checked> Emails</label><label class="chip"><input type="checkbox" id="socials" checked> Social profiles</label></fieldset><details><summary>Location and language</summary><div class="body">- <label class="wide" for="coords"><span>Map centre <span class="soft">optional</span></span>- <input id="coords" inputmode="decimal" autocomplete="off" placeholder="e.g. 30.2672, -97.7431">- <span class="soft">Right-click a spot in Google Maps, click the numbers to copy them, paste here. Left empty,- Google uses the city in your search.</span></label>+ <p class="wide soft">Each job centres the map on its city.</p><label for="zoom">Zoom<input id="zoom" type="number" min="1" max="21" value="15"></label><label for="lang">Language<input id="lang" value="en" maxlength="2" pattern="[A-Za-z]{{2}}"></label>- <label for="country"><span>Country <span class="soft">optional</span></span><input id="country" maxlength="2" pattern="[A-Za-z]{{2}}" placeholder="e.g. us"></label>+ <label for="country"><span>Country <span class="soft">optional</span></span><input id="country" maxlength="2" pattern="[A-Za-z]{{2}}" placeholder="the city's"></label></div></details><details><summary>Advanced options</summary><div class="body"><label for="depth">Depth<input id="depth" type="number" min="1" max="1000" value="5">@@ -395,64 +454,87 @@ async function load() {{if (!list.length) {{ const tr = document.createElement('tr'); cell(tr, 'No jobs yet. Start a search above.', 'soft').colSpan = 4; tb.appendChild(tr); }}}} catch (e) {{ say(e.message, 'bad'); }}}}+const TYPES = {types};+const whats = [], wheres = []; // picked business types, picked cities {{label, lat, lon, country}}+function tags(box, list) {{ // each pick is a button; pressing it removes the pick+ box.replaceChildren();+ list.forEach((v, i) => {{+ const b = document.createElement('button'), text = v.label || v; b.type = 'button'; b.className = 'tag';+ b.textContent = text + ' ×'; b.setAttribute('aria-label', 'Remove ' + text);+ b.addEventListener('click', () => {{ list.splice(i, 1); tags(box, list); }}); box.appendChild(b);+ }});+}}+function combo(input, box, source, choose) {{ // a box that takes only picks from its own list: arrows move, Enter or a click picks+ let items = [], active = -1;+ function show(list) {{+ items = list; active = list.length ? 0 : -1; box.replaceChildren();+ list.forEach((it, i) => {{+ const li = document.createElement('li'); li.id = box.id + '-' + i; li.setAttribute('role', 'option'); li.textContent = it.label;+ li.addEventListener('mousedown', ev => {{ ev.preventDefault(); pick(i); }}); box.appendChild(li);+ }});+ mark(); box.hidden = !list.length; input.setAttribute('aria-expanded', String(!box.hidden));+ }}+ function mark() {{+ [...box.children].forEach((li, i) => li.setAttribute('aria-selected', String(i === active)));+ if (active < 0) return input.removeAttribute('aria-activedescendant');+ input.setAttribute('aria-activedescendant', box.id + '-' + active); box.children[active].scrollIntoView({{block: 'nearest'}});+ }}+ function pick(i) {{ choose(items[i]); input.value = ''; say('', ''); source('', show); }}+ input.addEventListener('input', () => {{ if ($('msg').className === 'bad') say('', ''); source(input.value.trim(), show); }});+ input.addEventListener('focus', () => source(input.value.trim(), show));+ input.addEventListener('blur', () => show([]));+ input.addEventListener('keydown', ev => {{+ if (ev.key === 'ArrowDown' || ev.key === 'ArrowUp') {{+ ev.preventDefault(); if (!items.length) return;+ active = (active + (ev.key === 'ArrowDown' ? 1 : -1) + items.length) % items.length; mark();+ }} else if (ev.key === 'Enter') {{ // never submits the form+ ev.preventDefault();+ if (active >= 0) pick(active);+ else if (input.value.trim()) say('"' + input.value.trim() + '" is not in the list. Type a few letters, then pick from it.', 'bad');+ }} else if (ev.key === 'Escape') show([]);+ }});+}}+combo($('what'), $('what-opts'), (q, show) => show(TYPES.filter(t => t.toLowerCase().includes(q.toLowerCase())).map(t => ({{label: t}}))),+ t => {{ if (!whats.includes(t.label)) whats.push(t.label); tags($('whats'), whats); }});+let timer = 0;+combo($('where'), $('where-opts'), (q, show) => {{ // the biggest cities that start with q; nothing typed, the biggest US ones+ clearTimeout(timer);+ timer = setTimeout(async () => {{+ try {{+ const r = await api('GET', '/api/v1/cities?q=' + encodeURIComponent(q)); if (!r.ok) return;+ const found = await r.json();+ if (q === $('where').value.trim() && document.activeElement === $('where')) show(found); // else a newer lookup is coming+ }} catch (e) {{ say(e.message, 'bad'); }}+ }}, 120);+}}, c => {{ if (!wheres.some(w => w.label === c.label)) wheres.push(c); tags($('wheres'), wheres); }});$('f').addEventListener('submit', async ev => {{ev.preventDefault();- const keywords = $('keywords').value.split('\\n').map(s => s.trim()).filter(Boolean);- const at = $('coords').value.trim(), m = at.match(/^(-?\\d+(?:\\.\\d+)?)\\s*[,\\s]\\s*(-?\\d+(?:\\.\\d+)?)$/);- if (at && !m) {{ $('coords').closest('details').open = true; $('coords').focus();- return say('Map centre: paste two numbers such as 30.2672, -97.7431, or leave it empty.', 'bad'); }}+ for (const box of [$('what'), $('where')]) if (box.value.trim()) {{ box.focus();+ return say('"' + box.value.trim() + '" is not picked yet. Pick it from the list, or clear the box.', 'bad'); }}+ if (!whats.length || !wheres.length) {{ (whats.length ? $('where') : $('what')).focus();+ return say('Pick at least one business type and one city from the lists.', 'bad'); }}const proxies = $('proxies').value.split('\\n').map(s => s.trim()).filter(Boolean);- const body = {{name: Array.from(keywords[0] || 'job').slice(0, 200).join(''), keywords, lang: $('lang').value.trim().toLowerCase(),- country: $('country').value.trim().toLowerCase(), lat: m ? m[1] : '', lon: m ? m[2] : '',- zoom: +$('zoom').value || 15, depth: +$('depth').value, max_time: +$('max_time').value,- email: $('email').checked, socials: $('socials').checked, verify_emails: $('verify').checked,- reviews_max: +$('reviews').value || 0}};- if (proxies.length) body.proxies = proxies;- const go = $('go'), bare = m ? [] : keywords.filter(k => !/\\b(in|near)\\s+\\S|,/i.test(k));- if (bare.length && go.dataset.anyway !== '1') {{ // Google would search around this computer's location- go.dataset.anyway = '1'; go.textContent = 'Start anyway';- return say('No city in: ' + bare.slice(0, 3).map(k => '"' + k + '"').join(', ') + (bare.length > 3 ? '…' : '')- + '. Google would search near this computer. Add a city (dentist in Denver CO), set a map centre, or press Start anyway.', 'bad');- }}- delete go.dataset.anyway; go.disabled = true; go.textContent = 'Starting…';+ const go = $('go'); go.disabled = true; go.textContent = 'Starting…';+ let started = 0;try {{- const r = await api('POST', '/api/v1/jobs', body), d = await r.json();- if (r.ok) say('Started. It runs in the background, and the list below updates by itself.', 'ok');- else say('Not started: ' + d.message, 'bad');- load();+ for (const c of wheres) {{ // one job per city, its map centred on that city+ const keywords = whats.map(w => w + ' in ' + c.label);+ const body = {{name: Array.from(keywords[0] + (keywords.length > 1 ? ' +' + (keywords.length - 1) + ' more' : '')).slice(0, 200).join(''),+ keywords, lang: $('lang').value.trim().toLowerCase(), country: $('country').value.trim().toLowerCase() || c.country.toLowerCase(),+ lat: c.lat, lon: c.lon, zoom: +$('zoom').value || 15, depth: +$('depth').value, max_time: +$('max_time').value,+ email: $('email').checked, socials: $('socials').checked, verify_emails: $('verify').checked,+ reviews_max: +$('reviews').value || 0}};+ if (proxies.length) body.proxies = proxies;+ const r = await api('POST', '/api/v1/jobs', body), d = await r.json();+ if (!r.ok) return say('Not started' + (started ? ' after ' + started + ' job(s)' : '') + ': ' + d.message, 'bad');+ started++;+ }}+ say(started === 1 ? 'Started. It runs in the background, and the list below updates by itself.'+ : 'Started ' + started + ' jobs, one per city. They run one after another, and the list below updates by itself.', 'ok');}} catch (e) {{ say(e.message, 'bad'); }}- finally {{ go.disabled = false; go.textContent = 'Start scraping'; }}+ finally {{ go.disabled = false; go.textContent = 'Start scraping'; load(); }}}});load(); setInterval(load, 3000);-function pick() {{ // a picked business type becomes a keyword line that waits for its city: "Bakery in "- const v = $('kwpick').value.trim(); if (!v) return;- const k = $('keywords'); k.value = (k.value.trim() ? k.value.trim() + '\\n' : '') + v + ' in ';- $('kwpick').value = ''; k.focus(); k.setSelectionRange(k.value.length, k.value.length); unwarn();- say('Type the city after "' + v + ' in".', '');-}}-function unwarn() {{ const go = $('go'); if (go.dataset.anyway) {{ delete go.dataset.anyway; go.textContent = 'Start scraping'; }} }}-$('kwpick').addEventListener('change', pick);-$('kwpick').addEventListener('keydown', ev => {{ if (ev.key === 'Enter') {{ ev.preventDefault(); pick(); }} }});-$('keywords').addEventListener('input', unwarn); $('coords').addEventListener('input', unwarn);-(function () {{ // keyword ideas: Google's business category names, cached for 30 days, as text only- const key = 'gmskit-categories', month = 30 * 86400000;- function fill(names) {{- const list = $('kwlist'); list.replaceChildren();- for (const n of names) {{ const o = document.createElement('option'); o.value = n; list.appendChild(o); }}- }}- function clean(text) {{- return text.split('\\n').map(s => s.trim()).filter(s => s && s.length <= 80).slice(0, 5000);- }}- try {{- const c = JSON.parse(localStorage.getItem(key) || 'null');- if (c && Date.now() - c.t < month && Array.isArray(c.names)) return fill(c.names);- }} catch (e) {{}}- fetch('{categories}', {{credentials: 'omit'}}).then(r => r.ok ? r.text() : '').then(t => {{- const names = clean(t); if (!names.length) return;- fill(names);- try {{ localStorage.setItem(key, JSON.stringify({{t: Date.now(), names}})); }} catch (e) {{}}- }}).catch(() => {{}});-}})();(function () {{ // sponsor tiles: only the entries Mahan marks "ui": true in sponsors.json, as text, https links onlyconst box = $('sponsors'), grid = $('sp-grid'), key = 'gmskit-sponsors', day = 86400000;$('sp-close').addEventListener('click', () => {{ box.hidden = true; }}); // as gosom: hidden until the page reloads@@ -602,6 +684,8 @@ class Handler(BaseHTTPRequestHandler):csp = f"default-src 'none'; style-src 'nonce-{nonce}'; base-uri 'none'; form-action 'none'; frame-ancestors 'none'"return self.reply(200, docs_page(nonce).encode(), "text/html; charset=utf-8",[("Content-Security-Policy", csp)])+ if path == "/api/v1/cities":+ return self.reply(200, find_cities(urllib.parse.parse_qs(query).get("q", [""])[0][:100]))if path == "/api/openapi.json":return self.reply(200, OPENAPI)if path == "/":@@ -610,8 +694,7 @@ class Handler(BaseHTTPRequestHandler):"connect-src 'self' https://raw.githubusercontent.com; img-src 'self' data:; ""form-action 'self'; base-uri 'none'; ""frame-ancestors 'none'")- page = PAGE.format(nonce=nonce, version=VERSION, repo=REPO, sponsors=SPONSORS, categories=CATEGORIES,- categories_repo=CATEGORIES_REPO).encode()+ page = PAGE.format(nonce=nonce, version=VERSION, repo=REPO, sponsors=SPONSORS, types=TYPES_JS).encode()return self.reply(200, page, "text/html; charset=utf-8", [("Content-Security-Policy", csp)])self.fail(404, "not found")diff --git a/engine/test_engine.py b/engine/test_engine.pyindex aff2dd9..dea7ce1 100644--- a/engine/test_engine.py+++ b/engine/test_engine.py@@ -8,7 +8,7 @@ No network, no Chrome, no Docker."""import csv, hashlib, http.client, importlib.util, io, json, os, queue, re, shutil, socket, sqlite3, subprocess, sysimport tempfile-import threading, time, unittest, unittest.mock, urllib.request, uuid, zipfile+import threading, time, unittest, unittest.mock, urllib.parse, urllib.request, uuid, zipfileHERE = os.path.dirname(os.path.abspath(__file__))sys.path.insert(0, HERE)@@ -1237,26 +1237,30 @@ class ApiP3(Server):["schema"]["required"], ["name", "keywords", "lang", "max_time"])ui = self.call("GET", "/")[1].decode()self.assertIn('id="country"', ui)- self.assertIn('list="kwlist"', ui)+ self.assertIn('<input id="what" role="combobox"', ui)+ self.assertIn('<input id="where" role="combobox"', ui)+ self.assertIn("const TYPES = " + json.dumps(serve.TYPES) + ";", ui) # the What list is in the page, no downloadfront = ui.split("<details>")[0] # what a beginner sees before opening a sectionself.assertIn('<input type="checkbox" id="email" checked>', front)self.assertIn('<input type="checkbox" id="socials" checked>', front) # socials on by default- for el in ('id="coords"', 'id="depth"', 'id="lang"', 'id="max_time"', 'id="proxies"', 'id="reviews"'):+ for el in ('id="depth"', 'id="lang"', 'id="max_time"', 'id="proxies"', 'id="reviews"'):self.assertNotIn(el, front)self.assertIn(el, ui)self.assertEqual(ui.count("<details>"), 3) # location and language, advanced options, proxiesself.assertNotIn("lobstr.io</a>", ui)self.assertIn("socials: $('socials').checked", ui)self.assertIn("if (proxies.length) body.proxies = proxies", ui)- # newcomer test: Enter in the type box never submits; a picked type waits for its city; a search- # with no place warns once; downloads are named after the search; local times; a favicon- self.assertIn("if (ev.key === 'Enter') {{ ev.preventDefault(); pick(); }}".replace("{{", "{").replace("}}", "}"), ui)- self.assertIn("v + ' in '", ui)- self.assertIn("Start anyway", ui)+ # preset only: Enter in a box never submits; text not in a list is refused; each type is searched in+ # each city, one job per city centred on it; downloads are named after the search; local times; a favicon+ for rule in ("} else if (ev.key === 'Enter') { // never submits the form", "if (active >= 0) pick(active);",+ "is not in the list", "is not picked yet", "whats.map(w => w + ' in ' + c.label)",+ "lat: c.lat, lon: c.lon", "for (const c of wheres)", "'/api/v1/cities?q=' + encodeURIComponent(q)"):+ self.assertIn(rule, ui)+ for gone in ('id="keywords"', 'id="coords"', 'id="kwpick"', "<datalist", "Start anyway", "raw.githubusercontent.com/lobstrio"):+ self.assertNotIn(gone, ui)self.assertIn("a.download = fileName(j) + '.' + fmt", ui)self.assertIn("toLocaleString", ui)self.assertIn('<link rel="icon" href="data:image/svg+xml,', ui)- self.assertIn('placeholder="e.g. coffee shops in Austin TX"', ui)# as gosom: a star button, and a sponsor box that can be closed; paid sponsors in it only when# sponsors.json lists one Mahan marked "ui": true, https links only, four at mostself.assertIn('Star on GitHub</a>', ui)@@ -1266,8 +1270,6 @@ class ApiP3(Server):for rule in ("s.ui === true", "s.url.startsWith('https://')", ".slice(0, 4)", "noopener noreferrer sponsored"):self.assertIn(rule, ui)self.assertNotIn(".innerHTML", ui) # sponsor text is set as text, never parsed as HTML- self.assertIn(serve.CATEGORIES, ui) # loaded by the browser from GitHub, pinned to a commit- self.assertRegex(serve.CATEGORIES, r"/[0-9a-f]{40}/")script = re.search(r"<script nonce=[^>]+>(.*)</script>", ui, re.S).group(1)self.assertNotIn("'\n'", script) # PAGE must say \\n: a bare newline in a JS string breaks the page@@ -1279,6 +1281,40 @@ class ApiKeyDocs(Server):self.assertEqual(self.call("GET", "/api/docs")[0], 200)self.assertEqual(self.call("GET", "/api/openapi.json")[0], 200)self.assertEqual(self.call("GET", "/api/v1/jobs/00000000-0000-0000-0000-000000000000/download")[0], 401)+ self.assertEqual(self.call("GET", "/api/v1/cities?q=austin")[0], 401)+++class Presets(Server):+ def cities(self, q):+ code, body, _ = self.call("GET", "/api/v1/cities?q=" + urllib.parse.quote(q))+ self.assertEqual(code, 200)+ return json.loads(body)++ def test_where(self):+ top = self.cities("") # nothing typed: the biggest US cities+ self.assertEqual((len(top), top[0]["label"]), (20, "New York City, New York, US"))+ self.assertTrue(all(c["country"] == "US" for c in top))+ self.assertEqual(self.cities("austin")[0], {"label": "Austin, Texas, US", "lat": "30.2672", "lon": "-97.7431",+ "country": "US"})+ self.assertEqual([c["label"] for c in self.cities("Austin, Tex")], ["Austin, Texas, US"])+ self.assertEqual(self.cities("sao paulo")[0]["label"], "São Paulo, BR") # accents fold+ self.assertEqual(self.cities("lodz")[0]["label"], "Łódź, Łódź Voivodeship, PL")+ self.assertEqual(self.cities("shanghai")[0]["label"], "Shanghai, CN") # region == name: said once+ self.assertEqual(self.cities("no such place xq"), [])+ self.assertLessEqual(len(self.cities("s")), 20)++ def test_data(self):+ self.assertEqual(len({t.lower() for t in serve.TYPES}), len(serve.TYPES)) # no duplicate business types+ self.assertTrue(all(t == t.strip() and 0 < len(t) <= 80 for t in serve.TYPES))+ labels = set()+ for row in serve.cities():+ key, label, lat, lon, cc = row.split("\t")+ self.assertTrue(-90 <= float(lat) <= 90 and -180 <= float(lon) <= 180, row)+ self.assertRegex(cc, r"^[A-Z]{2}$")+ self.assertEqual(key, serve.fold(label))+ labels.add(label)+ self.assertEqual(len(labels), len(serve.cities())) # each label names one city+ self.assertGreater(len(labels), 150000)