Skip to content

Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.

feat(engine): business photos in images; optional place details; P4 audit loop 2 fixes

Mahan Jafari8fcce979aebf
diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.md
index e5409b0..b3927fe 100644
--- a/.claude/skills/google-maps-scraper/SKILL.md
+++ b/.claude/skills/google-maps-scraper/SKILL.md
@@ -123,7 +123,9 @@ columns are exactly these.
`review_count`, `open_hours` (the week) and `reviews_link` are filled when Google sends them: the
engine's headless Chrome sends its own user agent with `HeadlessChrome` replaced by `Chrome`, which
gets Google's full view. Google can still leave a field out. Review texts come only through the raw
-API's `reviews_max`. Images stay empty.
+API's `reviews_max`. `images` holds business photo URLs from the search results; `scripts/scrape.py
+--details` (API `extra_details`) adds the place panel's photos and `popular_times`, one page load
+per place, only when the user asks for them.
### Return only the 8 lead fields (default)
The raw CSV has 36 columns and most are noise for outreach. **By default, return only the 8 lead
diff --git a/README.md b/README.md
index e854c20..2da0e88 100644
--- a/README.md
+++ b/README.md
@@ -111,6 +111,9 @@ Illustrative rows:
`review_count` works in headless mode; see [Honest limits](#honest-limits).
+`--details` reads each place panel for additional business photo URLs and published hourly popular
+times. It implies `--full` and is slower. Missing schedules remain blank; `{}` is not counted as data.
+
`--socials` adds `instagram`, `facebook`, `linkedin`, `tiktok`, `youtube` and `x` columns. `--tech` adds
`fb_pixel`, `google_ads_tag`, `analytics`, `bing_ads`, `linkedin_insight`, `tiktok_pixel`, `cms`,
`mobile_friendly` and `intent_score`, and sorts the hottest leads first. `--verify-emails` adds
@@ -207,8 +210,9 @@ One keyword per line. One job. One CSV.
OS, same version, nothing else changed), and Google returns `review_count`, the week's
`open_hours` and `reviews_link`. Google can still leave a field out; if every count is missing,
`--min-reviews` warns and skips that filter. Review texts come through the raw API's
- `reviews_max` (up to 5 per place, one page load each). Photos stay empty. Rather run a real
- window, with the agent unchanged? [SETUP §3](SETUP.md#review-counts-hours-and-reviews).
+ `reviews_max` (up to 5 per place, one page load each). Business photo URLs come in `images`
+ (`--full`) with no extra page loads; `--details` adds the place panel's photos and published
+ popular times, one page load per place. Rather run a real window, with the agent unchanged? [SETUP §3](SETUP.md#review-counts-hours-and-reviews).
- **Emails only when a business publishes one.** The script reads the business homepage, then its
contact and about pages. Listings with no site, or a site with no visible
email, return an empty `emails` cell. Instagram bios are not read. Expect emails on a fraction of
diff --git a/SETUP.md b/SETUP.md
index efb472e..bcf80f5 100644
--- a/SETUP.md
+++ b/SETUP.md
@@ -175,6 +175,23 @@ each place's panel, one page load per place. It fills `user_reviews` (a JSON lis
`{"Rating", "When", "Description"}`; reviewer names and photos are left out) and
`reviews_per_rating` (`{"1": n, ... "5": n}`). `scripts/scrape.py` has no flag for reviews.
+Business photo URLs from search results are included in the `images` column without additional
+page loads. Use `--full` to keep that column in CLI output. Photos are linked, not downloaded.
+
+For additional place-panel data, use:
+
+```bash
+python3 scripts/scrape.py "coffee shops in Austin TX" 30.2672 -97.7431 --details --no-email
+```
+
+`--details` implies `--full` and requests `extra_details: true` from the API. The web UI has a
+**Place details** checkbox. The engine reads each place panel once for available photo URLs and
+published popular times; jobs requesting reviews share this pass. This adds one page load per place.
+`popular_times` is JSON such as `{"Monday": {"8": 20, "9": 50}}`: local hour and relative busyness
+from 0 to 100, not visitor counts. Google may provide no schedule. Missing data stays blank, and the
+job note reports how many schedules were unavailable; an empty object does not count as a schedule.
+No reviewer identities or reviewer photos are added.
+
Rather not change the user agent? Set `SCRAPER_HEADLESS=0` in `.env` and restart with
`python3 scripts/engine.py restart`: Chrome then runs as a normal window placed off-screen, with its
user agent unchanged, and gets the same data. A window needs a display (not Docker, not a server
@@ -302,7 +319,8 @@ Nothing is lost:
- The jobs in 1.x's web UI come along. On its first start the new engine reads 1.x's job volume
(read-only) and lists those jobs with their CSVs; a job 1.x had not finished shows as `failed`,
with its rows, and a job that never produced a CSV is left out. This happens once, so a job you
- delete afterwards stays deleted.
+ delete afterwards stays deleted; a job whose data cannot be read is skipped, and the engine's log
+ names its id (`docker compose logs`).
- The old volume itself is kept as it was. Once you no longer need it, free its disk space:
`docker compose down`, then `docker volume rm` with the name `docker volume ls` shows ending in
`_gmaps_data`, then `docker compose up -d` (Compose creates an empty one in its place).
diff --git a/docs/google-maps-scraper-python.md b/docs/google-maps-scraper-python.md
index 8b7af06..3b9762b 100644
--- a/docs/google-maps-scraper-python.md
+++ b/docs/google-maps-scraper-python.md
@@ -30,6 +30,7 @@ On Windows, run the same commands with `py` or `python3`. The engine needs Pytho
| `--max-time` | Job time limit in seconds. Default `600`. |
| `--out` | Output path. Default `results-<short job id>.csv`. A `.json` extension writes JSON. |
| `--json` | Writes JSON instead of CSV. |
+| `--details` | Reads each place panel for additional business photo URLs and published popular times; implies `--full`, adds one page load per place. Missing schedules stay blank. |
| `--full` | Keeps all raw columns (the 36 classic ones plus `categories`) instead of the 8 lead fields. Review counts and weekly opening hours work headless ([SETUP §3](../SETUP.md#review-counts-hours-and-reviews)). Review texts stay empty: the script does not ask for them. |
| `--fields` | Comma-separated columns to keep, overriding the lead fields. |
| `--socials` | Adds Instagram, Facebook, LinkedIn, TikTok, YouTube and X from each website. Slower. Zero AI tokens. |
diff --git a/engine/details.py b/engine/details.py
index 0d2bc1e..6374120 100644
--- a/engine/details.py
+++ b/engine/details.py
@@ -4,11 +4,11 @@ While the result list scrolls, the page fetches batches of about 20 places from
`https://www.google.com/search?tbm=map…` as `)]}'\\n[...]` JSON; each batch carries the full place
entity (name, address, phone, website, rating, coordinates, ids…). The engine captures those
responses in the browser (cdp Network events) and parses them here, so no place page is opened for
-them: the spike showed the place page document itself holds no entity data any more. Review texts
-are the one thing the batches lack: they come from the `/maps/preview/place` response a place's
-panel downloads (parse_place), only for jobs that ask for them. Standard library only.
+them. Requested reviews and extra details come from the `/maps/preview/place` response a place's
+panel downloads (parse_place). Business photo URLs in search batches need no panel load.
+Standard library only.
"""
-import json
+import json, urllib.parse
# Index path of each field inside a place entity. Checked on live batches in the spike
# (2026-10-31): title, category, address, rating, lat/lon, place_id, data_id, timezone, thumbnail
@@ -38,6 +38,7 @@ FIELDS = {
"description": (32, 1, 1),
}
HOURS = (203, 0) # [[day name, 1-7 (Monday-Sunday), date, [[text, ...], ...], ...], ...], from today on
+POPULAR = (84, 0) # [[weekday 1-7, [[hour 0-23, busyness 0-100, ...], ...]], ...]
HISTOGRAM = (175, 3) # place panel: [one-star count, ..., five-star count]
REVIEWS = (175, 9, 0, 0) # place panel: the reviews it shows first (about 5)
LISTS = {"categories": (13,), "address_lines": (2,)}
@@ -68,6 +69,12 @@ def parse_entity(entity):
hours = open_hours(dig(entity, HOURS))
if hours:
out["open_hours"] = hours
+ photos = images(entity)
+ if photos:
+ out["images"] = photos
+ traffic = popular_times(dig(entity, POPULAR))
+ if traffic:
+ out["popular_times"] = traffic
if "data_id" in out and ":" in out["data_id"]:
try:
out["cid"] = str(int(out["data_id"].split(":")[1], 16))
@@ -96,9 +103,55 @@ def open_hours(days):
return json.dumps({name: spans for _, (name, spans) in sorted(week.items())}, ensure_ascii=False)
+def images(entity):
+ """Photo URLs already supplied by Maps, as the existing images column's JSON objects."""
+ found, seen = [], set()
+ # Search cards and panels use different gallery layouts. Never include reviewer profiles.
+ groups = [(dig(entity, (171, 0)), (3, 0, 6, 0), (2,)),
+ (dig(entity, (37, 0)), (6, 0), ()),
+ (dig(entity, (72, 0)), (6, 0), ())]
+ for items, path, title_path in groups:
+ if not isinstance(items, list):
+ continue
+ for item in items:
+ url = dig(item, path)
+ if not isinstance(url, str) or url in seen:
+ continue
+ try:
+ parsed = urllib.parse.urlsplit(url)
+ if parsed.scheme != "https" or not parsed.hostname:
+ continue
+ except ValueError:
+ continue
+ seen.add(url)
+ title = _text(dig(item, title_path)) if title_path else ""
+ found.append({"title": title, "image": url})
+ return json.dumps(found, ensure_ascii=False) if found else ""
+
+
+def popular_times(days):
+ """Published hourly busyness by weekday; missing data stays absent, never an empty object."""
+ if not isinstance(days, list):
+ return ""
+ names = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")
+ week = {}
+ for day in days:
+ number, slots = dig(day, (0,)), dig(day, (1,))
+ if type(number) is not int or not 1 <= number <= 7 or not isinstance(slots, list):
+ continue
+ hours = {}
+ for slot in slots:
+ hour, busy = dig(slot, (0,)), dig(slot, (1,))
+ if type(hour) is int and 0 <= hour <= 23 and type(busy) is int and 0 <= busy <= 100:
+ hours[str(hour)] = busy
+ if hours:
+ week[number] = hours
+ return json.dumps({names[d - 1]: week[d] for d in sorted(week)}) if week else ""
+
+
def parse_place(body, limit=5):
"""review_count, reviews_per_rating and up to `limit` user_reviews from one `/maps/preview/place`
- response. Reviewer names and photos are left out: the kit collects business data, not people."""
+ response, plus available business photos and popular times. Reviewer identities are omitted."""
if body.startswith(")]}'"):
body = body.split("\n", 1)[-1]
try:
@@ -107,12 +160,12 @@ def parse_place(body, limit=5):
return {}
if not isinstance(entity, list):
return {}
- out = {k: v for k, v in parse_entity(entity).items() if k in ("review_count", "reviews_link", "open_hours")}
+ out = {k: v for k, v in parse_entity(entity).items() if k in ("review_count", "reviews_link", "open_hours", "images", "popular_times")}
hist = dig(entity, HISTOGRAM)
if isinstance(hist, list) and len(hist) == 5 and all(isinstance(n, int) for n in hist):
out["reviews_per_rating"] = json.dumps({str(i): n for i, n in enumerate(hist, 1)})
reviews = []
- for r in dig(entity, REVIEWS) or []:
+ for r in (dig(entity, REVIEWS) or []) if limit > 0 else []:
rating, text = dig(r, (0, 2, 0, 0)), _text(dig(r, (0, 2, 15, 0, 0)))
if isinstance(rating, int) and not isinstance(rating, bool):
reviews.append({"Rating": rating, "When": _text(dig(r, (0, 1, 6))), "Description": text})
diff --git a/engine/jobs.py b/engine/jobs.py
index 799604c..80a5723 100644
--- a/engine/jobs.py
+++ b/engine/jobs.py
@@ -105,6 +105,8 @@ def validate(body):
raise ValueError("invalid geo coordinates")
if body.get("max_time") in (None, 0):
raise ValueError("missing max time")
+ if not isinstance(body.get("extra_details", False), bool):
+ raise ValueError("invalid extra_details: true or false")
data = {"keywords": [k.strip() for k in kws], "lang": lang.lower(), "country": country.lower(),
"lat": str(lat).strip(), "lon": str(lon).strip(),
"zoom": _int(body, "zoom", 15, 1, 21, "zoom"),
@@ -114,6 +116,7 @@ def validate(body):
"fast_mode": bool(body.get("fast_mode")), "email": bool(body.get("email")),
"verify_emails": bool(body.get("verify_emails")), "socials": bool(body.get("socials")),
"reviews_max": _int(body, "reviews_max", 0, 0, REVIEWS_MAX, "reviews_max"),
+ "extra_details": body.get("extra_details", False),
"proxies": _proxies(body.get("proxies"))}
return name.strip(), data
@@ -322,6 +325,8 @@ class Store:
d = {}
if not isinstance(d, dict):
d = {}
+ if not isinstance(d.get("keywords"), list):
+ d["keywords"] = []
if isinstance(d.get("max_time"), int):
d["max_time"] //= 10 ** 9 # nanoseconds in 1.x, seconds here
with open(src, newline="", encoding="utf-8", errors="replace") as f:
@@ -497,7 +502,7 @@ class Worker(threading.Thread):
self._running = (None, None)
return
done, stage = min(p.get("done", 0), len(kws)), p.get("stage", "search")
- reviews = d.get("reviews_max") and stage in ("search", "reviews")
+ reviews = (d.get("reviews_max") or d.get("extra_details")) and stage in ("search", "reviews")
if stage == "search":
self.store.cut(jid, p.get("bytes", 0)) # the rows of a keyword cut short are searched again
rows = self.store.read_rows(jid) # [] when the cut removed the file
@@ -612,11 +617,13 @@ class Worker(threading.Thread):
break
if reason == "no-feed" and not again and time.monotonic() + 30 < deadline:
again = True # Google sometimes shows no result list once (seen in CI): look once more
+ if len(proxies) > 1:
+ rot += 1 # through the next proxy
self.log(f"job {jid} keyword {i + 1}/{len(kws)}: no result list; one more try")
continue
if reason == "no-feed":
- notes.append(f"keyword {i + 1}: Google showed no result list twice (no results for it, "
- "or a page the engine cannot read)")
+ notes.append(f"keyword {i + 1}: Google showed no result list{' twice' if again else ''} "
+ "(no results for it, or a page the engine cannot read)")
retries, again = 0, False
if reason.startswith("timeout"): # "timeout+dom-fallback" too
timed_out["yes"] = True
@@ -646,7 +653,7 @@ class Worker(threading.Thread):
n = self.add_reviews(rows, d, deadline, stop, [x for x in proxies if x not in bad] or proxies,
save, notes)
self.store.rewrite(jid, rows, cols) # keep the reviews if the engine stops during the next pass
- self.log(f"job {jid}: reviews for {n}/{sum(1 for r in rows if r.get('place_id'))} places")
+ self.log(f"job {jid}: place details for {n}/{sum(1 for r in rows if r.get('place_id'))} places")
if stage == "reviews" and not stop.is_set():
stage = "websites"
save(stage=stage)
@@ -721,9 +728,8 @@ class Worker(threading.Thread):
self._browser.dispose(ctx)
def add_reviews(self, rows, d, deadline, stop, proxies, save=lambda: None, notes=None):
- """Review count, stars histogram and up to reviews_max reviews per place, from each place's
- panel (the result batches carry no review texts): one page load per place, until the
- deadline. The number of places that got reviews."""
+ """Read requested place panels once for reviews and extra details, within the job deadline.
+ The number of panels that supplied parsed data; missing fields stay blank."""
todo = [r for r in rows if r.get("place_id") and not r.get("user_reviews")]
ctx = page = auth = None
n = 0
@@ -751,9 +757,9 @@ class Worker(threading.Thread):
break
got = details.parse_place(body or "", d["reviews_max"])
for k, v in got.items():
- if k in ("user_reviews", "reviews_per_rating") or not r.get(k):
+ if k in ("user_reviews", "reviews_per_rating", "images") or not r.get(k):
r[k] = str(v)
- n += "user_reviews" in got
+ n += bool(got)
save() # elapsed time, for a resume after a crash
except (CDPError, OSError) as e: # Chrome or the proxy went away: the reviews so far are kept
if notes is not None:
@@ -765,6 +771,10 @@ class Worker(threading.Thread):
page.close()
if ctx and not self._browser.cdp.closed:
self._browser.dispose(ctx)
+ if d.get("extra_details") and notes is not None:
+ missing = sum(not r.get("popular_times") for r in rows)
+ if missing:
+ notes.append(f"popular times unavailable for {missing}/{len(rows)} places")
return n
diff --git a/engine/serve.py b/engine/serve.py
index 9b752bf..fdc6685 100644
--- a/engine/serve.py
+++ b/engine/serve.py
@@ -87,6 +87,7 @@ JOB_PROPS = {
"drop domains that do not exist, add email_host"},
"socials": {"type": "boolean", "description": "visit each website's homepage for Instagram, Facebook, LinkedIn, "
"TikTok, YouTube and X links"},
+ "extra_details": {"type": "boolean", "description": "read each place panel for business photos and published popular times; one page load per place, shared with reviews"},
"reviews_max": {"type": "integer", "description": "0-5 reviews per place from its panel, one page load per place (headless supported)"},
"proxies": {"type": "array", "items": {"type": "string"},
"description": "http(s)://[user:pass@]host:port or socks5://host:port; one per keyword in turn, "
@@ -183,6 +184,7 @@ A job the engine was stopped in the middle of carries on when it starts again.</
<label>Country (optional)<input id="country" maxlength="2" pattern="[A-Za-z]{{2}}" placeholder="us"></label>
<label>Time limit (seconds)<input id="max_time" type="number" min="1" max="86400" value="600"></label>
<label><span>Emails</span><span><input type="checkbox" id="email" checked> visit each website</span></label>
+ <label><span>Place details</span><span><input type="checkbox" id="details"> photos and published popular times (slower)</span></label>
<div class="wide row"><button class="primary">Start scraping</button>
<span class="muted">Coordinates: right-click a spot in Google Maps and click the numbers.</span></div>
<label class="wide" id="keybox" hidden>API key (SCRAPER_API_KEY)<input id="key" type="password" autocomplete="off"></label>
@@ -216,7 +218,7 @@ function progress(j) {{
if (j.Status === 'pending') return p.keywords_done ? 'pending (resumes at keyword ' + (p.keywords_done + 1) + ')' : 'pending';
if (j.Status !== 'working') return j.Status + (p.places ? ' · ' + p.places + ' places' : '');
const stage = {{search: 'keyword ' + Math.min(p.keywords_done + 1, p.keywords_total) + '/' + p.keywords_total,
- reviews: 'reading reviews', websites: 'checking websites'}}[p.stage] || p.stage || '';
+ reviews: 'reading place details', websites: 'checking websites'}}[p.stage] || p.stage || '';
return 'working · ' + stage + ' · ' + (p.places || 0) + ' places · ' + (p.percent || 0) + '%';
}}
async function remove(j) {{
@@ -245,7 +247,7 @@ $('f').addEventListener('submit', async ev => {{
const body = {{name: keywords[0] || 'job', keywords, lang: $('lang').value.trim().toLowerCase(),
country: $('country').value.trim().toLowerCase(),
lat: $('lat').value.trim(), lon: $('lon').value.trim(), depth: +$('depth').value,
- max_time: +$('max_time').value, email: $('email').checked, zoom: 15}};
+ max_time: +$('max_time').value, email: $('email').checked, extra_details: $('details').checked, zoom: 15}};
try {{
const r = await api('POST', '/api/v1/jobs', body), d = await r.json();
say(r.ok ? 'Job created. It runs in the background; this list refreshes by itself.' : 'Not started: ' + d.message);
diff --git a/engine/test_engine.py b/engine/test_engine.py
index 52203c5..40313c5 100644
--- a/engine/test_engine.py
+++ b/engine/test_engine.py
@@ -974,6 +974,44 @@ def entity(days=7, reviews=False):
return e
+class ExtraDetailsParser(unittest.TestCase):
+ def test_photos_from_search_and_panel_are_valid_json_and_deduplicated(self):
+ e = [None] * 204
+ photo = [None] * 7
+ photo[6] = ["https://lh3.googleusercontent.com/business-photo"]
+ e[37] = [[photo]]
+ e[72] = [[photo, None, [None] * 7]]
+ panel = [None] * 4
+ panel[2], panel[3] = "Exterior", [[None] * 6 + [["https://lh3.googleusercontent.com/exterior"]]]
+ e[171] = [[panel]]
+ out = details.parse_entity(e)
+ images = json.loads(out["images"])
+ self.assertEqual(images, [{"title": "Exterior", "image": "https://lh3.googleusercontent.com/exterior"},
+ {"title": "", "image": "https://lh3.googleusercontent.com/business-photo"}])
+ row = jobs.to_row(out, "x")
+ self.assertEqual(json.loads(row["images"]), images)
+ photo[6] = ["javascript:alert(1)"]
+ e[171] = None
+ self.assertNotIn("images", details.parse_entity(e))
+
+ def test_popular_times_preserves_zero_and_rejects_invalid_hours(self):
+ days = [[7, [[12, 50], [24, 80], [13, True]]],
+ [1, [[8, 0], [9, 100], [-1, 1], [10, 101], None]],
+ [True, [[8, 20]]], [8, [[8, 10]]], [2, None], None]
+ self.assertEqual(json.loads(details.popular_times(days)),
+ {"Monday": {"8": 0, "9": 100}, "Sunday": {"12": 50}})
+ for absent in (None, [], {}, [None], [[1, []]]):
+ self.assertEqual(details.popular_times(absent), "")
+
+ def test_details_only_panel_does_not_return_an_unrequested_review(self):
+ e = entity(reviews=True)
+ e[84] = [[[1, [[8, 20]]]]]
+ body = ")]}'\n" + json.dumps([None] * 6 + [e])
+ out = details.parse_place(body, limit=0)
+ self.assertNotIn("user_reviews", out)
+ self.assertEqual(json.loads(out["popular_times"]), {"Monday": {"8": 20}})
+
+
class FullView(unittest.TestCase):
"""details.py fields that only Google's full view carries (a Chrome window), from made-up data."""
@@ -1270,6 +1308,29 @@ class LoopTwo(Fakes):
_, job = self.run_one(keywords=["a", "b", "c", "d"], proxies=["http://dead:1", "http://ok:2"])
self.assertEqual((job["Status"], calls), ("ok", ["http://dead:1"] + ["http://ok:2"] * 4))
+ def test_extra_details_shares_the_review_pass_and_warns_for_missing_popular_times(self):
+ RESULTS["cafes in Austin"] = ([place(1)], "end-of-list")
+ calls = []
+ def panel(page, url):
+ calls.append(url)
+ return "body"
+ with unittest.mock.patch.object(maps, "place_panel", side_effect=panel), \
+ unittest.mock.patch.object(details, "parse_place", return_value={"images": '[{"title":"","image":"https://example.com/photo"}]'}):
+ jid, job = self.run_one(extra_details=True, reviews_max=1)
+ self.assertEqual(len(calls), 1)
+ self.assertTrue(json.loads(self.rows(jid)[0]["images"]))
+ self.assertEqual(job["Status"], "ok")
+ self.assertIn("popular times unavailable for 1/1 places", job["Error"])
+
+ def test_details_only_runs_without_requesting_reviews(self):
+ RESULTS["cafes in Austin"] = ([place(1)], "end-of-list")
+ calls = []
+ self.worker.add_reviews = lambda rows, d, *a, **k: calls.append(d["reviews_max"]) or 1
+ _, job = self.run_one(extra_details=True)
+ self.assertEqual((calls, job["Status"]), ([0], "ok"))
+ with self.assertRaisesRegex(ValueError, "invalid extra_details"):
+ jobs.validate(dict(GOOD, extra_details="false"))
+
def test_headless_runs_the_reviews_pass(self):
RESULTS["cafes in Austin"] = ([place(1)], "end-of-list")
calls = []
diff --git a/scripts/scrape.py b/scripts/scrape.py
index ef55693..6c7297a 100755
--- a/scripts/scrape.py
+++ b/scripts/scrape.py
@@ -263,7 +263,7 @@ def progress_text(job):
if not isinstance(p, dict) or not p.get("keywords_total"):
return ""
total = p["keywords_total"]
- where = {"reviews": "reading reviews", "websites": "checking websites"}.get(
+ where = {"reviews": "reading place details", "websites": "checking websites"}.get(
p.get("stage"), f"keyword {min(p.get('keywords_done', 0) + 1, total)}/{total}")
return f", {where}, {p.get('places', 0)} places, {p.get('percent', 0)}%"
@@ -314,6 +314,14 @@ def _poll(body, job_id):
print()
if BLOCKED in error: # the rows before the block are kept; searching on would extend it
raise Blocked(f"⚠ The engine says: {error}", job_id, download_rows(job_id))
+ # An unpublished optional schedule is source availability, not an interrupted job.
+ errors = []
+ for part in error.split("; "):
+ if re.fullmatch(r"popular times unavailable for [0-9]+/[0-9]+ places", part):
+ print(f"⚠ {part}")
+ elif part:
+ errors.append(part)
+ error = "; ".join(errors)
if error: # stopped at max_time, or some keywords failed: the rows count, the job is incomplete
raise Partial(f"⚠ The engine says: {error}", job_id, download_rows(job_id))
return job_id, download_rows(job_id)
@@ -824,6 +832,9 @@ def main():
ap.add_argument("--out", default=None, help="output file path (default: results-<id>.csv). "
"The extension picks the format: .csv (default) or .json")
ap.add_argument("--json", action="store_true", help="write JSON instead of the default CSV")
+ ap.add_argument("--details", action="store_true",
+ help="read each place panel for business photos and published popular times; "
+ "implies --full, adds one page load per place")
ap.add_argument("--full", action="store_true", help="keep ALL raw columns (default: lean lead fields)")
ap.add_argument("--fields", help="comma-separated columns to keep (overrides the default lead set)")
ap.add_argument("--socials", action="store_true",
@@ -924,6 +935,8 @@ def main():
# The engine's own email pass reads homepages only; the crawl below also reads contact and
# about pages and decodes obfuscated addresses, so the engine only reads Google Maps.
"depth": a.depth, "email": False, "max_time": a.max_time}
+ if a.details:
+ body["extra_details"] = True
if a.proxy:
body["proxies"] = a.proxy
if a.country:
@@ -956,7 +969,7 @@ def main():
print(f" filters and --exclude kept {len(rows)} of {before} rows")
if a.limit and not a.has_email:
rows = rows[:a.limit]
- if a.full:
+ if a.full or a.details:
fields = list(rows[0].keys()) if rows else LEAD
elif a.fields:
fields = [c.strip() for c in a.fields.split(",") if c.strip()]
diff --git a/scripts/test_scrape.py b/scripts/test_scrape.py
index b6a5616..126dee2 100644
--- a/scripts/test_scrape.py
+++ b/scripts/test_scrape.py
@@ -829,6 +829,43 @@ class EngineProgress(unittest.TestCase):
self.assertIn("keyword 2/2, 20 places, 50%", out.getvalue())
self.assertFalse(any(c.endswith("/download") for c in calls[:-1])) # no download per poll any more
+ def test_unpublished_popular_times_warns_without_marking_completed_search_partial(self):
+ for note, partial in (("popular times unavailable for 1/1 places", False),
+ ("stopped at max_time; popular times unavailable for 1/1 places", True)):
+ def fake_req(method, path, body=None):
+ if method == "POST":
+ return 201, b'{"id":"j1"}'
+ if path.endswith("/download"):
+ return 200, b"title\nA\n"
+ return 200, json.dumps({"Status": "ok", "Error": note}).encode()
+ out = io.StringIO()
+ with mock.patch.object(scrape, "req", fake_req), redirect_stdout(out):
+ if partial:
+ with self.assertRaises(scrape.Partial):
+ scrape.run_job({"max_time": 60, "keywords": ["a"]})
+ else:
+ self.assertEqual(scrape.run_job({"max_time": 60, "keywords": ["a"]})[0], "j1")
+ self.assertIn("popular times unavailable", out.getvalue())
+
+ def test_details_requests_enrichment_and_keeps_raw_columns(self):
+ bodies = []
+ def fake_run_job(body):
+ bodies.append(body)
+ return "j1", [{"title": "Example", "images": '[{"image":"https://example.com/photo"}]',
+ "popular_times": '{"Monday":{"8":20}}'}]
+ with tempfile.TemporaryDirectory() as tmp:
+ out = os.path.join(tmp, "o.csv")
+ argv = ["scrape.py", "x in y", "1", "2", "--details", "--no-email", "--out", out]
+ with mock.patch.object(scrape, "req", return_value=(200, b"null")), \
+ mock.patch.object(scrape, "run_job", fake_run_job), mock.patch.object(sys, "argv", argv), \
+ redirect_stdout(io.StringIO()):
+ scrape.main()
+ with open(out) as f:
+ row = next(csv.DictReader(f))
+ self.assertEqual(json.loads(row["popular_times"]), {"Monday": {"8": 20}})
+ self.assertTrue(json.loads(row["images"]))
+ self.assertIs(bodies[0]["extra_details"], True)
+
def test_country_goes_into_the_job(self):
bodies = []
def fake_run_job(body):