Free, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.
Python agent-skillsb2b-leadsclaude-codeclaude-skillscodex-skillsemail-extractoremail-scrapergoogle-mapsgoogle-maps-extractorgoogle-maps-leadsgoogle-maps-scrapergoogle-maps-scraper-pythongoogle-maps-scrapinggooglemaps-scraperheadless-chromelead-generationlead-scraperno-api-keypythonweb-scraping
fix(engine): photos only from the search results for v2.0.0; place details held back
Mahan Jafari75863d182e49
diff --git a/.claude/skills/google-maps-scraper/SKILL.md b/.claude/skills/google-maps-scraper/SKILL.mdindex b3927fe..61799a4 100644--- a/.claude/skills/google-maps-scraper/SKILL.md+++ b/.claude/skills/google-maps-scraper/SKILL.md@@ -123,9 +123,7 @@ columns are exactly these.`review_count`, `open_hours` (the week) and `reviews_link` are filled when Google sends them: theengine's headless Chrome sends its own user agent with `HeadlessChrome` replaced by `Chrome`, whichgets Google's full view. Google can still leave a field out. Review texts come only through the raw-API's `reviews_max`. `images` holds business photo URLs from the search results; `scripts/scrape.py---details` (API `extra_details`) adds the place panel's photos and `popular_times`, one page load-per place, only when the user asks for them.+API's `reviews_max`. `images` holds business photo URLs from the search results.### Return only the 8 lead fields (default)The raw CSV has 36 columns and most are noise for outreach. **By default, return only the 8 leaddiff --git a/.github/live_reviews_check.py b/.github/live_reviews_check.pyindex cb2cf02..6f6de66 100644--- a/.github/live_reviews_check.py+++ b/.github/live_reviews_check.py@@ -1,5 +1,5 @@"""engine-live.yml: one depth-1 job with reviews_max=1 against the engine on localhost (SCRAPER_PORT, default 8080); exits 1-unless Google's full view came back (review counts and reviews filled). Prints counts only."""+unless Google's full view came back (review counts, reviews and photo URLs filled). Prints counts only."""import json, os, sys, time, urllib.requestBASE = f"http://localhost:{os.environ.get('SCRAPER_PORT', '8080')}/api/v1/jobs"@@ -21,6 +21,6 @@ for _ in range(200):breaktime.sleep(2)rows = json.loads(urllib.request.urlopen(f"{BASE}/{jid}/download?format=json").read())-fill = {k: sum(1 for r in rows if r.get(k)) for k in ("review_count", "open_hours", "user_reviews")}+fill = {k: sum(1 for r in rows if r.get(k)) for k in ("review_count", "open_hours", "user_reviews", "images")}print(f"{sys.argv[1]}: status={job['Status']} places={len(rows)}", fill, job["Progress"])-sys.exit(0 if rows and fill["review_count"] and fill["user_reviews"] else 1)+sys.exit(0 if rows and fill["review_count"] and fill["user_reviews"] and fill["images"] else 1)diff --git a/README.md b/README.mdindex 2da0e88..e582cc0 100644--- a/README.md+++ b/README.md@@ -106,14 +106,11 @@ Illustrative rows:| title | phone | emails | website | category | address | review_rating | review_count ||---|---|---|---|---|---|---|---|-| Example Roasters | +1 512-555-0101 | [email protected] | example.com | Coffee shop | 100 Congress Ave, Austin, TX | 4.7 | |-| Sample Espresso Bar | +1 512-555-0102 | [email protected] | example.org | Cafe | 200 S Lamar Blvd, Austin, TX | 4.5 | |+| Example Roasters | +1 512-555-0101 | [email protected] | example.com | Coffee shop | 100 Congress Ave, Austin, TX | 4.7 | 312 |+| Sample Espresso Bar | +1 512-555-0102 | [email protected] | example.org | Cafe | 200 S Lamar Blvd, Austin, TX | 4.5 | 128 |`review_count` works in headless mode; see [Honest limits](#honest-limits).-`--details` reads each place panel for additional business photo URLs and published hourly popular-times. It implies `--full` and is slower. Missing schedules remain blank; `{}` is not counted as data.-`--socials` adds `instagram`, `facebook`, `linkedin`, `tiktok`, `youtube` and `x` columns. `--tech` adds`fb_pixel`, `google_ads_tag`, `analytics`, `bing_ads`, `linkedin_insight`, `tiktok_pixel`, `cms`,`mobile_friendly` and `intent_score`, and sorts the hottest leads first. `--verify-emails` adds@@ -211,8 +208,7 @@ One keyword per line. One job. One CSV.`open_hours` and `reviews_link`. Google can still leave a field out; if every count is missing,`--min-reviews` warns and skips that filter. Review texts come through the raw API's`reviews_max` (up to 5 per place, one page load each). Business photo URLs come in `images`- (`--full`) with no extra page loads; `--details` adds the place panel's photos and published- popular times, one page load per place. Rather run a real window, with the agent unchanged? [SETUP §3](SETUP.md#review-counts-hours-and-reviews).+ (`--full`), with no extra page loads. Rather run a real window, with the agent unchanged? [SETUP §3](SETUP.md#review-counts-hours-and-reviews).- **Emails only when a business publishes one.** The script reads the business homepage, then itscontact and about pages. Listings with no site, or a site with no visibleemail, return an empty `emails` cell. Instagram bios are not read. Expect emails on a fraction ofdiff --git a/SETUP.md b/SETUP.mdindex bcf80f5..a1c99c2 100644--- a/SETUP.md+++ b/SETUP.md@@ -178,20 +178,6 @@ each place's panel, one page load per place. It fills `user_reviews` (a JSON lisBusiness photo URLs from search results are included in the `images` column without additionalpage loads. Use `--full` to keep that column in CLI output. Photos are linked, not downloaded.-For additional place-panel data, use:--```bash-python3 scripts/scrape.py "coffee shops in Austin TX" 30.2672 -97.7431 --details --no-email-```--`--details` implies `--full` and requests `extra_details: true` from the API. The web UI has a-**Place details** checkbox. The engine reads each place panel once for available photo URLs and-published popular times; jobs requesting reviews share this pass. This adds one page load per place.-`popular_times` is JSON such as `{"Monday": {"8": 20, "9": 50}}`: local hour and relative busyness-from 0 to 100, not visitor counts. Google may provide no schedule. Missing data stays blank, and the-job note reports how many schedules were unavailable; an empty object does not count as a schedule.-No reviewer identities or reviewer photos are added.-Rather not change the user agent? Set `SCRAPER_HEADLESS=0` in `.env` and restart with`python3 scripts/engine.py restart`: Chrome then runs as a normal window placed off-screen, with itsuser agent unchanged, and gets the same data. A window needs a display (not Docker, not a serverdiff --git a/docs/google-maps-scraper-python.md b/docs/google-maps-scraper-python.mdindex 3b9762b..8b7af06 100644--- a/docs/google-maps-scraper-python.md+++ b/docs/google-maps-scraper-python.md@@ -30,7 +30,6 @@ On Windows, run the same commands with `py` or `python3`. The engine needs Pytho| `--max-time` | Job time limit in seconds. Default `600`. || `--out` | Output path. Default `results-<short job id>.csv`. A `.json` extension writes JSON. || `--json` | Writes JSON instead of CSV. |-| `--details` | Reads each place panel for additional business photo URLs and published popular times; implies `--full`, adds one page load per place. Missing schedules stay blank. || `--full` | Keeps all raw columns (the 36 classic ones plus `categories`) instead of the 8 lead fields. Review counts and weekly opening hours work headless ([SETUP §3](../SETUP.md#review-counts-hours-and-reviews)). Review texts stay empty: the script does not ask for them. || `--fields` | Comma-separated columns to keep, overriding the lead fields. || `--socials` | Adds Instagram, Facebook, LinkedIn, TikTok, YouTube and X from each website. Slower. Zero AI tokens. |diff --git a/engine/details.py b/engine/details.pyindex 6374120..c472be0 100644--- a/engine/details.py+++ b/engine/details.py@@ -4,8 +4,9 @@ While the result list scrolls, the page fetches batches of about 20 places from`https://www.google.com/search?tbm=map…` as `)]}'\\n[...]` JSON; each batch carries the full placeentity (name, address, phone, website, rating, coordinates, ids…). The engine captures thoseresponses in the browser (cdp Network events) and parses them here, so no place page is opened for-them. Requested reviews and extra details come from the `/maps/preview/place` response a place's-panel downloads (parse_place). Business photo URLs in search batches need no panel load.+them, and business photo URLs come with them too. Review texts are the one thing the batches lack:+they come from the `/maps/preview/place` response a place's panel downloads (parse_place), only for+jobs that ask for them.Standard library only."""import json, urllib.parse@@ -38,7 +39,6 @@ FIELDS = {"description": (32, 1, 1),}HOURS = (203, 0) # [[day name, 1-7 (Monday-Sunday), date, [[text, ...], ...], ...], ...], from today on-POPULAR = (84, 0) # [[weekday 1-7, [[hour 0-23, busyness 0-100, ...], ...]], ...]HISTOGRAM = (175, 3) # place panel: [one-star count, ..., five-star count]REVIEWS = (175, 9, 0, 0) # place panel: the reviews it shows first (about 5)LISTS = {"categories": (13,), "address_lines": (2,)}@@ -72,9 +72,6 @@ def parse_entity(entity):photos = images(entity)if photos:out["images"] = photos- traffic = popular_times(dig(entity, POPULAR))- if traffic:- out["popular_times"] = trafficif "data_id" in out and ":" in out["data_id"]:try:out["cid"] = str(int(out["data_id"].split(":")[1], 16))@@ -129,29 +126,9 @@ def images(entity):return json.dumps(found, ensure_ascii=False) if found else ""-def popular_times(days):- """Published hourly busyness by weekday; missing data stays absent, never an empty object."""- if not isinstance(days, list):- return ""- names = ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday")- week = {}- for day in days:- number, slots = dig(day, (0,)), dig(day, (1,))- if type(number) is not int or not 1 <= number <= 7 or not isinstance(slots, list):- continue- hours = {}- for slot in slots:- hour, busy = dig(slot, (0,)), dig(slot, (1,))- if type(hour) is int and 0 <= hour <= 23 and type(busy) is int and 0 <= busy <= 100:- hours[str(hour)] = busy- if hours:- week[number] = hours- return json.dumps({names[d - 1]: week[d] for d in sorted(week)}) if week else ""--def parse_place(body, limit=5):"""review_count, reviews_per_rating and up to `limit` user_reviews from one `/maps/preview/place`- response, plus available business photos and popular times. Reviewer identities are omitted."""+ response. Reviewer names and photos are left out: the kit collects business data, not people."""if body.startswith(")]}'"):body = body.split("\n", 1)[-1]try:@@ -160,12 +137,12 @@ def parse_place(body, limit=5):return {}if not isinstance(entity, list):return {}- out = {k: v for k, v in parse_entity(entity).items() if k in ("review_count", "reviews_link", "open_hours", "images", "popular_times")}+ out = {k: v for k, v in parse_entity(entity).items() if k in ("review_count", "reviews_link", "open_hours")}hist = dig(entity, HISTOGRAM)if isinstance(hist, list) and len(hist) == 5 and all(isinstance(n, int) for n in hist):out["reviews_per_rating"] = json.dumps({str(i): n for i, n in enumerate(hist, 1)})reviews = []- for r in (dig(entity, REVIEWS) or []) if limit > 0 else []:+ for r in dig(entity, REVIEWS) or []:rating, text = dig(r, (0, 2, 0, 0)), _text(dig(r, (0, 2, 15, 0, 0)))if isinstance(rating, int) and not isinstance(rating, bool):reviews.append({"Rating": rating, "When": _text(dig(r, (0, 1, 6))), "Description": text})diff --git a/engine/jobs.py b/engine/jobs.pyindex 80a5723..b6e7498 100644--- a/engine/jobs.py+++ b/engine/jobs.py@@ -105,8 +105,6 @@ def validate(body):raise ValueError("invalid geo coordinates")if body.get("max_time") in (None, 0):raise ValueError("missing max time")- if not isinstance(body.get("extra_details", False), bool):- raise ValueError("invalid extra_details: true or false")data = {"keywords": [k.strip() for k in kws], "lang": lang.lower(), "country": country.lower(),"lat": str(lat).strip(), "lon": str(lon).strip(),"zoom": _int(body, "zoom", 15, 1, 21, "zoom"),@@ -116,7 +114,6 @@ def validate(body):"fast_mode": bool(body.get("fast_mode")), "email": bool(body.get("email")),"verify_emails": bool(body.get("verify_emails")), "socials": bool(body.get("socials")),"reviews_max": _int(body, "reviews_max", 0, 0, REVIEWS_MAX, "reviews_max"),- "extra_details": body.get("extra_details", False),"proxies": _proxies(body.get("proxies"))}return name.strip(), data@@ -502,7 +499,7 @@ class Worker(threading.Thread):self._running = (None, None)returndone, stage = min(p.get("done", 0), len(kws)), p.get("stage", "search")- reviews = (d.get("reviews_max") or d.get("extra_details")) and stage in ("search", "reviews")+ reviews = d.get("reviews_max") and stage in ("search", "reviews")if stage == "search":self.store.cut(jid, p.get("bytes", 0)) # the rows of a keyword cut short are searched againrows = self.store.read_rows(jid) # [] when the cut removed the file@@ -653,7 +650,7 @@ class Worker(threading.Thread):n = self.add_reviews(rows, d, deadline, stop, [x for x in proxies if x not in bad] or proxies,save, notes)self.store.rewrite(jid, rows, cols) # keep the reviews if the engine stops during the next pass- self.log(f"job {jid}: place details for {n}/{sum(1 for r in rows if r.get('place_id'))} places")+ self.log(f"job {jid}: reviews for {n}/{sum(1 for r in rows if r.get('place_id'))} places")if stage == "reviews" and not stop.is_set():stage = "websites"save(stage=stage)@@ -728,8 +725,9 @@ class Worker(threading.Thread):self._browser.dispose(ctx)def add_reviews(self, rows, d, deadline, stop, proxies, save=lambda: None, notes=None):- """Read requested place panels once for reviews and extra details, within the job deadline.- The number of panels that supplied parsed data; missing fields stay blank."""+ """Review count, stars histogram and up to reviews_max reviews per place, from each place's+ panel (the result batches carry no review texts): one page load per place, until the+ deadline. The number of places that got reviews."""todo = [r for r in rows if r.get("place_id") and not r.get("user_reviews")]ctx = page = auth = Nonen = 0@@ -757,9 +755,9 @@ class Worker(threading.Thread):breakgot = details.parse_place(body or "", d["reviews_max"])for k, v in got.items():- if k in ("user_reviews", "reviews_per_rating", "images") or not r.get(k):+ if k in ("user_reviews", "reviews_per_rating") or not r.get(k):r[k] = str(v)- n += bool(got)+ n += "user_reviews" in gotsave() # elapsed time, for a resume after a crashexcept (CDPError, OSError) as e: # Chrome or the proxy went away: the reviews so far are keptif notes is not None:@@ -771,10 +769,6 @@ class Worker(threading.Thread):page.close()if ctx and not self._browser.cdp.closed:self._browser.dispose(ctx)- if d.get("extra_details") and notes is not None:- missing = sum(not r.get("popular_times") for r in rows)- if missing:- notes.append(f"popular times unavailable for {missing}/{len(rows)} places")return ndiff --git a/engine/serve.py b/engine/serve.pyindex fdc6685..9b752bf 100644--- a/engine/serve.py+++ b/engine/serve.py@@ -87,7 +87,6 @@ JOB_PROPS = {"drop domains that do not exist, add email_host"},"socials": {"type": "boolean", "description": "visit each website's homepage for Instagram, Facebook, LinkedIn, ""TikTok, YouTube and X links"},- "extra_details": {"type": "boolean", "description": "read each place panel for business photos and published popular times; one page load per place, shared with reviews"},"reviews_max": {"type": "integer", "description": "0-5 reviews per place from its panel, one page load per place (headless supported)"},"proxies": {"type": "array", "items": {"type": "string"},"description": "http(s)://[user:pass@]host:port or socks5://host:port; one per keyword in turn, "@@ -184,7 +183,6 @@ A job the engine was stopped in the middle of carries on when it starts again.</<label>Country (optional)<input id="country" maxlength="2" pattern="[A-Za-z]{{2}}" placeholder="us"></label><label>Time limit (seconds)<input id="max_time" type="number" min="1" max="86400" value="600"></label><label><span>Emails</span><span><input type="checkbox" id="email" checked> visit each website</span></label>- <label><span>Place details</span><span><input type="checkbox" id="details"> photos and published popular times (slower)</span></label><div class="wide row"><button class="primary">Start scraping</button><span class="muted">Coordinates: right-click a spot in Google Maps and click the numbers.</span></div><label class="wide" id="keybox" hidden>API key (SCRAPER_API_KEY)<input id="key" type="password" autocomplete="off"></label>@@ -218,7 +216,7 @@ function progress(j) {{if (j.Status === 'pending') return p.keywords_done ? 'pending (resumes at keyword ' + (p.keywords_done + 1) + ')' : 'pending';if (j.Status !== 'working') return j.Status + (p.places ? ' · ' + p.places + ' places' : '');const stage = {{search: 'keyword ' + Math.min(p.keywords_done + 1, p.keywords_total) + '/' + p.keywords_total,- reviews: 'reading place details', websites: 'checking websites'}}[p.stage] || p.stage || '';+ reviews: 'reading reviews', websites: 'checking websites'}}[p.stage] || p.stage || '';return 'working · ' + stage + ' · ' + (p.places || 0) + ' places · ' + (p.percent || 0) + '%';}}async function remove(j) {{@@ -247,7 +245,7 @@ $('f').addEventListener('submit', async ev => {{const body = {{name: keywords[0] || 'job', keywords, lang: $('lang').value.trim().toLowerCase(),country: $('country').value.trim().toLowerCase(),lat: $('lat').value.trim(), lon: $('lon').value.trim(), depth: +$('depth').value,- max_time: +$('max_time').value, email: $('email').checked, extra_details: $('details').checked, zoom: 15}};+ max_time: +$('max_time').value, email: $('email').checked, zoom: 15}};try {{const r = await api('POST', '/api/v1/jobs', body), d = await r.json();say(r.ok ? 'Job created. It runs in the background; this list refreshes by itself.' : 'Not started: ' + d.message);diff --git a/engine/test_engine.py b/engine/test_engine.pyindex 40313c5..32fb796 100644--- a/engine/test_engine.py+++ b/engine/test_engine.py@@ -994,24 +994,6 @@ class ExtraDetailsParser(unittest.TestCase):e[171] = Noneself.assertNotIn("images", details.parse_entity(e))- def test_popular_times_preserves_zero_and_rejects_invalid_hours(self):- days = [[7, [[12, 50], [24, 80], [13, True]]],- [1, [[8, 0], [9, 100], [-1, 1], [10, 101], None]],- [True, [[8, 20]]], [8, [[8, 10]]], [2, None], None]- self.assertEqual(json.loads(details.popular_times(days)),- {"Monday": {"8": 0, "9": 100}, "Sunday": {"12": 50}})- for absent in (None, [], {}, [None], [[1, []]]):- self.assertEqual(details.popular_times(absent), "")-- def test_details_only_panel_does_not_return_an_unrequested_review(self):- e = entity(reviews=True)- e[84] = [[[1, [[8, 20]]]]]- body = ")]}'\n" + json.dumps([None] * 6 + [e])- out = details.parse_place(body, limit=0)- self.assertNotIn("user_reviews", out)- self.assertEqual(json.loads(out["popular_times"]), {"Monday": {"8": 20}})--class FullView(unittest.TestCase):"""details.py fields that only Google's full view carries (a Chrome window), from made-up data."""@@ -1308,29 +1290,6 @@ class LoopTwo(Fakes):_, job = self.run_one(keywords=["a", "b", "c", "d"], proxies=["http://dead:1", "http://ok:2"])self.assertEqual((job["Status"], calls), ("ok", ["http://dead:1"] + ["http://ok:2"] * 4))- def test_extra_details_shares_the_review_pass_and_warns_for_missing_popular_times(self):- RESULTS["cafes in Austin"] = ([place(1)], "end-of-list")- calls = []- def panel(page, url):- calls.append(url)- return "body"- with unittest.mock.patch.object(maps, "place_panel", side_effect=panel), \- unittest.mock.patch.object(details, "parse_place", return_value={"images": '[{"title":"","image":"https://example.com/photo"}]'}):- jid, job = self.run_one(extra_details=True, reviews_max=1)- self.assertEqual(len(calls), 1)- self.assertTrue(json.loads(self.rows(jid)[0]["images"]))- self.assertEqual(job["Status"], "ok")- self.assertIn("popular times unavailable for 1/1 places", job["Error"])-- def test_details_only_runs_without_requesting_reviews(self):- RESULTS["cafes in Austin"] = ([place(1)], "end-of-list")- calls = []- self.worker.add_reviews = lambda rows, d, *a, **k: calls.append(d["reviews_max"]) or 1- _, job = self.run_one(extra_details=True)- self.assertEqual((calls, job["Status"]), ([0], "ok"))- with self.assertRaisesRegex(ValueError, "invalid extra_details"):- jobs.validate(dict(GOOD, extra_details="false"))-def test_headless_runs_the_reviews_pass(self):RESULTS["cafes in Austin"] = ([place(1)], "end-of-list")calls = []diff --git a/scripts/scrape.py b/scripts/scrape.pyindex 6c7297a..ef55693 100755--- a/scripts/scrape.py+++ b/scripts/scrape.py@@ -263,7 +263,7 @@ def progress_text(job):if not isinstance(p, dict) or not p.get("keywords_total"):return ""total = p["keywords_total"]- where = {"reviews": "reading place details", "websites": "checking websites"}.get(+ where = {"reviews": "reading reviews", "websites": "checking websites"}.get(p.get("stage"), f"keyword {min(p.get('keywords_done', 0) + 1, total)}/{total}")return f", {where}, {p.get('places', 0)} places, {p.get('percent', 0)}%"@@ -314,14 +314,6 @@ def _poll(body, job_id):print()if BLOCKED in error: # the rows before the block are kept; searching on would extend itraise Blocked(f"⚠ The engine says: {error}", job_id, download_rows(job_id))- # An unpublished optional schedule is source availability, not an interrupted job.- errors = []- for part in error.split("; "):- if re.fullmatch(r"popular times unavailable for [0-9]+/[0-9]+ places", part):- print(f"⚠ {part}")- elif part:- errors.append(part)- error = "; ".join(errors)if error: # stopped at max_time, or some keywords failed: the rows count, the job is incompleteraise Partial(f"⚠ The engine says: {error}", job_id, download_rows(job_id))return job_id, download_rows(job_id)@@ -832,9 +824,6 @@ def main():ap.add_argument("--out", default=None, help="output file path (default: results-<id>.csv). ""The extension picks the format: .csv (default) or .json")ap.add_argument("--json", action="store_true", help="write JSON instead of the default CSV")- ap.add_argument("--details", action="store_true",- help="read each place panel for business photos and published popular times; "- "implies --full, adds one page load per place")ap.add_argument("--full", action="store_true", help="keep ALL raw columns (default: lean lead fields)")ap.add_argument("--fields", help="comma-separated columns to keep (overrides the default lead set)")ap.add_argument("--socials", action="store_true",@@ -935,8 +924,6 @@ def main():# The engine's own email pass reads homepages only; the crawl below also reads contact and# about pages and decodes obfuscated addresses, so the engine only reads Google Maps."depth": a.depth, "email": False, "max_time": a.max_time}- if a.details:- body["extra_details"] = Trueif a.proxy:body["proxies"] = a.proxyif a.country:@@ -969,7 +956,7 @@ def main():print(f" filters and --exclude kept {len(rows)} of {before} rows")if a.limit and not a.has_email:rows = rows[:a.limit]- if a.full or a.details:+ if a.full:fields = list(rows[0].keys()) if rows else LEADelif a.fields:fields = [c.strip() for c in a.fields.split(",") if c.strip()]diff --git a/scripts/test_scrape.py b/scripts/test_scrape.pyindex 126dee2..b6a5616 100644--- a/scripts/test_scrape.py+++ b/scripts/test_scrape.py@@ -829,43 +829,6 @@ class EngineProgress(unittest.TestCase):self.assertIn("keyword 2/2, 20 places, 50%", out.getvalue())self.assertFalse(any(c.endswith("/download") for c in calls[:-1])) # no download per poll any more- def test_unpublished_popular_times_warns_without_marking_completed_search_partial(self):- for note, partial in (("popular times unavailable for 1/1 places", False),- ("stopped at max_time; popular times unavailable for 1/1 places", True)):- def fake_req(method, path, body=None):- if method == "POST":- return 201, b'{"id":"j1"}'- if path.endswith("/download"):- return 200, b"title\nA\n"- return 200, json.dumps({"Status": "ok", "Error": note}).encode()- out = io.StringIO()- with mock.patch.object(scrape, "req", fake_req), redirect_stdout(out):- if partial:- with self.assertRaises(scrape.Partial):- scrape.run_job({"max_time": 60, "keywords": ["a"]})- else:- self.assertEqual(scrape.run_job({"max_time": 60, "keywords": ["a"]})[0], "j1")- self.assertIn("popular times unavailable", out.getvalue())-- def test_details_requests_enrichment_and_keeps_raw_columns(self):- bodies = []- def fake_run_job(body):- bodies.append(body)- return "j1", [{"title": "Example", "images": '[{"image":"https://example.com/photo"}]',- "popular_times": '{"Monday":{"8":20}}'}]- with tempfile.TemporaryDirectory() as tmp:- out = os.path.join(tmp, "o.csv")- argv = ["scrape.py", "x in y", "1", "2", "--details", "--no-email", "--out", out]- with mock.patch.object(scrape, "req", return_value=(200, b"null")), \- mock.patch.object(scrape, "run_job", fake_run_job), mock.patch.object(sys, "argv", argv), \- redirect_stdout(io.StringIO()):- scrape.main()- with open(out) as f:- row = next(csv.DictReader(f))- self.assertEqual(json.loads(row["popular_times"]), {"Monday": {"8": 20}})- self.assertTrue(json.loads(row["images"]))- self.assertIs(bodies[0]["extra_details"], True)-def test_country_goes_into_the_job(self):bodies = []def fake_run_job(body):