AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary
Python ai-promptsai-skillblueskyclaudeclaude-codeclawhubdeep-researchhackernewsinstagramopenclawpolymarketrecencyredditresearchsocial-mediatiktoktrendstwitterweb-searchyoutube
fix(amazon): start review lane at search time so multi-source runs keep a real budget (#999)
Matt Van Hornacdc397fec2e
diff --git a/changelog.d/+amazon-review-lane-at-search-time.fixed.md b/changelog.d/+amazon-review-lane-at-search-time.fixed.mdnew file mode 100644index 00000000..a535e4db--- /dev/null+++ b/changelog.d/+amazon-review-lane-at-search-time.fixed.md@@ -0,0 +1 @@+Amazon review enrichment now starts at search time instead of after all other sources finish, ensuring multi-source runs have a useful budget (up to 180s) rather than leftover crumbs. Previously, a run that spent 269s on retrieval would leave only 11s for reviews, causing all Bright Data pulls to time out. Budgets below 90s now skip the lane entirely instead of firing doomed short pulls that spend credits without returning reviews.diff --git a/docs/plans/2026-08-14-fix-amazon-review-budget-plan.md b/docs/plans/2026-08-14-fix-amazon-review-budget-plan.mdnew file mode 100644index 00000000..c254a682--- /dev/null+++ b/docs/plans/2026-08-14-fix-amazon-review-budget-plan.md@@ -0,0 +1,74 @@+# Fix Amazon Review Budget by Starting at Search Time++**Date:** 2026-08-14+**Status:** Implemented++## Problem (measured 2026-08-14 Bentgo run)++- Full multi-source run. Amazon search listings returned (12 products, stars, rating counts).+- Review lane logged: "pulling up to 50 reviews for 3 products (budget 11s)"+- "lane deadline 11s hit; dropped 3 straggling pull(s)"+- Bright Data timed out after 11s x3. Zero review bodies. Credits spent.+- Isolated Amazon-only re-run got the full 180s, finished in 124s, reviews landed.+- Cause: `_remaining_lane_budget` = min(LANE_DEADLINE=180, max(0, FOREGROUND_CONTRACT=300 - elapsed - RENDER_MARGIN=20)). Enrichment runs after all other sources. Elapsed ~269s → 11s.++## Why the floor-up fix was wrong++`max(120, leftover)` at elapsed=269 means the run goes to ~389–449s. The host Bash contract is 300s (`SKILL.md` 300000ms). That kills the **whole** report. Do not do that.++The measured failure is **when** the review lane starts, not pull quality. Isolated Amazon-only already finishes in 124s of 180s. On a full run, search has already landed products, then reviews wait through every other source plus Phase 2/2b, then get 11s.++## What to build++1. Start `enrich_with_reviews` when Amazon search returns, inside `_retrieve_stream` (amazon branch ~4284), overlapping other source futures. Pass real `elapsed = time.monotonic() - run_started` (thread `run_started` into retrieve). After a 30–90s search, leftover is 190–250s → clamp to 180. Isolated Amazon-only unchanged.++2. Keep finalize (`_finalize_items_by_source` ~2956) as attach-if-missing only. `enrich_source_items` already no-ops if `top_comments` is set. Do not make finalize the only start. Do not enrich inline on the collect loop (that serializes other sources for 124s).++3. Leftover below a useful floor → budget **0**, skip the lane. Do not fire doomed 11s pulls (Bright Data `cli_timeout = max(5, timeout-10)` so budget 11s → CLI timeout 1s, still spends 3 credits). Suggested `MIN_USEFUL_REVIEW_BUDGET = 90`. Crumbs are a skip, not a short timeout.++4. If the lane is skipped or all pulls drop: Amazon `source_status` **PARTIAL** with detail like `review lane timed out` / `review lane skipped (budget 0s)`. Listings stay. Do not flip the whole source to `timeout` (search succeeded). Footer already shows ⚠ when state != ok — do not change render.py.++5. `depth=quick` still 0 pulls. `mock=True` still skips. No env knob. No raising LANE_DEADLINE or FOREGROUND_CONTRACT.++## Out of scope++- Do NOT change footer/render.py.+- Do NOT change X search or Grok auth.+- Do NOT add an env knob.+- Do NOT reorder the whole source schedule (deferred).++## Implementation++### amazon.py++- Added `MIN_USEFUL_REVIEW_BUDGET = 90`+- Changed `_remaining_lane_budget` to return 0 when below floor (not floor up)+- Changed `enrich_with_reviews` to return `(products, status_detail)` tuple where status_detail is:+ - `None` for normal success+ - `"review lane skipped (budget 0s)"` when budget is below floor+ - `"review lane timed out"` when all pulls dropped++### pipeline.py++- Added `run_started` parameter to `_retrieve_stream`, `_retrieve_stream_impl`, and `_retry_thin_sources`+- Updated all call sites to pass `run_started`+- In the Amazon branch of `_retrieve_stream_impl`:+ - Run search as before+ - Calculate `elapsed = time.monotonic() - run_started`+ - Call `enrich_with_reviews` immediately after search returns+ - If enrichment reports a degraded status, add `_source_outcome` with `state=PARTIAL` to the artifact+- Updated finalize comments to note it's now attach-if-missing only++### Tests (test_amazon.py)++Retargeted existing tests:+- `test_lane_budget_shrinks_as_the_run_clock_advances` — now tests floor behavior+- `test_dropped_straggler_keeps_its_product_with_search_stats` — uses patched short LANE_DEADLINE+- `test_exhausted_wall_clock_skips_the_lane_entirely` — unchanged++New tests:+- `test_lane_budget_floor_prevents_doomed_pulls` — verifies elapsed=269 returns 0+- `test_lane_budget_constants_are_sane` — guards against constant drift+- `test_crumb_budget_skips_not_fires_doomed_pulls` — regression test for the Bentgo bug+- `test_early_elapsed_gets_full_budget` — verifies elapsed=40 gets 180s timeout+- `test_all_pulls_dropped_reports_timed_out_status` — verifies PARTIAL status on all-droppeddiff --git a/skills/last30days/scripts/lib/amazon.py b/skills/last30days/scripts/lib/amazon.pyindex b2f4b265..289ea2ba 100644--- a/skills/last30days/scripts/lib/amazon.py+++ b/skills/last30days/scripts/lib/amazon.py@@ -74,6 +74,12 @@ LANE_DEADLINE = 180FOREGROUND_CONTRACT = 300RENDER_MARGIN = 20+# Minimum useful budget for the review lane. Below this threshold, Bright+# Data pulls reliably time out (cli_timeout = max(5, timeout-10), so budget+# 11s → CLI timeout 1s). Crumbs are a skip, not a short timeout: firing+# doomed pulls still spends 3 credits with no reviews returned.+MIN_USEFUL_REVIEW_BUDGET = 90+# Minimum dated reviews inside the window before a drift arrow is honest.# Live census: a 50-cap pull returned 31 records of which only 5 were# inside 30 days, so an unguarded arrow would routinely publish a "trend"@@ -526,9 +532,17 @@ def parse_reviews(response: Dict[str, Any]) -> Tuple[List[Dict[str, Any]], Dict[def _remaining_lane_budget(elapsed: float) -> int:- """Clamp the lane deadline against what is left of the run's contract."""+ """Compute the review lane's wall-clock budget.++ Returns the lesser of LANE_DEADLINE and whatever remains of the foreground+ contract. If the remaining time is below MIN_USEFUL_REVIEW_BUDGET, returns+ 0 (skip the lane entirely) rather than firing doomed short pulls that spend+ Bright Data credits without returning reviews.+ """remaining = FOREGROUND_CONTRACT - elapsed - RENDER_MARGIN- return int(min(LANE_DEADLINE, max(0, remaining)))+ if remaining < MIN_USEFUL_REVIEW_BUDGET:+ return 0+ return int(min(LANE_DEADLINE, remaining))def enrich_with_reviews(@@ -541,7 +555,7 @@ def enrich_with_reviews(brand: str = "",keyword: str = "",fetcher=None,-) -> List[Dict[str, Any]]:+) -> Tuple[List[Dict[str, Any]], Optional[str]]:"""Attach review samples to the top products, in parallel, under a deadline.Every product is returned either way. A product whose pull is dropped@@ -549,23 +563,28 @@ def enrich_with_reviews(review sample -- it renders as ``quiet`` rather than vanishing, becauselosing a top product entirely is a worse failure than losing itsrecent-window read. The dropped pull has spent its credit regardless.++ Returns (enriched_products, status_detail). status_detail is None when+ enrichment succeeded normally, or a string describing a degraded outcome:+ - ``"review lane skipped (budget 0s)"`` -- crumb budget, lane did not run+ - ``"review lane timed out"`` -- all pulls dropped by the deadline"""enriched = [dict(p) for p in products]pull_count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])if pull_count <= 0:_log(f"depth={depth}: discovery only, no review pulls")- return enriched+ return enriched, Nonebudget = _remaining_lane_budget(elapsed)if budget <= 0:- _log("no wall-clock budget left for review pulls; skipping the lane")- return enriched+ _log(f"review lane skipped (budget {budget}s, floor {MIN_USEFUL_REVIEW_BUDGET}s)")+ return enriched, "review lane skipped (budget 0s)"targets = select_enrichment_targets(enriched, limit=pull_count, brand=brand, keyword=keyword)if not targets:- return enriched+ return enriched, Noneby_asin = {p["asin"]: p for p in enriched}pull = fetcher or (@@ -577,6 +596,8 @@ def enrich_with_reviews(_log(f"pulling up to {max_reviews} reviews for {len(targets)} products (budget {budget}s)")started = time.monotonic()+ completed_count = 0+ dropped_count = 0# Not a `with` block on purpose. Every future is already running (one# worker per target), so `future.cancel()` can never succeed, and# ThreadPoolExecutor's context-manager exit calls shutdown(wait=True) --@@ -604,14 +625,21 @@ def enrich_with_reviews(continueproduct["top_comments"] = commentsproduct.update({k: v for k, v in stats.items() if v})+ completed_count += 1except TimeoutError:- dropped = sum(1 for f in futures if not f.done())- _log(f"lane deadline {budget}s hit; dropped {dropped} straggling pull(s)")+ dropped_count = sum(1 for f in futures if not f.done())+ _log(f"lane deadline {budget}s hit; dropped {dropped_count} straggling pull(s)")finally:pool.shutdown(wait=False, cancel_futures=True)_log(f"review lane finished in {time.monotonic() - started:.0f}s")- return enriched++ # Report degraded outcome if all pulls dropped (none completed)+ status_detail = None+ if completed_count == 0 and dropped_count > 0:+ status_detail = "review lane timed out"++ return enriched, status_detaildef enrich_source_items(@@ -657,7 +685,7 @@ def enrich_source_items(if not products:return items- enriched = enrich_with_reviews(+ enriched, _status = enrich_with_reviews(products, depth=depth, config=config, elapsed=elapsed,max_reviews=max_reviews, keyword=keyword, fetcher=fetcher,)diff --git a/skills/last30days/scripts/lib/pipeline.py b/skills/last30days/scripts/lib/pipeline.pyindex bbb1ab8a..d07e5e8c 100644--- a/skills/last30days/scripts/lib/pipeline.py+++ b/skills/last30days/scripts/lib/pipeline.py@@ -1955,10 +1955,9 @@ def run(youtube_yt.reset_search_cache()settings = _resolve_depth_settings(depth, config)requested_sources = normalize_requested_sources(requested_sources)- # Wall-clock origin for budget-aware enrichment lanes. Retrieval can eat- # most of the foreground contract before enrichment starts, so a lane that- # sizes its deadline against a constant alone can push a run past the- # contract; the amazon review lane clamps against what is actually left.+ # Wall-clock origin for budget-aware enrichment lanes. Amazon review+ # enrichment starts at search time (inside _retrieve_stream_impl) so it+ # overlaps other sources instead of waiting for them all to finish.run_started = time.monotonic()from_date, to_date = dates.get_date_range(lookback_days, as_of_date=as_of_date)resolved_corpus_dirs = corpus.resolve_directories(@@ -2327,6 +2326,7 @@ def run(ig_creators=ig_creators,trustpilot_domain=trustpilot_domain,trustpilot_domain_is_hint=trustpilot_domain_is_hint,+ run_started=run_started,)] = (subquery, source)@@ -2361,6 +2361,7 @@ def run(ig_creators=ig_creators,trustpilot_domain=trustpilot_domain,trustpilot_domain_is_hint=trustpilot_domain_is_hint,+ run_started=run_started,)except Exception as retry_exc:detail = f"{exc} (retried once, still failed: {retry_exc})"@@ -2444,6 +2445,7 @@ def run(tiktok_creators=tiktok_creators,ig_creators=ig_creators,first_party_handles=explicit_first_party,+ run_started=run_started,)# Reclassify partial failures as DEGRADED instead of silently dropping them.@@ -2954,9 +2956,11 @@ def _finalize_items_by_source(digg.enrich_source_items(items, top_k=3)http.fixture_source_record(enrichment_request, schema.to_dict(items))if source == "amazon" and items and not mock:- # Same budget-at-the-survivors principle: review pulls are the- # expensive half of this lane (one credit each, and the slow- # half by wall clock), so they go to the products dedupe kept.+ # Attach-if-missing: review enrichment now runs at search time in+ # _retrieve_stream_impl, so items arriving here should already have+ # top_comments. enrich_source_items no-ops when top_comments is set.+ # This path handles fixture replay and any edge cases where retrieve+ # didn't enrich (e.g., run_started was not passed).matched, replayed = http.fixture_source_replay(enrichment_request)if matched:items = _merge_replayed_enrichment(items, replayed)@@ -2966,9 +2970,6 @@ def _finalize_items_by_source(depth=depth,config=config,keyword=str((config or {}).get("_amazon_query") or "").strip() or topic,- # Retrieval time already spent, so the lane sizes its- # deadline against what remains of the 300s contract- # rather than always claiming its full constant.elapsed=elapsed,)http.fixture_source_record(enrichment_request, schema.to_dict(items))@@ -3689,6 +3690,7 @@ def _retry_thin_sources(tiktok_creators: list[str] | None = None,ig_creators: list[str] | None = None,first_party_handles: Iterable[str] | None = None,+ run_started: float | None = None,) -> None:"""Retry sources with thin results using simplified core subject query."""if depth == "quick":@@ -3755,6 +3757,11 @@ def _retry_thin_sources(tiktok_hashtags=tiktok_hashtags,tiktok_creators=tiktok_creators,ig_creators=ig_creators,+ run_started=run_started,+ # Skip Amazon review enrichment here to avoid duplicate Bright Data+ # pulls for ASINs already enriched in Phase 1. Finalize will enrich+ # any genuinely new products that weren't in Phase 1.+ skip_amazon_enrichment=True,)outcome_note = artifact.get("_source_outcome") if isinstance(artifact, dict) else Nonenormalized = _normalize_score_dedupe(@@ -3864,6 +3871,7 @@ def _merge_reddit_items(free: list[dict], sc: list[dict]) -> list[dict]:def _retrieve_stream(*args, **kwargs) -> tuple[list[dict], dict]:"""Run one stream and retain HTTP failures swallowed by source adapters."""+ # run_started is passed through but not used here; it goes to _retrieve_stream_implsource = str(kwargs.get("source") or "")fixture_request = {"source": source,@@ -3934,6 +3942,8 @@ def _retrieve_stream_impl(ig_creators: list[str] | None = None,trustpilot_domain: str | None = None,trustpilot_domain_is_hint: bool = False,+ run_started: float | None = None,+ skip_amazon_enrichment: bool = False,) -> tuple[list[dict], dict]:# Early exit if source was rate-limited by a sibling futureif rate_limited_sources is not None and source in rate_limited_sources:@@ -4285,18 +4295,47 @@ def _retrieve_stream_impl(# The search keyword is model-supplied and may differ from the topic# ("Matt Van Horn" searches "June Oven"), so it keys off the stable# research topic rather than the narrowed per-subquery search_query.- # Review enrichment is deferred to _finalize_items_by_source so the- # credit budget lands on products that survive dedupe.keyword = (str((config or {}).get("_amazon_query") or "").strip()or raw_topic or topic or subquery.search_query)domain = str((config or {}).get("LAST30DAYS_AMAZON_DOMAIN") or amazon.DEFAULT_DOMAIN)result = amazon.search_products(keyword, domain=domain, config=config)- return (- amazon.parse_search_response(result, keyword, domain=domain),- _result_outcome_artifact(source, result),+ products = amazon.parse_search_response(result, keyword, domain=domain)+ artifact = _result_outcome_artifact(source, result)++ # Skip enrichment when called from thin retry (_retry_thin_sources) to+ # avoid duplicate Bright Data pulls for ASINs already enriched in Phase 1.+ # Finalize will enrich any NEW products (enrich_source_items no-ops when+ # top_comments is already set, so duplicates get skipped there too).+ if skip_amazon_enrichment:+ return products, artifact++ # Start review enrichment now, while other sources are still running.+ # Elapsed is measured from run_started so multi-source runs that finish+ # search quickly (30-90s) still have 190-250s of budget (clamped to 180).+ # This replaces the old deferred-to-finalize path which left only crumbs+ # (e.g. 11s) after long retrieval phases.+ elapsed = time.monotonic() - run_started if run_started else 0.0+ enriched, review_status = amazon.enrich_with_reviews(+ products,+ depth=depth,+ config=config,+ elapsed=elapsed,+ keyword=keyword,)++ # Record PARTIAL status if review lane was skipped or all pulls dropped+ if review_status:+ artifact = artifact or {}+ artifact = dict(artifact) if artifact else {}+ artifact["_source_outcome"] = {+ "state": schema.PARTIAL,+ "detail": review_status,+ "attempted": True,+ }++ return enriched, artifactif source == "bluesky":result = bluesky.search_bluesky(subquery.search_query, from_date, to_date, depth=depth, config=config)return bluesky.parse_bluesky_response(result), _result_outcome_artifact(source, result)diff --git a/tests/test_amazon.py b/tests/test_amazon.pyindex 77882214..da9da1e4 100644--- a/tests/test_amazon.py+++ b/tests/test_amazon.py@@ -278,24 +278,26 @@ class TestEnrichmentLane:def test_quick_depth_spawns_no_review_pulls(self):calls = []- out = amazon.enrich_with_reviews(+ out, status = amazon.enrich_with_reviews(self._products(), depth="quick",fetcher=lambda url: calls.append(url) or {"records": []},)assert calls == []assert len(out) == 4+ assert status is None # quick depth is not a degraded outcomedef test_default_depth_pulls_exactly_three(self):calls = []- amazon.enrich_with_reviews(+ _, status = amazon.enrich_with_reviews(self._products(), depth="default",fetcher=lambda url: calls.append(url) or {"records": [review_record(2, 5)]},)assert len(calls) == 3+ assert status is Nonedef test_deep_depth_pulls_five(self):calls = []- amazon.enrich_with_reviews(+ _, status = amazon.enrich_with_reviews(self._products(6), depth="deep",fetcher=lambda url: calls.append(url) or {"records": []},)@@ -320,12 +322,13 @@ class TestEnrichmentLane:assert seen["params"] == ["https://www.amazon.com/dp/B000000001", "50"]def test_reviews_attach_to_the_right_product(self):- out = amazon.enrich_with_reviews(+ out, status = amazon.enrich_with_reviews(self._products(3), depth="default",fetcher=lambda url: {"records": [review_record(2, 5, review_id=url)]},)assert all(p.get("top_comments") for p in out)assert out[0]["product_rating_count"] == 459+ assert status is Nonedef test_one_failing_pull_does_not_discard_its_siblings(self):def fetcher(url):@@ -333,11 +336,12 @@ class TestEnrichmentLane:return {"records": [], "error": "snapshot failed"}return {"records": [review_record(2, 5)]}- out = amazon.enrich_with_reviews(self._products(3), depth="default", fetcher=fetcher)+ out, status = amazon.enrich_with_reviews(self._products(3), depth="default", fetcher=fetcher)by_asin = {p["asin"]: p for p in out}assert not by_asin["B000000001"].get("top_comments")assert by_asin["B000000000"].get("top_comments")assert by_asin["B000000002"].get("top_comments")+ assert status is None # partial success is not reported as statusdef test_one_raising_pull_does_not_kill_the_lane(self):def fetcher(url):@@ -345,36 +349,127 @@ class TestEnrichmentLane:raise RuntimeError("boom")return {"records": [review_record(2, 5)]}- out = amazon.enrich_with_reviews(self._products(3), depth="default", fetcher=fetcher)+ out, status = amazon.enrich_with_reviews(self._products(3), depth="default", fetcher=fetcher)assert sum(1 for p in out if p.get("top_comments")) == 2+ assert status is None # partial success is not reported as statusdef test_dropped_straggler_keeps_its_product_with_search_stats(self):- """A deadline drop must never delete the product from the report."""+ """A deadline drop must never delete the product from the report.++ Use a patched short LANE_DEADLINE to trigger the straggler case without+ relying on crumb budgets (which now skip the lane entirely).+ """import time as _time+ import unittest.mockdef slow(url):if url.endswith("B000000000"):_time.sleep(3)return {"records": [review_record(2, 5)]}- out = amazon.enrich_with_reviews(- self._products(2), depth="default", fetcher=slow,- elapsed=amazon.FOREGROUND_CONTRACT - amazon.RENDER_MARGIN - 1,- )+ # Patch LANE_DEADLINE to 1s so the slow product times out but other+ # products have time to complete. elapsed=0 keeps the full 1s budget.+ with unittest.mock.patch.object(amazon, "LANE_DEADLINE", 1):+ out, status = amazon.enrich_with_reviews(+ self._products(2), depth="default", fetcher=slow,+ elapsed=0.0,+ )by_asin = {p["asin"]: p for p in out}assert set(by_asin) == {"B000000000", "B000000001"}# Search-record stats survive on the dropped product.assert by_asin["B000000000"]["rating"] == 4.4assert by_asin["B000000000"]["num_ratings"] == 900+ # One straggler dropped, so status should be "review lane timed out"+ # (since all 3 pulls didn't complete, but some did)+ # Actually, B000000001 completed, so status is None+ # Let me check: if completed_count > 0, status is Nonedef test_exhausted_wall_clock_skips_the_lane_entirely(self):calls = []- amazon.enrich_with_reviews(+ out, status = amazon.enrich_with_reviews(self._products(), depth="default",fetcher=lambda url: calls.append(url) or {"records": []},elapsed=amazon.FOREGROUND_CONTRACT,)assert calls == []+ assert status == "review lane skipped (budget 0s)"++ def test_crumb_budget_skips_not_fires_doomed_pulls(self):+ """Regression test for the Bentgo bug: elapsed=269 must skip, not fire doomed 11s pulls.++ The bug: a multi-source run took 269s before reaching Amazon enrichment,+ leaving only 11s of budget (300 - 269 - 20 = 11). All Bright Data pulls+ timed out (cli_timeout = max(5, timeout-10) = 1s), spending credits+ without returning reviews.++ The fix: crumb budgets (below MIN_USEFUL_REVIEW_BUDGET=90) return 0,+ skipping the lane entirely instead of firing doomed short pulls.+ """+ calls = []+ out, status = amazon.enrich_with_reviews(+ self._products(), depth="default",+ fetcher=lambda url: calls.append(url) or {"records": []},+ elapsed=269.0,+ )+ # Fetcher should never be called+ assert calls == []+ # Products keep their search stats (no top_comments)+ assert all(not p.get("top_comments") for p in out)+ assert all(p.get("rating") == 4.4 for p in out)+ assert all(p.get("num_ratings") for p in out)+ # Status should indicate the lane was skipped+ assert status == "review lane skipped (budget 0s)"++ def test_early_elapsed_gets_full_budget(self):+ """A quick search (40s elapsed) should get the full LANE_DEADLINE budget."""+ timeouts_seen = []++ def capturing_fetcher(url):+ return {"records": [review_record(2, 5)]}++ # Patch fetch_reviews to capture the timeout+ original_fetch = amazon.fetch_reviews+ captured_timeout = []++ def mock_fetch(url, *, max_reviews=50, config=None, timeout=180):+ captured_timeout.append(timeout)+ return {"records": [review_record(2, 5)]}++ amazon.fetch_reviews = mock_fetch+ try:+ out, status = amazon.enrich_with_reviews(+ self._products(1), depth="default",+ elapsed=40.0,+ )+ finally:+ amazon.fetch_reviews = original_fetch++ assert captured_timeout, "fetch_reviews was not called"+ # elapsed=40 → remaining = 300-40-20 = 240 → clamped to LANE_DEADLINE=180+ assert captured_timeout[0] == amazon.LANE_DEADLINE+ assert status is None++ def test_all_pulls_dropped_reports_timed_out_status(self):+ """When all pulls drop (none complete), status should be 'review lane timed out'."""+ import time as _time+ import unittest.mock++ def very_slow(url):+ _time.sleep(5) # Longer than the deadline+ return {"records": [review_record(2, 5)]}++ # Use a very short deadline so all pulls time out+ with unittest.mock.patch.object(amazon, "LANE_DEADLINE", 1):+ with unittest.mock.patch.object(amazon, "MIN_USEFUL_REVIEW_BUDGET", 1):+ out, status = amazon.enrich_with_reviews(+ self._products(2), depth="default", fetcher=very_slow,+ elapsed=0.0,+ )++ # No products should have top_comments (all dropped)+ assert all(not p.get("top_comments") for p in out)+ # Status should indicate timeout+ assert status == "review lane timed out"# --------------------------------------------------------------- stats@@ -540,6 +635,15 @@ class TestSourceItemEnrichment:assert "top_comments" not in other.metadatadef test_already_enriched_items_are_not_re_pulled(self):+ """enrich_source_items no-ops when top_comments is already set.++ This is critical for the thin-retry path: Phase 1 enriches products at+ search time, then thin retry (Phase 2b) may return the same ASINs. The+ pipeline passes skip_amazon_enrichment=True in thin retry, so products+ arrive at finalize without re-enrichment. Finalize calls enrich_source_items,+ which skips already-enriched items (top_comments set) and only enriches+ genuinely new products. This prevents duplicate Bright Data pulls.+ """item = _Item("B000000001", top_comments=[{"excerpt": "cached", "score": 0, "rating": 5, "date": None}])calls = []amazon.enrich_source_items(@@ -719,11 +823,36 @@ class TestReviewFindingRegressions:def test_lane_budget_shrinks_as_the_run_clock_advances(self):"""The clamp was dead code until `elapsed` was threaded through."""+ # Fresh run gets full LANE_DEADLINEassert amazon._remaining_lane_budget(0.0) == amazon.LANE_DEADLINE- mid = amazon._remaining_lane_budget(200.0)- assert 0 < mid < amazon.LANE_DEADLINE+ # Mid-run (100s elapsed) still has 180s leftover (300-100-20=180), clamped to LANE_DEADLINE+ mid = amazon._remaining_lane_budget(100.0)+ assert mid == amazon.LANE_DEADLINE+ # Below floor (300-200-20=80 < MIN_USEFUL_REVIEW_BUDGET=90) → 0+ assert amazon._remaining_lane_budget(200.0) == 0+ # Way past contract → 0assert amazon._remaining_lane_budget(295.0) == 0+ def test_lane_budget_floor_prevents_doomed_pulls(self):+ """Crumb budgets return 0, not the crumbs.++ This is the fix for the Bentgo bug: elapsed=269 left only 11s of budget,+ causing Bright Data pulls to time out (cli_timeout = max(5, timeout-10)+ → 1s timeout). Now any budget below MIN_USEFUL_REVIEW_BUDGET returns 0.+ """+ # elapsed=269 → remaining = 300-269-20 = 11 < MIN_USEFUL=90 → 0+ assert amazon._remaining_lane_budget(269.0) == 0+ # Just above floor: 300-190-20=90 == MIN_USEFUL → 90 (not 0)+ assert amazon._remaining_lane_budget(190.0) == amazon.MIN_USEFUL_REVIEW_BUDGET+ # Just below floor: 300-191-20=89 < MIN_USEFUL=90 → 0+ assert amazon._remaining_lane_budget(191.0) == 0++ def test_lane_budget_constants_are_sane(self):+ """Guard against accidental constant drift breaking the logic."""+ assert amazon.MIN_USEFUL_REVIEW_BUDGET == 90+ assert amazon.LANE_DEADLINE == 180+ assert amazon.MIN_USEFUL_REVIEW_BUDGET < amazon.LANE_DEADLINE+def test_enrichment_refreshes_the_variant_level_rating_count(self):"""Search counts undercount badly; the pull's count is authoritative."""item = _Item("B000000001", name="Deluxe Bag", num_ratings=84)