AI agent skill that researches any topic across Reddit, X, YouTube, HN, Polymarket, and the web - then synthesizes a grounded summary
Python ai-promptsai-skillblueskyclaudeclaude-codeclawhubdeep-researchhackernewsinstagramopenclawpolymarketrecencyredditresearchsocial-mediatiktoktrendstwitterweb-searchyoutube
feat(amazon): buyer-signal source via Bright Data CLI (#988)
Matt Van Horne93c8249d8ba
diff --git a/CONFIGURATION.md b/CONFIGURATION.mdindex 46247652..f4d808d6 100644--- a/CONFIGURATION.md+++ b/CONFIGURATION.md@@ -107,6 +107,10 @@ The project-scoped file is useful for **intentional per-client setups**: drop a**`LAST30DAYS_API_KEY`** + **`LAST30DAYS_API_BASE`** - optional remote-API backend. Set BOTH to route research through a remote API endpoint instead of running the local sources: `LAST30DAYS_API_BASE` is the endpoint (there is no built-in default), and `LAST30DAYS_API_KEY` is the bearer key for it. When both are set (and `--mock` is not passed), the engine submits the topic to that endpoint, polls with progress on stderr, and prints the server's report; none of the per-source keys below are used for that run. A configured local corpus is the privacy exception: the engine bypasses the hosted backend and runs locally rather than forwarding file-derived input. Non-default `--register` selections are forwarded with the request so server-side synthesis uses the same audience preset. Leave either unset to run local sources exactly as normal. Unlike the other keys here, these two are read only from the **process environment** (export them in your shell or host config) - they are deliberately not loaded from the `.env` files above, so a project-scoped `.env` can never silently redirect research to a remote endpoint. The remote endpoint does not return the local `Report` needed for the versioned agent JSON profile; use `--emit=json --json-profile=raw` for its existing server-response JSON contract.+**`BRIGHTDATA_API_KEY`** - optional, for the `amazon` source. The Bright Data CLI normally owns its own auth via `brightdata login`, so this is only needed if you would rather keep an explicit key in `.env` or the keychain. It is resolved through the standard config layering and passed to the CLI through the child process environment, never on the command line (where it would be readable from `/proc/<pid>/cmdline` by other local users on a shared host).++**`LAST30DAYS_AMAZON_DOMAIN`** - optional, default `https://www.amazon.com`. The marketplace the `amazon` source searches; set it to `https://www.amazon.co.uk`, `https://www.amazon.de`, and so on. Product URLs are validated against this host, so records from other marketplaces are rejected.+### Local corpus (your files)Register persistent directories with `LAST30DAYS_CORPUS_DIRS`. Separate paths with `:` on macOS/Linux (the platform path separator is `;` on Windows):@@ -145,6 +149,7 @@ python3 skills/last30days/scripts/last30days.py "MCP servers" \| arXiv | `arxiv-pp-cli` on PATH (auto-installed during first-run setup via `npx -y @mvanhorn/[email protected] install arxiv --cli-only`) | always on if `arxiv-pp-cli` on PATH; fires on research/technical topics and stays quiet otherwise (relevance + 365-day recency gating) | yes (free, keyless) || Techmeme | `techmeme-pp-cli` on PATH (auto-installed via `... install techmeme --cli-only`) | always on if `techmeme-pp-cli` on PATH; searches Techmeme's live archive and keeps only headlines dated within the research window (undated headlines flow through as low-confidence) | yes (free, keyless) || Trustpilot | `trustpilot-pp-cli` on PATH (NOT auto-installed; install on demand via `npx -y @mvanhorn/[email protected] install trustpilot --cli-only`) + (`INCLUDE_SOURCES` contains `trustpilot` **or** an explicit `--trustpilot-domain` / plan-level `trustpilot_domain`) | **opt-in, off by default**; `--trustpilot-domain=<domain>` (and per-entity `trustpilot_domain` in `--competitors-plan`) auto-activates the source for that run and bypasses the brand-shape gate. Persist with `INCLUDE_SOURCES=trustpilot` to skip per-run auto-enable. `EXCLUDE_SOURCES=trustpilot` still wins. Bare company names auto-resolve to the review-page domain via the CLI's search only when the source is already active. The session warms once before the search fan-out; a stale session does a ~10s headless-Chrome WAF-cookie harvest (set `LAST30DAYS_TRUSTPILOT_NO_BROWSER=1` to disable in cron/CI) | yes (no API key; cookie-replay after the one-time harvest) |+| Amazon | `brightdata` CLI on PATH **and logged in** (NOT auto-installed: `npm i -g @brightdata/cli` then `brightdata login`) + (`INCLUDE_SOURCES` contains `amazon` **or** `--search` includes `amazon`) | product records with live rating, rating count, and price, plus a capped sample of recent written reviews woven as buyer voice; the emoji footer shows each product's all-time-vs-last-30-days drift | **opt-in, off by default**. Free tier is 5,000 requests/month (~$7.50 equivalent); a typical run spends 4 (1 product search + up to 3 review pulls) regardless of how many reviews come back, since billing is per request. Past the free tier it bills the account balance at $1.50 per 1,000 records (~$0.32 for a default run). `--amazon-query=<keyword>` sets the product keyword when it differs from the topic; `LAST30DAYS_AMAZON_DOMAIN` selects a non-US marketplace. `EXCLUDE_SOURCES=amazon` wins. Never auto-fires: the model requests it per run or the user enables it durably || X / Twitter | one of: `AUTH_TOKEN` + `CT0` (browser cookies, Bird CLI), `XAI_API_KEY`, `XQUIK_API_KEY`, `SCRAPECREATORS_API_KEY`, or `FROM_BROWSER` (cookie-jar auth) | X items in results | cookie-jar / Bird = free; Xquik / xAI / ScrapeCreators = key-based || TikTok | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `tiktok` | TikTok items | 10K free calls || Instagram | `SCRAPECREATORS_API_KEY` + `INCLUDE_SOURCES` contains `instagram` | Instagram Reels | 10K free calls; raise `LAST30DAYS_TRANSCRIPT_TIMEOUT` (default 30s) if SC is slow on your network |diff --git a/changelog.d/+amazon-buyer-signals.added.md b/changelog.d/+amazon-buyer-signals.added.mdnew file mode 100644index 00000000..b75e08be--- /dev/null+++ b/changelog.d/+amazon-buyer-signals.added.md@@ -0,0 +1,7 @@+**Amazon buyer signals** — a new opt-in `amazon` source, backed by the Bright Data CLI. On shopping-intent topics it pulls discovered products with live ratings and prices, plus a capped sample of recent written reviews woven in as buyer voice.++The signal it exists for is *drift*: an all-time rating from thousands of ratings set against the average of only the reviews inside the last 30 days. When those disagree, something changed this month, and the review text says what. The emoji footer names each product and the direction it moved — `📦 Amazon: 3 products │ Chill Max XL 4.4★→3.8★ ↓, Deluxe Bag 4.7★→5.0★, BLUEY Set 4.8★ new` — rather than reporting inventory counts.++Off by default and dual-gated: the `brightdata` CLI must be on PATH and logged in, *and* the run must ask for the source (`--search ...,amazon` or `INCLUDE_SOURCES=amazon`). It never auto-fires from inferred intent. Use `--amazon-query` when the product keyword differs from the topic — a person topic searches their company's product line, not their name. `LAST30DAYS_AMAZON_DOMAIN` selects a non-US marketplace.++Billing is one credit per request against a 5,000/month free tier, so a typical run costs 4 credits regardless of how many reviews come back.diff --git a/skills/last30days/SKILL.md b/skills/last30days/SKILL.mdindex dc19ed57..0a65e243 100644--- a/skills/last30days/SKILL.md+++ b/skills/last30days/SKILL.md@@ -858,6 +858,7 @@ Before running the engine, determine which flags apply to this topic and resolve| `--github-user={user}` | Step 0.5b | Topic is a person who ships code (developer, engineer, CEO-who-codes, researcher) || `--github-repo={owner/repo}` | Step 0.5c | Topic is a product / project / open-source tool || `--trustpilot-domain={domain}` | Step 0.5d | Topic is a company / brand / service with a Trustpilot presence (passing the flag also auto-activates the opt-in Trustpilot source for this run) |+| `--amazon-query={keyword}` | Step 0.5e | Recent buyer sentiment would materially inform the report AND `brightdata` is on PATH and logged in. Keyword is brand-plus-category (`Weber grill`), and for a person topic it is their company's product line (`June Oven`), not their name. Also add `amazon` to `--search` || `--subreddits={sub1,sub2,...}` | Step 0.55 | Always — almost every topic has active Reddit communities || `--tiktok-hashtags={h1,h2,...}` | Step 0.55 | Always — inferred from topic || `--tiktok-creators={c1,c2,...}` | Step 0.55 | Creator / influencer / brand topics |@@ -1016,6 +1017,36 @@ Store: `RESOLVED_TRUSTPILOT_DOMAIN = {domain or empty}`---+### Step 0.5e: Decide the Amazon Buyer-Signal Lane (if `brightdata` is available)++**Availability first.** This lane exists only when the Bright Data CLI is on PATH and logged in (`--diagnose` reports `brightdata_installed` and `brightdata_authenticated`). If either is false the source does not exist, nothing changes, and you should skip this step entirely — do not mention it, do not suggest installing it mid-run.++**The one question to ask:** *would recent Amazon buyer sentiment materially inform this report?* Not "is this shopping" — the test is whether buyer evidence is real evidence for this topic.++| Topic | Fires? | `--amazon-query` |+|---|---|---|+| "Weber Grills" | Yes — brand topic where review signal is core evidence | `Weber grill` |+| "best bluetooth speaker under $100" | Yes — buying question, the whole point | `bluetooth speaker` |+| "Bentgo Box" | Yes — brand line | `Bentgo lunch box` |+| "Matt Van Horn" (CEO of June) | Yes — **and the keyword is the company's product, not the person** | `June Oven` |+| "Kanye West" | No — person/culture topic, buyer reviews are noise | — |+| "the 2026 election" | No — nothing to buy | — |++**Two mechanics that matter:**++1. **The keyword is yours to choose and is often not the topic.** Map person → company → product line using what you know plus what Step 0.55 surfaced. A "Matt Van Horn" run that searches Amazon for his name returns nothing; searching `June Oven` returns his company's product reviews, which is the actual signal.+2. **Phrase it as brand plus category, never bare brand.** A bare brand keyword lands on Amazon's ad-heavy page 1 and can miss the brand's own bestsellers — a live `Bentgo` search returned 57 competitor ads and missed the flagship, while `Bentgo lunch box` surfaced it. Say `Weber grill`, not `Weber`.++**`--search` is replace-not-add.** Passing `--search` narrows the run to exactly the sources listed, so include the full intended set: `--search reddit,x,youtube,amazon` — never a bare `--search amazon`, which would silently drop every other source.++**Cost and latency, so you can set expectations:** one credit for the product search plus one per review pull, 4 per typical run against a 5,000/month free tier. Review sampling adds roughly 30 seconds to 2 minutes at default depth. Quick depth pulls no reviews at all.++Store: `AMAZON_QUERY = {product keyword or empty}` — pass as `--amazon-query="{AMAZON_QUERY}"` and add `amazon` to `--search`.++**Skip this step if:** the CLI is unavailable, the topic has no consumer-product dimension, or the user set `EXCLUDE_SOURCES=amazon`.++---+## Agent Mode (--agent flag)If `--agent` appears in ARGUMENTS (e.g., `/last30days plaud granola --agent`):@@ -1367,7 +1398,7 @@ Only show lines for platforms where something was resolved. Skip empty lines. On- For how_to: prioritize YouTube (tutorials) and Reddit (guides)- Primary subquery weight = 1.0, secondary = 0.6-0.8, peripheral = 0.3-0.5-**Available sources (include ALL in primary subquery):** reddit, x, youtube, tiktok, instagram, hackernews, polymarket. Optional: bluesky, truthsocial, threads, pinterest, grounding (web search - only if user has Brave/Exa/Serper key), digg (Digg clusters - only if `digg-pp-cli` is on PATH)+**Available sources (include ALL in primary subquery):** reddit, x, youtube, tiktok, instagram, hackernews, polymarket. Optional: bluesky, truthsocial, threads, pinterest, grounding (web search - only if user has Brave/Exa/Serper key), digg (Digg clusters - only if `digg-pp-cli` is on PATH), amazon (buyer reviews - only if `brightdata` is on PATH and logged in; see Step 0.5e)**Intent → freshness_mode mapping:**- breaking_news, prediction → `strict_recent`diff --git a/skills/last30days/scripts/last30days.py b/skills/last30days/scripts/last30days.pyindex 8e1c85b6..f238ac0a 100644--- a/skills/last30days/scripts/last30days.py+++ b/skills/last30days/scripts/last30days.py@@ -732,6 +732,18 @@ def build_parser() -> argparse.ArgumentParser:"Find the domain with `trustpilot-pp-cli search '<name>'`."),)+ parser.add_argument(+ "--amazon-query",+ help=(+ "Product keyword the amazon source searches, when that source is active. "+ "Defaults to the topic. Supply it whenever the topic is not the product: "+ "a person topic searches their company's product line "+ "(--amazon-query='June Oven'), and a brand searches brand-plus-category "+ "(--amazon-query='Weber grill', not 'Weber' -- a bare brand keyword lands "+ "on an ad-heavy page that can miss the brand's own bestsellers). "+ "Requires the brightdata CLI on PATH and logged in."+ ),+ )parser.add_argument("--competitors",nargs="?",@@ -3308,6 +3320,28 @@ def _main(if keywords:config["_polymarket_keywords"] = keywords+ # Product keyword for the amazon source. Carried on config rather than+ # threaded through the run signature (the _polymarket_keywords idiom):+ # it is one optional string consumed in exactly two places.+ if getattr(args, "amazon_query", None):+ config["_amazon_query"] = args.amazon_query.strip()+ # Unlike --trustpilot-domain, this flag deliberately does NOT+ # auto-activate its source: the lane spends metered credits, so+ # turning it on stays an explicit request. But silence is the+ # wrong failure mode -- a model that resolves the keyword and+ # forgets the --search token would otherwise get no signal at+ # all that the flag did nothing.+ _amazon_requested = (+ (requested_sources and "amazon" in requested_sources)+ or "amazon" in str(config.get("INCLUDE_SOURCES") or "").lower()+ )+ if not _amazon_requested:+ sys.stderr.write(+ "[Amazon] --amazon-query was set but the amazon source was not "+ "requested; add it to --search (e.g. --search reddit,x,amazon) "+ "or set INCLUDE_SOURCES=amazon. Ignoring the keyword.\n"+ )+# vs-mode / plan routing: split a vs-topic into main + peers unless# discover-N or an explicit --competitors-list already decided who runs.topic, comp_enabled, comp_count, comp_explicit = apply_vs_competitor_routing(@@ -3428,6 +3462,14 @@ def _main(# leak across sub-runs. Each sub-run writes its own# `_auto_resolve_context` into its local config copy.entity_config = dict(config)+ # The Amazon keyword is entity-SPECIFIC, unlike the depth caps+ # this shallow copy exists to inherit. Leaving the main topic's+ # keyword in place would search Weber SKUs for a Traeger peer,+ # render a rival's products as that peer's buyer evidence, and+ # multiply the metered spend by the number of entities. Drop it+ # so each peer derives its own keyword from its own topic; a+ # per-entity keyword can ride in the --competitors-plan entry.+ entity_config.pop("_amazon_query", None)plan_entry = comp_plan.get(entity.strip().lower(), {})resolved = {"entity": entity,diff --git a/skills/last30days/scripts/lib/amazon.py b/skills/last30days/scripts/lib/amazon.pynew file mode 100644index 00000000..b2f4b265--- /dev/null+++ b/skills/last30days/scripts/lib/amazon.py@@ -0,0 +1,849 @@+"""Amazon product and review signals via the Bright Data CLI.++Two-stage source, following the digg discover-then-enrich shape:++1. **Discovery** -- one ``amazon_product_search`` per run turns a+ model-supplied product keyword into product records carrying live+ aggregate stats (rating, rating count, price). Cheap and fast.+2. **Enrichment** -- ``amazon_product_reviews`` pulls a capped sample of+ written reviews for the top few surviving products, in parallel, under+ a lane deadline. Reviews ride on their product item as metadata+ comments and feed community-voice weaving.++The signature signal is the fusion of those two: an all-time rating from+thousands of ratings, set against the average of just the reviews inside+the last-30-day window. When those disagree, something changed this month,+and the review text says what. No Amazon page shows that.++Metering (R13): one credit per pipeline request regardless of records+returned, so the caps here bound paid-tier *records*, not credits. A+default run is 1 search + up to 3 review pulls = 4 requests.++Field names and quirks below are verified against live payloads pulled+2026-08-13; see the plan's schema block. Three fields arrive doubled+(``review_posted_date``, ``review_header``, ``badge``) and are repaired+here rather than downstream.+"""++from __future__ import annotations++import re+import time+from concurrent.futures import ThreadPoolExecutor, as_completed+from datetime import datetime, timezone+from typing import Any, Dict, Iterable, List, Optional, Sequence, Tuple+from urllib.parse import urlparse++from . import brightdata, log+from .relevance import token_overlap_relevance+++SEARCH_PIPELINE = "amazon_product_search"+REVIEWS_PIPELINE = "amazon_product_reviews"++DEFAULT_DOMAIN = "https://www.amazon.com"++# Reviews requested per pull. Uniform across topic shapes and depths by+# decision: billing is per *request*, not per record, so a bigger cap is+# free on the monthly credit tier, and the in-window sample is what the+# drift signal rests on. Live-verified that latency does not scale with+# this number (50 reviews in 22s vs 20 reviews in 115s on a slower SKU).+#+# It is a ceiling, never a quota -- a SKU with 31 total reviews returns 31.+MAX_REVIEWS = 50++# How many products get a review pull, per depth. Quick spends one credit+# on discovery only: aggregate stats with no recent window.+DEPTH_CONFIG = {+ "quick": 0,+ "default": 3,+ "deep": 5,+}++SEARCH_TIMEOUT = 90+REVIEW_TIMEOUT = 180++# Wall-clock ceiling for the whole parallel review lane. Pulls that miss it+# are abandoned, and their products degrade to `quiet` rather than+# disappearing (a slow SKU is real and unrelated to the cap: one live pull+# took 115s).+LANE_DEADLINE = 180++# The engine's foreground contract. The lane deadline is clamped against+# whatever remains of it, minus room to render.+FOREGROUND_CONTRACT = 300+RENDER_MARGIN = 20++# Minimum dated reviews inside the window before a drift arrow is honest.+# Live census: a 50-cap pull returned 31 records of which only 5 were+# inside 30 days, so an unguarded arrow would routinely publish a "trend"+# computed from one or two reviews.+MIN_DRIFT_SAMPLE = 5++RECENT_WINDOW_DAYS = 30++# Product names run long and pipe-delimited; the footer needs a scannable+# handle, not a title.+SHORT_NAME_MAX = 18++_STAR_FIELDS = (+ ("one_star", 1),+ ("two_star", 2),+ ("three_star", 3),+ ("four_star", 4),+ ("five_star", 5),+)+++def _log(msg: str) -> None:+ log.source_log("Amazon", msg, tty_only=False)+++def _today() -> datetime:+ return datetime.now(timezone.utc)+++# --------------------------------------------------------------- parsing+++def undouble(text: str) -> str:+ """Repair the CLI's doubled string fields.++ Observed live: ``review_header`` arrives as ``"Best Box!Best Box!"`` and+ ``badge`` as ``"Verified Purchase, Verified Purchase"``. Handles the+ exact-repeat case and the comma-joined repeat, and leaves anything else+ untouched -- a genuinely repetitive title must survive intact.+ """+ value = (text or "").strip()+ if not value:+ return ""+ half, odd = divmod(len(value), 2)+ # Only treat an exact repeat as doubling when the halves are substantial+ # and look like a phrase rather than a syllable -- otherwise a real title+ # of "ByeBye" or "NoNo" gets silently truncated to half of itself. The+ # observed artifact doubles whole headlines, so requiring some length and+ # either whitespace or terminal punctuation keeps the repair targeted.+ if not odd and half >= 6 and value[:half] == value[half:]:+ first = value[:half]+ if " " in first or first[-1] in ".!?":+ return first.strip()+ parts = [p.strip() for p in value.split(",")]+ if len(parts) == 2 and parts[0] and parts[0] == parts[1]:+ return parts[0]+ return value+++_DATE_HEAD = re.compile(r"^([A-Z][a-z]+ \d{1,2}, \d{4})")+++def parse_review_date(raw: Any) -> Optional[str]:+ """Pull the ISO date out of the CLI's prose-wrapped date field.++ Live shape: ``"August 3, 2026Reviewed in the United States on August 3,+ 2026"``. Only the leading ``%B %d, %Y`` is trustworthy; the tail is+ localized prose that varies by marketplace.++ Returns ``YYYY-MM-DD`` or None.+ """+ match = _DATE_HEAD.match(str(raw or "").strip())+ if not match:+ return None+ try:+ return datetime.strptime(match.group(1), "%B %d, %Y").date().isoformat()+ except ValueError:+ return None+++def short_name(name: str, brand: str = "") -> str:+ """Derive a scannable footer handle from a long product name.++ Live names are pipe-delimited marketing strings with the brand carried+ in its own field rather than as a prefix ("Chill Max Leak-Proof XL+ Bento-Style Lunch Box | Included Ice Pack Keeps Food Cold"). Take the+ segment before the first delimiter, drop a leading brand token if one+ did sneak in, and clip to a scannable width on a word boundary.+ """+ text = re.split(r"[|(–—]", str(name or ""), maxsplit=1)[0].strip(" -,")+ brand_token = str(brand or "").strip()+ if brand_token:+ # Word-boundary anchored: a bare startswith() eats into sub-brands and+ # coincidental prefixes ("AnkerWork" under brand "Anker" would become+ # "Work", "Chillax" under "Chill" would become "ax").+ stripped = re.sub(+ rf"^{re.escape(brand_token)}\b[\s\-,]*", "", text, count=1, flags=re.IGNORECASE+ )+ if stripped:+ text = stripped.strip(" -,")+ if len(text) <= SHORT_NAME_MAX:+ return text+ clipped = text[:SHORT_NAME_MAX].rsplit(" ", 1)[0].strip(" -,")+ return clipped or text[:SHORT_NAME_MAX].strip()+++def _as_float(value: Any) -> Optional[float]:+ try:+ result = float(value)+ except (TypeError, ValueError):+ return None+ return result+++def _as_int(value: Any) -> int:+ try:+ return int(value)+ except (TypeError, ValueError):+ return 0+++def _is_sponsored(value: Any) -> bool:+ """The flag arrives as the string 'true'/'false', not a bool.++ Recorded in metadata but never used to filter (R4): its distribution+ swings hard with keyword phrasing, so filtering on it can blank the+ lane on an unlucky query.+ """+ if isinstance(value, bool):+ return value+ return str(value or "").strip().lower() == "true"+++def _valid_product_url(url: str, domain: str) -> bool:+ """Accept only https URLs on the configured Amazon host."""+ try:+ parsed = urlparse(url)+ expected = urlparse(domain or DEFAULT_DOMAIN)+ except ValueError:+ return False+ if parsed.scheme != "https" or not parsed.netloc:+ return False+ host = parsed.netloc.lower().removeprefix("www.")+ want = (expected.netloc or "").lower().removeprefix("www.")+ return bool(want) and host == want+++# Amazon ASINs are a fixed shape. Validating it matters because the value+# is interpolated into a URL that is then refetched through the CLI *and*+# rendered as a link in the report -- two sinks, one unvalidated API field.+_ASIN_RE = re.compile(r"^[A-Za-z0-9]{10}$")+++def _valid_asin(asin: str) -> bool:+ return bool(_ASIN_RE.match(asin or ""))+++def canonical_product_url(url: str, asin: str, domain: str) -> str:+ """Strip Amazon's tracking tail down to a stable /dp/<asin> link.++ Search records carry 200+ character URLs with session-scoped ``dib``+ tokens. Those work but are unreadable in a report and unstable across+ runs, which breaks dedupe on re-runs of the same topic.++ Falls back to the (already host-validated) original URL if the ASIN is+ not well-formed, so a malformed record can never shape the rebuilt URL.+ """+ if not _valid_asin(asin):+ return url+ base = (domain or DEFAULT_DOMAIN).rstrip("/")+ return f"{base}/dp/{asin}"+++# ------------------------------------------------------------- discovery+++def search_products(+ keyword: str,+ *,+ domain: str = DEFAULT_DOMAIN,+ config: Optional[Dict[str, Any]] = None,+ timeout: int = SEARCH_TIMEOUT,+) -> Dict[str, Any]:+ """Run one product search. Never raises; returns the adapter envelope."""+ query = (keyword or "").strip()+ if not query:+ return {"records": []}+ # A leading dash would be parsed as a CLI option rather than a search+ # term. The keyword is model-supplied and can be influenced by+ # pre-research over untrusted web content, so reject rather than+ # sanitize -- a keyword starting with '-' is never a real product.+ if query.startswith("-"):+ _log(f"rejecting option-shaped keyword: {query!r}")+ return {"records": [], "error": "amazon keyword may not begin with '-'"}+ _log(f"search '{query}' on {domain}")+ response = brightdata.run_pipeline(+ SEARCH_PIPELINE, [query, domain or DEFAULT_DOMAIN],+ timeout=timeout, config=config,+ )+ if response.get("error"):+ _log(f"search failed: {response['error']}")+ else:+ _log(f"search returned {len(response.get('records') or [])} records")+ return response+++def parse_search_response(+ response: Dict[str, Any],+ keyword: str,+ *,+ domain: str = DEFAULT_DOMAIN,+ min_relevance: float = 0.15,+) -> List[Dict[str, Any]]:+ """Turn raw search records into deduped, relevance-gated product dicts.++ Dedupe is by ASIN: live payloads repeat a single product up to five+ times across the result set (64 unique of 66 records on one pull).+ Relevance is scored against the *supplied keyword*, not the run topic,+ because the model may search "June Oven" on a topic about a person.+ """+ records = response.get("records") if isinstance(response, dict) else None+ if not isinstance(records, list):+ return []++ today = _today().date().isoformat()+ seen: Dict[str, Dict[str, Any]] = {}+ for record in records:+ if not isinstance(record, dict):+ continue+ asin = str(record.get("asin") or "").strip()+ raw_url = str(record.get("url") or "").strip()+ if not _valid_asin(asin) or not _valid_product_url(raw_url, domain):+ continue++ name = str(record.get("name") or "").strip()+ brand = str(record.get("brand") or "").strip()+ if not name:+ continue++ relevance = token_overlap_relevance(keyword, f"{brand} {name}".strip())+ if relevance < min_relevance:+ continue++ num_ratings = _as_int(record.get("num_ratings"))+ existing = seen.get(asin)+ # Duplicates of one ASIN can disagree on rating count (variant-level+ # records); keep the richest.+ if existing and _as_int(existing.get("num_ratings")) >= num_ratings:+ continue++ seen[asin] = {+ "asin": asin,+ # Current-date stamped (KTD6, trustpilot precedent): a live+ # aggregate rating is a fact about now, not about the product's+ # launch date, so it must not be dropped by the 30-day filter.+ "date": today,+ "name": name,+ "short_name": short_name(name, brand),+ "brand": brand,+ "url": canonical_product_url(raw_url, asin, domain),+ "rating": _as_float(record.get("rating")),+ "num_ratings": num_ratings,+ "price": _as_float(record.get("final_price")),+ "currency": str(record.get("currency") or "").strip(),+ "badge": undouble(str(record.get("badge") or "")),+ "sponsored": _is_sponsored(record.get("sponsored")),+ "bought_past_month": _as_int(record.get("bought_past_month")),+ "rank_on_page": _as_int(record.get("rank_on_page")),+ "relevance": relevance,+ }++ products = sorted(+ seen.values(),+ key=lambda p: (p["num_ratings"], p["relevance"]),+ reverse=True,+ )+ _log(f"{len(products)} unique on-keyword products after dedupe")+ return products+++def infer_brand(products: Sequence[Dict[str, Any]], keyword: str) -> str:+ """Detect a brand topic by matching record brands against the keyword.++ This is the guard against paying to review a competitor. Rival brands+ buy ads against a brand keyword and can outrank the brand's own catalog+ on raw rating count: on a live "bentgo lunch box" search a competitor+ held the top two slots and would have taken two of the three review+ pulls, putting a rival's reviews in a Bentgo report.++ Matching the *keyword's own tokens*, rather than picking the most+ common brand in the results, is what keeps category topics unfiltered.+ "best bluetooth speaker" names no brand, so nothing is constrained and+ the top products across brands compete on merit -- which is exactly+ what that topic shape wants.+ """+ normalized_keyword = " ".join(re.findall(r"[a-z0-9]+", (keyword or "").lower()))+ if not normalized_keyword:+ return ""+ keyword_tokens = set(normalized_keyword.split())++ # Keyed by the lowercased brand so one vendor spelled two ways ("Bentgo"+ # and "BENTGO" in the same result set) reads as one candidate. Without+ # this the set has two members, the function bails, and the guard it+ # exists to provide silently turns off.+ candidates: Dict[str, str] = {}+ for product in products:+ brand = str(product.get("brand") or "").strip()+ if not brand:+ continue+ brand_tokens = re.findall(r"[a-z0-9]+", brand.lower())+ if not brand_tokens:+ continue+ # Multi-word brands ("Hydro Flask") can never match a single-token+ # test, so compare the brand's whole token sequence against the+ # keyword's -- otherwise the guard is off for every two-word brand.+ if len(brand_tokens) == 1:+ matched = brand_tokens[0] in keyword_tokens and len(brand_tokens[0]) > 2+ else:+ matched = " ".join(brand_tokens) in normalized_keyword+ if matched:+ # First spelling wins, so the result is deterministic across runs.+ candidates.setdefault(brand.lower(), brand)+ return next(iter(candidates.values())) if len(candidates) == 1 else ""+++def select_enrichment_targets(+ products: Sequence[Dict[str, Any]],+ *,+ limit: int,+ brand: str = "",+ keyword: str = "",+) -> List[Dict[str, Any]]:+ """Pick which products get a review pull.++ Ranked by rating count, which is a coarse signal: search records carry+ variant-level counts that can undercount badly (84 on a record whose+ review pull reported 8,446). The review pull's own+ ``product_rating_count`` is authoritative once available.++ Two filters run before the cut:++ * **Brand**, supplied or inferred from the keyword (see ``infer_brand``).+ The record's own ``brand`` field does the work, which also solves+ accessory contamination outright -- a "grill brush for Weber" carries+ the brush maker's brand, not Weber. A front-anchored name match covers+ the few records where ``brand`` is null.+ * **Variant collapse.** Live results repeat one product across colors+ and sizes under distinct ASINs with near-identical names. Two of those+ would burn two of three pulls on the same product and render as+ duplicate footer entries, so only the best-ranked of each short-name+ group stays eligible.+ """+ if limit <= 0:+ return []+ pool = list(products)++ wanted = (brand or "").strip().lower() or infer_brand(pool, keyword).lower()+ if wanted:+ matched = [+ p for p in pool+ if (p.get("brand") or "").strip().lower() == wanted+ or (not (p.get("brand") or "").strip()+ and str(p.get("name") or "").strip().lower().startswith(wanted))+ ]+ if matched:+ pool = matched++ deduped: List[Dict[str, Any]] = []+ seen_names: set[str] = set()+ for product in pool:+ key = (product.get("short_name") or "").strip().lower()+ if key and key in seen_names:+ continue+ if key:+ seen_names.add(key)+ deduped.append(product)+ return deduped[:limit]+++# ------------------------------------------------------------ enrichment+++def fetch_reviews(+ product_url: str,+ *,+ max_reviews: int = MAX_REVIEWS,+ config: Optional[Dict[str, Any]] = None,+ timeout: int = REVIEW_TIMEOUT,+) -> Dict[str, Any]:+ """Pull a capped review sample for one product. Never raises."""+ if not product_url:+ return {"records": []}+ return brightdata.run_pipeline(+ REVIEWS_PIPELINE, [product_url, str(max_reviews)],+ timeout=timeout, config=config,+ )+++def parse_reviews(response: Dict[str, Any]) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:+ """Split a review payload into comment dicts and product-level stats.++ Product-level fields (``product_rating``, ``product_rating_count``, the+ ``product_rating_object`` star distribution) ride on *every* review+ record, so they are read off the first one.++ Comments are built directly in the shared score/excerpt shape rather+ than routed through ``normalize._remap_comments``, which strips keys it+ does not know -- and rating, date, and verified are exactly the keys+ this source needs to keep. Sorted newest first so the woven sample+ favors recent voices.+ """+ records = response.get("records") if isinstance(response, dict) else None+ if not isinstance(records, list) or not records:+ return [], {}++ first = records[0]+ distribution = first.get("product_rating_object")+ stats: Dict[str, Any] = {+ "product_rating": _as_float(first.get("product_rating")),+ "product_rating_count": _as_int(first.get("product_rating_count")),+ "star_distribution": distribution if isinstance(distribution, dict) else {},+ }++ comments: List[Dict[str, Any]] = []+ for record in records:+ if not isinstance(record, dict):+ continue+ body = str(record.get("review_text") or "").strip()+ header = undouble(str(record.get("review_header") or ""))+ excerpt = body or header+ if not excerpt:+ continue+ comments.append(+ {+ # Shared comment shape: downstream weaving reads score/excerpt.+ "score": _as_int(record.get("helpful_count")),+ "excerpt": excerpt,+ "author": str(record.get("author_name") or "").strip(),+ "rating": _as_int(record.get("rating")),+ "date": parse_review_date(record.get("review_posted_date")),+ "verified": bool(record.get("is_verified")),+ "vine": bool(record.get("is_amazon_vine")),+ "title": header,+ }+ )++ # Newest first; undated records sink rather than disappear (R2a).+ comments.sort(key=lambda c: (c["date"] or "", c["score"]), reverse=True)+ return comments, stats+++def _remaining_lane_budget(elapsed: float) -> int:+ """Clamp the lane deadline against what is left of the run's contract."""+ remaining = FOREGROUND_CONTRACT - elapsed - RENDER_MARGIN+ return int(min(LANE_DEADLINE, max(0, remaining)))+++def enrich_with_reviews(+ products: Sequence[Dict[str, Any]],+ *,+ depth: str = "default",+ config: Optional[Dict[str, Any]] = None,+ elapsed: float = 0.0,+ max_reviews: int = MAX_REVIEWS,+ brand: str = "",+ keyword: str = "",+ fetcher=None,+) -> List[Dict[str, Any]]:+ """Attach review samples to the top products, in parallel, under a deadline.++ Every product is returned either way. A product whose pull is dropped+ by the deadline keeps its search-record stats and simply carries no+ review sample -- it renders as ``quiet`` rather than vanishing, because+ losing a top product entirely is a worse failure than losing its+ recent-window read. The dropped pull has spent its credit regardless.+ """+ enriched = [dict(p) for p in products]+ pull_count = DEPTH_CONFIG.get(depth, DEPTH_CONFIG["default"])+ if pull_count <= 0:+ _log(f"depth={depth}: discovery only, no review pulls")+ return enriched++ budget = _remaining_lane_budget(elapsed)+ if budget <= 0:+ _log("no wall-clock budget left for review pulls; skipping the lane")+ return enriched++ targets = select_enrichment_targets(+ enriched, limit=pull_count, brand=brand, keyword=keyword+ )+ if not targets:+ return enriched++ by_asin = {p["asin"]: p for p in enriched}+ pull = fetcher or (+ lambda url: fetch_reviews(+ url, max_reviews=max_reviews, config=config,+ timeout=min(REVIEW_TIMEOUT, budget),+ )+ )++ _log(f"pulling up to {max_reviews} reviews for {len(targets)} products (budget {budget}s)")+ started = time.monotonic()+ # Not a `with` block on purpose. Every future is already running (one+ # worker per target), so `future.cancel()` can never succeed, and+ # ThreadPoolExecutor's context-manager exit calls shutdown(wait=True) --+ # which would block on the very straggler the deadline just declared+ # dropped, making the deadline advisory rather than real. Shutting down+ # without waiting lets the abandoned thread finish and discard its result+ # in the background while the run proceeds.+ pool = ThreadPoolExecutor(max_workers=max(1, len(targets)))+ try:+ futures = {pool.submit(pull, t["url"]): t["asin"] for t in targets}+ try:+ for future in as_completed(futures, timeout=budget):+ asin = futures[future]+ try:+ response = future.result()+ except Exception as exc: # never let one pull kill siblings+ _log(f"review pull failed for {asin}: {exc}")+ continue+ if response.get("error"):+ _log(f"review pull error for {asin}: {response['error']}")+ continue+ comments, stats = parse_reviews(response)+ product = by_asin.get(asin)+ if product is None:+ continue+ product["top_comments"] = comments+ product.update({k: v for k, v in stats.items() if v})+ except TimeoutError:+ dropped = sum(1 for f in futures if not f.done())+ _log(f"lane deadline {budget}s hit; dropped {dropped} straggling pull(s)")+ finally:+ pool.shutdown(wait=False, cancel_futures=True)++ _log(f"review lane finished in {time.monotonic() - started:.0f}s")+ return enriched+++def enrich_source_items(+ items: List[Any],+ *,+ depth: str = "default",+ config: Optional[Dict[str, Any]] = None,+ keyword: str = "",+ elapsed: float = 0.0,+ max_reviews: int = MAX_REVIEWS,+ fetcher=None,+) -> List[Any]:+ """Attach review samples to the amazon SourceItems that survived dedupe.++ Reads product identity out of ``metadata`` and writes ``top_comments``+ plus the computed stat block back into it, in place. Runs from+ ``pipeline._finalize_items_by_source`` so the review budget is spent on+ the products the brief will actually show, not on the top of the raw+ fanout (the digg enrichment precedent).+ """+ products: List[Dict[str, Any]] = []+ by_asin: Dict[str, Any] = {}+ for item in items:+ if getattr(item, "source", None) != "amazon":+ continue+ metadata = getattr(item, "metadata", None) or {}+ asin = str(metadata.get("asin") or "").strip()+ if not asin or metadata.get("top_comments"):+ continue+ products.append(+ {+ "asin": asin,+ "url": getattr(item, "url", "") or metadata.get("url", ""),+ "name": metadata.get("name") or getattr(item, "title", ""),+ "short_name": metadata.get("short_name") or "",+ "brand": metadata.get("brand") or "",+ "num_ratings": metadata.get("num_ratings") or 0,+ "rating": metadata.get("rating"),+ }+ )+ by_asin[asin] = item++ if not products:+ return items++ enriched = enrich_with_reviews(+ products, depth=depth, config=config, elapsed=elapsed,+ max_reviews=max_reviews, keyword=keyword, fetcher=fetcher,+ )+ for product in enriched:+ item = by_asin.get(product["asin"])+ if item is None:+ continue+ metadata = getattr(item, "metadata", None)+ if metadata is None:+ continue+ if product.get("top_comments"):+ metadata["top_comments"] = product["top_comments"]+ stats = product_stats(product)+ metadata["stats"] = stats+ # The review pull's product_rating_count supersedes the search+ # record's, which is variant-level and can undercount by orders of+ # magnitude (84 on a record whose pull reported 8,446). Normalization+ # ran before enrichment, so refresh the surfaces that already baked+ # the old number in -- otherwise one product shows two different+ # rating counts in the same report.+ for key in ("product_rating", "product_rating_count", "star_distribution"):+ if product.get(key):+ metadata[key] = product[key]+ authoritative = stats.get("ratings_total") or 0+ if authoritative and getattr(item, "engagement", None) is not None:+ item.engagement["ratings"] = authoritative+ metadata["num_ratings"] = authoritative+ _refresh_title(item, stats)+ return items+++def _refresh_title(item: Any, stats: Dict[str, Any]) -> None:+ """Rewrite the trailing "- 4.4/5 (N ratings)" headline after enrichment."""+ title = getattr(item, "title", "") or ""+ rating = stats.get("all_time")+ total = stats.get("ratings_total") or 0+ if not title or rating is None or not total:+ return+ headline = f"{rating}/5 ({total:,} ratings)"+ base = title.rsplit(" - ", 1)[0] if " - " in title else title+ item.title = f"{base} - {headline}"+++# ------------------------------------------------------------------ stats+++def stats_from_item(item: Any, *, today: Optional[datetime] = None) -> Dict[str, Any]:+ """Compute the stat block for a rendered SourceItem.++ Enrichment stores a precomputed block, but mock runs and replayed+ fixtures skip enrichment entirely, so render recomputes from metadata+ when it is absent. Cheap and pure -- all the inputs already live on+ the item.+ """+ metadata = getattr(item, "metadata", None) or {}+ cached = metadata.get("stats")+ if isinstance(cached, dict) and cached:+ return cached+ return product_stats(+ {+ "short_name": metadata.get("short_name") or "",+ "name": metadata.get("name") or getattr(item, "title", ""),+ "url": getattr(item, "url", "") or "",+ "rating": metadata.get("rating"),+ "num_ratings": metadata.get("num_ratings") or 0,+ "product_rating": metadata.get("product_rating"),+ "product_rating_count": metadata.get("product_rating_count") or 0,+ "star_distribution": metadata.get("star_distribution") or {},+ "top_comments": metadata.get("top_comments") or [],+ },+ today=today,+ )+++def footer_entry(stats: Dict[str, Any], *, quote: str = "") -> str:+ """Render one product's segment of the emoji-footer line (R1c).++ Shapes, by drift state::++ Chill Max XL 4.4★→3.8★ ↓ "the lid jams" negative drift (+ quote)+ Deluxe Bag 4.7★→5.0★ positive or flat drift+ Spirit E-325 4.4★ quiet too few in-window reviews+ BLUEY Set new no all-time baseline++ The ``↓`` is asymmetric on purpose: a sagging product is the alarm+ worth catching at a glance, and a healthy one needs no decoration.+ """+ name = stats.get("short_name") or "Product"+ all_time = stats.get("all_time")+ recent = stats.get("recent_avg")+ drift = stats.get("drift")++ if drift == "new" or all_time is None:+ return f"{name} new"+ if drift == "quiet" or recent is None:+ return f"{name} {all_time}★ quiet"++ entry = f"{name} {all_time}★→{recent}★"+ if drift == "down":+ entry += " ↓"+ if quote:+ entry += f' "{quote}"'+ return entry+++def five_star_share(distribution: Dict[str, Any]) -> Optional[float]:+ """Share of ratings that are 5-star, from the star-distribution object."""+ if not isinstance(distribution, dict) or not distribution:+ return None+ total = sum(_as_int(distribution.get(key)) for key, _ in _STAR_FIELDS)+ if total <= 0:+ return None+ return _as_int(distribution.get("five_star")) / total+++def recent_window_stats(+ comments: Iterable[Dict[str, Any]],+ *,+ today: Optional[datetime] = None,+ window_days: int = RECENT_WINDOW_DAYS,+) -> Dict[str, Any]:+ """Average rating and sample size inside the recent window."""+ reference = (today or _today()).date()+ ratings: List[int] = []+ for comment in comments or []:+ iso = comment.get("date")+ if not iso:+ continue+ try:+ posted = datetime.strptime(iso, "%Y-%m-%d").date()+ except (TypeError, ValueError):+ continue+ if 0 <= (reference - posted).days <= window_days:+ rating = _as_int(comment.get("rating"))+ if rating:+ ratings.append(rating)+ if not ratings:+ return {"recent_n": 0, "recent_avg": None}+ return {"recent_n": len(ratings), "recent_avg": sum(ratings) / len(ratings)}+++def product_stats(+ product: Dict[str, Any],+ *,+ today: Optional[datetime] = None,+) -> Dict[str, Any]:+ """Compute the render-facing stat block for one product.++ ``drift`` is one of:+ * ``"new"`` -- no all-time baseline to move away from+ * ``"quiet"`` -- baseline exists but the window has too few dated+ reviews to average honestly (below MIN_DRIFT_SAMPLE)+ * ``"up"`` / ``"down"`` / ``"flat"`` -- a real, sample-backed move++ The engine owns every number here; the model owns the words (R1b).+ """+ # The review pull's rating count supersedes the search record's, which+ # can be variant-level and badly low.+ all_time = product.get("product_rating")+ if all_time is None:+ all_time = product.get("rating")+ ratings_total = product.get("product_rating_count") or product.get("num_ratings") or 0++ window = recent_window_stats(product.get("top_comments") or [], today=today)+ recent_avg = window["recent_avg"]+ recent_n = window["recent_n"]++ if all_time is None:+ drift = "new"+ elif recent_n < MIN_DRIFT_SAMPLE or recent_avg is None:+ drift = "quiet"+ elif round(recent_avg, 1) > round(float(all_time), 1):+ drift = "up"+ elif round(recent_avg, 1) < round(float(all_time), 1):+ drift = "down"+ else:+ drift = "flat"++ return {+ "short_name": product.get("short_name") or short_name(product.get("name", "")),+ "url": product.get("url", ""),+ "all_time": round(float(all_time), 1) if all_time is not None else None,+ "ratings_total": _as_int(ratings_total),+ "five_star_share": five_star_share(product.get("star_distribution") or {}),+ "recent_avg": round(recent_avg, 1) if recent_avg is not None else None,+ "recent_n": recent_n,+ "reviews_pulled": len(product.get("top_comments") or []),+ "drift": drift,+ }diff --git a/skills/last30days/scripts/lib/brightdata.py b/skills/last30days/scripts/lib/brightdata.pynew file mode 100644index 00000000..1bd3772b--- /dev/null+++ b/skills/last30days/scripts/lib/brightdata.py@@ -0,0 +1,290 @@+"""Bright Data CLI adapter for last30days.++Shells out to the ``brightdata`` CLI (``@brightdata/cli``) to run Bright+Data Pipelines. The CLI owns authentication end to end -- ``brightdata+login`` does a gh-style zero-click browser flow and stores credentials in+a platform config directory -- so this module never handles a login, and+never reads credential *contents*: the auth probe is presence-only.++Activation gate: two-way, mirroring the digg CLI-gated precedent but with+an auth dimension the digg source does not have.++1. ``shutil.which("brightdata")`` must resolve on the **agent subprocess+ PATH** (not merely exist on disk -- Hermes/OpenClaw gateways often drop+ ``~/.local/bin``).+2. A credential signal must be present: either ``BRIGHTDATA_API_KEY``+ resolved through the normal config layering, or the CLI's own+ credentials file in the platform config dir.++The second check is deliberately offline. A stale token passes it and+then 401s fast at call time; that path degrades to empty results with the+CLI's own error line preserved in the envelope, which is the AE2 contract.++Metering note (R13): no pricing logic lives here. One pipeline request+costs one credit against the account's monthly free tier regardless of how+many records come back, so caps in the calling adapter bound *records*+(paid-tier cost), not credits. Credit and auth warnings from the CLI are+passed through verbatim rather than interpreted.+"""++from __future__ import annotations++import json+import os+import shutil+import sys+from pathlib import Path+from typing import Any, Dict, List, Optional, Sequence++from . import log, subproc+++CLI_BIN = "brightdata"++# Env var carrying an explicit API key. Registered in env.py so `.env` file+# and keychain users pass the gate the same way process-env users do; when+# it resolves from a non-process-env layer we hand it to the CLI via -k.+API_KEY_ENV = "BRIGHTDATA_API_KEY"++# Credentials filename written by `brightdata login`. Probed for existence+# only -- never opened, parsed, or logged.+_CREDENTIALS_FILENAME = "credentials.json"+_CONFIG_DIRNAME = "brightdata-cli"++# The CLI's own polling timeout sits below our subprocess timeout so the CLI+# exits cleanly with its own error rather than being SIGTERM'd mid-poll. Its+# timeout path throws with zero records (verified in its polling module --+# never partial output), so a timed-out pull is a clean parseable failure.+_CLI_TIMEOUT_MARGIN = 10+++def _log(msg: str) -> None:+ log.source_log("BrightData", msg, tty_only=False)+++def _config_dir() -> Path:+ """Platform config directory the Bright Data CLI stores credentials in.++ Mirrors the CLI's own credentials module: APPDATA on Windows, the+ Application Support tree on macOS, XDG_CONFIG_HOME (or ~/.config) on+ everything else.+ """+ if sys.platform == "win32":+ base = os.environ.get("APPDATA")+ root = Path(base) if base else Path.home() / "AppData" / "Roaming"+ elif sys.platform == "darwin":+ root = Path.home() / "Library" / "Application Support"+ else:+ base = os.environ.get("XDG_CONFIG_HOME")+ root = Path(base) if base else Path.home() / ".config"+ return root / _CONFIG_DIRNAME+++def is_installed() -> bool:+ """True when the brightdata binary resolves on the agent subprocess PATH."""+ return shutil.which(CLI_BIN) is not None+++def _api_key(config: Optional[Dict[str, Any]]) -> str:+ if not config:+ return ""+ return str(config.get(API_KEY_ENV) or "").strip()+++def has_credentials(config: Optional[Dict[str, Any]] = None) -> bool:+ """True when some credential signal exists, without reading any secret.++ Presence-only by design: an explicit API key resolved through config+ layering, or the existence of the CLI's credentials file. The file is+ never opened. This cannot distinguish a live token from an expired one+ -- that is what the fast 401 at call time is for.+ """+ if _api_key(config):+ return True+ try:+ return (_config_dir() / _CREDENTIALS_FILENAME).exists()+ except OSError:+ return False+++def is_available(config: Optional[Dict[str, Any]] = None) -> bool:+ """The full activation gate: binary on PATH *and* a credential signal."""+ return is_installed() and has_credentials(config)+++def gate_status(config: Optional[Dict[str, Any]] = None) -> Dict[str, bool]:+ """Two-field probe for ``pipeline.diagnose`` (bird_installed precedent).++ Network-free, so it is safe on the ``--diagnose`` / doctor path.+ """+ installed = is_installed()+ return {+ "brightdata_installed": installed,+ "brightdata_authenticated": installed and has_credentials(config),+ }+++def _build_args(+ pipeline_type: str,+ params: Sequence[str],+ *,+ cli_timeout: int,+) -> List[str]:+ """Assemble the CLI invocation.++ The API key is deliberately **absent** here -- it travels in the child's+ environment instead (see ``_child_env``). Process arguments are not a+ secret channel: ``/proc/<pid>/cmdline`` is world-readable under the+ default ``hidepid=0``, and a review pull lives for up to 180s, so a key+ on the command line is readable by any other local user and is captured+ verbatim by execve auditing, process accounting, and any monitoring+ agent that snapshots ``ps``. Mirrors the ``bird_x`` cookie-injection+ precedent.++ Positional params are fenced behind ``--`` so a keyword that happens to+ begin with a dash is parsed as a search term rather than as an option.+ """+ return [+ CLI_BIN,+ "pipelines",+ pipeline_type,+ "--json",+ "--timeout",+ str(cli_timeout),+ "--",+ *(str(p) for p in params),+ ]+++def _child_env(api_key: str) -> Optional[Dict[str, str]]:+ """Environment for the child process, carrying the key when we have one.++ Returns None when there is nothing to inject, so the child simply+ inherits the parent environment (the common case: the CLI owns its own+ credentials file, or the key is already exported).+ """+ if not api_key:+ return None+ return {**os.environ, API_KEY_ENV: api_key}+++def _scrub(text: str, secret: str) -> str:+ """Remove a secret from text before it is logged or returned.++ Defense in depth for the passthrough paths: the stderr lines this+ module deliberately surfaces are auth and quota failures, which are+ exactly the messages a CLI is most likely to echo the rejected+ credential back in.+ """+ if not secret or not text:+ return text+ return text.replace(secret, "***")+++def _extract_records(payload: Any) -> List[Dict[str, Any]]:+ """Pull the record list out of a parsed CLI payload.++ Verified live (2026-08-13): both amazon pipelines return a **bare JSON+ array** of flat record dicts, not the ``{"results": [...]}`` envelope the+ digg CLI uses. The dict branches below are defensive against CLI churn,+ which is a live risk on a package this young.+ """+ if isinstance(payload, list):+ return [r for r in payload if isinstance(r, dict)]+ if isinstance(payload, dict):+ for key in ("records", "results", "data"):+ value = payload.get(key)+ if isinstance(value, list):+ return [r for r in value if isinstance(r, dict)]+ return []+++def run_pipeline(+ pipeline_type: str,+ params: Sequence[str],+ *,+ timeout: int,+ config: Optional[Dict[str, Any]] = None,+) -> Dict[str, Any]:+ """Run one Bright Data pipeline and return ``{"records", "error"}``.++ Never raises. Every failure mode -- missing binary, spawn failure,+ subprocess timeout, non-zero exit, unparseable stdout -- returns empty+ records plus a one-line ``error`` string, so callers can record the+ failure in ``errors_by_source`` without branching on exception types.++ The CLI's first stderr line is preserved verbatim as the error (auth+ 401s and low-credit warnings are the cases that matter), and also+ mirrored to ``source_log`` so the failure is visible in non-TTY hosts.++ Args:+ pipeline_type: pipeline name, e.g. ``amazon_product_search``.+ params: positional pipeline params, passed through in order.+ timeout: subprocess timeout in seconds. The CLI's own polling+ timeout is set just below this so it can fail cleanly first.+ config: resolved config dict, consulted only for the API key.++ Returns:+ ``{"records": [...]}`` on success, else ``{"records": [], "error": str}``.+ """+ if not is_installed():+ return {"records": [], "error": f"{CLI_BIN} not on PATH"}++ cli_timeout = max(5, int(timeout) - _CLI_TIMEOUT_MARGIN)+ key = _api_key(config)+ cmd = _build_args(pipeline_type, params, cli_timeout=cli_timeout)++ try:+ result = subproc.run_with_timeout(cmd, timeout=timeout, env=_child_env(key))+ except subproc.SubprocTimeout as exc:+ _log(f"Timeout: {exc}")+ return {"records": [], "error": str(exc)}+ except FileNotFoundError as exc:+ _log(f"Binary missing: {exc}")+ return {"records": [], "error": str(exc)}+ except OSError as exc:+ _log(f"Spawn failed: {exc}")+ return {"records": [], "error": str(exc)}++ stderr = _scrub(result.stderr or "", key)+ _passthrough_warnings(stderr)++ if result.returncode != 0:+ lines = [ln.strip() for ln in stderr.strip().splitlines() if ln.strip()]+ # The CLI narrates polling progress on stderr, so the *last* line is+ # the actual failure; the first line is "Triggering pipeline...".+ first = lines[-1] if lines else f"exit {result.returncode}"+ _log(f"CLI exit {result.returncode}: {first}")+ return {"records": [], "error": first}++ stdout = result.stdout or ""+ if not stdout.strip():+ return {"records": []}+ try:+ payload = json.loads(stdout)+ except json.JSONDecodeError as exc:+ _log(f"JSON decode failed: {exc}")+ return {"records": [], "error": f"json decode: {exc}"}++ return {"records": _extract_records(payload)}+++# Substrings that mark a stderr line worth surfacing even on a successful+# run -- credit exhaustion and auth trouble are the two the user must see.+# Matched case-insensitively against the CLI's own wording, and echoed+# verbatim rather than reworded (R13: no pricing logic, no interpretation).+_WARNING_MARKERS = ("credit", "quota", "balance", "unauthor", "401", "expired", "login")+++def _passthrough_warnings(stderr: str) -> None:+ """Echo credit/auth warning lines from the CLI verbatim.++ Skips the routine polling narration so a normal run stays quiet.+ """+ for line in (stderr or "").splitlines():+ text = line.strip()+ if not text or text.lower().startswith(("status:", "triggering", "triggered", "data received")):+ continue+ lowered = text.lower()+ if any(marker in lowered for marker in _WARNING_MARKERS):+ _log(text)diff --git a/skills/last30days/scripts/lib/doctor.py b/skills/last30days/scripts/lib/doctor.pyindex 49d71f39..bd94161f 100644--- a/skills/last30days/scripts/lib/doctor.py+++ b/skills/last30days/scripts/lib/doctor.py@@ -53,7 +53,7 @@ import urllib.requestfrom pathlib import Pathfrom typing import Any, Callable, Dict, List, Optional-from . import backends, env, health, http, prescriptions+from . import backends, brightdata, env, health, http, prescriptionsfrom .backends import TIER_ERROR, TIER_OK, TIER_WARN# Rollup tiers (R1). ok/warn/error are U2's; only "off" is doctor's own.@@ -157,6 +157,7 @@ SOURCE_ORDER = ("techmeme","arxiv","trustpilot",+ "amazon","tiktok","instagram","threads",@@ -181,6 +182,9 @@ CLI_DEPENDENCIES = {"techmeme": "techmeme-pp-cli","arxiv": "arxiv-pp-cli","trustpilot": "trustpilot-pp-cli",+ # The only entry that also needs auth; _amazon_record reports the+ # installed-but-unauthenticated state the shared CLI helper cannot.+ "amazon": "brightdata","github": "gh",}_OPTIONAL_CLI_SOURCES = frozenset({"github"})@@ -540,6 +544,39 @@ def _trustpilot_record(config):return _cli_gated_record(config, "trustpilot-pp-cli", "trustpilot")+def _amazon_record(config):+ """Amazon buyer signals: CLI-gated *and* auth-gated.++ Unlike the other CLI-gated sources, a present binary is not enough --+ the Bright Data CLI owns its own login, so a user can have `brightdata`+ on PATH and still get nothing. Report those states separately: an+ unauthenticated install is configured-but-broken (a real fix exists and+ the user wants to hear it), while a missing binary is just an optional+ source nobody opted into.+ """+ probe = health.probe_dependency(brightdata.CLI_BIN)+ requires = f"{brightdata.CLI_BIN} on the agent-subprocess PATH, logged in"+ if probe.ok:+ if brightdata.has_credentials(config):+ return _record(status=health.OK, detail=probe.detail, requires=requires)+ return _record(+ status="unconfigured",+ fix="run `brightdata login` to activate the amazon source",+ detail="brightdata is installed but has no credentials",+ requires=requires,+ )+ entry = prescriptions.for_dependency_probe(probe)+ fix = _fix_text(entry) if entry else probe.prescription+ if probe.status == health.MISSING and not probe.off_path:+ return _record(+ status="opt-in",+ fix="npm i -g @brightdata/cli && brightdata login",+ detail=probe.detail,+ requires=requires,+ )+ return _record(status=probe.status, fix=fix, detail=probe.detail, requires=requires)++def _tiktok_record(config):return _sc_gated_record(config, "tiktok")@@ -716,6 +753,7 @@ _SOURCE_BUILDERS: Dict[str, Callable[[Dict[str, Any]], Dict[str, Any]]] = {"techmeme": _techmeme_record,"arxiv": _arxiv_record,"trustpilot": _trustpilot_record,+ "amazon": _amazon_record,"tiktok": _tiktok_record,"instagram": _instagram_record,"threads": _threads_record,diff --git a/skills/last30days/scripts/lib/env.py b/skills/last30days/scripts/lib/env.pyindex 78da3a1e..651b6de8 100644--- a/skills/last30days/scripts/lib/env.py+++ b/skills/last30days/scripts/lib/env.py@@ -60,7 +60,7 @@ KEYCHAIN_KEYS = ("AUTH_TOKEN", "CT0", "BSKY_HANDLE", "BSKY_APP_PASSWORD","TRUTHSOCIAL_TOKEN", "BRAVE_API_KEY", "EXA_API_KEY", "SERPER_API_KEY","OPENROUTER_API_KEY", "PERPLEXITY_API_KEY", "PARALLEL_API_KEY", "XQUIK_API_KEY",- "XIAOHONGSHU_API_BASE", "GITHUB_TOKEN",+ "XIAOHONGSHU_API_BASE", "GITHUB_TOKEN", "BRIGHTDATA_API_KEY",)# pass(1) integration: Linux/Unix analog of the Keychain source. Each key in@@ -530,6 +530,14 @@ def get_config(policy: ConfigLoadPolicy | None = None) -> dict[str, Any]:('LAST30DAYS_PERPLEXITY_DEEP_TIMEOUT_SECONDS', '600'),('PARALLEL_API_KEY', None),('XQUIK_API_KEY', None),+ # Bright Data CLI. Optional: the CLI normally owns its own auth via+ # `brightdata login`, so this only matters for users who prefer an+ # explicit key in a `.env` file or the keychain. Registered here so+ # those layers reach the gate and the subprocess (-k) alike.+ ('BRIGHTDATA_API_KEY', None),+ # Amazon marketplace the amazon source searches. Non-US users point+ # this at their own storefront (e.g. https://www.amazon.co.uk).+ ('LAST30DAYS_AMAZON_DOMAIN', 'https://www.amazon.com'),# Host-native search signal: set by the SKILL.md agent-host path when the# invoking runtime has its own (better) web-search tool, so the engine's# keyless search floor stays off there. Defaults unset -> floor allowed.diff --git a/skills/last30days/scripts/lib/normalize.py b/skills/last30days/scripts/lib/normalize.pyindex cc3eacad..ed3e0544 100644--- a/skills/last30days/scripts/lib/normalize.py+++ b/skills/last30days/scripts/lib/normalize.py@@ -65,6 +65,7 @@ def normalize_source_items("arxiv": _normalize_arxiv,"techmeme": _normalize_techmeme,"trustpilot": _normalize_trustpilot,+ "amazon": _normalize_amazon,"grounding": _normalize_grounding,"xiaohongshu": _normalize_grounding,"github": _normalize_github,@@ -736,6 +737,87 @@ def _normalize_trustpilot()+def _normalize_amazon(+ source: str,+ item: dict[str, Any],+ index: int,+ from_date: str,+ to_date: str,+) -> schema.SourceItem:+ """Normalizer for Amazon product-and-review signals.++ One item per product. The aggregate rating is current-state evidence, so+ the item is stamped with today's date on the Trustpilot precedent -- a+ live 4.4-star average is a fact about now, not about whenever the+ product launched.++ Reviews arrive already in the shared score/excerpt comment shape (built+ in the amazon adapter, deliberately not routed through _remap_comments,+ which would strip the rating/date/verified keys this source needs), so+ they pass straight through to metadata.+ """+ name = str(item.get("name") or "").strip()+ brand = str(item.get("brand") or "").strip()+ top_comments = item.get("top_comments") or []+ comment_text = _join_comment_excerpts(top_comments, "excerpt")+ rating = item.get("product_rating") if item.get("product_rating") is not None else item.get("rating")+ ratings_total = item.get("product_rating_count") or item.get("num_ratings") or 0++ headline = " ".join(+ part for part in [+ f"{rating}/5" if rating is not None else "",+ f"({ratings_total:,} ratings)" if ratings_total else "",+ ] if part+ )+ # The brand rides in its own field and is usually absent from the name,+ # so prepend it -- unless the name already leads with it, which would+ # otherwise read "Weber Weber Spirit E-325".+ if brand and not name.lower().startswith(brand.lower()):+ product_label = f"{brand} {name}".strip()+ else:+ product_label = name or brand+ title = " - ".join(part for part in [product_label, headline] if part)+ body = "\n".join(part for part in [title, comment_text] if part)++ return _source_item(+ item_id=str(item.get("asin") or f"AMZ{index + 1}"),+ source=source,+ title=title or f"Amazon product {index + 1}",+ body=body,+ url=str(item.get("url") or ""),+ author=brand or None,+ container="Amazon",+ published_at=item.get("date"),+ date_confidence=_date_confidence(item, from_date, to_date, default="low"),+ engagement=item.get("engagement") or {"ratings": ratings_total},+ relevance_hint=item.get("relevance", 0.6),+ why_relevant=str(item.get("why_relevant") or ""),+ snippet=comment_text[:400],+ metadata={+ "asin": str(item.get("asin") or ""),+ "name": name,+ "short_name": item.get("short_name") or "",+ "brand": brand,+ "rating": item.get("rating"),+ "num_ratings": item.get("num_ratings") or 0,+ "price": item.get("price"),+ "currency": item.get("currency") or "",+ "badge": item.get("badge") or "",+ # Recorded, never used as a filter: the flag's distribution+ # swings with keyword phrasing, so filtering can blank the lane.+ "sponsored": bool(item.get("sponsored")),+ "top_comments": top_comments,+ "product_rating": item.get("product_rating"),+ "product_rating_count": item.get("product_rating_count") or 0,+ "star_distribution": item.get("star_distribution") or {},+ # Relevant by construction: the adapter already gated products+ # against the model-supplied keyword, and review text rarely+ # names the product (KTD8).+ "grounding_exempt": True,+ },+ )++def _normalize_polymarket(source: str,item: dict[str, Any],diff --git a/skills/last30days/scripts/lib/pipeline.py b/skills/last30days/scripts/lib/pipeline.pyindex 41f76148..db191369 100644--- a/skills/last30days/scripts/lib/pipeline.py+++ b/skills/last30days/scripts/lib/pipeline.py@@ -19,9 +19,11 @@ from shutil import whichfrom typing import Anyfrom . import (+ amazon,arxiv,bird_x,bluesky,+ brightdata,corpus,dates,dedupe,@@ -96,7 +98,12 @@ SEARCH_ALIAS = {# trustpilot is capped at 1: every subquery would use the identical company# identifier, so N streams are pure redundancy -- and each extra stream risks# its own WAF-cookie Chrome harvest.-MAX_SOURCE_FETCHES: dict[str, int] = {"x": 2, "jobs": 1, "linkedin": 1, "stocktwits": 1, "trustpilot": 1}+# amazon is capped at 1 for the same reason as trustpilot: the model supplies+# one product keyword for the run, so every subquery would issue the identical+# product search. Extra streams would be pure redundancy at one credit each.+MAX_SOURCE_FETCHES: dict[str, int] = {+ "x": 2, "jobs": 1, "linkedin": 1, "stocktwits": 1, "trustpilot": 1, "amazon": 1,+}def _resolve_depth_settings(depth: str, config: dict[str, Any]) -> dict[str, int]:@@ -152,6 +159,7 @@ MOCK_AVAILABLE_SOURCES = ["arxiv","techmeme","trustpilot",+ "amazon","jobs","linkedin","corpus",@@ -275,6 +283,16 @@ def available_sources("trustpilot" in include_sources or (requested_sources and "trustpilot" in requested_sources)):available.append("trustpilot")+ # Amazon: opt-in additive source, dual-gated. The Bright Data CLI must be+ # on the agent subprocess PATH and carry a credential signal, AND the run+ # must ask for it -- the model per-run via --search, or the user durably+ # via INCLUDE_SOURCES=amazon. Never inferred from topic shape: the engine+ # misroutes most shopping phrasings, and auto-firing would spend a CLI+ # owner's credits on runs that have nothing to do with products.+ if brightdata.is_available(config) and (+ "amazon" in include_sources or (requested_sources and "amazon" in requested_sources)+ ):+ available.append("amazon")if ("xiaohongshu" in include_sourcesor (requested_sources and "xiaohongshu" in requested_sources)@@ -1735,8 +1753,13 @@ def diagnose("arxiv-pp-cli": bool(which("arxiv-pp-cli")),"techmeme-pp-cli": bool(which("techmeme-pp-cli")),"trustpilot-pp-cli": bool(which("trustpilot-pp-cli")),+ "brightdata": bool(which("brightdata")),"gh": bool(which("gh")),}+ # Network-free two-field probe (bird_installed/bird_authenticated+ # precedent): "installed" is PATH resolution, "authenticated" is a+ # presence-only credential signal that never reads the secret.+ brightdata_status = brightdata.gate_status(config)credential_destinations = {"global_env": str(env.CONFIG_FILE) if env.CONFIG_FILE else None,}@@ -1768,6 +1791,8 @@ def diagnose("native_search": env.is_native_search(config),"has_scrapecreators": bool(config.get("SCRAPECREATORS_API_KEY")),"has_github": bool(config.get("GITHUB_TOKEN") or which("gh")),+ "brightdata_installed": brightdata_status["brightdata_installed"],+ "brightdata_authenticated": brightdata_status["brightdata_authenticated"],# safe=True (doctor/--diagnose/--preflight) must stay network-free:# answer X availability from local evidence only. x_pending is# precomputed by diagnose() to avoid double evaluation.@@ -1918,6 +1943,11 @@ def run(youtube_yt.reset_search_cache()settings = _resolve_depth_settings(depth, config)requested_sources = normalize_requested_sources(requested_sources)+ # Wall-clock origin for budget-aware enrichment lanes. Retrieval can eat+ # most of the foreground contract before enrichment starts, so a lane that+ # sizes its deadline against a constant alone can push a run past the+ # contract; the amazon review lane clamps against what is actually left.+ run_started = time.monotonic()from_date, to_date = dates.get_date_range(lookback_days, as_of_date=as_of_date)resolved_corpus_dirs = corpus.resolve_directories(corpus_dirs or config.get("_CORPUS_DIRS"),@@ -2078,6 +2108,14 @@ def run(# the LAW 7 degraded-run and Step 0.55 pre-research banners do not apply -# they would contradict the documented jobs-scoped flow. Suppress them.bundle.artifacts["hiring_signals_mode"] = hiring_signals_mode+ # Record the resolved Amazon keyword whenever the lane is active, so the+ # footer can name it on an empty result. A search that matched nothing+ # still spent a credit, and the fix is almost always the keyword -- a+ # suppressed line means nobody ever learns it was wrong.+ if "amazon" in (available or []):+ bundle.artifacts["amazon_query"] = (+ str(config.get("_amazon_query") or "").strip() or topic+ )# Project-mode or person-mode GitHub: run once before the main subquery loop_github_custom_done = False@@ -2385,6 +2423,7 @@ def run(items_by_source = _finalize_items_by_source(bundle.items_by_source, topic=topic, config=config, depth=depth, mock=mock,+ elapsed=time.monotonic() - run_started,)source_status = _finalize_source_status(bundle.source_status, items_by_source)candidates = weighted_rrf(bundle.items_by_source_and_query, plan, pool_limit=settings["pool_limit"])@@ -2726,6 +2765,7 @@ def _finalize_items_by_source(config: dict | None = None,depth: str = "default",mock: bool = False,+ elapsed: float = 0.0,) -> dict[str, list[schema.SourceItem]]:finalized = {}for source, items in items_by_source_raw.items():@@ -2779,6 +2819,25 @@ def _finalize_items_by_source(else:digg.enrich_source_items(items, top_k=3)http.fixture_source_record(enrichment_request, schema.to_dict(items))+ if source == "amazon" and items and not mock:+ # Same budget-at-the-survivors principle: review pulls are the+ # expensive half of this lane (one credit each, and the slow+ # half by wall clock), so they go to the products dedupe kept.+ matched, replayed = http.fixture_source_replay(enrichment_request)+ if matched:+ items = _merge_replayed_enrichment(items, replayed)+ else:+ amazon.enrich_source_items(+ items,+ depth=depth,+ config=config,+ keyword=str((config or {}).get("_amazon_query") or "").strip() or topic,+ # Retrieval time already spent, so the lane sizes its+ # deadline against what remains of the 300s contract+ # rather than always claiming its full constant.+ elapsed=elapsed,+ )+ http.fixture_source_record(enrichment_request, schema.to_dict(items))finalized[source] = itemsreturn finalized@@ -3912,6 +3971,22 @@ def _retrieve_stream_impl(trustpilot.parse_trustpilot_response(result, query=relevance_topic),_result_outcome_artifact(source, result),)+ if source == "amazon":+ # The search keyword is model-supplied and may differ from the topic+ # ("Matt Van Horn" searches "June Oven"), so it keys off the stable+ # research topic rather than the narrowed per-subquery search_query.+ # Review enrichment is deferred to _finalize_items_by_source so the+ # credit budget lands on products that survive dedupe.+ keyword = (+ str((config or {}).get("_amazon_query") or "").strip()+ or raw_topic or topic or subquery.search_query+ )+ domain = str((config or {}).get("LAST30DAYS_AMAZON_DOMAIN") or amazon.DEFAULT_DOMAIN)+ result = amazon.search_products(keyword, domain=domain, config=config)+ return (+ amazon.parse_search_response(result, keyword, domain=domain),+ _result_outcome_artifact(source, result),+ )if source == "bluesky":result = bluesky.search_bluesky(subquery.search_query, from_date, to_date, depth=depth, config=config)return bluesky.parse_bluesky_response(result), _result_outcome_artifact(source, result)@@ -4119,6 +4194,110 @@ def _mock_stream_results(source: str, subquery: schema.SubQuery) -> tuple[list[d"why_relevant": "Mock Trustpilot sentiment",},],+ # Three products spanning the drift states the footer renders: one+ # sagging below its all-time average (with enough in-window reviews+ # to clear the arrow threshold), one steady, and one too new to have+ # a baseline. Mock runs exercise the full R1c line without a CLI.+ "amazon": [+ {+ "asin": "B0MOCK00X1",+ "date": dates.get_date_range(1)[1],+ "name": f"{subquery.search_query} Pro Model | Flagship Edition",+ "short_name": "Pro Model",+ "brand": subquery.search_query.split()[0].title() if subquery.search_query else "Example",+ "url": "https://www.amazon.com/dp/B0MOCK00X1",+ "rating": 4.4,+ "num_ratings": 459,+ "price": 39.99,+ "currency": "USD",+ "badge": "Best Seller",+ "sponsored": False,+ "relevance": 0.85,+ "why_relevant": "Mock Amazon product",+ "product_rating": 4.4,+ "product_rating_count": 459,+ "star_distribution": {+ "one_star": 28, "two_star": 9, "three_star": 28,+ "four_star": 60, "five_star": 335,+ },+ "top_comments": [+ {+ "score": 3, "rating": 2, "verified": True,+ "date": dates.get_date_range(3)[1],+ "excerpt": "The tray shifts in transit and the lid jams shut.",+ "title": "Lid jams",+ },+ {+ "score": 1, "rating": 4, "verified": True,+ "date": dates.get_date_range(9)[1],+ "excerpt": "Solid build, but arrived with a dented panel.",+ "title": "Shipping dent",+ },+ {+ "score": 0, "rating": 5, "verified": True,+ "date": dates.get_date_range(14)[1],+ "excerpt": "Keeps everything cold through a full school day.",+ "title": "Works great",+ },+ {+ "score": 0, "rating": 4, "verified": True,+ "date": dates.get_date_range(19)[1],+ "excerpt": "Good size for the price.",+ "title": "Good value",+ },+ {+ "score": 0, "rating": 4, "verified": False,+ "date": dates.get_date_range(24)[1],+ "excerpt": "Does the job, nothing fancy.",+ "title": "Fine",+ },+ ],+ },+ {+ "asin": "B0MOCK00X2",+ "date": dates.get_date_range(1)[1],+ "name": f"{subquery.search_query} Classic | Everyday Model",+ "short_name": "Classic",+ "brand": subquery.search_query.split()[0].title() if subquery.search_query else "Example",+ "url": "https://www.amazon.com/dp/B0MOCK00X2",+ "rating": 4.7,+ "num_ratings": 8446,+ "price": 24.99,+ "currency": "USD",+ "sponsored": False,+ "relevance": 0.8,+ "why_relevant": "Mock Amazon product",+ "product_rating": 4.7,+ "product_rating_count": 8446,+ "star_distribution": {+ "one_star": 120, "two_star": 90, "three_star": 300,+ "four_star": 1010, "five_star": 6926,+ },+ "top_comments": [+ {+ "score": 12, "rating": 5, "verified": True,+ "date": dates.get_date_range(4)[1],+ "excerpt": "Third one we've bought. They last.",+ "title": "Repeat buyer",+ },+ ],+ },+ {+ "asin": "B0MOCK00X3",+ "date": dates.get_date_range(1)[1],+ "name": f"{subquery.search_query} Mini | New Release",+ "short_name": "Mini",+ "brand": subquery.search_query.split()[0].title() if subquery.search_query else "Example",+ "url": "https://www.amazon.com/dp/B0MOCK00X3",+ "rating": None,+ "num_ratings": 57,+ "price": 19.99,+ "currency": "USD",+ "sponsored": False,+ "relevance": 0.72,+ "why_relevant": "Mock Amazon product",+ },+ ],"jobs": [{"id": "J1",diff --git a/skills/last30days/scripts/lib/planner.py b/skills/last30days/scripts/lib/planner.pyindex 0137c3df..d74653a8 100644--- a/skills/last30days/scripts/lib/planner.py+++ b/skills/last30days/scripts/lib/planner.py@@ -150,6 +150,7 @@ SOURCE_CAPABILITIES = {"arxiv": {"reference", "analysis", "link"},"techmeme": {"discussion", "link", "reference"},"trustpilot": {"reference", "company_signal", "social"},+ "amazon": {"reference", "company_signal", "product_signal"},"xiaohongshu": {"video", "video_shortform", "social"},"github": {"discussion", "link"},"grounding": {"web", "reference", "link"},diff --git a/skills/last30days/scripts/lib/render.py b/skills/last30days/scripts/lib/render.pyindex 4184b152..2df23fab 100644--- a/skills/last30days/scripts/lib/render.py+++ b/skills/last30days/scripts/lib/render.py@@ -10,6 +10,7 @@ from datetime import datefrom urllib.parse import urlparsefrom . import (+ amazon,dates,health,hiring_signals,@@ -226,6 +227,7 @@ SOURCE_LABELS = {"arxiv": "arXiv","techmeme": "Techmeme","trustpilot": "Trustpilot",+ "amazon": "Amazon","perplexity": "Perplexity","jobs": "Jobs","corpus": "Your files",@@ -1795,6 +1797,18 @@ def render_full(report: schema.Report, save_path: str | None = None) -> str:"perplexity","jobs",]+ # The list above fixes the display order for the sources it names, but it+ # is not the source registry and drifts every time one is added: amazon,+ # arxiv, techmeme, trustpilot, linkedin, and dripstack were all silently+ # absent from this dump while appearing normally in the ranked section+ # above, so the saved artifact -- the copy users keep -- was missing+ # evidence the run actually collected. Append whatever else the report+ # carries, sorted for determinism, so a new source is visible here the+ # day it lands rather than the day someone notices.+ source_order += sorted(+ source for source in evidence_report.items_by_source+ if source not in source_order+ )for source in source_order:items = evidence_report.items_by_source.get(source, [])if not items:@@ -1926,10 +1940,21 @@ def _format_item_engagement(item: schema.SourceItem) -> str:"digg_count","share_count","num_comments",+ "ratings",]:val = eng.get(key)if val is not None and val != 0:parts.append(f"{val} {key}")+ # Same drift as the source list above: this allowlist silently blanks the+ # engagement of any source whose metric is not on it (trustpilot's+ # `reviews`/`trustScore` today), so the item renders an empty `[]` in the+ # saved dump. Fall through only when nothing matched, which fixes the+ # blank case without adding previously-unshown keys to sources that+ # already render fine.+ if not parts:+ for key, val in sorted(eng.items()):+ if val not in (None, 0, ""):+ parts.append(f"{val} {key}")return ", ".join(parts) if parts else ""@@ -2793,6 +2818,10 @@ def _build_source_footer_lines(report: schema.Report) -> list[str]:line += f" │ ⚠ {_format_outcome(outcome)}"out.append(line)+ amazon_line = _amazon_footer_line(report)+ if amazon_line:+ out.append(amazon_line)+# Web (sources from grounding)web_items = report.items_by_source.get("grounding") or []if web_items:@@ -2818,6 +2847,100 @@ def _build_source_footer_lines(report: schema.Report) -> list[str]:return out+def _amazon_footer_line(report: schema.Report) -> str | None:+ """Build the 📦 Amazon emoji-footer line (R1c).++ Follows the Polymarket shape -- unit count, then *named products+ carrying their own numbers* -- rather than the three-count inventory+ shape every social source uses. ``3 products │ 611 ratings │ 56+ reviews`` fits the box and says nothing; a named product with the+ direction its rating moved this month is the whole reason the source+ exists.++ Three renderings:++ * **Default/deep** -- per-product drift entries (see+ ``amazon.footer_entry``).+ * **Quick depth** -- no review pulls means no recent window, so the+ inventory form is the honest one here and only here. Never render a+ ``→`` against a null window.+ * **Empty search** -- name the keyword rather than suppressing the+ line. Observability, not spend: an empty result means the model's+ keyword or its relevance judgment was wrong, and a hidden line means+ nobody ever finds out.+ """+ items = report.items_by_source.get("amazon") or []+ keyword = str((report.artifacts or {}).get("amazon_query") or "").strip()+ outcome = report.source_status.get("amazon")+ failed = bool(outcome and outcome.state != health.OK)++ if not items:+ if not keyword:+ return None+ # An empty result is only a keyword problem when the search actually+ # ran and came back empty. On an expired token or a CLI failure,+ # "no products matched" sends the user to fix the wrong thing --+ # so lead with the real outcome, same as every other footer branch.+ if failed:+ return f'📦 Amazon: no results for "{keyword}" │ ⚠ {_format_outcome(outcome)}'+ return f'📦 Amazon: no products matched "{keyword}"'++ count = len(items)+ plural = "products" if count != 1 else "product"+ all_stats = [amazon.stats_from_item(item) for item in items]++ # Quick depth pulls no reviews at all, so nothing has a recent window.+ if not any(s.get("reviews_pulled") for s in all_stats):+ rated = [s["all_time"] for s in all_stats if s.get("all_time") is not None]+ total_ratings = sum(s.get("ratings_total") or 0 for s in all_stats)+ parts = [f"{count} {plural}"]+ if rated:+ parts.append(f"{sum(rated) / len(rated):.1f}★ average")+ if total_ratings:+ parts.append(f"{total_ratings:,} ratings")+ line = f"📦 Amazon: {' │ '.join(parts)}"+ if failed:+ line += f" │ ⚠ {_format_outcome(outcome)}"+ return line++ # Only the *sampled* products earn a slot. A run can carry a dozen+ # discovered products, but only the two or three that got a review pull+ # have a recent window at all -- rendering the rest appends a string of+ # `quiet` entries that push the line past every other source in the box+ # while adding nothing (observed live: 12 entries, 9 of them padding).+ # The count still reports everything found, so nothing is hidden.+ sampled = [+ (s, item) for s, item in zip(all_stats, items) if s.get("reviews_pulled")+ ]+ # Variants of one product share a short name; showing both reads as a+ # rendering bug even though the ASINs differ.+ stats, shown_items, seen_names = [], [], set()+ for stat, item in sampled:+ key = (stat.get("short_name") or "").strip().lower()+ if key and key in seen_names:+ continue+ if key:+ seen_names.add(key)+ stats.append(stat)+ shown_items.append(item)+ items = shown_items++ # Deliberately no quote fragment here. The design called for one+ # model-written phrase on the sharpest negative drift ("... ↓ \"the lid+ # jams\""), but this footer is rendered by the engine *before* the model+ # ever sees the report, and the model passes it through verbatim -- so+ # there is no weave-time write path for the model to supply one. Rather+ # than ship a branch that can never fire, the quote is deferred: the+ # same evidence reaches the reader through the body section's+ # Loved/Gripes/Watch line, which the model does author. `footer_entry`+ # still accepts a quote so a future writer can supply one.+ entries = [amazon.footer_entry(s) for s in stats]+ line = f"📦 Amazon: {count} {plural} │ {', '.join(entries)}"+ if failed:+ line += f" │ ⚠ {_format_outcome(outcome)}"+ return line++def _top_voices_footer_line(report: schema.Report) -> str | None:"""Return the 🗣️ Top voices line or None if no meaningful voices exist.@@ -3021,6 +3144,7 @@ ENGAGEMENT_DISPLAY: dict[str, list[tuple[str, str]]] = {"perplexity": [("citations", "cite")],"digg": [("postCount", "posts"), ("uniqueAuthors", "auth")],"trustpilot": [("reviews", "reviews")],+ "amazon": [("ratings", "ratings")],}diff --git a/skills/last30days/scripts/lib/rerank.py b/skills/last30days/scripts/lib/rerank.pyindex 4fcd8f26..eb04d3a2 100644--- a/skills/last30days/scripts/lib/rerank.py+++ b/skills/last30days/scripts/lib/rerank.py@@ -642,6 +642,17 @@ def _fallback_tuple(if resolved_handles and _is_first_party(candidate, resolved_handles):score += FIRST_PARTY_AUTHOR_CREDITreturn max(0.0, min(100.0, score)), "fallback-local-score (first-party authorship)"+ # Grounding-exempt evidence (currently Amazon): the adapter gated these+ # against the model-supplied keyword before they existed, so the+ # entity-miss demotion below would punish them for a match they were+ # never going to make -- a "Weber Grills" run legitimately surfaces a+ # product called "Spirit E-325" whose reviews discuss searing, not Weber.+ # Returning here also skips _final_score's secondary penalty, which greps+ # the reason string for "entity-miss": one flag, both paths, per the+ # propagation pattern in+ # docs/solutions/logic-errors/entity-grounding-full-phrase-false-demotion.md+ if _is_grounding_exempt(candidate):+ return max(0.0, min(100.0, score)), "fallback-local-score (grounding-exempt source)"# Entity-grounding demotion: subtract ENTITY_MISS_PENALTY when the candidate# never mentions the primary entity's head token, across all text surfaces# (title, snippet, transcript, transcript highlights, top comments,@@ -670,6 +681,23 @@ def _primary_entity(topic: str) -> str:return stripped+def _is_grounding_exempt(candidate: schema.Candidate) -> bool:+ """True when the candidate carries the relevant-by-construction label.++ Set by adapters that already gated their results against an explicit+ keyword at retrieval time (see normalize._normalize_amazon). Checked on+ the candidate's own metadata and on any of its source items, since+ clustering can build a candidate from several items.+ """+ metadata = candidate.metadata or {}+ if isinstance(metadata, dict) and metadata.get("grounding_exempt"):+ return True+ return any(+ isinstance(item.metadata, dict) and item.metadata.get("grounding_exempt")+ for item in candidate.source_items+ )++def _is_corpus_candidate(candidate: schema.Candidate) -> bool:"""True when the candidate carries private corpus evidence."""if candidate.source == "corpus":diff --git a/skills/last30days/scripts/lib/setup_wizard.py b/skills/last30days/scripts/lib/setup_wizard.pyindex 464dabc6..6fad0f86 100644--- a/skills/last30days/scripts/lib/setup_wizard.py+++ b/skills/last30days/scripts/lib/setup_wizard.py@@ -17,6 +17,8 @@ from typing import Any, Dict, Optional, Tuplefrom urllib.error import HTTPError, URLErrorfrom urllib.request import Request, urlopen+from . import brightdata+logger = logging.getLogger(__name__)@@ -159,6 +161,12 @@ def run_auto_setup(config: Dict[str, Any], *, allow_browser_cookies: bool = Fals# Per-CLI status for the additional default-on Printing Press sources# (arxiv, techmeme, trustpilot): {source: {installed, action, ...}}."pp_sources": pp_sources,+ # Reported, never installed: this CLI spends the user's own metered+ # credits, so acquiring it stays their decision (U5/R11). Passing+ # config matters: a user whose key lives in a .env file or the+ # keychain (rather than a `brightdata login` credentials file) is+ # active in the engine, and setup must not tell them otherwise.+ "brightdata": brightdata_status(config),"env_written": False,}if ytdlp_action == "install_failed":@@ -322,6 +330,85 @@ PP_DEFAULT_SOURCES: list[tuple[str, str, str]] = [("techmeme", "techmeme", "techmeme-pp-cli"),]+# Bright Data is deliberately absent from PP_DEFAULT_SOURCES: it is not a+# Printing Press CLI, it is opt-in like Trustpilot, and it spends the user's+# own metered credits. Setup reports its state and never installs it.+BRIGHTDATA_BIN = "brightdata"+++def _brightdata_off_path_binary() -> Optional[str]:+ """Locate a brightdata binary that exists on disk but not on PATH.++ Covers the common npm global prefixes. The distinction matters because+ Hermes and OpenClaw gateways routinely run the engine with a PATH that+ excludes the user's npm bin directory, so "installed" and "the engine+ can see it" are different questions.+ """+ home = Path.home()+ candidates = [+ home / ".local" / "bin" / BRIGHTDATA_BIN,+ home / ".npm-global" / "bin" / BRIGHTDATA_BIN,+ Path("/opt/homebrew/bin") / BRIGHTDATA_BIN,+ Path("/usr/local/bin") / BRIGHTDATA_BIN,+ ]+ npm_prefix = os.environ.get("NPM_CONFIG_PREFIX")+ if npm_prefix:+ candidates.insert(0, Path(npm_prefix) / "bin" / BRIGHTDATA_BIN)+ for candidate in candidates:+ try:+ if candidate.is_file() and os.access(candidate, os.X_OK):+ return str(candidate)+ except OSError:+ continue+ return None+++def brightdata_status(config: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:+ """Report the Bright Data install and auth state honestly.++ Deliberately never claims the source is active unless the engine's own+ gate would pass -- ``brightdata.is_available`` is the single predicate,+ so setup and the engine cannot drift apart. Three states matter:++ * ``already_installed`` -- on PATH; ``authenticated`` says whether the+ amazon lane will actually run.+ * ``installed_off_path`` -- on disk but invisible to the engine, which+ is the Hermes/OpenClaw failure mode. Carries the path so the user can+ fix their PATH.+ * ``not_installed`` -- nothing found. No auto-install: this CLI+ spends the user's metered credits, so acquiring it is their call.+ """+ installed = brightdata.is_installed()+ authenticated = brightdata.has_credentials(config)+ if installed:+ action = "already_installed"+ off_path = ""+ else:+ off_path = _brightdata_off_path_binary() or ""+ action = "installed_off_path" if off_path else "not_installed"++ status: Dict[str, Any] = {+ "installed": installed,+ "action": action,+ "authenticated": installed and authenticated,+ # The engine gate, verbatim. Never report active on anything else.+ "engine_active": brightdata.is_available(config),+ }+ if off_path:+ status["path"] = off_path+ status["hint"] = (+ f"brightdata found at {off_path} but not on PATH; add its directory "+ "to PATH so the engine subprocess can see it"+ )+ elif installed and not authenticated:+ status["hint"] = "run `brightdata login` to activate the amazon source"+ elif not installed:+ status["hint"] = (+ "install with `npm i -g @brightdata/cli` then `brightdata login` "+ "to enable the amazon source"+ )+ return status+def _pp_bin_candidate_paths(bin_name: str) -> list[Path]:"""Known install locations for a Printing Press CLI binary (slug-parameterized@@ -653,6 +740,33 @@ def get_setup_status_text(results: Dict[str, Any]) -> str:f"then: `npx -y {PRINTING_PRESS_NPM} install {source_key} --cli-only`")+ # Bright Data / Amazon. Reported but never installed (it spends the user's+ # own metered credits), so the only useful thing setup can do is say+ # precisely why the lane is or is not active -- the three states below are+ # otherwise invisible, since SKILL.md tells the model not to raise the+ # subject mid-run.+ brightdata_status_entry = results.get("brightdata") or {}+ bd_action = brightdata_status_entry.get("action", "")+ if brightdata_status_entry.get("engine_active"):+ lines.append(" - Bright Data CLI ready (Amazon buyer signals available)")+ elif bd_action == "already_installed":+ lines.append(+ " - Bright Data CLI installed but not logged in — run "+ "`brightdata login` to enable Amazon buyer signals (optional)"+ )+ elif bd_action == "installed_off_path":+ bd_path = brightdata_status_entry.get("path", "")+ lines.append(+ f" - Bright Data CLI found at {bd_path} but not on PATH — add "+ f"{os.path.dirname(os.path.expanduser(bd_path))} to PATH and restart "+ "your agent session/gateway for Amazon buyer signals to activate"+ )+ elif bd_action == "not_installed":+ lines.append(+ " - Amazon buyer signals not installed (optional; 5,000 free "+ "requests/month). Install with: npm i -g @brightdata/cli && brightdata login"+ )+env_written = results.get("env_written", False)if env_written:lines.append("")diff --git a/skills/last30days/scripts/lib/signals.py b/skills/last30days/scripts/lib/signals.pyindex 740984d6..4d5eb4f9 100644--- a/skills/last30days/scripts/lib/signals.py+++ b/skills/last30days/scripts/lib/signals.py@@ -16,6 +16,9 @@ SOURCE_QUALITY = {"arxiv": 0.9,"techmeme": 0.85,"trustpilot": 0.78,+ # Verified-purchase reviews on a live aggregate rating: high-quality+ # buyer evidence, a notch above Trustpilot's open review model.+ "amazon": 0.8,"reddit": 0.6,"x": 0.68,"bluesky": 0.66,@@ -59,6 +62,17 @@ def local_relevance(if "project-mode" in labels:score = max(score, 0.8)+ # Grounding-exempt floor (currently Amazon): the adapter already gated+ # these against the model-supplied product keyword before creating them,+ # so they are relevant by construction. Their text is marketing copy plus+ # buyer reviews, which rarely repeats the topic phrasing -- a "Weber+ # Grills" run surfaces a product named "Spirit E-325" whose reviews talk+ # about searing, not about Weber. Without the floor, correctly-retrieved+ # evidence gets pruned for failing a keyword match it was never going to+ # win. Mirrors the project-mode GitHub floor above.+ if isinstance(item.metadata, dict) and item.metadata.get("grounding_exempt"):+ score = max(score, 0.8)+return score@@ -167,6 +181,7 @@ ENGAGEMENT_WEIGHTS: dict[str, list[tuple[str, float]]] = {"polymarket": [("volume", 0.60), ("liquidity", 0.40)],"digg": [("postCount", 0.40), ("uniqueAuthors", 0.30), ("rank_score", 0.30)],"trustpilot": [("reviews", 1.0)],+ "amazon": [("ratings", 1.0)],}diff --git a/skills/last30days/scripts/lib/ui.py b/skills/last30days/scripts/lib/ui.pyindex 0a166a0c..bbac763b 100644--- a/skills/last30days/scripts/lib/ui.py+++ b/skills/last30days/scripts/lib/ui.py@@ -130,6 +130,7 @@ SOURCE_COMPLETION_ORDER = ["arxiv","techmeme","trustpilot",+ "amazon",]SOURCE_COMPLETION_META = {@@ -148,6 +149,7 @@ SOURCE_COMPLETION_META = {"arxiv": ("arXiv", "paper", "papers", Colors.RED),"techmeme": ("Techmeme", "headline", "headlines", Colors.CYAN),"trustpilot": ("Trustpilot", "review", "reviews", Colors.GREEN),+ "amazon": ("Amazon", "product", "products", Colors.YELLOW),}diff --git a/skills/last30days/scripts/setup-keychain.sh b/skills/last30days/scripts/setup-keychain.shindex e8f1e2fa..b22e0f72 100755--- a/skills/last30days/scripts/setup-keychain.sh+++ b/skills/last30days/scripts/setup-keychain.sh@@ -41,6 +41,7 @@ ALL_KEYS=(XQUIK_API_KEYXIAOHONGSHU_API_BASEGITHUB_TOKEN+ BRIGHTDATA_API_KEY)if [[ "${OSTYPE:-}" != darwin* ]]; thendiff --git a/skills/last30days/scripts/setup-pass.sh b/skills/last30days/scripts/setup-pass.shindex fa549146..65799f08 100755--- a/skills/last30days/scripts/setup-pass.sh+++ b/skills/last30days/scripts/setup-pass.sh@@ -43,6 +43,7 @@ ALL_KEYS=(XQUIK_API_KEYXIAOHONGSHU_API_BASEGITHUB_TOKEN+ BRIGHTDATA_API_KEY)REPLACE=0diff --git a/tests/test_amazon.py b/tests/test_amazon.pynew file mode 100644index 00000000..77882214--- /dev/null+++ b/tests/test_amazon.py@@ -0,0 +1,800 @@+"""Tests for the Amazon source: discovery, enrichment, stats, footer (U2, U3).++Fixtures mirror live payload shapes pulled 2026-08-13, including the three+fields that arrive doubled and the fact that ``max_reviews`` is a ceiling+rather than a quota.++Nothing here spawns a subprocess or touches the network.+"""++from __future__ import annotations++import sys+from datetime import datetime, timedelta, timezone+from pathlib import Path++import pytest++sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))++from lib import amazon # noqa: E402+++TODAY = datetime(2026, 8, 13, tzinfo=timezone.utc)+DOMAIN = "https://www.amazon.com"+++def _days_ago(n: int) -> str:+ return (TODAY - timedelta(days=n)).date().isoformat()+++def search_record(**over):+ base = {+ "asin": "B0AAA00001",+ "url": "https://www.amazon.com/Bentgo-Chill/dp/B0AAA00001/ref=sr_1_1?dib=xyz",+ "name": "Chill Max Leak-Proof XL Bento-Style Lunch Box | Ice Pack Included",+ "brand": "Bentgo",+ "sponsored": "false",+ "rating": 4.4,+ "num_ratings": 459,+ "final_price": 39.99,+ "currency": "USD",+ }+ base.update(over)+ return base+++def review_record(days_ago: int, rating: int, **over):+ posted = (TODAY - timedelta(days=days_ago)).strftime("%B %-d, %Y")+ base = {+ "review_id": f"R{days_ago}{rating}",+ # Live shape: the date is doubled and prose-wrapped.+ "review_posted_date": f"{posted}Reviewed in the United States on {posted}",+ "review_header": "Great box!Great box!",+ "review_text": f"Review body from {days_ago} days ago.",+ "rating": rating,+ "helpful_count": 0,+ "is_verified": True,+ "is_amazon_vine": False,+ "author_name": "A Buyer",+ "product_rating": 4.4,+ "product_rating_count": 459,+ "product_rating_object": {+ "one_star": 28, "two_star": 9, "three_star": 28,+ "four_star": 60, "five_star": 335,+ },+ }+ base.update(over)+ return base+++# ------------------------------------------------------- field repair+++class TestFieldRepair:+ def test_doubled_header_is_repaired(self):+ assert amazon.undouble("Best Box!Best Box!") == "Best Box!"++ def test_comma_doubled_badge_is_repaired(self):+ assert amazon.undouble("Verified Purchase, Verified Purchase") == "Verified Purchase"++ def test_genuinely_repetitive_text_survives(self):+ assert amazon.undouble("Great great product") == "Great great product"+ assert amazon.undouble("Buy one, buy two") == "Buy one, buy two"++ def test_prose_wrapped_date_yields_the_leading_date(self):+ raw = "August 3, 2026Reviewed in the United States on August 3, 2026"+ assert amazon.parse_review_date(raw) == "2026-08-03"++ def test_unparseable_date_is_none_not_an_exception(self):+ assert amazon.parse_review_date("") is None+ assert amazon.parse_review_date("sometime last year") is None+ assert amazon.parse_review_date(None) is None+++class TestShortName:+ def test_takes_the_segment_before_the_delimiter(self):+ assert amazon.short_name("Chill Max XL | Ice Pack Included") == "Chill Max XL"++ def test_strips_a_leading_brand_when_present(self):+ assert amazon.short_name("Weber Spirit E-325", "Weber") == "Spirit E-325"++ def test_clips_long_names_on_a_word_boundary(self):+ out = amazon.short_name("Kids Prints Leak-Proof 5-Compartment Bento-Style Box")+ assert len(out) <= amazon.SHORT_NAME_MAX+ assert not out.endswith("-")+ assert " " not in out[-1:]+++# ---------------------------------------------------------- discovery+++class TestDiscovery:+ def test_parses_products_with_rating_and_price(self):+ products = amazon.parse_search_response({"records": [search_record()]}, "bentgo lunch box")+ assert len(products) == 1+ product = products[0]+ assert product["rating"] == 4.4+ assert product["price"] == 39.99+ assert product["num_ratings"] == 459+ assert product["brand"] == "Bentgo"++ def test_urls_are_canonicalized_to_dp_asin(self):+ """Live URLs carry 200+ chars of session-scoped tracking tail."""+ products = amazon.parse_search_response({"records": [search_record()]}, "bentgo")+ assert products[0]["url"] == "https://www.amazon.com/dp/B0AAA00001"++ def test_duplicate_asins_collapse_keeping_the_richest_record(self):+ records = [+ search_record(num_ratings=84),+ search_record(num_ratings=459),+ search_record(num_ratings=12),+ ]+ products = amazon.parse_search_response({"records": records}, "bentgo")+ assert len(products) == 1+ assert products[0]["num_ratings"] == 459++ def test_off_keyword_products_are_gated_out(self):+ records = [+ search_record(),+ search_record(asin="B0ZZZ00001", name="Cordless Drill Driver Kit", brand="DeWalt",+ url="https://www.amazon.com/dp/B0ZZZ00001"),+ ]+ products = amazon.parse_search_response({"records": records}, "bentgo lunch box")+ assert [p["asin"] for p in products] == ["B0AAA00001"]++ def test_non_amazon_and_non_https_urls_are_rejected(self):+ records = [+ search_record(asin="B000000001", url="http://www.amazon.com/dp/B000000001"),+ search_record(asin="B000000002", url="https://evil.example.com/dp/B2"),+ search_record(asin="B000000003", url="https://www.amazon.com/dp/B000000003"),+ ]+ products = amazon.parse_search_response({"records": records}, "bentgo chill max lunch box")+ assert [p["asin"] for p in products] == ["B000000003"]++ def test_alternate_marketplace_domain_is_honored(self):+ record = search_record(url="https://www.amazon.co.uk/dp/B0AAA00001")+ products = amazon.parse_search_response(+ {"records": [record]}, "bentgo", domain="https://www.amazon.co.uk"+ )+ assert products and products[0]["url"].startswith("https://www.amazon.co.uk/dp/")++ def test_sponsored_string_is_recorded_as_bool_never_filtered(self):+ """R4: the flag is metadata only. Filtering can blank the lane."""+ records = [+ search_record(asin="B000000001", sponsored="true", url="https://www.amazon.com/dp/B000000001"),+ search_record(asin="B000000002", sponsored="false", url="https://www.amazon.com/dp/B000000002"),+ ]+ products = amazon.parse_search_response({"records": records}, "bentgo chill max lunch box")+ assert len(products) == 2+ assert {p["asin"]: p["sponsored"] for p in products} == {"B000000001": True, "B000000002": False}++ def test_error_envelope_yields_no_products(self):+ assert amazon.parse_search_response({"records": [], "error": "401"}, "x") == []+++class TestTargetSelection:+ def _pool(self):+ return amazon.parse_search_response(+ {+ "records": [+ search_record(asin="C000000001", brand="Fimibuke", num_ratings=901,+ name="60oz Leakproof Bento Lunch Box",+ url="https://www.amazon.com/dp/C000000001"),+ search_record(asin="B000000001", brand="Bentgo", num_ratings=821,+ name="Kids Insulated Lunch Bag",+ url="https://www.amazon.com/dp/B000000001"),+ search_record(asin="B000000002", brand="Bentgo", num_ratings=710,+ name="MicroSteel Bento Lunch Box",+ url="https://www.amazon.com/dp/B000000002"),+ search_record(asin="B000000003", brand="Bentgo", num_ratings=623,+ name="Classic Stackable Lunch Box",+ url="https://www.amazon.com/dp/B000000003"),+ ]+ },+ "bentgo lunch box",+ )++ def test_brand_topic_excludes_competitors_buying_the_keyword(self):+ """The guard against paying to review a rival's product."""+ targets = amazon.select_enrichment_targets(self._pool(), limit=3, keyword="bentgo lunch box")+ assert [t["asin"] for t in targets] == ["B000000001", "B000000002", "B000000003"]+ assert all(t["brand"] == "Bentgo" for t in targets)++ def test_category_topic_stays_unfiltered_and_ranks_on_merit(self):+ targets = amazon.select_enrichment_targets(self._pool(), limit=3, keyword="kids lunch box")+ assert targets[0]["asin"] == "C000000001"++ def test_infer_brand_needs_the_keyword_to_name_it(self):+ pool = self._pool()+ assert amazon.infer_brand(pool, "bentgo lunch box") == "Bentgo"+ assert amazon.infer_brand(pool, "best kids lunch box") == ""++ def test_near_identical_variants_do_not_take_two_pulls(self):+ pool = amazon.parse_search_response(+ {+ "records": [+ search_record(asin="V000000001", num_ratings=901, name="60oz Leakproof Box | Blue",+ url="https://www.amazon.com/dp/V000000001"),+ search_record(asin="V000000002", num_ratings=900, name="60oz Leakproof Box | Pink",+ url="https://www.amazon.com/dp/V000000002"),+ search_record(asin="V000000003", num_ratings=500, name="Chill Max XL Box",+ url="https://www.amazon.com/dp/V000000003"),+ ]+ },+ "bentgo lunch box",+ )+ targets = amazon.select_enrichment_targets(pool, limit=2, keyword="bentgo lunch box")+ assert [t["asin"] for t in targets] == ["V000000001", "V000000003"]++ def test_zero_limit_selects_nothing(self):+ assert amazon.select_enrichment_targets(self._pool(), limit=0) == []+++# --------------------------------------------------------- enrichment+++class TestReviewParsing:+ def test_reviews_become_comments_with_stats(self):+ response = {"records": [review_record(3, 5), review_record(10, 4)]}+ comments, stats = amazon.parse_reviews(response)+ assert len(comments) == 2+ assert stats["product_rating"] == 4.4+ assert stats["product_rating_count"] == 459+ assert stats["star_distribution"]["five_star"] == 335++ def test_comments_carry_the_keys_remap_would_strip(self):+ """Rating, date, and verified are exactly what this source needs."""+ comments, _ = amazon.parse_reviews({"records": [review_record(3, 5)]})+ comment = comments[0]+ assert set(comment) >= {"score", "excerpt", "rating", "date", "verified"}+ assert comment["rating"] == 5+ assert comment["date"] == _days_ago(3)++ def test_doubled_header_is_repaired_in_the_comment_title(self):+ comments, _ = amazon.parse_reviews({"records": [review_record(3, 5)]})+ assert comments[0]["title"] == "Great box!"++ def test_woven_sample_is_newest_first(self):+ """R2a: recency is enforced client-side, not assumed from the API."""+ response = {"records": [+ review_record(400, 5), review_record(2, 3), review_record(45, 4),+ ]}+ comments, _ = amazon.parse_reviews(response)+ assert [c["date"] for c in comments] == [_days_ago(2), _days_ago(45), _days_ago(400)]++ def test_empty_payload_is_not_an_error(self):+ assert amazon.parse_reviews({"records": []}) == ([], {})+++class TestEnrichmentLane:+ def _products(self, n=4):+ return [+ {"asin": f"B00000000{i}", "url": f"https://www.amazon.com/dp/B00000000{i}",+ "name": f"Product {i}", "short_name": f"Product {i}",+ "brand": "Bentgo", "num_ratings": 900 - i, "rating": 4.4}+ for i in range(n)+ ]++ def test_quick_depth_spawns_no_review_pulls(self):+ calls = []+ out = amazon.enrich_with_reviews(+ self._products(), depth="quick",+ fetcher=lambda url: calls.append(url) or {"records": []},+ )+ assert calls == []+ assert len(out) == 4++ def test_default_depth_pulls_exactly_three(self):+ calls = []+ amazon.enrich_with_reviews(+ self._products(), depth="default",+ fetcher=lambda url: calls.append(url) or {"records": [review_record(2, 5)]},+ )+ assert len(calls) == 3++ def test_deep_depth_pulls_five(self):+ calls = []+ amazon.enrich_with_reviews(+ self._products(6), depth="deep",+ fetcher=lambda url: calls.append(url) or {"records": []},+ )+ assert len(calls) == 5++ def test_cap_is_fifty_per_pull(self):+ seen = {}++ def fetcher(url):+ return {"records": []}++ # The cap reaches the CLI through fetch_reviews; assert the constant+ # and the plumbed default together.+ assert amazon.MAX_REVIEWS == 50+ import lib.brightdata as bd+ original = bd.run_pipeline+ bd.run_pipeline = lambda p, params, **k: seen.update(params=params) or {"records": []}+ try:+ amazon.fetch_reviews("https://www.amazon.com/dp/B000000001")+ finally:+ bd.run_pipeline = original+ assert seen["params"] == ["https://www.amazon.com/dp/B000000001", "50"]++ def test_reviews_attach_to_the_right_product(self):+ out = amazon.enrich_with_reviews(+ self._products(3), depth="default",+ fetcher=lambda url: {"records": [review_record(2, 5, review_id=url)]},+ )+ assert all(p.get("top_comments") for p in out)+ assert out[0]["product_rating_count"] == 459++ def test_one_failing_pull_does_not_discard_its_siblings(self):+ def fetcher(url):+ if url.endswith("B000000001"):+ return {"records": [], "error": "snapshot failed"}+ return {"records": [review_record(2, 5)]}++ out = amazon.enrich_with_reviews(self._products(3), depth="default", fetcher=fetcher)+ by_asin = {p["asin"]: p for p in out}+ assert not by_asin["B000000001"].get("top_comments")+ assert by_asin["B000000000"].get("top_comments")+ assert by_asin["B000000002"].get("top_comments")++ def test_one_raising_pull_does_not_kill_the_lane(self):+ def fetcher(url):+ if url.endswith("B000000001"):+ raise RuntimeError("boom")+ return {"records": [review_record(2, 5)]}++ out = amazon.enrich_with_reviews(self._products(3), depth="default", fetcher=fetcher)+ assert sum(1 for p in out if p.get("top_comments")) == 2++ def test_dropped_straggler_keeps_its_product_with_search_stats(self):+ """A deadline drop must never delete the product from the report."""+ import time as _time++ def slow(url):+ if url.endswith("B000000000"):+ _time.sleep(3)+ return {"records": [review_record(2, 5)]}++ out = amazon.enrich_with_reviews(+ self._products(2), depth="default", fetcher=slow,+ elapsed=amazon.FOREGROUND_CONTRACT - amazon.RENDER_MARGIN - 1,+ )+ by_asin = {p["asin"]: p for p in out}+ assert set(by_asin) == {"B000000000", "B000000001"}+ # Search-record stats survive on the dropped product.+ assert by_asin["B000000000"]["rating"] == 4.4+ assert by_asin["B000000000"]["num_ratings"] == 900++ def test_exhausted_wall_clock_skips_the_lane_entirely(self):+ calls = []+ amazon.enrich_with_reviews(+ self._products(), depth="default",+ fetcher=lambda url: calls.append(url) or {"records": []},+ elapsed=amazon.FOREGROUND_CONTRACT,+ )+ assert calls == []+++# --------------------------------------------------------------- stats+++class TestStats:+ def test_five_star_share_from_the_distribution_object(self):+ share = amazon.five_star_share(+ {"one_star": 28, "two_star": 9, "three_star": 28, "four_star": 60, "five_star": 335}+ )+ assert round(share * 100) == 73++ def test_five_star_share_is_none_without_a_distribution(self):+ assert amazon.five_star_share({}) is None++ def test_recent_window_counts_only_dated_reviews_inside_it(self):+ comments = [+ {"date": _days_ago(2), "rating": 4},+ {"date": _days_ago(29), "rating": 4},+ {"date": _days_ago(31), "rating": 1},+ {"date": None, "rating": 5},+ ]+ window = amazon.recent_window_stats(comments, today=TODAY)+ assert window["recent_n"] == 2+ assert window["recent_avg"] == 4++ @pytest.mark.parametrize(+ "ratings,expected",+ [+ ([1, 2, 3, 4, 5], "down"), # avg 3.0 vs 4.4+ ([5, 5, 5, 5, 5], "up"), # avg 5.0 vs 4.4+ ([4, 4, 5, 4, 5], "flat"), # avg 4.4 vs 4.4+ ],+ )+ def test_drift_direction(self, ratings, expected):+ product = {+ "product_rating": 4.4,+ "product_rating_count": 459,+ "top_comments": [+ {"date": _days_ago(i + 1), "rating": r} for i, r in enumerate(ratings)+ ],+ }+ assert amazon.product_stats(product, today=TODAY)["drift"] == expected++ def test_below_threshold_sample_renders_quiet_not_a_drift(self):+ """Live census: a 50-cap pull can land only a handful in-window."""+ product = {+ "product_rating": 4.4,+ "top_comments": [{"date": _days_ago(i + 1), "rating": 1} for i in range(4)],+ }+ assert amazon.product_stats(product, today=TODAY)["drift"] == "quiet"++ def test_threshold_is_exactly_five(self):+ product = {+ "product_rating": 4.4,+ "top_comments": [{"date": _days_ago(i + 1), "rating": 1} for i in range(5)],+ }+ assert amazon.product_stats(product, today=TODAY)["drift"] == "down"++ def test_no_baseline_renders_new(self):+ assert amazon.product_stats({"top_comments": []}, today=TODAY)["drift"] == "new"++ def test_review_pull_rating_count_supersedes_the_search_record(self):+ """Search counts are variant-level and can undercount 100x."""+ stats = amazon.product_stats(+ {"num_ratings": 84, "product_rating_count": 8446, "product_rating": 4.7},+ today=TODAY,+ )+ assert stats["ratings_total"] == 8446++ def test_reproduces_the_live_chill_max_reading(self):+ """End-to-end against the real 2026-08-13 payload shape."""+ records = (+ [review_record(i, r) for i, r in ((1, 5), (2, 1), (5, 5), (9, 4), (11, 4))]+ + [review_record(200, 5), review_record(400, 5)]+ )+ comments, stats = amazon.parse_reviews({"records": records})+ product = {"short_name": "Chill Max XL", **stats, "top_comments": comments}+ out = amazon.product_stats(product, today=TODAY)+ assert out["all_time"] == 4.4+ assert out["ratings_total"] == 459+ assert round(out["five_star_share"] * 100) == 73+ assert out["recent_n"] == 5+ assert out["recent_avg"] == 3.8+ assert out["drift"] == "down"+++# -------------------------------------------------------------- footer+++class TestFooterEntry:+ def test_negative_drift_gets_the_arrow_marker(self):+ entry = amazon.footer_entry(+ {"short_name": "Chill Max XL", "all_time": 4.4, "recent_avg": 3.8, "drift": "down"}+ )+ assert entry == "Chill Max XL 4.4★→3.8★ ↓"++ def test_positive_drift_gets_no_marker(self):+ entry = amazon.footer_entry(+ {"short_name": "Deluxe Bag", "all_time": 4.7, "recent_avg": 5.0, "drift": "up"}+ )+ assert entry == "Deluxe Bag 4.7★→5.0★"+ assert "↓" not in entry++ def test_quiet_state_shows_the_baseline_without_an_arrow(self):+ entry = amazon.footer_entry(+ {"short_name": "Genesis E-325", "all_time": 4.4, "recent_avg": None, "drift": "quiet"}+ )+ assert entry == "Genesis E-325 4.4★ quiet"+ assert "→" not in entry++ def test_new_state_claims_no_baseline(self):+ entry = amazon.footer_entry({"short_name": "BLUEY Set", "all_time": None, "drift": "new"})+ assert entry == "BLUEY Set new"++ def test_quote_renders_only_on_negative_drift(self):+ sagging = amazon.footer_entry(+ {"short_name": "Chill Max XL", "all_time": 4.4, "recent_avg": 3.8, "drift": "down"},+ quote="the lid jams",+ )+ assert sagging == 'Chill Max XL 4.4★→3.8★ ↓ "the lid jams"'+ healthy = amazon.footer_entry(+ {"short_name": "Deluxe Bag", "all_time": 4.7, "recent_avg": 5.0, "drift": "up"},+ quote="the lid jams",+ )+ assert '"' not in healthy++ def test_absent_quote_renders_the_clean_numeric_entry(self):+ entry = amazon.footer_entry(+ {"short_name": "Chill Max XL", "all_time": 4.4, "recent_avg": 3.8, "drift": "down"},+ quote="",+ )+ assert entry == "Chill Max XL 4.4★→3.8★ ↓"+++class _Item:+ """Minimal SourceItem stand-in for the enrichment adapter."""++ def __init__(self, asin, **meta):+ self.source = "amazon"+ self.url = f"https://www.amazon.com/dp/{asin}"+ self.title = meta.get("name", asin)+ self.metadata = {"asin": asin, "short_name": asin, "brand": "Bentgo", **meta}+++class TestSourceItemEnrichment:+ def test_reviews_and_stats_land_on_item_metadata(self):+ items = [_Item("B000000001"), _Item("B000000002")]+ amazon.enrich_source_items(+ items, depth="default", keyword="bentgo lunch box",+ fetcher=lambda url: {"records": [+ review_record(i + 1, r) for i, r in enumerate([1, 1, 1, 1, 1])+ ]},+ )+ for item in items:+ assert item.metadata["top_comments"]+ assert item.metadata["stats"]["drift"] == "down"++ def test_non_amazon_items_are_untouched(self):+ other = _Item("B000000001")+ other.source = "reddit"+ amazon.enrich_source_items([other], depth="default", fetcher=lambda url: {"records": []})+ assert "top_comments" not in other.metadata++ def test_already_enriched_items_are_not_re_pulled(self):+ item = _Item("B000000001", top_comments=[{"excerpt": "cached", "score": 0, "rating": 5, "date": None}])+ calls = []+ amazon.enrich_source_items(+ [item], depth="default",+ fetcher=lambda url: calls.append(url) or {"records": []},+ )+ assert calls == []++ def test_quick_depth_touches_nothing(self):+ items = [_Item("B000000001")]+ calls = []+ amazon.enrich_source_items(+ items, depth="quick",+ fetcher=lambda url: calls.append(url) or {"records": []},+ )+ assert calls == []+ assert "top_comments" not in items[0].metadata+++class TestStatsFromItem:+ def test_uses_the_cached_block_when_enrichment_already_ran(self):+ item = _Item("B000000001", stats={"short_name": "Cached", "drift": "up"})+ assert amazon.stats_from_item(item)["short_name"] == "Cached"++ def test_recomputes_from_metadata_when_absent(self):+ """Mock runs and replayed fixtures skip enrichment entirely."""+ item = _Item(+ "B1",+ product_rating=4.4,+ product_rating_count=459,+ star_distribution={"one_star": 28, "two_star": 9, "three_star": 28,+ "four_star": 60, "five_star": 335},+ top_comments=[{"date": _days_ago(i + 1), "rating": 1} for i in range(5)],+ )+ stats = amazon.stats_from_item(item, today=TODAY)+ assert stats["drift"] == "down"+ assert stats["all_time"] == 4.4+ assert round(stats["five_star_share"] * 100) == 73+++class _FooterItem:+ def __init__(self, short_name, *, reviews=0, all_time=4.4, recent=None,+ drift="quiet", quote=None):+ self.source = "amazon"+ self.url = "https://www.amazon.com/dp/X"+ self.title = short_name+ self.metadata = {+ "asin": short_name,+ "stats": {+ "short_name": short_name, "all_time": all_time,+ "recent_avg": recent, "drift": drift,+ "reviews_pulled": reviews, "ratings_total": 100,+ "five_star_share": 0.7, "recent_n": 5 if recent else 0,+ "url": self.url,+ },+ }+ if quote:+ self.metadata["pulse_quote"] = quote+++def _report(items, *, keyword="bentgo lunch box"):+ from lib import schema+ report = object.__new__(schema.Report)+ object.__setattr__(report, "items_by_source", {"amazon": items})+ object.__setattr__(report, "artifacts", {"amazon_query": keyword})+ object.__setattr__(report, "source_status", {})+ return report+++class TestFooterLine:+ def _line(self, items, **kw):+ from lib import render+ return render._amazon_footer_line(_report(items, **kw))++ def test_only_sampled_products_get_a_slot(self):+ """A dozen discovered products must not become a dozen entries."""+ items = [_FooterItem("Sampled", reviews=20, recent=3.8, drift="down")]+ items += [_FooterItem(f"Unsampled{i}") for i in range(9)]+ line = self._line(items)+ assert "Sampled 4.4★→3.8★ ↓" in line+ assert "Unsampled" not in line+ # The count still reports everything discovered.+ assert line.startswith("📦 Amazon: 10 products │")++ def test_duplicate_variant_names_are_collapsed(self):+ items = [+ _FooterItem("Kids Bento", reviews=20, recent=4.9, drift="up"),+ _FooterItem("Kids Bento", reviews=20, recent=4.8, drift="up"),+ ]+ assert self._line(items).count("Kids Bento") == 1++ def test_quick_depth_renders_the_inventory_form(self):+ items = [_FooterItem("A"), _FooterItem("B")]+ line = self._line(items)+ assert "→" not in line+ assert "average" in line and "ratings" in line++ def test_footer_renders_no_quote_today(self):+ """The engine renders this line before the model sees the report, so+ there is no weave-time path for a model-written quote. Deferred."""+ items = [_FooterItem("Sagging", reviews=20, recent=3.8, drift="down",+ quote="the lid jams")]+ line = self._line(items)+ assert "↓" in line+ assert '"' not in line++ def test_footer_entry_still_accepts_a_quote_for_a_future_writer(self):+ entry = amazon.footer_entry(+ {"short_name": "X", "all_time": 4.4, "recent_avg": 3.8, "drift": "down"},+ quote="the lid jams",+ )+ assert '"the lid jams"' in entry++ def test_empty_result_names_the_keyword(self):+ assert self._line([]) == '📦 Amazon: no products matched "bentgo lunch box"'++ def test_no_line_at_all_when_the_source_never_ran(self):+ assert self._line([], keyword="") is None++ def test_malformed_asin_records_are_rejected(self):+ """The ASIN is interpolated into a URL that is refetched and rendered."""+ records = [+ search_record(asin="../../etc/passwd", url="https://www.amazon.com/dp/x"),+ search_record(asin="B0AAA0000", url="https://www.amazon.com/dp/y"), # 9 chars+ search_record(asin="B0AAA000012", url="https://www.amazon.com/dp/z"), # 11 chars+ search_record(asin="B0AAA00001", url="https://www.amazon.com/dp/ok"),+ ]+ products = amazon.parse_search_response({"records": records}, "bentgo chill max lunch box")+ assert [p["asin"] for p in products] == ["B0AAA00001"]++ def test_canonical_url_falls_back_when_the_asin_is_malformed(self):+ original = "https://www.amazon.com/dp/legit"+ assert amazon.canonical_product_url(original, "not-an-asin", DOMAIN) == original++ def test_option_shaped_keyword_is_rejected_before_the_cli_runs(self):+ """A leading dash would be parsed as a flag, not a search term."""+ calls = []+ import lib.brightdata as bd+ original = bd.run_pipeline+ bd.run_pipeline = lambda *a, **k: calls.append(a) or {"records": []}+ try:+ out = amazon.search_products("--help")+ finally:+ bd.run_pipeline = original+ assert calls == []+ assert "may not begin" in out["error"]+++class TestReviewFindingRegressions:+ """Regressions caught by the correctness review pass."""++ def test_brand_inference_survives_mixed_casing(self):+ """One vendor spelled two ways must not disable the guard."""+ pool = [{"brand": "Bentgo"}, {"brand": "BENTGO"}, {"brand": "Umi"}]+ assert amazon.infer_brand(pool, "bentgo lunch box") == "Bentgo"++ def test_multi_word_brands_are_matched(self):+ pool = [{"brand": "Hydro Flask"}, {"brand": "Iron Flask"}]+ assert amazon.infer_brand(pool, "hydro flask water bottle") == "Hydro Flask"++ def test_two_distinct_brands_in_the_keyword_stay_ambiguous(self):+ pool = [{"brand": "Yeti"}, {"brand": "Stanley"}]+ assert amazon.infer_brand(pool, "yeti vs stanley tumbler") == ""++ def test_short_name_respects_word_boundaries_on_the_brand(self):+ """A bare startswith() ate into sub-brands and coincidental prefixes."""+ assert amazon.short_name("AnkerWork B600 Video Bar", "Anker") == "AnkerWork B600"+ assert amazon.short_name("Chillax Bento Lunch Box", "Chill") == "Chillax Bento"++ def test_undouble_leaves_short_repeated_words_alone(self):+ assert amazon.undouble("ByeBye") == "ByeBye"+ assert amazon.undouble("NoNo") == "NoNo"+ # Real doubling of a whole headline still repairs.+ assert amazon.undouble("Best Box!Best Box!") == "Best Box!"+ assert amazon.undouble("Great value for money.Great value for money.") == \+ "Great value for money."++ def test_lane_budget_shrinks_as_the_run_clock_advances(self):+ """The clamp was dead code until `elapsed` was threaded through."""+ assert amazon._remaining_lane_budget(0.0) == amazon.LANE_DEADLINE+ mid = amazon._remaining_lane_budget(200.0)+ assert 0 < mid < amazon.LANE_DEADLINE+ assert amazon._remaining_lane_budget(295.0) == 0++ def test_enrichment_refreshes_the_variant_level_rating_count(self):+ """Search counts undercount badly; the pull's count is authoritative."""+ item = _Item("B000000001", name="Deluxe Bag", num_ratings=84)+ item.title = "Bentgo Deluxe Bag - 4.7/5 (84 ratings)"+ item.engagement = {"ratings": 84}+ amazon.enrich_source_items(+ [item], depth="default",+ fetcher=lambda url: {"records": [+ review_record(i + 1, 5, product_rating=4.7, product_rating_count=8446)+ for i in range(5)+ ]},+ )+ assert item.engagement["ratings"] == 8446+ assert "8,446 ratings" in item.title+ assert "84 ratings" not in item.title+ assert item.metadata["stats"]["ratings_total"] == 8446+++class TestSavedArtifactCompleteness:+ """The saved report is the copy users keep; it must not drop evidence."""++ def _saved(self, sources):+ from lib import render, schema+ report = object.__new__(schema.Report)+ for field, value in {+ "topic": "bentgo", "range_from": "2026-07-14", "range_to": "2026-08-13",+ "generated_at": "2026-08-13", "clusters": [], "ranked_candidates": [],+ "items_by_source": sources, "errors_by_source": {}, "source_status": {},+ "freshness_verdicts": [], "warnings": [], "artifacts": {},+ "library_context": [], "drill_of": None,+ "provider_runtime": schema.ProviderRuntime(+ reasoning_provider="local", planner_model="m", rerank_model="m",+ ),+ "query_plan": schema.QueryPlan(+ intent="product", freshness_mode="balanced_recent", cluster_mode="none",+ raw_topic="bentgo", subqueries=[],+ source_weights={s: 1.0 for s in sources},+ ),+ }.items():+ object.__setattr__(report, field, value)+ return render.render_full(report)++ def _item(self, source, item_id, engagement):+ from lib import schema+ return schema.SourceItem(+ item_id=item_id, source=source, title=f"{source} item", body="b",+ url="https://example.com", author="A", container=None,+ published_at="2026-08-10", date_confidence="high",+ engagement=engagement, relevance_hint=0.5, why_relevant="",+ snippet="", metadata={},+ )++ def test_amazon_items_appear_in_the_per_source_dump(self):+ """Regression: a hardcoded source list silently dropped this section."""+ out = self._saved({"amazon": [self._item("amazon", "B0AAA00001", {"ratings": 459})]})+ assert "### Amazon (1 items)" in out+ assert "B0AAA00001" in out++ def test_a_source_absent_from_the_fixed_order_still_renders(self):+ """The list is display order, not the source registry."""+ out = self._saved({"trustpilot": [self._item("trustpilot", "TP1", {"reviews": 12})]})+ assert "TP1" in out++ def test_engagement_is_not_blank_for_a_non_allowlisted_metric(self):+ out = self._saved({"amazon": [self._item("amazon", "B0AAA00001", {"ratings": 459})]})+ assert "459 ratings" in out++ def test_allowlisted_sources_keep_their_existing_engagement_format(self):+ """The fall-through must not add previously-unshown keys."""+ out = self._saved({"reddit": [self._item(+ "reddit", "R1", {"score": 120, "num_comments": 48, "upvote_ratio": 0.91}+ )]})+ assert "120 score, 48 num_comments" in out+ assert "upvote_ratio" not in outdiff --git a/tests/test_brightdata.py b/tests/test_brightdata.pynew file mode 100644index 00000000..061d7d1d--- /dev/null+++ b/tests/test_brightdata.py@@ -0,0 +1,313 @@+"""Tests for the Bright Data CLI adapter (U1).++Covers the activation gate (PATH + presence-only credential probe), the+never-raise error envelope across every failure mode, and the verbatim+passthrough of the CLI's own auth/credit warnings.++No test in this file spawns a real subprocess or touches the network.+"""++from __future__ import annotations++import json+import sys+from pathlib import Path++import pytest++sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "skills" / "last30days" / "scripts"))++from lib import brightdata, subproc # noqa: E402+++class _Result:+ """Stand-in for subproc.SubprocResult."""++ def __init__(self, returncode=0, stdout="", stderr=""):+ self.returncode = returncode+ self.stdout = stdout+ self.stderr = stderr+++# One real search record, trimmed. Field names verified against a live+# amazon_product_search payload on 2026-08-13.+SEARCH_RECORD = {+ "asin": "B0DWD47BW7",+ "url": "https://www.amazon.com/dp/B0DWD47BW7",+ "name": "Chill Max Leak-Proof XL Bento-Style Lunch Box | Ice Pack Included",+ "brand": "Bentgo",+ "sponsored": "false",+ "rating": 4.4,+ "num_ratings": 459,+ "final_price": 39.99,+ "currency": "USD",+}+++# ---------------------------------------------------------------- gate+++def test_gate_false_when_binary_missing(monkeypatch):+ monkeypatch.setattr(brightdata.shutil, "which", lambda _: None)+ assert brightdata.is_installed() is False+ assert brightdata.is_available({}) is False+++def test_gate_reports_unauthenticated_when_no_credential_signal(monkeypatch, tmp_path):+ monkeypatch.setattr(brightdata.shutil, "which", lambda _: "/usr/local/bin/brightdata")+ monkeypatch.setattr(brightdata, "_config_dir", lambda: tmp_path / "absent")+ status = brightdata.gate_status({})+ assert status["brightdata_installed"] is True+ assert status["brightdata_authenticated"] is False+ assert brightdata.is_available({}) is False+++def test_api_key_alone_satisfies_the_credential_probe(monkeypatch, tmp_path):+ monkeypatch.setattr(brightdata.shutil, "which", lambda _: "/usr/local/bin/brightdata")+ monkeypatch.setattr(brightdata, "_config_dir", lambda: tmp_path / "absent")+ assert brightdata.is_available({"BRIGHTDATA_API_KEY": "dummy-key-not-real"}) is True+++def test_credentials_file_satisfies_the_probe_without_being_read(monkeypatch, tmp_path):+ cfg = tmp_path / "brightdata-cli"+ cfg.mkdir()+ creds = cfg / "credentials.json"+ creds.write_text('{"token": "dummy-not-real"}')+ creds.chmod(0o600)+ monkeypatch.setattr(brightdata.shutil, "which", lambda _: "/usr/local/bin/brightdata")+ monkeypatch.setattr(brightdata, "_config_dir", lambda: cfg)++ def _explode(*a, **k): # pragma: no cover - must never run+ raise AssertionError("credential contents must never be read")++ monkeypatch.setattr(Path, "read_text", _explode)+ assert brightdata.has_credentials({}) is True+++def test_gate_does_not_spawn_a_subprocess(monkeypatch, tmp_path):+ monkeypatch.setattr(brightdata.shutil, "which", lambda _: None)++ def _explode(*a, **k): # pragma: no cover - must never run+ raise AssertionError("gate must stay offline")++ monkeypatch.setattr(subproc, "run_with_timeout", _explode)+ brightdata.gate_status({})+ assert brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)["records"] == []+++# ------------------------------------------------------------ envelope+++def _installed(monkeypatch):+ monkeypatch.setattr(brightdata.shutil, "which", lambda _: "/usr/local/bin/brightdata")+++def test_happy_path_parses_a_bare_json_array(monkeypatch):+ """Live payloads are a top-level list, not a {"results": ...} envelope."""+ _installed(monkeypatch)+ monkeypatch.setattr(+ subproc, "run_with_timeout",+ lambda *a, **k: _Result(stdout=json.dumps([SEARCH_RECORD, SEARCH_RECORD])),+ )+ out = brightdata.run_pipeline("amazon_product_search", ["bentgo", "https://www.amazon.com"], timeout=60)+ assert "error" not in out+ assert len(out["records"]) == 2+ assert out["records"][0]["asin"] == "B0DWD47BW7"++[email protected]("wrapper", ["records", "results", "data"])+def test_dict_envelopes_are_tolerated_for_cli_churn(monkeypatch, wrapper):+ _installed(monkeypatch)+ monkeypatch.setattr(+ subproc, "run_with_timeout",+ lambda *a, **k: _Result(stdout=json.dumps({wrapper: [SEARCH_RECORD]})),+ )+ assert len(brightdata.run_pipeline("amazon_product", ["u"], timeout=30)["records"]) == 1+++def test_auth_401_surfaces_the_cli_error_without_raising(monkeypatch):+ _installed(monkeypatch)+ stderr = "Triggering pipeline collection...\nError: 401 Unauthorized - run `brightdata login`"+ monkeypatch.setattr(+ subproc, "run_with_timeout", lambda *a, **k: _Result(returncode=1, stderr=stderr)+ )+ out = brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)+ assert out["records"] == []+ assert "401" in out["error"]+++def test_error_uses_the_last_stderr_line_not_the_polling_narration(monkeypatch):+ """The CLI narrates polling on stderr; the failure is the final line."""+ _installed(monkeypatch)+ stderr = (+ "Triggering pipeline collection for amazon_product_reviews...\n"+ "Status: running - polling again (attempt 1/600)\n"+ "Error: snapshot failed"+ )+ monkeypatch.setattr(+ subproc, "run_with_timeout", lambda *a, **k: _Result(returncode=1, stderr=stderr)+ )+ assert brightdata.run_pipeline("amazon_product_reviews", ["u", "50"], timeout=30)["error"] == "Error: snapshot failed"+++def test_subprocess_timeout_returns_empty_records_and_error(monkeypatch):+ _installed(monkeypatch)++ def _timeout(*a, **k):+ raise subproc.SubprocTimeout("Command brightdata timed out after 180s")++ monkeypatch.setattr(subproc, "run_with_timeout", _timeout)+ out = brightdata.run_pipeline("amazon_product_reviews", ["u", "50"], timeout=180)+ assert out["records"] == []+ assert "timed out" in out["error"]+++def test_malformed_json_returns_empty_records_and_error(monkeypatch):+ _installed(monkeypatch)+ monkeypatch.setattr(subproc, "run_with_timeout", lambda *a, **k: _Result(stdout="not json{"))+ out = brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)+ assert out["records"] == []+ assert "json decode" in out["error"]+++def test_empty_stdout_is_not_an_error(monkeypatch):+ _installed(monkeypatch)+ monkeypatch.setattr(subproc, "run_with_timeout", lambda *a, **k: _Result(stdout=" \n"))+ out = brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)+ assert out["records"] == []+ assert "error" not in out+++def test_spawn_failures_never_raise(monkeypatch):+ _installed(monkeypatch)+ for exc in (FileNotFoundError("no binary"), OSError("spawn failed")):+ def _raise(*a, __e=exc, **k):+ raise __e++ monkeypatch.setattr(subproc, "run_with_timeout", _raise)+ out = brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)+ assert out["records"] == [] and out["error"]+++# ------------------------------------------------------------ arguments+++def test_cli_timeout_sits_below_the_subprocess_timeout(monkeypatch):+ """The CLI must fail cleanly on its own before we SIGTERM it."""+ _installed(monkeypatch)+ seen = {}+ monkeypatch.setattr(+ subproc, "run_with_timeout",+ lambda cmd, **k: seen.update(cmd=cmd, timeout=k["timeout"]) or _Result(stdout="[]"),+ )+ brightdata.run_pipeline("amazon_product_reviews", ["u", "50"], timeout=180)+ cli_timeout = int(seen["cmd"][seen["cmd"].index("--timeout") + 1])+ assert cli_timeout < seen["timeout"] == 180+++def test_params_are_passed_positionally_in_order(monkeypatch):+ _installed(monkeypatch)+ seen = {}+ monkeypatch.setattr(+ subproc, "run_with_timeout",+ lambda cmd, **k: seen.update(cmd=cmd) or _Result(stdout="[]"),+ )+ brightdata.run_pipeline("amazon_product_reviews", ["https://a/dp/X", 50], timeout=60)+ cmd = seen["cmd"]+ assert cmd[:3] == ["brightdata", "pipelines", "amazon_product_reviews"]+ # Positionals sit after the option fence, in the order supplied.+ assert cmd[cmd.index("--") + 1:] == ["https://a/dp/X", "50"]+ assert "--json" in cmd+++def test_api_key_travels_in_the_environment_never_in_argv(monkeypatch):+ """argv is not a secret channel: /proc/<pid>/cmdline is world-readable."""+ _installed(monkeypatch)+ seen = {}+ monkeypatch.setattr(+ subproc, "run_with_timeout",+ lambda cmd, **k: seen.update(cmd=cmd, env=k.get("env")) or _Result(stdout="[]"),+ )+ monkeypatch.delenv("BRIGHTDATA_API_KEY", raising=False)+ brightdata.run_pipeline(+ "amazon_product", ["u"], timeout=30, config={"BRIGHTDATA_API_KEY": "dummy-key"}+ )+ assert "-k" not in seen["cmd"]+ assert "dummy-key" not in " ".join(seen["cmd"])+ assert seen["env"]["BRIGHTDATA_API_KEY"] == "dummy-key"+++def test_no_key_means_the_child_simply_inherits_the_parent_env(monkeypatch):+ _installed(monkeypatch)+ seen = {}+ monkeypatch.setattr(+ subproc, "run_with_timeout",+ lambda cmd, **k: seen.update(env=k.get("env")) or _Result(stdout="[]"),+ )+ brightdata.run_pipeline("amazon_product", ["u"], timeout=30, config={})+ assert seen["env"] is None+++def test_positional_params_are_fenced_behind_a_double_dash(monkeypatch):+ """A keyword beginning with '-' must not be parsed as an option."""+ _installed(monkeypatch)+ seen = {}+ monkeypatch.setattr(+ subproc, "run_with_timeout",+ lambda cmd, **k: seen.update(cmd=cmd) or _Result(stdout="[]"),+ )+ brightdata.run_pipeline("amazon_product_search", ["--help", "https://a"], timeout=30)+ cmd = seen["cmd"]+ assert "--" in cmd+ assert cmd.index("--") < cmd.index("--help")+++def test_the_key_is_scrubbed_out_of_surfaced_stderr(monkeypatch):+ """Auth failures are exactly where a CLI echoes the rejected key back."""+ _installed(monkeypatch)+ logged = []+ monkeypatch.setattr(brightdata.log, "source_log", lambda n, m, **k: logged.append(m))+ monkeypatch.setattr(+ subproc, "run_with_timeout",+ lambda *a, **k: _Result(returncode=1, stderr="Error: 401 for key dummy-key"),+ )+ out = brightdata.run_pipeline(+ "amazon_product", ["u"], timeout=30, config={"BRIGHTDATA_API_KEY": "dummy-key"}+ )+ assert "dummy-key" not in out["error"]+ assert not any("dummy-key" in m for m in logged)+++# ------------------------------------------------------------- warnings+++def test_credit_and_auth_warnings_pass_through_verbatim(monkeypatch):+ _installed(monkeypatch)+ warned = []+ monkeypatch.setattr(brightdata.log, "source_log", lambda name, msg, **k: warned.append(msg))+ stderr = (+ "Triggering pipeline collection for amazon_product_search...\n"+ "Status: running - polling again (attempt 1/600)\n"+ "Warning: only 12 credits remaining on your free tier\n"+ )+ monkeypatch.setattr(+ subproc, "run_with_timeout", lambda *a, **k: _Result(stdout="[]", stderr=stderr)+ )+ brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)+ assert "Warning: only 12 credits remaining on your free tier" in warned+ # Routine polling narration stays out of the log.+ assert not any("polling again" in m for m in warned)+++def test_source_log_is_called_with_tty_only_false(monkeypatch):+ """Repo rule: tty_only=True silently drops output in non-TTY hosts."""+ _installed(monkeypatch)+ kwargs = {}+ monkeypatch.setattr(+ brightdata.log, "source_log", lambda name, msg, **k: kwargs.update(k)+ )+ monkeypatch.setattr(+ subproc, "run_with_timeout", lambda *a, **k: _Result(returncode=1, stderr="Error: 401")+ )+ brightdata.run_pipeline("amazon_product_search", ["x"], timeout=30)+ assert kwargs.get("tty_only") is Falsediff --git a/tests/test_diagnose_compat.py b/tests/test_diagnose_compat.pyindex e065c48b..3f0d2658 100644--- a/tests/test_diagnose_compat.py+++ b/tests/test_diagnose_compat.py@@ -66,6 +66,8 @@ DIAGNOSE_TOP_KEYS = {"native_search","has_scrapecreators","has_github",+ "brightdata_installed",+ "brightdata_authenticated","x_pending_browser_auth","available_sources","safe",@@ -82,7 +84,10 @@ DIAGNOSE_TOP_KEYS = {DIAGNOSE_PROVIDERS_KEYS = {"google", "openai", "xai", "openrouter", "perplexity"}DIAGNOSE_BROWSER_COOKIES_KEYS = {"mode", "browsers", "reads_values"}-DIAGNOSE_EXTERNAL_COMMANDS_KEYS = {"yt-dlp", "digg-pp-cli", "arxiv-pp-cli", "techmeme-pp-cli", "trustpilot-pp-cli", "gh"}+DIAGNOSE_EXTERNAL_COMMANDS_KEYS = {+ "yt-dlp", "digg-pp-cli", "arxiv-pp-cli", "techmeme-pp-cli", "trustpilot-pp-cli",+ "brightdata", "gh",+}DIAGNOSE_CREDENTIAL_DESTINATIONS_KEYS = {"global_env"}PREFLIGHT_TOP_KEYS = {diff --git a/tests/test_pipeline_v3.py b/tests/test_pipeline_v3.pyindex 1006a320..a509d582 100644--- a/tests/test_pipeline_v3.py+++ b/tests/test_pipeline_v3.py@@ -1667,5 +1667,41 @@ class TestScrapeCreatorsTierGating(unittest.TestCase):self.assertIn("pinterest", avail)+class TestAmazonSourceGating:+ """U2: the amazon source is dual-gated -- CLI available AND requested."""++ def _config(self, include=""):+ return {"INCLUDE_SOURCES": include}++ def test_unavailable_without_the_cli_even_when_requested(self):+ with patch.object(pipeline.brightdata, "is_available", return_value=False):+ available = pipeline.available_sources(self._config(), ["amazon"])+ assert "amazon" not in available++ def test_unavailable_with_the_cli_when_not_requested(self):+ """Installing the CLI for other work must not start spending credits."""+ with patch.object(pipeline.brightdata, "is_available", return_value=True):+ available = pipeline.available_sources(self._config(), None)+ assert "amazon" not in available++ def test_available_via_per_run_request(self):+ with patch.object(pipeline.brightdata, "is_available", return_value=True):+ available = pipeline.available_sources(self._config(), ["amazon"])+ assert "amazon" in available++ def test_available_via_durable_include_sources(self):+ with patch.object(pipeline.brightdata, "is_available", return_value=True):+ available = pipeline.available_sources(self._config("amazon"), None)+ assert "amazon" in available++ def test_search_flag_accepts_the_amazon_token(self):+ import last30days+ assert "amazon" in last30days.parse_search_flag("reddit,x,amazon")++ def test_capped_at_one_fetch_per_run(self):+ """One model-supplied keyword per run: extra streams are pure cost."""+ assert pipeline.MAX_SOURCE_FETCHES["amazon"] == 1++if __name__ == "__main__":unittest.main()diff --git a/tests/test_setup_wizard.py b/tests/test_setup_wizard.pyindex 31ad4c42..7a896ab1 100644--- a/tests/test_setup_wizard.py+++ b/tests/test_setup_wizard.py@@ -833,3 +833,89 @@ class TestSetupSubcommand:args = parser.parse_args(["AI", "video", "tools"])topic = " ".join(args.topic) if args.topic else Noneassert topic.strip().lower() != "setup"+++class TestBrightDataStatusHonesty:+ """U5/R11: setup must never claim active unless the engine gate passes."""++ def _patched(self, *, installed, credentialed, off_path=None):+ available = installed and credentialed+ return (+ patch.object(setup_wizard.brightdata, "is_installed", return_value=installed),+ patch.object(setup_wizard.brightdata, "has_credentials", return_value=credentialed),+ patch.object(setup_wizard.brightdata, "is_available", return_value=available),+ patch.object(setup_wizard, "_brightdata_off_path_binary", return_value=off_path),+ )++ def test_on_path_and_credentialed_reports_engine_active(self):+ a, b, c, d = self._patched(installed=True, credentialed=True)+ with a, b, c, d:+ status = setup_wizard.brightdata_status({})+ assert status["action"] == "already_installed"+ assert status["authenticated"] is True+ assert status["engine_active"] is True++ def test_on_path_without_credentials_is_not_active_and_names_login(self):+ a, b, c, d = self._patched(installed=True, credentialed=False)+ with a, b, c, d:+ status = setup_wizard.brightdata_status({})+ assert status["action"] == "already_installed"+ assert status["authenticated"] is False+ assert status["engine_active"] is False+ assert "brightdata login" in status["hint"]++ def test_off_path_binary_is_reported_with_its_path(self):+ """The Hermes/OpenClaw failure mode: on disk, invisible to the engine."""+ a, b, c, d = self._patched(+ installed=False, credentialed=True, off_path="/Users/x/.npm-global/bin/brightdata"+ )+ with a, b, c, d:+ status = setup_wizard.brightdata_status({})+ assert status["action"] == "installed_off_path"+ assert status["engine_active"] is False+ assert status["path"] == "/Users/x/.npm-global/bin/brightdata"+ assert "PATH" in status["hint"]++ def test_absent_binary_recommends_but_never_installs(self):+ a, b, c, d = self._patched(installed=False, credentialed=False)+ with a, b, c, d, patch.object(setup_wizard.subprocess, "run") as run:+ status = setup_wizard.brightdata_status({})+ assert status["action"] == "not_installed"+ assert status["engine_active"] is False+ run.assert_not_called()++ def test_brightdata_is_excluded_from_auto_installed_pp_sources(self):+ slugs = {slug for _, slug, _ in setup_wizard.PP_DEFAULT_SOURCES}+ assert "brightdata" not in slugs+++class TestBrightDataSetupSurface:+ """The three states must be visible somewhere, or the honesty is moot."""++ def _text(self, status):+ return setup_wizard.get_setup_status_text({+ "cookies_found": {}, "ytdlp_installed": True,+ "ytdlp_action": "already_installed", "digg_installed": True,+ "digg_action": "already_installed", "pp_sources": {},+ "brightdata": status, "env_written": False,+ })++ def test_active_state_is_reported(self):+ text = self._text({"action": "already_installed", "engine_active": True})+ assert "Bright Data CLI ready" in text++ def test_installed_but_not_logged_in_names_the_fix(self):+ text = self._text({"action": "already_installed", "engine_active": False})+ assert "brightdata login" in text++ def test_off_path_reports_the_path_and_the_fix(self):+ text = self._text({+ "action": "installed_off_path", "engine_active": False,+ "path": "/Users/x/.npm-global/bin/brightdata",+ })+ assert "/Users/x/.npm-global/bin/brightdata" in text+ assert "PATH" in text++ def test_absent_offers_the_install_command_without_running_it(self):+ text = self._text({"action": "not_installed", "engine_active": False})+ assert "npm i -g @brightdata/cli" in text