:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(scrunch): add 4 missing tables to the scrunch source (#129126)
Tom Owers5a19c2b104fd
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex e9dac686c355..f9f994c60c76 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -10665,13 +10665,13 @@ Today (5): `brands`, `competitors`, `personas`, `prompts`, `responses`Diffed against: <https://developers.scrunch.com/api-reference/openapi.json>-- [ ] `Agent Traffic (GET /v1/{brand_id}/sites/{site_id}/agent-traffic)` — measure AI crawler traffic by date, page, agent source, and agent type (high)-- [ ] `Sitemap Page Metrics (GET /v1/{brand_id}/sitemap/pages/{page_id}/metrics)` — compare citations, AI referrals, and agent traffic over time for each page (high)-- [ ] `Site Audit Score (GET /v1/{brand_id}/audit-scores/site)` — track site visibility scores, prior scores, and potential improvement (high)+- [x] `Agent Traffic (GET /v1/{brand_id}/sites/{site_id}/agent-traffic)` — measure AI crawler traffic by date, page, agent source, and agent type (high)+- [x] `Sitemap Page Metrics (GET /v1/{brand_id}/sitemap/pages/{page_id}/metrics)` — compare citations, AI referrals, and agent traffic over time for each page (high)+- [x] `Site Audit Score (GET /v1/{brand_id}/audit-scores/site)` — track site visibility scores, prior scores, and potential improvement (high)- [ ] `Sitemap Pages (GET /v1/{brand_id}/sitemap/pages)` — inventory discovered pages and compare audit scores, crawl depth, and optimized content coverage (medium)- [ ] `Page Audits (GET /v1/{brand_id}/page-audits)` — track page audit activity, completion status, and affected URLs (medium)- [ ] `Page Audit Scores (GET /v1/{brand_id}/audit-scores/pages)` — compare page retrieval and citation scores with identified improvement opportunities (medium)-- [ ] `Signals (GET /v1/{brand_id}/signals)` — analyze detected changes in brand and competitor visibility by platform, metric, and confidence (medium)+- [x] `Signals (GET /v1/{brand_id}/signals)` — analyze detected changes in brand and competitor visibility by platform, metric, and confidence (medium)- [ ] `Page Recommendations (GET /v1/{brand_id}/site-optimization/recommendations/by-page)` — prioritize page improvements using recommendations, scores, and existing strengths (medium)Note: The official OpenAPI JSON was fetched with curl and all GET paths, parameters, and response schemas were compared with the source map. Paths include the specification server prefix, /v1. Traffic needs site IDs supplied separately, while page metrics need requests for each sitemap page. Query aggregates duplicate metrics obtainable from imported responses; ingestion writes, deployment operations, and connection configuration are excluded.diff --git a/products/warehouse_sources/sources/scrunch/canonical_descriptions.py b/products/warehouse_sources/sources/scrunch/canonical_descriptions.pyindex 7dd508645d36..cdfd6d02dba5 100644--- a/products/warehouse_sources/sources/scrunch/canonical_descriptions.py+++ b/products/warehouse_sources/sources/scrunch/canonical_descriptions.py@@ -48,6 +48,76 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"description": "The description of the persona.",},},+ "site_audit_score": {+ "description": "The latest site-level AI visibility score for each brand, from its most recent bulk audit.",+ "docs_url": "https://developers.scrunch.com/api-reference/openapi.json",+ "columns": {+ "brand_id": "The identifier of the brand.",+ "model_version": "The version of the scoring model.",+ "generated_at": "The time when Scrunch scored the site.",+ "n_pages": "The number of pages in the audit.",+ "total_prompts": "The number of prompts in the audit.",+ "score": "The retrievability and citability scores, both from 0 to 100.",+ "previous": "The scores from the previous audit run, if comparable.",+ "previous_absent_reason": "Why no previous score is present: no_prior_run or not_comparable.",+ "worst_page_url": "The URL of the page with the lowest score.",+ "best_page_url": "The URL of the page with the highest score.",+ "headroom": "The projected site score per metric if every surfaced opportunity is fixed.",+ "selection_changed_since_audit": "Whether the page selection changed after the audit.",+ },+ },+ "agent_traffic": {+ "description": "Daily AI agent and bot requests to each site with agent traffic enabled, by path, agent source, and agent type.",+ "docs_url": "https://developers.scrunch.com/api-reference/openapi.json",+ "columns": {+ "brand_id": "The identifier of the brand that owns the site.",+ "site_id": "The identifier of the site.",+ "date": "The day of the traffic, in YYYY-MM-DD format.",+ "site": "The registered domain of the site, in lowercase, without scheme or www.",+ "path": "The URL path on the site.",+ "agent_source": "The origin of the agent, for example OpenAI, Google, or Anthropic.",+ "agent_type": "The agent classification, for example GPTBot, ClaudeBot, or Googlebot.",+ "requests": "The number of requests for this combination of dimensions.",+ },+ },+ "sitemap_page_metrics": {+ "description": "Citations, AI referrals, and AI agent traffic for each sitemap page over the last 30 days, by AI platform.",+ "docs_url": "https://developers.scrunch.com/api-reference/openapi.json",+ "columns": {+ "brand_id": "The identifier of the brand.",+ "page_id": "The identifier of the sitemap page.",+ "page_url": "The full URL of the page.",+ "granularity": "The bucket size that Scrunch chose for the date range: daily or weekly.",+ "metric": "The metric: citations, ai_referrals, or agent_traffic.",+ "date": "The start date of the bucket, in YYYY-MM-DD format.",+ "platform": "The AI platform, for example OpenAI, Perplexity, or Gemini.",+ "count": "The count for this metric, bucket, and platform.",+ },+ },+ "signals": {+ "description": "Statistically tested changes in brand and competitor AI visibility, from the nightly detection sweep.",+ "docs_url": "https://developers.scrunch.com/api-reference/openapi.json",+ "columns": {+ "id": "The identifier of one detection of the signal.",+ "brand_id": "The identifier of the brand.",+ "detected_for_date": "The anchor date of the nightly detection.",+ "fingerprint": "The stable identity of the underlying issue across nightly re-detections.",+ "subject_kind": "Whose movement the signal describes: brand or competitor.",+ "alert_type": "The detection kind: level_change or trend.",+ "scope": "The slice granularity: account, account_platform, topic, or topic_platform.",+ "metric": "The metric the signal fired on, for example presence_rate.",+ "platform": "The AI platform. A value of (multi) means the slice spans several platforms.",+ "direction": "The direction of the movement: up, down, or none.",+ "tier": "The confidence tier, for example high, confident, worth_a_look, or provisional.",+ "current_value": "The metric value over the current window, as a rate from 0 to 1.",+ "baseline_value": "The metric value over the baseline window, as a rate from 0 to 1.",+ "delta_absolute": "The current value minus the baseline value.",+ "score": "The engine priority score.",+ "narrative_what": "A one-line headline of what changed.",+ "slice": "The platforms, topics, and country that the signal covers.",+ "created_at": "The UTC time when Scrunch created the signal row.",+ },+ },"responses": {"description": "AI answers to tracked prompts, with brand evaluations, citations, and competitor evaluations.","docs_url": "https://developers.scrunch.com/api-reference/responses/overview",diff --git a/products/warehouse_sources/sources/scrunch/scrunch.py b/products/warehouse_sources/sources/scrunch/scrunch.pyindex f0901ad9ac93..e8209a9d725a 100644--- a/products/warehouse_sources/sources/scrunch/scrunch.py+++ b/products/warehouse_sources/sources/scrunch/scrunch.py@@ -1,17 +1,27 @@-from datetime import UTC, datetime+from datetime import UTC, datetime, timedelta+from functools import partialfrom typing import Anyfrom requests import HTTPErrorfrom products.warehouse_sources.sources.scrunch.settings import (+ AGENT_TRAFFIC_DIMENSIONS,+ AGENT_TRAFFIC_HISTORY_DAYS,+ AGENT_TRAFFIC_PAGE_SIZE,BASE_URL,ENDPOINTS,+ FANOUT_PARENTS,+ INCREMENTAL_FIELDS,NON_RETRYABLE_ERRORS,PAGE_SIZE,PRIMARY_KEYS,+ SIGNAL_ANCHOR_LIMIT,+ SITEMAP_METRICS,+ FanoutParent,)from products.warehouse_sources.sources.sdk import (BearerTokenAuth,+ Endpoint,EndpointResource,RESTAPIConfig,RESTClient,@@ -20,6 +30,7 @@ from products.warehouse_sources.sources.sdk import (SourceInputs,SourceResponse,frozen,+ make_parent_key_name,parse_datetime_value,rename_parent_fields,rest_api_resources,@@ -61,54 +72,177 @@ def validate_credentials(api_key: str) -> tuple[bool, str | None]:return True, None+def _watermark_date(inputs: SourceInputs) -> str | None:+ if not inputs.should_use_incremental_field or inputs.db_incremental_field_last_value is None:+ return None+ watermark = parse_datetime_value(inputs.db_incremental_field_last_value)+ if watermark is None:+ raise ValueError(f"Invalid Scrunch {inputs.schema_name} watermark. Reset this table's sync.")+ # Date filters include the whole watermark day; merging prevents duplicate rows.+ return watermark.date().isoformat()+++@frozen+class _DateWindow:+ start_date: str | None+ end_date: str | None+++def _date_window(inputs: SourceInputs) -> _DateWindow:+ today = datetime.now(UTC).date()+ start_date = _watermark_date(inputs) if inputs.schema_name in INCREMENTAL_FIELDS else None+ if inputs.schema_name == "responses":+ # Exclude the open UTC day so new responses cannot move offsets during a sync.+ return _DateWindow(start_date=start_date, end_date=today.isoformat())+ if inputs.schema_name == "agent_traffic":+ if start_date is None:+ history_start = (+ inputs.history_start.date()+ if inputs.history_start+ else today - timedelta(days=AGENT_TRAFFIC_HISTORY_DAYS)+ )+ start_date = history_start.isoformat()+ return _DateWindow(start_date=start_date, end_date=today.isoformat())+ return _DateWindow(start_date=start_date, end_date=None)+++BRAND_KEY = make_parent_key_name("brands", "id")+++def _explode_page_metrics(row: dict[str, Any]) -> list[dict[str, Any]]:+ page = {+ "brand_id": row[make_parent_key_name("sitemap_pages", BRAND_KEY)],+ "page_id": row[make_parent_key_name("sitemap_pages", "id")],+ "page_url": row[make_parent_key_name("sitemap_pages", "url")],+ "granularity": row.get("granularity"),+ }+ metrics = row.get("metrics") or {}+ return [+ {**page, "metric": metric, "date": bucket["date"], "platform": platform, "count": count}+ for metric in SITEMAP_METRICS+ for bucket in metrics.get(metric) or []+ for platform, count in (bucket.get("count_by_platform") or {}).items()+ ]+++def _detection_days(start_date: str | None, row: dict[str, Any]) -> list[dict[str, Any]]:+ return [+ {"date": day, BRAND_KEY: row[BRAND_KEY]}+ for day in row.get("dates") or []+ if start_date is None or day >= start_date+ ]+++def _middle_resource(parent: FanoutParent, window: _DateWindow) -> EndpointResource:+ params: dict[str, Any] = {"brand_id": {"type": "resolve", "resource": "brands", "field": "id"}}+ endpoint: Endpoint = {"path": parent.path, "params": params}+ middle: EndpointResource = {"name": parent.name, "include_from_parent": ["id"], "endpoint": endpoint}+ if parent.name == "sites":+ # The sites list takes no paging params and returns every site at once.+ endpoint["paginator"] = "single_page"+ # Other sites of the brand return 404 on the agent traffic route.+ middle["data_map"] = lambda site: site if site.get("agent_traffic_active") else []+ elif parent.name == "sitemap_pages":+ # A brand without a completed sitemap crawl answers 404.+ endpoint["response_actions"] = [{"status_code": 404, "action": "ignore"}]+ else:+ params["limit"] = SIGNAL_ANCHOR_LIMIT+ endpoint |= {"paginator": "single_page", "data_selector": None, "data_selector_required": False}+ middle["data_map"] = partial(_detection_days, window.start_date)+ return middle+++def _child_endpoint(schema_name: str, path: str, window: _DateWindow) -> Endpoint:+ parent = FANOUT_PARENTS.get(schema_name)+ params: dict[str, Any] = {"brand_id": {"type": "resolve", "resource": "brands", "field": "id"}}+ if parent is not None:+ params = {+ "brand_id": {"type": "resolve", "resource": parent.name, "field": BRAND_KEY},+ parent.param: {"type": "resolve", "resource": parent.name, "field": parent.field},+ }+ endpoint: Endpoint = {"path": path, "params": params}+ if schema_name == "prompts":+ params["status"] = "all"+ elif schema_name == "responses":+ params["end_date"] = window.end_date+ if window.start_date is not None:+ params["start_date"] = window.start_date+ elif schema_name == "signals":+ params["sort"] = "detected_desc"+ # A path that ends in a placeholder defaults to a single page.+ endpoint["paginator"] = {"type": "offset", "limit": PAGE_SIZE, "total_path": "total"}+ elif schema_name == "agent_traffic":+ params |= {+ "start_date": window.start_date,+ "end_date": window.end_date,+ "fields": ",".join(AGENT_TRAFFIC_DIMENSIONS),+ "time_bucket": "day",+ }+ # The response carries no total, so a short page ends the walk.+ endpoint["paginator"] = {"type": "offset", "limit": AGENT_TRAFFIC_PAGE_SIZE, "total_path": None}+ endpoint["data_selector"] = "data"+ elif schema_name in ("site_audit_score", "sitemap_page_metrics"):+ # One object per request; the site score is null for a brand with no audit yet.+ endpoint |= {"paginator": "single_page", "data_selector": None, "data_selector_required": False}+ return endpoint+++def _child_resources(schema_name: str, path: str, window: _DateWindow) -> list[EndpointResource]:+ endpoint = _child_endpoint(schema_name, path, window)+ parent = FANOUT_PARENTS.get(schema_name)+ if parent is None:+ return [+ {+ "name": schema_name,+ "include_from_parent": ["id"],+ "endpoint": endpoint,+ "data_map": rename_parent_fields("brands", {"id": "brand_id"}),+ }+ ]++ child: EndpointResource = {"name": schema_name, "endpoint": endpoint}+ if schema_name == "agent_traffic":+ child["include_from_parent"] = ["id", BRAND_KEY]+ child["data_map"] = rename_parent_fields(parent.name, {"id": "site_id", BRAND_KEY: "brand_id"})+ elif schema_name == "sitemap_page_metrics":+ child["include_from_parent"] = ["id", "url", BRAND_KEY]+ child["data_map"] = _explode_page_metrics+ else:+ child["include_from_parent"] = [BRAND_KEY]+ child["data_map"] = rename_parent_fields(parent.name, {BRAND_KEY: "brand_id"})+ return [_middle_resource(parent, window), child]++def scrunch_source(api_key: str,inputs: SourceInputs,manager: ResumableSourceManager[ScrunchResumeConfig],) -> SourceResponse:- path = schema_for_resource(ENDPOINTS, inputs.schema_name)- is_responses = inputs.schema_name == "responses"+ schema_name = inputs.schema_name+ path = schema_for_resource(ENDPOINTS, schema_name)+ is_responses = schema_name == "responses"+ is_partitioned = schema_name in ("responses", "signals")resume = manager.load_state() if manager.can_resume() else None- start_date = None- # Exclude the open UTC day so new responses cannot move offsets during a sync.- end_date = datetime.now(UTC).date().isoformat() if is_responses else None- if resume is not None:- start_date, end_date = resume.start_date, resume.end_date- elif is_responses and inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:- watermark = parse_datetime_value(inputs.db_incremental_field_last_value)- if watermark is None:- raise ValueError("Invalid Scrunch response watermark. Reset this table's sync.")- # Date filters include the whole watermark day; merging prevents duplicate rows.- start_date = watermark.date().isoformat()+ window = (+ _DateWindow(start_date=resume.start_date, end_date=resume.end_date)+ if resume is not None+ else _date_window(inputs)+ )def save_checkpoint(state: dict[str, Any] | None) -> None:manager.save_state(ScrunchResumeConfig(paginator_state=state,- start_date=start_date,- end_date=end_date,+ start_date=window.start_date,+ end_date=window.end_date,complete=state is None,))parent: EndpointResource = {"name": "brands", "endpoint": {"path": ENDPOINTS["brands"]}}resources: list[str | EndpointResource] = [parent]- if inputs.schema_name != "brands":- params: dict[str, Any] = {"brand_id": {"type": "resolve", "resource": "brands", "field": "id"}}- if inputs.schema_name == "prompts":- params["status"] = "all"- if is_responses:- params["end_date"] = end_date- if start_date is not None:- params["start_date"] = start_date- resources.append(- {- "name": inputs.schema_name,- "include_from_parent": ["id"],- "endpoint": {"path": path, "params": params},- "data_map": rename_parent_fields("brands", {"id": "brand_id"}),- }- )+ if schema_name != "brands":+ resources.extend(_child_resources(schema_name, path, window))config: RESTAPIConfig = {"client": {@@ -132,14 +266,14 @@ def scrunch_source(resume_hook=save_checkpoint,initial_paginator_state=resume.paginator_state if resume else None,)- if resource.name == inputs.schema_name+ if resource.name == schema_name)return SourceResponse(- name=inputs.schema_name,+ name=schema_name,items=(lambda: []) if resume and resume.complete else (lambda: resource),- primary_keys=PRIMARY_KEYS[inputs.schema_name],+ primary_keys=PRIMARY_KEYS[schema_name],sort_mode="desc" if is_responses else None,- partition_keys=["created_at"] if is_responses else None,- partition_mode="datetime" if is_responses else None,- partition_format="month" if is_responses else None,+ partition_keys=["created_at"] if is_partitioned else None,+ partition_mode="datetime" if is_partitioned else None,+ partition_format="month" if is_partitioned else None,)diff --git a/products/warehouse_sources/sources/scrunch/settings.py b/products/warehouse_sources/sources/scrunch/settings.pyindex f5f733278f28..867574e66f29 100644--- a/products/warehouse_sources/sources/scrunch/settings.py+++ b/products/warehouse_sources/sources/scrunch/settings.py@@ -1,18 +1,53 @@-from products.warehouse_sources.sources.sdk import incremental_field+from products.warehouse_sources.sources.sdk import IncrementalFieldType, frozen, incremental_fieldAPI_VERSION = "v1"BASE_URL = f"https://api.scrunchai.com/{API_VERSION}"PAGE_SIZE = 100+AGENT_TRAFFIC_PAGE_SIZE = 10000+AGENT_TRAFFIC_HISTORY_DAYS = 365+AGENT_TRAFFIC_DIMENSIONS = ("date", "site", "path", "agent_source", "agent_type")+SITEMAP_METRICS = ("citations", "ai_referrals", "agent_traffic")+SIGNAL_ANCHOR_LIMIT = 365ENDPOINTS = {"brands": "brands","prompts": "{brand_id}/prompts","competitors": "brands/{brand_id}/competitors","personas": "brands/{brand_id}/personas",+ "site_audit_score": "{brand_id}/audit-scores/site",+ "agent_traffic": "{brand_id}/sites/{site_id}/agent-traffic",+ "sitemap_page_metrics": "{brand_id}/sitemap/pages/{page_id}/metrics",+ # Over a multi-day range the API keeps only the latest detection per fingerprint, so read one day at a time.+ "signals": "{brand_id}/signals?anchor_from={anchor}&anchor_to={anchor}","responses": "{brand_id}/responses",}-INCREMENTAL_FIELDS = {"responses": [incremental_field("created_at")]}-PRIMARY_KEYS = {name: ["id"] if name == "brands" else ["brand_id", "id"] for name in ENDPOINTS}+++@frozen+class FanoutParent:+ name: str+ path: str+ param: str+ field: str+++FANOUT_PARENTS = {+ "agent_traffic": FanoutParent(name="sites", path="{brand_id}/sites", param="site_id", field="id"),+ "sitemap_page_metrics": FanoutParent(+ name="sitemap_pages", path="{brand_id}/sitemap/pages", param="page_id", field="id"+ ),+ "signals": FanoutParent(name="signal_anchors", path="{brand_id}/signals/anchors", param="anchor", field="date"),+}+INCREMENTAL_FIELDS = {+ "responses": [incremental_field("created_at")],+ "signals": [incremental_field("detected_for_date", IncrementalFieldType.Date)],+ "agent_traffic": [incremental_field("date", IncrementalFieldType.Date)],+}+PRIMARY_KEYS = {name: ["id"] if name == "brands" else ["brand_id", "id"] for name in ENDPOINTS} | {+ "site_audit_score": ["brand_id"],+ "agent_traffic": ["brand_id", "site_id", *AGENT_TRAFFIC_DIMENSIONS],+ "sitemap_page_metrics": ["brand_id", "page_id", "metric", "date", "platform"],+}AUTH_ERROR = "Scrunch rejected the API key. Check the key in your organization's API Keys menu."PERMISSION_ERROR = (diff --git a/products/warehouse_sources/sources/scrunch/source.py b/products/warehouse_sources/sources/scrunch/source.pyindex c53579ad9901..4ebe8859c8eb 100644--- a/products/warehouse_sources/sources/scrunch/source.py+++ b/products/warehouse_sources/sources/scrunch/source.py@@ -1,10 +1,13 @@+from datetime import timedeltafrom typing import castfrom products.warehouse_sources.sources.scrunch._config import ScrunchSourceConfigfrom products.warehouse_sources.sources.scrunch.scrunch import ScrunchResumeConfig, scrunch_source, validate_credentialsfrom products.warehouse_sources.sources.scrunch.settings import (+ AGENT_TRAFFIC_HISTORY_DAYS,API_VERSION,ENDPOINTS,+ FANOUT_PARENTS,INCREMENTAL_FIELDS,NON_RETRYABLE_ERRORS,)@@ -41,6 +44,16 @@ class ScrunchSource(ResumableSource[ScrunchSourceConfig, ScrunchResumeConfig]):def source_type(self) -> ExternalDataSourceType:return ExternalDataSourceType.SCRUNCH+ def history_lookback_for_schema(+ self, schema_name: str, config: ScrunchSourceConfig | None = None+ ) -> timedelta | None:+ # Agent traffic requires a start date; the other tables return every row.+ return timedelta(days=AGENT_TRAFFIC_HISTORY_DAYS) if schema_name == "agent_traffic" else None++ def resume_covers_run(self, *, incremental_or_append: bool, schema_name: str | None = None) -> bool:+ # A two-level fan-out carries no resume state, so a retry reads it again from the start.+ return schema_name not in FANOUT_PARENTS+def get_non_retryable_errors(self) -> dict[str, str | None]:return NON_RETRYABLE_ERRORS@@ -58,7 +71,7 @@ class ScrunchSource(ResumableSource[ScrunchSourceConfig, ScrunchResumeConfig]):force_refresh: bool = False,api_version: str | None = None,) -> list[SourceSchema]:- return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names, merge_only={"responses"})+ return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names, merge_only=set(INCREMENTAL_FIELDS))def validate_credentials(self,diff --git a/products/warehouse_sources/sources/scrunch/tests/test_scrunch.py b/products/warehouse_sources/sources/scrunch/tests/test_scrunch.pyindex 3e0b35ae5a12..0f596e770015 100644--- a/products/warehouse_sources/sources/scrunch/tests/test_scrunch.py+++ b/products/warehouse_sources/sources/scrunch/tests/test_scrunch.py@@ -1,4 +1,4 @@-from datetime import UTC, datetime+from datetime import UTC, date, datetime, timedeltafrom typing import Anyimport pytest@@ -16,6 +16,7 @@ from products.warehouse_sources.sources.sdk.testing import DriveResult, Scripted@pytest.fixture(autouse=True)def small_pages(monkeypatch: pytest.MonkeyPatch) -> None:monkeypatch.setattr(scrunch, "PAGE_SIZE", 2)+ monkeypatch.setattr(scrunch, "AGENT_TRAFFIC_PAGE_SIZE", 2)def page(rows: list[dict[str, Any]], total: int, status: int = 200) -> ScriptedResponse:@@ -249,3 +250,152 @@ def test_invalid_watermark_fails_before_requests() -> None:assert isinstance(result.raised, ValueError)assert "watermark" in str(result.raised)assert result.requests == []+++SITE_A = "01AAAAAAAAAAAAAAAAAAAAAAAA"+SITE_B = "01BBBBBBBBBBBBBBBBBBBBBBBB"+++def traffic(rows: list[dict[str, Any]]) -> ScriptedResponse:+ return ScriptedResponse(json={"meta": {"start_date": "x", "end_date": "y", "time_bucket": "day"}, "data": rows})+++ "inputs,expected_start",+ [+ ({}, None),+ ({"history_start": datetime(2026, 1, 5, 12, tzinfo=UTC)}, "2026-01-05"),+ (+ {+ "incremental_field": "date",+ "db_incremental_field_last_value": "2026-08-01",+ "history_start": datetime(2026, 1, 5, tzinfo=UTC),+ },+ "2026-08-01",+ ),+ ({"incremental_field": "date", "db_incremental_field_last_value": date(2026, 8, 1)}, "2026-08-01"),+ ],+)+def test_agent_traffic_walks_active_sites_until_a_short_page(+ inputs: dict[str, Any], expected_start: str | None+) -> None:+ today = datetime.now(UTC).date()+ rows = [{"requests": n, "date": "2026-10-01", "path": f"/{n}", "agent_source": None} for n in range(3)]++ result = run(+ "agent_traffic",+ [+ page([{"id": 11}], 1),+ page([{"id": SITE_A, "agent_traffic_active": False}, {"id": SITE_B, "agent_traffic_active": True}], 2),+ traffic(rows[:2]),+ traffic(rows[2:]),+ ],+ **inputs,+ )++ assert result.raised is None+ assert result.rows == [{**row, "brand_id": 11, "site_id": SITE_B} for row in rows]+ assert result.paths == ["/v1/brands", "/v1/11/sites", *[f"/v1/11/sites/{SITE_B}/agent-traffic"] * 2]+ assert query(result, 1) == {}+ start = expected_start or (today - timedelta(days=365)).isoformat()+ for index, offset in ((2, "0"), (3, "2")):+ assert query(result, index) == {+ "offset": [offset],+ "limit": ["2"],+ "start_date": [start],+ "end_date": [today.isoformat()],+ "fields": ["date,site,path,agent_source,agent_type"],+ "time_bucket": ["day"],+ }+++def test_sitemap_page_metrics_explode_into_one_row_per_platform_bucket() -> None:+ def metrics(**buckets: list[dict[str, Any]]) -> ScriptedResponse:+ empty: dict[str, list[dict[str, Any]]] = {"citations": [], "ai_referrals": [], "agent_traffic": []}+ return ScriptedResponse(json={"granularity": "daily", "metrics": empty | buckets})++ result = run(+ "sitemap_page_metrics",+ [+ page([{"id": 10}, {"id": 11}], 2),+ ScriptedResponse(status=404, json={"detail": "No completed sitemap crawl found"}),+ page([{"id": 5, "url": "https://example.com/a"}, {"id": 6, "url": "https://example.com/b"}], 2),+ metrics(+ citations=[{"date": "2026-10-01", "count_by_platform": {"OpenAI": 2, "Gemini": 1}}],+ ai_referrals=[{"date": "2026-10-01", "count_by_platform": {}}],+ ),+ metrics(agent_traffic=[{"date": "2026-10-02", "count_by_platform": {"OpenAI": 7}}]),+ ],+ )++ assert result.raised is None+ page_a = {"brand_id": 11, "page_id": 5, "page_url": "https://example.com/a", "granularity": "daily"}+ page_b = {"brand_id": 11, "page_id": 6, "page_url": "https://example.com/b", "granularity": "daily"}+ assert result.rows == [+ {**page_a, "metric": "citations", "date": "2026-10-01", "platform": "OpenAI", "count": 2},+ {**page_a, "metric": "citations", "date": "2026-10-01", "platform": "Gemini", "count": 1},+ {**page_b, "metric": "agent_traffic", "date": "2026-10-02", "platform": "OpenAI", "count": 7},+ ]+ assert result.paths[1:] == [+ "/v1/10/sitemap/pages",+ "/v1/11/sitemap/pages",+ "/v1/11/sitemap/pages/5/metrics",+ "/v1/11/sitemap/pages/6/metrics",+ ]+ assert query(result, 2) == {"offset": ["0"], "limit": ["2"]}+ assert query(result, 3) == query(result, 4) == {}+++def test_site_audit_score_keeps_one_row_per_audited_brand() -> None:+ score = {"brand_id": 11, "generated_at": "2026-10-01T00:00:00Z", "score": {"retrievability": 70, "citability": 50}}++ result = run(+ "site_audit_score",+ [page([{"id": 11}, {"id": 22}], 2), ScriptedResponse(json=score), ScriptedResponse(body=b"null")],+ )++ assert result.raised is None+ assert result.rows == [score]+ assert result.paths == ["/v1/brands", "/v1/11/audit-scores/site", "/v1/22/audit-scores/site"]+ assert query(result, 1) == query(result, 2) == {}+++ "inputs,expected_days",+ [+ ({}, ["2026-08-03", "2026-08-02", "2026-08-01"]),+ (+ {"incremental_field": "detected_for_date", "db_incremental_field_last_value": "2026-08-02"},+ ["2026-08-03", "2026-08-02"],+ ),+ (+ {"incremental_field": "detected_for_date", "db_incremental_field_last_value": date(2026, 8, 2)},+ ["2026-08-03", "2026-08-02"],+ ),+ ],+)+def test_signals_read_each_detection_day_on_its_own(inputs: dict[str, Any], expected_days: list[str]) -> None:+ detections = {+ "2026-08-03": [{"id": 3, "fingerprint": "f1"}, {"id": 4, "fingerprint": "f2"}, {"id": 5, "fingerprint": "f3"}],+ "2026-08-02": [{"id": 2, "fingerprint": "f1"}],+ "2026-08-01": [{"id": 1, "fingerprint": "f1"}],+ }+ script = [+ page([{"id": 11}], 1),+ ScriptedResponse(json={"dates": ["2026-08-03", "2026-08-02", "2026-08-01"]}),+ ]+ for day in expected_days:+ rows = detections[day]+ script += [page(rows[index : index + 2], len(rows)) for index in range(0, len(rows), 2)]++ result = run("signals", script, **inputs)++ assert result.raised is None+ assert result.rows == [{**row, "brand_id": 11} for day in expected_days for row in detections[day]]+ assert result.paths[1] == "/v1/11/signals/anchors"+ assert query(result, 1) == {"limit": ["365"]}+ signal_queries = [query(result, index) for index in range(2, len(result.requests))]+ assert [(q["anchor_from"], q["anchor_to"], q["offset"]) for q in signal_queries] == [+ ([day], [day], [str(offset)]) for day in expected_days for offset in range(0, len(detections[day]), 2)+ ]+ assert all(q["sort"] == ["detected_desc"] for q in signal_queries)