Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(scrunch): add 4 missing tables to the scrunch source (#129126)

Tom Owers5a19c2b104fd
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index e9dac686c355..f9f994c60c76 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -10665,13 +10665,13 @@ Today (5): `brands`, `competitors`, `personas`, `prompts`, `responses`
Diffed against: <https://developers.scrunch.com/api-reference/openapi.json>
-- [ ] `Agent Traffic (GET /v1/{brand_id}/sites/{site_id}/agent-traffic)` — measure AI crawler traffic by date, page, agent source, and agent type (high)
-- [ ] `Sitemap Page Metrics (GET /v1/{brand_id}/sitemap/pages/{page_id}/metrics)` — compare citations, AI referrals, and agent traffic over time for each page (high)
-- [ ] `Site Audit Score (GET /v1/{brand_id}/audit-scores/site)` — track site visibility scores, prior scores, and potential improvement (high)
+- [x] `Agent Traffic (GET /v1/{brand_id}/sites/{site_id}/agent-traffic)` — measure AI crawler traffic by date, page, agent source, and agent type (high)
+- [x] `Sitemap Page Metrics (GET /v1/{brand_id}/sitemap/pages/{page_id}/metrics)` — compare citations, AI referrals, and agent traffic over time for each page (high)
+- [x] `Site Audit Score (GET /v1/{brand_id}/audit-scores/site)` — track site visibility scores, prior scores, and potential improvement (high)
- [ ] `Sitemap Pages (GET /v1/{brand_id}/sitemap/pages)` — inventory discovered pages and compare audit scores, crawl depth, and optimized content coverage (medium)
- [ ] `Page Audits (GET /v1/{brand_id}/page-audits)` — track page audit activity, completion status, and affected URLs (medium)
- [ ] `Page Audit Scores (GET /v1/{brand_id}/audit-scores/pages)` — compare page retrieval and citation scores with identified improvement opportunities (medium)
-- [ ] `Signals (GET /v1/{brand_id}/signals)` — analyze detected changes in brand and competitor visibility by platform, metric, and confidence (medium)
+- [x] `Signals (GET /v1/{brand_id}/signals)` — analyze detected changes in brand and competitor visibility by platform, metric, and confidence (medium)
- [ ] `Page Recommendations (GET /v1/{brand_id}/site-optimization/recommendations/by-page)` — prioritize page improvements using recommendations, scores, and existing strengths (medium)
Note: The official OpenAPI JSON was fetched with curl and all GET paths, parameters, and response schemas were compared with the source map. Paths include the specification server prefix, /v1. Traffic needs site IDs supplied separately, while page metrics need requests for each sitemap page. Query aggregates duplicate metrics obtainable from imported responses; ingestion writes, deployment operations, and connection configuration are excluded.
diff --git a/products/warehouse_sources/sources/scrunch/canonical_descriptions.py b/products/warehouse_sources/sources/scrunch/canonical_descriptions.py
index 7dd508645d36..cdfd6d02dba5 100644
--- a/products/warehouse_sources/sources/scrunch/canonical_descriptions.py
+++ b/products/warehouse_sources/sources/scrunch/canonical_descriptions.py
@@ -48,6 +48,76 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"description": "The description of the persona.",
},
},
+ "site_audit_score": {
+ "description": "The latest site-level AI visibility score for each brand, from its most recent bulk audit.",
+ "docs_url": "https://developers.scrunch.com/api-reference/openapi.json",
+ "columns": {
+ "brand_id": "The identifier of the brand.",
+ "model_version": "The version of the scoring model.",
+ "generated_at": "The time when Scrunch scored the site.",
+ "n_pages": "The number of pages in the audit.",
+ "total_prompts": "The number of prompts in the audit.",
+ "score": "The retrievability and citability scores, both from 0 to 100.",
+ "previous": "The scores from the previous audit run, if comparable.",
+ "previous_absent_reason": "Why no previous score is present: no_prior_run or not_comparable.",
+ "worst_page_url": "The URL of the page with the lowest score.",
+ "best_page_url": "The URL of the page with the highest score.",
+ "headroom": "The projected site score per metric if every surfaced opportunity is fixed.",
+ "selection_changed_since_audit": "Whether the page selection changed after the audit.",
+ },
+ },
+ "agent_traffic": {
+ "description": "Daily AI agent and bot requests to each site with agent traffic enabled, by path, agent source, and agent type.",
+ "docs_url": "https://developers.scrunch.com/api-reference/openapi.json",
+ "columns": {
+ "brand_id": "The identifier of the brand that owns the site.",
+ "site_id": "The identifier of the site.",
+ "date": "The day of the traffic, in YYYY-MM-DD format.",
+ "site": "The registered domain of the site, in lowercase, without scheme or www.",
+ "path": "The URL path on the site.",
+ "agent_source": "The origin of the agent, for example OpenAI, Google, or Anthropic.",
+ "agent_type": "The agent classification, for example GPTBot, ClaudeBot, or Googlebot.",
+ "requests": "The number of requests for this combination of dimensions.",
+ },
+ },
+ "sitemap_page_metrics": {
+ "description": "Citations, AI referrals, and AI agent traffic for each sitemap page over the last 30 days, by AI platform.",
+ "docs_url": "https://developers.scrunch.com/api-reference/openapi.json",
+ "columns": {
+ "brand_id": "The identifier of the brand.",
+ "page_id": "The identifier of the sitemap page.",
+ "page_url": "The full URL of the page.",
+ "granularity": "The bucket size that Scrunch chose for the date range: daily or weekly.",
+ "metric": "The metric: citations, ai_referrals, or agent_traffic.",
+ "date": "The start date of the bucket, in YYYY-MM-DD format.",
+ "platform": "The AI platform, for example OpenAI, Perplexity, or Gemini.",
+ "count": "The count for this metric, bucket, and platform.",
+ },
+ },
+ "signals": {
+ "description": "Statistically tested changes in brand and competitor AI visibility, from the nightly detection sweep.",
+ "docs_url": "https://developers.scrunch.com/api-reference/openapi.json",
+ "columns": {
+ "id": "The identifier of one detection of the signal.",
+ "brand_id": "The identifier of the brand.",
+ "detected_for_date": "The anchor date of the nightly detection.",
+ "fingerprint": "The stable identity of the underlying issue across nightly re-detections.",
+ "subject_kind": "Whose movement the signal describes: brand or competitor.",
+ "alert_type": "The detection kind: level_change or trend.",
+ "scope": "The slice granularity: account, account_platform, topic, or topic_platform.",
+ "metric": "The metric the signal fired on, for example presence_rate.",
+ "platform": "The AI platform. A value of (multi) means the slice spans several platforms.",
+ "direction": "The direction of the movement: up, down, or none.",
+ "tier": "The confidence tier, for example high, confident, worth_a_look, or provisional.",
+ "current_value": "The metric value over the current window, as a rate from 0 to 1.",
+ "baseline_value": "The metric value over the baseline window, as a rate from 0 to 1.",
+ "delta_absolute": "The current value minus the baseline value.",
+ "score": "The engine priority score.",
+ "narrative_what": "A one-line headline of what changed.",
+ "slice": "The platforms, topics, and country that the signal covers.",
+ "created_at": "The UTC time when Scrunch created the signal row.",
+ },
+ },
"responses": {
"description": "AI answers to tracked prompts, with brand evaluations, citations, and competitor evaluations.",
"docs_url": "https://developers.scrunch.com/api-reference/responses/overview",
diff --git a/products/warehouse_sources/sources/scrunch/scrunch.py b/products/warehouse_sources/sources/scrunch/scrunch.py
index f0901ad9ac93..e8209a9d725a 100644
--- a/products/warehouse_sources/sources/scrunch/scrunch.py
+++ b/products/warehouse_sources/sources/scrunch/scrunch.py
@@ -1,17 +1,27 @@
-from datetime import UTC, datetime
+from datetime import UTC, datetime, timedelta
+from functools import partial
from typing import Any
from requests import HTTPError
from products.warehouse_sources.sources.scrunch.settings import (
+ AGENT_TRAFFIC_DIMENSIONS,
+ AGENT_TRAFFIC_HISTORY_DAYS,
+ AGENT_TRAFFIC_PAGE_SIZE,
BASE_URL,
ENDPOINTS,
+ FANOUT_PARENTS,
+ INCREMENTAL_FIELDS,
NON_RETRYABLE_ERRORS,
PAGE_SIZE,
PRIMARY_KEYS,
+ SIGNAL_ANCHOR_LIMIT,
+ SITEMAP_METRICS,
+ FanoutParent,
)
from products.warehouse_sources.sources.sdk import (
BearerTokenAuth,
+ Endpoint,
EndpointResource,
RESTAPIConfig,
RESTClient,
@@ -20,6 +30,7 @@ from products.warehouse_sources.sources.sdk import (
SourceInputs,
SourceResponse,
frozen,
+ make_parent_key_name,
parse_datetime_value,
rename_parent_fields,
rest_api_resources,
@@ -61,54 +72,177 @@ def validate_credentials(api_key: str) -> tuple[bool, str | None]:
return True, None
+def _watermark_date(inputs: SourceInputs) -> str | None:
+ if not inputs.should_use_incremental_field or inputs.db_incremental_field_last_value is None:
+ return None
+ watermark = parse_datetime_value(inputs.db_incremental_field_last_value)
+ if watermark is None:
+ raise ValueError(f"Invalid Scrunch {inputs.schema_name} watermark. Reset this table's sync.")
+ # Date filters include the whole watermark day; merging prevents duplicate rows.
+ return watermark.date().isoformat()
+
+
+@frozen
+class _DateWindow:
+ start_date: str | None
+ end_date: str | None
+
+
+def _date_window(inputs: SourceInputs) -> _DateWindow:
+ today = datetime.now(UTC).date()
+ start_date = _watermark_date(inputs) if inputs.schema_name in INCREMENTAL_FIELDS else None
+ if inputs.schema_name == "responses":
+ # Exclude the open UTC day so new responses cannot move offsets during a sync.
+ return _DateWindow(start_date=start_date, end_date=today.isoformat())
+ if inputs.schema_name == "agent_traffic":
+ if start_date is None:
+ history_start = (
+ inputs.history_start.date()
+ if inputs.history_start
+ else today - timedelta(days=AGENT_TRAFFIC_HISTORY_DAYS)
+ )
+ start_date = history_start.isoformat()
+ return _DateWindow(start_date=start_date, end_date=today.isoformat())
+ return _DateWindow(start_date=start_date, end_date=None)
+
+
+BRAND_KEY = make_parent_key_name("brands", "id")
+
+
+def _explode_page_metrics(row: dict[str, Any]) -> list[dict[str, Any]]:
+ page = {
+ "brand_id": row[make_parent_key_name("sitemap_pages", BRAND_KEY)],
+ "page_id": row[make_parent_key_name("sitemap_pages", "id")],
+ "page_url": row[make_parent_key_name("sitemap_pages", "url")],
+ "granularity": row.get("granularity"),
+ }
+ metrics = row.get("metrics") or {}
+ return [
+ {**page, "metric": metric, "date": bucket["date"], "platform": platform, "count": count}
+ for metric in SITEMAP_METRICS
+ for bucket in metrics.get(metric) or []
+ for platform, count in (bucket.get("count_by_platform") or {}).items()
+ ]
+
+
+def _detection_days(start_date: str | None, row: dict[str, Any]) -> list[dict[str, Any]]:
+ return [
+ {"date": day, BRAND_KEY: row[BRAND_KEY]}
+ for day in row.get("dates") or []
+ if start_date is None or day >= start_date
+ ]
+
+
+def _middle_resource(parent: FanoutParent, window: _DateWindow) -> EndpointResource:
+ params: dict[str, Any] = {"brand_id": {"type": "resolve", "resource": "brands", "field": "id"}}
+ endpoint: Endpoint = {"path": parent.path, "params": params}
+ middle: EndpointResource = {"name": parent.name, "include_from_parent": ["id"], "endpoint": endpoint}
+ if parent.name == "sites":
+ # The sites list takes no paging params and returns every site at once.
+ endpoint["paginator"] = "single_page"
+ # Other sites of the brand return 404 on the agent traffic route.
+ middle["data_map"] = lambda site: site if site.get("agent_traffic_active") else []
+ elif parent.name == "sitemap_pages":
+ # A brand without a completed sitemap crawl answers 404.
+ endpoint["response_actions"] = [{"status_code": 404, "action": "ignore"}]
+ else:
+ params["limit"] = SIGNAL_ANCHOR_LIMIT
+ endpoint |= {"paginator": "single_page", "data_selector": None, "data_selector_required": False}
+ middle["data_map"] = partial(_detection_days, window.start_date)
+ return middle
+
+
+def _child_endpoint(schema_name: str, path: str, window: _DateWindow) -> Endpoint:
+ parent = FANOUT_PARENTS.get(schema_name)
+ params: dict[str, Any] = {"brand_id": {"type": "resolve", "resource": "brands", "field": "id"}}
+ if parent is not None:
+ params = {
+ "brand_id": {"type": "resolve", "resource": parent.name, "field": BRAND_KEY},
+ parent.param: {"type": "resolve", "resource": parent.name, "field": parent.field},
+ }
+ endpoint: Endpoint = {"path": path, "params": params}
+ if schema_name == "prompts":
+ params["status"] = "all"
+ elif schema_name == "responses":
+ params["end_date"] = window.end_date
+ if window.start_date is not None:
+ params["start_date"] = window.start_date
+ elif schema_name == "signals":
+ params["sort"] = "detected_desc"
+ # A path that ends in a placeholder defaults to a single page.
+ endpoint["paginator"] = {"type": "offset", "limit": PAGE_SIZE, "total_path": "total"}
+ elif schema_name == "agent_traffic":
+ params |= {
+ "start_date": window.start_date,
+ "end_date": window.end_date,
+ "fields": ",".join(AGENT_TRAFFIC_DIMENSIONS),
+ "time_bucket": "day",
+ }
+ # The response carries no total, so a short page ends the walk.
+ endpoint["paginator"] = {"type": "offset", "limit": AGENT_TRAFFIC_PAGE_SIZE, "total_path": None}
+ endpoint["data_selector"] = "data"
+ elif schema_name in ("site_audit_score", "sitemap_page_metrics"):
+ # One object per request; the site score is null for a brand with no audit yet.
+ endpoint |= {"paginator": "single_page", "data_selector": None, "data_selector_required": False}
+ return endpoint
+
+
+def _child_resources(schema_name: str, path: str, window: _DateWindow) -> list[EndpointResource]:
+ endpoint = _child_endpoint(schema_name, path, window)
+ parent = FANOUT_PARENTS.get(schema_name)
+ if parent is None:
+ return [
+ {
+ "name": schema_name,
+ "include_from_parent": ["id"],
+ "endpoint": endpoint,
+ "data_map": rename_parent_fields("brands", {"id": "brand_id"}),
+ }
+ ]
+
+ child: EndpointResource = {"name": schema_name, "endpoint": endpoint}
+ if schema_name == "agent_traffic":
+ child["include_from_parent"] = ["id", BRAND_KEY]
+ child["data_map"] = rename_parent_fields(parent.name, {"id": "site_id", BRAND_KEY: "brand_id"})
+ elif schema_name == "sitemap_page_metrics":
+ child["include_from_parent"] = ["id", "url", BRAND_KEY]
+ child["data_map"] = _explode_page_metrics
+ else:
+ child["include_from_parent"] = [BRAND_KEY]
+ child["data_map"] = rename_parent_fields(parent.name, {BRAND_KEY: "brand_id"})
+ return [_middle_resource(parent, window), child]
+
+
def scrunch_source(
api_key: str,
inputs: SourceInputs,
manager: ResumableSourceManager[ScrunchResumeConfig],
) -> SourceResponse:
- path = schema_for_resource(ENDPOINTS, inputs.schema_name)
- is_responses = inputs.schema_name == "responses"
+ schema_name = inputs.schema_name
+ path = schema_for_resource(ENDPOINTS, schema_name)
+ is_responses = schema_name == "responses"
+ is_partitioned = schema_name in ("responses", "signals")
resume = manager.load_state() if manager.can_resume() else None
- start_date = None
- # Exclude the open UTC day so new responses cannot move offsets during a sync.
- end_date = datetime.now(UTC).date().isoformat() if is_responses else None
- if resume is not None:
- start_date, end_date = resume.start_date, resume.end_date
- elif is_responses and inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:
- watermark = parse_datetime_value(inputs.db_incremental_field_last_value)
- if watermark is None:
- raise ValueError("Invalid Scrunch response watermark. Reset this table's sync.")
- # Date filters include the whole watermark day; merging prevents duplicate rows.
- start_date = watermark.date().isoformat()
+ window = (
+ _DateWindow(start_date=resume.start_date, end_date=resume.end_date)
+ if resume is not None
+ else _date_window(inputs)
+ )
def save_checkpoint(state: dict[str, Any] | None) -> None:
manager.save_state(
ScrunchResumeConfig(
paginator_state=state,
- start_date=start_date,
- end_date=end_date,
+ start_date=window.start_date,
+ end_date=window.end_date,
complete=state is None,
)
)
parent: EndpointResource = {"name": "brands", "endpoint": {"path": ENDPOINTS["brands"]}}
resources: list[str | EndpointResource] = [parent]
- if inputs.schema_name != "brands":
- params: dict[str, Any] = {"brand_id": {"type": "resolve", "resource": "brands", "field": "id"}}
- if inputs.schema_name == "prompts":
- params["status"] = "all"
- if is_responses:
- params["end_date"] = end_date
- if start_date is not None:
- params["start_date"] = start_date
- resources.append(
- {
- "name": inputs.schema_name,
- "include_from_parent": ["id"],
- "endpoint": {"path": path, "params": params},
- "data_map": rename_parent_fields("brands", {"id": "brand_id"}),
- }
- )
+ if schema_name != "brands":
+ resources.extend(_child_resources(schema_name, path, window))
config: RESTAPIConfig = {
"client": {
@@ -132,14 +266,14 @@ def scrunch_source(
resume_hook=save_checkpoint,
initial_paginator_state=resume.paginator_state if resume else None,
)
- if resource.name == inputs.schema_name
+ if resource.name == schema_name
)
return SourceResponse(
- name=inputs.schema_name,
+ name=schema_name,
items=(lambda: []) if resume and resume.complete else (lambda: resource),
- primary_keys=PRIMARY_KEYS[inputs.schema_name],
+ primary_keys=PRIMARY_KEYS[schema_name],
sort_mode="desc" if is_responses else None,
- partition_keys=["created_at"] if is_responses else None,
- partition_mode="datetime" if is_responses else None,
- partition_format="month" if is_responses else None,
+ partition_keys=["created_at"] if is_partitioned else None,
+ partition_mode="datetime" if is_partitioned else None,
+ partition_format="month" if is_partitioned else None,
)
diff --git a/products/warehouse_sources/sources/scrunch/settings.py b/products/warehouse_sources/sources/scrunch/settings.py
index f5f733278f28..867574e66f29 100644
--- a/products/warehouse_sources/sources/scrunch/settings.py
+++ b/products/warehouse_sources/sources/scrunch/settings.py
@@ -1,18 +1,53 @@
-from products.warehouse_sources.sources.sdk import incremental_field
+from products.warehouse_sources.sources.sdk import IncrementalFieldType, frozen, incremental_field
API_VERSION = "v1"
BASE_URL = f"https://api.scrunchai.com/{API_VERSION}"
PAGE_SIZE = 100
+AGENT_TRAFFIC_PAGE_SIZE = 10000
+AGENT_TRAFFIC_HISTORY_DAYS = 365
+AGENT_TRAFFIC_DIMENSIONS = ("date", "site", "path", "agent_source", "agent_type")
+SITEMAP_METRICS = ("citations", "ai_referrals", "agent_traffic")
+SIGNAL_ANCHOR_LIMIT = 365
ENDPOINTS = {
"brands": "brands",
"prompts": "{brand_id}/prompts",
"competitors": "brands/{brand_id}/competitors",
"personas": "brands/{brand_id}/personas",
+ "site_audit_score": "{brand_id}/audit-scores/site",
+ "agent_traffic": "{brand_id}/sites/{site_id}/agent-traffic",
+ "sitemap_page_metrics": "{brand_id}/sitemap/pages/{page_id}/metrics",
+ # Over a multi-day range the API keeps only the latest detection per fingerprint, so read one day at a time.
+ "signals": "{brand_id}/signals?anchor_from={anchor}&anchor_to={anchor}",
"responses": "{brand_id}/responses",
}
-INCREMENTAL_FIELDS = {"responses": [incremental_field("created_at")]}
-PRIMARY_KEYS = {name: ["id"] if name == "brands" else ["brand_id", "id"] for name in ENDPOINTS}
+
+
+@frozen
+class FanoutParent:
+ name: str
+ path: str
+ param: str
+ field: str
+
+
+FANOUT_PARENTS = {
+ "agent_traffic": FanoutParent(name="sites", path="{brand_id}/sites", param="site_id", field="id"),
+ "sitemap_page_metrics": FanoutParent(
+ name="sitemap_pages", path="{brand_id}/sitemap/pages", param="page_id", field="id"
+ ),
+ "signals": FanoutParent(name="signal_anchors", path="{brand_id}/signals/anchors", param="anchor", field="date"),
+}
+INCREMENTAL_FIELDS = {
+ "responses": [incremental_field("created_at")],
+ "signals": [incremental_field("detected_for_date", IncrementalFieldType.Date)],
+ "agent_traffic": [incremental_field("date", IncrementalFieldType.Date)],
+}
+PRIMARY_KEYS = {name: ["id"] if name == "brands" else ["brand_id", "id"] for name in ENDPOINTS} | {
+ "site_audit_score": ["brand_id"],
+ "agent_traffic": ["brand_id", "site_id", *AGENT_TRAFFIC_DIMENSIONS],
+ "sitemap_page_metrics": ["brand_id", "page_id", "metric", "date", "platform"],
+}
AUTH_ERROR = "Scrunch rejected the API key. Check the key in your organization's API Keys menu."
PERMISSION_ERROR = (
diff --git a/products/warehouse_sources/sources/scrunch/source.py b/products/warehouse_sources/sources/scrunch/source.py
index c53579ad9901..4ebe8859c8eb 100644
--- a/products/warehouse_sources/sources/scrunch/source.py
+++ b/products/warehouse_sources/sources/scrunch/source.py
@@ -1,10 +1,13 @@
+from datetime import timedelta
from typing import cast
from products.warehouse_sources.sources.scrunch._config import ScrunchSourceConfig
from products.warehouse_sources.sources.scrunch.scrunch import ScrunchResumeConfig, scrunch_source, validate_credentials
from products.warehouse_sources.sources.scrunch.settings import (
+ AGENT_TRAFFIC_HISTORY_DAYS,
API_VERSION,
ENDPOINTS,
+ FANOUT_PARENTS,
INCREMENTAL_FIELDS,
NON_RETRYABLE_ERRORS,
)
@@ -41,6 +44,16 @@ class ScrunchSource(ResumableSource[ScrunchSourceConfig, ScrunchResumeConfig]):
def source_type(self) -> ExternalDataSourceType:
return ExternalDataSourceType.SCRUNCH
+ def history_lookback_for_schema(
+ self, schema_name: str, config: ScrunchSourceConfig | None = None
+ ) -> timedelta | None:
+ # Agent traffic requires a start date; the other tables return every row.
+ return timedelta(days=AGENT_TRAFFIC_HISTORY_DAYS) if schema_name == "agent_traffic" else None
+
+ def resume_covers_run(self, *, incremental_or_append: bool, schema_name: str | None = None) -> bool:
+ # A two-level fan-out carries no resume state, so a retry reads it again from the start.
+ return schema_name not in FANOUT_PARENTS
+
def get_non_retryable_errors(self) -> dict[str, str | None]:
return NON_RETRYABLE_ERRORS
@@ -58,7 +71,7 @@ class ScrunchSource(ResumableSource[ScrunchSourceConfig, ScrunchResumeConfig]):
force_refresh: bool = False,
api_version: str | None = None,
) -> list[SourceSchema]:
- return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names, merge_only={"responses"})
+ return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names, merge_only=set(INCREMENTAL_FIELDS))
def validate_credentials(
self,
diff --git a/products/warehouse_sources/sources/scrunch/tests/test_scrunch.py b/products/warehouse_sources/sources/scrunch/tests/test_scrunch.py
index 3e0b35ae5a12..0f596e770015 100644
--- a/products/warehouse_sources/sources/scrunch/tests/test_scrunch.py
+++ b/products/warehouse_sources/sources/scrunch/tests/test_scrunch.py
@@ -1,4 +1,4 @@
-from datetime import UTC, datetime
+from datetime import UTC, date, datetime, timedelta
from typing import Any
import pytest
@@ -16,6 +16,7 @@ from products.warehouse_sources.sources.sdk.testing import DriveResult, Scripted
@pytest.fixture(autouse=True)
def small_pages(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(scrunch, "PAGE_SIZE", 2)
+ monkeypatch.setattr(scrunch, "AGENT_TRAFFIC_PAGE_SIZE", 2)
def page(rows: list[dict[str, Any]], total: int, status: int = 200) -> ScriptedResponse:
@@ -249,3 +250,152 @@ def test_invalid_watermark_fails_before_requests() -> None:
assert isinstance(result.raised, ValueError)
assert "watermark" in str(result.raised)
assert result.requests == []
+
+
+SITE_A = "01AAAAAAAAAAAAAAAAAAAAAAAA"
+SITE_B = "01BBBBBBBBBBBBBBBBBBBBBBBB"
+
+
+def traffic(rows: list[dict[str, Any]]) -> ScriptedResponse:
+ return ScriptedResponse(json={"meta": {"start_date": "x", "end_date": "y", "time_bucket": "day"}, "data": rows})
+
+
+ "inputs,expected_start",
+ [
+ ({}, None),
+ ({"history_start": datetime(2026, 1, 5, 12, tzinfo=UTC)}, "2026-01-05"),
+ (
+ {
+ "incremental_field": "date",
+ "db_incremental_field_last_value": "2026-08-01",
+ "history_start": datetime(2026, 1, 5, tzinfo=UTC),
+ },
+ "2026-08-01",
+ ),
+ ({"incremental_field": "date", "db_incremental_field_last_value": date(2026, 8, 1)}, "2026-08-01"),
+ ],
+)
+def test_agent_traffic_walks_active_sites_until_a_short_page(
+ inputs: dict[str, Any], expected_start: str | None
+) -> None:
+ today = datetime.now(UTC).date()
+ rows = [{"requests": n, "date": "2026-10-01", "path": f"/{n}", "agent_source": None} for n in range(3)]
+
+ result = run(
+ "agent_traffic",
+ [
+ page([{"id": 11}], 1),
+ page([{"id": SITE_A, "agent_traffic_active": False}, {"id": SITE_B, "agent_traffic_active": True}], 2),
+ traffic(rows[:2]),
+ traffic(rows[2:]),
+ ],
+ **inputs,
+ )
+
+ assert result.raised is None
+ assert result.rows == [{**row, "brand_id": 11, "site_id": SITE_B} for row in rows]
+ assert result.paths == ["/v1/brands", "/v1/11/sites", *[f"/v1/11/sites/{SITE_B}/agent-traffic"] * 2]
+ assert query(result, 1) == {}
+ start = expected_start or (today - timedelta(days=365)).isoformat()
+ for index, offset in ((2, "0"), (3, "2")):
+ assert query(result, index) == {
+ "offset": [offset],
+ "limit": ["2"],
+ "start_date": [start],
+ "end_date": [today.isoformat()],
+ "fields": ["date,site,path,agent_source,agent_type"],
+ "time_bucket": ["day"],
+ }
+
+
+def test_sitemap_page_metrics_explode_into_one_row_per_platform_bucket() -> None:
+ def metrics(**buckets: list[dict[str, Any]]) -> ScriptedResponse:
+ empty: dict[str, list[dict[str, Any]]] = {"citations": [], "ai_referrals": [], "agent_traffic": []}
+ return ScriptedResponse(json={"granularity": "daily", "metrics": empty | buckets})
+
+ result = run(
+ "sitemap_page_metrics",
+ [
+ page([{"id": 10}, {"id": 11}], 2),
+ ScriptedResponse(status=404, json={"detail": "No completed sitemap crawl found"}),
+ page([{"id": 5, "url": "https://example.com/a"}, {"id": 6, "url": "https://example.com/b"}], 2),
+ metrics(
+ citations=[{"date": "2026-10-01", "count_by_platform": {"OpenAI": 2, "Gemini": 1}}],
+ ai_referrals=[{"date": "2026-10-01", "count_by_platform": {}}],
+ ),
+ metrics(agent_traffic=[{"date": "2026-10-02", "count_by_platform": {"OpenAI": 7}}]),
+ ],
+ )
+
+ assert result.raised is None
+ page_a = {"brand_id": 11, "page_id": 5, "page_url": "https://example.com/a", "granularity": "daily"}
+ page_b = {"brand_id": 11, "page_id": 6, "page_url": "https://example.com/b", "granularity": "daily"}
+ assert result.rows == [
+ {**page_a, "metric": "citations", "date": "2026-10-01", "platform": "OpenAI", "count": 2},
+ {**page_a, "metric": "citations", "date": "2026-10-01", "platform": "Gemini", "count": 1},
+ {**page_b, "metric": "agent_traffic", "date": "2026-10-02", "platform": "OpenAI", "count": 7},
+ ]
+ assert result.paths[1:] == [
+ "/v1/10/sitemap/pages",
+ "/v1/11/sitemap/pages",
+ "/v1/11/sitemap/pages/5/metrics",
+ "/v1/11/sitemap/pages/6/metrics",
+ ]
+ assert query(result, 2) == {"offset": ["0"], "limit": ["2"]}
+ assert query(result, 3) == query(result, 4) == {}
+
+
+def test_site_audit_score_keeps_one_row_per_audited_brand() -> None:
+ score = {"brand_id": 11, "generated_at": "2026-10-01T00:00:00Z", "score": {"retrievability": 70, "citability": 50}}
+
+ result = run(
+ "site_audit_score",
+ [page([{"id": 11}, {"id": 22}], 2), ScriptedResponse(json=score), ScriptedResponse(body=b"null")],
+ )
+
+ assert result.raised is None
+ assert result.rows == [score]
+ assert result.paths == ["/v1/brands", "/v1/11/audit-scores/site", "/v1/22/audit-scores/site"]
+ assert query(result, 1) == query(result, 2) == {}
+
+
+ "inputs,expected_days",
+ [
+ ({}, ["2026-08-03", "2026-08-02", "2026-08-01"]),
+ (
+ {"incremental_field": "detected_for_date", "db_incremental_field_last_value": "2026-08-02"},
+ ["2026-08-03", "2026-08-02"],
+ ),
+ (
+ {"incremental_field": "detected_for_date", "db_incremental_field_last_value": date(2026, 8, 2)},
+ ["2026-08-03", "2026-08-02"],
+ ),
+ ],
+)
+def test_signals_read_each_detection_day_on_its_own(inputs: dict[str, Any], expected_days: list[str]) -> None:
+ detections = {
+ "2026-08-03": [{"id": 3, "fingerprint": "f1"}, {"id": 4, "fingerprint": "f2"}, {"id": 5, "fingerprint": "f3"}],
+ "2026-08-02": [{"id": 2, "fingerprint": "f1"}],
+ "2026-08-01": [{"id": 1, "fingerprint": "f1"}],
+ }
+ script = [
+ page([{"id": 11}], 1),
+ ScriptedResponse(json={"dates": ["2026-08-03", "2026-08-02", "2026-08-01"]}),
+ ]
+ for day in expected_days:
+ rows = detections[day]
+ script += [page(rows[index : index + 2], len(rows)) for index in range(0, len(rows), 2)]
+
+ result = run("signals", script, **inputs)
+
+ assert result.raised is None
+ assert result.rows == [{**row, "brand_id": 11} for day in expected_days for row in detections[day]]
+ assert result.paths[1] == "/v1/11/signals/anchors"
+ assert query(result, 1) == {"limit": ["365"]}
+ signal_queries = [query(result, index) for index in range(2, len(result.requests))]
+ assert [(q["anchor_from"], q["anchor_to"], q["offset"]) for q in signal_queries] == [
+ ([day], [day], [str(offset)]) for day in expected_days for offset in range(0, len(detections[day]), 2)
+ ]
+ assert all(q["sort"] == ["detected_desc"] for q in signal_queries)