:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(semrush): add 4 missing tables to the semrush source (#130160)
Tom Owers44574dd14613
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 15c8d48ab553..1a28e50599ca 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -10792,16 +10792,16 @@ Note: The dependency and scan list endpoints are POST-with-body (paginated searc## Semrush — **thin**-Today (3): `site_audit`, `site_audit_issue_descriptions`, `site_audit_snapshots`+Today (7): `domain_organic_competitors`, `domain_organic_keywords`, `site_audit`, `site_audit_history`, `site_audit_issue_descriptions`, `site_audit_issue_details`, `site_audit_snapshots`Diffed against: <https://developer.semrush.com/api/v3/projects/site-audit/>-- [ ] `Snapshots history (GET /reports/v1/projects/{ID}/siteaudit/history)` — track historical site health scores and issue counts, beyond the snapshot identifiers currently imported (high)-- [ ] `Snapshot issue details (GET /reports/v1/projects/{ID}/siteaudit/snapshot/{snapshotId}/issue/{issueId})` — identify affected URLs and measure the business impact of specific technical SEO issues (high)-- [ ] `Domain Organic Search Keywords (GET /?type=domain_organic)` — analyze organic rankings, search volume, landing URLs, and estimated traffic for a domain (high)+- [x] `Snapshots history (GET /reports/v1/projects/{ID}/siteaudit/history)` — track historical site health scores and issue counts, beyond the snapshot identifiers currently imported (high)+- [x] `Snapshot issue details (GET /reports/v1/projects/{ID}/siteaudit/snapshot/{snapshotId}/issue/{issueId})` — identify affected URLs and measure the business impact of specific technical SEO issues (high)+- [x] `Domain Organic Search Keywords (GET /?type=domain_organic)` — analyze organic rankings, search volume, landing URLs, and estimated traffic for a domain (high)- [ ] `Page list (GET /reports/v1/projects/{ID}/siteaudit/page/list)` — compare crawled pages and connect page identifiers to URLs (medium)- [ ] `Domain Paid Search Keywords (GET /?type=domain_adwords)` — compare paid keywords, positions, costs, and estimated traffic with organic performance (medium)-- [ ] `Competitors in Organic Search (GET /?type=domain_organic_organic)` — measure keyword overlap and competitive search visibility (medium)+- [x] `Competitors in Organic Search (GET /?type=domain_organic_organic)` — measure keyword overlap and competitive search visibility (medium)- [ ] `Domain Organic Pages (GET /?type=domain_organic_unique)` — compare landing pages by ranking keywords and estimated organic traffic (medium)Note: Parsed the Site Audit reference's operations and request examples. Also checked https://developer.semrush.com/api/v3/analytics/domain-reports/ for domain report operations. Current tables cover one project's audit metadata; domain reports need domain and database inputs. Issue details require snapshot and issue IDs; excluded audit launch and configuration operations.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv b/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csvindex acd7b35ad41f..0b2e02e9c551 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv@@ -1099,7 +1099,7 @@ SelectStar,api_token,,,,,select-starSemanticScholar,api_key,,,query,,semantic-scholarSemaphore,api_token organization project_id,,,,,semaphoreSemgrep,api_token,,,,,semgrep-Semrush,api_key project_id,,,,,semrush+Semrush,api_key project_id,,,domain database,,semrushSendGrid,api_key,,,,,send-gridSendPulse,,,,,,send-pulseSendowl,api_key api_secret,,,,,sendowldiff --git a/products/warehouse_sources/sources/semrush/_config.py b/products/warehouse_sources/sources/semrush/_config.pyindex 67f1ead65ab4..adcb8d48d809 100644--- a/products/warehouse_sources/sources/semrush/_config.py+++ b/products/warehouse_sources/sources/semrush/_config.py@@ -8,3 +8,5 @@ from products.warehouse_sources.sources.sdk import configclass SemrushSourceConfig(config.Config):api_key: strproject_id: str+ domain: str | None = None+ database: str | None = Nonediff --git a/products/warehouse_sources/sources/semrush/canonical_descriptions.py b/products/warehouse_sources/sources/semrush/canonical_descriptions.pyindex d324af86653c..3a9422a7d51b 100644--- a/products/warehouse_sources/sources/semrush/canonical_descriptions.py+++ b/products/warehouse_sources/sources/semrush/canonical_descriptions.py@@ -1,5 +1,5 @@from products.warehouse_sources.sources.sdk import CanonicalDescriptions-from products.warehouse_sources.sources.semrush.settings import API_DOCS_URL+from products.warehouse_sources.sources.semrush.settings import API_DOCS_URL, DOMAIN_REPORTS_DOCS_URLCANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"site_audit": {@@ -40,4 +40,67 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"info_column": "Label for the issue information column.",},},+ "site_audit_history": {+ "description": "Site health score and issue counts for each completed Site Audit snapshot of the configured project.",+ "docs_url": API_DOCS_URL + "#get-information-about-snapshot",+ "columns": {+ "project_id": "Project ID supplied in the source configuration.",+ "snapshot_id": "Audit snapshot ID.",+ "finish_date": "Audit completion time in Unix milliseconds.",+ "pages_crawled": "Number of crawled pages.",+ "quality": "Site health score (value) and its change from the previous audit (delta).",+ "errors": "Error issue IDs with their count, change from the previous audit, and number of checks.",+ "warnings": "Warning issue IDs with their count, change from the previous audit, and number of checks.",+ "notices": "Notice issue IDs with their count, change from the previous audit, and number of checks.",+ },+ },+ "site_audit_issue_details": {+ "description": "Pages affected by each issue that the latest Site Audit snapshot found.",+ "docs_url": API_DOCS_URL + "#detailed-report-for-issue",+ "columns": {+ "project_id": "Project ID supplied in the source configuration.",+ "snapshot_id": "Audit snapshot ID the rows come from.",+ "issue_id": "Issue type ID. Join to site_audit_issue_descriptions.id for the issue title.",+ "source_url": "URL of the page on which the issue was detected.",+ "target_url": "Target URL of the issue, for example the broken page a link points to.",+ "page_id": "Semrush page ID.",+ },+ },+ "domain_organic_keywords": {+ "description": "Keywords that bring users to the domain via Google's top 100 organic search results, sorted by traffic share.",+ "docs_url": DOMAIN_REPORTS_DOCS_URL + "#domain-organic-search-keywords",+ "columns": {+ "domain": "Domain the report was run for.",+ "database": "Semrush regional database code.",+ "keyword": "Keyword bringing users to the domain via Google's top 100 organic search results.",+ "position": "Position the URL gets for the keyword. 0 means not ranked.",+ "previous_position": "Previous position for the keyword. 0 means not ranked.",+ "position_difference": "Difference between the previous position and the current position.",+ "search_volume": "Average number of monthly searches for the keyword over the last 12 months.",+ "cpc": "Average price in USD advertisers pay for a click on an ad for the keyword.",+ "url": "URL of the target page.",+ "traffic_percent": "Share of the domain's traffic driven by the keyword.",+ "traffic_cost_percent": "Share of the domain's total traffic cost attributed to the keyword.",+ "competition": "Competitive density of advertisers for the keyword, from 0 to 1.",+ "number_of_results": "Total number of organic results returned for the keyword.",+ "trends": "Comma-separated monthly search interest for the last 12 months.",+ "keyword_difficulty": "Estimated difficulty of ranking for the keyword, from 0 to 100.",+ "timestamp": "Unix timestamp of the data.",+ },+ },+ "domain_organic_competitors": {+ "description": "Domains that compete with the domain in organic search, sorted by common keywords.",+ "docs_url": DOMAIN_REPORTS_DOCS_URL + "#competitors-in-organic-search",+ "columns": {+ "domain": "Domain the report was run for.",+ "database": "Semrush regional database code.",+ "competitor_domain": "Competitor domain name.",+ "competitor_relevance": "Competition level based on the keywords of each domain and the keywords they share.",+ "common_keywords": "Keywords both domains rank for in Google's top 100 organic search results.",+ "organic_keywords": "Keywords that bring users to the competitor via Google's top 100 organic results.",+ "organic_traffic": "Traffic brought to the competitor via Google's top 100 organic search results.",+ "organic_cost": "Estimated price of the competitor's organic keywords in Google Ads.",+ "adwords_keywords": "Keywords the competitor buys in Google Ads.",+ },+ },}diff --git a/products/warehouse_sources/sources/semrush/semrush.py b/products/warehouse_sources/sources/semrush/semrush.pyindex b743dc648745..19ca9a88bdc7 100644--- a/products/warehouse_sources/sources/semrush/semrush.py+++ b/products/warehouse_sources/sources/semrush/semrush.py@@ -1,28 +1,56 @@+import ioimport re+import csv+from collections.abc import Callable, Iterable, Iterator+from typing import Any, Optional+from urllib.parse import urlparse-from requests import RequestException+from requests import RequestException, Responsefrom products.warehouse_sources.sources.sdk import (+ ClientConfig,+ EndpointResource,+ PageNumberPaginator,Resource,ResponseAction,RESTAPIConfig,RESTClientRetryableError,SourceResponse,+ build_chained_resource,+ make_tracked_session,rest_api_resource,schema_for_resource,)from products.warehouse_sources.sources.semrush.settings import (ACCESS_ERROR,+ API_HOST,AUTH_ERROR,BASE_URL,+ DEFAULT_DATABASE,+ DOMAIN_ERROR,+ DOMAIN_REPORT_ERROR,+ DOMAIN_REPORT_ERROR_MESSAGES,+ DOMAIN_REPORT_NOTHING_FOUND,+ DOMAIN_REPORT_ROW_LIMIT,+ DOMAIN_REPORT_TIMEOUT,+ DOMAIN_REPORTS,ENDPOINTS,ERROR_MESSAGES,+ ISSUE_DETAILS_PAGE_SIZE,MAX_REQUEST_ATTEMPTS,PROJECT_ID_ERROR,REQUEST_ERROR,+ REQUEST_TIMEOUT,+ SITE_AUDIT_HISTORY,+ SITE_AUDIT_ISSUE_DETAILS,UNAVAILABLE_ERROR,+ DomainReportColumn,)+MISSING_ID_ERROR = "Semrush returned a record without its ID."+DOMAIN_REPORT_ROW_ERROR = "Semrush returned a domain report row with an unexpected number of columns."+DOMAIN_REPORT_TRANSIENT_CODES = {429, 500}+def validate_project_id(project_id: str) -> None:if re.fullmatch(r"[0-9]+", project_id) is None:@@ -31,13 +59,11 @@ def validate_project_id(project_id: str) -> None:def normalize_row(row: dict[str, object], project_id: str, primary_key: str) -> dict[str, object]:if row.get(primary_key) is None:- raise ValueError("Semrush returned a record without its ID.")+ raise ValueError(MISSING_ID_ERROR)return {**row, "project_id": project_id}-def semrush_resource(api_key: str, project_id: str, endpoint: str, team_id: int, job_id: str) -> Resource:- validate_project_id(project_id)- settings = schema_for_resource(ENDPOINTS, endpoint)+def _response_actions(ignore_not_found: bool = False) -> list[ResponseAction]:error_actions: list[ResponseAction] = [{"json_field": "code", "json_values": [code, str(code)], "action": "raise", "message": message}for code, message in ERROR_MESSAGES.items()@@ -47,23 +73,35 @@ def semrush_resource(api_key: str, project_id: str, endpoint: str, team_id: int,for status in range(400, 500)if status != 429]- actions: list[ResponseAction] = [+ # An issue the latest audit no longer reports can 404 between the info and detail requests.+ not_found: list[ResponseAction] = [{"status_code": 404, "action": "ignore"}] if ignore_not_found else []+ return [{"status_code": 401, "action": "raise", "message": AUTH_ERROR},{"status_code": 403, "action": "raise", "message": ACCESS_ERROR},+ *not_found,*error_actions,{"json_field": "code", "json_values": [511, "511"], "action": "retry", "message": "Semrush temporary error."},*client_error_actions,]+++def _client_config(api_key: str) -> ClientConfig:+ return {+ "base_url": BASE_URL,+ "auth": {"type": "api_key", "name": "key", "api_key": api_key, "location": "query"},+ "paginator": "single_page",+ "max_retries": MAX_REQUEST_ATTEMPTS,+ "allowed_hosts": [],+ "allow_redirects": False,+ "request_timeout": REQUEST_TIMEOUT,+ }+++def semrush_resource(api_key: str, project_id: str, endpoint: str, team_id: int, job_id: str) -> Resource:+ validate_project_id(project_id)+ settings = schema_for_resource(ENDPOINTS, endpoint)config: RESTAPIConfig = {- "client": {- "base_url": BASE_URL,- "auth": {"type": "api_key", "name": "key", "api_key": api_key, "location": "query"},- "paginator": "single_page",- "max_retries": MAX_REQUEST_ATTEMPTS,- "allowed_hosts": [],- "allow_redirects": False,- "request_timeout": (10, 30),- },+ "client": _client_config(api_key),"resources": [{"name": endpoint,@@ -72,7 +110,7 @@ def semrush_resource(api_key: str, project_id: str, endpoint: str, team_id: int,"path": f"{project_id}/siteaudit/{settings['path']}","data_selector": settings["selector"],"data_selector_required": True,- "response_actions": actions,+ "response_actions": _response_actions(),},"data_map": lambda row: normalize_row(row, project_id, settings["primary_key"]),}@@ -81,6 +119,221 @@ def semrush_resource(api_key: str, project_id: str, endpoint: str, team_id: int,return rest_api_resource(config, team_id, job_id, None)+def _first_row(resource: Iterable[Any]) -> dict[str, Any] | None:+ for page in resource:+ for row in page:+ return row+ return None+++def _snapshot_id(row: dict[str, Any]) -> str:+ snapshot_id = row.get("snapshot_id")+ if not isinstance(snapshot_id, str) or re.fullmatch(r"[0-9A-Za-z_-]+", snapshot_id) is None:+ raise ValueError(MISSING_ID_ERROR)+ return snapshot_id+++class IssueRowsPaginator(PageNumberPaginator):+ """Stops once the rows read reach `total`, which Semrush reports as a row count, not a page count."""++ def __init__(self) -> None:+ super().__init__(base_page=1)+ self.rows_read = 0++ def update_state(self, response: Response, data: Optional[list[Any]] = None) -> None:+ rows = len(data or [])+ self.rows_read += rows+ try:+ body = response.json()+ except ValueError:+ body = None+ total = body.get("total") if isinstance(body, dict) else None+ if rows == 0 or (isinstance(total, int) and self.rows_read >= total):+ self._has_next_page = False+ return+ self.page += 1+ self._has_next_page = True+++def _history_resource(+ api_key: str, project_id: str, team_id: int, job_id: str, last_finish_date: Any, incremental: bool+) -> Iterable[Any]:+ def new_snapshots(row: dict[str, Any]) -> list[dict[str, Any]]:+ _snapshot_id(row)+ finish_date = row.get("finish_date")+ if last_finish_date is not None and isinstance(finish_date, int) and finish_date <= last_finish_date:+ return []+ return [row]++ parent: EndpointResource = {+ "name": "site_audit_snapshots",+ "write_disposition": "replace",+ "endpoint": {+ "path": f"{project_id}/siteaudit/snapshots",+ "data_selector": "snapshots",+ "data_selector_required": True,+ "paginator": "single_page",+ "response_actions": _response_actions(),+ },+ # The snapshot list costs 100 API units; each snapshot's details cost 10,000. An+ # incremental sync reads the list and fetches details only for snapshots it has not loaded.+ "data_map": new_snapshots,+ }+ child: EndpointResource = {+ "name": SITE_AUDIT_HISTORY,+ "write_disposition": {"disposition": "merge", "strategy": "upsert"} if incremental else "replace",+ "include_from_parent": ["snapshot_id", "finish_date"],+ "endpoint": {+ "path": f"{project_id}/siteaudit/snapshot?snapshot_id={{snapshot_id}}",+ "params": {"snapshot_id": {"type": "resolve", "resource": "site_audit_snapshots", "field": "snapshot_id"}},+ "data_selector": "$",+ "data_selector_required": True,+ "paginator": "single_page",+ "response_actions": _response_actions(),+ },+ "data_map": lambda row: {**row, "project_id": project_id},+ }+ return build_chained_resource(+ resources=[parent, child],+ child_name=SITE_AUDIT_HISTORY,+ parent_name="site_audit_snapshots",+ parent_field_renames={"snapshot_id": "snapshot_id", "finish_date": "finish_date"},+ client_config=_client_config(api_key),+ team_id=team_id,+ job_id=job_id,+ )+++def _detected_issues(info: dict[str, Any]) -> list[dict[str, Any]]:+ defects = info.get("defects")+ if not isinstance(defects, dict):+ return []+ return [{"issue_id": int(issue_id)} for issue_id, count in defects.items() if str(issue_id).isdigit() and count]+++def _issue_details_resource(api_key: str, project_id: str, team_id: int, job_id: str) -> Iterable[Any]:+ snapshots = [+ row for page in semrush_resource(api_key, project_id, "site_audit_snapshots", team_id, job_id) for row in page+ ]+ if not snapshots:+ return []+ latest = max(snapshots, key=lambda row: row.get("finish_date") or 0)+ snapshot_id = _snapshot_id(latest)++ parent: EndpointResource = {+ "name": "site_audit",+ "write_disposition": "replace",+ "endpoint": {+ "path": f"{project_id}/siteaudit/info",+ "data_selector": "$",+ "data_selector_required": True,+ "paginator": "single_page",+ "response_actions": _response_actions(),+ },+ # Only issues the latest audit found have affected URLs, so each one costs a request.+ "data_map": _detected_issues,+ }+ child: EndpointResource = {+ "name": SITE_AUDIT_ISSUE_DETAILS,+ "write_disposition": "replace",+ "include_from_parent": ["issue_id"],+ "endpoint": {+ "path": f"{project_id}/siteaudit/snapshot/{snapshot_id}/issue/{{issue_id}}",+ "params": {+ "issue_id": {"type": "resolve", "resource": "site_audit", "field": "issue_id"},+ "limit": ISSUE_DETAILS_PAGE_SIZE,+ },+ "data_selector": "data",+ "paginator": IssueRowsPaginator(),+ "response_actions": _response_actions(ignore_not_found=True),+ },+ "data_map": lambda row: {**row, "project_id": project_id, "snapshot_id": snapshot_id},+ }+ return build_chained_resource(+ resources=[parent, child],+ child_name=SITE_AUDIT_ISSUE_DETAILS,+ parent_name="site_audit",+ parent_field_renames={"issue_id": "issue_id"},+ client_config=_client_config(api_key),+ team_id=team_id,+ job_id=job_id,+ )+++def _report_domain(api_key: str, project_id: str, domain: str | None, team_id: int, job_id: str) -> str:+ value = (domain or "").strip()+ if not value:+ info = _first_row(semrush_resource(api_key, project_id, "site_audit", team_id, job_id)) or {}+ value = str(info.get("url") or "").strip()+ host = urlparse(value if "//" in value else f"//{value}").hostname+ if not host:+ raise ValueError(DOMAIN_ERROR)+ return host+++def _parse_value(column: DomainReportColumn, value: str) -> object:+ if column["kind"] == "str":+ return value+ if value == "":+ return None+ try:+ number = float(value)+ except ValueError:+ return None+ return int(number) if column["kind"] == "int" else number+++def _domain_report_rows(+ api_key: str, endpoint: str, domain: Callable[[], str], database: str+) -> Iterator[list[dict[str, Any]]]:+ report = DOMAIN_REPORTS[endpoint]+ columns = report["columns"]+ report_domain = domain()+ params: dict[str, str | int] = {+ "type": report["type"],+ "key": api_key,+ "domain": report_domain,+ "database": database,+ "display_limit": DOMAIN_REPORT_ROW_LIMIT,+ "display_sort": report["sort"],+ "export_columns": ",".join(column["code"] for column in columns),+ "export_escape": 1,+ }+ session = make_tracked_session(redact_values=(api_key,), allow_redirects=False)+ response = session.get(+ API_HOST,+ params=params,+ timeout=DOMAIN_REPORT_TIMEOUT,+ )+ text = response.text.strip()+ error = re.match(r"ERROR\s+(\d+)\s*::", text)+ if error:+ code = int(error.group(1))+ if code == DOMAIN_REPORT_NOTHING_FOUND:+ return+ if code in DOMAIN_REPORT_TRANSIENT_CODES:+ raise RESTClientRetryableError(UNAVAILABLE_ERROR)+ raise ValueError(DOMAIN_REPORT_ERROR_MESSAGES.get(code, DOMAIN_REPORT_ERROR))+ if response.status_code == 401:+ raise ValueError(AUTH_ERROR)+ if response.status_code == 403:+ raise ValueError(ACCESS_ERROR)+ response.raise_for_status()++ reader = csv.reader(io.StringIO(text), delimiter=";")+ next(reader, None)+ rows: list[dict[str, Any]] = []+ for line in reader:+ if not line:+ continue+ if len(line) != len(columns):+ raise ValueError(DOMAIN_REPORT_ROW_ERROR)+ row: dict[str, Any] = {column["name"]: _parse_value(column, value) for column, value in zip(columns, line)}+ rows.append({"domain": report_domain, "database": database, **row})+ if rows:+ yield rows++def validate_credentials(api_key: str, project_id: str, team_id: int) -> tuple[bool, str | None]:try:list(semrush_resource(api_key, project_id, "site_audit", team_id, ""))@@ -94,7 +347,49 @@ def validate_credentials(api_key: str, project_id: str, team_id: int) -> tuple[breturn True, None-def semrush_source(api_key: str, project_id: str, endpoint: str, team_id: int, job_id: str) -> SourceResponse:+def semrush_source(+ api_key: str,+ project_id: str,+ endpoint: str,+ team_id: int,+ job_id: str,+ domain: str | None = None,+ database: str | None = None,+ should_use_incremental_field: bool = False,+ db_incremental_field_last_value: Any = None,+) -> SourceResponse:+ validate_project_id(project_id)++ if endpoint == SITE_AUDIT_HISTORY:+ last_value = db_incremental_field_last_value if should_use_incremental_field else None+ return SourceResponse(+ name=endpoint,+ items=lambda: _history_resource(+ api_key, project_id, team_id, job_id, last_value, should_use_incremental_field+ ),+ primary_keys=["project_id", "snapshot_id"],+ )++ if endpoint == SITE_AUDIT_ISSUE_DETAILS:+ return SourceResponse(+ name=endpoint,+ items=lambda: _issue_details_resource(api_key, project_id, team_id, job_id),+ primary_keys=["project_id", "snapshot_id", "issue_id", "source_url", "target_url"],+ )++ if endpoint in DOMAIN_REPORTS:+ report_database = (database or "").strip().lower() or DEFAULT_DATABASE+ return SourceResponse(+ name=endpoint,+ items=lambda: _domain_report_rows(+ api_key,+ endpoint,+ lambda: _report_domain(api_key, project_id, domain, team_id, job_id),+ report_database,+ ),+ primary_keys=DOMAIN_REPORTS[endpoint]["primary_keys"],+ )+resource = semrush_resource(api_key, project_id, endpoint, team_id, job_id)return SourceResponse(name=endpoint,diff --git a/products/warehouse_sources/sources/semrush/settings.py b/products/warehouse_sources/sources/semrush/settings.pyindex 3c829cb6e9ab..58f93fc0dc11 100644--- a/products/warehouse_sources/sources/semrush/settings.py+++ b/products/warehouse_sources/sources/semrush/settings.py@@ -1,4 +1,6 @@-from typing import TypedDict+from typing import Literal, TypedDict++from products.warehouse_sources.sources.sdk import IncrementalField, IncrementalFieldTypeclass SemrushEndpoint(TypedDict):@@ -7,10 +9,27 @@ class SemrushEndpoint(TypedDict):primary_key: str+class DomainReportColumn(TypedDict):+ code: str+ name: str+ kind: Literal["str", "int", "float"]+++class DomainReport(TypedDict):+ type: str+ sort: str+ columns: list[DomainReportColumn]+ primary_keys: list[str]++API_VERSION = "v1"-BASE_URL = f"https://api.semrush.com/reports/{API_VERSION}/projects/"+API_HOST = "https://api.semrush.com/"+BASE_URL = f"{API_HOST}reports/{API_VERSION}/projects/"API_DOCS_URL = "https://developer.semrush.com/api/v3/projects/site-audit/"+DOMAIN_REPORTS_DOCS_URL = "https://developer.semrush.com/api/v3/analytics/domain-reports/"MAX_REQUEST_ATTEMPTS = 3+REQUEST_TIMEOUT = (10, 30)+DOMAIN_REPORT_TIMEOUT = (10, 120)ENDPOINTS: dict[str, SemrushEndpoint] = {"site_audit": {"path": "info", "selector": "$", "primary_key": "id"},@@ -18,6 +37,67 @@ ENDPOINTS: dict[str, SemrushEndpoint] = {"site_audit_issue_descriptions": {"path": "meta/issues", "selector": "issues", "primary_key": "id"},}+SITE_AUDIT_HISTORY = "site_audit_history"+SITE_AUDIT_ISSUE_DETAILS = "site_audit_issue_details"+ISSUE_DETAILS_PAGE_SIZE = 100++DOMAIN_ORGANIC_KEYWORDS = "domain_organic_keywords"+DOMAIN_ORGANIC_COMPETITORS = "domain_organic_competitors"+DEFAULT_DATABASE = "us"+# Domain reports cost API units per returned line, so a sync reads the top rows only.+DOMAIN_REPORT_ROW_LIMIT = 10_000++DOMAIN_REPORTS: dict[str, DomainReport] = {+ DOMAIN_ORGANIC_KEYWORDS: {+ "type": "domain_organic",+ "sort": "tr_desc",+ "columns": [+ {"code": "Ph", "name": "keyword", "kind": "str"},+ {"code": "Po", "name": "position", "kind": "int"},+ {"code": "Pp", "name": "previous_position", "kind": "int"},+ {"code": "Pd", "name": "position_difference", "kind": "int"},+ {"code": "Nq", "name": "search_volume", "kind": "int"},+ {"code": "Cp", "name": "cpc", "kind": "float"},+ {"code": "Ur", "name": "url", "kind": "str"},+ {"code": "Tr", "name": "traffic_percent", "kind": "float"},+ {"code": "Tc", "name": "traffic_cost_percent", "kind": "float"},+ {"code": "Co", "name": "competition", "kind": "float"},+ {"code": "Nr", "name": "number_of_results", "kind": "int"},+ {"code": "Td", "name": "trends", "kind": "str"},+ {"code": "Kd", "name": "keyword_difficulty", "kind": "float"},+ {"code": "Ts", "name": "timestamp", "kind": "int"},+ ],+ "primary_keys": ["domain", "database", "keyword", "url"],+ },+ DOMAIN_ORGANIC_COMPETITORS: {+ "type": "domain_organic_organic",+ "sort": "np_desc",+ "columns": [+ {"code": "Dn", "name": "competitor_domain", "kind": "str"},+ {"code": "Cr", "name": "competitor_relevance", "kind": "float"},+ {"code": "Np", "name": "common_keywords", "kind": "int"},+ {"code": "Or", "name": "organic_keywords", "kind": "int"},+ {"code": "Ot", "name": "organic_traffic", "kind": "int"},+ {"code": "Oc", "name": "organic_cost", "kind": "float"},+ {"code": "Ad", "name": "adwords_keywords", "kind": "int"},+ ],+ "primary_keys": ["domain", "database", "competitor_domain"],+ },+}++TABLES = [*ENDPOINTS, SITE_AUDIT_HISTORY, SITE_AUDIT_ISSUE_DETAILS, *DOMAIN_REPORTS]++INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {+ SITE_AUDIT_HISTORY: [+ {+ "label": "Audit finish date",+ "type": IncrementalFieldType.Integer,+ "field": "finish_date",+ "field_type": IncrementalFieldType.Integer,+ }+ ],+}+AUTH_ERROR = "Semrush authentication failed. Check your v3 API key in your Semrush profile."ACCESS_ERROR = "Semrush API access is disabled. Check your SEO Business subscription and project access."QUOTA_ERROR = "Semrush API units or request limits are exhausted. Add API units or contact Semrush support."@@ -25,6 +105,9 @@ PROJECT_ERROR = "Semrush could not find this project. Check the project ID and yPROJECT_ID_ERROR = "Enter the numeric project ID from your Semrush project URL."REQUEST_ERROR = "Semrush rejected the request. Check your project ID, Site Audit setup, and API access."UNAVAILABLE_ERROR = "Semrush is unavailable or rate-limiting requests. Try again later."+DATABASE_ACCESS_ERROR = "Your Semrush plan can't access this regional database. Choose another database."+DOMAIN_REPORT_ERROR = "Semrush rejected the domain report request. Check the domain and regional database settings."+DOMAIN_ERROR = "Semrush returned no URL for this project. Enter a domain in the source settings."ERROR_MESSAGES = {70: AUTH_ERROR,@@ -37,3 +120,16 @@ ERROR_MESSAGES = {134: QUOTA_ERROR,512: PROJECT_ERROR,}++DOMAIN_REPORT_ERROR_MESSAGES = {+ 110: AUTH_ERROR,+ 120: AUTH_ERROR,+ 130: ACCESS_ERROR,+ 131: QUOTA_ERROR,+ 132: QUOTA_ERROR,+ 133: DATABASE_ACCESS_ERROR,+ 134: QUOTA_ERROR,+ 135: ACCESS_ERROR,+}+# "ERROR 50 :: NOTHING FOUND" means the domain has no rows in this database.+DOMAIN_REPORT_NOTHING_FOUND = 50diff --git a/products/warehouse_sources/sources/semrush/source.py b/products/warehouse_sources/sources/semrush/source.pyindex 8300a46a1a12..e0b85fca1117 100644--- a/products/warehouse_sources/sources/semrush/source.py+++ b/products/warehouse_sources/sources/semrush/source.py@@ -25,10 +25,14 @@ from products.warehouse_sources.sources.semrush.settings import (API_DOCS_URL,API_VERSION,AUTH_ERROR,- ENDPOINTS,+ DATABASE_ACCESS_ERROR,+ DOMAIN_ERROR,+ DOMAIN_REPORT_ERROR,ERROR_MESSAGES,+ INCREMENTAL_FIELDS,PROJECT_ID_ERROR,REQUEST_ERROR,+ TABLES,)@@ -50,6 +54,9 @@ class SemrushSource(SimpleSource[SemrushSourceConfig]):**{message: message for message in ERROR_MESSAGES.values()},PROJECT_ID_ERROR: PROJECT_ID_ERROR,REQUEST_ERROR: REQUEST_ERROR,+ DATABASE_ACCESS_ERROR: DATABASE_ACCESS_ERROR,+ DOMAIN_REPORT_ERROR: DOMAIN_REPORT_ERROR,+ DOMAIN_ERROR: DOMAIN_ERROR,"401 Client Error": AUTH_ERROR,"403 Client Error": ACCESS_ERROR,}@@ -66,7 +73,7 @@ class SemrushSource(SimpleSource[SemrushSourceConfig]):force_refresh: bool = False,api_version: str | None = None,) -> list[SourceSchema]:- return build_endpoint_schemas(ENDPOINTS, {}, names)+ return build_endpoint_schemas(TABLES, INCREMENTAL_FIELDS, names)def validate_credentials(self,@@ -78,7 +85,17 @@ class SemrushSource(SimpleSource[SemrushSourceConfig]):return validate_credentials(config.api_key, config.project_id, team_id)def source_for_pipeline(self, config: SemrushSourceConfig, inputs: SourceInputs) -> SourceResponse:- return semrush_source(config.api_key, config.project_id, inputs.schema_name, inputs.team_id, inputs.job_id)+ return semrush_source(+ config.api_key,+ config.project_id,+ inputs.schema_name,+ inputs.team_id,+ inputs.job_id,+ domain=config.domain,+ database=config.database,+ should_use_incremental_field=inputs.should_use_incremental_field,+ db_incremental_field_last_value=inputs.db_incremental_field_last_value,+ )@propertydef get_source_config(self) -> SourceConfig:@@ -90,7 +107,10 @@ class SemrushSource(SimpleSource[SemrushSourceConfig]):"Requires an SEO Business subscription and a project with Site Audit enabled. ""Find your v3 API key under [API keys](https://www.semrush.com/accounts/api-keys/active) in your Semrush profile. ""Syncs use your account's API units: 100 units per table request. Retries can use more units. "- "Connection validation also uses 100 units."+ "Connection validation also uses 100 units. "+ "Site audit history costs 10,000 units per audit snapshot, and incremental syncs fetch only new snapshots. "+ "Domain organic reports cost 10 units (keywords) or 40 units (competitors) per row, "+ "and each sync reads up to 10,000 rows."),iconPath="/static/services/semrush.png",fields=cast(@@ -112,6 +132,26 @@ class SemrushSource(SimpleSource[SemrushSourceConfig]):placeholder="123456",secret=False,),+ SourceFieldInputConfig(+ name="domain",+ label="Domain for organic reports",+ type=SourceFieldInputConfigType.TEXT,+ required=False,+ placeholder="example.com",+ caption="Leave empty to use your Semrush project's domain.",+ secret=False,+ sync_setting=True,+ ),+ SourceFieldInputConfig(+ name="database",+ label="Regional database for organic reports",+ type=SourceFieldInputConfigType.TEXT,+ required=False,+ placeholder="us",+ caption="A Semrush regional database code, such as `us`, `uk` or `de`. Defaults to `us`.",+ secret=False,+ sync_setting=True,+ ),],),releaseStatus=ReleaseStatus.GA,diff --git a/products/warehouse_sources/sources/semrush/tests/test_semrush.py b/products/warehouse_sources/sources/semrush/tests/test_semrush.pyindex 1208b64f1e0e..c5d286543589 100644--- a/products/warehouse_sources/sources/semrush/tests/test_semrush.py+++ b/products/warehouse_sources/sources/semrush/tests/test_semrush.py@@ -1,16 +1,26 @@+import reimport json-from collections.abc import Iterable+from collections.abc import Callable, Iterablefrom typing import Any, cast+from urllib.parse import parse_qs, urlparseimport pytestfrom unittest.mock import patch-from requests import RequestException, Response, Session+from requests import PreparedRequest, RequestException, Response, Session-from products.warehouse_sources.sources.semrush.semrush import semrush_source, validate_credentials+from products.warehouse_sources.sources.sdk import RESTClientRetryableError+from products.warehouse_sources.sources.semrush.semrush import (+ DOMAIN_REPORT_ROW_ERROR,+ semrush_source,+ validate_credentials,+)from products.warehouse_sources.sources.semrush.settings import (ACCESS_ERROR,AUTH_ERROR,+ DATABASE_ACCESS_ERROR,+ DOMAIN_ERROR,+ DOMAIN_REPORT_ERROR,PROJECT_ERROR,PROJECT_ID_ERROR,QUOTA_ERROR,@@ -97,3 +107,267 @@ def test_malformed_records_fail_instead_of_erasing_rows(endpoint: str, body: objwith patch.object(Session, "send", return_value=response(body)):with pytest.raises(ValueError, match=message):list(cast(Iterable[Any], semrush_source("fake-api-key", "123", endpoint, 1, "job").items()))+++def text_response(body: str, status: int = 200) -> Response:+ result = Response()+ result.status_code = status+ result._content = body.encode()+ result.headers["Content-Type"] = "text/plain"+ return result+++def routed(routes: dict[str, Callable[[dict[str, list[str]]], Response]], seen: list[str]) -> Callable[..., Response]:+ def send(request: PreparedRequest, **kwargs: Any) -> Response:+ url = urlparse(request.url or "")+ seen.append(f"{url.path}?{url.query}")+ query = parse_qs(url.query)+ for prefix, handler in routes.items():+ if url.path == prefix:+ return handler(query)+ raise AssertionError(f"unexpected request {url.path}")++ return send+++def rows_of(endpoint: str, **kwargs: Any) -> list[dict[str, Any]]:+ response = semrush_source("fake-api-key", "123", endpoint, 1, "job", **kwargs)+ return [row for page in cast(Iterable[Any], response.items()) for row in page]+++SNAPSHOTS = {+ "snapshots": [+ {"snapshot_id": "aaa111", "finish_date": 1000},+ {"snapshot_id": "bbb222", "finish_date": 3000},+ {"snapshot_id": "ccc333", "finish_date": 2000},+ ]+}+++def snapshot_details(query: dict[str, list[str]]) -> Response:+ snapshot_id = query["snapshot_id"][0]+ return response({"snapshot_id": snapshot_id, "quality": {"value": 90, "delta": 1}, "pages_crawled": 10})+++ "incremental,last_value,expected",+ [+ (False, None, ["aaa111", "bbb222", "ccc333"]),+ (False, 2500, ["aaa111", "bbb222", "ccc333"]),+ (True, None, ["aaa111", "bbb222", "ccc333"]),+ (True, 1000, ["bbb222", "ccc333"]),+ (True, 3000, []),+ ],+)+def test_history_fetches_details_only_for_snapshots_past_the_watermark(+ incremental: bool, last_value: int | None, expected: list[str]+) -> None:+ seen: list[str] = []+ routes = {+ "/reports/v1/projects/123/siteaudit/snapshots": lambda _: response(SNAPSHOTS),+ "/reports/v1/projects/123/siteaudit/snapshot": snapshot_details,+ }+ with patch.object(Session, "send", side_effect=routed(routes, seen)):+ rows = rows_of(+ "site_audit_history", should_use_incremental_field=incremental, db_incremental_field_last_value=last_value+ )++ assert sorted(row["snapshot_id"] for row in rows) == expected+ assert len(seen) == 1 + len(expected)+ finish_dates = {item["snapshot_id"]: item["finish_date"] for item in SNAPSHOTS["snapshots"]}+ for row in rows:+ assert row["project_id"] == "123"+ assert row["finish_date"] == finish_dates[row["snapshot_id"]]+ assert row["quality"] == {"value": 90, "delta": 1}+ assert not any(key.startswith("_site_audit_snapshots") for key in row)++[email protected]("snapshot_id", [None, "../456", "abc?key=x"])+def test_history_rejects_snapshot_ids_that_would_change_the_request(snapshot_id: str | None) -> None:+ seen: list[str] = []+ routes = {+ "/reports/v1/projects/123/siteaudit/snapshots": lambda _: response(+ {"snapshots": [{"snapshot_id": snapshot_id, "finish_date": 1}]}+ ),+ }+ with patch.object(Session, "send", side_effect=routed(routes, seen)):+ with pytest.raises(ValueError, match="record without its ID"):+ rows_of("site_audit_history")+ assert len(seen) == 1+++def issue_pages(query: dict[str, list[str]], total: int, page_size: int) -> Response:+ page = int(query["page"][0])+ start = (page - 1) * page_size+ count = max(0, min(page_size, total - start))+ data = [+ {"source_url": f"https://example.com/{start + i}", "target_url": None, "page_id": f"p{start + i}"}+ for i in range(count)+ ]+ return response({"limit": page_size, "page": page, "total": total, "data": data})+++def test_issue_details_walks_pages_of_detected_issues_in_the_latest_snapshot() -> None:+ seen: list[str] = []+ base = "/reports/v1/projects/123/siteaudit"+ routes = {+ f"{base}/snapshots": lambda _: response(SNAPSHOTS),+ f"{base}/info": lambda _: response({"id": 123, "defects": {"8": 150, "109": 2, "110": 0}}),+ f"{base}/snapshot/bbb222/issue/8": lambda query: issue_pages(query, total=150, page_size=100),+ f"{base}/snapshot/bbb222/issue/109": lambda query: issue_pages(query, total=2, page_size=100),+ }+ with patch.object(Session, "send", side_effect=routed(routes, seen)):+ rows = rows_of("site_audit_issue_details")++ assert sum(row["issue_id"] == 8 for row in rows) == 150+ assert sum(row["issue_id"] == 109 for row in rows) == 2+ assert all(row["snapshot_id"] == "bbb222" and row["project_id"] == "123" for row in rows)+ issue_requests = [request for request in seen if "/issue/" in request]+ assert len(issue_requests) == 3+ assert all("limit=100" in request for request in issue_requests)+++def test_issue_details_skips_an_issue_that_is_gone() -> None:+ seen: list[str] = []+ base = "/reports/v1/projects/123/siteaudit"+ routes = {+ f"{base}/snapshots": lambda _: response(SNAPSHOTS),+ f"{base}/info": lambda _: response({"id": 123, "defects": {"8": 1, "9": 1}}),+ f"{base}/snapshot/bbb222/issue/8": lambda _: response({"message": "Not found"}, 404),+ f"{base}/snapshot/bbb222/issue/9": lambda query: issue_pages(query, total=1, page_size=100),+ }+ with patch.object(Session, "send", side_effect=routed(routes, seen)):+ rows = rows_of("site_audit_issue_details")++ assert [row["issue_id"] for row in rows] == [9]+++def test_issue_details_without_audits_reads_nothing_else() -> None:+ seen: list[str] = []+ routes = {"/reports/v1/projects/123/siteaudit/snapshots": lambda _: response({"snapshots": []})}+ with patch.object(Session, "send", side_effect=routed(routes, seen)):+ assert rows_of("site_audit_issue_details") == []+ assert len(seen) == 1+++KEYWORDS_CSV = (+ "Keyword;Position;Previous Position;Position Difference;Search Volume;CPC;Url;Traffic (%);"+ "Traffic Cost (%);Competition;Number of Results;Trends;Keyword Difficulty Index;Timestamp\r\n"+ '"seo; tools";"6";"7";"1";"8100";"10.54";"http://example.com/";"2.15";"3.87";"0.54";"321000000";'+ '"0.67,0.82";"71";"1700000000"\r\n'+ '"seo";"9";"";"0";"110000";"14.82";"http://example.com/a";"17.53";"44.40";"0.50";"0";"0.81";"";"1700000000"\r\n'+)+++ "domain,database,expected_domain,expected_database,info_requests",+ [+ ("example.com", "UK", "example.com", "uk", 0),+ ("https://www.example.com/blog", None, "www.example.com", "us", 0),+ (None, "", "example.org", "us", 1),+ (" ", "de", "example.org", "de", 1),+ ],+)+def test_domain_organic_keywords_request_and_rows(+ domain: str | None, database: str | None, expected_domain: str, expected_database: str, info_requests: int+) -> None:+ seen: list[str] = []+ queries: list[dict[str, list[str]]] = []++ def report(query: dict[str, list[str]]) -> Response:+ queries.append(query)+ return text_response(KEYWORDS_CSV)++ routes = {+ "/reports/v1/projects/123/siteaudit/info": lambda _: response({"id": 123, "url": "https://example.org/"}),+ "/": report,+ }+ with patch.object(Session, "send", side_effect=routed(routes, seen)):+ rows = rows_of("domain_organic_keywords", domain=domain, database=database)++ assert len(seen) == 1 + info_requests+ (query,) = queries+ assert query["type"] == ["domain_organic"]+ assert query["domain"] == [expected_domain]+ assert query["database"] == [expected_database]+ assert query["display_limit"] == ["10000"]+ assert query["export_columns"] == ["Ph,Po,Pp,Pd,Nq,Cp,Ur,Tr,Tc,Co,Nr,Td,Kd,Ts"]+ assert rows[0] == {+ "domain": expected_domain,+ "database": expected_database,+ "keyword": "seo; tools",+ "position": 6,+ "previous_position": 7,+ "position_difference": 1,+ "search_volume": 8100,+ "cpc": 10.54,+ "url": "http://example.com/",+ "traffic_percent": 2.15,+ "traffic_cost_percent": 3.87,+ "competition": 0.54,+ "number_of_results": 321000000,+ "trends": "0.67,0.82",+ "keyword_difficulty": 71.0,+ "timestamp": 1700000000,+ }+ assert rows[1]["previous_position"] is None+ assert rows[1]["keyword_difficulty"] is None+++def test_domain_organic_competitors_rows() -> None:+ body = (+ "Domain;Competitor Relevance;Common Keywords;Organic Keywords;Organic Traffic;Organic Cost;Adwords Keywords\n"+ "rival.example.com;0.13;338;11021;5640;9690;0\n"+ )+ seen: list[str] = []+ with patch.object(Session, "send", side_effect=routed({"/": lambda _: text_response(body)}, seen)):+ rows = rows_of("domain_organic_competitors", domain="example.com")++ assert "type=domain_organic_organic" in seen[0]+ assert rows == [+ {+ "domain": "example.com",+ "database": "us",+ "competitor_domain": "rival.example.com",+ "competitor_relevance": 0.13,+ "common_keywords": 338,+ "organic_keywords": 11021,+ "organic_traffic": 5640,+ "organic_cost": 9690.0,+ "adwords_keywords": 0,+ }+ ]+++ "status,body,error,message",+ [+ (200, "ERROR 50 :: NOTHING FOUND", None, None),+ (400, "ERROR 120 :: WRONG KEY - ID PAIR", ValueError, AUTH_ERROR),+ (200, "ERROR 132 :: API UNITS BALANCE IS ZERO", ValueError, QUOTA_ERROR),+ (200, "ERROR 133 :: DB ACCESS DENIED", ValueError, DATABASE_ACCESS_ERROR),+ (400, "ERROR 10000 :: Wrong parameter 'database'.", ValueError, DOMAIN_REPORT_ERROR),+ (200, "ERROR 500 :: Internal Error", RESTClientRetryableError, UNAVAILABLE_ERROR),+ (403, "Forbidden", ValueError, ACCESS_ERROR),+ (200, "Domain;Competitor Relevance\nrival.example.com;0.13\n", ValueError, DOMAIN_REPORT_ROW_ERROR),+ ],+)+def test_domain_report_errors(status: int, body: str, error: type[Exception] | None, message: str | None) -> None:+ seen: list[str] = []+ with patch.object(Session, "send", side_effect=routed({"/": lambda _: text_response(body, status)}, seen)):+ if error is None:+ assert rows_of("domain_organic_competitors", domain="example.com") == []+ else:+ with pytest.raises(error, match=re.escape(message or "")):+ rows_of("domain_organic_competitors", domain="example.com")+ if message in {DATABASE_ACCESS_ERROR, DOMAIN_REPORT_ERROR}:+ assert message in SemrushSource().get_non_retryable_errors()+++def test_domain_report_without_a_project_url_asks_for_a_domain() -> None:+ seen: list[str] = []+ routes = {"/reports/v1/projects/123/siteaudit/info": lambda _: response({"id": 123, "url": ""})}+ with patch.object(Session, "send", side_effect=routed(routes, seen)):+ with pytest.raises(ValueError, match=DOMAIN_ERROR):+ rows_of("domain_organic_keywords")+ assert len(seen) == 1