Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(semrush): add 4 missing tables to the semrush source (#130160)

Tom Owers44574dd14613
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 15c8d48ab553..1a28e50599ca 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -10792,16 +10792,16 @@ Note: The dependency and scan list endpoints are POST-with-body (paginated searc
## Semrush — **thin**
-Today (3): `site_audit`, `site_audit_issue_descriptions`, `site_audit_snapshots`
+Today (7): `domain_organic_competitors`, `domain_organic_keywords`, `site_audit`, `site_audit_history`, `site_audit_issue_descriptions`, `site_audit_issue_details`, `site_audit_snapshots`
Diffed against: <https://developer.semrush.com/api/v3/projects/site-audit/>
-- [ ] `Snapshots history (GET /reports/v1/projects/{ID}/siteaudit/history)` — track historical site health scores and issue counts, beyond the snapshot identifiers currently imported (high)
-- [ ] `Snapshot issue details (GET /reports/v1/projects/{ID}/siteaudit/snapshot/{snapshotId}/issue/{issueId})` — identify affected URLs and measure the business impact of specific technical SEO issues (high)
-- [ ] `Domain Organic Search Keywords (GET /?type=domain_organic)` — analyze organic rankings, search volume, landing URLs, and estimated traffic for a domain (high)
+- [x] `Snapshots history (GET /reports/v1/projects/{ID}/siteaudit/history)` — track historical site health scores and issue counts, beyond the snapshot identifiers currently imported (high)
+- [x] `Snapshot issue details (GET /reports/v1/projects/{ID}/siteaudit/snapshot/{snapshotId}/issue/{issueId})` — identify affected URLs and measure the business impact of specific technical SEO issues (high)
+- [x] `Domain Organic Search Keywords (GET /?type=domain_organic)` — analyze organic rankings, search volume, landing URLs, and estimated traffic for a domain (high)
- [ ] `Page list (GET /reports/v1/projects/{ID}/siteaudit/page/list)` — compare crawled pages and connect page identifiers to URLs (medium)
- [ ] `Domain Paid Search Keywords (GET /?type=domain_adwords)` — compare paid keywords, positions, costs, and estimated traffic with organic performance (medium)
-- [ ] `Competitors in Organic Search (GET /?type=domain_organic_organic)` — measure keyword overlap and competitive search visibility (medium)
+- [x] `Competitors in Organic Search (GET /?type=domain_organic_organic)` — measure keyword overlap and competitive search visibility (medium)
- [ ] `Domain Organic Pages (GET /?type=domain_organic_unique)` — compare landing pages by ranking keywords and estimated organic traffic (medium)
Note: Parsed the Site Audit reference's operations and request examples. Also checked https://developer.semrush.com/api/v3/analytics/domain-reports/ for domain report operations. Current tables cover one project's audit metadata; domain reports need domain and database inputs. Issue details require snapshot and issue IDs; excluded audit launch and configuration operations.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv b/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv
index acd7b35ad41f..0b2e02e9c551 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/field_classification.csv
@@ -1099,7 +1099,7 @@ SelectStar,api_token,,,,,select-star
SemanticScholar,api_key,,,query,,semantic-scholar
Semaphore,api_token organization project_id,,,,,semaphore
Semgrep,api_token,,,,,semgrep
-Semrush,api_key project_id,,,,,semrush
+Semrush,api_key project_id,,,domain database,,semrush
SendGrid,api_key,,,,,send-grid
SendPulse,,,,,,send-pulse
Sendowl,api_key api_secret,,,,,sendowl
diff --git a/products/warehouse_sources/sources/semrush/_config.py b/products/warehouse_sources/sources/semrush/_config.py
index 67f1ead65ab4..adcb8d48d809 100644
--- a/products/warehouse_sources/sources/semrush/_config.py
+++ b/products/warehouse_sources/sources/semrush/_config.py
@@ -8,3 +8,5 @@ from products.warehouse_sources.sources.sdk import config
class SemrushSourceConfig(config.Config):
api_key: str
project_id: str
+ domain: str | None = None
+ database: str | None = None
diff --git a/products/warehouse_sources/sources/semrush/canonical_descriptions.py b/products/warehouse_sources/sources/semrush/canonical_descriptions.py
index d324af86653c..3a9422a7d51b 100644
--- a/products/warehouse_sources/sources/semrush/canonical_descriptions.py
+++ b/products/warehouse_sources/sources/semrush/canonical_descriptions.py
@@ -1,5 +1,5 @@
from products.warehouse_sources.sources.sdk import CanonicalDescriptions
-from products.warehouse_sources.sources.semrush.settings import API_DOCS_URL
+from products.warehouse_sources.sources.semrush.settings import API_DOCS_URL, DOMAIN_REPORTS_DOCS_URL
CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"site_audit": {
@@ -40,4 +40,67 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"info_column": "Label for the issue information column.",
},
},
+ "site_audit_history": {
+ "description": "Site health score and issue counts for each completed Site Audit snapshot of the configured project.",
+ "docs_url": API_DOCS_URL + "#get-information-about-snapshot",
+ "columns": {
+ "project_id": "Project ID supplied in the source configuration.",
+ "snapshot_id": "Audit snapshot ID.",
+ "finish_date": "Audit completion time in Unix milliseconds.",
+ "pages_crawled": "Number of crawled pages.",
+ "quality": "Site health score (value) and its change from the previous audit (delta).",
+ "errors": "Error issue IDs with their count, change from the previous audit, and number of checks.",
+ "warnings": "Warning issue IDs with their count, change from the previous audit, and number of checks.",
+ "notices": "Notice issue IDs with their count, change from the previous audit, and number of checks.",
+ },
+ },
+ "site_audit_issue_details": {
+ "description": "Pages affected by each issue that the latest Site Audit snapshot found.",
+ "docs_url": API_DOCS_URL + "#detailed-report-for-issue",
+ "columns": {
+ "project_id": "Project ID supplied in the source configuration.",
+ "snapshot_id": "Audit snapshot ID the rows come from.",
+ "issue_id": "Issue type ID. Join to site_audit_issue_descriptions.id for the issue title.",
+ "source_url": "URL of the page on which the issue was detected.",
+ "target_url": "Target URL of the issue, for example the broken page a link points to.",
+ "page_id": "Semrush page ID.",
+ },
+ },
+ "domain_organic_keywords": {
+ "description": "Keywords that bring users to the domain via Google's top 100 organic search results, sorted by traffic share.",
+ "docs_url": DOMAIN_REPORTS_DOCS_URL + "#domain-organic-search-keywords",
+ "columns": {
+ "domain": "Domain the report was run for.",
+ "database": "Semrush regional database code.",
+ "keyword": "Keyword bringing users to the domain via Google's top 100 organic search results.",
+ "position": "Position the URL gets for the keyword. 0 means not ranked.",
+ "previous_position": "Previous position for the keyword. 0 means not ranked.",
+ "position_difference": "Difference between the previous position and the current position.",
+ "search_volume": "Average number of monthly searches for the keyword over the last 12 months.",
+ "cpc": "Average price in USD advertisers pay for a click on an ad for the keyword.",
+ "url": "URL of the target page.",
+ "traffic_percent": "Share of the domain's traffic driven by the keyword.",
+ "traffic_cost_percent": "Share of the domain's total traffic cost attributed to the keyword.",
+ "competition": "Competitive density of advertisers for the keyword, from 0 to 1.",
+ "number_of_results": "Total number of organic results returned for the keyword.",
+ "trends": "Comma-separated monthly search interest for the last 12 months.",
+ "keyword_difficulty": "Estimated difficulty of ranking for the keyword, from 0 to 100.",
+ "timestamp": "Unix timestamp of the data.",
+ },
+ },
+ "domain_organic_competitors": {
+ "description": "Domains that compete with the domain in organic search, sorted by common keywords.",
+ "docs_url": DOMAIN_REPORTS_DOCS_URL + "#competitors-in-organic-search",
+ "columns": {
+ "domain": "Domain the report was run for.",
+ "database": "Semrush regional database code.",
+ "competitor_domain": "Competitor domain name.",
+ "competitor_relevance": "Competition level based on the keywords of each domain and the keywords they share.",
+ "common_keywords": "Keywords both domains rank for in Google's top 100 organic search results.",
+ "organic_keywords": "Keywords that bring users to the competitor via Google's top 100 organic results.",
+ "organic_traffic": "Traffic brought to the competitor via Google's top 100 organic search results.",
+ "organic_cost": "Estimated price of the competitor's organic keywords in Google Ads.",
+ "adwords_keywords": "Keywords the competitor buys in Google Ads.",
+ },
+ },
}
diff --git a/products/warehouse_sources/sources/semrush/semrush.py b/products/warehouse_sources/sources/semrush/semrush.py
index b743dc648745..19ca9a88bdc7 100644
--- a/products/warehouse_sources/sources/semrush/semrush.py
+++ b/products/warehouse_sources/sources/semrush/semrush.py
@@ -1,28 +1,56 @@
+import io
import re
+import csv
+from collections.abc import Callable, Iterable, Iterator
+from typing import Any, Optional
+from urllib.parse import urlparse
-from requests import RequestException
+from requests import RequestException, Response
from products.warehouse_sources.sources.sdk import (
+ ClientConfig,
+ EndpointResource,
+ PageNumberPaginator,
Resource,
ResponseAction,
RESTAPIConfig,
RESTClientRetryableError,
SourceResponse,
+ build_chained_resource,
+ make_tracked_session,
rest_api_resource,
schema_for_resource,
)
from products.warehouse_sources.sources.semrush.settings import (
ACCESS_ERROR,
+ API_HOST,
AUTH_ERROR,
BASE_URL,
+ DEFAULT_DATABASE,
+ DOMAIN_ERROR,
+ DOMAIN_REPORT_ERROR,
+ DOMAIN_REPORT_ERROR_MESSAGES,
+ DOMAIN_REPORT_NOTHING_FOUND,
+ DOMAIN_REPORT_ROW_LIMIT,
+ DOMAIN_REPORT_TIMEOUT,
+ DOMAIN_REPORTS,
ENDPOINTS,
ERROR_MESSAGES,
+ ISSUE_DETAILS_PAGE_SIZE,
MAX_REQUEST_ATTEMPTS,
PROJECT_ID_ERROR,
REQUEST_ERROR,
+ REQUEST_TIMEOUT,
+ SITE_AUDIT_HISTORY,
+ SITE_AUDIT_ISSUE_DETAILS,
UNAVAILABLE_ERROR,
+ DomainReportColumn,
)
+MISSING_ID_ERROR = "Semrush returned a record without its ID."
+DOMAIN_REPORT_ROW_ERROR = "Semrush returned a domain report row with an unexpected number of columns."
+DOMAIN_REPORT_TRANSIENT_CODES = {429, 500}
+
def validate_project_id(project_id: str) -> None:
if re.fullmatch(r"[0-9]+", project_id) is None:
@@ -31,13 +59,11 @@ def validate_project_id(project_id: str) -> None:
def normalize_row(row: dict[str, object], project_id: str, primary_key: str) -> dict[str, object]:
if row.get(primary_key) is None:
- raise ValueError("Semrush returned a record without its ID.")
+ raise ValueError(MISSING_ID_ERROR)
return {**row, "project_id": project_id}
-def semrush_resource(api_key: str, project_id: str, endpoint: str, team_id: int, job_id: str) -> Resource:
- validate_project_id(project_id)
- settings = schema_for_resource(ENDPOINTS, endpoint)
+def _response_actions(ignore_not_found: bool = False) -> list[ResponseAction]:
error_actions: list[ResponseAction] = [
{"json_field": "code", "json_values": [code, str(code)], "action": "raise", "message": message}
for code, message in ERROR_MESSAGES.items()
@@ -47,23 +73,35 @@ def semrush_resource(api_key: str, project_id: str, endpoint: str, team_id: int,
for status in range(400, 500)
if status != 429
]
- actions: list[ResponseAction] = [
+ # An issue the latest audit no longer reports can 404 between the info and detail requests.
+ not_found: list[ResponseAction] = [{"status_code": 404, "action": "ignore"}] if ignore_not_found else []
+ return [
{"status_code": 401, "action": "raise", "message": AUTH_ERROR},
{"status_code": 403, "action": "raise", "message": ACCESS_ERROR},
+ *not_found,
*error_actions,
{"json_field": "code", "json_values": [511, "511"], "action": "retry", "message": "Semrush temporary error."},
*client_error_actions,
]
+
+
+def _client_config(api_key: str) -> ClientConfig:
+ return {
+ "base_url": BASE_URL,
+ "auth": {"type": "api_key", "name": "key", "api_key": api_key, "location": "query"},
+ "paginator": "single_page",
+ "max_retries": MAX_REQUEST_ATTEMPTS,
+ "allowed_hosts": [],
+ "allow_redirects": False,
+ "request_timeout": REQUEST_TIMEOUT,
+ }
+
+
+def semrush_resource(api_key: str, project_id: str, endpoint: str, team_id: int, job_id: str) -> Resource:
+ validate_project_id(project_id)
+ settings = schema_for_resource(ENDPOINTS, endpoint)
config: RESTAPIConfig = {
- "client": {
- "base_url": BASE_URL,
- "auth": {"type": "api_key", "name": "key", "api_key": api_key, "location": "query"},
- "paginator": "single_page",
- "max_retries": MAX_REQUEST_ATTEMPTS,
- "allowed_hosts": [],
- "allow_redirects": False,
- "request_timeout": (10, 30),
- },
+ "client": _client_config(api_key),
"resources": [
{
"name": endpoint,
@@ -72,7 +110,7 @@ def semrush_resource(api_key: str, project_id: str, endpoint: str, team_id: int,
"path": f"{project_id}/siteaudit/{settings['path']}",
"data_selector": settings["selector"],
"data_selector_required": True,
- "response_actions": actions,
+ "response_actions": _response_actions(),
},
"data_map": lambda row: normalize_row(row, project_id, settings["primary_key"]),
}
@@ -81,6 +119,221 @@ def semrush_resource(api_key: str, project_id: str, endpoint: str, team_id: int,
return rest_api_resource(config, team_id, job_id, None)
+def _first_row(resource: Iterable[Any]) -> dict[str, Any] | None:
+ for page in resource:
+ for row in page:
+ return row
+ return None
+
+
+def _snapshot_id(row: dict[str, Any]) -> str:
+ snapshot_id = row.get("snapshot_id")
+ if not isinstance(snapshot_id, str) or re.fullmatch(r"[0-9A-Za-z_-]+", snapshot_id) is None:
+ raise ValueError(MISSING_ID_ERROR)
+ return snapshot_id
+
+
+class IssueRowsPaginator(PageNumberPaginator):
+ """Stops once the rows read reach `total`, which Semrush reports as a row count, not a page count."""
+
+ def __init__(self) -> None:
+ super().__init__(base_page=1)
+ self.rows_read = 0
+
+ def update_state(self, response: Response, data: Optional[list[Any]] = None) -> None:
+ rows = len(data or [])
+ self.rows_read += rows
+ try:
+ body = response.json()
+ except ValueError:
+ body = None
+ total = body.get("total") if isinstance(body, dict) else None
+ if rows == 0 or (isinstance(total, int) and self.rows_read >= total):
+ self._has_next_page = False
+ return
+ self.page += 1
+ self._has_next_page = True
+
+
+def _history_resource(
+ api_key: str, project_id: str, team_id: int, job_id: str, last_finish_date: Any, incremental: bool
+) -> Iterable[Any]:
+ def new_snapshots(row: dict[str, Any]) -> list[dict[str, Any]]:
+ _snapshot_id(row)
+ finish_date = row.get("finish_date")
+ if last_finish_date is not None and isinstance(finish_date, int) and finish_date <= last_finish_date:
+ return []
+ return [row]
+
+ parent: EndpointResource = {
+ "name": "site_audit_snapshots",
+ "write_disposition": "replace",
+ "endpoint": {
+ "path": f"{project_id}/siteaudit/snapshots",
+ "data_selector": "snapshots",
+ "data_selector_required": True,
+ "paginator": "single_page",
+ "response_actions": _response_actions(),
+ },
+ # The snapshot list costs 100 API units; each snapshot's details cost 10,000. An
+ # incremental sync reads the list and fetches details only for snapshots it has not loaded.
+ "data_map": new_snapshots,
+ }
+ child: EndpointResource = {
+ "name": SITE_AUDIT_HISTORY,
+ "write_disposition": {"disposition": "merge", "strategy": "upsert"} if incremental else "replace",
+ "include_from_parent": ["snapshot_id", "finish_date"],
+ "endpoint": {
+ "path": f"{project_id}/siteaudit/snapshot?snapshot_id={{snapshot_id}}",
+ "params": {"snapshot_id": {"type": "resolve", "resource": "site_audit_snapshots", "field": "snapshot_id"}},
+ "data_selector": "$",
+ "data_selector_required": True,
+ "paginator": "single_page",
+ "response_actions": _response_actions(),
+ },
+ "data_map": lambda row: {**row, "project_id": project_id},
+ }
+ return build_chained_resource(
+ resources=[parent, child],
+ child_name=SITE_AUDIT_HISTORY,
+ parent_name="site_audit_snapshots",
+ parent_field_renames={"snapshot_id": "snapshot_id", "finish_date": "finish_date"},
+ client_config=_client_config(api_key),
+ team_id=team_id,
+ job_id=job_id,
+ )
+
+
+def _detected_issues(info: dict[str, Any]) -> list[dict[str, Any]]:
+ defects = info.get("defects")
+ if not isinstance(defects, dict):
+ return []
+ return [{"issue_id": int(issue_id)} for issue_id, count in defects.items() if str(issue_id).isdigit() and count]
+
+
+def _issue_details_resource(api_key: str, project_id: str, team_id: int, job_id: str) -> Iterable[Any]:
+ snapshots = [
+ row for page in semrush_resource(api_key, project_id, "site_audit_snapshots", team_id, job_id) for row in page
+ ]
+ if not snapshots:
+ return []
+ latest = max(snapshots, key=lambda row: row.get("finish_date") or 0)
+ snapshot_id = _snapshot_id(latest)
+
+ parent: EndpointResource = {
+ "name": "site_audit",
+ "write_disposition": "replace",
+ "endpoint": {
+ "path": f"{project_id}/siteaudit/info",
+ "data_selector": "$",
+ "data_selector_required": True,
+ "paginator": "single_page",
+ "response_actions": _response_actions(),
+ },
+ # Only issues the latest audit found have affected URLs, so each one costs a request.
+ "data_map": _detected_issues,
+ }
+ child: EndpointResource = {
+ "name": SITE_AUDIT_ISSUE_DETAILS,
+ "write_disposition": "replace",
+ "include_from_parent": ["issue_id"],
+ "endpoint": {
+ "path": f"{project_id}/siteaudit/snapshot/{snapshot_id}/issue/{{issue_id}}",
+ "params": {
+ "issue_id": {"type": "resolve", "resource": "site_audit", "field": "issue_id"},
+ "limit": ISSUE_DETAILS_PAGE_SIZE,
+ },
+ "data_selector": "data",
+ "paginator": IssueRowsPaginator(),
+ "response_actions": _response_actions(ignore_not_found=True),
+ },
+ "data_map": lambda row: {**row, "project_id": project_id, "snapshot_id": snapshot_id},
+ }
+ return build_chained_resource(
+ resources=[parent, child],
+ child_name=SITE_AUDIT_ISSUE_DETAILS,
+ parent_name="site_audit",
+ parent_field_renames={"issue_id": "issue_id"},
+ client_config=_client_config(api_key),
+ team_id=team_id,
+ job_id=job_id,
+ )
+
+
+def _report_domain(api_key: str, project_id: str, domain: str | None, team_id: int, job_id: str) -> str:
+ value = (domain or "").strip()
+ if not value:
+ info = _first_row(semrush_resource(api_key, project_id, "site_audit", team_id, job_id)) or {}
+ value = str(info.get("url") or "").strip()
+ host = urlparse(value if "//" in value else f"//{value}").hostname
+ if not host:
+ raise ValueError(DOMAIN_ERROR)
+ return host
+
+
+def _parse_value(column: DomainReportColumn, value: str) -> object:
+ if column["kind"] == "str":
+ return value
+ if value == "":
+ return None
+ try:
+ number = float(value)
+ except ValueError:
+ return None
+ return int(number) if column["kind"] == "int" else number
+
+
+def _domain_report_rows(
+ api_key: str, endpoint: str, domain: Callable[[], str], database: str
+) -> Iterator[list[dict[str, Any]]]:
+ report = DOMAIN_REPORTS[endpoint]
+ columns = report["columns"]
+ report_domain = domain()
+ params: dict[str, str | int] = {
+ "type": report["type"],
+ "key": api_key,
+ "domain": report_domain,
+ "database": database,
+ "display_limit": DOMAIN_REPORT_ROW_LIMIT,
+ "display_sort": report["sort"],
+ "export_columns": ",".join(column["code"] for column in columns),
+ "export_escape": 1,
+ }
+ session = make_tracked_session(redact_values=(api_key,), allow_redirects=False)
+ response = session.get(
+ API_HOST,
+ params=params,
+ timeout=DOMAIN_REPORT_TIMEOUT,
+ )
+ text = response.text.strip()
+ error = re.match(r"ERROR\s+(\d+)\s*::", text)
+ if error:
+ code = int(error.group(1))
+ if code == DOMAIN_REPORT_NOTHING_FOUND:
+ return
+ if code in DOMAIN_REPORT_TRANSIENT_CODES:
+ raise RESTClientRetryableError(UNAVAILABLE_ERROR)
+ raise ValueError(DOMAIN_REPORT_ERROR_MESSAGES.get(code, DOMAIN_REPORT_ERROR))
+ if response.status_code == 401:
+ raise ValueError(AUTH_ERROR)
+ if response.status_code == 403:
+ raise ValueError(ACCESS_ERROR)
+ response.raise_for_status()
+
+ reader = csv.reader(io.StringIO(text), delimiter=";")
+ next(reader, None)
+ rows: list[dict[str, Any]] = []
+ for line in reader:
+ if not line:
+ continue
+ if len(line) != len(columns):
+ raise ValueError(DOMAIN_REPORT_ROW_ERROR)
+ row: dict[str, Any] = {column["name"]: _parse_value(column, value) for column, value in zip(columns, line)}
+ rows.append({"domain": report_domain, "database": database, **row})
+ if rows:
+ yield rows
+
+
def validate_credentials(api_key: str, project_id: str, team_id: int) -> tuple[bool, str | None]:
try:
list(semrush_resource(api_key, project_id, "site_audit", team_id, ""))
@@ -94,7 +347,49 @@ def validate_credentials(api_key: str, project_id: str, team_id: int) -> tuple[b
return True, None
-def semrush_source(api_key: str, project_id: str, endpoint: str, team_id: int, job_id: str) -> SourceResponse:
+def semrush_source(
+ api_key: str,
+ project_id: str,
+ endpoint: str,
+ team_id: int,
+ job_id: str,
+ domain: str | None = None,
+ database: str | None = None,
+ should_use_incremental_field: bool = False,
+ db_incremental_field_last_value: Any = None,
+) -> SourceResponse:
+ validate_project_id(project_id)
+
+ if endpoint == SITE_AUDIT_HISTORY:
+ last_value = db_incremental_field_last_value if should_use_incremental_field else None
+ return SourceResponse(
+ name=endpoint,
+ items=lambda: _history_resource(
+ api_key, project_id, team_id, job_id, last_value, should_use_incremental_field
+ ),
+ primary_keys=["project_id", "snapshot_id"],
+ )
+
+ if endpoint == SITE_AUDIT_ISSUE_DETAILS:
+ return SourceResponse(
+ name=endpoint,
+ items=lambda: _issue_details_resource(api_key, project_id, team_id, job_id),
+ primary_keys=["project_id", "snapshot_id", "issue_id", "source_url", "target_url"],
+ )
+
+ if endpoint in DOMAIN_REPORTS:
+ report_database = (database or "").strip().lower() or DEFAULT_DATABASE
+ return SourceResponse(
+ name=endpoint,
+ items=lambda: _domain_report_rows(
+ api_key,
+ endpoint,
+ lambda: _report_domain(api_key, project_id, domain, team_id, job_id),
+ report_database,
+ ),
+ primary_keys=DOMAIN_REPORTS[endpoint]["primary_keys"],
+ )
+
resource = semrush_resource(api_key, project_id, endpoint, team_id, job_id)
return SourceResponse(
name=endpoint,
diff --git a/products/warehouse_sources/sources/semrush/settings.py b/products/warehouse_sources/sources/semrush/settings.py
index 3c829cb6e9ab..58f93fc0dc11 100644
--- a/products/warehouse_sources/sources/semrush/settings.py
+++ b/products/warehouse_sources/sources/semrush/settings.py
@@ -1,4 +1,6 @@
-from typing import TypedDict
+from typing import Literal, TypedDict
+
+from products.warehouse_sources.sources.sdk import IncrementalField, IncrementalFieldType
class SemrushEndpoint(TypedDict):
@@ -7,10 +9,27 @@ class SemrushEndpoint(TypedDict):
primary_key: str
+class DomainReportColumn(TypedDict):
+ code: str
+ name: str
+ kind: Literal["str", "int", "float"]
+
+
+class DomainReport(TypedDict):
+ type: str
+ sort: str
+ columns: list[DomainReportColumn]
+ primary_keys: list[str]
+
+
API_VERSION = "v1"
-BASE_URL = f"https://api.semrush.com/reports/{API_VERSION}/projects/"
+API_HOST = "https://api.semrush.com/"
+BASE_URL = f"{API_HOST}reports/{API_VERSION}/projects/"
API_DOCS_URL = "https://developer.semrush.com/api/v3/projects/site-audit/"
+DOMAIN_REPORTS_DOCS_URL = "https://developer.semrush.com/api/v3/analytics/domain-reports/"
MAX_REQUEST_ATTEMPTS = 3
+REQUEST_TIMEOUT = (10, 30)
+DOMAIN_REPORT_TIMEOUT = (10, 120)
ENDPOINTS: dict[str, SemrushEndpoint] = {
"site_audit": {"path": "info", "selector": "$", "primary_key": "id"},
@@ -18,6 +37,67 @@ ENDPOINTS: dict[str, SemrushEndpoint] = {
"site_audit_issue_descriptions": {"path": "meta/issues", "selector": "issues", "primary_key": "id"},
}
+SITE_AUDIT_HISTORY = "site_audit_history"
+SITE_AUDIT_ISSUE_DETAILS = "site_audit_issue_details"
+ISSUE_DETAILS_PAGE_SIZE = 100
+
+DOMAIN_ORGANIC_KEYWORDS = "domain_organic_keywords"
+DOMAIN_ORGANIC_COMPETITORS = "domain_organic_competitors"
+DEFAULT_DATABASE = "us"
+# Domain reports cost API units per returned line, so a sync reads the top rows only.
+DOMAIN_REPORT_ROW_LIMIT = 10_000
+
+DOMAIN_REPORTS: dict[str, DomainReport] = {
+ DOMAIN_ORGANIC_KEYWORDS: {
+ "type": "domain_organic",
+ "sort": "tr_desc",
+ "columns": [
+ {"code": "Ph", "name": "keyword", "kind": "str"},
+ {"code": "Po", "name": "position", "kind": "int"},
+ {"code": "Pp", "name": "previous_position", "kind": "int"},
+ {"code": "Pd", "name": "position_difference", "kind": "int"},
+ {"code": "Nq", "name": "search_volume", "kind": "int"},
+ {"code": "Cp", "name": "cpc", "kind": "float"},
+ {"code": "Ur", "name": "url", "kind": "str"},
+ {"code": "Tr", "name": "traffic_percent", "kind": "float"},
+ {"code": "Tc", "name": "traffic_cost_percent", "kind": "float"},
+ {"code": "Co", "name": "competition", "kind": "float"},
+ {"code": "Nr", "name": "number_of_results", "kind": "int"},
+ {"code": "Td", "name": "trends", "kind": "str"},
+ {"code": "Kd", "name": "keyword_difficulty", "kind": "float"},
+ {"code": "Ts", "name": "timestamp", "kind": "int"},
+ ],
+ "primary_keys": ["domain", "database", "keyword", "url"],
+ },
+ DOMAIN_ORGANIC_COMPETITORS: {
+ "type": "domain_organic_organic",
+ "sort": "np_desc",
+ "columns": [
+ {"code": "Dn", "name": "competitor_domain", "kind": "str"},
+ {"code": "Cr", "name": "competitor_relevance", "kind": "float"},
+ {"code": "Np", "name": "common_keywords", "kind": "int"},
+ {"code": "Or", "name": "organic_keywords", "kind": "int"},
+ {"code": "Ot", "name": "organic_traffic", "kind": "int"},
+ {"code": "Oc", "name": "organic_cost", "kind": "float"},
+ {"code": "Ad", "name": "adwords_keywords", "kind": "int"},
+ ],
+ "primary_keys": ["domain", "database", "competitor_domain"],
+ },
+}
+
+TABLES = [*ENDPOINTS, SITE_AUDIT_HISTORY, SITE_AUDIT_ISSUE_DETAILS, *DOMAIN_REPORTS]
+
+INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {
+ SITE_AUDIT_HISTORY: [
+ {
+ "label": "Audit finish date",
+ "type": IncrementalFieldType.Integer,
+ "field": "finish_date",
+ "field_type": IncrementalFieldType.Integer,
+ }
+ ],
+}
+
AUTH_ERROR = "Semrush authentication failed. Check your v3 API key in your Semrush profile."
ACCESS_ERROR = "Semrush API access is disabled. Check your SEO Business subscription and project access."
QUOTA_ERROR = "Semrush API units or request limits are exhausted. Add API units or contact Semrush support."
@@ -25,6 +105,9 @@ PROJECT_ERROR = "Semrush could not find this project. Check the project ID and y
PROJECT_ID_ERROR = "Enter the numeric project ID from your Semrush project URL."
REQUEST_ERROR = "Semrush rejected the request. Check your project ID, Site Audit setup, and API access."
UNAVAILABLE_ERROR = "Semrush is unavailable or rate-limiting requests. Try again later."
+DATABASE_ACCESS_ERROR = "Your Semrush plan can't access this regional database. Choose another database."
+DOMAIN_REPORT_ERROR = "Semrush rejected the domain report request. Check the domain and regional database settings."
+DOMAIN_ERROR = "Semrush returned no URL for this project. Enter a domain in the source settings."
ERROR_MESSAGES = {
70: AUTH_ERROR,
@@ -37,3 +120,16 @@ ERROR_MESSAGES = {
134: QUOTA_ERROR,
512: PROJECT_ERROR,
}
+
+DOMAIN_REPORT_ERROR_MESSAGES = {
+ 110: AUTH_ERROR,
+ 120: AUTH_ERROR,
+ 130: ACCESS_ERROR,
+ 131: QUOTA_ERROR,
+ 132: QUOTA_ERROR,
+ 133: DATABASE_ACCESS_ERROR,
+ 134: QUOTA_ERROR,
+ 135: ACCESS_ERROR,
+}
+# "ERROR 50 :: NOTHING FOUND" means the domain has no rows in this database.
+DOMAIN_REPORT_NOTHING_FOUND = 50
diff --git a/products/warehouse_sources/sources/semrush/source.py b/products/warehouse_sources/sources/semrush/source.py
index 8300a46a1a12..e0b85fca1117 100644
--- a/products/warehouse_sources/sources/semrush/source.py
+++ b/products/warehouse_sources/sources/semrush/source.py
@@ -25,10 +25,14 @@ from products.warehouse_sources.sources.semrush.settings import (
API_DOCS_URL,
API_VERSION,
AUTH_ERROR,
- ENDPOINTS,
+ DATABASE_ACCESS_ERROR,
+ DOMAIN_ERROR,
+ DOMAIN_REPORT_ERROR,
ERROR_MESSAGES,
+ INCREMENTAL_FIELDS,
PROJECT_ID_ERROR,
REQUEST_ERROR,
+ TABLES,
)
@@ -50,6 +54,9 @@ class SemrushSource(SimpleSource[SemrushSourceConfig]):
**{message: message for message in ERROR_MESSAGES.values()},
PROJECT_ID_ERROR: PROJECT_ID_ERROR,
REQUEST_ERROR: REQUEST_ERROR,
+ DATABASE_ACCESS_ERROR: DATABASE_ACCESS_ERROR,
+ DOMAIN_REPORT_ERROR: DOMAIN_REPORT_ERROR,
+ DOMAIN_ERROR: DOMAIN_ERROR,
"401 Client Error": AUTH_ERROR,
"403 Client Error": ACCESS_ERROR,
}
@@ -66,7 +73,7 @@ class SemrushSource(SimpleSource[SemrushSourceConfig]):
force_refresh: bool = False,
api_version: str | None = None,
) -> list[SourceSchema]:
- return build_endpoint_schemas(ENDPOINTS, {}, names)
+ return build_endpoint_schemas(TABLES, INCREMENTAL_FIELDS, names)
def validate_credentials(
self,
@@ -78,7 +85,17 @@ class SemrushSource(SimpleSource[SemrushSourceConfig]):
return validate_credentials(config.api_key, config.project_id, team_id)
def source_for_pipeline(self, config: SemrushSourceConfig, inputs: SourceInputs) -> SourceResponse:
- return semrush_source(config.api_key, config.project_id, inputs.schema_name, inputs.team_id, inputs.job_id)
+ return semrush_source(
+ config.api_key,
+ config.project_id,
+ inputs.schema_name,
+ inputs.team_id,
+ inputs.job_id,
+ domain=config.domain,
+ database=config.database,
+ should_use_incremental_field=inputs.should_use_incremental_field,
+ db_incremental_field_last_value=inputs.db_incremental_field_last_value,
+ )
@property
def get_source_config(self) -> SourceConfig:
@@ -90,7 +107,10 @@ class SemrushSource(SimpleSource[SemrushSourceConfig]):
"Requires an SEO Business subscription and a project with Site Audit enabled. "
"Find your v3 API key under [API keys](https://www.semrush.com/accounts/api-keys/active) in your Semrush profile. "
"Syncs use your account's API units: 100 units per table request. Retries can use more units. "
- "Connection validation also uses 100 units."
+ "Connection validation also uses 100 units. "
+ "Site audit history costs 10,000 units per audit snapshot, and incremental syncs fetch only new snapshots. "
+ "Domain organic reports cost 10 units (keywords) or 40 units (competitors) per row, "
+ "and each sync reads up to 10,000 rows."
),
iconPath="/static/services/semrush.png",
fields=cast(
@@ -112,6 +132,26 @@ class SemrushSource(SimpleSource[SemrushSourceConfig]):
placeholder="123456",
secret=False,
),
+ SourceFieldInputConfig(
+ name="domain",
+ label="Domain for organic reports",
+ type=SourceFieldInputConfigType.TEXT,
+ required=False,
+ placeholder="example.com",
+ caption="Leave empty to use your Semrush project's domain.",
+ secret=False,
+ sync_setting=True,
+ ),
+ SourceFieldInputConfig(
+ name="database",
+ label="Regional database for organic reports",
+ type=SourceFieldInputConfigType.TEXT,
+ required=False,
+ placeholder="us",
+ caption="A Semrush regional database code, such as `us`, `uk` or `de`. Defaults to `us`.",
+ secret=False,
+ sync_setting=True,
+ ),
],
),
releaseStatus=ReleaseStatus.GA,
diff --git a/products/warehouse_sources/sources/semrush/tests/test_semrush.py b/products/warehouse_sources/sources/semrush/tests/test_semrush.py
index 1208b64f1e0e..c5d286543589 100644
--- a/products/warehouse_sources/sources/semrush/tests/test_semrush.py
+++ b/products/warehouse_sources/sources/semrush/tests/test_semrush.py
@@ -1,16 +1,26 @@
+import re
import json
-from collections.abc import Iterable
+from collections.abc import Callable, Iterable
from typing import Any, cast
+from urllib.parse import parse_qs, urlparse
import pytest
from unittest.mock import patch
-from requests import RequestException, Response, Session
+from requests import PreparedRequest, RequestException, Response, Session
-from products.warehouse_sources.sources.semrush.semrush import semrush_source, validate_credentials
+from products.warehouse_sources.sources.sdk import RESTClientRetryableError
+from products.warehouse_sources.sources.semrush.semrush import (
+ DOMAIN_REPORT_ROW_ERROR,
+ semrush_source,
+ validate_credentials,
+)
from products.warehouse_sources.sources.semrush.settings import (
ACCESS_ERROR,
AUTH_ERROR,
+ DATABASE_ACCESS_ERROR,
+ DOMAIN_ERROR,
+ DOMAIN_REPORT_ERROR,
PROJECT_ERROR,
PROJECT_ID_ERROR,
QUOTA_ERROR,
@@ -97,3 +107,267 @@ def test_malformed_records_fail_instead_of_erasing_rows(endpoint: str, body: obj
with patch.object(Session, "send", return_value=response(body)):
with pytest.raises(ValueError, match=message):
list(cast(Iterable[Any], semrush_source("fake-api-key", "123", endpoint, 1, "job").items()))
+
+
+def text_response(body: str, status: int = 200) -> Response:
+ result = Response()
+ result.status_code = status
+ result._content = body.encode()
+ result.headers["Content-Type"] = "text/plain"
+ return result
+
+
+def routed(routes: dict[str, Callable[[dict[str, list[str]]], Response]], seen: list[str]) -> Callable[..., Response]:
+ def send(request: PreparedRequest, **kwargs: Any) -> Response:
+ url = urlparse(request.url or "")
+ seen.append(f"{url.path}?{url.query}")
+ query = parse_qs(url.query)
+ for prefix, handler in routes.items():
+ if url.path == prefix:
+ return handler(query)
+ raise AssertionError(f"unexpected request {url.path}")
+
+ return send
+
+
+def rows_of(endpoint: str, **kwargs: Any) -> list[dict[str, Any]]:
+ response = semrush_source("fake-api-key", "123", endpoint, 1, "job", **kwargs)
+ return [row for page in cast(Iterable[Any], response.items()) for row in page]
+
+
+SNAPSHOTS = {
+ "snapshots": [
+ {"snapshot_id": "aaa111", "finish_date": 1000},
+ {"snapshot_id": "bbb222", "finish_date": 3000},
+ {"snapshot_id": "ccc333", "finish_date": 2000},
+ ]
+}
+
+
+def snapshot_details(query: dict[str, list[str]]) -> Response:
+ snapshot_id = query["snapshot_id"][0]
+ return response({"snapshot_id": snapshot_id, "quality": {"value": 90, "delta": 1}, "pages_crawled": 10})
+
+
+ "incremental,last_value,expected",
+ [
+ (False, None, ["aaa111", "bbb222", "ccc333"]),
+ (False, 2500, ["aaa111", "bbb222", "ccc333"]),
+ (True, None, ["aaa111", "bbb222", "ccc333"]),
+ (True, 1000, ["bbb222", "ccc333"]),
+ (True, 3000, []),
+ ],
+)
+def test_history_fetches_details_only_for_snapshots_past_the_watermark(
+ incremental: bool, last_value: int | None, expected: list[str]
+) -> None:
+ seen: list[str] = []
+ routes = {
+ "/reports/v1/projects/123/siteaudit/snapshots": lambda _: response(SNAPSHOTS),
+ "/reports/v1/projects/123/siteaudit/snapshot": snapshot_details,
+ }
+ with patch.object(Session, "send", side_effect=routed(routes, seen)):
+ rows = rows_of(
+ "site_audit_history", should_use_incremental_field=incremental, db_incremental_field_last_value=last_value
+ )
+
+ assert sorted(row["snapshot_id"] for row in rows) == expected
+ assert len(seen) == 1 + len(expected)
+ finish_dates = {item["snapshot_id"]: item["finish_date"] for item in SNAPSHOTS["snapshots"]}
+ for row in rows:
+ assert row["project_id"] == "123"
+ assert row["finish_date"] == finish_dates[row["snapshot_id"]]
+ assert row["quality"] == {"value": 90, "delta": 1}
+ assert not any(key.startswith("_site_audit_snapshots") for key in row)
+
+
[email protected]("snapshot_id", [None, "../456", "abc?key=x"])
+def test_history_rejects_snapshot_ids_that_would_change_the_request(snapshot_id: str | None) -> None:
+ seen: list[str] = []
+ routes = {
+ "/reports/v1/projects/123/siteaudit/snapshots": lambda _: response(
+ {"snapshots": [{"snapshot_id": snapshot_id, "finish_date": 1}]}
+ ),
+ }
+ with patch.object(Session, "send", side_effect=routed(routes, seen)):
+ with pytest.raises(ValueError, match="record without its ID"):
+ rows_of("site_audit_history")
+ assert len(seen) == 1
+
+
+def issue_pages(query: dict[str, list[str]], total: int, page_size: int) -> Response:
+ page = int(query["page"][0])
+ start = (page - 1) * page_size
+ count = max(0, min(page_size, total - start))
+ data = [
+ {"source_url": f"https://example.com/{start + i}", "target_url": None, "page_id": f"p{start + i}"}
+ for i in range(count)
+ ]
+ return response({"limit": page_size, "page": page, "total": total, "data": data})
+
+
+def test_issue_details_walks_pages_of_detected_issues_in_the_latest_snapshot() -> None:
+ seen: list[str] = []
+ base = "/reports/v1/projects/123/siteaudit"
+ routes = {
+ f"{base}/snapshots": lambda _: response(SNAPSHOTS),
+ f"{base}/info": lambda _: response({"id": 123, "defects": {"8": 150, "109": 2, "110": 0}}),
+ f"{base}/snapshot/bbb222/issue/8": lambda query: issue_pages(query, total=150, page_size=100),
+ f"{base}/snapshot/bbb222/issue/109": lambda query: issue_pages(query, total=2, page_size=100),
+ }
+ with patch.object(Session, "send", side_effect=routed(routes, seen)):
+ rows = rows_of("site_audit_issue_details")
+
+ assert sum(row["issue_id"] == 8 for row in rows) == 150
+ assert sum(row["issue_id"] == 109 for row in rows) == 2
+ assert all(row["snapshot_id"] == "bbb222" and row["project_id"] == "123" for row in rows)
+ issue_requests = [request for request in seen if "/issue/" in request]
+ assert len(issue_requests) == 3
+ assert all("limit=100" in request for request in issue_requests)
+
+
+def test_issue_details_skips_an_issue_that_is_gone() -> None:
+ seen: list[str] = []
+ base = "/reports/v1/projects/123/siteaudit"
+ routes = {
+ f"{base}/snapshots": lambda _: response(SNAPSHOTS),
+ f"{base}/info": lambda _: response({"id": 123, "defects": {"8": 1, "9": 1}}),
+ f"{base}/snapshot/bbb222/issue/8": lambda _: response({"message": "Not found"}, 404),
+ f"{base}/snapshot/bbb222/issue/9": lambda query: issue_pages(query, total=1, page_size=100),
+ }
+ with patch.object(Session, "send", side_effect=routed(routes, seen)):
+ rows = rows_of("site_audit_issue_details")
+
+ assert [row["issue_id"] for row in rows] == [9]
+
+
+def test_issue_details_without_audits_reads_nothing_else() -> None:
+ seen: list[str] = []
+ routes = {"/reports/v1/projects/123/siteaudit/snapshots": lambda _: response({"snapshots": []})}
+ with patch.object(Session, "send", side_effect=routed(routes, seen)):
+ assert rows_of("site_audit_issue_details") == []
+ assert len(seen) == 1
+
+
+KEYWORDS_CSV = (
+ "Keyword;Position;Previous Position;Position Difference;Search Volume;CPC;Url;Traffic (%);"
+ "Traffic Cost (%);Competition;Number of Results;Trends;Keyword Difficulty Index;Timestamp\r\n"
+ '"seo; tools";"6";"7";"1";"8100";"10.54";"http://example.com/";"2.15";"3.87";"0.54";"321000000";'
+ '"0.67,0.82";"71";"1700000000"\r\n'
+ '"seo";"9";"";"0";"110000";"14.82";"http://example.com/a";"17.53";"44.40";"0.50";"0";"0.81";"";"1700000000"\r\n'
+)
+
+
+ "domain,database,expected_domain,expected_database,info_requests",
+ [
+ ("example.com", "UK", "example.com", "uk", 0),
+ ("https://www.example.com/blog", None, "www.example.com", "us", 0),
+ (None, "", "example.org", "us", 1),
+ (" ", "de", "example.org", "de", 1),
+ ],
+)
+def test_domain_organic_keywords_request_and_rows(
+ domain: str | None, database: str | None, expected_domain: str, expected_database: str, info_requests: int
+) -> None:
+ seen: list[str] = []
+ queries: list[dict[str, list[str]]] = []
+
+ def report(query: dict[str, list[str]]) -> Response:
+ queries.append(query)
+ return text_response(KEYWORDS_CSV)
+
+ routes = {
+ "/reports/v1/projects/123/siteaudit/info": lambda _: response({"id": 123, "url": "https://example.org/"}),
+ "/": report,
+ }
+ with patch.object(Session, "send", side_effect=routed(routes, seen)):
+ rows = rows_of("domain_organic_keywords", domain=domain, database=database)
+
+ assert len(seen) == 1 + info_requests
+ (query,) = queries
+ assert query["type"] == ["domain_organic"]
+ assert query["domain"] == [expected_domain]
+ assert query["database"] == [expected_database]
+ assert query["display_limit"] == ["10000"]
+ assert query["export_columns"] == ["Ph,Po,Pp,Pd,Nq,Cp,Ur,Tr,Tc,Co,Nr,Td,Kd,Ts"]
+ assert rows[0] == {
+ "domain": expected_domain,
+ "database": expected_database,
+ "keyword": "seo; tools",
+ "position": 6,
+ "previous_position": 7,
+ "position_difference": 1,
+ "search_volume": 8100,
+ "cpc": 10.54,
+ "url": "http://example.com/",
+ "traffic_percent": 2.15,
+ "traffic_cost_percent": 3.87,
+ "competition": 0.54,
+ "number_of_results": 321000000,
+ "trends": "0.67,0.82",
+ "keyword_difficulty": 71.0,
+ "timestamp": 1700000000,
+ }
+ assert rows[1]["previous_position"] is None
+ assert rows[1]["keyword_difficulty"] is None
+
+
+def test_domain_organic_competitors_rows() -> None:
+ body = (
+ "Domain;Competitor Relevance;Common Keywords;Organic Keywords;Organic Traffic;Organic Cost;Adwords Keywords\n"
+ "rival.example.com;0.13;338;11021;5640;9690;0\n"
+ )
+ seen: list[str] = []
+ with patch.object(Session, "send", side_effect=routed({"/": lambda _: text_response(body)}, seen)):
+ rows = rows_of("domain_organic_competitors", domain="example.com")
+
+ assert "type=domain_organic_organic" in seen[0]
+ assert rows == [
+ {
+ "domain": "example.com",
+ "database": "us",
+ "competitor_domain": "rival.example.com",
+ "competitor_relevance": 0.13,
+ "common_keywords": 338,
+ "organic_keywords": 11021,
+ "organic_traffic": 5640,
+ "organic_cost": 9690.0,
+ "adwords_keywords": 0,
+ }
+ ]
+
+
+ "status,body,error,message",
+ [
+ (200, "ERROR 50 :: NOTHING FOUND", None, None),
+ (400, "ERROR 120 :: WRONG KEY - ID PAIR", ValueError, AUTH_ERROR),
+ (200, "ERROR 132 :: API UNITS BALANCE IS ZERO", ValueError, QUOTA_ERROR),
+ (200, "ERROR 133 :: DB ACCESS DENIED", ValueError, DATABASE_ACCESS_ERROR),
+ (400, "ERROR 10000 :: Wrong parameter 'database'.", ValueError, DOMAIN_REPORT_ERROR),
+ (200, "ERROR 500 :: Internal Error", RESTClientRetryableError, UNAVAILABLE_ERROR),
+ (403, "Forbidden", ValueError, ACCESS_ERROR),
+ (200, "Domain;Competitor Relevance\nrival.example.com;0.13\n", ValueError, DOMAIN_REPORT_ROW_ERROR),
+ ],
+)
+def test_domain_report_errors(status: int, body: str, error: type[Exception] | None, message: str | None) -> None:
+ seen: list[str] = []
+ with patch.object(Session, "send", side_effect=routed({"/": lambda _: text_response(body, status)}, seen)):
+ if error is None:
+ assert rows_of("domain_organic_competitors", domain="example.com") == []
+ else:
+ with pytest.raises(error, match=re.escape(message or "")):
+ rows_of("domain_organic_competitors", domain="example.com")
+ if message in {DATABASE_ACCESS_ERROR, DOMAIN_REPORT_ERROR}:
+ assert message in SemrushSource().get_non_retryable_errors()
+
+
+def test_domain_report_without_a_project_url_asks_for_a_domain() -> None:
+ seen: list[str] = []
+ routes = {"/reports/v1/projects/123/siteaudit/info": lambda _: response({"id": 123, "url": ""})}
+ with patch.object(Session, "send", side_effect=routed(routes, seen)):
+ with pytest.raises(ValueError, match=DOMAIN_ERROR):
+ rows_of("domain_organic_keywords")
+ assert len(seen) == 1