Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 6 missing tables to the hyros source (#122036)

Tom Owers7a3b452d3b5c
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index c9c97a27ef73..50c4ee489ce3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -6175,16 +6175,16 @@ Note: Enumerated the API reference from https://docs.hyperspell.com/llms.txt, th
## Hyros — gaps
-Today (8): `Calls`, `Keywords`, `Leads`, `Sales`, `Sources`, `Stages`, `Subscriptions`, `Tags`
+Today (14): `AdAccounts`, `Ads`, `CallConversionPaths`, `Calls`, `Keywords`, `LeadConversionPaths`, `Leads`, `Products`, `SaleConversionPaths`, `Sales`, `Sources`, `Stages`, `Subscriptions`, `Tags`
Diffed against: <https://api-docs.hyros.com/ai-context/rest-api.txt>
-- [ ] `Ad Accounts (GET /api/v1.0/ad-accounts)` — resolve adAccountId on sources and attribution records to account names and platforms (high)
-- [ ] `Ads (GET /api/v1.0/ads)` — resolve adSourceId on sources and attribution records to individual ads (high)
-- [ ] `Products (GET /api/v1.0/products)` — add product prices and cost of goods to sales for profit analysis, using their existing product IDs (high)
+- [x] `Ad Accounts (GET /api/v1.0/ad-accounts)` — resolve adAccountId on sources and attribution records to account names and platforms (high)
+- [x] `Ads (GET /api/v1.0/ads)` — resolve adSourceId on sources and attribution records to individual ads (high)
+- [x] `Products (GET /api/v1.0/products)` — add product prices and cost of goods to sales for profit analysis, using their existing product IDs (high)
- [ ] `Ads Attribution Report (GET /api/v1.0/attribution)` — produce vendor attribution metrics and ad spend across attribution models and reporting levels (high)
- [ ] `ROAS (GET /api/v1.0/attribution/roas)` — measure collected revenue against ad spend using the vendor's fixed return calculation (high)
-- [ ] `Conversion Paths (GET /api/v1.0/conversion-paths)` — analyze ordered source touches for each conversion and calculate custom attribution models (high)
+- [x] `Conversion Paths (GET /api/v1.0/conversion-paths)` — analyze ordered source touches for each conversion and calculate custom attribution models (high)
- [ ] `Lead Clicks (GET /api/v1.0/leads/clicks)` — analyze click histories behind attributed sales instead of only first and last touches (medium)
- [ ] `Leads Journey (GET /api/v1.0/leads/journey)` — measure stage changes, opt-ins, email activity, and other events across each lead's timeline (medium)
- [ ] `Custom Costs (GET /api/v1.0/custom-costs)` — include recurring and one-time business costs in campaign profitability calculations (medium)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/canonical_descriptions.py
index b84250a2d2a4..9e5158e25719 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/canonical_descriptions.py
@@ -125,4 +125,79 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"amount": "Number of leads currently in this stage.",
},
},
+ "AdAccounts": {
+ "description": "An ad account connected to Hyros, such as a Meta or Google Ads account.",
+ "docs_url": "https://api-docs.hyros.com",
+ "columns": {
+ "id": "Ad account identifier on the ad platform. Matches `adAccountId` on sources and attribution records.",
+ "name": "Name of the ad account.",
+ "type": "Ad platform the account belongs to, for example FACEBOOK or GOOGLE.",
+ },
+ },
+ "Ads": {
+ "description": "An individual ad from a connected ad platform, with the Hyros source it is tracked by.",
+ "docs_url": "https://api-docs.hyros.com",
+ "columns": {
+ "adSourceId": "Ad identifier on the ad platform. Matches `adSourceId` on sources and attribution records.",
+ "platform": "Ad platform the ad runs on, for example FACEBOOK or GOOGLE.",
+ "name": "Name of the ad.",
+ "adSource": "Ad platform details: ad id, ad account id and platform.",
+ "source": "Hyros source that tracks the ad.",
+ "creationDate": "Date the ad was created in Hyros, in epoch milliseconds.",
+ },
+ },
+ "Products": {
+ "description": "A product sold in the account, with its price and cost of goods.",
+ "docs_url": "https://api-docs.hyros.com",
+ "columns": {
+ "id": "Unique identifier for the product.",
+ "name": "Name of the product.",
+ "tag": "Product tag, prefixed with `$`, used to tag sales of the product.",
+ "sku": "Stock keeping unit of the product.",
+ "price": "Price of the product.",
+ "customCost": "Cost of goods of the product, used in profit and ROAS reporting.",
+ "recurring": "Whether the product is sold as a recurring sale.",
+ "callProduct": "Whether the product is a call product.",
+ "category": "Name of the product category.",
+ },
+ },
+ "SaleConversionPaths": {
+ "description": "A sale with the ordered source touches it was attributed with, up to the sale date.",
+ "docs_url": "https://api-docs.hyros.com",
+ "columns": {
+ "id": "Identifier of the conversion: the sale id (`sle-` prefixed) for sales, the call id (`cll-` prefixed) for calls, or the lead id for leads.",
+ "leadId": "Identifier of the lead the conversion belongs to.",
+ "orderId": "Identifier of the order the sale belongs to. Sales that share it were made in the same purchase.",
+ "creationDate": "Date of the conversion in ISO 8601, in the account timezone: the sale date for sales and calls, the join date for leads.",
+ "path": "Source touches the conversion was attributed with, ordered by click date ascending.",
+ "price": "Price of the conversion, with discount, hard cost, refunded amount and currency.",
+ "usdPrice": "Same amounts as `price`, converted to USD.",
+ "firstSale": "Whether this is the lead's first conversion of this type, which separates it from repeat conversions.",
+ },
+ },
+ "CallConversionPaths": {
+ "description": "A call with the ordered source touches it was attributed with, up to the call date.",
+ "docs_url": "https://api-docs.hyros.com",
+ "columns": {
+ "id": "Identifier of the conversion: the sale id (`sle-` prefixed) for sales, the call id (`cll-` prefixed) for calls, or the lead id for leads.",
+ "leadId": "Identifier of the lead the conversion belongs to.",
+ "orderId": "Identifier of the order the sale belongs to. Sales that share it were made in the same purchase.",
+ "creationDate": "Date of the conversion in ISO 8601, in the account timezone: the sale date for sales and calls, the join date for leads.",
+ "path": "Source touches the conversion was attributed with, ordered by click date ascending.",
+ "price": "Price of the conversion, with discount, hard cost, refunded amount and currency.",
+ "usdPrice": "Same amounts as `price`, converted to USD.",
+ "firstSale": "Whether this is the lead's first conversion of this type, which separates it from repeat conversions.",
+ },
+ },
+ "LeadConversionPaths": {
+ "description": "A lead with its full ordered list of source touches.",
+ "docs_url": "https://api-docs.hyros.com",
+ "columns": {
+ "id": "Identifier of the conversion: the sale id (`sle-` prefixed) for sales, the call id (`cll-` prefixed) for calls, or the lead id for leads.",
+ "leadId": "Identifier of the lead the conversion belongs to.",
+ "orderId": "Identifier of the order the sale belongs to. Sales that share it were made in the same purchase.",
+ "creationDate": "Date of the conversion in ISO 8601, in the account timezone: the sale date for sales and calls, the join date for leads.",
+ "path": "Source touches the conversion was attributed with, ordered by click date ascending.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/hyros.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/hyros.py
index b1ecba49032d..4ae3dd7529eb 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/hyros.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/hyros.py
@@ -42,11 +42,16 @@ def _format_hyros_date(value: Any) -> Optional[str]:
return str(value)
+def _lift_ad_source_keys(item: dict[str, Any]) -> dict[str, Any]:
+ ad_source = item.get("adSource") or {}
+ return {**item, "platform": ad_source.get("platform"), "adSourceId": ad_source.get("adSourceId")}
+
+
def _resource_for(endpoint: str, should_use_incremental_field: bool) -> EndpointResource:
config = HYROS_ENDPOINTS[endpoint]
is_incremental = should_use_incremental_field and config.incremental_query_param is not None
- params: dict[str, Any] = {"pageSize": PAGE_SIZE}
+ params: dict[str, Any] = {**config.params, "pageSize": PAGE_SIZE}
if config.incremental_query_param is not None:
params[config.incremental_query_param] = (
{
@@ -59,7 +64,7 @@ def _resource_for(endpoint: str, should_use_incremental_field: bool) -> Endpoint
else None
)
- return {
+ resource: EndpointResource = {
"name": endpoint,
"table_name": endpoint.lower(),
"write_disposition": {"disposition": "merge", "strategy": "upsert"} if is_incremental else "replace",
@@ -70,6 +75,9 @@ def _resource_for(endpoint: str, should_use_incremental_field: bool) -> Endpoint
},
"table_format": "delta",
}
+ if endpoint == "Ads":
+ resource["data_map"] = _lift_ad_source_keys
+ return resource
def _client_config(api_key: str) -> ClientConfig:
@@ -132,10 +140,7 @@ def hyros_source(
partition_mode="datetime" if has_partition else None,
partition_format="month" if has_partition else None,
partition_keys=[config.partition_key] if config.partition_key else None,
- # Hyros doesn't document the order `pageId` cursoring returns rows in. Ascending is the
- # conservative default the framework and every other cursor-paginated source here assume;
- # flag it explicitly since it's unverified rather than confirmed against the live API.
- sort_mode="asc",
+ sort_mode=config.sort_mode,
column_hints=resource.column_hints,
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/settings.py
index f4075c353c55..d68fc1c2a841 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/settings.py
@@ -1,8 +1,9 @@
from dataclasses import field
-from typing import Optional
+from typing import Any, Optional
from posthog.dataclasses import frozen
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SortMode
from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldType
# Hyros caps `pageSize` at 250 on every cursor-paginated list endpoint.
@@ -31,6 +32,12 @@ class HyrosEndpointConfig:
incremental_fields: list[IncrementalField] = field(default_factory=list)
# Stable creation-time field to partition by. None when there's no reliable created-date column.
partition_key: Optional[str] = None
+ # Fixed query parameters sent on every request, e.g. a required enum filter.
+ params: dict[str, Any] = field(default_factory=dict)
+ # Hyros doesn't document the order `pageId` cursoring returns rows in. Ascending is the
+ # conservative default the framework and every other cursor-paginated source here assume;
+ # flag it explicitly since it's unverified rather than confirmed against the live API.
+ sort_mode: Optional[SortMode] = "asc"
@property
def incremental_field_name(self) -> Optional[str]:
@@ -98,6 +105,51 @@ HYROS_ENDPOINTS: dict[str, HyrosEndpointConfig] = {
path="/api/v1.0/stages",
primary_keys=["name"],
),
+ "AdAccounts": HyrosEndpointConfig(
+ name="AdAccounts",
+ path="/api/v1.0/ad-accounts",
+ primary_keys=["id"],
+ ),
+ "Ads": HyrosEndpointConfig(
+ name="Ads",
+ path="/api/v1.0/ads",
+ # Ad rows carry no `id`; the platform ad id lives on the nested `adSource` and is lifted to
+ # the row root in transport. Ad ids are only unique within a platform.
+ primary_keys=["platform", "adSourceId"],
+ ),
+ "Products": HyrosEndpointConfig(
+ name="Products",
+ path="/api/v1.0/products",
+ primary_keys=["id"],
+ ),
+ **{
+ name: HyrosEndpointConfig(
+ name=name,
+ path="/api/v1.0/conversion-paths",
+ primary_keys=["id"],
+ # `fromDate` filters on the sale date, which is what `creationDate` holds. The path only
+ # covers touches up to that date, so a synced row doesn't change later.
+ incremental_query_param="fromDate",
+ incremental_fields=_incremental_field("creationDate"),
+ partition_key="creationDate",
+ params={"conversionType": conversion_type},
+ # The API doesn't document the order conversions are paged in.
+ sort_mode=None,
+ )
+ for name, conversion_type in (
+ ("SaleConversionPaths", "SALE"),
+ ("CallConversionPaths", "CALL"),
+ )
+ },
+ "LeadConversionPaths": HyrosEndpointConfig(
+ name="LeadConversionPaths",
+ path="/api/v1.0/conversion-paths",
+ primary_keys=["id"],
+ # Full refresh only: `fromDate` filters on the join date, but `path` is the lead's full touch
+ # list and keeps growing after it joins.
+ partition_key="creationDate",
+ params={"conversionType": "LEAD"},
+ ),
}
ENDPOINTS = tuple(HYROS_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/tests/test_hyros.py b/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/tests/test_hyros.py
index 90877aab2087..f6985a11a3ac 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/tests/test_hyros.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/hyros/tests/test_hyros.py
@@ -99,6 +99,46 @@ class TestHyrosSourcePagination:
assert result.raised is None
assert result.requests[0].param(param_name) == "2026-03-04T02:58:14+00:00"
+ @parameterized.expand(
+ [
+ ("sale_full_refresh", "SaleConversionPaths", "SALE", False, False),
+ ("call_incremental", "CallConversionPaths", "CALL", True, True),
+ ("lead_never_filters_by_join_date", "LeadConversionPaths", "LEAD", True, False),
+ ]
+ )
+ def test_conversion_paths_send_conversion_type(
+ self, _name: str, endpoint: str, conversion_type: str, incremental: bool, expect_from_date: bool
+ ) -> None:
+ result = _driver().run(
+ endpoint,
+ [_response([{"id": "1"}], next_page_id="cursor-1"), _response([{"id": "2"}])],
+ should_use_incremental_field=incremental,
+ db_incremental_field_last_value=datetime(2026, 3, 4, tzinfo=UTC) if incremental else None,
+ )
+
+ assert result.raised is None
+ assert result.paths == ["/v1/api/v1.0/conversion-paths"] * 2
+ expected: dict[str, tuple[str, ...]] = {"conversionType": (conversion_type,), "pageSize": ("250",)}
+ if expect_from_date:
+ expected["fromDate"] = ("2026-03-04T00:00:00+00:00",)
+ assert result.queries == [expected, {**expected, "pageId": ("cursor-1",)}]
+
+ def test_ads_lift_ad_source_keys_to_row_root(self) -> None:
+ result = _driver().run(
+ "Ads",
+ [
+ _response(
+ [
+ {"name": "ad", "adSource": {"adSourceId": "238", "adAccountId": "887", "platform": "FACEBOOK"}},
+ {"name": "orphan"},
+ ]
+ )
+ ],
+ )
+
+ assert result.raised is None
+ assert [(row["platform"], row["adSourceId"]) for row in result.rows] == [("FACEBOOK", "238"), (None, None)]
+
def test_full_refresh_endpoint_never_filters_by_date(self) -> None:
result = _driver().run(
"Stages",