:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse-sources): add commercetools variants table (#119233)
Tom Owerse9a617b24d07
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 4b8070eb91ee..377c33a9eb84 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -3087,7 +3087,7 @@ Note: coinmarketcap.com/api/documentation is a client-rendered zudoku app, but e## Commercetools — gaps-Today (16): `carts`, `categories`, `channels`, `customer_groups`, `customers`, `discount_codes`, `inventory`, `messages`, `orders`, `payments`, `product_projections`, `product_types`, `shopping_lists`, `standalone_prices`, `states`, `stores`+Today (17): `carts`, `categories`, `channels`, `customer_groups`, `customers`, `discount_codes`, `inventory`, `messages`, `orders`, `payments`, `product_projections`, `product_types`, `shopping_lists`, `standalone_prices`, `states`, `stores`, `variants`Diffed against: <https://raw.githubusercontent.com/commercetools/commercetools-api-reference/main/oas/api/openapi.yaml>@@ -3103,6 +3103,7 @@ Diffed against: <https://raw.githubusercontent.com/commercetools/commercetools-a- [ ] `/{projectKey}/product-discounts` — explains discounted prices appearing on product projections (medium)- [x] `/{projectKey}/standalone-prices` — prices held outside the product, so price analysis on product-projections alone is incomplete (medium)- [x] `/{projectKey}/shopping-lists` — wishlist and saved-cart behavior, a common pre-purchase funnel table (medium)+- [x] `/{projectKey}/variants` — standalone Product Variants on Modular Catalog projects, whose product projections carry no variants (medium)Note: The full OpenAPI (292 paths) is published at commercetools/commercetools-api-reference under oas/api/openapi.yaml; the docs site itself is navigational only. Static endpoint config in sources/commercetools/settings.py, no dynamic discovery. Also uncovered but lower value: reviews, payment-methods, business-units and associate-roles (B2B), quotes/quote-requests/staged-quotes, recurring-orders and recurrence-policies, tax-categories, zones, orders/edits. Excluded as config or plumbing: subscriptions, extensions, api-clients, types, custom-objects, product-selections.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/canonical_descriptions.pyindex 16097d1c257e..e70ef6cfad71 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/canonical_descriptions.py@@ -114,6 +114,17 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {published="Whether the product is currently published.",),},+ "variants": {+ "description": "A Product Variant managed as a standalone resource, with its current (published) and staged (draft) data. Only available for projects using the Modular catalog model, where product projections carry no variants.",+ "docs_url": "https://docs.commercetools.com/api/projects/variants",+ "columns": _columns(+ variantId="Sequential identifier of the variant within its product.",+ product="Reference to the product the variant belongs to.",+ published="Whether the variant is currently published.",+ current="Current (published) data of the variant, including SKU, attributes and images.",+ staged="Staged (draft) data of the variant, including unpublished changes.",+ ),+ },"product_types": {"description": "A product type defining the common custom attributes shared by many products. Products and product projections reference the product type they were created from.","docs_url": "https://docs.commercetools.com/api/projects/productTypes",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/settings.pyindex d1dc40fa4b53..d3e346950658 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/settings.py@@ -26,6 +26,9 @@ class CommercetoolsEndpointConfig:partition_key: str = "createdAt"+# Endpoints that only exist on some projects (variants: Modular catalog model only).+_OPT_IN_ENDPOINTS = frozenset({"variants"})+COMMERCETOOLS_ENDPOINTS: dict[str, CommercetoolsEndpointConfig] = {"orders": CommercetoolsEndpointConfig(name="orders",@@ -62,6 +65,13 @@ COMMERCETOOLS_ENDPOINTS: dict[str, CommercetoolsEndpointConfig] = {path="/product-types",scope="view_product_types",),+ # Only available on projects using the Modular catalog model; Classic projects+ # keep variants embedded in product_projections, so it stays off by default.+ "variants": CommercetoolsEndpointConfig(+ name="variants",+ path="/variants",+ scope="view_products",+ ),"categories": CommercetoolsEndpointConfig(name="categories",path="/categories",@@ -118,3 +128,6 @@ ENDPOINTS = tuple(COMMERCETOOLS_ENDPOINTS.keys())INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {name: config.incremental_fields for name, config in COMMERCETOOLS_ENDPOINTS.items() if config.incremental_fields}++# Tables that only exist on some projects stay off until the user opts in.+SHOULD_SYNC_DEFAULT: dict[str, bool] = {name: name not in _OPT_IN_ENDPOINTS for name in COMMERCETOOLS_ENDPOINTS}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/source.pyindex 2d1bb3564aba..c898daeaab23 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/source.py@@ -18,6 +18,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.commercetofrom products.warehouse_sources.backend.temporal.data_imports.sources.commercetools.settings import (ENDPOINTS,INCREMENTAL_FIELDS,+ SHOULD_SYNC_DEFAULT,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import FieldType, ResumableSourcefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.canonical_descriptions import (@@ -242,7 +243,7 @@ The messages table also needs the Messages Query feature turned on in your projeforce_refresh: bool = False,api_version: str | None = None,) -> list[SourceSchema]:- return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names)+ return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names, should_sync_default=SHOULD_SYNC_DEFAULT)def validate_credentials(self,