Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse-sources): add commercetools variants table (#119233)

Tom Owerse9a617b24d07
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 4b8070eb91ee..377c33a9eb84 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -3087,7 +3087,7 @@ Note: coinmarketcap.com/api/documentation is a client-rendered zudoku app, but e
## Commercetools — gaps
-Today (16): `carts`, `categories`, `channels`, `customer_groups`, `customers`, `discount_codes`, `inventory`, `messages`, `orders`, `payments`, `product_projections`, `product_types`, `shopping_lists`, `standalone_prices`, `states`, `stores`
+Today (17): `carts`, `categories`, `channels`, `customer_groups`, `customers`, `discount_codes`, `inventory`, `messages`, `orders`, `payments`, `product_projections`, `product_types`, `shopping_lists`, `standalone_prices`, `states`, `stores`, `variants`
Diffed against: <https://raw.githubusercontent.com/commercetools/commercetools-api-reference/main/oas/api/openapi.yaml>
@@ -3103,6 +3103,7 @@ Diffed against: <https://raw.githubusercontent.com/commercetools/commercetools-a
- [ ] `/{projectKey}/product-discounts` — explains discounted prices appearing on product projections (medium)
- [x] `/{projectKey}/standalone-prices` — prices held outside the product, so price analysis on product-projections alone is incomplete (medium)
- [x] `/{projectKey}/shopping-lists` — wishlist and saved-cart behavior, a common pre-purchase funnel table (medium)
+- [x] `/{projectKey}/variants` — standalone Product Variants on Modular Catalog projects, whose product projections carry no variants (medium)
Note: The full OpenAPI (292 paths) is published at commercetools/commercetools-api-reference under oas/api/openapi.yaml; the docs site itself is navigational only. Static endpoint config in sources/commercetools/settings.py, no dynamic discovery. Also uncovered but lower value: reviews, payment-methods, business-units and associate-roles (B2B), quotes/quote-requests/staged-quotes, recurring-orders and recurrence-policies, tax-categories, zones, orders/edits. Excluded as config or plumbing: subscriptions, extensions, api-clients, types, custom-objects, product-selections.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/canonical_descriptions.py
index 16097d1c257e..e70ef6cfad71 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/canonical_descriptions.py
@@ -114,6 +114,17 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
published="Whether the product is currently published.",
),
},
+ "variants": {
+ "description": "A Product Variant managed as a standalone resource, with its current (published) and staged (draft) data. Only available for projects using the Modular catalog model, where product projections carry no variants.",
+ "docs_url": "https://docs.commercetools.com/api/projects/variants",
+ "columns": _columns(
+ variantId="Sequential identifier of the variant within its product.",
+ product="Reference to the product the variant belongs to.",
+ published="Whether the variant is currently published.",
+ current="Current (published) data of the variant, including SKU, attributes and images.",
+ staged="Staged (draft) data of the variant, including unpublished changes.",
+ ),
+ },
"product_types": {
"description": "A product type defining the common custom attributes shared by many products. Products and product projections reference the product type they were created from.",
"docs_url": "https://docs.commercetools.com/api/projects/productTypes",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/settings.py
index d1dc40fa4b53..d3e346950658 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/settings.py
@@ -26,6 +26,9 @@ class CommercetoolsEndpointConfig:
partition_key: str = "createdAt"
+# Endpoints that only exist on some projects (variants: Modular catalog model only).
+_OPT_IN_ENDPOINTS = frozenset({"variants"})
+
COMMERCETOOLS_ENDPOINTS: dict[str, CommercetoolsEndpointConfig] = {
"orders": CommercetoolsEndpointConfig(
name="orders",
@@ -62,6 +65,13 @@ COMMERCETOOLS_ENDPOINTS: dict[str, CommercetoolsEndpointConfig] = {
path="/product-types",
scope="view_product_types",
),
+ # Only available on projects using the Modular catalog model; Classic projects
+ # keep variants embedded in product_projections, so it stays off by default.
+ "variants": CommercetoolsEndpointConfig(
+ name="variants",
+ path="/variants",
+ scope="view_products",
+ ),
"categories": CommercetoolsEndpointConfig(
name="categories",
path="/categories",
@@ -118,3 +128,6 @@ ENDPOINTS = tuple(COMMERCETOOLS_ENDPOINTS.keys())
INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {
name: config.incremental_fields for name, config in COMMERCETOOLS_ENDPOINTS.items() if config.incremental_fields
}
+
+# Tables that only exist on some projects stay off until the user opts in.
+SHOULD_SYNC_DEFAULT: dict[str, bool] = {name: name not in _OPT_IN_ENDPOINTS for name in COMMERCETOOLS_ENDPOINTS}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/source.py
index 2d1bb3564aba..c898daeaab23 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/commercetools/source.py
@@ -18,6 +18,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.commerceto
from products.warehouse_sources.backend.temporal.data_imports.sources.commercetools.settings import (
ENDPOINTS,
INCREMENTAL_FIELDS,
+ SHOULD_SYNC_DEFAULT,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.base import FieldType, ResumableSource
from products.warehouse_sources.backend.temporal.data_imports.sources.common.canonical_descriptions import (
@@ -242,7 +243,7 @@ The messages table also needs the Messages Query feature turned on in your proje
force_refresh: bool = False,
api_version: str | None = None,
) -> list[SourceSchema]:
- return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names)
+ return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names, should_sync_default=SHOULD_SYNC_DEFAULT)
def validate_credentials(
self,