:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add membrain process, label and opportunity product tables (#123044)
Tom Owerse3e7dbb24295
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex fee96f25e757..e29f47de6b80 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -7770,15 +7770,15 @@ Today (11): `account_growth_items`, `activities`, `companies`, `contacts`, `custDiffed against: <https://www.membrain.com/developers/api-documentation>-- [ ] `Sales processes (GET /API/v2/salesProcesses/)` — resolve opportunity process and stage IDs, including stage order and process-specific fields (high)-- [ ] `Prospecting processes (GET /API/v2/prospectingProcesses/)` — resolve prospect process and stage IDs for conversion analysis (high)+- [x] `Sales processes (GET /API/v2/salesProcesses/)` — resolve opportunity process and stage IDs, including stage order and process-specific fields (high)+- [x] `Prospecting processes (GET /API/v2/prospectingProcesses/)` — resolve prospect process and stage IDs for conversion analysis (high)- [x] `Account Growth processes (GET /API/v2/accountGrowthProcesses/)` — resolve account-growth process and stage IDs for expansion analysis (high)- [x] `Ticket processes (GET /API/v2/ticketProcesses/)` — resolve ticket process and stage IDs for support workflow analysis (high)- [x] `Currencies (GET /API/v2/currencies/)` — resolve ValueCurrencyId and supply exchange rates for comparable pipeline values (high)-- [ ] `Labels (GET /API/v2/labels/)` — resolve LabelIds on companies and contacts for consistent segmentation (high)+- [x] `Labels (GET /API/v2/labels/)` — resolve LabelIds on companies and contacts for consistent segmentation (high)- [x] `Activity types (GET /API/v2/activityTypes/)` — resolve ActivityTypeId and provide the complete activity taxonomy (high)- [ ] `Process Container Contacts (GET /API/v2/processContainerContacts/)` — connect contacts to sales processes and their stakeholder roles (medium)-- [ ] `Opportunity Products (GET /API/v2/opportunityProducts/)` — join products to opportunities for product-level pipeline and revenue analysis (medium)+- [x] `Opportunity Products (GET /API/v2/opportunityProducts/)` — join products to opportunities for product-level pipeline and revenue analysis (medium)- [ ] `Orders and Order Rows (GET /API/v2/orders/; GET /API/v2/orderRows/)` — measure booked sales and item-level amounts against opportunities and products (medium)Note: Fetched the complete official v2 reference and extracted top-level and nested GET operations. Process endpoints embed stages, steps, and custom fields; these definitions differ from the existing generic custom-fields table. Existing activities already cover notes, calls, emails, and tasks, so duplicate activity tables are excluded. Order rows and contact associations need joins; excluded automation configuration and batch transport.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/canonical_descriptions.pyindex 0e2c43520a5f..30ecaf3934bf 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/canonical_descriptions.py@@ -64,6 +64,28 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"LostReasonName": "Name of the recorded lost reason.",},},+ "opportunity_products": {+ "description": "Products added to opportunities (Sales Projects), one row per product line on an opportunity.",+ "docs_url": _DOCS_URL,+ "columns": {+ **_AUDIT_COLUMNS,+ "Id": "Unique identifier (GUID) of the opportunity product line.",+ "OpportunityId": "GUID of the opportunity (Sales Project) this product line belongs to.",+ "ProductId": "GUID of the product. Matches Id on products.",+ "Name": "Product name.",+ "Description": "Product description.",+ "Price": "Product price, in PriceCurrency.",+ "PriceCurrency": "Currency of the price. Matches Id on currencies.",+ "PriceType": "Pricing model: Recurring or Fixed.",+ "PriceInterval": "How often a recurring price repeats (for example 1M or 12M). Only applies when PriceType is Recurring.",+ "Quantity": "Quantity of the product.",+ "TermLength": "Length of the term. Only applies when PriceType is Recurring.",+ "TermStartDate": "Timestamp when the recurrence starts. Only applies when PriceType is Recurring.",+ "Discount": "Discount, as a percentage or an absolute amount in PriceCurrency.",+ "DiscountType": "Discount type: Absolute or Percentage.",+ "Cost": "Cost of the product, in PriceCurrency.",+ },+ },"account_growth_items": {"description": "Account growth projects. An account growth project's human-readable key has the form AGP-{n}.","docs_url": _DOCS_URL,@@ -171,6 +193,35 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"Name": "Activity type name.",},},+ "labels": {+ "description": "Labels used to tag companies and contacts.",+ "docs_url": _DOCS_URL,+ "columns": {+ "Id": "Unique identifier (GUID) of the label.",+ "Name": "Label name.",+ "UseFor": "Entity the label applies to: Company or Contact.",+ },+ },+ "sales_processes": {+ "description": "Active sales process definitions for opportunities (Sales Projects), including their stages, steps and custom fields.",+ "docs_url": _DOCS_URL,+ "columns": {+ "Id": "Unique identifier (GUID) of the sales process.",+ "Name": "Name of the sales process.",+ "Stages": "Stages in the process, as a list of {Id, Name, Steps} objects. Each Steps entry is a list of {Id, Name} objects.",+ "CustomFields": "Custom fields referenced in the process, with the same properties as the custom_fields table.",+ },+ },+ "prospecting_processes": {+ "description": "Active prospecting process definitions for prospects, including their stages, steps and custom fields.",+ "docs_url": _DOCS_URL,+ "columns": {+ "Id": "Unique identifier (GUID) of the prospecting process.",+ "Name": "Name of the prospecting process.",+ "Stages": "Stages in the process, as a list of {Id, Name, Steps} objects. Each Steps entry is a list of {Id, Name} objects.",+ "CustomFields": "Custom fields referenced in the process, with the same properties as the custom_fields table.",+ },+ },"ticket_processes": {"description": "Active ticket process definitions, including their stages, steps and custom fields.","docs_url": _DOCS_URL,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/membrain.py b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/membrain.pyindex 8362aadf18f7..6d997896cd86 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/membrain.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/membrain.py@@ -209,7 +209,7 @@ def membrain_source(],}- # Cross-run offset resume is only safe under the explicit stable sort we request; the four+ # Cross-run offset resume is only safe under the explicit stable sort we request; the# paged endpoints with no documented SortBy have an unestablished default order, where a# `From=N` resume could silently skip or duplicate rows if that order shifts between runs.# Gate both loading and saving on the same predicate and those endpoints restart at offset 0diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/settings.pyindex 743fe35f6e18..cce8573775e9 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/settings.py@@ -62,6 +62,13 @@ MEMBRAIN_ENDPOINTS: dict[str, MembrainEndpointConfig] = {paged=True,partition_key="CreatedDate",),+ # Products attached to an opportunity (Sales Project), one row per line item.+ "opportunity_products": MembrainEndpointConfig(+ name="opportunity_products",+ path="/opportunityProducts/",+ paged=True,+ partition_key="CreatedDate",+ ),"account_growth_items": MembrainEndpointConfig(name="account_growth_items",path="/accountGrowthItems/",@@ -95,8 +102,11 @@ MEMBRAIN_ENDPOINTS: dict[str, MembrainEndpointConfig] = {"products": MembrainEndpointConfig(name="products", path="/products/"),"currencies": MembrainEndpointConfig(name="currencies", path="/currencies/"),"activity_types": MembrainEndpointConfig(name="activity_types", path="/activityTypes/"),+ "labels": MembrainEndpointConfig(name="labels", path="/labels/"),# Process definitions return one row per active process with its Stages (each holding Steps)# and CustomFields as nested arrays.+ "sales_processes": MembrainEndpointConfig(name="sales_processes", path="/salesProcesses/"),+ "prospecting_processes": MembrainEndpointConfig(name="prospecting_processes", path="/prospectingProcesses/"),"ticket_processes": MembrainEndpointConfig(name="ticket_processes", path="/ticketProcesses/"),"account_growth_processes": MembrainEndpointConfig(name="account_growth_processes", path="/accountGrowthProcesses/"@@ -108,9 +118,9 @@ ENDPOINTS = tuple(MEMBRAIN_ENDPOINTS.keys())# All endpoints are full refresh only for now. The main list endpoints do document server-side# ChangedFromDate/ChangedToDate and CreatedFromDate/CreatedToDate range filters (both ends of a# range must be provided together), but the reference is the only evidence available: the filters-# could not be verified against a live instance here (no credentials), and four of the paged-# endpoints (prospects, opportunities, accountGrowthItems, tickets) document no SortBy, so row-# order inside a window is unknown. Enabling incremental is a matter of populating this dict,-# mapping the user's cursor into the range params in `membrain.py`, and passing an explicit upper-# bound, once the filters are verified live.+# could not be verified against a live instance here (no credentials), and five of the paged+# endpoints (prospects, opportunities, opportunityProducts, accountGrowthItems, tickets) document+# no SortBy, so row order inside a window is unknown. Enabling incremental is a matter of+# populating this dict, mapping the user's cursor into the range params in `membrain.py`, and+# passing an explicit upper bound, once the filters are verified live.INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/tests/test_membrain.py b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/tests/test_membrain.pyindex 965d64cd97de..4de594f1d97e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/tests/test_membrain.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/tests/test_membrain.py@@ -128,6 +128,7 @@ class TestPagination:# These endpoints document neither SortBy nor Include* params, so none are sent.("prospects", {}),("opportunities", {}),+ ("opportunity_products", {}),("account_growth_items", {}),("tickets", {}),]@@ -138,7 +139,9 @@ class TestPagination:assert result.raised is Noneassert {key: values[0] for key, values in result.queries[0].items()} == expected- @parameterized.expand([("prospects",), ("opportunities",), ("account_growth_items",), ("tickets",)])+ @parameterized.expand(+ [("prospects",), ("opportunities",), ("opportunity_products",), ("account_growth_items",), ("tickets",)]+ )def test_unsorted_endpoints_never_save_or_load_resume_state(self, endpoint: str) -> None:# No documented SortBy means no established stable order, so an offset resume across runs# could silently skip or duplicate rows if that order shifts between them.@@ -164,6 +167,9 @@ class TestBareArrayEndpoints:("products", "/products/"),("currencies", "/currencies/"),("activity_types", "/activityTypes/"),+ ("labels", "/labels/"),+ ("sales_processes", "/salesProcesses/"),+ ("prospecting_processes", "/prospectingProcesses/"),("ticket_processes", "/ticketProcesses/"),("account_growth_processes", "/accountGrowthProcesses/"),]