Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add membrain process, label and opportunity product tables (#123044)

Tom Owerse3e7dbb24295
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index fee96f25e757..e29f47de6b80 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -7770,15 +7770,15 @@ Today (11): `account_growth_items`, `activities`, `companies`, `contacts`, `cust
Diffed against: <https://www.membrain.com/developers/api-documentation>
-- [ ] `Sales processes (GET /API/v2/salesProcesses/)` — resolve opportunity process and stage IDs, including stage order and process-specific fields (high)
-- [ ] `Prospecting processes (GET /API/v2/prospectingProcesses/)` — resolve prospect process and stage IDs for conversion analysis (high)
+- [x] `Sales processes (GET /API/v2/salesProcesses/)` — resolve opportunity process and stage IDs, including stage order and process-specific fields (high)
+- [x] `Prospecting processes (GET /API/v2/prospectingProcesses/)` — resolve prospect process and stage IDs for conversion analysis (high)
- [x] `Account Growth processes (GET /API/v2/accountGrowthProcesses/)` — resolve account-growth process and stage IDs for expansion analysis (high)
- [x] `Ticket processes (GET /API/v2/ticketProcesses/)` — resolve ticket process and stage IDs for support workflow analysis (high)
- [x] `Currencies (GET /API/v2/currencies/)` — resolve ValueCurrencyId and supply exchange rates for comparable pipeline values (high)
-- [ ] `Labels (GET /API/v2/labels/)` — resolve LabelIds on companies and contacts for consistent segmentation (high)
+- [x] `Labels (GET /API/v2/labels/)` — resolve LabelIds on companies and contacts for consistent segmentation (high)
- [x] `Activity types (GET /API/v2/activityTypes/)` — resolve ActivityTypeId and provide the complete activity taxonomy (high)
- [ ] `Process Container Contacts (GET /API/v2/processContainerContacts/)` — connect contacts to sales processes and their stakeholder roles (medium)
-- [ ] `Opportunity Products (GET /API/v2/opportunityProducts/)` — join products to opportunities for product-level pipeline and revenue analysis (medium)
+- [x] `Opportunity Products (GET /API/v2/opportunityProducts/)` — join products to opportunities for product-level pipeline and revenue analysis (medium)
- [ ] `Orders and Order Rows (GET /API/v2/orders/; GET /API/v2/orderRows/)` — measure booked sales and item-level amounts against opportunities and products (medium)
Note: Fetched the complete official v2 reference and extracted top-level and nested GET operations. Process endpoints embed stages, steps, and custom fields; these definitions differ from the existing generic custom-fields table. Existing activities already cover notes, calls, emails, and tasks, so duplicate activity tables are excluded. Order rows and contact associations need joins; excluded automation configuration and batch transport.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/canonical_descriptions.py
index 0e2c43520a5f..30ecaf3934bf 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/canonical_descriptions.py
@@ -64,6 +64,28 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"LostReasonName": "Name of the recorded lost reason.",
},
},
+ "opportunity_products": {
+ "description": "Products added to opportunities (Sales Projects), one row per product line on an opportunity.",
+ "docs_url": _DOCS_URL,
+ "columns": {
+ **_AUDIT_COLUMNS,
+ "Id": "Unique identifier (GUID) of the opportunity product line.",
+ "OpportunityId": "GUID of the opportunity (Sales Project) this product line belongs to.",
+ "ProductId": "GUID of the product. Matches Id on products.",
+ "Name": "Product name.",
+ "Description": "Product description.",
+ "Price": "Product price, in PriceCurrency.",
+ "PriceCurrency": "Currency of the price. Matches Id on currencies.",
+ "PriceType": "Pricing model: Recurring or Fixed.",
+ "PriceInterval": "How often a recurring price repeats (for example 1M or 12M). Only applies when PriceType is Recurring.",
+ "Quantity": "Quantity of the product.",
+ "TermLength": "Length of the term. Only applies when PriceType is Recurring.",
+ "TermStartDate": "Timestamp when the recurrence starts. Only applies when PriceType is Recurring.",
+ "Discount": "Discount, as a percentage or an absolute amount in PriceCurrency.",
+ "DiscountType": "Discount type: Absolute or Percentage.",
+ "Cost": "Cost of the product, in PriceCurrency.",
+ },
+ },
"account_growth_items": {
"description": "Account growth projects. An account growth project's human-readable key has the form AGP-{n}.",
"docs_url": _DOCS_URL,
@@ -171,6 +193,35 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"Name": "Activity type name.",
},
},
+ "labels": {
+ "description": "Labels used to tag companies and contacts.",
+ "docs_url": _DOCS_URL,
+ "columns": {
+ "Id": "Unique identifier (GUID) of the label.",
+ "Name": "Label name.",
+ "UseFor": "Entity the label applies to: Company or Contact.",
+ },
+ },
+ "sales_processes": {
+ "description": "Active sales process definitions for opportunities (Sales Projects), including their stages, steps and custom fields.",
+ "docs_url": _DOCS_URL,
+ "columns": {
+ "Id": "Unique identifier (GUID) of the sales process.",
+ "Name": "Name of the sales process.",
+ "Stages": "Stages in the process, as a list of {Id, Name, Steps} objects. Each Steps entry is a list of {Id, Name} objects.",
+ "CustomFields": "Custom fields referenced in the process, with the same properties as the custom_fields table.",
+ },
+ },
+ "prospecting_processes": {
+ "description": "Active prospecting process definitions for prospects, including their stages, steps and custom fields.",
+ "docs_url": _DOCS_URL,
+ "columns": {
+ "Id": "Unique identifier (GUID) of the prospecting process.",
+ "Name": "Name of the prospecting process.",
+ "Stages": "Stages in the process, as a list of {Id, Name, Steps} objects. Each Steps entry is a list of {Id, Name} objects.",
+ "CustomFields": "Custom fields referenced in the process, with the same properties as the custom_fields table.",
+ },
+ },
"ticket_processes": {
"description": "Active ticket process definitions, including their stages, steps and custom fields.",
"docs_url": _DOCS_URL,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/membrain.py b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/membrain.py
index 8362aadf18f7..6d997896cd86 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/membrain.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/membrain.py
@@ -209,7 +209,7 @@ def membrain_source(
],
}
- # Cross-run offset resume is only safe under the explicit stable sort we request; the four
+ # Cross-run offset resume is only safe under the explicit stable sort we request; the
# paged endpoints with no documented SortBy have an unestablished default order, where a
# `From=N` resume could silently skip or duplicate rows if that order shifts between runs.
# Gate both loading and saving on the same predicate and those endpoints restart at offset 0
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/settings.py
index 743fe35f6e18..cce8573775e9 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/settings.py
@@ -62,6 +62,13 @@ MEMBRAIN_ENDPOINTS: dict[str, MembrainEndpointConfig] = {
paged=True,
partition_key="CreatedDate",
),
+ # Products attached to an opportunity (Sales Project), one row per line item.
+ "opportunity_products": MembrainEndpointConfig(
+ name="opportunity_products",
+ path="/opportunityProducts/",
+ paged=True,
+ partition_key="CreatedDate",
+ ),
"account_growth_items": MembrainEndpointConfig(
name="account_growth_items",
path="/accountGrowthItems/",
@@ -95,8 +102,11 @@ MEMBRAIN_ENDPOINTS: dict[str, MembrainEndpointConfig] = {
"products": MembrainEndpointConfig(name="products", path="/products/"),
"currencies": MembrainEndpointConfig(name="currencies", path="/currencies/"),
"activity_types": MembrainEndpointConfig(name="activity_types", path="/activityTypes/"),
+ "labels": MembrainEndpointConfig(name="labels", path="/labels/"),
# Process definitions return one row per active process with its Stages (each holding Steps)
# and CustomFields as nested arrays.
+ "sales_processes": MembrainEndpointConfig(name="sales_processes", path="/salesProcesses/"),
+ "prospecting_processes": MembrainEndpointConfig(name="prospecting_processes", path="/prospectingProcesses/"),
"ticket_processes": MembrainEndpointConfig(name="ticket_processes", path="/ticketProcesses/"),
"account_growth_processes": MembrainEndpointConfig(
name="account_growth_processes", path="/accountGrowthProcesses/"
@@ -108,9 +118,9 @@ ENDPOINTS = tuple(MEMBRAIN_ENDPOINTS.keys())
# All endpoints are full refresh only for now. The main list endpoints do document server-side
# ChangedFromDate/ChangedToDate and CreatedFromDate/CreatedToDate range filters (both ends of a
# range must be provided together), but the reference is the only evidence available: the filters
-# could not be verified against a live instance here (no credentials), and four of the paged
-# endpoints (prospects, opportunities, accountGrowthItems, tickets) document no SortBy, so row
-# order inside a window is unknown. Enabling incremental is a matter of populating this dict,
-# mapping the user's cursor into the range params in `membrain.py`, and passing an explicit upper
-# bound, once the filters are verified live.
+# could not be verified against a live instance here (no credentials), and five of the paged
+# endpoints (prospects, opportunities, opportunityProducts, accountGrowthItems, tickets) document
+# no SortBy, so row order inside a window is unknown. Enabling incremental is a matter of
+# populating this dict, mapping the user's cursor into the range params in `membrain.py`, and
+# passing an explicit upper bound, once the filters are verified live.
INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/tests/test_membrain.py b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/tests/test_membrain.py
index 965d64cd97de..4de594f1d97e 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/tests/test_membrain.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/membrain/tests/test_membrain.py
@@ -128,6 +128,7 @@ class TestPagination:
# These endpoints document neither SortBy nor Include* params, so none are sent.
("prospects", {}),
("opportunities", {}),
+ ("opportunity_products", {}),
("account_growth_items", {}),
("tickets", {}),
]
@@ -138,7 +139,9 @@ class TestPagination:
assert result.raised is None
assert {key: values[0] for key, values in result.queries[0].items()} == expected
- @parameterized.expand([("prospects",), ("opportunities",), ("account_growth_items",), ("tickets",)])
+ @parameterized.expand(
+ [("prospects",), ("opportunities",), ("opportunity_products",), ("account_growth_items",), ("tickets",)]
+ )
def test_unsorted_endpoints_never_save_or_load_resume_state(self, endpoint: str) -> None:
# No documented SortBy means no established stable order, so an offset resume across runs
# could silently skip or duplicate rows if that order shifts between them.
@@ -164,6 +167,9 @@ class TestBareArrayEndpoints:
("products", "/products/"),
("currencies", "/currencies/"),
("activity_types", "/activityTypes/"),
+ ("labels", "/labels/"),
+ ("sales_processes", "/salesProcesses/"),
+ ("prospecting_processes", "/prospectingProcesses/"),
("ticket_processes", "/ticketProcesses/"),
("account_growth_processes", "/accountGrowthProcesses/"),
]