:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add pipeliner line item, sales unit and currency tables (#126266)
Tom Owerse9519bb69146
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex aae1b8fa97ab..3daae95fa533 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -9469,14 +9469,14 @@ Note: Diffed against both specs - v2 (https://developers.pipedrive.com/docs/api/## Pipeliner — **thin**-Today (11): `accounts`, `appointments`, `clients`, `contacts`, `leads`, `notes`, `opportunities`, `pipelines`, `products`, `steps`, `tasks`+Today (17): `accounts`, `appointments`, `clients`, `contacts`, `currencies`, `currency_exchange_rates`, `currency_exchange_rates_lists`, `leads`, `notes`, `opportunities`, `oppty_product_relations`, `pipelines`, `products`, `sales_unit_client_relations`, `sales_units`, `steps`, `tasks`Diffed against: <https://pipelinercrm.eu.apidog.com/>-- [ ] `OpptyProductRelations` — product line items linking opportunities to products with quantity and price - synced opportunities and products are otherwise unjoinable (high)-- [ ] `ProcessActivityLogs and ProcessActivityLogLines` — step/stage transition history for opportunities, the basis of sales-velocity and stage-conversion analysis (high)-- [ ] `SalesUnits (and SalesUnitClientRelations)` — lookup resolving the sales unit IDs stamped on every account, contact, lead and opportunity we sync (high)-- [ ] `Currencies and CurrencyExchangeRates` — lookup for the currency IDs on opportunity and product values; without it multi-currency pipelines can't be normalized (high)+- [x] `OpptyProductRelations` — product line items linking opportunities to products with quantity and price - synced opportunities and products are otherwise unjoinable (high)+- [ ] `ProcessActivityLogs and ProcessActivityLogLines` — step/stage transition history for opportunities, the basis of sales-velocity and stage-conversion analysis (high) — skipped: these are Automatizer workflow run logs, not opportunity stage history+- [x] `SalesUnits (and SalesUnitClientRelations)` — lookup resolving the sales unit IDs stamped on every account, contact, lead and opportunity we sync (high)+- [x] `Currencies and CurrencyExchangeRates` — lookup for the currency IDs on opportunity and product values; without it multi-currency pipelines can't be normalized (high)- [ ] `Processes (and ProcessTemplates)` — the sales process each Step belongs to - the parent lookup for the synced steps table (high)- [ ] `AccountTypes, ContactTypes, LeadTypes, OpportunityTypes, TaskTypes, AppointmentTypes` — type lookups resolving the type IDs on the core records already synced (high)- [ ] `Quotes (and QuoteProcesses, QuoteTypes)` — quoting is a first-class Pipeliner object tied to opportunities and completely absent today (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/canonical_descriptions.pyindex 3138b8806e4f..43da8ae6b4d5 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/canonical_descriptions.py@@ -96,6 +96,34 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"is_archived": "True when the contact has been archived.",},},+ "currencies": {+ "description": "Currencies enabled in the team space, including the single base currency all amounts convert from.",+ "docs_url": _DOCS_URL,+ "columns": {+ **_COMMON_COLUMNS,+ "code": "Currency code and its default text representation (e.g. USD, EUR).",+ "is_base": "True for the space's base currency. Each space has exactly one base currency.",+ "symbol": "Display symbol of the currency (e.g. $, €).",+ },+ },+ "currency_exchange_rates": {+ "description": "Exchange rates from the base currency into each other currency, grouped into dated rate lists.",+ "docs_url": _DOCS_URL,+ "columns": {+ **_COMMON_COLUMNS,+ "currency_exchange_rate_list_id": "Id of the exchange rate list (effective period) this rate belongs to.",+ "currency_id": "Id of the currency the exchange rate applies to. Never the base currency.",+ "exchange_rate": "Multiplier that converts an amount in the base currency into this currency.",+ },+ },+ "currency_exchange_rates_lists": {+ "description": "Dated sets of exchange rates. Opportunities and line items reference the list used to convert their values.",+ "docs_url": _DOCS_URL,+ "columns": {+ **_COMMON_COLUMNS,+ "valid_from": "Date from which the exchange rates in this list apply.",+ },+ },"leads": {"description": "Potential deals before qualification, with source, ranking, process step, and close details.","docs_url": _DOCS_URL,@@ -152,6 +180,25 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"unit_id": "Id of the sales unit this opportunity is assigned to.",},},+ "oppty_product_relations": {+ "description": "Product line items on opportunities and quotes, with quantity, price, and discounts.",+ "docs_url": _DOCS_URL,+ "columns": {+ **_COMMON_COLUMNS,+ "oppty_id": "Id of the opportunity this line item belongs to. Mutually exclusive with quote_id.",+ "quote_id": "Id of the quote this line item belongs to. Mutually exclusive with oppty_id.",+ "product_id": "Id of the product on this line item.",+ "quantity": "Number of product units on this line item.",+ "price": "Per-unit price of the line item.",+ "amount": "Total amount of the line item (price times quantity, with discounts applied).",+ "discount_value": "Absolute discount applied to this line item versus the price-list price.",+ "discount_percentage": "Percentage discount applied to this line item versus the price-list price.",+ "currency_exchange_rates_list_id": "Id of the exchange rate list used to convert this line item.",+ "oppty_product_relation_type_id": "Id of the line item type.",+ "comment": "Free-form note describing this line item.",+ "sort_order": "Display order of the line item within its section.",+ },+ },"pipelines": {"description": "Sales pipelines that opportunities move through, each composed of ordered steps.","docs_url": _DOCS_URL,@@ -173,6 +220,25 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"unit_symbol": "Unit of measure shown next to product quantities (e.g. pcs, kg, hours).",},},+ "sales_unit_client_relations": {+ "description": "Membership of users in sales units, with each user's role in the unit.",+ "docs_url": _DOCS_URL,+ "columns": {+ **_COMMON_COLUMNS,+ "client_id": "Id of the user.",+ "unit_id": "Id of the sales unit.",+ "unit_role": "Role of the user in the sales unit: Member, or Manager (which also covers all subunits).",+ },+ },+ "sales_units": {+ "description": "Sales units (teams or regions) that accounts, contacts, leads, and opportunities are assigned to.",+ "docs_url": _DOCS_URL,+ "columns": {+ **_COMMON_COLUMNS,+ "name": "Name of the sales unit and its default text representation.",+ "parent_id": "Id of the parent unit, or empty for the root unit.",+ },+ },"steps": {"description": "Individual stages within a sales pipeline or lead/quote process, with win probability and expected timeframe.","docs_url": _DOCS_URL,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/settings.pyindex 45909bc52d39..9a3cc4cf3fdc 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/settings.py@@ -38,11 +38,21 @@ PIPELINER_ENDPOINTS: dict[str, PipelinerEndpointConfig] = {"appointments": PipelinerEndpointConfig(name="appointments", entity="Appointments"),"clients": PipelinerEndpointConfig(name="clients", entity="Clients"),"contacts": PipelinerEndpointConfig(name="contacts", entity="Contacts"),+ "currencies": PipelinerEndpointConfig(name="currencies", entity="Currencies"),+ "currency_exchange_rates": PipelinerEndpointConfig(name="currency_exchange_rates", entity="CurrencyExchangeRates"),+ "currency_exchange_rates_lists": PipelinerEndpointConfig(+ name="currency_exchange_rates_lists", entity="CurrencyExchangeRatesLists"+ ),"leads": PipelinerEndpointConfig(name="leads", entity="Leads"),"notes": PipelinerEndpointConfig(name="notes", entity="Notes"),"opportunities": PipelinerEndpointConfig(name="opportunities", entity="Opportunities"),+ "oppty_product_relations": PipelinerEndpointConfig(name="oppty_product_relations", entity="OpptyProductRelations"),"pipelines": PipelinerEndpointConfig(name="pipelines", entity="Pipelines"),"products": PipelinerEndpointConfig(name="products", entity="Products"),+ "sales_unit_client_relations": PipelinerEndpointConfig(+ name="sales_unit_client_relations", entity="SalesUnitClientRelations"+ ),+ "sales_units": PipelinerEndpointConfig(name="sales_units", entity="SalesUnits"),"steps": PipelinerEndpointConfig(name="steps", entity="Steps"),"tasks": PipelinerEndpointConfig(name="tasks", entity="Tasks"),}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/tests/test_pipeliner.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/tests/test_pipeliner.pyindex 6377d9581074..91aa7bffc82d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/tests/test_pipeliner.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/tests/test_pipeliner.py@@ -62,9 +62,18 @@ class TestFormatIncrementalValue:class TestPipelinerSourceTransport:- def test_paginates_with_after_cursor_until_last_page(self) -> None:+ @pytest.mark.parametrize(+ "endpoint, entity",+ [+ ("accounts", "Accounts"),+ ("oppty_product_relations", "OpptyProductRelations"),+ ("sales_unit_client_relations", "SalesUnitClientRelations"),+ ("currency_exchange_rates_lists", "CurrencyExchangeRatesLists"),+ ],+ )+ def test_paginates_with_after_cursor_until_last_page(self, endpoint: str, entity: str) -> None:result = _driver().run(- "accounts",+ endpoint,[_response([{"id": "1"}, {"id": "2"}], end_cursor="cur1", has_next_page=True),_response([{"id": "3"}], end_cursor="cur2", has_next_page=False),@@ -75,6 +84,7 @@ class TestPipelinerSourceTransport:assert [row["id"] for row in result.rows] == ["1", "2", "3"]assert result.params("after") == [None, "cur1"]assert result.params("first") == [str(PAGE_SIZE), str(PAGE_SIZE)]+ assert result.paths == [f"/api/v100/rest/spaces/space-1/entities/{entity}"] * 2# The last page still has an end cursor. The has_next_page flag ends the walk.assert len(result.requests) == 2