Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add pipeliner line item, sales unit and currency tables (#126266)

Tom Owerse9519bb69146
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index aae1b8fa97ab..3daae95fa533 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -9469,14 +9469,14 @@ Note: Diffed against both specs - v2 (https://developers.pipedrive.com/docs/api/
## Pipeliner — **thin**
-Today (11): `accounts`, `appointments`, `clients`, `contacts`, `leads`, `notes`, `opportunities`, `pipelines`, `products`, `steps`, `tasks`
+Today (17): `accounts`, `appointments`, `clients`, `contacts`, `currencies`, `currency_exchange_rates`, `currency_exchange_rates_lists`, `leads`, `notes`, `opportunities`, `oppty_product_relations`, `pipelines`, `products`, `sales_unit_client_relations`, `sales_units`, `steps`, `tasks`
Diffed against: <https://pipelinercrm.eu.apidog.com/>
-- [ ] `OpptyProductRelations` — product line items linking opportunities to products with quantity and price - synced opportunities and products are otherwise unjoinable (high)
-- [ ] `ProcessActivityLogs and ProcessActivityLogLines` — step/stage transition history for opportunities, the basis of sales-velocity and stage-conversion analysis (high)
-- [ ] `SalesUnits (and SalesUnitClientRelations)` — lookup resolving the sales unit IDs stamped on every account, contact, lead and opportunity we sync (high)
-- [ ] `Currencies and CurrencyExchangeRates` — lookup for the currency IDs on opportunity and product values; without it multi-currency pipelines can't be normalized (high)
+- [x] `OpptyProductRelations` — product line items linking opportunities to products with quantity and price - synced opportunities and products are otherwise unjoinable (high)
+- [ ] `ProcessActivityLogs and ProcessActivityLogLines` — step/stage transition history for opportunities, the basis of sales-velocity and stage-conversion analysis (high) — skipped: these are Automatizer workflow run logs, not opportunity stage history
+- [x] `SalesUnits (and SalesUnitClientRelations)` — lookup resolving the sales unit IDs stamped on every account, contact, lead and opportunity we sync (high)
+- [x] `Currencies and CurrencyExchangeRates` — lookup for the currency IDs on opportunity and product values; without it multi-currency pipelines can't be normalized (high)
- [ ] `Processes (and ProcessTemplates)` — the sales process each Step belongs to - the parent lookup for the synced steps table (high)
- [ ] `AccountTypes, ContactTypes, LeadTypes, OpportunityTypes, TaskTypes, AppointmentTypes` — type lookups resolving the type IDs on the core records already synced (high)
- [ ] `Quotes (and QuoteProcesses, QuoteTypes)` — quoting is a first-class Pipeliner object tied to opportunities and completely absent today (medium)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/canonical_descriptions.py
index 3138b8806e4f..43da8ae6b4d5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/canonical_descriptions.py
@@ -96,6 +96,34 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"is_archived": "True when the contact has been archived.",
},
},
+ "currencies": {
+ "description": "Currencies enabled in the team space, including the single base currency all amounts convert from.",
+ "docs_url": _DOCS_URL,
+ "columns": {
+ **_COMMON_COLUMNS,
+ "code": "Currency code and its default text representation (e.g. USD, EUR).",
+ "is_base": "True for the space's base currency. Each space has exactly one base currency.",
+ "symbol": "Display symbol of the currency (e.g. $, €).",
+ },
+ },
+ "currency_exchange_rates": {
+ "description": "Exchange rates from the base currency into each other currency, grouped into dated rate lists.",
+ "docs_url": _DOCS_URL,
+ "columns": {
+ **_COMMON_COLUMNS,
+ "currency_exchange_rate_list_id": "Id of the exchange rate list (effective period) this rate belongs to.",
+ "currency_id": "Id of the currency the exchange rate applies to. Never the base currency.",
+ "exchange_rate": "Multiplier that converts an amount in the base currency into this currency.",
+ },
+ },
+ "currency_exchange_rates_lists": {
+ "description": "Dated sets of exchange rates. Opportunities and line items reference the list used to convert their values.",
+ "docs_url": _DOCS_URL,
+ "columns": {
+ **_COMMON_COLUMNS,
+ "valid_from": "Date from which the exchange rates in this list apply.",
+ },
+ },
"leads": {
"description": "Potential deals before qualification, with source, ranking, process step, and close details.",
"docs_url": _DOCS_URL,
@@ -152,6 +180,25 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"unit_id": "Id of the sales unit this opportunity is assigned to.",
},
},
+ "oppty_product_relations": {
+ "description": "Product line items on opportunities and quotes, with quantity, price, and discounts.",
+ "docs_url": _DOCS_URL,
+ "columns": {
+ **_COMMON_COLUMNS,
+ "oppty_id": "Id of the opportunity this line item belongs to. Mutually exclusive with quote_id.",
+ "quote_id": "Id of the quote this line item belongs to. Mutually exclusive with oppty_id.",
+ "product_id": "Id of the product on this line item.",
+ "quantity": "Number of product units on this line item.",
+ "price": "Per-unit price of the line item.",
+ "amount": "Total amount of the line item (price times quantity, with discounts applied).",
+ "discount_value": "Absolute discount applied to this line item versus the price-list price.",
+ "discount_percentage": "Percentage discount applied to this line item versus the price-list price.",
+ "currency_exchange_rates_list_id": "Id of the exchange rate list used to convert this line item.",
+ "oppty_product_relation_type_id": "Id of the line item type.",
+ "comment": "Free-form note describing this line item.",
+ "sort_order": "Display order of the line item within its section.",
+ },
+ },
"pipelines": {
"description": "Sales pipelines that opportunities move through, each composed of ordered steps.",
"docs_url": _DOCS_URL,
@@ -173,6 +220,25 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"unit_symbol": "Unit of measure shown next to product quantities (e.g. pcs, kg, hours).",
},
},
+ "sales_unit_client_relations": {
+ "description": "Membership of users in sales units, with each user's role in the unit.",
+ "docs_url": _DOCS_URL,
+ "columns": {
+ **_COMMON_COLUMNS,
+ "client_id": "Id of the user.",
+ "unit_id": "Id of the sales unit.",
+ "unit_role": "Role of the user in the sales unit: Member, or Manager (which also covers all subunits).",
+ },
+ },
+ "sales_units": {
+ "description": "Sales units (teams or regions) that accounts, contacts, leads, and opportunities are assigned to.",
+ "docs_url": _DOCS_URL,
+ "columns": {
+ **_COMMON_COLUMNS,
+ "name": "Name of the sales unit and its default text representation.",
+ "parent_id": "Id of the parent unit, or empty for the root unit.",
+ },
+ },
"steps": {
"description": "Individual stages within a sales pipeline or lead/quote process, with win probability and expected timeframe.",
"docs_url": _DOCS_URL,
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/settings.py
index 45909bc52d39..9a3cc4cf3fdc 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/settings.py
@@ -38,11 +38,21 @@ PIPELINER_ENDPOINTS: dict[str, PipelinerEndpointConfig] = {
"appointments": PipelinerEndpointConfig(name="appointments", entity="Appointments"),
"clients": PipelinerEndpointConfig(name="clients", entity="Clients"),
"contacts": PipelinerEndpointConfig(name="contacts", entity="Contacts"),
+ "currencies": PipelinerEndpointConfig(name="currencies", entity="Currencies"),
+ "currency_exchange_rates": PipelinerEndpointConfig(name="currency_exchange_rates", entity="CurrencyExchangeRates"),
+ "currency_exchange_rates_lists": PipelinerEndpointConfig(
+ name="currency_exchange_rates_lists", entity="CurrencyExchangeRatesLists"
+ ),
"leads": PipelinerEndpointConfig(name="leads", entity="Leads"),
"notes": PipelinerEndpointConfig(name="notes", entity="Notes"),
"opportunities": PipelinerEndpointConfig(name="opportunities", entity="Opportunities"),
+ "oppty_product_relations": PipelinerEndpointConfig(name="oppty_product_relations", entity="OpptyProductRelations"),
"pipelines": PipelinerEndpointConfig(name="pipelines", entity="Pipelines"),
"products": PipelinerEndpointConfig(name="products", entity="Products"),
+ "sales_unit_client_relations": PipelinerEndpointConfig(
+ name="sales_unit_client_relations", entity="SalesUnitClientRelations"
+ ),
+ "sales_units": PipelinerEndpointConfig(name="sales_units", entity="SalesUnits"),
"steps": PipelinerEndpointConfig(name="steps", entity="Steps"),
"tasks": PipelinerEndpointConfig(name="tasks", entity="Tasks"),
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/tests/test_pipeliner.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/tests/test_pipeliner.py
index 6377d9581074..91aa7bffc82d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/tests/test_pipeliner.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pipeliner/tests/test_pipeliner.py
@@ -62,9 +62,18 @@ class TestFormatIncrementalValue:
class TestPipelinerSourceTransport:
- def test_paginates_with_after_cursor_until_last_page(self) -> None:
+ @pytest.mark.parametrize(
+ "endpoint, entity",
+ [
+ ("accounts", "Accounts"),
+ ("oppty_product_relations", "OpptyProductRelations"),
+ ("sales_unit_client_relations", "SalesUnitClientRelations"),
+ ("currency_exchange_rates_lists", "CurrencyExchangeRatesLists"),
+ ],
+ )
+ def test_paginates_with_after_cursor_until_last_page(self, endpoint: str, entity: str) -> None:
result = _driver().run(
- "accounts",
+ endpoint,
[
_response([{"id": "1"}, {"id": "2"}], end_cursor="cur1", has_next_page=True),
_response([{"id": "3"}], end_cursor="cur2", has_next_page=False),
@@ -75,6 +84,7 @@ class TestPipelinerSourceTransport:
assert [row["id"] for row in result.rows] == ["1", "2", "3"]
assert result.params("after") == [None, "cur1"]
assert result.params("first") == [str(PAGE_SIZE), str(PAGE_SIZE)]
+ assert result.paths == [f"/api/v100/rest/spaces/space-1/entities/{entity}"] * 2
# The last page still has an end cursor. The has_next_page flag ends the walk.
assert len(result.requests) == 2