:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse-sources): add emails, lifecycle and page view tables to pardot (#124776)
Tom Owers320fac83572b
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 9d3612fdd590..ffe145a6c7a0 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -9144,14 +9144,14 @@ Today (13): `campaigns`, `custom_fields`, `forms`, `list_emails`, `list_membershDiffed against: <https://developer.salesforce.com/docs/marketing/pardot/guide/version5overview.html>-- [ ] `Lifecycle Stages (GET /api/v5/objects/lifecycle-stages)` — resolve lifecycleStageId values on prospects into sales journey stages (high)-- [ ] `Lifecycle Histories (GET /api/v5/objects/lifecycle-histories)` — measure time between funnel stages and progression from prospect to won opportunity (high)-- [ ] `Emails (GET /api/v5/objects/emails)` — resolve emailId values on visitor activities and analyze individual email performance (high)+- [x] `Lifecycle Stages (GET /api/v5/objects/lifecycle-stages)` — resolve lifecycleStageId values on prospects into sales journey stages (high)+- [x] `Lifecycle Histories (GET /api/v5/objects/lifecycle-histories)` — measure time between funnel stages and progression from prospect to won opportunity (high)+- [x] `Emails (GET /api/v5/objects/emails)` — resolve emailId values on visitor activities and analyze individual email performance (high)- [ ] `Email Templates (GET /api/v5/objects/email-templates)` — resolve emailTemplateId values on visitor activities and list emails for performance comparisons (high)- [ ] `Custom Redirects (GET /api/v5/objects/custom-redirects)` — resolve customRedirectId values on visitor activities into tracked links and campaigns (high)- [ ] `Form Handlers (GET /api/v5/objects/form-handlers)` — resolve formHandlerId values on visitor activities for external form conversion analysis (high)- [ ] `Landing Pages (GET /api/v5/objects/landing-pages)` — resolve landingPageId values on visitor activities into conversion pages (high)-- [ ] `Visitor Page Views (GET /api/v5/objects/visitor-page-views)` — resolve visitorPageViewId values on visitor activities into individual page visits (high)+- [x] `Visitor Page Views (GET /api/v5/objects/visitor-page-views)` — resolve visitorPageViewId values on visitor activities into individual page visits (high)Note: Read the official v5 overview and each listed object reference, then checked their Query GET operations. Existing visitor activities carry IDs for several missing marketing assets. Lifecycle history adds stage transitions beyond the current prospect stage. Excluded exports, authentication, write operations, and layout configuration.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/canonical_descriptions.pyindex 5ff8b96c6a53..bfcdc2145b74 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/canonical_descriptions.py@@ -242,4 +242,54 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"sentAt": "Timestamp the email was sent at.",},},+ "emails": {+ "description": "An individual email sent to one prospect, either one-to-one or as part of a list email send.",+ "docs_url": f"{_DOCS_BASE}/email-v5.html",+ "columns": {+ **_shared("emails"),+ "subject": "Subject line of the email.",+ "prospectId": "ID of the prospect the email was sent to.",+ "listEmailId": "ID of the list email this email was sent as part of, if any.",+ "emailTemplateId": "ID of the template the email was built from.",+ "trackerDomainId": "ID of the tracker domain used for links in the email.",+ "clientType": "Client the email was sent from: Builder for Lightning Email Builder, Web for the legacy builder.",+ "type": "Content type of the email (html, text or htmlAndText).",+ "isOperational": "Whether the email is operational and ignores opt-outs.",+ "salesforceCmsId": "ID of the copy of the email in Salesforce CMS, if it was copied.",+ "sentAt": "Timestamp the email was sent at.",+ },+ },+ "lifecycle_stages": {+ "description": "A step in the prospect lifecycle, from visitor through to won opportunity.",+ "docs_url": f"{_DOCS_BASE}/lifecycle-stage-v5.html",+ "columns": {+ **_shared("lifecycle_stages"),+ "position": "Position of the stage in the lifecycle.",+ "matchType": "Whether a prospect must match all of the stage's rules or any of them.",+ "isLocked": "Whether the stage is locked.",+ },+ },+ "lifecycle_histories": {+ "description": "A move of one prospect from one lifecycle stage to the next.",+ "docs_url": f"{_DOCS_BASE}/lifecycle-history-v5.html",+ "columns": {+ **_shared("lifecycle_histories"),+ "prospectId": "ID of the prospect that changed stage.",+ "previousStageId": "ID of the lifecycle stage the prospect left.",+ "nextStageId": "ID of the lifecycle stage the prospect entered.",+ "secondsElapsed": "Number of seconds the prospect took to reach the new stage.",+ },+ },+ "visitor_page_views": {+ "description": "A single page view on your tracked site.",+ "docs_url": f"{_DOCS_BASE}/visitor-page-view-v5.html",+ "columns": {+ **_shared("visitor_page_views"),+ "url": "URL of the page that was viewed.",+ "title": "Title of the page that was viewed.",+ "visitorId": "ID of the visitor who viewed the page.",+ "visitId": "ID of the visit the page view belongs to.",+ "durationInSeconds": "Length of the page view in seconds.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/settings.pyindex 6ee8aa988b85..30f9aae29d25 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/settings.py@@ -391,6 +391,83 @@ PARDOT_ENDPOINTS: dict[str, PardotEndpointConfig] = {*_TIMESTAMPS,),),+ "emails": PardotEndpointConfig(+ name="emails",+ path="emails",+ # Emails have no createdAt/updatedAt; sentAt is the only datetime they filter and+ # sort on. Unsent scheduled emails carry a null sentAt, so it can't partition.+ order_by="id",+ incremental_fields=[incremental_field("sentAt")],+ # textMessage/htmlMessage aren't queryable and scheduledTime is write-only on+ # create — v5 400s if any of them is in a query's `fields` list.+ fields=(+ "id",+ "name",+ "subject",+ "campaignId",+ "prospectId",+ "listEmailId",+ "emailTemplateId",+ "trackerDomainId",+ "folderId",+ "clientType",+ "type",+ "isOperational",+ "salesforceCmsId",+ "sentAt",+ "createdById",+ ),+ ),+ "lifecycle_stages": PardotEndpointConfig(+ name="lifecycle_stages",+ path="lifecycle-stages",+ order_by="id",+ incremental_fields=_CREATED_AND_UPDATED,+ fields=(+ "id",+ "name",+ "position",+ "matchType",+ "isLocked",+ "isDeleted",+ "createdAt",+ "updatedAt",+ ),+ ),+ "lifecycle_histories": PardotEndpointConfig(+ name="lifecycle_histories",+ path="lifecycle-histories",+ # Lifecycle histories are append-only and only filter/sort on createdAt.+ order_by="id",+ incremental_fields=[incremental_field("createdAt")],+ partition_key="createdAt",+ fields=(+ "id",+ "prospectId",+ "previousStageId",+ "nextStageId",+ "secondsElapsed",+ "createdAt",+ ),+ ),+ "visitor_page_views": PardotEndpointConfig(+ name="visitor_page_views",+ path="visitor-page-views",+ order_by="id",+ incremental_fields=[incremental_field("createdAt")],+ partition_key="createdAt",+ fields=(+ "id",+ "url",+ "title",+ "visitorId",+ "visitId",+ "campaignId",+ "durationInSeconds",+ "salesforceId",+ "createdAt",+ ),+ ),}ENDPOINTS = tuple(PARDOT_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/tests/test_pardot.py b/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/tests/test_pardot.pyindex 21f018b93535..6421d18ba748 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/tests/test_pardot.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/pardot/tests/test_pardot.py@@ -118,16 +118,24 @@ class TestFormatDatetime:class TestBuildQueryParams:- def test_incremental_filters_and_sorts_on_the_chosen_cursor(self) -> None:+ @pytest.mark.parametrize(+ "endpoint, incremental_field",+ [+ ("prospects", "updatedAt"),+ # Emails have no createdAt/updatedAt filters, only sentAt.+ ("emails", "sentAt"),+ ],+ )+ def test_incremental_filters_and_sorts_on_the_chosen_cursor(self, endpoint: str, incremental_field: str) -> None:params = _build_query_params(- PARDOT_ENDPOINTS["prospects"],+ PARDOT_ENDPOINTS[endpoint],should_use_incremental_field=True,db_incremental_field_last_value=datetime(2024, 5, 1, tzinfo=UTC),- incremental_field="updatedAt",+ incremental_field=incremental_field,)- assert params["orderBy"] == "updatedAt"- assert params["updatedAtAfterOrEqualTo"] == "2024-05-01T00:00:00+00:00"+ assert params["orderBy"] == incremental_field+ assert params[f"{incremental_field}AfterOrEqualTo"] == "2024-05-01T00:00:00+00:00"@pytest.mark.parametrize("endpoint, incremental_field",@@ -161,6 +169,10 @@ class TestFieldLists:# v5 400s a query whose `fields` list names a write-only-on-create field.("custom_fields", "valuesPrefill"),("list_emails", "scheduledTime"),+ ("emails", "scheduledTime"),+ # Email bodies aren't queryable and 400 the same way.+ ("emails", "htmlMessage"),+ ("emails", "textMessage"),],)def test_does_not_request_write_only_fields(self, endpoint: str, write_only_field: str) -> None: