:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 2 missing tables to the Mailjet source (#124966)
Tom Owersbd3d9c9ac667
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 4b230b3e7bf0..326bac90a6a5 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -7604,7 +7604,7 @@ Note: The URLs supplied in the payload (documentation.mailgun.com/docs/mailgun/a## Mailjet — gaps-Today (13): `bouncestatistics`, `campaign`, `campaigndraft`, `clickstatistics`, `contact`, `contactmetadata`, `contactslist`, `listrecipient`, `message`, `messageevent`, `openinformation`, `statcounters`, `template`+Today (15): `bouncestatistics`, `campaign`, `campaigndraft`, `clickstatistics`, `contact`, `contactdata`, `contactmetadata`, `contactslist`, `listrecipient`, `message`, `messageevent`, `messageinformation`, `openinformation`, `statcounters`, `template`Diffed against: <https://dev.mailjet.com/email/reference/messages/>@@ -7612,10 +7612,10 @@ Diffed against: <https://dev.mailjet.com/email/reference/messages/>- [ ] `messagesentstatistics` — skipped: Mailjet lists it as a deprecated legacy statistics resource replaced by `/statcounters`.- [ ] `messagehistory` — skipped: only exposed as `/messagehistory/{message_ID}`, so it would take one request per message ever sent. The webhook-fed `messageevent` table already carries this event stream.- [x] `bouncestatistics` — Bounce events with reason codes; we sync clicks and opens but not bounces (high)-- [ ] `campaignstatistics` — Aggregate performance per campaign, resolving the campaign rows we already sync (high)-- [ ] `contactdata` — Actual custom property values per contact - we sync contactmetadata (the schema) but not the values (high)-- [ ] `openstatistics` — Open events aggregated per message/campaign, complementing the openinformation rows we sync (medium)-- [ ] `messageinformation` — Per-message metadata linking messages to campaigns, senders and spam assassin scores (medium)+- [ ] `campaignstatistics` — skipped: Mailjet's OpenAPI spec marks it as a deprecated legacy endpoint and points to `/statcounters`, which we already sync.+- [x] `contactdata` — Actual custom property values per contact - we sync contactmetadata (the schema) but not the values (high). Added as `contactdata`: full refresh, because the API has no change filter.+- [ ] `openstatistics` — skipped: Mailjet's OpenAPI spec marks it as a deprecated legacy endpoint and points to `/statcounters`, which we already sync.+- [x] `messageinformation` — Per-message metadata linking messages to campaigns, senders and spam assassin scores (medium). Added as `messageinformation`: incremental on `CreatedAt` through `FromTS`.- [ ] `newsletter` — Newsletter objects that campaign and campaigndraft rows reference (medium)- [ ] `sender` — Lookup table resolving the sender IDs carried on campaigns and messages (medium)- [ ] `listrecipientstatistics` — Engagement metrics per list subscription, joining to the listrecipient rows we sync (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/canonical_descriptions.pyindex 737e3f4aea7d..0d2bc1e9fd82 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/canonical_descriptions.py@@ -123,6 +123,33 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"CreatedAt": "Time the template was created.",},},+ "contactdata": {+ "description": "The custom contact property values stored for each contact in Mailjet.",+ "docs_url": "https://dev.mailjet.com/openapi/openapi-mailjet/contacts/get_v3_contactdata",+ "columns": {+ "ID": "Unique identifier of the contact's data object. Always equal to ContactID.",+ "ContactID": "The identifier of the contact the property values belong to.",+ "Data": "List of the contact's property values, each with the property Name and its Value.",+ },+ },+ "messageinformation": {+ "description": "Sending, size and spam information for each message processed by Mailjet.",+ "docs_url": "https://dev.mailjet.com/openapi/openapi-mailjet/messages/get_v3_messageinformation",+ "columns": {+ "ID": "Unique identifier of the message.",+ "CampaignID": "The identifier of the campaign the message is part of.",+ "ContactID": "The identifier of the recipient contact.",+ "CreatedAt": "Time the message was created.",+ "SendEndAt": "Time the last message of the campaign was sent.",+ "MessageSize": "The size of the message in bytes.",+ "SentCount": "The number of send attempts.",+ "QueuedCount": "The number of messages waiting in the send queue.",+ "OpenTrackedCount": "The number of messages with open tracking enabled.",+ "ClickTrackedCount": "The number of messages with click tracking enabled.",+ "SpamAssassinScore": "The SpamAssassin score of the message.",+ "SpamAssassinRules": "The SpamAssassin rules the message matched.",+ },+ },"openinformation": {"description": "An open event recording that a recipient opened a message in Mailjet.","docs_url": "https://dev.mailjet.com/email/reference/messages/#v3_get_openinformation",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/mailjet.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/mailjet.pyindex 4a9c1fc3971e..f1dc5f3bea0d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/mailjet.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/mailjet.py@@ -317,7 +317,8 @@ def mailjet_source(partition_mode="datetime" if config.partition_key else None,partition_format="month" if config.partition_key else None,partition_keys=[config.partition_key] if config.partition_key else None,- sort_mode="asc",+ # An incremental read is only ordered by its cursor when the pages are sorted on that field.+ sort_mode="asc" if config.from_ts_field in (None, config.sort) else None,column_hints=resource.column_hints,)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/settings.pyindex c4d47d2db1a9..60d06bb7f843 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/settings.py@@ -78,6 +78,22 @@ MAILJET_ENDPOINTS: dict[str, MailjetEndpointConfig] = {name="template",path="template",),+ # Property values per contact. Row `ID` equals `ContactID`. No server-side change filter exists,+ # so this is full refresh only.+ "contactdata": MailjetEndpointConfig(+ name="contactdata",+ path="contactdata",+ ),+ # `FromTS` filters on send time, which is never before `CreatedAt`, so a `CreatedAt` watermark+ # loses no rows. Mailjet does not document `Sort=CreatedAt` here, so pages stay sorted by ID.+ "messageinformation": MailjetEndpointConfig(+ name="messageinformation",+ path="messageinformation",+ partition_key="CreatedAt",+ sort="ID",+ from_ts_field="CreatedAt",+ incremental_fields=[_datetime_incremental_field("CreatedAt")],+ ),"openinformation": MailjetEndpointConfig(name="openinformation",path="openinformation",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/tests/test_mailjet.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/tests/test_mailjet.pyindex 879dee879ca4..5d9a6ba9f108 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/tests/test_mailjet.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/tests/test_mailjet.py@@ -125,9 +125,10 @@ class TestResume:class TestIncremental:- def test_from_ts_applied_for_statistics_endpoint(self) -> None:+ @parameterized.expand([("openinformation",), ("messageinformation",)])+ def test_from_ts_applied_for_time_filtered_endpoint(self, endpoint: str) -> None:result = _driver().run(- "openinformation",+ endpoint,[_response([{"ID": 1}], total=1)],should_use_incremental_field=True,db_incremental_field_last_value=datetime(2026, 1, 1, tzinfo=UTC),@@ -228,7 +229,8 @@ class TestSourceResponseShape:assert response.name == endpointassert response.primary_keys == [config.primary_key]- assert response.sort_mode == "asc"+ # messageinformation filters on CreatedAt but pages by ID, so it must not promise ascending order.+ assert response.sort_mode == (None if endpoint == "messageinformation" else "asc")if config.partition_key:assert response.partition_mode == "datetime"assert response.partition_format == "month"