Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 2 missing tables to the Mailjet source (#124966)

Tom Owersbd3d9c9ac667
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 4b230b3e7bf0..326bac90a6a5 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -7604,7 +7604,7 @@ Note: The URLs supplied in the payload (documentation.mailgun.com/docs/mailgun/a
## Mailjet — gaps
-Today (13): `bouncestatistics`, `campaign`, `campaigndraft`, `clickstatistics`, `contact`, `contactmetadata`, `contactslist`, `listrecipient`, `message`, `messageevent`, `openinformation`, `statcounters`, `template`
+Today (15): `bouncestatistics`, `campaign`, `campaigndraft`, `clickstatistics`, `contact`, `contactdata`, `contactmetadata`, `contactslist`, `listrecipient`, `message`, `messageevent`, `messageinformation`, `openinformation`, `statcounters`, `template`
Diffed against: <https://dev.mailjet.com/email/reference/messages/>
@@ -7612,10 +7612,10 @@ Diffed against: <https://dev.mailjet.com/email/reference/messages/>
- [ ] `messagesentstatistics` — skipped: Mailjet lists it as a deprecated legacy statistics resource replaced by `/statcounters`.
- [ ] `messagehistory` — skipped: only exposed as `/messagehistory/{message_ID}`, so it would take one request per message ever sent. The webhook-fed `messageevent` table already carries this event stream.
- [x] `bouncestatistics` — Bounce events with reason codes; we sync clicks and opens but not bounces (high)
-- [ ] `campaignstatistics` — Aggregate performance per campaign, resolving the campaign rows we already sync (high)
-- [ ] `contactdata` — Actual custom property values per contact - we sync contactmetadata (the schema) but not the values (high)
-- [ ] `openstatistics` — Open events aggregated per message/campaign, complementing the openinformation rows we sync (medium)
-- [ ] `messageinformation` — Per-message metadata linking messages to campaigns, senders and spam assassin scores (medium)
+- [ ] `campaignstatistics` — skipped: Mailjet's OpenAPI spec marks it as a deprecated legacy endpoint and points to `/statcounters`, which we already sync.
+- [x] `contactdata` — Actual custom property values per contact - we sync contactmetadata (the schema) but not the values (high). Added as `contactdata`: full refresh, because the API has no change filter.
+- [ ] `openstatistics` — skipped: Mailjet's OpenAPI spec marks it as a deprecated legacy endpoint and points to `/statcounters`, which we already sync.
+- [x] `messageinformation` — Per-message metadata linking messages to campaigns, senders and spam assassin scores (medium). Added as `messageinformation`: incremental on `CreatedAt` through `FromTS`.
- [ ] `newsletter` — Newsletter objects that campaign and campaigndraft rows reference (medium)
- [ ] `sender` — Lookup table resolving the sender IDs carried on campaigns and messages (medium)
- [ ] `listrecipientstatistics` — Engagement metrics per list subscription, joining to the listrecipient rows we sync (medium)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/canonical_descriptions.py
index 737e3f4aea7d..0d2bc1e9fd82 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/canonical_descriptions.py
@@ -123,6 +123,33 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"CreatedAt": "Time the template was created.",
},
},
+ "contactdata": {
+ "description": "The custom contact property values stored for each contact in Mailjet.",
+ "docs_url": "https://dev.mailjet.com/openapi/openapi-mailjet/contacts/get_v3_contactdata",
+ "columns": {
+ "ID": "Unique identifier of the contact's data object. Always equal to ContactID.",
+ "ContactID": "The identifier of the contact the property values belong to.",
+ "Data": "List of the contact's property values, each with the property Name and its Value.",
+ },
+ },
+ "messageinformation": {
+ "description": "Sending, size and spam information for each message processed by Mailjet.",
+ "docs_url": "https://dev.mailjet.com/openapi/openapi-mailjet/messages/get_v3_messageinformation",
+ "columns": {
+ "ID": "Unique identifier of the message.",
+ "CampaignID": "The identifier of the campaign the message is part of.",
+ "ContactID": "The identifier of the recipient contact.",
+ "CreatedAt": "Time the message was created.",
+ "SendEndAt": "Time the last message of the campaign was sent.",
+ "MessageSize": "The size of the message in bytes.",
+ "SentCount": "The number of send attempts.",
+ "QueuedCount": "The number of messages waiting in the send queue.",
+ "OpenTrackedCount": "The number of messages with open tracking enabled.",
+ "ClickTrackedCount": "The number of messages with click tracking enabled.",
+ "SpamAssassinScore": "The SpamAssassin score of the message.",
+ "SpamAssassinRules": "The SpamAssassin rules the message matched.",
+ },
+ },
"openinformation": {
"description": "An open event recording that a recipient opened a message in Mailjet.",
"docs_url": "https://dev.mailjet.com/email/reference/messages/#v3_get_openinformation",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/mailjet.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/mailjet.py
index 4a9c1fc3971e..f1dc5f3bea0d 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/mailjet.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/mailjet.py
@@ -317,7 +317,8 @@ def mailjet_source(
partition_mode="datetime" if config.partition_key else None,
partition_format="month" if config.partition_key else None,
partition_keys=[config.partition_key] if config.partition_key else None,
- sort_mode="asc",
+ # An incremental read is only ordered by its cursor when the pages are sorted on that field.
+ sort_mode="asc" if config.from_ts_field in (None, config.sort) else None,
column_hints=resource.column_hints,
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/settings.py
index c4d47d2db1a9..60d06bb7f843 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/settings.py
@@ -78,6 +78,22 @@ MAILJET_ENDPOINTS: dict[str, MailjetEndpointConfig] = {
name="template",
path="template",
),
+ # Property values per contact. Row `ID` equals `ContactID`. No server-side change filter exists,
+ # so this is full refresh only.
+ "contactdata": MailjetEndpointConfig(
+ name="contactdata",
+ path="contactdata",
+ ),
+ # `FromTS` filters on send time, which is never before `CreatedAt`, so a `CreatedAt` watermark
+ # loses no rows. Mailjet does not document `Sort=CreatedAt` here, so pages stay sorted by ID.
+ "messageinformation": MailjetEndpointConfig(
+ name="messageinformation",
+ path="messageinformation",
+ partition_key="CreatedAt",
+ sort="ID",
+ from_ts_field="CreatedAt",
+ incremental_fields=[_datetime_incremental_field("CreatedAt")],
+ ),
"openinformation": MailjetEndpointConfig(
name="openinformation",
path="openinformation",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/tests/test_mailjet.py b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/tests/test_mailjet.py
index 879dee879ca4..5d9a6ba9f108 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/tests/test_mailjet.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/mailjet/tests/test_mailjet.py
@@ -125,9 +125,10 @@ class TestResume:
class TestIncremental:
- def test_from_ts_applied_for_statistics_endpoint(self) -> None:
+ @parameterized.expand([("openinformation",), ("messageinformation",)])
+ def test_from_ts_applied_for_time_filtered_endpoint(self, endpoint: str) -> None:
result = _driver().run(
- "openinformation",
+ endpoint,
[_response([{"ID": 1}], total=1)],
should_use_incremental_field=True,
db_incremental_field_last_value=datetime(2026, 1, 1, tzinfo=UTC),
@@ -228,7 +229,8 @@ class TestSourceResponseShape:
assert response.name == endpoint
assert response.primary_keys == [config.primary_key]
- assert response.sort_mode == "asc"
+ # messageinformation filters on CreatedAt but pages by ID, so it must not promise ascending order.
+ assert response.sort_mode == (None if endpoint == "messageinformation" else "asc")
if config.partition_key:
assert response.partition_mode == "datetime"
assert response.partition_format == "month"