Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse-sources): add 9 missing tables to the SendGrid source (#130172)

Tom Owers2abaf078d1c3
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 534b4ffa519b..97b7ac6025fa 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -10823,9 +10823,9 @@ Diffed against: <https://github.com/twilio/sendgrid-oai/tree/main/spec/json (off
- [x] `GET /v3/categories/stats (and /v3/categories/stats/sums)` — Email stats broken down by category, the main way transactional senders segment performance (medium)
- [x] `GET /v3/categories` — Category lookup resolving the category tags used on stats and messages (medium)
- [x] `GET /v3/marketing/stats/automations (+ /{id}/links)` — Automation (journey) performance stats (medium)
-- [ ] `GET /v3/marketing/segments/2.0` — Marketing segment definitions; marketing_lists is synced but segments are not (medium)
-- [ ] `GET /v3/geo/stats, /v3/mailbox_providers/stats, /v3/devices/stats, /v3/clients/stats, /v3/browsers/stats` — Breakdown dimensions on the same stats fact (country, mailbox provider, device, client, browser) (medium)
-- [ ] `GET /v3/subusers and /v3/subusers/stats (+ /stats/monthly, /stats/sums, /reputations)` — Per-subuser volume and reputation — required for any agency or multi-tenant SendGrid account (medium)
+- [x] `GET /v3/marketing/segments/2.0` — Marketing segment definitions; marketing_lists is synced but segments are not (medium)
+- [x] `GET /v3/geo/stats, /v3/mailbox_providers/stats, /v3/devices/stats, /v3/clients/stats, /v3/browsers/stats` — Breakdown dimensions on the same stats fact (country, mailbox provider, device, client, browser) (medium)
+- [x] `GET /v3/subusers and /v3/subusers/stats (+ /stats/monthly, /stats/sums, /reputations)` — Per-subuser volume and reputation — required for any agency or multi-tenant SendGrid account (medium)
Note: Current coverage is essentially only the suppression family plus templates and lists — 8 tables against a spec set of 46 OpenAPI files. Not a single statistics or activity endpoint is synced, which is the bulk of analytical value. /v3/marketing/contacts returns a sample only; the full contact set requires the async exports job (POST /v3/marketing/contacts/exports then poll and download).
diff --git a/products/warehouse_sources/sources/sendgrid/api_inventory.md b/products/warehouse_sources/sources/sendgrid/api_inventory.md
index 248371d67c52..16e2a337ae4a 100644
--- a/products/warehouse_sources/sources/sendgrid/api_inventory.md
+++ b/products/warehouse_sources/sources/sendgrid/api_inventory.md
@@ -36,6 +36,15 @@ on the primary key — wasted API calls, not corrupted data.
| `category_stats` | `/categories/stats` (fan-out over `categories`) | none (single) | nested daily stats | `category`, `date` | `date` via `start_date` (YYYY-MM-DD) |
| `automation_stats` | `/marketing/stats/automations` | `_metadata` | `{"results": [...]}` | `id`, `aggregation`, `step_id` | full refresh |
| `automation_link_stats` | `/marketing/stats/automations/{id}/links` (fan-out) | `_metadata` | `{"results": [...]}` | `automation_id`, `url`, `url_location`, `step_id` | full refresh |
+| `geo_stats` | `/geo/stats` | limit/offset | nested daily stats | `date`, `type`, `name` | `date` via `start_date` (YYYY-MM-DD) |
+| `mailbox_provider_stats` | `/mailbox_providers/stats` | limit/offset | nested daily stats | `date`, `type`, `name` | `date` via `start_date` (YYYY-MM-DD) |
+| `device_stats` | `/devices/stats` | limit/offset | nested daily stats | `date`, `type`, `name` | `date` via `start_date` (YYYY-MM-DD) |
+| `client_stats` | `/clients/stats` | none (single) | nested daily stats | `date`, `type`, `name` | `date` via `start_date` (YYYY-MM-DD) |
+| `browser_stats` | `/browsers/stats` | limit/offset | nested daily stats | `date`, `type`, `name` | `date` via `start_date` (YYYY-MM-DD) |
+| `segments` | `/marketing/segments/2.0` | `_metadata` | `{"results": [...]}` | `id` | full refresh |
+| `subusers` | `/subusers` | limit/offset | bare array | `id` | full refresh |
+| `subuser_stats` | `/subusers/stats` (fan-out over `subusers`) | limit/offset | nested daily stats | `username`, `date` | `date` via `start_date` (YYYY-MM-DD) |
+| `subuser_reputations` | `/subusers/reputations` (fan-out over `subusers`) | none (single) | bare array | `username` | full refresh |
Notes:
@@ -109,6 +118,12 @@ Notes:
- `/marketing/contacts` is not synced. The GET returns a sample of at most 50 recent contacts. The full
set needs an asynchronous export job (`POST /marketing/contacts/exports`, then poll, then download
files from S3 links), which creates a job in the customer's account.
+- The advanced stats breakdowns (`geo_stats`, `mailbox_provider_stats`, `device_stats`, `client_stats`, `browser_stats`) return the same `{date, stats: [{type, name, metrics}]}` nesting as `/stats`, with one `stats` entry per breakdown value. Each entry flattens to one row, so `type` and `name` are in the primary key. `start_date` is required and filters server-side, so they sync incrementally on `date` like `stats`. The docs say these endpoints return 500 items per request by default. All but `/clients/stats` document `limit`/`offset`, so those send `limit=500` and stop on a short page. Each endpoint has its own scope (`geo.stats.read`, `mailbox_providers.stats.read`, `devices.stats.read`, `clients.stats.read`, `browsers.stats.read`). `/geo/stats` is not available for Regional (EU) subusers. The per-client-type `/clients/{client_type}/stats` is not synced, because `/clients/stats` covers every client.
+- `segments` reads `/marketing/segments/2.0`. The prose of the docs says it returns `{"results": [...]}`, but the published schema lists the segment fields at the top level. It documents no page params, so the sync sends no `page_size` and follows `_metadata.next` only when the API returns one. The legacy `/marketing/segments` (v1) is not synced.
+- `subusers` reads `/subusers`, a bare array paged by `limit`/`offset`. No maximum `limit` is documented, so it pages at 100. Only a parent account has subusers (`subusers.read`).
+- `/subusers/stats` requires `subusers` (up to 10 names) and `start_date`, so `subuser_stats` fans out over `subusers` with one subuser per request, like `category_stats`. The username is percent-encoded before it goes into the query string. It needs `subusers.stats.read`, so the permission probe calls `/subusers/stats` with a placeholder subuser.
+- `subuser_reputations` fans out over `subusers` and calls `/subusers/reputations?usernames=<name>`, because the spec does not say what the endpoint returns without `usernames`. It needs `subusers.reputations.read`. It is a snapshot with no history, so it is full refresh.
+- `/subusers/stats/sums` and `/subusers/stats/monthly` are not synced: they sum the `subuser_stats` metrics over a date range or a month, which a query over `subuser_stats` can do.
- `/marketing/lists` is the outlier: it needs `marketing.read` **and** an account with Marketing Campaigns.
Accounts without it, and accounts still on legacy Marketing Campaigns (a different API, scoped
`marketing_campaigns.read`), return 403 no matter how the key is scoped, so the table can be genuinely
diff --git a/products/warehouse_sources/sources/sendgrid/canonical_descriptions.py b/products/warehouse_sources/sources/sendgrid/canonical_descriptions.py
index ff738f4fa2db..3c442d635ec6 100644
--- a/products/warehouse_sources/sources/sendgrid/canonical_descriptions.py
+++ b/products/warehouse_sources/sources/sendgrid/canonical_descriptions.py
@@ -16,6 +16,16 @@ _SUPPRESSION_COLUMNS = {
"status": "Status code associated with the suppression event.",
}
+_BREAKDOWN_STATS_COLUMNS = {
+ "date": "The date the statistics were gathered, in YYYY-MM-DD format.",
+ "type": "The type of breakdown the row is for.",
+ "name": "The breakdown value the statistics are for.",
+ "opens": "Total number of times the emails were opened by recipients.",
+ "unique_opens": "Number of unique recipients who opened the emails.",
+ "clicks": "Number of links that were clicked in the emails.",
+ "unique_clicks": "Number of unique recipients who clicked links in the emails.",
+}
+
CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"bounces": {
"description": "Email addresses that bounced — the receiving server rejected the message.",
@@ -72,6 +82,42 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"invalid_emails": "Number of emails dropped because the recipient address was invalid.",
},
},
+ "geo_stats": {
+ "description": "Daily email open and click statistics for each recipient country.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/stats/retrieve-email-statistics-by-country-and-stateprovince",
+ "columns": {**_BREAKDOWN_STATS_COLUMNS, "name": "The country or state the statistics are for."},
+ },
+ "mailbox_provider_stats": {
+ "description": "Daily email statistics for each recipient mailbox provider, such as Gmail or Yahoo.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/stats/retrieve-email-statistics-by-mailbox-provider",
+ "columns": {
+ **_BREAKDOWN_STATS_COLUMNS,
+ "name": "The mailbox provider the statistics are for.",
+ "requests": "Number of emails that were requested to be delivered.",
+ "processed": "Number of requests that SendGrid processed.",
+ "delivered": "Number of emails that SendGrid confirmed were delivered to a recipient.",
+ "deferred": "Number of emails that temporarily could not be delivered.",
+ "bounces": "Number of emails that bounced instead of being delivered.",
+ "blocks": "Number of emails that were not allowed to be delivered by ISPs.",
+ "drops": "Number of emails that were not delivered because of a previous suppression.",
+ "spam_reports": "Number of recipients who marked the email as spam.",
+ },
+ },
+ "device_stats": {
+ "description": "Daily email open statistics for each recipient device type, such as desktop or phone.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/stats/retrieve-email-statistics-by-device-type",
+ "columns": {**_BREAKDOWN_STATS_COLUMNS, "name": "The device type the statistics are for."},
+ },
+ "client_stats": {
+ "description": "Daily email open statistics for each email client, such as Gmail or Apple Mail.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/stats/retrieve-email-statistics-by-client-type",
+ "columns": {**_BREAKDOWN_STATS_COLUMNS, "name": "The email client the statistics are for."},
+ },
+ "browser_stats": {
+ "description": "Daily email click statistics for each recipient browser.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/stats/retrieve-email-statistics-by-browser",
+ "columns": {**_BREAKDOWN_STATS_COLUMNS, "name": "The browser the statistics are for."},
+ },
"categories": {
"description": "Categories: the tags senders attach to emails to group them by topic, used to segment statistics.",
"docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/categories/retrieve-all-categories",
@@ -133,6 +179,22 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"contact_count": "Number of contacts on the list.",
},
},
+ "segments": {
+ "description": "Marketing segments, which select contacts with a query.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/segmenting-contacts-v2/get-list-of-segments",
+ "columns": {
+ "id": "Unique identifier for the segment.",
+ "name": "Name of the segment.",
+ "contacts_count": "Number of contacts in the segment.",
+ "created_at": "Time at which the segment was created.",
+ "updated_at": "Time at which the segment was last updated.",
+ "sample_updated_at": "Time at which the segment's contact sample was last updated.",
+ "next_sample_update": "Time at which the segment's contact sample is next updated.",
+ "parent_list_ids": "IDs of the lists the segment filters contacts from.",
+ "query_version": "Version of the segment query language.",
+ "status": "Validation status of the segment query.",
+ },
+ },
"singlesends": {
"description": "Single Sends: one-time Marketing Campaigns emails sent to lists or segments.",
"docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/single-sends/get-all-single-sends",
@@ -192,6 +254,49 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"clicks": "Number of clicks on this link.",
},
},
+ "subusers": {
+ "description": "Subusers of a parent account.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/subusers-api/list-all-subusers",
+ "columns": {
+ "id": "Unique identifier for the subuser.",
+ "username": "Username of the subuser.",
+ "email": "Contact email address of the subuser.",
+ "disabled": "Whether the subuser is disabled.",
+ "region": "Region the subuser is pinned to: global or eu.",
+ },
+ },
+ "subuser_stats": {
+ "description": "Daily email statistics for each subuser, such as requests, deliveries, opens, clicks, and bounces.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/subuser-statistics/retrieve-email-statistics-for-your-subuser",
+ "columns": {
+ "date": "The date the statistics were gathered, in YYYY-MM-DD format.",
+ "username": "Username of the subuser the statistics are for.",
+ "requests": "Number of emails that were requested to be delivered.",
+ "processed": "Number of requests that SendGrid processed.",
+ "delivered": "Number of emails that SendGrid confirmed were delivered to a recipient.",
+ "deferred": "Number of emails that temporarily could not be delivered.",
+ "opens": "Total number of times the emails were opened by recipients.",
+ "unique_opens": "Number of unique recipients who opened the emails.",
+ "clicks": "Number of links that were clicked.",
+ "unique_clicks": "Number of unique recipients who clicked links in the emails.",
+ "bounces": "Number of emails that bounced instead of being delivered.",
+ "bounce_drops": "Number of emails that were dropped because of a bounce.",
+ "blocks": "Number of emails that were not allowed to be delivered by ISPs.",
+ "invalid_emails": "Number of recipients with malformed addresses or addresses that the mail provider reported as invalid.",
+ "spam_reports": "Number of recipients who marked the email as spam.",
+ "spam_report_drops": "Number of emails that were dropped because the recipient previously marked emails as spam.",
+ "unsubscribes": "Number of recipients who unsubscribed.",
+ "unsubscribe_drops": "Number of emails that were dropped because the recipient unsubscribed.",
+ },
+ },
+ "subuser_reputations": {
+ "description": "Current sender reputation of each subuser.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/subusers-api/retrieve-subuser-reputations",
+ "columns": {
+ "username": "Username of the subuser.",
+ "reputation": "Sender reputation of the subuser, from 0 to 100.",
+ },
+ },
"templates": {
"description": "Email templates (legacy and dynamic) used to compose messages.",
"docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/transactional-templates/retrieve-paged-transactional-templates",
diff --git a/products/warehouse_sources/sources/sendgrid/sendgrid.py b/products/warehouse_sources/sources/sendgrid/sendgrid.py
index 73300b8e0ba8..7d1168e78105 100644
--- a/products/warehouse_sources/sources/sendgrid/sendgrid.py
+++ b/products/warehouse_sources/sources/sendgrid/sendgrid.py
@@ -111,15 +111,28 @@ def _flatten_daily_stats(item: dict[str, Any]) -> list[dict[str, Any]]:
return rows
-def _flatten_category_stats(item: dict[str, Any]) -> list[dict[str, Any]]:
- """Flatten one `{date, stats: [{name, metrics: {...}}]}` bucket of a single-category request
- into a row per metrics entry, keyed by the requested category from the parent row."""
+def _flatten_breakdown_stats(item: dict[str, Any]) -> list[dict[str, Any]]:
+ """Flatten one `{date, stats: [{type, name, metrics: {...}}]}` bucket into a row per breakdown
+ value (country, device, browser, ...), with the bucket `date` merged in."""
return [
- {"date": item.get("date"), "category": item.get("category"), **(entry.get("metrics") or {})}
+ {"date": item.get("date"), "type": entry.get("type"), "name": entry.get("name"), **(entry.get("metrics") or {})}
for entry in item.get("stats") or []
]
+def _parent_stats_flattener(parent_field: str) -> Callable[[dict[str, Any]], list[dict[str, Any]]]:
+ """Flatten one `{date, stats: [{name, metrics: {...}}]}` bucket of a single-parent request
+ into a row per metrics entry, keyed by the requested parent value (category, subuser)."""
+
+ def _flatten(item: dict[str, Any]) -> list[dict[str, Any]]:
+ return [
+ {"date": item.get("date"), parent_field: item.get(parent_field), **(entry.get("metrics") or {})}
+ for entry in item.get("stats") or []
+ ]
+
+ return _flatten
+
+
_ACTIVITY_TIMESTAMP_FORMAT = "%Y-%m-%dT%H:%M:%SZ"
@@ -315,7 +328,7 @@ def _build_params(
# Metadata endpoints carry the page size as a query param; offset endpoints get limit/offset
# from the paginator, single endpoints take no pagination params.
- if config.pagination == "metadata":
+ if config.pagination == "metadata" and config.send_page_size:
params["page_size"] = config.page_size
if config.incremental_param:
@@ -433,7 +446,7 @@ def _fanout_resource(
# The cursor is already bound into `child_params` as the endpoint's start param.
incremental_config_factory=lambda _field: None,
# Offset parents get `limit` from their paginator, single-page children take no page size.
- page_size_param="page_size" if config.pagination == "metadata" else None,
+ page_size_param="page_size" if config.pagination == "metadata" and config.send_page_size else None,
child_params_extra=child_params,
parent_data_map=(
_encoded_parent_field(config.encode_parent_field, config.fanout.resolve_field)
@@ -453,9 +466,10 @@ def _fanout_resource(
"data_selector_required": True,
},
)
- if config.response_shape == "category_stats":
+ if config.response_shape == "parent_stats":
assert isinstance(resource, Resource)
- resource.add_map(_flatten_category_stats)
+ assert config.parent_stats_field is not None
+ resource.add_map(_parent_stats_flattener(config.parent_stats_field))
return resource
@@ -519,6 +533,8 @@ def sendgrid_source(
if config.response_shape == "daily_stats":
# stats rows are nested one level deeper than the bare array; flatten them into flat daily rows.
resource_config["data_map"] = _flatten_daily_stats
+ elif config.response_shape == "breakdown_stats":
+ resource_config["data_map"] = _flatten_breakdown_stats
# Activity pagination owns its own request params (`query` window + `limit`), including the
# server-side incremental filter, so it is built with the cursor instead of via `params`.
@@ -586,7 +602,7 @@ def _probe_params(config: SendGridEndpointConfig) -> dict[str, Any]:
params: dict[str, Any] = dict(config.extra_params)
if config.pagination == "offset":
params["limit"] = 1
- elif config.pagination == "metadata":
+ elif config.pagination == "metadata" and config.send_page_size:
params["page_size"] = 1
elif config.pagination == "activity":
# `query` is required on /messages, so probe a one-day window for one row. The 403 for a
diff --git a/products/warehouse_sources/sources/sendgrid/settings.py b/products/warehouse_sources/sources/sendgrid/settings.py
index 752d6c3a3680..c7c9011c7020 100644
--- a/products/warehouse_sources/sources/sendgrid/settings.py
+++ b/products/warehouse_sources/sources/sendgrid/settings.py
@@ -24,9 +24,11 @@ IncrementalParamFormat = Literal["epoch", "date"]
# - "array": the rows are the JSON body itself, or the array under `data_key`.
# - "daily_stats": nested `[{date, stats: [{metrics: {...}}]}]` that flattens to one row per
# metrics bucket with `date` merged in (the /stats family).
-# - "category_stats": the same nesting, flattened to one row per date with the requested
-# `category` merged in (category_stats).
-ResponseShape = Literal["array", "daily_stats", "category_stats"]
+# - "breakdown_stats": the same nesting, flattened to one row per `stats` entry with the
+# entry's `type` and `name` kept (the advanced stats breakdowns).
+# - "parent_stats": the same nesting, flattened to one row per date with the fan-out parent
+# value named by `parent_stats_field` merged in (category_stats, subuser_stats).
+ResponseShape = Literal["array", "daily_stats", "breakdown_stats", "parent_stats"]
# /stats requires `start_date` on every request, so the first sync (no cursor yet) backfills this
# far back. Daily aggregation means one row per day, so a year is cheap and gives enough history to
@@ -46,6 +48,8 @@ class SendGridEndpointConfig:
# Stable creation field used for datetime partitioning. Never use a "modified" field here.
partition_key: Optional[str] = None
page_size: int = 500
+ # False for a metadata endpoint that documents no `page_size` param and returns every row at once.
+ send_page_size: bool = True
incremental_fields: list[IncrementalField] = field(default_factory=list)
# Server-side query param that filters by the incremental field.
incremental_param: Optional[str] = None
@@ -57,6 +61,8 @@ class SendGridEndpointConfig:
default_backfill_days: Optional[int] = None
# How the response body is shaped, so the transport can flatten nested stats responses.
response_shape: ResponseShape = "array"
+ # Parent row field merged into each flattened row of a "parent_stats" response.
+ parent_stats_field: Optional[str] = None
# Order rows arrive in across the whole sync. The Email Activity walk pages newest to oldest,
# so it declares "desc" and the pipeline defers the incremental watermark to the end of a
# successful run instead of checkpointing it per batch. None when rows come one parent at a
@@ -94,6 +100,8 @@ _MARKETING_CAMPAIGNS_NOTE = (
"Accounts without Marketing Campaigns, or on legacy Marketing Campaigns, cannot sync this table."
)
+_SUBUSERS_NOTE = "Only a parent account with Subusers can sync this table."
+
def _epoch_field(name: str) -> IncrementalField:
return {
@@ -122,9 +130,37 @@ def _datetime_field(name: str) -> IncrementalField:
}
+def _breakdown_stats_endpoint(
+ name: str,
+ path: str,
+ required_scope: str,
+ pagination: PaginationMode = "offset",
+ permission_note: Optional[str] = None,
+) -> SendGridEndpointConfig:
+ return SendGridEndpointConfig(
+ name=name,
+ path=path,
+ # One row per breakdown value per day. `type` stays in the key because geo rows can be a
+ # country or a state.
+ primary_keys=["date", "type", "name"],
+ pagination=pagination,
+ partition_key="date",
+ incremental_fields=[_date_field("date")],
+ incremental_param="start_date",
+ incremental_param_format="date",
+ default_backfill_days=STATS_DEFAULT_BACKFILL_DAYS,
+ response_shape="breakdown_stats",
+ # The API does not document the order of the date buckets.
+ sort_mode=None,
+ extra_params={"aggregated_by": "day"},
+ required_scope=required_scope,
+ permission_note=permission_note,
+ )
+
+
# SendGrid (v3) endpoint catalog. The suppression endpoints expose a genuine server-side
# `start_time` filter (Unix epoch seconds) over their immutable `created` field, so they sync
-# incrementally. `stats` and `category_stats` filter on `start_date` (YYYY-MM-DD) over their daily
+# incrementally. The stats tables filter on `start_date` (YYYY-MM-DD) over their daily
# aggregate date and also sync incrementally. Marketing/asm metadata endpoints offer no timestamp filter and ship as
# full refresh. See api_inventory.md for the per-endpoint research notes.
SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {
@@ -195,6 +231,22 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {
extra_params={"aggregated_by": "day"},
required_scope="stats.read",
),
+ # The advanced stats breakdowns. They return 500 items per request by default, and all but
+ # /clients/stats document `limit`/`offset`, so those page with an explicit limit.
+ "geo_stats": _breakdown_stats_endpoint(
+ "geo_stats",
+ "/geo/stats",
+ "geo.stats.read",
+ permission_note="Regional (EU) subusers cannot sync this table.",
+ ),
+ "mailbox_provider_stats": _breakdown_stats_endpoint(
+ "mailbox_provider_stats", "/mailbox_providers/stats", "mailbox_providers.stats.read"
+ ),
+ "device_stats": _breakdown_stats_endpoint("device_stats", "/devices/stats", "devices.stats.read"),
+ "client_stats": _breakdown_stats_endpoint(
+ "client_stats", "/clients/stats", "clients.stats.read", pagination="single"
+ ),
+ "browser_stats": _breakdown_stats_endpoint("browser_stats", "/browsers/stats", "browsers.stats.read"),
"categories": SendGridEndpointConfig(
name="categories",
path="/categories",
@@ -218,7 +270,8 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {
incremental_param="start_date",
incremental_param_format="date",
default_backfill_days=STATS_DEFAULT_BACKFILL_DAYS,
- response_shape="category_stats",
+ response_shape="parent_stats",
+ parent_stats_field="category",
sort_mode=None,
extra_params={"aggregated_by": "day"},
required_scope="categories.stats.read",
@@ -263,6 +316,18 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {
required_scope="marketing.read",
permission_note=_MARKETING_CAMPAIGNS_NOTE,
),
+ # Segments v2 returns every segment in one `results` array and documents no page params.
+ "segments": SendGridEndpointConfig(
+ name="segments",
+ path="/marketing/segments/2.0",
+ primary_keys=["id"],
+ pagination="metadata",
+ data_key="results",
+ send_page_size=False,
+ partition_key="created_at",
+ required_scope="marketing.read",
+ permission_note=_MARKETING_CAMPAIGNS_NOTE,
+ ),
"singlesends": SendGridEndpointConfig(
name="singlesends",
path="/marketing/singlesends",
@@ -338,6 +403,67 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {
child_response_actions=[{"status_code": 404, "action": "ignore"}],
),
),
+ "subusers": SendGridEndpointConfig(
+ name="subusers",
+ path="/subusers",
+ primary_keys=["id"],
+ pagination="offset",
+ # No maximum is documented, and a silently capped page would read as the short last page.
+ page_size=100,
+ extra_params={"include_region": "true"},
+ required_scope="subusers.read",
+ permission_note=_SUBUSERS_NOTE,
+ ),
+ # /subusers/stats requires the `subusers` param, so it fans out over the subuser listing one
+ # subuser per request.
+ "subuser_stats": SendGridEndpointConfig(
+ name="subuser_stats",
+ path="/subusers/stats?subusers={username}",
+ primary_keys=["username", "date"],
+ pagination="offset",
+ partition_key="date",
+ incremental_fields=[_date_field("date")],
+ incremental_param="start_date",
+ incremental_param_format="date",
+ default_backfill_days=STATS_DEFAULT_BACKFILL_DAYS,
+ response_shape="parent_stats",
+ parent_stats_field="username",
+ sort_mode=None,
+ extra_params={"aggregated_by": "day"},
+ required_scope="subusers.stats.read",
+ permission_note=_SUBUSERS_NOTE,
+ default_incremental_field="date",
+ fanout=DependentEndpointConfig(
+ parent_name="subusers",
+ resolve_param="username",
+ resolve_field="username_param",
+ include_from_parent=["username"],
+ parent_field_renames={"username": "username"},
+ child_response_actions=[{"status_code": 404, "action": "ignore"}],
+ ),
+ probe_resolve_value="posthog",
+ encode_parent_field="username",
+ ),
+ # The spec does not say what `usernames` defaults to, so each subuser is requested by name.
+ "subuser_reputations": SendGridEndpointConfig(
+ name="subuser_reputations",
+ path="/subusers/reputations?usernames={username}",
+ primary_keys=["username"],
+ pagination="single",
+ sort_mode=None,
+ required_scope="subusers.reputations.read",
+ permission_note=_SUBUSERS_NOTE,
+ fanout=DependentEndpointConfig(
+ parent_name="subusers",
+ resolve_param="username",
+ resolve_field="username_param",
+ include_from_parent=["username"],
+ parent_field_renames={"username": "username"},
+ child_response_actions=[{"status_code": 404, "action": "ignore"}],
+ ),
+ probe_resolve_value="posthog",
+ encode_parent_field="username",
+ ),
"templates": SendGridEndpointConfig(
name="templates",
path="/templates",
diff --git a/products/warehouse_sources/sources/sendgrid/source.py b/products/warehouse_sources/sources/sendgrid/source.py
index 611592a4eedf..bd6d60d96c95 100644
--- a/products/warehouse_sources/sources/sendgrid/source.py
+++ b/products/warehouse_sources/sources/sendgrid/source.py
@@ -198,9 +198,10 @@ You can create an API key in your [SendGrid account settings](https://app.sendgr
Grant the following read access (Restricted Access) so the key can reach the data you want to sync:
- **Suppressions**: bounces, blocks, invalid emails, spam reports, global unsubscribes, unsubscribe groups and their members
-- **Stats**: global email statistics (daily requests, delivered, opens, clicks, bounces)
+- **Stats**: global email statistics (daily requests, delivered, opens, clicks, bounces), and daily breakdowns by country, mailbox provider, device, email client and browser
- **Category Management**: categories and their daily statistics
-- **Marketing**: marketing lists, single sends, single send stats, and automation stats. This also needs an account with Marketing Campaigns enabled.
+- **Marketing**: marketing lists, segments, single sends, single send stats, and automation stats. This also needs an account with Marketing Campaigns enabled.
+- **Subusers**: subusers, their daily statistics and their sender reputations. Only a parent account with Subusers has this data.
- **Template Engine**: templates
- **Email Activity**: message activity. This table is off by default and also needs SendGrid's paid additional email activity history add-on.
diff --git a/products/warehouse_sources/sources/sendgrid/tests/test_sendgrid.py b/products/warehouse_sources/sources/sendgrid/tests/test_sendgrid.py
index 4f0fdc0ae6cc..1eee6fdb0ae6 100644
--- a/products/warehouse_sources/sources/sendgrid/tests/test_sendgrid.py
+++ b/products/warehouse_sources/sources/sendgrid/tests/test_sendgrid.py
@@ -92,6 +92,50 @@ class TestStats:
assert result.requests[0].param("start_date") == "2024-03-10"
+class TestBreakdownStats:
+ @pytest.mark.parametrize(
+ ("endpoint", "path", "expected_limit"),
+ [
+ ("geo_stats", "/v3/geo/stats", "500"),
+ # /clients/stats documents no `limit`/`offset`, so it gets none.
+ ("client_stats", "/v3/clients/stats", None),
+ ],
+ )
+ def test_flattens_one_row_per_breakdown_value_per_day(
+ self, endpoint: str, path: str, expected_limit: str | None
+ ) -> None:
+ body = [
+ {
+ "date": "2026-08-01",
+ "stats": [
+ {"type": "country", "name": "US", "metrics": {"opens": 3, "unique_opens": 2}},
+ {"type": "country", "name": "CA", "metrics": {"opens": 1, "unique_opens": 1}},
+ ],
+ },
+ ]
+
+ result = _run(
+ endpoint,
+ [_response(body)],
+ should_use_incremental_field=True,
+ db_incremental_field_last_value="2026-08-01",
+ incremental_field="date",
+ )
+
+ assert result.raised is None
+ # Without `name` every breakdown value of a day collides on the primary key.
+ assert result.rows == [
+ {"date": "2026-08-01", "type": "country", "name": "US", "opens": 3, "unique_opens": 2},
+ {"date": "2026-08-01", "type": "country", "name": "CA", "opens": 1, "unique_opens": 1},
+ ]
+ assert len(result.requests) == 1
+ request = result.requests[0]
+ assert request.path == path
+ assert request.param("limit") == expected_limit
+ assert request.param("start_date") == "2026-08-01"
+ assert request.param("aggregated_by") == "day"
+
+
class TestOffsetPagination:
def test_paginates_until_short_page_and_saves_state(self) -> None:
page1 = [{"email": f"a{i}@x.com", "created": i} for i in range(500)]
@@ -141,6 +185,13 @@ class TestMetadataPagination:
assert result.urls[1] == next_url
assert result.saved_states == [SendGridResumeConfig(next_url=next_url)]
+ def test_endpoint_without_a_page_size_param_gets_none(self) -> None:
+ result = _run("segments", [_response({"results": [{"id": "seg1"}], "_metadata": {}})])
+
+ assert result.raised is None
+ assert result.rows == [{"id": "seg1"}]
+ assert result.requests[0].param("page_size") is None
+
def test_missing_result_key_raises_loudly(self) -> None:
result = _run("marketing_lists", [_response({"_metadata": {}})])
@@ -365,6 +416,65 @@ class TestCategoryStatsFanOut:
assert [r.param("offset") for r in result.requests if r.path == "/v3/categories"] == ["0", "50"]
+def _subuser_stats_bucket(day: str, username: str, delivered: int) -> dict[str, Any]:
+ return {"date": day, "stats": [{"type": "subuser", "name": username, "metrics": {"delivered": delivered}}]}
+
+
+class TestSubuserFanOut:
+ @pytest.mark.parametrize(
+ ("endpoint", "child_path", "child_param", "child_bodies", "expected_rows"),
+ [
+ (
+ "subuser_stats",
+ "/v3/subusers/stats",
+ "subusers",
+ [[_subuser_stats_bucket("2026-08-01", "team+a", 3)], [_subuser_stats_bucket("2026-08-01", "b", 1)]],
+ [
+ {"date": "2026-08-01", "username": "team+a", "delivered": 3},
+ {"date": "2026-08-01", "username": "b", "delivered": 1},
+ ],
+ ),
+ (
+ "subuser_reputations",
+ "/v3/subusers/reputations",
+ "usernames",
+ [[{"username": "team+a", "reputation": 99}], [{"username": "b", "reputation": 95.2}]],
+ [{"username": "team+a", "reputation": 99}, {"username": "b", "reputation": 95.2}],
+ ),
+ ],
+ )
+ def test_requests_each_subuser_and_keys_rows_by_it(
+ self,
+ endpoint: str,
+ child_path: str,
+ child_param: str,
+ child_bodies: list[Any],
+ expected_rows: list[dict[str, Any]],
+ ) -> None:
+ subusers = [{"id": 1, "username": "team+a"}, {"id": 2, "username": "gone"}, {"id": 3, "username": "b"}]
+
+ result = _run(
+ endpoint,
+ route(
+ {
+ "/v3/subusers": [_response(subusers)],
+ child_path: [
+ _response(child_bodies[0]),
+ # A subuser deleted after the listing must not fail the table.
+ ScriptedResponse(status=404, json={"errors": [{"message": "not found"}]}),
+ _response(child_bodies[1]),
+ ],
+ }
+ ),
+ )
+
+ assert result.raised is None
+ assert result.rows == expected_rows
+ child_requests = [r for r in result.requests if r.path == child_path]
+ # A `+` in a username must reach the API whole, not decoded to a space.
+ assert [r.param(child_param) for r in child_requests] == ["team+a", "gone", "b"]
+
+
def _messages_page(count: int, last_event_time: str) -> list[dict[str, Any]]:
return [{"msg_id": f"m{last_event_time}-{i}", "last_event_time": last_event_time} for i in range(count)]
@@ -525,6 +635,7 @@ class TestGetEndpointPermissions:
# The category listing reads with `categories.read`, so it can't stand in for the
# `categories.stats.read` this child needs.
("category_stats", "/v3/categories/stats"),
+ ("subuser_stats", "/v3/subusers/stats"),
],
)
def test_fanout_endpoint_probe_path(self, endpoint: str, expected_path: str) -> None:
@@ -553,6 +664,8 @@ class TestGetEndpointPermissions:
("marketing_lists", {"page_size": "1"}),
("templates", {"page_size": "1", "generations": "legacy,dynamic"}),
("unsubscribe_groups", {}),
+ # Segments v2 documents no `page_size` param.
+ ("segments", {}),
# /categories/stats requires `categories` and `start_date`.
("category_stats", {"categories": "posthog", "start_date": "2026-08-07", "aggregated_by": "day"}),
# /messages requires `query`, so a probe without one risks the same masking 400.