:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse-sources): add 9 missing tables to the SendGrid source (#130172)
Tom Owers2abaf078d1c3
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 534b4ffa519b..97b7ac6025fa 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -10823,9 +10823,9 @@ Diffed against: <https://github.com/twilio/sendgrid-oai/tree/main/spec/json (off- [x] `GET /v3/categories/stats (and /v3/categories/stats/sums)` — Email stats broken down by category, the main way transactional senders segment performance (medium)- [x] `GET /v3/categories` — Category lookup resolving the category tags used on stats and messages (medium)- [x] `GET /v3/marketing/stats/automations (+ /{id}/links)` — Automation (journey) performance stats (medium)-- [ ] `GET /v3/marketing/segments/2.0` — Marketing segment definitions; marketing_lists is synced but segments are not (medium)-- [ ] `GET /v3/geo/stats, /v3/mailbox_providers/stats, /v3/devices/stats, /v3/clients/stats, /v3/browsers/stats` — Breakdown dimensions on the same stats fact (country, mailbox provider, device, client, browser) (medium)-- [ ] `GET /v3/subusers and /v3/subusers/stats (+ /stats/monthly, /stats/sums, /reputations)` — Per-subuser volume and reputation — required for any agency or multi-tenant SendGrid account (medium)+- [x] `GET /v3/marketing/segments/2.0` — Marketing segment definitions; marketing_lists is synced but segments are not (medium)+- [x] `GET /v3/geo/stats, /v3/mailbox_providers/stats, /v3/devices/stats, /v3/clients/stats, /v3/browsers/stats` — Breakdown dimensions on the same stats fact (country, mailbox provider, device, client, browser) (medium)+- [x] `GET /v3/subusers and /v3/subusers/stats (+ /stats/monthly, /stats/sums, /reputations)` — Per-subuser volume and reputation — required for any agency or multi-tenant SendGrid account (medium)Note: Current coverage is essentially only the suppression family plus templates and lists — 8 tables against a spec set of 46 OpenAPI files. Not a single statistics or activity endpoint is synced, which is the bulk of analytical value. /v3/marketing/contacts returns a sample only; the full contact set requires the async exports job (POST /v3/marketing/contacts/exports then poll and download).diff --git a/products/warehouse_sources/sources/sendgrid/api_inventory.md b/products/warehouse_sources/sources/sendgrid/api_inventory.mdindex 248371d67c52..16e2a337ae4a 100644--- a/products/warehouse_sources/sources/sendgrid/api_inventory.md+++ b/products/warehouse_sources/sources/sendgrid/api_inventory.md@@ -36,6 +36,15 @@ on the primary key — wasted API calls, not corrupted data.| `category_stats` | `/categories/stats` (fan-out over `categories`) | none (single) | nested daily stats | `category`, `date` | `date` via `start_date` (YYYY-MM-DD) || `automation_stats` | `/marketing/stats/automations` | `_metadata` | `{"results": [...]}` | `id`, `aggregation`, `step_id` | full refresh || `automation_link_stats` | `/marketing/stats/automations/{id}/links` (fan-out) | `_metadata` | `{"results": [...]}` | `automation_id`, `url`, `url_location`, `step_id` | full refresh |+| `geo_stats` | `/geo/stats` | limit/offset | nested daily stats | `date`, `type`, `name` | `date` via `start_date` (YYYY-MM-DD) |+| `mailbox_provider_stats` | `/mailbox_providers/stats` | limit/offset | nested daily stats | `date`, `type`, `name` | `date` via `start_date` (YYYY-MM-DD) |+| `device_stats` | `/devices/stats` | limit/offset | nested daily stats | `date`, `type`, `name` | `date` via `start_date` (YYYY-MM-DD) |+| `client_stats` | `/clients/stats` | none (single) | nested daily stats | `date`, `type`, `name` | `date` via `start_date` (YYYY-MM-DD) |+| `browser_stats` | `/browsers/stats` | limit/offset | nested daily stats | `date`, `type`, `name` | `date` via `start_date` (YYYY-MM-DD) |+| `segments` | `/marketing/segments/2.0` | `_metadata` | `{"results": [...]}` | `id` | full refresh |+| `subusers` | `/subusers` | limit/offset | bare array | `id` | full refresh |+| `subuser_stats` | `/subusers/stats` (fan-out over `subusers`) | limit/offset | nested daily stats | `username`, `date` | `date` via `start_date` (YYYY-MM-DD) |+| `subuser_reputations` | `/subusers/reputations` (fan-out over `subusers`) | none (single) | bare array | `username` | full refresh |Notes:@@ -109,6 +118,12 @@ Notes:- `/marketing/contacts` is not synced. The GET returns a sample of at most 50 recent contacts. The fullset needs an asynchronous export job (`POST /marketing/contacts/exports`, then poll, then downloadfiles from S3 links), which creates a job in the customer's account.+- The advanced stats breakdowns (`geo_stats`, `mailbox_provider_stats`, `device_stats`, `client_stats`, `browser_stats`) return the same `{date, stats: [{type, name, metrics}]}` nesting as `/stats`, with one `stats` entry per breakdown value. Each entry flattens to one row, so `type` and `name` are in the primary key. `start_date` is required and filters server-side, so they sync incrementally on `date` like `stats`. The docs say these endpoints return 500 items per request by default. All but `/clients/stats` document `limit`/`offset`, so those send `limit=500` and stop on a short page. Each endpoint has its own scope (`geo.stats.read`, `mailbox_providers.stats.read`, `devices.stats.read`, `clients.stats.read`, `browsers.stats.read`). `/geo/stats` is not available for Regional (EU) subusers. The per-client-type `/clients/{client_type}/stats` is not synced, because `/clients/stats` covers every client.+- `segments` reads `/marketing/segments/2.0`. The prose of the docs says it returns `{"results": [...]}`, but the published schema lists the segment fields at the top level. It documents no page params, so the sync sends no `page_size` and follows `_metadata.next` only when the API returns one. The legacy `/marketing/segments` (v1) is not synced.+- `subusers` reads `/subusers`, a bare array paged by `limit`/`offset`. No maximum `limit` is documented, so it pages at 100. Only a parent account has subusers (`subusers.read`).+- `/subusers/stats` requires `subusers` (up to 10 names) and `start_date`, so `subuser_stats` fans out over `subusers` with one subuser per request, like `category_stats`. The username is percent-encoded before it goes into the query string. It needs `subusers.stats.read`, so the permission probe calls `/subusers/stats` with a placeholder subuser.+- `subuser_reputations` fans out over `subusers` and calls `/subusers/reputations?usernames=<name>`, because the spec does not say what the endpoint returns without `usernames`. It needs `subusers.reputations.read`. It is a snapshot with no history, so it is full refresh.+- `/subusers/stats/sums` and `/subusers/stats/monthly` are not synced: they sum the `subuser_stats` metrics over a date range or a month, which a query over `subuser_stats` can do.- `/marketing/lists` is the outlier: it needs `marketing.read` **and** an account with Marketing Campaigns.Accounts without it, and accounts still on legacy Marketing Campaigns (a different API, scoped`marketing_campaigns.read`), return 403 no matter how the key is scoped, so the table can be genuinelydiff --git a/products/warehouse_sources/sources/sendgrid/canonical_descriptions.py b/products/warehouse_sources/sources/sendgrid/canonical_descriptions.pyindex ff738f4fa2db..3c442d635ec6 100644--- a/products/warehouse_sources/sources/sendgrid/canonical_descriptions.py+++ b/products/warehouse_sources/sources/sendgrid/canonical_descriptions.py@@ -16,6 +16,16 @@ _SUPPRESSION_COLUMNS = {"status": "Status code associated with the suppression event.",}+_BREAKDOWN_STATS_COLUMNS = {+ "date": "The date the statistics were gathered, in YYYY-MM-DD format.",+ "type": "The type of breakdown the row is for.",+ "name": "The breakdown value the statistics are for.",+ "opens": "Total number of times the emails were opened by recipients.",+ "unique_opens": "Number of unique recipients who opened the emails.",+ "clicks": "Number of links that were clicked in the emails.",+ "unique_clicks": "Number of unique recipients who clicked links in the emails.",+}+CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"bounces": {"description": "Email addresses that bounced — the receiving server rejected the message.",@@ -72,6 +82,42 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"invalid_emails": "Number of emails dropped because the recipient address was invalid.",},},+ "geo_stats": {+ "description": "Daily email open and click statistics for each recipient country.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/stats/retrieve-email-statistics-by-country-and-stateprovince",+ "columns": {**_BREAKDOWN_STATS_COLUMNS, "name": "The country or state the statistics are for."},+ },+ "mailbox_provider_stats": {+ "description": "Daily email statistics for each recipient mailbox provider, such as Gmail or Yahoo.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/stats/retrieve-email-statistics-by-mailbox-provider",+ "columns": {+ **_BREAKDOWN_STATS_COLUMNS,+ "name": "The mailbox provider the statistics are for.",+ "requests": "Number of emails that were requested to be delivered.",+ "processed": "Number of requests that SendGrid processed.",+ "delivered": "Number of emails that SendGrid confirmed were delivered to a recipient.",+ "deferred": "Number of emails that temporarily could not be delivered.",+ "bounces": "Number of emails that bounced instead of being delivered.",+ "blocks": "Number of emails that were not allowed to be delivered by ISPs.",+ "drops": "Number of emails that were not delivered because of a previous suppression.",+ "spam_reports": "Number of recipients who marked the email as spam.",+ },+ },+ "device_stats": {+ "description": "Daily email open statistics for each recipient device type, such as desktop or phone.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/stats/retrieve-email-statistics-by-device-type",+ "columns": {**_BREAKDOWN_STATS_COLUMNS, "name": "The device type the statistics are for."},+ },+ "client_stats": {+ "description": "Daily email open statistics for each email client, such as Gmail or Apple Mail.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/stats/retrieve-email-statistics-by-client-type",+ "columns": {**_BREAKDOWN_STATS_COLUMNS, "name": "The email client the statistics are for."},+ },+ "browser_stats": {+ "description": "Daily email click statistics for each recipient browser.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/stats/retrieve-email-statistics-by-browser",+ "columns": {**_BREAKDOWN_STATS_COLUMNS, "name": "The browser the statistics are for."},+ },"categories": {"description": "Categories: the tags senders attach to emails to group them by topic, used to segment statistics.","docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/categories/retrieve-all-categories",@@ -133,6 +179,22 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"contact_count": "Number of contacts on the list.",},},+ "segments": {+ "description": "Marketing segments, which select contacts with a query.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/segmenting-contacts-v2/get-list-of-segments",+ "columns": {+ "id": "Unique identifier for the segment.",+ "name": "Name of the segment.",+ "contacts_count": "Number of contacts in the segment.",+ "created_at": "Time at which the segment was created.",+ "updated_at": "Time at which the segment was last updated.",+ "sample_updated_at": "Time at which the segment's contact sample was last updated.",+ "next_sample_update": "Time at which the segment's contact sample is next updated.",+ "parent_list_ids": "IDs of the lists the segment filters contacts from.",+ "query_version": "Version of the segment query language.",+ "status": "Validation status of the segment query.",+ },+ },"singlesends": {"description": "Single Sends: one-time Marketing Campaigns emails sent to lists or segments.","docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/single-sends/get-all-single-sends",@@ -192,6 +254,49 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"clicks": "Number of clicks on this link.",},},+ "subusers": {+ "description": "Subusers of a parent account.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/subusers-api/list-all-subusers",+ "columns": {+ "id": "Unique identifier for the subuser.",+ "username": "Username of the subuser.",+ "email": "Contact email address of the subuser.",+ "disabled": "Whether the subuser is disabled.",+ "region": "Region the subuser is pinned to: global or eu.",+ },+ },+ "subuser_stats": {+ "description": "Daily email statistics for each subuser, such as requests, deliveries, opens, clicks, and bounces.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/subuser-statistics/retrieve-email-statistics-for-your-subuser",+ "columns": {+ "date": "The date the statistics were gathered, in YYYY-MM-DD format.",+ "username": "Username of the subuser the statistics are for.",+ "requests": "Number of emails that were requested to be delivered.",+ "processed": "Number of requests that SendGrid processed.",+ "delivered": "Number of emails that SendGrid confirmed were delivered to a recipient.",+ "deferred": "Number of emails that temporarily could not be delivered.",+ "opens": "Total number of times the emails were opened by recipients.",+ "unique_opens": "Number of unique recipients who opened the emails.",+ "clicks": "Number of links that were clicked.",+ "unique_clicks": "Number of unique recipients who clicked links in the emails.",+ "bounces": "Number of emails that bounced instead of being delivered.",+ "bounce_drops": "Number of emails that were dropped because of a bounce.",+ "blocks": "Number of emails that were not allowed to be delivered by ISPs.",+ "invalid_emails": "Number of recipients with malformed addresses or addresses that the mail provider reported as invalid.",+ "spam_reports": "Number of recipients who marked the email as spam.",+ "spam_report_drops": "Number of emails that were dropped because the recipient previously marked emails as spam.",+ "unsubscribes": "Number of recipients who unsubscribed.",+ "unsubscribe_drops": "Number of emails that were dropped because the recipient unsubscribed.",+ },+ },+ "subuser_reputations": {+ "description": "Current sender reputation of each subuser.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/subusers-api/retrieve-subuser-reputations",+ "columns": {+ "username": "Username of the subuser.",+ "reputation": "Sender reputation of the subuser, from 0 to 100.",+ },+ },"templates": {"description": "Email templates (legacy and dynamic) used to compose messages.","docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/transactional-templates/retrieve-paged-transactional-templates",diff --git a/products/warehouse_sources/sources/sendgrid/sendgrid.py b/products/warehouse_sources/sources/sendgrid/sendgrid.pyindex 73300b8e0ba8..7d1168e78105 100644--- a/products/warehouse_sources/sources/sendgrid/sendgrid.py+++ b/products/warehouse_sources/sources/sendgrid/sendgrid.py@@ -111,15 +111,28 @@ def _flatten_daily_stats(item: dict[str, Any]) -> list[dict[str, Any]]:return rows-def _flatten_category_stats(item: dict[str, Any]) -> list[dict[str, Any]]:- """Flatten one `{date, stats: [{name, metrics: {...}}]}` bucket of a single-category request- into a row per metrics entry, keyed by the requested category from the parent row."""+def _flatten_breakdown_stats(item: dict[str, Any]) -> list[dict[str, Any]]:+ """Flatten one `{date, stats: [{type, name, metrics: {...}}]}` bucket into a row per breakdown+ value (country, device, browser, ...), with the bucket `date` merged in."""return [- {"date": item.get("date"), "category": item.get("category"), **(entry.get("metrics") or {})}+ {"date": item.get("date"), "type": entry.get("type"), "name": entry.get("name"), **(entry.get("metrics") or {})}for entry in item.get("stats") or []]+def _parent_stats_flattener(parent_field: str) -> Callable[[dict[str, Any]], list[dict[str, Any]]]:+ """Flatten one `{date, stats: [{name, metrics: {...}}]}` bucket of a single-parent request+ into a row per metrics entry, keyed by the requested parent value (category, subuser)."""++ def _flatten(item: dict[str, Any]) -> list[dict[str, Any]]:+ return [+ {"date": item.get("date"), parent_field: item.get(parent_field), **(entry.get("metrics") or {})}+ for entry in item.get("stats") or []+ ]++ return _flatten++_ACTIVITY_TIMESTAMP_FORMAT = "%Y-%m-%dT%H:%M:%SZ"@@ -315,7 +328,7 @@ def _build_params(# Metadata endpoints carry the page size as a query param; offset endpoints get limit/offset# from the paginator, single endpoints take no pagination params.- if config.pagination == "metadata":+ if config.pagination == "metadata" and config.send_page_size:params["page_size"] = config.page_sizeif config.incremental_param:@@ -433,7 +446,7 @@ def _fanout_resource(# The cursor is already bound into `child_params` as the endpoint's start param.incremental_config_factory=lambda _field: None,# Offset parents get `limit` from their paginator, single-page children take no page size.- page_size_param="page_size" if config.pagination == "metadata" else None,+ page_size_param="page_size" if config.pagination == "metadata" and config.send_page_size else None,child_params_extra=child_params,parent_data_map=(_encoded_parent_field(config.encode_parent_field, config.fanout.resolve_field)@@ -453,9 +466,10 @@ def _fanout_resource("data_selector_required": True,},)- if config.response_shape == "category_stats":+ if config.response_shape == "parent_stats":assert isinstance(resource, Resource)- resource.add_map(_flatten_category_stats)+ assert config.parent_stats_field is not None+ resource.add_map(_parent_stats_flattener(config.parent_stats_field))return resource@@ -519,6 +533,8 @@ def sendgrid_source(if config.response_shape == "daily_stats":# stats rows are nested one level deeper than the bare array; flatten them into flat daily rows.resource_config["data_map"] = _flatten_daily_stats+ elif config.response_shape == "breakdown_stats":+ resource_config["data_map"] = _flatten_breakdown_stats# Activity pagination owns its own request params (`query` window + `limit`), including the# server-side incremental filter, so it is built with the cursor instead of via `params`.@@ -586,7 +602,7 @@ def _probe_params(config: SendGridEndpointConfig) -> dict[str, Any]:params: dict[str, Any] = dict(config.extra_params)if config.pagination == "offset":params["limit"] = 1- elif config.pagination == "metadata":+ elif config.pagination == "metadata" and config.send_page_size:params["page_size"] = 1elif config.pagination == "activity":# `query` is required on /messages, so probe a one-day window for one row. The 403 for adiff --git a/products/warehouse_sources/sources/sendgrid/settings.py b/products/warehouse_sources/sources/sendgrid/settings.pyindex 752d6c3a3680..c7c9011c7020 100644--- a/products/warehouse_sources/sources/sendgrid/settings.py+++ b/products/warehouse_sources/sources/sendgrid/settings.py@@ -24,9 +24,11 @@ IncrementalParamFormat = Literal["epoch", "date"]# - "array": the rows are the JSON body itself, or the array under `data_key`.# - "daily_stats": nested `[{date, stats: [{metrics: {...}}]}]` that flattens to one row per# metrics bucket with `date` merged in (the /stats family).-# - "category_stats": the same nesting, flattened to one row per date with the requested-# `category` merged in (category_stats).-ResponseShape = Literal["array", "daily_stats", "category_stats"]+# - "breakdown_stats": the same nesting, flattened to one row per `stats` entry with the+# entry's `type` and `name` kept (the advanced stats breakdowns).+# - "parent_stats": the same nesting, flattened to one row per date with the fan-out parent+# value named by `parent_stats_field` merged in (category_stats, subuser_stats).+ResponseShape = Literal["array", "daily_stats", "breakdown_stats", "parent_stats"]# /stats requires `start_date` on every request, so the first sync (no cursor yet) backfills this# far back. Daily aggregation means one row per day, so a year is cheap and gives enough history to@@ -46,6 +48,8 @@ class SendGridEndpointConfig:# Stable creation field used for datetime partitioning. Never use a "modified" field here.partition_key: Optional[str] = Nonepage_size: int = 500+ # False for a metadata endpoint that documents no `page_size` param and returns every row at once.+ send_page_size: bool = Trueincremental_fields: list[IncrementalField] = field(default_factory=list)# Server-side query param that filters by the incremental field.incremental_param: Optional[str] = None@@ -57,6 +61,8 @@ class SendGridEndpointConfig:default_backfill_days: Optional[int] = None# How the response body is shaped, so the transport can flatten nested stats responses.response_shape: ResponseShape = "array"+ # Parent row field merged into each flattened row of a "parent_stats" response.+ parent_stats_field: Optional[str] = None# Order rows arrive in across the whole sync. The Email Activity walk pages newest to oldest,# so it declares "desc" and the pipeline defers the incremental watermark to the end of a# successful run instead of checkpointing it per batch. None when rows come one parent at a@@ -94,6 +100,8 @@ _MARKETING_CAMPAIGNS_NOTE = ("Accounts without Marketing Campaigns, or on legacy Marketing Campaigns, cannot sync this table.")+_SUBUSERS_NOTE = "Only a parent account with Subusers can sync this table."+def _epoch_field(name: str) -> IncrementalField:return {@@ -122,9 +130,37 @@ def _datetime_field(name: str) -> IncrementalField:}+def _breakdown_stats_endpoint(+ name: str,+ path: str,+ required_scope: str,+ pagination: PaginationMode = "offset",+ permission_note: Optional[str] = None,+) -> SendGridEndpointConfig:+ return SendGridEndpointConfig(+ name=name,+ path=path,+ # One row per breakdown value per day. `type` stays in the key because geo rows can be a+ # country or a state.+ primary_keys=["date", "type", "name"],+ pagination=pagination,+ partition_key="date",+ incremental_fields=[_date_field("date")],+ incremental_param="start_date",+ incremental_param_format="date",+ default_backfill_days=STATS_DEFAULT_BACKFILL_DAYS,+ response_shape="breakdown_stats",+ # The API does not document the order of the date buckets.+ sort_mode=None,+ extra_params={"aggregated_by": "day"},+ required_scope=required_scope,+ permission_note=permission_note,+ )++# SendGrid (v3) endpoint catalog. The suppression endpoints expose a genuine server-side# `start_time` filter (Unix epoch seconds) over their immutable `created` field, so they sync-# incrementally. `stats` and `category_stats` filter on `start_date` (YYYY-MM-DD) over their daily+# incrementally. The stats tables filter on `start_date` (YYYY-MM-DD) over their daily# aggregate date and also sync incrementally. Marketing/asm metadata endpoints offer no timestamp filter and ship as# full refresh. See api_inventory.md for the per-endpoint research notes.SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {@@ -195,6 +231,22 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {extra_params={"aggregated_by": "day"},required_scope="stats.read",),+ # The advanced stats breakdowns. They return 500 items per request by default, and all but+ # /clients/stats document `limit`/`offset`, so those page with an explicit limit.+ "geo_stats": _breakdown_stats_endpoint(+ "geo_stats",+ "/geo/stats",+ "geo.stats.read",+ permission_note="Regional (EU) subusers cannot sync this table.",+ ),+ "mailbox_provider_stats": _breakdown_stats_endpoint(+ "mailbox_provider_stats", "/mailbox_providers/stats", "mailbox_providers.stats.read"+ ),+ "device_stats": _breakdown_stats_endpoint("device_stats", "/devices/stats", "devices.stats.read"),+ "client_stats": _breakdown_stats_endpoint(+ "client_stats", "/clients/stats", "clients.stats.read", pagination="single"+ ),+ "browser_stats": _breakdown_stats_endpoint("browser_stats", "/browsers/stats", "browsers.stats.read"),"categories": SendGridEndpointConfig(name="categories",path="/categories",@@ -218,7 +270,8 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {incremental_param="start_date",incremental_param_format="date",default_backfill_days=STATS_DEFAULT_BACKFILL_DAYS,- response_shape="category_stats",+ response_shape="parent_stats",+ parent_stats_field="category",sort_mode=None,extra_params={"aggregated_by": "day"},required_scope="categories.stats.read",@@ -263,6 +316,18 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {required_scope="marketing.read",permission_note=_MARKETING_CAMPAIGNS_NOTE,),+ # Segments v2 returns every segment in one `results` array and documents no page params.+ "segments": SendGridEndpointConfig(+ name="segments",+ path="/marketing/segments/2.0",+ primary_keys=["id"],+ pagination="metadata",+ data_key="results",+ send_page_size=False,+ partition_key="created_at",+ required_scope="marketing.read",+ permission_note=_MARKETING_CAMPAIGNS_NOTE,+ ),"singlesends": SendGridEndpointConfig(name="singlesends",path="/marketing/singlesends",@@ -338,6 +403,67 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {child_response_actions=[{"status_code": 404, "action": "ignore"}],),),+ "subusers": SendGridEndpointConfig(+ name="subusers",+ path="/subusers",+ primary_keys=["id"],+ pagination="offset",+ # No maximum is documented, and a silently capped page would read as the short last page.+ page_size=100,+ extra_params={"include_region": "true"},+ required_scope="subusers.read",+ permission_note=_SUBUSERS_NOTE,+ ),+ # /subusers/stats requires the `subusers` param, so it fans out over the subuser listing one+ # subuser per request.+ "subuser_stats": SendGridEndpointConfig(+ name="subuser_stats",+ path="/subusers/stats?subusers={username}",+ primary_keys=["username", "date"],+ pagination="offset",+ partition_key="date",+ incremental_fields=[_date_field("date")],+ incremental_param="start_date",+ incremental_param_format="date",+ default_backfill_days=STATS_DEFAULT_BACKFILL_DAYS,+ response_shape="parent_stats",+ parent_stats_field="username",+ sort_mode=None,+ extra_params={"aggregated_by": "day"},+ required_scope="subusers.stats.read",+ permission_note=_SUBUSERS_NOTE,+ default_incremental_field="date",+ fanout=DependentEndpointConfig(+ parent_name="subusers",+ resolve_param="username",+ resolve_field="username_param",+ include_from_parent=["username"],+ parent_field_renames={"username": "username"},+ child_response_actions=[{"status_code": 404, "action": "ignore"}],+ ),+ probe_resolve_value="posthog",+ encode_parent_field="username",+ ),+ # The spec does not say what `usernames` defaults to, so each subuser is requested by name.+ "subuser_reputations": SendGridEndpointConfig(+ name="subuser_reputations",+ path="/subusers/reputations?usernames={username}",+ primary_keys=["username"],+ pagination="single",+ sort_mode=None,+ required_scope="subusers.reputations.read",+ permission_note=_SUBUSERS_NOTE,+ fanout=DependentEndpointConfig(+ parent_name="subusers",+ resolve_param="username",+ resolve_field="username_param",+ include_from_parent=["username"],+ parent_field_renames={"username": "username"},+ child_response_actions=[{"status_code": 404, "action": "ignore"}],+ ),+ probe_resolve_value="posthog",+ encode_parent_field="username",+ ),"templates": SendGridEndpointConfig(name="templates",path="/templates",diff --git a/products/warehouse_sources/sources/sendgrid/source.py b/products/warehouse_sources/sources/sendgrid/source.pyindex 611592a4eedf..bd6d60d96c95 100644--- a/products/warehouse_sources/sources/sendgrid/source.py+++ b/products/warehouse_sources/sources/sendgrid/source.py@@ -198,9 +198,10 @@ You can create an API key in your [SendGrid account settings](https://app.sendgrGrant the following read access (Restricted Access) so the key can reach the data you want to sync:- **Suppressions**: bounces, blocks, invalid emails, spam reports, global unsubscribes, unsubscribe groups and their members-- **Stats**: global email statistics (daily requests, delivered, opens, clicks, bounces)+- **Stats**: global email statistics (daily requests, delivered, opens, clicks, bounces), and daily breakdowns by country, mailbox provider, device, email client and browser- **Category Management**: categories and their daily statistics-- **Marketing**: marketing lists, single sends, single send stats, and automation stats. This also needs an account with Marketing Campaigns enabled.+- **Marketing**: marketing lists, segments, single sends, single send stats, and automation stats. This also needs an account with Marketing Campaigns enabled.+- **Subusers**: subusers, their daily statistics and their sender reputations. Only a parent account with Subusers has this data.- **Template Engine**: templates- **Email Activity**: message activity. This table is off by default and also needs SendGrid's paid additional email activity history add-on.diff --git a/products/warehouse_sources/sources/sendgrid/tests/test_sendgrid.py b/products/warehouse_sources/sources/sendgrid/tests/test_sendgrid.pyindex 4f0fdc0ae6cc..1eee6fdb0ae6 100644--- a/products/warehouse_sources/sources/sendgrid/tests/test_sendgrid.py+++ b/products/warehouse_sources/sources/sendgrid/tests/test_sendgrid.py@@ -92,6 +92,50 @@ class TestStats:assert result.requests[0].param("start_date") == "2024-03-10"+class TestBreakdownStats:+ @pytest.mark.parametrize(+ ("endpoint", "path", "expected_limit"),+ [+ ("geo_stats", "/v3/geo/stats", "500"),+ # /clients/stats documents no `limit`/`offset`, so it gets none.+ ("client_stats", "/v3/clients/stats", None),+ ],+ )+ def test_flattens_one_row_per_breakdown_value_per_day(+ self, endpoint: str, path: str, expected_limit: str | None+ ) -> None:+ body = [+ {+ "date": "2026-08-01",+ "stats": [+ {"type": "country", "name": "US", "metrics": {"opens": 3, "unique_opens": 2}},+ {"type": "country", "name": "CA", "metrics": {"opens": 1, "unique_opens": 1}},+ ],+ },+ ]++ result = _run(+ endpoint,+ [_response(body)],+ should_use_incremental_field=True,+ db_incremental_field_last_value="2026-08-01",+ incremental_field="date",+ )++ assert result.raised is None+ # Without `name` every breakdown value of a day collides on the primary key.+ assert result.rows == [+ {"date": "2026-08-01", "type": "country", "name": "US", "opens": 3, "unique_opens": 2},+ {"date": "2026-08-01", "type": "country", "name": "CA", "opens": 1, "unique_opens": 1},+ ]+ assert len(result.requests) == 1+ request = result.requests[0]+ assert request.path == path+ assert request.param("limit") == expected_limit+ assert request.param("start_date") == "2026-08-01"+ assert request.param("aggregated_by") == "day"++class TestOffsetPagination:def test_paginates_until_short_page_and_saves_state(self) -> None:page1 = [{"email": f"a{i}@x.com", "created": i} for i in range(500)]@@ -141,6 +185,13 @@ class TestMetadataPagination:assert result.urls[1] == next_urlassert result.saved_states == [SendGridResumeConfig(next_url=next_url)]+ def test_endpoint_without_a_page_size_param_gets_none(self) -> None:+ result = _run("segments", [_response({"results": [{"id": "seg1"}], "_metadata": {}})])++ assert result.raised is None+ assert result.rows == [{"id": "seg1"}]+ assert result.requests[0].param("page_size") is None+def test_missing_result_key_raises_loudly(self) -> None:result = _run("marketing_lists", [_response({"_metadata": {}})])@@ -365,6 +416,65 @@ class TestCategoryStatsFanOut:assert [r.param("offset") for r in result.requests if r.path == "/v3/categories"] == ["0", "50"]+def _subuser_stats_bucket(day: str, username: str, delivered: int) -> dict[str, Any]:+ return {"date": day, "stats": [{"type": "subuser", "name": username, "metrics": {"delivered": delivered}}]}+++class TestSubuserFanOut:+ @pytest.mark.parametrize(+ ("endpoint", "child_path", "child_param", "child_bodies", "expected_rows"),+ [+ (+ "subuser_stats",+ "/v3/subusers/stats",+ "subusers",+ [[_subuser_stats_bucket("2026-08-01", "team+a", 3)], [_subuser_stats_bucket("2026-08-01", "b", 1)]],+ [+ {"date": "2026-08-01", "username": "team+a", "delivered": 3},+ {"date": "2026-08-01", "username": "b", "delivered": 1},+ ],+ ),+ (+ "subuser_reputations",+ "/v3/subusers/reputations",+ "usernames",+ [[{"username": "team+a", "reputation": 99}], [{"username": "b", "reputation": 95.2}]],+ [{"username": "team+a", "reputation": 99}, {"username": "b", "reputation": 95.2}],+ ),+ ],+ )+ def test_requests_each_subuser_and_keys_rows_by_it(+ self,+ endpoint: str,+ child_path: str,+ child_param: str,+ child_bodies: list[Any],+ expected_rows: list[dict[str, Any]],+ ) -> None:+ subusers = [{"id": 1, "username": "team+a"}, {"id": 2, "username": "gone"}, {"id": 3, "username": "b"}]++ result = _run(+ endpoint,+ route(+ {+ "/v3/subusers": [_response(subusers)],+ child_path: [+ _response(child_bodies[0]),+ # A subuser deleted after the listing must not fail the table.+ ScriptedResponse(status=404, json={"errors": [{"message": "not found"}]}),+ _response(child_bodies[1]),+ ],+ }+ ),+ )++ assert result.raised is None+ assert result.rows == expected_rows+ child_requests = [r for r in result.requests if r.path == child_path]+ # A `+` in a username must reach the API whole, not decoded to a space.+ assert [r.param(child_param) for r in child_requests] == ["team+a", "gone", "b"]++def _messages_page(count: int, last_event_time: str) -> list[dict[str, Any]]:return [{"msg_id": f"m{last_event_time}-{i}", "last_event_time": last_event_time} for i in range(count)]@@ -525,6 +635,7 @@ class TestGetEndpointPermissions:# The category listing reads with `categories.read`, so it can't stand in for the# `categories.stats.read` this child needs.("category_stats", "/v3/categories/stats"),+ ("subuser_stats", "/v3/subusers/stats"),],)def test_fanout_endpoint_probe_path(self, endpoint: str, expected_path: str) -> None:@@ -553,6 +664,8 @@ class TestGetEndpointPermissions:("marketing_lists", {"page_size": "1"}),("templates", {"page_size": "1", "generations": "legacy,dynamic"}),("unsubscribe_groups", {}),+ # Segments v2 documents no `page_size` param.+ ("segments", {}),# /categories/stats requires `categories` and `start_date`.("category_stats", {"categories": "posthog", "start_date": "2026-08-07", "aggregated_by": "day"}),# /messages requires `query`, so a probe without one risks the same masking 400.