:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 4 missing tables to the SendGrid source (#125316)
Tom Owersb560a9280e39
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex f2af31fac015..22db70f4ab77 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -10774,9 +10774,9 @@ Diffed against: <https://github.com/twilio/sendgrid-oai/tree/main/spec/json (off- [x] `GET /v3/messages (Email Activity)` — Per-message event feed (delivered, opened, clicked, bounced) — the row-level fact table behind all aggregate stats (high)- [ ] `GET /v3/marketing/contacts (+ /contacts/exports)` — Contact records; no recipient dimension exists today so lists and suppressions cannot be joined to people (high)- [x] `GET /v3/asm/groups/{group_id}/suppressions` — Membership rows linking email addresses to the unsubscribe_groups already synced — the join table is missing (high)-- [ ] `GET /v3/categories/stats (and /v3/categories/stats/sums)` — Email stats broken down by category, the main way transactional senders segment performance (medium)-- [ ] `GET /v3/categories` — Category lookup resolving the category tags used on stats and messages (medium)-- [ ] `GET /v3/marketing/stats/automations (+ /{id}/links)` — Automation (journey) performance stats (medium)+- [x] `GET /v3/categories/stats (and /v3/categories/stats/sums)` — Email stats broken down by category, the main way transactional senders segment performance (medium)+- [x] `GET /v3/categories` — Category lookup resolving the category tags used on stats and messages (medium)+- [x] `GET /v3/marketing/stats/automations (+ /{id}/links)` — Automation (journey) performance stats (medium)- [ ] `GET /v3/marketing/segments/2.0` — Marketing segment definitions; marketing_lists is synced but segments are not (medium)- [ ] `GET /v3/geo/stats, /v3/mailbox_providers/stats, /v3/devices/stats, /v3/clients/stats, /v3/browsers/stats` — Breakdown dimensions on the same stats fact (country, mailbox provider, device, client, browser) (medium)- [ ] `GET /v3/subusers and /v3/subusers/stats (+ /stats/monthly, /stats/sums, /reputations)` — Per-subuser volume and reputation — required for any agency or multi-tenant SendGrid account (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/api_inventory.md b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/api_inventory.mdindex 7ef1b0ea7b87..248371d67c52 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/api_inventory.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/api_inventory.md@@ -32,6 +32,10 @@ on the primary key — wasted API calls, not corrupted data.| `singlesends` | `/marketing/singlesends` | `_metadata` | `{"result": [...]}` | `id` | full refresh || `singlesend_stats` | `/marketing/stats/singlesends` | `_metadata` | `{"results": [...]}` | `id`, `ab_variation`, `ab_phase` | full refresh || `singlesend_link_stats` | `/marketing/stats/singlesends/{id}/links` (fan-out over `singlesends`) | `_metadata` | `{"results": [...]}` | `singlesend_id`, `url`, `url_location`, `ab_variation`, `ab_phase` | full refresh |+| `categories` | `/categories` | limit/offset | bare array | `category` | full refresh |+| `category_stats` | `/categories/stats` (fan-out over `categories`) | none (single) | nested daily stats | `category`, `date` | `date` via `start_date` (YYYY-MM-DD) |+| `automation_stats` | `/marketing/stats/automations` | `_metadata` | `{"results": [...]}` | `id`, `aggregation`, `step_id` | full refresh |+| `automation_link_stats` | `/marketing/stats/automations/{id}/links` (fan-out) | `_metadata` | `{"results": [...]}` | `automation_id`, `url`, `url_location`, `step_id` | full refresh |Notes:@@ -78,6 +82,25 @@ Notes:over `/asm/groups` and still lack the creation time. Neither takes a time filter.- The Single Send endpoints (`singlesends`, `singlesend_stats`, `singlesend_link_stats`) follow`_metadata.next` like the other marketing endpoints. The stats endpoints cap `page_size` at 50.+- `categories` reads `/categories`, a bare array of `{category}` paged by `limit`/`offset`. No maximum+ `limit` is documented, so it pages at the documented default of 50: a silently capped page would read+ as the short last page and end the walk early.+- `/categories/stats` requires `categories` (up to 10 names) and `start_date`, so `category_stats` fans+ out over `categories` with one category per request and `aggregated_by=day`. Each response is the+ same `{date, stats: [{name, type, metrics}]}` nesting as `/stats`, flattened to one row per category+ and day. Category names are free text, so the name is percent-encoded before it goes into the query+ string. `start_date` filters server-side, so the table syncs incrementally on `date`. Rows come one+ category at a time, so they declare no sort order. The listing needs `categories.read` and the stats+ need `categories.stats.read`, so the permission probe calls `/categories/stats` itself with a+ placeholder category. `/categories/stats/sums` is not synced: it sums the same metrics over a date+ range, which a query over `category_stats` can do.+- The Automation endpoints (`automation_stats`, `automation_link_stats`) follow `_metadata.next` and+ cap `page_size` at 50, like the Single Send stats. The API has no Automation listing, so the link+ fan-out reads the Automation ids from the stats rows (one lifetime-total row per Automation without+ `group_by`).+- `/marketing/contacts` is not synced. It returns a sample of up to 50 recent contacts, and its pagination+ is deprecated. The full set needs the asynchronous export job (`POST /marketing/contacts/exports`, poll+ the status, then download the files), and the public spec does not document the file layout.`singlesend_stats` sends no `singlesend_ids`, so it returns lifetime totals for every Single Send;without `group_by` each row has `ab_variation` and `ab_phase` set to `all`. `singlesend_link_stats`fans out over `singlesends`, and a 404 for a Single Send deleted mid-sync is skipped. No Single Senddiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/canonical_descriptions.pyindex 6640d3147c8b..23365dbefe9c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/canonical_descriptions.py@@ -74,6 +74,37 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"invalid_emails": "Number of emails dropped because the recipient address was invalid.",},},+ "categories": {+ "description": "Categories: the tags senders attach to emails to group them by topic, used to segment statistics.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/categories/retrieve-all-categories",+ "columns": {+ "category": "Name of the category used to group emails by broad topic.",+ },+ },+ "category_stats": {+ "description": "Daily email statistics for each category, such as requests, deliveries, opens, clicks, and bounces.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/categories/retrieve-email-statistics-for-categories",+ "columns": {+ "date": "The date the statistics were gathered, in YYYY-MM-DD format.",+ "category": "Name of the category that the statistics are for.",+ "requests": "Number of emails that were requested to be delivered.",+ "processed": "Number of requests that SendGrid processed.",+ "delivered": "Number of emails that SendGrid confirmed were delivered to a recipient.",+ "deferred": "Number of emails that temporarily could not be delivered.",+ "opens": "Total number of times the emails were opened by recipients.",+ "unique_opens": "Number of unique recipients who opened the emails.",+ "clicks": "Number of links that were clicked.",+ "unique_clicks": "Number of unique recipients who clicked links in the emails.",+ "bounces": "Number of emails that bounced instead of being delivered.",+ "bounce_drops": "Number of emails that were dropped because of a bounce.",+ "blocks": "Number of emails that were not allowed to be delivered by ISPs.",+ "invalid_emails": "Number of recipients with malformed addresses or addresses that the mail provider reported as invalid.",+ "spam_reports": "Number of recipients who marked the email as spam.",+ "spam_report_drops": "Number of emails that were dropped because the recipient previously marked emails as spam.",+ "unsubscribes": "Number of recipients who unsubscribed.",+ "unsubscribe_drops": "Number of emails that were dropped because the recipient unsubscribed.",+ },+ },"unsubscribe_groups": {"description": "Suppression (unsubscribe) groups used to let recipients opt out of specific categories of mail.","docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/suppressions-unsubscribe-groups/retrieve-all-suppression-groups-associated-with-the-user",@@ -142,6 +173,27 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"clicks": "Number of clicks on this link.",},},+ "automation_stats": {+ "description": "Lifetime engagement statistics for each Marketing Campaigns Automation, such as requests, deliveries, opens, and clicks.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/marketing-campaign-stats/get-all-automation-stats",+ "columns": {+ "id": "Unique identifier for the Automation that the statistics are for.",+ "aggregation": "The time unit that the statistics are rolled up to, total for lifetime totals.",+ "step_id": "The Automation step that the statistics are for, or all when they cover every step.",+ "stats": "The metrics: requests, delivered, opens, unique_opens, clicks, unique_clicks, bounces, bounce_drops, invalid_emails, spam_reports, spam_report_drops, and unsubscribes.",+ },+ },+ "automation_link_stats": {+ "description": "Click counts for each link in each Marketing Campaigns Automation.",+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/marketing-campaign-stats/get-automation-click-tracking-stats-by-id",+ "columns": {+ "automation_id": "Unique identifier for the Automation that contains the link.",+ "url": "The URL of the clicked link, including any custom fields in the URL.",+ "url_location": "The position of the link in the Automation step, numbered from the top down starting at 0.",+ "step_id": "The Automation step that contains the link.",+ "clicks": "Number of clicks on this link.",+ },+ },"templates": {"description": "Email templates (legacy and dynamic) used to compose messages.","docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/transactional-templates/retrieve-paged-transactional-templates",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/sendgrid.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/sendgrid.pyindex ebe74ddadd7d..3398efedde84 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/sendgrid.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/sendgrid.py@@ -1,8 +1,9 @@import loggingimport dataclasses+from collections.abc import Callablefrom datetime import UTC, date, datetime, timedeltafrom typing import Any, Optional-from urllib.parse import parse_qs, urlencode, urlparse+from urllib.parse import parse_qs, quote, urlencode, urlparsefrom requests import Request, Response@@ -20,6 +21,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resOffsetPaginator,SinglePagePaginator,)+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.resource import Resourcefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (ClientConfig,EndpointResource,@@ -115,6 +117,15 @@ def _flatten_daily_stats(item: dict[str, Any]) -> list[dict[str, Any]]:return rows+def _flatten_category_stats(item: dict[str, Any]) -> list[dict[str, Any]]:+ """Flatten one `{date, stats: [{name, metrics: {...}}]}` bucket of a single-category request+ into a row per metrics entry, keyed by the requested category from the parent row."""+ return [+ {"date": item.get("date"), "category": item.get("category"), **(entry.get("metrics") or {})}+ for entry in item.get("stats") or []+ ]++_ACTIVITY_TIMESTAMP_FORMAT = "%Y-%m-%dT%H:%M:%SZ"@@ -379,12 +390,23 @@ def _client_config(api_key: str, paginator: BasePaginator) -> ClientConfig:}+def _encoded_parent_field(source_field: str, target_field: str) -> Callable[[dict[str, Any]], dict[str, Any]]:+ def _map(row: dict[str, Any]) -> dict[str, Any]:+ row[target_field] = quote(str(row[source_field]), safe="")+ return row++ return _map++def _fanout_resource(api_key: str,config: SendGridEndpointConfig,team_id: int,job_id: str,resumable_source_manager: ResumableSourceManager[SendGridResumeConfig],+ should_use_incremental_field: bool,+ db_incremental_field_last_value: Any,+ incremental_field: Optional[str],) -> Any:assert config.fanout is not Noneparent_config = SENDGRID_ENDPOINTS[config.fanout.parent_name]@@ -399,29 +421,48 @@ def _fanout_resource(if state is not None:resumable_source_manager.save_state(SendGridResumeConfig(fanout_state=state))- return build_dependent_resource(+ child_params = _build_params(+ config, should_use_incremental_field, db_incremental_field_last_value, incremental_field+ )++ resource = build_dependent_resource(endpoint_configs=SENDGRID_ENDPOINTS,child_endpoint=config.name,fanout=config.fanout,- client_config=_client_config(api_key, SendGridMetadataPaginator()),+ client_config=_client_config(api_key, _build_paginator(config)),path_format_values={},team_id=team_id,job_id=job_id,db_incremental_field_last_value=None,- page_size_param="page_size",+ should_use_incremental_field=should_use_incremental_field,+ incremental_field=incremental_field,+ # The cursor is already bound into `child_params` as the endpoint's start param.+ incremental_config_factory=lambda _field: None,+ # Offset parents get `limit` from their paginator, single-page children take no page size.+ page_size_param="page_size" if config.pagination == "metadata" else None,+ child_params_extra=child_params,+ parent_data_map=(+ _encoded_parent_field(config.encode_parent_field, config.fanout.resolve_field)+ if config.encode_parent_field+ else None+ ),resume_hook=save_checkpoint,initial_paginator_state=initial_paginator_state,parent_endpoint_extra={- "paginator": SendGridMetadataPaginator(),+ "paginator": _build_paginator(parent_config),"data_selector": parent_config.data_key,"data_selector_required": True,},child_endpoint_extra={- "paginator": SendGridMetadataPaginator(),+ "paginator": _build_paginator(config),"data_selector": config.data_key,"data_selector_required": True,},)+ if config.response_shape == "category_stats":+ assert isinstance(resource, Resource)+ resource.add_map(_flatten_category_stats)+ return resourcedef _source_response(endpoint: str, config: SendGridEndpointConfig, resource: Any) -> SourceResponse:@@ -452,7 +493,18 @@ def sendgrid_source(if config.fanout is not None:return _source_response(- endpoint, config, _fanout_resource(api_key, config, team_id, job_id, resumable_source_manager)+ endpoint,+ config,+ _fanout_resource(+ api_key,+ config,+ team_id,+ job_id,+ resumable_source_manager,+ should_use_incremental_field,+ db_incremental_field_last_value,+ incremental_field,+ ),)params = _build_params(config, should_use_incremental_field, db_incremental_field_last_value, incremental_field)@@ -525,9 +577,10 @@ def get_status_code(api_key: str, path: str, params: Optional[dict[str, Any]] ="""Probe an endpoint to classify the credentials. Returns the HTTP status, or None on atransport error."""query = urlencode(params if params is not None else {"limit": 1})+ separator = "&" if "?" in path else "?"_ok, status = validate_via_probe(lambda: make_tracked_session(redact_values=(api_key,)),- f"{SENDGRID_BASE_URL}{path}?{query}",+ f"{SENDGRID_BASE_URL}{path}{separator}{query}",headers=_get_headers(api_key),)return status@@ -547,12 +600,18 @@ def _probe_params(config: SendGridEndpointConfig) -> dict[str, Any]:now = datetime.now(UTC)params["limit"] = 1params["query"] = _activity_query(_to_activity_timestamp(now - timedelta(days=1)), _to_activity_timestamp(now))+ if config.incremental_param and config.default_backfill_days is not None:+ # The start param is required, so a probe without it risks a 400 that hides a denial.+ params[config.incremental_param] = _format_cursor(datetime.now(UTC), config.incremental_param_format)return paramsdef get_endpoint_status_code(api_key: str, config: SendGridEndpointConfig) -> Optional[int]:- # A fan-out child path needs a parent id, and its parent reads with the same scope, so the- # parent listing stands in for it.+ if config.fanout is not None and config.probe_resolve_value is not None:+ path = config.path.replace(f"{{{config.fanout.resolve_param}}}", config.probe_resolve_value)+ return get_status_code(api_key, path, _probe_params(config))+ # Any other fan-out child path needs a parent id, and its parent reads with the same scope, so+ # the parent listing stands in for it.probe_config = configif config.fanout is not None:probe_config = SENDGRID_ENDPOINTS[config.fanout.parent_name]diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/settings.pyindex 01394716ccc8..a415e8488db8 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/settings.py@@ -27,7 +27,9 @@ IncrementalParamFormat = Literal["epoch", "date"]# - "array": the rows are the JSON body itself, or the array under `data_key`.# - "daily_stats": nested `[{date, stats: [{metrics: {...}}]}]` that flattens to one row per# metrics bucket with `date` merged in (the /stats family).-ResponseShape = Literal["array", "daily_stats"]+# - "category_stats": the same nesting, flattened to one row per date with the requested+# `category` merged in (category_stats).+ResponseShape = Literal["array", "daily_stats", "category_stats"]# /stats requires `start_date` on every request, so the first sync (no cursor yet) backfills this# far back. Daily aggregation means one row per day, so a year is cheap and gives enough history to@@ -60,8 +62,9 @@ class SendGridEndpointConfig:response_shape: ResponseShape = "array"# Order rows arrive in across the whole sync. The Email Activity walk pages newest to oldest,# so it declares "desc" and the pipeline defers the incremental watermark to the end of a- # successful run instead of checkpointing it per batch.- sort_mode: Literal["asc", "desc"] = "asc"+ # successful run instead of checkpointing it per batch. None when rows come one parent at a+ # time, so they form no single ordered series.+ sort_mode: Optional[Literal["asc", "desc"]] = "asc"# False for tables whose sync needs grants beyond what source creation validated: the schema# picker and one-shot source creation leave them unselected until the user opts in.should_sync_default: bool = True@@ -73,11 +76,16 @@ class SendGridEndpointConfig:required_scope: str = ""# Appended to the missing-scope message where granting the scope may not be enough on its own.permission_note: Optional[str] = None- # Structural requirement of `FanoutEndpointLike` (see common/rest_source/fanout.py); no- # fan-out endpoint here syncs incrementally.+ # Structural requirement of `FanoutEndpointLike` (see common/rest_source/fanout.py).default_incremental_field: Optional[str] = None# Set on a child endpoint whose path takes one id per row of a parent endpoint.fanout: Optional[DependentEndpointConfig] = None+ # Placeholder parent value to probe the child path with, for a fan-out child that reads with a+ # different scope than its parent. None probes the parent listing instead.+ probe_resolve_value: Optional[str] = None+ # Parent field to percent-encode into the fan-out `resolve_field`, for a parent value that is+ # free text bound into the child path's query string.+ encode_parent_field: Optional[str] = None# Endpoint to probe for permissions instead of this one, when this response is too large to# download on every schema-picker load. It must read with the same scope.probe_endpoint: Optional[str] = None@@ -119,8 +127,8 @@ def _datetime_field(name: str) -> IncrementalField:# SendGrid (v3) endpoint catalog. The suppression endpoints expose a genuine server-side# `start_time` filter (Unix epoch seconds) over their immutable `created` field, so they sync-# incrementally. `stats` filters on `start_date` (YYYY-MM-DD) over its daily aggregate date and-# also syncs incrementally. Marketing/asm metadata endpoints offer no timestamp filter and ship as+# incrementally. `stats` and `category_stats` filter on `start_date` (YYYY-MM-DD) over their daily+# aggregate date and also sync incrementally. Marketing/asm metadata endpoints offer no timestamp filter and ship as# full refresh. See api_inventory.md for the per-endpoint research notes.SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {"bounces": SendGridEndpointConfig(@@ -190,6 +198,45 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {extra_params={"aggregated_by": "day"},required_scope="stats.read",),+ "categories": SendGridEndpointConfig(+ name="categories",+ path="/categories",+ primary_keys=["category"],+ pagination="offset",+ # The documented default. No maximum is documented, and a silently capped page would read+ # as the short last page and end the walk early.+ page_size=50,+ required_scope="categories.read",+ ),+ # /categories/stats requires the `categories` param, so it fans out over the category listing+ # one category per request. The listing reads with a different scope, so the probe hits this+ # path itself.+ "category_stats": SendGridEndpointConfig(+ name="category_stats",+ path="/categories/stats?categories={category}",+ primary_keys=["category", "date"],+ pagination="single",+ partition_key="date",+ incremental_fields=[_date_field("date")],+ incremental_param="start_date",+ incremental_param_format="date",+ default_backfill_days=STATS_DEFAULT_BACKFILL_DAYS,+ response_shape="category_stats",+ sort_mode=None,+ extra_params={"aggregated_by": "day"},+ required_scope="categories.stats.read",+ default_incremental_field="date",+ fanout=DependentEndpointConfig(+ parent_name="categories",+ resolve_param="category",+ resolve_field="category_param",+ include_from_parent=["category"],+ parent_field_renames={"category": "category"},+ ),+ probe_resolve_value="posthog",+ # Category names are free text, and the resolved value is bound into the path unescaped.+ encode_parent_field="category",+ ),"unsubscribe_groups": SendGridEndpointConfig(name="unsubscribe_groups",path="/asm/groups",@@ -263,6 +310,37 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {child_response_actions=[{"status_code": 404, "action": "ignore"}],),),+ "automation_stats": SendGridEndpointConfig(+ name="automation_stats",+ path="/marketing/stats/automations",+ # Without `group_by`, step_id is "all" and aggregation is "total", so each automation has+ # one lifetime-total row. They stay in the key so a per-step or per-day row can never collide.+ primary_keys=["id", "aggregation", "step_id"],+ pagination="metadata",+ data_key="results",+ page_size=50,+ required_scope="marketing.read",+ permission_note=_MARKETING_CAMPAIGNS_NOTE,+ ),+ # The API has no automation listing, so the stats rows supply the automation ids.+ "automation_link_stats": SendGridEndpointConfig(+ name="automation_link_stats",+ path="/marketing/stats/automations/{automation_id}/links",+ primary_keys=["automation_id", "url", "url_location", "step_id"],+ pagination="metadata",+ data_key="results",+ page_size=50,+ required_scope="marketing.read",+ permission_note=_MARKETING_CAMPAIGNS_NOTE,+ fanout=DependentEndpointConfig(+ parent_name="automation_stats",+ resolve_param="automation_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "automation_id"},+ child_response_actions=[{"status_code": 404, "action": "ignore"}],+ ),+ ),"templates": SendGridEndpointConfig(name="templates",path="/templates",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/source.pyindex dd975f09708c..7ecf9667945a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/source.py@@ -202,7 +202,8 @@ You can create an API key in your [SendGrid account settings](https://app.sendgrGrant the following read access (Restricted Access) so the key can reach the data you want to sync:- **Suppressions**: bounces, blocks, invalid emails, spam reports, global unsubscribes, unsubscribe groups and their members- **Stats**: global email statistics (daily requests, delivered, opens, clicks, bounces)-- **Marketing**: marketing lists, single sends, and single send stats. This also needs an account with Marketing Campaigns enabled.+- **Category Management**: categories and their daily statistics+- **Marketing**: marketing lists, single sends, single send stats, and automation stats. This also needs an account with Marketing Campaigns enabled.- **Template Engine**: templates- **Email Activity**: message activity. This table is off by default and also needs SendGrid's paid additional email activity history add-on.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/tests/test_sendgrid.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/tests/test_sendgrid.pyindex 7918d9f558b2..af15bca95f85 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/tests/test_sendgrid.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/tests/test_sendgrid.py@@ -267,6 +267,106 @@ class TestSingleSendLinkStatsFanOut:assert result.rows == [{"url": "https://example.com/b", "singlesend_id": "s2"}]+def _automation_links_path(automation_id: str) -> str:+ return f"/marketing/stats/automations/{automation_id}/links"+++class TestAutomationLinkStatsFanOut:+ def test_walks_every_automation_and_tags_rows_with_its_id(self) -> None:+ stats_next = "https://api.sendgrid.com/v3/marketing/stats/automations?page_token=p2&page_size=50"+ link = {"url": "https://example.com/a", "url_location": 0, "step_id": "all"}++ result = _run(+ "automation_link_stats",+ route(+ {+ "/marketing/stats/automations": [+ _response(+ {+ "results": [+ {"id": "a1", "aggregation": "total", "step_id": "all"},+ {"id": "gone", "aggregation": "total", "step_id": "all"},+ ],+ "_metadata": {"next": stats_next},+ }+ ),+ _response(+ {"results": [{"id": "a3", "aggregation": "total", "step_id": "all"}], "_metadata": {}}+ ),+ ],+ _automation_links_path("a1"): [_response({"results": [{**link, "clicks": 4}], "_metadata": {}})],+ _automation_links_path("gone"): [ScriptedResponse(status=404, json={"errors": [{"message": "x"}]})],+ _automation_links_path("a3"): [_response({"results": [{**link, "clicks": 9}], "_metadata": {}})],+ }+ ),+ )++ assert result.raised is None+ assert result.rows == [+ {**link, "clicks": 4, "automation_id": "a1"},+ {**link, "clicks": 9, "automation_id": "a3"},+ ]+ assert stats_next in result.urls+++def _category_stats_bucket(day: str, category: str, delivered: int) -> dict[str, Any]:+ return {"date": day, "stats": [{"type": "category", "name": category, "metrics": {"delivered": delivered}}]}+++class TestCategoryStatsFanOut:+ @pytest.mark.parametrize(+ ("inputs", "expected_start_date"),+ [+ # start_date is required, so a full refresh backfills a fixed window.+ ({}, "2025-08-07"),+ (+ {+ "should_use_incremental_field": True,+ "db_incremental_field_last_value": date(2026, 8, 1),+ "incremental_field": "date",+ },+ "2026-08-01",+ ),+ ],+ )+ @time_machine.travel("2026-08-07T12:00:00Z", tick=False)+ def test_requests_each_category_and_keys_rows_by_it(self, inputs: dict[str, Any], expected_start_date: str) -> None:+ categories_page = [{"category": f"c{i}"} for i in range(49)] + [{"category": "news & offers"}]++ result = _run(+ "category_stats",+ route(+ {+ "/v3/categories": [_response(categories_page), _response([{"category": "a/b"}])],+ "/v3/categories/stats": [+ *[_response([]) for _ in range(49)],+ _response(+ [+ _category_stats_bucket("2026-08-01", "news & offers", 3),+ _category_stats_bucket("2026-08-02", "news & offers", 5),+ ]+ ),+ _response([_category_stats_bucket("2026-08-01", "a/b", 7)]),+ ],+ }+ ),+ **inputs,+ )++ assert result.raised is None+ assert result.rows == [+ {"date": "2026-08-01", "category": "news & offers", "delivered": 3},+ {"date": "2026-08-02", "category": "news & offers", "delivered": 5},+ {"date": "2026-08-01", "category": "a/b", "delivered": 7},+ ]+ stats_requests = [r for r in result.requests if r.path == "/v3/categories/stats"]+ # Names with `&` or `/` must reach the API whole, not split into extra params or path segments.+ assert [r.param("categories") for r in stats_requests[-2:]] == ["news & offers", "a/b"]+ assert {r.param("start_date") for r in stats_requests} == {expected_start_date}+ assert {r.param("aggregated_by") for r in stats_requests} == {"day"}+ assert [r.param("offset") for r in result.requests if r.path == "/v3/categories"] == ["0", "50"]++def _messages_page(count: int, last_event_time: str) -> list[dict[str, Any]]:return [{"msg_id": f"m{last_event_time}-{i}", "last_event_time": last_event_time} for i in range(count)]@@ -418,13 +518,23 @@ class TestGetEndpointPermissions:for fragment in fragments:assert fragment in reason- def test_fanout_endpoint_probes_its_parent_listing(self) -> None:- # The child path needs a single send id, so probing it as is would 404 and hide a denial.+ @pytest.mark.parametrize(+ ("endpoint", "expected_path"),+ [+ # The child path needs a parent id, so probing it as is would 404 and hide a denial.+ ("singlesend_link_stats", "/v3/marketing/singlesends"),+ ("automation_link_stats", "/v3/marketing/stats/automations"),+ # The category listing reads with `categories.read`, so it can't stand in for the+ # `categories.stats.read` this child needs.+ ("category_stats", "/v3/categories/stats"),+ ],+ )+ def test_fanout_endpoint_probe_path(self, endpoint: str, expected_path: str) -> None:with scripted_network(always(ScriptedResponse(status=403, json={}))) as network:- permissions = get_endpoint_permissions("k", ["singlesend_link_stats"])+ permissions = get_endpoint_permissions("k", [endpoint])- assert network.requests_log[0].path == "/v3/marketing/singlesends"- assert permissions["singlesend_link_stats"] is not None+ assert network.requests_log[0].path == expected_path+ assert permissions[endpoint] is not Nonedef test_unbounded_endpoint_probes_a_small_one_with_the_same_scope(self) -> None:with scripted_network(always(ScriptedResponse(status=403, json={}))) as network:@@ -445,6 +555,8 @@ class TestGetEndpointPermissions:("marketing_lists", {"page_size": "1"}),("templates", {"page_size": "1", "generations": "legacy,dynamic"}),("unsubscribe_groups", {}),+ # /categories/stats requires `categories` and `start_date`.+ ("category_stats", {"categories": "posthog", "start_date": "2026-08-07", "aggregated_by": "day"}),# /messages requires `query`, so a probe without one risks the same masking 400.("message_activity",