Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 4 missing tables to the SendGrid source (#125316)

Tom Owersb560a9280e39
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index f2af31fac015..22db70f4ab77 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -10774,9 +10774,9 @@ Diffed against: <https://github.com/twilio/sendgrid-oai/tree/main/spec/json (off
- [x] `GET /v3/messages (Email Activity)` — Per-message event feed (delivered, opened, clicked, bounced) — the row-level fact table behind all aggregate stats (high)
- [ ] `GET /v3/marketing/contacts (+ /contacts/exports)` — Contact records; no recipient dimension exists today so lists and suppressions cannot be joined to people (high)
- [x] `GET /v3/asm/groups/{group_id}/suppressions` — Membership rows linking email addresses to the unsubscribe_groups already synced — the join table is missing (high)
-- [ ] `GET /v3/categories/stats (and /v3/categories/stats/sums)` — Email stats broken down by category, the main way transactional senders segment performance (medium)
-- [ ] `GET /v3/categories` — Category lookup resolving the category tags used on stats and messages (medium)
-- [ ] `GET /v3/marketing/stats/automations (+ /{id}/links)` — Automation (journey) performance stats (medium)
+- [x] `GET /v3/categories/stats (and /v3/categories/stats/sums)` — Email stats broken down by category, the main way transactional senders segment performance (medium)
+- [x] `GET /v3/categories` — Category lookup resolving the category tags used on stats and messages (medium)
+- [x] `GET /v3/marketing/stats/automations (+ /{id}/links)` — Automation (journey) performance stats (medium)
- [ ] `GET /v3/marketing/segments/2.0` — Marketing segment definitions; marketing_lists is synced but segments are not (medium)
- [ ] `GET /v3/geo/stats, /v3/mailbox_providers/stats, /v3/devices/stats, /v3/clients/stats, /v3/browsers/stats` — Breakdown dimensions on the same stats fact (country, mailbox provider, device, client, browser) (medium)
- [ ] `GET /v3/subusers and /v3/subusers/stats (+ /stats/monthly, /stats/sums, /reputations)` — Per-subuser volume and reputation — required for any agency or multi-tenant SendGrid account (medium)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/api_inventory.md b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/api_inventory.md
index 7ef1b0ea7b87..248371d67c52 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/api_inventory.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/api_inventory.md
@@ -32,6 +32,10 @@ on the primary key — wasted API calls, not corrupted data.
| `singlesends` | `/marketing/singlesends` | `_metadata` | `{"result": [...]}` | `id` | full refresh |
| `singlesend_stats` | `/marketing/stats/singlesends` | `_metadata` | `{"results": [...]}` | `id`, `ab_variation`, `ab_phase` | full refresh |
| `singlesend_link_stats` | `/marketing/stats/singlesends/{id}/links` (fan-out over `singlesends`) | `_metadata` | `{"results": [...]}` | `singlesend_id`, `url`, `url_location`, `ab_variation`, `ab_phase` | full refresh |
+| `categories` | `/categories` | limit/offset | bare array | `category` | full refresh |
+| `category_stats` | `/categories/stats` (fan-out over `categories`) | none (single) | nested daily stats | `category`, `date` | `date` via `start_date` (YYYY-MM-DD) |
+| `automation_stats` | `/marketing/stats/automations` | `_metadata` | `{"results": [...]}` | `id`, `aggregation`, `step_id` | full refresh |
+| `automation_link_stats` | `/marketing/stats/automations/{id}/links` (fan-out) | `_metadata` | `{"results": [...]}` | `automation_id`, `url`, `url_location`, `step_id` | full refresh |
Notes:
@@ -78,6 +82,25 @@ Notes:
over `/asm/groups` and still lack the creation time. Neither takes a time filter.
- The Single Send endpoints (`singlesends`, `singlesend_stats`, `singlesend_link_stats`) follow
`_metadata.next` like the other marketing endpoints. The stats endpoints cap `page_size` at 50.
+- `categories` reads `/categories`, a bare array of `{category}` paged by `limit`/`offset`. No maximum
+ `limit` is documented, so it pages at the documented default of 50: a silently capped page would read
+ as the short last page and end the walk early.
+- `/categories/stats` requires `categories` (up to 10 names) and `start_date`, so `category_stats` fans
+ out over `categories` with one category per request and `aggregated_by=day`. Each response is the
+ same `{date, stats: [{name, type, metrics}]}` nesting as `/stats`, flattened to one row per category
+ and day. Category names are free text, so the name is percent-encoded before it goes into the query
+ string. `start_date` filters server-side, so the table syncs incrementally on `date`. Rows come one
+ category at a time, so they declare no sort order. The listing needs `categories.read` and the stats
+ need `categories.stats.read`, so the permission probe calls `/categories/stats` itself with a
+ placeholder category. `/categories/stats/sums` is not synced: it sums the same metrics over a date
+ range, which a query over `category_stats` can do.
+- The Automation endpoints (`automation_stats`, `automation_link_stats`) follow `_metadata.next` and
+ cap `page_size` at 50, like the Single Send stats. The API has no Automation listing, so the link
+ fan-out reads the Automation ids from the stats rows (one lifetime-total row per Automation without
+ `group_by`).
+- `/marketing/contacts` is not synced. It returns a sample of up to 50 recent contacts, and its pagination
+ is deprecated. The full set needs the asynchronous export job (`POST /marketing/contacts/exports`, poll
+ the status, then download the files), and the public spec does not document the file layout.
`singlesend_stats` sends no `singlesend_ids`, so it returns lifetime totals for every Single Send;
without `group_by` each row has `ab_variation` and `ab_phase` set to `all`. `singlesend_link_stats`
fans out over `singlesends`, and a 404 for a Single Send deleted mid-sync is skipped. No Single Send
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/canonical_descriptions.py
index 6640d3147c8b..23365dbefe9c 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/canonical_descriptions.py
@@ -74,6 +74,37 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"invalid_emails": "Number of emails dropped because the recipient address was invalid.",
},
},
+ "categories": {
+ "description": "Categories: the tags senders attach to emails to group them by topic, used to segment statistics.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/categories/retrieve-all-categories",
+ "columns": {
+ "category": "Name of the category used to group emails by broad topic.",
+ },
+ },
+ "category_stats": {
+ "description": "Daily email statistics for each category, such as requests, deliveries, opens, clicks, and bounces.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/categories/retrieve-email-statistics-for-categories",
+ "columns": {
+ "date": "The date the statistics were gathered, in YYYY-MM-DD format.",
+ "category": "Name of the category that the statistics are for.",
+ "requests": "Number of emails that were requested to be delivered.",
+ "processed": "Number of requests that SendGrid processed.",
+ "delivered": "Number of emails that SendGrid confirmed were delivered to a recipient.",
+ "deferred": "Number of emails that temporarily could not be delivered.",
+ "opens": "Total number of times the emails were opened by recipients.",
+ "unique_opens": "Number of unique recipients who opened the emails.",
+ "clicks": "Number of links that were clicked.",
+ "unique_clicks": "Number of unique recipients who clicked links in the emails.",
+ "bounces": "Number of emails that bounced instead of being delivered.",
+ "bounce_drops": "Number of emails that were dropped because of a bounce.",
+ "blocks": "Number of emails that were not allowed to be delivered by ISPs.",
+ "invalid_emails": "Number of recipients with malformed addresses or addresses that the mail provider reported as invalid.",
+ "spam_reports": "Number of recipients who marked the email as spam.",
+ "spam_report_drops": "Number of emails that were dropped because the recipient previously marked emails as spam.",
+ "unsubscribes": "Number of recipients who unsubscribed.",
+ "unsubscribe_drops": "Number of emails that were dropped because the recipient unsubscribed.",
+ },
+ },
"unsubscribe_groups": {
"description": "Suppression (unsubscribe) groups used to let recipients opt out of specific categories of mail.",
"docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/suppressions-unsubscribe-groups/retrieve-all-suppression-groups-associated-with-the-user",
@@ -142,6 +173,27 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"clicks": "Number of clicks on this link.",
},
},
+ "automation_stats": {
+ "description": "Lifetime engagement statistics for each Marketing Campaigns Automation, such as requests, deliveries, opens, and clicks.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/marketing-campaign-stats/get-all-automation-stats",
+ "columns": {
+ "id": "Unique identifier for the Automation that the statistics are for.",
+ "aggregation": "The time unit that the statistics are rolled up to, total for lifetime totals.",
+ "step_id": "The Automation step that the statistics are for, or all when they cover every step.",
+ "stats": "The metrics: requests, delivered, opens, unique_opens, clicks, unique_clicks, bounces, bounce_drops, invalid_emails, spam_reports, spam_report_drops, and unsubscribes.",
+ },
+ },
+ "automation_link_stats": {
+ "description": "Click counts for each link in each Marketing Campaigns Automation.",
+ "docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/marketing-campaign-stats/get-automation-click-tracking-stats-by-id",
+ "columns": {
+ "automation_id": "Unique identifier for the Automation that contains the link.",
+ "url": "The URL of the clicked link, including any custom fields in the URL.",
+ "url_location": "The position of the link in the Automation step, numbered from the top down starting at 0.",
+ "step_id": "The Automation step that contains the link.",
+ "clicks": "Number of clicks on this link.",
+ },
+ },
"templates": {
"description": "Email templates (legacy and dynamic) used to compose messages.",
"docs_url": "https://www.twilio.com/docs/sendgrid/api-reference/transactional-templates/retrieve-paged-transactional-templates",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/sendgrid.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/sendgrid.py
index ebe74ddadd7d..3398efedde84 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/sendgrid.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/sendgrid.py
@@ -1,8 +1,9 @@
import logging
import dataclasses
+from collections.abc import Callable
from datetime import UTC, date, datetime, timedelta
from typing import Any, Optional
-from urllib.parse import parse_qs, urlencode, urlparse
+from urllib.parse import parse_qs, quote, urlencode, urlparse
from requests import Request, Response
@@ -20,6 +21,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
OffsetPaginator,
SinglePagePaginator,
)
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.resource import Resource
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (
ClientConfig,
EndpointResource,
@@ -115,6 +117,15 @@ def _flatten_daily_stats(item: dict[str, Any]) -> list[dict[str, Any]]:
return rows
+def _flatten_category_stats(item: dict[str, Any]) -> list[dict[str, Any]]:
+ """Flatten one `{date, stats: [{name, metrics: {...}}]}` bucket of a single-category request
+ into a row per metrics entry, keyed by the requested category from the parent row."""
+ return [
+ {"date": item.get("date"), "category": item.get("category"), **(entry.get("metrics") or {})}
+ for entry in item.get("stats") or []
+ ]
+
+
_ACTIVITY_TIMESTAMP_FORMAT = "%Y-%m-%dT%H:%M:%SZ"
@@ -379,12 +390,23 @@ def _client_config(api_key: str, paginator: BasePaginator) -> ClientConfig:
}
+def _encoded_parent_field(source_field: str, target_field: str) -> Callable[[dict[str, Any]], dict[str, Any]]:
+ def _map(row: dict[str, Any]) -> dict[str, Any]:
+ row[target_field] = quote(str(row[source_field]), safe="")
+ return row
+
+ return _map
+
+
def _fanout_resource(
api_key: str,
config: SendGridEndpointConfig,
team_id: int,
job_id: str,
resumable_source_manager: ResumableSourceManager[SendGridResumeConfig],
+ should_use_incremental_field: bool,
+ db_incremental_field_last_value: Any,
+ incremental_field: Optional[str],
) -> Any:
assert config.fanout is not None
parent_config = SENDGRID_ENDPOINTS[config.fanout.parent_name]
@@ -399,29 +421,48 @@ def _fanout_resource(
if state is not None:
resumable_source_manager.save_state(SendGridResumeConfig(fanout_state=state))
- return build_dependent_resource(
+ child_params = _build_params(
+ config, should_use_incremental_field, db_incremental_field_last_value, incremental_field
+ )
+
+ resource = build_dependent_resource(
endpoint_configs=SENDGRID_ENDPOINTS,
child_endpoint=config.name,
fanout=config.fanout,
- client_config=_client_config(api_key, SendGridMetadataPaginator()),
+ client_config=_client_config(api_key, _build_paginator(config)),
path_format_values={},
team_id=team_id,
job_id=job_id,
db_incremental_field_last_value=None,
- page_size_param="page_size",
+ should_use_incremental_field=should_use_incremental_field,
+ incremental_field=incremental_field,
+ # The cursor is already bound into `child_params` as the endpoint's start param.
+ incremental_config_factory=lambda _field: None,
+ # Offset parents get `limit` from their paginator, single-page children take no page size.
+ page_size_param="page_size" if config.pagination == "metadata" else None,
+ child_params_extra=child_params,
+ parent_data_map=(
+ _encoded_parent_field(config.encode_parent_field, config.fanout.resolve_field)
+ if config.encode_parent_field
+ else None
+ ),
resume_hook=save_checkpoint,
initial_paginator_state=initial_paginator_state,
parent_endpoint_extra={
- "paginator": SendGridMetadataPaginator(),
+ "paginator": _build_paginator(parent_config),
"data_selector": parent_config.data_key,
"data_selector_required": True,
},
child_endpoint_extra={
- "paginator": SendGridMetadataPaginator(),
+ "paginator": _build_paginator(config),
"data_selector": config.data_key,
"data_selector_required": True,
},
)
+ if config.response_shape == "category_stats":
+ assert isinstance(resource, Resource)
+ resource.add_map(_flatten_category_stats)
+ return resource
def _source_response(endpoint: str, config: SendGridEndpointConfig, resource: Any) -> SourceResponse:
@@ -452,7 +493,18 @@ def sendgrid_source(
if config.fanout is not None:
return _source_response(
- endpoint, config, _fanout_resource(api_key, config, team_id, job_id, resumable_source_manager)
+ endpoint,
+ config,
+ _fanout_resource(
+ api_key,
+ config,
+ team_id,
+ job_id,
+ resumable_source_manager,
+ should_use_incremental_field,
+ db_incremental_field_last_value,
+ incremental_field,
+ ),
)
params = _build_params(config, should_use_incremental_field, db_incremental_field_last_value, incremental_field)
@@ -525,9 +577,10 @@ def get_status_code(api_key: str, path: str, params: Optional[dict[str, Any]] =
"""Probe an endpoint to classify the credentials. Returns the HTTP status, or None on a
transport error."""
query = urlencode(params if params is not None else {"limit": 1})
+ separator = "&" if "?" in path else "?"
_ok, status = validate_via_probe(
lambda: make_tracked_session(redact_values=(api_key,)),
- f"{SENDGRID_BASE_URL}{path}?{query}",
+ f"{SENDGRID_BASE_URL}{path}{separator}{query}",
headers=_get_headers(api_key),
)
return status
@@ -547,12 +600,18 @@ def _probe_params(config: SendGridEndpointConfig) -> dict[str, Any]:
now = datetime.now(UTC)
params["limit"] = 1
params["query"] = _activity_query(_to_activity_timestamp(now - timedelta(days=1)), _to_activity_timestamp(now))
+ if config.incremental_param and config.default_backfill_days is not None:
+ # The start param is required, so a probe without it risks a 400 that hides a denial.
+ params[config.incremental_param] = _format_cursor(datetime.now(UTC), config.incremental_param_format)
return params
def get_endpoint_status_code(api_key: str, config: SendGridEndpointConfig) -> Optional[int]:
- # A fan-out child path needs a parent id, and its parent reads with the same scope, so the
- # parent listing stands in for it.
+ if config.fanout is not None and config.probe_resolve_value is not None:
+ path = config.path.replace(f"{{{config.fanout.resolve_param}}}", config.probe_resolve_value)
+ return get_status_code(api_key, path, _probe_params(config))
+ # Any other fan-out child path needs a parent id, and its parent reads with the same scope, so
+ # the parent listing stands in for it.
probe_config = config
if config.fanout is not None:
probe_config = SENDGRID_ENDPOINTS[config.fanout.parent_name]
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/settings.py
index 01394716ccc8..a415e8488db8 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/settings.py
@@ -27,7 +27,9 @@ IncrementalParamFormat = Literal["epoch", "date"]
# - "array": the rows are the JSON body itself, or the array under `data_key`.
# - "daily_stats": nested `[{date, stats: [{metrics: {...}}]}]` that flattens to one row per
# metrics bucket with `date` merged in (the /stats family).
-ResponseShape = Literal["array", "daily_stats"]
+# - "category_stats": the same nesting, flattened to one row per date with the requested
+# `category` merged in (category_stats).
+ResponseShape = Literal["array", "daily_stats", "category_stats"]
# /stats requires `start_date` on every request, so the first sync (no cursor yet) backfills this
# far back. Daily aggregation means one row per day, so a year is cheap and gives enough history to
@@ -60,8 +62,9 @@ class SendGridEndpointConfig:
response_shape: ResponseShape = "array"
# Order rows arrive in across the whole sync. The Email Activity walk pages newest to oldest,
# so it declares "desc" and the pipeline defers the incremental watermark to the end of a
- # successful run instead of checkpointing it per batch.
- sort_mode: Literal["asc", "desc"] = "asc"
+ # successful run instead of checkpointing it per batch. None when rows come one parent at a
+ # time, so they form no single ordered series.
+ sort_mode: Optional[Literal["asc", "desc"]] = "asc"
# False for tables whose sync needs grants beyond what source creation validated: the schema
# picker and one-shot source creation leave them unselected until the user opts in.
should_sync_default: bool = True
@@ -73,11 +76,16 @@ class SendGridEndpointConfig:
required_scope: str = ""
# Appended to the missing-scope message where granting the scope may not be enough on its own.
permission_note: Optional[str] = None
- # Structural requirement of `FanoutEndpointLike` (see common/rest_source/fanout.py); no
- # fan-out endpoint here syncs incrementally.
+ # Structural requirement of `FanoutEndpointLike` (see common/rest_source/fanout.py).
default_incremental_field: Optional[str] = None
# Set on a child endpoint whose path takes one id per row of a parent endpoint.
fanout: Optional[DependentEndpointConfig] = None
+ # Placeholder parent value to probe the child path with, for a fan-out child that reads with a
+ # different scope than its parent. None probes the parent listing instead.
+ probe_resolve_value: Optional[str] = None
+ # Parent field to percent-encode into the fan-out `resolve_field`, for a parent value that is
+ # free text bound into the child path's query string.
+ encode_parent_field: Optional[str] = None
# Endpoint to probe for permissions instead of this one, when this response is too large to
# download on every schema-picker load. It must read with the same scope.
probe_endpoint: Optional[str] = None
@@ -119,8 +127,8 @@ def _datetime_field(name: str) -> IncrementalField:
# SendGrid (v3) endpoint catalog. The suppression endpoints expose a genuine server-side
# `start_time` filter (Unix epoch seconds) over their immutable `created` field, so they sync
-# incrementally. `stats` filters on `start_date` (YYYY-MM-DD) over its daily aggregate date and
-# also syncs incrementally. Marketing/asm metadata endpoints offer no timestamp filter and ship as
+# incrementally. `stats` and `category_stats` filter on `start_date` (YYYY-MM-DD) over their daily
+# aggregate date and also sync incrementally. Marketing/asm metadata endpoints offer no timestamp filter and ship as
# full refresh. See api_inventory.md for the per-endpoint research notes.
SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {
"bounces": SendGridEndpointConfig(
@@ -190,6 +198,45 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {
extra_params={"aggregated_by": "day"},
required_scope="stats.read",
),
+ "categories": SendGridEndpointConfig(
+ name="categories",
+ path="/categories",
+ primary_keys=["category"],
+ pagination="offset",
+ # The documented default. No maximum is documented, and a silently capped page would read
+ # as the short last page and end the walk early.
+ page_size=50,
+ required_scope="categories.read",
+ ),
+ # /categories/stats requires the `categories` param, so it fans out over the category listing
+ # one category per request. The listing reads with a different scope, so the probe hits this
+ # path itself.
+ "category_stats": SendGridEndpointConfig(
+ name="category_stats",
+ path="/categories/stats?categories={category}",
+ primary_keys=["category", "date"],
+ pagination="single",
+ partition_key="date",
+ incremental_fields=[_date_field("date")],
+ incremental_param="start_date",
+ incremental_param_format="date",
+ default_backfill_days=STATS_DEFAULT_BACKFILL_DAYS,
+ response_shape="category_stats",
+ sort_mode=None,
+ extra_params={"aggregated_by": "day"},
+ required_scope="categories.stats.read",
+ default_incremental_field="date",
+ fanout=DependentEndpointConfig(
+ parent_name="categories",
+ resolve_param="category",
+ resolve_field="category_param",
+ include_from_parent=["category"],
+ parent_field_renames={"category": "category"},
+ ),
+ probe_resolve_value="posthog",
+ # Category names are free text, and the resolved value is bound into the path unescaped.
+ encode_parent_field="category",
+ ),
"unsubscribe_groups": SendGridEndpointConfig(
name="unsubscribe_groups",
path="/asm/groups",
@@ -263,6 +310,37 @@ SENDGRID_ENDPOINTS: dict[str, SendGridEndpointConfig] = {
child_response_actions=[{"status_code": 404, "action": "ignore"}],
),
),
+ "automation_stats": SendGridEndpointConfig(
+ name="automation_stats",
+ path="/marketing/stats/automations",
+ # Without `group_by`, step_id is "all" and aggregation is "total", so each automation has
+ # one lifetime-total row. They stay in the key so a per-step or per-day row can never collide.
+ primary_keys=["id", "aggregation", "step_id"],
+ pagination="metadata",
+ data_key="results",
+ page_size=50,
+ required_scope="marketing.read",
+ permission_note=_MARKETING_CAMPAIGNS_NOTE,
+ ),
+ # The API has no automation listing, so the stats rows supply the automation ids.
+ "automation_link_stats": SendGridEndpointConfig(
+ name="automation_link_stats",
+ path="/marketing/stats/automations/{automation_id}/links",
+ primary_keys=["automation_id", "url", "url_location", "step_id"],
+ pagination="metadata",
+ data_key="results",
+ page_size=50,
+ required_scope="marketing.read",
+ permission_note=_MARKETING_CAMPAIGNS_NOTE,
+ fanout=DependentEndpointConfig(
+ parent_name="automation_stats",
+ resolve_param="automation_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "automation_id"},
+ child_response_actions=[{"status_code": 404, "action": "ignore"}],
+ ),
+ ),
"templates": SendGridEndpointConfig(
name="templates",
path="/templates",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/source.py
index dd975f09708c..7ecf9667945a 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/source.py
@@ -202,7 +202,8 @@ You can create an API key in your [SendGrid account settings](https://app.sendgr
Grant the following read access (Restricted Access) so the key can reach the data you want to sync:
- **Suppressions**: bounces, blocks, invalid emails, spam reports, global unsubscribes, unsubscribe groups and their members
- **Stats**: global email statistics (daily requests, delivered, opens, clicks, bounces)
-- **Marketing**: marketing lists, single sends, and single send stats. This also needs an account with Marketing Campaigns enabled.
+- **Category Management**: categories and their daily statistics
+- **Marketing**: marketing lists, single sends, single send stats, and automation stats. This also needs an account with Marketing Campaigns enabled.
- **Template Engine**: templates
- **Email Activity**: message activity. This table is off by default and also needs SendGrid's paid additional email activity history add-on.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/tests/test_sendgrid.py b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/tests/test_sendgrid.py
index 7918d9f558b2..af15bca95f85 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/tests/test_sendgrid.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/sendgrid/tests/test_sendgrid.py
@@ -267,6 +267,106 @@ class TestSingleSendLinkStatsFanOut:
assert result.rows == [{"url": "https://example.com/b", "singlesend_id": "s2"}]
+def _automation_links_path(automation_id: str) -> str:
+ return f"/marketing/stats/automations/{automation_id}/links"
+
+
+class TestAutomationLinkStatsFanOut:
+ def test_walks_every_automation_and_tags_rows_with_its_id(self) -> None:
+ stats_next = "https://api.sendgrid.com/v3/marketing/stats/automations?page_token=p2&page_size=50"
+ link = {"url": "https://example.com/a", "url_location": 0, "step_id": "all"}
+
+ result = _run(
+ "automation_link_stats",
+ route(
+ {
+ "/marketing/stats/automations": [
+ _response(
+ {
+ "results": [
+ {"id": "a1", "aggregation": "total", "step_id": "all"},
+ {"id": "gone", "aggregation": "total", "step_id": "all"},
+ ],
+ "_metadata": {"next": stats_next},
+ }
+ ),
+ _response(
+ {"results": [{"id": "a3", "aggregation": "total", "step_id": "all"}], "_metadata": {}}
+ ),
+ ],
+ _automation_links_path("a1"): [_response({"results": [{**link, "clicks": 4}], "_metadata": {}})],
+ _automation_links_path("gone"): [ScriptedResponse(status=404, json={"errors": [{"message": "x"}]})],
+ _automation_links_path("a3"): [_response({"results": [{**link, "clicks": 9}], "_metadata": {}})],
+ }
+ ),
+ )
+
+ assert result.raised is None
+ assert result.rows == [
+ {**link, "clicks": 4, "automation_id": "a1"},
+ {**link, "clicks": 9, "automation_id": "a3"},
+ ]
+ assert stats_next in result.urls
+
+
+def _category_stats_bucket(day: str, category: str, delivered: int) -> dict[str, Any]:
+ return {"date": day, "stats": [{"type": "category", "name": category, "metrics": {"delivered": delivered}}]}
+
+
+class TestCategoryStatsFanOut:
+ @pytest.mark.parametrize(
+ ("inputs", "expected_start_date"),
+ [
+ # start_date is required, so a full refresh backfills a fixed window.
+ ({}, "2025-08-07"),
+ (
+ {
+ "should_use_incremental_field": True,
+ "db_incremental_field_last_value": date(2026, 8, 1),
+ "incremental_field": "date",
+ },
+ "2026-08-01",
+ ),
+ ],
+ )
+ @time_machine.travel("2026-08-07T12:00:00Z", tick=False)
+ def test_requests_each_category_and_keys_rows_by_it(self, inputs: dict[str, Any], expected_start_date: str) -> None:
+ categories_page = [{"category": f"c{i}"} for i in range(49)] + [{"category": "news & offers"}]
+
+ result = _run(
+ "category_stats",
+ route(
+ {
+ "/v3/categories": [_response(categories_page), _response([{"category": "a/b"}])],
+ "/v3/categories/stats": [
+ *[_response([]) for _ in range(49)],
+ _response(
+ [
+ _category_stats_bucket("2026-08-01", "news & offers", 3),
+ _category_stats_bucket("2026-08-02", "news & offers", 5),
+ ]
+ ),
+ _response([_category_stats_bucket("2026-08-01", "a/b", 7)]),
+ ],
+ }
+ ),
+ **inputs,
+ )
+
+ assert result.raised is None
+ assert result.rows == [
+ {"date": "2026-08-01", "category": "news & offers", "delivered": 3},
+ {"date": "2026-08-02", "category": "news & offers", "delivered": 5},
+ {"date": "2026-08-01", "category": "a/b", "delivered": 7},
+ ]
+ stats_requests = [r for r in result.requests if r.path == "/v3/categories/stats"]
+ # Names with `&` or `/` must reach the API whole, not split into extra params or path segments.
+ assert [r.param("categories") for r in stats_requests[-2:]] == ["news & offers", "a/b"]
+ assert {r.param("start_date") for r in stats_requests} == {expected_start_date}
+ assert {r.param("aggregated_by") for r in stats_requests} == {"day"}
+ assert [r.param("offset") for r in result.requests if r.path == "/v3/categories"] == ["0", "50"]
+
+
def _messages_page(count: int, last_event_time: str) -> list[dict[str, Any]]:
return [{"msg_id": f"m{last_event_time}-{i}", "last_event_time": last_event_time} for i in range(count)]
@@ -418,13 +518,23 @@ class TestGetEndpointPermissions:
for fragment in fragments:
assert fragment in reason
- def test_fanout_endpoint_probes_its_parent_listing(self) -> None:
- # The child path needs a single send id, so probing it as is would 404 and hide a denial.
+ @pytest.mark.parametrize(
+ ("endpoint", "expected_path"),
+ [
+ # The child path needs a parent id, so probing it as is would 404 and hide a denial.
+ ("singlesend_link_stats", "/v3/marketing/singlesends"),
+ ("automation_link_stats", "/v3/marketing/stats/automations"),
+ # The category listing reads with `categories.read`, so it can't stand in for the
+ # `categories.stats.read` this child needs.
+ ("category_stats", "/v3/categories/stats"),
+ ],
+ )
+ def test_fanout_endpoint_probe_path(self, endpoint: str, expected_path: str) -> None:
with scripted_network(always(ScriptedResponse(status=403, json={}))) as network:
- permissions = get_endpoint_permissions("k", ["singlesend_link_stats"])
+ permissions = get_endpoint_permissions("k", [endpoint])
- assert network.requests_log[0].path == "/v3/marketing/singlesends"
- assert permissions["singlesend_link_stats"] is not None
+ assert network.requests_log[0].path == expected_path
+ assert permissions[endpoint] is not None
def test_unbounded_endpoint_probes_a_small_one_with_the_same_scope(self) -> None:
with scripted_network(always(ScriptedResponse(status=403, json={}))) as network:
@@ -445,6 +555,8 @@ class TestGetEndpointPermissions:
("marketing_lists", {"page_size": "1"}),
("templates", {"page_size": "1", "generations": "legacy,dynamic"}),
("unsubscribe_groups", {}),
+ # /categories/stats requires `categories` and `start_date`.
+ ("category_stats", {"categories": "posthog", "start_date": "2026-08-07", "aggregated_by": "day"}),
# /messages requires `query`, so a probe without one risks the same masking 400.
(
"message_activity",