Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add proofpoint tap threat, campaign and people tables (#126784)

Tom Owers4fd7719e8f2e
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 679b87bae903..7cf66a363a04 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -9842,10 +9842,10 @@ Today (4): `clicks_blocked`, `clicks_permitted`, `messages_blocked`, `messages_d
Diffed against: <https://help.proofpoint.com/Threat_Insight_Dashboard/API_Documentation>
-- [ ] `Threat summary (GET /v2/threat/summary/{threatId})` — resolve threat identifiers from existing SIEM events into threat details and classifications (high)
-- [ ] `Campaign IDs and details (GET /v2/campaign/ids; GET /v2/campaign/{campaignId})` — resolve campaign identifiers in SIEM events and analyze related threat activity (high)
-- [ ] `Very Attacked People (GET /v2/people/vap)` — measure the vendor attack index and identify highly targeted users (high)
-- [ ] `Top clickers (GET /v2/people/top-clickers)` — rank risky clicking behavior with user identity and threat family breakdowns (medium)
+- [x] `Threat summary (GET /v2/threat/summary/{threatId})` — resolve threat identifiers from existing SIEM events into threat details and classifications (high)
+- [x] `Campaign IDs and details (GET /v2/campaign/ids; GET /v2/campaign/{campaignId})` — resolve campaign identifiers in SIEM events and analyze related threat activity (high)
+- [x] `Very Attacked People (GET /v2/people/vap)` — measure the vendor attack index and identify highly targeted users (high)
+- [x] `Top clickers (GET /v2/people/top-clickers)` — rank risky clicking behavior with user identity and threat family breakdowns (medium)
- [ ] `Forensics (GET /v2/forensics)` — analyze threat evidence and indicators of compromise linked to events (medium)
Note: The official API index links the People, Campaign, Threat, and Forensics references, whose GET operations were inspected. Threat summaries and forensics need IDs from existing events; campaign details need an ID listing or event fan-out. People results use rolling windows and quotas. SIEM aggregate endpoints duplicate existing tables, and URL decoding is excluded.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/canonical_descriptions.py
index 0ab59c02e5dd..bc13a4a184c8 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/canonical_descriptions.py
@@ -3,6 +3,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.can
)
from products.warehouse_sources.backend.temporal.data_imports.sources.proofpoint_tap.settings import API_DOCS_URL
+DOCS_ROOT = "https://help.proofpoint.com/Threat_Insight_Dashboard/API_Documentation"
+
CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"clicks_blocked": {
"description": "Blocked clicks on malicious URLs.",
@@ -44,4 +46,56 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"query_end_time": "End of the query interval that returned this event, added by PostHog.",
},
},
+ "threats": {
+ "description": "Threat summaries for the threats that SIEM events reference.",
+ "docs_url": f"{DOCS_ROOT}/Threat_API",
+ "columns": {
+ "id": "Unique threat identifier.",
+ "identifiedAt": "Time when Proofpoint identified the threat.",
+ "name": "Threat name, such as a URL or an attachment hash.",
+ "type": "Threat type, such as url or attachment.",
+ "category": "Threat category, such as malware or phish.",
+ "status": "Threat status, such as active or cleared.",
+ "severity": "Threat severity score from 0 to 1000.",
+ "attackSpread": "Number of other customers that received this threat.",
+ "families": "Threat families of the threat.",
+ "query_end_time": "End of the SIEM query interval that first returned this threat in a sync, added by PostHog.",
+ },
+ },
+ "campaigns": {
+ "description": "Campaigns updated in the queried interval, with their threats, actors and malware.",
+ "docs_url": f"{DOCS_ROOT}/Campaign_API",
+ "columns": {
+ "id": "Unique campaign identifier.",
+ "name": "Campaign name.",
+ "description": "Campaign description.",
+ "startDate": "Start date of the campaign.",
+ "lastUpdatedAt": "Time of the last update to the campaign.",
+ "campaignMembers": "Threats that belong to the campaign.",
+ "actors": "Threat actors associated with the campaign.",
+ "malware": "Malware families associated with the campaign.",
+ "techniques": "Attack techniques associated with the campaign.",
+ "query_end_time": "End of the campaign ID query interval that returned this campaign, added by PostHog.",
+ },
+ },
+ "very_attacked_people": {
+ "description": "Very Attacked People in your organization over the last 90 days, ranked by attack index.",
+ "docs_url": f"{DOCS_ROOT}/People_API",
+ "columns": {
+ "guid": "Unique identifier of the user, added by PostHog from the identity.",
+ "identity": "User identity: emails, name, department, location, title and VIP status.",
+ "threatStatistics": "Attack index of the user and the score of each threat family.",
+ "interval": "Time interval of the data, in ISO 8601 format.",
+ },
+ },
+ "top_clickers": {
+ "description": "Users who clicked the most malicious URLs over the last 90 days.",
+ "docs_url": f"{DOCS_ROOT}/People_API",
+ "columns": {
+ "guid": "Unique identifier of the user, added by PostHog from the identity.",
+ "identity": "User identity: emails, name, department, location, title and VIP status.",
+ "clickStatistics": "Click count of the user and the clicks for each threat family.",
+ "interval": "Time interval of the data, in ISO 8601 format.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/proofpoint_tap.py b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/proofpoint_tap.py
index 4c980a2180fe..4152e5c45ae4 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/proofpoint_tap.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/proofpoint_tap.py
@@ -1,6 +1,7 @@
-from collections.abc import Iterator
-from datetime import UTC, datetime
+from collections.abc import Callable, Iterator
+from datetime import UTC, datetime, timedelta
from typing import Any, ClassVar
+from urllib.parse import quote
from requests import HTTPError
@@ -12,6 +13,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
rest_api_resource,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.resource import Resource
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import Endpoint
from products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManager
from products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import schema_for_resource
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponse
@@ -21,11 +23,17 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_
from products.warehouse_sources.backend.temporal.data_imports.sources.proofpoint_tap.settings import (
AUTH_ERRORS,
BASE_URL,
+ CAMPAIGN_IDS_PATH,
+ CAMPAIGN_WINDOW,
ENDPOINTS,
MIN_WINDOW,
OVERLAP,
+ PEOPLE_WINDOW_DAYS,
RETENTION,
+ SIEM_ALL_PATH,
+ SIEM_SELECTORS,
WINDOW,
+ TapEndpoint,
)
@@ -37,8 +45,19 @@ class TapCursor:
@frozen
class TapResumeState:
- next_start: str
- end: str
+ next_start: str = ""
+ end: str = ""
+ # Lookup tables: the IDs already read in the window that starts at next_start.
+ done: tuple[str, ...] = ()
+ # People tables: the next page to read.
+ page: int = 1
+
+
+@frozen
+class TapWindowBounds:
+ start: datetime
+ end: datetime
+ resume: TapResumeState | None
def parse_timestamp(value: str | datetime) -> datetime:
@@ -50,42 +69,109 @@ def parse_timestamp(value: str | datetime) -> datetime:
def get_resource(
config: ProofpointTapSourceConfig,
- endpoint: str,
+ name: str,
+ path: str,
params: dict[str, Any],
team_id: int,
job_id: str,
api_version: str,
+ ignore_not_found: bool = False,
) -> Resource:
- settings = schema_for_resource(ENDPOINTS, endpoint)
+ endpoint: Endpoint = {
+ "path": path,
+ "params": params,
+ "data_selector": "$",
+ "data_selector_required": True,
+ }
+ if ignore_not_found:
+ endpoint["response_actions"] = [{"status_code": 404, "action": "ignore"}]
rest_config: RESTAPIConfig = {
"client": {
- "base_url": f"{BASE_URL}/{api_version}/siem/",
+ "base_url": f"{BASE_URL}/{api_version}/",
"auth": {"type": "http_basic", "username": config.service_principal, "password": config.secret},
"paginator": "single_page",
"allowed_hosts": [],
"allow_redirects": False,
"request_timeout": 60,
},
- "resources": [
- {
- "name": endpoint,
- "endpoint": {
- "path": settings.path,
- "params": {"format": "json", **params},
- "data_selector": "$",
- "data_selector_required": True,
- },
- }
- ],
+ "resources": [{"name": name, "endpoint": endpoint}],
}
return rest_api_resource(rest_config, team_id, job_id, None)
+def fetch_payload(
+ config: ProofpointTapSourceConfig,
+ inputs: SourceInputs,
+ path: str,
+ params: dict[str, Any],
+ api_version: str,
+ ignore_not_found: bool = False,
+) -> dict[str, Any] | None:
+ resource = get_resource(
+ config, inputs.schema_name, path, params, inputs.team_id, inputs.job_id, api_version, ignore_not_found
+ )
+ payloads = [payload for page in resource for payload in page]
+ if ignore_not_found and not payloads:
+ return None
+ if len(payloads) != 1 or not isinstance(payloads[0], dict):
+ raise ValueError("Proofpoint TAP returned an invalid response. Retry the sync.")
+ return payloads[0]
+
+
+def fetch_siem_window(
+ config: ProofpointTapSourceConfig,
+ inputs: SourceInputs,
+ path: str,
+ start: datetime,
+ window_end: datetime,
+ api_version: str,
+) -> tuple[datetime, dict[str, Any]]:
+ payload = fetch_payload(
+ config,
+ inputs,
+ path,
+ {"format": "json", "interval": f"{start.isoformat()}/{window_end.isoformat()}"},
+ api_version,
+ )
+ assert payload is not None
+ raw_end = payload.get("queryEndTime")
+ if not isinstance(raw_end, str):
+ raise ValueError("Proofpoint TAP returned an invalid response. Retry the sync.")
+ query_end = parse_timestamp(raw_end)
+ if query_end != window_end:
+ raise ValueError("Proofpoint TAP returned an invalid query end time. Retry the sync.")
+ return query_end, payload
+
+
+def dict_rows(value: Any) -> list[dict[str, Any]]:
+ if not isinstance(value, list) or any(not isinstance(row, dict) for row in value):
+ raise ValueError("Proofpoint TAP returned an invalid response. Retry the sync.")
+ return value
+
+
def validate_credentials(
config: ProofpointTapSourceConfig, team_id: int, schema_name: str | None, api_version: str
) -> tuple[bool, str | None]:
+ endpoint = ENDPOINTS.get(schema_name or "")
+ # The campaign and People APIs allow 50 requests a day, so a SIEM request checks the credentials.
+ if endpoint is None or endpoint.kind in ("campaigns", "people"):
+ path = ENDPOINTS["clicks_blocked"].path
+ elif endpoint.kind == "threats":
+ path = SIEM_ALL_PATH
+ else:
+ path = endpoint.path
try:
- list(get_resource(config, schema_name or "clicks_blocked", {"sinceSeconds": 30}, team_id, "", api_version))
+ list(
+ get_resource(
+ config,
+ schema_name or "clicks_blocked",
+ path,
+ {"format": "json", "sinceSeconds": 30},
+ team_id,
+ "",
+ api_version,
+ )
+ )
except HTTPError as error:
status = error.response.status_code if error.response is not None else None
if status == 403 and schema_name is None:
@@ -96,6 +182,191 @@ def validate_credentials(
return True, None
+def window_bounds(
+ inputs: SourceInputs,
+ manager: ResumableSourceManager[TapResumeState],
+ cursor_manager: SourceCursorManager[TapCursor],
+) -> TapWindowBounds:
+ now = datetime.now(UTC).replace(second=0, microsecond=0)
+ # Leave time for the first request before the oldest interval expires.
+ earliest = now - RETENTION + 2 * OVERLAP
+ end = now - OVERLAP
+ start = earliest
+ cursor = cursor_manager.load() if inputs.should_use_incremental_field else None
+ if cursor is not None:
+ start = max(earliest, parse_timestamp(cursor.query_end_time) - OVERLAP)
+ elif inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:
+ start = max(earliest, parse_timestamp(inputs.db_incremental_field_last_value) - OVERLAP)
+
+ resume = manager.load_state() if manager.can_resume() else None
+ if resume is not None:
+ start = max(earliest, parse_timestamp(resume.next_start))
+ end = min(end, parse_timestamp(resume.end))
+ return TapWindowBounds(start=start, end=end, resume=resume)
+
+
+def siem_items(
+ config: ProofpointTapSourceConfig,
+ inputs: SourceInputs,
+ settings: TapEndpoint,
+ manager: ResumableSourceManager[TapResumeState],
+ cursor_manager: SourceCursorManager[TapCursor],
+ api_version: str,
+) -> Iterator[list[dict[str, Any]]]:
+ bounds = window_bounds(inputs, manager, cursor_manager)
+ start, end, resume = bounds.start, bounds.end, bounds.resume
+ last_end: datetime | None = None
+ while end - start >= MIN_WINDOW:
+ window_end = min(start + WINDOW, end)
+ query_end, payload = fetch_siem_window(config, inputs, settings.path, start, window_end, api_version)
+ rows = dict_rows(payload.get(settings.selector))
+ if any(not row.get(settings.primary_key) for row in rows):
+ raise ValueError("Proofpoint TAP returned an event without its unique ID. Retry the sync.")
+
+ # Event timestamps can predate detection, so only the queried interval can advance the cursor.
+ for row in rows:
+ row["query_end_time"] = query_end
+ manager.save_state(TapResumeState(next_start=query_end.isoformat(), end=end.isoformat()))
+ if rows:
+ yield rows
+ manager.safe_point()
+ last_end = query_end
+ start = query_end
+
+ if last_end is not None:
+ cursor_manager.stage(TapCursor(query_end_time=last_end.isoformat()))
+ elif resume is not None and start >= end:
+ cursor_manager.stage(TapCursor(query_end_time=end.isoformat()))
+
+
+def siem_threat_ids(
+ config: ProofpointTapSourceConfig, inputs: SourceInputs, api_version: str
+) -> Callable[[datetime, datetime], Iterator[dict[str, Any]]]:
+ def list_window(start: datetime, window_end: datetime) -> Iterator[dict[str, Any]]:
+ _, payload = fetch_siem_window(config, inputs, SIEM_ALL_PATH, start, window_end, api_version)
+ for selector in SIEM_SELECTORS:
+ for event in dict_rows(payload.get(selector) or []):
+ if event.get("threatID"):
+ yield {"id": event["threatID"]}
+ for threat in event.get("threatsInfoMap") or []:
+ if isinstance(threat, dict) and threat.get("threatID"):
+ yield {"id": threat["threatID"]}
+
+ return list_window
+
+
+def campaign_ids(
+ config: ProofpointTapSourceConfig, inputs: SourceInputs, settings: TapEndpoint, api_version: str
+) -> Callable[[datetime, datetime], Iterator[dict[str, Any]]]:
+ page_size = settings.page_size or 100
+
+ def list_window(start: datetime, window_end: datetime) -> Iterator[dict[str, Any]]:
+ page = 1
+ while True:
+ payload = fetch_payload(
+ config,
+ inputs,
+ CAMPAIGN_IDS_PATH,
+ {"interval": f"{start.isoformat()}/{window_end.isoformat()}", "size": page_size, "page": page},
+ api_version,
+ )
+ assert payload is not None
+ campaigns = dict_rows(payload.get(settings.selector) or [])
+ yield from campaigns
+ if len(campaigns) < page_size:
+ return
+ page += 1
+
+ return list_window
+
+
+def lookup_items(
+ config: ProofpointTapSourceConfig,
+ inputs: SourceInputs,
+ settings: TapEndpoint,
+ manager: ResumableSourceManager[TapResumeState],
+ cursor_manager: SourceCursorManager[TapCursor],
+ api_version: str,
+ window: timedelta,
+ list_window: Callable[[datetime, datetime], Iterator[dict[str, Any]]],
+) -> Iterator[list[dict[str, Any]]]:
+ """Look up the details of every ID that a time window lists, one window at a time."""
+ bounds = window_bounds(inputs, manager, cursor_manager)
+ start, end, resume = bounds.start, bounds.end, bounds.resume
+ # A resumed run lists its window again and skips the IDs it already read there.
+ done = list(resume.done) if resume is not None else []
+ seen = set(done)
+ last_end: datetime | None = None
+ while end - start >= MIN_WINDOW:
+ window_end = min(start + window, end)
+ manager.save_state(TapResumeState(next_start=start.isoformat(), end=end.isoformat(), done=tuple(done)))
+ for entry in list_window(start, window_end):
+ item_id = entry.get("id")
+ if not isinstance(item_id, str) or not item_id:
+ raise ValueError("Proofpoint TAP returned an item without its unique ID. Retry the sync.")
+ if item_id in seen:
+ continue
+ seen.add(item_id)
+ # An ID that SIEM events still reference can be gone from the lookup API.
+ detail = fetch_payload(
+ config, inputs, settings.path.format(id=quote(item_id, safe="")), {}, api_version, ignore_not_found=True
+ )
+ if detail is not None and detail.get("id") != item_id:
+ raise ValueError("Proofpoint TAP returned details for a different item. Retry the sync.")
+ done.append(item_id)
+ manager.save_state(TapResumeState(next_start=start.isoformat(), end=end.isoformat(), done=tuple(done)))
+ if detail is None:
+ manager.safe_point()
+ continue
+ yield [{**entry, **detail, "query_end_time": window_end}]
+ manager.safe_point()
+ done = []
+ last_end = window_end
+ start = window_end
+
+ if last_end is not None:
+ cursor_manager.stage(TapCursor(query_end_time=last_end.isoformat()))
+ elif resume is not None and start >= end:
+ cursor_manager.stage(TapCursor(query_end_time=end.isoformat()))
+
+
+def people_items(
+ config: ProofpointTapSourceConfig,
+ inputs: SourceInputs,
+ settings: TapEndpoint,
+ manager: ResumableSourceManager[TapResumeState],
+ api_version: str,
+) -> Iterator[list[dict[str, Any]]]:
+ page_size = settings.page_size or 100
+ seen: set[str] = set()
+ # The endpoint allows 50 requests a day, so a retry after the quota resets continues at the next page.
+ resume = manager.load_state() if manager.can_resume() else None
+ page = resume.page if resume is not None else 1
+ while True:
+ payload = fetch_payload(
+ config, inputs, settings.path, {"window": PEOPLE_WINDOW_DAYS, "size": page_size, "page": page}, api_version
+ )
+ assert payload is not None
+ users = dict_rows(payload.get(settings.selector))
+ rows = []
+ for user in users:
+ identity = user.get("identity")
+ guid = identity.get("guid") if isinstance(identity, dict) else None
+ if not isinstance(guid, str) or not guid:
+ raise ValueError("Proofpoint TAP returned a user without its unique ID. Retry the sync.")
+ # Results are ranked, so a user can move to another page while the sync reads them.
+ if guid not in seen:
+ seen.add(guid)
+ rows.append({**user, "guid": guid, "interval": payload.get("interval")})
+ manager.save_state(TapResumeState(page=page + 1))
+ if rows:
+ yield rows
+ manager.safe_point()
+ if len(users) < page_size:
+ return
+ page += 1
+
+
def proofpoint_tap_source(
config: ProofpointTapSourceConfig,
inputs: SourceInputs,
@@ -106,73 +377,27 @@ def proofpoint_tap_source(
settings = schema_for_resource(ENDPOINTS, inputs.schema_name)
def items() -> Iterator[list[dict[str, Any]]]:
- now = datetime.now(UTC).replace(second=0, microsecond=0)
- # Leave time for the first request before the oldest interval expires.
- earliest = now - RETENTION + 2 * OVERLAP
- end = now - OVERLAP
- start = earliest
- cursor = cursor_manager.load() if inputs.should_use_incremental_field else None
- if cursor is not None:
- start = max(earliest, parse_timestamp(cursor.query_end_time) - OVERLAP)
- elif inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:
- start = max(earliest, parse_timestamp(inputs.db_incremental_field_last_value) - OVERLAP)
-
- resume = manager.load_state() if manager.can_resume() else None
- if resume is not None:
- start = max(earliest, parse_timestamp(resume.next_start))
- end = min(end, parse_timestamp(resume.end))
-
- last_end: datetime | None = None
- while end - start >= MIN_WINDOW:
- window_end = min(start + WINDOW, end)
- resource = get_resource(
- config,
- inputs.schema_name,
- {"interval": f"{start.isoformat()}/{window_end.isoformat()}"},
- inputs.team_id,
- inputs.job_id,
- api_version,
+ if settings.kind == "siem":
+ yield from siem_items(config, inputs, settings, manager, cursor_manager, api_version)
+ elif settings.kind == "threats":
+ list_window = siem_threat_ids(config, inputs, api_version)
+ yield from lookup_items(config, inputs, settings, manager, cursor_manager, api_version, WINDOW, list_window)
+ elif settings.kind == "campaigns":
+ list_window = campaign_ids(config, inputs, settings, api_version)
+ yield from lookup_items(
+ config, inputs, settings, manager, cursor_manager, api_version, CAMPAIGN_WINDOW, list_window
)
- payloads = [payload for page in resource for payload in page]
- if len(payloads) != 1 or not isinstance(payloads[0], dict):
- raise ValueError("Proofpoint TAP returned an invalid response. Retry the sync.")
- payload = payloads[0]
- raw_end = payload.get("queryEndTime")
- rows = payload.get(settings.selector)
- if (
- not isinstance(raw_end, str)
- or not isinstance(rows, list)
- or any(not isinstance(row, dict) for row in rows)
- ):
- raise ValueError("Proofpoint TAP returned an invalid response. Retry the sync.")
- query_end = parse_timestamp(raw_end)
- if query_end != window_end:
- raise ValueError("Proofpoint TAP returned an invalid query end time. Retry the sync.")
- if any(not row.get(settings.primary_key) for row in rows):
- raise ValueError("Proofpoint TAP returned an event without its unique ID. Retry the sync.")
-
- # Event timestamps can predate detection, so only the queried interval can advance the cursor.
- for row in rows:
- row["query_end_time"] = query_end
- manager.save_state(TapResumeState(next_start=query_end.isoformat(), end=end.isoformat()))
- if rows:
- yield rows
- manager.safe_point()
- last_end = query_end
- start = query_end
-
- if last_end is not None:
- cursor_manager.stage(TapCursor(query_end_time=last_end.isoformat()))
- elif resume is not None and start >= end:
- cursor_manager.stage(TapCursor(query_end_time=end.isoformat()))
+ else:
+ yield from people_items(config, inputs, settings, manager, api_version)
+ partitioned = settings.partition_key is not None
return SourceResponse(
name=inputs.schema_name,
items=items,
primary_keys=[settings.primary_key],
- partition_keys=[settings.event_time],
- partition_mode="datetime",
- partition_format="month",
- sort_mode="desc",
+ partition_keys=[settings.partition_key] if settings.partition_key else None,
+ partition_mode="datetime" if partitioned else None,
+ partition_format="month" if partitioned else None,
+ sort_mode="desc" if settings.kind == "siem" else None,
on_complete=manager.clear_state,
)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/settings.py
index 4ea90b419cf8..7c9d03c7aae3 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/settings.py
@@ -1,4 +1,5 @@
from datetime import timedelta
+from typing import Literal
from posthog.dataclasses import frozen
@@ -13,30 +14,74 @@ MIN_WINDOW = timedelta(seconds=30)
OVERLAP = timedelta(minutes=1)
+# Campaign ID listings accept at most one day per request and allow 50 requests a day.
+CAMPAIGN_WINDOW = timedelta(days=1)
+CAMPAIGN_PAGE_SIZE = 200
+# The People API accepts 14, 30 or 90 days and allows 50 requests a day.
+PEOPLE_WINDOW_DAYS = 90
+
+
@frozen
class TapEndpoint:
+ kind: Literal["siem", "threats", "campaigns", "people"]
path: str
selector: str
primary_key: str
- event_time: str
+ partition_key: str | None = None
+ page_size: int | None = None
ENDPOINTS = {
"clicks_blocked": TapEndpoint(
- path="clicks/blocked", selector="clicksBlocked", primary_key="id", event_time="clickTime"
+ kind="siem", path="siem/clicks/blocked", selector="clicksBlocked", primary_key="id", partition_key="clickTime"
),
"clicks_permitted": TapEndpoint(
- path="clicks/permitted", selector="clicksPermitted", primary_key="id", event_time="clickTime"
+ kind="siem",
+ path="siem/clicks/permitted",
+ selector="clicksPermitted",
+ primary_key="id",
+ partition_key="clickTime",
),
"messages_blocked": TapEndpoint(
- path="messages/blocked", selector="messagesBlocked", primary_key="GUID", event_time="messageTime"
+ kind="siem",
+ path="siem/messages/blocked",
+ selector="messagesBlocked",
+ primary_key="GUID",
+ partition_key="messageTime",
),
"messages_delivered": TapEndpoint(
- path="messages/delivered", selector="messagesDelivered", primary_key="GUID", event_time="messageTime"
+ kind="siem",
+ path="siem/messages/delivered",
+ selector="messagesDelivered",
+ primary_key="GUID",
+ partition_key="messageTime",
+ ),
+ "threats": TapEndpoint(
+ kind="threats", path="threat/summary/{id}", selector="", primary_key="id", partition_key="identifiedAt"
+ ),
+ "campaigns": TapEndpoint(
+ kind="campaigns",
+ path="campaign/{id}",
+ selector="campaigns",
+ primary_key="id",
+ partition_key="startDate",
+ page_size=CAMPAIGN_PAGE_SIZE,
+ ),
+ "very_attacked_people": TapEndpoint(
+ kind="people", path="people/vap", selector="users", primary_key="guid", page_size=1000
+ ),
+ "top_clickers": TapEndpoint(
+ kind="people", path="people/top-clickers", selector="users", primary_key="guid", page_size=200
),
}
-INCREMENTAL_FIELDS = {name: [incremental_field("query_end_time")] for name in ENDPOINTS}
+SIEM_ALL_PATH = "siem/all"
+CAMPAIGN_IDS_PATH = "campaign/ids"
+SIEM_SELECTORS = ("clicksBlocked", "clicksPermitted", "messagesBlocked", "messagesDelivered")
+
+INCREMENTAL_FIELDS = {
+ name: [incremental_field("query_end_time")] for name, endpoint in ENDPOINTS.items() if endpoint.kind != "people"
+}
AUTH_ERRORS = {
401: "Proofpoint TAP authentication failed. Check your service principal and secret.",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/tests/test_proofpoint_tap.py b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/tests/test_proofpoint_tap.py
index b36ea0c52e4e..096b7dc74376 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/tests/test_proofpoint_tap.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/tests/test_proofpoint_tap.py
@@ -1,3 +1,4 @@
+import re
import json
from collections.abc import Iterable
from datetime import UTC, datetime, timedelta
@@ -31,7 +32,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.proofpoint
from products.warehouse_sources.backend.temporal.data_imports.sources.proofpoint_tap.source import ProofpointTapSource
NOW = datetime(2026, 1, 8, 12, tzinfo=UTC)
-BASE = "https://tap-api-v2.proofpoint.com/v2/siem"
+ROOT = "https://tap-api-v2.proofpoint.com/v2"
+BASE = f"{ROOT}/siem"
CONFIG = ProofpointTapSourceConfig.from_dict({"service_principal": "example-principal", "secret": "example-secret"})
@@ -211,15 +213,19 @@ def test_error_classification_preserves_checkpoint(status: int) -> None:
@pytest.mark.parametrize(
- ("status", "schema", "valid", "message"),
+ ("status", "schema", "valid", "message", "path"),
[
- (403, None, True, None),
- (403, "messages_blocked", False, "Check access in the TAP dashboard"),
+ (403, None, True, None, "/v2/siem/clicks/blocked"),
+ (403, "messages_blocked", False, "Check access in the TAP dashboard", "/v2/siem/messages/blocked"),
+ (200, "threats", True, None, "/v2/siem/all"),
+ (200, "campaigns", True, None, "/v2/siem/clicks/blocked"),
+ (200, "top_clickers", True, None, "/v2/siem/clicks/blocked"),
],
)
-def test_credential_probe_status_mapping(status: int, schema: str | None, valid: bool, message: str | None) -> None:
- selector = "messagesBlocked" if schema else "clicksBlocked"
- response = ScriptedResponse(status=status, json={"queryEndTime": NOW.isoformat(), selector: []})
+def test_credential_probe_status_mapping(
+ status: int, schema: str | None, valid: bool, message: str | None, path: str
+) -> None:
+ response = ScriptedResponse(status=status, json={"queryEndTime": NOW.isoformat()})
with scripted_network([response]) as network:
result, error = ProofpointTapSource().validate_credentials(CONFIG, 1, schema)
assert result is valid
@@ -229,7 +235,7 @@ def test_credential_probe_status_mapping(status: int, schema: str | None, valid:
assert error is not None and message in error
assert len(network.requests_log) == 1
request = network.requests_log[0]
- assert request.path == ("/v2/siem/messages/blocked" if schema else "/v2/siem/clicks/blocked")
+ assert request.path == path
assert request.query == {"format": ("json",), "sinceSeconds": ("30",)}
@@ -237,3 +243,152 @@ def test_unexpected_probe_error_is_not_bad_credentials() -> None:
with scripted_network([ScriptedResponse(status=400, json={})]):
with pytest.raises(HTTPError):
ProofpointTapSource().validate_credentials(CONFIG, 1)
+
+
+ ("resume", "lookups", "saved"),
+ [
+ (
+ None,
+ ["threat-a", "gone", "threat-b"],
+ [
+ ("2026-01-08T09:59:00+00:00", ()),
+ ("2026-01-08T09:59:00+00:00", ("threat-a",)),
+ ("2026-01-08T09:59:00+00:00", ("threat-a", "gone")),
+ ("2026-01-08T09:59:00+00:00", ("threat-a", "gone", "threat-b")),
+ ("2026-01-08T10:59:00+00:00", ()),
+ ],
+ ),
+ (
+ TapResumeState(next_start="2026-01-08T09:59:00+00:00", end="2026-01-08T11:59:00+00:00", done=("threat-a",)),
+ ["gone", "threat-b"],
+ [
+ ("2026-01-08T09:59:00+00:00", ("threat-a",)),
+ ("2026-01-08T09:59:00+00:00", ("threat-a", "gone")),
+ ("2026-01-08T09:59:00+00:00", ("threat-a", "gone", "threat-b")),
+ ("2026-01-08T10:59:00+00:00", ()),
+ ],
+ ),
+ ],
+)
+@time_machine.travel(NOW, tick=False)
+def test_threats_resolve_each_siem_threat_once_and_skip_missing(
+ resume: TapResumeState | None, lookups: list[str], saved: list[tuple[str, tuple[str, ...]]]
+) -> None:
+ def siem_all(request: PreparedRequest) -> tuple[int, dict[str, str], str]:
+ end = query(request)["interval"][0].split("/")[1]
+ payload = {
+ "queryEndTime": end,
+ "clicksBlocked": [{"id": "click-a", "threatID": "threat-a"}],
+ "clicksPermitted": [],
+ "messagesBlocked": [
+ {"GUID": "message-a", "threatsInfoMap": [{"threatID": "threat-a"}, {"threatID": "gone"}]}
+ ],
+ "messagesDelivered": [{"GUID": "message-b", "threatsInfoMap": [{"threatID": "threat-b"}]}],
+ }
+ return 200, {"Content-Type": "application/json"}, json.dumps(payload)
+
+ responses.add_callback(responses.GET, f"{BASE}/all", callback=siem_all)
+ for threat_id in ("threat-a", "threat-b"):
+ responses.get(
+ f"{ROOT}/threat/summary/{threat_id}", json={"id": threat_id, "identifiedAt": "2026-01-01T00:00:00Z"}
+ )
+ responses.get(f"{ROOT}/threat/summary/gone", status=404, json={})
+
+ harness = Harness(endpoint="threats", resume=resume)
+ rows = harness.rows()
+
+ expected_rows = [threat_id for threat_id in lookups if threat_id != "gone"]
+ assert [(row["id"], row["query_end_time"]) for row in rows] == [
+ (threat_id, datetime(2026, 1, 8, 10, 59, tzinfo=UTC)) for threat_id in expected_rows
+ ]
+ assert [urlparse(call.request.url or "").path for call in responses.calls] == [
+ "/v2/siem/all",
+ *[f"/v2/threat/summary/{threat_id}" for threat_id in lookups],
+ "/v2/siem/all",
+ ]
+ assert [(call.args[0].next_start, call.args[0].done) for call in harness.manager.save_state.call_args_list] == saved
+ assert harness.response.primary_keys == ["id"]
+ assert harness.response.sort_mode is None
+ assert harness.cursor.staged == TapCursor(query_end_time="2026-01-08T11:59:00+00:00")
+
+
+@time_machine.travel(NOW, tick=False)
+def test_campaigns_page_through_one_day_windows_and_merge_details() -> None:
+ full_page = [{"id": f"campaign-{i}", "lastUpdatedAt": "2026-01-07T00:00:00Z", "notable": False} for i in range(200)]
+ responses.get(f"{ROOT}/campaign/ids", json={"campaigns": full_page})
+ responses.get(f"{ROOT}/campaign/ids", json={"campaigns": [{"id": "campaign-0"}, {"id": "campaign-x"}]})
+ responses.get(f"{ROOT}/campaign/ids", json={"campaigns": []})
+ responses.add_callback(
+ responses.GET,
+ re.compile(f"{ROOT}/campaign/campaign-.+"),
+ callback=lambda request: (
+ 200,
+ {},
+ json.dumps({"id": urlparse(request.url or "").path.rsplit("/", 1)[1], "name": "Example"}),
+ ),
+ )
+
+ harness = Harness(endpoint="campaigns", watermark=NOW - timedelta(days=1, hours=1))
+ rows = harness.rows()
+
+ listings = [
+ query(call.request) for call in responses.calls if call.request.url and "/campaign/ids" in call.request.url
+ ]
+ assert [(q["interval"][0], q["page"], q["size"]) for q in listings] == [
+ ("2026-01-07T10:59:00+00:00/2026-01-08T10:59:00+00:00", ["1"], ["200"]),
+ ("2026-01-07T10:59:00+00:00/2026-01-08T10:59:00+00:00", ["2"], ["200"]),
+ ("2026-01-08T10:59:00+00:00/2026-01-08T11:59:00+00:00", ["1"], ["200"]),
+ ]
+ assert len(rows) == 201
+ assert rows[0] == {
+ "id": "campaign-0",
+ "lastUpdatedAt": "2026-01-07T00:00:00Z",
+ "notable": False,
+ "name": "Example",
+ "query_end_time": datetime(2026, 1, 8, 10, 59, tzinfo=UTC),
+ }
+ assert rows[-1]["id"] == "campaign-x"
+ assert harness.cursor.staged == TapCursor(query_end_time="2026-01-08T11:59:00+00:00")
+
+
+ ("endpoint", "path", "size", "first_page"),
+ [
+ ("very_attacked_people", "people/vap", 1000, 1),
+ ("top_clickers", "people/top-clickers", 200, 1),
+ ("top_clickers", "people/top-clickers", 200, 3),
+ ],
+)
+def test_people_walk_pages_and_key_rows_by_identity(endpoint: str, path: str, size: int, first_page: int) -> None:
+ def user(guid: str) -> dict[str, Any]:
+ return {"identity": {"guid": guid, "emails": [f"{guid}@example.com"]}, "threatStatistics": {"attackIndex": 1}}
+
+ interval = "2025-10-10T00:00:00Z/2026-01-08T00:00:00Z"
+ responses.get(f"{ROOT}/{path}", json={"users": [user(f"user-{i}") for i in range(size)], "interval": interval})
+ responses.get(f"{ROOT}/{path}", json={"users": [user("user-0"), user("user-last")], "interval": interval})
+
+ resume = TapResumeState(page=first_page) if first_page > 1 else None
+ harness = Harness(endpoint=endpoint, incremental=False, watermark=None, resume=resume)
+ rows = harness.rows()
+
+ assert [query(call.request) for call in responses.calls] == [
+ {"window": ["90"], "size": [str(size)], "page": [str(first_page)]},
+ {"window": ["90"], "size": [str(size)], "page": [str(first_page + 1)]},
+ ]
+ assert [call.args[0].page for call in harness.manager.save_state.call_args_list] == [first_page + 1, first_page + 2]
+ assert len(rows) == size + 1
+ assert rows[-1]["guid"] == "user-last"
+ assert rows[-1]["interval"] == interval
+ assert harness.response.primary_keys == ["guid"]
+ assert harness.response.partition_keys is None
+
+
+def test_people_user_without_guid_fails() -> None:
+ responses.get(f"{ROOT}/people/vap", json={"users": [{"identity": {}}]})
+ with pytest.raises(ValueError):
+ Harness(endpoint="very_attacked_people", incremental=False, watermark=None).rows()