:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add proofpoint tap threat, campaign and people tables (#126784)
Tom Owers4fd7719e8f2e
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 679b87bae903..7cf66a363a04 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -9842,10 +9842,10 @@ Today (4): `clicks_blocked`, `clicks_permitted`, `messages_blocked`, `messages_dDiffed against: <https://help.proofpoint.com/Threat_Insight_Dashboard/API_Documentation>-- [ ] `Threat summary (GET /v2/threat/summary/{threatId})` — resolve threat identifiers from existing SIEM events into threat details and classifications (high)-- [ ] `Campaign IDs and details (GET /v2/campaign/ids; GET /v2/campaign/{campaignId})` — resolve campaign identifiers in SIEM events and analyze related threat activity (high)-- [ ] `Very Attacked People (GET /v2/people/vap)` — measure the vendor attack index and identify highly targeted users (high)-- [ ] `Top clickers (GET /v2/people/top-clickers)` — rank risky clicking behavior with user identity and threat family breakdowns (medium)+- [x] `Threat summary (GET /v2/threat/summary/{threatId})` — resolve threat identifiers from existing SIEM events into threat details and classifications (high)+- [x] `Campaign IDs and details (GET /v2/campaign/ids; GET /v2/campaign/{campaignId})` — resolve campaign identifiers in SIEM events and analyze related threat activity (high)+- [x] `Very Attacked People (GET /v2/people/vap)` — measure the vendor attack index and identify highly targeted users (high)+- [x] `Top clickers (GET /v2/people/top-clickers)` — rank risky clicking behavior with user identity and threat family breakdowns (medium)- [ ] `Forensics (GET /v2/forensics)` — analyze threat evidence and indicators of compromise linked to events (medium)Note: The official API index links the People, Campaign, Threat, and Forensics references, whose GET operations were inspected. Threat summaries and forensics need IDs from existing events; campaign details need an ID listing or event fan-out. People results use rolling windows and quotas. SIEM aggregate endpoints duplicate existing tables, and URL decoding is excluded.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/canonical_descriptions.pyindex 0ab59c02e5dd..bc13a4a184c8 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/canonical_descriptions.py@@ -3,6 +3,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.can)from products.warehouse_sources.backend.temporal.data_imports.sources.proofpoint_tap.settings import API_DOCS_URL+DOCS_ROOT = "https://help.proofpoint.com/Threat_Insight_Dashboard/API_Documentation"+CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"clicks_blocked": {"description": "Blocked clicks on malicious URLs.",@@ -44,4 +46,56 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"query_end_time": "End of the query interval that returned this event, added by PostHog.",},},+ "threats": {+ "description": "Threat summaries for the threats that SIEM events reference.",+ "docs_url": f"{DOCS_ROOT}/Threat_API",+ "columns": {+ "id": "Unique threat identifier.",+ "identifiedAt": "Time when Proofpoint identified the threat.",+ "name": "Threat name, such as a URL or an attachment hash.",+ "type": "Threat type, such as url or attachment.",+ "category": "Threat category, such as malware or phish.",+ "status": "Threat status, such as active or cleared.",+ "severity": "Threat severity score from 0 to 1000.",+ "attackSpread": "Number of other customers that received this threat.",+ "families": "Threat families of the threat.",+ "query_end_time": "End of the SIEM query interval that first returned this threat in a sync, added by PostHog.",+ },+ },+ "campaigns": {+ "description": "Campaigns updated in the queried interval, with their threats, actors and malware.",+ "docs_url": f"{DOCS_ROOT}/Campaign_API",+ "columns": {+ "id": "Unique campaign identifier.",+ "name": "Campaign name.",+ "description": "Campaign description.",+ "startDate": "Start date of the campaign.",+ "lastUpdatedAt": "Time of the last update to the campaign.",+ "campaignMembers": "Threats that belong to the campaign.",+ "actors": "Threat actors associated with the campaign.",+ "malware": "Malware families associated with the campaign.",+ "techniques": "Attack techniques associated with the campaign.",+ "query_end_time": "End of the campaign ID query interval that returned this campaign, added by PostHog.",+ },+ },+ "very_attacked_people": {+ "description": "Very Attacked People in your organization over the last 90 days, ranked by attack index.",+ "docs_url": f"{DOCS_ROOT}/People_API",+ "columns": {+ "guid": "Unique identifier of the user, added by PostHog from the identity.",+ "identity": "User identity: emails, name, department, location, title and VIP status.",+ "threatStatistics": "Attack index of the user and the score of each threat family.",+ "interval": "Time interval of the data, in ISO 8601 format.",+ },+ },+ "top_clickers": {+ "description": "Users who clicked the most malicious URLs over the last 90 days.",+ "docs_url": f"{DOCS_ROOT}/People_API",+ "columns": {+ "guid": "Unique identifier of the user, added by PostHog from the identity.",+ "identity": "User identity: emails, name, department, location, title and VIP status.",+ "clickStatistics": "Click count of the user and the clicks for each threat family.",+ "interval": "Time interval of the data, in ISO 8601 format.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/proofpoint_tap.py b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/proofpoint_tap.pyindex 4c980a2180fe..4152e5c45ae4 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/proofpoint_tap.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/proofpoint_tap.py@@ -1,6 +1,7 @@-from collections.abc import Iterator-from datetime import UTC, datetime+from collections.abc import Callable, Iterator+from datetime import UTC, datetime, timedeltafrom typing import Any, ClassVar+from urllib.parse import quotefrom requests import HTTPError@@ -12,6 +13,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resrest_api_resource,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.resource import Resource+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import Endpointfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import schema_for_resourcefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponse@@ -21,11 +23,17 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_from products.warehouse_sources.backend.temporal.data_imports.sources.proofpoint_tap.settings import (AUTH_ERRORS,BASE_URL,+ CAMPAIGN_IDS_PATH,+ CAMPAIGN_WINDOW,ENDPOINTS,MIN_WINDOW,OVERLAP,+ PEOPLE_WINDOW_DAYS,RETENTION,+ SIEM_ALL_PATH,+ SIEM_SELECTORS,WINDOW,+ TapEndpoint,)@@ -37,8 +45,19 @@ class TapCursor:@frozenclass TapResumeState:- next_start: str- end: str+ next_start: str = ""+ end: str = ""+ # Lookup tables: the IDs already read in the window that starts at next_start.+ done: tuple[str, ...] = ()+ # People tables: the next page to read.+ page: int = 1+++@frozen+class TapWindowBounds:+ start: datetime+ end: datetime+ resume: TapResumeState | Nonedef parse_timestamp(value: str | datetime) -> datetime:@@ -50,42 +69,109 @@ def parse_timestamp(value: str | datetime) -> datetime:def get_resource(config: ProofpointTapSourceConfig,- endpoint: str,+ name: str,+ path: str,params: dict[str, Any],team_id: int,job_id: str,api_version: str,+ ignore_not_found: bool = False,) -> Resource:- settings = schema_for_resource(ENDPOINTS, endpoint)+ endpoint: Endpoint = {+ "path": path,+ "params": params,+ "data_selector": "$",+ "data_selector_required": True,+ }+ if ignore_not_found:+ endpoint["response_actions"] = [{"status_code": 404, "action": "ignore"}]rest_config: RESTAPIConfig = {"client": {- "base_url": f"{BASE_URL}/{api_version}/siem/",+ "base_url": f"{BASE_URL}/{api_version}/","auth": {"type": "http_basic", "username": config.service_principal, "password": config.secret},"paginator": "single_page","allowed_hosts": [],"allow_redirects": False,"request_timeout": 60,},- "resources": [- {- "name": endpoint,- "endpoint": {- "path": settings.path,- "params": {"format": "json", **params},- "data_selector": "$",- "data_selector_required": True,- },- }- ],+ "resources": [{"name": name, "endpoint": endpoint}],}return rest_api_resource(rest_config, team_id, job_id, None)+def fetch_payload(+ config: ProofpointTapSourceConfig,+ inputs: SourceInputs,+ path: str,+ params: dict[str, Any],+ api_version: str,+ ignore_not_found: bool = False,+) -> dict[str, Any] | None:+ resource = get_resource(+ config, inputs.schema_name, path, params, inputs.team_id, inputs.job_id, api_version, ignore_not_found+ )+ payloads = [payload for page in resource for payload in page]+ if ignore_not_found and not payloads:+ return None+ if len(payloads) != 1 or not isinstance(payloads[0], dict):+ raise ValueError("Proofpoint TAP returned an invalid response. Retry the sync.")+ return payloads[0]+++def fetch_siem_window(+ config: ProofpointTapSourceConfig,+ inputs: SourceInputs,+ path: str,+ start: datetime,+ window_end: datetime,+ api_version: str,+) -> tuple[datetime, dict[str, Any]]:+ payload = fetch_payload(+ config,+ inputs,+ path,+ {"format": "json", "interval": f"{start.isoformat()}/{window_end.isoformat()}"},+ api_version,+ )+ assert payload is not None+ raw_end = payload.get("queryEndTime")+ if not isinstance(raw_end, str):+ raise ValueError("Proofpoint TAP returned an invalid response. Retry the sync.")+ query_end = parse_timestamp(raw_end)+ if query_end != window_end:+ raise ValueError("Proofpoint TAP returned an invalid query end time. Retry the sync.")+ return query_end, payload+++def dict_rows(value: Any) -> list[dict[str, Any]]:+ if not isinstance(value, list) or any(not isinstance(row, dict) for row in value):+ raise ValueError("Proofpoint TAP returned an invalid response. Retry the sync.")+ return value++def validate_credentials(config: ProofpointTapSourceConfig, team_id: int, schema_name: str | None, api_version: str) -> tuple[bool, str | None]:+ endpoint = ENDPOINTS.get(schema_name or "")+ # The campaign and People APIs allow 50 requests a day, so a SIEM request checks the credentials.+ if endpoint is None or endpoint.kind in ("campaigns", "people"):+ path = ENDPOINTS["clicks_blocked"].path+ elif endpoint.kind == "threats":+ path = SIEM_ALL_PATH+ else:+ path = endpoint.pathtry:- list(get_resource(config, schema_name or "clicks_blocked", {"sinceSeconds": 30}, team_id, "", api_version))+ list(+ get_resource(+ config,+ schema_name or "clicks_blocked",+ path,+ {"format": "json", "sinceSeconds": 30},+ team_id,+ "",+ api_version,+ )+ )except HTTPError as error:status = error.response.status_code if error.response is not None else Noneif status == 403 and schema_name is None:@@ -96,6 +182,191 @@ def validate_credentials(return True, None+def window_bounds(+ inputs: SourceInputs,+ manager: ResumableSourceManager[TapResumeState],+ cursor_manager: SourceCursorManager[TapCursor],+) -> TapWindowBounds:+ now = datetime.now(UTC).replace(second=0, microsecond=0)+ # Leave time for the first request before the oldest interval expires.+ earliest = now - RETENTION + 2 * OVERLAP+ end = now - OVERLAP+ start = earliest+ cursor = cursor_manager.load() if inputs.should_use_incremental_field else None+ if cursor is not None:+ start = max(earliest, parse_timestamp(cursor.query_end_time) - OVERLAP)+ elif inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:+ start = max(earliest, parse_timestamp(inputs.db_incremental_field_last_value) - OVERLAP)++ resume = manager.load_state() if manager.can_resume() else None+ if resume is not None:+ start = max(earliest, parse_timestamp(resume.next_start))+ end = min(end, parse_timestamp(resume.end))+ return TapWindowBounds(start=start, end=end, resume=resume)+++def siem_items(+ config: ProofpointTapSourceConfig,+ inputs: SourceInputs,+ settings: TapEndpoint,+ manager: ResumableSourceManager[TapResumeState],+ cursor_manager: SourceCursorManager[TapCursor],+ api_version: str,+) -> Iterator[list[dict[str, Any]]]:+ bounds = window_bounds(inputs, manager, cursor_manager)+ start, end, resume = bounds.start, bounds.end, bounds.resume+ last_end: datetime | None = None+ while end - start >= MIN_WINDOW:+ window_end = min(start + WINDOW, end)+ query_end, payload = fetch_siem_window(config, inputs, settings.path, start, window_end, api_version)+ rows = dict_rows(payload.get(settings.selector))+ if any(not row.get(settings.primary_key) for row in rows):+ raise ValueError("Proofpoint TAP returned an event without its unique ID. Retry the sync.")++ # Event timestamps can predate detection, so only the queried interval can advance the cursor.+ for row in rows:+ row["query_end_time"] = query_end+ manager.save_state(TapResumeState(next_start=query_end.isoformat(), end=end.isoformat()))+ if rows:+ yield rows+ manager.safe_point()+ last_end = query_end+ start = query_end++ if last_end is not None:+ cursor_manager.stage(TapCursor(query_end_time=last_end.isoformat()))+ elif resume is not None and start >= end:+ cursor_manager.stage(TapCursor(query_end_time=end.isoformat()))+++def siem_threat_ids(+ config: ProofpointTapSourceConfig, inputs: SourceInputs, api_version: str+) -> Callable[[datetime, datetime], Iterator[dict[str, Any]]]:+ def list_window(start: datetime, window_end: datetime) -> Iterator[dict[str, Any]]:+ _, payload = fetch_siem_window(config, inputs, SIEM_ALL_PATH, start, window_end, api_version)+ for selector in SIEM_SELECTORS:+ for event in dict_rows(payload.get(selector) or []):+ if event.get("threatID"):+ yield {"id": event["threatID"]}+ for threat in event.get("threatsInfoMap") or []:+ if isinstance(threat, dict) and threat.get("threatID"):+ yield {"id": threat["threatID"]}++ return list_window+++def campaign_ids(+ config: ProofpointTapSourceConfig, inputs: SourceInputs, settings: TapEndpoint, api_version: str+) -> Callable[[datetime, datetime], Iterator[dict[str, Any]]]:+ page_size = settings.page_size or 100++ def list_window(start: datetime, window_end: datetime) -> Iterator[dict[str, Any]]:+ page = 1+ while True:+ payload = fetch_payload(+ config,+ inputs,+ CAMPAIGN_IDS_PATH,+ {"interval": f"{start.isoformat()}/{window_end.isoformat()}", "size": page_size, "page": page},+ api_version,+ )+ assert payload is not None+ campaigns = dict_rows(payload.get(settings.selector) or [])+ yield from campaigns+ if len(campaigns) < page_size:+ return+ page += 1++ return list_window+++def lookup_items(+ config: ProofpointTapSourceConfig,+ inputs: SourceInputs,+ settings: TapEndpoint,+ manager: ResumableSourceManager[TapResumeState],+ cursor_manager: SourceCursorManager[TapCursor],+ api_version: str,+ window: timedelta,+ list_window: Callable[[datetime, datetime], Iterator[dict[str, Any]]],+) -> Iterator[list[dict[str, Any]]]:+ """Look up the details of every ID that a time window lists, one window at a time."""+ bounds = window_bounds(inputs, manager, cursor_manager)+ start, end, resume = bounds.start, bounds.end, bounds.resume+ # A resumed run lists its window again and skips the IDs it already read there.+ done = list(resume.done) if resume is not None else []+ seen = set(done)+ last_end: datetime | None = None+ while end - start >= MIN_WINDOW:+ window_end = min(start + window, end)+ manager.save_state(TapResumeState(next_start=start.isoformat(), end=end.isoformat(), done=tuple(done)))+ for entry in list_window(start, window_end):+ item_id = entry.get("id")+ if not isinstance(item_id, str) or not item_id:+ raise ValueError("Proofpoint TAP returned an item without its unique ID. Retry the sync.")+ if item_id in seen:+ continue+ seen.add(item_id)+ # An ID that SIEM events still reference can be gone from the lookup API.+ detail = fetch_payload(+ config, inputs, settings.path.format(id=quote(item_id, safe="")), {}, api_version, ignore_not_found=True+ )+ if detail is not None and detail.get("id") != item_id:+ raise ValueError("Proofpoint TAP returned details for a different item. Retry the sync.")+ done.append(item_id)+ manager.save_state(TapResumeState(next_start=start.isoformat(), end=end.isoformat(), done=tuple(done)))+ if detail is None:+ manager.safe_point()+ continue+ yield [{**entry, **detail, "query_end_time": window_end}]+ manager.safe_point()+ done = []+ last_end = window_end+ start = window_end++ if last_end is not None:+ cursor_manager.stage(TapCursor(query_end_time=last_end.isoformat()))+ elif resume is not None and start >= end:+ cursor_manager.stage(TapCursor(query_end_time=end.isoformat()))+++def people_items(+ config: ProofpointTapSourceConfig,+ inputs: SourceInputs,+ settings: TapEndpoint,+ manager: ResumableSourceManager[TapResumeState],+ api_version: str,+) -> Iterator[list[dict[str, Any]]]:+ page_size = settings.page_size or 100+ seen: set[str] = set()+ # The endpoint allows 50 requests a day, so a retry after the quota resets continues at the next page.+ resume = manager.load_state() if manager.can_resume() else None+ page = resume.page if resume is not None else 1+ while True:+ payload = fetch_payload(+ config, inputs, settings.path, {"window": PEOPLE_WINDOW_DAYS, "size": page_size, "page": page}, api_version+ )+ assert payload is not None+ users = dict_rows(payload.get(settings.selector))+ rows = []+ for user in users:+ identity = user.get("identity")+ guid = identity.get("guid") if isinstance(identity, dict) else None+ if not isinstance(guid, str) or not guid:+ raise ValueError("Proofpoint TAP returned a user without its unique ID. Retry the sync.")+ # Results are ranked, so a user can move to another page while the sync reads them.+ if guid not in seen:+ seen.add(guid)+ rows.append({**user, "guid": guid, "interval": payload.get("interval")})+ manager.save_state(TapResumeState(page=page + 1))+ if rows:+ yield rows+ manager.safe_point()+ if len(users) < page_size:+ return+ page += 1++def proofpoint_tap_source(config: ProofpointTapSourceConfig,inputs: SourceInputs,@@ -106,73 +377,27 @@ def proofpoint_tap_source(settings = schema_for_resource(ENDPOINTS, inputs.schema_name)def items() -> Iterator[list[dict[str, Any]]]:- now = datetime.now(UTC).replace(second=0, microsecond=0)- # Leave time for the first request before the oldest interval expires.- earliest = now - RETENTION + 2 * OVERLAP- end = now - OVERLAP- start = earliest- cursor = cursor_manager.load() if inputs.should_use_incremental_field else None- if cursor is not None:- start = max(earliest, parse_timestamp(cursor.query_end_time) - OVERLAP)- elif inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:- start = max(earliest, parse_timestamp(inputs.db_incremental_field_last_value) - OVERLAP)-- resume = manager.load_state() if manager.can_resume() else None- if resume is not None:- start = max(earliest, parse_timestamp(resume.next_start))- end = min(end, parse_timestamp(resume.end))-- last_end: datetime | None = None- while end - start >= MIN_WINDOW:- window_end = min(start + WINDOW, end)- resource = get_resource(- config,- inputs.schema_name,- {"interval": f"{start.isoformat()}/{window_end.isoformat()}"},- inputs.team_id,- inputs.job_id,- api_version,+ if settings.kind == "siem":+ yield from siem_items(config, inputs, settings, manager, cursor_manager, api_version)+ elif settings.kind == "threats":+ list_window = siem_threat_ids(config, inputs, api_version)+ yield from lookup_items(config, inputs, settings, manager, cursor_manager, api_version, WINDOW, list_window)+ elif settings.kind == "campaigns":+ list_window = campaign_ids(config, inputs, settings, api_version)+ yield from lookup_items(+ config, inputs, settings, manager, cursor_manager, api_version, CAMPAIGN_WINDOW, list_window)- payloads = [payload for page in resource for payload in page]- if len(payloads) != 1 or not isinstance(payloads[0], dict):- raise ValueError("Proofpoint TAP returned an invalid response. Retry the sync.")- payload = payloads[0]- raw_end = payload.get("queryEndTime")- rows = payload.get(settings.selector)- if (- not isinstance(raw_end, str)- or not isinstance(rows, list)- or any(not isinstance(row, dict) for row in rows)- ):- raise ValueError("Proofpoint TAP returned an invalid response. Retry the sync.")- query_end = parse_timestamp(raw_end)- if query_end != window_end:- raise ValueError("Proofpoint TAP returned an invalid query end time. Retry the sync.")- if any(not row.get(settings.primary_key) for row in rows):- raise ValueError("Proofpoint TAP returned an event without its unique ID. Retry the sync.")-- # Event timestamps can predate detection, so only the queried interval can advance the cursor.- for row in rows:- row["query_end_time"] = query_end- manager.save_state(TapResumeState(next_start=query_end.isoformat(), end=end.isoformat()))- if rows:- yield rows- manager.safe_point()- last_end = query_end- start = query_end-- if last_end is not None:- cursor_manager.stage(TapCursor(query_end_time=last_end.isoformat()))- elif resume is not None and start >= end:- cursor_manager.stage(TapCursor(query_end_time=end.isoformat()))+ else:+ yield from people_items(config, inputs, settings, manager, api_version)+ partitioned = settings.partition_key is not Nonereturn SourceResponse(name=inputs.schema_name,items=items,primary_keys=[settings.primary_key],- partition_keys=[settings.event_time],- partition_mode="datetime",- partition_format="month",- sort_mode="desc",+ partition_keys=[settings.partition_key] if settings.partition_key else None,+ partition_mode="datetime" if partitioned else None,+ partition_format="month" if partitioned else None,+ sort_mode="desc" if settings.kind == "siem" else None,on_complete=manager.clear_state,)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/settings.pyindex 4ea90b419cf8..7c9d03c7aae3 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/settings.py@@ -1,4 +1,5 @@from datetime import timedelta+from typing import Literalfrom posthog.dataclasses import frozen@@ -13,30 +14,74 @@ MIN_WINDOW = timedelta(seconds=30)OVERLAP = timedelta(minutes=1)+# Campaign ID listings accept at most one day per request and allow 50 requests a day.+CAMPAIGN_WINDOW = timedelta(days=1)+CAMPAIGN_PAGE_SIZE = 200+# The People API accepts 14, 30 or 90 days and allows 50 requests a day.+PEOPLE_WINDOW_DAYS = 90++@frozenclass TapEndpoint:+ kind: Literal["siem", "threats", "campaigns", "people"]path: strselector: strprimary_key: str- event_time: str+ partition_key: str | None = None+ page_size: int | None = NoneENDPOINTS = {"clicks_blocked": TapEndpoint(- path="clicks/blocked", selector="clicksBlocked", primary_key="id", event_time="clickTime"+ kind="siem", path="siem/clicks/blocked", selector="clicksBlocked", primary_key="id", partition_key="clickTime"),"clicks_permitted": TapEndpoint(- path="clicks/permitted", selector="clicksPermitted", primary_key="id", event_time="clickTime"+ kind="siem",+ path="siem/clicks/permitted",+ selector="clicksPermitted",+ primary_key="id",+ partition_key="clickTime",),"messages_blocked": TapEndpoint(- path="messages/blocked", selector="messagesBlocked", primary_key="GUID", event_time="messageTime"+ kind="siem",+ path="siem/messages/blocked",+ selector="messagesBlocked",+ primary_key="GUID",+ partition_key="messageTime",),"messages_delivered": TapEndpoint(- path="messages/delivered", selector="messagesDelivered", primary_key="GUID", event_time="messageTime"+ kind="siem",+ path="siem/messages/delivered",+ selector="messagesDelivered",+ primary_key="GUID",+ partition_key="messageTime",+ ),+ "threats": TapEndpoint(+ kind="threats", path="threat/summary/{id}", selector="", primary_key="id", partition_key="identifiedAt"+ ),+ "campaigns": TapEndpoint(+ kind="campaigns",+ path="campaign/{id}",+ selector="campaigns",+ primary_key="id",+ partition_key="startDate",+ page_size=CAMPAIGN_PAGE_SIZE,+ ),+ "very_attacked_people": TapEndpoint(+ kind="people", path="people/vap", selector="users", primary_key="guid", page_size=1000+ ),+ "top_clickers": TapEndpoint(+ kind="people", path="people/top-clickers", selector="users", primary_key="guid", page_size=200),}-INCREMENTAL_FIELDS = {name: [incremental_field("query_end_time")] for name in ENDPOINTS}+SIEM_ALL_PATH = "siem/all"+CAMPAIGN_IDS_PATH = "campaign/ids"+SIEM_SELECTORS = ("clicksBlocked", "clicksPermitted", "messagesBlocked", "messagesDelivered")++INCREMENTAL_FIELDS = {+ name: [incremental_field("query_end_time")] for name, endpoint in ENDPOINTS.items() if endpoint.kind != "people"+}AUTH_ERRORS = {401: "Proofpoint TAP authentication failed. Check your service principal and secret.",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/tests/test_proofpoint_tap.py b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/tests/test_proofpoint_tap.pyindex b36ea0c52e4e..096b7dc74376 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/tests/test_proofpoint_tap.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/proofpoint_tap/tests/test_proofpoint_tap.py@@ -1,3 +1,4 @@+import reimport jsonfrom collections.abc import Iterablefrom datetime import UTC, datetime, timedelta@@ -31,7 +32,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.proofpointfrom products.warehouse_sources.backend.temporal.data_imports.sources.proofpoint_tap.source import ProofpointTapSourceNOW = datetime(2026, 1, 8, 12, tzinfo=UTC)-BASE = "https://tap-api-v2.proofpoint.com/v2/siem"+ROOT = "https://tap-api-v2.proofpoint.com/v2"+BASE = f"{ROOT}/siem"CONFIG = ProofpointTapSourceConfig.from_dict({"service_principal": "example-principal", "secret": "example-secret"})@@ -211,15 +213,19 @@ def test_error_classification_preserves_checkpoint(status: int) -> None:@pytest.mark.parametrize(- ("status", "schema", "valid", "message"),+ ("status", "schema", "valid", "message", "path"),[- (403, None, True, None),- (403, "messages_blocked", False, "Check access in the TAP dashboard"),+ (403, None, True, None, "/v2/siem/clicks/blocked"),+ (403, "messages_blocked", False, "Check access in the TAP dashboard", "/v2/siem/messages/blocked"),+ (200, "threats", True, None, "/v2/siem/all"),+ (200, "campaigns", True, None, "/v2/siem/clicks/blocked"),+ (200, "top_clickers", True, None, "/v2/siem/clicks/blocked"),],)-def test_credential_probe_status_mapping(status: int, schema: str | None, valid: bool, message: str | None) -> None:- selector = "messagesBlocked" if schema else "clicksBlocked"- response = ScriptedResponse(status=status, json={"queryEndTime": NOW.isoformat(), selector: []})+def test_credential_probe_status_mapping(+ status: int, schema: str | None, valid: bool, message: str | None, path: str+) -> None:+ response = ScriptedResponse(status=status, json={"queryEndTime": NOW.isoformat()})with scripted_network([response]) as network:result, error = ProofpointTapSource().validate_credentials(CONFIG, 1, schema)assert result is valid@@ -229,7 +235,7 @@ def test_credential_probe_status_mapping(status: int, schema: str | None, valid:assert error is not None and message in errorassert len(network.requests_log) == 1request = network.requests_log[0]- assert request.path == ("/v2/siem/messages/blocked" if schema else "/v2/siem/clicks/blocked")+ assert request.path == pathassert request.query == {"format": ("json",), "sinceSeconds": ("30",)}@@ -237,3 +243,152 @@ def test_unexpected_probe_error_is_not_bad_credentials() -> None:with scripted_network([ScriptedResponse(status=400, json={})]):with pytest.raises(HTTPError):ProofpointTapSource().validate_credentials(CONFIG, 1)+++ ("resume", "lookups", "saved"),+ [+ (+ None,+ ["threat-a", "gone", "threat-b"],+ [+ ("2026-01-08T09:59:00+00:00", ()),+ ("2026-01-08T09:59:00+00:00", ("threat-a",)),+ ("2026-01-08T09:59:00+00:00", ("threat-a", "gone")),+ ("2026-01-08T09:59:00+00:00", ("threat-a", "gone", "threat-b")),+ ("2026-01-08T10:59:00+00:00", ()),+ ],+ ),+ (+ TapResumeState(next_start="2026-01-08T09:59:00+00:00", end="2026-01-08T11:59:00+00:00", done=("threat-a",)),+ ["gone", "threat-b"],+ [+ ("2026-01-08T09:59:00+00:00", ("threat-a",)),+ ("2026-01-08T09:59:00+00:00", ("threat-a", "gone")),+ ("2026-01-08T09:59:00+00:00", ("threat-a", "gone", "threat-b")),+ ("2026-01-08T10:59:00+00:00", ()),+ ],+ ),+ ],+)+@time_machine.travel(NOW, tick=False)+def test_threats_resolve_each_siem_threat_once_and_skip_missing(+ resume: TapResumeState | None, lookups: list[str], saved: list[tuple[str, tuple[str, ...]]]+) -> None:+ def siem_all(request: PreparedRequest) -> tuple[int, dict[str, str], str]:+ end = query(request)["interval"][0].split("/")[1]+ payload = {+ "queryEndTime": end,+ "clicksBlocked": [{"id": "click-a", "threatID": "threat-a"}],+ "clicksPermitted": [],+ "messagesBlocked": [+ {"GUID": "message-a", "threatsInfoMap": [{"threatID": "threat-a"}, {"threatID": "gone"}]}+ ],+ "messagesDelivered": [{"GUID": "message-b", "threatsInfoMap": [{"threatID": "threat-b"}]}],+ }+ return 200, {"Content-Type": "application/json"}, json.dumps(payload)++ responses.add_callback(responses.GET, f"{BASE}/all", callback=siem_all)+ for threat_id in ("threat-a", "threat-b"):+ responses.get(+ f"{ROOT}/threat/summary/{threat_id}", json={"id": threat_id, "identifiedAt": "2026-01-01T00:00:00Z"}+ )+ responses.get(f"{ROOT}/threat/summary/gone", status=404, json={})++ harness = Harness(endpoint="threats", resume=resume)+ rows = harness.rows()++ expected_rows = [threat_id for threat_id in lookups if threat_id != "gone"]+ assert [(row["id"], row["query_end_time"]) for row in rows] == [+ (threat_id, datetime(2026, 1, 8, 10, 59, tzinfo=UTC)) for threat_id in expected_rows+ ]+ assert [urlparse(call.request.url or "").path for call in responses.calls] == [+ "/v2/siem/all",+ *[f"/v2/threat/summary/{threat_id}" for threat_id in lookups],+ "/v2/siem/all",+ ]+ assert [(call.args[0].next_start, call.args[0].done) for call in harness.manager.save_state.call_args_list] == saved+ assert harness.response.primary_keys == ["id"]+ assert harness.response.sort_mode is None+ assert harness.cursor.staged == TapCursor(query_end_time="2026-01-08T11:59:00+00:00")+++@time_machine.travel(NOW, tick=False)+def test_campaigns_page_through_one_day_windows_and_merge_details() -> None:+ full_page = [{"id": f"campaign-{i}", "lastUpdatedAt": "2026-01-07T00:00:00Z", "notable": False} for i in range(200)]+ responses.get(f"{ROOT}/campaign/ids", json={"campaigns": full_page})+ responses.get(f"{ROOT}/campaign/ids", json={"campaigns": [{"id": "campaign-0"}, {"id": "campaign-x"}]})+ responses.get(f"{ROOT}/campaign/ids", json={"campaigns": []})+ responses.add_callback(+ responses.GET,+ re.compile(f"{ROOT}/campaign/campaign-.+"),+ callback=lambda request: (+ 200,+ {},+ json.dumps({"id": urlparse(request.url or "").path.rsplit("/", 1)[1], "name": "Example"}),+ ),+ )++ harness = Harness(endpoint="campaigns", watermark=NOW - timedelta(days=1, hours=1))+ rows = harness.rows()++ listings = [+ query(call.request) for call in responses.calls if call.request.url and "/campaign/ids" in call.request.url+ ]+ assert [(q["interval"][0], q["page"], q["size"]) for q in listings] == [+ ("2026-01-07T10:59:00+00:00/2026-01-08T10:59:00+00:00", ["1"], ["200"]),+ ("2026-01-07T10:59:00+00:00/2026-01-08T10:59:00+00:00", ["2"], ["200"]),+ ("2026-01-08T10:59:00+00:00/2026-01-08T11:59:00+00:00", ["1"], ["200"]),+ ]+ assert len(rows) == 201+ assert rows[0] == {+ "id": "campaign-0",+ "lastUpdatedAt": "2026-01-07T00:00:00Z",+ "notable": False,+ "name": "Example",+ "query_end_time": datetime(2026, 1, 8, 10, 59, tzinfo=UTC),+ }+ assert rows[-1]["id"] == "campaign-x"+ assert harness.cursor.staged == TapCursor(query_end_time="2026-01-08T11:59:00+00:00")+++ ("endpoint", "path", "size", "first_page"),+ [+ ("very_attacked_people", "people/vap", 1000, 1),+ ("top_clickers", "people/top-clickers", 200, 1),+ ("top_clickers", "people/top-clickers", 200, 3),+ ],+)+def test_people_walk_pages_and_key_rows_by_identity(endpoint: str, path: str, size: int, first_page: int) -> None:+ def user(guid: str) -> dict[str, Any]:+ return {"identity": {"guid": guid, "emails": [f"{guid}@example.com"]}, "threatStatistics": {"attackIndex": 1}}++ interval = "2025-10-10T00:00:00Z/2026-01-08T00:00:00Z"+ responses.get(f"{ROOT}/{path}", json={"users": [user(f"user-{i}") for i in range(size)], "interval": interval})+ responses.get(f"{ROOT}/{path}", json={"users": [user("user-0"), user("user-last")], "interval": interval})++ resume = TapResumeState(page=first_page) if first_page > 1 else None+ harness = Harness(endpoint=endpoint, incremental=False, watermark=None, resume=resume)+ rows = harness.rows()++ assert [query(call.request) for call in responses.calls] == [+ {"window": ["90"], "size": [str(size)], "page": [str(first_page)]},+ {"window": ["90"], "size": [str(size)], "page": [str(first_page + 1)]},+ ]+ assert [call.args[0].page for call in harness.manager.save_state.call_args_list] == [first_page + 1, first_page + 2]+ assert len(rows) == size + 1+ assert rows[-1]["guid"] == "user-last"+ assert rows[-1]["interval"] == interval+ assert harness.response.primary_keys == ["guid"]+ assert harness.response.partition_keys is None+++def test_people_user_without_guid_fails() -> None:+ responses.get(f"{ROOT}/people/vap", json={"users": [{"identity": {}}]})+ with pytest.raises(ValueError):+ Harness(endpoint="very_attacked_people", incremental=False, watermark=None).rows()