:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add frontegg audits and tenants tables (#120219)
Tom Owerse26aa9469236
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex eb942bf3064c..2e20dcda875e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -5052,15 +5052,15 @@ Note: Diffed against Front's published OpenAPI 3.0 spec (128 GET operations). Th## Frontegg — **thin**-Today (3): `permissions`, `roles`, `users`+Today (5): `audits`, `permissions`, `roles`, `tenants`, `users`Diffed against: <https://raw.githubusercontent.com/frontegg/openapi-public/master/combined.json>-- [ ] `Get accounts (tenants) (GET /tenants/resources/tenants/v2)` — resolve tenantId and tenantIds on users into customer account names and metadata (high)-- [ ] `Get audits (V2) (GET /audits/resources/audits/v2)` — analyze login activity, user changes, and account events over time (high)-- [ ] `Get users roles (GET /identity/resources/users/v3/roles)` — join synced users and roles through account-specific assignments (medium)+- [x] `Get accounts (tenants) (GET /tenants/resources/tenants/v2)` — resolve tenantId and tenantIds on users into customer account names and metadata (high). Added as `tenants`.+- [x] `Get audits (V2) (GET /audits/resources/audits/v2)` — analyze login activity, user changes, and account events over time (high). Added as `audits`, incremental on `createdAt`.+- [ ] `Get users roles (GET /identity/resources/users/v3/roles)` — join synced users and roles through account-specific assignments (medium); skipped: a lookup that needs user IDs and an account header, and each `users` row already carries its per-account roles in `tenants`- [ ] `Get all groups paginated (GET /identity/resources/groups/v2)` — describe customer user groups for account segmentation and membership analysis (medium)-- [ ] `Get users groups (GET /identity/resources/users/v3/groups)` — connect users to their customer groups (medium)+- [ ] `Get users groups (GET /identity/resources/users/v3/groups)` — connect users to their customer groups (medium); skipped: a lookup that needs user IDs and an account header, with no documented response shape or key. `Get all groups paginated` is the better route to membership- [ ] `Get sub-accounts hierarchy tree (GET /tenants/resources/hierarchy/v1/tree)` — roll up account activity through parent and child customer accounts (medium)Note: Parsed the official combined OpenAPI file and checked identity.json, tenants.json, and audits.json for service prefixes. The user schema exposes tenant IDs but the source only imports users, roles, and permissions. Account-scoped lists need tenant fan-out and management-token permissions. Excluded authentication, keys, session configuration, and invitations that expose tokens.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/canonical_descriptions.pyindex 0882a5d1f9bf..6f34fc928b49 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/canonical_descriptions.py@@ -34,4 +34,26 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"createdAt": "The time when the permission was created.",},},+ "tenants": {+ "description": "Accounts (tenants) in the Frontegg environment, with their names and metadata.",+ "docs_url": "https://developers.frontegg.com/ciam/api/tenants/accounts/tenantcontrollerv2_gettenants",+ "columns": {+ "tenantId": "The account (tenant) identifier. Users reference it in tenantId and tenantIds.",+ "name": "The account name.",+ "metadata": "Custom metadata stored on the account.",+ "createdAt": "The time when the account was created.",+ },+ },+ "audits": {+ "description": "Audit log events in the Frontegg environment, such as logins and user or account changes.",+ "docs_url": "https://developers.frontegg.com/ciam/api/audits/main/auditscontroller_v2_getaudits",+ "columns": {+ "frontegg_id": "The audit event identifier.",+ "tenantId": "The account (tenant) identifier for the event.",+ "email": "The email address of the user who performed the action.",+ "action": "The action that the event records.",+ "severity": "The event severity: Info, Medium, High, Critical, or Error.",+ "createdAt": "The time when the event was created.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/frontegg.py b/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/frontegg.pyindex ae31f5c0e479..cd69ae36f0ff 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/frontegg.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/frontegg.py@@ -1,3 +1,4 @@+from datetime import UTC, datetimefrom typing import Anyfrom requests import HTTPError@@ -14,6 +15,8 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceInputs, SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.frontegg.settings import (ENDPOINTS,+ PARTITION_KEYS,+ PRIMARY_KEYS,REGION_ERROR,REGIONS,)@@ -24,7 +27,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.generated_@frozenclass FronteggResumeConfig:- page: int+ page: int | None = None+ offset: int | None = None+ created_from: str | None = Noneclass FronteggAuth(OAuth2Auth):@@ -58,6 +63,12 @@ class FronteggAuth(OAuth2Auth):self._apply_token_response(payload if isinstance(payload, dict) else None)+def _created_from(value: Any) -> str:+ if isinstance(value, datetime):+ return (value if value.tzinfo else value.replace(tzinfo=UTC)).isoformat()+ return str(value)++def frontegg_source(config: FronteggSourceConfig,manager: ResumableSourceManager[FronteggResumeConfig],@@ -72,15 +83,35 @@ def frontegg_source(if path is None:raise ValueError(f"Frontegg table has no endpoint path: {inputs.schema_name}")endpoint["path"] = path.format(api_version=api_version)+ resume = manager.load_state() if manager.can_resume() else None+ if resume is not None and resume.offset is not None:+ # The offset counts rows within the window of the attempt that saved it.+ created_from = resume.created_from+ elif inputs.should_use_incremental_field and inputs.db_incremental_field_last_value is not None:+ created_from = _created_from(inputs.db_incremental_field_last_value)+ else:+ created_from = None+ if created_from is not None:+ endpoint["params"] = {**(endpoint.get("params") or {}), "created_from": created_from}rest_config: RESTAPIConfig = {"client": {"base_url": REGIONS[config.region], "auth": auth, "request_timeout": (10, 60)},"resources": [{"name": inputs.schema_name, "endpoint": endpoint}],}- resume = manager.load_state() if manager.can_resume() else None+ initial_state = (+ {key: value for key, value in (("page", resume.page), ("offset", resume.offset)) if value is not None}+ if resume+ else None+ )def save_checkpoint(state: dict[str, Any] | None) -> None:if state is not None:- manager.save_state(FronteggResumeConfig(page=int(state["page"])))+ manager.save_state(+ FronteggResumeConfig(+ page=int(state["page"]) if "page" in state else None,+ offset=int(state["offset"]) if "offset" in state else None,+ created_from=created_from,+ )+ )resource = rest_api_resource(rest_config,@@ -88,6 +119,13 @@ def frontegg_source(inputs.job_id,None,resume_hook=save_checkpoint,- initial_paginator_state={"page": resume.page} if resume else None,+ initial_paginator_state=initial_state or None,+ )+ partition_key = PARTITION_KEYS.get(inputs.schema_name)+ return SourceResponse(+ name=inputs.schema_name,+ items=lambda: resource,+ primary_keys=PRIMARY_KEYS.get(inputs.schema_name, ["id"]),+ partition_mode="datetime" if partition_key else None,+ partition_keys=[partition_key] if partition_key else None,)- return SourceResponse(name=inputs.schema_name, items=lambda: resource, primary_keys=["id"])diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/settings.pyindex 929ebb73e873..9089fcf7004a 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/settings.py@@ -1,4 +1,5 @@from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import Endpoint+from products.warehouse_sources.backend.types import IncrementalField, IncrementalFieldTypeREGIONS = {"EU": "https://api.frontegg.com",@@ -25,6 +26,39 @@ ENDPOINTS: dict[str, Endpoint] = {"data_selector": "$","paginator": "single_page",},+ "tenants": {+ "path": "tenants/resources/tenants/v2",+ "data_selector": "items",+ "params": {"_limit": 200, "_sortBy": "createdAt", "_order": "ASC"},+ "paginator": {"type": "page_number", "page_param": "_offset", "total_path": "_metadata.totalPages"},+ },+ "audits": {+ "path": "audits/resources/audits/v2",+ "data_selector": "data",+ "params": {"sortBy": "createdAt", "sortDirection": "asc"},+ "paginator": {+ "type": "offset",+ "limit": 200,+ "offset_param": "offset",+ "limit_param": "count",+ "total_path": "total",+ },+ },+}++PRIMARY_KEYS: dict[str, list[str]] = {"tenants": ["tenantId"], "audits": ["frontegg_id"]}++PARTITION_KEYS: dict[str, str] = {"audits": "createdAt"}++INCREMENTAL_FIELDS: dict[str, list[IncrementalField]] = {+ "audits": [+ {+ "label": "createdAt",+ "type": IncrementalFieldType.DateTime,+ "field": "createdAt",+ "field_type": IncrementalFieldType.DateTime,+ }+ ],}AUTH_ERROR = "Frontegg authentication failed. Check your client ID, API key, and region."diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/source.pyindex 33d9dfd34d34..faaf556a7d49 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/source.py@@ -37,6 +37,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.frontegg.ffrom products.warehouse_sources.backend.temporal.data_imports.sources.frontegg.settings import (AUTH_ERROR,ENDPOINTS,+ INCREMENTAL_FIELDS,PERMISSION_ERROR,REGION_ERROR,REGIONS,@@ -68,7 +69,7 @@ class FronteggSource(ResumableSource[FronteggSourceConfig, FronteggResumeConfig])lists_tables_without_credentials = True- # The pin selects the users API; roles and permissions have independent endpoint versions.+ # The pin selects the users API; the other tables have independent endpoint versions.supported_versions = ("v3",)default_version = "v3"api_docs_url = "https://developers.frontegg.com/ciam/api/identity"@@ -101,7 +102,7 @@ class FronteggSource(ResumableSource[FronteggSourceConfig, FronteggResumeConfig]force_refresh: bool = False,api_version: str | None = None,) -> list[SourceSchema]:- return build_endpoint_schemas(ENDPOINTS, {}, names)+ return build_endpoint_schemas(ENDPOINTS, INCREMENTAL_FIELDS, names)def validate_credentials(self,diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/tests/test_frontegg.py b/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/tests/test_frontegg.pyindex 458e22b68d3f..fac0cc1361d8 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/tests/test_frontegg.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/frontegg/tests/test_frontegg.py@@ -28,11 +28,18 @@ def config(region: Region = "EU") -> FronteggSourceConfig:class TestFronteggTransport:- @pytest.mark.parametrize("table,version", [("users", "v3"), ("roles", "v2")])+ @pytest.mark.parametrize(+ "table,endpoint_path",+ [+ ("users", "/identity/resources/users/v3"),+ ("roles", "/identity/resources/roles/v2"),+ ("tenants", "/tenants/resources/tenants/v2"),+ ],+ )@pytest.mark.parametrize("resume_page", [None, 1])@pytest.mark.parametrize("empty_terminal", [False, True])def test_pagination_and_resume(- self, table: str, version: str, resume_page: int | None, empty_terminal: bool+ self, table: str, endpoint_path: str, resume_page: int | None, empty_terminal: bool) -> None:first_page = resume_page or 0result = SourceDriver(FronteggSource(), config()).run(@@ -51,7 +58,6 @@ class TestFronteggTransport:)assert result.raised is Noneassert result.rows == [{"id": "first"}] + ([] if empty_terminal else [{"id": "last"}])- endpoint_path = f"/identity/resources/{table}/{version}"assert result.paths == ["/auth/vendor", endpoint_path, endpoint_path]for index, request in enumerate(result.requests[1:]):assert request.query == {@@ -63,6 +69,58 @@ class TestFronteggTransport:assert request.headers["authorization"] == "Bearer fake-token"assert result.saved_states == [FronteggResumeConfig(page=first_page + 1)]+ @pytest.mark.parametrize(+ "should_use_incremental_field,last_value,created_from",+ [+ (False, datetime(2026, 1, 2, 3, 4, 5), None),+ (True, None, None),+ (True, datetime(2026, 1, 2, 3, 4, 5), "2026-01-02T03:04:05+00:00"),+ ],+ )+ @pytest.mark.parametrize(+ "resume",+ [+ None,+ FronteggResumeConfig(offset=200),+ FronteggResumeConfig(offset=200, created_from="2025-12-01T00:00:00+00:00"),+ ],+ )+ def test_audits_offset_pagination_incremental_and_resume(+ self,+ should_use_incremental_field: bool,+ last_value: datetime | None,+ created_from: str | None,+ resume: FronteggResumeConfig | None,+ ) -> None:+ first_offset = resume.offset if resume and resume.offset is not None else 0+ window = resume.created_from if resume else created_from+ full_page = [{"frontegg_id": f"event-{index}"} for index in range(200)]+ result = SourceDriver(FronteggSource(), config()).run(+ "audits",+ [+ ScriptedResponse(json={"token": "fake-token", "expiresIn": 3600}),+ ScriptedResponse(json={"data": full_page, "total": first_offset + 201}),+ ScriptedResponse(json={"data": [{"frontegg_id": "last"}], "total": first_offset + 201}),+ ],+ resume_state=resume,+ should_use_incremental_field=should_use_incremental_field,+ db_incremental_field_last_value=last_value,+ )+ assert result.raised is None+ assert result.rows == [*full_page, {"frontegg_id": "last"}]+ assert result.paths == ["/auth/vendor", "/audits/resources/audits/v2", "/audits/resources/audits/v2"]+ for index, request in enumerate(result.requests[1:]):+ expected = {+ "count": ("200",),+ "offset": (str(first_offset + index * 200),),+ "sortBy": ("createdAt",),+ "sortDirection": ("asc",),+ }+ if window is not None:+ expected["created_from"] = (window,)+ assert request.query == expected+ assert result.saved_states == [FronteggResumeConfig(offset=first_offset + 200, created_from=window)]+@pytest.mark.parametrize("region,host",[