:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add power bi admin dashboard tiles and refreshables (#126275)
Tom Owerseb7f96964613
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex da288ec760e9..0182b5b2c252 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -9671,19 +9671,19 @@ Note: The official reference navigation and embedded OpenAPI definitions expose## Power BI admin — gaps-Today (7): `activity_events`, `capacities`, `dashboards`, `dataflows`, `datasets`, `groups`, `reports`+Today (9): `activity_events`, `capacities`, `dashboard_tiles`, `dashboards`, `dataflows`, `datasets`, `groups`, `refreshables`, `reports`Diffed against: <https://raw.githubusercontent.com/microsoft/PowerBI-CSharp/master/sdk/swaggers/swagger.json>- [ ] `Workspace users (GET /v1.0/myorg/admin/groups/{groupId}/users)` — connect workspace access and roles to the existing workspace inventory (high)-- [ ] `Refreshables (GET /v1.0/myorg/admin/capacities/refreshables)` — measure refresh failures, duration, and load across capacities and datasets (high)+- [x] `Refreshables (GET /v1.0/myorg/admin/capacities/refreshables)` — measure refresh failures, duration, and load across capacities and datasets (high)- [ ] `Dataset data sources (GET /v1.0/myorg/admin/datasets/{datasetId}/datasources)` — identify upstream databases and connections for semantic model lineage (medium)- [ ] `Dataset users (GET /v1.0/myorg/admin/datasets/{datasetId}/users)` — audit direct access and permissions for existing semantic models (medium)- [ ] `Report users (GET /v1.0/myorg/admin/reports/{reportId}/users)` — compare report access with actual viewing activity (medium)- [ ] `Apps (GET /v1.0/myorg/admin/apps)` — track published analytics apps and their workspace distribution (medium)- [ ] `Dataset to dataflow links (GET /v1.0/myorg/admin/groups/{groupId}/datasets/upstreamDataflows)` — connect existing datasets and dataflows to analyze dependencies (medium)- [ ] `Dataflow data sources (GET /v1.0/myorg/admin/dataflows/{dataflowId}/datasources)` — trace dataflow dependencies to external systems (medium)-- [ ] `Dashboard tiles (GET /v1.0/myorg/admin/dashboards/{dashboardId}/tiles)` — connect dashboard content to reports and datasets (medium)+- [x] `Dashboard tiles (GET /v1.0/myorg/admin/dashboards/{dashboardId}/tiles)` — connect dashboard content to reports and datasets (medium)- [ ] `Published to web artifacts (GET /v1.0/myorg/admin/widelySharedArtifacts/publishedToWeb)` — track public exposure of analytics artifacts across the tenant (medium)Note: The Microsoft SDK Swagger spec lists tenant inventory and nested admin GET operations. Parent IDs drive most new collections; the existing workspace request does not expand users. Encryption keys, scan status, and duplicate workspace inventory endpoints are excluded.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/canonical_descriptions.pyindex 6c1ae845e2de..45804c071069 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/canonical_descriptions.py@@ -153,4 +153,40 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"tenantKeyId": "Identifier of the encryption key assigned to the capacity.",},},+ "dashboard_tiles": {+ "description": "A tile on a dashboard, one row per dashboard and tile.",+ "docs_url": "https://learn.microsoft.com/en-us/rest/api/power-bi/admin/dashboards-get-tiles-as-admin",+ "columns": {+ "dashboard_id": "Identifier of the dashboard the tile is on.",+ "id": "Identifier of the tile.",+ "title": "Display name of the tile.",+ "reportId": "Identifier of the report the tile was pinned from, for tiles created from a report.",+ "datasetId": "Identifier of the semantic model behind the tile, for tiles created from a report or a dataset.",+ "rowSpan": "Number of rows the tile spans.",+ "colSpan": "Number of columns the tile spans.",+ "embedUrl": "Embed URL of the tile.",+ "embedData": "Embed data for the tile.",+ },+ },+ "refreshables": {+ "description": "Refresh statistics for a semantic model on a capacity, over the seven-day refresh history Power BI keeps.",+ "docs_url": "https://learn.microsoft.com/en-us/rest/api/power-bi/admin/get-refreshables",+ "columns": {+ "id": "Identifier of the refreshable item.",+ "name": "Display name of the refreshable item.",+ "kind": "Kind of refreshable item, for example Dataset.",+ "startTime": "UTC start of the window the refresh statistics cover.",+ "endTime": "UTC end of the window the refresh statistics cover.",+ "refreshCount": "Number of refreshes in the window.",+ "refreshFailures": "Number of failed refreshes in the window.",+ "averageDuration": "Average refresh duration in the window, in seconds.",+ "medianDuration": "Median refresh duration in the window, in seconds.",+ "refreshesPerDay": "Number of scheduled and on-demand refreshes per day in the window.",+ "lastRefresh": "Most recent refresh history entry, with its type, status, and start and end times.",+ "refreshSchedule": "Configured refresh schedule: days, times, time zone, and notification option.",+ "configuredBy": "Owners of the refreshable item.",+ "capacity": "Capacity the refreshable item runs on.",+ "group": "Workspace the refreshable item belongs to.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/power_bi_admin.py b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/power_bi_admin.pyindex 7c6f6050438c..3b55683ea437 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/power_bi_admin.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/power_bi_admin.py@@ -17,9 +17,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.power_bi_aLOGIN_BASE_URL,MAX_CONTINUATION_PAGES,MAX_ODATA_PAGES,- ODATA_PAGE_SIZE,POWER_BI_ADMIN_ENDPOINTS,TOKEN_SCOPE,+ PowerBiAdminEndpointConfig,)REQUEST_TIMEOUT_SECONDS = 120@@ -230,13 +230,16 @@ def _odata_pages(skip = start_skipfor page in range(MAX_ODATA_PAGES):- params = {"$top": str(ODATA_PAGE_SIZE), "$skip": str(skip)}+ params = {"$top": str(config.page_size), "$skip": str(skip), **config.params}body = client.get(config.path, params=params)rows = body.get(config.data_key) or []- if rows:- yield rows+ output = _flatten_children(rows, config) if config.child_key else rows+ if output:+ yield output+ else:+ resumable_source_manager.safe_point()- if len(rows) < ODATA_PAGE_SIZE:+ if len(rows) < config.page_size:returnskip += len(rows)@@ -246,6 +249,15 @@ def _odata_pages(logger.warning(f"Power BI admin: hit the {MAX_ODATA_PAGES}-page cap for {endpoint}, later rows are skipped")+def _flatten_children(parents: list[dict[str, Any]], config: PowerBiAdminEndpointConfig) -> list[dict[str, Any]]:+ assert config.child_key is not None and config.parent_id_column is not None+ return [+ {**child, config.parent_id_column: parent.get("id")}+ for parent in parents+ for child in parent.get(config.child_key) or []+ ]++def _single_page(client: PowerBiAdminClient, endpoint: str) -> Iterator[list[dict[str, Any]]]:config = POWER_BI_ADMIN_ENDPOINTS[endpoint]body = client.get(config.path)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/settings.pyindex 976c88b39477..c2a2063bce87 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/settings.py@@ -18,6 +18,9 @@ ACTIVITY_EVENTS_RETENTION_DAYS = 28# `$top` ceiling documented for the AsAdmin inventory list endpoints.ODATA_PAGE_SIZE = 5000+# The refreshables reference only documents paging "beyond the first 1000", so ask for no more than+# that: a page the server silently truncates below `$top` would end the walk early.+REFRESHABLES_PAGE_SIZE = 1000# Hard caps so a misbehaving continuation token or an `$skip` walk that never shortens# can't spin forever. Both are far above any realistic tenant.@@ -42,6 +45,14 @@ class PowerBiAdminEndpointConfig:# OData `$top`/`$skip` paging. Activity events use continuation tokens instead, and# `capacities` returns the whole tenant list in one response.supports_odata_paging: bool = True+ page_size: int = ODATA_PAGE_SIZE+ # Extra query params sent on every page, such as `$expand`.+ params: dict[str, str] = field(default_factory=dict)+ # When set, each parent row's list under `child_key` becomes the table's rows, stamped+ # with the parent `id` as `parent_id_column`. Expanding children inline on the parent+ # list costs one request per page instead of one per parent against a 200/hour limit.+ child_key: Optional[str] = None+ parent_id_column: Optional[str] = NonePOWER_BI_ADMIN_ENDPOINTS: dict[str, PowerBiAdminEndpointConfig] = {@@ -94,6 +105,23 @@ POWER_BI_ADMIN_ENDPOINTS: dict[str, PowerBiAdminEndpointConfig] = {description="Premium and Fabric capacities assigned to the tenant",supports_odata_paging=False,),+ "dashboard_tiles": PowerBiAdminEndpointConfig(+ name="dashboard_tiles",+ path="/dashboards",+ primary_keys=["dashboard_id", "id"],+ description="Tiles on every dashboard in the tenant, with the report and dataset behind each tile",+ params={"$expand": "tiles"},+ child_key="tiles",+ parent_id_column="dashboard_id",+ ),+ "refreshables": PowerBiAdminEndpointConfig(+ name="refreshables",+ path="/capacities/refreshables",+ primary_keys=["id"],+ description="Refresh statistics for every semantic model on a capacity over the last seven days",+ page_size=REFRESHABLES_PAGE_SIZE,+ params={"$expand": "capacity,group"},+ ),}ENDPOINTS = tuple(POWER_BI_ADMIN_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/source.pyindex c6df873ac3d0..69a38795bb06 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/source.py@@ -143,8 +143,8 @@ class PowerBiAdminSource(ResumableSource[PowerBiAdminSourceConfig, PowerBiAdminRkeywords=["power bi", "powerbi", "fabric", "microsoft"],caption=("Sync tenant-wide Power BI and Fabric governance data: the audit log of views, edits, exports "- "and shares, plus the workspace, semantic model, report, dashboard, dataflow, and capacity "- "inventories.\n\n"+ "and shares, the workspace, semantic model, report, dashboard, dataflow, and capacity "+ "inventories, plus dashboard tiles and capacity refresh statistics.\n\n""Create an Entra ID app registration, add a client secret, and put the app in a security group ""that is allowed to use the read-only admin APIs in the Fabric admin portal. The app must not ""have any admin-consent Power BI permissions set on it. Then enter the directory (tenant) ID, "diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/tests/test_power_bi_admin.py b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/tests/test_power_bi_admin.pyindex cd792621ae52..acccca138140 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/tests/test_power_bi_admin.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/tests/test_power_bi_admin.py@@ -37,6 +37,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.power_bi_aACTIVITY_EVENTS_ENDPOINT,ODATA_PAGE_SIZE,POWER_BI_ADMIN_ENDPOINTS,+ REFRESHABLES_PAGE_SIZE,)from products.warehouse_sources.backend.temporal.data_imports.sources.power_bi_admin.source import PowerBiAdminSource@@ -372,14 +373,44 @@ class TestInventoryEndpoints:assert len(_api_requests(result.requests)) == 1assert _activity_params(result) == [{"$top": str(ODATA_PAGE_SIZE), "$skip": "0"}]- def test_full_page_advances_skip_until_a_short_page(self) -> None:- full_page = [{"id": str(i)} for i in range(ODATA_PAGE_SIZE)]+ @parameterized.expand(+ [+ ("groups", ODATA_PAGE_SIZE),+ ("refreshables", REFRESHABLES_PAGE_SIZE),+ # A full page of dashboards with no tiles yields no rows but must not end the walk.+ ("dashboard_tiles", ODATA_PAGE_SIZE),+ ]+ )+ def test_full_page_advances_skip_until_a_short_page(self, endpoint: str, page_size: int) -> None:+ full_page = [{"id": str(i)} for i in range(page_size)]- result = _run("groups", [_body(200, {"value": full_page}), _body(200, {"value": [{"id": "last"}]})])+ result = _run(endpoint, [_body(200, {"value": full_page}), _body(200, {"value": [{"id": "last"}]})])- assert len(result.items) == 2- assert _skips(result) == ["0", str(ODATA_PAGE_SIZE)]- assert result.saved_states[0] == PowerBiAdminResumeConfig(skip=ODATA_PAGE_SIZE)+ assert _skips(result) == ["0", str(page_size)]+ assert {request.param("$top") for request in _api_requests(result.requests)} == {str(page_size)}+ assert result.saved_states[0] == PowerBiAdminResumeConfig(skip=page_size)++ @parameterized.expand(+ [+ ("dashboard_tiles", "tiles", "dashboard_id", {"id": "tile-1"}),+ ]+ )+ def test_expanded_children_become_rows_stamped_with_their_parent(+ self, endpoint: str, child_key: str, parent_column: str, child: dict[str, Any]+ ) -> None:+ result = _run(+ endpoint,+ [_body(200, {"value": [{"id": "p1", child_key: [child]}, {"id": "p2", child_key: []}, {"id": "p3"}]})],+ )++ assert result.items == [[{**child, parent_column: "p1"}]]+ assert _api_requests(result.requests)[0].param("$expand") == child_key++ def test_refreshables_expand_their_capacity_and_workspace(self) -> None:+ result = _run("refreshables", [_body(200, {"value": [{"id": "r1"}]})])++ assert result.items == [[{"id": "r1"}]]+ assert _api_requests(result.requests)[0].param("$expand") == "capacity,group"def test_resume_starts_from_the_saved_skip(self) -> None:result = _run(