Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add power bi admin dashboard tiles and refreshables (#126275)

Tom Owerseb7f96964613
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index da288ec760e9..0182b5b2c252 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -9671,19 +9671,19 @@ Note: The official reference navigation and embedded OpenAPI definitions expose
## Power BI admin — gaps
-Today (7): `activity_events`, `capacities`, `dashboards`, `dataflows`, `datasets`, `groups`, `reports`
+Today (9): `activity_events`, `capacities`, `dashboard_tiles`, `dashboards`, `dataflows`, `datasets`, `groups`, `refreshables`, `reports`
Diffed against: <https://raw.githubusercontent.com/microsoft/PowerBI-CSharp/master/sdk/swaggers/swagger.json>
- [ ] `Workspace users (GET /v1.0/myorg/admin/groups/{groupId}/users)` — connect workspace access and roles to the existing workspace inventory (high)
-- [ ] `Refreshables (GET /v1.0/myorg/admin/capacities/refreshables)` — measure refresh failures, duration, and load across capacities and datasets (high)
+- [x] `Refreshables (GET /v1.0/myorg/admin/capacities/refreshables)` — measure refresh failures, duration, and load across capacities and datasets (high)
- [ ] `Dataset data sources (GET /v1.0/myorg/admin/datasets/{datasetId}/datasources)` — identify upstream databases and connections for semantic model lineage (medium)
- [ ] `Dataset users (GET /v1.0/myorg/admin/datasets/{datasetId}/users)` — audit direct access and permissions for existing semantic models (medium)
- [ ] `Report users (GET /v1.0/myorg/admin/reports/{reportId}/users)` — compare report access with actual viewing activity (medium)
- [ ] `Apps (GET /v1.0/myorg/admin/apps)` — track published analytics apps and their workspace distribution (medium)
- [ ] `Dataset to dataflow links (GET /v1.0/myorg/admin/groups/{groupId}/datasets/upstreamDataflows)` — connect existing datasets and dataflows to analyze dependencies (medium)
- [ ] `Dataflow data sources (GET /v1.0/myorg/admin/dataflows/{dataflowId}/datasources)` — trace dataflow dependencies to external systems (medium)
-- [ ] `Dashboard tiles (GET /v1.0/myorg/admin/dashboards/{dashboardId}/tiles)` — connect dashboard content to reports and datasets (medium)
+- [x] `Dashboard tiles (GET /v1.0/myorg/admin/dashboards/{dashboardId}/tiles)` — connect dashboard content to reports and datasets (medium)
- [ ] `Published to web artifacts (GET /v1.0/myorg/admin/widelySharedArtifacts/publishedToWeb)` — track public exposure of analytics artifacts across the tenant (medium)
Note: The Microsoft SDK Swagger spec lists tenant inventory and nested admin GET operations. Parent IDs drive most new collections; the existing workspace request does not expand users. Encryption keys, scan status, and duplicate workspace inventory endpoints are excluded.
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/canonical_descriptions.py
index 6c1ae845e2de..45804c071069 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/canonical_descriptions.py
@@ -153,4 +153,40 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"tenantKeyId": "Identifier of the encryption key assigned to the capacity.",
},
},
+ "dashboard_tiles": {
+ "description": "A tile on a dashboard, one row per dashboard and tile.",
+ "docs_url": "https://learn.microsoft.com/en-us/rest/api/power-bi/admin/dashboards-get-tiles-as-admin",
+ "columns": {
+ "dashboard_id": "Identifier of the dashboard the tile is on.",
+ "id": "Identifier of the tile.",
+ "title": "Display name of the tile.",
+ "reportId": "Identifier of the report the tile was pinned from, for tiles created from a report.",
+ "datasetId": "Identifier of the semantic model behind the tile, for tiles created from a report or a dataset.",
+ "rowSpan": "Number of rows the tile spans.",
+ "colSpan": "Number of columns the tile spans.",
+ "embedUrl": "Embed URL of the tile.",
+ "embedData": "Embed data for the tile.",
+ },
+ },
+ "refreshables": {
+ "description": "Refresh statistics for a semantic model on a capacity, over the seven-day refresh history Power BI keeps.",
+ "docs_url": "https://learn.microsoft.com/en-us/rest/api/power-bi/admin/get-refreshables",
+ "columns": {
+ "id": "Identifier of the refreshable item.",
+ "name": "Display name of the refreshable item.",
+ "kind": "Kind of refreshable item, for example Dataset.",
+ "startTime": "UTC start of the window the refresh statistics cover.",
+ "endTime": "UTC end of the window the refresh statistics cover.",
+ "refreshCount": "Number of refreshes in the window.",
+ "refreshFailures": "Number of failed refreshes in the window.",
+ "averageDuration": "Average refresh duration in the window, in seconds.",
+ "medianDuration": "Median refresh duration in the window, in seconds.",
+ "refreshesPerDay": "Number of scheduled and on-demand refreshes per day in the window.",
+ "lastRefresh": "Most recent refresh history entry, with its type, status, and start and end times.",
+ "refreshSchedule": "Configured refresh schedule: days, times, time zone, and notification option.",
+ "configuredBy": "Owners of the refreshable item.",
+ "capacity": "Capacity the refreshable item runs on.",
+ "group": "Workspace the refreshable item belongs to.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/power_bi_admin.py b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/power_bi_admin.py
index 7c6f6050438c..3b55683ea437 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/power_bi_admin.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/power_bi_admin.py
@@ -17,9 +17,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.power_bi_a
LOGIN_BASE_URL,
MAX_CONTINUATION_PAGES,
MAX_ODATA_PAGES,
- ODATA_PAGE_SIZE,
POWER_BI_ADMIN_ENDPOINTS,
TOKEN_SCOPE,
+ PowerBiAdminEndpointConfig,
)
REQUEST_TIMEOUT_SECONDS = 120
@@ -230,13 +230,16 @@ def _odata_pages(
skip = start_skip
for page in range(MAX_ODATA_PAGES):
- params = {"$top": str(ODATA_PAGE_SIZE), "$skip": str(skip)}
+ params = {"$top": str(config.page_size), "$skip": str(skip), **config.params}
body = client.get(config.path, params=params)
rows = body.get(config.data_key) or []
- if rows:
- yield rows
+ output = _flatten_children(rows, config) if config.child_key else rows
+ if output:
+ yield output
+ else:
+ resumable_source_manager.safe_point()
- if len(rows) < ODATA_PAGE_SIZE:
+ if len(rows) < config.page_size:
return
skip += len(rows)
@@ -246,6 +249,15 @@ def _odata_pages(
logger.warning(f"Power BI admin: hit the {MAX_ODATA_PAGES}-page cap for {endpoint}, later rows are skipped")
+def _flatten_children(parents: list[dict[str, Any]], config: PowerBiAdminEndpointConfig) -> list[dict[str, Any]]:
+ assert config.child_key is not None and config.parent_id_column is not None
+ return [
+ {**child, config.parent_id_column: parent.get("id")}
+ for parent in parents
+ for child in parent.get(config.child_key) or []
+ ]
+
+
def _single_page(client: PowerBiAdminClient, endpoint: str) -> Iterator[list[dict[str, Any]]]:
config = POWER_BI_ADMIN_ENDPOINTS[endpoint]
body = client.get(config.path)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/settings.py
index 976c88b39477..c2a2063bce87 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/settings.py
@@ -18,6 +18,9 @@ ACTIVITY_EVENTS_RETENTION_DAYS = 28
# `$top` ceiling documented for the AsAdmin inventory list endpoints.
ODATA_PAGE_SIZE = 5000
+# The refreshables reference only documents paging "beyond the first 1000", so ask for no more than
+# that: a page the server silently truncates below `$top` would end the walk early.
+REFRESHABLES_PAGE_SIZE = 1000
# Hard caps so a misbehaving continuation token or an `$skip` walk that never shortens
# can't spin forever. Both are far above any realistic tenant.
@@ -42,6 +45,14 @@ class PowerBiAdminEndpointConfig:
# OData `$top`/`$skip` paging. Activity events use continuation tokens instead, and
# `capacities` returns the whole tenant list in one response.
supports_odata_paging: bool = True
+ page_size: int = ODATA_PAGE_SIZE
+ # Extra query params sent on every page, such as `$expand`.
+ params: dict[str, str] = field(default_factory=dict)
+ # When set, each parent row's list under `child_key` becomes the table's rows, stamped
+ # with the parent `id` as `parent_id_column`. Expanding children inline on the parent
+ # list costs one request per page instead of one per parent against a 200/hour limit.
+ child_key: Optional[str] = None
+ parent_id_column: Optional[str] = None
POWER_BI_ADMIN_ENDPOINTS: dict[str, PowerBiAdminEndpointConfig] = {
@@ -94,6 +105,23 @@ POWER_BI_ADMIN_ENDPOINTS: dict[str, PowerBiAdminEndpointConfig] = {
description="Premium and Fabric capacities assigned to the tenant",
supports_odata_paging=False,
),
+ "dashboard_tiles": PowerBiAdminEndpointConfig(
+ name="dashboard_tiles",
+ path="/dashboards",
+ primary_keys=["dashboard_id", "id"],
+ description="Tiles on every dashboard in the tenant, with the report and dataset behind each tile",
+ params={"$expand": "tiles"},
+ child_key="tiles",
+ parent_id_column="dashboard_id",
+ ),
+ "refreshables": PowerBiAdminEndpointConfig(
+ name="refreshables",
+ path="/capacities/refreshables",
+ primary_keys=["id"],
+ description="Refresh statistics for every semantic model on a capacity over the last seven days",
+ page_size=REFRESHABLES_PAGE_SIZE,
+ params={"$expand": "capacity,group"},
+ ),
}
ENDPOINTS = tuple(POWER_BI_ADMIN_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/source.py
index c6df873ac3d0..69a38795bb06 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/source.py
@@ -143,8 +143,8 @@ class PowerBiAdminSource(ResumableSource[PowerBiAdminSourceConfig, PowerBiAdminR
keywords=["power bi", "powerbi", "fabric", "microsoft"],
caption=(
"Sync tenant-wide Power BI and Fabric governance data: the audit log of views, edits, exports "
- "and shares, plus the workspace, semantic model, report, dashboard, dataflow, and capacity "
- "inventories.\n\n"
+ "and shares, the workspace, semantic model, report, dashboard, dataflow, and capacity "
+ "inventories, plus dashboard tiles and capacity refresh statistics.\n\n"
"Create an Entra ID app registration, add a client secret, and put the app in a security group "
"that is allowed to use the read-only admin APIs in the Fabric admin portal. The app must not "
"have any admin-consent Power BI permissions set on it. Then enter the directory (tenant) ID, "
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/tests/test_power_bi_admin.py b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/tests/test_power_bi_admin.py
index cd792621ae52..acccca138140 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/tests/test_power_bi_admin.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/power_bi_admin/tests/test_power_bi_admin.py
@@ -37,6 +37,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.power_bi_a
ACTIVITY_EVENTS_ENDPOINT,
ODATA_PAGE_SIZE,
POWER_BI_ADMIN_ENDPOINTS,
+ REFRESHABLES_PAGE_SIZE,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.power_bi_admin.source import PowerBiAdminSource
@@ -372,14 +373,44 @@ class TestInventoryEndpoints:
assert len(_api_requests(result.requests)) == 1
assert _activity_params(result) == [{"$top": str(ODATA_PAGE_SIZE), "$skip": "0"}]
- def test_full_page_advances_skip_until_a_short_page(self) -> None:
- full_page = [{"id": str(i)} for i in range(ODATA_PAGE_SIZE)]
+ @parameterized.expand(
+ [
+ ("groups", ODATA_PAGE_SIZE),
+ ("refreshables", REFRESHABLES_PAGE_SIZE),
+ # A full page of dashboards with no tiles yields no rows but must not end the walk.
+ ("dashboard_tiles", ODATA_PAGE_SIZE),
+ ]
+ )
+ def test_full_page_advances_skip_until_a_short_page(self, endpoint: str, page_size: int) -> None:
+ full_page = [{"id": str(i)} for i in range(page_size)]
- result = _run("groups", [_body(200, {"value": full_page}), _body(200, {"value": [{"id": "last"}]})])
+ result = _run(endpoint, [_body(200, {"value": full_page}), _body(200, {"value": [{"id": "last"}]})])
- assert len(result.items) == 2
- assert _skips(result) == ["0", str(ODATA_PAGE_SIZE)]
- assert result.saved_states[0] == PowerBiAdminResumeConfig(skip=ODATA_PAGE_SIZE)
+ assert _skips(result) == ["0", str(page_size)]
+ assert {request.param("$top") for request in _api_requests(result.requests)} == {str(page_size)}
+ assert result.saved_states[0] == PowerBiAdminResumeConfig(skip=page_size)
+
+ @parameterized.expand(
+ [
+ ("dashboard_tiles", "tiles", "dashboard_id", {"id": "tile-1"}),
+ ]
+ )
+ def test_expanded_children_become_rows_stamped_with_their_parent(
+ self, endpoint: str, child_key: str, parent_column: str, child: dict[str, Any]
+ ) -> None:
+ result = _run(
+ endpoint,
+ [_body(200, {"value": [{"id": "p1", child_key: [child]}, {"id": "p2", child_key: []}, {"id": "p3"}]})],
+ )
+
+ assert result.items == [[{**child, parent_column: "p1"}]]
+ assert _api_requests(result.requests)[0].param("$expand") == child_key
+
+ def test_refreshables_expand_their_capacity_and_workspace(self) -> None:
+ result = _run("refreshables", [_body(200, {"value": [{"id": "r1"}]})])
+
+ assert result.items == [[{"id": "r1"}]]
+ assert _api_requests(result.requests)[0].param("$expand") == "capacity,group"
def test_resume_starts_from_the_saved_skip(self) -> None:
result = _run(