:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(select-star): add lineage, dashboard users and tagged items tables (#129118)
Tom Owers3361baa63ea7
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 18e707d2ed50..e355d688e176 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -10743,10 +10743,10 @@ Today (6): `Columns`, `Dashboards`, `Databases`, `Schemas`, `Tables`, `Tags`Diffed against: <https://docs.selectstar.com/select-star-api/api-examples.md>-- [ ] `Lineage (GET /v1/lineage/{guid}/)` — analyze table and column dependencies and assess the impact of data changes (high)-- [ ] `Dashboard Frequent Users (GET /v1/bi/dashboards/{guid}/frequent-dsusers/)` — measure dashboard adoption and query activity by individual user (medium)-- [ ] `Explore Related Dashboards (GET /v1/bi/explores/{guid}/related-dashboards/)` — connect Looker explores with dependent dashboards and their usage (medium)-- [ ] `Tagged Items (GET /v1/tags/items/{guid}/)` — analyze asset membership for assigned and suggested tags, including sensitive-data classifications (medium)+- [x] `Lineage (GET /v1/lineage/{guid}/)` — analyze table and column dependencies and assess the impact of data changes (high)+- [x] `Dashboard Frequent Users (GET /v1/bi/dashboards/{guid}/frequent-dsusers/)` — measure dashboard adoption and query activity by individual user (medium)+- [ ] `Explore Related Dashboards (GET /v1/bi/explores/{guid}/related-dashboards/)` — connect Looker explores with dependent dashboards and their usage (medium) — skipped: the API documents no list call for Looker Explores, so there are no explore GUIDs to request it for+- [x] `Tagged Items (GET /v1/tags/items/{guid}/)` — analyze asset membership for assigned and suggested tags, including sensitive-data classifications (medium)Note: Official API examples and the API overview were fetched with curl and their documented GET operations were compared with source paths. The live reference and likely Swagger endpoints returned HTTP 503, so findings use explicit official request examples. These resources need requests for each parent, and Looker explore IDs need separate discovery because explores are not imported. SCIM provisioning and metadata writes are excluded.diff --git a/products/warehouse_sources/sources/select_star/canonical_descriptions.py b/products/warehouse_sources/sources/select_star/canonical_descriptions.pyindex b955ae342da1..8bf4af627833 100644--- a/products/warehouse_sources/sources/select_star/canonical_descriptions.py+++ b/products/warehouse_sources/sources/select_star/canonical_descriptions.py@@ -1,6 +1,7 @@from products.warehouse_sources.sources.sdk import CanonicalDescriptions_DOCS_URL = "https://docs.selectstar.com/select-star-api/overview"+_EXAMPLES_URL = "https://docs.selectstar.com/select-star-api/api-examples"CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"Tables": {@@ -92,4 +93,51 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"view_count": "How many times the dashboard has been viewed.",},},+ "Lineage": {+ "description": "Direct downstream table-level lineage for each table: the table itself plus every "+ "table, view, or dashboard that reads from it, one level deep.",+ "docs_url": _EXAMPLES_URL,+ "columns": {+ "table_guid": "GUID of the table whose lineage this row belongs to.",+ "key": "Identifier of the lineage object within this table's lineage graph.",+ "guid": "Select Star's unique identifier for the lineage object.",+ "object_type": "The kind of lineage object, e.g. table or dashboard.",+ "name": "The lineage object's name.",+ "full_name": "Full path for the lineage object, e.g. DATABASE.SCHEMA.TABLE.",+ "data_type": "The object's type in its data source, e.g. table, view, or workbook.",+ "data_source_type": "The data source the object comes from, e.g. snowflake or tableau.",+ "source_table_guids": "GUIDs of the objects this object reads from.",+ "target_table_guids": "GUIDs of the objects that read from this object.",+ "popularity": "Usage statistics for the object over a recent timeframe.",+ "tag_guids": "GUIDs of the tags on the object.",+ },+ },+ "DashboardFrequentUsers": {+ "description": "The users who run each BI dashboard most often, with their query counts.",+ "docs_url": _EXAMPLES_URL,+ "columns": {+ "dashboard_guid": "GUID of the dashboard.",+ "guid": "GUID of the data source user.",+ "name": "The user's name in the data source.",+ "display_name": "The username the data source uses for the user.",+ "user": "The matching Select Star user, if one exists.",+ "query_count": "How many times the user ran the dashboard.",+ "last_queried": "The date the user last ran the dashboard.",+ "data_source": "The BI data source the user belongs to.",+ },+ },+ "TaggedItems": {+ "description": "The assets that carry each tag, both user-defined and suggested by Select Star.",+ "docs_url": _EXAMPLES_URL,+ "columns": {+ "tag_guid": "GUID of the tag.",+ "tag": "The tag applied to the asset.",+ "kind": "Whether a user applied the tag (user-defined) or Select Star suggested it.",+ "suggested_source_kind": "Where a suggested tag came from.",+ "suggestion_source": "The object a suggested tag was derived from, if any.",+ "item": "The tagged asset, with its object type and metadata.",+ "item_guid": "GUID of the tagged asset.",+ "item_object_type": "The kind of tagged asset, e.g. table or column.",+ },+ },}diff --git a/products/warehouse_sources/sources/select_star/select_star.py b/products/warehouse_sources/sources/select_star/select_star.pyindex 31f739800212..878697b3e751 100644--- a/products/warehouse_sources/sources/select_star/select_star.py+++ b/products/warehouse_sources/sources/select_star/select_star.py@@ -1,21 +1,33 @@-from typing import Any, Optional+from typing import Any, Optional, castfrom products.warehouse_sources.sources.sdk import (+ ClientConfig,JSONResponsePaginator,+ Resource,RESTAPIConfig,ResumableSourceManager,+ SinglePagePaginator,SourceResponse,+ build_dependent_resource,frozen,make_tracked_session,rest_api_resource,validate_via_probe,)-from products.warehouse_sources.sources.select_star.settings import PAGE_SIZE, SELECTSTAR_BASE_URL, SELECTSTAR_ENDPOINTS+from products.warehouse_sources.sources.select_star.settings import (+ PAGE_SIZE,+ SELECTSTAR_BASE_URL,+ SELECTSTAR_ENDPOINTS,+ SelectStarEndpointConfig,+)@frozenclass SelectStarResumeConfig:- next_url: str+ next_url: Optional[str] = None+ # Framework fan-out state for the per-parent endpoints, passed straight back to+ # `build_dependent_resource`.+ fanout_state: Optional[dict[str, Any]] = Nonedef _auth_header_value(api_token: str) -> str:@@ -27,6 +39,86 @@ def _incremental_param(field: str) -> dict[str, Any]:return {"type": "incremental", "cursor_path": field, "initial_value": None}+def _client_config(api_token: str) -> ClientConfig:+ return {+ "base_url": SELECTSTAR_BASE_URL,+ "auth": {+ "type": "api_key",+ "api_key": _auth_header_value(api_token),+ "name": "Authorization",+ "location": "header",+ },+ "paginator": JSONResponsePaginator(next_url_path="next"),+ # Pagination follows the absolute `next` URL straight out of the response body, and+ # the API token is replayed on every request via the Authorization header, so pin+ # follow-up requests to the API host and refuse redirects: a tampered `next` link must+ # not be able to retarget the token off-host (SSRF).+ "allowed_hosts": [],+ "allow_redirects": False,+ }+++def _flatten_tagged_item(row: dict[str, Any]) -> dict[str, Any]:+ item = row.get("item") or {}+ obj = item.get("obj") or {}+ row["item_guid"] = obj.get("guid")+ row["item_object_type"] = item.get("object_type")+ return row+++def _fanout_source(+ api_token: str,+ endpoint: str,+ config: SelectStarEndpointConfig,+ team_id: int,+ job_id: str,+ resumable_source_manager: ResumableSourceManager[SelectStarResumeConfig],+) -> SourceResponse:+ assert config.fanout is not None+ parent_config = SELECTSTAR_ENDPOINTS[config.fanout.parent_name]++ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None++ def save_checkpoint(state: Optional[dict[str, Any]]) -> None:+ if state:+ resumable_source_manager.save_state(SelectStarResumeConfig(fanout_state=state))++ resource = cast(+ Resource,+ build_dependent_resource(+ endpoint_configs=SELECTSTAR_ENDPOINTS,+ child_endpoint=endpoint,+ fanout=config.fanout,+ client_config=_client_config(api_token),+ path_format_values={},+ team_id=team_id,+ job_id=job_id,+ db_incremental_field_last_value=None,+ # The parent page size rides in `parent_params`; the child endpoints document no page size.+ page_size_param=None,+ parent_endpoint_extra={"data_selector": parent_config.data_selector},+ child_endpoint_extra={+ "data_selector": config.data_selector,+ **({} if config.paginated else {"paginator": SinglePagePaginator()}),+ },+ resume_hook=save_checkpoint,+ initial_paginator_state=resume.fanout_state if resume else None,+ # Saves the parent page too, so the checkpoint keeps two pages of finished parents+ # rather than every table in a large catalog.+ resume_parent_pagination=True,+ ),+ )+ if endpoint == "TaggedItems":+ resource.add_map(_flatten_tagged_item)++ return SourceResponse(+ name=endpoint,+ items=lambda: resource,+ primary_keys=config.primary_keys,+ column_hints=resource.column_hints,+ )++def select_star_source(api_token: str,endpoint: str,@@ -38,6 +130,8 @@ def select_star_source(incremental_field: Optional[str] = None,) -> SourceResponse:config = SELECTSTAR_ENDPOINTS[endpoint]+ if config.fanout is not None:+ return _fanout_source(api_token, endpoint, config, team_id, job_id, resumable_source_manager)# inputs.incremental_field is the user's chosen cursor (menu = config.incremental_fields);# fall back to the first advertised option so a schema saved before the user picks stays usable.@@ -49,22 +143,7 @@ def select_star_source(params[f"{filter_field}__gte"] = _incremental_param(filter_field) if use_incremental else Nonerest_config: RESTAPIConfig = {- "client": {- "base_url": SELECTSTAR_BASE_URL,- "auth": {- "type": "api_key",- "api_key": _auth_header_value(api_token),- "name": "Authorization",- "location": "header",- },- "paginator": JSONResponsePaginator(next_url_path="next"),- # Pagination follows the absolute `next` URL straight out of the response body, and- # the API token is replayed on every request via the Authorization header, so pin- # follow-up requests to the API host and refuse redirects: a tampered `next` link must- # not be able to retarget the token off-host (SSRF).- "allowed_hosts": [],- "allow_redirects": False,- },+ "client": _client_config(api_token),"resource_defaults": {"write_disposition": {"disposition": "merge", "strategy": "upsert"} if use_incremental else "replace",},@@ -84,7 +163,7 @@ def select_star_source(initial_paginator_state: Optional[dict[str, Any]] = Noneif resumable_source_manager.can_resume():resume = resumable_source_manager.load_state()- if resume is not None:+ if resume is not None and resume.next_url:initial_paginator_state = {"next_url": resume.next_url}def save_checkpoint(state: Optional[dict[str, Any]]) -> None:diff --git a/products/warehouse_sources/sources/select_star/settings.py b/products/warehouse_sources/sources/select_star/settings.pyindex 6565555e9c08..1acbe021823e 100644--- a/products/warehouse_sources/sources/select_star/settings.py+++ b/products/warehouse_sources/sources/select_star/settings.py@@ -1,6 +1,13 @@from dataclasses import field+from typing import Any-from products.warehouse_sources.sources.sdk import IncrementalField, IncrementalFieldType, frozen+from products.warehouse_sources.sources.sdk import (+ DependentEndpointConfig,+ IncrementalField,+ IncrementalFieldType,+ ResponseAction,+ frozen,+)SELECTSTAR_BASE_URL = "https://api.production.selectstar.com"@@ -8,15 +15,27 @@ SELECTSTAR_BASE_URL = "https://api.production.selectstar.com"# their documented burst limit (1000 requests/60s) for organizations with large catalogs.PAGE_SIZE = 100+# A parent deleted between the parent listing and its child fetch 404s; skip it instead of+# failing the whole fan-out.+_SKIP_MISSING_PARENT: list[ResponseAction] = [{"status_code": 404, "action": "ignore"}]+@frozenclass SelectStarEndpointConfig:+ name: strpath: strprimary_keys: list[str]incremental_fields: list[IncrementalField] = field(default_factory=list)# Stable "created" field to partition by. None when the response carries no field that# never changes after creation (never partition on a field that can be edited later).partition_key: str | None = None+ # Set where the resource only exists per parent row (e.g. /v1/lineage/{guid}/).+ fanout: DependentEndpointConfig | None = None+ data_selector: str = "results"+ # False for responses that return the whole collection in one body with no `next` link.+ paginated: bool = True+ default_incremental_field: str | None = None+ page_size: int = PAGE_SIZEdef _datetime_field(name: str) -> IncrementalField:@@ -28,8 +47,22 @@ def _datetime_field(name: str) -> IncrementalField:}-SELECTSTAR_ENDPOINTS: dict[str, SelectStarEndpointConfig] = {- "Tables": SelectStarEndpointConfig(+def _fanout(parent_name: str, parent_key: str, child_params: dict[str, Any] | None = None) -> DependentEndpointConfig:+ return DependentEndpointConfig(+ parent_name=parent_name,+ resolve_param="guid",+ resolve_field="guid",+ include_from_parent=["guid"],+ parent_field_renames={"guid": parent_key},+ parent_params={"page_size": PAGE_SIZE},+ child_params=child_params or {},+ child_response_actions=_SKIP_MISSING_PARENT,+ )+++_ENDPOINT_LIST: tuple[SelectStarEndpointConfig, ...] = (+ SelectStarEndpointConfig(+ name="Tables",path="/v1/tables/",primary_keys=["guid"],# Both filters are server-side (`<field>__gte`); the row also carries the field back,@@ -38,7 +71,8 @@ SELECTSTAR_ENDPOINTS: dict[str, SelectStarEndpointConfig] = {incremental_fields=[_datetime_field("updated_on"), _datetime_field("last_queried_on")],partition_key="db_created_on",),- "Columns": SelectStarEndpointConfig(+ SelectStarEndpointConfig(+ name="Columns",path="/v1/columns/",primary_keys=["guid"],# The endpoint accepts an `updated_on__gte` filter, but the Column object itself never@@ -46,30 +80,59 @@ SELECTSTAR_ENDPOINTS: dict[str, SelectStarEndpointConfig] = {# back as a watermark. Full refresh only.incremental_fields=[],),- "Databases": SelectStarEndpointConfig(+ SelectStarEndpointConfig(+ name="Databases",path="/v1/databases/",primary_keys=["guid"],# Same gap as Columns: `updated_on__gte` filters, but no timestamp comes back on the row.incremental_fields=[],),- "Schemas": SelectStarEndpointConfig(+ SelectStarEndpointConfig(+ name="Schemas",path="/v1/schemas/",primary_keys=["guid"],incremental_fields=[],),- "Tags": SelectStarEndpointConfig(+ SelectStarEndpointConfig(+ name="Tags",path="/v1/tags/",primary_keys=["guid"],incremental_fields=[_datetime_field("updated_on")],),- "Dashboards": SelectStarEndpointConfig(+ SelectStarEndpointConfig(+ name="Dashboards",path="/v1/bi/dashboards/",primary_keys=["guid"],# No `updated_on`-style filter param on this endpoint, so full refresh only.incremental_fields=[],partition_key="dashboard_created_at",),-}+ SelectStarEndpointConfig(+ name="Lineage",+ path="/v1/lineage/{guid}/",+ # Each table's response lists the table itself plus its direct downstream objects, so an+ # object appears once per upstream table: the root table is part of the key.+ primary_keys=["table_guid", "key"],+ fanout=_fanout("Tables", "table_guid", {"max_depth": "1", "direction": "right", "mode": "table"}),+ data_selector="table_lineage",+ paginated=False,+ ),+ SelectStarEndpointConfig(+ name="DashboardFrequentUsers",+ path="/v1/bi/dashboards/{guid}/frequent-dsusers/",+ primary_keys=["dashboard_guid", "guid"],+ fanout=_fanout("Dashboards", "dashboard_guid"),+ ),+ SelectStarEndpointConfig(+ name="TaggedItems",+ path="/v1/tags/items/{guid}/",+ # One asset can carry the same tag as both a user-defined and a suggested tag.+ primary_keys=["tag_guid", "kind", "item_guid"],+ fanout=_fanout("Tags", "tag_guid"),+ ),+)++SELECTSTAR_ENDPOINTS: dict[str, SelectStarEndpointConfig] = {config.name: config for config in _ENDPOINT_LIST}ENDPOINTS = tuple(SELECTSTAR_ENDPOINTS.keys())diff --git a/products/warehouse_sources/sources/select_star/tests/test_select_star.py b/products/warehouse_sources/sources/select_star/tests/test_select_star.pyindex 1ae02db624aa..4e8f9604ec98 100644--- a/products/warehouse_sources/sources/select_star/tests/test_select_star.py+++ b/products/warehouse_sources/sources/select_star/tests/test_select_star.py@@ -62,7 +62,7 @@ class TestValidateCredentials:class TestSelectStarSourceResponse:- @parameterized.expand([(e,) for e in ENDPOINTS])+ @parameterized.expand([(e,) for e in ENDPOINTS if SELECTSTAR_ENDPOINTS[e].fanout is None])def test_source_response_shape(self, endpoint: str) -> None:result = _driver().run(endpoint, [_page([])])@@ -71,6 +71,133 @@ class TestSelectStarSourceResponse:assert result.response.primary_keys == ["guid"]assert result.response.sort_mode == "asc"- def test_every_endpoint_uses_guid_primary_key(self) -> None:- assert all(config.primary_keys == ["guid"] for config in SELECTSTAR_ENDPOINTS.values())- assert set(SELECTSTAR_ENDPOINTS) == set(ENDPOINTS)++def _lineage(nodes: list[dict[str, Any]]) -> ScriptedResponse:+ return ScriptedResponse(json={"table_lineage": nodes, "column_lineage": []})+++class TestFanout:+ @parameterized.expand(+ [+ (+ "lineage",+ "Lineage",+ "/v1/tables/",+ "/v1/lineage/{}/",+ "table_guid",+ _lineage([{"key": "ta_1", "guid": "ta_1"}]),+ ),+ (+ "dashboard_frequent_users",+ "DashboardFrequentUsers",+ "/v1/bi/dashboards/",+ "/v1/bi/dashboards/{}/frequent-dsusers/",+ "dashboard_guid",+ _page([{"guid": "du_1", "query_count": 7}]),+ ),+ (+ "tagged_items",+ "TaggedItems",+ "/v1/tags/",+ "/v1/tags/items/{}/",+ "tag_guid",+ _page([{"kind": "user-defined", "item": {"object_type": "column", "obj": {"guid": "co_1"}}}]),+ ),+ ]+ )+ def test_walks_every_parent_page_and_tags_child_rows(+ self,+ _name: str,+ endpoint: str,+ parent_path: str,+ child_path: str,+ parent_key: str,+ child_response: ScriptedResponse,+ ) -> None:+ parent_page_2 = f"{SELECTSTAR_BASE_URL}{parent_path}?page=2"++ result = _driver().run(+ endpoint,+ [+ _page([{"guid": "p1"}, {"guid": "p2"}], next_url=parent_page_2),+ child_response,+ child_response,+ _page([{"guid": "p3"}]),+ child_response,+ ],+ )++ assert result.paths == [+ parent_path,+ child_path.format("p1"),+ child_path.format("p2"),+ parent_path,+ child_path.format("p3"),+ ]+ assert [row[parent_key] for row in result.rows] == ["p1", "p2", "p3"]+ assert result.requests[0].param("page_size") == "100"+ assert all(request.param("page_size") is None for request in result.requests[1:3])+ assert result.response is not None+ assert result.response.primary_keys == SELECTSTAR_ENDPOINTS[endpoint].primary_keys+ assert all(key in result.rows[0] for key in result.response.primary_keys)+ assert result.response.sort_mode is None++ def test_lineage_requests_one_level_of_downstream_table_lineage(self) -> None:+ result = _driver().run("Lineage", [_page([{"guid": "ta_root"}]), _lineage([])])++ child = result.requests[1]+ assert (child.param("max_depth"), child.param("direction"), child.param("mode")) == ("1", "right", "table")++ def test_tagged_items_flatten_the_tagged_asset(self) -> None:+ result = _driver().run(+ "TaggedItems",+ [+ _page([{"guid": "tg_1"}]),+ _page(+ [+ {"kind": "suggested", "item": {"object_type": "table", "obj": {"guid": "ta_1"}}},+ {"kind": "user-defined", "item": None},+ ]+ ),+ ],+ )++ assert [(r["item_guid"], r["item_object_type"]) for r in result.rows] == [("ta_1", "table"), (None, None)]++ def test_follows_child_next_url_and_skips_a_deleted_parent(self) -> None:+ child_page_2 = f"{SELECTSTAR_BASE_URL}/v1/bi/dashboards/da_1/frequent-dsusers/?page=2"++ result = _driver().run(+ "DashboardFrequentUsers",+ [+ _page([{"guid": "da_gone"}, {"guid": "da_1"}]),+ ScriptedResponse(status=404, body=b"Not Found"),+ _page([{"guid": "du_1"}], next_url=child_page_2),+ _page([{"guid": "du_2"}]),+ ],+ )++ assert [(r["dashboard_guid"], r["guid"]) for r in result.rows] == [("da_1", "du_1"), ("da_1", "du_2")]+ assert result.urls[-1] == child_page_2++ def test_resumes_from_saved_fanout_state(self) -> None:+ first = _driver().run(+ "DashboardFrequentUsers",+ [+ _page([{"guid": "da_1"}, {"guid": "da_2"}]),+ _page([{"guid": "du_1"}]),+ _page([{"guid": "du_2"}]),+ ],+ )+ assert first.saved_states+ saved = first.saved_states[0]+ assert isinstance(saved, SelectStarResumeConfig) and saved.fanout_state is not None++ resumed = _driver().run(+ "DashboardFrequentUsers",+ [_page([{"guid": "da_1"}, {"guid": "da_2"}]), _page([{"guid": "du_2"}])],+ resume_state=saved,+ )++ assert resumed.paths == ["/v1/bi/dashboards/", "/v1/bi/dashboards/da_2/frequent-dsusers/"]+ assert [r["guid"] for r in resumed.rows] == ["du_2"]