:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(harness): add organizations, projects and infrastructures tables (#121151)
Tom Owers2d1399a18a88
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 2f6eed7d0319..68ccda0c82df 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -5695,13 +5695,13 @@ Note: Used the current documentation index and parsed version 2026-06-15 OpenAPI## Harness — **thin**-Today (4): `environments`, `executions`, `pipelines`, `services`+Today (7): `environments`, `executions`, `infrastructures`, `organizations`, `pipelines`, `projects`, `services`Diffed against: <https://apidocs.harness.io/_bundle/index.yaml>-- [ ] `Projects (GET /ng/api/projects)` — resolve project identifiers on synced resources into project names and organizational context (high)-- [ ] `Organizations (GET /ng/api/organizations)` — resolve organization identifiers on synced resources for named organizational reporting (high)-- [ ] `Infrastructures (GET /ng/api/infrastructures)` — explain infrastructure references and deployment targets within synced pipeline definitions and environments (high)+- [x] `Projects (GET /ng/api/projects)` — resolve project identifiers on synced resources into project names and organizational context (high)+- [x] `Organizations (GET /ng/api/organizations)` — resolve organization identifiers on synced resources for named organizational reporting (high)+- [x] `Infrastructures (GET /ng/api/infrastructures)` — explain infrastructure references and deployment targets within synced pipeline definitions and environments (high)- [ ] `Approval instances (GET /v1/orgs/{org}/projects/{project}/approvals/execution/{execution-id})` — measure manual approval bottlenecks and identify approval outcomes within pipeline executions (medium)- [ ] `Pipeline execution notes (GET /pipeline/api/pipelines/execution/{planExecutionId}/notes)` — connect execution outcomes with operator comments and explanations (medium)- [ ] `Pipeline execution annotations (GET /pipeline/api/pipelines/execution/{planExecutionId}/annotations)` — join execution outcomes with structured annotations and deployment context (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/harness/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/harness/canonical_descriptions.pyindex 71f58faa17c7..bb79c706bd4b 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/harness/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/harness/canonical_descriptions.py@@ -52,4 +52,45 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"lastModifiedAt": "Last modification time in milliseconds since the Unix epoch.",},},+ "infrastructures": {+ "description": "Infrastructure definitions for each environment in the selected Harness project.",+ "docs_url": "https://apidocs.harness.io/infrastructures/getinfrastructurelist",+ "columns": {+ "identifier": "Infrastructure identifier within the environment.",+ "environmentRef": "Identifier of the environment the infrastructure belongs to.",+ "name": "Infrastructure name.",+ "type": "Infrastructure type, such as KubernetesDirect or SshWinRmAws.",+ "deploymentType": "Deployment type the infrastructure supports.",+ "yaml": "YAML definition of the infrastructure.",+ "createdAt": "Creation time in milliseconds since the Unix epoch.",+ "lastModifiedAt": "Last modification time in milliseconds since the Unix epoch.",+ },+ },+ "organizations": {+ "description": "Organizations in the Harness account.",+ "docs_url": "https://apidocs.harness.io/organization/getorganizationlist",+ "columns": {+ "identifier": "Organization identifier within the account.",+ "name": "Organization name.",+ "description": "Organization description.",+ "tags": "Tags set on the organization.",+ "createdAt": "Creation time in milliseconds since the Unix epoch.",+ "lastModifiedAt": "Last modification time in milliseconds since the Unix epoch.",+ "harnessManaged": "Whether Harness manages the organization.",+ },+ },+ "projects": {+ "description": "Projects in the Harness account that the API key's user is a member of.",+ "docs_url": "https://apidocs.harness.io/project/getprojectlist",+ "columns": {+ "identifier": "Project identifier within the organization.",+ "orgIdentifier": "Identifier of the organization that owns the project.",+ "name": "Project name.",+ "description": "Project description.",+ "modules": "Harness modules enabled for the project.",+ "tags": "Tags set on the project.",+ "createdAt": "Creation time in milliseconds since the Unix epoch.",+ "lastModifiedAt": "Last modification time in milliseconds since the Unix epoch.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/harness/harness.py b/products/warehouse_sources/backend/temporal/data_imports/sources/harness/harness.pyindex c1957860f124..f7caa8c3b8e9 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/harness/harness.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/harness/harness.py@@ -1,3 +1,4 @@+from collections.abc import Callablefrom typing import TYPE_CHECKING, Anyfrom requests.exceptions import HTTPError@@ -7,13 +8,20 @@ from posthog.dataclasses import frozenfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source import (RESTAPIConfig,rest_api_resource,+ rest_api_resources,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import APIKeyAuth+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.config_setup import (+ make_parent_key_name,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (SinglePagePaginator,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.rest_client import RESTClient-from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import Endpoint+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (+ Endpoint,+ EndpointResource,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.schema import schema_for_resourcefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.harness.settings import (@@ -33,7 +41,11 @@ if TYPE_CHECKING:@frozenclass HarnessResumeConfig:- page: int+ page: int | None = None+ # Fan-out tables resume by parent, see `common.rest_source.__init__._make_paginate_dependent_resource`.+ completed: list[str] | None = None+ current: str | None = None+ child_state: dict[str, Any] | None = Nonedef base_url(config: "HarnessSourceConfig") -> str:@@ -44,15 +56,20 @@ def base_url(config: "HarnessSourceConfig") -> str:def endpoint_config(config: "HarnessSourceConfig", name: str, size: int = 100) -> Endpoint:endpoint = schema_for_resource(ENDPOINTS, name)+ params: dict[str, Any] = {"accountIdentifier": config.account_id}+ if endpoint.scope == "project":+ params["orgIdentifier"] = config.organization_id+ params["projectIdentifier"] = config.project_id+ params[endpoint.size_param] = size+ path = endpoint.path+ if endpoint.parent is not None:+ params["parent_identifier"] = {"type": "resolve", "resource": endpoint.parent.name, "field": "identifier"}+ path = f"{path}?{endpoint.parent.param}={{parent_identifier}}"request: Endpoint = {- "path": endpoint.path,+ "path": path,"method": endpoint.method,- "params": {- "accountIdentifier": config.account_id,- "orgIdentifier": config.organization_id,- "projectIdentifier": config.project_id,- "size": size,- },+ "params": params,+ "paginator": {"type": "page_number", "page_param": endpoint.page_param, "total_path": "data.totalPages"},"data_selector": "data.content","data_selector_required": True,}@@ -67,7 +84,13 @@ def validate_credentials(config: "HarnessSourceConfig", schema_name: str | Noneexcept ValueError as error:return False, str(error)- endpoint = endpoint_config(config, schema_name or "pipelines", size=1)+ probe_name = schema_name or "pipelines"+ parent = schema_for_resource(ENDPOINTS, probe_name).parent+ # Harness governs a fan-out table with its parent's permission, and its list needs a parent id.+ if parent is not None:+ probe_name = parent.name+ endpoint = endpoint_config(config, probe_name, size=1)+ page_param = schema_for_resource(ENDPOINTS, probe_name).page_paramclient = RESTClient(base_url=url,auth=APIKeyAuth(api_key=config.api_key, name="x-api-key"),@@ -81,7 +104,7 @@ def validate_credentials(config: "HarnessSourceConfig", schema_name: str | Noneclient.paginate(path=endpoint["path"] or "",method=endpoint["method"] or "GET",- params={**(endpoint["params"] or {}), "page": 0},+ params={**(endpoint["params"] or {}), page_param: 0},json=endpoint.get("json"),data_selector="data.content",data_selector_required=True,@@ -100,6 +123,24 @@ def validate_credentials(config: "HarnessSourceConfig", schema_name: str | Nonereturn True, None+def row_normalizer(name: str) -> Callable[[dict[str, Any]], dict[str, Any]]:+ endpoint = schema_for_resource(ENDPOINTS, name)++ def normalize_row(row: dict[str, Any]) -> dict[str, Any]:+ if endpoint.wrapper is not None:+ details = row.get(endpoint.wrapper)+ if not isinstance(details, dict):+ raise ValueError(f"Harness returned a {name} row without an identifier.")+ row = {**details, **{key: value for key, value in row.items() if key != endpoint.wrapper}}+ if endpoint.parent is not None:+ row[endpoint.parent.column] = row.pop(make_parent_key_name(endpoint.parent.name, "identifier"))+ if not all(row.get(key) for key in endpoint.primary_keys):+ raise ValueError(f"Harness returned a {name} row without an identifier.")+ return row++ return normalize_row++def harness_source(config: "HarnessSourceConfig",name: str,@@ -108,47 +149,84 @@ def harness_source(manager: "ResumableSourceManager[HarnessResumeConfig]",) -> SourceResponse:endpoint = schema_for_resource(ENDPOINTS, name)+ resource_config: EndpointResource = {+ "name": name,+ "endpoint": endpoint_config(config, name),+ "write_disposition": "replace",+ }+ resources: list[str | EndpointResource] = [resource_config]+ if endpoint.parent is not None:+ resource_config["include_from_parent"] = ["identifier"]+ resources.insert(+ 0,+ {+ "name": endpoint.parent.name,+ "endpoint": endpoint_config(config, endpoint.parent.name),+ "write_disposition": "replace",+ "data_map": row_normalizer(endpoint.parent.name),+ },+ )rest_config: RESTAPIConfig = {"client": {"base_url": base_url(config),"auth": {"type": "api_key", "name": "x-api-key", "api_key": config.api_key, "location": "header"},- "paginator": {"type": "page_number", "page_param": "page", "total_path": "data.totalPages"},"allowed_hosts": [],"allow_redirects": False,"request_timeout": 60,},- "resources": [{"name": name, "endpoint": endpoint_config(config, name), "write_disposition": "replace"}],+ "resources": resources,}state = manager.load_state() if manager.can_resume() else None- def save_checkpoint(paginator_state: dict[str, Any] | None) -> None:- if paginator_state is not None:- manager.save_state(HarnessResumeConfig(page=int(paginator_state["page"])))-- resource = rest_api_resource(- rest_config,- team_id,- job_id,- None,- resume_hook=save_checkpoint,- initial_paginator_state={"page": state.page} if state is not None else None,- )+ if endpoint.parent is None:- def normalize_row(row: dict[str, Any]) -> dict[str, Any]:- if endpoint.wrapper is not None:- details = row.get(endpoint.wrapper)- if not isinstance(details, dict) or not details.get(endpoint.primary_key):- raise ValueError(f"Harness returned a {name} row without an identifier.")- return {**details, **{key: value for key, value in row.items() if key != endpoint.wrapper}}- if not row.get(endpoint.primary_key):- raise ValueError(f"Harness returned a {name} row without an identifier.")- return row+ def save_checkpoint(paginator_state: dict[str, Any] | None) -> None:+ if paginator_state is not None:+ manager.save_state(HarnessResumeConfig(page=int(paginator_state["page"])))++ resource = rest_api_resource(+ rest_config,+ team_id,+ job_id,+ None,+ resume_hook=save_checkpoint,+ initial_paginator_state={"page": state.page} if state is not None and state.page is not None else None,+ )+ else:++ def save_fanout_checkpoint(fanout_state: dict[str, Any] | None) -> None:+ if fanout_state is not None:+ manager.save_state(+ HarnessResumeConfig(+ completed=fanout_state.get("completed"),+ current=fanout_state.get("current"),+ child_state=fanout_state.get("child_state"),+ )+ )++ initial_state = (+ {"completed": state.completed or [], "current": state.current, "child_state": state.child_state}+ if state is not None and (state.completed or state.current)+ else None+ )+ resource = next(+ item+ for item in rest_api_resources(+ rest_config,+ team_id,+ job_id,+ None,+ resume_hook=save_fanout_checkpoint,+ initial_paginator_state=initial_state,+ )+ if getattr(item, "name", None) == name+ )- resource.add_map(normalize_row)+ resource.add_map(row_normalizer(name))return SourceResponse(name=name,items=lambda: resource,- primary_keys=[endpoint.primary_key],+ primary_keys=list(endpoint.primary_keys),sort_mode=None,on_complete=manager.clear_state,)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/harness/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/harness/settings.pyindex e20775b1a30b..ec052a2162d1 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/harness/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/harness/settings.py@@ -14,23 +14,56 @@ PERMISSION_ERROR = "Your Harness token needs view permission for this table in tREGION_ERROR = "Select a supported Harness region."+@frozen+class HarnessParent:+ name: str+ param: str+ column: str++@frozenclass HarnessEndpoint:path: str- primary_key: str = "identifier"+ primary_keys: tuple[str, ...] = ("identifier",)method: Literal["GET", "POST"] = "GET"filter_type: str | None = Nonewrapper: str | None = None+ scope: Literal["account", "project"] = "project"+ page_param: str = "page"+ size_param: str = "size"+ parent: HarnessParent | None = NoneENDPOINTS = {"pipelines": HarnessEndpoint(path="/pipeline/api/pipelines/list", method="POST", filter_type="PipelineSetup"),"executions": HarnessEndpoint(path="/pipeline/api/pipelines/execution/summary",- primary_key="planExecutionId",+ primary_keys=("planExecutionId",),method="POST",filter_type="PipelineExecution",),"services": HarnessEndpoint(path="/ng/api/servicesV2", wrapper="service"),"environments": HarnessEndpoint(path="/ng/api/environmentsV2", wrapper="environment"),+ # Harness requires an environment for this list, so it fans out over the environments list.+ "infrastructures": HarnessEndpoint(+ path="/ng/api/infrastructures",+ primary_keys=("environmentRef", "identifier"),+ wrapper="infrastructure",+ parent=HarnessParent(name="environments", param="environmentIdentifier", column="environmentRef"),+ ),+ "organizations": HarnessEndpoint(+ path="/ng/api/organizations",+ wrapper="organization",+ scope="account",+ page_param="pageIndex",+ size_param="pageSize",+ ),+ "projects": HarnessEndpoint(+ path="/ng/api/projects",+ primary_keys=("orgIdentifier", "identifier"),+ wrapper="project",+ scope="account",+ page_param="pageIndex",+ size_param="pageSize",+ ),}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/harness/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/harness/source.pyindex 497a5ff6dd76..38ada9b8503e 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/harness/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/harness/source.py@@ -113,8 +113,10 @@ class HarnessSource(ResumableSource[HarnessSourceConfig, HarnessResumeConfig]):releaseStatus=ReleaseStatus.BETA,caption=("In Harness, open **My Profile > My API Keys** to create an API key and token. "- "Paste the token below. Give it view permission for pipelines, executions, services, and environments in your project. "- "Each connection imports one project. All tables use full refresh."+ "Paste the token below. Give it view permission for pipelines, executions, services, and environments in your project, "+ "and for organizations and projects in your account. "+ "The organizations and projects tables list your whole account. The other tables import the selected project. "+ "All tables use full refresh."),fields=cast(list[FieldType],diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/harness/tests/test_harness.py b/products/warehouse_sources/backend/temporal/data_imports/sources/harness/tests/test_harness.pyindex 9e58bcd36536..7702274bfcb7 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/harness/tests/test_harness.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/harness/tests/test_harness.py@@ -12,6 +12,7 @@ from requests.exceptions import HTTPErrorfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.resumable import ResumableSourceManagerfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.testing import (+ RecordedRequest,ScriptedResponse,SourceDriver,always,@@ -149,17 +150,42 @@ def test_requests_and_pagination(@pytest.mark.parametrize(- ("status", "schema", "valid", "message"),+ ("status", "schema", "valid", "message", "path", "page_param", "size_param"),[- (200, None, True, None),- (403, None, True, None),- (403, "services", False, "view permission"),- (400, None, False, "account, organization, project, and region"),- (404, None, False, "account, organization, project, and region"),+ (200, None, True, None, "/pipeline/api/pipelines/list", "page", "size"),+ (403, None, True, None, "/pipeline/api/pipelines/list", "page", "size"),+ (403, "services", False, "view permission", "/ng/api/servicesV2", "page", "size"),+ (403, "infrastructures", False, "view permission", "/ng/api/environmentsV2", "page", "size"),+ (200, "organizations", True, None, "/ng/api/organizations", "pageIndex", "pageSize"),+ (+ 400,+ None,+ False,+ "account, organization, project, and region",+ "/pipeline/api/pipelines/list",+ "page",+ "size",+ ),+ (+ 404,+ None,+ False,+ "account, organization, project, and region",+ "/pipeline/api/pipelines/list",+ "page",+ "size",+ ),],)def test_credential_probe(- config: HarnessSourceConfig, status: int, schema: str | None, valid: bool, message: str | None+ config: HarnessSourceConfig,+ status: int,+ schema: str | None,+ valid: bool,+ message: str | None,+ path: str,+ page_param: str,+ size_param: str,) -> None:with scripted_network([ScriptedResponse(status=status, json={"data": {"content": [], "totalPages": 0}})]@@ -173,8 +199,9 @@ def test_credential_probe(assert len(network.requests_log) == 1request = network.requests_log[0]assert request.headers["x-api-key"] == "fake-token"- assert request.param("size") == "1"- assert request.path == ("/ng/api/servicesV2" if schema == "services" else "/pipeline/api/pipelines/list")+ assert request.param(size_param) == "1"+ assert request.param(page_param) == "0"+ assert request.path == path@pytest.mark.parametrize("status", [401, 403])@@ -214,6 +241,7 @@ def test_invalid_region_never_sends_token(config: HarnessSourceConfig, region: s("pipelines", {"data": {"content": [{}]}}, "without an identifier"),("services", {"data": {"content": [{"service": None}]}}, "without an identifier"),("environments", {"data": {"content": [{"environment": {}}]}}, "without an identifier"),+ ("projects", {"data": {"content": [{"project": {"identifier": "web"}}]}}, "without an identifier"),],)def test_malformed_response_fails(driver: SourceDriver, name: str, body: dict[str, Any], error: str) -> None:@@ -221,3 +249,87 @@ def test_malformed_response_fails(driver: SourceDriver, name: str, body: dict[stassert isinstance(result.raised, ValueError)assert error in str(result.raised)assert result.saved_states == []+++ ("name", "path", "row", "expected"),+ [+ (+ "organizations",+ "/ng/api/organizations",+ {"organization": {"identifier": "default", "name": "Default"}, "harnessManaged": True},+ {"identifier": "default", "name": "Default", "harnessManaged": True},+ ),+ (+ "projects",+ "/ng/api/projects",+ {"project": {"orgIdentifier": "default", "identifier": "web"}, "isFavorite": False},+ {"orgIdentifier": "default", "identifier": "web", "isFavorite": False},+ ),+ ],+)+def test_account_tables_page_across_the_account(+ driver: SourceDriver, name: str, path: str, row: dict[str, Any], expected: dict[str, Any]+) -> None:+ page = {"data": {"content": [row], "totalPages": 2}}+ result = driver.run(name, [ScriptedResponse(json=page), ScriptedResponse(json=page)])+ assert result.raised is None+ assert result.rows == [expected, expected]+ assert result.paths == [path, path]+ assert result.params("pageIndex") == ["0", "1"]+ assert result.params("pageSize") == ["100", "100"]+ assert result.params("accountIdentifier") == ["example-account", "example-account"]+ assert result.params("orgIdentifier") == [None, None]+ assert result.params("projectIdentifier") == [None, None]+++def infrastructure_responder(request: RecordedRequest) -> ScriptedResponse:+ if request.path == "/ng/api/environmentsV2":+ return ScriptedResponse(+ json={+ "data": {+ "content": [{"environment": {"identifier": "staging"}}, {"environment": {"identifier": "prod"}}],+ "totalPages": 1,+ }+ }+ )+ page = request.param("page")+ total_pages = 2 if request.param("environmentIdentifier") == "staging" else 1+ row = {"infrastructure": {"identifier": "k8s", "type": "KubernetesDirect"}, "createdAt": 100}+ return ScriptedResponse(+ json={"data": {"content": [row] if page != "1" else [{**row, "createdAt": 200}], "totalPages": total_pages}}+ )+++def test_infrastructures_fan_out_over_environments(driver: SourceDriver) -> None:+ result = driver.run("infrastructures", infrastructure_responder)+ assert result.raised is None+ assert result.rows == [+ {"identifier": "k8s", "type": "KubernetesDirect", "createdAt": 100, "environmentRef": "staging"},+ {"identifier": "k8s", "type": "KubernetesDirect", "createdAt": 200, "environmentRef": "staging"},+ {"identifier": "k8s", "type": "KubernetesDirect", "createdAt": 100, "environmentRef": "prod"},+ ]+ infrastructure_requests = [r for r in result.requests if r.path == "/ng/api/infrastructures"]+ assert [(r.param("environmentIdentifier"), r.param("page")) for r in infrastructure_requests] == [+ ("staging", "0"),+ ("staging", "1"),+ ("prod", "0"),+ ]+ assert {r.param("projectIdentifier") for r in result.requests} == {"example-project"}+ assert result.committed_states[-1].completed == [+ "/ng/api/infrastructures?environmentIdentifier=prod",+ "/ng/api/infrastructures?environmentIdentifier=staging",+ ]+++def test_infrastructures_resume_skips_completed_environments(driver: SourceDriver) -> None:+ result = driver.run(+ "infrastructures",+ infrastructure_responder,+ resume_state=HarnessResumeConfig(completed=["/ng/api/infrastructures?environmentIdentifier=staging"]),+ )+ assert result.raised is None+ assert [row["environmentRef"] for row in result.rows] == ["prod"]+ assert [r.param("environmentIdentifier") for r in result.requests if r.path == "/ng/api/infrastructures"] == [+ "prod"+ ]