:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add three missing tables to the cloudsmith source (#119249)
Tom Owers59e989782bf0
diff --git a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt b/posthog/test/repo_invariants/dataclass_frozen_baseline.txtindex 35878a60ba44..3d44d567ed91 100644--- a/posthog/test/repo_invariants/dataclass_frozen_baseline.txt+++ b/posthog/test/repo_invariants/dataclass_frozen_baseline.txt@@ -392,7 +392,6 @@1 products/warehouse_sources/backend/temporal/data_imports/sources/cloudbeds/settings.py1 products/warehouse_sources/backend/temporal/data_imports/sources/cloudflare/settings.py1 products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/cloudsmith.py-1 products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/settings.py1 products/warehouse_sources/backend/temporal/data_imports/sources/cloudzero/cloudzero.py1 products/warehouse_sources/backend/temporal/data_imports/sources/coassemble/coassemble.py1 products/warehouse_sources/backend/temporal/data_imports/sources/coassemble/settings.pydiff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 214cef483b06..bac7f0928ccd 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -2815,14 +2815,14 @@ Today (8): `audit_log`, `entitlements`, `members`, `packages`, `repositories`, `Diffed against: <https://api.cloudsmith.io/openapi/?format=openapi>-- [ ] `Package usage metrics (GET /metrics/packages/{owner}/{repo}/)` — analyze package downloads over time and identify heavily used versions (high)-- [ ] `Quota history (GET /quota/history/{owner}/)` — track bandwidth and storage consumption over time to forecast package hosting costs (high)-- [ ] `Package dependencies (GET /packages/{owner}/{repo}/{identifier}/dependencies/)` — connect packages to direct and transitive dependencies for vulnerability impact analysis (medium)+- [ ] `Package usage metrics (GET /metrics/packages/{owner}/{repo}/)` — analyze package downloads over time and identify heavily used versions (high); skipped, the endpoint returns one repository-wide summary object (active and inactive package counts plus download and bandwidth aggregates), not per-package or per-period rows+- [x] `Quota history (GET /quota/history/{owner}/)` — track bandwidth and storage consumption over time to forecast package hosting costs (high). Added as `quota_history`.+- [x] `Package dependencies (GET /packages/{owner}/{repo}/{identifier}/dependencies/)` — connect packages to direct and transitive dependencies for vulnerability impact analysis (medium). Added as `package_dependencies` (fan-out over repositories, then packages).- [ ] `Entitlement usage metrics (GET /metrics/entitlements/{owner}/)` — attribute package consumption to customers or distribution entitlements (medium)- [ ] `Team members (GET /orgs/{org}/teams/{team}/members)` — connect synced teams to organization members for access and ownership analysis (medium)- [ ] `License policy violations (GET /orgs/{org}/license-policy-violation/)` — identify packages that breach license policies and measure compliance exposure (medium)- [ ] `Vulnerability policy violations (GET /orgs/{org}/vulnerability-policy-violation/)` — distinguish policy failures from scan findings and prioritize affected packages (medium)-- [ ] `Repository audit log (GET /audit-log/{owner}/{repo}/)` — analyze repository activity through its dedicated log beyond the namespace log currently synced (medium)+- [x] `Repository audit log (GET /audit-log/{owner}/{repo}/)` — analyze repository activity through its dedicated log beyond the namespace log currently synced (medium). Added as `repository_audit_log` (fan-out over repositories).Note: The official OpenAPI specification was parsed for all GET operations and response schemas. Existing repository requests already cover packages, entitlements, and webhooks; the audit table calls the namespace log. Dependencies need requests per package, while team membership needs requests per team. Credentials, policy configuration, and upload operations were excluded.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/canonical_descriptions.pyindex a0fa1a67bfda..819bf682b883 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/canonical_descriptions.py@@ -182,6 +182,49 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"target_slug_perm": "Permanent identifier for the target.",},},+ "repository_audit_log": {+ "description": "An audit log entry for a single repository: who did what, to which object, and when. Availability depends on your Cloudsmith plan.",+ "docs_url": "https://help.cloudsmith.io/reference/audit_log_repo_list",+ "columns": {+ "repository_slug": "Slug of the repository the entry was listed from. Added by PostHog so entries stay distinguishable across repositories.",+ "uuid": "Identifier for the audit log entry.",+ "event": "The event that was recorded.",+ "event_at": "The time the event happened.",+ "context": "The context the event happened in.",+ "actor": "The user or service that performed the action.",+ "actor_kind": "Whether the actor was a user, a service or something else.",+ "actor_slug_perm": "Permanent identifier for the actor.",+ "actor_ip_address": "IP address the action came from.",+ "actor_location": "Geographic location resolved from the actor's IP address.",+ "object": "The object the action was performed on.",+ "object_kind": "The kind of object the action was performed on.",+ "object_slug_perm": "Permanent identifier for the object.",+ },+ },+ "package_dependencies": {+ "description": "A dependency declared by a package, including transitive dependencies where the package format supports them.",+ "docs_url": "https://help.cloudsmith.io/reference/packages_dependencies",+ "columns": {+ "repository_slug": "Slug of the repository the package belongs to. Added by PostHog.",+ "package_slug_perm": "Permanent identifier of the package that declares the dependency. Added by PostHog.",+ "name": "Name of the package that is depended on.",+ "version": "Version constraint for the dependency, if any.",+ "operator": "Comparison operator that applies to the version constraint, for example = or >=.",+ "dep_type": "Kind of dependency, for example Depends, Runtime, Development or Test.",+ },+ },+ "quota_history": {+ "description": "Bandwidth and storage usage for the workspace over each quota period, measured against the plan limits.",+ "docs_url": "https://help.cloudsmith.io/reference/quota_history_read",+ "columns": {+ "start": "Start of the quota period.",+ "end": "End of the quota period.",+ "days": "Number of days in the quota period.",+ "plan": "The Cloudsmith plan that applied in the period.",+ "raw": "Usage in the period as numbers: downloaded and uploaded bytes and storage used, each with its limit and percentage of the limit.",+ "display": "Usage in the period as human-readable strings.",+ },+ },"members": {"description": "A member of the Cloudsmith organization, with their role, team memberships and login activity.","docs_url": "https://help.cloudsmith.io/reference/orgs_members_list",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/cloudsmith.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/cloudsmith.pyindex 28dd38bf926c..e466023aba87 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/cloudsmith.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/cloudsmith.py@@ -20,10 +20,13 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resrest_api_resource,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (+ build_chained_resource,build_dependent_resource,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.paginators import (+ BasePaginator,PageNumberPaginator,+ SinglePagePaginator,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.typing import (ClientConfig,@@ -163,18 +166,35 @@ def validate_credentials(api_key: str, workspace: str, schema_name: Optional[strreturn False, f"Cloudsmith returned an unexpected response (HTTP {response.status_code})."-def get_resource(config: CloudsmithEndpointConfig, workspace: str) -> EndpointResource:- if config.fanout:- raise ValueError(f"Fan-out endpoint '{config.name}' must use the fan-out path")+def _paginator(config: CloudsmithEndpointConfig) -> BasePaginator:+ return CloudsmithPaginator(page_size=config.page_size) if config.paginated else SinglePagePaginator()- endpoint_config: Endpoint = {++def _endpoint(+ config: CloudsmithEndpointConfig, workspace: str, extra_params: Optional[dict[str, Any]] = None+) -> Endpoint:+ params: dict[str, Any] = {"page_size": config.page_size} if config.paginated else {}+ params.update(config.params)+ params.update(extra_params or {})+ endpoint: Endpoint = {"path": config.path.replace("{owner}", workspace),- "params": {"page_size": config.page_size, **config.params},- "paginator": CloudsmithPaginator(page_size=config.page_size),- # Every Cloudsmith list endpoint returns a bare JSON array; anything else is a shape- # change we would rather fail on than silently sync as a single row.+ "params": params,+ "paginator": _paginator(config),+ # Every Cloudsmith list endpoint returns a bare JSON array or an object with a known rows+ # key; anything else is a shape change we would rather fail on than silently sync as a+ # single row."data_selector_required": True,}+ if config.data_selector:+ endpoint["data_selector"] = config.data_selector+ return endpoint+++def get_resource(config: CloudsmithEndpointConfig, workspace: str) -> EndpointResource:+ if config.fanout or config.chained_fanout:+ raise ValueError(f"Fan-out endpoint '{config.name}' must use the fan-out path")++ endpoint_config = _endpoint(config, workspace)resource: EndpointResource = {"name": config.name,@@ -188,10 +208,83 @@ def get_resource(config: CloudsmithEndpointConfig, workspace: str) -> EndpointRereturn resource-def _make_source_response(config: CloudsmithEndpointConfig, items_fn: Callable[[], Any]) -> SourceResponse:+def _chained_resource(+ config: CloudsmithEndpointConfig, api_key: str, workspace: str, team_id: int, job_id: str+) -> Iterable[Any]:+ """Build the repositories -> parent -> child chain for a second-level fan-out endpoint."""+ chained = config.chained_fanout+ assert chained is not None+ middle_config = CLOUDSMITH_ENDPOINTS[chained.parent_name]+ root_fanout = middle_config.fanout+ if root_fanout is None:+ raise ValueError(f"'{chained.parent_name}' does not fan out from repositories")+ root_config = CLOUDSMITH_ENDPOINTS[root_fanout.parent_name]++ root_endpoint = _endpoint(root_config, workspace, root_fanout.parent_params)+ middle_endpoint = _endpoint(+ middle_config,+ workspace,+ {+ root_fanout.resolve_param: {+ "type": "resolve",+ "resource": root_config.name,+ "field": root_fanout.resolve_field,+ }+ },+ )+ child_endpoint = _endpoint(+ config,+ workspace,+ {+ param: {"type": "resolve", "resource": middle_config.name, "field": field_name}+ for param, field_name in chained.resolve_fields.items()+ },+ )+ # A package deleted between its packages page and this fetch must not sink the run.+ child_endpoint["response_actions"] = [{"status_code": 404, "action": "ignore"}]++ root_resource: EndpointResource = {+ "name": root_config.name,+ "table_name": root_config.name,+ "write_disposition": "replace",+ "endpoint": root_endpoint,+ "table_format": "delta",+ }+ middle_resource: EndpointResource = {+ "name": middle_config.name,+ "table_name": middle_config.name,+ "write_disposition": "replace",+ "include_from_parent": root_fanout.include_from_parent,+ "endpoint": middle_endpoint,+ "table_format": "delta",+ }+ child_resource: EndpointResource = {+ "name": config.name,+ "table_name": config.name,+ "write_disposition": "replace",+ "include_from_parent": chained.include_from_parent,+ "endpoint": child_endpoint,+ "table_format": "delta",+ }++ return build_chained_resource(+ resources=[root_resource, middle_resource, child_resource],+ child_name=config.name,+ parent_name=middle_config.name,+ parent_field_renames=chained.parent_field_renames,+ client_config=_client_config(api_key),+ team_id=team_id,+ job_id=job_id,+ )+++def _make_source_response(+ config: CloudsmithEndpointConfig, items_fn: Callable[[], Any], supports_resume: bool = True+) -> SourceResponse:return SourceResponse(name=config.name,items=items_fn,+ supports_resume=supports_resume,primary_keys=config.primary_key if isinstance(config.primary_key, list) else [config.primary_key],sort_mode=config.sort_mode,partition_count=1,@@ -229,6 +322,11 @@ def cloudsmith_source(if state:resumable_source_manager.save_state(CloudsmithResumeConfig(paginator_state=dict(state)))+ if endpoint_config.chained_fanout is not None:+ # A two-level chain saves no checkpoint, so a shutdown cannot hand it off mid-walk.+ chained_items = _chained_resource(endpoint_config, api_key, workspace, team_id, job_id)+ return _make_source_response(endpoint_config, lambda: chained_items, supports_resume=False)+if endpoint_config.fanout is not None:parent_config = CLOUDSMITH_ENDPOINTS[endpoint_config.fanout.parent_name]dependent_resource: Any = build_dependent_resource(diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/settings.pyindex 2700f7fce3ff..2ec0759faca3 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/settings.py@@ -24,7 +24,21 @@ UPLOADED_AT_INCREMENTAL: IncrementalField = {}-@dataclass+@dataclass(frozen=True)+class CloudsmithChainedFanoutConfig:+ """A second-level fan-out whose child path binds several values from one parent row.++ `parent_name` names an endpoint that itself fans out from repositories, so the built chain+ is repositories -> parent -> child. `build_dependent_resource` covers one hop only.+ """++ parent_name: str+ resolve_fields: dict[str, str]+ include_from_parent: list[str]+ parent_field_renames: dict[str, str]+++@dataclass(frozen=True)class CloudsmithEndpointConfig:name: strpath: str@@ -43,6 +57,12 @@ class CloudsmithEndpointConfig:# inert for them rather than a claim about the order Cloudsmith returns rows in.sort_mode: Literal["asc", "desc"] = "asc"fanout: DependentEndpointConfig | None = None+ chained_fanout: CloudsmithChainedFanoutConfig | None = None+ # Key that holds the rows in an object response. None for the endpoints that return a bare+ # JSON array.+ data_selector: str | None = None+ # False for endpoints that return everything in one response and take no page params.+ paginated: bool = True# Response fields dropped from every row before it is yielded.strip_fields: tuple[str, ...] = ()@@ -131,6 +151,37 @@ CLOUDSMITH_ENDPOINTS: dict[str, CloudsmithEndpointConfig] = {primary_key="uuid",partition_key="event_at",),+ "repository_audit_log": CloudsmithEndpointConfig(+ name="repository_audit_log",+ path="/audit-log/{owner}/{repo}/",+ primary_key=["repository_slug", "uuid"],+ partition_key="event_at",+ fanout=_REPOSITORY_FANOUT,+ ),+ "package_dependencies": CloudsmithEndpointConfig(+ name="package_dependencies",+ path="/packages/{owner}/{repo}/{identifier}/dependencies/",+ # Dependency rows carry no identifier of their own. A package can depend on the same+ # name more than once with a different relationship or version constraint.+ primary_key=["repository_slug", "package_slug_perm", "dep_type", "name", "operator", "version"],+ data_selector="dependencies",+ paginated=False,+ chained_fanout=CloudsmithChainedFanoutConfig(+ parent_name="packages",+ # The package rows carry the repository slug the packages request was bound to.+ resolve_fields={"repo": "_repositories_slug", "identifier": "slug_perm"},+ include_from_parent=["_repositories_slug", "slug_perm"],+ parent_field_renames={"_repositories_slug": "repository_slug", "slug_perm": "package_slug_perm"},+ ),+ ),+ "quota_history": CloudsmithEndpointConfig(+ name="quota_history",+ path="/quota/history/{owner}/",+ primary_key="start",+ data_selector="history",+ paginated=False,+ partition_key="start",+ ),"members": CloudsmithEndpointConfig(name="members",path="/orgs/{owner}/members/",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/source.pyindex 7dd35b9eda73..5d80a6f0a183 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/source.py@@ -116,7 +116,7 @@ class CloudsmithSource(ResumableSource[CloudsmithSourceConfig, CloudsmithResumeCname=ExternalDataSourceType.CLOUDSMITH,category=DataWarehouseSourceCategory.ENGINEERING___MONITORING,label="Cloudsmith",- caption="""Sync your Cloudsmith repositories, packages, entitlements, webhooks, vulnerability scans, audit log, members and teams.+ caption="""Sync your Cloudsmith repositories, packages, package dependencies, entitlements, webhooks, vulnerability scans, audit logs, quota history, members and teams.Create an API key in Cloudsmith under **Account settings → API settings**. The key inherits your own access, so use a key from a user who can see every repository you want to sync. The workspace is the slug in your Cloudsmith URLs, for example `acme` in `https://cloudsmith.io/~acme/`.""",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/tests/test_cloudsmith.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/tests/test_cloudsmith.pyindex 206271010eb0..f9d8239d377d 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/tests/test_cloudsmith.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/tests/test_cloudsmith.py@@ -156,7 +156,9 @@ class TestCloudsmithTransport:assert isinstance(resource["endpoint"]["paginator"], CloudsmithPaginator)assert resource["endpoint"]["data_selector_required"] is True- @parameterized.expand([("packages",), ("entitlements",), ("webhooks",)])+ @parameterized.expand(+ [("packages",), ("entitlements",), ("webhooks",), ("repository_audit_log",), ("package_dependencies",)]+ )def test_get_resource_rejects_fanout_endpoints(self, endpoint) -> None:with pytest.raises(ValueError, match="Fan-out endpoint"):get_resource(CLOUDSMITH_ENDPOINTS[endpoint], "acme")@@ -219,11 +221,13 @@ class TestCloudsmithTransport:("packages", {"sort": "date"}),("entitlements", {"sort": "name"}),("webhooks", None),+ ("repository_audit_log", None),])def test_fanout_wiring(self, endpoint: str, expected_child_params: dict[str, str] | None) -> None:- skipped_path = f"/{endpoint}/acme/skip/"- child_path = f"/{endpoint}/acme/prod/"+ child_prefix = CLOUDSMITH_ENDPOINTS[endpoint].path.split("/")[1]+ skipped_path = f"/{child_prefix}/acme/skip/"+ child_path = f"/{child_prefix}/acme/prod/"result = self._driver().run(endpoint,[self._page([{"slug": "skip"}, {"slug": "prod"}]), self._page([{"slug_perm": "item"}])],@@ -231,7 +235,7 @@ class TestCloudsmithTransport:)assert result.raised is None- assert result.paths == ["/v1/repos/acme/", f"/v1/{endpoint}/acme/prod/"]+ assert result.paths == ["/v1/repos/acme/", f"/v1{child_path}"]assert result.params("page_size") == ["100", "100"]# `{owner}` has to be pre-formatted: the fan-out helper only substitutes `{repo}`.assert result.params("page") == ["1", "1"]@@ -268,3 +272,63 @@ class TestCloudsmithTransport:if should_use_incremental_fieldelse None)++ def test_quota_history_reads_rows_from_single_object_response(self) -> None:+ periods = [+ {"start": "2026-08-01T00:00:00Z", "end": "2026-08-31T23:59:59Z", "plan": "Pro", "days": 31},+ {"start": "2026-09-01T00:00:00Z", "end": "2026-09-30T23:59:59Z", "plan": "Pro", "days": 30},+ ]+ result = self._driver().run("quota_history", [ScriptedResponse(json={"history": periods})])++ assert result.raised is None+ assert result.paths == ["/v1/quota/history/acme/"]+ # The endpoint is not paginated, so no page params are sent.+ assert result.params("page") == [None]+ assert result.params("page_size") == [None]+ assert result.rows == periods++ def test_package_dependencies_chains_repositories_then_packages(self) -> None:+ result = self._driver().run(+ "package_dependencies",+ [+ self._page([{"slug": "prod"}]),+ self._page([{"slug_perm": "pkg1", "repository": "prod"}, {"slug_perm": "gone", "repository": "prod"}]),+ ScriptedResponse(+ json={+ "dependencies": [+ {"name": "requests", "version": "2.31", "operator": ">=", "dep_type": "Requires"},+ {"name": "urllib3", "version": None, "operator": None, "dep_type": "Requires"},+ ]+ }+ ),+ # A package deleted between the packages page and its dependencies fetch.+ ScriptedResponse(status=404, json={"detail": "Not found."}),+ ],+ )++ assert result.raised is None+ assert result.paths == [+ "/v1/repos/acme/",+ "/v1/packages/acme/prod/",+ "/v1/packages/acme/prod/pkg1/dependencies/",+ "/v1/packages/acme/prod/gone/dependencies/",+ ]+ assert result.params("page_size") == ["100", "100", None, None]+ assert result.rows == [+ {+ "name": "requests",+ "version": "2.31",+ "operator": ">=",+ "dep_type": "Requires",+ "repository_slug": "prod",+ "package_slug_perm": "pkg1",+ },+ {+ "name": "urllib3",+ "version": None,+ "operator": None,+ "dep_type": "Requires",+ "repository_slug": "prod",+ "package_slug_perm": "pkg1",+ },+ ]diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/tests/test_cloudsmith_source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/tests/test_cloudsmith_source.pyindex 75f36a7bf427..8526240655d2 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/tests/test_cloudsmith_source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/cloudsmith/tests/test_cloudsmith_source.py@@ -30,7 +30,7 @@ class TestCloudsmithSource:# Only the packages list has a server-side time filter (`query=uploaded:>=...`);# everything else would have to fetch every page anyway, so it stays full refresh.- for name in ("repositories", "entitlements", "webhooks", "vulnerabilities", "audit_log", "members", "teams"):+ for name in set(CLOUDSMITH_ENDPOINTS) - {"packages"}:assert schemas[name].supports_incremental is Falseassert schemas[name].incremental_fields == []