:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 4 missing tables to the Codescene source (#120177)
Tom Owers9b6008d23bd6
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 4ac48c4dd843..e3bd6879c233 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -2975,7 +2975,7 @@ Note: The whole public REST API is four doc pages (applications, artifacts, buil## Codescene — **thin**-Today (7): `Analyses`, `AuthorStatistics`, `Components`, `Files`, `Issues`, `Projects`, `TechnicalDebt`+Today (11): `Analyses`, `AuthorStatistics`, `BranchStatistics`, `CommitActivity`, `Commits`, `Components`, `DeltaAnalyses`, `Files`, `Issues`, `Projects`, `TechnicalDebt`Diffed against: <https://docs.enterprise.codescene.io/latest/integrations/rest-api.html>@@ -2983,16 +2983,16 @@ Diffed against: <https://docs.enterprise.codescene.io/latest/integrations/rest-a- [x] `projects/{project-id}/analyses/latest/issues` — issues from the project management integration, with status, cycle time and the commits and files changed while each was open (high)- [x] `projects/{project-id}/analyses/latest/technical-debt` — technical debt and refactoring targets, CodeScene's headline metric (high)- [x] `projects/{project-id}/analyses/latest/author-statistics` — per-author contribution stats; the only way to join code health to people (high)-- [ ] `projects/{project-id}/analyses/latest/commits` — commit-level rows underpinning every aggregate CodeScene reports (medium)-- [ ] `projects/{project-id}/analyses/latest/commit-activity` — commit activity time series for delivery-rate dashboards (medium)-- [ ] `projects/{project-id}/analyses/latest/branch-statistics` — per-branch stats for branching-strategy and lead-time analysis (medium)-- [ ] `projects/{project-id}/delta-analyses` — per-PR delta analyses - the state/transition history of quality per change (medium)+- [x] `projects/{project-id}/analyses/latest/commits` — commit-level rows underpinning every aggregate CodeScene reports (medium)+- [x] `projects/{project-id}/analyses/latest/commit-activity` — commit activity time series for delivery-rate dashboards (medium)+- [x] `projects/{project-id}/analyses/latest/branch-statistics` — per-branch stats for branching-strategy and lead-time analysis (medium)+- [x] `projects/{project-id}/delta-analyses` — per-PR delta analyses - the state/transition history of quality per change (medium)- [ ] `code-coverage/projects/{project-id}/gate-results/outcomes` — coverage quality-gate pass/fail outcomes over a date range (medium)- [ ] `code-health/projects/{project-id}/safeguards/pr/outcomes` — PR safeguard outcomes, the enforcement signal teams report on (medium)- [ ] `active-authors` — authoritative author roster; a lookup table for the author ids appearing in analyses (medium)- [ ] `projects/{project-id}/analyses/latest/experience/languages` — author language experience, used for knowledge-risk and bus-factor reporting (low)-Note: Static endpoint list; no dynamic table discovery in products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.py (hardcoded CODESCENE_ENDPOINTS). The v2 API exposes ~100 paths; PostHog covers 7. The per-analysis tables still read the 'latest' analysis only, but `Analyses` now carries the run history so an analysis id can be pinned.+Note: Static endpoint list; no dynamic table discovery in products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.py (hardcoded CODESCENE_ENDPOINTS). The v2 API exposes ~100 paths; PostHog covers 11. The per-analysis tables still read the 'latest' analysis only, but `Analyses` now carries the run history so an analysis id can be pinned.## Cody — adequatediff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/canonical_descriptions.pyindex c343e887de3e..86f43369352f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/canonical_descriptions.py@@ -94,4 +94,59 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"former_contributor": "Whether the author is marked as a former contributor in CodeScene.",},},+ "Commits": {+ "description": "The commits included in a project's latest analysis, with the lines changed per commit.",+ "docs_url": "https://docs.enterprise.codescene.io/latest/integrations/rest-api.html",+ "columns": {+ "project_id": "Identifier of the CodeScene project this commit belongs to.",+ "rev": "Hash of the commit.",+ "date": "Date of the commit.",+ "author": "Name of the commit author, as recorded in the commit history.",+ "repository": "Name of the repository that holds the commit.",+ "files": "Number of files the commit changed.",+ "loc_added": "Lines of code the commit added.",+ "loc_deleted": "Lines of code the commit deleted.",+ "ticket_id": "Issue identifier referenced by the commit message, if any.",+ },+ },+ "CommitActivity": {+ "description": "Weekly commit activity trend from a project's latest analysis.",+ "docs_url": "https://docs.enterprise.codescene.io/latest/integrations/rest-api.html",+ "columns": {+ "project_id": "Identifier of the CodeScene project this trend covers.",+ "date": "Date of the week the data point covers.",+ "authors_at_date": "Number of authors who committed in the week.",+ "revisions_at_date": "Number of commits made in the week.",+ },+ },+ "BranchStatistics": {+ "description": "Per-branch delivery statistics from a project's latest analysis, with lead times and delivery risk.",+ "docs_url": "https://docs.enterprise.codescene.io/latest/integrations/rest-api.html",+ "columns": {+ "project_id": "Identifier of the CodeScene project this branch belongs to.",+ "repository_name": "Name of the repository that holds the branch.",+ "branch": "Name of the branch.",+ "commits": "Number of commits on the branch.",+ "contributing_authors": "Number of authors who committed on the branch.",+ "first_commit_date": "Date of the first commit on the branch.",+ "last_commit_date": "Date of the most recent commit on the branch.",+ "branch_duration_in_hours": "Hours between the first and last commit on the branch.",+ "lead_time_first_commit_in_hours": "Lead time in hours, counted from the first commit on the branch.",+ "lead_time_to_merge_in_hours": "Lead time in hours until the branch was merged.",+ "merged": "Whether the branch has been merged.",+ "delivery_risk": "CodeScene's delivery risk score for the branch.",+ "risk_description": "Explanation of the delivery risk score.",+ },+ },+ "DeltaAnalyses": {+ "description": "Delta analyses for a project: the Code Health checks CodeScene runs on individual changes such as pull requests.",+ "docs_url": "https://docs.enterprise.codescene.io/latest/integrations/rest-api.html",+ "columns": {+ "project_id": "Identifier of the CodeScene project this delta analysis ran for.",+ "id": "Unique identifier of the delta analysis.",+ "ref": "API path of the delta analysis resource.",+ "analysistime": "Time the delta analysis ran.",+ "repository": "Name of the repository the analyzed change belongs to.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.pyindex ad95d202a09d..d8058488007c 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.py@@ -22,8 +22,9 @@ class CodesceneEndpointConfig:primary_key: str | list[str] = "id"page_size: int = 100# The endpoints differ in what they page on: most take `page` + `page_size` and report- # `max_pages`, `analyses` pages on `page` alone, and `author-statistics` returns the whole- # collection in a single unpaginated response.+ # `max_pages`, `analyses` and `delta-analyses` page on `page` alone, and `author-statistics`,+ # `branch-statistics` and `commit-activity` return the whole collection in a single+ # unpaginated response.paginated: bool = Truepage_size_param: str | None = "page_size"fanout: DependentEndpointConfig | None = None@@ -126,6 +127,65 @@ CODESCENE_ENDPOINTS: dict[str, CodesceneEndpointConfig] = {parent_field_renames={"id": "project_id"},),),+ "Commits": CodesceneEndpointConfig(+ name="Commits",+ path="/projects/{project_id}/analyses/latest/commits",+ data_selector="commits",+ # A project can span several repositories, so a commit hash is only unique per repository.+ primary_key=["project_id", "repository", "rev"],+ page_size=200,+ fanout=DependentEndpointConfig(+ parent_name="Projects",+ resolve_param="project_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "project_id"},+ ),+ ),+ "CommitActivity": CodesceneEndpointConfig(+ name="CommitActivity",+ path="/projects/{project_id}/analyses/latest/commit-activity",+ data_selector="commit_activity_trend",+ primary_key=["project_id", "date"],+ paginated=False,+ page_size_param=None,+ fanout=DependentEndpointConfig(+ parent_name="Projects",+ resolve_param="project_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "project_id"},+ ),+ ),+ "BranchStatistics": CodesceneEndpointConfig(+ name="BranchStatistics",+ path="/projects/{project_id}/analyses/latest/branch-statistics",+ data_selector=None,+ primary_key=["project_id", "repository_name", "branch"],+ paginated=False,+ page_size_param=None,+ fanout=DependentEndpointConfig(+ parent_name="Projects",+ resolve_param="project_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "project_id"},+ ),+ ),+ "DeltaAnalyses": CodesceneEndpointConfig(+ name="DeltaAnalyses",+ path="/projects/{project_id}/delta-analyses",+ data_selector="delta_analyses",+ primary_key=["project_id", "id"],+ page_size_param=None,+ fanout=DependentEndpointConfig(+ parent_name="Projects",+ resolve_param="project_id",+ resolve_field="id",+ include_from_parent=["id"],+ parent_field_renames={"id": "project_id"},+ ),+ ),}ENDPOINTS = tuple(CODESCENE_ENDPOINTS)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/source.pyindex bc42c7f57807..879fb75b27e8 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/source.py@@ -59,7 +59,7 @@ class CodesceneSource(ResumableSource[CodesceneSourceConfig, CodesceneResumeConfcategory=DataWarehouseSourceCategory.ENGINEERING___MONITORING,label="CodeScene",releaseStatus=ReleaseStatus.ALPHA,- caption="""Sync projects, analysis runs, per-file Code Health metrics, architectural components, code health issues, technical debt refactoring targets, and author statistics from CodeScene.+ caption="""Sync projects, analysis runs, delta analyses, per-file Code Health metrics, architectural components, code health issues, technical debt refactoring targets, commits, commit activity, branch statistics, and author statistics from CodeScene.Create a Personal Access Token from the CodeScene API tokens page with an Admin, Architect, or RestApi role. Leave the API base URL blank to use CodeScene Cloud, or enter your on-prem CodeScene server's API URL (for example `https://codescene.yourcompany.com:3003/api/v2`).""",iconPath="/static/services/codescene.png",diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/tests/test_codescene.py b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/tests/test_codescene.pyindex 29a08ca514a4..76caae9b36d4 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/tests/test_codescene.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/tests/test_codescene.py@@ -157,6 +157,11 @@ class TestCodesceneSourceFanout:("technical_debt", "TechnicalDebt", "result", "page_size", PageNumberPaginator),# Author statistics comes back as a bare array with no envelope and no pagination.("author_statistics", "AuthorStatistics", None, None, SinglePagePaginator),+ ("branch_statistics", "BranchStatistics", None, None, SinglePagePaginator),+ ("commits", "Commits", "commits", "page_size", PageNumberPaginator),+ # Commit activity is wrapped in an envelope but has no pagination.+ ("commit_activity", "CommitActivity", "commit_activity_trend", None, SinglePagePaginator),+ ("delta_analyses", "DeltaAnalyses", "delta_analyses", None, PageNumberPaginator),])def test_fanout_wiring(