Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add 4 missing tables to the Codescene source (#120177)

Tom Owers9b6008d23bd6
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 4ac48c4dd843..e3bd6879c233 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -2975,7 +2975,7 @@ Note: The whole public REST API is four doc pages (applications, artifacts, buil
## Codescene — **thin**
-Today (7): `Analyses`, `AuthorStatistics`, `Components`, `Files`, `Issues`, `Projects`, `TechnicalDebt`
+Today (11): `Analyses`, `AuthorStatistics`, `BranchStatistics`, `CommitActivity`, `Commits`, `Components`, `DeltaAnalyses`, `Files`, `Issues`, `Projects`, `TechnicalDebt`
Diffed against: <https://docs.enterprise.codescene.io/latest/integrations/rest-api.html>
@@ -2983,16 +2983,16 @@ Diffed against: <https://docs.enterprise.codescene.io/latest/integrations/rest-a
- [x] `projects/{project-id}/analyses/latest/issues` — issues from the project management integration, with status, cycle time and the commits and files changed while each was open (high)
- [x] `projects/{project-id}/analyses/latest/technical-debt` — technical debt and refactoring targets, CodeScene's headline metric (high)
- [x] `projects/{project-id}/analyses/latest/author-statistics` — per-author contribution stats; the only way to join code health to people (high)
-- [ ] `projects/{project-id}/analyses/latest/commits` — commit-level rows underpinning every aggregate CodeScene reports (medium)
-- [ ] `projects/{project-id}/analyses/latest/commit-activity` — commit activity time series for delivery-rate dashboards (medium)
-- [ ] `projects/{project-id}/analyses/latest/branch-statistics` — per-branch stats for branching-strategy and lead-time analysis (medium)
-- [ ] `projects/{project-id}/delta-analyses` — per-PR delta analyses - the state/transition history of quality per change (medium)
+- [x] `projects/{project-id}/analyses/latest/commits` — commit-level rows underpinning every aggregate CodeScene reports (medium)
+- [x] `projects/{project-id}/analyses/latest/commit-activity` — commit activity time series for delivery-rate dashboards (medium)
+- [x] `projects/{project-id}/analyses/latest/branch-statistics` — per-branch stats for branching-strategy and lead-time analysis (medium)
+- [x] `projects/{project-id}/delta-analyses` — per-PR delta analyses - the state/transition history of quality per change (medium)
- [ ] `code-coverage/projects/{project-id}/gate-results/outcomes` — coverage quality-gate pass/fail outcomes over a date range (medium)
- [ ] `code-health/projects/{project-id}/safeguards/pr/outcomes` — PR safeguard outcomes, the enforcement signal teams report on (medium)
- [ ] `active-authors` — authoritative author roster; a lookup table for the author ids appearing in analyses (medium)
- [ ] `projects/{project-id}/analyses/latest/experience/languages` — author language experience, used for knowledge-risk and bus-factor reporting (low)
-Note: Static endpoint list; no dynamic table discovery in products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.py (hardcoded CODESCENE_ENDPOINTS). The v2 API exposes ~100 paths; PostHog covers 7. The per-analysis tables still read the 'latest' analysis only, but `Analyses` now carries the run history so an analysis id can be pinned.
+Note: Static endpoint list; no dynamic table discovery in products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.py (hardcoded CODESCENE_ENDPOINTS). The v2 API exposes ~100 paths; PostHog covers 11. The per-analysis tables still read the 'latest' analysis only, but `Analyses` now carries the run history so an analysis id can be pinned.
## Cody — adequate
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/canonical_descriptions.py
index c343e887de3e..86f43369352f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/canonical_descriptions.py
@@ -94,4 +94,59 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"former_contributor": "Whether the author is marked as a former contributor in CodeScene.",
},
},
+ "Commits": {
+ "description": "The commits included in a project's latest analysis, with the lines changed per commit.",
+ "docs_url": "https://docs.enterprise.codescene.io/latest/integrations/rest-api.html",
+ "columns": {
+ "project_id": "Identifier of the CodeScene project this commit belongs to.",
+ "rev": "Hash of the commit.",
+ "date": "Date of the commit.",
+ "author": "Name of the commit author, as recorded in the commit history.",
+ "repository": "Name of the repository that holds the commit.",
+ "files": "Number of files the commit changed.",
+ "loc_added": "Lines of code the commit added.",
+ "loc_deleted": "Lines of code the commit deleted.",
+ "ticket_id": "Issue identifier referenced by the commit message, if any.",
+ },
+ },
+ "CommitActivity": {
+ "description": "Weekly commit activity trend from a project's latest analysis.",
+ "docs_url": "https://docs.enterprise.codescene.io/latest/integrations/rest-api.html",
+ "columns": {
+ "project_id": "Identifier of the CodeScene project this trend covers.",
+ "date": "Date of the week the data point covers.",
+ "authors_at_date": "Number of authors who committed in the week.",
+ "revisions_at_date": "Number of commits made in the week.",
+ },
+ },
+ "BranchStatistics": {
+ "description": "Per-branch delivery statistics from a project's latest analysis, with lead times and delivery risk.",
+ "docs_url": "https://docs.enterprise.codescene.io/latest/integrations/rest-api.html",
+ "columns": {
+ "project_id": "Identifier of the CodeScene project this branch belongs to.",
+ "repository_name": "Name of the repository that holds the branch.",
+ "branch": "Name of the branch.",
+ "commits": "Number of commits on the branch.",
+ "contributing_authors": "Number of authors who committed on the branch.",
+ "first_commit_date": "Date of the first commit on the branch.",
+ "last_commit_date": "Date of the most recent commit on the branch.",
+ "branch_duration_in_hours": "Hours between the first and last commit on the branch.",
+ "lead_time_first_commit_in_hours": "Lead time in hours, counted from the first commit on the branch.",
+ "lead_time_to_merge_in_hours": "Lead time in hours until the branch was merged.",
+ "merged": "Whether the branch has been merged.",
+ "delivery_risk": "CodeScene's delivery risk score for the branch.",
+ "risk_description": "Explanation of the delivery risk score.",
+ },
+ },
+ "DeltaAnalyses": {
+ "description": "Delta analyses for a project: the Code Health checks CodeScene runs on individual changes such as pull requests.",
+ "docs_url": "https://docs.enterprise.codescene.io/latest/integrations/rest-api.html",
+ "columns": {
+ "project_id": "Identifier of the CodeScene project this delta analysis ran for.",
+ "id": "Unique identifier of the delta analysis.",
+ "ref": "API path of the delta analysis resource.",
+ "analysistime": "Time the delta analysis ran.",
+ "repository": "Name of the repository the analyzed change belongs to.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.py
index ad95d202a09d..d8058488007c 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/settings.py
@@ -22,8 +22,9 @@ class CodesceneEndpointConfig:
primary_key: str | list[str] = "id"
page_size: int = 100
# The endpoints differ in what they page on: most take `page` + `page_size` and report
- # `max_pages`, `analyses` pages on `page` alone, and `author-statistics` returns the whole
- # collection in a single unpaginated response.
+ # `max_pages`, `analyses` and `delta-analyses` page on `page` alone, and `author-statistics`,
+ # `branch-statistics` and `commit-activity` return the whole collection in a single
+ # unpaginated response.
paginated: bool = True
page_size_param: str | None = "page_size"
fanout: DependentEndpointConfig | None = None
@@ -126,6 +127,65 @@ CODESCENE_ENDPOINTS: dict[str, CodesceneEndpointConfig] = {
parent_field_renames={"id": "project_id"},
),
),
+ "Commits": CodesceneEndpointConfig(
+ name="Commits",
+ path="/projects/{project_id}/analyses/latest/commits",
+ data_selector="commits",
+ # A project can span several repositories, so a commit hash is only unique per repository.
+ primary_key=["project_id", "repository", "rev"],
+ page_size=200,
+ fanout=DependentEndpointConfig(
+ parent_name="Projects",
+ resolve_param="project_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "project_id"},
+ ),
+ ),
+ "CommitActivity": CodesceneEndpointConfig(
+ name="CommitActivity",
+ path="/projects/{project_id}/analyses/latest/commit-activity",
+ data_selector="commit_activity_trend",
+ primary_key=["project_id", "date"],
+ paginated=False,
+ page_size_param=None,
+ fanout=DependentEndpointConfig(
+ parent_name="Projects",
+ resolve_param="project_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "project_id"},
+ ),
+ ),
+ "BranchStatistics": CodesceneEndpointConfig(
+ name="BranchStatistics",
+ path="/projects/{project_id}/analyses/latest/branch-statistics",
+ data_selector=None,
+ primary_key=["project_id", "repository_name", "branch"],
+ paginated=False,
+ page_size_param=None,
+ fanout=DependentEndpointConfig(
+ parent_name="Projects",
+ resolve_param="project_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "project_id"},
+ ),
+ ),
+ "DeltaAnalyses": CodesceneEndpointConfig(
+ name="DeltaAnalyses",
+ path="/projects/{project_id}/delta-analyses",
+ data_selector="delta_analyses",
+ primary_key=["project_id", "id"],
+ page_size_param=None,
+ fanout=DependentEndpointConfig(
+ parent_name="Projects",
+ resolve_param="project_id",
+ resolve_field="id",
+ include_from_parent=["id"],
+ parent_field_renames={"id": "project_id"},
+ ),
+ ),
}
ENDPOINTS = tuple(CODESCENE_ENDPOINTS)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/source.py
index bc42c7f57807..879fb75b27e8 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/source.py
@@ -59,7 +59,7 @@ class CodesceneSource(ResumableSource[CodesceneSourceConfig, CodesceneResumeConf
category=DataWarehouseSourceCategory.ENGINEERING___MONITORING,
label="CodeScene",
releaseStatus=ReleaseStatus.ALPHA,
- caption="""Sync projects, analysis runs, per-file Code Health metrics, architectural components, code health issues, technical debt refactoring targets, and author statistics from CodeScene.
+ caption="""Sync projects, analysis runs, delta analyses, per-file Code Health metrics, architectural components, code health issues, technical debt refactoring targets, commits, commit activity, branch statistics, and author statistics from CodeScene.
Create a Personal Access Token from the CodeScene API tokens page with an Admin, Architect, or RestApi role. Leave the API base URL blank to use CodeScene Cloud, or enter your on-prem CodeScene server's API URL (for example `https://codescene.yourcompany.com:3003/api/v2`).""",
iconPath="/static/services/codescene.png",
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/tests/test_codescene.py b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/tests/test_codescene.py
index 29a08ca514a4..76caae9b36d4 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/tests/test_codescene.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/codescene/tests/test_codescene.py
@@ -157,6 +157,11 @@ class TestCodesceneSourceFanout:
("technical_debt", "TechnicalDebt", "result", "page_size", PageNumberPaginator),
# Author statistics comes back as a bare array with no envelope and no pagination.
("author_statistics", "AuthorStatistics", None, None, SinglePagePaginator),
+ ("branch_statistics", "BranchStatistics", None, None, SinglePagePaginator),
+ ("commits", "Commits", "commits", "page_size", PageNumberPaginator),
+ # Commit activity is wrapped in an envelope but has no pagination.
+ ("commit_activity", "CommitActivity", "commit_activity_trend", None, SinglePagePaginator),
+ ("delta_analyses", "DeltaAnalyses", "delta_analyses", None, PageNumberPaginator),
]
)
def test_fanout_wiring(