Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(data-warehouse): add jira changelog, comment, group and version tables (#122041)

Tom Owers5b37d01280e8
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 239fe4f87839..cc07c02931ff 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -6626,16 +6626,16 @@ Today (14): `boards`, `dashboards`, `fields`, `filters`, `issue_types`, `issues`
Diffed against: <https://developer.atlassian.com/cloud/jira/platform/swagger-v3.v3.json>
-- [ ] `/rest/api/3/issue/{id}/changelog (and /changelog/bulkfetch)` — issue field transition history - the only source for cycle time, time-in-status and reopen rate (high)
+- [x] `/rest/api/3/issue/{id}/changelog (and /changelog/bulkfetch)` — issue field transition history - the only source for cycle time, time-in-status and reopen rate (high). Added as `issue_changelogs` via `/rest/api/3/search/jql` plus `POST /changelog/bulkfetch`, which avoids one request per issue.
- [x] `/rest/agile/1.0/board/{boardId}/sprint` — sprints, the unit every agile velocity and commitment metric is built on (high)
- [x] `/rest/agile/1.0/board` — boards lookup that scopes sprints and backlogs to teams (high)
- [x] `/rest/agile/1.0/sprint/{sprintId}/issue` — sprint-to-issue membership edges needed for velocity and scope-change analysis (high). Added as `sprint_issues` from the Sprint field via `/rest/api/3/search/jql`, because the spec marks this endpoint and the other agile issue listings deprecated.
-- [ ] `/rest/api/3/issue/{id}/comment (or /rest/api/3/comment/list)` — comment volume and response latency, the main collaboration signal on an issue (high)
+- [x] `/rest/api/3/issue/{id}/comment (or /rest/api/3/comment/list)` — comment volume and response latency, the main collaboration signal on an issue (high). Added as `issue_comments` from the comment field of `/rest/api/3/search/jql`; only issues whose embedded list is truncated are read again through `/issue/{id}/comment`.
- [x] `/rest/api/3/issue/{id}/worklog (with /worklog/updated for incrementals)` — logged time per issue and user, the basis for effort and cost reporting (high). Added as `worklogs` via `/worklog/updated` plus `POST /worklog/list`, which avoids one request per issue.
-- [ ] `/rest/api/3/project/{projectIdOrKey}/versions` — lookup resolving the fixVersions/affectedVersions IDs carried on synced issues (medium)
+- [x] `/rest/api/3/project/{projectIdOrKey}/versions` — lookup resolving the fixVersions/affectedVersions IDs carried on synced issues (medium). Added as `versions` via the paginated `/project/{projectIdOrKey}/version`.
- [ ] `/rest/api/3/project/{projectIdOrKey}/components` — lookup resolving the component IDs carried on synced issues (medium)
- [ ] `/rest/api/3/statuscategory` — lookup rolling the already-synced statuses into To Do / In Progress / Done buckets (medium)
-- [ ] `/rest/api/3/group/bulk and /rest/api/3/group/member` — group definitions plus user-to-group membership edges for team-level rollups (medium)
+- [x] `/rest/api/3/group/bulk and /rest/api/3/group/member` — group definitions plus user-to-group membership edges for team-level rollups (medium). Added as `groups` and `group_members`.
- [ ] `/rest/agile/1.0/board/{boardId}/epic` — epic catalog for rolling issues up to initiatives (medium)
- [ ] `/rest/api/3/issueLinkType (plus issue links)` — blocks/duplicates/relates edges between issues (low)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/canonical_descriptions.py
index b5bc0d78b9dc..a987a83adbb6 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/canonical_descriptions.py
@@ -192,4 +192,74 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"issue_key": "Human-readable key of the issue (e.g. PROJ-123).",
},
},
+ "issue_changelogs": {
+ "description": "One change to a Jira issue, with the field transitions it made. Use it for cycle time, time in status, and reopen rates.",
+ "docs_url": "https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issues/#api-rest-api-3-changelog-bulkfetch-post",
+ "columns": {
+ "id": "ID of the changelog entry.",
+ "issue_id": "ID of the issue that changed.",
+ "author": "The user who made the change.",
+ "created": "When the change took place.",
+ "items": "The fields that changed, each with its value before (from, fromString) and after (to, toString) the change.",
+ "historyMetadata": "Metadata about the change, such as the activity or app that made it.",
+ },
+ },
+ "issue_comments": {
+ "description": "A comment on a Jira issue.",
+ "docs_url": "https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issue-comments/#api-rest-api-3-issue-issueidorkey-comment-get",
+ "columns": {
+ "id": "Unique identifier for the comment.",
+ "issue_id": "ID of the issue the comment is on.",
+ "issue_key": "Human-readable key of the issue (e.g. PROJ-123).",
+ "author": "The user who created the comment.",
+ "updateAuthor": "The user who last updated the comment.",
+ "body": "The comment text, in Atlassian Document Format.",
+ "created": "When the comment was created.",
+ "updated": "When the comment was last updated.",
+ "jsdPublic": "Whether the comment is visible in Jira Service Desk.",
+ "visibility": "Restriction on who can view the comment, if any.",
+ "self": "URL of the comment's REST resource.",
+ },
+ },
+ "groups": {
+ "description": "A user group on the Jira site.",
+ "docs_url": "https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-groups/#api-rest-api-3-group-bulk-get",
+ "columns": {
+ "groupId": "Unique identifier for the group.",
+ "name": "The group's name. A group's name can change, so join on groupId.",
+ },
+ },
+ "group_members": {
+ "description": "Membership of a user in a group, one row per user and group, including inactive users.",
+ "docs_url": "https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-groups/#api-rest-api-3-group-member-get",
+ "columns": {
+ "group_id": "ID of the group.",
+ "group_name": "Name of the group.",
+ "accountId": "Account ID of the member.",
+ "accountType": "The account type (e.g. atlassian, app, customer).",
+ "displayName": "The member's display name.",
+ "emailAddress": "The member's email address, if visible.",
+ "active": "Whether the member's account is active.",
+ "timeZone": "The member's time zone.",
+ "self": "URL of the user's REST resource.",
+ },
+ },
+ "versions": {
+ "description": "A project version (release). Issues reference versions in their fixVersions and versions fields.",
+ "docs_url": "https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-project-versions/#api-rest-api-3-project-projectidorkey-version-get",
+ "columns": {
+ "id": "Unique identifier for the version.",
+ "projectId": "ID of the project the version belongs to.",
+ "name": "The version's name.",
+ "description": "The version's description.",
+ "archived": "Whether the version is archived.",
+ "released": "Whether the version is released.",
+ "overdue": "Whether the version is overdue.",
+ "startDate": "The version's start date.",
+ "releaseDate": "The version's release date.",
+ "userStartDate": "The start date, in the format set by the site's date settings.",
+ "userReleaseDate": "The release date, in the format set by the site's date settings.",
+ "self": "URL of the version's REST resource.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/jira.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/jira.py
index 25b956b3ebe0..3a6629883595 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/jira.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/jira.py
@@ -14,6 +14,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
rest_api_resource,
)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import HttpBasicAuth
+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.config_setup import (
+ create_response_hooks,
+)
from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (
build_dependent_resource,
)
@@ -29,6 +32,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.res
from products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probe
from products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponse
from products.warehouse_sources.backend.temporal.data_imports.sources.jira.settings import (
+ DEFAULT_PAGE_SIZE,
JIRA_ENDPOINTS,
JiraEndpointConfig,
)
@@ -49,6 +53,8 @@ JQL_FLOOR_DATETIME = "1970-01-01 00:00"
# that shares the watermark's timestamp is not skipped; merge dedupes the overlap.
WORKLOG_SINCE_OVERLAP = timedelta(minutes=1)
+SEARCH_JQL_PATH = "/rest/api/3/search/jql"
+
SPRINT_FIELD_SCHEMA = "com.pyxis.greenhopper.jira:gh-sprint"
SPRINT_ISSUES_JQL = f'sprint is not EMPTY AND created >= "{JQL_FLOOR_DATETIME}" ORDER BY created ASC'
@@ -194,6 +200,37 @@ def _find_sprint_field(fields: list[Any]) -> Optional[str]:
return None
+def _changelog_rows(
+ issue_changelogs: list[dict[str, Any]], created_floor_millis: Optional[int]
+) -> list[dict[str, Any]]:
+ return [
+ {**history, "issue_id": issue_changelog.get("issueId")}
+ for issue_changelog in issue_changelogs
+ for history in issue_changelog.get("changeHistories") or []
+ if created_floor_millis is None or _created_at_or_after(history, created_floor_millis)
+ ]
+
+
+def _created_at_or_after(history: dict[str, Any], floor_millis: int) -> bool:
+ try:
+ return _to_epoch_millis(history["created"]) >= floor_millis
+ except (KeyError, TypeError, ValueError):
+ return True
+
+
+def _comment_rows(issue: dict[str, Any], comments: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ return [{**comment, "issue_id": issue["id"], "issue_key": issue.get("key")} for comment in comments]
+
+
+def _resume_token_state(
+ resumable_source_manager: ResumableSourceManager[JiraResumeConfig], jql: str
+) -> Optional[dict[str, Any]]:
+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
+ if resume and resume.next_page_token and resume.jql == jql:
+ return {"cursor": resume.next_page_token}
+ return None
+
+
def _rest_client(subdomain: str, email: str, api_token: str) -> RESTClient:
return RESTClient(
base_url=base_url(subdomain),
@@ -344,16 +381,171 @@ def _sprint_issues_source(
)
-def _sprints_source(subdomain: str, email: str, api_token: str, team_id: int, job_id: str) -> SourceResponse:
- config = JIRA_ENDPOINTS["sprints"]
+def _issue_changelogs_source(
+ subdomain: str,
+ email: str,
+ api_token: str,
+ resumable_source_manager: ResumableSourceManager[JiraResumeConfig],
+ db_incremental_field_last_value: Optional[Any],
+) -> SourceResponse:
+ config = JIRA_ENDPOINTS["issue_changelogs"]
+ # A new changelog entry moves the issue's ``updated``, so issues updated since the watermark carry
+ # every changelog entry created since it.
+ jql = _build_issues_jql("updated", db_incremental_field_last_value)
+ initial_paginator_state = _resume_token_state(resumable_source_manager, jql)
+ # Bulkfetch returns the whole history of each issue. Changelog entries never change, so an
+ # incremental run keeps only the ones inside the JQL lookback window.
+ created_floor_millis = (
+ _to_epoch_millis(db_incremental_field_last_value) - int(INCREMENTAL_LOOKBACK.total_seconds() * 1000)
+ if db_incremental_field_last_value
+ else None
+ )
+
+ def items() -> Iterator[list[dict[str, Any]]]:
+ client = _rest_client(subdomain, email, api_token)
+ next_page: dict[str, Any] = {}
+
+ def stage_next_page(state: Optional[dict[str, Any]], has_next: bool) -> None:
+ next_page["cursor"] = state.get("cursor") if state and has_next else None
+
+ def stage_more_changelogs(_state: Optional[dict[str, Any]], has_next: bool) -> None:
+ next_page["more_changelogs"] = has_next
+
+ def checkpoint() -> None:
+ if next_page.get("cursor") is not None:
+ resumable_source_manager.save_state(JiraResumeConfig(next_page_token=next_page["cursor"], jql=jql))
+
+ for issues in client.paginate(
+ SEARCH_JQL_PATH,
+ params={"jql": jql, "maxResults": config.page_size, "fields": "id"},
+ paginator=JiraTokenPaginator(),
+ data_selector="issues",
+ initial_paginator_state=initial_paginator_state,
+ page_state_hook=stage_next_page,
+ ):
+ issue_ids = [str(issue["id"]) for issue in issues if issue.get("id") is not None]
+ if not issue_ids:
+ checkpoint()
+ resumable_source_manager.safe_point()
+ continue
+
+ for issue_changelogs in client.paginate(
+ config.path,
+ method="post",
+ json={"issueIdsOrKeys": issue_ids, "maxResults": config.page_size},
+ paginator=JSONResponseCursorPaginator(
+ cursor_path="nextPageToken", cursor_param="nextPageToken", param_location="json"
+ ),
+ data_selector="issueChangeLogs",
+ page_state_hook=stage_more_changelogs,
+ ):
+ rows = _changelog_rows(issue_changelogs, created_floor_millis)
+ # The next issue page is only safe to resume from once this page's last changelogs are in hand.
+ if not next_page["more_changelogs"]:
+ checkpoint()
+ if rows:
+ yield rows
+ else:
+ resumable_source_manager.safe_point()
+
+ return SourceResponse(
+ name="issue_changelogs",
+ items=items,
+ primary_keys=config.primary_key,
+ partition_count=1,
+ partition_size=1,
+ partition_mode="datetime",
+ partition_format="month",
+ partition_keys=[config.partition_key] if config.partition_key else None,
+ )
+
+
+def _issue_comments_source(
+ subdomain: str,
+ email: str,
+ api_token: str,
+ resumable_source_manager: ResumableSourceManager[JiraResumeConfig],
+ db_incremental_field_last_value: Optional[Any],
+) -> SourceResponse:
+ config = JIRA_ENDPOINTS["issue_comments"]
+ # Adding or editing a comment moves the issue's ``updated``, whichever comment field is the cursor.
+ jql = _build_issues_jql("updated", db_incremental_field_last_value)
+ initial_paginator_state = _resume_token_state(resumable_source_manager, jql)
+
+ def items() -> Iterator[list[dict[str, Any]]]:
+ client = _rest_client(subdomain, email, api_token)
+ # An issue deleted after the search answers 404, which reads as an issue with no comments.
+ skip_deleted_issue = create_response_hooks(
+ [{"status_code": 404, "action": "ignore"}], resource_name="issue_comments"
+ )
+ next_page: dict[str, Any] = {}
+
+ def stage_next_page(state: Optional[dict[str, Any]], has_next: bool) -> None:
+ next_page["cursor"] = state.get("cursor") if state and has_next else None
+
+ for issues in client.paginate(
+ config.path,
+ params={"jql": jql, "maxResults": config.page_size, "fields": "comment"},
+ paginator=JiraTokenPaginator(),
+ data_selector=config.data_key,
+ initial_paginator_state=initial_paginator_state,
+ page_state_hook=stage_next_page,
+ ):
+ rows: list[dict[str, Any]] = []
+ for issue in issues:
+ embedded = (issue.get("fields") or {}).get("comment") or {}
+ comments = embedded.get("comments") or []
+ if (embedded.get("total") or 0) <= len(comments):
+ rows.extend(_comment_rows(issue, comments))
+ continue
+
+ # The search embeds a capped list, so read the whole list for this issue.
+ for page in client.paginate(
+ f"/rest/api/3/issue/{issue['id']}/comment",
+ params={"orderBy": "created"},
+ paginator=JiraOffsetPaginator(
+ limit=DEFAULT_PAGE_SIZE, offset_param="startAt", limit_param="maxResults", total_path="total"
+ ),
+ data_selector="comments",
+ hooks=skip_deleted_issue,
+ ):
+ rows.extend(_comment_rows(issue, page))
+ # Hand rows on after each request so a worker shutdown can stop between them.
+ if rows:
+ yield rows
+ rows = []
+ else:
+ resumable_source_manager.safe_point()
+
+ if next_page.get("cursor") is not None:
+ resumable_source_manager.save_state(JiraResumeConfig(next_page_token=next_page["cursor"], jql=jql))
+ if rows:
+ yield rows
+ else:
+ resumable_source_manager.safe_point()
+
+ return SourceResponse(
+ name="issue_comments",
+ items=items,
+ primary_keys=config.primary_key,
+ partition_count=1,
+ partition_size=1,
+ partition_mode="datetime",
+ partition_format="month",
+ partition_keys=[config.partition_key] if config.partition_key else None,
+ )
+
+
+def _fanout_resource(endpoint: str, subdomain: str, email: str, api_token: str, team_id: int, job_id: str) -> Resource:
+ config = JIRA_ENDPOINTS[endpoint]
assert config.fanout is not None
parent_config = JIRA_ENDPOINTS[config.fanout.parent_name]
- resource = cast(
+ return cast(
Resource,
build_dependent_resource(
endpoint_configs=JIRA_ENDPOINTS,
- child_endpoint="sprints",
+ child_endpoint=endpoint,
fanout=config.fanout,
client_config=_client_config(subdomain, email, api_token),
path_format_values={},
@@ -369,6 +561,24 @@ def _sprints_source(subdomain: str, email: str, api_token: str, team_id: int, jo
),
)
+
+def _fanout_source(
+ endpoint: str, subdomain: str, email: str, api_token: str, team_id: int, job_id: str
+) -> SourceResponse:
+ resource = _fanout_resource(endpoint, subdomain, email, api_token, team_id, job_id)
+ return SourceResponse(
+ name=endpoint,
+ items=lambda: resource,
+ primary_keys=JIRA_ENDPOINTS[endpoint].primary_key,
+ partition_count=1,
+ partition_size=1,
+ )
+
+
+def _sprints_source(subdomain: str, email: str, api_token: str, team_id: int, job_id: str) -> SourceResponse:
+ config = JIRA_ENDPOINTS["sprints"]
+ resource = _fanout_resource("sprints", subdomain, email, api_token, team_id, job_id)
+
seen: set[Any] = set()
def first_listing(sprint: dict[str, Any]) -> bool:
@@ -428,6 +638,24 @@ def jira_source(
return _sprint_issues_source(subdomain, email, api_token, logger, resumable_source_manager)
if endpoint == "sprints":
return _sprints_source(subdomain, email, api_token, team_id, job_id)
+ if endpoint in ("group_members", "versions"):
+ return _fanout_source(endpoint, subdomain, email, api_token, team_id, job_id)
+ if endpoint == "issue_changelogs":
+ return _issue_changelogs_source(
+ subdomain,
+ email,
+ api_token,
+ resumable_source_manager,
+ db_incremental_field_last_value if should_use_incremental_field else None,
+ )
+ if endpoint == "issue_comments":
+ return _issue_comments_source(
+ subdomain,
+ email,
+ api_token,
+ resumable_source_manager,
+ db_incremental_field_last_value if should_use_incremental_field else None,
+ )
resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None
initial_paginator_state: Optional[dict[str, Any]] = None
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/settings.py
index 2acef111cecb..8bbbe2c969c4 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/settings.py
@@ -18,6 +18,12 @@ DEFAULT_PAGE_SIZE = 100
AGILE_PAGE_SIZE = 50
# Both ``/worklog/updated`` and ``/worklog/list`` cap a page at 1000 worklogs.
WORKLOG_PAGE_SIZE = 1000
+# ``/changelog/bulkfetch`` takes at most 1000 issue ids per request.
+CHANGELOG_PAGE_SIZE = 1000
+# ``/group/member`` caps a page at 50; ``/group/bulk`` defaults to 50 and documents no maximum.
+GROUP_PAGE_SIZE = 50
+# ``/project/{id}/version`` defaults to 50 and documents no maximum.
+VERSION_PAGE_SIZE = 50
@dataclass(frozen=True)
@@ -50,10 +56,10 @@ def _datetime_field(name: str) -> IncrementalField:
# Endpoint catalog. Jira Cloud REST API v3 (https://developer.atlassian.com/cloud/jira/platform/rest/v3/).
-# We deliberately exclude the per-issue fan-out streams (comments, votes, watchers, remote links,
-# properties, transitions) — each requires one HTTP request per issue and degrades badly on large
-# instances. Worklogs avoid that through the instance-wide ``/worklog/updated`` feed. Issues carry the
-# bulk of the useful data and sync incrementally.
+# We deliberately exclude the per-issue fan-out streams (votes, watchers, remote links, properties,
+# transitions) — each requires one HTTP request per issue and degrades badly on large instances.
+# Worklogs, changelogs and comments avoid that by reading through instance-wide or bulk endpoints.
+# Issues carry the bulk of the useful data and sync incrementally.
JIRA_ENDPOINTS: dict[str, JiraEndpointConfig] = {
"issues": JiraEndpointConfig(
name="issues",
@@ -177,6 +183,73 @@ JIRA_ENDPOINTS: dict[str, JiraEndpointConfig] = {
pagination="custom",
data_key="issues",
),
+ "issue_changelogs": JiraEndpointConfig(
+ name="issue_changelogs",
+ # Issue ids come from the enhanced JQL search, then their changelogs are read in bulk with
+ # ``POST /changelog/bulkfetch``, which avoids one request per issue.
+ path="/rest/api/3/changelog/bulkfetch",
+ primary_key=["issue_id", "id"],
+ pagination="custom",
+ supports_incremental=True,
+ partition_key="created",
+ incremental_fields=[_datetime_field("created")],
+ page_size=CHANGELOG_PAGE_SIZE,
+ ),
+ "issue_comments": JiraEndpointConfig(
+ name="issue_comments",
+ # Comments ride along on the enhanced JQL search (``fields=comment``). Only an issue whose
+ # embedded list is truncated is read again through ``/issue/{id}/comment``.
+ path="/rest/api/3/search/jql",
+ primary_key=["id"],
+ pagination="custom",
+ data_key="issues",
+ supports_incremental=True,
+ partition_key="created",
+ incremental_fields=[_datetime_field("updated"), _datetime_field("created")],
+ ),
+ "groups": JiraEndpointConfig(
+ name="groups",
+ path="/rest/api/3/group/bulk",
+ primary_key=["groupId"],
+ pagination="offset",
+ data_key="values",
+ page_size=GROUP_PAGE_SIZE,
+ ),
+ "group_members": JiraEndpointConfig(
+ name="group_members",
+ path="/rest/api/3/group/member?groupId={groupId}",
+ primary_key=["group_id", "accountId"],
+ pagination="offset",
+ data_key="values",
+ page_size=GROUP_PAGE_SIZE,
+ fanout=DependentEndpointConfig(
+ parent_name="groups",
+ resolve_param="groupId",
+ resolve_field="groupId",
+ include_from_parent=["groupId", "name"],
+ parent_field_renames={"groupId": "group_id", "name": "group_name"},
+ child_params={"includeInactiveUsers": "true"},
+ # A group deleted after the listing answers 404.
+ child_response_actions=[{"status_code": 404, "action": "ignore"}],
+ ),
+ ),
+ "versions": JiraEndpointConfig(
+ name="versions",
+ # The paginated listing; ``/project/{id}/versions`` returns every version in one response.
+ path="/rest/api/3/project/{projectId}/version",
+ primary_key=["id"],
+ pagination="offset",
+ data_key="values",
+ page_size=VERSION_PAGE_SIZE,
+ fanout=DependentEndpointConfig(
+ parent_name="projects",
+ resolve_param="projectId",
+ resolve_field="id",
+ include_from_parent=[],
+ # A project deleted after the listing answers 404.
+ child_response_actions=[{"status_code": 404, "action": "ignore"}],
+ ),
+ ),
}
ENDPOINTS = tuple(JIRA_ENDPOINTS.keys())
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/tests/test_jira.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/tests/test_jira.py
index f7c40b5d0904..351130732d87 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/tests/test_jira.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/tests/test_jira.py
@@ -334,3 +334,173 @@ class TestSprintIssues:
assert result.raised is None
assert result.rows == []
assert result.paths == ["/rest/api/3/field"]
+
+
+class TestIssueChangelogs:
+ def test_bulk_fetches_each_issue_page_and_checkpoints_after_its_last_changelog_page(self) -> None:
+ result = _driver().run(
+ "issue_changelogs",
+ route(
+ {
+ "/rest/api/3/search/jql": [
+ ScriptedResponse(json={"issues": [{"id": "10"}, {"id": "11"}], "nextPageToken": "tok2"}),
+ ScriptedResponse(json={"issues": [{"id": "12"}], "isLast": True}),
+ ],
+ "/rest/api/3/changelog/bulkfetch": [
+ ScriptedResponse(
+ json={
+ "issueChangeLogs": [
+ {"issueId": "10", "changeHistories": [{"id": "100", "created": "c1"}]},
+ {"issueId": "11", "changeHistories": [{"id": "101", "created": "c2"}]},
+ ],
+ "nextPageToken": "cl2",
+ }
+ ),
+ ScriptedResponse(
+ json={"issueChangeLogs": [{"issueId": "10", "changeHistories": [{"id": "102"}]}]}
+ ),
+ ScriptedResponse(json={"issueChangeLogs": []}),
+ ],
+ }
+ ),
+ )
+
+ assert result.raised is None
+ assert result.params("fields") == ["id", None, None, "id", None]
+ assert [request.json() for request in result.requests if request.method == "POST"] == [
+ {"issueIdsOrKeys": ["10", "11"], "maxResults": 1000},
+ {"issueIdsOrKeys": ["10", "11"], "maxResults": 1000, "nextPageToken": "cl2"},
+ {"issueIdsOrKeys": ["12"], "maxResults": 1000},
+ ]
+ assert [(row["issue_id"], row["id"]) for row in result.rows] == [("10", "100"), ("11", "101"), ("10", "102")]
+ full_sync_jql = 'updated >= "1970-01-01 00:00" ORDER BY updated ASC'
+ assert result.saved_states == [JiraResumeConfig(next_page_token="tok2", jql=full_sync_jql)]
+
+ @parameterized.expand(
+ [
+ ("full_refresh", {}, 'updated >= "1970-01-01 00:00" ORDER BY updated ASC', ["old", "recent"]),
+ # The JQL and the changelog filter both use the watermark less the one day lookback.
+ (
+ "incremental",
+ {
+ "incremental_field": "created",
+ "db_incremental_field_last_value": datetime(2026, 3, 4, 2, 58, tzinfo=UTC),
+ },
+ 'updated >= "2026-03-03 02:58" ORDER BY updated ASC',
+ ["recent"],
+ ),
+ ]
+ )
+ def test_incremental_shapes_jql_and_drops_history_before_the_window(
+ self, _name: str, run_kwargs: dict[str, Any], expected_jql: str, expected_ids: list[str]
+ ) -> None:
+ result = _driver().run(
+ "issue_changelogs",
+ route(
+ {
+ "/rest/api/3/search/jql": [ScriptedResponse(json={"issues": [{"id": "10"}], "isLast": True})],
+ "/rest/api/3/changelog/bulkfetch": [
+ ScriptedResponse(
+ json={
+ "issueChangeLogs": [
+ {
+ "issueId": "10",
+ "changeHistories": [
+ {"id": "old", "created": "2026-03-03T02:57:00.000+0000"},
+ {"id": "recent", "created": "2026-03-03T02:58:00.000+0000"},
+ ],
+ }
+ ]
+ }
+ )
+ ],
+ }
+ ),
+ **run_kwargs,
+ )
+
+ assert result.raised is None
+ assert result.params("jql")[0] == expected_jql
+ assert [row["id"] for row in result.rows] == expected_ids
+
+
+class TestIssueComments:
+ def test_reads_embedded_comments_and_refetches_truncated_lists_of_existing_issues(self) -> None:
+ result = _driver().run(
+ "issue_comments",
+ route(
+ {
+ "/rest/api/3/search/jql": [
+ ScriptedResponse(
+ json={
+ "issues": [
+ {
+ "id": "1",
+ "key": "P-1",
+ "fields": {"comment": {"comments": [{"id": "a"}], "total": 1}},
+ },
+ {
+ "id": "2",
+ "key": "P-2",
+ "fields": {"comment": {"comments": [{"id": "b"}], "total": 2}},
+ },
+ {"id": "3", "key": "P-3", "fields": {"comment": {"comments": [], "total": 0}}},
+ {
+ "id": "4",
+ "key": "P-4",
+ "fields": {"comment": {"comments": [{"id": "d"}], "total": 2}},
+ },
+ ],
+ "nextPageToken": "tok2",
+ }
+ ),
+ ScriptedResponse(json={"issues": [], "isLast": True}),
+ ],
+ "/rest/api/3/issue/2/comment": [
+ ScriptedResponse(json={"comments": [{"id": "b"}, {"id": "c"}], "startAt": 0, "total": 2})
+ ],
+ # Deleted after the search returned it.
+ "/rest/api/3/issue/4/comment": [
+ ScriptedResponse(status=404, json={"errorMessages": ["Issue does not exist"]})
+ ],
+ }
+ ),
+ )
+
+ assert result.raised is None
+ assert result.paths == [
+ "/rest/api/3/search/jql",
+ "/rest/api/3/issue/2/comment",
+ "/rest/api/3/issue/4/comment",
+ "/rest/api/3/search/jql",
+ ]
+ assert result.params("fields") == ["comment", None, None, "comment"]
+ assert [(row["issue_key"], row["id"]) for row in result.rows] == [("P-1", "a"), ("P-2", "b"), ("P-2", "c")]
+
+
+class TestGroupMembers:
+ def test_fans_out_over_groups_with_group_columns_and_skips_deleted_groups(self) -> None:
+ result = _driver().run(
+ "group_members",
+ route(
+ {
+ "/rest/api/3/group/bulk": [
+ ScriptedResponse(
+ json={
+ "values": [{"groupId": "g1", "name": "devs"}, {"groupId": "g2", "name": "gone"}],
+ "isLast": True,
+ }
+ )
+ ],
+ "/rest/api/3/group/member": [
+ ScriptedResponse(json={"values": [{"accountId": "u1"}], "isLast": True}),
+ ScriptedResponse(status=404, json={"errorMessages": ["Group not found"]}),
+ ],
+ }
+ ),
+ )
+
+ assert result.raised is None
+ assert result.params("groupId") == [None, "g1", "g2"]
+ assert result.params("includeInactiveUsers") == [None, "true", "true"]
+ assert result.rows == [{"accountId": "u1", "group_id": "g1", "group_name": "devs"}]