:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add jira changelog, comment, group and version tables (#122041)
Tom Owers5b37d01280e8
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 239fe4f87839..cc07c02931ff 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -6626,16 +6626,16 @@ Today (14): `boards`, `dashboards`, `fields`, `filters`, `issue_types`, `issues`Diffed against: <https://developer.atlassian.com/cloud/jira/platform/swagger-v3.v3.json>-- [ ] `/rest/api/3/issue/{id}/changelog (and /changelog/bulkfetch)` — issue field transition history - the only source for cycle time, time-in-status and reopen rate (high)+- [x] `/rest/api/3/issue/{id}/changelog (and /changelog/bulkfetch)` — issue field transition history - the only source for cycle time, time-in-status and reopen rate (high). Added as `issue_changelogs` via `/rest/api/3/search/jql` plus `POST /changelog/bulkfetch`, which avoids one request per issue.- [x] `/rest/agile/1.0/board/{boardId}/sprint` — sprints, the unit every agile velocity and commitment metric is built on (high)- [x] `/rest/agile/1.0/board` — boards lookup that scopes sprints and backlogs to teams (high)- [x] `/rest/agile/1.0/sprint/{sprintId}/issue` — sprint-to-issue membership edges needed for velocity and scope-change analysis (high). Added as `sprint_issues` from the Sprint field via `/rest/api/3/search/jql`, because the spec marks this endpoint and the other agile issue listings deprecated.-- [ ] `/rest/api/3/issue/{id}/comment (or /rest/api/3/comment/list)` — comment volume and response latency, the main collaboration signal on an issue (high)+- [x] `/rest/api/3/issue/{id}/comment (or /rest/api/3/comment/list)` — comment volume and response latency, the main collaboration signal on an issue (high). Added as `issue_comments` from the comment field of `/rest/api/3/search/jql`; only issues whose embedded list is truncated are read again through `/issue/{id}/comment`.- [x] `/rest/api/3/issue/{id}/worklog (with /worklog/updated for incrementals)` — logged time per issue and user, the basis for effort and cost reporting (high). Added as `worklogs` via `/worklog/updated` plus `POST /worklog/list`, which avoids one request per issue.-- [ ] `/rest/api/3/project/{projectIdOrKey}/versions` — lookup resolving the fixVersions/affectedVersions IDs carried on synced issues (medium)+- [x] `/rest/api/3/project/{projectIdOrKey}/versions` — lookup resolving the fixVersions/affectedVersions IDs carried on synced issues (medium). Added as `versions` via the paginated `/project/{projectIdOrKey}/version`.- [ ] `/rest/api/3/project/{projectIdOrKey}/components` — lookup resolving the component IDs carried on synced issues (medium)- [ ] `/rest/api/3/statuscategory` — lookup rolling the already-synced statuses into To Do / In Progress / Done buckets (medium)-- [ ] `/rest/api/3/group/bulk and /rest/api/3/group/member` — group definitions plus user-to-group membership edges for team-level rollups (medium)+- [x] `/rest/api/3/group/bulk and /rest/api/3/group/member` — group definitions plus user-to-group membership edges for team-level rollups (medium). Added as `groups` and `group_members`.- [ ] `/rest/agile/1.0/board/{boardId}/epic` — epic catalog for rolling issues up to initiatives (medium)- [ ] `/rest/api/3/issueLinkType (plus issue links)` — blocks/duplicates/relates edges between issues (low)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/canonical_descriptions.pyindex b5bc0d78b9dc..a987a83adbb6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/canonical_descriptions.py@@ -192,4 +192,74 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"issue_key": "Human-readable key of the issue (e.g. PROJ-123).",},},+ "issue_changelogs": {+ "description": "One change to a Jira issue, with the field transitions it made. Use it for cycle time, time in status, and reopen rates.",+ "docs_url": "https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issues/#api-rest-api-3-changelog-bulkfetch-post",+ "columns": {+ "id": "ID of the changelog entry.",+ "issue_id": "ID of the issue that changed.",+ "author": "The user who made the change.",+ "created": "When the change took place.",+ "items": "The fields that changed, each with its value before (from, fromString) and after (to, toString) the change.",+ "historyMetadata": "Metadata about the change, such as the activity or app that made it.",+ },+ },+ "issue_comments": {+ "description": "A comment on a Jira issue.",+ "docs_url": "https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-issue-comments/#api-rest-api-3-issue-issueidorkey-comment-get",+ "columns": {+ "id": "Unique identifier for the comment.",+ "issue_id": "ID of the issue the comment is on.",+ "issue_key": "Human-readable key of the issue (e.g. PROJ-123).",+ "author": "The user who created the comment.",+ "updateAuthor": "The user who last updated the comment.",+ "body": "The comment text, in Atlassian Document Format.",+ "created": "When the comment was created.",+ "updated": "When the comment was last updated.",+ "jsdPublic": "Whether the comment is visible in Jira Service Desk.",+ "visibility": "Restriction on who can view the comment, if any.",+ "self": "URL of the comment's REST resource.",+ },+ },+ "groups": {+ "description": "A user group on the Jira site.",+ "docs_url": "https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-groups/#api-rest-api-3-group-bulk-get",+ "columns": {+ "groupId": "Unique identifier for the group.",+ "name": "The group's name. A group's name can change, so join on groupId.",+ },+ },+ "group_members": {+ "description": "Membership of a user in a group, one row per user and group, including inactive users.",+ "docs_url": "https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-groups/#api-rest-api-3-group-member-get",+ "columns": {+ "group_id": "ID of the group.",+ "group_name": "Name of the group.",+ "accountId": "Account ID of the member.",+ "accountType": "The account type (e.g. atlassian, app, customer).",+ "displayName": "The member's display name.",+ "emailAddress": "The member's email address, if visible.",+ "active": "Whether the member's account is active.",+ "timeZone": "The member's time zone.",+ "self": "URL of the user's REST resource.",+ },+ },+ "versions": {+ "description": "A project version (release). Issues reference versions in their fixVersions and versions fields.",+ "docs_url": "https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-project-versions/#api-rest-api-3-project-projectidorkey-version-get",+ "columns": {+ "id": "Unique identifier for the version.",+ "projectId": "ID of the project the version belongs to.",+ "name": "The version's name.",+ "description": "The version's description.",+ "archived": "Whether the version is archived.",+ "released": "Whether the version is released.",+ "overdue": "Whether the version is overdue.",+ "startDate": "The version's start date.",+ "releaseDate": "The version's release date.",+ "userStartDate": "The start date, in the format set by the site's date settings.",+ "userReleaseDate": "The release date, in the format set by the site's date settings.",+ "self": "URL of the version's REST resource.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/jira.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/jira.pyindex 25b956b3ebe0..3a6629883595 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/jira.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/jira.py@@ -14,6 +14,9 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resrest_api_resource,)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.auth import HttpBasicAuth+from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.config_setup import (+ create_response_hooks,+)from products.warehouse_sources.backend.temporal.data_imports.sources.common.rest_source.fanout import (build_dependent_resource,)@@ -29,6 +32,7 @@ from products.warehouse_sources.backend.temporal.data_imports.sources.common.resfrom products.warehouse_sources.backend.temporal.data_imports.sources.common.source_helpers import validate_via_probefrom products.warehouse_sources.backend.temporal.data_imports.sources.common.typings import SourceResponsefrom products.warehouse_sources.backend.temporal.data_imports.sources.jira.settings import (+ DEFAULT_PAGE_SIZE,JIRA_ENDPOINTS,JiraEndpointConfig,)@@ -49,6 +53,8 @@ JQL_FLOOR_DATETIME = "1970-01-01 00:00"# that shares the watermark's timestamp is not skipped; merge dedupes the overlap.WORKLOG_SINCE_OVERLAP = timedelta(minutes=1)+SEARCH_JQL_PATH = "/rest/api/3/search/jql"+SPRINT_FIELD_SCHEMA = "com.pyxis.greenhopper.jira:gh-sprint"SPRINT_ISSUES_JQL = f'sprint is not EMPTY AND created >= "{JQL_FLOOR_DATETIME}" ORDER BY created ASC'@@ -194,6 +200,37 @@ def _find_sprint_field(fields: list[Any]) -> Optional[str]:return None+def _changelog_rows(+ issue_changelogs: list[dict[str, Any]], created_floor_millis: Optional[int]+) -> list[dict[str, Any]]:+ return [+ {**history, "issue_id": issue_changelog.get("issueId")}+ for issue_changelog in issue_changelogs+ for history in issue_changelog.get("changeHistories") or []+ if created_floor_millis is None or _created_at_or_after(history, created_floor_millis)+ ]+++def _created_at_or_after(history: dict[str, Any], floor_millis: int) -> bool:+ try:+ return _to_epoch_millis(history["created"]) >= floor_millis+ except (KeyError, TypeError, ValueError):+ return True+++def _comment_rows(issue: dict[str, Any], comments: list[dict[str, Any]]) -> list[dict[str, Any]]:+ return [{**comment, "issue_id": issue["id"], "issue_key": issue.get("key")} for comment in comments]+++def _resume_token_state(+ resumable_source_manager: ResumableSourceManager[JiraResumeConfig], jql: str+) -> Optional[dict[str, Any]]:+ resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else None+ if resume and resume.next_page_token and resume.jql == jql:+ return {"cursor": resume.next_page_token}+ return None++def _rest_client(subdomain: str, email: str, api_token: str) -> RESTClient:return RESTClient(base_url=base_url(subdomain),@@ -344,16 +381,171 @@ def _sprint_issues_source()-def _sprints_source(subdomain: str, email: str, api_token: str, team_id: int, job_id: str) -> SourceResponse:- config = JIRA_ENDPOINTS["sprints"]+def _issue_changelogs_source(+ subdomain: str,+ email: str,+ api_token: str,+ resumable_source_manager: ResumableSourceManager[JiraResumeConfig],+ db_incremental_field_last_value: Optional[Any],+) -> SourceResponse:+ config = JIRA_ENDPOINTS["issue_changelogs"]+ # A new changelog entry moves the issue's ``updated``, so issues updated since the watermark carry+ # every changelog entry created since it.+ jql = _build_issues_jql("updated", db_incremental_field_last_value)+ initial_paginator_state = _resume_token_state(resumable_source_manager, jql)+ # Bulkfetch returns the whole history of each issue. Changelog entries never change, so an+ # incremental run keeps only the ones inside the JQL lookback window.+ created_floor_millis = (+ _to_epoch_millis(db_incremental_field_last_value) - int(INCREMENTAL_LOOKBACK.total_seconds() * 1000)+ if db_incremental_field_last_value+ else None+ )++ def items() -> Iterator[list[dict[str, Any]]]:+ client = _rest_client(subdomain, email, api_token)+ next_page: dict[str, Any] = {}++ def stage_next_page(state: Optional[dict[str, Any]], has_next: bool) -> None:+ next_page["cursor"] = state.get("cursor") if state and has_next else None++ def stage_more_changelogs(_state: Optional[dict[str, Any]], has_next: bool) -> None:+ next_page["more_changelogs"] = has_next++ def checkpoint() -> None:+ if next_page.get("cursor") is not None:+ resumable_source_manager.save_state(JiraResumeConfig(next_page_token=next_page["cursor"], jql=jql))++ for issues in client.paginate(+ SEARCH_JQL_PATH,+ params={"jql": jql, "maxResults": config.page_size, "fields": "id"},+ paginator=JiraTokenPaginator(),+ data_selector="issues",+ initial_paginator_state=initial_paginator_state,+ page_state_hook=stage_next_page,+ ):+ issue_ids = [str(issue["id"]) for issue in issues if issue.get("id") is not None]+ if not issue_ids:+ checkpoint()+ resumable_source_manager.safe_point()+ continue++ for issue_changelogs in client.paginate(+ config.path,+ method="post",+ json={"issueIdsOrKeys": issue_ids, "maxResults": config.page_size},+ paginator=JSONResponseCursorPaginator(+ cursor_path="nextPageToken", cursor_param="nextPageToken", param_location="json"+ ),+ data_selector="issueChangeLogs",+ page_state_hook=stage_more_changelogs,+ ):+ rows = _changelog_rows(issue_changelogs, created_floor_millis)+ # The next issue page is only safe to resume from once this page's last changelogs are in hand.+ if not next_page["more_changelogs"]:+ checkpoint()+ if rows:+ yield rows+ else:+ resumable_source_manager.safe_point()++ return SourceResponse(+ name="issue_changelogs",+ items=items,+ primary_keys=config.primary_key,+ partition_count=1,+ partition_size=1,+ partition_mode="datetime",+ partition_format="month",+ partition_keys=[config.partition_key] if config.partition_key else None,+ )+++def _issue_comments_source(+ subdomain: str,+ email: str,+ api_token: str,+ resumable_source_manager: ResumableSourceManager[JiraResumeConfig],+ db_incremental_field_last_value: Optional[Any],+) -> SourceResponse:+ config = JIRA_ENDPOINTS["issue_comments"]+ # Adding or editing a comment moves the issue's ``updated``, whichever comment field is the cursor.+ jql = _build_issues_jql("updated", db_incremental_field_last_value)+ initial_paginator_state = _resume_token_state(resumable_source_manager, jql)++ def items() -> Iterator[list[dict[str, Any]]]:+ client = _rest_client(subdomain, email, api_token)+ # An issue deleted after the search answers 404, which reads as an issue with no comments.+ skip_deleted_issue = create_response_hooks(+ [{"status_code": 404, "action": "ignore"}], resource_name="issue_comments"+ )+ next_page: dict[str, Any] = {}++ def stage_next_page(state: Optional[dict[str, Any]], has_next: bool) -> None:+ next_page["cursor"] = state.get("cursor") if state and has_next else None++ for issues in client.paginate(+ config.path,+ params={"jql": jql, "maxResults": config.page_size, "fields": "comment"},+ paginator=JiraTokenPaginator(),+ data_selector=config.data_key,+ initial_paginator_state=initial_paginator_state,+ page_state_hook=stage_next_page,+ ):+ rows: list[dict[str, Any]] = []+ for issue in issues:+ embedded = (issue.get("fields") or {}).get("comment") or {}+ comments = embedded.get("comments") or []+ if (embedded.get("total") or 0) <= len(comments):+ rows.extend(_comment_rows(issue, comments))+ continue++ # The search embeds a capped list, so read the whole list for this issue.+ for page in client.paginate(+ f"/rest/api/3/issue/{issue['id']}/comment",+ params={"orderBy": "created"},+ paginator=JiraOffsetPaginator(+ limit=DEFAULT_PAGE_SIZE, offset_param="startAt", limit_param="maxResults", total_path="total"+ ),+ data_selector="comments",+ hooks=skip_deleted_issue,+ ):+ rows.extend(_comment_rows(issue, page))+ # Hand rows on after each request so a worker shutdown can stop between them.+ if rows:+ yield rows+ rows = []+ else:+ resumable_source_manager.safe_point()++ if next_page.get("cursor") is not None:+ resumable_source_manager.save_state(JiraResumeConfig(next_page_token=next_page["cursor"], jql=jql))+ if rows:+ yield rows+ else:+ resumable_source_manager.safe_point()++ return SourceResponse(+ name="issue_comments",+ items=items,+ primary_keys=config.primary_key,+ partition_count=1,+ partition_size=1,+ partition_mode="datetime",+ partition_format="month",+ partition_keys=[config.partition_key] if config.partition_key else None,+ )+++def _fanout_resource(endpoint: str, subdomain: str, email: str, api_token: str, team_id: int, job_id: str) -> Resource:+ config = JIRA_ENDPOINTS[endpoint]assert config.fanout is not Noneparent_config = JIRA_ENDPOINTS[config.fanout.parent_name]- resource = cast(+ return cast(Resource,build_dependent_resource(endpoint_configs=JIRA_ENDPOINTS,- child_endpoint="sprints",+ child_endpoint=endpoint,fanout=config.fanout,client_config=_client_config(subdomain, email, api_token),path_format_values={},@@ -369,6 +561,24 @@ def _sprints_source(subdomain: str, email: str, api_token: str, team_id: int, jo),)++def _fanout_source(+ endpoint: str, subdomain: str, email: str, api_token: str, team_id: int, job_id: str+) -> SourceResponse:+ resource = _fanout_resource(endpoint, subdomain, email, api_token, team_id, job_id)+ return SourceResponse(+ name=endpoint,+ items=lambda: resource,+ primary_keys=JIRA_ENDPOINTS[endpoint].primary_key,+ partition_count=1,+ partition_size=1,+ )+++def _sprints_source(subdomain: str, email: str, api_token: str, team_id: int, job_id: str) -> SourceResponse:+ config = JIRA_ENDPOINTS["sprints"]+ resource = _fanout_resource("sprints", subdomain, email, api_token, team_id, job_id)+seen: set[Any] = set()def first_listing(sprint: dict[str, Any]) -> bool:@@ -428,6 +638,24 @@ def jira_source(return _sprint_issues_source(subdomain, email, api_token, logger, resumable_source_manager)if endpoint == "sprints":return _sprints_source(subdomain, email, api_token, team_id, job_id)+ if endpoint in ("group_members", "versions"):+ return _fanout_source(endpoint, subdomain, email, api_token, team_id, job_id)+ if endpoint == "issue_changelogs":+ return _issue_changelogs_source(+ subdomain,+ email,+ api_token,+ resumable_source_manager,+ db_incremental_field_last_value if should_use_incremental_field else None,+ )+ if endpoint == "issue_comments":+ return _issue_comments_source(+ subdomain,+ email,+ api_token,+ resumable_source_manager,+ db_incremental_field_last_value if should_use_incremental_field else None,+ )resume = resumable_source_manager.load_state() if resumable_source_manager.can_resume() else Noneinitial_paginator_state: Optional[dict[str, Any]] = Nonediff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/settings.pyindex 2acef111cecb..8bbbe2c969c4 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/settings.py@@ -18,6 +18,12 @@ DEFAULT_PAGE_SIZE = 100AGILE_PAGE_SIZE = 50# Both ``/worklog/updated`` and ``/worklog/list`` cap a page at 1000 worklogs.WORKLOG_PAGE_SIZE = 1000+# ``/changelog/bulkfetch`` takes at most 1000 issue ids per request.+CHANGELOG_PAGE_SIZE = 1000+# ``/group/member`` caps a page at 50; ``/group/bulk`` defaults to 50 and documents no maximum.+GROUP_PAGE_SIZE = 50+# ``/project/{id}/version`` defaults to 50 and documents no maximum.+VERSION_PAGE_SIZE = 50@dataclass(frozen=True)@@ -50,10 +56,10 @@ def _datetime_field(name: str) -> IncrementalField:# Endpoint catalog. Jira Cloud REST API v3 (https://developer.atlassian.com/cloud/jira/platform/rest/v3/).-# We deliberately exclude the per-issue fan-out streams (comments, votes, watchers, remote links,-# properties, transitions) — each requires one HTTP request per issue and degrades badly on large-# instances. Worklogs avoid that through the instance-wide ``/worklog/updated`` feed. Issues carry the-# bulk of the useful data and sync incrementally.+# We deliberately exclude the per-issue fan-out streams (votes, watchers, remote links, properties,+# transitions) — each requires one HTTP request per issue and degrades badly on large instances.+# Worklogs, changelogs and comments avoid that by reading through instance-wide or bulk endpoints.+# Issues carry the bulk of the useful data and sync incrementally.JIRA_ENDPOINTS: dict[str, JiraEndpointConfig] = {"issues": JiraEndpointConfig(name="issues",@@ -177,6 +183,73 @@ JIRA_ENDPOINTS: dict[str, JiraEndpointConfig] = {pagination="custom",data_key="issues",),+ "issue_changelogs": JiraEndpointConfig(+ name="issue_changelogs",+ # Issue ids come from the enhanced JQL search, then their changelogs are read in bulk with+ # ``POST /changelog/bulkfetch``, which avoids one request per issue.+ path="/rest/api/3/changelog/bulkfetch",+ primary_key=["issue_id", "id"],+ pagination="custom",+ supports_incremental=True,+ partition_key="created",+ incremental_fields=[_datetime_field("created")],+ page_size=CHANGELOG_PAGE_SIZE,+ ),+ "issue_comments": JiraEndpointConfig(+ name="issue_comments",+ # Comments ride along on the enhanced JQL search (``fields=comment``). Only an issue whose+ # embedded list is truncated is read again through ``/issue/{id}/comment``.+ path="/rest/api/3/search/jql",+ primary_key=["id"],+ pagination="custom",+ data_key="issues",+ supports_incremental=True,+ partition_key="created",+ incremental_fields=[_datetime_field("updated"), _datetime_field("created")],+ ),+ "groups": JiraEndpointConfig(+ name="groups",+ path="/rest/api/3/group/bulk",+ primary_key=["groupId"],+ pagination="offset",+ data_key="values",+ page_size=GROUP_PAGE_SIZE,+ ),+ "group_members": JiraEndpointConfig(+ name="group_members",+ path="/rest/api/3/group/member?groupId={groupId}",+ primary_key=["group_id", "accountId"],+ pagination="offset",+ data_key="values",+ page_size=GROUP_PAGE_SIZE,+ fanout=DependentEndpointConfig(+ parent_name="groups",+ resolve_param="groupId",+ resolve_field="groupId",+ include_from_parent=["groupId", "name"],+ parent_field_renames={"groupId": "group_id", "name": "group_name"},+ child_params={"includeInactiveUsers": "true"},+ # A group deleted after the listing answers 404.+ child_response_actions=[{"status_code": 404, "action": "ignore"}],+ ),+ ),+ "versions": JiraEndpointConfig(+ name="versions",+ # The paginated listing; ``/project/{id}/versions`` returns every version in one response.+ path="/rest/api/3/project/{projectId}/version",+ primary_key=["id"],+ pagination="offset",+ data_key="values",+ page_size=VERSION_PAGE_SIZE,+ fanout=DependentEndpointConfig(+ parent_name="projects",+ resolve_param="projectId",+ resolve_field="id",+ include_from_parent=[],+ # A project deleted after the listing answers 404.+ child_response_actions=[{"status_code": 404, "action": "ignore"}],+ ),+ ),}ENDPOINTS = tuple(JIRA_ENDPOINTS.keys())diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/tests/test_jira.py b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/tests/test_jira.pyindex f7c40b5d0904..351130732d87 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/jira/tests/test_jira.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/jira/tests/test_jira.py@@ -334,3 +334,173 @@ class TestSprintIssues:assert result.raised is Noneassert result.rows == []assert result.paths == ["/rest/api/3/field"]+++class TestIssueChangelogs:+ def test_bulk_fetches_each_issue_page_and_checkpoints_after_its_last_changelog_page(self) -> None:+ result = _driver().run(+ "issue_changelogs",+ route(+ {+ "/rest/api/3/search/jql": [+ ScriptedResponse(json={"issues": [{"id": "10"}, {"id": "11"}], "nextPageToken": "tok2"}),+ ScriptedResponse(json={"issues": [{"id": "12"}], "isLast": True}),+ ],+ "/rest/api/3/changelog/bulkfetch": [+ ScriptedResponse(+ json={+ "issueChangeLogs": [+ {"issueId": "10", "changeHistories": [{"id": "100", "created": "c1"}]},+ {"issueId": "11", "changeHistories": [{"id": "101", "created": "c2"}]},+ ],+ "nextPageToken": "cl2",+ }+ ),+ ScriptedResponse(+ json={"issueChangeLogs": [{"issueId": "10", "changeHistories": [{"id": "102"}]}]}+ ),+ ScriptedResponse(json={"issueChangeLogs": []}),+ ],+ }+ ),+ )++ assert result.raised is None+ assert result.params("fields") == ["id", None, None, "id", None]+ assert [request.json() for request in result.requests if request.method == "POST"] == [+ {"issueIdsOrKeys": ["10", "11"], "maxResults": 1000},+ {"issueIdsOrKeys": ["10", "11"], "maxResults": 1000, "nextPageToken": "cl2"},+ {"issueIdsOrKeys": ["12"], "maxResults": 1000},+ ]+ assert [(row["issue_id"], row["id"]) for row in result.rows] == [("10", "100"), ("11", "101"), ("10", "102")]+ full_sync_jql = 'updated >= "1970-01-01 00:00" ORDER BY updated ASC'+ assert result.saved_states == [JiraResumeConfig(next_page_token="tok2", jql=full_sync_jql)]++ @parameterized.expand(+ [+ ("full_refresh", {}, 'updated >= "1970-01-01 00:00" ORDER BY updated ASC', ["old", "recent"]),+ # The JQL and the changelog filter both use the watermark less the one day lookback.+ (+ "incremental",+ {+ "incremental_field": "created",+ "db_incremental_field_last_value": datetime(2026, 3, 4, 2, 58, tzinfo=UTC),+ },+ 'updated >= "2026-03-03 02:58" ORDER BY updated ASC',+ ["recent"],+ ),+ ]+ )+ def test_incremental_shapes_jql_and_drops_history_before_the_window(+ self, _name: str, run_kwargs: dict[str, Any], expected_jql: str, expected_ids: list[str]+ ) -> None:+ result = _driver().run(+ "issue_changelogs",+ route(+ {+ "/rest/api/3/search/jql": [ScriptedResponse(json={"issues": [{"id": "10"}], "isLast": True})],+ "/rest/api/3/changelog/bulkfetch": [+ ScriptedResponse(+ json={+ "issueChangeLogs": [+ {+ "issueId": "10",+ "changeHistories": [+ {"id": "old", "created": "2026-03-03T02:57:00.000+0000"},+ {"id": "recent", "created": "2026-03-03T02:58:00.000+0000"},+ ],+ }+ ]+ }+ )+ ],+ }+ ),+ **run_kwargs,+ )++ assert result.raised is None+ assert result.params("jql")[0] == expected_jql+ assert [row["id"] for row in result.rows] == expected_ids+++class TestIssueComments:+ def test_reads_embedded_comments_and_refetches_truncated_lists_of_existing_issues(self) -> None:+ result = _driver().run(+ "issue_comments",+ route(+ {+ "/rest/api/3/search/jql": [+ ScriptedResponse(+ json={+ "issues": [+ {+ "id": "1",+ "key": "P-1",+ "fields": {"comment": {"comments": [{"id": "a"}], "total": 1}},+ },+ {+ "id": "2",+ "key": "P-2",+ "fields": {"comment": {"comments": [{"id": "b"}], "total": 2}},+ },+ {"id": "3", "key": "P-3", "fields": {"comment": {"comments": [], "total": 0}}},+ {+ "id": "4",+ "key": "P-4",+ "fields": {"comment": {"comments": [{"id": "d"}], "total": 2}},+ },+ ],+ "nextPageToken": "tok2",+ }+ ),+ ScriptedResponse(json={"issues": [], "isLast": True}),+ ],+ "/rest/api/3/issue/2/comment": [+ ScriptedResponse(json={"comments": [{"id": "b"}, {"id": "c"}], "startAt": 0, "total": 2})+ ],+ # Deleted after the search returned it.+ "/rest/api/3/issue/4/comment": [+ ScriptedResponse(status=404, json={"errorMessages": ["Issue does not exist"]})+ ],+ }+ ),+ )++ assert result.raised is None+ assert result.paths == [+ "/rest/api/3/search/jql",+ "/rest/api/3/issue/2/comment",+ "/rest/api/3/issue/4/comment",+ "/rest/api/3/search/jql",+ ]+ assert result.params("fields") == ["comment", None, None, "comment"]+ assert [(row["issue_key"], row["id"]) for row in result.rows] == [("P-1", "a"), ("P-2", "b"), ("P-2", "c")]+++class TestGroupMembers:+ def test_fans_out_over_groups_with_group_columns_and_skips_deleted_groups(self) -> None:+ result = _driver().run(+ "group_members",+ route(+ {+ "/rest/api/3/group/bulk": [+ ScriptedResponse(+ json={+ "values": [{"groupId": "g1", "name": "devs"}, {"groupId": "g2", "name": "gone"}],+ "isLast": True,+ }+ )+ ],+ "/rest/api/3/group/member": [+ ScriptedResponse(json={"values": [{"accountId": "u1"}], "isLast": True}),+ ScriptedResponse(status=404, json={"errorMessages": ["Group not found"]}),+ ],+ }+ ),+ )++ assert result.raised is None+ assert result.params("groupId") == [None, "g1", "g2"]+ assert result.params("includeInactiveUsers") == [None, "true", "true"]+ assert result.rows == [{"accountId": "u1", "group_id": "g1", "group_name": "devs"}]