Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(warehouse): add aws glue data quality results v2 table (#119182)

Tom Owers78fa79ab074b
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
index 93ef7cc44750..00249ea086da 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md
@@ -1182,11 +1182,12 @@ Note: Compared the official botocore operation model with the static table map a
## AWS Glue Data Catalog — **thin**
-Today (10): `crawlers`, `crawls`, `data_quality_results`, `databases`, `job_runs`, `jobs`, `partitions`, `table_versions`, `tables`, `workflow_runs`
+Today (11): `crawlers`, `crawls`, `data_quality_results`, `data_quality_results_v2`, `databases`, `job_runs`, `jobs`, `partitions`, `table_versions`, `tables`, `workflow_runs`
Diffed against: <https://raw.githubusercontent.com/boto/botocore/develop/botocore/data/glue/2017-03-31/service-2.json>
- [x] `Data quality results (ListDataQualityResults, POST /)` — links data quality scores and failed rules to synced tables and job runs (high)
+- [x] `Data quality results V2 (ListDataQualityResultsV2, POST /)` — lists data quality results published from any supported engine, not only Glue jobs (medium)
- [x] `Table versions (GetTableVersions, POST /)` — preserves schema and metadata history for existing catalog tables (medium)
- [x] `Crawler history (ListCrawls, POST /)` — tracks crawl duration, failures, catalog changes, and compute use beyond the latest crawler result (medium)
- [x] `Workflow runs (GetWorkflowRuns, POST /)` — measures complete pipeline duration and failures across related jobs and crawlers (medium)
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.py
index ff44fd9afd63..4acbed02e160 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.py
@@ -149,4 +149,17 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {
"region": "AWS region selected for this source.",
},
},
+ "data_quality_results_v2": {
+ "description": "Data quality results published from any supported data processing engine, not only Glue jobs.",
+ "docs_url": "https://docs.aws.amazon.com/glue/latest/webapi/API_ListDataQualityResultsV2.html",
+ "columns": {
+ "result_id": "Identifier of the data quality result.",
+ "data_source": "Table associated with the data quality result, if any.",
+ "result_published_on": "Time when the data quality result was published.",
+ "dimensions": "Platform identifiers, such as cluster, application, workgroup, or project IDs, for where the evaluation ran.",
+ "evaluation_context": "Evaluation context that identifies a result, or a series of related results, within a run.",
+ "descriptors": "User-provided descriptors that add context to the result, such as database name, table name, team, or environment.",
+ "region": "AWS region selected for this source.",
+ },
+ },
}
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.py
index 262f783324e5..d2b689ec1d5f 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.py
@@ -83,6 +83,12 @@ ENDPOINTS: dict[str, GlueEndpoint] = {
primary_keys=("region", "result_id"),
timestamp_columns=("started_on",),
),
+ "data_quality_results_v2": GlueEndpoint(
+ operation="ListDataQualityResultsV2",
+ result_key="Results",
+ primary_keys=("region", "result_id"),
+ timestamp_columns=("result_published_on",),
+ ),
}
# ListWorkflows returns only names, so it feeds workflow_runs without being a table of its own.
@@ -107,6 +113,7 @@ ENDPOINT_DESCRIPTIONS = {
"table_versions": "Schema and metadata history for each catalog table.",
"workflow_runs": "Workflow run history, status, timing, and action statistics.",
"data_quality_results": "Data quality evaluation runs, with the evaluated table and the job run.",
+ "data_quality_results_v2": "Data quality results published from any supported engine, with the evaluated table and run context.",
}
ERROR_MESSAGES = {
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.py
index a39b6feb2767..a39a0535d4a6 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.py
@@ -123,7 +123,7 @@ class AwsGlueDataCatalogSource(ResumableSource[AwsGlueDataCatalogSourceConfig, A
"for catalog tables. "
"Grant `glue:GetJobs`, `glue:GetJobRuns`, `glue:GetCrawlers`, and `glue:ListCrawls` for jobs and crawlers. "
"Grant `glue:ListWorkflows` and `glue:GetWorkflowRuns` for workflow runs, "
- "and `glue:ListDataQualityResults` for data quality results. "
+ "and `glue:ListDataQualityResults` and `glue:ListDataQualityResultsV2` for data quality results. "
"Lake Formation permissions can also restrict catalog access. "
"Enter the access key ID and secret access key. Add a session token for temporary credentials."
),
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.py
index 5a61030c12ab..88434c5cefa6 100644
--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.py
+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.py
@@ -184,6 +184,7 @@ def test_body_retry_does_not_repeat_transport_retries(status: int, attempts: int
("jobs", "Jobs"),
("crawlers", "Crawlers"),
("data_quality_results", "Results"),
+ ("data_quality_results_v2", "Results"),
],
)
def test_top_level_resume_checkpoints_before_yield_and_clears_after_completion(endpoint: str, result_key: str) -> None: