:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(warehouse): add aws glue data quality results v2 table (#119182)
Tom Owers78fa79ab074b
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 93ef7cc44750..00249ea086da 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -1182,11 +1182,12 @@ Note: Compared the official botocore operation model with the static table map a## AWS Glue Data Catalog — **thin**-Today (10): `crawlers`, `crawls`, `data_quality_results`, `databases`, `job_runs`, `jobs`, `partitions`, `table_versions`, `tables`, `workflow_runs`+Today (11): `crawlers`, `crawls`, `data_quality_results`, `data_quality_results_v2`, `databases`, `job_runs`, `jobs`, `partitions`, `table_versions`, `tables`, `workflow_runs`Diffed against: <https://raw.githubusercontent.com/boto/botocore/develop/botocore/data/glue/2017-03-31/service-2.json>- [x] `Data quality results (ListDataQualityResults, POST /)` — links data quality scores and failed rules to synced tables and job runs (high)+- [x] `Data quality results V2 (ListDataQualityResultsV2, POST /)` — lists data quality results published from any supported engine, not only Glue jobs (medium)- [x] `Table versions (GetTableVersions, POST /)` — preserves schema and metadata history for existing catalog tables (medium)- [x] `Crawler history (ListCrawls, POST /)` — tracks crawl duration, failures, catalog changes, and compute use beyond the latest crawler result (medium)- [x] `Workflow runs (GetWorkflowRuns, POST /)` — measures complete pipeline duration and failures across related jobs and crawlers (medium)diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.pyindex ff44fd9afd63..4acbed02e160 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.py@@ -149,4 +149,17 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"region": "AWS region selected for this source.",},},+ "data_quality_results_v2": {+ "description": "Data quality results published from any supported data processing engine, not only Glue jobs.",+ "docs_url": "https://docs.aws.amazon.com/glue/latest/webapi/API_ListDataQualityResultsV2.html",+ "columns": {+ "result_id": "Identifier of the data quality result.",+ "data_source": "Table associated with the data quality result, if any.",+ "result_published_on": "Time when the data quality result was published.",+ "dimensions": "Platform identifiers, such as cluster, application, workgroup, or project IDs, for where the evaluation ran.",+ "evaluation_context": "Evaluation context that identifies a result, or a series of related results, within a run.",+ "descriptors": "User-provided descriptors that add context to the result, such as database name, table name, team, or environment.",+ "region": "AWS region selected for this source.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.pyindex 262f783324e5..d2b689ec1d5f 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.py@@ -83,6 +83,12 @@ ENDPOINTS: dict[str, GlueEndpoint] = {primary_keys=("region", "result_id"),timestamp_columns=("started_on",),),+ "data_quality_results_v2": GlueEndpoint(+ operation="ListDataQualityResultsV2",+ result_key="Results",+ primary_keys=("region", "result_id"),+ timestamp_columns=("result_published_on",),+ ),}# ListWorkflows returns only names, so it feeds workflow_runs without being a table of its own.@@ -107,6 +113,7 @@ ENDPOINT_DESCRIPTIONS = {"table_versions": "Schema and metadata history for each catalog table.","workflow_runs": "Workflow run history, status, timing, and action statistics.","data_quality_results": "Data quality evaluation runs, with the evaluated table and the job run.",+ "data_quality_results_v2": "Data quality results published from any supported engine, with the evaluated table and run context.",}ERROR_MESSAGES = {diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.pyindex a39b6feb2767..a39a0535d4a6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.py@@ -123,7 +123,7 @@ class AwsGlueDataCatalogSource(ResumableSource[AwsGlueDataCatalogSourceConfig, A"for catalog tables. ""Grant `glue:GetJobs`, `glue:GetJobRuns`, `glue:GetCrawlers`, and `glue:ListCrawls` for jobs and crawlers. ""Grant `glue:ListWorkflows` and `glue:GetWorkflowRuns` for workflow runs, "- "and `glue:ListDataQualityResults` for data quality results. "+ "and `glue:ListDataQualityResults` and `glue:ListDataQualityResultsV2` for data quality results. ""Lake Formation permissions can also restrict catalog access. ""Enter the access key ID and secret access key. Add a session token for temporary credentials."),diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.pyindex 5a61030c12ab..88434c5cefa6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.py@@ -184,6 +184,7 @@ def test_body_retry_does_not_repeat_transport_retries(status: int, attempts: int("jobs", "Jobs"),("crawlers", "Crawlers"),("data_quality_results", "Results"),+ ("data_quality_results_v2", "Results"),],)def test_top_level_resume_checkpoints_before_yield_and_clears_after_completion(endpoint: str, result_key: str) -> None: