:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(data-warehouse): add 3 missing tables to the AWS Glue Data Catalog source (#120762)
Tom Owersdd235a5ab695
diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.mdindex 563fc3d3ffa0..becaa6e1bd13 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/COVERAGE_GAPS_APPENDIX.md@@ -1182,7 +1182,7 @@ Note: Compared the official botocore operation model with the static table map a## AWS Glue Data Catalog — **thin**-Today (11): `crawlers`, `crawls`, `data_quality_results`, `data_quality_results_v2`, `databases`, `job_runs`, `jobs`, `partitions`, `table_versions`, `tables`, `workflow_runs`+Today (14): `column_statistics`, `crawlers`, `crawls`, `data_quality_results`, `data_quality_results_v2`, `databases`, `job_runs`, `jobs`, `partitions`, `sessions`, `table_optimizer_runs`, `table_versions`, `tables`, `workflow_runs`Diffed against: <https://raw.githubusercontent.com/boto/botocore/develop/botocore/data/glue/2017-03-31/service-2.json>@@ -1191,9 +1191,9 @@ Diffed against: <https://raw.githubusercontent.com/boto/botocore/develop/botocor- [x] `Table versions (GetTableVersions, POST /)` — preserves schema and metadata history for existing catalog tables (medium)- [x] `Crawler history (ListCrawls, POST /)` — tracks crawl duration, failures, catalog changes, and compute use beyond the latest crawler result (medium)- [x] `Workflow runs (GetWorkflowRuns, POST /)` — measures complete pipeline duration and failures across related jobs and crawlers (medium)-- [ ] `Table column statistics (GetColumnStatisticsForTable, POST /)` — provides column distributions, null counts, and cardinality for data profiling (medium)-- [ ] `Table optimizer runs (ListTableOptimizerRuns, POST /)` — tracks compaction and cleanup execution history, failures, and resource use for catalog tables (medium)-- [ ] `Interactive sessions (ListSessions, POST /)` — measures interactive ETL session duration, status, and compute use alongside scheduled job runs (medium)+- [x] `Table column statistics (GetColumnStatisticsForTable, POST /)` — provides column distributions, null counts, and cardinality for data profiling (medium)+- [x] `Table optimizer runs (ListTableOptimizerRuns, POST /)` — tracks compaction and cleanup execution history, failures, and resource use for catalog tables (medium)+- [x] `Interactive sessions (ListSessions, POST /)` — measures interactive ETL session duration, status, and compute use alongside scheduled job runs (medium)Note: Compared the official botocore operation model with the static table map and extraction code; reads use POST /. Existing traversal already covers database tables, table partitions, and job runs. Data quality results require GetDataQualityResult hydration; workflow runs require ListWorkflows discovery and calls for each workflow. Table history, statistics, and optimizer runs require table traversal; connection settings, security configuration, and write operations were excluded.diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/aws_glue_data_catalog.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/aws_glue_data_catalog.pyindex e23c011de9a8..1236da1a83f5 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/aws_glue_data_catalog.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/aws_glue_data_catalog.py@@ -16,11 +16,13 @@ from posthog.dataclasses import frozenfrom products.warehouse_sources.backend.temporal.data_imports.sources.aws_glue_data_catalog.settings import (BODY_RETRY_CODES,+ COLUMN_STATISTICS_BATCH_SIZE,CONTENT_TYPE,ENDPOINTS,ERROR_MESSAGES,PARENT_ENDPOINTS,REQUEST_TIMEOUT_SECONDS,+ TABLE_OPTIMIZER_TYPES,TARGET_PREFIXES,GlueEndpoint,)@@ -146,6 +148,15 @@ class AwsGlueClient:raise ValueError("AWS Glue returned an invalid response.")return result+ def request_endpoint(self, endpoint: GlueEndpoint, params: dict[str, Any]) -> dict[str, Any] | None:+ try:+ return self.request(endpoint.operation, params)+ except AwsGlueError as error:+ # ListTableOptimizerRuns fails when the table has no optimizer of the requested type.+ if endpoint.missing_is_empty and error.code == "EntityNotFoundException":+ return None+ raise+def pages(self,endpoint: GlueEndpoint,@@ -154,10 +165,12 @@ class AwsGlueClient:) -> Iterator[GluePage]:token = start_tokenwhile True:- params = {"MaxResults": endpoint.page_size, **payload}+ params = {"MaxResults": endpoint.page_size, **payload} if endpoint.paginated else {**payload}if token:params["NextToken"] = token- body = self.request(endpoint.operation, params)+ body = self.request_endpoint(endpoint, params)+ if body is None:+ returnitems = page_items(endpoint, body)next_token = body.get("NextToken") or Noneif next_token is not None and not isinstance(next_token, str):@@ -181,7 +194,7 @@ def normalize_row(endpoint: str, item: dict[str, Any], region: str, parent_paylovalue = row.get(column)if isinstance(value, int | float) and not isinstance(value, bool):row[column] = datetime.fromtimestamp(value, tz=UTC)- for key in ("DatabaseName", "TableName", "JobName", "CrawlerName", "CatalogId"):+ for key in ("DatabaseName", "TableName", "JobName", "CrawlerName", "CatalogId", "Type"):if key in parent_payload:row[_CAMEL_BOUNDARY.sub("_", key).lower()] = parent_payload[key]if endpoint == "partitions":@@ -200,11 +213,29 @@ def parent_payload(endpoint: str, parent: dict[str, Any]) -> dict[str, Any]:payload = {"DatabaseName": parent["name"] if endpoint == "tables" else parent["database_name"]}if parent.get("catalog_id"):payload["CatalogId"] = parent["catalog_id"]- if endpoint in ("partitions", "table_versions"):+ if endpoint in ("partitions", "table_versions", "column_statistics", "table_optimizer_runs"):payload["TableName"] = parent["name"]return payload+def parent_payloads(endpoint: str, parent: dict[str, Any]) -> list[dict[str, Any]]:+ payload = parent_payload(endpoint, parent)+ if endpoint == "column_statistics":+ columns = (parent.get("storage_descriptor") or {}).get("Columns") or []+ names = [column["Name"] for column in columns if isinstance(column, dict) and column.get("Name")]+ return [+ {**payload, "ColumnNames": names[start : start + COLUMN_STATISTICS_BATCH_SIZE]}+ for start in range(0, len(names), COLUMN_STATISTICS_BATCH_SIZE)+ ]+ if endpoint == "table_optimizer_runs":+ # Table optimizers exist only for Iceberg tables, and the request requires a catalog ID.+ table_type = str((parent.get("parameters") or {}).get("table_type", ""))+ if table_type.upper() != "ICEBERG" or "CatalogId" not in payload:+ return []+ return [{**payload, "Type": optimizer_type} for optimizer_type in TABLE_OPTIMIZER_TYPES]+ return [payload]++def walk_rows(client: AwsGlueClient,endpoint: str,@@ -215,11 +246,11 @@ def walk_rows(if spec.parent:for parents in walk_rows(client, spec.parent, manager):for parent in parents:- payload = parent_payload(endpoint, parent)- for page in client.pages(spec, payload):- if page.items:- yield [normalize_row(endpoint, item, client.region, payload) for item in page.items]- manager.safe_point()+ for payload in parent_payloads(endpoint, parent):+ for page in client.pages(spec, payload):+ if page.items:+ yield [normalize_row(endpoint, item, client.region, payload) for item in page.items]+ manager.safe_point()returnstate = manager.load_state() if checkpoint and manager.can_resume() else None@@ -272,8 +303,13 @@ def probe_endpoint(client: AwsGlueClient, endpoint: str) -> dict[str, Any] | Nonparent = probe_endpoint(client, spec.parent)if parent is None:return None- payload = parent_payload(endpoint, parent)- body = client.request(spec.operation, {**payload, "MaxResults": 1})+ payloads = parent_payloads(endpoint, parent)+ if not payloads:+ return None+ payload = payloads[0]+ body = client.request_endpoint(spec, {**payload, "MaxResults": 1} if spec.paginated else payload)+ if body is None:+ return Noneitems = page_items(spec, body)return normalize_row(endpoint, items[0], client.region, payload) if items else Nonediff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.pyindex 4acbed02e160..1453c3644526 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/canonical_descriptions.py@@ -162,4 +162,57 @@ CANONICAL_DESCRIPTIONS: CanonicalDescriptions = {"region": "AWS region selected for this source.",},},+ "column_statistics": {+ "description": "Column statistics for catalog tables, such as null counts, distinct values, and value ranges.",+ "docs_url": "https://docs.aws.amazon.com/glue/latest/webapi/API_ColumnStatistics.html",+ "columns": {+ "database_name": "Name of the database that contains the table.",+ "table_name": "Name of the table that contains the column.",+ "catalog_id": "Identifier of the Data Catalog that contains the table.",+ "column_name": "Name of the column that the statistics belong to.",+ "column_type": "Data type of the column.",+ "analyzed_time": "Time when the column statistics were generated.",+ "statistics_data": "Statistics for the column, with a type and the values for that type, such as null and distinct value counts.",+ "region": "AWS region selected for this source.",+ },+ },+ "table_optimizer_runs": {+ "description": "History of table optimizer runs for Iceberg tables, one row for each run.",+ "docs_url": "https://docs.aws.amazon.com/glue/latest/webapi/API_TableOptimizerRun.html",+ "columns": {+ "database_name": "Name of the database that contains the table.",+ "table_name": "Name of the table that the optimizer ran on.",+ "catalog_id": "Identifier of the Data Catalog that contains the table.",+ "type": "Type of table optimizer: compaction, retention, or orphan_file_deletion.",+ "event_type": "Status of the run, such as starting, in_progress, completed, or failed.",+ "start_timestamp": "Time when the run started.",+ "end_timestamp": "Time when the run ended.",+ "error": "Error that occurred during the run, if any.",+ "compaction_strategy": "Strategy that the compaction run used, such as binpack, sort, or z-order.",+ "compaction_metrics": "Metrics for a compaction run.",+ "retention_metrics": "Metrics for a snapshot retention run.",+ "orphan_file_deletion_metrics": "Metrics for an orphan file deletion run.",+ "region": "AWS region selected for this source.",+ },+ },+ "sessions": {+ "description": "Interactive sessions that run ETL code on demand.",+ "docs_url": "https://docs.aws.amazon.com/glue/latest/webapi/API_Session.html",+ "columns": {+ "id": "Identifier of the session.",+ "created_on": "Time when the session was created.",+ "completed_on": "Time when the session ended.",+ "status": "Status of the session, such as READY, FAILED, TIMEOUT, or STOPPED.",+ "error_message": "Error message shown when the session fails.",+ "execution_time": "Total time the session ran.",+ "dpu_seconds": "DPUs that the session consumed, in DPU-seconds.",+ "max_capacity": "Number of DPUs that can be allocated when the session runs.",+ "number_of_workers": "Number of workers of the defined worker type that the session uses.",+ "worker_type": "Type of predefined worker that the session uses.",+ "glue_version": "AWS Glue version that the session uses.",+ "idle_timeout": "Number of minutes the session can stay idle before it times out.",+ "session_type": "Type of the session, such as LIVY or SPARK_CONNECT.",+ "region": "AWS region selected for this source.",+ },+ },}diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.pyindex d2b689ec1d5f..48bd25c1d9cc 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/settings.py@@ -15,6 +15,8 @@ class GlueEndpoint:page_size: int = 100timestamp_columns: tuple[str, ...] = ()names_only: bool = False+ paginated: bool = True+ missing_is_empty: bool = FalseENDPOINTS: dict[str, GlueEndpoint] = {@@ -89,8 +91,34 @@ ENDPOINTS: dict[str, GlueEndpoint] = {primary_keys=("region", "result_id"),timestamp_columns=("result_published_on",),),+ "column_statistics": GlueEndpoint(+ operation="GetColumnStatisticsForTable",+ result_key="ColumnStatisticsList",+ primary_keys=("region", "database_name", "table_name", "column_name"),+ parent="tables",+ timestamp_columns=("analyzed_time",),+ paginated=False,+ ),+ "table_optimizer_runs": GlueEndpoint(+ operation="ListTableOptimizerRuns",+ result_key="TableOptimizerRuns",+ primary_keys=("region", "database_name", "table_name", "type", "start_timestamp"),+ parent="tables",+ timestamp_columns=("start_timestamp", "end_timestamp"),+ missing_is_empty=True,+ ),+ "sessions": GlueEndpoint(+ operation="ListSessions",+ result_key="Sessions",+ primary_keys=("region", "id"),+ timestamp_columns=("created_on", "completed_on"),+ ),}+# GetColumnStatisticsForTable accepts at most 100 column names per request.+COLUMN_STATISTICS_BATCH_SIZE = 100+TABLE_OPTIMIZER_TYPES = ("compaction", "retention", "orphan_file_deletion")+# ListWorkflows returns only names, so it feeds workflow_runs without being a table of its own.PARENT_ENDPOINTS: dict[str, GlueEndpoint] = {"workflows": GlueEndpoint(@@ -114,6 +142,9 @@ ENDPOINT_DESCRIPTIONS = {"workflow_runs": "Workflow run history, status, timing, and action statistics.","data_quality_results": "Data quality evaluation runs, with the evaluated table and the job run.","data_quality_results_v2": "Data quality results published from any supported engine, with the evaluated table and run context.",+ "column_statistics": "Column statistics for each catalog table, such as null counts, distinct values, and value ranges.",+ "table_optimizer_runs": "Compaction, snapshot retention, and orphan file deletion runs for each Iceberg table.",+ "sessions": "Interactive sessions, with status, duration, and compute use.",}ERROR_MESSAGES = {diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.pyindex fd9ff5334b3f..b3de4f3b5ef8 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/source.py@@ -124,6 +124,8 @@ class AwsGlueDataCatalogSource(ResumableSource[AwsGlueDataCatalogSourceConfig, A"Grant `glue:GetJobs`, `glue:GetJobRuns`, `glue:GetCrawlers`, and `glue:ListCrawls` for jobs and crawlers. ""Grant `glue:ListWorkflows` and `glue:GetWorkflowRuns` for workflow runs, ""and `glue:ListDataQualityResults` and `glue:ListDataQualityResultsV2` for data quality results. "+ "Grant `glue:GetTable` for column statistics, `glue:ListTableOptimizerRuns` for table optimizer runs, "+ "and `glue:ListSessions` for interactive sessions. ""Lake Formation permissions can also restrict catalog access. ""Enter the access key ID and secret access key. Add a session token for temporary credentials."),diff --git a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.py b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.pyindex 88434c5cefa6..e31a87774fd6 100644--- a/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.py+++ b/products/warehouse_sources/backend/temporal/data_imports/sources/aws_glue_data_catalog/tests/test_aws_glue_data_catalog.py@@ -185,6 +185,7 @@ def test_body_retry_does_not_repeat_transport_retries(status: int, attempts: int("crawlers", "Crawlers"),("data_quality_results", "Results"),("data_quality_results_v2", "Results"),+ ("sessions", "Sessions"),],)def test_top_level_resume_checkpoints_before_yield_and_clears_after_completion(endpoint: str, result_key: str) -> None:@@ -372,6 +373,185 @@ def test_history_fan_out_pages_each_parent_and_keys_rows_by_parent(assert result.rows[0]["dpu_hour"] == 0.5+DATABASE_PAGE = ScriptedResponse(json={"DatabaseList": [{"Name": "analytics", "CatalogId": "123456789012"}]})+NOT_FOUND = ScriptedResponse(status=400, json={"__type": "EntityNotFoundException", "Message": "Not found"})+++def iceberg_table(name: str) -> dict[str, Any]:+ return {"Name": name, "CatalogId": "123456789012", "Parameters": {"table_type": "iceberg"}}+++def table_with_columns(name: str, count: int) -> dict[str, Any]:+ return {"Name": name, "StorageDescriptor": {"Columns": [{"Name": f"col_{i}", "Type": "int"} for i in range(count)]}}+++ "endpoint,script,expected_requests,expected_keys",+ [+ (+ "column_statistics",+ [+ DATABASE_PAGE,+ ScriptedResponse(+ json={+ "TableList": [+ table_with_columns("wide", 101),+ table_with_columns("view", 0),+ table_with_columns("events", 1),+ ]+ }+ ),+ ScriptedResponse(+ json={+ "ColumnStatisticsList": [+ {+ "ColumnName": "col_0",+ "ColumnType": "int",+ "AnalyzedTime": 1735689600,+ "StatisticsData": {},+ }+ ]+ }+ ),+ ScriptedResponse(json={"ColumnStatisticsList": [], "Errors": [{"ColumnName": "col_100"}]}),+ ScriptedResponse(json={"ColumnStatisticsList": [{"ColumnName": "col_0", "ColumnType": "int"}]}),+ ],+ [+ {"MaxResults": 100},+ {"MaxResults": 100, "DatabaseName": "analytics", "CatalogId": "123456789012"},+ {+ "DatabaseName": "analytics",+ "CatalogId": "123456789012",+ "TableName": "wide",+ "ColumnNames": [f"col_{i}" for i in range(100)],+ },+ {+ "DatabaseName": "analytics",+ "CatalogId": "123456789012",+ "TableName": "wide",+ "ColumnNames": ["col_100"],+ },+ {+ "DatabaseName": "analytics",+ "CatalogId": "123456789012",+ "TableName": "events",+ "ColumnNames": ["col_0"],+ },+ ],+ [("eu-west-1", "analytics", "wide", "col_0"), ("eu-west-1", "analytics", "events", "col_0")],+ ),+ (+ "table_optimizer_runs",+ [+ DATABASE_PAGE,+ ScriptedResponse(+ json={+ "TableList": [+ iceberg_table("events"),+ {"Name": "hive", "CatalogId": "123456789012", "Parameters": {"classification": "parquet"}},+ ]+ }+ ),+ ScriptedResponse(+ json={+ "TableOptimizerRuns": [{"eventType": "completed", "startTimestamp": 1735689600}],+ "NextToken": "older",+ }+ ),+ ScriptedResponse(json={"TableOptimizerRuns": [{"eventType": "failed", "startTimestamp": 0}]}),+ NOT_FOUND,+ ScriptedResponse(+ json={"TableOptimizerRuns": [{"eventType": "completed", "startTimestamp": 1735689600}]}+ ),+ ],+ [+ {"MaxResults": 100},+ {"MaxResults": 100, "DatabaseName": "analytics", "CatalogId": "123456789012"},+ {+ "MaxResults": 100,+ "DatabaseName": "analytics",+ "CatalogId": "123456789012",+ "TableName": "events",+ "Type": "compaction",+ },+ {+ "MaxResults": 100,+ "DatabaseName": "analytics",+ "CatalogId": "123456789012",+ "TableName": "events",+ "Type": "compaction",+ "NextToken": "older",+ },+ {+ "MaxResults": 100,+ "DatabaseName": "analytics",+ "CatalogId": "123456789012",+ "TableName": "events",+ "Type": "retention",+ },+ {+ "MaxResults": 100,+ "DatabaseName": "analytics",+ "CatalogId": "123456789012",+ "TableName": "events",+ "Type": "orphan_file_deletion",+ },+ ],+ [+ ("eu-west-1", "analytics", "events", "compaction", datetime(2025, 1, 1, tzinfo=UTC)),+ ("eu-west-1", "analytics", "events", "compaction", datetime(1970, 1, 1, tzinfo=UTC)),+ ("eu-west-1", "analytics", "events", "orphan_file_deletion", datetime(2025, 1, 1, tzinfo=UTC)),+ ],+ ),+ ],+)+def test_table_fan_out_shapes_requests_for_each_table(+ endpoint: str,+ script: list[ScriptedResponse],+ expected_requests: list[dict[str, Any]],+ expected_keys: list[tuple[Any, ...]],+) -> None:+ result = driver().run(endpoint, script)+ assert result.raised is None+ assert [request.json() for request in result.requests] == expected_requests+ primary_keys = glue.ENDPOINTS[endpoint].primary_keys+ assert [tuple(row[key] for key in primary_keys) for row in result.rows] == expected_keys+ assert all(row["catalog_id"] == "123456789012" for row in result.rows)+ if endpoint == "column_statistics":+ assert result.rows[0]["analyzed_time"] == datetime(2025, 1, 1, tzinfo=UTC)+ assert "column_names" not in result.rows[0]+++ "schema_name,tables,child,expected_requests",+ [+ ("table_optimizer_runs", [iceberg_table("events")], NOT_FOUND, 3),+ ("table_optimizer_runs", [{"Name": "hive", "CatalogId": "123456789012"}], None, 2),+ (+ "column_statistics",+ [table_with_columns("events", 2)],+ ScriptedResponse(json={"ColumnStatisticsList": []}),+ 3,+ ),+ ("column_statistics", [table_with_columns("view", 0)], None, 2),+ ],+)+def test_table_fan_out_probe_accepts_tables_without_child_data(+ schema_name: str, tables: list[dict[str, Any]], child: ScriptedResponse | None, expected_requests: int+) -> None:+ script = [DATABASE_PAGE, ScriptedResponse(json={"TableList": tables}), *([child] if child else [])]+ with scripted_network(script) as network:+ assert glue.validate_credentials(config(), GLUE_API_VERSION, schema_name) == (True, None)+ assert len(network.requests_log) == expected_requests+ if schema_name == "column_statistics" and child:+ assert network.requests_log[-1].json() == {+ "DatabaseName": "analytics",+ "CatalogId": "123456789012",+ "TableName": "events",+ "ColumnNames": ["col_0", "col_1"],+ }++@pytest.mark.parametrize("schema_name", [None, "databases"])@pytest.mark.parametrize("code",