Skip to content

Graph-Native Infrastructure for Context and Accountable AI Systems

Address review: column lineage, connection reuse, UC name validation

KaifAhmad12d5bd18fa4f4
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 03eed002..951c0053 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,12 +14,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Databricks Connector (Unity Catalog + Delta Lake ingestion)** (#747) by @KaifAhmad1
- Added `DatabricksIngestor` (`semantica/ingest/databricks_ingestor.py`), mirroring `SnowflakeIngestor`'s structure and public API shape: a `DatabricksConnector` connection handler, a `DatabricksData` dataclass, and an optional-import guard for `databricks-sdk`/`databricks-sql-connector`
- Supports personal access token and OAuth M2M (service principal `client_id`/`client_secret`) authentication, configurable via constructor args or `DATABRICKS_*` environment variables
- - `ingest_table()`/`ingest_query()` run against a SQL warehouse or cluster via `databricks-sql-connector`, with `where`/`order_by`/`limit`/`offset` support and the same identifier-escaping and unsafe-`ORDER BY` rejection as `SnowflakeIngestor`
- - `get_table_schema()`, `list_catalogs()`, `list_schemas()`, and `list_tables()` introspect Unity Catalog via `databricks-sdk`'s `WorkspaceClient`; `get_table_lineage()` calls Unity Catalog's table-lineage REST API to surface upstream/downstream dependencies for `Table --DEPENDS_ON--> Table` graph edges
+ - `ingest_table()`/`ingest_query()` run against a SQL warehouse or cluster via `databricks-sql-connector`, with `where`/`order_by`/`limit`/`offset` support and the same identifier-escaping and unsafe-`ORDER BY` rejection as `SnowflakeIngestor`; each call closes the SQL connection it opened unless one is already open (e.g. via the `with DatabricksIngestor(...)` context manager), which reuses and closes it exactly once instead of leaking a second connection per call
+ - `get_table_schema()`, `list_catalogs()`, `list_schemas()`, and `list_tables()` introspect Unity Catalog via `databricks-sdk`'s `WorkspaceClient`, validating both catalog and schema are resolved before calling the SDK; `get_table_lineage()` calls Unity Catalog's table-lineage REST API for upstream/downstream `Table --DEPENDS_ON--> Table` dependencies, plus an opt-in `include_column_lineage=True` that resolves per-column lineage via the column-lineage API
- `export_as_documents()` converts ingested rows into Semantica document dicts for KG construction, matching `SnowflakeIngestor.export_as_documents()`'s shape
- Registered as a lazy export in `semantica.ingest` (`DatabricksIngestor`, `DatabricksData`, `DatabricksConnector`) and as the `db-databricks` optional extra (`pip install "semantica[db-databricks]"`) in `pyproject.toml`, included in `db-all`
- New `docs/integrations/databricks.md` page modeled on `docs/integrations/snowflake.md`, plus a `DatabricksIngestor` section and table row in `docs/reference/ingest.md` and cross-links between the two integration pages
- - 27 unit tests in `tests/test_databricks_ingestor.py` covering both auth methods, table/query ingestion, pagination, unsafe `ORDER BY` rejection, schema/catalog/table listing, lineage, document export, the context manager, and the missing-dependency error path, closing #747
+ - 35 unit tests in `tests/test_databricks_ingestor.py` covering both auth methods, table/query ingestion, connection lifecycle (including reuse under the context manager), pagination, unsafe `ORDER BY` rejection, catalog/schema validation, schema/catalog/table listing, table and column lineage, document export, and the missing-dependency error path, closing #747
- **SQLite Vector Store Backend (`sqlite-vec`)** (#726) by @Luffy2208 and @KaifAhmad1
- Added `SQLiteVecStore` (`semantica/vector_store/sqlite_vec_store.py`), a disk-backed local vector store using the `sqlite-vec` extension's `vec0` virtual tables, closing #240
diff --git a/docs/integrations/databricks.md b/docs/integrations/databricks.md
index a58dc50a..f3ef7b65 100644
--- a/docs/integrations/databricks.md
+++ b/docs/integrations/databricks.md
@@ -116,7 +116,7 @@ schemas = ingestor.list_schemas(catalog="main")
tables = ingestor.list_tables(catalog="main", schema="default")
```
-### Table lineage
+### Table and column lineage
```python
lineage = ingestor.get_table_lineage("customers", catalog="main", schema="default")
@@ -126,6 +126,19 @@ print(lineage["downstream"]) # tables derived from `customers`
Use `get_table_lineage` to build `Table --DEPENDS_ON--> Table` edges in the knowledge graph directly from Unity Catalog's lineage tracking, without re-deriving lineage from query logs.
+<Tip>
+Pass `include_column_lineage=True` to also resolve per-column upstream/downstream references (one extra Unity Catalog request per column, so it's opt-in):
+
+```python
+lineage = ingestor.get_table_lineage(
+ "customers", catalog="main", schema="default", include_column_lineage=True,
+)
+print(lineage["columns"]["email"])
+# {"upstream": ["main.default.raw_customers.email_address"], "downstream": []}
+```
+
+</Tip>
+
## Export as Semantica Documents
diff --git a/semantica/ingest/databricks_ingestor.py b/semantica/ingest/databricks_ingestor.py
index 1d90a28f..d6ee8d46 100644
--- a/semantica/ingest/databricks_ingestor.py
+++ b/semantica/ingest/databricks_ingestor.py
@@ -11,7 +11,7 @@ Key Features:
- Multiple authentication methods (personal access token, OAuth M2M)
- Query execution with streaming and pagination
- Table and schema introspection
- - Table-level lineage via Unity Catalog's lineage API
+ - Table-level and (optional) column-level lineage via Unity Catalog's lineage API
- Progress tracking and error handling
- Connection management with proper cleanup
@@ -166,6 +166,10 @@ class DatabricksConnector:
"""
Establish a Databricks SQL connection for table/query ingestion.
+ If a connection is already open (e.g. because the ingestor is being
+ used as a context manager), it is reused instead of opening a second
+ one, which would otherwise be left unclosed.
+
Returns:
Connection: databricks-sql-connector connection object
@@ -173,6 +177,9 @@ class DatabricksConnector:
ProcessingError: If connection fails
ValidationError: If 'http_path' is not configured
"""
+ if self.connection is not None:
+ return self.connection
+
if not self.http_path:
raise ValidationError(
"Databricks 'http_path' is required for SQL ingestion. "
@@ -434,6 +441,7 @@ class DatabricksIngestor:
)
try:
+ already_connected = self.connector.connection is not None
conn = self.connector.connect()
try:
table_ref = self._escaped_table_ref(table_name, catalog, schema)
@@ -506,7 +514,8 @@ class DatabricksIngestor:
metadata={"query": query},
)
finally:
- self.connector.disconnect()
+ if not already_connected:
+ self.connector.disconnect()
except Exception as e:
self.progress_tracker.stop_tracking(
@@ -554,6 +563,7 @@ class DatabricksIngestor:
)
try:
+ already_connected = self.connector.connection is not None
conn = self.connector.connect()
try:
self.logger.debug(f"Executing query: {query[:100]}...")
@@ -610,7 +620,8 @@ class DatabricksIngestor:
metadata={"parameters": parameters} if parameters else {},
)
finally:
- self.connector.disconnect()
+ if not already_connected:
+ self.connector.disconnect()
except Exception as e:
self.progress_tracker.stop_tracking(
@@ -651,6 +662,12 @@ class DatabricksIngestor:
"Provide via 'catalog' parameter or set default catalog in connector."
)
+ if not schema:
+ raise ValidationError(
+ "Schema name is required for schema introspection. "
+ "Provide via 'schema' parameter or set default schema in connector."
+ )
+
client = self.connector.get_workspace_client()
full_name = self._full_table_name(table_name, catalog, schema)
@@ -785,19 +802,27 @@ class DatabricksIngestor:
table_name: str,
catalog: Optional[str] = None,
schema: Optional[str] = None,
- ) -> Dict[str, List[str]]:
+ include_column_lineage: bool = False,
+ ) -> Dict[str, Any]:
"""
- Get table-level lineage from Unity Catalog.
+ Get table-level (and optionally column-level) lineage from Unity Catalog.
Args:
table_name: Name of the table
catalog: Catalog name (uses default if not provided)
schema: Schema name (uses default if not provided)
+ include_column_lineage: If True, also fetch per-column upstream/
+ downstream lineage via Unity Catalog's column-lineage API. This
+ issues one additional request per column in the table, so it is
+ slower than table-level lineage alone and is opt-in.
Returns:
dict: Lineage information containing:
- upstream: List of fully-qualified upstream table names
- downstream: List of fully-qualified downstream table names
+ - columns: (only when include_column_lineage=True) dict mapping
+ each column name to {"upstream": [...], "downstream": [...]}
+ fully-qualified column references
Raises:
ProcessingError: If lineage retrieval fails
@@ -805,6 +830,19 @@ class DatabricksIngestor:
try:
catalog = catalog or self.connector.catalog
schema = schema or self.connector.schema
+
+ if not catalog:
+ raise ValidationError(
+ "Catalog name is required for lineage retrieval. "
+ "Provide via 'catalog' parameter or set default catalog in connector."
+ )
+
+ if not schema:
+ raise ValidationError(
+ "Schema name is required for lineage retrieval. "
+ "Provide via 'schema' parameter or set default schema in connector."
+ )
+
full_name = self._full_table_name(table_name, catalog, schema)
client = self.connector.get_workspace_client()
@@ -831,12 +869,84 @@ class DatabricksIngestor:
f"{len(upstream)} upstream, {len(downstream)} downstream"
)
- return {"upstream": upstream, "downstream": downstream}
+ result: Dict[str, Any] = {"upstream": upstream, "downstream": downstream}
+
+ if include_column_lineage:
+ result["columns"] = self._get_column_lineage(
+ client, full_name, table_name, catalog, schema
+ )
+
+ return result
except Exception as e:
self.logger.error(f"Failed to get table lineage: {e}")
raise ProcessingError(f"Failed to get table lineage: {e}") from e
+ def _get_column_lineage(
+ self,
+ client: "WorkspaceClient",
+ full_name: str,
+ table_name: str,
+ catalog: str,
+ schema: str,
+ ) -> Dict[str, Dict[str, List[str]]]:
+ """Fetch per-column lineage for every column in a table.
+
+ Unity Catalog's column-lineage API resolves one column at a time, so
+ this issues one lineage-tracking request per column in the table's
+ schema. Columns that fail to resolve lineage (no permissions, no
+ tracked lineage, etc.) are skipped rather than failing the whole call.
+ """
+ schema_info = self.get_table_schema(table_name, catalog=catalog, schema=schema)
+ columns_lineage: Dict[str, Dict[str, List[str]]] = {}
+
+ for column in schema_info["columns"]:
+ column_name = column["name"]
+ try:
+ response = client.api_client.do(
+ "GET",
+ "/api/2.0/lineage-tracking/column-lineage",
+ query={"table_name": full_name, "column_name": column_name},
+ )
+ except Exception as e:
+ self.logger.debug(
+ f"Skipping column lineage for {full_name}.{column_name}: {e}"
+ )
+ continue
+
+ columns_lineage[column_name] = {
+ "upstream": [
+ ref
+ for ref in (
+ self._format_column_ref(item)
+ for item in response.get("upstream_cols", []) or []
+ )
+ if ref
+ ],
+ "downstream": [
+ ref
+ for ref in (
+ self._format_column_ref(item)
+ for item in response.get("downstream_cols", []) or []
+ )
+ if ref
+ ],
+ }
+
+ return columns_lineage
+
+ @staticmethod
+ def _format_column_ref(item: Dict[str, Any]) -> Optional[str]:
+ """Format a Unity Catalog column-lineage entry as 'catalog.schema.table.column'."""
+ table = item.get("table_name") or item.get("tableName")
+ column = item.get("name") or item.get("column_name") or item.get("columnName")
+ if not table or not column:
+ return None
+ catalog_name = item.get("catalog_name") or item.get("catalogName")
+ schema_name = item.get("schema_name") or item.get("schemaName")
+ parts = [part for part in (catalog_name, schema_name, table, column) if part]
+ return ".".join(parts)
+
def export_as_documents(
self,
data: DatabricksData,
diff --git a/tests/test_databricks_ingestor.py b/tests/test_databricks_ingestor.py
index 02ecce12..d78292f3 100644
--- a/tests/test_databricks_ingestor.py
+++ b/tests/test_databricks_ingestor.py
@@ -512,6 +512,30 @@ class TestDatabricksIngestor:
full_name="TEST_CATALOG.TEST_SCHEMA.customers"
)
+ @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+ @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+ @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
+ def test_get_table_schema_requires_schema(self, mock_ws_client_cls, mock_sql):
+ """Test that get_table_schema() raises instead of calling the SDK with a missing schema."""
+ from semantica.ingest.databricks_ingestor import DatabricksIngestor
+ from semantica.utils.exceptions import ProcessingError
+
+ mock_ws_client = Mock()
+ mock_ws_client_cls.return_value = mock_ws_client
+
+ ingestor = DatabricksIngestor(
+ host="https://adb-xxx.azuredatabricks.net",
+ token="test_token",
+ http_path="/sql/1.0/warehouses/xxxx",
+ catalog="TEST_CATALOG",
+ )
+ ingestor.connector.schema = None
+
+ with pytest.raises(ProcessingError, match="Schema name is required"):
+ ingestor.get_table_schema("customers")
+
+ mock_ws_client.tables.get.assert_not_called()
+
@patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
@patch("semantica.ingest.databricks_ingestor.databricks_sql")
@patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
@@ -624,6 +648,95 @@ class TestDatabricksIngestor:
assert lineage["upstream"] == ["main.default.raw_customers"]
assert lineage["downstream"] == ["main.default.customer_summary"]
+ assert "columns" not in lineage
+
+ @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+ @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+ @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
+ def test_get_table_lineage_with_column_lineage(self, mock_ws_client_cls, mock_sql):
+ """Test that include_column_lineage=True fetches per-column lineage."""
+ from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+ mock_column_1 = Mock()
+ mock_column_1.name = "id"
+ mock_column_1.type_text = "BIGINT"
+ mock_column_1.nullable = False
+ mock_column_1.comment = None
+
+ mock_column_2 = Mock()
+ mock_column_2.name = "name"
+ mock_column_2.type_text = "STRING"
+ mock_column_2.nullable = True
+ mock_column_2.comment = None
+
+ mock_table_info = Mock()
+ mock_table_info.columns = [mock_column_1, mock_column_2]
+
+ def do_side_effect(method, path, query=None):
+ if path.endswith("table-lineage"):
+ return {
+ "upstreams": [{"tableInfo": {"name": "main.default.raw_customers"}}],
+ "downstreams": [],
+ }
+ if path.endswith("column-lineage"):
+ if query["column_name"] == "id":
+ return {
+ "upstream_cols": [
+ {
+ "catalog_name": "main",
+ "schema_name": "default",
+ "table_name": "raw_customers",
+ "name": "customer_id",
+ }
+ ],
+ "downstream_cols": [],
+ }
+ return {"upstream_cols": [], "downstream_cols": []}
+ raise AssertionError(f"unexpected path: {path}")
+
+ mock_ws_client = Mock()
+ mock_ws_client.tables.get = Mock(return_value=mock_table_info)
+ mock_ws_client.api_client.do = Mock(side_effect=do_side_effect)
+ mock_ws_client_cls.return_value = mock_ws_client
+
+ ingestor = DatabricksIngestor(
+ host="https://adb-xxx.azuredatabricks.net",
+ token="test_token",
+ http_path="/sql/1.0/warehouses/xxxx",
+ catalog="main",
+ schema="default",
+ )
+
+ lineage = ingestor.get_table_lineage("customers", include_column_lineage=True)
+
+ assert lineage["upstream"] == ["main.default.raw_customers"]
+ assert lineage["columns"]["id"]["upstream"] == ["main.default.raw_customers.customer_id"]
+ assert lineage["columns"]["id"]["downstream"] == []
+ assert lineage["columns"]["name"]["upstream"] == []
+
+ @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+ @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+ @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")
+ def test_get_table_lineage_requires_schema(self, mock_ws_client_cls, mock_sql):
+ """Test that get_table_lineage() validates schema before calling the REST API."""
+ from semantica.ingest.databricks_ingestor import DatabricksIngestor
+ from semantica.utils.exceptions import ProcessingError
+
+ mock_ws_client = Mock()
+ mock_ws_client_cls.return_value = mock_ws_client
+
+ ingestor = DatabricksIngestor(
+ host="https://adb-xxx.azuredatabricks.net",
+ token="test_token",
+ http_path="/sql/1.0/warehouses/xxxx",
+ catalog="main",
+ )
+ ingestor.connector.schema = None
+
+ with pytest.raises(ProcessingError, match="Schema name is required"):
+ ingestor.get_table_lineage("customers")
+
+ mock_ws_client.api_client.do.assert_not_called()
@patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
@patch("semantica.ingest.databricks_ingestor.databricks_sql")
@@ -677,6 +790,39 @@ class TestDatabricksIngestor:
mock_conn.close.assert_called()
+ @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
+ @patch("semantica.ingest.databricks_ingestor.databricks_sql")
+ def test_context_manager_reuses_connection_across_calls(
+ self, mock_sql, mock_databricks_connection
+ ):
+ """Test that ingest_table()/ingest_query() reuse (not leak) the connection
+ opened by __enter__, and only close it once on __exit__."""
+ from semantica.ingest.databricks_ingestor import DatabricksIngestor
+
+ mock_conn, mock_cursor = mock_databricks_connection
+ mock_sql.connect = Mock(return_value=mock_conn)
+
+ with DatabricksIngestor(
+ host="https://adb-xxx.azuredatabricks.net",
+ token="test_token",
+ http_path="/sql/1.0/warehouses/xxxx",
+ ) as ingestor:
+ ingestor.ingest_table("customers")
+ # The connection opened by __enter__ must still be the live one:
+ # ingest_table() should not have closed it out from under the
+ # context manager.
+ assert ingestor.connector.connection == mock_conn
+ mock_conn.close.assert_not_called()
+
+ ingestor.ingest_query("SELECT * FROM sales")
+ assert ingestor.connector.connection == mock_conn
+ mock_conn.close.assert_not_called()
+
+ # databricks_sql.connect() should only have been called once (by
+ # __enter__): both ingestion calls reused that same connection.
+ mock_sql.connect.assert_called_once()
+ mock_conn.close.assert_called_once()
+
@patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)
@patch("semantica.ingest.databricks_ingestor.databricks_sql")
def test_convert_datetime(self, mock_sql):