Graph-Native Infrastructure for Context and Accountable AI Systems
Python agent-memoryaiai-governanceai-infrastructureartificial-intelligencecontext-engineeringcontext-graphsdata-engineeringdecision-intelligencedeveloper-toolsexplainable-aigenerative-aigraph-ragknowledge-graphllmontologyprovenancepythonreasoningsemantic-search
Address review: column lineage, connection reuse, UC name validation
KaifAhmad12d5bd18fa4f4
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 03eed002..951c0053 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -14,12 +14,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0- **Databricks Connector (Unity Catalog + Delta Lake ingestion)** (#747) by @KaifAhmad1- Added `DatabricksIngestor` (`semantica/ingest/databricks_ingestor.py`), mirroring `SnowflakeIngestor`'s structure and public API shape: a `DatabricksConnector` connection handler, a `DatabricksData` dataclass, and an optional-import guard for `databricks-sdk`/`databricks-sql-connector`- Supports personal access token and OAuth M2M (service principal `client_id`/`client_secret`) authentication, configurable via constructor args or `DATABRICKS_*` environment variables- - `ingest_table()`/`ingest_query()` run against a SQL warehouse or cluster via `databricks-sql-connector`, with `where`/`order_by`/`limit`/`offset` support and the same identifier-escaping and unsafe-`ORDER BY` rejection as `SnowflakeIngestor`- - `get_table_schema()`, `list_catalogs()`, `list_schemas()`, and `list_tables()` introspect Unity Catalog via `databricks-sdk`'s `WorkspaceClient`; `get_table_lineage()` calls Unity Catalog's table-lineage REST API to surface upstream/downstream dependencies for `Table --DEPENDS_ON--> Table` graph edges+ - `ingest_table()`/`ingest_query()` run against a SQL warehouse or cluster via `databricks-sql-connector`, with `where`/`order_by`/`limit`/`offset` support and the same identifier-escaping and unsafe-`ORDER BY` rejection as `SnowflakeIngestor`; each call closes the SQL connection it opened unless one is already open (e.g. via the `with DatabricksIngestor(...)` context manager), which reuses and closes it exactly once instead of leaking a second connection per call+ - `get_table_schema()`, `list_catalogs()`, `list_schemas()`, and `list_tables()` introspect Unity Catalog via `databricks-sdk`'s `WorkspaceClient`, validating both catalog and schema are resolved before calling the SDK; `get_table_lineage()` calls Unity Catalog's table-lineage REST API for upstream/downstream `Table --DEPENDS_ON--> Table` dependencies, plus an opt-in `include_column_lineage=True` that resolves per-column lineage via the column-lineage API- `export_as_documents()` converts ingested rows into Semantica document dicts for KG construction, matching `SnowflakeIngestor.export_as_documents()`'s shape- Registered as a lazy export in `semantica.ingest` (`DatabricksIngestor`, `DatabricksData`, `DatabricksConnector`) and as the `db-databricks` optional extra (`pip install "semantica[db-databricks]"`) in `pyproject.toml`, included in `db-all`- New `docs/integrations/databricks.md` page modeled on `docs/integrations/snowflake.md`, plus a `DatabricksIngestor` section and table row in `docs/reference/ingest.md` and cross-links between the two integration pages- - 27 unit tests in `tests/test_databricks_ingestor.py` covering both auth methods, table/query ingestion, pagination, unsafe `ORDER BY` rejection, schema/catalog/table listing, lineage, document export, the context manager, and the missing-dependency error path, closing #747+ - 35 unit tests in `tests/test_databricks_ingestor.py` covering both auth methods, table/query ingestion, connection lifecycle (including reuse under the context manager), pagination, unsafe `ORDER BY` rejection, catalog/schema validation, schema/catalog/table listing, table and column lineage, document export, and the missing-dependency error path, closing #747- **SQLite Vector Store Backend (`sqlite-vec`)** (#726) by @Luffy2208 and @KaifAhmad1- Added `SQLiteVecStore` (`semantica/vector_store/sqlite_vec_store.py`), a disk-backed local vector store using the `sqlite-vec` extension's `vec0` virtual tables, closing #240diff --git a/docs/integrations/databricks.md b/docs/integrations/databricks.mdindex a58dc50a..f3ef7b65 100644--- a/docs/integrations/databricks.md+++ b/docs/integrations/databricks.md@@ -116,7 +116,7 @@ schemas = ingestor.list_schemas(catalog="main")tables = ingestor.list_tables(catalog="main", schema="default")```-### Table lineage+### Table and column lineage```pythonlineage = ingestor.get_table_lineage("customers", catalog="main", schema="default")@@ -126,6 +126,19 @@ print(lineage["downstream"]) # tables derived from `customers`Use `get_table_lineage` to build `Table --DEPENDS_ON--> Table` edges in the knowledge graph directly from Unity Catalog's lineage tracking, without re-deriving lineage from query logs.+<Tip>+Pass `include_column_lineage=True` to also resolve per-column upstream/downstream references (one extra Unity Catalog request per column, so it's opt-in):++```python+lineage = ingestor.get_table_lineage(+ "customers", catalog="main", schema="default", include_column_lineage=True,+)+print(lineage["columns"]["email"])+# {"upstream": ["main.default.raw_customers.email_address"], "downstream": []}+```++</Tip>+## Export as Semantica Documentsdiff --git a/semantica/ingest/databricks_ingestor.py b/semantica/ingest/databricks_ingestor.pyindex 1d90a28f..d6ee8d46 100644--- a/semantica/ingest/databricks_ingestor.py+++ b/semantica/ingest/databricks_ingestor.py@@ -11,7 +11,7 @@ Key Features:- Multiple authentication methods (personal access token, OAuth M2M)- Query execution with streaming and pagination- Table and schema introspection- - Table-level lineage via Unity Catalog's lineage API+ - Table-level and (optional) column-level lineage via Unity Catalog's lineage API- Progress tracking and error handling- Connection management with proper cleanup@@ -166,6 +166,10 @@ class DatabricksConnector:"""Establish a Databricks SQL connection for table/query ingestion.+ If a connection is already open (e.g. because the ingestor is being+ used as a context manager), it is reused instead of opening a second+ one, which would otherwise be left unclosed.+Returns:Connection: databricks-sql-connector connection object@@ -173,6 +177,9 @@ class DatabricksConnector:ProcessingError: If connection failsValidationError: If 'http_path' is not configured"""+ if self.connection is not None:+ return self.connection+if not self.http_path:raise ValidationError("Databricks 'http_path' is required for SQL ingestion. "@@ -434,6 +441,7 @@ class DatabricksIngestor:)try:+ already_connected = self.connector.connection is not Noneconn = self.connector.connect()try:table_ref = self._escaped_table_ref(table_name, catalog, schema)@@ -506,7 +514,8 @@ class DatabricksIngestor:metadata={"query": query},)finally:- self.connector.disconnect()+ if not already_connected:+ self.connector.disconnect()except Exception as e:self.progress_tracker.stop_tracking(@@ -554,6 +563,7 @@ class DatabricksIngestor:)try:+ already_connected = self.connector.connection is not Noneconn = self.connector.connect()try:self.logger.debug(f"Executing query: {query[:100]}...")@@ -610,7 +620,8 @@ class DatabricksIngestor:metadata={"parameters": parameters} if parameters else {},)finally:- self.connector.disconnect()+ if not already_connected:+ self.connector.disconnect()except Exception as e:self.progress_tracker.stop_tracking(@@ -651,6 +662,12 @@ class DatabricksIngestor:"Provide via 'catalog' parameter or set default catalog in connector.")+ if not schema:+ raise ValidationError(+ "Schema name is required for schema introspection. "+ "Provide via 'schema' parameter or set default schema in connector."+ )+client = self.connector.get_workspace_client()full_name = self._full_table_name(table_name, catalog, schema)@@ -785,19 +802,27 @@ class DatabricksIngestor:table_name: str,catalog: Optional[str] = None,schema: Optional[str] = None,- ) -> Dict[str, List[str]]:+ include_column_lineage: bool = False,+ ) -> Dict[str, Any]:"""- Get table-level lineage from Unity Catalog.+ Get table-level (and optionally column-level) lineage from Unity Catalog.Args:table_name: Name of the tablecatalog: Catalog name (uses default if not provided)schema: Schema name (uses default if not provided)+ include_column_lineage: If True, also fetch per-column upstream/+ downstream lineage via Unity Catalog's column-lineage API. This+ issues one additional request per column in the table, so it is+ slower than table-level lineage alone and is opt-in.Returns:dict: Lineage information containing:- upstream: List of fully-qualified upstream table names- downstream: List of fully-qualified downstream table names+ - columns: (only when include_column_lineage=True) dict mapping+ each column name to {"upstream": [...], "downstream": [...]}+ fully-qualified column referencesRaises:ProcessingError: If lineage retrieval fails@@ -805,6 +830,19 @@ class DatabricksIngestor:try:catalog = catalog or self.connector.catalogschema = schema or self.connector.schema++ if not catalog:+ raise ValidationError(+ "Catalog name is required for lineage retrieval. "+ "Provide via 'catalog' parameter or set default catalog in connector."+ )++ if not schema:+ raise ValidationError(+ "Schema name is required for lineage retrieval. "+ "Provide via 'schema' parameter or set default schema in connector."+ )+full_name = self._full_table_name(table_name, catalog, schema)client = self.connector.get_workspace_client()@@ -831,12 +869,84 @@ class DatabricksIngestor:f"{len(upstream)} upstream, {len(downstream)} downstream")- return {"upstream": upstream, "downstream": downstream}+ result: Dict[str, Any] = {"upstream": upstream, "downstream": downstream}++ if include_column_lineage:+ result["columns"] = self._get_column_lineage(+ client, full_name, table_name, catalog, schema+ )++ return resultexcept Exception as e:self.logger.error(f"Failed to get table lineage: {e}")raise ProcessingError(f"Failed to get table lineage: {e}") from e+ def _get_column_lineage(+ self,+ client: "WorkspaceClient",+ full_name: str,+ table_name: str,+ catalog: str,+ schema: str,+ ) -> Dict[str, Dict[str, List[str]]]:+ """Fetch per-column lineage for every column in a table.++ Unity Catalog's column-lineage API resolves one column at a time, so+ this issues one lineage-tracking request per column in the table's+ schema. Columns that fail to resolve lineage (no permissions, no+ tracked lineage, etc.) are skipped rather than failing the whole call.+ """+ schema_info = self.get_table_schema(table_name, catalog=catalog, schema=schema)+ columns_lineage: Dict[str, Dict[str, List[str]]] = {}++ for column in schema_info["columns"]:+ column_name = column["name"]+ try:+ response = client.api_client.do(+ "GET",+ "/api/2.0/lineage-tracking/column-lineage",+ query={"table_name": full_name, "column_name": column_name},+ )+ except Exception as e:+ self.logger.debug(+ f"Skipping column lineage for {full_name}.{column_name}: {e}"+ )+ continue++ columns_lineage[column_name] = {+ "upstream": [+ ref+ for ref in (+ self._format_column_ref(item)+ for item in response.get("upstream_cols", []) or []+ )+ if ref+ ],+ "downstream": [+ ref+ for ref in (+ self._format_column_ref(item)+ for item in response.get("downstream_cols", []) or []+ )+ if ref+ ],+ }++ return columns_lineage++ @staticmethod+ def _format_column_ref(item: Dict[str, Any]) -> Optional[str]:+ """Format a Unity Catalog column-lineage entry as 'catalog.schema.table.column'."""+ table = item.get("table_name") or item.get("tableName")+ column = item.get("name") or item.get("column_name") or item.get("columnName")+ if not table or not column:+ return None+ catalog_name = item.get("catalog_name") or item.get("catalogName")+ schema_name = item.get("schema_name") or item.get("schemaName")+ parts = [part for part in (catalog_name, schema_name, table, column) if part]+ return ".".join(parts)+def export_as_documents(self,data: DatabricksData,diff --git a/tests/test_databricks_ingestor.py b/tests/test_databricks_ingestor.pyindex 02ecce12..d78292f3 100644--- a/tests/test_databricks_ingestor.py+++ b/tests/test_databricks_ingestor.py@@ -512,6 +512,30 @@ class TestDatabricksIngestor:full_name="TEST_CATALOG.TEST_SCHEMA.customers")+ @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)+ @patch("semantica.ingest.databricks_ingestor.databricks_sql")+ @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")+ def test_get_table_schema_requires_schema(self, mock_ws_client_cls, mock_sql):+ """Test that get_table_schema() raises instead of calling the SDK with a missing schema."""+ from semantica.ingest.databricks_ingestor import DatabricksIngestor+ from semantica.utils.exceptions import ProcessingError++ mock_ws_client = Mock()+ mock_ws_client_cls.return_value = mock_ws_client++ ingestor = DatabricksIngestor(+ host="https://adb-xxx.azuredatabricks.net",+ token="test_token",+ http_path="/sql/1.0/warehouses/xxxx",+ catalog="TEST_CATALOG",+ )+ ingestor.connector.schema = None++ with pytest.raises(ProcessingError, match="Schema name is required"):+ ingestor.get_table_schema("customers")++ mock_ws_client.tables.get.assert_not_called()+@patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)@patch("semantica.ingest.databricks_ingestor.databricks_sql")@patch("semantica.ingest.databricks_ingestor.WorkspaceClient")@@ -624,6 +648,95 @@ class TestDatabricksIngestor:assert lineage["upstream"] == ["main.default.raw_customers"]assert lineage["downstream"] == ["main.default.customer_summary"]+ assert "columns" not in lineage++ @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)+ @patch("semantica.ingest.databricks_ingestor.databricks_sql")+ @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")+ def test_get_table_lineage_with_column_lineage(self, mock_ws_client_cls, mock_sql):+ """Test that include_column_lineage=True fetches per-column lineage."""+ from semantica.ingest.databricks_ingestor import DatabricksIngestor++ mock_column_1 = Mock()+ mock_column_1.name = "id"+ mock_column_1.type_text = "BIGINT"+ mock_column_1.nullable = False+ mock_column_1.comment = None++ mock_column_2 = Mock()+ mock_column_2.name = "name"+ mock_column_2.type_text = "STRING"+ mock_column_2.nullable = True+ mock_column_2.comment = None++ mock_table_info = Mock()+ mock_table_info.columns = [mock_column_1, mock_column_2]++ def do_side_effect(method, path, query=None):+ if path.endswith("table-lineage"):+ return {+ "upstreams": [{"tableInfo": {"name": "main.default.raw_customers"}}],+ "downstreams": [],+ }+ if path.endswith("column-lineage"):+ if query["column_name"] == "id":+ return {+ "upstream_cols": [+ {+ "catalog_name": "main",+ "schema_name": "default",+ "table_name": "raw_customers",+ "name": "customer_id",+ }+ ],+ "downstream_cols": [],+ }+ return {"upstream_cols": [], "downstream_cols": []}+ raise AssertionError(f"unexpected path: {path}")++ mock_ws_client = Mock()+ mock_ws_client.tables.get = Mock(return_value=mock_table_info)+ mock_ws_client.api_client.do = Mock(side_effect=do_side_effect)+ mock_ws_client_cls.return_value = mock_ws_client++ ingestor = DatabricksIngestor(+ host="https://adb-xxx.azuredatabricks.net",+ token="test_token",+ http_path="/sql/1.0/warehouses/xxxx",+ catalog="main",+ schema="default",+ )++ lineage = ingestor.get_table_lineage("customers", include_column_lineage=True)++ assert lineage["upstream"] == ["main.default.raw_customers"]+ assert lineage["columns"]["id"]["upstream"] == ["main.default.raw_customers.customer_id"]+ assert lineage["columns"]["id"]["downstream"] == []+ assert lineage["columns"]["name"]["upstream"] == []++ @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)+ @patch("semantica.ingest.databricks_ingestor.databricks_sql")+ @patch("semantica.ingest.databricks_ingestor.WorkspaceClient")+ def test_get_table_lineage_requires_schema(self, mock_ws_client_cls, mock_sql):+ """Test that get_table_lineage() validates schema before calling the REST API."""+ from semantica.ingest.databricks_ingestor import DatabricksIngestor+ from semantica.utils.exceptions import ProcessingError++ mock_ws_client = Mock()+ mock_ws_client_cls.return_value = mock_ws_client++ ingestor = DatabricksIngestor(+ host="https://adb-xxx.azuredatabricks.net",+ token="test_token",+ http_path="/sql/1.0/warehouses/xxxx",+ catalog="main",+ )+ ingestor.connector.schema = None++ with pytest.raises(ProcessingError, match="Schema name is required"):+ ingestor.get_table_lineage("customers")++ mock_ws_client.api_client.do.assert_not_called()@patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)@patch("semantica.ingest.databricks_ingestor.databricks_sql")@@ -677,6 +790,39 @@ class TestDatabricksIngestor:mock_conn.close.assert_called()+ @patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)+ @patch("semantica.ingest.databricks_ingestor.databricks_sql")+ def test_context_manager_reuses_connection_across_calls(+ self, mock_sql, mock_databricks_connection+ ):+ """Test that ingest_table()/ingest_query() reuse (not leak) the connection+ opened by __enter__, and only close it once on __exit__."""+ from semantica.ingest.databricks_ingestor import DatabricksIngestor++ mock_conn, mock_cursor = mock_databricks_connection+ mock_sql.connect = Mock(return_value=mock_conn)++ with DatabricksIngestor(+ host="https://adb-xxx.azuredatabricks.net",+ token="test_token",+ http_path="/sql/1.0/warehouses/xxxx",+ ) as ingestor:+ ingestor.ingest_table("customers")+ # The connection opened by __enter__ must still be the live one:+ # ingest_table() should not have closed it out from under the+ # context manager.+ assert ingestor.connector.connection == mock_conn+ mock_conn.close.assert_not_called()++ ingestor.ingest_query("SELECT * FROM sales")+ assert ingestor.connector.connection == mock_conn+ mock_conn.close.assert_not_called()++ # databricks_sql.connect() should only have been called once (by+ # __enter__): both ingestion calls reused that same connection.+ mock_sql.connect.assert_called_once()+ mock_conn.close.assert_called_once()+@patch("semantica.ingest.databricks_ingestor.DATABRICKS_AVAILABLE", True)@patch("semantica.ingest.databricks_ingestor.databricks_sql")def test_convert_datetime(self, mock_sql):