Build and run agents you can see, understand and trust.
Python agentchatbotlarge-language-modelsllmllm-agentmcpmulti-agentmulti-modalreact-agentrealtime-agentvoice-agentvoice-assistant
feat(vdb): add Alibaba Cloud MySQL vector store implementation (#857)
carribeanea8cb9ddfe7a
diff --git a/examples/functionality/vector_store/alibabacloud_mysql_vector/README.md b/examples/functionality/vector_store/alibabacloud_mysql_vector/README.mdnew file mode 100644index 000000000..4e2cbf8c1--- /dev/null+++ b/examples/functionality/vector_store/alibabacloud_mysql_vector/README.md@@ -0,0 +1,422 @@+# AlibabaCloud MySQL Vector Store Example++This example demonstrates how to use the `AlibabaCloudMySQLStore` class in AgentScope's RAG system for vector storage and similarity search operations using AlibabaCloud MySQL (RDS) with native vector functions.++## Features++AlibabaCloudMySQLStore provides:+- Vector storage using MySQL's native VECTOR data type+- Automatic vector index creation (CREATE VECTOR INDEX) based on distance metric+- Vector functions (VEC_FROMTEXT, VEC_DISTANCE_COSINE, VEC_DISTANCE_EUCLIDEAN)+- Database-level distance calculation and sorting via ORDER BY+- Two distance metrics: COSINE and EUCLIDEAN (supported by AlibabaCloud MySQL)+- Metadata filtering support+- CRUD operations (Create, Read, Update, Delete)+- Support for chunked documents+- Direct access to the underlying MySQL connection for advanced operations+- Full integration with AlibabaCloud RDS MySQL instances++## Prerequisites++### 1. AlibabaCloud RDS MySQL Instance++You need an AlibabaCloud RDS MySQL instance with vector support:++- **Version**: MySQL 8.0++- **Vector Plugin**: Ensure the vector search plugin is enabled (check `vidx_disabled` parameter is OFF)+- **Network Access**: Configure security group and whitelist to allow access++#### Create RDS MySQL Instance on AlibabaCloud:++1. Go to [AlibabaCloud RDS Console](https://rdsnext.console.aliyun.com/)+2. Click "Create Instance"+3. Select MySQL 8.0 or higher+4. Configure specifications based on your needs+5. Set up network and security settings+6. Note down the connection endpoint (e.g., `rm-xxxxx.mysql.rds.aliyuncs.com`)++#### Configure Database:++```sql+-- Connect to your RDS MySQL instance+mysql -h rm-xxxxx.mysql.rds.aliyuncs.com -P 3306 -u your_username -p++-- Check if vector capability is enabled (vidx_disabled should be OFF)+SHOW VARIABLES LIKE 'vidx_disabled';+-- Expected result: vidx_disabled | OFF+-- If OFF, vector capability is enabled+-- If ON, contact AlibabaCloud support to enable vector search plugin++-- Create database+CREATE DATABASE agentscope_test;++-- Use the database+USE agentscope_test;++-- Verify vector functions are available+SELECT VEC_FROMTEXT('[1,2,3]');+```++### 2. Python Dependencies++```bash+pip install mysql-connector-python agentscope+```++### 3. Network Configuration++Ensure your local machine or server can access the RDS instance:+- Add your IP to the RDS whitelist+- Configure security group rules+- Use SSL connection if required++## Configuration++Update the connection parameters in `main.py`:++```python+store = AlibabaCloudMySQLStore(+ host="rm-xxxxx.mysql.rds.aliyuncs.com", # Your RDS endpoint+ port=3306,+ user="your_username", # Your RDS username+ password="your_password", # Your RDS password+ database="agentscope_test",+ table_name="test_vectors",+ dimensions=768, # Set to your embedding dimension+ distance="COSINE",+ # Optional: SSL configuration+ # connection_kwargs={+ # "ssl_ca": "/path/to/ca.pem",+ # "ssl_verify_cert": True,+ # }+)+```++## Running the Example++```bash+python main.py+```++## Example Tests++The example includes three comprehensive tests:++### 1. Basic CRUD Operations+- Initialize AlibabaCloudMySQLStore+- Add documents with embeddings+- Search for similar documents+- Delete documents+- Get the underlying MySQL connection++### 2. Search with Metadata Filtering+- Add documents with different categories+- Search with and without filters+- Use SQL WHERE clauses for filtering++### 3. Different Distance Metrics+- Test COSINE similarity (best for normalized vectors)+- Test EUCLIDEAN distance (best for absolute distance)++## Key Features Explained++### Distance Metrics++AlibabaCloud MySQL supports two distance metrics:++- **COSINE**: Measures the cosine of the angle between vectors. Values range from 0 (identical) to 2 (opposite). Best for text embeddings and normalized vectors.+- **EUCLIDEAN**: Measures the straight-line Euclidean distance between vectors. Lower values indicate similarity. Best for absolute distance measurements.++**Note**: Unlike some other vector databases, AlibabaCloud MySQL currently only supports COSINE and EUCLIDEAN distance functions. Inner Product (IP) is not supported.++### Metadata Filtering++Use SQL WHERE clauses to filter search results:++```python+results = await store.search(+ query_embedding=embedding,+ limit=10,+ filter='doc_id LIKE "ai%" AND chunk_id > 0',+)+```++### Table Structure++The implementation automatically creates a table with the following structure:++```sql+CREATE TABLE IF NOT EXISTS table_name (+ id VARCHAR(255) PRIMARY KEY,+ embedding VECTOR(dimensions) NOT NULL,+ doc_id VARCHAR(255) NOT NULL,+ chunk_id INT NOT NULL,+ content TEXT NOT NULL,+ total_chunks INT NOT NULL,+ INDEX idx_doc_id (doc_id),+ INDEX idx_chunk_id (chunk_id),+ VECTOR INDEX (embedding) M=16 DISTANCE=cosine -- or DISTANCE=euclidean+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;+```++**Note**: The vector index is created directly within the `CREATE TABLE` statement, not as a separate SQL command. The `M` parameter controls the HNSW algorithm's graph connectivity (default: 16).++### Performance Considerations++- **VECTOR Data Type**: Uses MySQL's native VECTOR type for efficient storage+- **Vector Index**: Automatically creates a vector index with the specified distance metric for fast similarity search+- **Database-Level Distance Calculation**: Vector distance calculations are performed at the database level using MySQL's native vector functions (VEC_DISTANCE_COSINE, VEC_DISTANCE_EUCLIDEAN), with sorting done via SQL ORDER BY+- **Native Vector Support**: MySQL 8.0+ has built-in vector functions that are highly optimized for vector operations+- **Supported Distance Metrics**: Only COSINE and EUCLIDEAN are supported+- **Small to Medium Datasets**: AlibabaCloudMySQLStore performs well for datasets up to 100K vectors+- **Large Datasets**: For datasets with millions of vectors, consider using dedicated vector databases (MilvusLite, Qdrant) with specialized indexing (HNSW, IVF, etc.)+- **RDS Performance**: Leverage AlibabaCloud RDS features like read replicas, backup, and monitoring++## Advanced Usage++### Direct Database Access++```python+# Get the MySQL connection for advanced operations+conn = store.get_client()+cursor = conn.cursor()++# Execute custom SQL queries+cursor.execute("SELECT COUNT(*) FROM test_vectors")+count = cursor.fetchone()+print(f"Total vectors: {count[0]}")+```++### Using MySQL Native Vector Functions++MySQL's native vector functions can be used directly in SQL queries:++```python+conn = store.get_client()+cursor = conn.cursor()++# Use MySQL native vector functions directly+query_vector = "[0.1,0.2,0.3,0.4]"+cursor.execute("""+ SELECT+ doc_id,+ VEC_DISTANCE_COSINE(vector, VEC_FROMTEXT(%s)) as distance+ FROM test_vectors+ ORDER BY distance ASC+ LIMIT 10+""", (query_vector,))++results = cursor.fetchall()++# Available MySQL vector functions in AlibabaCloud:+# - VEC_FROMTEXT(text) - Convert text "[1,2,3]" to vector+# - VEC_DISTANCE_COSINE(v1, v2) - Cosine distance+# - VEC_DISTANCE_EUCLIDEAN(v1, v2) - Euclidean distance+```++### SSL Connection++For secure connections to AlibabaCloud RDS:++```python+store = AlibabaCloudMySQLStore(+ host="rm-xxxxx.mysql.rds.aliyuncs.com",+ port=3306,+ user="your_username",+ password="your_password",+ database="agentscope_test",+ table_name="vectors",+ dimensions=768,+ distance="COSINE",+ connection_kwargs={+ "ssl_ca": "/path/to/ca.pem",+ "ssl_verify_cert": True,+ "ssl_verify_identity": True,+ },+)+```++### Batch Operations++```python+# Add large batches of documents+batch_size = 1000+for i in range(0, len(all_documents), batch_size):+ batch = all_documents[i:i + batch_size]+ await store.add(batch)+```++### Connection Pooling++```python+store = AlibabaCloudMySQLStore(+ host="rm-xxxxx.mysql.rds.aliyuncs.com",+ port=3306,+ user="your_username",+ password="your_password",+ database="agentscope_test",+ table_name="vectors",+ dimensions=768,+ distance="COSINE",+ connection_kwargs={+ "pool_name": "mypool",+ "pool_size": 10,+ "pool_reset_session": True,+ },+)+```++## Troubleshooting++### MySQL Version Check++Ensure your RDS MySQL version supports vector functions:++```sql+SELECT VERSION();+-- Should be MySQL 8.0 or higher++-- Check if vector capability is enabled (critical check)+SHOW VARIABLES LIKE 'vidx_disabled';+-- Expected result: vidx_disabled | OFF (vector capability enabled)++-- Test vector functions+SELECT VEC_FROMTEXT('[1,2,3]');+```++### Connection Errors++If you get connection errors:++1. **Check Whitelist**: Ensure your IP is in the RDS whitelist+2. **Security Group**: Verify security group rules allow port 3306+3. **Network Type**: Ensure you're using the correct endpoint (public/private)+4. **Credentials**: Double-check username and password++```bash+# Test connection from command line+mysql -h rm-xxxxx.mysql.rds.aliyuncs.com -P 3306 -u your_username -p+```++### Vector Function Errors++If you get errors about VEC_DISTANCE_COSINE or VECTOR type not being recognized:++1. **Check if vector capability is enabled**:++```sql+-- Check vidx_disabled parameter (must be OFF)+SHOW VARIABLES LIKE 'vidx_disabled';+-- Expected result: vidx_disabled | OFF+-- If ON, vector capability is disabled, contact AlibabaCloud support+```++2. Verify MySQL version is 8.0 or higher++```sql+SELECT VERSION();+```++3. Test vector functions availability:++```sql+-- Check if vector functions are available+SELECT VEC_FROMTEXT('[1,2,3]');++-- Check if VECTOR type is supported+CREATE TABLE test_vector (v VECTOR(3));+DROP TABLE test_vector;++-- List vector indexes+SHOW INDEX FROM your_table_name WHERE Index_type = 'VECTOR';+```++If `vidx_disabled` is ON, contact AlibabaCloud support to enable the vector search plugin for your RDS instance.++### Performance Optimization++For large datasets on AlibabaCloud RDS:++1. **Upgrade Instance**: Consider higher specifications (CPU, Memory)+2. **Read Replicas**: Use read replicas for read-heavy workloads+3. **Indexes**: Add indexes on frequently filtered columns+4. **Connection Pool**: Use connection pooling for concurrent operations+5. **Monitor**: Use AlibabaCloud CloudMonitor for performance insights++### Timeout Errors++If you experience timeout errors:++```python+store = AlibabaCloudMySQLStore(+ host="rm-xxxxx.mysql.rds.aliyuncs.com",+ port=3306,+ user="your_username",+ password="your_password",+ database="agentscope_test",+ table_name="vectors",+ dimensions=768,+ distance="COSINE",+ connection_kwargs={+ "connect_timeout": 30,+ "read_timeout": 60,+ "write_timeout": 60,+ },+)+```++## AlibabaCloud RDS Best Practices++1. **Backup**: Enable automatic backups in RDS console+2. **Monitoring**: Set up alerts for CPU, memory, and connection usage+3. **Security**: Use private network connections when possible+4. **Scaling**: Consider read-only instances for read-heavy workloads+5. **Cost Optimization**: Use reserved instances for long-term usage++## Related Resources++- [AlibabaCloud RDS Documentation](https://www.alibabacloud.com/help/en/apsaradb-for-rds)+- [AlibabaCloud MySQL Vector Functions](https://www.alibabacloud.com/help/en/rds/apsaradb-rds-for-mysql/vector-storage-1)+- [AgentScope RAG Tutorial](https://doc.agentscope.io/tutorial/task_rag.html)+- [MySQL Connector Python](https://dev.mysql.com/doc/connector-python/en/)++## Example Use Cases++### RAG System with AlibabaCloud++```python+from agentscope.rag import AlibabaCloudMySQLStore, KnowledgeBase++# Initialize vector store with AlibabaCloud RDS+store = AlibabaCloudMySQLStore(+ host="rm-xxxxx.mysql.rds.aliyuncs.com",+ port=3306,+ user="your_username",+ password="your_password",+ database="rag_system",+ table_name="knowledge_vectors",+ dimensions=768,+ distance="COSINE",+)++# Create knowledge base+kb = KnowledgeBase(store=store)++# Add documents+await kb.add_documents(documents)++# Search+results = await kb.search("What is AI?", top_k=5)+```++## Support++For issues related to:+- **AlibabaCloudMySQLStore**: Open an issue on AgentScope GitHub+- **RDS MySQL**: Contact AlibabaCloud Support+- **Vector Functions**: Check MySQL documentation or AlibabaCloud support++## License++This example is part of the AgentScope project and follows the same license.+diff --git a/examples/functionality/vector_store/alibabacloud_mysql_vector/main.py b/examples/functionality/vector_store/alibabacloud_mysql_vector/main.pynew file mode 100644index 000000000..f53d97b9f--- /dev/null+++ b/examples/functionality/vector_store/alibabacloud_mysql_vector/main.py@@ -0,0 +1,282 @@+# -*- coding: utf-8 -*-+"""Example of using AlibabaCloudMySQLStore in AgentScope RAG system."""+import asyncio+from agentscope.rag import (+ AlibabaCloudMySQLStore,+ Document,+ DocMetadata,+)+from agentscope.message import TextBlock+++async def example_basic_operations() -> None:+ """The example of basic CRUD operations with AlibabaCloudMySQLStore."""+ print("\n" + "=" * 60)+ print("Test 1: Basic CRUD Operations")+ print("=" * 60)++ # Initialize AlibabaCloudMySQLStore+ # Replace with your AlibabaCloud MySQL connection details+ store = AlibabaCloudMySQLStore(+ host="rm-xxxxx.mysql.rds.aliyuncs.com", # Your RDS endpoint+ port=3306,+ user="your_username",+ password="your_password",+ database="agentscope_test",+ table_name="test_vectors",+ dimensions=4, # Small dimension for testing+ distance="COSINE",+ )++ print("✓ AlibabaCloudMySQLStore initialized")++ # Create test documents with embeddings+ test_docs = [+ Document(+ metadata=DocMetadata(+ content=TextBlock(+ text="Artificial Intelligence is the future",+ ),+ doc_id="doc_1",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.1, 0.2, 0.3, 0.4],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Machine Learning is a subset of AI"),+ doc_id="doc_2",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.2, 0.3, 0.4, 0.5],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Deep Learning uses neural networks"),+ doc_id="doc_3",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.3, 0.4, 0.5, 0.6],+ ),+ ]++ # Test add operation+ await store.add(test_docs)+ print(f"✓ Added {len(test_docs)} documents to the store")++ # Test search operation+ query_embedding = [0.15, 0.25, 0.35, 0.45]+ results = await store.search(+ query_embedding=query_embedding,+ limit=2,+ )++ print(f"\n✓ Search completed, found {len(results)} results:")+ for i, result in enumerate(results, 1):+ print(f" {i}. Score: {result.score:.4f}")+ print(f" Content: {result.metadata.content}")+ print(f" Doc ID: {result.metadata.doc_id}")++ # Test search with score threshold+ results_filtered = await store.search(+ query_embedding=query_embedding,+ limit=5,+ score_threshold=0.9,+ )+ print(f"\n✓ Search with threshold (>0.9): {len(results_filtered)} results")++ # Test delete operation+ await store.delete(filter='doc_id = "doc_2"')+ print("\n✓ Deleted document with doc_id='doc_2'")++ # Verify deletion+ results_after_delete = await store.search(+ query_embedding=query_embedding,+ limit=5,+ )+ print(f"✓ After deletion: {len(results_after_delete)} documents remain")++ # Get client for advanced operations+ client = store.get_client()+ print(f"\n✓ Got MySQL connection: {type(client).__name__}")++ # Close connection+ store.close()+ print("✓ Connection closed")+++async def example_filter_search() -> None:+ """The example of search with metadata filtering."""+ print("\n" + "=" * 60)+ print("Test 2: Search with Metadata Filtering")+ print("=" * 60)++ store = AlibabaCloudMySQLStore(+ host="rm-xxxxx.mysql.rds.aliyuncs.com",+ port=3306,+ user="your_username",+ password="your_password",+ database="agentscope_test",+ table_name="filter_vectors",+ dimensions=4,+ distance="COSINE",+ )++ # Create documents with different categories+ docs = [+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Python is a programming language"),+ doc_id="prog_1",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.1, 0.2, 0.3, 0.4],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(+ text="Java is used for enterprise applications",+ ),+ doc_id="prog_2",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.2, 0.3, 0.4, 0.5],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Neural networks are used in AI"),+ doc_id="ai_1",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.3, 0.4, 0.5, 0.6],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Deep learning requires GPUs"),+ doc_id="ai_2",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.4, 0.5, 0.6, 0.7],+ ),+ ]++ await store.add(docs)+ print(f"✓ Added {len(docs)} documents with different doc_id prefixes")++ # Search without filter+ query_embedding = [0.25, 0.35, 0.45, 0.55]+ all_results = await store.search(+ query_embedding=query_embedding,+ limit=4,+ )+ print(f"\n✓ Search without filter: {len(all_results)} results")+ for i, result in enumerate(all_results, 1):+ doc_id = result.metadata.doc_id+ score = result.score+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")++ # Search with filter for programming docs+ prog_results = await store.search(+ query_embedding=query_embedding,+ limit=4,+ filter='doc_id LIKE "prog%"',+ )+ filter_msg = 'doc_id LIKE "prog%"'+ print(f"\n✓ Search with filter ({filter_msg}): {len(prog_results)}")+ for i, result in enumerate(prog_results, 1):+ doc_id = result.metadata.doc_id+ score = result.score+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")++ # Search with filter for AI docs+ ai_results = await store.search(+ query_embedding=query_embedding,+ limit=4,+ filter='doc_id LIKE "ai%"',+ )+ filter_msg = 'doc_id LIKE "ai%"'+ print(f"\n✓ Search with filter ({filter_msg}): {len(ai_results)}")+ for i, result in enumerate(ai_results, 1):+ doc_id = result.metadata.doc_id+ score = result.score+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")++ store.close()+++async def example_distance_metrics() -> None:+ """The example of different distance metrics."""+ print("\n" + "=" * 60)+ print("Test 3: Different Distance Metrics")+ print("=" * 60)++ # Test with different metrics+ # Note: AlibabaCloud MySQL only supports COSINE and EUCLIDEAN+ metrics = ["COSINE", "EUCLIDEAN"]++ for metric in metrics:+ print(f"\n--- Testing {metric} metric ---")+ store = AlibabaCloudMySQLStore(+ host="rm-xxxxx.mysql.rds.aliyuncs.com",+ port=3306,+ user="your_username",+ password="your_password",+ database="agentscope_test",+ table_name=f"{metric.lower()}_vectors",+ dimensions=4,+ distance=metric,+ )++ docs = [+ Document(+ metadata=DocMetadata(+ content=TextBlock(text=f"Test doc for {metric}"),+ doc_id=f"doc_{metric}_1",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.1, 0.2, 0.3, 0.4],+ ),+ ]++ await store.add(docs)+ results = await store.search(+ query_embedding=[0.1, 0.2, 0.3, 0.4],+ limit=1,+ )++ print(f"✓ {metric} metric: Score = {results[0].score:.4f}")+ store.close()+++async def main() -> None:+ """Run all examples."""+ print("\n" + "=" * 60)+ print("AlibabaCloud MySQL Vector Store Test Suite")+ print("=" * 60)++ try:+ await example_basic_operations()+ await example_filter_search()+ await example_distance_metrics()++ print("\n" + "=" * 60)+ print("✓ All tests completed successfully!")+ print("=" * 60)++ except Exception as e:+ print(f"\n✗ Test failed with error: {e}")+ import traceback++ traceback.print_exc()+++if __name__ == "__main__":+ asyncio.run(main())diff --git a/pyproject.toml b/pyproject.tomlindex 4a351bd23..104b45a74 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -69,6 +69,8 @@ full = [# The qdrant-client >= 1.16.0 has conflicts with pymilvus, so we fix# the version to 1.15.1 here."qdrant-client==1.15.1",+ "mysql-connector-python",+]dev = [diff --git a/src/agentscope/rag/__init__.py b/src/agentscope/rag/__init__.pyindex 66f82489e..146b35aa6 100644--- a/src/agentscope/rag/__init__.py+++ b/src/agentscope/rag/__init__.py@@ -16,6 +16,7 @@ from ._store import (VDBStoreBase,QdrantStore,MilvusLiteStore,+ AlibabaCloudMySQLStore,)from ._knowledge_base import KnowledgeBasefrom ._simple_knowledge import SimpleKnowledge@@ -32,6 +33,7 @@ __all__ = ["VDBStoreBase","QdrantStore","MilvusLiteStore",+ "AlibabaCloudMySQLStore","KnowledgeBase","SimpleKnowledge",]diff --git a/src/agentscope/rag/_store/__init__.py b/src/agentscope/rag/_store/__init__.pyindex ceebe517e..c107eae5d 100644--- a/src/agentscope/rag/_store/__init__.py+++ b/src/agentscope/rag/_store/__init__.py@@ -6,9 +6,11 @@ from ._store_base import ()from ._qdrant_store import QdrantStorefrom ._milvuslite_store import MilvusLiteStore+from ._alibabacloud_mysql_store import AlibabaCloudMySQLStore__all__ = ["VDBStoreBase","QdrantStore","MilvusLiteStore",+ "AlibabaCloudMySQLStore",]diff --git a/src/agentscope/rag/_store/_alibabacloud_mysql_store.py b/src/agentscope/rag/_store/_alibabacloud_mysql_store.pynew file mode 100644index 000000000..d62cab0a6--- /dev/null+++ b/src/agentscope/rag/_store/_alibabacloud_mysql_store.py@@ -0,0 +1,402 @@+# -*- coding: utf-8 -*-+"""The AlibabaCloud MySQL vector store implementation."""+import json+from typing import Any, Literal, TYPE_CHECKING++from .._reader import Document+from ._store_base import VDBStoreBase+from .._document import DocMetadata++from ..._utils._common import _map_text_to_uuid+from ...types import Embedding++if TYPE_CHECKING:+ from mysql.connector import MySQLConnection+else:+ MySQLConnection = "mysql.connector.MySQLConnection"+++class AlibabaCloudMySQLStore(VDBStoreBase):+ """The AlibabaCloud MySQL vector store implementation, supporting vector+ search operations using MySQL's native vector functions.++ .. note:: AlibabaCloud MySQL vector search requires MySQL 8.0+.+ This implementation uses MySQL's native vector functions+ (VEC_DISTANCE_COSINE, VEC_DISTANCE_EUCLIDEAN, VEC_FROMTEXT) for+ efficient vector similarity search with ORDER BY in SQL.+ Only COSINE and EUCLIDEAN distance metrics are supported.++ .. note:: Requires mysql-connector-python package. Install with:+ `pip install mysql-connector-python`++ .. note:: For AlibabaCloud MySQL instances, ensure vector search plugin+ is enabled. Contact AlibabaCloud support if needed.++ """++ def __init__(+ self,+ host: str,+ port: int,+ user: str,+ password: str,+ database: str,+ table_name: str,+ dimensions: int,+ distance: Literal["COSINE", "EUCLIDEAN"] = "COSINE",+ hnsw_m: int = 16,+ connection_kwargs: dict[str, Any] | None = None,+ ) -> None:+ """Initialize the AlibabaCloud MySQL vector store.++ Args:+ host (`str`):+ The hostname of the AlibabaCloud MySQL server.+ Example: "rm-xxxxx.mysql.rds.aliyuncs.com"+ port (`int`):+ The port number of the MySQL server (typically 3306).+ user (`str`):+ The username for authentication.+ password (`str`):+ The password for authentication.+ database (`str`):+ The database name to use.+ table_name (`str`):+ The name of the table to store the embeddings.+ dimensions (`int`):+ The dimension of the embeddings.+ distance (`Literal["COSINE", "EUCLIDEAN"]`, default to "COSINE"):+ The distance metric to use for similarity search. Can be+ one of "COSINE" (cosine similarity) or "EUCLIDEAN"+ (Euclidean distance). Defaults to "COSINE".+ hnsw_m (`int`, default to 16):+ The M parameter for HNSW vector index, which controls+ the number of bi-directional links created for each node+ during construction. Higher values create denser graphs+ with better recall but use more memory. Typical values+ range from 4 to 64. Defaults to 16.+ connection_kwargs (`dict[str, Any] | None`, optional):+ Other keyword arguments for the MySQL connector.+ Example: {"ssl_ca": "/path/to/ca.pem", "charset": "utf8mb4"}+ """++ try:+ import mysql.connector+ except ImportError as e:+ raise ImportError(+ "MySQL connector is not installed. Please install it with "+ "`pip install mysql-connector-python`.",+ ) from e++ connection_kwargs = connection_kwargs or {}++ # Initialize connection parameters+ self.connection_params = {+ "host": host,+ "port": port,+ "user": user,+ "password": password,+ "database": database,+ **connection_kwargs,+ }++ self.table_name = table_name+ self.dimensions = dimensions+ self.distance = distance+ self.hnsw_m = hnsw_m++ # Initialize connection+ self._conn = mysql.connector.connect(**self.connection_params)+ self._cursor = self._conn.cursor(dictionary=True)++ def _get_distance_function(self) -> str:+ """Get the MySQL native vector distance function name based on the+ distance metric.++ Returns:+ `str`:+ The SQL vector distance function name.+ """+ if self.distance == "COSINE":+ return "VEC_DISTANCE_COSINE"+ elif self.distance == "EUCLIDEAN":+ return "VEC_DISTANCE_EUCLIDEAN"+ else:+ raise ValueError(+ f"Unsupported distance metric: {self.distance}. "+ f"AlibabaCloud MySQL only supports 'COSINE' and 'EUCLIDEAN'.",+ )++ def _format_vector_for_sql(self, vector: list[float]) -> str:+ """Format a vector as a string for MySQL VEC_FROMTEXT function.+ Args:+ vector (`list[float]`):+ The vector to format.+ Returns:+ `str`:+ The formatted vector string like "[1,2,3,4]".+ """+ return "[" + ",".join(map(str, vector)) + "]"++ async def _validate_table(self) -> None:+ """Validate the table exists, if not, create it.+ Creates a table with VECTOR type columns and automatically+ creates a vector index based on the specified distance metric+ using HNSW algorithm.+ """+ # Get distance metric in lowercase for SQL+ distance_metric = self.distance.lower()++ # Create table with VECTOR INDEX in a single statement+ # VECTOR(dimensions) type is available in AlibabaCloud MySQL 8.0++ # VECTOR INDEX uses HNSW algorithm with M parameter for+ # graph connectivity+ # IF NOT EXISTS prevents errors if table already exists+ create_table_sql = f"""+ CREATE TABLE IF NOT EXISTS {self.table_name} (+ id VARCHAR(255) PRIMARY KEY,+ embedding VECTOR({self.dimensions}) NOT NULL,+ doc_id VARCHAR(255) NOT NULL,+ chunk_id INT NOT NULL,+ content TEXT NOT NULL,+ total_chunks INT NOT NULL,+ INDEX idx_doc_id (doc_id),+ INDEX idx_chunk_id (chunk_id),+ VECTOR INDEX (embedding) M={self.hnsw_m} DISTANCE={distance_metric}+ ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci+ """+ self._cursor.execute(create_table_sql)+ self._conn.commit()++ async def add(self, documents: list[Document], **kwargs: Any) -> None:+ """Add embeddings to the AlibabaCloud MySQL vector store.++ Args:+ documents (`list[Document]`):+ A list of embedding records to be recorded in the MySQL store.+ **kwargs (`Any`):+ Additional arguments for the insert operation.+ """+ await self._validate_table()++ # Prepare data for insertion+ for doc in documents:+ # Generate a unique ID+ unique_string = json.dumps(+ {+ "doc_id": doc.metadata.doc_id,+ "chunk_id": doc.metadata.chunk_id,+ "content": doc.metadata.content,+ },+ ensure_ascii=False,+ )+ unique_id = _map_text_to_uuid(unique_string)++ # Format vector for MySQL VEC_FROMTEXT+ if doc.embedding is None:+ raise ValueError(+ f"Document embedding cannot be None for doc_id: "+ f"{doc.metadata.doc_id}",+ )+ vector_text = self._format_vector_for_sql(doc.embedding)++ # Insert data using VEC_FROMTEXT to convert text to vector+ insert_sql = f"""+ INSERT INTO {self.table_name}+ (id, embedding, doc_id, chunk_id, content, total_chunks)+ VALUES (%s, VEC_FROMTEXT(%s), %s, %s, %s, %s)+ ON DUPLICATE KEY UPDATE+ embedding = VEC_FROMTEXT(%s),+ doc_id = VALUES(doc_id),+ chunk_id = VALUES(chunk_id),+ content = VALUES(content),+ total_chunks = VALUES(total_chunks)+ """++ # Serialize content to JSON if it's not a string+ content_str = (+ doc.metadata.content+ if isinstance(doc.metadata.content, str)+ else json.dumps(doc.metadata.content, ensure_ascii=False)+ )++ self._cursor.execute(+ insert_sql,+ (+ unique_id,+ vector_text,+ doc.metadata.doc_id,+ doc.metadata.chunk_id,+ content_str,+ doc.metadata.total_chunks,+ vector_text, # For ON DUPLICATE KEY UPDATE+ ),+ )++ self._conn.commit()++ async def search(+ self,+ query_embedding: Embedding,+ limit: int,+ score_threshold: float | None = None,+ **kwargs: Any,+ ) -> list[Document]:+ """Search relevant documents from the AlibabaCloud MySQL vector store.++ Args:+ query_embedding (`Embedding`):+ The embedding of the query text.+ limit (`int`):+ The number of relevant documents to retrieve.+ score_threshold (`float | None`, optional):+ The minimum similarity score threshold to filter the+ results. Score is calculated as 1 - distance, where+ higher scores indicate higher similarity. Only documents+ with score >= score_threshold will be returned.+ **kwargs (`Any`):+ Additional arguments for the search operation.+ - filter (`str`): WHERE clause to filter the search results.+ """++ # Format query vector for MySQL VEC_FROMTEXT+ query_vector_text = self._format_vector_for_sql(query_embedding)++ # Get the distance function+ distance_func = self._get_distance_function()++ # Build WHERE clause+ where_conditions = []+ if "filter" in kwargs and kwargs["filter"]:+ where_conditions.append(kwargs["filter"])++ # Add score threshold condition if specified+ # Score is calculated as 1 - distance, so higher scores+ # indicate higher similarity+ # To filter by score_threshold, we need:+ # 1.0 - distance >= score_threshold+ if score_threshold is not None:+ where_conditions.append(+ f"{distance_func}(embedding, VEC_FROMTEXT(%s)) <= %s",+ )++ where_clause = ""+ if where_conditions:+ where_clause = "WHERE " + " AND ".join(where_conditions)++ # Build and execute the search query using MySQL native vector+ # functions with ORDER BY for efficient sorting+ search_sql = f"""+ SELECT+ id,+ doc_id,+ chunk_id,+ content,+ total_chunks,+ {distance_func}(embedding, VEC_FROMTEXT(%s)) as distance+ FROM {self.table_name}+ {where_clause}+ ORDER BY distance ASC+ LIMIT %s+ """++ # Prepare parameters+ params: list[str | float | int] = [query_vector_text]+ if score_threshold is not None:+ # Convert score threshold to distance threshold:+ # distance <= 1.0 - score+ params.extend([query_vector_text, 1.0 - score_threshold])+ params.append(limit)++ self._cursor.execute(search_sql, params)+ results = self._cursor.fetchall()++ # Process results+ collected_res = []+ for row in results:+ # Deserialize content from JSON if it's a JSON string+ content = row["content"]+ try:+ content = json.loads(content)+ except (json.JSONDecodeError, TypeError):+ # If it's not valid JSON, keep it as is (plain string)+ pass++ doc_metadata = DocMetadata(+ content=content,+ doc_id=row["doc_id"],+ chunk_id=row["chunk_id"],+ total_chunks=row["total_chunks"],+ )++ # Create Document+ # Convert distance to score: score = 1 - distance+ # Higher scores indicate higher similarity+ collected_res.append(+ Document(+ embedding=None, # Vector not returned for efficiency+ score=1.0 - row["distance"],+ metadata=doc_metadata,+ ),+ )++ return collected_res++ async def delete(+ self,+ ids: list[str] | None = None,+ filter: str | None = None, # pylint: disable=redefined-builtin+ **kwargs: Any,+ ) -> None:+ """Delete documents from the AlibabaCloud MySQL vector store.++ Args:+ ids (`list[str] | None`, optional):+ List of entity IDs to delete.+ filter (`str | None`, optional):+ WHERE clause expression to filter documents to delete.+ **kwargs (`Any`):+ Additional arguments for the delete operation.+ """+ if ids is None and filter is None:+ raise ValueError(+ "Either ids or filter must be provided for deletion.",+ )++ if ids is not None:+ # Delete by IDs+ placeholders = ",".join(["%s"] * len(ids))+ delete_sql = (+ f"DELETE FROM {self.table_name} WHERE id IN ({placeholders})"+ )+ self._cursor.execute(delete_sql, ids)+ elif filter is not None:+ # Delete by filter+ delete_sql = f"DELETE FROM {self.table_name} WHERE {filter}"+ self._cursor.execute(delete_sql)++ self._conn.commit()++ def get_client(self) -> MySQLConnection:+ """Get the underlying MySQL connection, so that developers can access+ the full functionality of AlibabaCloud MySQL.++ Returns:+ `MySQLConnection`:+ The underlying MySQL connection.+ """+ return self._conn++ def close(self) -> None:+ """Close the database connection."""+ if self._cursor:+ self._cursor.close()+ if self._conn:+ self._conn.close()++ def __del__(self) -> None:+ """Destructor to ensure connection is closed."""+ try:+ self.close()+ except Exception: # pylint: disable=broad-except+ passdiff --git a/tests/rag_store_test.py b/tests/rag_store_test.pyindex afbd08372..485dd2382 100644--- a/tests/rag_store_test.py+++ b/tests/rag_store_test.py@@ -2,6 +2,7 @@"""Test the RAG store implementations."""import osfrom unittest import IsolatedAsyncioTestCase+from unittest.mock import MagicMock, patchfrom agentscope.message import TextBlockfrom agentscope.rag import (@@ -9,6 +10,7 @@ from agentscope.rag import (Document,DocMetadata,MilvusLiteStore,+ AlibabaCloudMySQLStore,)@@ -128,6 +130,135 @@ class RAGStoreTest(IsolatedAsyncioTestCase):"This is a test document.",)+ async def test_alibabacloud_mysql_store(self) -> None:+ """Test the AlibabaCloudMySQLStore implementation using mocks."""+ # Create mock MySQL module and connector+ mock_mysql_connector = MagicMock()+ mock_mysql = MagicMock()+ mock_mysql.connector = mock_mysql_connector++ # Create mock cursor and connection+ mock_cursor = MagicMock()+ mock_conn = MagicMock()++ # Configure mock connection to return mock cursor+ mock_conn.cursor.return_value = mock_cursor+ mock_mysql_connector.connect.return_value = mock_conn++ # Mock the search query result+ # Simulate a database row returned by fetchall+ mock_search_result = [+ {+ "id": "test-uuid-1",+ "doc_id": "doc1",+ "chunk_id": 0,+ "content": (+ '{"type": "text", "text": "This is a test document."}'+ ),+ "total_chunks": 2,+ "distance": 0.03, # Low distance = high similarity+ },+ ]++ # Use patch.dict to mock sys.modules+ with patch.dict(+ "sys.modules",+ {+ "mysql": mock_mysql,+ "mysql.connector": mock_mysql_connector,+ },+ ):+ # Create store instance+ store = AlibabaCloudMySQLStore(+ host="test-host",+ port=3306,+ user="test-user",+ password="test-password",+ database="test-database",+ table_name="test_vectors",+ dimensions=3,+ )++ # Verify connection was established+ mock_mysql_connector.connect.assert_called_once()++ # Test add operation+ await store.add(+ [+ Document(+ embedding=[0.1, 0.2, 0.3],+ metadata=DocMetadata(+ content=TextBlock(+ type="text",+ text="This is a test document.",+ ),+ doc_id="doc1",+ chunk_id=0,+ total_chunks=2,+ ),+ ),+ Document(+ embedding=[2.0, 3.8, 2.7],+ metadata=DocMetadata(+ content=TextBlock(+ type="text",+ text="This is another test document.",+ ),+ doc_id="doc1",+ chunk_id=1,+ total_chunks=2,+ ),+ ),+ ],+ )++ # Verify add operations executed SQL+ self.assertTrue(mock_cursor.execute.called)+ self.assertTrue(mock_conn.commit.called)++ # Reset mock for search operation+ mock_cursor.reset_mock()+ mock_conn.reset_mock()++ # Configure mock to return search results+ mock_cursor.fetchall.return_value = mock_search_result++ # Test search operation+ res = await store.search(+ query_embedding=[0.15, 0.25, 0.35],+ limit=3,+ score_threshold=0.95,+ )++ # Verify search results+ self.assertEqual(len(res), 1)+ # Score = 1 - distance = 1 - 0.03 = 0.97+ self.assertAlmostEqual(res[0].score, 0.97, places=2)+ self.assertEqual(+ res[0].metadata.content["text"],+ "This is a test document.",+ )+ self.assertEqual(res[0].metadata.doc_id, "doc1")+ self.assertEqual(res[0].metadata.chunk_id, 0)+ self.assertEqual(res[0].metadata.total_chunks, 2)++ # Verify search executed SQL query+ self.assertTrue(mock_cursor.execute.called)+ self.assertTrue(mock_cursor.fetchall.called)++ # Test delete operation+ await store.delete(filter='doc_id = "doc1"')++ # Verify delete executed SQL+ self.assertTrue(mock_conn.commit.called)++ # Test close+ store.close()++ # Verify connections were closed+ mock_cursor.close.assert_called()+ mock_conn.close.assert_called()+async def asyncTearDown(self) -> None:"""Clean up after tests."""# Remove Milvus Lite database file