Skip to content

Build and run agents you can see, understand and trust.

feat(vdb): add Alibaba Cloud MySQL vector store implementation (#857)

carribeanea8cb9ddfe7a
diff --git a/examples/functionality/vector_store/alibabacloud_mysql_vector/README.md b/examples/functionality/vector_store/alibabacloud_mysql_vector/README.md
new file mode 100644
index 000000000..4e2cbf8c1
--- /dev/null
+++ b/examples/functionality/vector_store/alibabacloud_mysql_vector/README.md
@@ -0,0 +1,422 @@
+# AlibabaCloud MySQL Vector Store Example
+
+This example demonstrates how to use the `AlibabaCloudMySQLStore` class in AgentScope's RAG system for vector storage and similarity search operations using AlibabaCloud MySQL (RDS) with native vector functions.
+
+## Features
+
+AlibabaCloudMySQLStore provides:
+- Vector storage using MySQL's native VECTOR data type
+- Automatic vector index creation (CREATE VECTOR INDEX) based on distance metric
+- Vector functions (VEC_FROMTEXT, VEC_DISTANCE_COSINE, VEC_DISTANCE_EUCLIDEAN)
+- Database-level distance calculation and sorting via ORDER BY
+- Two distance metrics: COSINE and EUCLIDEAN (supported by AlibabaCloud MySQL)
+- Metadata filtering support
+- CRUD operations (Create, Read, Update, Delete)
+- Support for chunked documents
+- Direct access to the underlying MySQL connection for advanced operations
+- Full integration with AlibabaCloud RDS MySQL instances
+
+## Prerequisites
+
+### 1. AlibabaCloud RDS MySQL Instance
+
+You need an AlibabaCloud RDS MySQL instance with vector support:
+
+- **Version**: MySQL 8.0+
+- **Vector Plugin**: Ensure the vector search plugin is enabled (check `vidx_disabled` parameter is OFF)
+- **Network Access**: Configure security group and whitelist to allow access
+
+#### Create RDS MySQL Instance on AlibabaCloud:
+
+1. Go to [AlibabaCloud RDS Console](https://rdsnext.console.aliyun.com/)
+2. Click "Create Instance"
+3. Select MySQL 8.0 or higher
+4. Configure specifications based on your needs
+5. Set up network and security settings
+6. Note down the connection endpoint (e.g., `rm-xxxxx.mysql.rds.aliyuncs.com`)
+
+#### Configure Database:
+
+```sql
+-- Connect to your RDS MySQL instance
+mysql -h rm-xxxxx.mysql.rds.aliyuncs.com -P 3306 -u your_username -p
+
+-- Check if vector capability is enabled (vidx_disabled should be OFF)
+SHOW VARIABLES LIKE 'vidx_disabled';
+-- Expected result: vidx_disabled | OFF
+-- If OFF, vector capability is enabled
+-- If ON, contact AlibabaCloud support to enable vector search plugin
+
+-- Create database
+CREATE DATABASE agentscope_test;
+
+-- Use the database
+USE agentscope_test;
+
+-- Verify vector functions are available
+SELECT VEC_FROMTEXT('[1,2,3]');
+```
+
+### 2. Python Dependencies
+
+```bash
+pip install mysql-connector-python agentscope
+```
+
+### 3. Network Configuration
+
+Ensure your local machine or server can access the RDS instance:
+- Add your IP to the RDS whitelist
+- Configure security group rules
+- Use SSL connection if required
+
+## Configuration
+
+Update the connection parameters in `main.py`:
+
+```python
+store = AlibabaCloudMySQLStore(
+ host="rm-xxxxx.mysql.rds.aliyuncs.com", # Your RDS endpoint
+ port=3306,
+ user="your_username", # Your RDS username
+ password="your_password", # Your RDS password
+ database="agentscope_test",
+ table_name="test_vectors",
+ dimensions=768, # Set to your embedding dimension
+ distance="COSINE",
+ # Optional: SSL configuration
+ # connection_kwargs={
+ # "ssl_ca": "/path/to/ca.pem",
+ # "ssl_verify_cert": True,
+ # }
+)
+```
+
+## Running the Example
+
+```bash
+python main.py
+```
+
+## Example Tests
+
+The example includes three comprehensive tests:
+
+### 1. Basic CRUD Operations
+- Initialize AlibabaCloudMySQLStore
+- Add documents with embeddings
+- Search for similar documents
+- Delete documents
+- Get the underlying MySQL connection
+
+### 2. Search with Metadata Filtering
+- Add documents with different categories
+- Search with and without filters
+- Use SQL WHERE clauses for filtering
+
+### 3. Different Distance Metrics
+- Test COSINE similarity (best for normalized vectors)
+- Test EUCLIDEAN distance (best for absolute distance)
+
+## Key Features Explained
+
+### Distance Metrics
+
+AlibabaCloud MySQL supports two distance metrics:
+
+- **COSINE**: Measures the cosine of the angle between vectors. Values range from 0 (identical) to 2 (opposite). Best for text embeddings and normalized vectors.
+- **EUCLIDEAN**: Measures the straight-line Euclidean distance between vectors. Lower values indicate similarity. Best for absolute distance measurements.
+
+**Note**: Unlike some other vector databases, AlibabaCloud MySQL currently only supports COSINE and EUCLIDEAN distance functions. Inner Product (IP) is not supported.
+
+### Metadata Filtering
+
+Use SQL WHERE clauses to filter search results:
+
+```python
+results = await store.search(
+ query_embedding=embedding,
+ limit=10,
+ filter='doc_id LIKE "ai%" AND chunk_id > 0',
+)
+```
+
+### Table Structure
+
+The implementation automatically creates a table with the following structure:
+
+```sql
+CREATE TABLE IF NOT EXISTS table_name (
+ id VARCHAR(255) PRIMARY KEY,
+ embedding VECTOR(dimensions) NOT NULL,
+ doc_id VARCHAR(255) NOT NULL,
+ chunk_id INT NOT NULL,
+ content TEXT NOT NULL,
+ total_chunks INT NOT NULL,
+ INDEX idx_doc_id (doc_id),
+ INDEX idx_chunk_id (chunk_id),
+ VECTOR INDEX (embedding) M=16 DISTANCE=cosine -- or DISTANCE=euclidean
+) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+```
+
+**Note**: The vector index is created directly within the `CREATE TABLE` statement, not as a separate SQL command. The `M` parameter controls the HNSW algorithm's graph connectivity (default: 16).
+
+### Performance Considerations
+
+- **VECTOR Data Type**: Uses MySQL's native VECTOR type for efficient storage
+- **Vector Index**: Automatically creates a vector index with the specified distance metric for fast similarity search
+- **Database-Level Distance Calculation**: Vector distance calculations are performed at the database level using MySQL's native vector functions (VEC_DISTANCE_COSINE, VEC_DISTANCE_EUCLIDEAN), with sorting done via SQL ORDER BY
+- **Native Vector Support**: MySQL 8.0+ has built-in vector functions that are highly optimized for vector operations
+- **Supported Distance Metrics**: Only COSINE and EUCLIDEAN are supported
+- **Small to Medium Datasets**: AlibabaCloudMySQLStore performs well for datasets up to 100K vectors
+- **Large Datasets**: For datasets with millions of vectors, consider using dedicated vector databases (MilvusLite, Qdrant) with specialized indexing (HNSW, IVF, etc.)
+- **RDS Performance**: Leverage AlibabaCloud RDS features like read replicas, backup, and monitoring
+
+## Advanced Usage
+
+### Direct Database Access
+
+```python
+# Get the MySQL connection for advanced operations
+conn = store.get_client()
+cursor = conn.cursor()
+
+# Execute custom SQL queries
+cursor.execute("SELECT COUNT(*) FROM test_vectors")
+count = cursor.fetchone()
+print(f"Total vectors: {count[0]}")
+```
+
+### Using MySQL Native Vector Functions
+
+MySQL's native vector functions can be used directly in SQL queries:
+
+```python
+conn = store.get_client()
+cursor = conn.cursor()
+
+# Use MySQL native vector functions directly
+query_vector = "[0.1,0.2,0.3,0.4]"
+cursor.execute("""
+ SELECT
+ doc_id,
+ VEC_DISTANCE_COSINE(vector, VEC_FROMTEXT(%s)) as distance
+ FROM test_vectors
+ ORDER BY distance ASC
+ LIMIT 10
+""", (query_vector,))
+
+results = cursor.fetchall()
+
+# Available MySQL vector functions in AlibabaCloud:
+# - VEC_FROMTEXT(text) - Convert text "[1,2,3]" to vector
+# - VEC_DISTANCE_COSINE(v1, v2) - Cosine distance
+# - VEC_DISTANCE_EUCLIDEAN(v1, v2) - Euclidean distance
+```
+
+### SSL Connection
+
+For secure connections to AlibabaCloud RDS:
+
+```python
+store = AlibabaCloudMySQLStore(
+ host="rm-xxxxx.mysql.rds.aliyuncs.com",
+ port=3306,
+ user="your_username",
+ password="your_password",
+ database="agentscope_test",
+ table_name="vectors",
+ dimensions=768,
+ distance="COSINE",
+ connection_kwargs={
+ "ssl_ca": "/path/to/ca.pem",
+ "ssl_verify_cert": True,
+ "ssl_verify_identity": True,
+ },
+)
+```
+
+### Batch Operations
+
+```python
+# Add large batches of documents
+batch_size = 1000
+for i in range(0, len(all_documents), batch_size):
+ batch = all_documents[i:i + batch_size]
+ await store.add(batch)
+```
+
+### Connection Pooling
+
+```python
+store = AlibabaCloudMySQLStore(
+ host="rm-xxxxx.mysql.rds.aliyuncs.com",
+ port=3306,
+ user="your_username",
+ password="your_password",
+ database="agentscope_test",
+ table_name="vectors",
+ dimensions=768,
+ distance="COSINE",
+ connection_kwargs={
+ "pool_name": "mypool",
+ "pool_size": 10,
+ "pool_reset_session": True,
+ },
+)
+```
+
+## Troubleshooting
+
+### MySQL Version Check
+
+Ensure your RDS MySQL version supports vector functions:
+
+```sql
+SELECT VERSION();
+-- Should be MySQL 8.0 or higher
+
+-- Check if vector capability is enabled (critical check)
+SHOW VARIABLES LIKE 'vidx_disabled';
+-- Expected result: vidx_disabled | OFF (vector capability enabled)
+
+-- Test vector functions
+SELECT VEC_FROMTEXT('[1,2,3]');
+```
+
+### Connection Errors
+
+If you get connection errors:
+
+1. **Check Whitelist**: Ensure your IP is in the RDS whitelist
+2. **Security Group**: Verify security group rules allow port 3306
+3. **Network Type**: Ensure you're using the correct endpoint (public/private)
+4. **Credentials**: Double-check username and password
+
+```bash
+# Test connection from command line
+mysql -h rm-xxxxx.mysql.rds.aliyuncs.com -P 3306 -u your_username -p
+```
+
+### Vector Function Errors
+
+If you get errors about VEC_DISTANCE_COSINE or VECTOR type not being recognized:
+
+1. **Check if vector capability is enabled**:
+
+```sql
+-- Check vidx_disabled parameter (must be OFF)
+SHOW VARIABLES LIKE 'vidx_disabled';
+-- Expected result: vidx_disabled | OFF
+-- If ON, vector capability is disabled, contact AlibabaCloud support
+```
+
+2. Verify MySQL version is 8.0 or higher
+
+```sql
+SELECT VERSION();
+```
+
+3. Test vector functions availability:
+
+```sql
+-- Check if vector functions are available
+SELECT VEC_FROMTEXT('[1,2,3]');
+
+-- Check if VECTOR type is supported
+CREATE TABLE test_vector (v VECTOR(3));
+DROP TABLE test_vector;
+
+-- List vector indexes
+SHOW INDEX FROM your_table_name WHERE Index_type = 'VECTOR';
+```
+
+If `vidx_disabled` is ON, contact AlibabaCloud support to enable the vector search plugin for your RDS instance.
+
+### Performance Optimization
+
+For large datasets on AlibabaCloud RDS:
+
+1. **Upgrade Instance**: Consider higher specifications (CPU, Memory)
+2. **Read Replicas**: Use read replicas for read-heavy workloads
+3. **Indexes**: Add indexes on frequently filtered columns
+4. **Connection Pool**: Use connection pooling for concurrent operations
+5. **Monitor**: Use AlibabaCloud CloudMonitor for performance insights
+
+### Timeout Errors
+
+If you experience timeout errors:
+
+```python
+store = AlibabaCloudMySQLStore(
+ host="rm-xxxxx.mysql.rds.aliyuncs.com",
+ port=3306,
+ user="your_username",
+ password="your_password",
+ database="agentscope_test",
+ table_name="vectors",
+ dimensions=768,
+ distance="COSINE",
+ connection_kwargs={
+ "connect_timeout": 30,
+ "read_timeout": 60,
+ "write_timeout": 60,
+ },
+)
+```
+
+## AlibabaCloud RDS Best Practices
+
+1. **Backup**: Enable automatic backups in RDS console
+2. **Monitoring**: Set up alerts for CPU, memory, and connection usage
+3. **Security**: Use private network connections when possible
+4. **Scaling**: Consider read-only instances for read-heavy workloads
+5. **Cost Optimization**: Use reserved instances for long-term usage
+
+## Related Resources
+
+- [AlibabaCloud RDS Documentation](https://www.alibabacloud.com/help/en/apsaradb-for-rds)
+- [AlibabaCloud MySQL Vector Functions](https://www.alibabacloud.com/help/en/rds/apsaradb-rds-for-mysql/vector-storage-1)
+- [AgentScope RAG Tutorial](https://doc.agentscope.io/tutorial/task_rag.html)
+- [MySQL Connector Python](https://dev.mysql.com/doc/connector-python/en/)
+
+## Example Use Cases
+
+### RAG System with AlibabaCloud
+
+```python
+from agentscope.rag import AlibabaCloudMySQLStore, KnowledgeBase
+
+# Initialize vector store with AlibabaCloud RDS
+store = AlibabaCloudMySQLStore(
+ host="rm-xxxxx.mysql.rds.aliyuncs.com",
+ port=3306,
+ user="your_username",
+ password="your_password",
+ database="rag_system",
+ table_name="knowledge_vectors",
+ dimensions=768,
+ distance="COSINE",
+)
+
+# Create knowledge base
+kb = KnowledgeBase(store=store)
+
+# Add documents
+await kb.add_documents(documents)
+
+# Search
+results = await kb.search("What is AI?", top_k=5)
+```
+
+## Support
+
+For issues related to:
+- **AlibabaCloudMySQLStore**: Open an issue on AgentScope GitHub
+- **RDS MySQL**: Contact AlibabaCloud Support
+- **Vector Functions**: Check MySQL documentation or AlibabaCloud support
+
+## License
+
+This example is part of the AgentScope project and follows the same license.
+
diff --git a/examples/functionality/vector_store/alibabacloud_mysql_vector/main.py b/examples/functionality/vector_store/alibabacloud_mysql_vector/main.py
new file mode 100644
index 000000000..f53d97b9f
--- /dev/null
+++ b/examples/functionality/vector_store/alibabacloud_mysql_vector/main.py
@@ -0,0 +1,282 @@
+# -*- coding: utf-8 -*-
+"""Example of using AlibabaCloudMySQLStore in AgentScope RAG system."""
+import asyncio
+from agentscope.rag import (
+ AlibabaCloudMySQLStore,
+ Document,
+ DocMetadata,
+)
+from agentscope.message import TextBlock
+
+
+async def example_basic_operations() -> None:
+ """The example of basic CRUD operations with AlibabaCloudMySQLStore."""
+ print("\n" + "=" * 60)
+ print("Test 1: Basic CRUD Operations")
+ print("=" * 60)
+
+ # Initialize AlibabaCloudMySQLStore
+ # Replace with your AlibabaCloud MySQL connection details
+ store = AlibabaCloudMySQLStore(
+ host="rm-xxxxx.mysql.rds.aliyuncs.com", # Your RDS endpoint
+ port=3306,
+ user="your_username",
+ password="your_password",
+ database="agentscope_test",
+ table_name="test_vectors",
+ dimensions=4, # Small dimension for testing
+ distance="COSINE",
+ )
+
+ print("✓ AlibabaCloudMySQLStore initialized")
+
+ # Create test documents with embeddings
+ test_docs = [
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(
+ text="Artificial Intelligence is the future",
+ ),
+ doc_id="doc_1",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.1, 0.2, 0.3, 0.4],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Machine Learning is a subset of AI"),
+ doc_id="doc_2",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.2, 0.3, 0.4, 0.5],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Deep Learning uses neural networks"),
+ doc_id="doc_3",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.3, 0.4, 0.5, 0.6],
+ ),
+ ]
+
+ # Test add operation
+ await store.add(test_docs)
+ print(f"✓ Added {len(test_docs)} documents to the store")
+
+ # Test search operation
+ query_embedding = [0.15, 0.25, 0.35, 0.45]
+ results = await store.search(
+ query_embedding=query_embedding,
+ limit=2,
+ )
+
+ print(f"\n✓ Search completed, found {len(results)} results:")
+ for i, result in enumerate(results, 1):
+ print(f" {i}. Score: {result.score:.4f}")
+ print(f" Content: {result.metadata.content}")
+ print(f" Doc ID: {result.metadata.doc_id}")
+
+ # Test search with score threshold
+ results_filtered = await store.search(
+ query_embedding=query_embedding,
+ limit=5,
+ score_threshold=0.9,
+ )
+ print(f"\n✓ Search with threshold (>0.9): {len(results_filtered)} results")
+
+ # Test delete operation
+ await store.delete(filter='doc_id = "doc_2"')
+ print("\n✓ Deleted document with doc_id='doc_2'")
+
+ # Verify deletion
+ results_after_delete = await store.search(
+ query_embedding=query_embedding,
+ limit=5,
+ )
+ print(f"✓ After deletion: {len(results_after_delete)} documents remain")
+
+ # Get client for advanced operations
+ client = store.get_client()
+ print(f"\n✓ Got MySQL connection: {type(client).__name__}")
+
+ # Close connection
+ store.close()
+ print("✓ Connection closed")
+
+
+async def example_filter_search() -> None:
+ """The example of search with metadata filtering."""
+ print("\n" + "=" * 60)
+ print("Test 2: Search with Metadata Filtering")
+ print("=" * 60)
+
+ store = AlibabaCloudMySQLStore(
+ host="rm-xxxxx.mysql.rds.aliyuncs.com",
+ port=3306,
+ user="your_username",
+ password="your_password",
+ database="agentscope_test",
+ table_name="filter_vectors",
+ dimensions=4,
+ distance="COSINE",
+ )
+
+ # Create documents with different categories
+ docs = [
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Python is a programming language"),
+ doc_id="prog_1",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.1, 0.2, 0.3, 0.4],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(
+ text="Java is used for enterprise applications",
+ ),
+ doc_id="prog_2",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.2, 0.3, 0.4, 0.5],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Neural networks are used in AI"),
+ doc_id="ai_1",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.3, 0.4, 0.5, 0.6],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Deep learning requires GPUs"),
+ doc_id="ai_2",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.4, 0.5, 0.6, 0.7],
+ ),
+ ]
+
+ await store.add(docs)
+ print(f"✓ Added {len(docs)} documents with different doc_id prefixes")
+
+ # Search without filter
+ query_embedding = [0.25, 0.35, 0.45, 0.55]
+ all_results = await store.search(
+ query_embedding=query_embedding,
+ limit=4,
+ )
+ print(f"\n✓ Search without filter: {len(all_results)} results")
+ for i, result in enumerate(all_results, 1):
+ doc_id = result.metadata.doc_id
+ score = result.score
+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")
+
+ # Search with filter for programming docs
+ prog_results = await store.search(
+ query_embedding=query_embedding,
+ limit=4,
+ filter='doc_id LIKE "prog%"',
+ )
+ filter_msg = 'doc_id LIKE "prog%"'
+ print(f"\n✓ Search with filter ({filter_msg}): {len(prog_results)}")
+ for i, result in enumerate(prog_results, 1):
+ doc_id = result.metadata.doc_id
+ score = result.score
+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")
+
+ # Search with filter for AI docs
+ ai_results = await store.search(
+ query_embedding=query_embedding,
+ limit=4,
+ filter='doc_id LIKE "ai%"',
+ )
+ filter_msg = 'doc_id LIKE "ai%"'
+ print(f"\n✓ Search with filter ({filter_msg}): {len(ai_results)}")
+ for i, result in enumerate(ai_results, 1):
+ doc_id = result.metadata.doc_id
+ score = result.score
+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")
+
+ store.close()
+
+
+async def example_distance_metrics() -> None:
+ """The example of different distance metrics."""
+ print("\n" + "=" * 60)
+ print("Test 3: Different Distance Metrics")
+ print("=" * 60)
+
+ # Test with different metrics
+ # Note: AlibabaCloud MySQL only supports COSINE and EUCLIDEAN
+ metrics = ["COSINE", "EUCLIDEAN"]
+
+ for metric in metrics:
+ print(f"\n--- Testing {metric} metric ---")
+ store = AlibabaCloudMySQLStore(
+ host="rm-xxxxx.mysql.rds.aliyuncs.com",
+ port=3306,
+ user="your_username",
+ password="your_password",
+ database="agentscope_test",
+ table_name=f"{metric.lower()}_vectors",
+ dimensions=4,
+ distance=metric,
+ )
+
+ docs = [
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text=f"Test doc for {metric}"),
+ doc_id=f"doc_{metric}_1",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.1, 0.2, 0.3, 0.4],
+ ),
+ ]
+
+ await store.add(docs)
+ results = await store.search(
+ query_embedding=[0.1, 0.2, 0.3, 0.4],
+ limit=1,
+ )
+
+ print(f"✓ {metric} metric: Score = {results[0].score:.4f}")
+ store.close()
+
+
+async def main() -> None:
+ """Run all examples."""
+ print("\n" + "=" * 60)
+ print("AlibabaCloud MySQL Vector Store Test Suite")
+ print("=" * 60)
+
+ try:
+ await example_basic_operations()
+ await example_filter_search()
+ await example_distance_metrics()
+
+ print("\n" + "=" * 60)
+ print("✓ All tests completed successfully!")
+ print("=" * 60)
+
+ except Exception as e:
+ print(f"\n✗ Test failed with error: {e}")
+ import traceback
+
+ traceback.print_exc()
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/pyproject.toml b/pyproject.toml
index 4a351bd23..104b45a74 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -69,6 +69,8 @@ full = [
# The qdrant-client >= 1.16.0 has conflicts with pymilvus, so we fix
# the version to 1.15.1 here.
"qdrant-client==1.15.1",
+ "mysql-connector-python",
+
]
dev = [
diff --git a/src/agentscope/rag/__init__.py b/src/agentscope/rag/__init__.py
index 66f82489e..146b35aa6 100644
--- a/src/agentscope/rag/__init__.py
+++ b/src/agentscope/rag/__init__.py
@@ -16,6 +16,7 @@ from ._store import (
VDBStoreBase,
QdrantStore,
MilvusLiteStore,
+ AlibabaCloudMySQLStore,
)
from ._knowledge_base import KnowledgeBase
from ._simple_knowledge import SimpleKnowledge
@@ -32,6 +33,7 @@ __all__ = [
"VDBStoreBase",
"QdrantStore",
"MilvusLiteStore",
+ "AlibabaCloudMySQLStore",
"KnowledgeBase",
"SimpleKnowledge",
]
diff --git a/src/agentscope/rag/_store/__init__.py b/src/agentscope/rag/_store/__init__.py
index ceebe517e..c107eae5d 100644
--- a/src/agentscope/rag/_store/__init__.py
+++ b/src/agentscope/rag/_store/__init__.py
@@ -6,9 +6,11 @@ from ._store_base import (
)
from ._qdrant_store import QdrantStore
from ._milvuslite_store import MilvusLiteStore
+from ._alibabacloud_mysql_store import AlibabaCloudMySQLStore
__all__ = [
"VDBStoreBase",
"QdrantStore",
"MilvusLiteStore",
+ "AlibabaCloudMySQLStore",
]
diff --git a/src/agentscope/rag/_store/_alibabacloud_mysql_store.py b/src/agentscope/rag/_store/_alibabacloud_mysql_store.py
new file mode 100644
index 000000000..d62cab0a6
--- /dev/null
+++ b/src/agentscope/rag/_store/_alibabacloud_mysql_store.py
@@ -0,0 +1,402 @@
+# -*- coding: utf-8 -*-
+"""The AlibabaCloud MySQL vector store implementation."""
+import json
+from typing import Any, Literal, TYPE_CHECKING
+
+from .._reader import Document
+from ._store_base import VDBStoreBase
+from .._document import DocMetadata
+
+from ..._utils._common import _map_text_to_uuid
+from ...types import Embedding
+
+if TYPE_CHECKING:
+ from mysql.connector import MySQLConnection
+else:
+ MySQLConnection = "mysql.connector.MySQLConnection"
+
+
+class AlibabaCloudMySQLStore(VDBStoreBase):
+ """The AlibabaCloud MySQL vector store implementation, supporting vector
+ search operations using MySQL's native vector functions.
+
+ .. note:: AlibabaCloud MySQL vector search requires MySQL 8.0+.
+ This implementation uses MySQL's native vector functions
+ (VEC_DISTANCE_COSINE, VEC_DISTANCE_EUCLIDEAN, VEC_FROMTEXT) for
+ efficient vector similarity search with ORDER BY in SQL.
+ Only COSINE and EUCLIDEAN distance metrics are supported.
+
+ .. note:: Requires mysql-connector-python package. Install with:
+ `pip install mysql-connector-python`
+
+ .. note:: For AlibabaCloud MySQL instances, ensure vector search plugin
+ is enabled. Contact AlibabaCloud support if needed.
+
+ """
+
+ def __init__(
+ self,
+ host: str,
+ port: int,
+ user: str,
+ password: str,
+ database: str,
+ table_name: str,
+ dimensions: int,
+ distance: Literal["COSINE", "EUCLIDEAN"] = "COSINE",
+ hnsw_m: int = 16,
+ connection_kwargs: dict[str, Any] | None = None,
+ ) -> None:
+ """Initialize the AlibabaCloud MySQL vector store.
+
+ Args:
+ host (`str`):
+ The hostname of the AlibabaCloud MySQL server.
+ Example: "rm-xxxxx.mysql.rds.aliyuncs.com"
+ port (`int`):
+ The port number of the MySQL server (typically 3306).
+ user (`str`):
+ The username for authentication.
+ password (`str`):
+ The password for authentication.
+ database (`str`):
+ The database name to use.
+ table_name (`str`):
+ The name of the table to store the embeddings.
+ dimensions (`int`):
+ The dimension of the embeddings.
+ distance (`Literal["COSINE", "EUCLIDEAN"]`, default to "COSINE"):
+ The distance metric to use for similarity search. Can be
+ one of "COSINE" (cosine similarity) or "EUCLIDEAN"
+ (Euclidean distance). Defaults to "COSINE".
+ hnsw_m (`int`, default to 16):
+ The M parameter for HNSW vector index, which controls
+ the number of bi-directional links created for each node
+ during construction. Higher values create denser graphs
+ with better recall but use more memory. Typical values
+ range from 4 to 64. Defaults to 16.
+ connection_kwargs (`dict[str, Any] | None`, optional):
+ Other keyword arguments for the MySQL connector.
+ Example: {"ssl_ca": "/path/to/ca.pem", "charset": "utf8mb4"}
+ """
+
+ try:
+ import mysql.connector
+ except ImportError as e:
+ raise ImportError(
+ "MySQL connector is not installed. Please install it with "
+ "`pip install mysql-connector-python`.",
+ ) from e
+
+ connection_kwargs = connection_kwargs or {}
+
+ # Initialize connection parameters
+ self.connection_params = {
+ "host": host,
+ "port": port,
+ "user": user,
+ "password": password,
+ "database": database,
+ **connection_kwargs,
+ }
+
+ self.table_name = table_name
+ self.dimensions = dimensions
+ self.distance = distance
+ self.hnsw_m = hnsw_m
+
+ # Initialize connection
+ self._conn = mysql.connector.connect(**self.connection_params)
+ self._cursor = self._conn.cursor(dictionary=True)
+
+ def _get_distance_function(self) -> str:
+ """Get the MySQL native vector distance function name based on the
+ distance metric.
+
+ Returns:
+ `str`:
+ The SQL vector distance function name.
+ """
+ if self.distance == "COSINE":
+ return "VEC_DISTANCE_COSINE"
+ elif self.distance == "EUCLIDEAN":
+ return "VEC_DISTANCE_EUCLIDEAN"
+ else:
+ raise ValueError(
+ f"Unsupported distance metric: {self.distance}. "
+ f"AlibabaCloud MySQL only supports 'COSINE' and 'EUCLIDEAN'.",
+ )
+
+ def _format_vector_for_sql(self, vector: list[float]) -> str:
+ """Format a vector as a string for MySQL VEC_FROMTEXT function.
+ Args:
+ vector (`list[float]`):
+ The vector to format.
+ Returns:
+ `str`:
+ The formatted vector string like "[1,2,3,4]".
+ """
+ return "[" + ",".join(map(str, vector)) + "]"
+
+ async def _validate_table(self) -> None:
+ """Validate the table exists, if not, create it.
+ Creates a table with VECTOR type columns and automatically
+ creates a vector index based on the specified distance metric
+ using HNSW algorithm.
+ """
+ # Get distance metric in lowercase for SQL
+ distance_metric = self.distance.lower()
+
+ # Create table with VECTOR INDEX in a single statement
+ # VECTOR(dimensions) type is available in AlibabaCloud MySQL 8.0+
+ # VECTOR INDEX uses HNSW algorithm with M parameter for
+ # graph connectivity
+ # IF NOT EXISTS prevents errors if table already exists
+ create_table_sql = f"""
+ CREATE TABLE IF NOT EXISTS {self.table_name} (
+ id VARCHAR(255) PRIMARY KEY,
+ embedding VECTOR({self.dimensions}) NOT NULL,
+ doc_id VARCHAR(255) NOT NULL,
+ chunk_id INT NOT NULL,
+ content TEXT NOT NULL,
+ total_chunks INT NOT NULL,
+ INDEX idx_doc_id (doc_id),
+ INDEX idx_chunk_id (chunk_id),
+ VECTOR INDEX (embedding) M={self.hnsw_m} DISTANCE={distance_metric}
+ ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
+ """
+ self._cursor.execute(create_table_sql)
+ self._conn.commit()
+
+ async def add(self, documents: list[Document], **kwargs: Any) -> None:
+ """Add embeddings to the AlibabaCloud MySQL vector store.
+
+ Args:
+ documents (`list[Document]`):
+ A list of embedding records to be recorded in the MySQL store.
+ **kwargs (`Any`):
+ Additional arguments for the insert operation.
+ """
+ await self._validate_table()
+
+ # Prepare data for insertion
+ for doc in documents:
+ # Generate a unique ID
+ unique_string = json.dumps(
+ {
+ "doc_id": doc.metadata.doc_id,
+ "chunk_id": doc.metadata.chunk_id,
+ "content": doc.metadata.content,
+ },
+ ensure_ascii=False,
+ )
+ unique_id = _map_text_to_uuid(unique_string)
+
+ # Format vector for MySQL VEC_FROMTEXT
+ if doc.embedding is None:
+ raise ValueError(
+ f"Document embedding cannot be None for doc_id: "
+ f"{doc.metadata.doc_id}",
+ )
+ vector_text = self._format_vector_for_sql(doc.embedding)
+
+ # Insert data using VEC_FROMTEXT to convert text to vector
+ insert_sql = f"""
+ INSERT INTO {self.table_name}
+ (id, embedding, doc_id, chunk_id, content, total_chunks)
+ VALUES (%s, VEC_FROMTEXT(%s), %s, %s, %s, %s)
+ ON DUPLICATE KEY UPDATE
+ embedding = VEC_FROMTEXT(%s),
+ doc_id = VALUES(doc_id),
+ chunk_id = VALUES(chunk_id),
+ content = VALUES(content),
+ total_chunks = VALUES(total_chunks)
+ """
+
+ # Serialize content to JSON if it's not a string
+ content_str = (
+ doc.metadata.content
+ if isinstance(doc.metadata.content, str)
+ else json.dumps(doc.metadata.content, ensure_ascii=False)
+ )
+
+ self._cursor.execute(
+ insert_sql,
+ (
+ unique_id,
+ vector_text,
+ doc.metadata.doc_id,
+ doc.metadata.chunk_id,
+ content_str,
+ doc.metadata.total_chunks,
+ vector_text, # For ON DUPLICATE KEY UPDATE
+ ),
+ )
+
+ self._conn.commit()
+
+ async def search(
+ self,
+ query_embedding: Embedding,
+ limit: int,
+ score_threshold: float | None = None,
+ **kwargs: Any,
+ ) -> list[Document]:
+ """Search relevant documents from the AlibabaCloud MySQL vector store.
+
+ Args:
+ query_embedding (`Embedding`):
+ The embedding of the query text.
+ limit (`int`):
+ The number of relevant documents to retrieve.
+ score_threshold (`float | None`, optional):
+ The minimum similarity score threshold to filter the
+ results. Score is calculated as 1 - distance, where
+ higher scores indicate higher similarity. Only documents
+ with score >= score_threshold will be returned.
+ **kwargs (`Any`):
+ Additional arguments for the search operation.
+ - filter (`str`): WHERE clause to filter the search results.
+ """
+
+ # Format query vector for MySQL VEC_FROMTEXT
+ query_vector_text = self._format_vector_for_sql(query_embedding)
+
+ # Get the distance function
+ distance_func = self._get_distance_function()
+
+ # Build WHERE clause
+ where_conditions = []
+ if "filter" in kwargs and kwargs["filter"]:
+ where_conditions.append(kwargs["filter"])
+
+ # Add score threshold condition if specified
+ # Score is calculated as 1 - distance, so higher scores
+ # indicate higher similarity
+ # To filter by score_threshold, we need:
+ # 1.0 - distance >= score_threshold
+ if score_threshold is not None:
+ where_conditions.append(
+ f"{distance_func}(embedding, VEC_FROMTEXT(%s)) <= %s",
+ )
+
+ where_clause = ""
+ if where_conditions:
+ where_clause = "WHERE " + " AND ".join(where_conditions)
+
+ # Build and execute the search query using MySQL native vector
+ # functions with ORDER BY for efficient sorting
+ search_sql = f"""
+ SELECT
+ id,
+ doc_id,
+ chunk_id,
+ content,
+ total_chunks,
+ {distance_func}(embedding, VEC_FROMTEXT(%s)) as distance
+ FROM {self.table_name}
+ {where_clause}
+ ORDER BY distance ASC
+ LIMIT %s
+ """
+
+ # Prepare parameters
+ params: list[str | float | int] = [query_vector_text]
+ if score_threshold is not None:
+ # Convert score threshold to distance threshold:
+ # distance <= 1.0 - score
+ params.extend([query_vector_text, 1.0 - score_threshold])
+ params.append(limit)
+
+ self._cursor.execute(search_sql, params)
+ results = self._cursor.fetchall()
+
+ # Process results
+ collected_res = []
+ for row in results:
+ # Deserialize content from JSON if it's a JSON string
+ content = row["content"]
+ try:
+ content = json.loads(content)
+ except (json.JSONDecodeError, TypeError):
+ # If it's not valid JSON, keep it as is (plain string)
+ pass
+
+ doc_metadata = DocMetadata(
+ content=content,
+ doc_id=row["doc_id"],
+ chunk_id=row["chunk_id"],
+ total_chunks=row["total_chunks"],
+ )
+
+ # Create Document
+ # Convert distance to score: score = 1 - distance
+ # Higher scores indicate higher similarity
+ collected_res.append(
+ Document(
+ embedding=None, # Vector not returned for efficiency
+ score=1.0 - row["distance"],
+ metadata=doc_metadata,
+ ),
+ )
+
+ return collected_res
+
+ async def delete(
+ self,
+ ids: list[str] | None = None,
+ filter: str | None = None, # pylint: disable=redefined-builtin
+ **kwargs: Any,
+ ) -> None:
+ """Delete documents from the AlibabaCloud MySQL vector store.
+
+ Args:
+ ids (`list[str] | None`, optional):
+ List of entity IDs to delete.
+ filter (`str | None`, optional):
+ WHERE clause expression to filter documents to delete.
+ **kwargs (`Any`):
+ Additional arguments for the delete operation.
+ """
+ if ids is None and filter is None:
+ raise ValueError(
+ "Either ids or filter must be provided for deletion.",
+ )
+
+ if ids is not None:
+ # Delete by IDs
+ placeholders = ",".join(["%s"] * len(ids))
+ delete_sql = (
+ f"DELETE FROM {self.table_name} WHERE id IN ({placeholders})"
+ )
+ self._cursor.execute(delete_sql, ids)
+ elif filter is not None:
+ # Delete by filter
+ delete_sql = f"DELETE FROM {self.table_name} WHERE {filter}"
+ self._cursor.execute(delete_sql)
+
+ self._conn.commit()
+
+ def get_client(self) -> MySQLConnection:
+ """Get the underlying MySQL connection, so that developers can access
+ the full functionality of AlibabaCloud MySQL.
+
+ Returns:
+ `MySQLConnection`:
+ The underlying MySQL connection.
+ """
+ return self._conn
+
+ def close(self) -> None:
+ """Close the database connection."""
+ if self._cursor:
+ self._cursor.close()
+ if self._conn:
+ self._conn.close()
+
+ def __del__(self) -> None:
+ """Destructor to ensure connection is closed."""
+ try:
+ self.close()
+ except Exception: # pylint: disable=broad-except
+ pass
diff --git a/tests/rag_store_test.py b/tests/rag_store_test.py
index afbd08372..485dd2382 100644
--- a/tests/rag_store_test.py
+++ b/tests/rag_store_test.py
@@ -2,6 +2,7 @@
"""Test the RAG store implementations."""
import os
from unittest import IsolatedAsyncioTestCase
+from unittest.mock import MagicMock, patch
from agentscope.message import TextBlock
from agentscope.rag import (
@@ -9,6 +10,7 @@ from agentscope.rag import (
Document,
DocMetadata,
MilvusLiteStore,
+ AlibabaCloudMySQLStore,
)
@@ -128,6 +130,135 @@ class RAGStoreTest(IsolatedAsyncioTestCase):
"This is a test document.",
)
+ async def test_alibabacloud_mysql_store(self) -> None:
+ """Test the AlibabaCloudMySQLStore implementation using mocks."""
+ # Create mock MySQL module and connector
+ mock_mysql_connector = MagicMock()
+ mock_mysql = MagicMock()
+ mock_mysql.connector = mock_mysql_connector
+
+ # Create mock cursor and connection
+ mock_cursor = MagicMock()
+ mock_conn = MagicMock()
+
+ # Configure mock connection to return mock cursor
+ mock_conn.cursor.return_value = mock_cursor
+ mock_mysql_connector.connect.return_value = mock_conn
+
+ # Mock the search query result
+ # Simulate a database row returned by fetchall
+ mock_search_result = [
+ {
+ "id": "test-uuid-1",
+ "doc_id": "doc1",
+ "chunk_id": 0,
+ "content": (
+ '{"type": "text", "text": "This is a test document."}'
+ ),
+ "total_chunks": 2,
+ "distance": 0.03, # Low distance = high similarity
+ },
+ ]
+
+ # Use patch.dict to mock sys.modules
+ with patch.dict(
+ "sys.modules",
+ {
+ "mysql": mock_mysql,
+ "mysql.connector": mock_mysql_connector,
+ },
+ ):
+ # Create store instance
+ store = AlibabaCloudMySQLStore(
+ host="test-host",
+ port=3306,
+ user="test-user",
+ password="test-password",
+ database="test-database",
+ table_name="test_vectors",
+ dimensions=3,
+ )
+
+ # Verify connection was established
+ mock_mysql_connector.connect.assert_called_once()
+
+ # Test add operation
+ await store.add(
+ [
+ Document(
+ embedding=[0.1, 0.2, 0.3],
+ metadata=DocMetadata(
+ content=TextBlock(
+ type="text",
+ text="This is a test document.",
+ ),
+ doc_id="doc1",
+ chunk_id=0,
+ total_chunks=2,
+ ),
+ ),
+ Document(
+ embedding=[2.0, 3.8, 2.7],
+ metadata=DocMetadata(
+ content=TextBlock(
+ type="text",
+ text="This is another test document.",
+ ),
+ doc_id="doc1",
+ chunk_id=1,
+ total_chunks=2,
+ ),
+ ),
+ ],
+ )
+
+ # Verify add operations executed SQL
+ self.assertTrue(mock_cursor.execute.called)
+ self.assertTrue(mock_conn.commit.called)
+
+ # Reset mock for search operation
+ mock_cursor.reset_mock()
+ mock_conn.reset_mock()
+
+ # Configure mock to return search results
+ mock_cursor.fetchall.return_value = mock_search_result
+
+ # Test search operation
+ res = await store.search(
+ query_embedding=[0.15, 0.25, 0.35],
+ limit=3,
+ score_threshold=0.95,
+ )
+
+ # Verify search results
+ self.assertEqual(len(res), 1)
+ # Score = 1 - distance = 1 - 0.03 = 0.97
+ self.assertAlmostEqual(res[0].score, 0.97, places=2)
+ self.assertEqual(
+ res[0].metadata.content["text"],
+ "This is a test document.",
+ )
+ self.assertEqual(res[0].metadata.doc_id, "doc1")
+ self.assertEqual(res[0].metadata.chunk_id, 0)
+ self.assertEqual(res[0].metadata.total_chunks, 2)
+
+ # Verify search executed SQL query
+ self.assertTrue(mock_cursor.execute.called)
+ self.assertTrue(mock_cursor.fetchall.called)
+
+ # Test delete operation
+ await store.delete(filter='doc_id = "doc1"')
+
+ # Verify delete executed SQL
+ self.assertTrue(mock_conn.commit.called)
+
+ # Test close
+ store.close()
+
+ # Verify connections were closed
+ mock_cursor.close.assert_called()
+ mock_conn.close.assert_called()
+
async def asyncTearDown(self) -> None:
"""Clean up after tests."""
# Remove Milvus Lite database file