Build and run agents you can see, understand and trust.
Python agentchatbotlarge-language-modelsllmllm-agentmcpmulti-agentmulti-modalreact-agentrealtime-agentvoice-agentvoice-assistant
feat(rag): add MongoDB vector store in the RAG module. (#860)
qbc3d4ffb2890cb
diff --git a/examples/functionality/vector_store/mongodb/README.md b/examples/functionality/vector_store/mongodb/README.mdnew file mode 100644index 000000000..3e2dd6473--- /dev/null+++ b/examples/functionality/vector_store/mongodb/README.md@@ -0,0 +1,211 @@+# MongoDB Vector Store++This example demonstrates how to use **MongoDBStore** for vector storage and semantic search in AgentScope using MongoDB's Vector Search capabilities.+It includes comprehensive test scenarios covering CRUD operations, metadata filtering, document chunking, and distance metrics.++### Quick Start++Install agentscope first, and then the MongoDB dependency:++```bash+pip install pymongo+```++**Important:** Before running the example, you need to set the `MONGODB_HOST`+environment variable with your MongoDB connection string:++```bash+# For local MongoDB+export MONGODB_HOST="mongodb://localhost:27017/?directConnection=true"++# For MongoDB Atlas (replace with your connection string)+# export MONGODB_HOST=${YOUR_MONGODB_HOST}+```++Run the example script, which showcases adding, searching, and deleting in MongoDB vector store:++```bash+python main.py+```++> **Note:** The script connects to MongoDB Atlas or local MongoDB instance. Make sure you have a valid MongoDB connection string.++## Prerequisites++- Confirm your MongoDB instance supports Vector Search functionality+- Valid MongoDB connection string (local or Atlas)++## Usage++### Initialize Store++```python+from agentscope.rag import MongoDBStore++# For MongoDB Atlas+store = MongoDBStore(+ host="mongodb+srv://username:[email protected]/",+ db_name="test_db",+ collection_name="test_collection",+ dimensions=768, # Match your embedding model+ distance="cosine", # cosine, euclidean, or dotProduct+)++# For local MongoDB+store = MongoDBStore(+ host="mongodb://localhost:27017/?directConnection=true",+ db_name="test_db",+ collection_name="test_collection",+ dimensions=768,+ distance="cosine",+)++# To enable filtering in search, specify filter_fields:+store = MongoDBStore(+ host="mongodb://localhost:27017/?directConnection=true",+ db_name="test_db",+ collection_name="test_collection",+ dimensions=768,+ distance="cosine",+ filter_fields=["payload.doc_id", "payload.chunk_id"], # Fields for filtering+)++# No manual initialization needed - everything is automatic!+# Database, collection, and vector search index are created automatically+# when you first call add() or search()+```++### Add Documents++```python+from agentscope.rag import Document, DocMetadata+from agentscope.message import TextBlock++doc = Document(+ metadata=DocMetadata(+ content=TextBlock(type="text", text="Your document text"),+ doc_id="doc_1",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.1, 0.2, ...], # Your embedding vector+)++await store.add([doc])+```++### Search++```python+results = await store.search(+ query_embedding=[0.15, 0.25, ...],+ limit=5,+ score_threshold=0.9, # Optional+ filter={"payload.doc_id": {"$in": ["doc_1", "doc_2"]}}, # Optional filter+)+# Note:+# - To use filter, the field must be declared in filter_fields when creating store+# - MongoDB $vectorSearch filter supports: $gt, $gte, $lt, $lte,+# $eq, $ne, $in, $nin, $exists, $not (NOT $regex)+```++### Delete++```python+# Delete by document IDs (no initialization needed)+await store.delete(ids=["doc_1", "doc_2"])++# Delete entire collection (use with caution)+await store.delete_collection()++# Delete entire database (use with caution)+await store.delete_database()+```++## Distance Metrics++| Metric | Description | Best For |+|--------|-------------|----------|+| **cosine** | Cosine similarity | Text embeddings (recommended) |+| **euclidean** | Euclidean distance | Spatial data |+| **dotProduct** | Inner Product | Recommendation systems |++## Advanced Usage++### Access Underlying Client++```python+client = store.get_client()+# Use MongoDB client for advanced operations+stats = await client[store.db_name].command("collStats", store.collection_name)+```++### Document Metadata++- `content`: Text content (TextBlock)+- `doc_id`: Unique document identifier+- `chunk_id`: Chunk position (0-indexed)+- `total_chunks`: Total chunks in document++### Vector Search Index++MongoDBStore automatically creates vector search indexes with the following configuration:++```python+{+ "fields": [+ {+ "type": "vector",+ "path": "vector",+ "similarity": "cosine", # or euclidean, dotProduct+ "numDimensions": 768+ }+ ]+}+```++## Connection Examples++### MongoDB Atlas++```python+store = MongoDBStore(+ host="<YOUR_MONGO_ATLAS_CONNECTION_STRING>",+ db_name="production_db",+ collection_name="documents",+ dimensions=1536,+ distance="cosine",+)+```++### Local MongoDB++#### Without Authentication++```python+store = MongoDBStore(+ host="mongodb://localhost:27017?directConnection=true",+ db_name="local_db",+ collection_name="test_collection",+ dimensions=768,+ distance="cosine",+)+```++#### With Authentication++```python+store = MongoDBStore(+ host="mongodb://user:pass@localhost:27017/?directConnection=true",+ db_name="test_db",+ collection_name="test_collection",+ dimensions=768,+ distance="cosine",+)+```++## References++- [MongoDB Vector Search Documentation](https://www.mongodb.com/docs/atlas/atlas-search/vector-search/)+- [MongoDB Atlas Documentation](https://www.mongodb.com/docs/atlas/)+- [AgentScope RAG Tutorial](https://doc.agentscope.io/tutorial/task_rag.html)diff --git a/examples/functionality/vector_store/mongodb/main.py b/examples/functionality/vector_store/mongodb/main.pynew file mode 100644index 000000000..0212a6693--- /dev/null+++ b/examples/functionality/vector_store/mongodb/main.py@@ -0,0 +1,351 @@+# -*- coding: utf-8 -*-+"""Example of using MongoDBStore in AgentScope RAG system."""+import asyncio+import os++from agentscope.rag import (+ MongoDBStore,+ Document,+ DocMetadata,+)+from agentscope.message import TextBlock+++async def example_basic_operations() -> None:+ """The example of basic CRUD operations with MongoDBStore."""+ print("\n" + "=" * 60)+ print("Test 1: Basic CRUD Operations")+ print("=" * 60)++ # Initialize MongoDBStore with MongoDB connection+ store = MongoDBStore(+ host=os.getenv("MONGODB_HOST"),+ db_name="test_db",+ collection_name="test_collection",+ dimensions=4, # Small dimension for testing+ distance="cosine",+ )++ print("✓ MongoDBStore initialized")++ # Create test documents with embeddings+ test_docs = [+ Document(+ metadata=DocMetadata(+ content=TextBlock(+ text="Artificial Intelligence is the future",+ ),+ doc_id="doc_1",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.1, 0.2, 0.3, 0.4],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Machine Learning is a subset of AI"),+ doc_id="doc_2",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.2, 0.3, 0.4, 0.5],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Deep Learning uses neural networks"),+ doc_id="doc_3",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.3, 0.4, 0.5, 0.6],+ ),+ ]++ # Test add operation (automatically creates database, collection,+ # and index)+ await store.add(test_docs)+ print(f"✓ Added {len(test_docs)} documents to the store")++ # Test search operation (automatically waits for index to be ready)+ query_embedding = [0.15, 0.25, 0.35, 0.45]+ results = await store.search(+ query_embedding=query_embedding,+ limit=2,+ )++ print(f"\n✓ Search completed, found {len(results)} results:")+ for i, result in enumerate(results, 1):+ print(f" {i}. Score: {result.score:.4f}")+ print(f" Content: {result.metadata.content}")+ print(f" Doc ID: {result.metadata.doc_id}")++ # Test search with score threshold (also waits for index if needed)+ results_filtered = await store.search(+ query_embedding=query_embedding,+ limit=5,+ score_threshold=0.3,+ )+ print(f"\n✓ Search with threshold (>0.3): {len(results_filtered)} results")++ # Test delete operation (no initialization needed)+ # Note: MongoDBStore uses ids parameter for deletion+ await store.delete(ids=["doc_2", "doc_3", "doc_1"])+ print("\n✓ Deleted documents with specified doc_ids")++ # Verify deletion (search will wait for index if needed)+ results_after_delete = await store.search(+ query_embedding=query_embedding,+ limit=5,+ )+ print(f"✓ After deletion: {len(results_after_delete)} documents remain")++ # Get client for advanced operations+ client = store.get_client()+ print(f"\n✓ Got MongoDB Client: {type(client).__name__}")++ await store.close()+++async def example_filter_search() -> None:+ """The example of search with metadata filtering."""+ print("\n" + "=" * 60)+ print("Test 2: Search with Metadata Filtering")+ print("=" * 60)++ # To use filter in search, specify filter_fields when creating the store.+ # These fields will be indexed for filtering in $vectorSearch.+ store = MongoDBStore(+ host=os.getenv("MONGODB_HOST"),+ db_name="filter_test_db",+ collection_name="filter_collection",+ dimensions=4,+ distance="cosine",+ filter_fields=["payload.doc_id"], # Enable filtering on doc_id+ )++ # Create documents with different categories+ docs = [+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Python is a programming language"),+ doc_id="prog_1",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.1, 0.2, 0.3, 0.4],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(+ text="Java is used for enterprise applications",+ ),+ doc_id="prog_2",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.2, 0.3, 0.4, 0.5],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Neural networks are used in AI"),+ doc_id="ai_1",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.3, 0.4, 0.5, 0.6],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Deep learning requires GPUs"),+ doc_id="ai_2",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.4, 0.5, 0.6, 0.7],+ ),+ ]++ # Add documents (automatically creates database, collection, and index)+ await store.add(docs)+ print(f"✓ Added {len(docs)} documents with different doc_id prefixes")++ # Search without filter (automatically waits for index if needed)+ query_embedding = [0.25, 0.35, 0.45, 0.55]+ all_results = await store.search(+ query_embedding=query_embedding,+ limit=4,+ )+ print(f"\n✓ Search without filter: {len(all_results)} results")+ for i, result in enumerate(all_results, 1):+ doc_id = result.metadata.doc_id+ score = result.score+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")++ # Search with filter for programming docs+ # Note: doc_id is stored in payload.doc_id in MongoDB documents+ # MongoDB $vectorSearch filter supports: $gt, $gte, $lt, $lte, $eq, $ne,+ # $in, $nin, $exists, $not (NOT $regex)+ prog_results = await store.search(+ query_embedding=query_embedding,+ limit=4,+ filter={"payload.doc_id": {"$in": ["prog_1", "prog_2"]}},+ )+ print(f"\n✓ Search with filter (prog docs): {len(prog_results)} results")+ for i, result in enumerate(prog_results, 1):+ doc_id = result.metadata.doc_id+ score = result.score+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")++ # Search with filter for AI docs+ ai_results = await store.search(+ query_embedding=query_embedding,+ limit=4,+ filter={"payload.doc_id": {"$in": ["ai_1", "ai_2"]}},+ )+ print(f"\n✓ Search with filter (ai docs): {len(ai_results)} results")+ for i, result in enumerate(ai_results, 1):+ doc_id = result.metadata.doc_id+ score = result.score+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")++ await store.close()+++async def example_multiple_chunks() -> None:+ """The example of documents with multiple chunks."""+ print("\n" + "=" * 60)+ print("Test 3: Documents with Multiple Chunks")+ print("=" * 60)++ store = MongoDBStore(+ host=os.getenv("MONGODB_HOST"),+ db_name="chunks_test_db",+ collection_name="chunks_collection",+ dimensions=4,+ distance="cosine",+ )++ # Create a document split into multiple chunks+ chunks = [+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Chapter 1: Introduction to AI"),+ doc_id="book_1",+ chunk_id=0,+ total_chunks=3,+ ),+ embedding=[0.1, 0.2, 0.3, 0.4],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Chapter 2: Machine Learning Basics"),+ doc_id="book_1",+ chunk_id=1,+ total_chunks=3,+ ),+ embedding=[0.2, 0.3, 0.4, 0.5],+ ),+ Document(+ metadata=DocMetadata(+ content=TextBlock(text="Chapter 3: Deep Learning Advanced"),+ doc_id="book_1",+ chunk_id=2,+ total_chunks=3,+ ),+ embedding=[0.3, 0.4, 0.5, 0.6],+ ),+ ]++ # Add chunks (automatically creates database, collection, and index)+ await store.add(chunks)+ print(f"✓ Added document with {len(chunks)} chunks")++ # Search and verify chunk information (automatically waits for index if+ # needed)+ query_embedding = [0.2, 0.3, 0.4, 0.5]+ results = await store.search(+ query_embedding=query_embedding,+ limit=3,+ )++ print("\n✓ Search results for multi-chunk document:")+ for i, result in enumerate(results, 1):+ chunk_info = (+ f"{result.metadata.chunk_id}/{result.metadata.total_chunks}"+ )+ print(f" {i}. Chunk {chunk_info}")+ print(f" Content: {result.metadata.content}")+ print(f" Score: {result.score:.4f}")++ await store.close()+++async def example_distance_metrics() -> None:+ """The example of different distance metrics."""+ print("\n" + "=" * 60)+ print("Test 4: Different Distance Metrics")+ print("=" * 60)++ # Test with different metrics+ metrics = ["cosine", "euclidean", "dotProduct"]++ for metric in metrics:+ print(f"\n--- Testing {metric} metric ---")+ store = MongoDBStore(+ host=os.getenv("MONGODB_HOST"),+ db_name=f"{metric}_test_db",+ collection_name=f"{metric}_collection",+ dimensions=4,+ distance=metric,+ )++ docs = [+ Document(+ metadata=DocMetadata(+ content=TextBlock(text=f"Test doc for {metric}"),+ doc_id=f"doc_{metric}_1",+ chunk_id=0,+ total_chunks=1,+ ),+ embedding=[0.1, 0.2, 0.3, 0.4],+ ),+ ]++ # Add and search (automatically creates database/collection/index+ # and waits for index)+ await store.add(docs)+ results = await store.search(+ query_embedding=[0.1, 0.2, 0.3, 0.4],+ limit=1,+ )++ print(f"✓ {metric} metric: Score = {results[0].score:.4f}")++ await store.close()+++async def main() -> None:+ """Run all example."""+ print("\n" + "=" * 60)+ print("MongoDBStore Comprehensive Test Suite")+ print("=" * 60)++ try:+ # await example_basic_operations()+ # await example_filter_search()+ # await example_multiple_chunks()+ await example_distance_metrics()++ print("\n" + "=" * 60)+ print("✓ All tests completed successfully!")+ print("=" * 60)++ except Exception as e:+ print(f"\n✗ Test failed with error: {e}")+ import traceback++ traceback.print_exc()+++if __name__ == "__main__":+ asyncio.run(main())diff --git a/pyproject.toml b/pyproject.tomlindex 52f32d72f..d13147435 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -106,10 +106,12 @@ readers = [qdrant = ["qdrant-client==1.15.1"]milvus = ["pymilvus[milvus_lite]"]ali_mysql = ["mysql-connector-python"]+mongodb = ["pymongo"]vdbs = ["agentscope[ali_mysql]","agentscope[qdrant]","agentscope[milvus]",+ "agentscope[mongodb]",]diff --git a/src/agentscope/rag/__init__.py b/src/agentscope/rag/__init__.pyindex 146b35aa6..3dfe828ef 100644--- a/src/agentscope/rag/__init__.py+++ b/src/agentscope/rag/__init__.py@@ -16,6 +16,7 @@ from ._store import (VDBStoreBase,QdrantStore,MilvusLiteStore,+ MongoDBStore,AlibabaCloudMySQLStore,)from ._knowledge_base import KnowledgeBase@@ -33,6 +34,7 @@ __all__ = ["VDBStoreBase","QdrantStore","MilvusLiteStore",+ "MongoDBStore","AlibabaCloudMySQLStore","KnowledgeBase","SimpleKnowledge",diff --git a/src/agentscope/rag/_store/__init__.py b/src/agentscope/rag/_store/__init__.pyindex c107eae5d..96e0ce323 100644--- a/src/agentscope/rag/_store/__init__.py+++ b/src/agentscope/rag/_store/__init__.py@@ -6,11 +6,13 @@ from ._store_base import ()from ._qdrant_store import QdrantStorefrom ._milvuslite_store import MilvusLiteStore+from ._mongodb_store import MongoDBStorefrom ._alibabacloud_mysql_store import AlibabaCloudMySQLStore__all__ = ["VDBStoreBase","QdrantStore","MilvusLiteStore",+ "MongoDBStore","AlibabaCloudMySQLStore",]diff --git a/src/agentscope/rag/_store/_mongodb_store.py b/src/agentscope/rag/_store/_mongodb_store.pynew file mode 100644index 000000000..8da6e46f1--- /dev/null+++ b/src/agentscope/rag/_store/_mongodb_store.py@@ -0,0 +1,392 @@+# -*- coding: utf-8 -*-+"""The MongoDB vector store implementation using MongoDB Vector Search.++This implementation provides a vector database store using MongoDB's vector+ search capabilities. It requires MongoDB with vector search support and+ automatically creates vector search indexes.+"""+import asyncio+import time++from typing import Any, Literal, TYPE_CHECKING++from .._reader import Document+from ._store_base import VDBStoreBase+from .._document import DocMetadata+from ...types import Embedding++if TYPE_CHECKING:+ from pymongo import AsyncMongoClient+else:+ AsyncMongoClient = "pymongo.AsyncMongoClient"+++class MongoDBStore(VDBStoreBase):+ """MongoDB vector store using MongoDB Vector Search.++ This class provides a vector database store implementation using MongoDB's+ vector search capabilities. It requires MongoDB with vector search support+ and creates vector search indexes automatically.++ .. note:: Ensure your MongoDB instance supports Vector Search+ functionality.++ .. note:: The store automatically creates database, collection, and vector+ search index on first operation. No manual initialization is required.+ """++ def __init__(+ self,+ host: str,+ db_name: str,+ collection_name: str,+ dimensions: int,+ index_name: str = "vector_index",+ distance: Literal["cosine", "euclidean", "dotProduct"] = "cosine",+ filter_fields: list[str] | None = None,+ client_kwargs: dict[str, Any] | None = None,+ db_kwargs: dict[str, Any] | None = None,+ collection_kwargs: dict[str, Any] | None = None,+ ) -> None:+ """Initialize the MongoDB vector store.++ Args:+ host (`str`):+ MongoDB connection host, e.g., "mongodb://localhost:27017" or+ "mongodb+srv://cluster.mongodb.net/".+ db_name (`str`):+ Database name to store vector documents.+ collection_name (`str`):+ Collection name to store vector documents.+ dimensions (`int`):+ Embedding dimensions for the vector search index.+ index_name (`str`, defaults to "vector_index"):+ Vector search index name.+ distance (`Literal["cosine", "euclidean", "dotProduct"]`, \+ defaults to "cosine"):+ Distance metric for vector similarity. Can be one of "cosine",+ "euclidean", or "dotProduct".+ filter_fields (`list[str] | None`, optional):+ List of field paths to index for filtering in $vectorSearch.+ For example: ["payload.doc_id", "payload.chunk_id"].+ These fields can then be used in the `filter` parameter of+ the `search` method. MongoDB $vectorSearch filter supports:+ $gt, $gte, $lt, $lte, $eq, $ne, $in, $nin, $exists, $not.+ client_kwargs (`dict[str, Any] | None`, optional):+ Additional kwargs for MongoDB client.+ db_kwargs (`dict[str, Any] | None`, optional):+ Additional kwargs for database.+ collection_kwargs (`dict[str, Any] | None`, optional):+ Additional kwargs for collection.++ Raises:+ ImportError: If pymongo is not installed.+ """+ try:+ from pymongo import AsyncMongoClient+ except ImportError as e:+ raise ImportError(+ "Please install the latest pymongo package to use "+ "AsyncMongoClient: `pip install pymongo`",+ ) from e++ self._client: AsyncMongoClient = AsyncMongoClient(+ host,+ **(client_kwargs or {}),+ )+ self.db_name = db_name+ self.collection_name = collection_name+ self.index_name = index_name+ self.dimensions = dimensions+ self.distance = distance+ self.filter_fields = filter_fields or []+ self.db_kwargs = db_kwargs or {}+ self.collection_kwargs = collection_kwargs or {}++ self._db = None+ self._collection = None++ async def _validate_db_and_collection(self) -> None:+ """Validate the database and collection exist, create if necessary.++ This method ensures the database and collection are available,+ and creates a vector search index for the collection.++ Raises:+ Exception: If database or collection creation fails.+ """+ self._db = self._client.get_database(+ self.db_name,+ **self.db_kwargs,+ )++ if self.collection_name not in await self._db.list_collection_names():+ self._collection = await self._db.create_collection(+ self.collection_name,+ )+ else:+ self._collection = self._db.get_collection(+ self.collection_name,+ **self.collection_kwargs,+ )++ from pymongo.operations import SearchIndexModel++ # Build index fields: vector field + optional filter fields+ index_fields: list[dict[str, Any]] = [+ {+ "type": "vector",+ "path": "vector",+ "similarity": self.distance,+ "numDimensions": self.dimensions,+ },+ ]++ # Add user-specified filter fields+ for field_path in self.filter_fields:+ index_fields.append(+ {+ "type": "filter",+ "path": field_path,+ },+ )++ search_index_model = SearchIndexModel(+ definition={"fields": index_fields},+ name=self.index_name,+ type="vectorSearch",+ )++ await self._collection.create_search_index(+ model=search_index_model,+ )++ async def _wait_for_index_ready(self, timeout: int = 30) -> None:+ """Wait for the vector search index to be ready with timeout+ protection.++ Args:+ timeout (`int`, defaults to 30):+ Maximum time to wait in seconds.++ Raises:+ TimeoutError: If index is not ready within the timeout period.+ """++ start_time = time.time()+ while time.time() - start_time < timeout:+ try:+ indices = []+ async for idx in await self._collection.list_search_indexes(+ self.index_name,+ ):+ indices.append(idx)+ if indices and indices[0].get("queryable") is True:+ return+ except Exception:+ pass++ await asyncio.sleep(0.2)++ raise TimeoutError(f"Index not ready after {timeout} seconds")++ async def add(self, documents: list[Document], **kwargs: Any) -> None:+ """Insert documents with embeddings into MongoDB.++ This method automatically creates the database, collection, and vector+ search index if they don't exist.++ Args:+ documents (`list[Document]`):+ List of Document objects to insert.+ **kwargs (`Any`):+ Additional arguments (unused).++ .. note::+ Each inserted record has structure:++ .. code-block:: python++ {+ "id": str, # Document ID+ "vector": list[float], # Vector embedding+ "payload": dict, # DocMetadata as dict+ }+ """+ await self._validate_db_and_collection()++ # Prepare documents for insertion+ docs_to_insert = []+ for doc in documents:+ # Convert DocMetadata to dict for storage+ payload = {+ "doc_id": doc.metadata.doc_id,+ "chunk_id": doc.metadata.chunk_id,+ "total_chunks": doc.metadata.total_chunks,+ "content": doc.metadata.content,+ }++ # Create document record+ doc_record = {+ "id": f"{doc.metadata.doc_id}_{doc.metadata.chunk_id}",+ "vector": doc.embedding,+ "payload": payload,+ }+ docs_to_insert.append(doc_record)++ # Insert documents using upsert to handle duplicates+ if not docs_to_insert:+ return+ from pymongo import ReplaceOne++ operations = [+ ReplaceOne(+ {"id": doc_record["id"]},+ doc_record,+ upsert=True,+ )+ for doc_record in docs_to_insert+ ]+ await self._collection.bulk_write(operations)++ async def search(+ self,+ query_embedding: Embedding,+ limit: int,+ score_threshold: float | None = None,+ **kwargs: Any,+ ) -> list[Document]:+ """Search relevant documents using MongoDB Vector Search.++ This method uses MongoDB's $vectorSearch aggregation pipeline for+ vector similarity search. It automatically waits for the vector search+ index to be ready before performing the search.++ Args:+ query_embedding (`Embedding`):+ The embedding vector to search for.+ limit (`int`):+ Maximum number of documents to return.+ score_threshold (`float | None`, optional):+ Minimum similarity score threshold. Documents with scores below+ this threshold will be filtered out.+ **kwargs (`Any`):+ Additional arguments for the search operation.++ Returns:+ `list[Document]`: List of Document objects with embedding,+ score, and metadata.++ .. note::+ - Requires MongoDB with vector search support+ - Uses $vectorSearch aggregation pipeline+ """+ await self._validate_db_and_collection()+ # Wait for index to be ready before searching+ await self._wait_for_index_ready()++ # Construct aggregation pipeline for vector search+ # See: https://www.mongodb.com/docs/atlas/atlas-search/vector-search/+ num_candidates = int(+ kwargs.pop(+ "num_candidates",+ max(+ 100,+ limit * 20,+ ),+ ),+ )++ pipeline: list[dict[str, Any]] = [+ {+ "$vectorSearch": {+ "index": self.index_name,+ "path": "vector",+ "queryVector": list(query_embedding),+ "numCandidates": num_candidates,+ "limit": limit,+ **kwargs,+ },+ },+ {+ "$project": {+ "vector": 1,+ "payload": 1,+ "score": {"$meta": "vectorSearchScore"},+ },+ },+ ]++ cursor = await self._collection.aggregate(pipeline)+ results: list[Document] = []+ async for item in cursor:+ score_val = float(item.get("score", 0.0))+ if score_threshold is not None and score_val < score_threshold:+ continue++ payload = item.get("payload", {})+ # Rebuild Document+ metadata = DocMetadata(**payload)++ results.append(+ Document(+ embedding=[float(x) for x in item.get("vector", [])],+ score=score_val,+ metadata=metadata,+ ),+ )++ return results++ async def delete(+ self,+ ids: str | list[str] | None = None,+ ) -> None:+ """Delete documents from the MongoDB collection.++ Args:+ ids (`str | list[str] | None`, optional):+ List of document IDs to delete. If provided, deletes documents+ with matching doc_id in payload.+ """++ if not ids:+ return++ if isinstance(ids, str):+ ids = [ids]++ await self._collection.delete_many({"payload.doc_id": {"$in": ids}})++ def get_client(self) -> AsyncMongoClient:+ """Get the underlying MongoDB client for advanced operations.++ Returns:+ `AsyncMongoClient`: The AsyncMongoClient instance.+ """+ return self._client++ async def delete_collection(self) -> None:+ """Delete the entire collection.++ .. warning::+ This will permanently delete all documents in the collection.+ """+ await self._collection.drop()++ async def delete_database(self) -> None:+ """Delete the entire database.++ .. warning::+ This will permanently delete the entire database and all its+ collections.+ """+ await self._client.drop_database(self.db_name)++ async def close(self) -> None:+ """Close the MongoDB connection.++ This should be called when the store is no longer needed to properly+ clean up resources.+ """+ await self._client.close()diff --git a/tests/rag_store_test.py b/tests/rag_store_test.pyindex 485dd2382..b7db30f3a 100644--- a/tests/rag_store_test.py+++ b/tests/rag_store_test.py@@ -1,8 +1,9 @@# -*- coding: utf-8 -*-"""Test the RAG store implementations."""import os+from typing import AsyncGeneratorfrom unittest import IsolatedAsyncioTestCase-from unittest.mock import MagicMock, patch+from unittest.mock import MagicMock, patch, AsyncMockfrom agentscope.message import TextBlockfrom agentscope.rag import (@@ -11,6 +12,7 @@ from agentscope.rag import (DocMetadata,MilvusLiteStore,AlibabaCloudMySQLStore,+ MongoDBStore,)@@ -259,6 +261,178 @@ class RAGStoreTest(IsolatedAsyncioTestCase):mock_cursor.close.assert_called()mock_conn.close.assert_called()+ async def test_mongodb_store(self) -> None:+ """Test the MongoDBStore implementation using mocks."""+ # Create mock pymongo module+ mock_pymongo = MagicMock()+ mock_operations = MagicMock()+ mock_pymongo.operations = mock_operations++ # Create mock AsyncMongoClient+ mock_client = MagicMock()+ mock_db = MagicMock()+ mock_collection = MagicMock()++ # Configure mock client+ mock_pymongo.AsyncMongoClient.return_value = mock_client+ mock_client.get_database.return_value = mock_db++ # Configure mock database+ mock_db.list_collection_names = AsyncMock(return_value=[])+ mock_db.create_collection = AsyncMock(return_value=mock_collection)+ mock_db.get_collection.return_value = mock_collection++ # Configure mock collection+ mock_collection.create_search_index = AsyncMock()+ mock_collection.bulk_write = AsyncMock()+ mock_collection.delete_many = AsyncMock()+ mock_collection.drop = AsyncMock()++ # Mock list_search_indexes to return queryable index+ async def mock_index_iter() -> AsyncGenerator:+ yield {"queryable": True}++ mock_collection.list_search_indexes = AsyncMock(+ return_value=mock_index_iter(),+ )++ # Mock aggregate to return search results+ mock_search_results = [+ {+ "vector": [0.1, 0.2, 0.3],+ "payload": {+ "doc_id": "doc1",+ "chunk_id": 0,+ "total_chunks": 2,+ "content": {+ "type": "text",+ "text": "This is a test document.",+ },+ },+ "score": 0.97,+ },+ ]++ async def mock_aggregate_iter() -> AsyncGenerator:+ for item in mock_search_results:+ yield item++ mock_collection.aggregate = AsyncMock(+ return_value=mock_aggregate_iter(),+ )++ # Mock client close+ mock_client.close = AsyncMock()+ mock_client.drop_database = AsyncMock()++ # Mock ReplaceOne+ mock_replace_one = MagicMock()+ mock_pymongo.ReplaceOne = mock_replace_one++ with patch.dict(+ "sys.modules",+ {+ "pymongo": mock_pymongo,+ "pymongo.operations": mock_operations,+ },+ ):+ # Create store instance+ store = MongoDBStore(+ host="mongodb://localhost:27017",+ db_name="test_db",+ collection_name="test_collection",+ dimensions=3,+ distance="cosine",+ )++ # Verify client was created+ mock_pymongo.AsyncMongoClient.assert_called_once()++ # Test add operation+ await store.add(+ [+ Document(+ embedding=[0.1, 0.2, 0.3],+ metadata=DocMetadata(+ content=TextBlock(+ type="text",+ text="This is a test document.",+ ),+ doc_id="doc1",+ chunk_id=0,+ total_chunks=2,+ ),+ ),+ Document(+ embedding=[0.9, 0.1, 0.4],+ metadata=DocMetadata(+ content=TextBlock(+ type="text",+ text="This is another test document.",+ ),+ doc_id="doc1",+ chunk_id=1,+ total_chunks=2,+ ),+ ),+ ],+ )++ # Verify add operations+ self.assertTrue(mock_collection.bulk_write.called)++ # Reset mocks for search operation+ mock_collection.reset_mock()++ # Reconfigure list_search_indexes for search+ mock_collection.list_search_indexes = AsyncMock(+ return_value=mock_index_iter(),+ )+ mock_collection.aggregate = AsyncMock(+ return_value=mock_aggregate_iter(),+ )++ # Test search operation+ res = await store.search(+ query_embedding=[0.15, 0.25, 0.35],+ limit=3,+ score_threshold=0.8,+ )++ # Verify search results+ self.assertEqual(len(res), 1)+ self.assertAlmostEqual(res[0].score, 0.97, places=2)+ self.assertEqual(+ res[0].metadata.content["text"],+ "This is a test document.",+ )+ self.assertEqual(res[0].metadata.doc_id, "doc1")+ self.assertEqual(res[0].metadata.chunk_id, 0)+ self.assertEqual(res[0].metadata.total_chunks, 2)++ # Verify search executed aggregate+ self.assertTrue(mock_collection.aggregate.called)++ # Test delete operation+ await store.delete(ids="doc1")++ # Verify delete executed+ mock_collection.delete_many.assert_called_with(+ {"payload.doc_id": {"$in": ["doc1"]}},+ )++ # Test delete_collection+ await store.delete_collection()+ mock_collection.drop.assert_called()++ # Test delete_database+ await store.delete_database()+ mock_client.drop_database.assert_called_with("test_db")++ # Test close+ await store.close()+ mock_client.close.assert_called()+async def asyncTearDown(self) -> None:"""Clean up after tests."""# Remove Milvus Lite database file