Skip to content

Build and run agents you can see, understand and trust.

feat(rag): add MongoDB vector store in the RAG module. (#860)

qbc3d4ffb2890cb
diff --git a/examples/functionality/vector_store/mongodb/README.md b/examples/functionality/vector_store/mongodb/README.md
new file mode 100644
index 000000000..3e2dd6473
--- /dev/null
+++ b/examples/functionality/vector_store/mongodb/README.md
@@ -0,0 +1,211 @@
+# MongoDB Vector Store
+
+This example demonstrates how to use **MongoDBStore** for vector storage and semantic search in AgentScope using MongoDB's Vector Search capabilities.
+It includes comprehensive test scenarios covering CRUD operations, metadata filtering, document chunking, and distance metrics.
+
+### Quick Start
+
+Install agentscope first, and then the MongoDB dependency:
+
+```bash
+pip install pymongo
+```
+
+**Important:** Before running the example, you need to set the `MONGODB_HOST`
+environment variable with your MongoDB connection string:
+
+```bash
+# For local MongoDB
+export MONGODB_HOST="mongodb://localhost:27017/?directConnection=true"
+
+# For MongoDB Atlas (replace with your connection string)
+# export MONGODB_HOST=${YOUR_MONGODB_HOST}
+```
+
+Run the example script, which showcases adding, searching, and deleting in MongoDB vector store:
+
+```bash
+python main.py
+```
+
+> **Note:** The script connects to MongoDB Atlas or local MongoDB instance. Make sure you have a valid MongoDB connection string.
+
+## Prerequisites
+
+- Confirm your MongoDB instance supports Vector Search functionality
+- Valid MongoDB connection string (local or Atlas)
+
+## Usage
+
+### Initialize Store
+
+```python
+from agentscope.rag import MongoDBStore
+
+# For MongoDB Atlas
+store = MongoDBStore(
+ host="mongodb+srv://username:[email protected]/",
+ db_name="test_db",
+ collection_name="test_collection",
+ dimensions=768, # Match your embedding model
+ distance="cosine", # cosine, euclidean, or dotProduct
+)
+
+# For local MongoDB
+store = MongoDBStore(
+ host="mongodb://localhost:27017/?directConnection=true",
+ db_name="test_db",
+ collection_name="test_collection",
+ dimensions=768,
+ distance="cosine",
+)
+
+# To enable filtering in search, specify filter_fields:
+store = MongoDBStore(
+ host="mongodb://localhost:27017/?directConnection=true",
+ db_name="test_db",
+ collection_name="test_collection",
+ dimensions=768,
+ distance="cosine",
+ filter_fields=["payload.doc_id", "payload.chunk_id"], # Fields for filtering
+)
+
+# No manual initialization needed - everything is automatic!
+# Database, collection, and vector search index are created automatically
+# when you first call add() or search()
+```
+
+### Add Documents
+
+```python
+from agentscope.rag import Document, DocMetadata
+from agentscope.message import TextBlock
+
+doc = Document(
+ metadata=DocMetadata(
+ content=TextBlock(type="text", text="Your document text"),
+ doc_id="doc_1",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.1, 0.2, ...], # Your embedding vector
+)
+
+await store.add([doc])
+```
+
+### Search
+
+```python
+results = await store.search(
+ query_embedding=[0.15, 0.25, ...],
+ limit=5,
+ score_threshold=0.9, # Optional
+ filter={"payload.doc_id": {"$in": ["doc_1", "doc_2"]}}, # Optional filter
+)
+# Note:
+# - To use filter, the field must be declared in filter_fields when creating store
+# - MongoDB $vectorSearch filter supports: $gt, $gte, $lt, $lte,
+# $eq, $ne, $in, $nin, $exists, $not (NOT $regex)
+```
+
+### Delete
+
+```python
+# Delete by document IDs (no initialization needed)
+await store.delete(ids=["doc_1", "doc_2"])
+
+# Delete entire collection (use with caution)
+await store.delete_collection()
+
+# Delete entire database (use with caution)
+await store.delete_database()
+```
+
+## Distance Metrics
+
+| Metric | Description | Best For |
+|--------|-------------|----------|
+| **cosine** | Cosine similarity | Text embeddings (recommended) |
+| **euclidean** | Euclidean distance | Spatial data |
+| **dotProduct** | Inner Product | Recommendation systems |
+
+## Advanced Usage
+
+### Access Underlying Client
+
+```python
+client = store.get_client()
+# Use MongoDB client for advanced operations
+stats = await client[store.db_name].command("collStats", store.collection_name)
+```
+
+### Document Metadata
+
+- `content`: Text content (TextBlock)
+- `doc_id`: Unique document identifier
+- `chunk_id`: Chunk position (0-indexed)
+- `total_chunks`: Total chunks in document
+
+### Vector Search Index
+
+MongoDBStore automatically creates vector search indexes with the following configuration:
+
+```python
+{
+ "fields": [
+ {
+ "type": "vector",
+ "path": "vector",
+ "similarity": "cosine", # or euclidean, dotProduct
+ "numDimensions": 768
+ }
+ ]
+}
+```
+
+## Connection Examples
+
+### MongoDB Atlas
+
+```python
+store = MongoDBStore(
+ host="<YOUR_MONGO_ATLAS_CONNECTION_STRING>",
+ db_name="production_db",
+ collection_name="documents",
+ dimensions=1536,
+ distance="cosine",
+)
+```
+
+### Local MongoDB
+
+#### Without Authentication
+
+```python
+store = MongoDBStore(
+ host="mongodb://localhost:27017?directConnection=true",
+ db_name="local_db",
+ collection_name="test_collection",
+ dimensions=768,
+ distance="cosine",
+)
+```
+
+#### With Authentication
+
+```python
+store = MongoDBStore(
+ host="mongodb://user:pass@localhost:27017/?directConnection=true",
+ db_name="test_db",
+ collection_name="test_collection",
+ dimensions=768,
+ distance="cosine",
+)
+```
+
+## References
+
+- [MongoDB Vector Search Documentation](https://www.mongodb.com/docs/atlas/atlas-search/vector-search/)
+- [MongoDB Atlas Documentation](https://www.mongodb.com/docs/atlas/)
+- [AgentScope RAG Tutorial](https://doc.agentscope.io/tutorial/task_rag.html)
diff --git a/examples/functionality/vector_store/mongodb/main.py b/examples/functionality/vector_store/mongodb/main.py
new file mode 100644
index 000000000..0212a6693
--- /dev/null
+++ b/examples/functionality/vector_store/mongodb/main.py
@@ -0,0 +1,351 @@
+# -*- coding: utf-8 -*-
+"""Example of using MongoDBStore in AgentScope RAG system."""
+import asyncio
+import os
+
+from agentscope.rag import (
+ MongoDBStore,
+ Document,
+ DocMetadata,
+)
+from agentscope.message import TextBlock
+
+
+async def example_basic_operations() -> None:
+ """The example of basic CRUD operations with MongoDBStore."""
+ print("\n" + "=" * 60)
+ print("Test 1: Basic CRUD Operations")
+ print("=" * 60)
+
+ # Initialize MongoDBStore with MongoDB connection
+ store = MongoDBStore(
+ host=os.getenv("MONGODB_HOST"),
+ db_name="test_db",
+ collection_name="test_collection",
+ dimensions=4, # Small dimension for testing
+ distance="cosine",
+ )
+
+ print("✓ MongoDBStore initialized")
+
+ # Create test documents with embeddings
+ test_docs = [
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(
+ text="Artificial Intelligence is the future",
+ ),
+ doc_id="doc_1",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.1, 0.2, 0.3, 0.4],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Machine Learning is a subset of AI"),
+ doc_id="doc_2",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.2, 0.3, 0.4, 0.5],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Deep Learning uses neural networks"),
+ doc_id="doc_3",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.3, 0.4, 0.5, 0.6],
+ ),
+ ]
+
+ # Test add operation (automatically creates database, collection,
+ # and index)
+ await store.add(test_docs)
+ print(f"✓ Added {len(test_docs)} documents to the store")
+
+ # Test search operation (automatically waits for index to be ready)
+ query_embedding = [0.15, 0.25, 0.35, 0.45]
+ results = await store.search(
+ query_embedding=query_embedding,
+ limit=2,
+ )
+
+ print(f"\n✓ Search completed, found {len(results)} results:")
+ for i, result in enumerate(results, 1):
+ print(f" {i}. Score: {result.score:.4f}")
+ print(f" Content: {result.metadata.content}")
+ print(f" Doc ID: {result.metadata.doc_id}")
+
+ # Test search with score threshold (also waits for index if needed)
+ results_filtered = await store.search(
+ query_embedding=query_embedding,
+ limit=5,
+ score_threshold=0.3,
+ )
+ print(f"\n✓ Search with threshold (>0.3): {len(results_filtered)} results")
+
+ # Test delete operation (no initialization needed)
+ # Note: MongoDBStore uses ids parameter for deletion
+ await store.delete(ids=["doc_2", "doc_3", "doc_1"])
+ print("\n✓ Deleted documents with specified doc_ids")
+
+ # Verify deletion (search will wait for index if needed)
+ results_after_delete = await store.search(
+ query_embedding=query_embedding,
+ limit=5,
+ )
+ print(f"✓ After deletion: {len(results_after_delete)} documents remain")
+
+ # Get client for advanced operations
+ client = store.get_client()
+ print(f"\n✓ Got MongoDB Client: {type(client).__name__}")
+
+ await store.close()
+
+
+async def example_filter_search() -> None:
+ """The example of search with metadata filtering."""
+ print("\n" + "=" * 60)
+ print("Test 2: Search with Metadata Filtering")
+ print("=" * 60)
+
+ # To use filter in search, specify filter_fields when creating the store.
+ # These fields will be indexed for filtering in $vectorSearch.
+ store = MongoDBStore(
+ host=os.getenv("MONGODB_HOST"),
+ db_name="filter_test_db",
+ collection_name="filter_collection",
+ dimensions=4,
+ distance="cosine",
+ filter_fields=["payload.doc_id"], # Enable filtering on doc_id
+ )
+
+ # Create documents with different categories
+ docs = [
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Python is a programming language"),
+ doc_id="prog_1",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.1, 0.2, 0.3, 0.4],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(
+ text="Java is used for enterprise applications",
+ ),
+ doc_id="prog_2",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.2, 0.3, 0.4, 0.5],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Neural networks are used in AI"),
+ doc_id="ai_1",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.3, 0.4, 0.5, 0.6],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Deep learning requires GPUs"),
+ doc_id="ai_2",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.4, 0.5, 0.6, 0.7],
+ ),
+ ]
+
+ # Add documents (automatically creates database, collection, and index)
+ await store.add(docs)
+ print(f"✓ Added {len(docs)} documents with different doc_id prefixes")
+
+ # Search without filter (automatically waits for index if needed)
+ query_embedding = [0.25, 0.35, 0.45, 0.55]
+ all_results = await store.search(
+ query_embedding=query_embedding,
+ limit=4,
+ )
+ print(f"\n✓ Search without filter: {len(all_results)} results")
+ for i, result in enumerate(all_results, 1):
+ doc_id = result.metadata.doc_id
+ score = result.score
+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")
+
+ # Search with filter for programming docs
+ # Note: doc_id is stored in payload.doc_id in MongoDB documents
+ # MongoDB $vectorSearch filter supports: $gt, $gte, $lt, $lte, $eq, $ne,
+ # $in, $nin, $exists, $not (NOT $regex)
+ prog_results = await store.search(
+ query_embedding=query_embedding,
+ limit=4,
+ filter={"payload.doc_id": {"$in": ["prog_1", "prog_2"]}},
+ )
+ print(f"\n✓ Search with filter (prog docs): {len(prog_results)} results")
+ for i, result in enumerate(prog_results, 1):
+ doc_id = result.metadata.doc_id
+ score = result.score
+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")
+
+ # Search with filter for AI docs
+ ai_results = await store.search(
+ query_embedding=query_embedding,
+ limit=4,
+ filter={"payload.doc_id": {"$in": ["ai_1", "ai_2"]}},
+ )
+ print(f"\n✓ Search with filter (ai docs): {len(ai_results)} results")
+ for i, result in enumerate(ai_results, 1):
+ doc_id = result.metadata.doc_id
+ score = result.score
+ print(f" {i}. Doc ID: {doc_id}, Score: {score:.4f}")
+
+ await store.close()
+
+
+async def example_multiple_chunks() -> None:
+ """The example of documents with multiple chunks."""
+ print("\n" + "=" * 60)
+ print("Test 3: Documents with Multiple Chunks")
+ print("=" * 60)
+
+ store = MongoDBStore(
+ host=os.getenv("MONGODB_HOST"),
+ db_name="chunks_test_db",
+ collection_name="chunks_collection",
+ dimensions=4,
+ distance="cosine",
+ )
+
+ # Create a document split into multiple chunks
+ chunks = [
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Chapter 1: Introduction to AI"),
+ doc_id="book_1",
+ chunk_id=0,
+ total_chunks=3,
+ ),
+ embedding=[0.1, 0.2, 0.3, 0.4],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Chapter 2: Machine Learning Basics"),
+ doc_id="book_1",
+ chunk_id=1,
+ total_chunks=3,
+ ),
+ embedding=[0.2, 0.3, 0.4, 0.5],
+ ),
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text="Chapter 3: Deep Learning Advanced"),
+ doc_id="book_1",
+ chunk_id=2,
+ total_chunks=3,
+ ),
+ embedding=[0.3, 0.4, 0.5, 0.6],
+ ),
+ ]
+
+ # Add chunks (automatically creates database, collection, and index)
+ await store.add(chunks)
+ print(f"✓ Added document with {len(chunks)} chunks")
+
+ # Search and verify chunk information (automatically waits for index if
+ # needed)
+ query_embedding = [0.2, 0.3, 0.4, 0.5]
+ results = await store.search(
+ query_embedding=query_embedding,
+ limit=3,
+ )
+
+ print("\n✓ Search results for multi-chunk document:")
+ for i, result in enumerate(results, 1):
+ chunk_info = (
+ f"{result.metadata.chunk_id}/{result.metadata.total_chunks}"
+ )
+ print(f" {i}. Chunk {chunk_info}")
+ print(f" Content: {result.metadata.content}")
+ print(f" Score: {result.score:.4f}")
+
+ await store.close()
+
+
+async def example_distance_metrics() -> None:
+ """The example of different distance metrics."""
+ print("\n" + "=" * 60)
+ print("Test 4: Different Distance Metrics")
+ print("=" * 60)
+
+ # Test with different metrics
+ metrics = ["cosine", "euclidean", "dotProduct"]
+
+ for metric in metrics:
+ print(f"\n--- Testing {metric} metric ---")
+ store = MongoDBStore(
+ host=os.getenv("MONGODB_HOST"),
+ db_name=f"{metric}_test_db",
+ collection_name=f"{metric}_collection",
+ dimensions=4,
+ distance=metric,
+ )
+
+ docs = [
+ Document(
+ metadata=DocMetadata(
+ content=TextBlock(text=f"Test doc for {metric}"),
+ doc_id=f"doc_{metric}_1",
+ chunk_id=0,
+ total_chunks=1,
+ ),
+ embedding=[0.1, 0.2, 0.3, 0.4],
+ ),
+ ]
+
+ # Add and search (automatically creates database/collection/index
+ # and waits for index)
+ await store.add(docs)
+ results = await store.search(
+ query_embedding=[0.1, 0.2, 0.3, 0.4],
+ limit=1,
+ )
+
+ print(f"✓ {metric} metric: Score = {results[0].score:.4f}")
+
+ await store.close()
+
+
+async def main() -> None:
+ """Run all example."""
+ print("\n" + "=" * 60)
+ print("MongoDBStore Comprehensive Test Suite")
+ print("=" * 60)
+
+ try:
+ # await example_basic_operations()
+ # await example_filter_search()
+ # await example_multiple_chunks()
+ await example_distance_metrics()
+
+ print("\n" + "=" * 60)
+ print("✓ All tests completed successfully!")
+ print("=" * 60)
+
+ except Exception as e:
+ print(f"\n✗ Test failed with error: {e}")
+ import traceback
+
+ traceback.print_exc()
+
+
+if __name__ == "__main__":
+ asyncio.run(main())
diff --git a/pyproject.toml b/pyproject.toml
index 52f32d72f..d13147435 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -106,10 +106,12 @@ readers = [
qdrant = ["qdrant-client==1.15.1"]
milvus = ["pymilvus[milvus_lite]"]
ali_mysql = ["mysql-connector-python"]
+mongodb = ["pymongo"]
vdbs = [
"agentscope[ali_mysql]",
"agentscope[qdrant]",
"agentscope[milvus]",
+ "agentscope[mongodb]",
]
diff --git a/src/agentscope/rag/__init__.py b/src/agentscope/rag/__init__.py
index 146b35aa6..3dfe828ef 100644
--- a/src/agentscope/rag/__init__.py
+++ b/src/agentscope/rag/__init__.py
@@ -16,6 +16,7 @@ from ._store import (
VDBStoreBase,
QdrantStore,
MilvusLiteStore,
+ MongoDBStore,
AlibabaCloudMySQLStore,
)
from ._knowledge_base import KnowledgeBase
@@ -33,6 +34,7 @@ __all__ = [
"VDBStoreBase",
"QdrantStore",
"MilvusLiteStore",
+ "MongoDBStore",
"AlibabaCloudMySQLStore",
"KnowledgeBase",
"SimpleKnowledge",
diff --git a/src/agentscope/rag/_store/__init__.py b/src/agentscope/rag/_store/__init__.py
index c107eae5d..96e0ce323 100644
--- a/src/agentscope/rag/_store/__init__.py
+++ b/src/agentscope/rag/_store/__init__.py
@@ -6,11 +6,13 @@ from ._store_base import (
)
from ._qdrant_store import QdrantStore
from ._milvuslite_store import MilvusLiteStore
+from ._mongodb_store import MongoDBStore
from ._alibabacloud_mysql_store import AlibabaCloudMySQLStore
__all__ = [
"VDBStoreBase",
"QdrantStore",
"MilvusLiteStore",
+ "MongoDBStore",
"AlibabaCloudMySQLStore",
]
diff --git a/src/agentscope/rag/_store/_mongodb_store.py b/src/agentscope/rag/_store/_mongodb_store.py
new file mode 100644
index 000000000..8da6e46f1
--- /dev/null
+++ b/src/agentscope/rag/_store/_mongodb_store.py
@@ -0,0 +1,392 @@
+# -*- coding: utf-8 -*-
+"""The MongoDB vector store implementation using MongoDB Vector Search.
+
+This implementation provides a vector database store using MongoDB's vector
+ search capabilities. It requires MongoDB with vector search support and
+ automatically creates vector search indexes.
+"""
+import asyncio
+import time
+
+from typing import Any, Literal, TYPE_CHECKING
+
+from .._reader import Document
+from ._store_base import VDBStoreBase
+from .._document import DocMetadata
+from ...types import Embedding
+
+if TYPE_CHECKING:
+ from pymongo import AsyncMongoClient
+else:
+ AsyncMongoClient = "pymongo.AsyncMongoClient"
+
+
+class MongoDBStore(VDBStoreBase):
+ """MongoDB vector store using MongoDB Vector Search.
+
+ This class provides a vector database store implementation using MongoDB's
+ vector search capabilities. It requires MongoDB with vector search support
+ and creates vector search indexes automatically.
+
+ .. note:: Ensure your MongoDB instance supports Vector Search
+ functionality.
+
+ .. note:: The store automatically creates database, collection, and vector
+ search index on first operation. No manual initialization is required.
+ """
+
+ def __init__(
+ self,
+ host: str,
+ db_name: str,
+ collection_name: str,
+ dimensions: int,
+ index_name: str = "vector_index",
+ distance: Literal["cosine", "euclidean", "dotProduct"] = "cosine",
+ filter_fields: list[str] | None = None,
+ client_kwargs: dict[str, Any] | None = None,
+ db_kwargs: dict[str, Any] | None = None,
+ collection_kwargs: dict[str, Any] | None = None,
+ ) -> None:
+ """Initialize the MongoDB vector store.
+
+ Args:
+ host (`str`):
+ MongoDB connection host, e.g., "mongodb://localhost:27017" or
+ "mongodb+srv://cluster.mongodb.net/".
+ db_name (`str`):
+ Database name to store vector documents.
+ collection_name (`str`):
+ Collection name to store vector documents.
+ dimensions (`int`):
+ Embedding dimensions for the vector search index.
+ index_name (`str`, defaults to "vector_index"):
+ Vector search index name.
+ distance (`Literal["cosine", "euclidean", "dotProduct"]`, \
+ defaults to "cosine"):
+ Distance metric for vector similarity. Can be one of "cosine",
+ "euclidean", or "dotProduct".
+ filter_fields (`list[str] | None`, optional):
+ List of field paths to index for filtering in $vectorSearch.
+ For example: ["payload.doc_id", "payload.chunk_id"].
+ These fields can then be used in the `filter` parameter of
+ the `search` method. MongoDB $vectorSearch filter supports:
+ $gt, $gte, $lt, $lte, $eq, $ne, $in, $nin, $exists, $not.
+ client_kwargs (`dict[str, Any] | None`, optional):
+ Additional kwargs for MongoDB client.
+ db_kwargs (`dict[str, Any] | None`, optional):
+ Additional kwargs for database.
+ collection_kwargs (`dict[str, Any] | None`, optional):
+ Additional kwargs for collection.
+
+ Raises:
+ ImportError: If pymongo is not installed.
+ """
+ try:
+ from pymongo import AsyncMongoClient
+ except ImportError as e:
+ raise ImportError(
+ "Please install the latest pymongo package to use "
+ "AsyncMongoClient: `pip install pymongo`",
+ ) from e
+
+ self._client: AsyncMongoClient = AsyncMongoClient(
+ host,
+ **(client_kwargs or {}),
+ )
+ self.db_name = db_name
+ self.collection_name = collection_name
+ self.index_name = index_name
+ self.dimensions = dimensions
+ self.distance = distance
+ self.filter_fields = filter_fields or []
+ self.db_kwargs = db_kwargs or {}
+ self.collection_kwargs = collection_kwargs or {}
+
+ self._db = None
+ self._collection = None
+
+ async def _validate_db_and_collection(self) -> None:
+ """Validate the database and collection exist, create if necessary.
+
+ This method ensures the database and collection are available,
+ and creates a vector search index for the collection.
+
+ Raises:
+ Exception: If database or collection creation fails.
+ """
+ self._db = self._client.get_database(
+ self.db_name,
+ **self.db_kwargs,
+ )
+
+ if self.collection_name not in await self._db.list_collection_names():
+ self._collection = await self._db.create_collection(
+ self.collection_name,
+ )
+ else:
+ self._collection = self._db.get_collection(
+ self.collection_name,
+ **self.collection_kwargs,
+ )
+
+ from pymongo.operations import SearchIndexModel
+
+ # Build index fields: vector field + optional filter fields
+ index_fields: list[dict[str, Any]] = [
+ {
+ "type": "vector",
+ "path": "vector",
+ "similarity": self.distance,
+ "numDimensions": self.dimensions,
+ },
+ ]
+
+ # Add user-specified filter fields
+ for field_path in self.filter_fields:
+ index_fields.append(
+ {
+ "type": "filter",
+ "path": field_path,
+ },
+ )
+
+ search_index_model = SearchIndexModel(
+ definition={"fields": index_fields},
+ name=self.index_name,
+ type="vectorSearch",
+ )
+
+ await self._collection.create_search_index(
+ model=search_index_model,
+ )
+
+ async def _wait_for_index_ready(self, timeout: int = 30) -> None:
+ """Wait for the vector search index to be ready with timeout
+ protection.
+
+ Args:
+ timeout (`int`, defaults to 30):
+ Maximum time to wait in seconds.
+
+ Raises:
+ TimeoutError: If index is not ready within the timeout period.
+ """
+
+ start_time = time.time()
+ while time.time() - start_time < timeout:
+ try:
+ indices = []
+ async for idx in await self._collection.list_search_indexes(
+ self.index_name,
+ ):
+ indices.append(idx)
+ if indices and indices[0].get("queryable") is True:
+ return
+ except Exception:
+ pass
+
+ await asyncio.sleep(0.2)
+
+ raise TimeoutError(f"Index not ready after {timeout} seconds")
+
+ async def add(self, documents: list[Document], **kwargs: Any) -> None:
+ """Insert documents with embeddings into MongoDB.
+
+ This method automatically creates the database, collection, and vector
+ search index if they don't exist.
+
+ Args:
+ documents (`list[Document]`):
+ List of Document objects to insert.
+ **kwargs (`Any`):
+ Additional arguments (unused).
+
+ .. note::
+ Each inserted record has structure:
+
+ .. code-block:: python
+
+ {
+ "id": str, # Document ID
+ "vector": list[float], # Vector embedding
+ "payload": dict, # DocMetadata as dict
+ }
+ """
+ await self._validate_db_and_collection()
+
+ # Prepare documents for insertion
+ docs_to_insert = []
+ for doc in documents:
+ # Convert DocMetadata to dict for storage
+ payload = {
+ "doc_id": doc.metadata.doc_id,
+ "chunk_id": doc.metadata.chunk_id,
+ "total_chunks": doc.metadata.total_chunks,
+ "content": doc.metadata.content,
+ }
+
+ # Create document record
+ doc_record = {
+ "id": f"{doc.metadata.doc_id}_{doc.metadata.chunk_id}",
+ "vector": doc.embedding,
+ "payload": payload,
+ }
+ docs_to_insert.append(doc_record)
+
+ # Insert documents using upsert to handle duplicates
+ if not docs_to_insert:
+ return
+ from pymongo import ReplaceOne
+
+ operations = [
+ ReplaceOne(
+ {"id": doc_record["id"]},
+ doc_record,
+ upsert=True,
+ )
+ for doc_record in docs_to_insert
+ ]
+ await self._collection.bulk_write(operations)
+
+ async def search(
+ self,
+ query_embedding: Embedding,
+ limit: int,
+ score_threshold: float | None = None,
+ **kwargs: Any,
+ ) -> list[Document]:
+ """Search relevant documents using MongoDB Vector Search.
+
+ This method uses MongoDB's $vectorSearch aggregation pipeline for
+ vector similarity search. It automatically waits for the vector search
+ index to be ready before performing the search.
+
+ Args:
+ query_embedding (`Embedding`):
+ The embedding vector to search for.
+ limit (`int`):
+ Maximum number of documents to return.
+ score_threshold (`float | None`, optional):
+ Minimum similarity score threshold. Documents with scores below
+ this threshold will be filtered out.
+ **kwargs (`Any`):
+ Additional arguments for the search operation.
+
+ Returns:
+ `list[Document]`: List of Document objects with embedding,
+ score, and metadata.
+
+ .. note::
+ - Requires MongoDB with vector search support
+ - Uses $vectorSearch aggregation pipeline
+ """
+ await self._validate_db_and_collection()
+ # Wait for index to be ready before searching
+ await self._wait_for_index_ready()
+
+ # Construct aggregation pipeline for vector search
+ # See: https://www.mongodb.com/docs/atlas/atlas-search/vector-search/
+ num_candidates = int(
+ kwargs.pop(
+ "num_candidates",
+ max(
+ 100,
+ limit * 20,
+ ),
+ ),
+ )
+
+ pipeline: list[dict[str, Any]] = [
+ {
+ "$vectorSearch": {
+ "index": self.index_name,
+ "path": "vector",
+ "queryVector": list(query_embedding),
+ "numCandidates": num_candidates,
+ "limit": limit,
+ **kwargs,
+ },
+ },
+ {
+ "$project": {
+ "vector": 1,
+ "payload": 1,
+ "score": {"$meta": "vectorSearchScore"},
+ },
+ },
+ ]
+
+ cursor = await self._collection.aggregate(pipeline)
+ results: list[Document] = []
+ async for item in cursor:
+ score_val = float(item.get("score", 0.0))
+ if score_threshold is not None and score_val < score_threshold:
+ continue
+
+ payload = item.get("payload", {})
+ # Rebuild Document
+ metadata = DocMetadata(**payload)
+
+ results.append(
+ Document(
+ embedding=[float(x) for x in item.get("vector", [])],
+ score=score_val,
+ metadata=metadata,
+ ),
+ )
+
+ return results
+
+ async def delete(
+ self,
+ ids: str | list[str] | None = None,
+ ) -> None:
+ """Delete documents from the MongoDB collection.
+
+ Args:
+ ids (`str | list[str] | None`, optional):
+ List of document IDs to delete. If provided, deletes documents
+ with matching doc_id in payload.
+ """
+
+ if not ids:
+ return
+
+ if isinstance(ids, str):
+ ids = [ids]
+
+ await self._collection.delete_many({"payload.doc_id": {"$in": ids}})
+
+ def get_client(self) -> AsyncMongoClient:
+ """Get the underlying MongoDB client for advanced operations.
+
+ Returns:
+ `AsyncMongoClient`: The AsyncMongoClient instance.
+ """
+ return self._client
+
+ async def delete_collection(self) -> None:
+ """Delete the entire collection.
+
+ .. warning::
+ This will permanently delete all documents in the collection.
+ """
+ await self._collection.drop()
+
+ async def delete_database(self) -> None:
+ """Delete the entire database.
+
+ .. warning::
+ This will permanently delete the entire database and all its
+ collections.
+ """
+ await self._client.drop_database(self.db_name)
+
+ async def close(self) -> None:
+ """Close the MongoDB connection.
+
+ This should be called when the store is no longer needed to properly
+ clean up resources.
+ """
+ await self._client.close()
diff --git a/tests/rag_store_test.py b/tests/rag_store_test.py
index 485dd2382..b7db30f3a 100644
--- a/tests/rag_store_test.py
+++ b/tests/rag_store_test.py
@@ -1,8 +1,9 @@
# -*- coding: utf-8 -*-
"""Test the RAG store implementations."""
import os
+from typing import AsyncGenerator
from unittest import IsolatedAsyncioTestCase
-from unittest.mock import MagicMock, patch
+from unittest.mock import MagicMock, patch, AsyncMock
from agentscope.message import TextBlock
from agentscope.rag import (
@@ -11,6 +12,7 @@ from agentscope.rag import (
DocMetadata,
MilvusLiteStore,
AlibabaCloudMySQLStore,
+ MongoDBStore,
)
@@ -259,6 +261,178 @@ class RAGStoreTest(IsolatedAsyncioTestCase):
mock_cursor.close.assert_called()
mock_conn.close.assert_called()
+ async def test_mongodb_store(self) -> None:
+ """Test the MongoDBStore implementation using mocks."""
+ # Create mock pymongo module
+ mock_pymongo = MagicMock()
+ mock_operations = MagicMock()
+ mock_pymongo.operations = mock_operations
+
+ # Create mock AsyncMongoClient
+ mock_client = MagicMock()
+ mock_db = MagicMock()
+ mock_collection = MagicMock()
+
+ # Configure mock client
+ mock_pymongo.AsyncMongoClient.return_value = mock_client
+ mock_client.get_database.return_value = mock_db
+
+ # Configure mock database
+ mock_db.list_collection_names = AsyncMock(return_value=[])
+ mock_db.create_collection = AsyncMock(return_value=mock_collection)
+ mock_db.get_collection.return_value = mock_collection
+
+ # Configure mock collection
+ mock_collection.create_search_index = AsyncMock()
+ mock_collection.bulk_write = AsyncMock()
+ mock_collection.delete_many = AsyncMock()
+ mock_collection.drop = AsyncMock()
+
+ # Mock list_search_indexes to return queryable index
+ async def mock_index_iter() -> AsyncGenerator:
+ yield {"queryable": True}
+
+ mock_collection.list_search_indexes = AsyncMock(
+ return_value=mock_index_iter(),
+ )
+
+ # Mock aggregate to return search results
+ mock_search_results = [
+ {
+ "vector": [0.1, 0.2, 0.3],
+ "payload": {
+ "doc_id": "doc1",
+ "chunk_id": 0,
+ "total_chunks": 2,
+ "content": {
+ "type": "text",
+ "text": "This is a test document.",
+ },
+ },
+ "score": 0.97,
+ },
+ ]
+
+ async def mock_aggregate_iter() -> AsyncGenerator:
+ for item in mock_search_results:
+ yield item
+
+ mock_collection.aggregate = AsyncMock(
+ return_value=mock_aggregate_iter(),
+ )
+
+ # Mock client close
+ mock_client.close = AsyncMock()
+ mock_client.drop_database = AsyncMock()
+
+ # Mock ReplaceOne
+ mock_replace_one = MagicMock()
+ mock_pymongo.ReplaceOne = mock_replace_one
+
+ with patch.dict(
+ "sys.modules",
+ {
+ "pymongo": mock_pymongo,
+ "pymongo.operations": mock_operations,
+ },
+ ):
+ # Create store instance
+ store = MongoDBStore(
+ host="mongodb://localhost:27017",
+ db_name="test_db",
+ collection_name="test_collection",
+ dimensions=3,
+ distance="cosine",
+ )
+
+ # Verify client was created
+ mock_pymongo.AsyncMongoClient.assert_called_once()
+
+ # Test add operation
+ await store.add(
+ [
+ Document(
+ embedding=[0.1, 0.2, 0.3],
+ metadata=DocMetadata(
+ content=TextBlock(
+ type="text",
+ text="This is a test document.",
+ ),
+ doc_id="doc1",
+ chunk_id=0,
+ total_chunks=2,
+ ),
+ ),
+ Document(
+ embedding=[0.9, 0.1, 0.4],
+ metadata=DocMetadata(
+ content=TextBlock(
+ type="text",
+ text="This is another test document.",
+ ),
+ doc_id="doc1",
+ chunk_id=1,
+ total_chunks=2,
+ ),
+ ),
+ ],
+ )
+
+ # Verify add operations
+ self.assertTrue(mock_collection.bulk_write.called)
+
+ # Reset mocks for search operation
+ mock_collection.reset_mock()
+
+ # Reconfigure list_search_indexes for search
+ mock_collection.list_search_indexes = AsyncMock(
+ return_value=mock_index_iter(),
+ )
+ mock_collection.aggregate = AsyncMock(
+ return_value=mock_aggregate_iter(),
+ )
+
+ # Test search operation
+ res = await store.search(
+ query_embedding=[0.15, 0.25, 0.35],
+ limit=3,
+ score_threshold=0.8,
+ )
+
+ # Verify search results
+ self.assertEqual(len(res), 1)
+ self.assertAlmostEqual(res[0].score, 0.97, places=2)
+ self.assertEqual(
+ res[0].metadata.content["text"],
+ "This is a test document.",
+ )
+ self.assertEqual(res[0].metadata.doc_id, "doc1")
+ self.assertEqual(res[0].metadata.chunk_id, 0)
+ self.assertEqual(res[0].metadata.total_chunks, 2)
+
+ # Verify search executed aggregate
+ self.assertTrue(mock_collection.aggregate.called)
+
+ # Test delete operation
+ await store.delete(ids="doc1")
+
+ # Verify delete executed
+ mock_collection.delete_many.assert_called_with(
+ {"payload.doc_id": {"$in": ["doc1"]}},
+ )
+
+ # Test delete_collection
+ await store.delete_collection()
+ mock_collection.drop.assert_called()
+
+ # Test delete_database
+ await store.delete_database()
+ mock_client.drop_database.assert_called_with("test_db")
+
+ # Test close
+ await store.close()
+ mock_client.close.assert_called()
+
async def asyncTearDown(self) -> None:
"""Clean up after tests."""
# Remove Milvus Lite database file