Skip to content

Build and run agents you can see, understand and trust.

feat(rag): add MongoDB vector store backend (#2008)

xu-supingb2d92f31e0d7
diff --git a/examples/rag/README.md b/examples/rag/README.md
index 1e7e1abde..dc4589889 100644
--- a/examples/rag/README.md
+++ b/examples/rag/README.md
@@ -7,7 +7,7 @@ Two library-mode walk-throughs of `agentscope.rag` — no FastAPI service, no ma
| [`index_and_search.py`](./index_and_search.py) | The minimal pipeline: parse → chunk → embed → insert, then `KnowledgeBase.search`. Start here. |
| [`integrate_with_agent.py`](./integrate_with_agent.py) | Attaches the same `KnowledgeBase` to an `Agent` via `RAGMiddleware`, in both `static` (auto-inject) and `agentic` (tool-driven) modes. |
-Both examples use an in-memory Qdrant store (`location=":memory:"`) and the DashScope `text-embedding-v4` model, so no external services are required.
+Both examples use an in-memory Qdrant store (`location=":memory:"`) and the DashScope `text-embedding-v4` model, so no external services are required. The sections below show how to swap in Milvus Lite or MongoDB instead; those backends need additional setup.
## Install
@@ -19,6 +19,8 @@ uv pip install "agentscope[rag]"
uv pip install -e ".[rag]"
```
+### Milvus Lite (local persistence)
+
To use a local persistent Milvus Lite vector store instead of the
in-memory Qdrant store, install the optional extra:
@@ -28,7 +30,8 @@ uv pip install "agentscope[milvuslite]"
uv pip install -e ".[milvuslite]"
```
-Then replace the vector store construction:
+Then replace the vector store construction in `index_and_search.py`
+and/or `integrate_with_agent.py`:
```python
from agentscope.rag import MilvusLiteStore
@@ -36,6 +39,88 @@ from agentscope.rag import MilvusLiteStore
store = MilvusLiteStore(uri="./rag_demo.db")
```
+### MongoDB Vector Search
+
+To use MongoDB as the vector backend instead of the in-memory Qdrant
+store — useful when your team already runs MongoDB as the primary data
+store and wants to avoid maintaining a separate vector database — install
+the optional extra:
+
+```bash
+uv pip install "agentscope[mongodb]"
+# Or from source (repo root)
+uv pip install -e ".[mongodb]"
+```
+
+**Prerequisites**
+
+- A MongoDB deployment with [Vector Search](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/) enabled:
+ - **MongoDB Atlas** — create a cluster and enable Vector Search on the target database; or
+ - **Self-hosted** — MongoDB 7.0+ replica set with Vector Search enabled.
+- A connection URI available in the environment (do not hard-code credentials):
+
+```bash
+export MONGODB_URI="mongodb+srv://user:[email protected]/?retryWrites=true&w=majority"
+# Self-hosted example:
+# export MONGODB_URI="mongodb://localhost:27017"
+```
+
+Then replace the vector store construction in `index_and_search.py`
+and/or `integrate_with_agent.py`:
+
+```python
+import os
+
+from agentscope.rag import MongoDBStore
+
+store = MongoDBStore(
+ uri=os.environ["MONGODB_URI"],
+ database="agentscope_rag",
+ # Declare every field you plan to filter on in search().
+ # Required for metadata_filter; defaults to ["document_id"] only.
+ filter_fields=[
+ "document_id",
+ # "chunk.metadata.tenant_id", # uncomment if you use metadata_filter
+ ],
+)
+
+# MongoDBStore is also an async context manager — same as QdrantStore.
+async with store:
+ knowledge = KnowledgeBase(
+ name="demo-kb",
+ description="A toy corpus on cats and AgentScope.",
+ embedding_model=embedding_model,
+ vector_store=store,
+ collection=COLLECTION,
+ )
+ ...
+```
+
+**Notes**
+
+- The examples use DashScope `text-embedding-v4` with `dimensions=1024`.
+ `MongoDBStore.create_collection` is called automatically on the first
+ index operation with that dimension — keep the embedding model and
+ index dimensions aligned.
+- Unlike Qdrant `:memory:` or Milvus Lite (local `.db` file), MongoDB is
+ an external service; you must have a reachable cluster before running
+ the scripts.
+- If `search(..., metadata_filter={...})` returns no results or errors,
+ ensure each metadata key is listed in `filter_fields` as
+ `chunk.metadata.<key>` when constructing `MongoDBStore`.
+- For the full FastAPI RAG service, pass the same `MongoDBStore` instance
+ to `create_app(vector_store=...)` in `examples/agent_service/main.py`
+ (the default there uses in-memory Qdrant for zero-setup demos).
+
+### Choosing a vector backend
+
+| | Qdrant (default) | Milvus Lite | MongoDB |
+| --- | --- | --- | --- |
+| Install extra | `agentscope[rag]` | `agentscope[milvuslite]` | `agentscope[mongodb]` |
+| External service | No | No | Yes |
+| Persistence | No (`:memory:`) | Yes (local `.db`) | Yes (server) |
+| Best for | Quick start / tests | Local dev with persistence | Teams already on MongoDB |
+
`integrate_with_agent.py` additionally uses `DashScopeChatModel`, which is already in the base `agentscope` dependencies.
## Run
@@ -47,7 +132,8 @@ python examples/rag/index_and_search.py
python examples/rag/integrate_with_agent.py
```
+When using MongoDB, also export `MONGODB_URI` before running.
+
## Service mode
The two scripts above are library-mode — you drive the pipeline yourself in a single process. For the full service-mode experience (FastAPI endpoints for knowledge base CRUD, document upload, indexing workers, and search), see [`examples/agent_service`](../agent_service) for the backend and [`examples/web_ui`](../web_ui) for the chat-style UI.
-
diff --git a/pyproject.toml b/pyproject.toml
index 81158f37b..ffd601aec 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -88,6 +88,10 @@ milvuslite = [
"pymilvus[milvus-lite]>=2.4.0",
"milvus-lite>=3.0; sys_platform == 'win32'",
]
+# ------------ MongoDB vector store ------------
+mongodb = [
+ "pymongo>=4.7",
+]
# ------------ S3-compatible blob store ------------
s3 = [
@@ -114,6 +118,7 @@ full = [
"agentscope[tools]",
"agentscope[rag]",
"agentscope[milvuslite]",
+ "agentscope[mongodb]",
"agentscope[s3]",
"agentscope[mem0]",
"agentscope[reme]",
@@ -153,7 +158,7 @@ include-package-data = true
# ``"*"`` applies to every discovered package. Patterns:
# - ``py.typed`` marks the typed package per PEP 561.
# - ``_models/*.yaml`` reaches into the data-only ``_models/``
-# subdirs that ship with each model provider (they have no
+# subdirs that ship with each model provider (they have no
# ``__init__.py`` and are not packages, but the files still
# live alongside the parent package's source on disk).
"*" = [
@@ -162,7 +167,6 @@ include-package-data = true
"_cosyvoice_models/*.yaml",
]
"agentscope.workspace._docker" = ["Dockerfile*.template"]
-
[build-system]
requires = ["setuptools>=45", "wheel"]
build-backend = "setuptools.build_meta"
diff --git a/src/agentscope/rag/__init__.py b/src/agentscope/rag/__init__.py
index 78697cb40..5d2c3a12e 100644
--- a/src/agentscope/rag/__init__.py
+++ b/src/agentscope/rag/__init__.py
@@ -14,6 +14,7 @@ from ._vdb import (
VectorRecord,
VectorSearchResult,
QdrantStore,
+ MongoDBStore,
)
from ._knowledge import KnowledgeBase
@@ -34,4 +35,5 @@ __all__ = [
"VectorSearchResult",
"QdrantStore",
"KnowledgeBase",
+ "MongoDBStore",
]
diff --git a/src/agentscope/rag/_vdb/__init__.py b/src/agentscope/rag/_vdb/__init__.py
index 58774cfd9..68c712032 100644
--- a/src/agentscope/rag/_vdb/__init__.py
+++ b/src/agentscope/rag/_vdb/__init__.py
@@ -7,8 +7,9 @@ from ._vector_store import (
VectorSearchResult,
VectorStoreBase,
)
-from ._milvus_lite import MilvusLiteStore
from ._qdrant import QdrantStore
+from ._mongodb import MongoDBStore
+from ._milvus_lite import MilvusLiteStore
__all__ = [
"DocumentSummary",
@@ -17,4 +18,5 @@ __all__ = [
"VectorRecord",
"VectorSearchResult",
"QdrantStore",
+ "MongoDBStore",
]
diff --git a/src/agentscope/rag/_vdb/_mongodb.py b/src/agentscope/rag/_vdb/_mongodb.py
new file mode 100644
index 000000000..358c8a808
--- /dev/null
+++ b/src/agentscope/rag/_vdb/_mongodb.py
@@ -0,0 +1,464 @@
+# -*- coding: utf-8 -*-
+"""MongoDB implementation of the vector store backend.
+
+Built on ``pymongo.AsyncMongoClient`` and MongoDB Vector Search
+(``$vectorSearch``), so all operations are non-blocking and safe to
+call from the application's event loop.
+
+The same class supports MongoDB Atlas and self-hosted deployments
+through the constructor arguments:
+
+- ``uri="mongodb+srv://..."`` — MongoDB Atlas cluster
+- ``uri="mongodb://localhost:27017"`` — self-hosted replica set
+ (MongoDB 7.0+ with Vector Search enabled)
+"""
+import asyncio
+import time
+from typing import TYPE_CHECKING, Any, Literal
+
+from ._vector_store import (
+ DocumentSummary,
+ VectorRecord,
+ VectorSearchResult,
+ VectorStoreBase,
+)
+from .._document import Chunk
+
+if TYPE_CHECKING:
+ from pymongo import AsyncMongoClient
+
+
+class MongoDBStore(VectorStoreBase):
+ """Vector store backend backed by `MongoDB Vector Search \
+<https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/>`_.
+
+ Each knowledge base maps to one MongoDB collection inside a single
+ database. Every document stores the owning ``document_id`` plus
+ the serialized :class:`~agentscope.rag.Chunk`, which is
+ reconstructed on retrieval.
+
+ .. note:: Requires ``pymongo`` with ``AsyncMongoClient`` support
+ (``pymongo>=4.7``). Install with
+ ``pip install agentscope[mongodb]``.
+
+ .. note:: Fields used in :meth:`search` ``metadata_filter`` must be
+ declared in ``filter_fields`` at construction time so they are
+ included in the vector search index definition.
+
+ .. code-block:: python
+
+ # MongoDB Atlas
+ store = MongoDBStore(
+ uri="mongodb+srv://user:[email protected]",
+ database="agentscope_rag",
+ filter_fields=[
+ "document_id",
+ "chunk.metadata.tenant_id",
+ ],
+ )
+
+ async with store:
+ await store.create_collection("kb-1", dimensions=768)
+
+ """
+
+ def __init__(
+ self,
+ uri: str,
+ database: str,
+ distance: Literal["cosine", "euclidean", "dotProduct"] = "cosine",
+ index_name: str = "vector_index",
+ filter_fields: list[str] | None = None,
+ client_kwargs: dict[str, Any] | None = None,
+ ) -> None:
+ """Initialize the MongoDB vector store.
+
+ Args:
+ uri (`str`):
+ The MongoDB connection URI, e.g.
+ ``"mongodb+srv://..."`` for Atlas or
+ ``"mongodb://localhost:27017"`` for a self-hosted
+ replica set.
+ database (`str`):
+ The database name. Each knowledge base maps to one
+ collection inside this database.
+ distance (`Literal["cosine", "euclidean", "dotProduct"]`, \
+ defaults to ``"cosine"``):
+ The similarity metric used when creating vector search
+ indexes.
+ index_name (`str`, defaults to ``"vector_index"``):
+ The name of the vector search index created on each
+ collection. All collections share this index name.
+ filter_fields (`list[str] | None`, optional):
+ Field paths declared as filter fields in the vector
+ search index. Required for ``metadata_filter`` in
+ :meth:`search`. Defaults to ``["document_id"]``.
+ client_kwargs (`dict[str, Any] | None`, optional):
+ Extra keyword arguments forwarded to
+ :class:`~pymongo.AsyncMongoClient`.
+ """
+ self._uri = uri
+ self._database_name = database
+ self._distance = distance
+ self._index_name = index_name
+ self._filter_fields = filter_fields or ["document_id"]
+ self._client_kwargs = client_kwargs or {}
+ self._client: "AsyncMongoClient | None" = None
+
+ def get_client(self) -> "AsyncMongoClient":
+ """Lazily create and cache the async MongoDB client.
+
+ Returns:
+ `AsyncMongoClient`:
+ The shared async client instance.
+ """
+ if self._client is None:
+ from pymongo import AsyncMongoClient
+
+ self._client = AsyncMongoClient(
+ self._uri,
+ **self._client_kwargs,
+ )
+ return self._client
+
+ async def __aexit__(
+ self,
+ exc_type: type[BaseException] | None,
+ exc_val: BaseException | None,
+ exc_tb: Any,
+ ) -> None:
+ """Exit the async context — close the underlying client."""
+ if self._client is not None:
+ await self._client.close()
+ self._client = None
+
+ # ------------------------------------------------------------------
+ # Collection management
+ # ------------------------------------------------------------------
+
+ def _db(self) -> Any:
+ """Return the configured database handle."""
+ return self.get_client()[self._database_name]
+
+ def _col(self, collection: str) -> Any:
+ """Return a collection handle inside the configured database."""
+ return self._db()[collection]
+
+ async def create_collection(self, name: str, dimensions: int) -> None:
+ """Create a new MongoDB collection with a vector search index.
+
+ No-op if the collection already exists.
+
+ Args:
+ name (`str`):
+ The collection name. Typically, the knowledge base ID.
+ dimensions (`int`):
+ The fixed vector dimensionality for this collection.
+ """
+ if await self.has_collection(name):
+ return
+
+ from pymongo.operations import SearchIndexModel
+
+ await self._db().create_collection(name)
+ col = self._col(name)
+
+ fields: list[dict[str, Any]] = [
+ {
+ "type": "vector",
+ "path": "vector",
+ "numDimensions": dimensions,
+ "similarity": self._distance,
+ },
+ ]
+ fields.extend(
+ {"type": "filter", "path": field_path}
+ for field_path in self._filter_fields
+ )
+
+ model = SearchIndexModel(
+ definition={"fields": fields},
+ name=self._index_name,
+ type="vectorSearch",
+ )
+ await col.create_search_index(model)
+ await self._wait_for_index_ready(col, name)
+
+ async def delete_collection(self, name: str) -> None:
+ """Delete a collection and all its data.
+
+ Args:
+ name (`str`):
+ The collection name to delete.
+ """
+ await self._col(name).drop()
+
+ async def has_collection(self, name: str) -> bool:
+ """Check whether a collection exists.
+
+ Args:
+ name (`str`):
+ The collection name to check.
+
+ Returns:
+ `bool`: ``True`` if the collection exists.
+ """
+ return name in await self._db().list_collection_names()
+
+ async def _wait_for_index_ready(
+ self,
+ collection: Any,
+ collection_name: str,
+ timeout: float = 30.0,
+ ) -> None:
+ """Poll until the vector search index becomes queryable.
+
+ Args:
+ collection:
+ The MongoDB collection handle.
+ collection_name (`str`):
+ The collection name (used in timeout error messages).
+ timeout (`float`, defaults to ``30.0``):
+ Maximum seconds to wait before raising
+ :class:`TimeoutError`.
+
+ Raises:
+ `TimeoutError`:
+ If the index is not queryable within ``timeout`` seconds.
+ """
+ deadline = time.monotonic() + timeout
+ while time.monotonic() < deadline:
+ async for index in await collection.list_search_indexes(
+ self._index_name,
+ ):
+ if index.get("queryable"):
+ return
+ break
+ await asyncio.sleep(0.5)
+
+ raise TimeoutError(
+ f"Vector search index {self._index_name!r} on collection "
+ f"{collection_name!r} was not queryable within {timeout}s",
+ )
+
+ async def _ensure_index_ready(self, collection: str) -> None:
+ """Ensure the vector search index is queryable before reads."""
+ await self._wait_for_index_ready(self._col(collection), collection)
+
+ # ------------------------------------------------------------------
+ # Data operations
+ # ------------------------------------------------------------------
+
+ async def insert(
+ self,
+ collection: str,
+ records: list[VectorRecord],
+ ) -> None:
+ """Insert records into a collection.
+
+ Each document stores :attr:`VectorRecord.document_id` under the
+ ``document_id`` key and the serialized :class:`Chunk` under the
+ ``chunk`` key, so that :meth:`delete` can remove all records of
+ one document.
+
+ Args:
+ collection (`str`):
+ The target collection name.
+ records (`list[VectorRecord]`):
+ The records to insert (each carrying a
+ :class:`Chunk` and its embedding vector).
+ """
+ if not records:
+ return
+
+ from pymongo import ReplaceOne
+
+ col = self._col(collection)
+ operations = [
+ ReplaceOne(
+ {"_id": f"{record.document_id}_{record.chunk.chunk_index}"},
+ {
+ "_id": f"{record.document_id}_{record.chunk.chunk_index}",
+ "document_id": record.document_id,
+ "vector": record.vector,
+ "chunk": record.chunk.model_dump(mode="json"),
+ },
+ upsert=True,
+ )
+ for record in records
+ ]
+ await col.bulk_write(operations, ordered=False)
+
+ async def delete(
+ self,
+ collection: str,
+ document_id: str,
+ ) -> None:
+ """Delete all records belonging to one source document.
+
+ Matches the ``document_id`` field written by :meth:`insert` from
+ :attr:`VectorRecord.document_id`.
+
+ Args:
+ collection (`str`):
+ The target collection name.
+ document_id (`str`):
+ The source document ID whose records should be
+ removed.
+ """
+ await self._col(collection).delete_many({"document_id": document_id})
+
+ # ------------------------------------------------------------------
+ # Search
+ # ------------------------------------------------------------------
+
+ async def search(
+ self,
+ collection: str,
+ query_vector: list[float],
+ top_k: int = 5,
+ metadata_filter: dict[str, Any] | None = None,
+ ) -> list[VectorSearchResult]:
+ """Find the most similar records to a query vector.
+
+ Args:
+ collection (`str`):
+ The collection to search.
+ query_vector (`list[float]`):
+ The query embedding vector.
+ top_k (`int`, defaults to ``5``):
+ Maximum number of results to return.
+ metadata_filter (`dict[str, Any] | None`, optional):
+ If provided, restrict the search to records whose
+ ``chunk.metadata`` matches every ``key == value`` pair
+ in this dict (translated into a MongoDB filter against
+ ``chunk.metadata.<key>``). Each key must correspond to
+ a path declared in ``filter_fields``.
+
+ Returns:
+ `list[VectorSearchResult]`:
+ Results ordered by descending similarity score.
+ """
+ await self._ensure_index_ready(collection)
+ col = self._col(collection)
+
+ vector_search: dict[str, Any] = {
+ "index": self._index_name,
+ "path": "vector",
+ "queryVector": query_vector,
+ "numCandidates": max(100, top_k * 20),
+ "limit": top_k,
+ }
+ filter_expr = self._build_metadata_filter(metadata_filter)
+ if filter_expr is not None:
+ vector_search["filter"] = filter_expr
+
+ pipeline = [
+ {"$vectorSearch": vector_search},
+ {
+ "$project": {
+ "document_id": 1,
+ "chunk": 1,
+ "score": {"$meta": "vectorSearchScore"},
+ },
+ },
+ ]
+
+ results: list[VectorSearchResult] = []
+ async for doc in await col.aggregate(pipeline):
+ results.append(
+ VectorSearchResult(
+ score=float(doc["score"]),
+ document_id=doc["document_id"],
+ chunk=Chunk.model_validate(doc["chunk"]),
+ ),
+ )
+ return results
+
+ @staticmethod
+ def _build_metadata_filter(
+ metadata_filter: dict[str, Any] | None,
+ ) -> dict[str, Any] | None:
+ """Translate a flat ``{key: value}`` filter into a MongoDB filter.
+
+ Each ``key`` is matched against the corresponding nested path
+ ``chunk.metadata.<key>`` written by :meth:`insert`. Returns
+ ``None`` when ``metadata_filter`` is empty so that callers
+ skip the filter argument entirely.
+
+ Args:
+ metadata_filter (`dict[str, Any] | None`):
+ The flat filter, or ``None`` for no filter.
+
+ Returns:
+ `dict[str, Any] | None`:
+ A MongoDB filter document, or ``None``.
+ """
+ if not metadata_filter:
+ return None
+
+ return {
+ "$and": [
+ {f"chunk.metadata.{key}": {"$eq": value}}
+ for key, value in metadata_filter.items()
+ ],
+ }
+
+ # ------------------------------------------------------------------
+ # Document listing
+ # ------------------------------------------------------------------
+
+ async def list_documents(
+ self,
+ collection: str,
+ metadata_filter: dict[str, Any] | None = None,
+ ) -> list[DocumentSummary]:
+ """List all distinct source documents indexed in a collection.
+
+ Aggregates records by ``document_id``. The first chunk encountered
+ for each document supplies the ``source`` filename and the
+ document-level ``metadata``.
+
+ Args:
+ collection (`str`):
+ The target collection name.
+ metadata_filter (`dict[str, Any] | None`, optional):
+ If provided, restrict aggregation to records whose
+ ``chunk.metadata`` matches every ``key == value`` pair.
+
+ Returns:
+ `list[DocumentSummary]`:
+ One summary per distinct ``document_id``.
+ """
+ pipeline: list[dict[str, Any]] = []
+ if metadata_filter:
+ pipeline.append(
+ {
+ "$match": {
+ f"chunk.metadata.{key}": value
+ for key, value in metadata_filter.items()
+ },
+ },
+ )
+ pipeline.append(
+ {
+ "$group": {
+ "_id": "$document_id",
+ "source": {"$first": "$chunk.source"},
+ "metadata": {"$first": "$chunk.metadata"},
+ "chunk_count": {"$sum": 1},
+ },
+ },
+ )
+
+ summaries: list[DocumentSummary] = []
+ async for row in await self._col(collection).aggregate(pipeline):
+ summaries.append(
+ DocumentSummary(
+ document_id=row["_id"],
+ source=row["source"],
+ chunk_count=row["chunk_count"],
+ metadata=row["metadata"] or {},
+ ),
+ )
+ return summaries
diff --git a/tests/rag_vdb_mongodb_test.py b/tests/rag_vdb_mongodb_test.py
new file mode 100644
index 000000000..b4c3ed5a7
--- /dev/null
+++ b/tests/rag_vdb_mongodb_test.py
@@ -0,0 +1,647 @@
+# -*- coding: utf-8 -*-
+# pylint: disable=protected-access,missing-function-docstring
+"""Unit tests for the MongoDBStore class (mocked pymongo backend)."""
+from __future__ import annotations
+
+import math
+from contextlib import AsyncExitStack
+from typing import Any
+from unittest.async_case import IsolatedAsyncioTestCase
+from unittest.mock import patch
+
+from utils import AnyString
+
+from agentscope.message import TextBlock
+from agentscope.rag import (
+ Chunk,
+ MongoDBStore,
+ VectorRecord,
+ VectorSearchResult,
+)
+
+
+def _dump_results(results: list[VectorSearchResult]) -> list[dict]:
+ """Convert search results into plain dicts for whole-structure
+ comparison.
+
+ Args:
+ results (`list[VectorSearchResult]`):
+ The search results to convert.
+
+ Returns:
+ `list[dict]`:
+ The results as plain dicts.
+ """
+ return [result.model_dump() for result in results]
+
+
+def _make_record(
+ text: str,
+ vector: list[float],
+ document_id: str,
+ chunk_index: int = 0,
+ total_chunks: int = 1,
+) -> VectorRecord:
+ """Build a VectorRecord for testing.
+
+ Args:
+ text (`str`):
+ The chunk text content.
+ vector (`list[float]`):
+ The embedding vector.
+ document_id (`str`):
+ The ID of the source document the record belongs to.
+ chunk_index (`int`, defaults to ``0``):
+ The chunk index within the document.
+ total_chunks (`int`, defaults to ``1``):
+ The total number of chunks in the document.
+
+ Returns:
+ `VectorRecord`:
+ The constructed record.
+ """
+ return VectorRecord(
+ vector=vector,
+ document_id=document_id,
+ chunk=Chunk(
+ content=TextBlock(text=text),
+ source=f"{document_id}.txt",
+ chunk_index=chunk_index,
+ total_chunks=total_chunks,
+ ),
+ )
+
+
+def _cosine_similarity(a: list[float], b: list[float]) -> float:
+ """Cosine similarity aligned with Qdrant cosine scoring in tests."""
+ dot = sum(x * y for x, y in zip(a, b, strict=True))
+ norm_a = math.sqrt(sum(x * x for x in a))
+ norm_b = math.sqrt(sum(x * x for x in b))
+ if norm_a == 0.0 or norm_b == 0.0:
+ return 0.0
+ return dot / (norm_a * norm_b)
+
+
+class _FakeAsyncIterator:
+ """Minimal async iterator for mocked MongoDB cursors."""
+
+ def __init__(self, items: list[Any]) -> None:
+ self._items = items
+ self._index = 0
+
+ def __aiter__(self) -> "_FakeAsyncIterator":
+ return self
+
+ async def __anext__(self) -> Any:
+ if self._index >= len(self._items):
+ raise StopAsyncIteration
+ item = self._items[self._index]
+ self._index += 1
+ return item
+
+
+class _FakeMongoCollection:
+ """In-memory collection that implements the async API MongoDBStore uses."""
+
+ def __init__(self, database: "_FakeMongoDatabase", name: str) -> None:
+ self._database = database
+ self._name = name
+ self._docs: dict[str, dict[str, Any]] = {}
+ self._index_queryable = False
+
+ async def bulk_write(
+ self,
+ operations: list[Any],
+ ordered: bool = False,
+ ) -> None:
+ del ordered
+ for operation in operations:
+ self._docs[operation._doc["_id"]] = dict(operation._doc)
+
+ async def delete_many(self, filter_doc: dict[str, Any]) -> None:
+ document_id = filter_doc["document_id"]
+ to_remove = [
+ doc_id
+ for doc_id, doc in self._docs.items()
+ if doc.get("document_id") == document_id
+ ]
+ for doc_id in to_remove:
+ del self._docs[doc_id]
+
+ async def create_search_index(self, model: Any) -> None:
+ del model
+ self._index_queryable = True
+
+ async def list_search_indexes(self, name: str) -> _FakeAsyncIterator:
+ del name
+ return _FakeAsyncIterator([{"queryable": self._index_queryable}])
+
+ async def aggregate(
+ self,
+ pipeline: list[dict[str, Any]],
+ ) -> _FakeAsyncIterator:
+ return _FakeAsyncIterator(self._run_aggregate(pipeline))
+
+ async def drop(self) -> None:
+ self._database._collections.pop(self._name, None)
+
+ def _run_aggregate(
+ self,
+ pipeline: list[dict[str, Any]],
+ ) -> list[dict[str, Any]]:
+ if pipeline and "$vectorSearch" in pipeline[0]:
+ return self._vector_search(pipeline[0]["$vectorSearch"])
+ return self._list_documents(pipeline)
+
+ def _vector_search(self, stage: dict[str, Any]) -> list[dict[str, Any]]:
+ query_vector = stage["queryVector"]
+ top_k = stage["limit"]
+ metadata_filter = stage.get("filter")
+
+ scored: list[dict[str, Any]] = []
+ for doc in self._docs.values():
+ if not self._matches_metadata_filter(doc, metadata_filter):
+ continue
+ scored.append(
+ {
+ "document_id": doc["document_id"],
+ "chunk": doc["chunk"],
+ "score": _cosine_similarity(query_vector, doc["vector"]),
+ },
+ )
+
+ scored.sort(key=lambda item: item["score"], reverse=True)
+ return scored[:top_k]
+
+ def _list_documents(
+ self,
+ pipeline: list[dict[str, Any]],
+ ) -> list[dict[str, Any]]:
+ docs = list(self._docs.values())
+ for stage in pipeline:
+ if "$match" in stage:
+ docs = [
+ doc
+ for doc in docs
+ if self._matches_match_stage(doc, stage["$match"])
+ ]
+ elif "$group" in stage:
+ return self._group_documents(docs, stage["$group"])
+ return []
+
+ @staticmethod
+ def _matches_match_stage(
+ doc: dict[str, Any],
+ match: dict[str, Any],
+ ) -> bool:
+ for key, expected in match.items():
+ value = doc
+ for part in key.split("."):
+ value = value[part]
+ if value != expected:
+ return False
+ return True
+
+ @staticmethod
+ def _matches_metadata_filter(
+ doc: dict[str, Any],
+ metadata_filter: dict[str, Any] | None,
+ ) -> bool:
+ if not metadata_filter:
+ return True
+
+ for clause in metadata_filter.get("$and", []):
+ for key, condition in clause.items():
+ expected = condition["$eq"]
+ value = doc
+ for part in key.split("."):
+ value = value[part]
+ if value != expected:
+ return False
+ return True
+
+ @staticmethod
+ def _group_documents(
+ docs: list[dict[str, Any]],
+ group_stage: dict[str, Any],
+ ) -> list[dict[str, Any]]:
+ del group_stage
+ grouped: dict[str, dict[str, Any]] = {}
+ for doc in docs:
+ document_id = doc["document_id"]
+ if document_id not in grouped:
+ grouped[document_id] = {
+ "_id": document_id,
+ "source": doc["chunk"]["source"],
+ "metadata": doc["chunk"].get("metadata") or {},
+ "chunk_count": 0,
+ }
+ grouped[document_id]["chunk_count"] += 1
+ return list(grouped.values())
+
+
+class _FakeMongoDatabase:
+ """In-memory database handle."""
+
+ def __init__(self) -> None:
+ self._collections: dict[str, _FakeMongoCollection] = {}
+
+ async def list_collection_names(self) -> list[str]:
+ return list(self._collections.keys())
+
+ async def create_collection(self, name: str) -> _FakeMongoCollection:
+ collection = _FakeMongoCollection(self, name)
+ self._collections[name] = collection
+ return collection
+
+ def __getitem__(self, name: str) -> _FakeMongoCollection:
+ if name not in self._collections:
+ self._collections[name] = _FakeMongoCollection(self, name)
+ return self._collections[name]
+
+
+class _FakeMongoClient:
+ """In-memory async MongoDB client."""
+
+ def __init__(self, database_name: str) -> None:
+ self._database_name = database_name
+ self._databases: dict[str, _FakeMongoDatabase] = {}
+
+ def __getitem__(self, database_name: str) -> _FakeMongoDatabase:
+ if database_name not in self._databases:
+ self._databases[database_name] = _FakeMongoDatabase()
+ return self._databases[database_name]
+
+ async def close(self) -> None:
+ self._databases.clear()
+
+
+class MongoDBStoreTest(IsolatedAsyncioTestCase):
+ """The test cases for the MongoDBStore class."""
+
+ async def asyncSetUp(self) -> None:
+ """Create a MongoDB store with a mocked pymongo client."""
+ self._fake_client = _FakeMongoClient("test-db")
+ self._client_patcher = patch.object(
+ MongoDBStore,
+ "get_client",
+ return_value=self._fake_client,
+ )
+ self._client_patcher.start()
+
+ self._exit_stack = AsyncExitStack()
+ self.store = MongoDBStore(uri="mongodb://mock", database="test-db")
+ await self._exit_stack.enter_async_context(self.store)
+
+ async def asyncTearDown(self) -> None:
+ """Close the store and stop patches after each test."""
+ await self._exit_stack.aclose()
+ self._client_patcher.stop()
+
+ async def test_collection_lifecycle(self) -> None:
+ """Collections can be created, checked, and deleted."""
+ self.assertEqual(await self.store.has_collection("kb-1"), False)
+
+ await self.store.create_collection("kb-1", dimensions=3)
+ self.assertEqual(await self.store.has_collection("kb-1"), True)
+
+ # Creating an existing collection is a no-op
+ await self.store.create_collection("kb-1", dimensions=3)
+ self.assertEqual(await self.store.has_collection("kb-1"), True)
+
+ await self.store.delete_collection("kb-1")
+ self.assertEqual(await self.store.has_collection("kb-1"), False)
+
+ async def test_insert_and_search(self) -> None:
+ """Inserted records are searchable, ordered by similarity."""
+ await self.store.create_collection("kb-1", dimensions=3)
+ await self.store.insert(
+ "kb-1",
+ [
+ _make_record(
+ "Hello world!",
+ [1.0, 0.0, 0.0],
+ document_id="doc-1",
+ chunk_index=0,
+ total_chunks=2,
+ ),
+ _make_record(
+ "Goodbye world!",
+ [0.0, 1.0, 0.0],
+ document_id="doc-1",
+ chunk_index=1,
+ total_chunks=2,
+ ),
+ ],
+ )
+
+ results = await self.store.search(
+ "kb-1",
+ query_vector=[1.0, 0.0, 0.0],
+ top_k=2,
+ )
+
+ self.assertEqual(
+ _dump_results(results),
+ [
+ {
+ "score": 1.0,
+ "document_id": "doc-1",
+ "chunk": {
+ "content": {
+ "type": "text",
+ "text": "Hello world!",
+ "id": AnyString(),
+ },
+ "source": "doc-1.txt",
+ "chunk_index": 0,
+ "total_chunks": 2,
+ "metadata": {},
+ },
+ },
+ {
+ "score": 0.0,
+ "document_id": "doc-1",
+ "chunk": {
+ "content": {
+ "type": "text",
+ "text": "Goodbye world!",
+ "id": AnyString(),
+ },
+ "source": "doc-1.txt",
+ "chunk_index": 1,
+ "total_chunks": 2,
+ "metadata": {},
+ },
+ },
+ ],
+ )
+
+ async def test_search_top_k(self) -> None:
+ """top_k limits the number of returned results."""
+ await self.store.create_collection("kb-1", dimensions=3)
+ await self.store.insert(
+ "kb-1",
+ [
+ _make_record("A", [1.0, 0.0, 0.0], document_id="doc-1"),
+ _make_record("B", [0.9, 0.1, 0.0], document_id="doc-2"),
+ _make_record("C", [0.0, 0.0, 1.0], document_id="doc-3"),
+ ],
+ )
+
+ results = await self.store.search(
+ "kb-1",
+ query_vector=[1.0, 0.0, 0.0],
+ top_k=1,
+ )
+
+ self.assertEqual(
+ _dump_results(results),
+ [
+ {
+ "score": 1.0,
+ "document_id": "doc-1",
+ "chunk": {
+ "content": {
+ "type": "text",
+ "text": "A",
+ "id": AnyString(),
+ },
+ "source": "doc-1.txt",
+ "chunk_index": 0,
+ "total_chunks": 1,
+ "metadata": {},
+ },
+ },
+ ],
+ )
+
+ async def test_delete_by_document_id(self) -> None:
+ """delete removes all records of one document only."""
+ await self.store.create_collection("kb-1", dimensions=3)
+ await self.store.insert(
+ "kb-1",
+ [
+ _make_record(
+ "doc1-chunk0",
+ [1.0, 0.0, 0.0],
+ document_id="doc-1",
+ chunk_index=0,
+ total_chunks=2,
+ ),
+ _make_record(
+ "doc1-chunk1",
+ [0.9, 0.1, 0.0],
+ document_id="doc-1",
+ chunk_index=1,
+ total_chunks=2,
+ ),
+ _make_record(
+ "doc2-chunk0",
+ [0.0, 1.0, 0.0],
+ document_id="doc-2",
+ ),
+ ],
+ )
+
+ await self.store.delete("kb-1", document_id="doc-1")
+
+ results = await self.store.search(
+ "kb-1",
+ query_vector=[1.0, 0.0, 0.0],
+ top_k=5,
+ )
+
+ self.assertEqual(
+ _dump_results(results),
+ [
+ {
+ "score": 0.0,
+ "document_id": "doc-2",
+ "chunk": {
+ "content": {
+ "type": "text",
+ "text": "doc2-chunk0",
+ "id": AnyString(),
+ },
+ "source": "doc-2.txt",
+ "chunk_index": 0,
+ "total_chunks": 1,
+ "metadata": {},
+ },
+ },
+ ],
+ )
+
+ async def test_insert_empty_records(self) -> None:
+ """Inserting an empty record list is a no-op."""
+ await self.store.create_collection("kb-1", dimensions=3)
+ await self.store.insert("kb-1", [])
+
+ results = await self.store.search(
+ "kb-1",
+ query_vector=[1.0, 0.0, 0.0],
+ )
+
+ self.assertEqual(_dump_results(results), [])
+
+ async def test_list_documents_aggregates_by_document_id(self) -> None:
+ """list_documents groups chunks by document_id."""
+ await self.store.create_collection("kb-1", dimensions=3)
+
+ def _record_with_metadata(
+ text: str,
+ document_id: str,
+ metadata: dict,
+ chunk_index: int = 0,
+ total_chunks: int = 1,
+ ) -> VectorRecord:
+ return VectorRecord(
+ vector=[1.0, 0.0, 0.0],
+ document_id=document_id,
+ chunk=Chunk(
+ content=TextBlock(text=text),
+ source=metadata.get("filename", f"{document_id}.txt"),
+ chunk_index=chunk_index,
+ total_chunks=total_chunks,
+ metadata=metadata,
+ ),
+ )
+
+ await self.store.insert(
+ "kb-1",
+ [
+ _record_with_metadata(
+ "A",
+ "doc-1",
+ {"filename": "alpha.txt", "media_type": "text/plain"},
+ 0,
+ 2,
+ ),
+ _record_with_metadata(
+ "B",
+ "doc-1",
+ {"filename": "alpha.txt", "media_type": "text/plain"},
+ 1,
+ 2,
+ ),
+ _record_with_metadata(
+ "C",
+ "doc-2",
+ {"filename": "beta.md", "media_type": "text/markdown"},
+ 0,
+ 1,
+ ),
+ ],
+ )
+
+ summaries = sorted(
+ await self.store.list_documents("kb-1"),
+ key=lambda summary: summary.document_id,
+ )
+ self.assertEqual(
+ [summary.model_dump() for summary in summaries],
+ [
+ {
+ "document_id": "doc-1",
+ "source": "alpha.txt",
+ "chunk_count": 2,
+ "metadata": {
+ "filename": "alpha.txt",
+ "media_type": "text/plain",
+ },
+ },
+ {
+ "document_id": "doc-2",
+ "source": "beta.md",
+ "chunk_count": 1,
+ "metadata": {
+ "filename": "beta.md",
+ "media_type": "text/markdown",
+ },
+ },
+ ],
+ )
+
+ async def test_search_metadata_filter(self) -> None:
+ """search applies the metadata_filter as a payload predicate."""
+ await self.store.create_collection("kb-1", dimensions=3)
+
+ def _record(
+ text: str,
+ document_id: str,
+ kb_scope: str,
+ ) -> VectorRecord:
+ return VectorRecord(
+ vector=[1.0, 0.0, 0.0],
+ document_id=document_id,
+ chunk=Chunk(
+ content=TextBlock(text=text),
+ source=f"{document_id}.txt",
+ chunk_index=0,
+ total_chunks=1,
+ metadata={"kb_scope": kb_scope},
+ ),
+ )
+
+ await self.store.insert(
+ "kb-1",
+ [
+ _record("A", "doc-1", "kb-a"),
+ _record("B", "doc-2", "kb-b"),
+ ],
+ )
+
+ results = await self.store.search(
+ "kb-1",
+ query_vector=[1.0, 0.0, 0.0],
+ top_k=5,
+ metadata_filter={"kb_scope": "kb-a"},
+ )
+ self.assertEqual(
+ _dump_results(results),
+ [
+ {
+ "score": 1.0,
+ "document_id": "doc-1",
+ "chunk": {
+ "content": {
+ "type": "text",
+ "text": "A",
+ "id": AnyString(),
+ },
+ "source": "doc-1.txt",
+ "chunk_index": 0,
+ "total_chunks": 1,
+ "metadata": {"kb_scope": "kb-a"},
+ },
+ },
+ ],
+ )
+
+ results = await self.store.search(
+ "kb-1",
+ query_vector=[1.0, 0.0, 0.0],
+ top_k=5,
+ metadata_filter={"kb_scope": "kb-b"},
+ )
+ self.assertEqual(
+ _dump_results(results),
+ [
+ {
+ "score": 1.0,
+ "document_id": "doc-2",
+ "chunk": {
+ "content": {
+ "type": "text",
+ "text": "B",
+ "id": AnyString(),
+ },
+ "source": "doc-2.txt",
+ "chunk_index": 0,
+ "total_chunks": 1,
+ "metadata": {"kb_scope": "kb-b"},
+ },
+ },
+ ],
+ )