Skip to content

Build and run agents you can see, understand and trust.

feat(rag): add elasticsearch vector store backend (#2129)

ouyangwulin5c6960714297
diff --git a/examples/rag/README.md b/examples/rag/README.md
index dc4589889..cfd661dd9 100644
--- a/examples/rag/README.md
+++ b/examples/rag/README.md
@@ -7,7 +7,7 @@ Two library-mode walk-throughs of `agentscope.rag` — no FastAPI service, no ma
| [`index_and_search.py`](./index_and_search.py) | The minimal pipeline: parse → chunk → embed → insert, then `KnowledgeBase.search`. Start here. |
| [`integrate_with_agent.py`](./integrate_with_agent.py) | Attaches the same `KnowledgeBase` to an `Agent` via `RAGMiddleware`, in both `static` (auto-inject) and `agentic` (tool-driven) modes. |
-Both examples use an in-memory Qdrant store (`location=":memory:"`) and the DashScope `text-embedding-v4` model, so no external services are required. The sections below show how to swap in Milvus Lite or MongoDB instead; those backends need additional setup.
+Both examples use an in-memory Qdrant store (`location=":memory:"`) and the DashScope `text-embedding-v4` model, so no external services are required. The sections below show how to swap in Milvus Lite, MongoDB, or Elasticsearch instead; those backends need additional setup.
## Install
@@ -109,17 +109,131 @@ async with store:
ensure each metadata key is listed in `filter_fields` as
`chunk.metadata.<key>` when constructing `MongoDBStore`.
- For the full FastAPI RAG service, pass the same `MongoDBStore` instance
- to `create_app(vector_store=...)` in `examples/agent_service/main.py`
- (the default there uses in-memory Qdrant for zero-setup demos).
+ to `CollectionPerKbManager(storage=..., vector_store=...)` in
+ `examples/agent_service/main.py` (the default there uses in-memory Qdrant
+ for zero-setup demos).
+
+### Elasticsearch
+
+To use Elasticsearch as the vector backend, install the optional async
+client extra:
+
+```bash
+uv pip install "agentscope[elasticsearch]"
+# Or from source (repo root)
+uv pip install -e ".[elasticsearch]"
+```
+
+**Prerequisites**
+
+- A reachable Elasticsearch 8.12+ deployment. The AgentScope extra uses
+ the official Elasticsearch 8.x Python client, which is compatible with
+ Elasticsearch 8.x and 9.x.
+- An Elasticsearch URL available in the environment. Keep credentials and
+ TLS settings outside source code.
+
+For local development, the following starts a single-node Elasticsearch
+instance with security disabled:
+
+```bash
+docker run --rm --name agentscope-elasticsearch \
+ -p 9200:9200 \
+ -e discovery.type=single-node \
+ -e xpack.security.enabled=false \
+ docker.elastic.co/elasticsearch/elasticsearch:8.19.3
+
+export ELASTICSEARCH_URL="http://localhost:9200"
+```
+
+Do not disable security in shared or production environments. For an
+authenticated HTTPS deployment, also export credentials and the CA
+certificate path, then pass them through `client_kwargs` as shown below.
+
+Replace the vector store construction in `index_and_search.py` and/or
+`integrate_with_agent.py`:
+
+```python
+import os
+
+from agentscope.rag import ElasticsearchStore
+
+client_kwargs = {}
+if os.getenv("ELASTICSEARCH_API_KEY"):
+ client_kwargs["api_key"] = os.environ["ELASTICSEARCH_API_KEY"]
+if os.getenv("ELASTICSEARCH_CA_CERTS"):
+ client_kwargs["ca_certs"] = os.environ["ELASTICSEARCH_CA_CERTS"]
+
+store = ElasticsearchStore(
+ hosts=os.environ["ELASTICSEARCH_URL"],
+ # Number of HNSW candidates considered per shard. Higher values can
+ # improve recall at the cost of additional search work.
+ num_candidates=100,
+ # Use False for higher write throughput during large imports when
+ # immediate search visibility is not required.
+ refresh="wait_for",
+ client_kwargs=client_kwargs,
+)
+
+# ElasticsearchStore implements the same async context-manager and
+# VectorStoreBase contracts as QdrantStore.
+async with store:
+ knowledge = KnowledgeBase(
+ name="demo-kb",
+ description="A toy corpus on cats and AgentScope.",
+ embedding_model=embedding_model,
+ vector_store=store,
+ collection=COLLECTION,
+ )
+ ...
+```
+
+For username/password authentication, configure `basic_auth` instead of an
+API key:
+
+```python
+store = ElasticsearchStore(
+ hosts=os.environ["ELASTICSEARCH_URL"],
+ client_kwargs={
+ "basic_auth": (
+ os.environ["ELASTICSEARCH_USERNAME"],
+ os.environ["ELASTICSEARCH_PASSWORD"],
+ ),
+ "ca_certs": os.environ["ELASTICSEARCH_CA_CERTS"],
+ },
+)
+```
+
+**Notes**
+
+- Each knowledge base collection maps to a separate Elasticsearch index.
+- Collections use an indexed `dense_vector` field with cosine similarity.
+ Keep the embedding model dimensions unchanged after an index is created.
+- Chunk metadata is stored in an Elasticsearch `flattened` field. Any
+ top-level metadata key can therefore be used with
+ `search(..., metadata_filter={"tenant_id": "bank-a"})`; no filter-field
+ declaration is required.
+- Writes use a stable ID derived from `document_id` and `chunk_index`, so
+ retrying the same indexing operation replaces existing chunks instead of
+ creating duplicates.
+- Writes default to `refresh="wait_for"`, making indexed chunks searchable
+ before the operation returns. For large imports, set `refresh=False` and
+ allow Elasticsearch's normal refresh interval (or an explicit index
+ refresh) to make changes visible with higher throughput.
+- Elasticsearch transforms cosine similarity into a positive `_score`.
+ `ElasticsearchStore` converts it back to raw cosine similarity so score
+ thresholds behave consistently with the other vector backends.
+- For service mode, construct
+ `CollectionPerKbManager(storage=storage, vector_store=store)` and pass it
+ to `create_app(knowledge_base_manager=...)`.
### Choosing a vector backend
-| | Qdrant (default) | Milvus Lite | MongoDB |
-| --- | --- | --- | --- |
-| Install extra | `agentscope[rag]` | `agentscope[milvuslite]` | `agentscope[mongodb]` |
-| External service | No | No | Yes |
-| Persistence | No (`:memory:`) | Yes (local `.db`) | Yes (server) |
-| Best for | Quick start / tests | Local dev with persistence | Teams already on MongoDB |
+| | Qdrant (default) | Milvus Lite | MongoDB | Elasticsearch |
+| --- | --- | --- | --- | --- |
+| Install extra | `agentscope[rag]` | `agentscope[milvuslite]` | `agentscope[mongodb]` | `agentscope[elasticsearch]` |
+| External service | No | No | Yes | Yes |
+| Persistence | No (`:memory:`) | Yes (local `.db`) | Yes (server) | Yes (server) |
+| Best for | Quick start / tests | Local dev with persistence | Teams already on MongoDB | Teams already on Elastic or needing distributed kNN search |
`integrate_with_agent.py` additionally uses `DashScopeChatModel`, which is already in the base `agentscope` dependencies.
@@ -132,7 +246,9 @@ python examples/rag/index_and_search.py
python examples/rag/integrate_with_agent.py
```
-When using MongoDB, also export `MONGODB_URI` before running.
+When using MongoDB, also export `MONGODB_URI` before running. When using
+Elasticsearch, export `ELASTICSEARCH_URL` and any required authentication or
+TLS environment variables.
## Service mode
diff --git a/pyproject.toml b/pyproject.toml
index cc0fe4425..881146cec 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -105,6 +105,10 @@ milvuslite = [
mongodb = [
"pymongo>=4.7",
]
+# ------------ Elasticsearch vector store ------------
+elasticsearch = [
+ "elasticsearch[async]>=8.12",
+]
# ------------ S3-compatible blob store ------------
s3 = [
@@ -133,6 +137,7 @@ full = [
"agentscope[rag]",
"agentscope[milvuslite]",
"agentscope[mongodb]",
+ "agentscope[elasticsearch]",
"agentscope[s3]",
"agentscope[mem0]",
"agentscope[reme]",
diff --git a/src/agentscope/rag/__init__.py b/src/agentscope/rag/__init__.py
index 659c073ab..b4500d1aa 100644
--- a/src/agentscope/rag/__init__.py
+++ b/src/agentscope/rag/__init__.py
@@ -17,6 +17,7 @@ from ._parser import (
)
from ._vdb import (
DocumentSummary,
+ ElasticsearchStore,
MilvusLiteStore,
VectorStoreBase,
VectorRecord,
@@ -31,6 +32,7 @@ __all__ = [
"ChunkerBase",
"Chunk",
"DocumentSummary",
+ "ElasticsearchStore",
"ImageParser",
"MilvusLiteStore",
"ParserBase",
diff --git a/src/agentscope/rag/_vdb/__init__.py b/src/agentscope/rag/_vdb/__init__.py
index 68c712032..c9e6ede61 100644
--- a/src/agentscope/rag/_vdb/__init__.py
+++ b/src/agentscope/rag/_vdb/__init__.py
@@ -10,9 +10,11 @@ from ._vector_store import (
from ._qdrant import QdrantStore
from ._mongodb import MongoDBStore
from ._milvus_lite import MilvusLiteStore
+from ._elasticsearch import ElasticsearchStore
__all__ = [
"DocumentSummary",
+ "ElasticsearchStore",
"MilvusLiteStore",
"VectorStoreBase",
"VectorRecord",
diff --git a/src/agentscope/rag/_vdb/_elasticsearch.py b/src/agentscope/rag/_vdb/_elasticsearch.py
new file mode 100644
index 000000000..c0084c1be
--- /dev/null
+++ b/src/agentscope/rag/_vdb/_elasticsearch.py
@@ -0,0 +1,297 @@
+# -*- coding: utf-8 -*-
+"""Elasticsearch implementation of the vector store backend."""
+from __future__ import annotations
+
+import hashlib
+from typing import TYPE_CHECKING, Any, Literal
+
+from .._document import Chunk
+from ._vector_store import (
+ DocumentSummary,
+ VectorRecord,
+ VectorSearchResult,
+ VectorStoreBase,
+)
+
+if TYPE_CHECKING:
+ from elasticsearch import AsyncElasticsearch
+
+
+class ElasticsearchStore(VectorStoreBase):
+ """Vector store backed by Elasticsearch dense-vector indexes.
+
+ Each knowledge base maps to one Elasticsearch index. Vectors use
+ cosine similarity and approximate k-nearest-neighbour search. Chunk
+ payloads are retained in ``_source`` while metadata is duplicated into
+ a ``flattened`` field so exact-match filters do not create unbounded
+ dynamic mappings.
+
+ .. note:: Requires the official async Elasticsearch client. Install it
+ with ``pip install agentscope[elasticsearch]``.
+ """
+
+ def __init__(
+ self,
+ hosts: str | list[str],
+ *,
+ num_candidates: int = 100,
+ refresh: bool | Literal["wait_for"] = "wait_for",
+ client_kwargs: dict[str, Any] | None = None,
+ ) -> None:
+ """Initialize the Elasticsearch vector store.
+
+ Args:
+ hosts (`str | list[str]`):
+ Elasticsearch URL or list of URLs.
+ num_candidates (`int`, defaults to ``100``):
+ Minimum HNSW candidates considered per shard. The effective
+ value is raised to ``top_k`` when necessary.
+ refresh (`bool | Literal["wait_for"]`, defaults to \
+ ``"wait_for"``):
+ Refresh policy for writes. Set to ``False`` for higher
+ indexing throughput when immediate search visibility is not
+ required. Elasticsearch's delete-by-query API only accepts a
+ boolean, so ``"wait_for"`` maps to ``True`` for deletes.
+ client_kwargs (`dict[str, Any] | None`, optional):
+ Extra arguments forwarded to ``AsyncElasticsearch`` such as
+ ``api_key``, ``basic_auth`` or ``ca_certs``.
+ """
+ if num_candidates <= 0 or num_candidates > 10_000:
+ raise ValueError("num_candidates must be between 1 and 10000")
+ self._hosts = hosts
+ self._num_candidates = num_candidates
+ self._refresh = refresh
+ self._client_kwargs = client_kwargs or {}
+ self._client: "AsyncElasticsearch | None" = None
+
+ def get_client(self) -> "AsyncElasticsearch":
+ """Lazily create and cache the shared async client."""
+ if self._client is None:
+ from elasticsearch import AsyncElasticsearch
+
+ self._client = AsyncElasticsearch(
+ self._hosts,
+ **self._client_kwargs,
+ )
+ return self._client
+
+ async def __aexit__(
+ self,
+ exc_type: type[BaseException] | None,
+ exc_val: BaseException | None,
+ exc_tb: Any,
+ ) -> None:
+ """Close the underlying client if this store created it."""
+ if self._client is not None:
+ await self._client.close()
+ self._client = None
+
+ async def create_collection(self, name: str, dimensions: int) -> None:
+ """Create an Elasticsearch index for one knowledge base."""
+ if await self.has_collection(name):
+ return
+ await self.get_client().indices.create(
+ index=name,
+ mappings={
+ "dynamic": False,
+ "properties": {
+ "vector": {
+ "type": "dense_vector",
+ "dims": dimensions,
+ "index": True,
+ "similarity": "cosine",
+ },
+ "document_id": {"type": "keyword"},
+ "chunk": {"type": "object", "enabled": False},
+ "metadata": {"type": "object", "dynamic": "runtime"},
+ },
+ },
+ )
+
+ async def delete_collection(self, name: str) -> None:
+ """Delete an Elasticsearch index and all its records."""
+ await self.get_client().indices.delete(index=name)
+
+ async def has_collection(self, name: str) -> bool:
+ """Return whether an Elasticsearch index exists."""
+ return bool(await self.get_client().indices.exists(index=name))
+
+ async def insert(
+ self,
+ collection: str,
+ records: list[VectorRecord],
+ ) -> None:
+ """Bulk-index records using deterministic, retry-safe IDs."""
+ if not records:
+ return
+
+ operations: list[dict[str, Any]] = []
+ for record in records:
+ operations.extend(
+ [
+ {
+ "index": {
+ "_index": collection,
+ "_id": self._record_id(record),
+ },
+ },
+ {
+ "vector": record.vector,
+ "document_id": record.document_id,
+ "chunk": record.chunk.model_dump(mode="json"),
+ "metadata": record.chunk.metadata,
+ },
+ ],
+ )
+
+ response = await self.get_client().bulk(
+ operations=operations,
+ refresh=self._refresh,
+ )
+ if response.get("errors"):
+ failures = [
+ item
+ for item in response.get("items", [])
+ if next(iter(item.values())).get("error")
+ ]
+ raise RuntimeError(
+ f"Elasticsearch bulk insert failed for {len(failures)} "
+ "record(s)",
+ )
+
+ async def delete(self, collection: str, document_id: str) -> None:
+ """Delete every chunk belonging to one source document."""
+ await self.get_client().delete_by_query(
+ index=collection,
+ query={"term": {"document_id": document_id}},
+ conflicts="proceed",
+ refresh=self._refresh is not False,
+ )
+
+ async def search(
+ self,
+ collection: str,
+ query_vector: list[float],
+ top_k: int = 5,
+ metadata_filter: dict[str, Any] | None = None,
+ ) -> list[VectorSearchResult]:
+ """Run an approximate cosine kNN search."""
+ if top_k <= 0:
+ return []
+ if top_k > 10_000:
+ raise ValueError("top_k cannot exceed Elasticsearch's 10000 limit")
+ num_candidates = min(
+ max(self._num_candidates, top_k),
+ 10_000,
+ )
+ knn: dict[str, Any] = {
+ "field": "vector",
+ "query_vector": query_vector,
+ "k": top_k,
+ "num_candidates": num_candidates,
+ }
+ filters = self._metadata_filters(metadata_filter)
+ if filters:
+ knn["filter"] = filters
+
+ response = await self.get_client().search(
+ index=collection,
+ size=top_k,
+ knn=knn,
+ source_includes=["document_id", "chunk"],
+ )
+ return [
+ VectorSearchResult(
+ score=2.0 * float(hit["_score"]) - 1.0,
+ document_id=hit["_source"]["document_id"],
+ chunk=Chunk.model_validate(hit["_source"]["chunk"]),
+ )
+ for hit in response["hits"]["hits"]
+ ]
+
+ async def list_documents(
+ self,
+ collection: str,
+ metadata_filter: dict[str, Any] | None = None,
+ ) -> list[DocumentSummary]:
+ """Aggregate documents with paginated composite buckets."""
+ summaries: list[DocumentSummary] = []
+ after: dict[str, Any] | None = None
+
+ while True:
+ composite: dict[str, Any] = {
+ "size": 500,
+ "sources": [
+ {
+ "document_id": {
+ "terms": {"field": "document_id"},
+ },
+ },
+ ],
+ }
+ if after is not None:
+ composite["after"] = after
+
+ response = await self.get_client().search(
+ index=collection,
+ size=0,
+ query=self._filter_query(metadata_filter),
+ aggs={
+ "documents": {
+ "composite": composite,
+ "aggs": {
+ "sample": {
+ "top_hits": {
+ "size": 1,
+ "_source": ["chunk"],
+ },
+ },
+ },
+ },
+ },
+ )
+ aggregation = response["aggregations"]["documents"]
+ buckets = aggregation["buckets"]
+ for bucket in buckets:
+ source = bucket["sample"]["hits"]["hits"][0]["_source"]
+ chunk = Chunk.model_validate(source["chunk"])
+ summaries.append(
+ DocumentSummary(
+ document_id=bucket["key"]["document_id"],
+ source=chunk.source,
+ chunk_count=bucket["doc_count"],
+ metadata=chunk.metadata,
+ ),
+ )
+ if not buckets or "after_key" not in aggregation:
+ break
+ after = aggregation["after_key"]
+
+ return summaries
+
+ @staticmethod
+ def _record_id(record: VectorRecord) -> str:
+ """Build a stable ID so re-indexing replaces the same chunk."""
+ raw = f"{record.document_id}\0{record.chunk.chunk_index}"
+ return hashlib.sha256(raw.encode("utf-8")).hexdigest()
+
+ @staticmethod
+ def _metadata_filters(
+ metadata_filter: dict[str, Any] | None,
+ ) -> list[dict[str, Any]]:
+ """Translate exact metadata matches into Elasticsearch filters."""
+ return [
+ {"term": {f"metadata.{key}": value}}
+ for key, value in (metadata_filter or {}).items()
+ ]
+
+ @classmethod
+ def _filter_query(
+ cls,
+ metadata_filter: dict[str, Any] | None,
+ ) -> dict[str, Any]:
+ """Build a query for metadata-filtered aggregations."""
+ filters = cls._metadata_filters(metadata_filter)
+ if not filters:
+ return {"match_all": {}}
+ return {"bool": {"filter": filters}}
diff --git a/tests/rag_vdb_elasticsearch_test.py b/tests/rag_vdb_elasticsearch_test.py
new file mode 100644
index 000000000..1f70f7f2c
--- /dev/null
+++ b/tests/rag_vdb_elasticsearch_test.py
@@ -0,0 +1,249 @@
+# -*- coding: utf-8 -*-
+# pylint: disable=protected-access,missing-function-docstring
+"""Unit tests for the ElasticsearchStore class."""
+from __future__ import annotations
+
+from contextlib import AsyncExitStack
+from typing import Any
+from unittest.async_case import IsolatedAsyncioTestCase
+from unittest.mock import AsyncMock, patch
+
+from agentscope.message import TextBlock
+from agentscope.rag import (
+ Chunk,
+ ElasticsearchStore,
+ VectorRecord,
+)
+
+
+def _record(
+ document_id: str,
+ chunk_index: int,
+ metadata: dict[str, Any] | None = None,
+) -> VectorRecord:
+ return VectorRecord(
+ vector=[1.0, 0.0, 0.0],
+ document_id=document_id,
+ chunk=Chunk(
+ content=TextBlock(text=f"chunk-{chunk_index}"),
+ source=f"{document_id}.txt",
+ chunk_index=chunk_index,
+ total_chunks=2,
+ metadata=metadata or {},
+ ),
+ )
+
+
+class _FakeIndices:
+ """Minimal asynchronous indices namespace."""
+
+ def __init__(self) -> None:
+ self.exists = AsyncMock(return_value=False)
+ self.create = AsyncMock()
+ self.delete = AsyncMock()
+
+
+class _FakeClient:
+ """Minimal asynchronous Elasticsearch client."""
+
+ def __init__(self) -> None:
+ self.indices = _FakeIndices()
+ self.bulk = AsyncMock(return_value={"errors": False, "items": []})
+ self.delete_by_query = AsyncMock()
+ self.search = AsyncMock()
+ self.close = AsyncMock()
+
+
+class ElasticsearchStoreTest(IsolatedAsyncioTestCase):
+ """Elasticsearch vector-store contract tests."""
+
+ async def asyncSetUp(self) -> None:
+ self.client = _FakeClient()
+ self.client_patcher = patch.object(
+ ElasticsearchStore,
+ "get_client",
+ return_value=self.client,
+ )
+ self.client_patcher.start()
+ self.exit_stack = AsyncExitStack()
+ self.store = ElasticsearchStore(hosts="http://localhost:9200")
+ await self.exit_stack.enter_async_context(self.store)
+
+ async def asyncTearDown(self) -> None:
+ await self.exit_stack.aclose()
+ self.client_patcher.stop()
+
+ async def test_collection_lifecycle(self) -> None:
+ self.assertFalse(await self.store.has_collection("kb-1"))
+
+ await self.store.create_collection("kb-1", dimensions=3)
+
+ self.client.indices.create.assert_awaited_once_with(
+ index="kb-1",
+ mappings={
+ "dynamic": False,
+ "properties": {
+ "vector": {
+ "type": "dense_vector",
+ "dims": 3,
+ "index": True,
+ "similarity": "cosine",
+ },
+ "document_id": {"type": "keyword"},
+ "chunk": {"type": "object", "enabled": False},
+ "metadata": {"type": "object", "dynamic": "runtime"},
+ },
+ },
+ )
+
+ await self.store.delete_collection("kb-1")
+ self.client.indices.delete.assert_awaited_once_with(index="kb-1")
+
+ async def test_insert_uses_stable_ids(self) -> None:
+ records = [_record("doc-1", 0), _record("doc-1", 1)]
+ await self.store.insert("kb-1", records)
+ first_operations = self.client.bulk.await_args.kwargs["operations"]
+
+ await self.store.insert("kb-1", records)
+ second_operations = self.client.bulk.await_args.kwargs["operations"]
+
+ self.assertEqual(first_operations, second_operations)
+ self.assertEqual(first_operations[0]["index"]["_index"], "kb-1")
+ self.assertNotEqual(
+ first_operations[0]["index"]["_id"],
+ first_operations[2]["index"]["_id"],
+ )
+ self.assertEqual(first_operations[1]["document_id"], "doc-1")
+
+ async def test_insert_empty_records_is_noop(self) -> None:
+ await self.store.insert("kb-1", [])
+ self.client.bulk.assert_not_awaited()
+
+ async def test_refresh_policy_can_disable_write_refreshes(self) -> None:
+ store = ElasticsearchStore(
+ hosts="http://localhost:9200",
+ refresh=False,
+ )
+
+ await store.insert("kb-1", [_record("doc-1", 0)])
+ await store.delete("kb-1", "doc-1")
+
+ self.assertIs(self.client.bulk.await_args.kwargs["refresh"], False)
+ self.assertIs(
+ self.client.delete_by_query.await_args.kwargs["refresh"],
+ False,
+ )
+
+ async def test_insert_surfaces_bulk_item_failures(self) -> None:
+ self.client.bulk.return_value = {
+ "errors": True,
+ "items": [{"index": {"error": {"type": "mapper_error"}}}],
+ }
+
+ with self.assertRaisesRegex(RuntimeError, "1 record"):
+ await self.store.insert("kb-1", [_record("doc-1", 0)])
+
+ async def test_delete_by_document_id(self) -> None:
+ await self.store.delete("kb-1", "doc-1")
+
+ self.client.delete_by_query.assert_awaited_once_with(
+ index="kb-1",
+ query={"term": {"document_id": "doc-1"}},
+ conflicts="proceed",
+ refresh=True,
+ )
+
+ async def test_search_with_metadata_filter(self) -> None:
+ chunk = _record("doc-1", 0, {"tenant": "bank-a"}).chunk
+ self.client.search.return_value = {
+ "hits": {
+ "hits": [
+ {
+ "_score": 0.95,
+ "_source": {
+ "document_id": "doc-1",
+ "chunk": chunk.model_dump(mode="json"),
+ },
+ },
+ ],
+ },
+ }
+
+ results = await self.store.search(
+ "kb-1",
+ [1.0, 0.0, 0.0],
+ top_k=5,
+ metadata_filter={"tenant": "bank-a"},
+ )
+
+ self.client.search.assert_awaited_once_with(
+ index="kb-1",
+ size=5,
+ knn={
+ "field": "vector",
+ "query_vector": [1.0, 0.0, 0.0],
+ "k": 5,
+ "num_candidates": 100,
+ "filter": [{"term": {"metadata.tenant": "bank-a"}}],
+ },
+ source_includes=["document_id", "chunk"],
+ )
+ self.assertEqual(results[0].document_id, "doc-1")
+ # Elasticsearch maps cosine to (1 + cosine) / 2. The store
+ # normalizes it back to the raw cosine used by other backends.
+ self.assertAlmostEqual(results[0].score, 0.9)
+
+ async def test_search_rejects_top_k_above_elasticsearch_limit(
+ self,
+ ) -> None:
+ with self.assertRaisesRegex(ValueError, "10000"):
+ await self.store.search("kb-1", [1.0, 0.0, 0.0], top_k=10_001)
+ self.client.search.assert_not_awaited()
+
+ async def test_list_documents_uses_composite_pagination(self) -> None:
+ chunk = _record("doc-1", 0, {"tenant": "bank-a"}).chunk
+ self.client.search.side_effect = [
+ {
+ "aggregations": {
+ "documents": {
+ "buckets": [
+ {
+ "key": {"document_id": "doc-1"},
+ "doc_count": 2,
+ "sample": {
+ "hits": {
+ "hits": [
+ {
+ "_source": {
+ "chunk": chunk.model_dump(
+ mode="json",
+ ),
+ },
+ },
+ ],
+ },
+ },
+ },
+ ],
+ "after_key": {"document_id": "doc-1"},
+ },
+ },
+ },
+ {"aggregations": {"documents": {"buckets": []}}},
+ ]
+
+ summaries = await self.store.list_documents(
+ "kb-1",
+ metadata_filter={"tenant": "bank-a"},
+ )
+
+ self.assertEqual(len(summaries), 1)
+ self.assertEqual(summaries[0].document_id, "doc-1")
+ self.assertEqual(summaries[0].chunk_count, 2)
+ self.assertEqual(summaries[0].metadata, {"tenant": "bank-a"})
+ self.assertEqual(self.client.search.await_count, 2)
+ second_query = self.client.search.await_args_list[1].kwargs
+ self.assertEqual(
+ second_query["aggs"]["documents"]["composite"]["after"],
+ {"document_id": "doc-1"},
+ )