Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat: add Jev choice-based reranking (#5486)

chenxiaobin-monkey1394d4769a34
diff --git a/docs/en/configuration/01-server.md b/docs/en/configuration/01-server.md
index c19b91a6f..09454356d 100644
--- a/docs/en/configuration/01-server.md
+++ b/docs/en/configuration/01-server.md
@@ -144,13 +144,14 @@ Changing the model or `dimension` can make existing vector collections incompati
|---|---|---|---|
| `provider` | `vikingdb`, `cohere`, `openai`, `litellm`, `jev` / `null` | `null` | Rerank service; inferred from credentials when omitted |
| `model` | string / `null` | `null` | OpenAI-compatible, LiteLLM, or Jev rerank model |
+| `mode` | `noul`, `choice`, or `null` | `noul` | Jev rerank mode; `null` also uses `noul` |
| `threshold` | number | `0.1` | Minimum score considered relevant |
| `max_input_tokens` | integer; `0` or `>= 128` | `0` | Maximum estimated tokens per query-document pair; `0` disables truncation |
| `log_payloads` | boolean | `false` | Log complete rerank request and response payloads; may expose query and document content |
Rerank has no separate `enabled` field. It becomes available when the required provider credentials are configured.
-`jev` supports direct TypeSafe access (`https://api.typesafe.ai`, model `jev-latest`) and Vercel AI Gateway's TypeSafe-compatible endpoint (`https://ai-gateway.vercel.sh/typesafe`, model `typesafe-ai/jev`) through the existing `api_base` and `model` fields; both speak the same protocol. It sends the query and candidate documents as structured `state`, asks one independent relevance question per candidate, and uses each yes probability as its rerank score. Setting `provider` explicitly requires the credentials that provider needs: `ak` and `sk` for `vikingdb`, `api_key` for `cohere` and `jev`, `api_key` and `api_base` for `openai`, `model` for `litellm`. An incomplete block is rejected when the configuration loads.
+`jev` supports direct TypeSafe access (`https://api.typesafe.ai`, model `jev-latest`) and Vercel AI Gateway's TypeSafe-compatible endpoint (`https://ai-gateway.vercel.sh/typesafe`, model `typesafe-ai/jev`) through the existing `api_base` and `model` fields; both speak the same protocol. Its default `noul` mode scores each candidate independently. The optional `choice` mode compares all candidates and returns relative probabilities; set `threshold` to `0` when using it. Setting `provider` explicitly requires the credentials that provider needs: `ak` and `sk` for `vikingdb`, `api_key` for `cohere` and `jev`, `api_key` and `api_base` for `openai`, `model` for `litellm`. An incomplete block is rejected when the configuration loads.
## Retrieval Settings
diff --git a/docs/en/guides/01-configuration.md b/docs/en/guides/01-configuration.md
index 5586cac0b..597d21697 100644
--- a/docs/en/guides/01-configuration.md
+++ b/docs/en/guides/01-configuration.md
@@ -1072,10 +1072,21 @@ Notes for Vercel:
- `api_base` must include the `/typesafe` suffix. `https://ai-gateway.vercel.sh/v1`
is Vercel's own evaluate protocol and is not supported by the adapter.
-The Jev adapter sends the query and candidate documents as structured System One
-`state`, then asks one independent Noul relevance question per candidate. Each returned
-yes probability becomes that document's rerank score. All questions are evaluated in
-parallel in one request, and scores do not compete or have to sum to 1.
+The Jev adapter sends the query and candidate documents as shared System One `state`
+and supports two modes:
+
+- `noul` (default): one independent relevance question per document, using its returned yes
+ probability as the score. All questions are sent in one HTTP request, and document
+ scores do not have to sum to 1.
+- `choice`: one question compares all candidate documents, with `candidate_{index}`
+ mapped to input indices. Scores come from `probabilities` in document order.
+
+Select the mode with `rerank.mode`; omitting it or setting it to `null` keeps the
+default `noul` behavior. Choice scores are relative probabilities summing to 1 within
+the candidate pool, not absolute relevance probabilities, so they are unsuitable for
+absolute score filtering. When enabling Choice, set `"threshold": 0`. The MCP `find`
+and `search` `min_score` argument overrides that setting, so pass `min_score=0` when
+using Choice through MCP.
**Parameters**
@@ -1088,6 +1099,7 @@ parallel in one request, and scores do not compete or have to sum to 1.
| `api_key` | str | API key (for `openai`, `cohere`, or `jev` providers) |
| `api_base` | str | Endpoint URL (for `openai` or `jev`; Jev defaults to `https://api.typesafe.ai`, Vercel uses `https://ai-gateway.vercel.sh/typesafe`) |
| `model` | str | Model name for OpenAI-compatible, LiteLLM, or Jev providers |
+| `mode` | `"noul"`, `"choice"`, or `null` | Jev rerank mode. `null` and omission use `"noul"` |
| `timeout` | float | HTTP request timeout in seconds for HTTP rerank providers, including Jev. Default: `30.0` |
| `max_input_tokens` | int | Maximum estimated raw-text tokens in each query-document pair sent to the reranker. Oversized inputs retain their beginning and end. `0` disables. Default: `0` |
| `log_payloads` | bool | Log complete rerank request and response payloads. May expose query and document content. Default: `false` |
@@ -1099,7 +1111,7 @@ parallel in one request, and scores do not compete or have to sum to 1.
- `cohere`: Cohere Rerank API
- `openai`: OpenAI-compatible Rerank API
- `litellm`: LiteLLM Rerank API
-- `jev`: Jev (TypeSafe System One) structured-decision API; each document receives an independent Noul relevance score
+- `jev`: Jev (TypeSafe System One) structured-decision API with Choice comparison and independent Noul scoring
If rerank is not configured, search uses vector similarity only.
diff --git a/docs/zh/configuration/01-server.md b/docs/zh/configuration/01-server.md
index d9915f29d..5a9a225ae 100644
--- a/docs/zh/configuration/01-server.md
+++ b/docs/zh/configuration/01-server.md
@@ -144,13 +144,14 @@ API 型 `embedding`、`vlm`、`query_planner` 和 `rerank` 配置会复用部分
|---|---|---|---|
| `provider` | `vikingdb`、`cohere`、`openai`、`litellm`、`jev` / `null` | `null` | Rerank 服务类型;省略时根据凭证字段推断 |
| `model` | string / `null` | `null` | OpenAI 兼容、LiteLLM 或 Jev Rerank 模型 |
+| `mode` | `noul`、`choice` 或 `null` | `noul` | Jev Rerank 模式;`null` 同样使用 `noul` |
| `threshold` | number | `0.1` | 判定结果相关的最低分数 |
| `max_input_tokens` | integer;`0` 或 `>= 128` | `0` | 每个 query-document pair 的最大估算 token;`0` 表示不截断 |
| `log_payloads` | boolean | `false` | 记录完整 rerank 请求和响应;日志可能包含 query 和文档内容 |
Rerank 没有单独的 `enabled` 字段;配置了对应 provider 所需的凭证后才会启用。
-`jev` 通过现有 `api_base` 和 `model` 字段同时支持 TypeSafe 直连(`https://api.typesafe.ai`,模型 `jev-latest`)和 Vercel AI Gateway 的 TypeSafe 兼容端点(`https://ai-gateway.vercel.sh/typesafe`,模型 `typesafe-ai/jev`),两者协议相同。它将 query 和候选文档作为结构化 `state`,为每个候选提出一个独立的相关性问题,并将各自的 yes 概率作为 rerank 分数。显式指定 `provider` 时必须提供该 provider 所需的凭证:`vikingdb` 需要 `ak` 和 `sk`,`cohere` 和 `jev` 需要 `api_key`,`openai` 需要 `api_key` 和 `api_base`,`litellm` 需要 `model`。凭证不全的配置在加载时即被拒绝。
+`jev` 通过现有 `api_base` 和 `model` 字段同时支持 TypeSafe 直连(`https://api.typesafe.ai`,模型 `jev-latest`)和 Vercel AI Gateway 的 TypeSafe 兼容端点(`https://ai-gateway.vercel.sh/typesafe`,模型 `typesafe-ai/jev`),两者协议相同。默认 `noul` 模式独立评估每个候选;可选的 `choice` 模式横向比较所有候选并返回相对概率,使用时应将 `threshold` 设为 `0`。显式指定 `provider` 时必须提供该 provider 所需的凭证:`vikingdb` 需要 `ak` 和 `sk`,`cohere` 和 `jev` 需要 `api_key`,`openai` 需要 `api_key` 和 `api_base`,`litellm` 需要 `model`。凭证不全的配置在加载时即被拒绝。
## 检索配置
diff --git a/docs/zh/guides/01-configuration.md b/docs/zh/guides/01-configuration.md
index ab0c4461f..4264e3dba 100644
--- a/docs/zh/guides/01-configuration.md
+++ b/docs/zh/guides/01-configuration.md
@@ -1036,9 +1036,18 @@ PDF 解析配置。支持三种策略:`local`(本地 pdfplumber)、`mineru
- `api_base` 必须带 `/typesafe` 后缀;`https://ai-gateway.vercel.sh/v1` 是
Vercel 自有的 evaluate 协议,适配器不支持。
-Jev 适配器将 query 和候选文档作为结构化 System One `state`,并为每个候选
-提出一个独立的 Noul 相关性问题。每个问题返回的 yes 概率就是该文档的 rerank
-分数。所有问题在一次请求中并行计算,各文档分数互不竞争,也不要求总和为 1。
+Jev 适配器将 query 和候选文档作为共享的 System One `state`,支持两种模式:
+
+- `noul`(默认):为每篇文档提出独立的相关性问题,返回的 yes 概率作为分数。
+ 所有问题放在一次 HTTP 请求中,各文档分数不要求总和为 1。
+- `choice`:用一个问题横向比较所有候选文档,以 `candidate_{index}` 对应输入
+ 下标,并按原文档顺序读取 `probabilities` 作为分数。
+
+通过 `rerank.mode` 选择模式;未配置或配置为 `null` 时默认使用 `noul`。Choice
+返回候选池内总和为 1 的相对概率,不是绝对相关概率,不适合使用绝对分数阈值。
+启用 Choice 时建议同时设置 `"threshold": 0`。MCP `find` 和 `search` 的
+`min_score` 会覆盖该配置,因此通过 MCP 使用 Choice 时还需要显式传入
+`min_score=0`。
**参数**
@@ -1051,6 +1060,7 @@ Jev 适配器将 query 和候选文档作为结构化 System One `state`,并
| `api_key` | str | API Key(用于 `openai`、`cohere` 或 `jev` 提供方) |
| `api_base` | str | 接口地址(用于 `openai` 或 `jev`;Jev 默认为 `https://api.typesafe.ai`,Vercel 使用 `https://ai-gateway.vercel.sh/typesafe`) |
| `model` | str | 模型名称(用于 OpenAI 兼容、LiteLLM 或 `jev` 提供方) |
+| `mode` | `"noul"`、`"choice"` 或 `null` | Jev Rerank 模式;省略或 `null` 时使用 `"noul"` |
| `timeout` | float | HTTP Rerank provider(包括 Jev)的请求超时时间,单位为秒。默认:`30.0` |
| `max_input_tokens` | int | 每个 query-document 对发送给 reranker 的最大估算原始文本 token 数;超长输入会保留开头和结尾。`0` 表示不截断。默认:`0` |
| `log_payloads` | bool | 记录完整 rerank 请求和响应;日志可能包含 query 和文档内容。默认:`false` |
@@ -1062,7 +1072,7 @@ Jev 适配器将 query 和候选文档作为结构化 System One `state`,并
- `cohere`: Cohere Rerank API
- `openai`: OpenAI 兼容的 Rerank 接口
- `litellm`: LiteLLM Rerank 接口
-- `jev`: Jev (TypeSafe System One) 结构化判定接口,为每篇文档独立计算 Noul 相关性分数
+- `jev`: Jev (TypeSafe System One) 结构化判定接口,支持 Choice 横向比较和 Noul 独立评分
如果未配置 Rerank,搜索仅使用向量相似度。
diff --git a/openviking/models/rerank/jev_rerank.py b/openviking/models/rerank/jev_rerank.py
index 9f73831a3..351248cf2 100644
--- a/openviking/models/rerank/jev_rerank.py
+++ b/openviking/models/rerank/jev_rerank.py
@@ -3,9 +3,8 @@
"""
Jev (TypeSafe System One) Rerank API Client.
-TypeSafe has no native rerank endpoint; each document is evaluated by an
-independent Noul question against the query. The returned yes probability is
-used as the relevance score.
+Choice compares candidates in one question and returns relative probabilities.
+Noul evaluates each document independently and returns its yes probability.
Speaks only the TypeSafe System One protocol. Vercel AI Gateway exposes a
TypeSafe-compatible endpoint (https://ai-gateway.vercel.sh/typesafe), so routing
@@ -26,6 +25,81 @@ from openviking_cli.utils import get_logger
logger = get_logger(__name__)
+MODE_NOUL = "noul"
+MODE_CHOICE = "choice"
+
+
+def _validate_score(score: object) -> float:
+ if isinstance(score, bool) or not isinstance(score, (int, float)) or not 0 <= score <= 1:
+ raise ValueError(f"Invalid relevance score: {score!r}")
+ return float(score)
+
+
+class NoulRerankMode:
+ """One independent relevance question per document."""
+
+ def build_questions(self, documents: List[str]) -> Dict[str, dict]:
+ """Build one independent relevance question per document."""
+ return {
+ f"relevance_{index}": {
+ "type": MODE_NOUL,
+ "instructions": {
+ "candidate_index": index,
+ "question": (
+ "Does candidate_documents[candidate_index] answer or match "
+ "the retrieval intent of query?"
+ ),
+ },
+ "criteria": {
+ "true": "The candidate directly answers or is relevant to the query",
+ "false": "The candidate does not answer and is not relevant to the query",
+ },
+ }
+ for index in range(len(documents))
+ }
+
+ def parse_scores(self, answers: dict, document_count: int) -> List[float]:
+ scores = []
+ for index in range(document_count):
+ answer = answers.get(f"relevance_{index}")
+ if not isinstance(answer, dict) or answer.get("type") != MODE_NOUL:
+ raise ValueError(f"Missing or malformed Noul answer for item {index}")
+ scores.append(_validate_score(answer.get("noul")))
+ return scores
+
+
+class ChoiceRerankMode:
+ """One choice question comparing all documents."""
+
+ def build_questions(self, documents: List[str]) -> Dict[str, dict]:
+ return {
+ "relevance": {
+ "type": MODE_CHOICE,
+ "instructions": (
+ "Which candidate document best answers or matches the retrieval intent of query?"
+ ),
+ "criteria": {
+ f"candidate_{index}": (
+ f"candidate_documents[{index}] directly answers or is relevant to the query"
+ )
+ for index in range(len(documents))
+ },
+ }
+ }
+
+ def parse_scores(self, answers: dict, document_count: int) -> List[float]:
+ answer = answers.get("relevance")
+ if not isinstance(answer, dict) or answer.get("type") != MODE_CHOICE:
+ raise ValueError("Missing or malformed Choice answer")
+ probabilities = answer.get("probabilities")
+ if not isinstance(probabilities, dict):
+ raise ValueError("Choice answer has no probabilities object")
+ # Use candidate keys, never response iteration order or winner confidence.
+ return [
+ _validate_score(probabilities.get(f"candidate_{index}"))
+ for index in range(document_count)
+ ]
+
class JevRerankClient(RerankBase):
"""Jev rerank client — same interface as VikingDB RerankClient."""
@@ -37,8 +111,14 @@ class JevRerankClient(RerankBase):
api_base: str = "https://api.typesafe.ai",
timeout: float = 30.0,
log_payloads: bool = False,
+ mode: str = MODE_NOUL,
):
super().__init__()
+ modes = {MODE_NOUL: NoulRerankMode, MODE_CHOICE: ChoiceRerankMode}
+ if mode not in modes:
+ raise ValueError(f"Unknown Jev rerank mode: {mode!r}")
+ self.mode = mode
+ self._strategy = modes[mode]()
self.api_key = api_key
self.model_name = model_name
self.api_base = api_base.rstrip("/")
@@ -58,26 +138,6 @@ class JevRerankClient(RerankBase):
timeout=timeout,
)
- def _build_questions(self, documents: List[str]) -> Dict[str, dict]:
- """Build one independent relevance question per document."""
- return {
- f"relevance_{index}": {
- "type": "noul",
- "instructions": {
- "candidate_index": index,
- "question": (
- "Does candidate_documents[candidate_index] answer or match "
- "the retrieval intent of query?"
- ),
- },
- "criteria": {
- "true": "The candidate directly answers or is relevant to the query",
- "false": "The candidate does not answer and is not relevant to the query",
- },
- }
- for index in range(len(documents))
- }
-
def rerank_batch(self, query: str, documents: List[str]) -> Optional[List[float]]:
"""Rerank documents against a query using the Jev System One API.
@@ -87,14 +147,12 @@ class JevRerankClient(RerankBase):
if not documents:
return []
- questions = self._build_questions(documents)
- body = {
- "model": self.model_name,
- "state": {"query": query, "candidate_documents": documents},
- "questions": questions,
- }
-
try:
+ body = {
+ "model": self.model_name,
+ "state": {"query": query, "candidate_documents": documents},
+ "questions": self._strategy.build_questions(documents),
+ }
if self.log_payloads:
logger.warning(
"[JevRerank] Request items=%s payload=%s",
@@ -119,17 +177,7 @@ class JevRerankClient(RerankBase):
logger.warning("[JevRerank] Response has no answers object")
return None
- scores = []
- for index in range(len(documents)):
- answer = answers.get(f"relevance_{index}")
- if not isinstance(answer, dict) or answer.get("type") != "noul":
- logger.warning("[JevRerank] Missing or malformed answer for item %s", index)
- return None
- score = answer.get("noul")
- if not isinstance(score, (int, float)) or not 0 <= score <= 1:
- logger.warning("[JevRerank] Invalid relevance score for item %s", index)
- return None
- scores.append(float(score))
+ scores = self._strategy.parse_scores(answers, len(documents))
usage = data.get("usage") if isinstance(data.get("usage"), dict) else {}
input_tokens = usage.get("input_tokens", 0)
@@ -170,4 +218,5 @@ class JevRerankClient(RerankBase):
api_base=api_base,
timeout=config.timeout,
log_payloads=config.log_payloads,
+ mode=config.mode or MODE_NOUL,
)
diff --git a/openviking_cli/utils/config/rerank_config.py b/openviking_cli/utils/config/rerank_config.py
index 96454308c..62ece0963 100644
--- a/openviking_cli/utils/config/rerank_config.py
+++ b/openviking_cli/utils/config/rerank_config.py
@@ -30,6 +30,10 @@ class RerankConfig(BaseModel):
model: Optional[str] = Field(
default=None, description="Model name for OpenAI-compatible, LiteLLM, or Jev providers"
)
+ mode: Optional[str] = Field(
+ default="noul",
+ description="Jev rerank mode: 'noul' or 'choice'",
+ )
extra_headers: Optional[Dict[str, str]] = Field(
default=None, description="Extra HTTP headers for OpenAI-compatible providers"
@@ -84,6 +88,11 @@ class RerankConfig(BaseModel):
raise ValueError("Rerank max_input_tokens must be 0 or at least 128")
provider = self._effective_provider()
+ if provider == "jev" and self.mode is not None:
+ self.mode = self.mode.strip().lower()
+ if self.mode not in ("noul", "choice"):
+ raise ValueError("Jev rerank mode must be one of ['noul', 'choice']")
+
if provider and provider not in [
"vikingdb",
"cohere",
diff --git a/tests/unit/models/rerank/test_jev_rerank.py b/tests/unit/models/rerank/test_jev_rerank.py
index 25df84a03..6858e9dbe 100644
--- a/tests/unit/models/rerank/test_jev_rerank.py
+++ b/tests/unit/models/rerank/test_jev_rerank.py
@@ -8,6 +8,7 @@ import httpx
import pytest
from openviking.models.rerank import JevRerankClient, RerankClient
+from openviking.models.rerank.jev_rerank import MODE_CHOICE, MODE_NOUL
from openviking_cli.utils.config.rerank_config import RerankConfig
@@ -41,6 +42,7 @@ class TestJevRerankClient:
client = JevRerankClient(api_key="test-key")
scores = client.rerank_batch("What is UCW?", ["doc A", "doc B", "doc C"])
+ assert client.mode == MODE_NOUL
assert scores == [0.97, 0.04, 0.07]
body = mock_client.post.call_args[1]["json"]
assert body["model"] == "jev-latest"
@@ -64,6 +66,7 @@ class TestJevRerankClient:
api_key="vercel-key",
api_base="https://ai-gateway.vercel.sh/typesafe",
model_name="typesafe-ai/jev",
+ mode=MODE_NOUL,
)
assert client.rerank_batch("query", ["first", "second"]) == [0.91, 0.08]
@@ -80,7 +83,7 @@ class TestJevRerankClient:
mock_client_class.return_value = mock_client
mock_client.post.return_value = _mock_response(_systemone_payload([0.9]))
- client = JevRerankClient(api_key="secret-key", log_payloads=True)
+ client = JevRerankClient(api_key="secret-key", log_payloads=True, mode=MODE_NOUL)
assert client.rerank_batch("query", ["document"]) == [0.9]
messages = [str(call) for call in mock_log.call_args_list]
@@ -95,7 +98,7 @@ class TestJevRerankClient:
mock_client_class.return_value = mock_client
mock_client.post.return_value = _mock_response(_systemone_payload([0.9]))
- client = JevRerankClient(api_key="secret-key")
+ client = JevRerankClient(api_key="secret-key", mode=MODE_NOUL)
assert client.rerank_batch("query", ["document"]) == [0.9]
mock_log.assert_not_called()
@@ -111,7 +114,7 @@ class TestJevRerankClient:
error = httpx.HTTPStatusError("401", request=MagicMock(), response=MagicMock())
mock_client.post.return_value = _mock_response({}, status_error=error)
- client = JevRerankClient(api_key="bad-key")
+ client = JevRerankClient(api_key="bad-key", mode=MODE_NOUL)
assert client.rerank_batch("query", ["doc"]) is None
@patch("openviking.models.rerank.jev_rerank.httpx.Client")
@@ -122,7 +125,7 @@ class TestJevRerankClient:
{"answers": {}, "usage": {"input_tokens": 1, "output_tokens": 1}}
)
- client = JevRerankClient(api_key="key")
+ client = JevRerankClient(api_key="key", mode=MODE_NOUL)
assert client.rerank_batch("q", ["doc"]) is None
@patch("openviking.models.rerank.jev_rerank.httpx.Client")
@@ -131,7 +134,7 @@ class TestJevRerankClient:
mock_client_class.return_value = mock_client
mock_client.post.return_value = _mock_response(_systemone_payload([0.2, 0.8, 0.4]))
- client = JevRerankClient(api_key="key")
+ client = JevRerankClient(api_key="key", mode=MODE_NOUL)
scores = client.rerank_batch("q", ["a", "b", "c"])
assert scores == [0.2, 0.8, 0.4]
@@ -144,7 +147,7 @@ class TestJevRerankClient:
mock_client_class.return_value = mock_client
mock_client.post.return_value = _mock_response(_systemone_payload([1.1]))
- client = JevRerankClient(api_key="key")
+ client = JevRerankClient(api_key="key", mode=MODE_NOUL)
assert client.rerank_batch("q", ["doc"]) is None
@patch("openviking.models.rerank.jev_rerank.httpx.Client")
@@ -156,11 +159,141 @@ class TestJevRerankClient:
mock_client.close.assert_called_once()
+class TestChoiceRerank:
+ @patch("openviking.models.rerank.jev_rerank.httpx.Client")
+ def test_choice_preserves_document_order(self, mock_client_class):
+ mock_client = mock_client_class.return_value
+ mock_client.post.return_value = _mock_response(
+ {
+ "answers": {
+ "relevance": {
+ "type": "choice",
+ "choice": "candidate_1",
+ "confidence": 0.55,
+ "probabilities": {
+ "candidate_2": 0.2,
+ "candidate_1": 0.7,
+ "candidate_0": 0.1,
+ },
+ }
+ },
+ "usage": {"input_tokens": None, "output_tokens": 1},
+ }
+ )
+ client = JevRerankClient(api_key="key", mode=MODE_CHOICE)
+ assert client.mode == MODE_CHOICE
+ assert client.rerank_batch("query", ["first", "second", "third"]) == [0.1, 0.7, 0.2]
+ body = mock_client.post.call_args.kwargs["json"]
+ assert body["state"] == {
+ "query": "query",
+ "candidate_documents": ["first", "second", "third"],
+ }
+ assert body["questions"] == {
+ "relevance": {
+ "type": "choice",
+ "instructions": (
+ "Which candidate document best answers or matches the retrieval intent of query?"
+ ),
+ "criteria": {
+ "candidate_0": (
+ "candidate_documents[0] directly answers or is relevant to the query"
+ ),
+ "candidate_1": (
+ "candidate_documents[1] directly answers or is relevant to the query"
+ ),
+ "candidate_2": (
+ "candidate_documents[2] directly answers or is relevant to the query"
+ ),
+ },
+ }
+ }
+ mock_client.post.assert_called_once_with("https://api.typesafe.ai/v1/systemone", json=body)
+
+ @pytest.mark.parametrize("count", [1, 30])
+ @patch("openviking.models.rerank.jev_rerank.httpx.Client")
+ def test_choice_does_not_impose_candidate_limit(self, mock_client_class, count):
+ probabilities = {f"candidate_{index}": 1 / count for index in range(count)}
+ mock_client_class.return_value.post.return_value = _mock_response(
+ {"answers": {"relevance": {"type": "choice", "probabilities": probabilities}}}
+ )
+ client = JevRerankClient(api_key="key", mode=MODE_CHOICE)
+ assert client.rerank_batch("query", ["doc"] * count) == [1 / count] * count
+ criteria = mock_client_class.return_value.post.call_args.kwargs["json"]["questions"][
+ "relevance"
+ ]["criteria"]
+ assert list(criteria) == list(probabilities)
+
+ @patch("openviking.models.rerank.jev_rerank.httpx.Client")
+ def test_choice_empty_input_does_not_send_request(self, mock_client_class):
+ client = JevRerankClient(api_key="key", mode=MODE_CHOICE)
+ assert client.rerank_batch("query", []) == []
+ mock_client_class.return_value.post.assert_not_called()
+
+ @pytest.mark.parametrize(
+ "answer",
+ [
+ None,
+ {"type": "noul", "noul": 0.8},
+ {"type": "choice", "choice": "candidate_0", "confidence": 0.8},
+ {"type": "choice", "probabilities": []},
+ {"type": "choice", "probabilities": {"candidate_0": 0.8}},
+ {
+ "type": "choice",
+ "probabilities": {"candidate_0": 0.8, "candidate_1": "0.2"},
+ },
+ {"type": "choice", "probabilities": {"candidate_0": 0.8, "candidate_1": True}},
+ {"type": "choice", "probabilities": {"candidate_0": 0.8, "candidate_1": -0.1}},
+ {"type": "choice", "probabilities": {"candidate_0": 0.8, "candidate_1": 1.1}},
+ {
+ "type": "choice",
+ "probabilities": {"candidate_0": 0.8, "candidate_1": float("nan")},
+ },
+ ],
+ )
+ @patch("openviking.models.rerank.jev_rerank.httpx.Client")
+ def test_invalid_choice_answer_returns_none(self, mock_client_class, answer):
+ mock_client_class.return_value.post.return_value = _mock_response(
+ {"answers": {"relevance": answer}}
+ )
+ client = JevRerankClient(api_key="key", mode=MODE_CHOICE)
+ assert client.rerank_batch("query", ["first", "second"]) is None
+ mock_client_class.return_value.post.assert_called_once()
+
+ @patch("openviking.models.rerank.jev_rerank.httpx.Client")
+ def test_unknown_mode_rejected_before_client_creation(self, mock_client_class):
+ with pytest.raises(ValueError, match="Unknown Jev rerank mode"):
+ JevRerankClient(api_key="key", mode="unknown")
+ mock_client_class.assert_not_called()
+
+
class TestJevRerankConfig:
def test_explicit_jev_provider(self):
config = RerankConfig(provider="jev", api_key="key")
assert config._effective_provider() == "jev"
assert config.is_available() is True
+ assert config.mode == MODE_NOUL
+
+ @pytest.mark.parametrize("mode", [MODE_CHOICE, "CHOICE"])
+ def test_explicit_choice_mode(self, mode):
+ config = RerankConfig(provider="jev", api_key="key", mode=mode)
+ assert config.mode == MODE_CHOICE
+
+ def test_null_mode_uses_client_default(self):
+ config = RerankConfig(provider="jev", api_key="key", mode=None)
+ assert config.mode is None
+
+ def test_unknown_mode_rejected(self):
+ with pytest.raises(ValueError, match="Jev rerank mode"):
+ RerankConfig(provider="jev", api_key="key", mode="unknown")
+
+ def test_non_jev_provider_does_not_validate_jev_mode(self):
+ config = RerankConfig(
+ provider="openai",
+ api_key="key",
+ api_base="https://example.com/rerank",
+ mode="listwise",
+ )
+ assert config.mode == "listwise"
def test_jev_auto_detected_from_api_base(self):
config = RerankConfig(api_key="key", api_base="https://api.typesafe.ai")
@@ -191,6 +324,14 @@ class TestJevDispatch:
assert isinstance(client, JevRerankClient)
assert client.api_key == "jev-key"
assert client.model_name == "jev-latest"
+ assert client.mode == MODE_NOUL
+
+ @pytest.mark.parametrize("mode", [MODE_CHOICE, None])
+ @patch("openviking.models.rerank.jev_rerank.httpx.Client")
+ def test_from_config_resolves_mode(self, mock_client_class, mode):
+ config = RerankConfig(provider="jev", api_key="key", mode=mode)
+ client = RerankClient.from_config(config)
+ assert client.mode == (mode or MODE_NOUL)
@patch("openviking.models.rerank.jev_rerank.httpx.Client")
def test_from_config_uses_custom_model(self, mock_client_class):