Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

fix(benchmark): load WIKI-Dir path mapping (#3279)

Yuanqing ZHAO546da35cc8af
diff --git a/benchmark/vectordb_perf/README.md b/benchmark/vectordb_perf/README.md
index 913daabf4..6a470f3e7 100644
--- a/benchmark/vectordb_perf/README.md
+++ b/benchmark/vectordb_perf/README.md
@@ -101,23 +101,38 @@ mkdir -p benchmark/data/dir-vector-dataset-files
runner 会先校验 `wiki` 和 `arxiv` 所需文件;缺文件时会直接列出 missing files 和下载地址。
调试时可以用 `--dataset wiki` 或 `--dataset arxiv` 只跑单个数据集。各 dataset 对应文件见下方“真实数据文件”。
-真实数据少量抽样:
+真实数据少量抽样(Wiki 公开文件):
```bash
.venv/bin/python -m benchmark.vectordb_perf.run \
--config ./ov.conf \
--workload dir-vector \
+ --dataset wiki \
--dataset-root benchmark/data/dir-vector-dataset-files \
+ --dir-vector-query-scope derived_gt_lca_v1 \
--profile smoke
```
-真实数据全量:
+WIKI-Dir 的 corpus 目录来自单独发布的 `dbpedia_dir_2m_corpus_paths.json`,runner 会按
+`id` 将它与 corpus metadata 和 fvecs 逐行对齐,并在错位或缺行时直接报错。这个文件约
+600 MB,runner 使用流式解析,不会把全部 mapping 一次性加载到内存。
+
+当前公开的 Wiki query metadata 没有包含目录 constraint,因此默认的 `dataset` 口径会拒绝
+把所有 query 静默当成根目录查询。若要对公开文件做目录过滤的诊断性测试,可以显式使用
+`derived_gt_lca_v1`:对每个 query 的最高 relevance ground-truth 文档取目录最长公共前缀,
+并跳过只能得到根目录的 query。这个口径使用了 ground truth,不代表发布数据集的官方 query
+分布,也不能用来声明官方 Directory recall。该选项只改变 Wiki;在 `--dataset all` 中,
+其他 dataset 保持现有 loader 行为。
+
+真实数据全量(公开 Wiki 文件的诊断性目录口径):
```bash
.venv/bin/python -m benchmark.vectordb_perf.run \
--config ./ov.conf \
--workload dir-vector \
+ --dataset wiki \
--dataset-root ./benchmark/data/dir-vector-dataset-files \
+ --dir-vector-query-scope derived_gt_lca_v1 \
--full
```
@@ -135,7 +150,9 @@ runner 会把配置里的 `name` 改成 `<name>_bench_<run-id>`,避免覆盖
.venv/bin/python -m benchmark.vectordb_perf.run \
--config ./ov.conf \
--workload dir-vector \
+ --dataset wiki \
--dataset-root ./benchmark/data/dir-vector-dataset-files \
+ --dir-vector-query-scope derived_gt_lca_v1 \
--full \
--run-id ovbench_full_001 \
--mode write-only
@@ -147,7 +164,9 @@ runner 会把配置里的 `name` 改成 `<name>_bench_<run-id>`,避免覆盖
.venv/bin/python -m benchmark.vectordb_perf.run \
--config ./ov.conf \
--workload dir-vector \
+ --dataset wiki \
--dataset-root ./benchmark/data/dir-vector-dataset-files \
+ --dir-vector-query-scope derived_gt_lca_v1 \
--full \
--run-id ovbench_full_001 \
--mode read-only \
@@ -206,7 +225,7 @@ upsert;需要保留未提供字段的 partial update 仍使用单条 `upsert`
进度输出示例:
```text
-[vectordb_perf] start run_id=ovbench_full_001 workload=dir-vector:wiki mode=write-only records=1941679 queries=456 dim=1024
+[vectordb_perf] start run_id=ovbench_full_001 workload=dir-vector:wiki mode=write-only records=1941679 base_queries=456 directory_queries=217 dim=1024
[vectordb_perf] collection=vectordb_perf_bench_ovbench_full_001_wiki
[vectordb_perf] setup: create collection
[vectordb_perf] setup: collection ready
@@ -236,7 +255,7 @@ upsert;需要保留未提供字段的 partial update 仍使用单条 `upsert`
| `--dataset` | 期望文件 |
| --- | --- |
-| `wiki` | `dbpedia_dir_2m_corpus.jsonl`、`dbpedia_dir_2m_corpus_vectors.fvecs`、`dbpedia_dir_2m_query.jsonl`、`dbpedia_dir_2m_query_vectors.fvecs`、`dbpedia_dir_2m_groundtruth.tsv` |
+| `wiki` | `dbpedia_dir_2m_corpus.jsonl`、`dbpedia_dir_2m_corpus_paths.json`、`dbpedia_dir_2m_corpus_vectors.fvecs`、`dbpedia_dir_2m_query.jsonl`、`dbpedia_dir_2m_query_vectors.fvecs`、`dbpedia_dir_2m_groundtruth.tsv` |
| `arxiv` | `arxiv_corpus_metadata.json`、`arxiv_corpus_vectors.fvecs`、`arxiv_query_constraint.json`、`arxiv_query_vectors.fvecs`、`arxiv_ground_truth.txt` |
| `arxiv_category` | `arxiv_corpus_metadata.json`、`arxiv_corpus_vectors.fvecs`、`arxiv_category_query_constraint.json`、`arxiv_category_query_vectors.fvecs`、`arxiv_category_ground_truth.txt` |
@@ -303,6 +322,7 @@ Volcengine VikingDB:
| `--batch-size` | 每次 backend `upsert_many` 调用的记录数;远端 adapter 可按请求上限再拆分 |
| `--concurrency` | 查询并发 |
| `--top-k` | 检索返回条数 |
+| `--dir-vector-query-scope` | dir-vector 目录 query 来源;默认 `dataset`,Wiki 公开文件可显式使用诊断口径 `derived_gt_lca_v1` |
| `--distance` | `ip`、`l2`、`cosine` |
| `--drop-at-end` | 运行结束后删除测试 collection |
@@ -326,8 +346,10 @@ Volcengine VikingDB:
| 数据规模 | `records` 是本次计划读取的数据条数,`inserted` 是实际写入 backend 的条数,`queries` 是实际查询数 |
| 召回与 QPS | `vector_search` 和 `filtered_vector_search` 的 QPS、平均延迟、P95 延迟、gt_recall@K |
-gt_recall@K 按 query 的 ground truth 命中率统计。`--full` 才是官方全量 recall;
-非 `--full` 的真实数据报告会标成 `sampled_subset`,只用于 smoke sanity,不代表官方全量 recall。
+gt_recall@K 按 query 的 ground truth 命中率统计。全量 Wiki 的无过滤 `vector_search` 保留
+`official_full` 标记;其 `filtered_vector_search` 在诊断模式下标为 `derived_gt_lca_v1`。
+非 `--full` 会标成 `sampled_subset`。派生目录和抽样口径只用于诊断,不代表官方 Directory
+recall。
## 资源限制口径
diff --git a/benchmark/vectordb_perf/run.py b/benchmark/vectordb_perf/run.py
index a221da7c3..d37e2c275 100644
--- a/benchmark/vectordb_perf/run.py
+++ b/benchmark/vectordb_perf/run.py
@@ -19,6 +19,7 @@ import os
import platform
import random
import shutil
+import statistics
import struct
import subprocess
import sys
@@ -42,6 +43,7 @@ BENCH_USER_ID = "bench_user"
DIR_VECTOR_DATASET_URL = "https://github.com/KurtPatrickHere/dir-vector-dataset"
DEFAULT_DIR_VECTOR_DATASETS = ("wiki", "arxiv")
PROGRESS_INTERVAL_SECONDS = 5.0
+DIR_VECTOR_QUERY_SCOPES = ("dataset", "derived_gt_lca_v1")
PROFILE_DEFAULTS: dict[str, dict[str, Any]] = {
@@ -83,6 +85,7 @@ PROFILE_DEFAULTS: dict[str, dict[str, Any]] = {
DIR_VECTOR_FILES = {
"wiki": {
"corpus": "dbpedia_dir_2m_corpus.jsonl",
+ "corpus_paths": "dbpedia_dir_2m_corpus_paths.json",
"vectors": "dbpedia_dir_2m_corpus_vectors.fvecs",
"queries": "dbpedia_dir_2m_query.jsonl",
"query_vectors": "dbpedia_dir_2m_query_vectors.fvecs",
@@ -128,6 +131,7 @@ class BenchOptions:
seed: int
distance: Optional[str]
drop_at_end: bool
+ dir_vector_query_scope: str
@dataclass
@@ -138,6 +142,12 @@ class QueryCase:
ground_truth_ids: list[str] = field(default_factory=list)
+@dataclass(frozen=True)
+class GroundTruthItem:
+ corpus_id: str
+ score: float = 1.0
+
+
@dataclass
class Workload:
name: str
@@ -145,6 +155,8 @@ class Workload:
records: Callable[[], Iterator[dict[str, Any]]]
queries: list[QueryCase]
expected_rows: Optional[int]
+ directory_queries: Optional[list[QueryCase]] = None
+ path_diagnostics: dict[str, Any] = field(default_factory=dict)
@dataclass
@@ -173,6 +185,10 @@ class RunResult:
kept_collection: bool
+def directory_queries(workload: Workload) -> list[QueryCase]:
+ return workload.queries if workload.directory_queries is None else workload.directory_queries
+
+
def utc_now() -> str:
return datetime.now(UTC).isoformat()
@@ -312,6 +328,106 @@ def iter_fvecs(path: Path, limit: Optional[int] = None) -> Iterator[list[float]]
count += 1
+def iter_json_array_records(
+ path: Path,
+ limit: Optional[int] = None,
+ *,
+ chunk_size: int = 1024 * 1024,
+) -> Iterator[dict[str, Any]]:
+ """Stream a top-level JSON array without materializing the whole file."""
+ if chunk_size <= 0:
+ raise ValueError("chunk_size must be positive")
+ if limit is not None and limit <= 0:
+ return
+
+ decoder = json.JSONDecoder()
+ with path.open("r", encoding="utf-8") as handle:
+ buffer = ""
+ position = 0
+ eof = False
+ started = False
+ expect_value = True
+ after_comma = False
+ yielded = 0
+
+ def refill() -> None:
+ nonlocal buffer, position, eof
+ if position:
+ buffer = buffer[position:]
+ position = 0
+ chunk = handle.read(chunk_size)
+ if chunk:
+ buffer += chunk
+ else:
+ eof = True
+
+ while True:
+ if position >= len(buffer) and not eof:
+ refill()
+ while position < len(buffer) and buffer[position].isspace():
+ position += 1
+ if position >= len(buffer):
+ if eof:
+ raise ValueError(f"truncated JSON array in {path.name}")
+ refill()
+ continue
+
+ if not started:
+ if buffer[position] != "[":
+ raise ValueError(f"expected a top-level JSON array in {path.name}")
+ position += 1
+ started = True
+ continue
+
+ if not expect_value:
+ if buffer[position] == ",":
+ position += 1
+ expect_value = True
+ after_comma = True
+ continue
+ if buffer[position] == "]":
+ position += 1
+ while True:
+ while position < len(buffer) and buffer[position].isspace():
+ position += 1
+ if position < len(buffer):
+ raise ValueError(f"unexpected content after JSON array in {path.name}")
+ if eof:
+ return
+ refill()
+ raise ValueError(f"expected ',' or ']' in {path.name}")
+
+ if buffer[position] == "]":
+ if after_comma:
+ raise ValueError(f"trailing comma in JSON array in {path.name}")
+ position += 1
+ while True:
+ while position < len(buffer) and buffer[position].isspace():
+ position += 1
+ if position < len(buffer):
+ raise ValueError(f"unexpected content after JSON array in {path.name}")
+ if eof:
+ return
+ refill()
+
+ try:
+ item, end = decoder.raw_decode(buffer, position)
+ except json.JSONDecodeError as exc:
+ if eof:
+ raise ValueError(f"invalid JSON array in {path.name}: {exc}") from exc
+ refill()
+ continue
+ position = end
+ expect_value = False
+ after_comma = False
+ if not isinstance(item, dict):
+ raise ValueError(f"expected JSON objects in {path.name}")
+ yield item
+ yielded += 1
+ if limit is not None and yielded >= limit:
+ return
+
+
def iter_json_records(path: Path, limit: Optional[int] = None) -> Iterator[dict[str, Any]]:
if path.suffix == ".jsonl":
with path.open("r", encoding="utf-8") as handle:
@@ -325,6 +441,18 @@ def iter_json_records(path: Path, limit: Optional[int] = None) -> Iterator[dict[
yield item
return
+ with path.open("r", encoding="utf-8") as handle:
+ first = ""
+ while not first:
+ char = handle.read(1)
+ if not char:
+ raise ValueError(f"empty JSON file: {path}")
+ if not char.isspace():
+ first = char
+ if first == "[":
+ yield from iter_json_array_records(path, limit)
+ return
+
data = json.loads(path.read_text(encoding="utf-8"))
if isinstance(data, dict):
iterable: Iterable[Any] = data.values()
@@ -364,25 +492,46 @@ def normalize_path(path: Any) -> str:
return stripped.rstrip("/") if len(stripped) > 1 else stripped
-def extract_path(record: dict[str, Any]) -> str:
- value = pick_value(
- record,
- (
- "dir_path",
- "dir_paths",
- "path",
- "paths",
- "directory",
- "directories",
- "category_path",
- "category_paths",
- "date_path",
- ),
- )
+GENERIC_PATH_FIELDS = (
+ "dir_path",
+ "dir_paths",
+ "path",
+ "paths",
+ "directory",
+ "directories",
+ "category_path",
+ "category_paths",
+ "date_path",
+)
+
+
+def extract_declared_path(
+ record: dict[str, Any], fields: Sequence[str] = GENERIC_PATH_FIELDS
+) -> Optional[str]:
+ """Return only explicitly declared path metadata, never arbitrary prose."""
+ value = pick_value(record, fields)
if value is not None:
return normalize_path(value)
+ for container_name in ("constraints", "constraint", "metadata"):
+ nested = record.get(container_name)
+ if isinstance(nested, dict):
+ value = pick_value(nested, fields)
+ if value is not None:
+ return normalize_path(value)
+ elif container_name == "constraint" and isinstance(nested, (str, list)):
+ return normalize_path(nested)
+ return None
+
- # Last resort for dataset variants with nested constraint metadata.
+def extract_path(record: dict[str, Any]) -> str:
+ return extract_declared_path(record) or "/"
+
+
+def extract_legacy_path(record: dict[str, Any]) -> str:
+ """Keep the pre-existing best-effort behavior for non-Wiki datasets."""
+ declared = extract_declared_path(record)
+ if declared is not None:
+ return declared
for value in record.values():
if isinstance(value, str) and "/" in value:
return normalize_path(value)
@@ -396,7 +545,17 @@ def extract_path(record: dict[str, Any]) -> str:
def record_id(record: dict[str, Any], fallback: int) -> str:
value = pick_value(
record,
- ("id", "entity_id", "paper_id", "doc_id", "item_id", "source_id", "qid", "query_id"),
+ (
+ "id",
+ "_id",
+ "entity_id",
+ "paper_id",
+ "doc_id",
+ "item_id",
+ "source_id",
+ "qid",
+ "query_id",
+ ),
)
return str(value) if value is not None else str(fallback)
@@ -419,6 +578,65 @@ def path_matches(path_value: Any, prefix: str) -> bool:
return False
+def path_parts(path: str) -> tuple[str, ...]:
+ return tuple(part for part in normalize_path(path).strip("/").split("/") if part)
+
+
+def common_path_prefix(paths: Sequence[str]) -> str:
+ if not paths:
+ return "/"
+ prefix = list(path_parts(paths[0]))
+ for path in paths[1:]:
+ parts = path_parts(path)
+ index = 0
+ while index < min(len(prefix), len(parts)) and prefix[index] == parts[index]:
+ index += 1
+ del prefix[index:]
+ if not prefix:
+ return "/"
+ return "/" + "/".join(prefix) if prefix else "/"
+
+
+def iter_wiki_corpus_paths(path: Path, limit: Optional[int] = None) -> Iterator[tuple[str, str]]:
+ for index, item in enumerate(iter_json_array_records(path, limit)):
+ item_id = record_id(item, index)
+ if item_id != str(index):
+ raise ValueError(
+ f"Wiki corpus path mapping is not row-aligned at row {index}: id={item_id}"
+ )
+ raw_path = item.get("path")
+ normalized = normalize_path(raw_path)
+ if normalized == "/":
+ raise ValueError(f"Wiki corpus path mapping has an empty root path at row {index}")
+ yield item_id, normalized
+
+
+def collect_wiki_corpus_paths(path: Path, corpus_ids: set[str]) -> dict[str, str]:
+ if not corpus_ids:
+ return {}
+ found: dict[str, str] = {}
+ for item_id, item_path in iter_wiki_corpus_paths(path):
+ if item_id in corpus_ids:
+ found[item_id] = item_path
+ if len(found) == len(corpus_ids):
+ break
+ missing = corpus_ids - found.keys()
+ if missing:
+ preview = ", ".join(sorted(missing)[:5])
+ raise ValueError(
+ f"Wiki corpus path mapping is missing {len(missing)} ground-truth ids: {preview}"
+ )
+ return found
+
+
+def derived_gt_lca_scope(items: Sequence[GroundTruthItem], corpus_paths: dict[str, str]) -> str:
+ if not items:
+ return "/"
+ highest_score = max(item.score for item in items)
+ relevant_paths = [corpus_paths[item.corpus_id] for item in items if item.score == highest_score]
+ return common_path_prefix(relevant_paths)
+
+
def synthetic_vector(rng: random.Random, dim: int) -> list[float]:
return [rng.uniform(-1.0, 1.0) for _ in range(dim)]
@@ -542,8 +760,8 @@ def build_synthetic_workload(options: BenchOptions) -> Workload:
)
-def load_ground_truth(path: Path) -> dict[str, list[str]]:
- result: dict[str, list[str]] = {}
+def load_scored_ground_truth(path: Path) -> dict[str, list[GroundTruthItem]]:
+ result: dict[str, list[GroundTruthItem]] = {}
with path.open("r", encoding="utf-8") as handle:
for index, line in enumerate(handle):
stripped = line.strip()
@@ -554,20 +772,29 @@ def load_ground_truth(path: Path) -> dict[str, list[str]]:
if parts[0].lower() in {"query-id", "query_id"}:
continue
if len(parts) >= 3 and is_number(parts[2]):
- result.setdefault(str(parts[0]), []).append(str(parts[1]))
+ result.setdefault(str(parts[0]), []).append(
+ GroundTruthItem(str(parts[1]), float(parts[2]))
+ )
continue
if len(parts) >= 2:
- result[str(parts[0])] = [item for item in parts[1:] if item]
+ result[str(parts[0])] = [GroundTruthItem(item) for item in parts[1:] if item]
continue
else:
parts = stripped.replace(",", " ").split()
if len(parts) < 2:
continue
# Plain-text ground-truth files omit query ids; line number is the query id.
- result[str(index)] = [item for item in parts if item]
+ result[str(index)] = [GroundTruthItem(item) for item in parts if item]
return result
+def load_ground_truth(path: Path) -> dict[str, list[str]]:
+ return {
+ query_id: [item.corpus_id for item in items]
+ for query_id, items in load_scored_ground_truth(path).items()
+ }
+
+
def is_number(value: str) -> bool:
try:
float(value)
@@ -612,6 +839,50 @@ def build_dir_vector_workload(options: BenchOptions) -> Workload:
else {record_id(meta, index) for index, meta in enumerate(sample_metadata)}
)
+ query_records = load_query_records(paths["queries"], query_limit if options.full else None)
+ if options.full and len(query_records) != total_queries:
+ raise ValueError(
+ f"query metadata/vector count mismatch: {len(query_records)} vs {total_queries}"
+ )
+ query_vectors = list(iter_fvecs(paths["query_vectors"], len(query_records)))
+ if len(query_vectors) != len(query_records):
+ raise ValueError(
+ f"query metadata/vector count mismatch: {len(query_records)} vs {len(query_vectors)}"
+ )
+ scored_ground_truth = load_scored_ground_truth(paths["ground_truth"])
+
+ effective_query_scope = (
+ options.dir_vector_query_scope if options.dataset == "wiki" else "dataset"
+ )
+ query_ground_truth = [
+ scored_ground_truth.get(record_id(query, index))
+ or scored_ground_truth.get(str(index))
+ or []
+ for index, query in enumerate(query_records)
+ ]
+ missing_wiki_scopes = [
+ record_id(query, index)
+ for index, query in enumerate(query_records)
+ if extract_declared_path(query) is None
+ ]
+ if (
+ options.dataset == "wiki"
+ and effective_query_scope == "dataset"
+ and options.mode != "write-only"
+ and missing_wiki_scopes
+ ):
+ raise ValueError(
+ f"wiki query metadata has no directory constraint for "
+ f"{len(missing_wiki_scopes)} queries; refusing a root-only Directory benchmark. "
+ "Use a corrected dataset, or explicitly opt into the diagnostic "
+ "--dir-vector-query-scope derived_gt_lca_v1 mode for Wiki."
+ )
+
+ wiki_gt_paths: dict[str, str] = {}
+ if options.dataset == "wiki" and options.mode != "write-only":
+ ground_truth_ids = {item.corpus_id for items in query_ground_truth for item in items}
+ wiki_gt_paths = collect_wiki_corpus_paths(paths["corpus_paths"], ground_truth_ids)
+
def records() -> Iterator[dict[str, Any]]:
vector_iter = iter_fvecs(paths["vectors"], row_limit)
metadata_iter = (
@@ -619,51 +890,143 @@ def build_dir_vector_workload(options: BenchOptions) -> Workload:
if sample_metadata is not None
else iter_json_records(paths["corpus"], row_limit)
)
- for index, (meta, vector) in enumerate(zip(metadata_iter, vector_iter, strict=False)):
+ if options.dataset == "wiki":
+ mapping_iter = iter_wiki_corpus_paths(
+ paths["corpus_paths"], None if options.full else row_limit
+ )
+ combined: Iterable[tuple[dict[str, Any], list[float], tuple[str, str]]] = zip(
+ metadata_iter, vector_iter, mapping_iter, strict=True
+ )
+ else:
+ combined = (
+ (
+ meta,
+ vector,
+ (
+ record_id(meta, index),
+ extract_legacy_path(meta),
+ ),
+ )
+ for index, (meta, vector) in enumerate(zip(metadata_iter, vector_iter, strict=True))
+ )
+
+ for index, (meta, vector, (mapped_id, mapped_path)) in enumerate(combined):
rid = record_id(meta, index)
- path = extract_path(meta)
+ if options.dataset == "wiki" and (rid != mapped_id or rid != str(index)):
+ raise ValueError(
+ "Wiki corpus metadata/path/vector rows are not aligned at "
+ f"row {index}: metadata_id={rid}, path_id={mapped_id}"
+ )
+ if options.dataset == "wiki" and mapped_path == "/":
+ raise ValueError(f"dir-vector corpus row {index} has no declared directory path")
category = str(pick_value(meta, ("category", "primary_category", "type")) or "")
title = str(pick_value(meta, ("title", "name", "label")) or rid)
yield ov_context_record(
record_id=rid,
vector=vector,
- uri=resource_uri(options.dataset, path, rid),
+ uri=resource_uri(options.dataset, mapped_path, rid),
index=index,
category=category,
content=title,
)
- query_records = load_query_records(paths["queries"], query_limit if options.full else None)
- query_vectors = list(iter_fvecs(paths["query_vectors"], len(query_records)))
- ground_truth = load_ground_truth(paths["ground_truth"])
- covered_cases: list[QueryCase] = []
- fallback_cases: list[QueryCase] = []
- for index, (query_meta, vector) in enumerate(zip(query_records, query_vectors, strict=False)):
+ base_covered: list[QueryCase] = []
+ base_fallback: list[QueryCase] = []
+ directory_covered: list[QueryCase] = []
+ directory_fallback: list[QueryCase] = []
+ missing_dataset_scopes: list[str] = []
+ root_derived_scopes = 0
+ for index, (query_meta, vector) in enumerate(zip(query_records, query_vectors, strict=True)):
qid = record_id(query_meta, index)
- gt = [str(item) for item in (ground_truth.get(qid) or ground_truth.get(str(index)) or [])]
+ scored_gt = query_ground_truth[index]
+ full_gt = [item.corpus_id for item in scored_gt]
+ gt = full_gt
if sampled_ids is not None:
gt = [item for item in gt if item in sampled_ids]
- case = QueryCase(
+
+ declared_scope = (
+ extract_declared_path(query_meta)
+ if options.dataset == "wiki"
+ else extract_legacy_path(query_meta)
+ )
+ directory_scope = declared_scope or "/"
+ if effective_query_scope == "derived_gt_lca_v1" and options.mode != "write-only":
+ directory_scope = derived_gt_lca_scope(scored_gt, wiki_gt_paths)
+ elif options.dataset == "wiki" and directory_scope == "/":
+ missing_dataset_scopes.append(qid)
+
+ base_case = QueryCase(
query_id=qid,
vector=vector,
- filter_path=resource_uri(options.dataset, extract_path(query_meta)),
+ filter_path=resource_uri(options.dataset, directory_scope),
ground_truth_ids=gt,
)
if gt or sampled_ids is None:
- covered_cases.append(case)
+ base_covered.append(base_case)
+ else:
+ base_fallback.append(base_case)
+
+ if options.dataset == "wiki" and directory_scope == "/":
+ if effective_query_scope == "derived_gt_lca_v1":
+ root_derived_scopes += 1
+ continue
+ directory_gt = (
+ [item for item in gt if path_matches(wiki_gt_paths[item], directory_scope)]
+ if options.dataset == "wiki"
+ else gt
+ )
+ directory_case = QueryCase(
+ query_id=qid,
+ vector=vector,
+ filter_path=resource_uri(options.dataset, directory_scope),
+ ground_truth_ids=directory_gt,
+ )
+ if directory_gt or sampled_ids is None:
+ directory_covered.append(directory_case)
else:
- fallback_cases.append(case)
- cases = covered_cases[:query_limit]
- if len(cases) < query_limit:
- cases.extend(fallback_cases[: query_limit - len(cases)])
+ directory_fallback.append(directory_case)
+
+ def select_cases(covered: list[QueryCase], fallback: list[QueryCase]) -> list[QueryCase]:
+ selected = covered[:query_limit]
+ if len(selected) < query_limit:
+ selected.extend(fallback[: query_limit - len(selected)])
+ return selected
+
+ cases = select_cases(base_covered, base_fallback)
+ directory_cases = select_cases(directory_covered, directory_fallback)
if not cases:
raise ValueError("dir-vector query set is empty")
+ if options.mode != "write-only" and missing_dataset_scopes:
+ raise ValueError(
+ f"{options.dataset} query metadata has no directory constraint for "
+ f"{len(missing_dataset_scopes)} queries; refusing a root-only Directory benchmark. "
+ "Use a corrected dataset, or explicitly opt into the diagnostic "
+ "--dir-vector-query-scope derived_gt_lca_v1 mode for Wiki."
+ )
+ if options.mode != "write-only" and not directory_cases:
+ raise ValueError("dir-vector Directory query set is empty after path validation")
+
+ scope_depths = [len(path_parts(case.filter_path)) - 3 for case in directory_cases]
+ scope_depths = [max(0, depth) for depth in scope_depths]
+ path_diagnostics = {
+ "corpus_path_source": paths.get("corpus_paths", paths["corpus"]).name,
+ "query_scope_source": effective_query_scope,
+ "base_query_count": len(cases),
+ "directory_query_count": len(directory_cases),
+ "unique_directory_scopes": len({case.filter_path for case in directory_cases}),
+ "root_derived_scopes_omitted": root_derived_scopes,
+ "directory_scope_depth_min": min(scope_depths) if scope_depths else None,
+ "directory_scope_depth_median": statistics.median(scope_depths) if scope_depths else None,
+ "directory_scope_depth_max": max(scope_depths) if scope_depths else None,
+ }
return Workload(
name=f"dir-vector:{options.dataset}",
dim=dim,
records=records,
queries=cases,
expected_rows=row_limit,
+ directory_queries=directory_cases,
+ path_diagnostics=path_diagnostics,
)
@@ -901,7 +1264,9 @@ async def run_benchmark_async(options: BenchOptions) -> RunResult:
progress(
"start "
f"run_id={options.run_id} workload={workload.name} mode={options.mode} "
- f"records={workload.expected_rows} queries={len(workload.queries)} dim={workload.dim}"
+ f"records={workload.expected_rows} base_queries={len(workload.queries)} "
+ f"directory_queries={len(directory_queries(workload))} "
+ f"dim={workload.dim}"
)
progress(f"collection={collection_name}")
@@ -1023,6 +1388,7 @@ async def run_benchmark_async(options: BenchOptions) -> RunResult:
)
selected_queries = workload.queries
+ selected_directory_queries = directory_queries(workload)
search_events, errors, vector_quality = await run_search_phase(
backend=backend,
ctx=ctx,
@@ -1040,7 +1406,7 @@ async def run_benchmark_async(options: BenchOptions) -> RunResult:
backend=backend,
ctx=ctx,
phase="filtered_vector_search",
- queries=selected_queries,
+ queries=selected_directory_queries,
top_k=options.top_k,
concurrency=options.concurrency,
filtered=True,
@@ -1049,7 +1415,7 @@ async def run_benchmark_async(options: BenchOptions) -> RunResult:
validation_errors.extend(errors)
quality["filtered_vector_search"] = filtered_quality
- first_filter = selected_queries[0].filter_path
+ first_filter = selected_directory_queries[0].filter_path
event, _ = await async_timed_event(
"validate",
"count_filtered",
@@ -1112,10 +1478,12 @@ def finalize_result(
"dataset": options.dataset if options.workload == "dir-vector" else "synthetic",
"record_count": workload.expected_rows,
"query_count": len(workload.queries),
+ "directory_query_count": len(directory_queries(workload)),
"vector_dim": workload.dim,
"top_k": options.top_k,
"full": options.full,
"profile": options.profile,
+ "path_diagnostics": workload.path_diagnostics,
},
environment=collect_environment(),
kept_collection=kept_collection,
@@ -1295,6 +1663,7 @@ def write_suite_outputs(output_dir: Path, runs: list[tuple[BenchOptions, RunResu
"records": result.workload.get("record_count"),
"inserted": metric_value(run_summaries[run_options.dataset], "ingest", "count"),
"queries": result.workload.get("query_count"),
+ "directory_queries": result.workload.get("directory_query_count"),
"dim": result.workload.get("vector_dim"),
"top_k": result.workload.get("top_k"),
"vector_qps": metric_value(
@@ -1361,7 +1730,10 @@ def quality_rate(quality: dict[str, Any], phase: str) -> str:
return f"{float(value) * 100.0:.2f}%"
-def recall_scope(result: RunResult) -> str:
+def recall_scope(result: RunResult, phase: str = "filtered_vector_search") -> str:
+ query_scope_source = result.workload.get("path_diagnostics", {}).get("query_scope_source")
+ if phase == "filtered_vector_search" and query_scope_source and query_scope_source != "dataset":
+ return str(query_scope_source)
if result.workload.get("workload") == "dir-vector" and not result.workload.get("full"):
return "sampled_subset"
return "official_full"
@@ -1377,6 +1749,7 @@ def workload_rows(result: RunResult, summary_rows: list[dict[str, Any]]) -> list
"records": workload.get("record_count"),
"inserted": metric_value(summary_rows, "ingest", "count"),
"queries": workload.get("query_count"),
+ "directory_queries": workload.get("directory_query_count"),
"dim": workload.get("vector_dim"),
"top_k": workload.get("top_k"),
"full": workload.get("full"),
@@ -1394,7 +1767,7 @@ def recall_qps_rows(
"qps": metric_value(summary_rows, "vector_search", "throughput_per_sec"),
"avg_ms": metric_value(summary_rows, "vector_search", "avg_ms"),
"p95_ms": metric_value(summary_rows, "vector_search", "p95_ms"),
- "scope": recall_scope(result),
+ "scope": recall_scope(result, "vector_search"),
recall_header: quality_rate(result.quality, "vector_search"),
},
{
@@ -1402,7 +1775,7 @@ def recall_qps_rows(
"qps": metric_value(summary_rows, "filtered_vector_search", "throughput_per_sec"),
"avg_ms": metric_value(summary_rows, "filtered_vector_search", "avg_ms"),
"p95_ms": metric_value(summary_rows, "filtered_vector_search", "p95_ms"),
- "scope": recall_scope(result),
+ "scope": recall_scope(result, "filtered_vector_search"),
recall_header: quality_rate(result.quality, "filtered_vector_search"),
},
]
@@ -1438,11 +1811,19 @@ def build_markdown_report(
"",
markdown_table(workload_rows(result, summary_rows)),
"",
+ "## Directory path 口径",
+ "",
+ f"- Corpus path source: `{result.workload.get('path_diagnostics', {}).get('corpus_path_source', 'generated')}`",
+ f"- Query scope source: `{result.workload.get('path_diagnostics', {}).get('query_scope_source', 'generated')}`",
+ f"- Directory queries: `{result.workload.get('directory_query_count')}`",
+ f"- Unique directory scopes: `{result.workload.get('path_diagnostics', {}).get('unique_directory_scopes', '-')}`",
+ f"- Root-derived scopes omitted: `{result.workload.get('path_diagnostics', {}).get('root_derived_scopes_omitted', 0)}`",
+ "",
"## 召回与 QPS",
"",
markdown_table(recall_qps_rows(result, summary_rows, options)),
"",
- "gt_recall@K 按 query 的 ground truth 命中率统计;`sampled_subset` 只用于 smoke sanity,不代表官方全量 recall。官方 recall 请用 `--full`。",
+ "gt_recall@K 按 query 的 ground truth 命中率统计;`sampled_subset` 只用于 smoke sanity,不代表官方全量 recall。`derived_gt_lca_v1` 使用 ground truth 派生目录,只是诊断 workload,也不代表发布数据集的官方 query constraint。",
"",
"## 性能汇总",
"",
@@ -1551,6 +1932,17 @@ def parse_args(argv: Optional[Sequence[str]] = None) -> BenchOptions:
parser.add_argument("--path-depth", type=int)
parser.add_argument("--path-fanout", type=int)
parser.add_argument("--filter-selectivity", type=float)
+ parser.add_argument(
+ "--dir-vector-query-scope",
+ choices=DIR_VECTOR_QUERY_SCOPES,
+ default="dataset",
+ help=(
+ "Directory scope source for dir-vector queries. dataset requires published query "
+ "constraints; derived_gt_lca_v1 gives Wiki an explicit diagnostic scope derived "
+ "from the deepest common path of highest-score ground-truth items. Other datasets "
+ "continue to use their published constraints."
+ ),
+ )
parser.add_argument("--seed", type=int, default=42)
parser.add_argument("--distance", choices=["ip", "l2", "cosine"])
parser.add_argument("--drop-at-end", action="store_true", help="Drop benchmark collection")
@@ -1588,6 +1980,7 @@ def parse_args(argv: Optional[Sequence[str]] = None) -> BenchOptions:
seed=args.seed,
distance=args.distance,
drop_at_end=args.drop_at_end,
+ dir_vector_query_scope=args.dir_vector_query_scope,
)
diff --git a/tests/benchmark/test_vectordb_perf_run.py b/tests/benchmark/test_vectordb_perf_run.py
index fed1eb9e7..76bc97a33 100644
--- a/tests/benchmark/test_vectordb_perf_run.py
+++ b/tests/benchmark/test_vectordb_perf_run.py
@@ -1,9 +1,101 @@
# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.
# SPDX-License-Identifier: AGPL-3.0
+import json
+import struct
+
import pytest
-from benchmark.vectordb_perf.run import upsert_records
+from benchmark.vectordb_perf.run import (
+ RunResult,
+ build_dir_vector_workload,
+ directory_queries,
+ extract_path,
+ iter_json_array_records,
+ parse_args,
+ recall_scope,
+ record_id,
+ upsert_records,
+)
+
+
+def _write_fvecs(path, vectors):
+ with path.open("wb") as handle:
+ for vector in vectors:
+ handle.write(struct.pack("<i", len(vector)))
+ handle.write(struct.pack(f"<{len(vector)}f", *vector))
+
+
+def _write_wiki_fixture(tmp_path, *, query_paths=False, mapping_ids=(0, 1, 2)):
+ corpus = [
+ {"_id": 0, "title": "pronounced /not-a-directory/", "text": "zero"},
+ {"_id": 1, "title": "one", "text": "one"},
+ {"_id": 2, "title": "two", "text": "two"},
+ ]
+ paths = [
+ {"id": mapping_ids[0], "path": "/science/physics/quantum/"},
+ {"id": mapping_ids[1], "path": "/science/physics/classical/"},
+ {"id": mapping_ids[2], "path": "/history/ancient/"},
+ ]
+ queries = [
+ {"_id": 0, "text": "physics", "metadata": {}},
+ {"_id": 1, "text": "mixed", "metadata": {}},
+ {"_id": 2, "text": "history", "metadata": {}},
+ ]
+ if query_paths:
+ queries[0]["metadata"] = {"path": "/science/physics"}
+ queries[1]["metadata"] = {"path": "/science"}
+ queries[2]["metadata"] = {"path": "/history"}
+
+ (tmp_path / "dbpedia_dir_2m_corpus.jsonl").write_text(
+ "".join(json.dumps(item) + "\n" for item in corpus), encoding="utf-8"
+ )
+ (tmp_path / "dbpedia_dir_2m_corpus_paths.json").write_text(json.dumps(paths), encoding="utf-8")
+ (tmp_path / "dbpedia_dir_2m_query.jsonl").write_text(
+ "".join(json.dumps(item) + "\n" for item in queries), encoding="utf-8"
+ )
+ (tmp_path / "dbpedia_dir_2m_groundtruth.tsv").write_text(
+ "query-id\tcorpus-id\tscore\n0\t0\t2\n0\t1\t2\n1\t0\t2\n1\t2\t2\n2\t2\t1\n",
+ encoding="utf-8",
+ )
+ vectors = [[float(index), 0.0, 0.0, 1.0] for index in range(3)]
+ _write_fvecs(tmp_path / "dbpedia_dir_2m_corpus_vectors.fvecs", vectors)
+ _write_fvecs(tmp_path / "dbpedia_dir_2m_query_vectors.fvecs", vectors)
+
+
+def _wiki_options(tmp_path, *extra):
+ return parse_args(
+ [
+ "--workload",
+ "dir-vector",
+ "--dataset",
+ "wiki",
+ "--dataset-root",
+ str(tmp_path),
+ "--rows",
+ "3",
+ "--queries",
+ "3",
+ *extra,
+ ]
+ )
+
+
+def _write_arxiv_public_shape_fixture(tmp_path):
+ corpus = [
+ {"id": "0", "categories": ["cs.AI", "cs.LG"], "time": "2024/01"},
+ {"id": "1", "categories": ["math.OC"], "time": "2024/02"},
+ ]
+ queries = [
+ {"query_id": "0", "constraints": {"time": "2024/01"}},
+ {"query_id": "1", "constraints": {"category": "math.OC"}},
+ ]
+ (tmp_path / "arxiv_corpus_metadata.json").write_text(json.dumps(corpus), encoding="utf-8")
+ (tmp_path / "arxiv_query_constraint.json").write_text(json.dumps(queries), encoding="utf-8")
+ (tmp_path / "arxiv_ground_truth.txt").write_text("0 1\n1 0\n", encoding="utf-8")
+ vectors = [[0.0, 1.0, 0.0, 0.0], [1.0, 0.0, 0.0, 0.0]]
+ _write_fvecs(tmp_path / "arxiv_corpus_vectors.fvecs", vectors)
+ _write_fvecs(tmp_path / "arxiv_query_vectors.fvecs", vectors)
@pytest.mark.asyncio
@@ -41,3 +133,202 @@ async def test_upsert_records_rejects_incomplete_bulk_result():
object(),
[{"id": "rec-1"}, {"id": "rec-2"}],
)
+
+
+def test_iter_json_array_records_streams_across_small_chunks(tmp_path):
+ path = tmp_path / "records.json"
+ expected = [{"id": 0, "path": "/α/β"}, {"id": 1, "path": '/quoted/"x"'}]
+ path.write_text(json.dumps(expected, ensure_ascii=False), encoding="utf-8")
+
+ assert list(iter_json_array_records(path, chunk_size=3)) == expected
+ assert list(iter_json_array_records(path, limit=1, chunk_size=2)) == expected[:1]
+ assert list(iter_json_array_records(path, limit=0, chunk_size=2)) == []
+
+
+ "payload, message",
+ [
+ ('[{"id": 0},]', "trailing comma"),
+ ('[{"id": 0}] garbage', "unexpected content"),
+ ('[{"id": 0}', "truncated JSON array"),
+ ],
+)
+def test_iter_json_array_records_rejects_invalid_full_input(tmp_path, payload, message):
+ path = tmp_path / "invalid.json"
+ path.write_text(payload, encoding="utf-8")
+
+ with pytest.raises(ValueError, match=message):
+ list(iter_json_array_records(path, chunk_size=2))
+
+
+def test_path_extraction_does_not_treat_prose_slashes_as_a_directory():
+ assert extract_path({"title": "pronounced /foo/", "text": "a/b"}) == "/"
+ assert record_id({"_id": 17}, 0) == "17"
+
+
+def test_wiki_mapping_and_derived_scopes_use_real_hierarchy(tmp_path):
+ _write_wiki_fixture(tmp_path)
+ workload = build_dir_vector_workload(
+ _wiki_options(tmp_path, "--dir-vector-query-scope", "derived_gt_lca_v1")
+ )
+
+ records = list(workload.records())
+ assert [record["uri"] for record in records] == [
+ "viking://resources/bench/wiki/science/physics/quantum/0",
+ "viking://resources/bench/wiki/science/physics/classical/1",
+ "viking://resources/bench/wiki/history/ancient/2",
+ ]
+ assert len(workload.queries) == 3
+ assert [case.query_id for case in workload.directory_queries] == ["0", "2"]
+ assert workload.directory_queries[0].filter_path.endswith("/science/physics")
+ assert workload.directory_queries[0].ground_truth_ids == ["0", "1"]
+ assert workload.path_diagnostics == {
+ "corpus_path_source": "dbpedia_dir_2m_corpus_paths.json",
+ "query_scope_source": "derived_gt_lca_v1",
+ "base_query_count": 3,
+ "directory_query_count": 2,
+ "unique_directory_scopes": 2,
+ "root_derived_scopes_omitted": 1,
+ "directory_scope_depth_min": 2,
+ "directory_scope_depth_median": 2.0,
+ "directory_scope_depth_max": 2,
+ }
+
+
+def test_wiki_dataset_scope_refuses_missing_query_constraints(tmp_path):
+ _write_wiki_fixture(tmp_path)
+
+ with pytest.raises(ValueError, match="refusing a root-only Directory benchmark"):
+ build_dir_vector_workload(_wiki_options(tmp_path))
+
+
+def test_wiki_write_only_preserves_an_intentionally_empty_directory_set(tmp_path):
+ _write_wiki_fixture(tmp_path)
+
+ workload = build_dir_vector_workload(_wiki_options(tmp_path, "--mode", "write-only"))
+
+ assert workload.directory_queries == []
+ assert directory_queries(workload) == []
+
+
+def test_wiki_dataset_scope_accepts_explicit_query_paths(tmp_path):
+ _write_wiki_fixture(tmp_path, query_paths=True)
+
+ workload = build_dir_vector_workload(_wiki_options(tmp_path))
+
+ assert len(workload.directory_queries) == 3
+ assert workload.path_diagnostics["query_scope_source"] == "dataset"
+ assert workload.path_diagnostics["unique_directory_scopes"] == 3
+
+
+def test_wiki_mapping_requires_row_aligned_ids(tmp_path):
+ _write_wiki_fixture(tmp_path, mapping_ids=(0, 9, 2))
+
+ with pytest.raises(ValueError, match="not row-aligned at row 1"):
+ build_dir_vector_workload(
+ _wiki_options(tmp_path, "--dir-vector-query-scope", "derived_gt_lca_v1")
+ )
+
+
+def test_wiki_mapping_requires_all_rows_in_full_mode(tmp_path):
+ _write_wiki_fixture(tmp_path)
+ path = tmp_path / "dbpedia_dir_2m_corpus_paths.json"
+ mapping = json.loads(path.read_text(encoding="utf-8"))
+ mapping.append({"id": 3, "path": "/extra"})
+ path.write_text(json.dumps(mapping), encoding="utf-8")
+ workload = build_dir_vector_workload(
+ _wiki_options(tmp_path, "--full", "--dir-vector-query-scope", "derived_gt_lca_v1")
+ )
+
+ with pytest.raises(ValueError, match=r"zip\(\) argument 3 is longer"):
+ list(workload.records())
+
+
+def test_wiki_mapping_rejects_missing_ground_truth_rows(tmp_path):
+ _write_wiki_fixture(tmp_path)
+ path = tmp_path / "dbpedia_dir_2m_corpus_paths.json"
+ mapping = json.loads(path.read_text(encoding="utf-8"))
+ path.write_text(json.dumps(mapping[:2]), encoding="utf-8")
+
+ with pytest.raises(ValueError, match="missing 1 ground-truth ids"):
+ build_dir_vector_workload(
+ _wiki_options(tmp_path, "--dir-vector-query-scope", "derived_gt_lca_v1")
+ )
+
+
+def test_wiki_mapping_rejects_root_path(tmp_path):
+ _write_wiki_fixture(tmp_path)
+ path = tmp_path / "dbpedia_dir_2m_corpus_paths.json"
+ mapping = json.loads(path.read_text(encoding="utf-8"))
+ mapping[0]["path"] = None
+ path.write_text(json.dumps(mapping), encoding="utf-8")
+
+ with pytest.raises(ValueError, match="empty root path at row 0"):
+ build_dir_vector_workload(
+ _wiki_options(tmp_path, "--dir-vector-query-scope", "derived_gt_lca_v1")
+ )
+
+
+def test_full_wiki_requires_query_metadata_for_every_vector(tmp_path):
+ _write_wiki_fixture(tmp_path)
+ path = tmp_path / "dbpedia_dir_2m_query.jsonl"
+ lines = path.read_text(encoding="utf-8").splitlines()
+ path.write_text("\n".join(lines[:2]) + "\n", encoding="utf-8")
+
+ with pytest.raises(ValueError, match="query metadata/vector count mismatch: 2 vs 3"):
+ build_dir_vector_workload(
+ _wiki_options(tmp_path, "--full", "--dir-vector-query-scope", "derived_gt_lca_v1")
+ )
+
+
+def test_derived_scope_only_labels_filtered_phase(tmp_path):
+ _write_wiki_fixture(tmp_path)
+ workload = build_dir_vector_workload(
+ _wiki_options(tmp_path, "--full", "--dir-vector-query-scope", "derived_gt_lca_v1")
+ )
+ result = RunResult(
+ run_id="test",
+ collection_name="test",
+ output_dir=str(tmp_path),
+ events=[],
+ validation_errors=[],
+ quality={},
+ workload={
+ "workload": "dir-vector",
+ "full": True,
+ "path_diagnostics": workload.path_diagnostics,
+ },
+ environment={},
+ kept_collection=True,
+ )
+
+ assert recall_scope(result, "vector_search") == "official_full"
+ assert recall_scope(result, "filtered_vector_search") == "derived_gt_lca_v1"
+
+ result.workload["path_diagnostics"] = {}
+ assert recall_scope(result, "filtered_vector_search") == "official_full"
+
+
+def test_derived_wiki_option_keeps_non_wiki_loader_behavior(tmp_path):
+ _write_arxiv_public_shape_fixture(tmp_path)
+ options = parse_args(
+ [
+ "--workload",
+ "dir-vector",
+ "--dataset",
+ "arxiv",
+ "--dataset-root",
+ str(tmp_path),
+ "--rows",
+ "2",
+ "--queries",
+ "2",
+ "--dir-vector-query-scope",
+ "derived_gt_lca_v1",
+ ]
+ )
+
+ workload = build_dir_vector_workload(options)
+
+ assert workload.path_diagnostics["query_scope_source"] == "dataset"
+ assert len(workload.directory_queries) == 2