Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
fix(benchmark): load WIKI-Dir path mapping (#3279)
Yuanqing ZHAO546da35cc8af
diff --git a/benchmark/vectordb_perf/README.md b/benchmark/vectordb_perf/README.mdindex 913daabf4..6a470f3e7 100644--- a/benchmark/vectordb_perf/README.md+++ b/benchmark/vectordb_perf/README.md@@ -101,23 +101,38 @@ mkdir -p benchmark/data/dir-vector-dataset-filesrunner 会先校验 `wiki` 和 `arxiv` 所需文件;缺文件时会直接列出 missing files 和下载地址。调试时可以用 `--dataset wiki` 或 `--dataset arxiv` 只跑单个数据集。各 dataset 对应文件见下方“真实数据文件”。-真实数据少量抽样:+真实数据少量抽样(Wiki 公开文件):```bash.venv/bin/python -m benchmark.vectordb_perf.run \--config ./ov.conf \--workload dir-vector \+ --dataset wiki \--dataset-root benchmark/data/dir-vector-dataset-files \+ --dir-vector-query-scope derived_gt_lca_v1 \--profile smoke```-真实数据全量:+WIKI-Dir 的 corpus 目录来自单独发布的 `dbpedia_dir_2m_corpus_paths.json`,runner 会按+`id` 将它与 corpus metadata 和 fvecs 逐行对齐,并在错位或缺行时直接报错。这个文件约+600 MB,runner 使用流式解析,不会把全部 mapping 一次性加载到内存。++当前公开的 Wiki query metadata 没有包含目录 constraint,因此默认的 `dataset` 口径会拒绝+把所有 query 静默当成根目录查询。若要对公开文件做目录过滤的诊断性测试,可以显式使用+`derived_gt_lca_v1`:对每个 query 的最高 relevance ground-truth 文档取目录最长公共前缀,+并跳过只能得到根目录的 query。这个口径使用了 ground truth,不代表发布数据集的官方 query+分布,也不能用来声明官方 Directory recall。该选项只改变 Wiki;在 `--dataset all` 中,+其他 dataset 保持现有 loader 行为。++真实数据全量(公开 Wiki 文件的诊断性目录口径):```bash.venv/bin/python -m benchmark.vectordb_perf.run \--config ./ov.conf \--workload dir-vector \+ --dataset wiki \--dataset-root ./benchmark/data/dir-vector-dataset-files \+ --dir-vector-query-scope derived_gt_lca_v1 \--full```@@ -135,7 +150,9 @@ runner 会把配置里的 `name` 改成 `<name>_bench_<run-id>`,避免覆盖.venv/bin/python -m benchmark.vectordb_perf.run \--config ./ov.conf \--workload dir-vector \+ --dataset wiki \--dataset-root ./benchmark/data/dir-vector-dataset-files \+ --dir-vector-query-scope derived_gt_lca_v1 \--full \--run-id ovbench_full_001 \--mode write-only@@ -147,7 +164,9 @@ runner 会把配置里的 `name` 改成 `<name>_bench_<run-id>`,避免覆盖.venv/bin/python -m benchmark.vectordb_perf.run \--config ./ov.conf \--workload dir-vector \+ --dataset wiki \--dataset-root ./benchmark/data/dir-vector-dataset-files \+ --dir-vector-query-scope derived_gt_lca_v1 \--full \--run-id ovbench_full_001 \--mode read-only \@@ -206,7 +225,7 @@ upsert;需要保留未提供字段的 partial update 仍使用单条 `upsert`进度输出示例:```text-[vectordb_perf] start run_id=ovbench_full_001 workload=dir-vector:wiki mode=write-only records=1941679 queries=456 dim=1024+[vectordb_perf] start run_id=ovbench_full_001 workload=dir-vector:wiki mode=write-only records=1941679 base_queries=456 directory_queries=217 dim=1024[vectordb_perf] collection=vectordb_perf_bench_ovbench_full_001_wiki[vectordb_perf] setup: create collection[vectordb_perf] setup: collection ready@@ -236,7 +255,7 @@ upsert;需要保留未提供字段的 partial update 仍使用单条 `upsert`| `--dataset` | 期望文件 || --- | --- |-| `wiki` | `dbpedia_dir_2m_corpus.jsonl`、`dbpedia_dir_2m_corpus_vectors.fvecs`、`dbpedia_dir_2m_query.jsonl`、`dbpedia_dir_2m_query_vectors.fvecs`、`dbpedia_dir_2m_groundtruth.tsv` |+| `wiki` | `dbpedia_dir_2m_corpus.jsonl`、`dbpedia_dir_2m_corpus_paths.json`、`dbpedia_dir_2m_corpus_vectors.fvecs`、`dbpedia_dir_2m_query.jsonl`、`dbpedia_dir_2m_query_vectors.fvecs`、`dbpedia_dir_2m_groundtruth.tsv` || `arxiv` | `arxiv_corpus_metadata.json`、`arxiv_corpus_vectors.fvecs`、`arxiv_query_constraint.json`、`arxiv_query_vectors.fvecs`、`arxiv_ground_truth.txt` || `arxiv_category` | `arxiv_corpus_metadata.json`、`arxiv_corpus_vectors.fvecs`、`arxiv_category_query_constraint.json`、`arxiv_category_query_vectors.fvecs`、`arxiv_category_ground_truth.txt` |@@ -303,6 +322,7 @@ Volcengine VikingDB:| `--batch-size` | 每次 backend `upsert_many` 调用的记录数;远端 adapter 可按请求上限再拆分 || `--concurrency` | 查询并发 || `--top-k` | 检索返回条数 |+| `--dir-vector-query-scope` | dir-vector 目录 query 来源;默认 `dataset`,Wiki 公开文件可显式使用诊断口径 `derived_gt_lca_v1` || `--distance` | `ip`、`l2`、`cosine` || `--drop-at-end` | 运行结束后删除测试 collection |@@ -326,8 +346,10 @@ Volcengine VikingDB:| 数据规模 | `records` 是本次计划读取的数据条数,`inserted` 是实际写入 backend 的条数,`queries` 是实际查询数 || 召回与 QPS | `vector_search` 和 `filtered_vector_search` 的 QPS、平均延迟、P95 延迟、gt_recall@K |-gt_recall@K 按 query 的 ground truth 命中率统计。`--full` 才是官方全量 recall;-非 `--full` 的真实数据报告会标成 `sampled_subset`,只用于 smoke sanity,不代表官方全量 recall。+gt_recall@K 按 query 的 ground truth 命中率统计。全量 Wiki 的无过滤 `vector_search` 保留+`official_full` 标记;其 `filtered_vector_search` 在诊断模式下标为 `derived_gt_lca_v1`。+非 `--full` 会标成 `sampled_subset`。派生目录和抽样口径只用于诊断,不代表官方 Directory+recall。## 资源限制口径diff --git a/benchmark/vectordb_perf/run.py b/benchmark/vectordb_perf/run.pyindex a221da7c3..d37e2c275 100644--- a/benchmark/vectordb_perf/run.py+++ b/benchmark/vectordb_perf/run.py@@ -19,6 +19,7 @@ import osimport platformimport randomimport shutil+import statisticsimport structimport subprocessimport sys@@ -42,6 +43,7 @@ BENCH_USER_ID = "bench_user"DIR_VECTOR_DATASET_URL = "https://github.com/KurtPatrickHere/dir-vector-dataset"DEFAULT_DIR_VECTOR_DATASETS = ("wiki", "arxiv")PROGRESS_INTERVAL_SECONDS = 5.0+DIR_VECTOR_QUERY_SCOPES = ("dataset", "derived_gt_lca_v1")PROFILE_DEFAULTS: dict[str, dict[str, Any]] = {@@ -83,6 +85,7 @@ PROFILE_DEFAULTS: dict[str, dict[str, Any]] = {DIR_VECTOR_FILES = {"wiki": {"corpus": "dbpedia_dir_2m_corpus.jsonl",+ "corpus_paths": "dbpedia_dir_2m_corpus_paths.json","vectors": "dbpedia_dir_2m_corpus_vectors.fvecs","queries": "dbpedia_dir_2m_query.jsonl","query_vectors": "dbpedia_dir_2m_query_vectors.fvecs",@@ -128,6 +131,7 @@ class BenchOptions:seed: intdistance: Optional[str]drop_at_end: bool+ dir_vector_query_scope: str@dataclass@@ -138,6 +142,12 @@ class QueryCase:ground_truth_ids: list[str] = field(default_factory=list)+@dataclass(frozen=True)+class GroundTruthItem:+ corpus_id: str+ score: float = 1.0++@dataclassclass Workload:name: str@@ -145,6 +155,8 @@ class Workload:records: Callable[[], Iterator[dict[str, Any]]]queries: list[QueryCase]expected_rows: Optional[int]+ directory_queries: Optional[list[QueryCase]] = None+ path_diagnostics: dict[str, Any] = field(default_factory=dict)@dataclass@@ -173,6 +185,10 @@ class RunResult:kept_collection: bool+def directory_queries(workload: Workload) -> list[QueryCase]:+ return workload.queries if workload.directory_queries is None else workload.directory_queries++def utc_now() -> str:return datetime.now(UTC).isoformat()@@ -312,6 +328,106 @@ def iter_fvecs(path: Path, limit: Optional[int] = None) -> Iterator[list[float]]count += 1+def iter_json_array_records(+ path: Path,+ limit: Optional[int] = None,+ *,+ chunk_size: int = 1024 * 1024,+) -> Iterator[dict[str, Any]]:+ """Stream a top-level JSON array without materializing the whole file."""+ if chunk_size <= 0:+ raise ValueError("chunk_size must be positive")+ if limit is not None and limit <= 0:+ return++ decoder = json.JSONDecoder()+ with path.open("r", encoding="utf-8") as handle:+ buffer = ""+ position = 0+ eof = False+ started = False+ expect_value = True+ after_comma = False+ yielded = 0++ def refill() -> None:+ nonlocal buffer, position, eof+ if position:+ buffer = buffer[position:]+ position = 0+ chunk = handle.read(chunk_size)+ if chunk:+ buffer += chunk+ else:+ eof = True++ while True:+ if position >= len(buffer) and not eof:+ refill()+ while position < len(buffer) and buffer[position].isspace():+ position += 1+ if position >= len(buffer):+ if eof:+ raise ValueError(f"truncated JSON array in {path.name}")+ refill()+ continue++ if not started:+ if buffer[position] != "[":+ raise ValueError(f"expected a top-level JSON array in {path.name}")+ position += 1+ started = True+ continue++ if not expect_value:+ if buffer[position] == ",":+ position += 1+ expect_value = True+ after_comma = True+ continue+ if buffer[position] == "]":+ position += 1+ while True:+ while position < len(buffer) and buffer[position].isspace():+ position += 1+ if position < len(buffer):+ raise ValueError(f"unexpected content after JSON array in {path.name}")+ if eof:+ return+ refill()+ raise ValueError(f"expected ',' or ']' in {path.name}")++ if buffer[position] == "]":+ if after_comma:+ raise ValueError(f"trailing comma in JSON array in {path.name}")+ position += 1+ while True:+ while position < len(buffer) and buffer[position].isspace():+ position += 1+ if position < len(buffer):+ raise ValueError(f"unexpected content after JSON array in {path.name}")+ if eof:+ return+ refill()++ try:+ item, end = decoder.raw_decode(buffer, position)+ except json.JSONDecodeError as exc:+ if eof:+ raise ValueError(f"invalid JSON array in {path.name}: {exc}") from exc+ refill()+ continue+ position = end+ expect_value = False+ after_comma = False+ if not isinstance(item, dict):+ raise ValueError(f"expected JSON objects in {path.name}")+ yield item+ yielded += 1+ if limit is not None and yielded >= limit:+ return++def iter_json_records(path: Path, limit: Optional[int] = None) -> Iterator[dict[str, Any]]:if path.suffix == ".jsonl":with path.open("r", encoding="utf-8") as handle:@@ -325,6 +441,18 @@ def iter_json_records(path: Path, limit: Optional[int] = None) -> Iterator[dict[yield itemreturn+ with path.open("r", encoding="utf-8") as handle:+ first = ""+ while not first:+ char = handle.read(1)+ if not char:+ raise ValueError(f"empty JSON file: {path}")+ if not char.isspace():+ first = char+ if first == "[":+ yield from iter_json_array_records(path, limit)+ return+data = json.loads(path.read_text(encoding="utf-8"))if isinstance(data, dict):iterable: Iterable[Any] = data.values()@@ -364,25 +492,46 @@ def normalize_path(path: Any) -> str:return stripped.rstrip("/") if len(stripped) > 1 else stripped-def extract_path(record: dict[str, Any]) -> str:- value = pick_value(- record,- (- "dir_path",- "dir_paths",- "path",- "paths",- "directory",- "directories",- "category_path",- "category_paths",- "date_path",- ),- )+GENERIC_PATH_FIELDS = (+ "dir_path",+ "dir_paths",+ "path",+ "paths",+ "directory",+ "directories",+ "category_path",+ "category_paths",+ "date_path",+)+++def extract_declared_path(+ record: dict[str, Any], fields: Sequence[str] = GENERIC_PATH_FIELDS+) -> Optional[str]:+ """Return only explicitly declared path metadata, never arbitrary prose."""+ value = pick_value(record, fields)if value is not None:return normalize_path(value)+ for container_name in ("constraints", "constraint", "metadata"):+ nested = record.get(container_name)+ if isinstance(nested, dict):+ value = pick_value(nested, fields)+ if value is not None:+ return normalize_path(value)+ elif container_name == "constraint" and isinstance(nested, (str, list)):+ return normalize_path(nested)+ return None+- # Last resort for dataset variants with nested constraint metadata.+def extract_path(record: dict[str, Any]) -> str:+ return extract_declared_path(record) or "/"+++def extract_legacy_path(record: dict[str, Any]) -> str:+ """Keep the pre-existing best-effort behavior for non-Wiki datasets."""+ declared = extract_declared_path(record)+ if declared is not None:+ return declaredfor value in record.values():if isinstance(value, str) and "/" in value:return normalize_path(value)@@ -396,7 +545,17 @@ def extract_path(record: dict[str, Any]) -> str:def record_id(record: dict[str, Any], fallback: int) -> str:value = pick_value(record,- ("id", "entity_id", "paper_id", "doc_id", "item_id", "source_id", "qid", "query_id"),+ (+ "id",+ "_id",+ "entity_id",+ "paper_id",+ "doc_id",+ "item_id",+ "source_id",+ "qid",+ "query_id",+ ),)return str(value) if value is not None else str(fallback)@@ -419,6 +578,65 @@ def path_matches(path_value: Any, prefix: str) -> bool:return False+def path_parts(path: str) -> tuple[str, ...]:+ return tuple(part for part in normalize_path(path).strip("/").split("/") if part)+++def common_path_prefix(paths: Sequence[str]) -> str:+ if not paths:+ return "/"+ prefix = list(path_parts(paths[0]))+ for path in paths[1:]:+ parts = path_parts(path)+ index = 0+ while index < min(len(prefix), len(parts)) and prefix[index] == parts[index]:+ index += 1+ del prefix[index:]+ if not prefix:+ return "/"+ return "/" + "/".join(prefix) if prefix else "/"+++def iter_wiki_corpus_paths(path: Path, limit: Optional[int] = None) -> Iterator[tuple[str, str]]:+ for index, item in enumerate(iter_json_array_records(path, limit)):+ item_id = record_id(item, index)+ if item_id != str(index):+ raise ValueError(+ f"Wiki corpus path mapping is not row-aligned at row {index}: id={item_id}"+ )+ raw_path = item.get("path")+ normalized = normalize_path(raw_path)+ if normalized == "/":+ raise ValueError(f"Wiki corpus path mapping has an empty root path at row {index}")+ yield item_id, normalized+++def collect_wiki_corpus_paths(path: Path, corpus_ids: set[str]) -> dict[str, str]:+ if not corpus_ids:+ return {}+ found: dict[str, str] = {}+ for item_id, item_path in iter_wiki_corpus_paths(path):+ if item_id in corpus_ids:+ found[item_id] = item_path+ if len(found) == len(corpus_ids):+ break+ missing = corpus_ids - found.keys()+ if missing:+ preview = ", ".join(sorted(missing)[:5])+ raise ValueError(+ f"Wiki corpus path mapping is missing {len(missing)} ground-truth ids: {preview}"+ )+ return found+++def derived_gt_lca_scope(items: Sequence[GroundTruthItem], corpus_paths: dict[str, str]) -> str:+ if not items:+ return "/"+ highest_score = max(item.score for item in items)+ relevant_paths = [corpus_paths[item.corpus_id] for item in items if item.score == highest_score]+ return common_path_prefix(relevant_paths)++def synthetic_vector(rng: random.Random, dim: int) -> list[float]:return [rng.uniform(-1.0, 1.0) for _ in range(dim)]@@ -542,8 +760,8 @@ def build_synthetic_workload(options: BenchOptions) -> Workload:)-def load_ground_truth(path: Path) -> dict[str, list[str]]:- result: dict[str, list[str]] = {}+def load_scored_ground_truth(path: Path) -> dict[str, list[GroundTruthItem]]:+ result: dict[str, list[GroundTruthItem]] = {}with path.open("r", encoding="utf-8") as handle:for index, line in enumerate(handle):stripped = line.strip()@@ -554,20 +772,29 @@ def load_ground_truth(path: Path) -> dict[str, list[str]]:if parts[0].lower() in {"query-id", "query_id"}:continueif len(parts) >= 3 and is_number(parts[2]):- result.setdefault(str(parts[0]), []).append(str(parts[1]))+ result.setdefault(str(parts[0]), []).append(+ GroundTruthItem(str(parts[1]), float(parts[2]))+ )continueif len(parts) >= 2:- result[str(parts[0])] = [item for item in parts[1:] if item]+ result[str(parts[0])] = [GroundTruthItem(item) for item in parts[1:] if item]continueelse:parts = stripped.replace(",", " ").split()if len(parts) < 2:continue# Plain-text ground-truth files omit query ids; line number is the query id.- result[str(index)] = [item for item in parts if item]+ result[str(index)] = [GroundTruthItem(item) for item in parts if item]return result+def load_ground_truth(path: Path) -> dict[str, list[str]]:+ return {+ query_id: [item.corpus_id for item in items]+ for query_id, items in load_scored_ground_truth(path).items()+ }++def is_number(value: str) -> bool:try:float(value)@@ -612,6 +839,50 @@ def build_dir_vector_workload(options: BenchOptions) -> Workload:else {record_id(meta, index) for index, meta in enumerate(sample_metadata)})+ query_records = load_query_records(paths["queries"], query_limit if options.full else None)+ if options.full and len(query_records) != total_queries:+ raise ValueError(+ f"query metadata/vector count mismatch: {len(query_records)} vs {total_queries}"+ )+ query_vectors = list(iter_fvecs(paths["query_vectors"], len(query_records)))+ if len(query_vectors) != len(query_records):+ raise ValueError(+ f"query metadata/vector count mismatch: {len(query_records)} vs {len(query_vectors)}"+ )+ scored_ground_truth = load_scored_ground_truth(paths["ground_truth"])++ effective_query_scope = (+ options.dir_vector_query_scope if options.dataset == "wiki" else "dataset"+ )+ query_ground_truth = [+ scored_ground_truth.get(record_id(query, index))+ or scored_ground_truth.get(str(index))+ or []+ for index, query in enumerate(query_records)+ ]+ missing_wiki_scopes = [+ record_id(query, index)+ for index, query in enumerate(query_records)+ if extract_declared_path(query) is None+ ]+ if (+ options.dataset == "wiki"+ and effective_query_scope == "dataset"+ and options.mode != "write-only"+ and missing_wiki_scopes+ ):+ raise ValueError(+ f"wiki query metadata has no directory constraint for "+ f"{len(missing_wiki_scopes)} queries; refusing a root-only Directory benchmark. "+ "Use a corrected dataset, or explicitly opt into the diagnostic "+ "--dir-vector-query-scope derived_gt_lca_v1 mode for Wiki."+ )++ wiki_gt_paths: dict[str, str] = {}+ if options.dataset == "wiki" and options.mode != "write-only":+ ground_truth_ids = {item.corpus_id for items in query_ground_truth for item in items}+ wiki_gt_paths = collect_wiki_corpus_paths(paths["corpus_paths"], ground_truth_ids)+def records() -> Iterator[dict[str, Any]]:vector_iter = iter_fvecs(paths["vectors"], row_limit)metadata_iter = (@@ -619,51 +890,143 @@ def build_dir_vector_workload(options: BenchOptions) -> Workload:if sample_metadata is not Noneelse iter_json_records(paths["corpus"], row_limit))- for index, (meta, vector) in enumerate(zip(metadata_iter, vector_iter, strict=False)):+ if options.dataset == "wiki":+ mapping_iter = iter_wiki_corpus_paths(+ paths["corpus_paths"], None if options.full else row_limit+ )+ combined: Iterable[tuple[dict[str, Any], list[float], tuple[str, str]]] = zip(+ metadata_iter, vector_iter, mapping_iter, strict=True+ )+ else:+ combined = (+ (+ meta,+ vector,+ (+ record_id(meta, index),+ extract_legacy_path(meta),+ ),+ )+ for index, (meta, vector) in enumerate(zip(metadata_iter, vector_iter, strict=True))+ )++ for index, (meta, vector, (mapped_id, mapped_path)) in enumerate(combined):rid = record_id(meta, index)- path = extract_path(meta)+ if options.dataset == "wiki" and (rid != mapped_id or rid != str(index)):+ raise ValueError(+ "Wiki corpus metadata/path/vector rows are not aligned at "+ f"row {index}: metadata_id={rid}, path_id={mapped_id}"+ )+ if options.dataset == "wiki" and mapped_path == "/":+ raise ValueError(f"dir-vector corpus row {index} has no declared directory path")category = str(pick_value(meta, ("category", "primary_category", "type")) or "")title = str(pick_value(meta, ("title", "name", "label")) or rid)yield ov_context_record(record_id=rid,vector=vector,- uri=resource_uri(options.dataset, path, rid),+ uri=resource_uri(options.dataset, mapped_path, rid),index=index,category=category,content=title,)- query_records = load_query_records(paths["queries"], query_limit if options.full else None)- query_vectors = list(iter_fvecs(paths["query_vectors"], len(query_records)))- ground_truth = load_ground_truth(paths["ground_truth"])- covered_cases: list[QueryCase] = []- fallback_cases: list[QueryCase] = []- for index, (query_meta, vector) in enumerate(zip(query_records, query_vectors, strict=False)):+ base_covered: list[QueryCase] = []+ base_fallback: list[QueryCase] = []+ directory_covered: list[QueryCase] = []+ directory_fallback: list[QueryCase] = []+ missing_dataset_scopes: list[str] = []+ root_derived_scopes = 0+ for index, (query_meta, vector) in enumerate(zip(query_records, query_vectors, strict=True)):qid = record_id(query_meta, index)- gt = [str(item) for item in (ground_truth.get(qid) or ground_truth.get(str(index)) or [])]+ scored_gt = query_ground_truth[index]+ full_gt = [item.corpus_id for item in scored_gt]+ gt = full_gtif sampled_ids is not None:gt = [item for item in gt if item in sampled_ids]- case = QueryCase(++ declared_scope = (+ extract_declared_path(query_meta)+ if options.dataset == "wiki"+ else extract_legacy_path(query_meta)+ )+ directory_scope = declared_scope or "/"+ if effective_query_scope == "derived_gt_lca_v1" and options.mode != "write-only":+ directory_scope = derived_gt_lca_scope(scored_gt, wiki_gt_paths)+ elif options.dataset == "wiki" and directory_scope == "/":+ missing_dataset_scopes.append(qid)++ base_case = QueryCase(query_id=qid,vector=vector,- filter_path=resource_uri(options.dataset, extract_path(query_meta)),+ filter_path=resource_uri(options.dataset, directory_scope),ground_truth_ids=gt,)if gt or sampled_ids is None:- covered_cases.append(case)+ base_covered.append(base_case)+ else:+ base_fallback.append(base_case)++ if options.dataset == "wiki" and directory_scope == "/":+ if effective_query_scope == "derived_gt_lca_v1":+ root_derived_scopes += 1+ continue+ directory_gt = (+ [item for item in gt if path_matches(wiki_gt_paths[item], directory_scope)]+ if options.dataset == "wiki"+ else gt+ )+ directory_case = QueryCase(+ query_id=qid,+ vector=vector,+ filter_path=resource_uri(options.dataset, directory_scope),+ ground_truth_ids=directory_gt,+ )+ if directory_gt or sampled_ids is None:+ directory_covered.append(directory_case)else:- fallback_cases.append(case)- cases = covered_cases[:query_limit]- if len(cases) < query_limit:- cases.extend(fallback_cases[: query_limit - len(cases)])+ directory_fallback.append(directory_case)++ def select_cases(covered: list[QueryCase], fallback: list[QueryCase]) -> list[QueryCase]:+ selected = covered[:query_limit]+ if len(selected) < query_limit:+ selected.extend(fallback[: query_limit - len(selected)])+ return selected++ cases = select_cases(base_covered, base_fallback)+ directory_cases = select_cases(directory_covered, directory_fallback)if not cases:raise ValueError("dir-vector query set is empty")+ if options.mode != "write-only" and missing_dataset_scopes:+ raise ValueError(+ f"{options.dataset} query metadata has no directory constraint for "+ f"{len(missing_dataset_scopes)} queries; refusing a root-only Directory benchmark. "+ "Use a corrected dataset, or explicitly opt into the diagnostic "+ "--dir-vector-query-scope derived_gt_lca_v1 mode for Wiki."+ )+ if options.mode != "write-only" and not directory_cases:+ raise ValueError("dir-vector Directory query set is empty after path validation")++ scope_depths = [len(path_parts(case.filter_path)) - 3 for case in directory_cases]+ scope_depths = [max(0, depth) for depth in scope_depths]+ path_diagnostics = {+ "corpus_path_source": paths.get("corpus_paths", paths["corpus"]).name,+ "query_scope_source": effective_query_scope,+ "base_query_count": len(cases),+ "directory_query_count": len(directory_cases),+ "unique_directory_scopes": len({case.filter_path for case in directory_cases}),+ "root_derived_scopes_omitted": root_derived_scopes,+ "directory_scope_depth_min": min(scope_depths) if scope_depths else None,+ "directory_scope_depth_median": statistics.median(scope_depths) if scope_depths else None,+ "directory_scope_depth_max": max(scope_depths) if scope_depths else None,+ }return Workload(name=f"dir-vector:{options.dataset}",dim=dim,records=records,queries=cases,expected_rows=row_limit,+ directory_queries=directory_cases,+ path_diagnostics=path_diagnostics,)@@ -901,7 +1264,9 @@ async def run_benchmark_async(options: BenchOptions) -> RunResult:progress("start "f"run_id={options.run_id} workload={workload.name} mode={options.mode} "- f"records={workload.expected_rows} queries={len(workload.queries)} dim={workload.dim}"+ f"records={workload.expected_rows} base_queries={len(workload.queries)} "+ f"directory_queries={len(directory_queries(workload))} "+ f"dim={workload.dim}")progress(f"collection={collection_name}")@@ -1023,6 +1388,7 @@ async def run_benchmark_async(options: BenchOptions) -> RunResult:)selected_queries = workload.queries+ selected_directory_queries = directory_queries(workload)search_events, errors, vector_quality = await run_search_phase(backend=backend,ctx=ctx,@@ -1040,7 +1406,7 @@ async def run_benchmark_async(options: BenchOptions) -> RunResult:backend=backend,ctx=ctx,phase="filtered_vector_search",- queries=selected_queries,+ queries=selected_directory_queries,top_k=options.top_k,concurrency=options.concurrency,filtered=True,@@ -1049,7 +1415,7 @@ async def run_benchmark_async(options: BenchOptions) -> RunResult:validation_errors.extend(errors)quality["filtered_vector_search"] = filtered_quality- first_filter = selected_queries[0].filter_path+ first_filter = selected_directory_queries[0].filter_pathevent, _ = await async_timed_event("validate","count_filtered",@@ -1112,10 +1478,12 @@ def finalize_result("dataset": options.dataset if options.workload == "dir-vector" else "synthetic","record_count": workload.expected_rows,"query_count": len(workload.queries),+ "directory_query_count": len(directory_queries(workload)),"vector_dim": workload.dim,"top_k": options.top_k,"full": options.full,"profile": options.profile,+ "path_diagnostics": workload.path_diagnostics,},environment=collect_environment(),kept_collection=kept_collection,@@ -1295,6 +1663,7 @@ def write_suite_outputs(output_dir: Path, runs: list[tuple[BenchOptions, RunResu"records": result.workload.get("record_count"),"inserted": metric_value(run_summaries[run_options.dataset], "ingest", "count"),"queries": result.workload.get("query_count"),+ "directory_queries": result.workload.get("directory_query_count"),"dim": result.workload.get("vector_dim"),"top_k": result.workload.get("top_k"),"vector_qps": metric_value(@@ -1361,7 +1730,10 @@ def quality_rate(quality: dict[str, Any], phase: str) -> str:return f"{float(value) * 100.0:.2f}%"-def recall_scope(result: RunResult) -> str:+def recall_scope(result: RunResult, phase: str = "filtered_vector_search") -> str:+ query_scope_source = result.workload.get("path_diagnostics", {}).get("query_scope_source")+ if phase == "filtered_vector_search" and query_scope_source and query_scope_source != "dataset":+ return str(query_scope_source)if result.workload.get("workload") == "dir-vector" and not result.workload.get("full"):return "sampled_subset"return "official_full"@@ -1377,6 +1749,7 @@ def workload_rows(result: RunResult, summary_rows: list[dict[str, Any]]) -> list"records": workload.get("record_count"),"inserted": metric_value(summary_rows, "ingest", "count"),"queries": workload.get("query_count"),+ "directory_queries": workload.get("directory_query_count"),"dim": workload.get("vector_dim"),"top_k": workload.get("top_k"),"full": workload.get("full"),@@ -1394,7 +1767,7 @@ def recall_qps_rows("qps": metric_value(summary_rows, "vector_search", "throughput_per_sec"),"avg_ms": metric_value(summary_rows, "vector_search", "avg_ms"),"p95_ms": metric_value(summary_rows, "vector_search", "p95_ms"),- "scope": recall_scope(result),+ "scope": recall_scope(result, "vector_search"),recall_header: quality_rate(result.quality, "vector_search"),},{@@ -1402,7 +1775,7 @@ def recall_qps_rows("qps": metric_value(summary_rows, "filtered_vector_search", "throughput_per_sec"),"avg_ms": metric_value(summary_rows, "filtered_vector_search", "avg_ms"),"p95_ms": metric_value(summary_rows, "filtered_vector_search", "p95_ms"),- "scope": recall_scope(result),+ "scope": recall_scope(result, "filtered_vector_search"),recall_header: quality_rate(result.quality, "filtered_vector_search"),},]@@ -1438,11 +1811,19 @@ def build_markdown_report("",markdown_table(workload_rows(result, summary_rows)),"",+ "## Directory path 口径",+ "",+ f"- Corpus path source: `{result.workload.get('path_diagnostics', {}).get('corpus_path_source', 'generated')}`",+ f"- Query scope source: `{result.workload.get('path_diagnostics', {}).get('query_scope_source', 'generated')}`",+ f"- Directory queries: `{result.workload.get('directory_query_count')}`",+ f"- Unique directory scopes: `{result.workload.get('path_diagnostics', {}).get('unique_directory_scopes', '-')}`",+ f"- Root-derived scopes omitted: `{result.workload.get('path_diagnostics', {}).get('root_derived_scopes_omitted', 0)}`",+ "","## 召回与 QPS","",markdown_table(recall_qps_rows(result, summary_rows, options)),"",- "gt_recall@K 按 query 的 ground truth 命中率统计;`sampled_subset` 只用于 smoke sanity,不代表官方全量 recall。官方 recall 请用 `--full`。",+ "gt_recall@K 按 query 的 ground truth 命中率统计;`sampled_subset` 只用于 smoke sanity,不代表官方全量 recall。`derived_gt_lca_v1` 使用 ground truth 派生目录,只是诊断 workload,也不代表发布数据集的官方 query constraint。","","## 性能汇总","",@@ -1551,6 +1932,17 @@ def parse_args(argv: Optional[Sequence[str]] = None) -> BenchOptions:parser.add_argument("--path-depth", type=int)parser.add_argument("--path-fanout", type=int)parser.add_argument("--filter-selectivity", type=float)+ parser.add_argument(+ "--dir-vector-query-scope",+ choices=DIR_VECTOR_QUERY_SCOPES,+ default="dataset",+ help=(+ "Directory scope source for dir-vector queries. dataset requires published query "+ "constraints; derived_gt_lca_v1 gives Wiki an explicit diagnostic scope derived "+ "from the deepest common path of highest-score ground-truth items. Other datasets "+ "continue to use their published constraints."+ ),+ )parser.add_argument("--seed", type=int, default=42)parser.add_argument("--distance", choices=["ip", "l2", "cosine"])parser.add_argument("--drop-at-end", action="store_true", help="Drop benchmark collection")@@ -1588,6 +1980,7 @@ def parse_args(argv: Optional[Sequence[str]] = None) -> BenchOptions:seed=args.seed,distance=args.distance,drop_at_end=args.drop_at_end,+ dir_vector_query_scope=args.dir_vector_query_scope,)diff --git a/tests/benchmark/test_vectordb_perf_run.py b/tests/benchmark/test_vectordb_perf_run.pyindex fed1eb9e7..76bc97a33 100644--- a/tests/benchmark/test_vectordb_perf_run.py+++ b/tests/benchmark/test_vectordb_perf_run.py@@ -1,9 +1,101 @@# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.# SPDX-License-Identifier: AGPL-3.0+import json+import struct+import pytest-from benchmark.vectordb_perf.run import upsert_records+from benchmark.vectordb_perf.run import (+ RunResult,+ build_dir_vector_workload,+ directory_queries,+ extract_path,+ iter_json_array_records,+ parse_args,+ recall_scope,+ record_id,+ upsert_records,+)+++def _write_fvecs(path, vectors):+ with path.open("wb") as handle:+ for vector in vectors:+ handle.write(struct.pack("<i", len(vector)))+ handle.write(struct.pack(f"<{len(vector)}f", *vector))+++def _write_wiki_fixture(tmp_path, *, query_paths=False, mapping_ids=(0, 1, 2)):+ corpus = [+ {"_id": 0, "title": "pronounced /not-a-directory/", "text": "zero"},+ {"_id": 1, "title": "one", "text": "one"},+ {"_id": 2, "title": "two", "text": "two"},+ ]+ paths = [+ {"id": mapping_ids[0], "path": "/science/physics/quantum/"},+ {"id": mapping_ids[1], "path": "/science/physics/classical/"},+ {"id": mapping_ids[2], "path": "/history/ancient/"},+ ]+ queries = [+ {"_id": 0, "text": "physics", "metadata": {}},+ {"_id": 1, "text": "mixed", "metadata": {}},+ {"_id": 2, "text": "history", "metadata": {}},+ ]+ if query_paths:+ queries[0]["metadata"] = {"path": "/science/physics"}+ queries[1]["metadata"] = {"path": "/science"}+ queries[2]["metadata"] = {"path": "/history"}++ (tmp_path / "dbpedia_dir_2m_corpus.jsonl").write_text(+ "".join(json.dumps(item) + "\n" for item in corpus), encoding="utf-8"+ )+ (tmp_path / "dbpedia_dir_2m_corpus_paths.json").write_text(json.dumps(paths), encoding="utf-8")+ (tmp_path / "dbpedia_dir_2m_query.jsonl").write_text(+ "".join(json.dumps(item) + "\n" for item in queries), encoding="utf-8"+ )+ (tmp_path / "dbpedia_dir_2m_groundtruth.tsv").write_text(+ "query-id\tcorpus-id\tscore\n0\t0\t2\n0\t1\t2\n1\t0\t2\n1\t2\t2\n2\t2\t1\n",+ encoding="utf-8",+ )+ vectors = [[float(index), 0.0, 0.0, 1.0] for index in range(3)]+ _write_fvecs(tmp_path / "dbpedia_dir_2m_corpus_vectors.fvecs", vectors)+ _write_fvecs(tmp_path / "dbpedia_dir_2m_query_vectors.fvecs", vectors)+++def _wiki_options(tmp_path, *extra):+ return parse_args(+ [+ "--workload",+ "dir-vector",+ "--dataset",+ "wiki",+ "--dataset-root",+ str(tmp_path),+ "--rows",+ "3",+ "--queries",+ "3",+ *extra,+ ]+ )+++def _write_arxiv_public_shape_fixture(tmp_path):+ corpus = [+ {"id": "0", "categories": ["cs.AI", "cs.LG"], "time": "2024/01"},+ {"id": "1", "categories": ["math.OC"], "time": "2024/02"},+ ]+ queries = [+ {"query_id": "0", "constraints": {"time": "2024/01"}},+ {"query_id": "1", "constraints": {"category": "math.OC"}},+ ]+ (tmp_path / "arxiv_corpus_metadata.json").write_text(json.dumps(corpus), encoding="utf-8")+ (tmp_path / "arxiv_query_constraint.json").write_text(json.dumps(queries), encoding="utf-8")+ (tmp_path / "arxiv_ground_truth.txt").write_text("0 1\n1 0\n", encoding="utf-8")+ vectors = [[0.0, 1.0, 0.0, 0.0], [1.0, 0.0, 0.0, 0.0]]+ _write_fvecs(tmp_path / "arxiv_corpus_vectors.fvecs", vectors)+ _write_fvecs(tmp_path / "arxiv_query_vectors.fvecs", vectors)@pytest.mark.asyncio@@ -41,3 +133,202 @@ async def test_upsert_records_rejects_incomplete_bulk_result():object(),[{"id": "rec-1"}, {"id": "rec-2"}],)+++def test_iter_json_array_records_streams_across_small_chunks(tmp_path):+ path = tmp_path / "records.json"+ expected = [{"id": 0, "path": "/α/β"}, {"id": 1, "path": '/quoted/"x"'}]+ path.write_text(json.dumps(expected, ensure_ascii=False), encoding="utf-8")++ assert list(iter_json_array_records(path, chunk_size=3)) == expected+ assert list(iter_json_array_records(path, limit=1, chunk_size=2)) == expected[:1]+ assert list(iter_json_array_records(path, limit=0, chunk_size=2)) == []+++ "payload, message",+ [+ ('[{"id": 0},]', "trailing comma"),+ ('[{"id": 0}] garbage', "unexpected content"),+ ('[{"id": 0}', "truncated JSON array"),+ ],+)+def test_iter_json_array_records_rejects_invalid_full_input(tmp_path, payload, message):+ path = tmp_path / "invalid.json"+ path.write_text(payload, encoding="utf-8")++ with pytest.raises(ValueError, match=message):+ list(iter_json_array_records(path, chunk_size=2))+++def test_path_extraction_does_not_treat_prose_slashes_as_a_directory():+ assert extract_path({"title": "pronounced /foo/", "text": "a/b"}) == "/"+ assert record_id({"_id": 17}, 0) == "17"+++def test_wiki_mapping_and_derived_scopes_use_real_hierarchy(tmp_path):+ _write_wiki_fixture(tmp_path)+ workload = build_dir_vector_workload(+ _wiki_options(tmp_path, "--dir-vector-query-scope", "derived_gt_lca_v1")+ )++ records = list(workload.records())+ assert [record["uri"] for record in records] == [+ "viking://resources/bench/wiki/science/physics/quantum/0",+ "viking://resources/bench/wiki/science/physics/classical/1",+ "viking://resources/bench/wiki/history/ancient/2",+ ]+ assert len(workload.queries) == 3+ assert [case.query_id for case in workload.directory_queries] == ["0", "2"]+ assert workload.directory_queries[0].filter_path.endswith("/science/physics")+ assert workload.directory_queries[0].ground_truth_ids == ["0", "1"]+ assert workload.path_diagnostics == {+ "corpus_path_source": "dbpedia_dir_2m_corpus_paths.json",+ "query_scope_source": "derived_gt_lca_v1",+ "base_query_count": 3,+ "directory_query_count": 2,+ "unique_directory_scopes": 2,+ "root_derived_scopes_omitted": 1,+ "directory_scope_depth_min": 2,+ "directory_scope_depth_median": 2.0,+ "directory_scope_depth_max": 2,+ }+++def test_wiki_dataset_scope_refuses_missing_query_constraints(tmp_path):+ _write_wiki_fixture(tmp_path)++ with pytest.raises(ValueError, match="refusing a root-only Directory benchmark"):+ build_dir_vector_workload(_wiki_options(tmp_path))+++def test_wiki_write_only_preserves_an_intentionally_empty_directory_set(tmp_path):+ _write_wiki_fixture(tmp_path)++ workload = build_dir_vector_workload(_wiki_options(tmp_path, "--mode", "write-only"))++ assert workload.directory_queries == []+ assert directory_queries(workload) == []+++def test_wiki_dataset_scope_accepts_explicit_query_paths(tmp_path):+ _write_wiki_fixture(tmp_path, query_paths=True)++ workload = build_dir_vector_workload(_wiki_options(tmp_path))++ assert len(workload.directory_queries) == 3+ assert workload.path_diagnostics["query_scope_source"] == "dataset"+ assert workload.path_diagnostics["unique_directory_scopes"] == 3+++def test_wiki_mapping_requires_row_aligned_ids(tmp_path):+ _write_wiki_fixture(tmp_path, mapping_ids=(0, 9, 2))++ with pytest.raises(ValueError, match="not row-aligned at row 1"):+ build_dir_vector_workload(+ _wiki_options(tmp_path, "--dir-vector-query-scope", "derived_gt_lca_v1")+ )+++def test_wiki_mapping_requires_all_rows_in_full_mode(tmp_path):+ _write_wiki_fixture(tmp_path)+ path = tmp_path / "dbpedia_dir_2m_corpus_paths.json"+ mapping = json.loads(path.read_text(encoding="utf-8"))+ mapping.append({"id": 3, "path": "/extra"})+ path.write_text(json.dumps(mapping), encoding="utf-8")+ workload = build_dir_vector_workload(+ _wiki_options(tmp_path, "--full", "--dir-vector-query-scope", "derived_gt_lca_v1")+ )++ with pytest.raises(ValueError, match=r"zip\(\) argument 3 is longer"):+ list(workload.records())+++def test_wiki_mapping_rejects_missing_ground_truth_rows(tmp_path):+ _write_wiki_fixture(tmp_path)+ path = tmp_path / "dbpedia_dir_2m_corpus_paths.json"+ mapping = json.loads(path.read_text(encoding="utf-8"))+ path.write_text(json.dumps(mapping[:2]), encoding="utf-8")++ with pytest.raises(ValueError, match="missing 1 ground-truth ids"):+ build_dir_vector_workload(+ _wiki_options(tmp_path, "--dir-vector-query-scope", "derived_gt_lca_v1")+ )+++def test_wiki_mapping_rejects_root_path(tmp_path):+ _write_wiki_fixture(tmp_path)+ path = tmp_path / "dbpedia_dir_2m_corpus_paths.json"+ mapping = json.loads(path.read_text(encoding="utf-8"))+ mapping[0]["path"] = None+ path.write_text(json.dumps(mapping), encoding="utf-8")++ with pytest.raises(ValueError, match="empty root path at row 0"):+ build_dir_vector_workload(+ _wiki_options(tmp_path, "--dir-vector-query-scope", "derived_gt_lca_v1")+ )+++def test_full_wiki_requires_query_metadata_for_every_vector(tmp_path):+ _write_wiki_fixture(tmp_path)+ path = tmp_path / "dbpedia_dir_2m_query.jsonl"+ lines = path.read_text(encoding="utf-8").splitlines()+ path.write_text("\n".join(lines[:2]) + "\n", encoding="utf-8")++ with pytest.raises(ValueError, match="query metadata/vector count mismatch: 2 vs 3"):+ build_dir_vector_workload(+ _wiki_options(tmp_path, "--full", "--dir-vector-query-scope", "derived_gt_lca_v1")+ )+++def test_derived_scope_only_labels_filtered_phase(tmp_path):+ _write_wiki_fixture(tmp_path)+ workload = build_dir_vector_workload(+ _wiki_options(tmp_path, "--full", "--dir-vector-query-scope", "derived_gt_lca_v1")+ )+ result = RunResult(+ run_id="test",+ collection_name="test",+ output_dir=str(tmp_path),+ events=[],+ validation_errors=[],+ quality={},+ workload={+ "workload": "dir-vector",+ "full": True,+ "path_diagnostics": workload.path_diagnostics,+ },+ environment={},+ kept_collection=True,+ )++ assert recall_scope(result, "vector_search") == "official_full"+ assert recall_scope(result, "filtered_vector_search") == "derived_gt_lca_v1"++ result.workload["path_diagnostics"] = {}+ assert recall_scope(result, "filtered_vector_search") == "official_full"+++def test_derived_wiki_option_keeps_non_wiki_loader_behavior(tmp_path):+ _write_arxiv_public_shape_fixture(tmp_path)+ options = parse_args(+ [+ "--workload",+ "dir-vector",+ "--dataset",+ "arxiv",+ "--dataset-root",+ str(tmp_path),+ "--rows",+ "2",+ "--queries",+ "2",+ "--dir-vector-query-scope",+ "derived_gt_lca_v1",+ ]+ )++ workload = build_dir_vector_workload(options)++ assert workload.path_diagnostics["query_scope_source"] == "dataset"+ assert len(workload.directory_queries) == 2