Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(benchmark): 增加 OV 目录向量检索性能 benchmark (#3076)
Qin Haojie2eb61fbbad82
diff --git a/benchmark/.gitignore b/benchmark/.gitignoreindex fbca22537..760ae03e6 100644--- a/benchmark/.gitignore+++ b/benchmark/.gitignore@@ -1 +1,2 @@results/+./vectordb_perf/*.confdiff --git a/benchmark/vectordb_perf/README.md b/benchmark/vectordb_perf/README.mdnew file mode 100644index 000000000..e45d0c7e1--- /dev/null+++ b/benchmark/vectordb_perf/README.md@@ -0,0 +1,331 @@+# OpenViking Vector Backend 性能 Benchmark++这个 benchmark 用来快速验收新的 VectorDB storage backend 在 OpenViking 场景下的表现。+它不直接调用 `CollectionAdapter`,而是走 `VikingVectorIndexBackend`:++- 建表使用 `CollectionSchemas.context_collection`+- 写入使用 `VikingVectorIndexBackend.upsert(..., ctx=...)`+- 查询使用 `VikingVectorIndexBackend.search_in_tenant(...)`+- 目录范围使用 `target_directories`,内部会编译成 `PathScope("uri", ..., depth=-1)`++它仍然不经过 OpenViking Server、AGFS、embedding 服务和 rerank;向量直接来自模拟数据或+dir-vector-dataset 的 `.fvecs` 文件。++## 先选模式++| 模式 | 参数 | 数据来源 | 适合场景 |+| --- | --- | --- | --- |+| 模拟数据 | `--workload synthetic` | runner 生成向量和目录路径 | 新 storage backend 首次接入、稳定复现、CI smoke |+| 真实数据 | `--workload dir-vector` | dir-vector-dataset 的 metadata、fvecs、ground truth | 目录过滤 + 向量检索的真实 workload 验收 |++默认是 `synthetic`。++## 再选规模++| 规模 | 参数 | synthetic 行为 | dir-vector 行为 |+| --- | --- | --- | --- |+| 少量 smoke | `--profile smoke` | 生成 128 行、8 个 query | 抽样前 128 行、前 8 个 query |+| 常规 standard | `--profile standard` | 生成 10000 行、100 个 query | 抽样前 10000 行、前 100 个 query |+| 压力 stress | `--profile stress` | 生成 100000 行、500 个 query | 抽样前 100000 行、前 500 个 query |+| 全量真实数据 | `--workload dir-vector --full` | 不适用 | 读取 dataset 全量 corpus 和 query |++`--rows`、`--queries`、`--batch-size`、`--concurrency`、`--top-k` 可以覆盖 profile 默认值。+`--full` 只对 `dir-vector` 生效;真实数据的向量维度从 `.fvecs` 读取,`--dim` 只影响 synthetic。++## 再选读写阶段++| 阶段模式 | 参数 | 行为 |+| --- | --- | --- |+| 读写一体 | `--mode read-write` | 默认模式;创建 collection、写入 OV context row,然后跑 count/get/search |+| 只写不读 | `--mode write-only` | 创建 collection 并 upsert;不跑 count、get、向量检索、过滤检索 |+| 只读不写 | `--mode read-only` | 不创建 collection、不 upsert;直接连接同名 collection 跑 count/get/search |++`read-only` 用于压测一个已经准备好的 collection。它必须和写入阶段使用相同的 `--config`、+配置里的 `vectordb.name`、`--run-id`、`--workload`、`--dataset` / `--full` / 抽样参数和+`--distance`,否则 runner 可能连到不同 collection,或用不同 query/ground truth 口径算报告。++`read-only` 不允许 `--drop-at-end`。如果想先准备数据再反复测读性能,先固定 `--run-id` 跑一次+`write-only`,后续用同一个 `--run-id` 跑 `read-only`。++## 快速开始++先准备一个 `ov.conf`,最小本地配置如下:++```json+{+ "storage": {+ "workspace": "./benchmark/results/vectordb_perf/local_data",+ "vectordb": {+ "backend": "local",+ "name": "vectordb_perf",+ "project": "default",+ "index_name": "default",+ "distance_metric": "cosine"+ }+ }+}+```++模拟数据少量 smoke:++```bash+.venv/bin/python -m benchmark.vectordb_perf.run \+ --config ./ov.conf \+ --workload synthetic \+ --profile smoke+```++模拟数据常规 benchmark:++```bash+.venv/bin/python -m benchmark.vectordb_perf.run \+ --config ./ov.conf \+ --workload synthetic \+ --profile standard+```++真实数据必须先下载。`--workload dir-vector` 默认会跑 `wiki` 和 `arxiv` 两个真实数据集,+runner 不会自动下载数据,也不会从 GitHub clone 后直接生成 `.fvecs`。先到+dir-vector-dataset 仓库 README 的 File Download 链接下载这两组数据文件:++https://github.com/KurtPatrickHere/dir-vector-dataset++下载后把文件解压或移动到同一个本地目录,例如:++```bash+mkdir -p benchmark/data/dir-vector-dataset-files+# 把下载得到的 corpus/query/vector/ground-truth 文件放进这个目录+```++`--dataset-root` 必须指向“直接包含数据文件”的目录,不是 GitHub repo 目录,也不是一个空目录。+runner 会先校验 `wiki` 和 `arxiv` 所需文件;缺文件时会直接列出 missing files 和下载地址。+调试时可以用 `--dataset wiki` 或 `--dataset arxiv` 只跑单个数据集。各 dataset 对应文件见下方“真实数据文件”。++真实数据少量抽样:++```bash+.venv/bin/python -m benchmark.vectordb_perf.run \+ --config ./ov.conf \+ --workload dir-vector \+ --dataset-root benchmark/data/dir-vector-dataset-files \+ --profile smoke+```++真实数据全量:++```bash+.venv/bin/python -m benchmark.vectordb_perf.run \+ --config ./ov.conf \+ --workload dir-vector \+ --dataset-root ./benchmark/data/dir-vector-dataset-files \+ --full+```++默认保留测试 collection,方便复查。需要运行后清理:++```bash+--drop-at-end+```++runner 会把配置里的 `name` 改成 `<name>_bench_<run-id>`,避免覆盖正式 collection。++分离写入和读取时要固定 `--run-id`。例如先写入:++```bash+.venv/bin/python -m benchmark.vectordb_perf.run \+ --config ./ov.conf \+ --workload dir-vector \+ --dataset-root ./benchmark/data/dir-vector-dataset-files \+ --full \+ --run-id ovbench_full_001 \+ --mode write-only+```++再只跑读取和检索:++```bash+.venv/bin/python -m benchmark.vectordb_perf.run \+ --config ./ov.conf \+ --workload dir-vector \+ --dataset-root ./benchmark/data/dir-vector-dataset-files \+ --full \+ --run-id ovbench_full_001 \+ --mode read-only \+ --output-dir benchmark/results/vectordb_perf/ovbench_full_001_read+```++## 写入数据结构++两种 workload 最终都会转换成 OV context row,写入字段如下:++| 字段 | 示例 | 说明 |+| --- | --- | --- |+| `id` | `syn-1` | 主键;真实数据使用原始 doc id |+| `uri` | `viking://resources/bench/synthetic/d0_1/d1_0/syn-1` | OV URI;目录过滤只看这个字段 |+| `type` | `file` | OV context type 内的资源类型 |+| `context_type` | `resource` | 固定按资源检索 |+| `vector` | `[0.1, ...]` | dense vector;synthetic 生成,dir-vector 读取 `.fvecs` |+| `created_at` / `updated_at` | `2026-01-01T00:00:01Z` | 确定性时间戳 |+| `active_count` | `0` | OV 访问计数 |+| `level` | `2` | L2 detail/content 层;查询也限制 `level=[2]` |+| `name` | `syn-1` | 展示名 |+| `description` / `tags` / `search_tags` | `cat_1` | 标量字段 |+| `abstract` | `source_id:syn-1 category:cat_1` | 检索返回字段,也用于 ground truth 轻量匹配 |+| `content` | `benchmark content ...` | text 字段,匹配真实 OV schema |+| `account_id` | `bench_account` | 用于 backend tenant filter |+| `owner_user_id` | `bench_user` | OV owner 字段 |++所以这个 benchmark 不是“普通向量库 row + filter”,而是 OV 的 context collection row。++## 测试阶段++每次运行会创建一个带 run id 后缀的测试 collection,然后执行:++| 阶段 | 内容 |+| --- | --- |+| `setup` | 创建 collection 和索引 schema |+| `ingest` | 通过 backend upsert OV context row |+| `validate` | count、get、过滤 count |+| `vector_search` | `search_in_tenant`,无指定目录 |+| `filtered_vector_search` | `search_in_tenant`,带 `target_directories` |+| `cleanup` | 仅在传 `--drop-at-end` 时删除测试 collection |++功能错误会导致非零退出码;性能慢只记录到报告里。++`--mode write-only` 只会出现 `setup` / `ingest` / 可选 `cleanup` 阶段。+`--mode read-only` 只会出现 `validate` / `vector_search` / `filtered_vector_search` 阶段。+运行中会向 stderr 输出进度,包括当前 collection、写入条数、查询条数和速率。++进度输出示例:++```text+[vectordb_perf] start run_id=ovbench_full_001 workload=dir-vector:wiki mode=write-only records=1941679 queries=456 dim=1024+[vectordb_perf] collection=vectordb_perf_bench_ovbench_full_001_wiki+[vectordb_perf] setup: create collection+[vectordb_perf] setup: collection ready+[vectordb_perf] ingest: start batch_size=1000+[vectordb_perf] ingest: 120000/1941679 (6.2%), 3950.4/s+[vectordb_perf] ingest: 240000/1941679 (12.4%), 4021.7/s+```++## 真实数据文件++`--workload dir-vector` 使用+[KurtPatrickHere/dir-vector-dataset](https://github.com/KurtPatrickHere/dir-vector-dataset)+发布的数据文件。需要先手动下载;下载入口在该仓库 README 的 File Download / Google Drive+链接里。++默认真实数据 benchmark 会跑 `wiki` 和 `arxiv` 两行。`arxiv_category` 不在默认两数据集里,+需要时显式传 `--dataset arxiv_category`。++默认两个真实数据集的全量规模如下,行数按 `.fvecs` 文件统计,也是 runner 的实际读取口径:++| `--dataset` | corpus 向量条数 | query 条数 | 向量维度 |+| --- | ---: | ---: | ---: |+| `wiki` | 1,941,679 | 456 | 1024 |+| `arxiv` | 2,763,543 | 1,000 | 1024 |++`--dataset all --full` 会依次跑这两组数据,合计 4,705,222 条 corpus 向量和 1,456 个 query。++| `--dataset` | 期望文件 |+| --- | --- |+| `wiki` | `dbpedia_dir_2m_corpus.jsonl`、`dbpedia_dir_2m_corpus_vectors.fvecs`、`dbpedia_dir_2m_query.jsonl`、`dbpedia_dir_2m_query_vectors.fvecs`、`dbpedia_dir_2m_groundtruth.tsv` |+| `arxiv` | `arxiv_corpus_metadata.json`、`arxiv_corpus_vectors.fvecs`、`arxiv_query_constraint.json`、`arxiv_query_vectors.fvecs`、`arxiv_ground_truth.txt` |+| `arxiv_category` | `arxiv_corpus_metadata.json`、`arxiv_corpus_vectors.fvecs`、`arxiv_category_query_constraint.json`、`arxiv_category_query_vectors.fvecs`、`arxiv_category_ground_truth.txt` |++把这些文件放到同一个目录,然后用 `--dataset-root` 指向该目录。++## 其他 ov.conf 示例++Volcengine VikingDB:++```json+{+ "storage": {+ "vectordb": {+ "backend": "volcengine",+ "name": "vectordb_perf",+ "project": "default",+ "index_name": "default",+ "distance_metric": "cosine",+ "volcengine": {+ "region": "cn-beijing",+ "ak": "YOUR_AK",+ "sk": "YOUR_SK"+ }+ }+ }+}+```++自定义 adapter 类:++```json+{+ "storage": {+ "vectordb": {+ "backend": "my_project.adapters.MyCollectionAdapter",+ "name": "vectordb_perf",+ "project": "default",+ "index_name": "default",+ "custom_params": {+ "endpoint": "http://127.0.0.1:9000",+ "token": "YOUR_TOKEN"+ }+ }+ }+}+```++## 常用参数++| 参数 | 说明 |+| --- | --- |+| `--config` | OpenViking 配置文件路径 |+| `--output-dir` | 报告输出目录;默认在 `benchmark/results/vectordb_perf/<run-id>/` |+| `--run-id` | 本次运行标识;会进入 collection 名和报告 |+| `--profile` | `smoke`、`standard`、`stress` |+| `--mode` | `read-write`、`write-only`、`read-only`;默认 `read-write` |+| `--workload` | `synthetic` 或 `dir-vector` |+| `--dataset-root` | dir-vector 数据目录 |+| `--dataset` | `all`、`wiki`、`arxiv`、`arxiv_category`;默认 `all`,即依次跑 `wiki` 和 `arxiv` |+| `--full` | dir-vector 全量读取 corpus 和 query;不加时按 `--rows` / `--queries` 抽样 |+| `--rows` | synthetic 行数;dir-vector 抽样行数 |+| `--queries` | 查询数 |+| `--dim` | synthetic 向量维度 |+| `--batch-size` | upsert 批大小 |+| `--concurrency` | 查询并发 |+| `--top-k` | 检索返回条数 |+| `--distance` | `ip`、`l2`、`cosine` |+| `--drop-at-end` | 运行结束后删除测试 collection |++## 报告输出++主要输出文件:++| 文件 | 内容 |+| --- | --- |+| `summary_zh.md` | 中文摘要,优先看这个;包含数据规模、Recall@K、QPS、延迟和环境 |+| `run_summary.json` | 汇总结果,适合自动化读取;包含 `workload`、`quality`、`phase_summary` |+| `events.jsonl` | 每次操作的延迟、成功状态和错误 |+| `phase_summary.csv` | 按阶段聚合的吞吐和延迟 |+| `environment.json` | runner 本机环境观测 |+| `run_config.json` | 本次运行参数 |++`summary_zh.md` 里重点看两张表:++| 表 | 内容 |+| --- | --- |+| 数据规模 | `records` 是本次计划读取的数据条数,`inserted` 是实际写入 backend 的条数,`queries` 是实际查询数 |+| 召回与 QPS | `vector_search` 和 `filtered_vector_search` 的 QPS、平均延迟、P95 延迟、gt_recall@K |++gt_recall@K 按 query 的 ground truth 命中率统计。`--full` 才是官方全量 recall;+非 `--full` 的真实数据报告会标成 `sampled_subset`,只用于 smoke sanity,不代表官方全量 recall。++## 资源限制口径++runner 会记录本机 CPU、内存、平台、GPU 探测和 cgroup 信息,但不主动限制资源。+如果 backend 连接的是远端服务,远端实例规格需要人工记录或用部署系统控制。++本地要控制资源,建议在 Docker、cgroup 或 CI runner 层限制 CPU/内存,然后把报告里的+`environment.json` 和部署规格一起归档。diff --git a/benchmark/vectordb_perf/__init__.py b/benchmark/vectordb_perf/__init__.pynew file mode 100644index 000000000..bf08d1ddb--- /dev/null+++ b/benchmark/vectordb_perf/__init__.py@@ -0,0 +1 @@+"""VectorDB adapter performance benchmark."""diff --git a/benchmark/vectordb_perf/run.py b/benchmark/vectordb_perf/run.pynew file mode 100644index 000000000..6440bcc39--- /dev/null+++ b/benchmark/vectordb_perf/run.py@@ -0,0 +1,1604 @@+#!/usr/bin/env python3+# Copyright (c) 2026 Beijing Volcano Engine Technology Co., Ltd.+# SPDX-License-Identifier: AGPL-3.0+"""OpenViking vector backend performance benchmark.++This benchmark targets OpenViking's VikingVectorIndexBackend boundary. It uses+the real OV context schema, URI scope filters, tenant context, and backend calls+while still accepting precomputed vectors from synthetic or dir-vector data.+"""++from __future__ import annotations++import argparse+import asyncio+import csv+import json+import math+import os+import platform+import random+import shutil+import struct+import subprocess+import sys+import time+import uuid+from dataclasses import asdict, dataclass, field, replace+from datetime import UTC, datetime+from pathlib import Path+from typing import Any, Callable, Iterable, Iterator, Optional, Sequence++from openviking.storage.expr import PathScope+from openviking.server.identity import RequestContext, Role+from openviking.storage.collection_schemas import CollectionSchemas+from openviking.storage.viking_vector_index_backend import VikingVectorIndexBackend+from openviking_cli.session.user_id import UserIdentifier+from openviking_cli.utils.config import OpenVikingConfigSingleton+++BENCH_ACCOUNT_ID = "bench_account"+BENCH_USER_ID = "bench_user"+DIR_VECTOR_DATASET_URL = "https://github.com/KurtPatrickHere/dir-vector-dataset"+DEFAULT_DIR_VECTOR_DATASETS = ("wiki", "arxiv")+PROGRESS_INTERVAL_SECONDS = 5.0+++PROFILE_DEFAULTS: dict[str, dict[str, Any]] = {+ "smoke": {+ "rows": 128,+ "queries": 8,+ "dim": 32,+ "batch_size": 32,+ "concurrency": 1,+ "top_k": 10,+ "path_depth": 4,+ "path_fanout": 4,+ "filter_selectivity": 0.25,+ },+ "standard": {+ "rows": 10000,+ "queries": 100,+ "dim": 1024,+ "batch_size": 500,+ "concurrency": 4,+ "top_k": 20,+ "path_depth": 6,+ "path_fanout": 8,+ "filter_selectivity": 0.05,+ },+ "stress": {+ "rows": 100000,+ "queries": 500,+ "dim": 1024,+ "batch_size": 1000,+ "concurrency": 8,+ "top_k": 100,+ "path_depth": 8,+ "path_fanout": 10,+ "filter_selectivity": 0.01,+ },+}++DIR_VECTOR_FILES = {+ "wiki": {+ "corpus": "dbpedia_dir_2m_corpus.jsonl",+ "vectors": "dbpedia_dir_2m_corpus_vectors.fvecs",+ "queries": "dbpedia_dir_2m_query.jsonl",+ "query_vectors": "dbpedia_dir_2m_query_vectors.fvecs",+ "ground_truth": "dbpedia_dir_2m_groundtruth.tsv",+ },+ "arxiv": {+ "corpus": "arxiv_corpus_metadata.json",+ "vectors": "arxiv_corpus_vectors.fvecs",+ "queries": "arxiv_query_constraint.json",+ "query_vectors": "arxiv_query_vectors.fvecs",+ "ground_truth": "arxiv_ground_truth.txt",+ },+ "arxiv_category": {+ "corpus": "arxiv_corpus_metadata.json",+ "vectors": "arxiv_corpus_vectors.fvecs",+ "queries": "arxiv_category_query_constraint.json",+ "query_vectors": "arxiv_category_query_vectors.fvecs",+ "ground_truth": "arxiv_category_ground_truth.txt",+ },+}+++@dataclass+class BenchOptions:+ config: Optional[str]+ output_dir: Path+ run_id: str+ profile: str+ mode: str+ workload: str+ dataset_root: Optional[Path]+ dataset: str+ full: bool+ rows: int+ queries: int+ dim: int+ batch_size: int+ concurrency: int+ top_k: int+ path_depth: int+ path_fanout: int+ filter_selectivity: float+ seed: int+ distance: Optional[str]+ drop_at_end: bool+++@dataclass+class QueryCase:+ query_id: str+ vector: list[float]+ filter_path: str+ ground_truth_ids: list[str] = field(default_factory=list)+++@dataclass+class Workload:+ name: str+ dim: int+ records: Callable[[], Iterator[dict[str, Any]]]+ queries: list[QueryCase]+ expected_rows: Optional[int]+++@dataclass+class Event:+ phase: str+ operation: str+ started_at: str+ ended_at: str+ latency_ms: float+ success: bool+ count: int = 0+ error: Optional[str] = None+ extra: dict[str, Any] = field(default_factory=dict)+++@dataclass+class RunResult:+ run_id: str+ collection_name: str+ output_dir: str+ events: list[Event]+ validation_errors: list[str]+ quality: dict[str, Any]+ workload: dict[str, Any]+ environment: dict[str, Any]+ kept_collection: bool+++def utc_now() -> str:+ return datetime.now(UTC).isoformat()+++def progress(message: str) -> None:+ print(f"[vectordb_perf] {message}", file=sys.stderr, flush=True)+++def progress_count(+ phase: str,+ done: int,+ total: Optional[int],+ started_at: float,+ last_reported_at: float,+ *,+ force: bool = False,+) -> float:+ now = time.perf_counter()+ if not force and last_reported_at and now - last_reported_at < PROGRESS_INTERVAL_SECONDS:+ return last_reported_at+ elapsed = max(now - started_at, 1e-9)+ rate = done / elapsed+ if total:+ progress(f"{phase}: {done}/{total} ({done / total * 100.0:.1f}%), {rate:.1f}/s")+ else:+ progress(f"{phase}: {done}, {rate:.1f}/s")+ return now+++def timed_event(+ phase: str,+ operation: str,+ fn: Callable[[], Any],+ *,+ count: int = 0,+ extra: Optional[dict[str, Any]] = None,+) -> tuple[Event, Any]:+ started = utc_now()+ start = time.perf_counter()+ try:+ result = fn()+ success = True+ error = None+ except Exception as exc: # benchmark should record the failing operation+ result = None+ success = False+ error = f"{type(exc).__name__}: {exc}"+ ended = utc_now()+ latency_ms = (time.perf_counter() - start) * 1000.0+ return (+ Event(+ phase=phase,+ operation=operation,+ started_at=started,+ ended_at=ended,+ latency_ms=latency_ms,+ success=success,+ count=count,+ error=error,+ extra=extra or {},+ ),+ result,+ )+++async def async_timed_event(+ phase: str,+ operation: str,+ fn: Callable[[], Any],+ *,+ count: int = 0,+ extra: Optional[dict[str, Any]] = None,+) -> tuple[Event, Any]:+ started = utc_now()+ start = time.perf_counter()+ try:+ result = await fn()+ success = True+ error = None+ except Exception as exc: # benchmark should record the failing operation+ result = None+ success = False+ error = f"{type(exc).__name__}: {exc}"+ ended = utc_now()+ latency_ms = (time.perf_counter() - start) * 1000.0+ return (+ Event(+ phase=phase,+ operation=operation,+ started_at=started,+ ended_at=ended,+ latency_ms=latency_ms,+ success=success,+ count=count,+ error=error,+ extra=extra or {},+ ),+ result,+ )+++def read_fvecs_dim(path: Path) -> int:+ with path.open("rb") as handle:+ raw = handle.read(4)+ if len(raw) != 4:+ raise ValueError(f"empty fvecs file: {path}")+ return struct.unpack("<i", raw)[0]+++def count_fvecs(path: Path) -> int:+ dim = read_fvecs_dim(path)+ if dim <= 0:+ raise ValueError(f"invalid fvecs dimension {dim} in {path}")+ stride = 4 + dim * 4+ size = path.stat().st_size+ if size % stride != 0:+ raise ValueError(f"fvecs file size is not aligned with dim={dim}: {path}")+ return size // stride+++def iter_fvecs(path: Path, limit: Optional[int] = None) -> Iterator[list[float]]:+ with path.open("rb") as handle:+ count = 0+ while limit is None or count < limit:+ raw_dim = handle.read(4)+ if not raw_dim:+ return+ if len(raw_dim) != 4:+ raise ValueError(f"truncated fvecs dimension in {path}")+ dim = struct.unpack("<i", raw_dim)[0]+ if dim <= 0:+ raise ValueError(f"invalid fvecs dimension {dim} in {path}")+ raw_vec = handle.read(dim * 4)+ if len(raw_vec) != dim * 4:+ raise ValueError(f"truncated fvecs vector in {path}")+ yield list(struct.unpack(f"<{dim}f", raw_vec))+ count += 1+++def iter_json_records(path: Path, limit: Optional[int] = None) -> Iterator[dict[str, Any]]:+ if path.suffix == ".jsonl":+ with path.open("r", encoding="utf-8") as handle:+ for index, line in enumerate(handle):+ if limit is not None and index >= limit:+ return+ stripped = line.strip()+ if stripped:+ item = json.loads(stripped)+ if isinstance(item, dict):+ yield item+ return++ data = json.loads(path.read_text(encoding="utf-8"))+ if isinstance(data, dict):+ iterable: Iterable[Any] = data.values()+ elif isinstance(data, list):+ iterable = data+ else:+ raise ValueError(f"unsupported JSON corpus shape in {path}")+ for index, item in enumerate(iterable):+ if limit is not None and index >= limit:+ return+ if isinstance(item, dict):+ yield item+++def pick_value(record: dict[str, Any], candidates: Sequence[str]) -> Any:+ for key in candidates:+ value = record.get(key)+ if value not in (None, ""):+ return value+ return None+++def normalize_path(path: Any) -> str:+ if isinstance(path, list):+ if not path:+ return "/"+ return ";".join(normalize_path(item) for item in path if item not in (None, ""))+ if not isinstance(path, str):+ return "/"+ stripped = path.strip()+ if not stripped:+ return "/"+ if stripped.startswith("viking://"):+ stripped = "/" + stripped[len("viking://") :].strip("/")+ elif not stripped.startswith("/"):+ stripped = "/" + stripped+ return stripped.rstrip("/") if len(stripped) > 1 else stripped+++def extract_path(record: dict[str, Any]) -> str:+ value = pick_value(+ record,+ (+ "dir_path",+ "dir_paths",+ "path",+ "paths",+ "directory",+ "directories",+ "category_path",+ "category_paths",+ "date_path",+ ),+ )+ if value is not None:+ return normalize_path(value)++ # Last resort for dataset variants with nested constraint metadata.+ for value in record.values():+ if isinstance(value, str) and "/" in value:+ return normalize_path(value)+ if isinstance(value, list):+ path_items = [item for item in value if isinstance(item, str) and "/" in item]+ if path_items:+ return normalize_path(path_items)+ return "/"+++def record_id(record: dict[str, Any], fallback: int) -> str:+ value = pick_value(+ record,+ ("id", "entity_id", "paper_id", "doc_id", "item_id", "source_id", "qid", "query_id"),+ )+ return str(value) if value is not None else str(fallback)+++def path_prefix(path: str, segments: int) -> str:+ parts = [part for part in path.strip("/").split("/") if part]+ if not parts:+ return "/"+ selected = parts[: max(1, min(segments, len(parts)))]+ return "/" + "/".join(selected)+++def path_matches(path_value: Any, prefix: str) -> bool:+ paths = str(path_value or "").split(";")+ normalized_prefix = normalize_path(prefix)+ for item in paths:+ path = normalize_path(item)+ if path == normalized_prefix or path.startswith(normalized_prefix.rstrip("/") + "/"):+ return True+ return False+++def synthetic_vector(rng: random.Random, dim: int) -> list[float]:+ return [rng.uniform(-1.0, 1.0) for _ in range(dim)]+++def synthetic_path(index: int, *, depth: int, fanout: int) -> str:+ fanout = max(1, fanout)+ parts = []+ value = index+ for level in range(max(1, depth)):+ parts.append(f"d{level}_{value % fanout}")+ value //= fanout+ return "/" + "/".join(parts)+++def safe_uri_segment(value: Any) -> str:+ text = str(value).strip().replace("/", "_")+ return "".join(ch if ch.isalnum() or ch in ("_", "-", ".") else "_" for ch in text) or "item"+++def resource_uri(dataset: str, path: str, item_id: Optional[Any] = None) -> str:+ normalized = normalize_path(str(path or "/").split(";")[0])+ suffix = normalized.strip("/")+ base = f"viking://resources/bench/{safe_uri_segment(dataset)}"+ if suffix:+ base = f"{base}/{suffix}"+ if item_id is not None:+ base = f"{base}/{safe_uri_segment(item_id)}"+ return base+++def deterministic_time(index: int) -> str:+ return f"2026-01-01T00:{(index // 60) % 60:02d}:{index % 60:02d}Z"+++def ov_context_record(+ *,+ record_id: str,+ vector: list[float],+ uri: str,+ index: int,+ category: str = "",+ content: str = "",+) -> dict[str, Any]:+ timestamp = deterministic_time(index)+ label = category or "resource"+ return {+ "id": record_id,+ "uri": uri,+ "type": "file",+ "context_type": "resource",+ "vector": vector,+ "created_at": timestamp,+ "updated_at": timestamp,+ "active_count": 0,+ "level": 2,+ "name": safe_uri_segment(record_id),+ "description": label,+ "tags": label,+ "search_tags": [label],+ "abstract": f"source_id:{record_id} category:{label}",+ "content": content or f"benchmark content for {record_id} in {label}",+ "account_id": BENCH_ACCOUNT_ID,+ "owner_user_id": BENCH_USER_ID,+ }+++def filter_segments(*, selectivity: float, fanout: int, depth: int) -> int:+ selectivity = min(1.0, max(1e-9, selectivity))+ fanout = max(2, fanout)+ segments = math.ceil(math.log(selectivity) / math.log(1.0 / fanout))+ return max(1, min(depth, segments))+++def build_synthetic_workload(options: BenchOptions) -> Workload:+ rows = max(1, options.rows)+ dim = max(4, options.dim)+ segment_count = filter_segments(+ selectivity=options.filter_selectivity,+ fanout=options.path_fanout,+ depth=options.path_depth,+ )++ def records() -> Iterator[dict[str, Any]]:+ rng = random.Random(options.seed)+ for index in range(rows):+ record_path = synthetic_path(+ index, depth=options.path_depth, fanout=options.path_fanout+ )+ rid = f"syn-{index}"+ yield ov_context_record(+ record_id=rid,+ vector=synthetic_vector(rng, dim),+ uri=resource_uri("synthetic", record_path, rid),+ index=index,+ category=f"cat_{index % 16}",+ )++ query_rng = random.Random(options.seed)+ vectors = [synthetic_vector(query_rng, dim) for _ in range(rows)]+ query_count = max(1, min(options.queries, rows))+ cases: list[QueryCase] = []+ for qindex in range(query_count):+ record_index = (qindex * max(1, rows // query_count)) % rows+ full_path = synthetic_path(record_index, depth=options.path_depth, fanout=options.path_fanout)+ cases.append(+ QueryCase(+ query_id=f"syn-q{qindex}",+ vector=vectors[record_index],+ filter_path=resource_uri("synthetic", path_prefix(full_path, segment_count)),+ ground_truth_ids=[f"syn-{record_index}"],+ )+ )+ return Workload(+ name="synthetic",+ dim=dim,+ records=records,+ queries=cases,+ expected_rows=rows,+ )+++def load_ground_truth(path: Path) -> dict[str, list[str]]:+ result: dict[str, list[str]] = {}+ with path.open("r", encoding="utf-8") as handle:+ for index, line in enumerate(handle):+ stripped = line.strip()+ if not stripped:+ continue+ if "\t" in stripped:+ parts = stripped.split("\t")+ if parts[0].lower() in {"query-id", "query_id"}:+ continue+ if len(parts) >= 3 and is_number(parts[2]):+ result.setdefault(str(parts[0]), []).append(str(parts[1]))+ continue+ if len(parts) >= 2:+ result[str(parts[0])] = [item for item in parts[1:] if item]+ continue+ else:+ parts = stripped.replace(",", " ").split()+ if len(parts) < 2:+ continue+ # Plain-text ground-truth files omit query ids; line number is the query id.+ result[str(index)] = [item for item in parts if item]+ return result+++def is_number(value: str) -> bool:+ try:+ float(value)+ return True+ except ValueError:+ return False+++def load_query_records(path: Path, limit: Optional[int]) -> list[dict[str, Any]]:+ if path.suffix == ".jsonl":+ return list(iter_json_records(path, limit))+ data = json.loads(path.read_text(encoding="utf-8"))+ if isinstance(data, list):+ return [item for item in data[:limit] if isinstance(item, dict)] if limit else data+ if isinstance(data, dict):+ items = []+ for key, value in data.items():+ if limit is not None and len(items) >= limit:+ break+ if isinstance(value, dict):+ row = dict(value)+ else:+ row = {"constraint": value}+ row.setdefault("query_id", key)+ items.append(row)+ return items+ raise ValueError(f"unsupported query JSON shape in {path}")+++def build_dir_vector_workload(options: BenchOptions) -> Workload:+ paths = dir_vector_paths(options.dataset_root, options.dataset)++ dim = read_fvecs_dim(paths["vectors"])+ total_rows = count_fvecs(paths["vectors"])+ total_queries = count_fvecs(paths["query_vectors"])+ row_limit = total_rows if options.full else min(options.rows, total_rows)+ query_limit = total_queries if options.full else min(options.queries, total_queries)+ sample_metadata = None if options.full else list(iter_json_records(paths["corpus"], row_limit))+ sampled_ids = (+ None+ if sample_metadata is None+ else {record_id(meta, index) for index, meta in enumerate(sample_metadata)}+ )++ def records() -> Iterator[dict[str, Any]]:+ vector_iter = iter_fvecs(paths["vectors"], row_limit)+ metadata_iter = (+ iter(sample_metadata)+ if sample_metadata is not None+ else iter_json_records(paths["corpus"], row_limit)+ )+ for index, (meta, vector) in enumerate(zip(metadata_iter, vector_iter)):+ rid = record_id(meta, index)+ path = extract_path(meta)+ category = str(pick_value(meta, ("category", "primary_category", "type")) or "")+ title = str(pick_value(meta, ("title", "name", "label")) or rid)+ yield ov_context_record(+ record_id=rid,+ vector=vector,+ uri=resource_uri(options.dataset, path, rid),+ index=index,+ category=category,+ content=title,+ )++ query_records = load_query_records(paths["queries"], query_limit if options.full else None)+ query_vectors = list(iter_fvecs(paths["query_vectors"], len(query_records)))+ ground_truth = load_ground_truth(paths["ground_truth"])+ covered_cases: list[QueryCase] = []+ fallback_cases: list[QueryCase] = []+ for index, (query_meta, vector) in enumerate(zip(query_records, query_vectors)):+ qid = record_id(query_meta, index)+ gt = [str(item) for item in (ground_truth.get(qid) or ground_truth.get(str(index)) or [])]+ if sampled_ids is not None:+ gt = [item for item in gt if item in sampled_ids]+ case = QueryCase(+ query_id=qid,+ vector=vector,+ filter_path=resource_uri(options.dataset, extract_path(query_meta)),+ ground_truth_ids=gt,+ )+ if gt or sampled_ids is None:+ covered_cases.append(case)+ else:+ fallback_cases.append(case)+ cases = covered_cases[:query_limit]+ if len(cases) < query_limit:+ cases.extend(fallback_cases[: query_limit - len(cases)])+ if not cases:+ raise ValueError("dir-vector query set is empty")+ return Workload(+ name=f"dir-vector:{options.dataset}",+ dim=dim,+ records=records,+ queries=cases,+ expected_rows=row_limit,+ )+++def dir_vector_paths(dataset_root: Optional[Path], dataset: str) -> dict[str, Path]:+ if dataset_root is None:+ raise ValueError("--dataset-root is required for dir-vector workload")+ dataset_files = DIR_VECTOR_FILES[dataset]+ expected = ", ".join(dataset_files.values())+ if not dataset_root.exists():+ raise FileNotFoundError(+ f"--dataset-root does not exist: {dataset_root}\n"+ f"Download dir-vector data first: {DIR_VECTOR_DATASET_URL}\n"+ f"Expected --dataset {dataset} files in that directory: {expected}"+ )+ paths = {name: dataset_root / filename for name, filename in dataset_files.items()}+ missing = [str(path) for path in paths.values() if not path.exists()]+ if missing:+ raise FileNotFoundError(+ "dir-vector dataset files missing:\n"+ + "\n".join(f"- {path}" for path in missing)+ + f"\nDownload dir-vector data first: {DIR_VECTOR_DATASET_URL}"+ + f"\nExpected --dataset {dataset} files: {expected}"+ )+ return paths+++def build_schema(collection_name: str, dim: int) -> dict[str, Any]:+ return CollectionSchemas.context_collection(collection_name, dim)+++def load_backend_config(options: BenchOptions, collection_name: Optional[str] = None):+ OpenVikingConfigSingleton.reset_instance()+ config = OpenVikingConfigSingleton.initialize(config_path=options.config)+ vectordb = config.storage.vectordb.model_copy(deep=True)+ if collection_name is not None:+ vectordb.name = collection_name+ return vectordb+++def make_collection_name(config_name: str, run_id: str) -> str:+ base = "".join(ch if ch.isalnum() or ch == "_" else "_" for ch in (config_name or "context"))+ if not base or not base[0].isalpha():+ base = f"bench_{base}"+ suffix = "".join(ch if ch.isalnum() else "_" for ch in run_id)+ return f"{base}_bench_{suffix}"[:120]+++def batched(iterable: Iterable[dict[str, Any]], size: int) -> Iterator[list[dict[str, Any]]]:+ batch: list[dict[str, Any]] = []+ for item in iterable:+ batch.append(item)+ if len(batch) >= size:+ yield batch+ batch = []+ if batch:+ yield batch+++def sample_record_ids(workload: Workload, limit: int) -> list[str]:+ ids: list[str] = []+ for record in workload.records():+ value = record.get("id")+ if value is not None:+ ids.append(str(value))+ if len(ids) >= limit:+ break+ return ids+++async def run_search_phase(+ *,+ backend: VikingVectorIndexBackend,+ ctx: RequestContext,+ phase: str,+ queries: list[QueryCase],+ top_k: int,+ concurrency: int,+ filtered: bool,+) -> tuple[list[Event], list[str], dict[str, Any]]:+ validation_errors: list[str] = []+ quality_hits = 0+ quality_total = 0++ async def one_query(query: QueryCase) -> tuple[Event, list[dict[str, Any]]]:+ target_directories = [query.filter_path] if filtered else None+ return await async_timed_event(+ phase,+ "targeted_search_in_tenant" if filtered else "search_in_tenant",+ lambda: backend.search_in_tenant(+ ctx=ctx,+ query_vector=query.vector,+ context_type="resource",+ target_directories=target_directories,+ level=[2],+ limit=top_k,+ ),+ extra={"query_id": query.query_id, "filter_path": query.filter_path if filtered else ""},+ )++ events: list[Event] = []+ results: list[tuple[QueryCase, list[dict[str, Any]]]] = []+ workers = max(1, concurrency)+ progress(f"{phase}: start queries={len(queries)} concurrency={workers} top_k={top_k}")+ started_at = time.perf_counter()+ last_reported_at = 0.0+ done = 0+ if workers == 1:+ for query in queries:+ event, rows = await one_query(query)+ events.append(event)+ results.append((query, rows or []))+ done += 1+ last_reported_at = progress_count(+ phase,+ done,+ len(queries),+ started_at,+ last_reported_at,+ force=done == len(queries),+ )+ else:+ async def tagged_query(query: QueryCase):+ event, rows = await one_query(query)+ return query, event, rows++ pending = [asyncio.create_task(tagged_query(query)) for query in queries]+ for future in asyncio.as_completed(pending):+ query, event, rows = await future+ events.append(event)+ results.append((query, rows or []))+ done += 1+ last_reported_at = progress_count(+ phase,+ done,+ len(queries),+ started_at,+ last_reported_at,+ force=done == len(queries),+ )++ for query, rows in results:+ if not rows:+ validation_errors.append(f"{phase}: query {query.query_id} returned no results")+ continue+ if filtered:+ leaked = [+ row.get("id")+ for row in rows+ if not path_matches(row.get("uri", ""), query.filter_path)+ ]+ if leaked:+ validation_errors.append(+ f"{phase}: query {query.query_id} returned rows outside {query.filter_path}: {leaked[:5]}"+ )+ if query.ground_truth_ids:+ quality_total += 1+ if result_hits_ground_truth(rows, query.ground_truth_ids):+ quality_hits += 1++ quality = {+ "queries_with_ground_truth": quality_total,+ "hit_rate_at_k": (quality_hits / quality_total) if quality_total else None,+ "hits": quality_hits,+ }+ return events, validation_errors, quality+++def result_hits_ground_truth(rows: list[dict[str, Any]], ground_truth_ids: list[str]) -> bool:+ returned_ids = {str(row.get("id")) for row in rows if row.get("id") is not None}+ return any(str(item) in returned_ids for item in ground_truth_ids)+++def benchmark_context() -> RequestContext:+ return RequestContext(user=UserIdentifier(BENCH_ACCOUNT_ID, BENCH_USER_ID), role=Role.USER)+++async def upsert_records(+ backend: VikingVectorIndexBackend, ctx: RequestContext, records: list[dict[str, Any]]+) -> list[str]:+ ids: list[str] = []+ for record in records:+ inserted = await backend.upsert(record, ctx=ctx)+ if inserted:+ ids.append(inserted)+ return ids+++def run_benchmark(options: BenchOptions) -> RunResult:+ return asyncio.run(run_benchmark_async(options))+++def selected_dir_vector_datasets(dataset: str) -> tuple[str, ...]:+ return DEFAULT_DIR_VECTOR_DATASETS if dataset == "all" else (dataset,)+++def run_benchmark_suite(options: BenchOptions) -> list[tuple[BenchOptions, RunResult]]:+ if options.workload != "dir-vector" or options.dataset != "all":+ return [(options, run_benchmark(options))]++ datasets = selected_dir_vector_datasets(options.dataset)+ for dataset in datasets:+ dir_vector_paths(options.dataset_root, dataset)++ runs = []+ for dataset in datasets:+ child = replace(+ options,+ dataset=dataset,+ run_id=f"{options.run_id}_{dataset}",+ output_dir=options.output_dir / dataset,+ )+ runs.append((child, run_benchmark(child)))+ return runs+++async def run_benchmark_async(options: BenchOptions) -> RunResult:+ workload = (+ build_synthetic_workload(options)+ if options.workload == "synthetic"+ else build_dir_vector_workload(options)+ )+ output_dir = options.output_dir+ output_dir.mkdir(parents=True, exist_ok=True)++ # Load once to discover the configured base name, then reload with the run-specific name.+ base_config = load_backend_config(options)+ collection_name = make_collection_name(base_config.name or "context", options.run_id)+ vectordb_config = load_backend_config(options, collection_name)+ if options.distance:+ vectordb_config.distance_metric = options.distance+ vectordb_config.dimension = workload.dim+ backend = VikingVectorIndexBackend(config=vectordb_config)+ ctx = benchmark_context()++ progress(+ "start "+ f"run_id={options.run_id} workload={workload.name} mode={options.mode} "+ f"records={workload.expected_rows} queries={len(workload.queries)} dim={workload.dim}"+ )+ progress(f"collection={collection_name}")++ events: list[Event] = []+ validation_errors: list[str] = []+ quality: dict[str, Any] = {}+ kept_collection = True+ inserted = 0+ fetch_ids: list[str] = []++ if options.mode != "read-only":+ progress("setup: create collection")+ event, created = await async_timed_event(+ "setup",+ "create_collection",+ lambda: backend.create_collection(+ collection_name,+ build_schema(collection_name, workload.dim),+ ),+ )+ events.append(event)+ if not event.success:+ validation_errors.append(event.error or "create_collection failed")+ await close_backend(backend)+ return finalize_result(+ options, workload, collection_name, events, validation_errors, quality, True+ )+ if created is False:+ validation_errors.append(f"collection already exists: {collection_name}")+ await close_backend(backend)+ return finalize_result(+ options, workload, collection_name, events, validation_errors, quality, True+ )+ progress("setup: collection ready")++ batch_size = max(1, options.batch_size)+ progress(f"ingest: start batch_size={batch_size}")+ ingest_started_at = time.perf_counter()+ last_ingest_reported_at = 0.0+ for batch in batched(workload.records(), batch_size):+ if len(fetch_ids) < 20:+ fetch_ids.extend(str(row["id"]) for row in batch[: 20 - len(fetch_ids)])+ event, ids = await async_timed_event(+ "ingest",+ "upsert_rows_serial",+ lambda batch=batch: upsert_records(backend, ctx, batch),+ count=len(batch),+ )+ events.append(event)+ if not event.success:+ validation_errors.append(event.error or "upsert_rows_serial failed")+ break+ inserted += len(ids or [])+ last_ingest_reported_at = progress_count(+ "ingest",+ inserted,+ workload.expected_rows,+ ingest_started_at,+ last_ingest_reported_at,+ force=inserted == workload.expected_rows,+ )++ if options.mode != "read-only" and inserted == 0:+ validation_errors.append("no records inserted")+ await close_backend(backend)+ return finalize_result(+ options,+ workload,+ collection_name,+ events,+ validation_errors,+ quality,+ kept_collection=True,+ )++ if options.mode == "write-only":+ if options.drop_at_end:+ progress("cleanup: drop collection")+ event, dropped = await async_timed_event(+ "cleanup", "drop_collection", lambda: backend.drop_collection()+ )+ events.append(event)+ kept_collection = not bool(dropped)+ if not event.success:+ validation_errors.append(event.error or "drop_collection failed")+ await close_backend(backend)+ return finalize_result(+ options,+ workload,+ collection_name,+ events,+ validation_errors,+ quality,+ kept_collection=kept_collection,+ )++ if options.mode == "read-only":+ fetch_ids = sample_record_ids(workload, 20)++ progress("validate: count/get")+ event, count_result = await async_timed_event(+ "validate", "count_all", lambda: backend.count(ctx=ctx)+ )+ events.append(event)+ if event.success and workload.expected_rows is not None and int(count_result or 0) < inserted:+ validation_errors.append(f"count_all={count_result} smaller than inserted={inserted}")+ elif not event.success:+ validation_errors.append(event.error or "count_all failed")++ event, fetched = await async_timed_event(+ "validate", "fetch_by_ids", lambda: backend.get(fetch_ids, ctx=ctx), count=len(fetch_ids)+ )+ events.append(event)+ if not event.success:+ validation_errors.append(event.error or "fetch_by_ids failed")+ elif len(fetched or []) != len(fetch_ids):+ validation_errors.append(f"fetch_by_ids returned {len(fetched or [])}, expected {len(fetch_ids)}")++ selected_queries = workload.queries+ search_events, errors, vector_quality = await run_search_phase(+ backend=backend,+ ctx=ctx,+ phase="vector_search",+ queries=selected_queries,+ top_k=options.top_k,+ concurrency=options.concurrency,+ filtered=False,+ )+ events.extend(search_events)+ validation_errors.extend(errors)+ quality["vector_search"] = vector_quality++ filtered_events, errors, filtered_quality = await run_search_phase(+ backend=backend,+ ctx=ctx,+ phase="filtered_vector_search",+ queries=selected_queries,+ top_k=options.top_k,+ concurrency=options.concurrency,+ filtered=True,+ )+ events.extend(filtered_events)+ validation_errors.extend(errors)+ quality["filtered_vector_search"] = filtered_quality++ first_filter = selected_queries[0].filter_path+ event, _ = await async_timed_event(+ "validate",+ "count_filtered",+ lambda: backend.count(PathScope("uri", first_filter, depth=-1), ctx=ctx),+ extra={"filter_path": first_filter},+ )+ events.append(event)+ if not event.success:+ validation_errors.append(event.error or "count_filtered failed")++ if options.drop_at_end:+ progress("cleanup: drop collection")+ event, dropped = await async_timed_event(+ "cleanup", "drop_collection", lambda: backend.drop_collection()+ )+ events.append(event)+ kept_collection = not bool(dropped)+ if not event.success:+ validation_errors.append(event.error or "drop_collection failed")++ await close_backend(backend)+ return finalize_result(+ options,+ workload,+ collection_name,+ events,+ validation_errors,+ quality,+ kept_collection=kept_collection,+ )+++async def close_backend(backend: VikingVectorIndexBackend) -> None:+ try:+ await backend.close()+ except Exception:+ pass+++def finalize_result(+ options: BenchOptions,+ workload: Workload,+ collection_name: str,+ events: list[Event],+ validation_errors: list[str],+ quality: dict[str, Any],+ kept_collection: bool,+) -> RunResult:+ return RunResult(+ run_id=options.run_id,+ collection_name=collection_name,+ output_dir=str(options.output_dir),+ events=events,+ validation_errors=validation_errors,+ quality=quality,+ workload={+ "name": workload.name,+ "mode": options.mode,+ "workload": options.workload,+ "dataset": options.dataset if options.workload == "dir-vector" else "synthetic",+ "record_count": workload.expected_rows,+ "query_count": len(workload.queries),+ "vector_dim": workload.dim,+ "top_k": options.top_k,+ "full": options.full,+ "profile": options.profile,+ },+ environment=collect_environment(),+ kept_collection=kept_collection,+ )+++def percentile(values: Sequence[float], pct: float) -> float:+ if not values:+ return 0.0+ sorted_values = sorted(values)+ index = min(len(sorted_values) - 1, max(0, math.ceil((pct / 100.0) * len(sorted_values)) - 1))+ return sorted_values[index]+++def phase_summary_rows(events: list[Event]) -> list[dict[str, Any]]:+ grouped: dict[tuple[str, str], list[Event]] = {}+ for event in events:+ grouped.setdefault((event.phase, event.operation), []).append(event)++ rows: list[dict[str, Any]] = []+ for (phase, operation), items in sorted(grouped.items()):+ latencies = [item.latency_ms for item in items]+ wall_ms = phase_wall_ms(items)+ count = sum(item.count or 1 for item in items)+ rows.append(+ {+ "phase": phase,+ "operation": operation,+ "events": len(items),+ "count": count,+ "wall_ms": round(wall_ms, 3),+ "success": sum(1 for item in items if item.success),+ "errors": sum(1 for item in items if not item.success),+ "avg_ms": round(sum(latencies) / len(latencies), 3),+ "p50_ms": round(percentile(latencies, 50), 3),+ "p95_ms": round(percentile(latencies, 95), 3),+ "p99_ms": round(percentile(latencies, 99), 3),+ "max_ms": round(max(latencies), 3),+ "throughput_per_sec": round((count / wall_ms) * 1000.0, 3)+ if wall_ms > 0+ else 0.0,+ }+ )+ return rows+++def phase_wall_ms(events: list[Event]) -> float:+ starts: list[datetime] = []+ ends: list[datetime] = []+ for event in events:+ try:+ starts.append(datetime.fromisoformat(event.started_at))+ ends.append(datetime.fromisoformat(event.ended_at))+ except ValueError:+ continue+ if starts and ends:+ return max(0.0, (max(ends) - min(starts)).total_seconds() * 1000.0)+ return sum(event.latency_ms for event in events)+++def collect_environment() -> dict[str, Any]:+ env: dict[str, Any] = {+ "observed_at": utc_now(),+ "platform": platform.platform(),+ "machine": platform.machine(),+ "python": sys.version,+ "pid": os.getpid(),+ "cpu_count": os.cpu_count(),+ "memory_bytes": total_memory_bytes(),+ "cgroup": read_cgroup_limits(),+ "gpu": nvidia_smi(),+ "remote_resources_verified": False,+ }+ return env+++def total_memory_bytes() -> Optional[int]:+ try:+ pages = os.sysconf("SC_PHYS_PAGES")+ page_size = os.sysconf("SC_PAGE_SIZE")+ return int(pages * page_size)+ except (AttributeError, OSError, ValueError):+ return None+++def read_int_file(path: Path) -> Optional[int | str]:+ try:+ raw = path.read_text(encoding="utf-8").strip()+ except OSError:+ return None+ if raw == "max":+ return "max"+ try:+ return int(raw.split()[0])+ except (ValueError, IndexError):+ return raw+++def read_cgroup_limits() -> dict[str, Any]:+ return {+ "cpu_max": read_int_file(Path("/sys/fs/cgroup/cpu.max")),+ "memory_max": read_int_file(Path("/sys/fs/cgroup/memory.max")),+ "memory_current": read_int_file(Path("/sys/fs/cgroup/memory.current")),+ }+++def nvidia_smi() -> list[dict[str, str]]:+ if shutil.which("nvidia-smi") is None:+ return []+ try:+ result = subprocess.run(+ [+ "nvidia-smi",+ "--query-gpu=name,memory.total,memory.used,utilization.gpu",+ "--format=csv,noheader,nounits",+ ],+ check=False,+ stdout=subprocess.PIPE,+ stderr=subprocess.DEVNULL,+ text=True,+ timeout=3,+ )+ except (OSError, subprocess.TimeoutExpired):+ return []+ if result.returncode != 0:+ return []+ rows = []+ for line in result.stdout.splitlines():+ parts = [part.strip() for part in line.split(",")]+ if len(parts) >= 4:+ rows.append(+ {+ "name": parts[0],+ "memory_total_mb": parts[1],+ "memory_used_mb": parts[2],+ "utilization_gpu_percent": parts[3],+ }+ )+ return rows+++def write_outputs(result: RunResult, options: BenchOptions) -> None:+ output_dir = Path(result.output_dir)+ output_dir.mkdir(parents=True, exist_ok=True)+ summary_rows = phase_summary_rows(result.events)+ write_json(output_dir / "environment.json", result.environment)+ config_json = {+ **asdict(options),+ "output_dir": str(options.output_dir),+ "dataset_root": str(options.dataset_root) if options.dataset_root else None,+ }+ write_json(output_dir / "run_config.json", config_json)+ write_json(+ output_dir / "run_summary.json",+ {+ "run_id": result.run_id,+ "collection_name": result.collection_name,+ "kept_collection": result.kept_collection,+ "validation_errors": result.validation_errors,+ "workload": result.workload,+ "quality": result.quality,+ "phase_summary": summary_rows,+ },+ )+ write_jsonl(output_dir / "events.jsonl", [asdict(event) for event in result.events])+ write_csv(output_dir / "phase_summary.csv", summary_rows)+ write_text(output_dir / "summary_zh.md", build_markdown_report(result, summary_rows, options))+++def write_suite_outputs(+ output_dir: Path, runs: list[tuple[BenchOptions, RunResult]]+) -> None:+ output_dir.mkdir(parents=True, exist_ok=True)+ run_summaries = {+ run_options.dataset: phase_summary_rows(result.events) for run_options, result in runs+ }+ rows = [+ {+ "dataset": run_options.dataset,+ "mode": result.workload.get("mode"),+ "records": result.workload.get("record_count"),+ "inserted": metric_value(run_summaries[run_options.dataset], "ingest", "count"),+ "queries": result.workload.get("query_count"),+ "dim": result.workload.get("vector_dim"),+ "top_k": result.workload.get("top_k"),+ "vector_qps": metric_value(run_summaries[run_options.dataset], "vector_search", "throughput_per_sec"),+ "vector_recall": quality_rate(result.quality, "vector_search"),+ "filtered_qps": metric_value(+ run_summaries[run_options.dataset], "filtered_vector_search", "throughput_per_sec"+ ),+ "filtered_recall": quality_rate(result.quality, "filtered_vector_search"),+ "recall_scope": recall_scope(result),+ "summary": str(Path(result.output_dir) / "summary_zh.md"),+ "collection": result.collection_name,+ "errors": len(result.validation_errors),+ "kept_collection": result.kept_collection,+ }+ for run_options, result in runs+ ]+ write_json(+ output_dir / "run_summary.json",+ {+ "datasets": rows,+ "validation_errors": {+ run_options.dataset: result.validation_errors for run_options, result in runs+ },+ },+ )+ lines = [+ "# OpenViking Vector Backend 真实数据汇总",+ "",+ "本次 `dir-vector` 真实数据 benchmark 覆盖 `wiki` 和 `arxiv` 两个数据集。",+ "",+ markdown_table(rows),+ "",+ ]+ for run_options, result in runs:+ lines.extend(["", f"## {run_options.dataset} 明细", ""])+ detail = build_markdown_report(+ result, run_summaries[run_options.dataset], run_options+ ).strip().splitlines()+ for line in detail[1:]:+ lines.append("#" + line if line.startswith("#") else line)+ write_text(output_dir / "summary_zh.md", "\n".join(lines))+++def metric_row(summary_rows: list[dict[str, Any]], phase: str) -> dict[str, Any]:+ for row in summary_rows:+ if row.get("phase") == phase:+ return row+ return {}+++def metric_value(summary_rows: list[dict[str, Any]], phase: str, key: str) -> Any:+ value = metric_row(summary_rows, phase).get(key)+ return "-" if value is None else value+++def quality_rate(quality: dict[str, Any], phase: str) -> str:+ value = quality.get(phase, {}).get("hit_rate_at_k")+ if value is None:+ return "-"+ return f"{float(value) * 100.0:.2f}%"+++def recall_scope(result: RunResult) -> str:+ if result.workload.get("workload") == "dir-vector" and not result.workload.get("full"):+ return "sampled_subset"+ return "official_full"+++def workload_rows(result: RunResult, summary_rows: list[dict[str, Any]]) -> list[dict[str, Any]]:+ workload = result.workload+ return [+ {+ "workload": workload.get("name"),+ "mode": workload.get("mode"),+ "dataset": workload.get("dataset"),+ "records": workload.get("record_count"),+ "inserted": metric_value(summary_rows, "ingest", "count"),+ "queries": workload.get("query_count"),+ "dim": workload.get("vector_dim"),+ "top_k": workload.get("top_k"),+ "full": workload.get("full"),+ }+ ]+++def recall_qps_rows(+ result: RunResult, summary_rows: list[dict[str, Any]], options: BenchOptions+) -> list[dict[str, Any]]:+ recall_header = f"gt_recall@{options.top_k}"+ return [+ {+ "phase": "vector_search",+ "qps": metric_value(summary_rows, "vector_search", "throughput_per_sec"),+ "avg_ms": metric_value(summary_rows, "vector_search", "avg_ms"),+ "p95_ms": metric_value(summary_rows, "vector_search", "p95_ms"),+ "scope": recall_scope(result),+ recall_header: quality_rate(result.quality, "vector_search"),+ },+ {+ "phase": "filtered_vector_search",+ "qps": metric_value(summary_rows, "filtered_vector_search", "throughput_per_sec"),+ "avg_ms": metric_value(summary_rows, "filtered_vector_search", "avg_ms"),+ "p95_ms": metric_value(summary_rows, "filtered_vector_search", "p95_ms"),+ "scope": recall_scope(result),+ recall_header: quality_rate(result.quality, "filtered_vector_search"),+ },+ ]+++def build_markdown_report(+ result: RunResult, summary_rows: list[dict[str, Any]], options: BenchOptions+) -> str:+ lines = [+ "# OpenViking Vector Backend 性能验收报告",+ "",+ f"- Run ID: `{result.run_id}`",+ f"- Workload: `{options.workload}`",+ f"- Mode: `{options.mode}`",+ f"- Collection: `{result.collection_name}`",+ f"- 功能错误: `{len(result.validation_errors)}`",+ f"- Collection 保留: `{result.kept_collection}`",+ "",+ "## 结论",+ "",+ ]+ if result.validation_errors:+ lines.append("功能验收未通过。性能慢只记录,以下功能错误会导致非零退出码:")+ for error in result.validation_errors[:20]:+ lines.append(f"- {error}")+ else:+ lines.append("功能验收通过。")++ lines.extend(+ [+ "",+ "## 数据规模",+ "",+ markdown_table(workload_rows(result, summary_rows)),+ "",+ "## 召回与 QPS",+ "",+ markdown_table(recall_qps_rows(result, summary_rows, options)),+ "",+ "gt_recall@K 按 query 的 ground truth 命中率统计;`sampled_subset` 只用于 smoke sanity,不代表官方全量 recall。官方 recall 请用 `--full`。",+ "",+ "## 性能汇总",+ "",+ markdown_table(summary_rows),+ "",+ "## 环境说明",+ "",+ f"- Platform: `{result.environment.get('platform')}`",+ f"- CPU count: `{result.environment.get('cpu_count')}`",+ f"- Memory bytes: `{result.environment.get('memory_bytes')}`",+ f"- GPU: `{result.environment.get('gpu') or 'not detected'}`",+ "- 远端/GPU 服务资源未由本 runner 验证;报告只记录 runner 本机观测信息。",+ "",+ "## 清理",+ "",+ ]+ )+ if result.kept_collection:+ lines.append(+ "默认保留测试 collection。需要清理时,用相同配置连接后删除 "+ f"`{result.collection_name}`。"+ )+ else:+ lines.append("本次运行已尝试删除测试 collection。")++ lines.extend(+ [+ "",+ "## 输出文件",+ "",+ "- `run_summary.json`: 汇总数据",+ "- `events.jsonl`: 每次操作明细",+ "- `phase_summary.csv`: 阶段聚合",+ "- `environment.json`: 本机环境观测",+ ]+ )+ return "\n".join(lines) + "\n"+++def markdown_table(rows: list[dict[str, Any]]) -> str:+ if not rows:+ return "无数据。"+ headers = list(rows[0].keys())+ lines = [+ "| " + " | ".join(headers) + " |",+ "| " + " | ".join("---" for _ in headers) + " |",+ ]+ for row in rows:+ lines.append("| " + " | ".join(str(row.get(header, "")) for header in headers) + " |")+ return "\n".join(lines)+++def write_json(path: Path, value: Any) -> None:+ path.write_text(json.dumps(value, ensure_ascii=False, indent=2, default=str), encoding="utf-8")+++def write_text(path: Path, value: str) -> None:+ path.write_text(value, encoding="utf-8")+++def write_jsonl(path: Path, values: Iterable[Any]) -> None:+ with path.open("w", encoding="utf-8") as handle:+ for value in values:+ handle.write(json.dumps(value, ensure_ascii=False, default=str) + "\n")+++def write_csv(path: Path, rows: list[dict[str, Any]]) -> None:+ if not rows:+ path.write_text("", encoding="utf-8")+ return+ with path.open("w", newline="", encoding="utf-8") as handle:+ writer = csv.DictWriter(handle, fieldnames=list(rows[0].keys()))+ writer.writeheader()+ writer.writerows(rows)+++def parse_args(argv: Optional[Sequence[str]] = None) -> BenchOptions:+ parser = argparse.ArgumentParser(description="Run OpenViking vector backend performance benchmark")+ parser.add_argument("--config", help="Path to ov.conf. Defaults to OpenViking config lookup.")+ parser.add_argument("--output-dir", type=Path)+ parser.add_argument("--run-id")+ parser.add_argument("--profile", choices=sorted(PROFILE_DEFAULTS), default="smoke")+ parser.add_argument(+ "--mode",+ choices=["read-write", "write-only", "read-only"],+ default="read-write",+ help="read-write creates, writes, and reads; write-only skips reads; read-only skips create/upsert",+ )+ parser.add_argument("--workload", choices=["synthetic", "dir-vector"], default="synthetic")+ parser.add_argument("--dataset-root", type=Path)+ parser.add_argument(+ "--dataset",+ choices=["all", *sorted(DIR_VECTOR_FILES)],+ default="all",+ help="dir-vector dataset; default runs wiki and arxiv",+ )+ parser.add_argument("--full", action="store_true", help="Run full dir-vector corpus/query set")+ parser.add_argument("--rows", type=int)+ parser.add_argument("--queries", type=int)+ parser.add_argument("--dim", type=int)+ parser.add_argument("--batch-size", type=int)+ parser.add_argument("--concurrency", type=int)+ parser.add_argument("--top-k", type=int)+ parser.add_argument("--path-depth", type=int)+ parser.add_argument("--path-fanout", type=int)+ parser.add_argument("--filter-selectivity", type=float)+ parser.add_argument("--seed", type=int, default=42)+ parser.add_argument("--distance", choices=["ip", "l2", "cosine"])+ parser.add_argument("--drop-at-end", action="store_true", help="Drop benchmark collection")+ args = parser.parse_args(argv)++ if args.mode == "read-only" and args.drop_at_end:+ raise ValueError("--drop-at-end is not allowed with --mode read-only")++ defaults = PROFILE_DEFAULTS[args.profile]+ run_id = args.run_id or datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ") + "_" + uuid.uuid4().hex[:8]+ output_dir = args.output_dir or Path("benchmark/results/vectordb_perf") / run_id+ return BenchOptions(+ config=args.config,+ output_dir=output_dir,+ run_id=run_id,+ profile=args.profile,+ mode=args.mode,+ workload=args.workload,+ dataset_root=args.dataset_root,+ dataset=args.dataset,+ full=args.full,+ rows=args.rows if args.rows is not None else defaults["rows"],+ queries=args.queries if args.queries is not None else defaults["queries"],+ dim=args.dim if args.dim is not None else defaults["dim"],+ batch_size=args.batch_size if args.batch_size is not None else defaults["batch_size"],+ concurrency=args.concurrency if args.concurrency is not None else defaults["concurrency"],+ top_k=args.top_k if args.top_k is not None else defaults["top_k"],+ path_depth=args.path_depth if args.path_depth is not None else defaults["path_depth"],+ path_fanout=args.path_fanout if args.path_fanout is not None else defaults["path_fanout"],+ filter_selectivity=args.filter_selectivity+ if args.filter_selectivity is not None+ else defaults["filter_selectivity"],+ seed=args.seed,+ distance=args.distance,+ drop_at_end=args.drop_at_end,+ )+++def main(argv: Optional[Sequence[str]] = None) -> int:+ try:+ options = parse_args(argv)+ runs = run_benchmark_suite(options)+ except (FileNotFoundError, ValueError) as exc:+ print(f"error: {exc}", file=sys.stderr)+ return 2+ for run_options, result in runs:+ write_outputs(result, run_options)+ if len(runs) > 1:+ write_suite_outputs(options.output_dir, runs)+ print(f"summary: {options.output_dir / 'summary_zh.md'}")+ else:+ print(f"summary: {Path(runs[0][1].output_dir) / 'summary_zh.md'}")+ return 1 if any(result.validation_errors for _, result in runs) else 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/src/index/detail/scalar/bitmap_holder/bitmap_field_group.cpp b/src/index/detail/scalar/bitmap_holder/bitmap_field_group.cppindex 648ed55e3..1ec660a7e 100644--- a/src/index/detail/scalar/bitmap_holder/bitmap_field_group.cpp+++ b/src/index/detail/scalar/bitmap_holder/bitmap_field_group.cpp@@ -49,7 +49,11 @@ bool BitmapGroupBase::exist_bitmap(const std::string& key) {}const Bitmap* BitmapGroupBase::get_bitmap(const std::string& key) {- return get_editable_bitmap(key);+ auto it = bitmap_group_.find(key);+ if (it == bitmap_group_.end()) {+ return nullptr;+ }+ return it->second.get();}Bitmap* BitmapGroupBase::get_editable_bitmap(const std::string& key) {diff --git a/src/index/detail/scalar/bitmap_holder/bitmap_field_group.h b/src/index/detail/scalar/bitmap_holder/bitmap_field_group.hindex eef770250..d4804ff7c 100644--- a/src/index/detail/scalar/bitmap_holder/bitmap_field_group.h+++ b/src/index/detail/scalar/bitmap_holder/bitmap_field_group.h@@ -468,7 +468,11 @@ class FieldBitmapGroupSet {}const Bitmap* get_bitmap(const std::string& field, const std::string key) {- return get_editable_bitmap(field, key);+ auto itr = field_bitmap_groups_map_.find(field);+ if (itr == field_bitmap_groups_map_.end()) {+ return nullptr;+ }+ return itr->second->get_bitmap(key);}Bitmap* get_editable_bitmap(const std::string& field, const std::string key) {