Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(benchmark): update session load benchmark (#2031)
Qin Haojiee52d3446a0e9
diff --git a/benchmark/custom/README.md b/benchmark/custom/README.mdnew file mode 100644index 000000000..a1da5e826--- /dev/null+++ b/benchmark/custom/README.md@@ -0,0 +1,264 @@+# OpenViking Server 压测脚本使用指南++本目录包含面向 OpenViking 本地 Server 的自定义压测脚本。当前主要脚本是:++- `session_contention_benchmark.py`:通用 Server 压测框架,覆盖 SDK、CLI HTTP 封装和真实 `ov` 子进程三种调用路径。++## 压测目标++`session_contention_benchmark.py` 用来请求已经启动的 OpenViking Server,验证多类接口在并发和混合负载下的吞吐、延迟、失败率和后台任务积压情况。++脚本不会启动或停止 Server,只负责:++1. 生成本地 Markdown 测试文档。+2. 每次运行前默认清空上一次压测写入的数据目录。+3. 并发请求资源添加、检索、session 写入、session commit、任务轮询和观测接口。+4. 输出中文压测报告和机器可读明细文件。++## 前置条件++先在另一个终端启动 OpenViking Server:++```bash+openviking-server+```++如果 Server 启用了 API Key 或多租户,请准备好以下信息:++- Server 地址,例如 `http://127.0.0.1:1935`+- API Key,默认参数会使用 `test-root-api-key`+- Account,默认 `default`+- User,默认 `default`++真实 CLI 子进程模式还要求当前环境能执行 `ov`:++```bash+ov health+```++## 快速开始++在仓库根目录运行 smoke 压测:++```bash+.venv/bin/python benchmark/custom/session_contention_benchmark.py \+ --server-url http://127.0.0.1:1935 \+ --profile smoke+```++只测试 Python SDK 路径:++```bash+.venv/bin/python benchmark/custom/session_contention_benchmark.py \+ --server-url http://127.0.0.1:1935 \+ --profile smoke \+ --adapters sdk+```++同时测试 SDK、CLI HTTP 封装和真实 `ov` 子进程:++```bash+.venv/bin/python benchmark/custom/session_contention_benchmark.py \+ --server-url http://127.0.0.1:1935 \+ --profile standard \+ --adapters sdk,cli-http,cli-subprocess+```++## 调用路径说明++脚本支持三种 adapter:++| Adapter | 含义 | 适用场景 |+| --- | --- | --- |+| `sdk` | 通过 `openviking.AsyncHTTPClient` 请求 Server | 评估 Python SDK 的 HTTP 调用表现 |+| `cli-http` | 直接使用 `openviking_cli.client.http.AsyncHTTPClient` | 评估 CLI 共用 HTTP client 层表现 |+| `cli-subprocess` | 每次请求真实执行一次 `ov ... --output json` | 评估真实 CLI 进程启动、配置读取、上传和输出解析成本 |++默认会同时运行三种路径:++```bash+--adapters sdk,cli-http,cli-subprocess+```++如果只关心 Server 吞吐,建议先跑 `sdk` 或 `cli-http`。如果关心用户实际执行 CLI 命令的端到端成本,再加入 `cli-subprocess`。++## 压测场景++每个 adapter 会依次执行以下阶段:++| 阶段 | 内容 |+| --- | --- |+| `warmup` | 健康检查预热 |+| `add_resources` | 并发添加多个生成的 Markdown 文档 |+| `session_messages` | 并发向不同 session 写入多轮 user / assistant 消息 |+| `retrieval` | 并发执行 `find`、`search`、`grep`、`glob` |+| `session_commit` | 并发 commit 不同 session,并轮询后台任务 |+| `mixed` | 混合资源添加、检索、session 写入、commit、观测接口和任务轮询 |++这些阶段不是只压单个接口,目的是观察 OpenViking 在真实组合负载下的退化情况。++## 数据清理策略++默认每次运行前会清理:++- Server 侧资源目录:`viking://resources/bench/load_test`+- 旧 session:所有 `bench-load-` 前缀的 session+- 本地生成数据目录:`benchmark/results/openviking_server_load/data`++默认运行结束后保留本次写入的数据,方便人工复查。下一次运行前会再次清空。++如果不想在运行前清理:++```bash+--no-clear-before-run+```++如果希望运行结束后也清理:++```bash+--cleanup-at-end+```++## 常用参数++| 参数 | 默认值 | 说明 |+| --- | --- | --- |+| `--server-url` | `http://127.0.0.1:1935` | OpenViking Server 地址 |+| `--api-key` | `test-root-api-key` | 请求使用的 API Key |+| `--account` | `default` | 请求使用的 account |+| `--user` | `default` | 请求使用的 user |+| `--adapters` | `sdk,cli-http,cli-subprocess` | 要测试的调用路径 |+| `--profile` | `standard` | 压测规模:`smoke`、`standard`、`stress` |+| `--resource-count` | 跟随 profile | 每个 adapter 添加的初始文档数 |+| `--session-count` | 跟随 profile | 每个 adapter 使用的 session 数 |+| `--phase-seconds` | 跟随 profile | 单类持续压测阶段时长 |+| `--mixed-seconds` | 跟随 profile | 混合压测阶段时长 |+| `--drain-timeout` | `60` | 等待后台任务完成的最大秒数 |+| `--data-root-uri` | `viking://resources/bench/load_test` | Server 侧压测资源根目录 |+| `--output-dir` | 自动生成 | 报告输出目录 |+| `--ov-bin` | `ov` | 真实 CLI 子进程使用的可执行文件 |++## Profile 说明++| Profile | 用途 | 特点 |+| --- | --- | --- |+| `smoke` | 快速验证脚本、配置和 Server 可用性 | 时间短、并发低、数据少 |+| `standard` | 常规压测 | 默认推荐配置 |+| `stress` | 高压力压测 | 并发和数据量更高,耗时更长 |++建议先跑 `smoke`,确认报告正常生成后再跑 `standard` 或 `stress`。++## 报告输出++默认输出目录类似:++```text+benchmark/results/openviking_server_load/20260511T120000Z/+```++主要文件:++| 文件 | 内容 |+| --- | --- |+| `summary_zh.md` | 中文压测报告,优先阅读 |+| `run_summary.json` | 汇总结果,便于自动分析 |+| `request_events.jsonl` | 每次请求的明细事件 |+| `task_events.jsonl` | 后台任务完成和积压明细 |+| `request_summary.csv` | 按 adapter / 阶段 / 接口聚合的 QPS、成功率、延迟 |+| `request_windows.csv` | 按时间窗口聚合的请求表现 |+| `adapter_comparison.csv` | SDK / CLI 路径对比 |+| `errors.csv` | 错误 Top 明细 |++报告会重点展示:++- 总请求量、失败数和成功率。+- 各接口 p50 / p95 / p99 / max 延迟。+- `retrieval` 阶段到 `mixed` 阶段的检索延迟变化。+- SDK、CLI HTTP 封装、真实 CLI 子进程之间的差异。+- commit / add_resource 后台任务是否积压。+- Top 错误类型和发生位置。++## 示例:指定认证信息++```bash+.venv/bin/python benchmark/custom/session_contention_benchmark.py \+ --server-url http://127.0.0.1:1935 \+ --api-key your-root-api-key \+ --account default \+ --user default \+ --profile standard+```++也可以通过环境变量传入:++```bash+export OPENVIKING_SERVER_URL=http://127.0.0.1:1935+export OPENVIKING_API_KEY=your-root-api-key+export OPENVIKING_ACCOUNT=default+export OPENVIKING_USER=default++.venv/bin/python benchmark/custom/session_contention_benchmark.py --profile smoke+```++## 示例:降低真实 CLI 子进程开销++真实 `cli-subprocess` 会为每个请求启动一次 `ov` 进程,开销明显高于 SDK 路径。若只想压 Server 本身,可以先排除它:++```bash+.venv/bin/python benchmark/custom/session_contention_benchmark.py \+ --profile standard \+ --adapters sdk,cli-http+```++若必须测试真实 CLI,但本地 `ov` 不在 PATH 中,可以指定可执行文件:++```bash+.venv/bin/python benchmark/custom/session_contention_benchmark.py \+ --profile smoke \+ --adapters cli-subprocess \+ --ov-bin .venv/bin/ov+```++## 常见问题++### 1. Server 连接失败++先确认 Server 已启动:++```bash+curl http://127.0.0.1:1935/health+```++如果端口不同,请设置 `--server-url`。++### 2. 认证失败++确认 `--api-key`、`--account`、`--user` 与当前 Server 配置一致。多租户模式下,root key 请求通常还需要 account 和 user。++### 3. `cli-subprocess` 失败++确认 `ov` 可执行:++```bash+ov health+```++如果不可用,使用 `--ov-bin .venv/bin/ov` 或只运行 `sdk,cli-http`。++### 4. 后台任务没有在 drain 内完成++这通常说明资源处理或 session commit 的后台任务积压。可以:++- 增大 `--drain-timeout`+- 降低并发参数+- 查看 `task_events.jsonl`+- 查看 Server 日志和 observer queue 状态++### 5. 多次运行结果差异较大++压测会受本机 CPU、磁盘、模型配置、队列积压和真实 CLI 进程启动开销影响。建议:++1. 先跑 `smoke` 验证环境。+2. 连续跑多次 `standard`。+3. 对比 `adapter_comparison.csv` 和 `request_windows.csv`。+4. 避免在已有大量后台任务未完成时启动下一轮。diff --git a/benchmark/custom/session_contention_benchmark.py b/benchmark/custom/session_contention_benchmark.pyindex c351952ae..fa48f88e5 100644--- a/benchmark/custom/session_contention_benchmark.py+++ b/benchmark/custom/session_contention_benchmark.py@@ -1,5 +1,5 @@#!/usr/bin/env python3-"""Daily session mixed-load contention benchmark for OpenViking."""+"""OpenViking server mixed-load benchmark."""from __future__ import annotations@@ -10,103 +10,157 @@ import jsonimport mathimport osimport random+import shutilimport sysimport timefrom dataclasses import asdict, dataclass, fieldfrom datetime import UTC, datetime+from itertools import countfrom pathlib import Path-from typing import Any, Dict, Iterable, List, Optional+from typing import Any, Awaitable, Callable, Dict, Iterable, List, Optional, Sequence-import httpx--DEFAULT_FIND_QUERIES = [- "how to authenticate users",- "what is OpenViking",+DEFAULT_DATA_ROOT_URI = "viking://resources/bench/load_test"+DEFAULT_SESSION_PREFIX = "bench-load-"+DEFAULT_QUERIES = [+ "OpenViking server load test","session commit memory extraction",+ "resource ingestion benchmark document",+ "concurrent retrieval latency",]DEFAULT_SLOW_THRESHOLDS_MS = (1000, 3000, 5000)-MAX_ERROR_MESSAGE_LEN = 500+MAX_ERROR_MESSAGE_LEN = 800++PROFILE_DEFAULTS: Dict[str, Dict[str, Any]] = {+ "smoke": {+ "resource_count": 6,+ "session_count": 6,+ "messages_per_session": 3,+ "resource_concurrency": 2,+ "search_concurrency": 2,+ "session_concurrency": 2,+ "commit_concurrency": 2,+ "mixed_concurrency": 4,+ "phase_seconds": 8.0,+ "mixed_seconds": 12.0,+ "warmup_seconds": 1.0,+ },+ "standard": {+ "resource_count": 24,+ "session_count": 24,+ "messages_per_session": 6,+ "resource_concurrency": 6,+ "search_concurrency": 8,+ "session_concurrency": 8,+ "commit_concurrency": 6,+ "mixed_concurrency": 16,+ "phase_seconds": 30.0,+ "mixed_seconds": 60.0,+ "warmup_seconds": 3.0,+ },+ "stress": {+ "resource_count": 80,+ "session_count": 80,+ "messages_per_session": 8,+ "resource_concurrency": 16,+ "search_concurrency": 24,+ "session_concurrency": 24,+ "commit_concurrency": 16,+ "mixed_concurrency": 48,+ "phase_seconds": 60.0,+ "mixed_seconds": 180.0,+ "warmup_seconds": 5.0,+ },+}@dataclassclass BenchmarkConfig:server_url: str- api_key: str- account: str- user: str- request_timeout: float+ api_key: Optional[str]+ account: Optional[str]+ user: Optional[str]+ agent_id: Optional[str]+ timeout: float+ adapters: List[str]+ profile: str+ resource_count: intsession_count: int- writer_concurrency: int- reader_concurrency: int- extract_concurrency: int- messages_per_commit: int- extract_ratio: float- message_size: int- baseline_seconds: float+ messages_per_session: int+ resource_concurrency: int+ search_concurrency: int+ session_concurrency: int+ commit_concurrency: int+ mixed_concurrency: int+ phase_seconds: floatmixed_seconds: float- recovery_seconds: float- window_seconds: float- observer_interval: float+ warmup_seconds: float+ drain_timeout: floattask_poll_interval: float- task_drain_timeout: float+ request_window_seconds: float+ local_data_dir: stroutput_dir: str- cleanup: bool- require_extract_load: bool- find_queries: List[str]- find_limit: int- find_target_uri: str- find_score_threshold: Optional[float]+ data_root_uri: str+ session_prefix: str+ clear_before_run: bool+ cleanup_at_end: bool+ ov_bin: strseed: int+ find_limit: int+++def direct_http_agent_id(config: BenchmarkConfig) -> str:+ """Avoid loading the user's global ovcli.conf when agent_id is unset."""+ return config.agent_id or ""@dataclass-class PhaseMetadata:- phase: str- started_at: str- ended_at: str- duration_seconds: float+class AdapterResult:+ success: bool+ result: Any = None+ status_code: Optional[int] = None+ exception_type: Optional[str] = None+ error_message: Optional[str] = None+ raw_stdout: Optional[str] = None+ raw_stderr: Optional[str] = None+ command: Optional[List[str]] = None@dataclassclass RequestEvent:- api: str- method: str- path: str- phase: str+ adapter: str+ scenario: str+ operation: strstarted_at: strended_at: strelapsed_ms_since_run_start: floatlatency_ms: floatsuccess: boolstatus_code: Optional[int]- timeout: boolexception_type: Optional[str]- error_code: Optional[str]error_message: Optional[str]session_id: Optional[str] = None- cycle_index: Optional[int] = None- worker_id: Optional[int] = None+ resource_uri: Optional[str] = Nonetask_id: Optional[str] = None+ worker_id: Optional[int] = None+ command: Optional[str] = Nonedef to_dict(self) -> Dict[str, Any]:return asdict(self)@dataclass-class CommitTaskEvent:+class TaskEvent:+ adapter: strtask_id: str- session_id: str- origin_phase: str- completion_phase: str+ task_type: str+ origin_scenario: str+ completion_scenario: strstatus: str- created_at: Optional[float]- updated_at: Optional[float]- server_duration_ms: Optional[float]+ resource_id: Optional[str]local_duration_ms: float- active_count_updated: Optional[int]- memories_extracted: Optional[Dict[str, int]]- error: Optional[str]- cycle_index: Optional[int]+ server_duration_ms: Optional[float]+ error_message: Optional[str]+ result: Anypolled_at: strdef to_dict(self) -> Dict[str, Any]:@@ -114,980 +168,1241 @@ class CommitTaskEvent:@dataclass-class ObserverSample:- api: str- phase: str- sampled_at: str- elapsed_ms_since_run_start: float- latency_ms: float- success: bool- is_healthy: Optional[bool]- has_errors: Optional[bool]- payload: Optional[Dict[str, Any]]- error_message: Optional[str] = None-- def to_dict(self) -> Dict[str, Any]:- return asdict(self)+class PendingTask:+ adapter_name: str+ task_id: str+ task_type: str+ origin_scenario: str+ resource_id: Optional[str]+ local_started_monotonic: float@dataclass-class PendingCommitTask:- task_id: str- session_id: str- origin_phase: str- cycle_index: int- local_started_monotonic: float+class PhaseMetadata:+ adapter: str+ scenario: str+ started_at: str+ ended_at: str+ duration_seconds: float++ def to_dict(self) -> Dict[str, Any]:+ return asdict(self)@dataclassclass Recorder:request_events: List[RequestEvent] = field(default_factory=list)- task_events: List[CommitTaskEvent] = field(default_factory=list)- observer_samples: List[ObserverSample] = field(default_factory=list)+ task_events: List[TaskEvent] = field(default_factory=list)+ phases: List[PhaseMetadata] = field(default_factory=list)notes: List[str] = field(default_factory=list)def add_request(self, event: RequestEvent) -> None:self.request_events.append(event)- def add_task(self, event: CommitTaskEvent) -> None:+ def add_task(self, event: TaskEvent) -> None:self.task_events.append(event)- def add_sample(self, sample: ObserverSample) -> None:- self.observer_samples.append(sample)+ def add_phase(self, phase: PhaseMetadata) -> None:+ self.phases.append(phase)def add_note(self, note: str) -> None:self.notes.append(note)-class PhaseState:- def __init__(self, initial: str = "setup") -> None:- self.current = initial+class LoadAdapter:+ name: str+ async def initialize(self) -> None: ...-class BenchmarkHTTPClient:- def __init__(self, config: BenchmarkConfig, recorder: Recorder) -> None:- self._config = config- self._recorder = recorder- self._run_start_monotonic = time.perf_counter()- self._client = httpx.AsyncClient(- base_url=config.server_url.rstrip("/"),- headers=self._default_headers(),- timeout=httpx.Timeout(config.request_timeout),- follow_redirects=True,- limits=httpx.Limits(- max_connections=max(- 32,- config.writer_concurrency- + config.reader_concurrency- + config.extract_concurrency- + 8,- ),- max_keepalive_connections=max(- 16,- config.writer_concurrency + config.reader_concurrency + 4,- ),- ),- )+ async def close(self) -> None: ...- @property- def run_start_monotonic(self) -> float:- return self._run_start_monotonic+ async def health(self) -> Any: ...- async def aclose(self) -> None:- await self._client.aclose()+ async def system_status(self) -> Any: ...- def _default_headers(self) -> Dict[str, str]:- headers = {- "Accept": "*/*",- "Content-Type": "application/json",- "User-Agent": "OpenViking-Session-Contention-Benchmark/1.0",- "X-OpenViking-Account": self._config.account,- "X-OpenViking-User": self._config.user,- }- if self._config.api_key:- headers["Authorization"] = f"Bearer {self._config.api_key}"- return headers+ async def observer_queue(self) -> Any: ...- async def request_json(- self,- *,- api: str,- method: str,- path: str,- phase: str,- session_id: Optional[str] = None,- cycle_index: Optional[int] = None,- worker_id: Optional[int] = None,- task_id: Optional[str] = None,- json_payload: Optional[Dict[str, Any]] = None,- params: Optional[Dict[str, Any]] = None,- ) -> tuple[Optional[httpx.Response], Optional[Dict[str, Any]]]:- started_monotonic = time.perf_counter()- started_wall = utc_now()- response: Optional[httpx.Response] = None- body: Optional[Dict[str, Any]] = None- status_code: Optional[int] = None- success = False- timeout = False- exception_type: Optional[str] = None- error_code: Optional[str] = None- error_message: Optional[str] = None+ async def add_resource(+ self, *, path: str, to: str, reason: str, wait: bool, timeout: Optional[float]+ ) -> Any: ...- try:- response = await self._client.request(- method=method,- url=path,- json=json_payload,- params=params,- )- status_code = response.status_code- body = maybe_json(response)- success = self._is_success(status_code, body)- if not success:- error_code, error_message = extract_error(body, status_code)- except httpx.TimeoutException as exc:- timeout = True- exception_type = type(exc).__name__- error_message = truncate_error_message(str(exc))- except Exception as exc: # pragma: no cover - exercised in real runs- exception_type = type(exc).__name__- error_message = truncate_error_message(str(exc))+ async def wait_processed(self, timeout: Optional[float]) -> Any: ...- ended_wall = utc_now()- ended_monotonic = time.perf_counter()- latency_ms = (ended_monotonic - started_monotonic) * 1000.0- elapsed_ms = (started_monotonic - self._run_start_monotonic) * 1000.0- self._recorder.add_request(- RequestEvent(- api=api,- method=method.upper(),- path=path,- phase=phase,- started_at=started_wall,- ended_at=ended_wall,- elapsed_ms_since_run_start=elapsed_ms,- latency_ms=latency_ms,- success=success,- status_code=status_code,- timeout=timeout,- exception_type=exception_type,- error_code=error_code,- error_message=error_message,- session_id=session_id,- cycle_index=cycle_index,- worker_id=worker_id,- task_id=task_id,- )- )- return response, body+ async def find(self, *, query: str, target_uri: str, limit: int) -> Any: ...- @staticmethod- def _is_success(status_code: Optional[int], body: Optional[Dict[str, Any]]) -> bool:- if status_code is None or status_code >= 400:- return False- if not isinstance(body, dict):- return status_code < 400- if "status" in body:- return body.get("status") == "ok"- return True+ async def search(+ self, *, query: str, target_uri: str, session_id: Optional[str], limit: int+ ) -> Any: ...+ async def grep(self, *, uri: str, pattern: str, limit: int) -> Any: ...++ async def glob(self, *, uri: str, pattern: str, limit: int) -> Any: ...++ async def add_message(self, *, session_id: str, role: str, content: str) -> Any: ...++ async def commit_session(self, *, session_id: str) -> Any: ...++ async def get_task(self, *, task_id: str) -> Any: ...-class CommitTaskPoller:- def __init__(- self,- client: BenchmarkHTTPClient,- recorder: Recorder,- phase_state: PhaseState,- poll_interval: float,- ) -> None:- self._client = client- self._recorder = recorder- self._phase_state = phase_state- self._poll_interval = poll_interval- self._pending: Dict[str, PendingCommitTask] = {}- self._closed = False- self._wake_event = asyncio.Event()- self._lock = asyncio.Lock()-- async def register(self, task: PendingCommitTask) -> None:- async with self._lock:- self._pending[task.task_id] = task- self._wake_event.set()++class AsyncHTTPAdapter(LoadAdapter):+ def __init__(self, *, name: str, config: BenchmarkConfig, sdk_import: bool) -> None:+ self.name = name+ self._config = config+ self._sdk_import = sdk_import+ self._client: Any = None++ async def initialize(self) -> None:+ if self._sdk_import:+ import openviking as ov++ client_cls = ov.AsyncHTTPClient+ else:+ from openviking_cli.client.http import AsyncHTTPClient++ client_cls = AsyncHTTPClient+ self._client = client_cls(+ url=self._config.server_url,+ api_key=self._config.api_key,+ account=self._config.account,+ user=self._config.user,+ agent_id=direct_http_agent_id(self._config),+ timeout=self._config.timeout,+ extra_headers={},+ )+ await self._client.initialize()async def close(self) -> None:- self._closed = True- self._wake_event.set()+ if self._client is not None:+ await self._client.close()++ async def health(self) -> Any:+ return {"healthy": await self._client.health()}++ async def system_status(self) -> Any:+ return await self._client._get_system_status()++ async def observer_queue(self) -> Any:+ return await self._client._get_queue_status()++ async def add_resource(+ self, *, path: str, to: str, reason: str, wait: bool, timeout: Optional[float]+ ) -> Any:+ return await self._client.add_resource(+ path=path,+ to=to,+ reason=reason,+ wait=wait,+ timeout=timeout,+ strict=False,+ )- async def drain(self, timeout: float) -> None:- deadline = time.perf_counter() + timeout- while True:- async with self._lock:- remaining = len(self._pending)- if remaining == 0:- return- if time.perf_counter() >= deadline:- return- await asyncio.sleep(min(self._poll_interval, 0.5))-- async def finalize_incomplete(self) -> None:- async with self._lock:- leftovers = list(self._pending.values())- self._pending.clear()- for item in leftovers:- local_duration_ms = (time.perf_counter() - item.local_started_monotonic) * 1000.0- self._recorder.add_task(- CommitTaskEvent(- task_id=item.task_id,- session_id=item.session_id,- origin_phase=item.origin_phase,- completion_phase=self._phase_state.current,- status="incomplete",- created_at=None,- updated_at=None,- server_duration_ms=None,- local_duration_ms=local_duration_ms,- active_count_updated=None,- memories_extracted=None,- error="task not completed before benchmark end",- cycle_index=item.cycle_index,- polled_at=utc_now(),- )- )+ async def wait_processed(self, timeout: Optional[float]) -> Any:+ return await self._client.wait_processed(timeout=timeout)- async def run(self) -> None:- while True:- await self._wake_event.wait()- self._wake_event.clear()+ async def find(self, *, query: str, target_uri: str, limit: int) -> Any:+ result = await self._client.find(query=query, target_uri=target_uri, limit=limit)+ return to_jsonable(result)- while True:- async with self._lock:- pending = list(self._pending.values())- if not pending:- break- await self._poll_pending(pending)- if self._closed:- return- await asyncio.sleep(self._poll_interval)+ async def search(+ self, *, query: str, target_uri: str, session_id: Optional[str], limit: int+ ) -> Any:+ result = await self._client.search(+ query=query,+ target_uri=target_uri,+ session_id=session_id,+ limit=limit,+ )+ return to_jsonable(result)- if self._closed:- return+ async def grep(self, *, uri: str, pattern: str, limit: int) -> Any:+ return await self._client.grep(uri=uri, pattern=pattern, node_limit=limit)- async def _poll_pending(self, pending: List[PendingCommitTask]) -> None:- coroutines = [self._poll_one(item) for item in pending]- results = await asyncio.gather(*coroutines, return_exceptions=True)- completed_ids = [task_id for task_id in results if isinstance(task_id, str)]- if not completed_ids:- return- async with self._lock:- for task_id in completed_ids:- self._pending.pop(task_id, None)-- async def _poll_one(self, item: PendingCommitTask) -> Optional[str]:- _, body = await self._client.request_json(- api="get_task",- method="GET",- path=f"/api/v1/tasks/{item.task_id}",- phase=self._phase_state.current,- session_id=item.session_id,- cycle_index=item.cycle_index,- task_id=item.task_id,+ async def glob(self, *, uri: str, pattern: str, limit: int) -> Any:+ return await self._client.glob(pattern=pattern, uri=uri)++ async def add_message(self, *, session_id: str, role: str, content: str) -> Any:+ return await self._client.add_message(session_id, role, content=content)++ async def commit_session(self, *, session_id: str) -> Any:+ return await self._client.commit_session(session_id)++ async def get_task(self, *, task_id: str) -> Any:+ return await self._client.get_task(task_id)+++class CliSubprocessAdapter(LoadAdapter):+ def __init__(self, *, config: BenchmarkConfig, cli_config_path: Path) -> None:+ self.name = "cli-subprocess"+ self._config = config+ self._cli_config_path = cli_config_path+ self._env = os.environ.copy()+ self._env["OPENVIKING_CLI_CONFIG_FILE"] = str(cli_config_path)++ async def initialize(self) -> None:+ self._cli_config_path.parent.mkdir(parents=True, exist_ok=True)+ self._cli_config_path.write_text(+ json.dumps(build_cli_config_payload(self._config), ensure_ascii=False, indent=2),+ encoding="utf-8",)- if not isinstance(body, dict) or body.get("status") != "ok":- return None- result = body.get("result") or {}- task_status = result.get("status")- if task_status not in {"completed", "failed"}:- return None+ result = await self.health()+ if isinstance(result, AdapterResult) and not result.success:+ raise RuntimeError(result.error_message or "ov health failed")- created_at = to_float(result.get("created_at"))- updated_at = to_float(result.get("updated_at"))- server_duration_ms = None- if created_at is not None and updated_at is not None:- server_duration_ms = max(updated_at - created_at, 0.0) * 1000.0- local_duration_ms = (time.perf_counter() - item.local_started_monotonic) * 1000.0- task_result = result.get("result") or {}- self._recorder.add_task(- CommitTaskEvent(- task_id=item.task_id,- session_id=item.session_id,- origin_phase=item.origin_phase,- completion_phase=self._phase_state.current,- status=task_status,- created_at=created_at,- updated_at=updated_at,- server_duration_ms=server_duration_ms,- local_duration_ms=local_duration_ms,- active_count_updated=task_result.get("active_count_updated"),- memories_extracted=task_result.get("memories_extracted"),- error=result.get("error"),- cycle_index=item.cycle_index,- polled_at=utc_now(),- )+ async def close(self) -> None:+ return None++ async def health(self) -> AdapterResult:+ return await self._run(["health"], operation_timeout=self._config.timeout)++ async def system_status(self) -> AdapterResult:+ return await self._run(["status"])++ async def observer_queue(self) -> AdapterResult:+ return await self._run(["observer", "queue"])++ async def add_resource(+ self, *, path: str, to: str, reason: str, wait: bool, timeout: Optional[float]+ ) -> AdapterResult:+ args = ["add-resource", path, "--to", to, "--reason", reason]+ if wait:+ args.append("--wait")+ if timeout is not None:+ args.extend(["--timeout", str(timeout)])+ return await self._run(args, operation_timeout=max(self._config.timeout, 60.0))++ async def wait_processed(self, timeout: Optional[float]) -> AdapterResult:+ args = ["wait"]+ if timeout is not None:+ args.extend(["--timeout", str(timeout)])+ return await self._run(args, operation_timeout=timeout or self._config.timeout)++ async def find(self, *, query: str, target_uri: str, limit: int) -> AdapterResult:+ return await self._run(["find", query, "--uri", target_uri, "-n", str(limit)])++ async def search(+ self, *, query: str, target_uri: str, session_id: Optional[str], limit: int+ ) -> AdapterResult:+ args = ["search", query, "--uri", target_uri, "-n", str(limit)]+ if session_id:+ args.extend(["--session-id", session_id])+ return await self._run(args)++ async def grep(self, *, uri: str, pattern: str, limit: int) -> AdapterResult:+ return await self._run(["grep", pattern, "--uri", uri, "-n", str(limit)])++ async def glob(self, *, uri: str, pattern: str, limit: int) -> AdapterResult:+ return await self._run(["glob", pattern, "--uri", uri, "-n", str(limit)])++ async def add_message(self, *, session_id: str, role: str, content: str) -> AdapterResult:+ return await self._run(+ ["session", "add-message", session_id, "--role", role, "--content", content])- return item.task_id++ async def commit_session(self, *, session_id: str) -> AdapterResult:+ return await self._run(["session", "commit", session_id])++ async def get_task(self, *, task_id: str) -> AdapterResult:+ return await self._run(["task", "status", task_id])++ def build_command(self, args: Sequence[str]) -> List[str]:+ return [+ self._config.ov_bin,+ "--output",+ "json",+ "--compact",+ "--no-progress",+ *args,+ ]++ async def _run(+ self, args: Sequence[str], *, operation_timeout: Optional[float] = None+ ) -> AdapterResult:+ command = self.build_command(args)+ try:+ proc = await asyncio.create_subprocess_exec(+ *command,+ stdout=asyncio.subprocess.PIPE,+ stderr=asyncio.subprocess.PIPE,+ env=self._env,+ )+ stdout_bytes, stderr_bytes = await asyncio.wait_for(+ proc.communicate(),+ timeout=operation_timeout or self._config.timeout,+ )+ except asyncio.TimeoutError:+ return AdapterResult(+ success=False,+ exception_type="TimeoutError",+ error_message=f"CLI command timed out after {operation_timeout or self._config.timeout}s",+ command=command,+ )+ except Exception as exc: # pragma: no cover - exercised in real runs+ return AdapterResult(+ success=False,+ exception_type=type(exc).__name__,+ error_message=truncate_error_message(str(exc)),+ command=command,+ )+ stdout = stdout_bytes.decode("utf-8", errors="replace")+ stderr = stderr_bytes.decode("utf-8", errors="replace")+ parsed = parse_cli_process_result(proc.returncode, stdout, stderr)+ parsed.command = command+ return parsedclass BenchmarkRunner:def __init__(self, config: BenchmarkConfig) -> None:self.config = config+ self.run_id = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")self.random = random.Random(config.seed)self.recorder = Recorder()- self.phase_state = PhaseState()- self.phase_metadata: List[PhaseMetadata] = []- self.phase_durations: Dict[str, float] = {}- self.session_ids: List[str] = []- self.session_queue: asyncio.Queue[str] = asyncio.Queue()- self.session_cycle_counts: Dict[str, int] = {}- self.extract_semaphore = asyncio.Semaphore(max(1, config.extract_concurrency))- self.client = BenchmarkHTTPClient(config, self.recorder)- self.task_poller = CommitTaskPoller(- client=self.client,- recorder=self.recorder,- phase_state=self.phase_state,- poll_interval=config.task_poll_interval,- )+ self.run_start_monotonic = time.perf_counter()+ self.pending_tasks: List[PendingTask] = []+ self.session_ids_by_adapter: Dict[str, List[str]] = {}+ self.adapters: Dict[str, LoadAdapter] = {}+ self.mixed_doc_counter = count()async def run(self) -> int:- poller_task = asyncio.create_task(self.task_poller.run())exit_code = 0try:- await self._preflight()- await self._create_sessions()- await self._run_phase(- phase="baseline",- duration_seconds=self.config.baseline_seconds,- enable_readers=self.config.reader_concurrency > 0,- enable_writers=False,- enable_sampler=self.config.observer_interval > 0,- )- await self._run_phase(- phase="mixed_load",- duration_seconds=self.config.mixed_seconds,- enable_readers=self.config.reader_concurrency > 0,- enable_writers=self.config.writer_concurrency > 0 and bool(self.session_ids),- enable_sampler=self.config.observer_interval > 0,- )- await self._run_phase(- phase="recovery",- duration_seconds=self.config.recovery_seconds,- enable_readers=self.config.reader_concurrency > 0,- enable_writers=False,- enable_sampler=self.config.observer_interval > 0,- )- if self.config.task_drain_timeout > 0:- self.phase_state.current = "drain"- await self.task_poller.drain(self.config.task_drain_timeout)- except RuntimeError as exc:- self.recorder.add_note(f"fatal: {exc}")- print(f"[fatal] {exc}", file=sys.stderr)+ await self._prepare_filesystem()+ await self._prepare_remote_state()+ self.adapters = await self._create_adapters()+ for adapter in self.adapters.values():+ await self._run_adapter(adapter)+ if self.pending_tasks:+ await self._drain_tasks("final_drain", self.config.drain_timeout)+ except Exception as exc:+ self.recorder.add_note(f"fatal: {type(exc).__name__}: {exc}")+ print(f"[fatal] {type(exc).__name__}: {exc}", file=sys.stderr)exit_code = 1finally:- await self.task_poller.close()- await poller_task- await self.task_poller.finalize_incomplete()- if self.config.cleanup and self.session_ids:- await self._cleanup_sessions()- await self.client.aclose()-- self._write_outputs()- self._print_summary()+ if self.config.cleanup_at_end:+ try:+ await self._clear_remote_state()+ except Exception as exc: # pragma: no cover - best effort cleanup+ self.recorder.add_note(f"cleanup_at_end failed: {type(exc).__name__}: {exc}")+ for adapter in self.adapters.values():+ await adapter.close()+ self._write_outputs()+ self._print_summary_path()return exit_code- async def _preflight(self) -> None:- self.phase_state.current = "setup"- _, health_body = await self.client.request_json(- api="health",- method="GET",- path="/health",- phase="setup",+ async def _create_adapters(self) -> Dict[str, LoadAdapter]:+ output_dir = Path(self.config.output_dir)+ adapters: Dict[str, LoadAdapter] = {}+ for name in self.config.adapters:+ if name == "sdk":+ adapter: LoadAdapter = AsyncHTTPAdapter(+ name="sdk", config=self.config, sdk_import=True+ )+ elif name == "cli-http":+ adapter = AsyncHTTPAdapter(name="cli-http", config=self.config, sdk_import=False)+ elif name == "cli-subprocess":+ adapter = CliSubprocessAdapter(+ config=self.config,+ cli_config_path=output_dir / "runtime" / "ovcli.conf",+ )+ else:+ raise ValueError(f"Unknown adapter: {name}")+ await adapter.initialize()+ adapters[name] = adapter+ return adapters++ async def _prepare_filesystem(self) -> None:+ local_dir = Path(self.config.local_data_dir)+ if self.config.clear_before_run and local_dir.exists():+ shutil.rmtree(local_dir)+ local_dir.mkdir(parents=True, exist_ok=True)+ Path(self.config.output_dir).mkdir(parents=True, exist_ok=True)+ for adapter_name in self.config.adapters:+ adapter_dir = local_dir / "seed" / adapter_name+ adapter_dir.mkdir(parents=True, exist_ok=True)+ for index in range(self.config.resource_count):+ write_benchmark_document(+ adapter_dir / f"resource-{index:04d}.md",+ adapter=adapter_name,+ index=index,+ run_id=self.run_id,+ )++ async def _prepare_remote_state(self) -> None:+ if self.config.clear_before_run:+ await self._clear_remote_state()+ await self._ensure_remote_tree()++ async def _clear_remote_state(self) -> None:+ from openviking_cli.client.http import AsyncHTTPClient+ from openviking_cli.exceptions import NotFoundError, OpenVikingError++ client = AsyncHTTPClient(+ url=self.config.server_url,+ api_key=self.config.api_key,+ account=self.config.account,+ user=self.config.user,+ agent_id=direct_http_agent_id(self.config),+ timeout=self.config.timeout,+ extra_headers={},)- if not isinstance(health_body, dict) or health_body.get("status") != "ok":- raise RuntimeError("server health check failed")-- _, status_body = await self.client.request_json(- api="system_status",- method="GET",- path="/api/v1/system/status",- phase="setup",+ await client.initialize()+ try:+ try:+ await client.rm(self.config.data_root_uri, recursive=True)+ except NotFoundError:+ pass+ except OpenVikingError as exc:+ if not is_not_found_error(exc):+ raise+ sessions = await client.list_sessions()+ for session_id in extract_session_ids(sessions):+ if session_id.startswith(self.config.session_prefix):+ try:+ await client.delete_session(session_id)+ except Exception:+ pass+ finally:+ await client.close()++ async def _ensure_remote_tree(self) -> None:+ from openviking_cli.client.http import AsyncHTTPClient+ from openviking_cli.exceptions import OpenVikingError++ client = AsyncHTTPClient(+ url=self.config.server_url,+ api_key=self.config.api_key,+ account=self.config.account,+ user=self.config.user,+ agent_id=direct_http_agent_id(self.config),+ timeout=self.config.timeout,+ extra_headers={},+ )+ await client.initialize()+ try:+ for uri in iter_resource_tree_uris(self.config.data_root_uri):+ try:+ await client.mkdir(uri)+ except OpenVikingError as exc:+ if not is_already_exists_error(exc):+ raise+ for adapter_name in self.config.adapters:+ try:+ await client.mkdir(f"{self.config.data_root_uri}/{adapter_name}")+ except OpenVikingError as exc:+ if not is_already_exists_error(exc):+ raise+ finally:+ await client.close()++ async def _run_adapter(self, adapter: LoadAdapter) -> None:+ self.session_ids_by_adapter[adapter.name] = [+ f"{self.config.session_prefix}{self.run_id}-{adapter.name}-{i:04d}"+ for i in range(self.config.session_count)+ ]+ await self._run_warmup(adapter)+ await self._run_add_resources(adapter)+ await self._drain_tasks(f"{adapter.name}:resource_drain", self.config.drain_timeout)+ await self._run_session_messages(adapter)+ await self._run_timed_scenario(+ adapter=adapter,+ scenario="retrieval",+ duration_seconds=self.config.phase_seconds,+ concurrency=self.config.search_concurrency,+ worker=self._retrieval_worker,)- if not isinstance(status_body, dict) or status_body.get("status") != "ok":- raise RuntimeError("authenticated system status request failed")-- _, models_body = await self.client.request_json(- api="observer_models",- method="GET",- path="/api/v1/observer/models",- phase="setup",+ await self._run_session_commit(adapter)+ await self._drain_tasks(f"{adapter.name}:commit_drain", self.config.drain_timeout)+ await self._run_timed_scenario(+ adapter=adapter,+ scenario="mixed",+ duration_seconds=self.config.mixed_seconds,+ concurrency=self.config.mixed_concurrency,+ worker=self._mixed_worker,)- model_result = (models_body or {}).get("result") if isinstance(models_body, dict) else None- model_note = self._extract_model_note(model_result)- if model_note:- self.recorder.add_note(model_note)-- if self.config.extract_ratio > 0:- preflight_result = await self._run_extract_preflight()- if preflight_result:- self.recorder.add_note(preflight_result)- if self.config.require_extract_load:- raise RuntimeError(preflight_result)-- async def _run_extract_preflight(self) -> Optional[str]:- _, create_body = await self.client.request_json(- api="create_session",- method="POST",- path="/api/v1/sessions",- phase="setup",+ await self._drain_tasks(f"{adapter.name}:mixed_drain", self.config.drain_timeout)++ async def _run_warmup(self, adapter: LoadAdapter) -> None:+ if self.config.warmup_seconds <= 0:+ return+ await self._run_timed_scenario(+ adapter=adapter,+ scenario="warmup",+ duration_seconds=self.config.warmup_seconds,+ concurrency=1,+ worker=self._warmup_worker,)- session_id = extract_session_id(create_body)- if not session_id:- return "extract preflight could not create session"- try:- payload = {- "role": "user",- "content": build_message_content(- session_id=session_id,- cycle_index=0,- message_index=0,- size=self.config.message_size,- ),- }- await self.client.request_json(- api="add_message",- method="POST",- path=f"/api/v1/sessions/{session_id}/messages",- phase="setup",- session_id=session_id,- cycle_index=0,- json_payload=payload,- )- _, extract_body = await self.client.request_json(- api="extract",- method="POST",- path=f"/api/v1/sessions/{session_id}/extract",- phase="setup",- session_id=session_id,- cycle_index=0,+ async def _warmup_worker(+ self, adapter: LoadAdapter, scenario: str, worker_id: int, stop_event: asyncio.Event+ ) -> None:+ while not stop_event.is_set():+ await self._record_request(+ adapter,+ scenario,+ "health",+ lambda: adapter.health(),+ worker_id=worker_id,)- if not isinstance(extract_body, dict) or extract_body.get("status") != "ok":- return "extract preflight request failed"- result = extract_body.get("result")- if isinstance(result, list) and not result:- return (- "extract preflight returned empty result; long-tail load may be weak if models are "- "not configured"+ await asyncio.sleep(0.2)++ async def _run_add_resources(self, adapter: LoadAdapter) -> None:+ scenario = "add_resources"+ started = time.perf_counter()+ started_wall = utc_now()+ files = sorted((Path(self.config.local_data_dir) / "seed" / adapter.name).glob("*.md"))+ queue: asyncio.Queue[Path] = asyncio.Queue()+ for path in files:+ await queue.put(path)++ async def worker(worker_id: int) -> None:+ while True:+ try:+ path = queue.get_nowait()+ except asyncio.QueueEmpty:+ return+ resource_uri = f"{self.config.data_root_uri}/{adapter.name}/{path.name}"+ result = await self._record_request(+ adapter,+ scenario,+ "add_resource",+ lambda p=path, uri=resource_uri: adapter.add_resource(+ path=str(p),+ to=uri,+ reason="OpenViking server load benchmark",+ wait=False,+ timeout=None,+ ),+ resource_uri=resource_uri,+ worker_id=worker_id,)- return None- finally:- await self.client.request_json(- api="delete_session",- method="DELETE",- path=f"/api/v1/sessions/{session_id}",- phase="setup",- session_id=session_id,+ self._register_task_if_present(+ adapter=adapter,+ scenario=scenario,+ task_type="add_resource",+ resource_id=resource_uri,+ result=result,+ )++ await asyncio.gather(+ *[+ asyncio.create_task(worker(worker_id))+ for worker_id in range(max(1, self.config.resource_concurrency))+ ]+ )+ await self._record_request(+ adapter,+ scenario,+ "wait_processed",+ lambda: adapter.wait_processed(timeout=self.config.drain_timeout),+ )+ self.recorder.add_phase(+ PhaseMetadata(+ adapter=adapter.name,+ scenario=scenario,+ started_at=started_wall,+ ended_at=utc_now(),+ duration_seconds=time.perf_counter() - started,)+ )- def _extract_model_note(self, model_result: Any) -> Optional[str]:- if not isinstance(model_result, dict):- return None- is_healthy = model_result.get("is_healthy")- status = model_result.get("status")- if is_healthy is False:- return f"observer/models reports unhealthy state; extract load may not be representative: {status}"- return None+ async def _run_session_messages(self, adapter: LoadAdapter) -> None:+ scenario = "session_messages"+ started = time.perf_counter()+ started_wall = utc_now()+ units = [+ (session_id, message_index)+ for session_id in self.session_ids_by_adapter[adapter.name]+ for message_index in range(self.config.messages_per_session)+ ]+ queue: asyncio.Queue[tuple[str, int]] = asyncio.Queue()+ for unit in units:+ await queue.put(unit)- async def _create_sessions(self) -> None:- if self.config.session_count <= 0:- return- for _ in range(self.config.session_count):- _, body = await self.client.request_json(- api="create_session",- method="POST",- path="/api/v1/sessions",- phase="setup",+ async def worker(worker_id: int) -> None:+ while True:+ try:+ session_id, message_index = queue.get_nowait()+ except asyncio.QueueEmpty:+ return+ role = "user" if message_index % 2 == 0 else "assistant"+ content = build_message_content(+ session_id=session_id,+ message_index=message_index,+ adapter=adapter.name,+ )+ await self._record_request(+ adapter,+ scenario,+ "add_message",+ lambda sid=session_id, r=role, c=content: adapter.add_message(+ session_id=sid,+ role=r,+ content=c,+ ),+ session_id=session_id,+ worker_id=worker_id,+ )++ await asyncio.gather(+ *[+ asyncio.create_task(worker(worker_id))+ for worker_id in range(max(1, self.config.session_concurrency))+ ]+ )+ self.recorder.add_phase(+ PhaseMetadata(+ adapter=adapter.name,+ scenario=scenario,+ started_at=started_wall,+ ended_at=utc_now(),+ duration_seconds=time.perf_counter() - started,)- session_id = extract_session_id(body)- if not session_id:- raise RuntimeError("failed to create benchmark sessions")- self.session_ids.append(session_id)- self.session_cycle_counts[session_id] = 0- await self.session_queue.put(session_id)-- async def _cleanup_sessions(self) -> None:- self.phase_state.current = "cleanup"- for session_id in self.session_ids:- await self.client.request_json(- api="delete_session",- method="DELETE",- path=f"/api/v1/sessions/{session_id}",- phase="cleanup",- session_id=session_id,+ )++ async def _run_session_commit(self, adapter: LoadAdapter) -> None:+ scenario = "session_commit"+ started = time.perf_counter()+ started_wall = utc_now()+ queue: asyncio.Queue[str] = asyncio.Queue()+ for session_id in self.session_ids_by_adapter[adapter.name]:+ await queue.put(session_id)++ async def worker(worker_id: int) -> None:+ while True:+ try:+ session_id = queue.get_nowait()+ except asyncio.QueueEmpty:+ return+ result = await self._record_request(+ adapter,+ scenario,+ "commit_session",+ lambda sid=session_id: adapter.commit_session(session_id=sid),+ session_id=session_id,+ worker_id=worker_id,+ )+ self._register_task_if_present(+ adapter=adapter,+ scenario=scenario,+ task_type="session_commit",+ resource_id=session_id,+ result=result,+ )++ await asyncio.gather(+ *[+ asyncio.create_task(worker(worker_id))+ for worker_id in range(max(1, self.config.commit_concurrency))+ ]+ )+ self.recorder.add_phase(+ PhaseMetadata(+ adapter=adapter.name,+ scenario=scenario,+ started_at=started_wall,+ ended_at=utc_now(),+ duration_seconds=time.perf_counter() - started,)+ )- async def _run_phase(+ async def _run_timed_scenario(self,*,- phase: str,+ adapter: LoadAdapter,+ scenario: str,duration_seconds: float,- enable_readers: bool,- enable_writers: bool,- enable_sampler: bool,+ concurrency: int,+ worker: Callable[[LoadAdapter, str, int, asyncio.Event], Awaitable[None]],) -> None:- if duration_seconds <= 0:+ if duration_seconds <= 0 or concurrency <= 0:return-- self.phase_state.current = phasestop_event = asyncio.Event()- tasks: List[asyncio.Task[Any]] = []-- if enable_readers:- for worker_id in range(self.config.reader_concurrency):- tasks.append(asyncio.create_task(self._reader_worker(phase, worker_id, stop_event)))- if enable_writers:- for worker_id in range(self.config.writer_concurrency):- tasks.append(asyncio.create_task(self._writer_worker(phase, worker_id, stop_event)))- if enable_sampler:- tasks.append(asyncio.create_task(self._sampler_worker(phase, stop_event)))-- phase_started = time.perf_counter()+ started = time.perf_counter()started_wall = utc_now()+ tasks = [+ asyncio.create_task(worker(adapter, scenario, worker_id, stop_event))+ for worker_id in range(concurrency)+ ]await asyncio.sleep(duration_seconds)stop_event.set()- if tasks:- await asyncio.gather(*tasks, return_exceptions=True)- phase_duration = time.perf_counter() - phase_started- ended_wall = utc_now()- self.phase_metadata.append(+ await asyncio.gather(*tasks, return_exceptions=True)+ self.recorder.add_phase(PhaseMetadata(- phase=phase,+ adapter=adapter.name,+ scenario=scenario,started_at=started_wall,- ended_at=ended_wall,- duration_seconds=phase_duration,+ ended_at=utc_now(),+ duration_seconds=time.perf_counter() - started,))- self.phase_durations[phase] = phase_duration- async def _writer_worker(self, phase: str, worker_id: int, stop_event: asyncio.Event) -> None:+ async def _retrieval_worker(+ self, adapter: LoadAdapter, scenario: str, worker_id: int, stop_event: asyncio.Event+ ) -> None:while not stop_event.is_set():- session_id = await self._borrow_session(stop_event)- if not session_id:- return- try:- cycle_index = self.session_cycle_counts[session_id]- self.session_cycle_counts[session_id] += 1- await self._run_session_cycle(- phase=phase,+ await self._run_retrieval_operation(adapter, scenario, worker_id)++ async def _mixed_worker(+ self, adapter: LoadAdapter, scenario: str, worker_id: int, stop_event: asyncio.Event+ ) -> None:+ choices = [+ ("find", 35),+ ("search", 20),+ ("add_message", 20),+ ("commit_session", 10),+ ("add_resource", 10),+ ("observer_queue", 3),+ ("system_status", 2),+ ]+ operations = [name for name, weight in choices for _ in range(weight)]+ while not stop_event.is_set():+ operation = self.random.choice(operations)+ if operation in {"find", "search"}:+ await self._run_retrieval_operation(adapter, scenario, worker_id, force=operation)+ elif operation == "add_message":+ session_id = self.random.choice(self.session_ids_by_adapter[adapter.name])+ content = build_message_content(+ session_id=session_id,+ message_index=next(self.mixed_doc_counter),+ adapter=adapter.name,+ )+ await self._record_request(+ adapter,+ scenario,+ "add_message",+ lambda sid=session_id, c=content: adapter.add_message(+ session_id=sid,+ role="user",+ content=c,+ ),+ session_id=session_id,worker_id=worker_id,+ )+ elif operation == "commit_session":+ session_id = self.random.choice(self.session_ids_by_adapter[adapter.name])+ result = await self._record_request(+ adapter,+ scenario,+ "commit_session",+ lambda sid=session_id: adapter.commit_session(session_id=sid),session_id=session_id,- cycle_index=cycle_index,+ worker_id=worker_id,+ )+ self._register_task_if_present(+ adapter=adapter,+ scenario=scenario,+ task_type="session_commit",+ resource_id=session_id,+ result=result,+ )+ elif operation == "add_resource":+ index = next(self.mixed_doc_counter)+ path = (+ Path(self.config.local_data_dir) / "mixed" / adapter.name / f"mixed-{index}.md"+ )+ write_benchmark_document(+ path, adapter=adapter.name, index=index, run_id=self.run_id+ )+ resource_uri = f"{self.config.data_root_uri}/{adapter.name}/{path.name}"+ result = await self._record_request(+ adapter,+ scenario,+ "add_resource",+ lambda p=path, uri=resource_uri: adapter.add_resource(+ path=str(p),+ to=uri,+ reason="OpenViking mixed load benchmark",+ wait=False,+ timeout=None,+ ),+ resource_uri=resource_uri,+ worker_id=worker_id,+ )+ self._register_task_if_present(+ adapter=adapter,+ scenario=scenario,+ task_type="add_resource",+ resource_id=resource_uri,+ result=result,+ )+ elif operation == "observer_queue":+ await self._record_request(+ adapter,+ scenario,+ "observer_queue",+ lambda: adapter.observer_queue(),+ worker_id=worker_id,+ )+ else:+ await self._record_request(+ adapter,+ scenario,+ "system_status",+ lambda: adapter.system_status(),+ worker_id=worker_id,)- finally:- await self.session_queue.put(session_id)- async def _run_session_cycle(+ async def _run_retrieval_operation(self,- *,- phase: str,+ adapter: LoadAdapter,+ scenario: str,worker_id: int,- session_id: str,- cycle_index: int,+ *,+ force: Optional[str] = None,) -> None:- successful_messages = 0- for message_index in range(self.config.messages_per_commit):- payload = {- "role": "user",- "content": build_message_content(- session_id=session_id,- cycle_index=cycle_index,- message_index=message_index,- size=self.config.message_size,+ operation = (+ force+ or self.random.choices(+ ["find", "search", "grep", "glob"], weights=[55, 25, 10, 10], k=1+ )[0]+ )+ query = self.random.choice(DEFAULT_QUERIES)+ if operation == "find":+ await self._record_request(+ adapter,+ scenario,+ "find",+ lambda: adapter.find(+ query=query,+ target_uri=self.config.data_root_uri,+ limit=self.config.find_limit,+ ),+ worker_id=worker_id,+ )+ elif operation == "search":+ session_id = self.random.choice(self.session_ids_by_adapter[adapter.name])+ await self._record_request(+ adapter,+ scenario,+ "search",+ lambda sid=session_id: adapter.search(+ query=query,+ target_uri=self.config.data_root_uri,+ session_id=sid,+ limit=self.config.find_limit,),- }- _, body = await self.client.request_json(- api="add_message",- method="POST",- path=f"/api/v1/sessions/{session_id}/messages",- phase=phase,session_id=session_id,- cycle_index=cycle_index,worker_id=worker_id,- json_payload=payload,)- if isinstance(body, dict) and body.get("status") == "ok":- successful_messages += 1-- if successful_messages <= 0:- return-- if self.config.extract_ratio > 0 and self.random.random() < self.config.extract_ratio:- async with self.extract_semaphore:- await self.client.request_json(- api="extract",- method="POST",- path=f"/api/v1/sessions/{session_id}/extract",- phase=phase,- session_id=session_id,- cycle_index=cycle_index,- worker_id=worker_id,- )+ elif operation == "grep":+ await self._record_request(+ adapter,+ scenario,+ "grep",+ lambda: adapter.grep(+ uri=self.config.data_root_uri,+ pattern="benchmark",+ limit=self.config.find_limit,+ ),+ worker_id=worker_id,+ )+ else:+ await self._record_request(+ adapter,+ scenario,+ "glob",+ lambda: adapter.glob(+ uri=self.config.data_root_uri,+ pattern="*.md",+ limit=self.config.find_limit,+ ),+ worker_id=worker_id,+ )- _, body = await self.client.request_json(- api="commit",- method="POST",- path=f"/api/v1/sessions/{session_id}/commit",- phase=phase,- session_id=session_id,- cycle_index=cycle_index,- worker_id=worker_id,- )- task_id = extract_task_id(body)- if task_id:- await self.task_poller.register(- PendingCommitTask(- task_id=task_id,- session_id=session_id,- origin_phase=phase,- cycle_index=cycle_index,- local_started_monotonic=time.perf_counter(),- )+ async def _record_request(+ self,+ adapter: LoadAdapter,+ scenario: str,+ operation: str,+ call: Callable[[], Awaitable[Any]],+ *,+ session_id: Optional[str] = None,+ resource_uri: Optional[str] = None,+ worker_id: Optional[int] = None,+ ) -> AdapterResult:+ started_monotonic = time.perf_counter()+ started_wall = utc_now()+ result: AdapterResult+ try:+ value = await call()+ if isinstance(value, AdapterResult):+ result = value+ else:+ result = AdapterResult(success=True, result=to_jsonable(value))+ except Exception as exc:+ result = AdapterResult(+ success=False,+ exception_type=type(exc).__name__,+ error_message=truncate_error_message(str(exc)),)- async def _reader_worker(self, phase: str, worker_id: int, stop_event: asyncio.Event) -> None:- while not stop_event.is_set():- payload = {- "query": self.random.choice(self.config.find_queries),- "limit": self.config.find_limit,- }- if self.config.find_target_uri:- payload["target_uri"] = self.config.find_target_uri- if self.config.find_score_threshold is not None:- payload["score_threshold"] = self.config.find_score_threshold- await self.client.request_json(- api="find",- method="POST",- path="/api/v1/search/find",- phase=phase,+ ended_monotonic = time.perf_counter()+ result_payload = normalize_result_payload(result.result)+ task_id = extract_task_id(result_payload)+ if task_id is None:+ task_id = extract_task_id_from_any(result.result)+ self.recorder.add_request(+ RequestEvent(+ adapter=adapter.name,+ scenario=scenario,+ operation=operation,+ started_at=started_wall,+ ended_at=utc_now(),+ elapsed_ms_since_run_start=(started_monotonic - self.run_start_monotonic) * 1000.0,+ latency_ms=(ended_monotonic - started_monotonic) * 1000.0,+ success=result.success,+ status_code=result.status_code,+ exception_type=result.exception_type,+ error_message=result.error_message,+ session_id=session_id,+ resource_uri=resource_uri or extract_root_uri(result_payload),+ task_id=task_id,worker_id=worker_id,- json_payload=payload,+ command=" ".join(result.command) if result.command else None,)+ )+ return result- async def _sampler_worker(self, phase: str, stop_event: asyncio.Event) -> None:- sample_specs = [- ("system_status", "GET", "/api/v1/system/status"),- ("observer_queue", "GET", "/api/v1/observer/queue"),- ("observer_system", "GET", "/api/v1/observer/system"),- ]- while not stop_event.is_set():- for api, method, path in sample_specs:- started = time.perf_counter()- response, body = await self.client.request_json(- api=api,- method=method,- path=path,- phase=phase,- )- latency_ms = (time.perf_counter() - started) * 1000.0- success = response is not None and self.client._is_success(- response.status_code if response else None,- body,+ def _register_task_if_present(+ self,+ *,+ adapter: LoadAdapter,+ scenario: str,+ task_type: str,+ resource_id: Optional[str],+ result: AdapterResult,+ ) -> None:+ if not result.success:+ return+ task_id = extract_task_id_from_any(result.result)+ if not task_id:+ return+ self.pending_tasks.append(+ PendingTask(+ adapter_name=adapter.name,+ task_id=task_id,+ task_type=task_type,+ origin_scenario=scenario,+ resource_id=resource_id,+ local_started_monotonic=time.perf_counter(),+ )+ )++ async def _drain_tasks(self, scenario: str, timeout: float) -> None:+ if not self.pending_tasks or timeout <= 0:+ self._finalize_incomplete_tasks(scenario)+ return+ deadline = time.perf_counter() + timeout+ while self.pending_tasks and time.perf_counter() < deadline:+ remaining: List[PendingTask] = []+ for task in list(self.pending_tasks):+ adapter = self.adapters.get(task.adapter_name)+ if adapter is None:+ remaining.append(task)+ continue+ result = await self._record_request(+ adapter,+ scenario,+ "get_task",+ lambda a=adapter, tid=task.task_id: a.get_task(task_id=tid),)- self.recorder.add_sample(- ObserverSample(- api=api,- phase=phase,- sampled_at=utc_now(),- elapsed_ms_since_run_start=(- time.perf_counter() - self.client.run_start_monotonic+ payload = normalize_result_payload(result.result)+ status = extract_task_status(payload)+ if result.success and status in {"completed", "failed"}:+ self.recorder.add_task(+ build_task_event(+ adapter=task.adapter_name,+ task=task,+ completion_scenario=scenario,+ payload=payload,)- * 1000.0,- latency_ms=latency_ms,- success=success,- is_healthy=extract_boolean(body, "result", "is_healthy"),- has_errors=extract_boolean(body, "result", "has_errors"),- payload=body if isinstance(body, dict) else None,- error_message=extract_error(- body, response.status_code if response else None- )[1]- if response is not None and not success- else None,)+ else:+ remaining.append(task)+ self.pending_tasks = remaining+ if self.pending_tasks:+ await asyncio.sleep(self.config.task_poll_interval)+ if scenario == "final_drain":+ self._finalize_incomplete_tasks(scenario)++ def _finalize_incomplete_tasks(self, scenario: str) -> None:+ for task in self.pending_tasks:+ self.recorder.add_task(+ TaskEvent(+ adapter=task.adapter_name,+ task_id=task.task_id,+ task_type=task.task_type,+ origin_scenario=task.origin_scenario,+ completion_scenario=scenario,+ status="incomplete",+ resource_id=task.resource_id,+ local_duration_ms=(time.perf_counter() - task.local_started_monotonic) * 1000.0,+ server_duration_ms=None,+ error_message="task not completed before benchmark end",+ result=None,+ polled_at=utc_now(),)- if stop_event.is_set():- break- if stop_event.is_set():- return- try:- await asyncio.wait_for(stop_event.wait(), timeout=self.config.observer_interval)- except asyncio.TimeoutError:- continue-- async def _borrow_session(self, stop_event: asyncio.Event) -> Optional[str]:- while not stop_event.is_set():- try:- return await asyncio.wait_for(self.session_queue.get(), timeout=0.2)- except asyncio.TimeoutError:- continue- return None+ )+ self.pending_tasks = []def _write_outputs(self) -> None:output_dir = Path(self.config.output_dir)output_dir.mkdir(parents=True, exist_ok=True)-- request_summary_rows = build_request_summary_rows(+ summary_rows = build_request_summary_rows(+ events=self.recorder.request_events,+ phases=self.recorder.phases,+ )+ window_rows = build_request_window_rows(events=self.recorder.request_events,- phase_durations=self.phase_durations,- total_run_duration=total_duration_seconds(self.phase_metadata),+ window_seconds=self.config.request_window_seconds,)- task_summary_rows = build_task_summary_rows(self.recorder.task_events)- human_summary_zh = render_human_summary_zh(+ task_rows = build_task_summary_rows(self.recorder.task_events)+ adapter_rows = build_adapter_comparison_rows(summary_rows)+ error_rows = build_error_rows(self.recorder.request_events)+ report = render_report_zh(config=self.config,- output_dir=self.config.output_dir,+ run_id=self.run_id,notes=self.recorder.notes,- phase_metadata=self.phase_metadata,- request_summary_rows=request_summary_rows,- request_events=self.recorder.request_events,- task_summary_rows=task_summary_rows,- task_events=self.recorder.task_events,+ phases=self.recorder.phases,+ summary_rows=summary_rows,+ task_rows=task_rows,+ adapter_rows=adapter_rows,+ error_rows=error_rows,+ output_dir=str(output_dir),)write_json(output_dir / "run_config.json", asdict(self.config))- write_json(- output_dir / "phases.json",- [asdict(item) for item in self.phase_metadata],- )+ write_json(output_dir / "phases.json", [phase.to_dict() for phase in self.recorder.phases])write_json(output_dir / "run_summary.json",- self._build_run_summary(- request_summary_rows=request_summary_rows,- task_summary_rows=task_summary_rows,- human_summary_zh=human_summary_zh,- ),+ {+ "run_id": self.run_id,+ "created_at": utc_now(),+ "notes": self.recorder.notes,+ "request_summary": summary_rows,+ "task_summary": task_rows,+ "adapter_comparison": adapter_rows,+ "errors": error_rows,+ "summary_zh": report,+ },)- write_text(output_dir / "summary_zh.txt", human_summary_zh)+ write_text(output_dir / "summary_zh.md", report)write_jsonl(output_dir / "request_events.jsonl", self.recorder.request_events)- write_jsonl(output_dir / "task_events.jsonl", self.recorder.task_events)- write_jsonl(output_dir / "observer_samples.jsonl", self.recorder.observer_samples)-- write_csv(- output_dir / "request_summary.csv",- request_summary_rows,- )- write_csv(- output_dir / "request_windows.csv",- build_request_window_rows(- events=self.recorder.request_events,- window_seconds=self.config.window_seconds,- ),- )- write_csv(- output_dir / "task_summary.csv",- task_summary_rows,- )-- def _build_run_summary(- self,- *,- request_summary_rows: List[Dict[str, Any]],- task_summary_rows: List[Dict[str, Any]],- human_summary_zh: str,- ) -> Dict[str, Any]:- find_delta = build_find_phase_delta(request_summary_rows)- return {- "notes": self.recorder.notes,- "phase_metadata": [asdict(item) for item in self.phase_metadata],- "request_summary": request_summary_rows,- "task_summary": task_summary_rows,- "find_phase_delta": find_delta,- "human_summary_zh": human_summary_zh,- "created_at": utc_now(),- }-- def _print_summary(self) -> None:- request_summary_rows = build_request_summary_rows(- events=self.recorder.request_events,- phase_durations=self.phase_durations,- total_run_duration=total_duration_seconds(self.phase_metadata),- )- task_summary_rows = build_task_summary_rows(self.recorder.task_events)- print(- "\n"- + render_human_summary_zh(- config=self.config,- output_dir=self.config.output_dir,- notes=self.recorder.notes,- phase_metadata=self.phase_metadata,- request_summary_rows=request_summary_rows,- request_events=self.recorder.request_events,- task_summary_rows=task_summary_rows,- task_events=self.recorder.task_events,- )- )---def parse_args(argv: Optional[List[str]] = None) -> BenchmarkConfig:- server_host = os.getenv("SERVER_HOST", "127.0.0.1")- server_port = int(os.getenv("SERVER_PORT", "1933"))- default_server_url = f"http://{server_host}:{server_port}"- default_output_dir = (- Path(__file__).resolve().parents[1]- / "results"- / "session_contention"- / datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")- )-- parser = argparse.ArgumentParser(- description="Reproduce session addMessage/extract/commit contention against concurrent find traffic.",- )- parser.add_argument("--server-url", default=default_server_url)- parser.add_argument("--api-key", default=os.getenv("OPENVIKING_API_KEY", "test-root-api-key"))- parser.add_argument("--account", default=os.getenv("OPENVIKING_ACCOUNT", "default"))- parser.add_argument("--user", default=os.getenv("OPENVIKING_USER", "default"))- parser.add_argument("--request-timeout", type=float, default=30.0)- parser.add_argument("--sessions", type=int, default=8)- parser.add_argument("--writer-concurrency", type=int, default=8)- parser.add_argument("--reader-concurrency", type=int, default=4)- parser.add_argument("--extract-concurrency", type=int, default=4)- parser.add_argument("--messages-per-commit", type=int, default=5)- parser.add_argument("--extract-ratio", type=float, default=0.5)- parser.add_argument("--message-size", type=int, default=768)- parser.add_argument("--baseline-seconds", type=float, default=30.0)- parser.add_argument("--mixed-seconds", type=float, default=120.0)- parser.add_argument("--recovery-seconds", type=float, default=30.0)- parser.add_argument("--window-seconds", type=float, default=5.0)- parser.add_argument("--observer-interval", type=float, default=5.0)- parser.add_argument("--task-poll-interval", type=float, default=1.0)- parser.add_argument("--task-drain-timeout", type=float, default=30.0)- parser.add_argument("--output-dir", default=str(default_output_dir))- parser.add_argument("--cleanup", action="store_true")- parser.add_argument("--require-extract-load", action="store_true")- parser.add_argument(- "--find-query",- action="append",- dest="find_queries",- default=[],- help="Repeat to add multiple find queries.",- )- parser.add_argument("--find-limit", type=int, default=10)- parser.add_argument("--find-target-uri", default="")- parser.add_argument("--find-score-threshold", type=float, default=None)- parser.add_argument("--seed", type=int, default=42)-- args = parser.parse_args(argv)- find_queries = args.find_queries or list(DEFAULT_FIND_QUERIES)-- config = BenchmarkConfig(- server_url=args.server_url,- api_key=args.api_key,- account=args.account,- user=args.user,- request_timeout=args.request_timeout,- session_count=max(0, args.sessions),- writer_concurrency=max(0, args.writer_concurrency),- reader_concurrency=max(0, args.reader_concurrency),- extract_concurrency=max(1, args.extract_concurrency),- messages_per_commit=max(1, args.messages_per_commit),- extract_ratio=min(max(args.extract_ratio, 0.0), 1.0),- message_size=max(128, args.message_size),- baseline_seconds=max(0.0, args.baseline_seconds),- mixed_seconds=max(0.0, args.mixed_seconds),- recovery_seconds=max(0.0, args.recovery_seconds),- window_seconds=max(args.window_seconds, 1.0),- observer_interval=0.0 if args.observer_interval <= 0 else max(args.observer_interval, 0.1),- task_poll_interval=max(args.task_poll_interval, 0.1),- task_drain_timeout=max(0.0, args.task_drain_timeout),- output_dir=args.output_dir,- cleanup=args.cleanup,- require_extract_load=args.require_extract_load,- find_queries=find_queries,- find_limit=max(1, args.find_limit),- find_target_uri=args.find_target_uri,- find_score_threshold=args.find_score_threshold,- seed=args.seed,+ write_jsonl(output_dir / "task_events.jsonl", self.recorder.task_events)+ write_csv(output_dir / "request_summary.csv", summary_rows)+ write_csv(output_dir / "request_windows.csv", window_rows)+ write_csv(output_dir / "task_summary.csv", task_rows)+ write_csv(output_dir / "adapter_comparison.csv", adapter_rows)+ write_csv(output_dir / "errors.csv", error_rows)++ def _print_summary_path(self) -> None:+ summary_path = Path(self.config.output_dir) / "summary_zh.md"+ print(f"\n压测报告: {summary_path}")+++def parse_cli_process_result(returncode: int, stdout: str, stderr: str) -> AdapterResult:+ stdout = stdout.strip()+ stderr = stderr.strip()+ parsed: Any = None+ if stdout:+ try:+ parsed = json.loads(stdout)+ except json.JSONDecodeError:+ parsed = None+ if returncode != 0:+ error_message = stderr or stdout or f"ov exited with code {returncode}"+ if parsed and isinstance(parsed, dict):+ error_message = extract_error_message(parsed) or error_message+ return AdapterResult(+ success=False,+ status_code=returncode,+ exception_type="CliExitError",+ error_message=truncate_error_message(error_message),+ result=parsed,+ raw_stdout=stdout,+ raw_stderr=stderr,+ )+ if isinstance(parsed, dict) and parsed.get("ok") is False:+ return AdapterResult(+ success=False,+ status_code=returncode,+ exception_type="CliApiError",+ error_message=truncate_error_message(extract_error_message(parsed) or stderr),+ result=parsed.get("result"),+ raw_stdout=stdout,+ raw_stderr=stderr,+ )+ if isinstance(parsed, dict) and "result" in parsed and parsed.get("ok") is True:+ return AdapterResult(+ success=True,+ status_code=returncode,+ result=parsed.get("result"),+ raw_stdout=stdout,+ raw_stderr=stderr,+ )+ return AdapterResult(+ success=True,+ status_code=returncode,+ result=parsed if parsed is not None else stdout,+ raw_stdout=stdout,+ raw_stderr=stderr,)- if config.writer_concurrency > 0 and config.session_count <= 0:- parser.error("--sessions must be > 0 when --writer-concurrency is enabled")- return config--def maybe_json(response: httpx.Response) -> Optional[Dict[str, Any]]:- try:- body = response.json()- except ValueError:- return None- return body if isinstance(body, dict) else {"value": body}+def build_cli_config_payload(config: BenchmarkConfig) -> Dict[str, Any]:+ return {+ "url": config.server_url,+ "api_key": config.api_key,+ "account": config.account,+ "user": config.user,+ "agent_id": config.agent_id,+ "timeout": config.timeout,+ "extra_headers": None,+ }-def extract_error(- body: Optional[Dict[str, Any]], status_code: Optional[int]-) -> tuple[Optional[str], Optional[str]]:- if not isinstance(body, dict):- if status_code is None:- return None, None- return None, f"http status {status_code}"- error = body.get("error")- if isinstance(error, dict):- return error.get("code"), truncate_error_message(error.get("message"))- if body.get("status") not in {None, "ok"}:- return body.get("status"), truncate_error_message(json.dumps(body, ensure_ascii=False))- if status_code is not None and status_code >= 400:- return None, f"http status {status_code}"- return None, None+def write_benchmark_document(path: Path, *, adapter: str, index: int, run_id: str) -> None:+ path.parent.mkdir(parents=True, exist_ok=True)+ content = f"""# OpenViking Load Benchmark Document {index}-def extract_session_id(body: Optional[Dict[str, Any]]) -> Optional[str]:- if not isinstance(body, dict):- return None- result = body.get("result")- if not isinstance(result, dict):- return None- session_id = result.get("session_id")- return session_id if isinstance(session_id, str) else None+Adapter: {adapter}+Run: {run_id}+Keywords: benchmark, concurrent retrieval, resource ingestion, session commit.+This generated document is used to test concurrent add_resource, find, search,+grep, glob, add_message, and commit traffic against a local OpenViking server.-def extract_task_id(body: Optional[Dict[str, Any]]) -> Optional[str]:- if not isinstance(body, dict):- return None- result = body.get("result")- if not isinstance(result, dict):- return None- task_id = result.get("task_id")- return task_id if isinstance(task_id, str) and task_id else None+The text intentionally repeats relevant terms so retrieval has stable targets:+OpenViking server benchmark benchmark benchmark. Session commit memory extraction.+Resource ingestion queue latency. Mixed SDK CLI subprocess load.+"""+ path.write_text(content, encoding="utf-8")-def build_message_content(- *, session_id: str, cycle_index: int, message_index: int, size: int-) -> str:- prefix = (- f"session={session_id} cycle={cycle_index} message={message_index}. "- "We discussed project goals, deployment constraints, user preferences, debugging notes, "- "timelines, risks, and follow-up actions. "- )- detail = (- "The user prefers production-safe changes, wants clear rollback steps, and asked for "- "memory extraction to keep decisions, entities, and events. "- "We also covered resource bottlenecks, queue backlog, response latency, and how read "- "traffic regressed during heavy write pressure. "+def build_message_content(*, session_id: str, message_index: int, adapter: str) -> str:+ return (+ f"[{adapter}] session={session_id} message={message_index}. "+ "The user discussed OpenViking server load testing, concurrent resource ingestion, "+ "retrieval latency, session commit background tasks, and mixed SDK/CLI traffic. "+ "Keep decisions, bottlenecks, follow-up actions, and observed queue behavior.")- content = prefix- while len(content) < size:- content += detail- return content[:size]-def truncate_error_message(message: Optional[str]) -> Optional[str]:- if message is None:- return None- if len(message) <= MAX_ERROR_MESSAGE_LEN:- return message- return message[:MAX_ERROR_MESSAGE_LEN] + "...[truncated]"+def extract_session_ids(payload: Any) -> List[str]:+ if isinstance(payload, dict):+ for key in ("sessions", "items", "result"):+ if key in payload:+ return extract_session_ids(payload[key])+ value = payload.get("session_id") or payload.get("id")+ return [value] if isinstance(value, str) else []+ if isinstance(payload, list):+ ids: List[str] = []+ for item in payload:+ ids.extend(extract_session_ids(item))+ return ids+ return []+++def iter_resource_tree_uris(root_uri: str) -> List[str]:+ prefix = "viking://resources"+ normalized = root_uri.rstrip("/")+ if not normalized.startswith(prefix):+ return [normalized]+ tail = normalized[len(prefix) :].strip("/")+ uris = [prefix]+ current = prefix+ for part in tail.split("/"):+ if not part:+ continue+ current = f"{current}/{part}"+ uris.append(current)+ return uris[1:]+++def to_jsonable(value: Any) -> Any:+ if hasattr(value, "to_dict") and callable(value.to_dict):+ return value.to_dict()+ if isinstance(value, dict):+ return {k: to_jsonable(v) for k, v in value.items()}+ if isinstance(value, list):+ return [to_jsonable(v) for v in value]+ if isinstance(value, tuple):+ return [to_jsonable(v) for v in value]+ return value+++def normalize_result_payload(value: Any) -> Any:+ if isinstance(value, AdapterResult):+ return normalize_result_payload(value.result)+ if isinstance(value, dict) and value.get("ok") is True and "result" in value:+ return value["result"]+ return value+++def extract_task_id_from_any(value: Any) -> Optional[str]:+ return extract_task_id(normalize_result_payload(value))+++def extract_task_id(value: Any) -> Optional[str]:+ if isinstance(value, dict):+ candidate = value.get("task_id")+ if isinstance(candidate, str) and candidate:+ return candidate+ for key in ("result", "task"):+ nested = extract_task_id(value.get(key))+ if nested:+ return nested+ return None-def utc_now() -> str:- return datetime.now(UTC).isoformat(timespec="milliseconds").replace("+00:00", "Z")+def extract_root_uri(value: Any) -> Optional[str]:+ if isinstance(value, dict):+ for key in ("root_uri", "uri", "resource_id"):+ candidate = value.get(key)+ if isinstance(candidate, str) and candidate:+ return candidate+ return None+++def extract_task_status(value: Any) -> Optional[str]:+ if not isinstance(value, dict):+ return None+ status = value.get("status")+ if isinstance(status, str):+ return status+ task = value.get("task")+ if isinstance(task, dict):+ status = task.get("status")+ if isinstance(status, str):+ return status+ return None-def total_duration_seconds(phases: List[PhaseMetadata]) -> float:- return sum(item.duration_seconds for item in phases)+def build_task_event(+ *, adapter: str, task: PendingTask, completion_scenario: str, payload: Any+) -> TaskEvent:+ created_at = to_float(payload.get("created_at")) if isinstance(payload, dict) else None+ updated_at = to_float(payload.get("updated_at")) if isinstance(payload, dict) else None+ server_duration_ms = None+ if created_at is not None and updated_at is not None:+ server_duration_ms = max(updated_at - created_at, 0.0) * 1000.0+ status = extract_task_status(payload) or "unknown"+ error_message = payload.get("error") if isinstance(payload, dict) else None+ task_result = payload.get("result") if isinstance(payload, dict) else None+ return TaskEvent(+ adapter=adapter,+ task_id=task.task_id,+ task_type=task.task_type,+ origin_scenario=task.origin_scenario,+ completion_scenario=completion_scenario,+ status=status,+ resource_id=task.resource_id,+ local_duration_ms=(time.perf_counter() - task.local_started_monotonic) * 1000.0,+ server_duration_ms=server_duration_ms,+ error_message=truncate_error_message(str(error_message)) if error_message else None,+ result=task_result,+ polled_at=utc_now(),+ )def percentile(values: Iterable[float], pct: float) -> Optional[float]:@@ -1106,95 +1421,79 @@ def percentile(values: Iterable[float], pct: float) -> Optional[float]:def build_request_summary_rows(- *,- events: List[RequestEvent],- phase_durations: Dict[str, float],- total_run_duration: float,+ *, events: List[RequestEvent], phases: List[PhaseMetadata]) -> List[Dict[str, Any]]:+ phase_durations = {+ (phase.adapter, phase.scenario): max(phase.duration_seconds, 1e-9) for phase in phases+ }+ groups: Dict[tuple[str, str, str], List[RequestEvent]] = {}+ for event in events:+ groups.setdefault((event.adapter, event.scenario, event.operation), []).append(event)rows: List[Dict[str, Any]] = []- rows.extend(- _build_request_summary_for_groups(- events=events,- grouping=lambda event: (event.phase, event.api),- duration_lookup=phase_durations,+ for (adapter, scenario, operation), group_events in sorted(groups.items()):+ duration = phase_durations.get((adapter, scenario)) or estimate_event_span_seconds(+ group_events)- )- overall_groups = _build_request_summary_for_groups(- events=events,- grouping=lambda event: ("ALL", event.api),- duration_lookup={"ALL": total_run_duration},- )- rows.extend(overall_groups)- return sorted(rows, key=lambda row: (row["phase"], row["api"]))+ rows.append(build_summary_row(adapter, scenario, operation, group_events, duration))+ overall: Dict[tuple[str, str], List[RequestEvent]] = {}+ for event in events:+ overall.setdefault((event.adapter, event.operation), []).append(event)+ for (adapter, operation), group_events in sorted(overall.items()):+ rows.append(build_summary_row(adapter, "ALL", operation, group_events, total_span(events)))+ return rows-def _build_request_summary_for_groups(- *,+def build_summary_row(+ adapter: str,+ scenario: str,+ operation: str,events: List[RequestEvent],- grouping,- duration_lookup: Dict[str, float],-) -> List[Dict[str, Any]]:- groups: Dict[tuple[str, str], List[RequestEvent]] = {}+ duration_seconds: float,+) -> Dict[str, Any]:+ latencies = [event.latency_ms for event in events]+ successes = sum(1 for event in events if event.success)+ failures = len(events) - successes+ status_counts: Dict[str, int] = {}for event in events:- key = grouping(event)- groups.setdefault(key, []).append(event)-- rows: List[Dict[str, Any]] = []- for (phase, api), api_events in groups.items():- latencies = [event.latency_ms for event in api_events]- successes = sum(1 for event in api_events if event.success)- failures = len(api_events) - successes- timeouts = sum(1 for event in api_events if event.timeout)- exceptions = sum(1 for event in api_events if event.exception_type)- status_counts: Dict[str, int] = {}- for event in api_events:- key = str(event.status_code) if event.status_code is not None else "exception"- status_counts[key] = status_counts.get(key, 0) + 1- duration = max(duration_lookup.get(phase, 0.0), 1e-9)- row = {- "phase": phase,- "api": api,- "requests": len(api_events),- "successes": successes,- "failures": failures,- "timeouts": timeouts,- "exceptions": exceptions,- "success_rate": round((successes / len(api_events)) * 100.0, 4),- "qps": round(len(api_events) / duration, 4),- "avg_ms": round(sum(latencies) / len(latencies), 4),- "p50_ms": round_optional(percentile(latencies, 50)),- "p90_ms": round_optional(percentile(latencies, 90)),- "p95_ms": round_optional(percentile(latencies, 95)),- "p99_ms": round_optional(percentile(latencies, 99)),- "max_ms": round_optional(max(latencies) if latencies else None),- "slow_gt_1s": sum(- 1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[0]- ),- "slow_gt_3s": sum(- 1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[1]- ),- "slow_gt_5s": sum(- 1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[2]- ),- "status_codes": json.dumps(status_counts, sort_keys=True),- }- rows.append(row)- return rows+ key = (+ str(event.status_code)+ if event.status_code is not None+ else (event.exception_type or "ok")+ )+ status_counts[key] = status_counts.get(key, 0) + 1+ return {+ "adapter": adapter,+ "scenario": scenario,+ "operation": operation,+ "requests": len(events),+ "successes": successes,+ "failures": failures,+ "success_rate": round((successes / len(events)) * 100.0, 4) if events else 0.0,+ "qps": round(len(events) / max(duration_seconds, 1e-9), 4),+ "avg_ms": round_optional(sum(latencies) / len(latencies) if latencies else None),+ "p50_ms": round_optional(percentile(latencies, 50)),+ "p90_ms": round_optional(percentile(latencies, 90)),+ "p95_ms": round_optional(percentile(latencies, 95)),+ "p99_ms": round_optional(percentile(latencies, 99)),+ "max_ms": round_optional(max(latencies) if latencies else None),+ "slow_gt_1s": sum(1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[0]),+ "slow_gt_3s": sum(1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[1]),+ "slow_gt_5s": sum(1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[2]),+ "status_codes": json.dumps(status_counts, sort_keys=True),+ }def build_request_window_rows(- *,- events: List[RequestEvent],- window_seconds: float,+ *, events: List[RequestEvent], window_seconds: float) -> List[Dict[str, Any]]:- groups: Dict[tuple[int, str, str], List[RequestEvent]] = {}+ groups: Dict[tuple[int, str, str, str], List[RequestEvent]] = {}for event in events:window_index = int((event.elapsed_ms_since_run_start / 1000.0) // window_seconds)- key = (window_index, event.phase, event.api)- groups.setdefault(key, []).append(event)-+ groups.setdefault(+ (window_index, event.adapter, event.scenario, event.operation), []+ ).append(event)rows: List[Dict[str, Any]] = []- for (window_index, phase, api), window_events in sorted(groups.items()):+ for (window_index, adapter, scenario, operation), window_events in sorted(groups.items()):latencies = [event.latency_ms for event in window_events]successes = sum(1 for event in window_events if event.success)rows.append(@@ -1202,13 +1501,14 @@ def build_request_window_rows("window_index": window_index,"window_start_sec": round(window_index * window_seconds, 4),"window_end_sec": round((window_index + 1) * window_seconds, 4),- "phase": phase,- "api": api,+ "adapter": adapter,+ "scenario": scenario,+ "operation": operation,"requests": len(window_events),"successes": successes,"failures": len(window_events) - successes,"success_rate": round((successes / len(window_events)) * 100.0, 4),- "qps": round(len(window_events) / window_seconds, 4),+ "qps": round(len(window_events) / max(window_seconds, 1e-9), 4),"p95_ms": round_optional(percentile(latencies, 95)),"p99_ms": round_optional(percentile(latencies, 99)),"max_ms": round_optional(max(latencies) if latencies else None),@@ -1217,441 +1517,438 @@ def build_request_window_rows(return rows-def build_task_summary_rows(events: List[CommitTaskEvent]) -> List[Dict[str, Any]]:- groups: Dict[str, List[CommitTaskEvent]] = {}+def build_task_summary_rows(events: List[TaskEvent]) -> List[Dict[str, Any]]:+ groups: Dict[tuple[str, str, str], List[TaskEvent]] = {}for event in events:- groups.setdefault(event.status, []).append(event)-+ groups.setdefault((event.adapter, event.task_type, event.status), []).append(event)rows: List[Dict[str, Any]] = []- for status, status_events in sorted(groups.items()):+ for (adapter, task_type, status), task_events in sorted(groups.items()):+ local_latencies = [event.local_duration_ms for event in task_events]server_latencies = [event.server_duration_ms- for event in status_events+ for event in task_eventsif event.server_duration_ms is not None]- local_latencies = [event.local_duration_ms for event in status_events]- successes = sum(1 for event in status_events if event.status == "completed")rows.append({+ "adapter": adapter,+ "task_type": task_type,"status": status,- "tasks": len(status_events),- "successes": successes,- "success_rate": round((successes / len(status_events)) * 100.0, 4),- "p50_server_duration_ms": round_optional(percentile(server_latencies, 50)),- "p95_server_duration_ms": round_optional(percentile(server_latencies, 95)),- "p99_server_duration_ms": round_optional(percentile(server_latencies, 99)),- "max_server_duration_ms": round_optional(+ "tasks": len(task_events),+ "p50_local_ms": round_optional(percentile(local_latencies, 50)),+ "p95_local_ms": round_optional(percentile(local_latencies, 95)),+ "p99_local_ms": round_optional(percentile(local_latencies, 99)),+ "max_local_ms": round_optional(max(local_latencies) if local_latencies else None),+ "p50_server_ms": round_optional(percentile(server_latencies, 50)),+ "p95_server_ms": round_optional(percentile(server_latencies, 95)),+ "p99_server_ms": round_optional(percentile(server_latencies, 99)),+ "max_server_ms": round_optional(max(server_latencies) if server_latencies else None),- "p50_local_duration_ms": round_optional(percentile(local_latencies, 50)),- "p95_local_duration_ms": round_optional(percentile(local_latencies, 95)),- "p99_local_duration_ms": round_optional(percentile(local_latencies, 99)),- "max_local_duration_ms": round_optional(- max(local_latencies) if local_latencies else None- ),})return rows-def build_find_phase_delta(summary_rows: List[Dict[str, Any]]) -> Optional[Dict[str, float]]:- baseline = next(- (row for row in summary_rows if row["phase"] == "baseline" and row["api"] == "find"),- None,- )- mixed = next(- (row for row in summary_rows if row["phase"] == "mixed_load" and row["api"] == "find"),- None,- )- if not baseline or not mixed:- return None- baseline_p95 = baseline.get("p95_ms")- baseline_p99 = baseline.get("p99_ms")- mixed_p95 = mixed.get("p95_ms")- mixed_p99 = mixed.get("p99_ms")- if not all(metric is not None for metric in [baseline_p95, baseline_p99, mixed_p95, mixed_p99]):- return None- return {- "baseline_p95_ms": baseline_p95,- "mixed_p95_ms": mixed_p95,- "p95_delta_percent": percent_change(baseline_p95, mixed_p95),- "baseline_p99_ms": baseline_p99,- "mixed_p99_ms": mixed_p99,- "p99_delta_percent": percent_change(baseline_p99, mixed_p99),- "baseline_success_rate": baseline["success_rate"],- "mixed_success_rate": mixed["success_rate"],- "success_rate_delta_percent": mixed["success_rate"] - baseline["success_rate"],- }---def find_request_summary_row(- summary_rows: List[Dict[str, Any]],- *,- api: str,- phase: str,-) -> Optional[Dict[str, Any]]:- return next((row for row in summary_rows if row["api"] == api and row["phase"] == phase), None)+def build_adapter_comparison_rows(summary_rows: List[Dict[str, Any]]) -> List[Dict[str, Any]]:+ rows = [+ row+ for row in summary_rows+ if row["scenario"] == "ALL"+ and row["operation"] in {"add_resource", "find", "search", "add_message", "commit_session"}+ ]+ return sorted(rows, key=lambda row: (row["operation"], row["adapter"]))-def phase_target_seconds(config: BenchmarkConfig, phase: str) -> Optional[float]:- mapping = {- "baseline": config.baseline_seconds,- "mixed_load": config.mixed_seconds,- "recovery": config.recovery_seconds,- }- return mapping.get(phase)+def build_error_rows(events: List[RequestEvent]) -> List[Dict[str, Any]]:+ groups: Dict[tuple[str, str, str, str], int] = {}+ for event in events:+ if event.success:+ continue+ key = (+ event.adapter,+ event.scenario,+ event.operation,+ event.exception_type or event.error_message or "unknown",+ )+ groups[key] = groups.get(key, 0) + 1+ return [+ {+ "adapter": adapter,+ "scenario": scenario,+ "operation": operation,+ "error": error,+ "count": count_value,+ }+ for (adapter, scenario, operation, error), count_value in sorted(+ groups.items(), key=lambda item: item[1], reverse=True+ )+ ]-def build_phase_overview_rows(+def render_report_zh(+ *,config: BenchmarkConfig,- phase_metadata: List[PhaseMetadata],-) -> List[Dict[str, Optional[float]]]:- rows: List[Dict[str, Optional[float]]] = []- for item in phase_metadata:- target = phase_target_seconds(config, item.phase)- delta = None if target is None else item.duration_seconds - target- rows.append(- {- "phase": item.phase,- "target_seconds": round_optional(target),- "actual_seconds": round_optional(item.duration_seconds),- "delta_seconds": round_optional(delta),- }+ run_id: str,+ notes: List[str],+ phases: List[PhaseMetadata],+ summary_rows: List[Dict[str, Any]],+ task_rows: List[Dict[str, Any]],+ adapter_rows: List[Dict[str, Any]],+ error_rows: List[Dict[str, Any]],+ output_dir: str,+) -> str:+ lines: List[str] = []+ lines.append("# OpenViking Server 压测报告")+ lines.append("")+ lines.append(f"- 运行 ID: `{run_id}`")+ lines.append(f"- Server: `{config.server_url}`")+ lines.append(f"- 调用路径: `{', '.join(config.adapters)}`")+ lines.append(f"- 测试数据根目录: `{config.data_root_uri}`")+ lines.append(f"- 输出目录: `{output_dir}`")+ if notes:+ lines.append("")+ lines.append("## 说明")+ for note in notes:+ lines.append(f"- {note}")+ lines.append("")+ lines.append("## 核心结论")+ total_requests = sum(row["requests"] for row in summary_rows if row["scenario"] != "ALL")+ total_failures = sum(row["failures"] for row in summary_rows if row["scenario"] != "ALL")+ success_rate = (+ (total_requests - total_failures) / total_requests * 100.0 if total_requests else 0.0+ )+ lines.append(+ f"- 本次共记录 {total_requests} 次请求,失败 {total_failures} 次,整体成功率 {success_rate:.2f}%。"+ )+ for adapter in config.adapters:+ mixed_find = find_summary(summary_rows, adapter=adapter, scenario="mixed", operation="find")+ retrieval_find = find_summary(+ summary_rows, adapter=adapter, scenario="retrieval", operation="find")- return rows+ if mixed_find and retrieval_find:+ lines.append(+ "- "+ f"`{adapter}` find p95: retrieval {fmt_ms(retrieval_find['p95_ms'])} -> "+ f"mixed {fmt_ms(mixed_find['p95_ms'])},变化 "+ f"{fmt_delta_percent(percent_change(retrieval_find['p95_ms'], mixed_find['p95_ms']))}。"+ )+ incomplete = [row for row in task_rows if row["status"] == "incomplete"]+ if incomplete:+ total_incomplete = sum(row["tasks"] for row in incomplete)+ lines.append(f"- 后台任务存在积压:{total_incomplete} 个任务在 drain 后仍未完成。")+ if error_rows:+ top = error_rows[0]+ lines.append(+ f"- Top 错误: `{top['adapter']}/{top['scenario']}/{top['operation']}` "+ f"`{top['error']}` 共 {top['count']} 次。"+ )+ lines.append("")+ lines.append("## 阶段耗时")+ lines.append(markdown_table([phase.to_dict() for phase in phases]))+ lines.append("")+ lines.append("## 请求汇总")+ lines.append(+ markdown_table(+ [+ pick(+ row,+ [+ "adapter",+ "scenario",+ "operation",+ "requests",+ "success_rate",+ "qps",+ "p50_ms",+ "p95_ms",+ "p99_ms",+ "max_ms",+ ],+ )+ for row in summary_rows+ if row["scenario"] != "ALL"+ ]+ )+ )+ lines.append("")+ lines.append("## SDK/CLI 对比")+ lines.append(+ markdown_table(+ [+ pick(+ row,+ ["adapter", "operation", "requests", "success_rate", "qps", "p95_ms", "p99_ms"],+ )+ for row in adapter_rows+ ]+ )+ )+ lines.append("")+ lines.append("## 后台任务")+ lines.append(markdown_table(task_rows) if task_rows else "无后台任务记录。")+ lines.append("")+ lines.append("## 错误 Top")+ lines.append(markdown_table(error_rows[:20]) if error_rows else "无请求错误。")+ lines.append("")+ lines.append("## 输出文件")+ lines.append("- `run_summary.json`: 汇总数据")+ lines.append("- `request_events.jsonl`: 每次请求明细")+ lines.append("- `task_events.jsonl`: 后台任务明细")+ lines.append("- `request_summary.csv`: 请求聚合")+ lines.append("- `request_windows.csv`: 时间窗口聚合")+ lines.append("- `adapter_comparison.csv`: SDK/CLI 对比")+ return "\n".join(lines) + "\n"-def build_api_error_breakdown(- events: List[RequestEvent],- *,- api: str,- phase: Optional[str] = None,-) -> Dict[str, Any]:- filtered = [- event for event in events if event.api == api and (phase is None or event.phase == phase)+def markdown_table(rows: List[Dict[str, Any]]) -> str:+ if not rows:+ return "_无数据_"+ columns = list(rows[0].keys())+ output = [+ "| " + " | ".join(columns) + " |",+ "| " + " | ".join("---" for _ in columns) + " |",]- exception_counts: Dict[str, int] = {}- error_counts: Dict[str, int] = {}- for event in filtered:- if event.exception_type:- exception_counts[event.exception_type] = (- exception_counts.get(event.exception_type, 0) + 1- )- key = event.error_code or event.exception_type- if key:- error_counts[key] = error_counts.get(key, 0) + 1- return {- "requests": len(filtered),- "successes": sum(1 for event in filtered if event.success),- "failures": sum(1 for event in filtered if not event.success),- "timeouts": sum(1 for event in filtered if event.timeout),- "exception_counts": exception_counts,- "error_counts": error_counts,- }+ for row in rows:+ output.append("| " + " | ".join(format_cell(row.get(col)) for col in columns) + " |")+ return "\n".join(output)-def format_phase_name_cn(phase: str) -> str:- mapping = {- "setup": "预热",- "baseline": "基线阶段",- "mixed_load": "混合压测阶段",- "recovery": "恢复阶段",- "drain": "收尾等待阶段",- "cleanup": "清理阶段",- "ALL": "全程",- }- return mapping.get(phase, phase)+def pick(row: Dict[str, Any], columns: List[str]) -> Dict[str, Any]:+ return {column: row.get(column) for column in columns}-def format_seconds(value: Optional[float]) -> str:+def format_cell(value: Any) -> str:if value is None:return "n/a"- return f"{value:.1f}s"+ text = str(value)+ return text.replace("|", "\\|").replace("\n", " ")-def format_percent(value: Optional[float]) -> str:- if value is None:- return "n/a"- return f"{value:.2f}%"+def find_summary(+ rows: List[Dict[str, Any]], *, adapter: str, scenario: str, operation: str+) -> Optional[Dict[str, Any]]:+ return next(+ (+ row+ for row in rows+ if row["adapter"] == adapter+ and row["scenario"] == scenario+ and row["operation"] == operation+ ),+ None,+ )-def format_delta_percent(value: Optional[float]) -> str:- if value is None:- return "n/a"- sign = "+" if value >= 0 else ""- return f"{sign}{value:.2f}%"+def estimate_event_span_seconds(events: List[RequestEvent]) -> float:+ if len(events) < 2:+ return 1e-9+ starts = [event.elapsed_ms_since_run_start for event in events]+ ends = [event.elapsed_ms_since_run_start + event.latency_ms for event in events]+ return max((max(ends) - min(starts)) / 1000.0, 1e-9)-def format_delta_seconds(value: Optional[float]) -> str:- if value is None:- return "n/a"- sign = "+" if value >= 0 else ""- return f"{sign}{value:.1f}s"+def total_span(events: List[RequestEvent]) -> float:+ if not events:+ return 1e-9+ starts = [event.elapsed_ms_since_run_start for event in events]+ ends = [event.elapsed_ms_since_run_start + event.latency_ms for event in events]+ return max((max(ends) - min(starts)) / 1000.0, 1e-9)-def format_change(old: Optional[float], new: Optional[float], *, unit: str = "ms") -> str:- if old is None or new is None:- return "n/a"- if unit == "ms":- return (- f"{old:.2f}{unit} -> {new:.2f}{unit} ({format_delta_percent(percent_change(old, new))})"- )- return f"{old:.2f} -> {new:.2f} ({format_delta_percent(percent_change(old, new))})"+def percent_change(old: Optional[float], new: Optional[float]) -> Optional[float]:+ if old is None or new is None or old == 0:+ return None+ return ((new - old) / old) * 100.0-def format_qps_change(old: Optional[float], new: Optional[float]) -> str:- if old is None or new is None:+def fmt_delta_percent(value: Optional[float]) -> str:+ if value is None:return "n/a"- return f"{old:.2f} -> {new:.2f} ({format_delta_percent(percent_change(old, new))})"+ sign = "+" if value >= 0 else ""+ return f"{sign}{value:.2f}%"-def render_human_summary_zh(- *,- config: BenchmarkConfig,- output_dir: str,- notes: List[str],- phase_metadata: List[PhaseMetadata],- request_summary_rows: List[Dict[str, Any]],- request_events: List[RequestEvent],- task_summary_rows: List[Dict[str, Any]],- task_events: List[CommitTaskEvent],-) -> str:- lines: List[str] = []- lines.append("=== OpenViking Session 竞争压测摘要 ===")- lines.append(f"结果目录: {output_dir}")+def fmt_ms(value: Optional[float]) -> str:+ return "n/a" if value is None else f"{value:.2f}ms"- if notes:- lines.append("")- lines.append("说明:")- for note in notes:- lines.append(f"- {note}")- phase_rows = build_phase_overview_rows(config, phase_metadata)- baseline_find = find_request_summary_row(request_summary_rows, api="find", phase="baseline")- mixed_find = find_request_summary_row(request_summary_rows, api="find", phase="mixed_load")- recovery_find = find_request_summary_row(request_summary_rows, api="find", phase="recovery")- mixed_add = find_request_summary_row(- request_summary_rows, api="add_message", phase="mixed_load"- )- mixed_commit = find_request_summary_row(request_summary_rows, api="commit", phase="mixed_load")- mixed_extract = find_request_summary_row(- request_summary_rows, api="extract", phase="mixed_load"- )- baseline_status = find_request_summary_row(- request_summary_rows, api="system_status", phase="baseline"- )- mixed_status = find_request_summary_row(- request_summary_rows, api="system_status", phase="mixed_load"- )- baseline_queue = find_request_summary_row(- request_summary_rows, api="observer_queue", phase="baseline"- )- mixed_queue = find_request_summary_row(- request_summary_rows, api="observer_queue", phase="mixed_load"- )- find_delta = build_find_phase_delta(request_summary_rows)- extract_breakdown = build_api_error_breakdown(request_events, api="extract", phase="mixed_load")- completed_tasks = next(- (row for row in task_summary_rows if row["status"] == "completed"),- None,- )- incomplete_tasks = next(- (row for row in task_summary_rows if row["status"] == "incomplete"),- None,- )- total_task_count = len(task_events)+def round_optional(value: Optional[float], digits: int = 4) -> Optional[float]:+ if value is None:+ return None+ return round(float(value), digits)- lines.append("")- lines.append("一、核心结论")- if baseline_find and mixed_find and find_delta:- lines.append(- "- 已明确复现读接口退化:`find` 在混合压测阶段的 p95 从 "- f"{baseline_find['p95_ms']:.2f}ms 升到 {mixed_find['p95_ms']:.2f}ms,"- f"增幅 {find_delta['p95_delta_percent']:.2f}%;p99 从 "- f"{baseline_find['p99_ms']:.2f}ms 升到 {mixed_find['p99_ms']:.2f}ms,"- f"增幅 {find_delta['p99_delta_percent']:.2f}%。"- )- lines.append(- "- `find` 吞吐也下降了:QPS 从 "- f"{baseline_find['qps']:.2f} 降到 {mixed_find['qps']:.2f},"- f"变化 {format_delta_percent(percent_change(baseline_find['qps'], mixed_find['qps']))}。"- )- if recovery_find and baseline_find and mixed_find:- lines.append(- "- 恢复阶段没有完全回到基线:`find` p95 为 "- f"{recovery_find['p95_ms']:.2f}ms,仍高于基线 "- f"{format_delta_percent(percent_change(baseline_find['p95_ms'], recovery_find['p95_ms']))};"- "但相比混合压测阶段已经有明显回落。"- )- if mixed_extract:- lines.append(- "- 长尾压力主要来自 `extract`:混合压测阶段共 "- f"{mixed_extract['requests']} 次调用,成功率 {mixed_extract['success_rate']:.2f}%,"- f"p95 {mixed_extract['p95_ms']:.2f}ms。"- )- if extract_breakdown["timeouts"] > 0:- lines.append(- "- `extract` 失败几乎全是客户端超时:"- f"{extract_breakdown['timeouts']}/{extract_breakdown['requests']} 次超时,"- f"主异常是 {format_top_counts(extract_breakdown['exception_counts'])}。"- )- if mixed_commit and completed_tasks:- lines.append(- "- `commit` 接口本身不是最重的部分:前台 `commit` p95 只有 "- f"{mixed_commit['p95_ms']:.2f}ms;真正重的是后台任务,已完成任务的后台 p95 达 "- f"{completed_tasks['p95_server_duration_ms']:.2f}ms。"- )- if incomplete_tasks:- lines.append(- "- 后台积压明显:本次共跟踪到 "- f"{total_task_count} 个 `commit` 背景任务,其中 {incomplete_tasks['tasks']} 个在压测结束"- "并等待 drain 后仍未完成。"- )- lines.append("")- lines.append("二、阶段时长")- for row in phase_rows:- extra = ""- if row["delta_seconds"] is not None and row["delta_seconds"] > 1:- extra = ",实际时长明显长于目标值,通常说明脚本在等待 in-flight 会话周期收尾"- lines.append(- f"- {format_phase_name_cn(row['phase'])}: 目标 {format_seconds(row['target_seconds'])},"- f"实际 {format_seconds(row['actual_seconds'])},偏差 {format_delta_seconds(row['delta_seconds'])}{extra}"- )+def to_float(value: Any) -> Optional[float]:+ try:+ if value is None:+ return None+ return float(value)+ except (TypeError, ValueError):+ return None- lines.append("")- lines.append("三、关键指标对比")- if baseline_find and mixed_find and recovery_find:- lines.append(- "- `find`:"- f" 基线 p95={baseline_find['p95_ms']:.2f}ms / p99={baseline_find['p99_ms']:.2f}ms / qps={baseline_find['qps']:.2f};"- f" 压测中 p95={mixed_find['p95_ms']:.2f}ms / p99={mixed_find['p99_ms']:.2f}ms / qps={mixed_find['qps']:.2f};"- f" 恢复期 p95={recovery_find['p95_ms']:.2f}ms / p99={recovery_find['p99_ms']:.2f}ms / qps={recovery_find['qps']:.2f}。"- )- if mixed_add:- lines.append(- "- `add_message`: 混合压测阶段 "- f"requests={mixed_add['requests']},p50={mixed_add['p50_ms']:.2f}ms,"- f"p95={mixed_add['p95_ms']:.2f}ms,p99={mixed_add['p99_ms']:.2f}ms。"- )- if mixed_commit:- lines.append(- "- `commit`: 混合压测阶段 "- f"requests={mixed_commit['requests']},p50={mixed_commit['p50_ms']:.2f}ms,"- f"p95={mixed_commit['p95_ms']:.2f}ms,p99={mixed_commit['p99_ms']:.2f}ms。"- )- if mixed_extract:- lines.append(- "- `extract`: 混合压测阶段 "- f"requests={mixed_extract['requests']},success_rate={mixed_extract['success_rate']:.2f}%,"- f"timeouts={extract_breakdown['timeouts']},p95={mixed_extract['p95_ms']:.2f}ms。"- )- if completed_tasks:- lines.append(- "- `commit` 背景任务(completed):"- f" tasks={completed_tasks['tasks']},p50={format_metric(completed_tasks['p50_server_duration_ms'])},"- f" p95={format_metric(completed_tasks['p95_server_duration_ms'])},"- f" p99={format_metric(completed_tasks['p99_server_duration_ms'])}。"- )- if incomplete_tasks:- lines.append(- "- `commit` 背景任务(incomplete):"- f" tasks={incomplete_tasks['tasks']},本地等待 p95={format_metric(incomplete_tasks['p95_local_duration_ms'])}。"- )- if baseline_status and mixed_status:- lines.append(- "- `system_status`: p95 "- f"{format_change(baseline_status['p95_ms'], mixed_status['p95_ms'])}。"- )- if baseline_queue and mixed_queue:- lines.append(- "- `observer_queue`: p95 "- f"{format_change(baseline_queue['p95_ms'], mixed_queue['p95_ms'])}。"- )- lines.append("")- lines.append("四、怎么理解这次结果")- lines.append(- "- `find` 没有报错,但延迟和吞吐同时变差,这比“报错”更说明问题:读请求被明显挤压了。"- )- lines.append("- `extract` 的大量 30 秒超时说明长尾请求已经被稳定制造出来了,压测目标基本达成。")- lines.append(- "- `commit` 前台接口看起来还好,但后台任务非常慢,说明资源竞争更可能发生在后续提取/索引阶段,而不是 HTTP 返回这一步。"- )- lines.append(- "- 如果你要拿这次结果给别人看,最应该盯的是三组数字:"- "`find` 基线 vs 压测 p95/p99、`extract` 超时比例、`commit` 背景任务完成时长。"- )+def truncate_error_message(message: Optional[str]) -> Optional[str]:+ if message is None:+ return None+ if len(message) <= MAX_ERROR_MESSAGE_LEN:+ return message+ return message[:MAX_ERROR_MESSAGE_LEN] + "...[truncated]"- return "\n".join(lines)+def extract_error_message(value: Any) -> Optional[str]:+ if not isinstance(value, dict):+ return None+ error = value.get("error")+ if isinstance(error, dict):+ message = error.get("message") or error.get("code")+ return str(message) if message is not None else None+ if error:+ return str(error)+ return None-def format_top_counts(counts: Dict[str, int], limit: int = 3) -> str:- if not counts:- return "无"- ordered = sorted(counts.items(), key=lambda item: (-item[1], item[0]))- return ", ".join(f"{key}={value}" for key, value in ordered[:limit])+def is_not_found_error(exc: Exception) -> bool:+ text = str(exc).lower()+ return "not_found" in text or "not found" in text-def percent_change(old: float, new: float) -> float:- if old == 0:- return 0.0 if new == 0 else 100.0- return ((new - old) / old) * 100.0+def is_already_exists_error(exc: Exception) -> bool:+ text = str(exc).lower()+ return "already_exists" in text or "already exists" in text or "exist" in text-def round_optional(value: Optional[float], ndigits: int = 4) -> Optional[float]:- if value is None:- return None- return round(value, ndigits)+def utc_now() -> str:+ return datetime.now(UTC).isoformat(timespec="milliseconds").replace("+00:00", "Z")-def write_json(path: Path, data: Any) -> None:- with path.open("w", encoding="utf-8") as handle:- json.dump(data, handle, indent=2, ensure_ascii=False)+def write_json(path: Path, value: Any) -> None:+ path.write_text(json.dumps(value, ensure_ascii=False, indent=2), encoding="utf-8")-def write_text(path: Path, content: str) -> None:- path.write_text(content, encoding="utf-8")++def write_text(path: Path, value: str) -> None:+ path.write_text(value, encoding="utf-8")-def write_jsonl(path: Path, rows: Iterable[Any]) -> None:+def write_jsonl(path: Path, values: Iterable[Any]) -> None:with path.open("w", encoding="utf-8") as handle:- for row in rows:- if hasattr(row, "to_dict"):- row = row.to_dict()- handle.write(json.dumps(row, ensure_ascii=False) + "\n")+ for value in values:+ payload = value.to_dict() if hasattr(value, "to_dict") else value+ handle.write(json.dumps(payload, ensure_ascii=False) + "\n")def write_csv(path: Path, rows: List[Dict[str, Any]]) -> None:if not rows:path.write_text("", encoding="utf-8")return- fieldnames = list(rows[0].keys())with path.open("w", encoding="utf-8", newline="") as handle:- writer = csv.DictWriter(handle, fieldnames=fieldnames)+ writer = csv.DictWriter(handle, fieldnames=list(rows[0].keys()))writer.writeheader()writer.writerows(rows)-def extract_boolean(body: Optional[Dict[str, Any]], *keys: str) -> Optional[bool]:- current: Any = body- for key in keys:- if not isinstance(current, dict):- return None- current = current.get(key)- return current if isinstance(current, bool) else None+def parse_args(argv: Optional[List[str]] = None) -> BenchmarkConfig:+ parser = argparse.ArgumentParser(+ description="Run SDK/CLI mixed load benchmark against a running OpenViking server."+ )+ parser.add_argument(+ "--server-url",+ default=os.getenv("OPENVIKING_SERVER_URL", "http://127.0.0.1:1935"),+ )+ parser.add_argument("--api-key", default=os.getenv("OPENVIKING_API_KEY", "test-root-api-key"))+ parser.add_argument("--account", default=os.getenv("OPENVIKING_ACCOUNT", "default"))+ parser.add_argument("--user", default=os.getenv("OPENVIKING_USER", "default"))+ parser.add_argument("--agent-id", default=os.getenv("OPENVIKING_AGENT_ID"))+ parser.add_argument("--timeout", type=float, default=60.0)+ parser.add_argument(+ "--adapters",+ default="sdk,cli-http,cli-subprocess",+ help="Comma separated: sdk,cli-http,cli-subprocess",+ )+ parser.add_argument("--profile", choices=sorted(PROFILE_DEFAULTS), default="standard")+ parser.add_argument("--resource-count", type=int)+ parser.add_argument("--session-count", type=int)+ parser.add_argument("--messages-per-session", type=int)+ parser.add_argument("--resource-concurrency", type=int)+ parser.add_argument("--search-concurrency", type=int)+ parser.add_argument("--session-concurrency", type=int)+ parser.add_argument("--commit-concurrency", type=int)+ parser.add_argument("--mixed-concurrency", type=int)+ parser.add_argument("--phase-seconds", type=float)+ parser.add_argument("--mixed-seconds", type=float)+ parser.add_argument("--warmup-seconds", type=float)+ parser.add_argument("--drain-timeout", type=float, default=60.0)+ parser.add_argument("--task-poll-interval", type=float, default=1.0)+ parser.add_argument("--request-window-seconds", type=float, default=5.0)+ parser.add_argument("--data-root-uri", default=DEFAULT_DATA_ROOT_URI)+ default_results_root = (+ Path(__file__).resolve().parents[1] / "results" / "openviking_server_load"+ )+ parser.add_argument("--local-data-dir", default=str(default_results_root / "data"))+ parser.add_argument(+ "--output-dir",+ default=str(default_results_root / datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")),+ )+ parser.add_argument("--clear-before-run", action=argparse.BooleanOptionalAction, default=True)+ parser.add_argument("--cleanup-at-end", action="store_true")+ parser.add_argument("--ov-bin", default=os.getenv("OV_BIN", "ov"))+ parser.add_argument("--seed", type=int, default=42)+ parser.add_argument("--find-limit", type=int, default=10)+ args = parser.parse_args(argv)+ profile = PROFILE_DEFAULTS[args.profile]-def to_float(value: Any) -> Optional[float]:- if isinstance(value, (float, int)):- return float(value)- return None+ def get_int(name: str) -> int:+ return max(+ 0, int(getattr(args, name) if getattr(args, name) is not None else profile[name])+ )+ def get_float(name: str) -> float:+ return max(+ 0.0, float(getattr(args, name) if getattr(args, name) is not None else profile[name])+ )-def format_metric(value: Optional[float]) -> str:- if value is None:- return "n/a"- return f"{value:.2f}ms"+ adapters = [item.strip() for item in args.adapters.split(",") if item.strip()]+ allowed = {"sdk", "cli-http", "cli-subprocess"}+ invalid = sorted(set(adapters) - allowed)+ if invalid:+ parser.error(f"Unsupported adapters: {', '.join(invalid)}")+ if not adapters:+ parser.error("--adapters must not be empty")++ return BenchmarkConfig(+ server_url=args.server_url.rstrip("/"),+ api_key=args.api_key or None,+ account=args.account or None,+ user=args.user or None,+ agent_id=args.agent_id or None,+ timeout=max(0.1, args.timeout),+ adapters=adapters,+ profile=args.profile,+ resource_count=get_int("resource_count"),+ session_count=max(1, get_int("session_count")),+ messages_per_session=max(1, get_int("messages_per_session")),+ resource_concurrency=max(1, get_int("resource_concurrency")),+ search_concurrency=max(1, get_int("search_concurrency")),+ session_concurrency=max(1, get_int("session_concurrency")),+ commit_concurrency=max(1, get_int("commit_concurrency")),+ mixed_concurrency=max(1, get_int("mixed_concurrency")),+ phase_seconds=get_float("phase_seconds"),+ mixed_seconds=get_float("mixed_seconds"),+ warmup_seconds=get_float("warmup_seconds"),+ drain_timeout=max(0.0, args.drain_timeout),+ task_poll_interval=max(0.1, args.task_poll_interval),+ request_window_seconds=max(1.0, args.request_window_seconds),+ local_data_dir=args.local_data_dir,+ output_dir=args.output_dir,+ data_root_uri=args.data_root_uri.rstrip("/"),+ session_prefix=DEFAULT_SESSION_PREFIX,+ clear_before_run=bool(args.clear_before_run),+ cleanup_at_end=bool(args.cleanup_at_end),+ ov_bin=args.ov_bin,+ seed=args.seed,+ find_limit=max(1, args.find_limit),+ )async def async_main(argv: Optional[List[str]] = None) -> int:@@ -1661,11 +1958,7 @@ async def async_main(argv: Optional[List[str]] = None) -> int:def main(argv: Optional[List[str]] = None) -> int:- try:- return asyncio.run(async_main(argv))- except KeyboardInterrupt:- print("\n[stopped] benchmark interrupted by user", file=sys.stderr)- return 130+ return asyncio.run(async_main(argv))if __name__ == "__main__":