Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(benchmark): update session load benchmark (#2031)

Qin Haojiee52d3446a0e9
diff --git a/benchmark/custom/README.md b/benchmark/custom/README.md
new file mode 100644
index 000000000..a1da5e826
--- /dev/null
+++ b/benchmark/custom/README.md
@@ -0,0 +1,264 @@
+# OpenViking Server 压测脚本使用指南
+
+本目录包含面向 OpenViking 本地 Server 的自定义压测脚本。当前主要脚本是:
+
+- `session_contention_benchmark.py`:通用 Server 压测框架,覆盖 SDK、CLI HTTP 封装和真实 `ov` 子进程三种调用路径。
+
+## 压测目标
+
+`session_contention_benchmark.py` 用来请求已经启动的 OpenViking Server,验证多类接口在并发和混合负载下的吞吐、延迟、失败率和后台任务积压情况。
+
+脚本不会启动或停止 Server,只负责:
+
+1. 生成本地 Markdown 测试文档。
+2. 每次运行前默认清空上一次压测写入的数据目录。
+3. 并发请求资源添加、检索、session 写入、session commit、任务轮询和观测接口。
+4. 输出中文压测报告和机器可读明细文件。
+
+## 前置条件
+
+先在另一个终端启动 OpenViking Server:
+
+```bash
+openviking-server
+```
+
+如果 Server 启用了 API Key 或多租户,请准备好以下信息:
+
+- Server 地址,例如 `http://127.0.0.1:1935`
+- API Key,默认参数会使用 `test-root-api-key`
+- Account,默认 `default`
+- User,默认 `default`
+
+真实 CLI 子进程模式还要求当前环境能执行 `ov`:
+
+```bash
+ov health
+```
+
+## 快速开始
+
+在仓库根目录运行 smoke 压测:
+
+```bash
+.venv/bin/python benchmark/custom/session_contention_benchmark.py \
+ --server-url http://127.0.0.1:1935 \
+ --profile smoke
+```
+
+只测试 Python SDK 路径:
+
+```bash
+.venv/bin/python benchmark/custom/session_contention_benchmark.py \
+ --server-url http://127.0.0.1:1935 \
+ --profile smoke \
+ --adapters sdk
+```
+
+同时测试 SDK、CLI HTTP 封装和真实 `ov` 子进程:
+
+```bash
+.venv/bin/python benchmark/custom/session_contention_benchmark.py \
+ --server-url http://127.0.0.1:1935 \
+ --profile standard \
+ --adapters sdk,cli-http,cli-subprocess
+```
+
+## 调用路径说明
+
+脚本支持三种 adapter:
+
+| Adapter | 含义 | 适用场景 |
+| --- | --- | --- |
+| `sdk` | 通过 `openviking.AsyncHTTPClient` 请求 Server | 评估 Python SDK 的 HTTP 调用表现 |
+| `cli-http` | 直接使用 `openviking_cli.client.http.AsyncHTTPClient` | 评估 CLI 共用 HTTP client 层表现 |
+| `cli-subprocess` | 每次请求真实执行一次 `ov ... --output json` | 评估真实 CLI 进程启动、配置读取、上传和输出解析成本 |
+
+默认会同时运行三种路径:
+
+```bash
+--adapters sdk,cli-http,cli-subprocess
+```
+
+如果只关心 Server 吞吐,建议先跑 `sdk` 或 `cli-http`。如果关心用户实际执行 CLI 命令的端到端成本,再加入 `cli-subprocess`。
+
+## 压测场景
+
+每个 adapter 会依次执行以下阶段:
+
+| 阶段 | 内容 |
+| --- | --- |
+| `warmup` | 健康检查预热 |
+| `add_resources` | 并发添加多个生成的 Markdown 文档 |
+| `session_messages` | 并发向不同 session 写入多轮 user / assistant 消息 |
+| `retrieval` | 并发执行 `find`、`search`、`grep`、`glob` |
+| `session_commit` | 并发 commit 不同 session,并轮询后台任务 |
+| `mixed` | 混合资源添加、检索、session 写入、commit、观测接口和任务轮询 |
+
+这些阶段不是只压单个接口,目的是观察 OpenViking 在真实组合负载下的退化情况。
+
+## 数据清理策略
+
+默认每次运行前会清理:
+
+- Server 侧资源目录:`viking://resources/bench/load_test`
+- 旧 session:所有 `bench-load-` 前缀的 session
+- 本地生成数据目录:`benchmark/results/openviking_server_load/data`
+
+默认运行结束后保留本次写入的数据,方便人工复查。下一次运行前会再次清空。
+
+如果不想在运行前清理:
+
+```bash
+--no-clear-before-run
+```
+
+如果希望运行结束后也清理:
+
+```bash
+--cleanup-at-end
+```
+
+## 常用参数
+
+| 参数 | 默认值 | 说明 |
+| --- | --- | --- |
+| `--server-url` | `http://127.0.0.1:1935` | OpenViking Server 地址 |
+| `--api-key` | `test-root-api-key` | 请求使用的 API Key |
+| `--account` | `default` | 请求使用的 account |
+| `--user` | `default` | 请求使用的 user |
+| `--adapters` | `sdk,cli-http,cli-subprocess` | 要测试的调用路径 |
+| `--profile` | `standard` | 压测规模:`smoke`、`standard`、`stress` |
+| `--resource-count` | 跟随 profile | 每个 adapter 添加的初始文档数 |
+| `--session-count` | 跟随 profile | 每个 adapter 使用的 session 数 |
+| `--phase-seconds` | 跟随 profile | 单类持续压测阶段时长 |
+| `--mixed-seconds` | 跟随 profile | 混合压测阶段时长 |
+| `--drain-timeout` | `60` | 等待后台任务完成的最大秒数 |
+| `--data-root-uri` | `viking://resources/bench/load_test` | Server 侧压测资源根目录 |
+| `--output-dir` | 自动生成 | 报告输出目录 |
+| `--ov-bin` | `ov` | 真实 CLI 子进程使用的可执行文件 |
+
+## Profile 说明
+
+| Profile | 用途 | 特点 |
+| --- | --- | --- |
+| `smoke` | 快速验证脚本、配置和 Server 可用性 | 时间短、并发低、数据少 |
+| `standard` | 常规压测 | 默认推荐配置 |
+| `stress` | 高压力压测 | 并发和数据量更高,耗时更长 |
+
+建议先跑 `smoke`,确认报告正常生成后再跑 `standard` 或 `stress`。
+
+## 报告输出
+
+默认输出目录类似:
+
+```text
+benchmark/results/openviking_server_load/20260511T120000Z/
+```
+
+主要文件:
+
+| 文件 | 内容 |
+| --- | --- |
+| `summary_zh.md` | 中文压测报告,优先阅读 |
+| `run_summary.json` | 汇总结果,便于自动分析 |
+| `request_events.jsonl` | 每次请求的明细事件 |
+| `task_events.jsonl` | 后台任务完成和积压明细 |
+| `request_summary.csv` | 按 adapter / 阶段 / 接口聚合的 QPS、成功率、延迟 |
+| `request_windows.csv` | 按时间窗口聚合的请求表现 |
+| `adapter_comparison.csv` | SDK / CLI 路径对比 |
+| `errors.csv` | 错误 Top 明细 |
+
+报告会重点展示:
+
+- 总请求量、失败数和成功率。
+- 各接口 p50 / p95 / p99 / max 延迟。
+- `retrieval` 阶段到 `mixed` 阶段的检索延迟变化。
+- SDK、CLI HTTP 封装、真实 CLI 子进程之间的差异。
+- commit / add_resource 后台任务是否积压。
+- Top 错误类型和发生位置。
+
+## 示例:指定认证信息
+
+```bash
+.venv/bin/python benchmark/custom/session_contention_benchmark.py \
+ --server-url http://127.0.0.1:1935 \
+ --api-key your-root-api-key \
+ --account default \
+ --user default \
+ --profile standard
+```
+
+也可以通过环境变量传入:
+
+```bash
+export OPENVIKING_SERVER_URL=http://127.0.0.1:1935
+export OPENVIKING_API_KEY=your-root-api-key
+export OPENVIKING_ACCOUNT=default
+export OPENVIKING_USER=default
+
+.venv/bin/python benchmark/custom/session_contention_benchmark.py --profile smoke
+```
+
+## 示例:降低真实 CLI 子进程开销
+
+真实 `cli-subprocess` 会为每个请求启动一次 `ov` 进程,开销明显高于 SDK 路径。若只想压 Server 本身,可以先排除它:
+
+```bash
+.venv/bin/python benchmark/custom/session_contention_benchmark.py \
+ --profile standard \
+ --adapters sdk,cli-http
+```
+
+若必须测试真实 CLI,但本地 `ov` 不在 PATH 中,可以指定可执行文件:
+
+```bash
+.venv/bin/python benchmark/custom/session_contention_benchmark.py \
+ --profile smoke \
+ --adapters cli-subprocess \
+ --ov-bin .venv/bin/ov
+```
+
+## 常见问题
+
+### 1. Server 连接失败
+
+先确认 Server 已启动:
+
+```bash
+curl http://127.0.0.1:1935/health
+```
+
+如果端口不同,请设置 `--server-url`。
+
+### 2. 认证失败
+
+确认 `--api-key`、`--account`、`--user` 与当前 Server 配置一致。多租户模式下,root key 请求通常还需要 account 和 user。
+
+### 3. `cli-subprocess` 失败
+
+确认 `ov` 可执行:
+
+```bash
+ov health
+```
+
+如果不可用,使用 `--ov-bin .venv/bin/ov` 或只运行 `sdk,cli-http`。
+
+### 4. 后台任务没有在 drain 内完成
+
+这通常说明资源处理或 session commit 的后台任务积压。可以:
+
+- 增大 `--drain-timeout`
+- 降低并发参数
+- 查看 `task_events.jsonl`
+- 查看 Server 日志和 observer queue 状态
+
+### 5. 多次运行结果差异较大
+
+压测会受本机 CPU、磁盘、模型配置、队列积压和真实 CLI 进程启动开销影响。建议:
+
+1. 先跑 `smoke` 验证环境。
+2. 连续跑多次 `standard`。
+3. 对比 `adapter_comparison.csv` 和 `request_windows.csv`。
+4. 避免在已有大量后台任务未完成时启动下一轮。
diff --git a/benchmark/custom/session_contention_benchmark.py b/benchmark/custom/session_contention_benchmark.py
index c351952ae..fa48f88e5 100644
--- a/benchmark/custom/session_contention_benchmark.py
+++ b/benchmark/custom/session_contention_benchmark.py
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
-"""Daily session mixed-load contention benchmark for OpenViking."""
+"""OpenViking server mixed-load benchmark."""
from __future__ import annotations
@@ -10,103 +10,157 @@ import json
import math
import os
import random
+import shutil
import sys
import time
from dataclasses import asdict, dataclass, field
from datetime import UTC, datetime
+from itertools import count
from pathlib import Path
-from typing import Any, Dict, Iterable, List, Optional
+from typing import Any, Awaitable, Callable, Dict, Iterable, List, Optional, Sequence
-import httpx
-
-DEFAULT_FIND_QUERIES = [
- "how to authenticate users",
- "what is OpenViking",
+DEFAULT_DATA_ROOT_URI = "viking://resources/bench/load_test"
+DEFAULT_SESSION_PREFIX = "bench-load-"
+DEFAULT_QUERIES = [
+ "OpenViking server load test",
"session commit memory extraction",
+ "resource ingestion benchmark document",
+ "concurrent retrieval latency",
]
DEFAULT_SLOW_THRESHOLDS_MS = (1000, 3000, 5000)
-MAX_ERROR_MESSAGE_LEN = 500
+MAX_ERROR_MESSAGE_LEN = 800
+
+PROFILE_DEFAULTS: Dict[str, Dict[str, Any]] = {
+ "smoke": {
+ "resource_count": 6,
+ "session_count": 6,
+ "messages_per_session": 3,
+ "resource_concurrency": 2,
+ "search_concurrency": 2,
+ "session_concurrency": 2,
+ "commit_concurrency": 2,
+ "mixed_concurrency": 4,
+ "phase_seconds": 8.0,
+ "mixed_seconds": 12.0,
+ "warmup_seconds": 1.0,
+ },
+ "standard": {
+ "resource_count": 24,
+ "session_count": 24,
+ "messages_per_session": 6,
+ "resource_concurrency": 6,
+ "search_concurrency": 8,
+ "session_concurrency": 8,
+ "commit_concurrency": 6,
+ "mixed_concurrency": 16,
+ "phase_seconds": 30.0,
+ "mixed_seconds": 60.0,
+ "warmup_seconds": 3.0,
+ },
+ "stress": {
+ "resource_count": 80,
+ "session_count": 80,
+ "messages_per_session": 8,
+ "resource_concurrency": 16,
+ "search_concurrency": 24,
+ "session_concurrency": 24,
+ "commit_concurrency": 16,
+ "mixed_concurrency": 48,
+ "phase_seconds": 60.0,
+ "mixed_seconds": 180.0,
+ "warmup_seconds": 5.0,
+ },
+}
@dataclass
class BenchmarkConfig:
server_url: str
- api_key: str
- account: str
- user: str
- request_timeout: float
+ api_key: Optional[str]
+ account: Optional[str]
+ user: Optional[str]
+ agent_id: Optional[str]
+ timeout: float
+ adapters: List[str]
+ profile: str
+ resource_count: int
session_count: int
- writer_concurrency: int
- reader_concurrency: int
- extract_concurrency: int
- messages_per_commit: int
- extract_ratio: float
- message_size: int
- baseline_seconds: float
+ messages_per_session: int
+ resource_concurrency: int
+ search_concurrency: int
+ session_concurrency: int
+ commit_concurrency: int
+ mixed_concurrency: int
+ phase_seconds: float
mixed_seconds: float
- recovery_seconds: float
- window_seconds: float
- observer_interval: float
+ warmup_seconds: float
+ drain_timeout: float
task_poll_interval: float
- task_drain_timeout: float
+ request_window_seconds: float
+ local_data_dir: str
output_dir: str
- cleanup: bool
- require_extract_load: bool
- find_queries: List[str]
- find_limit: int
- find_target_uri: str
- find_score_threshold: Optional[float]
+ data_root_uri: str
+ session_prefix: str
+ clear_before_run: bool
+ cleanup_at_end: bool
+ ov_bin: str
seed: int
+ find_limit: int
+
+
+def direct_http_agent_id(config: BenchmarkConfig) -> str:
+ """Avoid loading the user's global ovcli.conf when agent_id is unset."""
+ return config.agent_id or ""
@dataclass
-class PhaseMetadata:
- phase: str
- started_at: str
- ended_at: str
- duration_seconds: float
+class AdapterResult:
+ success: bool
+ result: Any = None
+ status_code: Optional[int] = None
+ exception_type: Optional[str] = None
+ error_message: Optional[str] = None
+ raw_stdout: Optional[str] = None
+ raw_stderr: Optional[str] = None
+ command: Optional[List[str]] = None
@dataclass
class RequestEvent:
- api: str
- method: str
- path: str
- phase: str
+ adapter: str
+ scenario: str
+ operation: str
started_at: str
ended_at: str
elapsed_ms_since_run_start: float
latency_ms: float
success: bool
status_code: Optional[int]
- timeout: bool
exception_type: Optional[str]
- error_code: Optional[str]
error_message: Optional[str]
session_id: Optional[str] = None
- cycle_index: Optional[int] = None
- worker_id: Optional[int] = None
+ resource_uri: Optional[str] = None
task_id: Optional[str] = None
+ worker_id: Optional[int] = None
+ command: Optional[str] = None
def to_dict(self) -> Dict[str, Any]:
return asdict(self)
@dataclass
-class CommitTaskEvent:
+class TaskEvent:
+ adapter: str
task_id: str
- session_id: str
- origin_phase: str
- completion_phase: str
+ task_type: str
+ origin_scenario: str
+ completion_scenario: str
status: str
- created_at: Optional[float]
- updated_at: Optional[float]
- server_duration_ms: Optional[float]
+ resource_id: Optional[str]
local_duration_ms: float
- active_count_updated: Optional[int]
- memories_extracted: Optional[Dict[str, int]]
- error: Optional[str]
- cycle_index: Optional[int]
+ server_duration_ms: Optional[float]
+ error_message: Optional[str]
+ result: Any
polled_at: str
def to_dict(self) -> Dict[str, Any]:
@@ -114,980 +168,1241 @@ class CommitTaskEvent:
@dataclass
-class ObserverSample:
- api: str
- phase: str
- sampled_at: str
- elapsed_ms_since_run_start: float
- latency_ms: float
- success: bool
- is_healthy: Optional[bool]
- has_errors: Optional[bool]
- payload: Optional[Dict[str, Any]]
- error_message: Optional[str] = None
-
- def to_dict(self) -> Dict[str, Any]:
- return asdict(self)
+class PendingTask:
+ adapter_name: str
+ task_id: str
+ task_type: str
+ origin_scenario: str
+ resource_id: Optional[str]
+ local_started_monotonic: float
@dataclass
-class PendingCommitTask:
- task_id: str
- session_id: str
- origin_phase: str
- cycle_index: int
- local_started_monotonic: float
+class PhaseMetadata:
+ adapter: str
+ scenario: str
+ started_at: str
+ ended_at: str
+ duration_seconds: float
+
+ def to_dict(self) -> Dict[str, Any]:
+ return asdict(self)
@dataclass
class Recorder:
request_events: List[RequestEvent] = field(default_factory=list)
- task_events: List[CommitTaskEvent] = field(default_factory=list)
- observer_samples: List[ObserverSample] = field(default_factory=list)
+ task_events: List[TaskEvent] = field(default_factory=list)
+ phases: List[PhaseMetadata] = field(default_factory=list)
notes: List[str] = field(default_factory=list)
def add_request(self, event: RequestEvent) -> None:
self.request_events.append(event)
- def add_task(self, event: CommitTaskEvent) -> None:
+ def add_task(self, event: TaskEvent) -> None:
self.task_events.append(event)
- def add_sample(self, sample: ObserverSample) -> None:
- self.observer_samples.append(sample)
+ def add_phase(self, phase: PhaseMetadata) -> None:
+ self.phases.append(phase)
def add_note(self, note: str) -> None:
self.notes.append(note)
-class PhaseState:
- def __init__(self, initial: str = "setup") -> None:
- self.current = initial
+class LoadAdapter:
+ name: str
+ async def initialize(self) -> None: ...
-class BenchmarkHTTPClient:
- def __init__(self, config: BenchmarkConfig, recorder: Recorder) -> None:
- self._config = config
- self._recorder = recorder
- self._run_start_monotonic = time.perf_counter()
- self._client = httpx.AsyncClient(
- base_url=config.server_url.rstrip("/"),
- headers=self._default_headers(),
- timeout=httpx.Timeout(config.request_timeout),
- follow_redirects=True,
- limits=httpx.Limits(
- max_connections=max(
- 32,
- config.writer_concurrency
- + config.reader_concurrency
- + config.extract_concurrency
- + 8,
- ),
- max_keepalive_connections=max(
- 16,
- config.writer_concurrency + config.reader_concurrency + 4,
- ),
- ),
- )
+ async def close(self) -> None: ...
- @property
- def run_start_monotonic(self) -> float:
- return self._run_start_monotonic
+ async def health(self) -> Any: ...
- async def aclose(self) -> None:
- await self._client.aclose()
+ async def system_status(self) -> Any: ...
- def _default_headers(self) -> Dict[str, str]:
- headers = {
- "Accept": "*/*",
- "Content-Type": "application/json",
- "User-Agent": "OpenViking-Session-Contention-Benchmark/1.0",
- "X-OpenViking-Account": self._config.account,
- "X-OpenViking-User": self._config.user,
- }
- if self._config.api_key:
- headers["Authorization"] = f"Bearer {self._config.api_key}"
- return headers
+ async def observer_queue(self) -> Any: ...
- async def request_json(
- self,
- *,
- api: str,
- method: str,
- path: str,
- phase: str,
- session_id: Optional[str] = None,
- cycle_index: Optional[int] = None,
- worker_id: Optional[int] = None,
- task_id: Optional[str] = None,
- json_payload: Optional[Dict[str, Any]] = None,
- params: Optional[Dict[str, Any]] = None,
- ) -> tuple[Optional[httpx.Response], Optional[Dict[str, Any]]]:
- started_monotonic = time.perf_counter()
- started_wall = utc_now()
- response: Optional[httpx.Response] = None
- body: Optional[Dict[str, Any]] = None
- status_code: Optional[int] = None
- success = False
- timeout = False
- exception_type: Optional[str] = None
- error_code: Optional[str] = None
- error_message: Optional[str] = None
+ async def add_resource(
+ self, *, path: str, to: str, reason: str, wait: bool, timeout: Optional[float]
+ ) -> Any: ...
- try:
- response = await self._client.request(
- method=method,
- url=path,
- json=json_payload,
- params=params,
- )
- status_code = response.status_code
- body = maybe_json(response)
- success = self._is_success(status_code, body)
- if not success:
- error_code, error_message = extract_error(body, status_code)
- except httpx.TimeoutException as exc:
- timeout = True
- exception_type = type(exc).__name__
- error_message = truncate_error_message(str(exc))
- except Exception as exc: # pragma: no cover - exercised in real runs
- exception_type = type(exc).__name__
- error_message = truncate_error_message(str(exc))
+ async def wait_processed(self, timeout: Optional[float]) -> Any: ...
- ended_wall = utc_now()
- ended_monotonic = time.perf_counter()
- latency_ms = (ended_monotonic - started_monotonic) * 1000.0
- elapsed_ms = (started_monotonic - self._run_start_monotonic) * 1000.0
- self._recorder.add_request(
- RequestEvent(
- api=api,
- method=method.upper(),
- path=path,
- phase=phase,
- started_at=started_wall,
- ended_at=ended_wall,
- elapsed_ms_since_run_start=elapsed_ms,
- latency_ms=latency_ms,
- success=success,
- status_code=status_code,
- timeout=timeout,
- exception_type=exception_type,
- error_code=error_code,
- error_message=error_message,
- session_id=session_id,
- cycle_index=cycle_index,
- worker_id=worker_id,
- task_id=task_id,
- )
- )
- return response, body
+ async def find(self, *, query: str, target_uri: str, limit: int) -> Any: ...
- @staticmethod
- def _is_success(status_code: Optional[int], body: Optional[Dict[str, Any]]) -> bool:
- if status_code is None or status_code >= 400:
- return False
- if not isinstance(body, dict):
- return status_code < 400
- if "status" in body:
- return body.get("status") == "ok"
- return True
+ async def search(
+ self, *, query: str, target_uri: str, session_id: Optional[str], limit: int
+ ) -> Any: ...
+ async def grep(self, *, uri: str, pattern: str, limit: int) -> Any: ...
+
+ async def glob(self, *, uri: str, pattern: str, limit: int) -> Any: ...
+
+ async def add_message(self, *, session_id: str, role: str, content: str) -> Any: ...
+
+ async def commit_session(self, *, session_id: str) -> Any: ...
+
+ async def get_task(self, *, task_id: str) -> Any: ...
-class CommitTaskPoller:
- def __init__(
- self,
- client: BenchmarkHTTPClient,
- recorder: Recorder,
- phase_state: PhaseState,
- poll_interval: float,
- ) -> None:
- self._client = client
- self._recorder = recorder
- self._phase_state = phase_state
- self._poll_interval = poll_interval
- self._pending: Dict[str, PendingCommitTask] = {}
- self._closed = False
- self._wake_event = asyncio.Event()
- self._lock = asyncio.Lock()
-
- async def register(self, task: PendingCommitTask) -> None:
- async with self._lock:
- self._pending[task.task_id] = task
- self._wake_event.set()
+
+class AsyncHTTPAdapter(LoadAdapter):
+ def __init__(self, *, name: str, config: BenchmarkConfig, sdk_import: bool) -> None:
+ self.name = name
+ self._config = config
+ self._sdk_import = sdk_import
+ self._client: Any = None
+
+ async def initialize(self) -> None:
+ if self._sdk_import:
+ import openviking as ov
+
+ client_cls = ov.AsyncHTTPClient
+ else:
+ from openviking_cli.client.http import AsyncHTTPClient
+
+ client_cls = AsyncHTTPClient
+ self._client = client_cls(
+ url=self._config.server_url,
+ api_key=self._config.api_key,
+ account=self._config.account,
+ user=self._config.user,
+ agent_id=direct_http_agent_id(self._config),
+ timeout=self._config.timeout,
+ extra_headers={},
+ )
+ await self._client.initialize()
async def close(self) -> None:
- self._closed = True
- self._wake_event.set()
+ if self._client is not None:
+ await self._client.close()
+
+ async def health(self) -> Any:
+ return {"healthy": await self._client.health()}
+
+ async def system_status(self) -> Any:
+ return await self._client._get_system_status()
+
+ async def observer_queue(self) -> Any:
+ return await self._client._get_queue_status()
+
+ async def add_resource(
+ self, *, path: str, to: str, reason: str, wait: bool, timeout: Optional[float]
+ ) -> Any:
+ return await self._client.add_resource(
+ path=path,
+ to=to,
+ reason=reason,
+ wait=wait,
+ timeout=timeout,
+ strict=False,
+ )
- async def drain(self, timeout: float) -> None:
- deadline = time.perf_counter() + timeout
- while True:
- async with self._lock:
- remaining = len(self._pending)
- if remaining == 0:
- return
- if time.perf_counter() >= deadline:
- return
- await asyncio.sleep(min(self._poll_interval, 0.5))
-
- async def finalize_incomplete(self) -> None:
- async with self._lock:
- leftovers = list(self._pending.values())
- self._pending.clear()
- for item in leftovers:
- local_duration_ms = (time.perf_counter() - item.local_started_monotonic) * 1000.0
- self._recorder.add_task(
- CommitTaskEvent(
- task_id=item.task_id,
- session_id=item.session_id,
- origin_phase=item.origin_phase,
- completion_phase=self._phase_state.current,
- status="incomplete",
- created_at=None,
- updated_at=None,
- server_duration_ms=None,
- local_duration_ms=local_duration_ms,
- active_count_updated=None,
- memories_extracted=None,
- error="task not completed before benchmark end",
- cycle_index=item.cycle_index,
- polled_at=utc_now(),
- )
- )
+ async def wait_processed(self, timeout: Optional[float]) -> Any:
+ return await self._client.wait_processed(timeout=timeout)
- async def run(self) -> None:
- while True:
- await self._wake_event.wait()
- self._wake_event.clear()
+ async def find(self, *, query: str, target_uri: str, limit: int) -> Any:
+ result = await self._client.find(query=query, target_uri=target_uri, limit=limit)
+ return to_jsonable(result)
- while True:
- async with self._lock:
- pending = list(self._pending.values())
- if not pending:
- break
- await self._poll_pending(pending)
- if self._closed:
- return
- await asyncio.sleep(self._poll_interval)
+ async def search(
+ self, *, query: str, target_uri: str, session_id: Optional[str], limit: int
+ ) -> Any:
+ result = await self._client.search(
+ query=query,
+ target_uri=target_uri,
+ session_id=session_id,
+ limit=limit,
+ )
+ return to_jsonable(result)
- if self._closed:
- return
+ async def grep(self, *, uri: str, pattern: str, limit: int) -> Any:
+ return await self._client.grep(uri=uri, pattern=pattern, node_limit=limit)
- async def _poll_pending(self, pending: List[PendingCommitTask]) -> None:
- coroutines = [self._poll_one(item) for item in pending]
- results = await asyncio.gather(*coroutines, return_exceptions=True)
- completed_ids = [task_id for task_id in results if isinstance(task_id, str)]
- if not completed_ids:
- return
- async with self._lock:
- for task_id in completed_ids:
- self._pending.pop(task_id, None)
-
- async def _poll_one(self, item: PendingCommitTask) -> Optional[str]:
- _, body = await self._client.request_json(
- api="get_task",
- method="GET",
- path=f"/api/v1/tasks/{item.task_id}",
- phase=self._phase_state.current,
- session_id=item.session_id,
- cycle_index=item.cycle_index,
- task_id=item.task_id,
+ async def glob(self, *, uri: str, pattern: str, limit: int) -> Any:
+ return await self._client.glob(pattern=pattern, uri=uri)
+
+ async def add_message(self, *, session_id: str, role: str, content: str) -> Any:
+ return await self._client.add_message(session_id, role, content=content)
+
+ async def commit_session(self, *, session_id: str) -> Any:
+ return await self._client.commit_session(session_id)
+
+ async def get_task(self, *, task_id: str) -> Any:
+ return await self._client.get_task(task_id)
+
+
+class CliSubprocessAdapter(LoadAdapter):
+ def __init__(self, *, config: BenchmarkConfig, cli_config_path: Path) -> None:
+ self.name = "cli-subprocess"
+ self._config = config
+ self._cli_config_path = cli_config_path
+ self._env = os.environ.copy()
+ self._env["OPENVIKING_CLI_CONFIG_FILE"] = str(cli_config_path)
+
+ async def initialize(self) -> None:
+ self._cli_config_path.parent.mkdir(parents=True, exist_ok=True)
+ self._cli_config_path.write_text(
+ json.dumps(build_cli_config_payload(self._config), ensure_ascii=False, indent=2),
+ encoding="utf-8",
)
- if not isinstance(body, dict) or body.get("status") != "ok":
- return None
- result = body.get("result") or {}
- task_status = result.get("status")
- if task_status not in {"completed", "failed"}:
- return None
+ result = await self.health()
+ if isinstance(result, AdapterResult) and not result.success:
+ raise RuntimeError(result.error_message or "ov health failed")
- created_at = to_float(result.get("created_at"))
- updated_at = to_float(result.get("updated_at"))
- server_duration_ms = None
- if created_at is not None and updated_at is not None:
- server_duration_ms = max(updated_at - created_at, 0.0) * 1000.0
- local_duration_ms = (time.perf_counter() - item.local_started_monotonic) * 1000.0
- task_result = result.get("result") or {}
- self._recorder.add_task(
- CommitTaskEvent(
- task_id=item.task_id,
- session_id=item.session_id,
- origin_phase=item.origin_phase,
- completion_phase=self._phase_state.current,
- status=task_status,
- created_at=created_at,
- updated_at=updated_at,
- server_duration_ms=server_duration_ms,
- local_duration_ms=local_duration_ms,
- active_count_updated=task_result.get("active_count_updated"),
- memories_extracted=task_result.get("memories_extracted"),
- error=result.get("error"),
- cycle_index=item.cycle_index,
- polled_at=utc_now(),
- )
+ async def close(self) -> None:
+ return None
+
+ async def health(self) -> AdapterResult:
+ return await self._run(["health"], operation_timeout=self._config.timeout)
+
+ async def system_status(self) -> AdapterResult:
+ return await self._run(["status"])
+
+ async def observer_queue(self) -> AdapterResult:
+ return await self._run(["observer", "queue"])
+
+ async def add_resource(
+ self, *, path: str, to: str, reason: str, wait: bool, timeout: Optional[float]
+ ) -> AdapterResult:
+ args = ["add-resource", path, "--to", to, "--reason", reason]
+ if wait:
+ args.append("--wait")
+ if timeout is not None:
+ args.extend(["--timeout", str(timeout)])
+ return await self._run(args, operation_timeout=max(self._config.timeout, 60.0))
+
+ async def wait_processed(self, timeout: Optional[float]) -> AdapterResult:
+ args = ["wait"]
+ if timeout is not None:
+ args.extend(["--timeout", str(timeout)])
+ return await self._run(args, operation_timeout=timeout or self._config.timeout)
+
+ async def find(self, *, query: str, target_uri: str, limit: int) -> AdapterResult:
+ return await self._run(["find", query, "--uri", target_uri, "-n", str(limit)])
+
+ async def search(
+ self, *, query: str, target_uri: str, session_id: Optional[str], limit: int
+ ) -> AdapterResult:
+ args = ["search", query, "--uri", target_uri, "-n", str(limit)]
+ if session_id:
+ args.extend(["--session-id", session_id])
+ return await self._run(args)
+
+ async def grep(self, *, uri: str, pattern: str, limit: int) -> AdapterResult:
+ return await self._run(["grep", pattern, "--uri", uri, "-n", str(limit)])
+
+ async def glob(self, *, uri: str, pattern: str, limit: int) -> AdapterResult:
+ return await self._run(["glob", pattern, "--uri", uri, "-n", str(limit)])
+
+ async def add_message(self, *, session_id: str, role: str, content: str) -> AdapterResult:
+ return await self._run(
+ ["session", "add-message", session_id, "--role", role, "--content", content]
)
- return item.task_id
+
+ async def commit_session(self, *, session_id: str) -> AdapterResult:
+ return await self._run(["session", "commit", session_id])
+
+ async def get_task(self, *, task_id: str) -> AdapterResult:
+ return await self._run(["task", "status", task_id])
+
+ def build_command(self, args: Sequence[str]) -> List[str]:
+ return [
+ self._config.ov_bin,
+ "--output",
+ "json",
+ "--compact",
+ "--no-progress",
+ *args,
+ ]
+
+ async def _run(
+ self, args: Sequence[str], *, operation_timeout: Optional[float] = None
+ ) -> AdapterResult:
+ command = self.build_command(args)
+ try:
+ proc = await asyncio.create_subprocess_exec(
+ *command,
+ stdout=asyncio.subprocess.PIPE,
+ stderr=asyncio.subprocess.PIPE,
+ env=self._env,
+ )
+ stdout_bytes, stderr_bytes = await asyncio.wait_for(
+ proc.communicate(),
+ timeout=operation_timeout or self._config.timeout,
+ )
+ except asyncio.TimeoutError:
+ return AdapterResult(
+ success=False,
+ exception_type="TimeoutError",
+ error_message=f"CLI command timed out after {operation_timeout or self._config.timeout}s",
+ command=command,
+ )
+ except Exception as exc: # pragma: no cover - exercised in real runs
+ return AdapterResult(
+ success=False,
+ exception_type=type(exc).__name__,
+ error_message=truncate_error_message(str(exc)),
+ command=command,
+ )
+ stdout = stdout_bytes.decode("utf-8", errors="replace")
+ stderr = stderr_bytes.decode("utf-8", errors="replace")
+ parsed = parse_cli_process_result(proc.returncode, stdout, stderr)
+ parsed.command = command
+ return parsed
class BenchmarkRunner:
def __init__(self, config: BenchmarkConfig) -> None:
self.config = config
+ self.run_id = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
self.random = random.Random(config.seed)
self.recorder = Recorder()
- self.phase_state = PhaseState()
- self.phase_metadata: List[PhaseMetadata] = []
- self.phase_durations: Dict[str, float] = {}
- self.session_ids: List[str] = []
- self.session_queue: asyncio.Queue[str] = asyncio.Queue()
- self.session_cycle_counts: Dict[str, int] = {}
- self.extract_semaphore = asyncio.Semaphore(max(1, config.extract_concurrency))
- self.client = BenchmarkHTTPClient(config, self.recorder)
- self.task_poller = CommitTaskPoller(
- client=self.client,
- recorder=self.recorder,
- phase_state=self.phase_state,
- poll_interval=config.task_poll_interval,
- )
+ self.run_start_monotonic = time.perf_counter()
+ self.pending_tasks: List[PendingTask] = []
+ self.session_ids_by_adapter: Dict[str, List[str]] = {}
+ self.adapters: Dict[str, LoadAdapter] = {}
+ self.mixed_doc_counter = count()
async def run(self) -> int:
- poller_task = asyncio.create_task(self.task_poller.run())
exit_code = 0
try:
- await self._preflight()
- await self._create_sessions()
- await self._run_phase(
- phase="baseline",
- duration_seconds=self.config.baseline_seconds,
- enable_readers=self.config.reader_concurrency > 0,
- enable_writers=False,
- enable_sampler=self.config.observer_interval > 0,
- )
- await self._run_phase(
- phase="mixed_load",
- duration_seconds=self.config.mixed_seconds,
- enable_readers=self.config.reader_concurrency > 0,
- enable_writers=self.config.writer_concurrency > 0 and bool(self.session_ids),
- enable_sampler=self.config.observer_interval > 0,
- )
- await self._run_phase(
- phase="recovery",
- duration_seconds=self.config.recovery_seconds,
- enable_readers=self.config.reader_concurrency > 0,
- enable_writers=False,
- enable_sampler=self.config.observer_interval > 0,
- )
- if self.config.task_drain_timeout > 0:
- self.phase_state.current = "drain"
- await self.task_poller.drain(self.config.task_drain_timeout)
- except RuntimeError as exc:
- self.recorder.add_note(f"fatal: {exc}")
- print(f"[fatal] {exc}", file=sys.stderr)
+ await self._prepare_filesystem()
+ await self._prepare_remote_state()
+ self.adapters = await self._create_adapters()
+ for adapter in self.adapters.values():
+ await self._run_adapter(adapter)
+ if self.pending_tasks:
+ await self._drain_tasks("final_drain", self.config.drain_timeout)
+ except Exception as exc:
+ self.recorder.add_note(f"fatal: {type(exc).__name__}: {exc}")
+ print(f"[fatal] {type(exc).__name__}: {exc}", file=sys.stderr)
exit_code = 1
finally:
- await self.task_poller.close()
- await poller_task
- await self.task_poller.finalize_incomplete()
- if self.config.cleanup and self.session_ids:
- await self._cleanup_sessions()
- await self.client.aclose()
-
- self._write_outputs()
- self._print_summary()
+ if self.config.cleanup_at_end:
+ try:
+ await self._clear_remote_state()
+ except Exception as exc: # pragma: no cover - best effort cleanup
+ self.recorder.add_note(f"cleanup_at_end failed: {type(exc).__name__}: {exc}")
+ for adapter in self.adapters.values():
+ await adapter.close()
+ self._write_outputs()
+ self._print_summary_path()
return exit_code
- async def _preflight(self) -> None:
- self.phase_state.current = "setup"
- _, health_body = await self.client.request_json(
- api="health",
- method="GET",
- path="/health",
- phase="setup",
+ async def _create_adapters(self) -> Dict[str, LoadAdapter]:
+ output_dir = Path(self.config.output_dir)
+ adapters: Dict[str, LoadAdapter] = {}
+ for name in self.config.adapters:
+ if name == "sdk":
+ adapter: LoadAdapter = AsyncHTTPAdapter(
+ name="sdk", config=self.config, sdk_import=True
+ )
+ elif name == "cli-http":
+ adapter = AsyncHTTPAdapter(name="cli-http", config=self.config, sdk_import=False)
+ elif name == "cli-subprocess":
+ adapter = CliSubprocessAdapter(
+ config=self.config,
+ cli_config_path=output_dir / "runtime" / "ovcli.conf",
+ )
+ else:
+ raise ValueError(f"Unknown adapter: {name}")
+ await adapter.initialize()
+ adapters[name] = adapter
+ return adapters
+
+ async def _prepare_filesystem(self) -> None:
+ local_dir = Path(self.config.local_data_dir)
+ if self.config.clear_before_run and local_dir.exists():
+ shutil.rmtree(local_dir)
+ local_dir.mkdir(parents=True, exist_ok=True)
+ Path(self.config.output_dir).mkdir(parents=True, exist_ok=True)
+ for adapter_name in self.config.adapters:
+ adapter_dir = local_dir / "seed" / adapter_name
+ adapter_dir.mkdir(parents=True, exist_ok=True)
+ for index in range(self.config.resource_count):
+ write_benchmark_document(
+ adapter_dir / f"resource-{index:04d}.md",
+ adapter=adapter_name,
+ index=index,
+ run_id=self.run_id,
+ )
+
+ async def _prepare_remote_state(self) -> None:
+ if self.config.clear_before_run:
+ await self._clear_remote_state()
+ await self._ensure_remote_tree()
+
+ async def _clear_remote_state(self) -> None:
+ from openviking_cli.client.http import AsyncHTTPClient
+ from openviking_cli.exceptions import NotFoundError, OpenVikingError
+
+ client = AsyncHTTPClient(
+ url=self.config.server_url,
+ api_key=self.config.api_key,
+ account=self.config.account,
+ user=self.config.user,
+ agent_id=direct_http_agent_id(self.config),
+ timeout=self.config.timeout,
+ extra_headers={},
)
- if not isinstance(health_body, dict) or health_body.get("status") != "ok":
- raise RuntimeError("server health check failed")
-
- _, status_body = await self.client.request_json(
- api="system_status",
- method="GET",
- path="/api/v1/system/status",
- phase="setup",
+ await client.initialize()
+ try:
+ try:
+ await client.rm(self.config.data_root_uri, recursive=True)
+ except NotFoundError:
+ pass
+ except OpenVikingError as exc:
+ if not is_not_found_error(exc):
+ raise
+ sessions = await client.list_sessions()
+ for session_id in extract_session_ids(sessions):
+ if session_id.startswith(self.config.session_prefix):
+ try:
+ await client.delete_session(session_id)
+ except Exception:
+ pass
+ finally:
+ await client.close()
+
+ async def _ensure_remote_tree(self) -> None:
+ from openviking_cli.client.http import AsyncHTTPClient
+ from openviking_cli.exceptions import OpenVikingError
+
+ client = AsyncHTTPClient(
+ url=self.config.server_url,
+ api_key=self.config.api_key,
+ account=self.config.account,
+ user=self.config.user,
+ agent_id=direct_http_agent_id(self.config),
+ timeout=self.config.timeout,
+ extra_headers={},
+ )
+ await client.initialize()
+ try:
+ for uri in iter_resource_tree_uris(self.config.data_root_uri):
+ try:
+ await client.mkdir(uri)
+ except OpenVikingError as exc:
+ if not is_already_exists_error(exc):
+ raise
+ for adapter_name in self.config.adapters:
+ try:
+ await client.mkdir(f"{self.config.data_root_uri}/{adapter_name}")
+ except OpenVikingError as exc:
+ if not is_already_exists_error(exc):
+ raise
+ finally:
+ await client.close()
+
+ async def _run_adapter(self, adapter: LoadAdapter) -> None:
+ self.session_ids_by_adapter[adapter.name] = [
+ f"{self.config.session_prefix}{self.run_id}-{adapter.name}-{i:04d}"
+ for i in range(self.config.session_count)
+ ]
+ await self._run_warmup(adapter)
+ await self._run_add_resources(adapter)
+ await self._drain_tasks(f"{adapter.name}:resource_drain", self.config.drain_timeout)
+ await self._run_session_messages(adapter)
+ await self._run_timed_scenario(
+ adapter=adapter,
+ scenario="retrieval",
+ duration_seconds=self.config.phase_seconds,
+ concurrency=self.config.search_concurrency,
+ worker=self._retrieval_worker,
)
- if not isinstance(status_body, dict) or status_body.get("status") != "ok":
- raise RuntimeError("authenticated system status request failed")
-
- _, models_body = await self.client.request_json(
- api="observer_models",
- method="GET",
- path="/api/v1/observer/models",
- phase="setup",
+ await self._run_session_commit(adapter)
+ await self._drain_tasks(f"{adapter.name}:commit_drain", self.config.drain_timeout)
+ await self._run_timed_scenario(
+ adapter=adapter,
+ scenario="mixed",
+ duration_seconds=self.config.mixed_seconds,
+ concurrency=self.config.mixed_concurrency,
+ worker=self._mixed_worker,
)
- model_result = (models_body or {}).get("result") if isinstance(models_body, dict) else None
- model_note = self._extract_model_note(model_result)
- if model_note:
- self.recorder.add_note(model_note)
-
- if self.config.extract_ratio > 0:
- preflight_result = await self._run_extract_preflight()
- if preflight_result:
- self.recorder.add_note(preflight_result)
- if self.config.require_extract_load:
- raise RuntimeError(preflight_result)
-
- async def _run_extract_preflight(self) -> Optional[str]:
- _, create_body = await self.client.request_json(
- api="create_session",
- method="POST",
- path="/api/v1/sessions",
- phase="setup",
+ await self._drain_tasks(f"{adapter.name}:mixed_drain", self.config.drain_timeout)
+
+ async def _run_warmup(self, adapter: LoadAdapter) -> None:
+ if self.config.warmup_seconds <= 0:
+ return
+ await self._run_timed_scenario(
+ adapter=adapter,
+ scenario="warmup",
+ duration_seconds=self.config.warmup_seconds,
+ concurrency=1,
+ worker=self._warmup_worker,
)
- session_id = extract_session_id(create_body)
- if not session_id:
- return "extract preflight could not create session"
- try:
- payload = {
- "role": "user",
- "content": build_message_content(
- session_id=session_id,
- cycle_index=0,
- message_index=0,
- size=self.config.message_size,
- ),
- }
- await self.client.request_json(
- api="add_message",
- method="POST",
- path=f"/api/v1/sessions/{session_id}/messages",
- phase="setup",
- session_id=session_id,
- cycle_index=0,
- json_payload=payload,
- )
- _, extract_body = await self.client.request_json(
- api="extract",
- method="POST",
- path=f"/api/v1/sessions/{session_id}/extract",
- phase="setup",
- session_id=session_id,
- cycle_index=0,
+ async def _warmup_worker(
+ self, adapter: LoadAdapter, scenario: str, worker_id: int, stop_event: asyncio.Event
+ ) -> None:
+ while not stop_event.is_set():
+ await self._record_request(
+ adapter,
+ scenario,
+ "health",
+ lambda: adapter.health(),
+ worker_id=worker_id,
)
- if not isinstance(extract_body, dict) or extract_body.get("status") != "ok":
- return "extract preflight request failed"
- result = extract_body.get("result")
- if isinstance(result, list) and not result:
- return (
- "extract preflight returned empty result; long-tail load may be weak if models are "
- "not configured"
+ await asyncio.sleep(0.2)
+
+ async def _run_add_resources(self, adapter: LoadAdapter) -> None:
+ scenario = "add_resources"
+ started = time.perf_counter()
+ started_wall = utc_now()
+ files = sorted((Path(self.config.local_data_dir) / "seed" / adapter.name).glob("*.md"))
+ queue: asyncio.Queue[Path] = asyncio.Queue()
+ for path in files:
+ await queue.put(path)
+
+ async def worker(worker_id: int) -> None:
+ while True:
+ try:
+ path = queue.get_nowait()
+ except asyncio.QueueEmpty:
+ return
+ resource_uri = f"{self.config.data_root_uri}/{adapter.name}/{path.name}"
+ result = await self._record_request(
+ adapter,
+ scenario,
+ "add_resource",
+ lambda p=path, uri=resource_uri: adapter.add_resource(
+ path=str(p),
+ to=uri,
+ reason="OpenViking server load benchmark",
+ wait=False,
+ timeout=None,
+ ),
+ resource_uri=resource_uri,
+ worker_id=worker_id,
)
- return None
- finally:
- await self.client.request_json(
- api="delete_session",
- method="DELETE",
- path=f"/api/v1/sessions/{session_id}",
- phase="setup",
- session_id=session_id,
+ self._register_task_if_present(
+ adapter=adapter,
+ scenario=scenario,
+ task_type="add_resource",
+ resource_id=resource_uri,
+ result=result,
+ )
+
+ await asyncio.gather(
+ *[
+ asyncio.create_task(worker(worker_id))
+ for worker_id in range(max(1, self.config.resource_concurrency))
+ ]
+ )
+ await self._record_request(
+ adapter,
+ scenario,
+ "wait_processed",
+ lambda: adapter.wait_processed(timeout=self.config.drain_timeout),
+ )
+ self.recorder.add_phase(
+ PhaseMetadata(
+ adapter=adapter.name,
+ scenario=scenario,
+ started_at=started_wall,
+ ended_at=utc_now(),
+ duration_seconds=time.perf_counter() - started,
)
+ )
- def _extract_model_note(self, model_result: Any) -> Optional[str]:
- if not isinstance(model_result, dict):
- return None
- is_healthy = model_result.get("is_healthy")
- status = model_result.get("status")
- if is_healthy is False:
- return f"observer/models reports unhealthy state; extract load may not be representative: {status}"
- return None
+ async def _run_session_messages(self, adapter: LoadAdapter) -> None:
+ scenario = "session_messages"
+ started = time.perf_counter()
+ started_wall = utc_now()
+ units = [
+ (session_id, message_index)
+ for session_id in self.session_ids_by_adapter[adapter.name]
+ for message_index in range(self.config.messages_per_session)
+ ]
+ queue: asyncio.Queue[tuple[str, int]] = asyncio.Queue()
+ for unit in units:
+ await queue.put(unit)
- async def _create_sessions(self) -> None:
- if self.config.session_count <= 0:
- return
- for _ in range(self.config.session_count):
- _, body = await self.client.request_json(
- api="create_session",
- method="POST",
- path="/api/v1/sessions",
- phase="setup",
+ async def worker(worker_id: int) -> None:
+ while True:
+ try:
+ session_id, message_index = queue.get_nowait()
+ except asyncio.QueueEmpty:
+ return
+ role = "user" if message_index % 2 == 0 else "assistant"
+ content = build_message_content(
+ session_id=session_id,
+ message_index=message_index,
+ adapter=adapter.name,
+ )
+ await self._record_request(
+ adapter,
+ scenario,
+ "add_message",
+ lambda sid=session_id, r=role, c=content: adapter.add_message(
+ session_id=sid,
+ role=r,
+ content=c,
+ ),
+ session_id=session_id,
+ worker_id=worker_id,
+ )
+
+ await asyncio.gather(
+ *[
+ asyncio.create_task(worker(worker_id))
+ for worker_id in range(max(1, self.config.session_concurrency))
+ ]
+ )
+ self.recorder.add_phase(
+ PhaseMetadata(
+ adapter=adapter.name,
+ scenario=scenario,
+ started_at=started_wall,
+ ended_at=utc_now(),
+ duration_seconds=time.perf_counter() - started,
)
- session_id = extract_session_id(body)
- if not session_id:
- raise RuntimeError("failed to create benchmark sessions")
- self.session_ids.append(session_id)
- self.session_cycle_counts[session_id] = 0
- await self.session_queue.put(session_id)
-
- async def _cleanup_sessions(self) -> None:
- self.phase_state.current = "cleanup"
- for session_id in self.session_ids:
- await self.client.request_json(
- api="delete_session",
- method="DELETE",
- path=f"/api/v1/sessions/{session_id}",
- phase="cleanup",
- session_id=session_id,
+ )
+
+ async def _run_session_commit(self, adapter: LoadAdapter) -> None:
+ scenario = "session_commit"
+ started = time.perf_counter()
+ started_wall = utc_now()
+ queue: asyncio.Queue[str] = asyncio.Queue()
+ for session_id in self.session_ids_by_adapter[adapter.name]:
+ await queue.put(session_id)
+
+ async def worker(worker_id: int) -> None:
+ while True:
+ try:
+ session_id = queue.get_nowait()
+ except asyncio.QueueEmpty:
+ return
+ result = await self._record_request(
+ adapter,
+ scenario,
+ "commit_session",
+ lambda sid=session_id: adapter.commit_session(session_id=sid),
+ session_id=session_id,
+ worker_id=worker_id,
+ )
+ self._register_task_if_present(
+ adapter=adapter,
+ scenario=scenario,
+ task_type="session_commit",
+ resource_id=session_id,
+ result=result,
+ )
+
+ await asyncio.gather(
+ *[
+ asyncio.create_task(worker(worker_id))
+ for worker_id in range(max(1, self.config.commit_concurrency))
+ ]
+ )
+ self.recorder.add_phase(
+ PhaseMetadata(
+ adapter=adapter.name,
+ scenario=scenario,
+ started_at=started_wall,
+ ended_at=utc_now(),
+ duration_seconds=time.perf_counter() - started,
)
+ )
- async def _run_phase(
+ async def _run_timed_scenario(
self,
*,
- phase: str,
+ adapter: LoadAdapter,
+ scenario: str,
duration_seconds: float,
- enable_readers: bool,
- enable_writers: bool,
- enable_sampler: bool,
+ concurrency: int,
+ worker: Callable[[LoadAdapter, str, int, asyncio.Event], Awaitable[None]],
) -> None:
- if duration_seconds <= 0:
+ if duration_seconds <= 0 or concurrency <= 0:
return
-
- self.phase_state.current = phase
stop_event = asyncio.Event()
- tasks: List[asyncio.Task[Any]] = []
-
- if enable_readers:
- for worker_id in range(self.config.reader_concurrency):
- tasks.append(asyncio.create_task(self._reader_worker(phase, worker_id, stop_event)))
- if enable_writers:
- for worker_id in range(self.config.writer_concurrency):
- tasks.append(asyncio.create_task(self._writer_worker(phase, worker_id, stop_event)))
- if enable_sampler:
- tasks.append(asyncio.create_task(self._sampler_worker(phase, stop_event)))
-
- phase_started = time.perf_counter()
+ started = time.perf_counter()
started_wall = utc_now()
+ tasks = [
+ asyncio.create_task(worker(adapter, scenario, worker_id, stop_event))
+ for worker_id in range(concurrency)
+ ]
await asyncio.sleep(duration_seconds)
stop_event.set()
- if tasks:
- await asyncio.gather(*tasks, return_exceptions=True)
- phase_duration = time.perf_counter() - phase_started
- ended_wall = utc_now()
- self.phase_metadata.append(
+ await asyncio.gather(*tasks, return_exceptions=True)
+ self.recorder.add_phase(
PhaseMetadata(
- phase=phase,
+ adapter=adapter.name,
+ scenario=scenario,
started_at=started_wall,
- ended_at=ended_wall,
- duration_seconds=phase_duration,
+ ended_at=utc_now(),
+ duration_seconds=time.perf_counter() - started,
)
)
- self.phase_durations[phase] = phase_duration
- async def _writer_worker(self, phase: str, worker_id: int, stop_event: asyncio.Event) -> None:
+ async def _retrieval_worker(
+ self, adapter: LoadAdapter, scenario: str, worker_id: int, stop_event: asyncio.Event
+ ) -> None:
while not stop_event.is_set():
- session_id = await self._borrow_session(stop_event)
- if not session_id:
- return
- try:
- cycle_index = self.session_cycle_counts[session_id]
- self.session_cycle_counts[session_id] += 1
- await self._run_session_cycle(
- phase=phase,
+ await self._run_retrieval_operation(adapter, scenario, worker_id)
+
+ async def _mixed_worker(
+ self, adapter: LoadAdapter, scenario: str, worker_id: int, stop_event: asyncio.Event
+ ) -> None:
+ choices = [
+ ("find", 35),
+ ("search", 20),
+ ("add_message", 20),
+ ("commit_session", 10),
+ ("add_resource", 10),
+ ("observer_queue", 3),
+ ("system_status", 2),
+ ]
+ operations = [name for name, weight in choices for _ in range(weight)]
+ while not stop_event.is_set():
+ operation = self.random.choice(operations)
+ if operation in {"find", "search"}:
+ await self._run_retrieval_operation(adapter, scenario, worker_id, force=operation)
+ elif operation == "add_message":
+ session_id = self.random.choice(self.session_ids_by_adapter[adapter.name])
+ content = build_message_content(
+ session_id=session_id,
+ message_index=next(self.mixed_doc_counter),
+ adapter=adapter.name,
+ )
+ await self._record_request(
+ adapter,
+ scenario,
+ "add_message",
+ lambda sid=session_id, c=content: adapter.add_message(
+ session_id=sid,
+ role="user",
+ content=c,
+ ),
+ session_id=session_id,
worker_id=worker_id,
+ )
+ elif operation == "commit_session":
+ session_id = self.random.choice(self.session_ids_by_adapter[adapter.name])
+ result = await self._record_request(
+ adapter,
+ scenario,
+ "commit_session",
+ lambda sid=session_id: adapter.commit_session(session_id=sid),
session_id=session_id,
- cycle_index=cycle_index,
+ worker_id=worker_id,
+ )
+ self._register_task_if_present(
+ adapter=adapter,
+ scenario=scenario,
+ task_type="session_commit",
+ resource_id=session_id,
+ result=result,
+ )
+ elif operation == "add_resource":
+ index = next(self.mixed_doc_counter)
+ path = (
+ Path(self.config.local_data_dir) / "mixed" / adapter.name / f"mixed-{index}.md"
+ )
+ write_benchmark_document(
+ path, adapter=adapter.name, index=index, run_id=self.run_id
+ )
+ resource_uri = f"{self.config.data_root_uri}/{adapter.name}/{path.name}"
+ result = await self._record_request(
+ adapter,
+ scenario,
+ "add_resource",
+ lambda p=path, uri=resource_uri: adapter.add_resource(
+ path=str(p),
+ to=uri,
+ reason="OpenViking mixed load benchmark",
+ wait=False,
+ timeout=None,
+ ),
+ resource_uri=resource_uri,
+ worker_id=worker_id,
+ )
+ self._register_task_if_present(
+ adapter=adapter,
+ scenario=scenario,
+ task_type="add_resource",
+ resource_id=resource_uri,
+ result=result,
+ )
+ elif operation == "observer_queue":
+ await self._record_request(
+ adapter,
+ scenario,
+ "observer_queue",
+ lambda: adapter.observer_queue(),
+ worker_id=worker_id,
+ )
+ else:
+ await self._record_request(
+ adapter,
+ scenario,
+ "system_status",
+ lambda: adapter.system_status(),
+ worker_id=worker_id,
)
- finally:
- await self.session_queue.put(session_id)
- async def _run_session_cycle(
+ async def _run_retrieval_operation(
self,
- *,
- phase: str,
+ adapter: LoadAdapter,
+ scenario: str,
worker_id: int,
- session_id: str,
- cycle_index: int,
+ *,
+ force: Optional[str] = None,
) -> None:
- successful_messages = 0
- for message_index in range(self.config.messages_per_commit):
- payload = {
- "role": "user",
- "content": build_message_content(
- session_id=session_id,
- cycle_index=cycle_index,
- message_index=message_index,
- size=self.config.message_size,
+ operation = (
+ force
+ or self.random.choices(
+ ["find", "search", "grep", "glob"], weights=[55, 25, 10, 10], k=1
+ )[0]
+ )
+ query = self.random.choice(DEFAULT_QUERIES)
+ if operation == "find":
+ await self._record_request(
+ adapter,
+ scenario,
+ "find",
+ lambda: adapter.find(
+ query=query,
+ target_uri=self.config.data_root_uri,
+ limit=self.config.find_limit,
+ ),
+ worker_id=worker_id,
+ )
+ elif operation == "search":
+ session_id = self.random.choice(self.session_ids_by_adapter[adapter.name])
+ await self._record_request(
+ adapter,
+ scenario,
+ "search",
+ lambda sid=session_id: adapter.search(
+ query=query,
+ target_uri=self.config.data_root_uri,
+ session_id=sid,
+ limit=self.config.find_limit,
),
- }
- _, body = await self.client.request_json(
- api="add_message",
- method="POST",
- path=f"/api/v1/sessions/{session_id}/messages",
- phase=phase,
session_id=session_id,
- cycle_index=cycle_index,
worker_id=worker_id,
- json_payload=payload,
)
- if isinstance(body, dict) and body.get("status") == "ok":
- successful_messages += 1
-
- if successful_messages <= 0:
- return
-
- if self.config.extract_ratio > 0 and self.random.random() < self.config.extract_ratio:
- async with self.extract_semaphore:
- await self.client.request_json(
- api="extract",
- method="POST",
- path=f"/api/v1/sessions/{session_id}/extract",
- phase=phase,
- session_id=session_id,
- cycle_index=cycle_index,
- worker_id=worker_id,
- )
+ elif operation == "grep":
+ await self._record_request(
+ adapter,
+ scenario,
+ "grep",
+ lambda: adapter.grep(
+ uri=self.config.data_root_uri,
+ pattern="benchmark",
+ limit=self.config.find_limit,
+ ),
+ worker_id=worker_id,
+ )
+ else:
+ await self._record_request(
+ adapter,
+ scenario,
+ "glob",
+ lambda: adapter.glob(
+ uri=self.config.data_root_uri,
+ pattern="*.md",
+ limit=self.config.find_limit,
+ ),
+ worker_id=worker_id,
+ )
- _, body = await self.client.request_json(
- api="commit",
- method="POST",
- path=f"/api/v1/sessions/{session_id}/commit",
- phase=phase,
- session_id=session_id,
- cycle_index=cycle_index,
- worker_id=worker_id,
- )
- task_id = extract_task_id(body)
- if task_id:
- await self.task_poller.register(
- PendingCommitTask(
- task_id=task_id,
- session_id=session_id,
- origin_phase=phase,
- cycle_index=cycle_index,
- local_started_monotonic=time.perf_counter(),
- )
+ async def _record_request(
+ self,
+ adapter: LoadAdapter,
+ scenario: str,
+ operation: str,
+ call: Callable[[], Awaitable[Any]],
+ *,
+ session_id: Optional[str] = None,
+ resource_uri: Optional[str] = None,
+ worker_id: Optional[int] = None,
+ ) -> AdapterResult:
+ started_monotonic = time.perf_counter()
+ started_wall = utc_now()
+ result: AdapterResult
+ try:
+ value = await call()
+ if isinstance(value, AdapterResult):
+ result = value
+ else:
+ result = AdapterResult(success=True, result=to_jsonable(value))
+ except Exception as exc:
+ result = AdapterResult(
+ success=False,
+ exception_type=type(exc).__name__,
+ error_message=truncate_error_message(str(exc)),
)
- async def _reader_worker(self, phase: str, worker_id: int, stop_event: asyncio.Event) -> None:
- while not stop_event.is_set():
- payload = {
- "query": self.random.choice(self.config.find_queries),
- "limit": self.config.find_limit,
- }
- if self.config.find_target_uri:
- payload["target_uri"] = self.config.find_target_uri
- if self.config.find_score_threshold is not None:
- payload["score_threshold"] = self.config.find_score_threshold
- await self.client.request_json(
- api="find",
- method="POST",
- path="/api/v1/search/find",
- phase=phase,
+ ended_monotonic = time.perf_counter()
+ result_payload = normalize_result_payload(result.result)
+ task_id = extract_task_id(result_payload)
+ if task_id is None:
+ task_id = extract_task_id_from_any(result.result)
+ self.recorder.add_request(
+ RequestEvent(
+ adapter=adapter.name,
+ scenario=scenario,
+ operation=operation,
+ started_at=started_wall,
+ ended_at=utc_now(),
+ elapsed_ms_since_run_start=(started_monotonic - self.run_start_monotonic) * 1000.0,
+ latency_ms=(ended_monotonic - started_monotonic) * 1000.0,
+ success=result.success,
+ status_code=result.status_code,
+ exception_type=result.exception_type,
+ error_message=result.error_message,
+ session_id=session_id,
+ resource_uri=resource_uri or extract_root_uri(result_payload),
+ task_id=task_id,
worker_id=worker_id,
- json_payload=payload,
+ command=" ".join(result.command) if result.command else None,
)
+ )
+ return result
- async def _sampler_worker(self, phase: str, stop_event: asyncio.Event) -> None:
- sample_specs = [
- ("system_status", "GET", "/api/v1/system/status"),
- ("observer_queue", "GET", "/api/v1/observer/queue"),
- ("observer_system", "GET", "/api/v1/observer/system"),
- ]
- while not stop_event.is_set():
- for api, method, path in sample_specs:
- started = time.perf_counter()
- response, body = await self.client.request_json(
- api=api,
- method=method,
- path=path,
- phase=phase,
- )
- latency_ms = (time.perf_counter() - started) * 1000.0
- success = response is not None and self.client._is_success(
- response.status_code if response else None,
- body,
+ def _register_task_if_present(
+ self,
+ *,
+ adapter: LoadAdapter,
+ scenario: str,
+ task_type: str,
+ resource_id: Optional[str],
+ result: AdapterResult,
+ ) -> None:
+ if not result.success:
+ return
+ task_id = extract_task_id_from_any(result.result)
+ if not task_id:
+ return
+ self.pending_tasks.append(
+ PendingTask(
+ adapter_name=adapter.name,
+ task_id=task_id,
+ task_type=task_type,
+ origin_scenario=scenario,
+ resource_id=resource_id,
+ local_started_monotonic=time.perf_counter(),
+ )
+ )
+
+ async def _drain_tasks(self, scenario: str, timeout: float) -> None:
+ if not self.pending_tasks or timeout <= 0:
+ self._finalize_incomplete_tasks(scenario)
+ return
+ deadline = time.perf_counter() + timeout
+ while self.pending_tasks and time.perf_counter() < deadline:
+ remaining: List[PendingTask] = []
+ for task in list(self.pending_tasks):
+ adapter = self.adapters.get(task.adapter_name)
+ if adapter is None:
+ remaining.append(task)
+ continue
+ result = await self._record_request(
+ adapter,
+ scenario,
+ "get_task",
+ lambda a=adapter, tid=task.task_id: a.get_task(task_id=tid),
)
- self.recorder.add_sample(
- ObserverSample(
- api=api,
- phase=phase,
- sampled_at=utc_now(),
- elapsed_ms_since_run_start=(
- time.perf_counter() - self.client.run_start_monotonic
+ payload = normalize_result_payload(result.result)
+ status = extract_task_status(payload)
+ if result.success and status in {"completed", "failed"}:
+ self.recorder.add_task(
+ build_task_event(
+ adapter=task.adapter_name,
+ task=task,
+ completion_scenario=scenario,
+ payload=payload,
)
- * 1000.0,
- latency_ms=latency_ms,
- success=success,
- is_healthy=extract_boolean(body, "result", "is_healthy"),
- has_errors=extract_boolean(body, "result", "has_errors"),
- payload=body if isinstance(body, dict) else None,
- error_message=extract_error(
- body, response.status_code if response else None
- )[1]
- if response is not None and not success
- else None,
)
+ else:
+ remaining.append(task)
+ self.pending_tasks = remaining
+ if self.pending_tasks:
+ await asyncio.sleep(self.config.task_poll_interval)
+ if scenario == "final_drain":
+ self._finalize_incomplete_tasks(scenario)
+
+ def _finalize_incomplete_tasks(self, scenario: str) -> None:
+ for task in self.pending_tasks:
+ self.recorder.add_task(
+ TaskEvent(
+ adapter=task.adapter_name,
+ task_id=task.task_id,
+ task_type=task.task_type,
+ origin_scenario=task.origin_scenario,
+ completion_scenario=scenario,
+ status="incomplete",
+ resource_id=task.resource_id,
+ local_duration_ms=(time.perf_counter() - task.local_started_monotonic) * 1000.0,
+ server_duration_ms=None,
+ error_message="task not completed before benchmark end",
+ result=None,
+ polled_at=utc_now(),
)
- if stop_event.is_set():
- break
- if stop_event.is_set():
- return
- try:
- await asyncio.wait_for(stop_event.wait(), timeout=self.config.observer_interval)
- except asyncio.TimeoutError:
- continue
-
- async def _borrow_session(self, stop_event: asyncio.Event) -> Optional[str]:
- while not stop_event.is_set():
- try:
- return await asyncio.wait_for(self.session_queue.get(), timeout=0.2)
- except asyncio.TimeoutError:
- continue
- return None
+ )
+ self.pending_tasks = []
def _write_outputs(self) -> None:
output_dir = Path(self.config.output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
-
- request_summary_rows = build_request_summary_rows(
+ summary_rows = build_request_summary_rows(
+ events=self.recorder.request_events,
+ phases=self.recorder.phases,
+ )
+ window_rows = build_request_window_rows(
events=self.recorder.request_events,
- phase_durations=self.phase_durations,
- total_run_duration=total_duration_seconds(self.phase_metadata),
+ window_seconds=self.config.request_window_seconds,
)
- task_summary_rows = build_task_summary_rows(self.recorder.task_events)
- human_summary_zh = render_human_summary_zh(
+ task_rows = build_task_summary_rows(self.recorder.task_events)
+ adapter_rows = build_adapter_comparison_rows(summary_rows)
+ error_rows = build_error_rows(self.recorder.request_events)
+ report = render_report_zh(
config=self.config,
- output_dir=self.config.output_dir,
+ run_id=self.run_id,
notes=self.recorder.notes,
- phase_metadata=self.phase_metadata,
- request_summary_rows=request_summary_rows,
- request_events=self.recorder.request_events,
- task_summary_rows=task_summary_rows,
- task_events=self.recorder.task_events,
+ phases=self.recorder.phases,
+ summary_rows=summary_rows,
+ task_rows=task_rows,
+ adapter_rows=adapter_rows,
+ error_rows=error_rows,
+ output_dir=str(output_dir),
)
write_json(output_dir / "run_config.json", asdict(self.config))
- write_json(
- output_dir / "phases.json",
- [asdict(item) for item in self.phase_metadata],
- )
+ write_json(output_dir / "phases.json", [phase.to_dict() for phase in self.recorder.phases])
write_json(
output_dir / "run_summary.json",
- self._build_run_summary(
- request_summary_rows=request_summary_rows,
- task_summary_rows=task_summary_rows,
- human_summary_zh=human_summary_zh,
- ),
+ {
+ "run_id": self.run_id,
+ "created_at": utc_now(),
+ "notes": self.recorder.notes,
+ "request_summary": summary_rows,
+ "task_summary": task_rows,
+ "adapter_comparison": adapter_rows,
+ "errors": error_rows,
+ "summary_zh": report,
+ },
)
- write_text(output_dir / "summary_zh.txt", human_summary_zh)
+ write_text(output_dir / "summary_zh.md", report)
write_jsonl(output_dir / "request_events.jsonl", self.recorder.request_events)
- write_jsonl(output_dir / "task_events.jsonl", self.recorder.task_events)
- write_jsonl(output_dir / "observer_samples.jsonl", self.recorder.observer_samples)
-
- write_csv(
- output_dir / "request_summary.csv",
- request_summary_rows,
- )
- write_csv(
- output_dir / "request_windows.csv",
- build_request_window_rows(
- events=self.recorder.request_events,
- window_seconds=self.config.window_seconds,
- ),
- )
- write_csv(
- output_dir / "task_summary.csv",
- task_summary_rows,
- )
-
- def _build_run_summary(
- self,
- *,
- request_summary_rows: List[Dict[str, Any]],
- task_summary_rows: List[Dict[str, Any]],
- human_summary_zh: str,
- ) -> Dict[str, Any]:
- find_delta = build_find_phase_delta(request_summary_rows)
- return {
- "notes": self.recorder.notes,
- "phase_metadata": [asdict(item) for item in self.phase_metadata],
- "request_summary": request_summary_rows,
- "task_summary": task_summary_rows,
- "find_phase_delta": find_delta,
- "human_summary_zh": human_summary_zh,
- "created_at": utc_now(),
- }
-
- def _print_summary(self) -> None:
- request_summary_rows = build_request_summary_rows(
- events=self.recorder.request_events,
- phase_durations=self.phase_durations,
- total_run_duration=total_duration_seconds(self.phase_metadata),
- )
- task_summary_rows = build_task_summary_rows(self.recorder.task_events)
- print(
- "\n"
- + render_human_summary_zh(
- config=self.config,
- output_dir=self.config.output_dir,
- notes=self.recorder.notes,
- phase_metadata=self.phase_metadata,
- request_summary_rows=request_summary_rows,
- request_events=self.recorder.request_events,
- task_summary_rows=task_summary_rows,
- task_events=self.recorder.task_events,
- )
- )
-
-
-def parse_args(argv: Optional[List[str]] = None) -> BenchmarkConfig:
- server_host = os.getenv("SERVER_HOST", "127.0.0.1")
- server_port = int(os.getenv("SERVER_PORT", "1933"))
- default_server_url = f"http://{server_host}:{server_port}"
- default_output_dir = (
- Path(__file__).resolve().parents[1]
- / "results"
- / "session_contention"
- / datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
- )
-
- parser = argparse.ArgumentParser(
- description="Reproduce session addMessage/extract/commit contention against concurrent find traffic.",
- )
- parser.add_argument("--server-url", default=default_server_url)
- parser.add_argument("--api-key", default=os.getenv("OPENVIKING_API_KEY", "test-root-api-key"))
- parser.add_argument("--account", default=os.getenv("OPENVIKING_ACCOUNT", "default"))
- parser.add_argument("--user", default=os.getenv("OPENVIKING_USER", "default"))
- parser.add_argument("--request-timeout", type=float, default=30.0)
- parser.add_argument("--sessions", type=int, default=8)
- parser.add_argument("--writer-concurrency", type=int, default=8)
- parser.add_argument("--reader-concurrency", type=int, default=4)
- parser.add_argument("--extract-concurrency", type=int, default=4)
- parser.add_argument("--messages-per-commit", type=int, default=5)
- parser.add_argument("--extract-ratio", type=float, default=0.5)
- parser.add_argument("--message-size", type=int, default=768)
- parser.add_argument("--baseline-seconds", type=float, default=30.0)
- parser.add_argument("--mixed-seconds", type=float, default=120.0)
- parser.add_argument("--recovery-seconds", type=float, default=30.0)
- parser.add_argument("--window-seconds", type=float, default=5.0)
- parser.add_argument("--observer-interval", type=float, default=5.0)
- parser.add_argument("--task-poll-interval", type=float, default=1.0)
- parser.add_argument("--task-drain-timeout", type=float, default=30.0)
- parser.add_argument("--output-dir", default=str(default_output_dir))
- parser.add_argument("--cleanup", action="store_true")
- parser.add_argument("--require-extract-load", action="store_true")
- parser.add_argument(
- "--find-query",
- action="append",
- dest="find_queries",
- default=[],
- help="Repeat to add multiple find queries.",
- )
- parser.add_argument("--find-limit", type=int, default=10)
- parser.add_argument("--find-target-uri", default="")
- parser.add_argument("--find-score-threshold", type=float, default=None)
- parser.add_argument("--seed", type=int, default=42)
-
- args = parser.parse_args(argv)
- find_queries = args.find_queries or list(DEFAULT_FIND_QUERIES)
-
- config = BenchmarkConfig(
- server_url=args.server_url,
- api_key=args.api_key,
- account=args.account,
- user=args.user,
- request_timeout=args.request_timeout,
- session_count=max(0, args.sessions),
- writer_concurrency=max(0, args.writer_concurrency),
- reader_concurrency=max(0, args.reader_concurrency),
- extract_concurrency=max(1, args.extract_concurrency),
- messages_per_commit=max(1, args.messages_per_commit),
- extract_ratio=min(max(args.extract_ratio, 0.0), 1.0),
- message_size=max(128, args.message_size),
- baseline_seconds=max(0.0, args.baseline_seconds),
- mixed_seconds=max(0.0, args.mixed_seconds),
- recovery_seconds=max(0.0, args.recovery_seconds),
- window_seconds=max(args.window_seconds, 1.0),
- observer_interval=0.0 if args.observer_interval <= 0 else max(args.observer_interval, 0.1),
- task_poll_interval=max(args.task_poll_interval, 0.1),
- task_drain_timeout=max(0.0, args.task_drain_timeout),
- output_dir=args.output_dir,
- cleanup=args.cleanup,
- require_extract_load=args.require_extract_load,
- find_queries=find_queries,
- find_limit=max(1, args.find_limit),
- find_target_uri=args.find_target_uri,
- find_score_threshold=args.find_score_threshold,
- seed=args.seed,
+ write_jsonl(output_dir / "task_events.jsonl", self.recorder.task_events)
+ write_csv(output_dir / "request_summary.csv", summary_rows)
+ write_csv(output_dir / "request_windows.csv", window_rows)
+ write_csv(output_dir / "task_summary.csv", task_rows)
+ write_csv(output_dir / "adapter_comparison.csv", adapter_rows)
+ write_csv(output_dir / "errors.csv", error_rows)
+
+ def _print_summary_path(self) -> None:
+ summary_path = Path(self.config.output_dir) / "summary_zh.md"
+ print(f"\n压测报告: {summary_path}")
+
+
+def parse_cli_process_result(returncode: int, stdout: str, stderr: str) -> AdapterResult:
+ stdout = stdout.strip()
+ stderr = stderr.strip()
+ parsed: Any = None
+ if stdout:
+ try:
+ parsed = json.loads(stdout)
+ except json.JSONDecodeError:
+ parsed = None
+ if returncode != 0:
+ error_message = stderr or stdout or f"ov exited with code {returncode}"
+ if parsed and isinstance(parsed, dict):
+ error_message = extract_error_message(parsed) or error_message
+ return AdapterResult(
+ success=False,
+ status_code=returncode,
+ exception_type="CliExitError",
+ error_message=truncate_error_message(error_message),
+ result=parsed,
+ raw_stdout=stdout,
+ raw_stderr=stderr,
+ )
+ if isinstance(parsed, dict) and parsed.get("ok") is False:
+ return AdapterResult(
+ success=False,
+ status_code=returncode,
+ exception_type="CliApiError",
+ error_message=truncate_error_message(extract_error_message(parsed) or stderr),
+ result=parsed.get("result"),
+ raw_stdout=stdout,
+ raw_stderr=stderr,
+ )
+ if isinstance(parsed, dict) and "result" in parsed and parsed.get("ok") is True:
+ return AdapterResult(
+ success=True,
+ status_code=returncode,
+ result=parsed.get("result"),
+ raw_stdout=stdout,
+ raw_stderr=stderr,
+ )
+ return AdapterResult(
+ success=True,
+ status_code=returncode,
+ result=parsed if parsed is not None else stdout,
+ raw_stdout=stdout,
+ raw_stderr=stderr,
)
- if config.writer_concurrency > 0 and config.session_count <= 0:
- parser.error("--sessions must be > 0 when --writer-concurrency is enabled")
- return config
-
-def maybe_json(response: httpx.Response) -> Optional[Dict[str, Any]]:
- try:
- body = response.json()
- except ValueError:
- return None
- return body if isinstance(body, dict) else {"value": body}
+def build_cli_config_payload(config: BenchmarkConfig) -> Dict[str, Any]:
+ return {
+ "url": config.server_url,
+ "api_key": config.api_key,
+ "account": config.account,
+ "user": config.user,
+ "agent_id": config.agent_id,
+ "timeout": config.timeout,
+ "extra_headers": None,
+ }
-def extract_error(
- body: Optional[Dict[str, Any]], status_code: Optional[int]
-) -> tuple[Optional[str], Optional[str]]:
- if not isinstance(body, dict):
- if status_code is None:
- return None, None
- return None, f"http status {status_code}"
- error = body.get("error")
- if isinstance(error, dict):
- return error.get("code"), truncate_error_message(error.get("message"))
- if body.get("status") not in {None, "ok"}:
- return body.get("status"), truncate_error_message(json.dumps(body, ensure_ascii=False))
- if status_code is not None and status_code >= 400:
- return None, f"http status {status_code}"
- return None, None
+def write_benchmark_document(path: Path, *, adapter: str, index: int, run_id: str) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ content = f"""# OpenViking Load Benchmark Document {index}
-def extract_session_id(body: Optional[Dict[str, Any]]) -> Optional[str]:
- if not isinstance(body, dict):
- return None
- result = body.get("result")
- if not isinstance(result, dict):
- return None
- session_id = result.get("session_id")
- return session_id if isinstance(session_id, str) else None
+Adapter: {adapter}
+Run: {run_id}
+Keywords: benchmark, concurrent retrieval, resource ingestion, session commit.
+This generated document is used to test concurrent add_resource, find, search,
+grep, glob, add_message, and commit traffic against a local OpenViking server.
-def extract_task_id(body: Optional[Dict[str, Any]]) -> Optional[str]:
- if not isinstance(body, dict):
- return None
- result = body.get("result")
- if not isinstance(result, dict):
- return None
- task_id = result.get("task_id")
- return task_id if isinstance(task_id, str) and task_id else None
+The text intentionally repeats relevant terms so retrieval has stable targets:
+OpenViking server benchmark benchmark benchmark. Session commit memory extraction.
+Resource ingestion queue latency. Mixed SDK CLI subprocess load.
+"""
+ path.write_text(content, encoding="utf-8")
-def build_message_content(
- *, session_id: str, cycle_index: int, message_index: int, size: int
-) -> str:
- prefix = (
- f"session={session_id} cycle={cycle_index} message={message_index}. "
- "We discussed project goals, deployment constraints, user preferences, debugging notes, "
- "timelines, risks, and follow-up actions. "
- )
- detail = (
- "The user prefers production-safe changes, wants clear rollback steps, and asked for "
- "memory extraction to keep decisions, entities, and events. "
- "We also covered resource bottlenecks, queue backlog, response latency, and how read "
- "traffic regressed during heavy write pressure. "
+def build_message_content(*, session_id: str, message_index: int, adapter: str) -> str:
+ return (
+ f"[{adapter}] session={session_id} message={message_index}. "
+ "The user discussed OpenViking server load testing, concurrent resource ingestion, "
+ "retrieval latency, session commit background tasks, and mixed SDK/CLI traffic. "
+ "Keep decisions, bottlenecks, follow-up actions, and observed queue behavior."
)
- content = prefix
- while len(content) < size:
- content += detail
- return content[:size]
-def truncate_error_message(message: Optional[str]) -> Optional[str]:
- if message is None:
- return None
- if len(message) <= MAX_ERROR_MESSAGE_LEN:
- return message
- return message[:MAX_ERROR_MESSAGE_LEN] + "...[truncated]"
+def extract_session_ids(payload: Any) -> List[str]:
+ if isinstance(payload, dict):
+ for key in ("sessions", "items", "result"):
+ if key in payload:
+ return extract_session_ids(payload[key])
+ value = payload.get("session_id") or payload.get("id")
+ return [value] if isinstance(value, str) else []
+ if isinstance(payload, list):
+ ids: List[str] = []
+ for item in payload:
+ ids.extend(extract_session_ids(item))
+ return ids
+ return []
+
+
+def iter_resource_tree_uris(root_uri: str) -> List[str]:
+ prefix = "viking://resources"
+ normalized = root_uri.rstrip("/")
+ if not normalized.startswith(prefix):
+ return [normalized]
+ tail = normalized[len(prefix) :].strip("/")
+ uris = [prefix]
+ current = prefix
+ for part in tail.split("/"):
+ if not part:
+ continue
+ current = f"{current}/{part}"
+ uris.append(current)
+ return uris[1:]
+
+
+def to_jsonable(value: Any) -> Any:
+ if hasattr(value, "to_dict") and callable(value.to_dict):
+ return value.to_dict()
+ if isinstance(value, dict):
+ return {k: to_jsonable(v) for k, v in value.items()}
+ if isinstance(value, list):
+ return [to_jsonable(v) for v in value]
+ if isinstance(value, tuple):
+ return [to_jsonable(v) for v in value]
+ return value
+
+
+def normalize_result_payload(value: Any) -> Any:
+ if isinstance(value, AdapterResult):
+ return normalize_result_payload(value.result)
+ if isinstance(value, dict) and value.get("ok") is True and "result" in value:
+ return value["result"]
+ return value
+
+
+def extract_task_id_from_any(value: Any) -> Optional[str]:
+ return extract_task_id(normalize_result_payload(value))
+
+
+def extract_task_id(value: Any) -> Optional[str]:
+ if isinstance(value, dict):
+ candidate = value.get("task_id")
+ if isinstance(candidate, str) and candidate:
+ return candidate
+ for key in ("result", "task"):
+ nested = extract_task_id(value.get(key))
+ if nested:
+ return nested
+ return None
-def utc_now() -> str:
- return datetime.now(UTC).isoformat(timespec="milliseconds").replace("+00:00", "Z")
+def extract_root_uri(value: Any) -> Optional[str]:
+ if isinstance(value, dict):
+ for key in ("root_uri", "uri", "resource_id"):
+ candidate = value.get(key)
+ if isinstance(candidate, str) and candidate:
+ return candidate
+ return None
+
+
+def extract_task_status(value: Any) -> Optional[str]:
+ if not isinstance(value, dict):
+ return None
+ status = value.get("status")
+ if isinstance(status, str):
+ return status
+ task = value.get("task")
+ if isinstance(task, dict):
+ status = task.get("status")
+ if isinstance(status, str):
+ return status
+ return None
-def total_duration_seconds(phases: List[PhaseMetadata]) -> float:
- return sum(item.duration_seconds for item in phases)
+def build_task_event(
+ *, adapter: str, task: PendingTask, completion_scenario: str, payload: Any
+) -> TaskEvent:
+ created_at = to_float(payload.get("created_at")) if isinstance(payload, dict) else None
+ updated_at = to_float(payload.get("updated_at")) if isinstance(payload, dict) else None
+ server_duration_ms = None
+ if created_at is not None and updated_at is not None:
+ server_duration_ms = max(updated_at - created_at, 0.0) * 1000.0
+ status = extract_task_status(payload) or "unknown"
+ error_message = payload.get("error") if isinstance(payload, dict) else None
+ task_result = payload.get("result") if isinstance(payload, dict) else None
+ return TaskEvent(
+ adapter=adapter,
+ task_id=task.task_id,
+ task_type=task.task_type,
+ origin_scenario=task.origin_scenario,
+ completion_scenario=completion_scenario,
+ status=status,
+ resource_id=task.resource_id,
+ local_duration_ms=(time.perf_counter() - task.local_started_monotonic) * 1000.0,
+ server_duration_ms=server_duration_ms,
+ error_message=truncate_error_message(str(error_message)) if error_message else None,
+ result=task_result,
+ polled_at=utc_now(),
+ )
def percentile(values: Iterable[float], pct: float) -> Optional[float]:
@@ -1106,95 +1421,79 @@ def percentile(values: Iterable[float], pct: float) -> Optional[float]:
def build_request_summary_rows(
- *,
- events: List[RequestEvent],
- phase_durations: Dict[str, float],
- total_run_duration: float,
+ *, events: List[RequestEvent], phases: List[PhaseMetadata]
) -> List[Dict[str, Any]]:
+ phase_durations = {
+ (phase.adapter, phase.scenario): max(phase.duration_seconds, 1e-9) for phase in phases
+ }
+ groups: Dict[tuple[str, str, str], List[RequestEvent]] = {}
+ for event in events:
+ groups.setdefault((event.adapter, event.scenario, event.operation), []).append(event)
rows: List[Dict[str, Any]] = []
- rows.extend(
- _build_request_summary_for_groups(
- events=events,
- grouping=lambda event: (event.phase, event.api),
- duration_lookup=phase_durations,
+ for (adapter, scenario, operation), group_events in sorted(groups.items()):
+ duration = phase_durations.get((adapter, scenario)) or estimate_event_span_seconds(
+ group_events
)
- )
- overall_groups = _build_request_summary_for_groups(
- events=events,
- grouping=lambda event: ("ALL", event.api),
- duration_lookup={"ALL": total_run_duration},
- )
- rows.extend(overall_groups)
- return sorted(rows, key=lambda row: (row["phase"], row["api"]))
+ rows.append(build_summary_row(adapter, scenario, operation, group_events, duration))
+ overall: Dict[tuple[str, str], List[RequestEvent]] = {}
+ for event in events:
+ overall.setdefault((event.adapter, event.operation), []).append(event)
+ for (adapter, operation), group_events in sorted(overall.items()):
+ rows.append(build_summary_row(adapter, "ALL", operation, group_events, total_span(events)))
+ return rows
-def _build_request_summary_for_groups(
- *,
+def build_summary_row(
+ adapter: str,
+ scenario: str,
+ operation: str,
events: List[RequestEvent],
- grouping,
- duration_lookup: Dict[str, float],
-) -> List[Dict[str, Any]]:
- groups: Dict[tuple[str, str], List[RequestEvent]] = {}
+ duration_seconds: float,
+) -> Dict[str, Any]:
+ latencies = [event.latency_ms for event in events]
+ successes = sum(1 for event in events if event.success)
+ failures = len(events) - successes
+ status_counts: Dict[str, int] = {}
for event in events:
- key = grouping(event)
- groups.setdefault(key, []).append(event)
-
- rows: List[Dict[str, Any]] = []
- for (phase, api), api_events in groups.items():
- latencies = [event.latency_ms for event in api_events]
- successes = sum(1 for event in api_events if event.success)
- failures = len(api_events) - successes
- timeouts = sum(1 for event in api_events if event.timeout)
- exceptions = sum(1 for event in api_events if event.exception_type)
- status_counts: Dict[str, int] = {}
- for event in api_events:
- key = str(event.status_code) if event.status_code is not None else "exception"
- status_counts[key] = status_counts.get(key, 0) + 1
- duration = max(duration_lookup.get(phase, 0.0), 1e-9)
- row = {
- "phase": phase,
- "api": api,
- "requests": len(api_events),
- "successes": successes,
- "failures": failures,
- "timeouts": timeouts,
- "exceptions": exceptions,
- "success_rate": round((successes / len(api_events)) * 100.0, 4),
- "qps": round(len(api_events) / duration, 4),
- "avg_ms": round(sum(latencies) / len(latencies), 4),
- "p50_ms": round_optional(percentile(latencies, 50)),
- "p90_ms": round_optional(percentile(latencies, 90)),
- "p95_ms": round_optional(percentile(latencies, 95)),
- "p99_ms": round_optional(percentile(latencies, 99)),
- "max_ms": round_optional(max(latencies) if latencies else None),
- "slow_gt_1s": sum(
- 1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[0]
- ),
- "slow_gt_3s": sum(
- 1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[1]
- ),
- "slow_gt_5s": sum(
- 1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[2]
- ),
- "status_codes": json.dumps(status_counts, sort_keys=True),
- }
- rows.append(row)
- return rows
+ key = (
+ str(event.status_code)
+ if event.status_code is not None
+ else (event.exception_type or "ok")
+ )
+ status_counts[key] = status_counts.get(key, 0) + 1
+ return {
+ "adapter": adapter,
+ "scenario": scenario,
+ "operation": operation,
+ "requests": len(events),
+ "successes": successes,
+ "failures": failures,
+ "success_rate": round((successes / len(events)) * 100.0, 4) if events else 0.0,
+ "qps": round(len(events) / max(duration_seconds, 1e-9), 4),
+ "avg_ms": round_optional(sum(latencies) / len(latencies) if latencies else None),
+ "p50_ms": round_optional(percentile(latencies, 50)),
+ "p90_ms": round_optional(percentile(latencies, 90)),
+ "p95_ms": round_optional(percentile(latencies, 95)),
+ "p99_ms": round_optional(percentile(latencies, 99)),
+ "max_ms": round_optional(max(latencies) if latencies else None),
+ "slow_gt_1s": sum(1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[0]),
+ "slow_gt_3s": sum(1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[1]),
+ "slow_gt_5s": sum(1 for latency in latencies if latency > DEFAULT_SLOW_THRESHOLDS_MS[2]),
+ "status_codes": json.dumps(status_counts, sort_keys=True),
+ }
def build_request_window_rows(
- *,
- events: List[RequestEvent],
- window_seconds: float,
+ *, events: List[RequestEvent], window_seconds: float
) -> List[Dict[str, Any]]:
- groups: Dict[tuple[int, str, str], List[RequestEvent]] = {}
+ groups: Dict[tuple[int, str, str, str], List[RequestEvent]] = {}
for event in events:
window_index = int((event.elapsed_ms_since_run_start / 1000.0) // window_seconds)
- key = (window_index, event.phase, event.api)
- groups.setdefault(key, []).append(event)
-
+ groups.setdefault(
+ (window_index, event.adapter, event.scenario, event.operation), []
+ ).append(event)
rows: List[Dict[str, Any]] = []
- for (window_index, phase, api), window_events in sorted(groups.items()):
+ for (window_index, adapter, scenario, operation), window_events in sorted(groups.items()):
latencies = [event.latency_ms for event in window_events]
successes = sum(1 for event in window_events if event.success)
rows.append(
@@ -1202,13 +1501,14 @@ def build_request_window_rows(
"window_index": window_index,
"window_start_sec": round(window_index * window_seconds, 4),
"window_end_sec": round((window_index + 1) * window_seconds, 4),
- "phase": phase,
- "api": api,
+ "adapter": adapter,
+ "scenario": scenario,
+ "operation": operation,
"requests": len(window_events),
"successes": successes,
"failures": len(window_events) - successes,
"success_rate": round((successes / len(window_events)) * 100.0, 4),
- "qps": round(len(window_events) / window_seconds, 4),
+ "qps": round(len(window_events) / max(window_seconds, 1e-9), 4),
"p95_ms": round_optional(percentile(latencies, 95)),
"p99_ms": round_optional(percentile(latencies, 99)),
"max_ms": round_optional(max(latencies) if latencies else None),
@@ -1217,441 +1517,438 @@ def build_request_window_rows(
return rows
-def build_task_summary_rows(events: List[CommitTaskEvent]) -> List[Dict[str, Any]]:
- groups: Dict[str, List[CommitTaskEvent]] = {}
+def build_task_summary_rows(events: List[TaskEvent]) -> List[Dict[str, Any]]:
+ groups: Dict[tuple[str, str, str], List[TaskEvent]] = {}
for event in events:
- groups.setdefault(event.status, []).append(event)
-
+ groups.setdefault((event.adapter, event.task_type, event.status), []).append(event)
rows: List[Dict[str, Any]] = []
- for status, status_events in sorted(groups.items()):
+ for (adapter, task_type, status), task_events in sorted(groups.items()):
+ local_latencies = [event.local_duration_ms for event in task_events]
server_latencies = [
event.server_duration_ms
- for event in status_events
+ for event in task_events
if event.server_duration_ms is not None
]
- local_latencies = [event.local_duration_ms for event in status_events]
- successes = sum(1 for event in status_events if event.status == "completed")
rows.append(
{
+ "adapter": adapter,
+ "task_type": task_type,
"status": status,
- "tasks": len(status_events),
- "successes": successes,
- "success_rate": round((successes / len(status_events)) * 100.0, 4),
- "p50_server_duration_ms": round_optional(percentile(server_latencies, 50)),
- "p95_server_duration_ms": round_optional(percentile(server_latencies, 95)),
- "p99_server_duration_ms": round_optional(percentile(server_latencies, 99)),
- "max_server_duration_ms": round_optional(
+ "tasks": len(task_events),
+ "p50_local_ms": round_optional(percentile(local_latencies, 50)),
+ "p95_local_ms": round_optional(percentile(local_latencies, 95)),
+ "p99_local_ms": round_optional(percentile(local_latencies, 99)),
+ "max_local_ms": round_optional(max(local_latencies) if local_latencies else None),
+ "p50_server_ms": round_optional(percentile(server_latencies, 50)),
+ "p95_server_ms": round_optional(percentile(server_latencies, 95)),
+ "p99_server_ms": round_optional(percentile(server_latencies, 99)),
+ "max_server_ms": round_optional(
max(server_latencies) if server_latencies else None
),
- "p50_local_duration_ms": round_optional(percentile(local_latencies, 50)),
- "p95_local_duration_ms": round_optional(percentile(local_latencies, 95)),
- "p99_local_duration_ms": round_optional(percentile(local_latencies, 99)),
- "max_local_duration_ms": round_optional(
- max(local_latencies) if local_latencies else None
- ),
}
)
return rows
-def build_find_phase_delta(summary_rows: List[Dict[str, Any]]) -> Optional[Dict[str, float]]:
- baseline = next(
- (row for row in summary_rows if row["phase"] == "baseline" and row["api"] == "find"),
- None,
- )
- mixed = next(
- (row for row in summary_rows if row["phase"] == "mixed_load" and row["api"] == "find"),
- None,
- )
- if not baseline or not mixed:
- return None
- baseline_p95 = baseline.get("p95_ms")
- baseline_p99 = baseline.get("p99_ms")
- mixed_p95 = mixed.get("p95_ms")
- mixed_p99 = mixed.get("p99_ms")
- if not all(metric is not None for metric in [baseline_p95, baseline_p99, mixed_p95, mixed_p99]):
- return None
- return {
- "baseline_p95_ms": baseline_p95,
- "mixed_p95_ms": mixed_p95,
- "p95_delta_percent": percent_change(baseline_p95, mixed_p95),
- "baseline_p99_ms": baseline_p99,
- "mixed_p99_ms": mixed_p99,
- "p99_delta_percent": percent_change(baseline_p99, mixed_p99),
- "baseline_success_rate": baseline["success_rate"],
- "mixed_success_rate": mixed["success_rate"],
- "success_rate_delta_percent": mixed["success_rate"] - baseline["success_rate"],
- }
-
-
-def find_request_summary_row(
- summary_rows: List[Dict[str, Any]],
- *,
- api: str,
- phase: str,
-) -> Optional[Dict[str, Any]]:
- return next((row for row in summary_rows if row["api"] == api and row["phase"] == phase), None)
+def build_adapter_comparison_rows(summary_rows: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
+ rows = [
+ row
+ for row in summary_rows
+ if row["scenario"] == "ALL"
+ and row["operation"] in {"add_resource", "find", "search", "add_message", "commit_session"}
+ ]
+ return sorted(rows, key=lambda row: (row["operation"], row["adapter"]))
-def phase_target_seconds(config: BenchmarkConfig, phase: str) -> Optional[float]:
- mapping = {
- "baseline": config.baseline_seconds,
- "mixed_load": config.mixed_seconds,
- "recovery": config.recovery_seconds,
- }
- return mapping.get(phase)
+def build_error_rows(events: List[RequestEvent]) -> List[Dict[str, Any]]:
+ groups: Dict[tuple[str, str, str, str], int] = {}
+ for event in events:
+ if event.success:
+ continue
+ key = (
+ event.adapter,
+ event.scenario,
+ event.operation,
+ event.exception_type or event.error_message or "unknown",
+ )
+ groups[key] = groups.get(key, 0) + 1
+ return [
+ {
+ "adapter": adapter,
+ "scenario": scenario,
+ "operation": operation,
+ "error": error,
+ "count": count_value,
+ }
+ for (adapter, scenario, operation, error), count_value in sorted(
+ groups.items(), key=lambda item: item[1], reverse=True
+ )
+ ]
-def build_phase_overview_rows(
+def render_report_zh(
+ *,
config: BenchmarkConfig,
- phase_metadata: List[PhaseMetadata],
-) -> List[Dict[str, Optional[float]]]:
- rows: List[Dict[str, Optional[float]]] = []
- for item in phase_metadata:
- target = phase_target_seconds(config, item.phase)
- delta = None if target is None else item.duration_seconds - target
- rows.append(
- {
- "phase": item.phase,
- "target_seconds": round_optional(target),
- "actual_seconds": round_optional(item.duration_seconds),
- "delta_seconds": round_optional(delta),
- }
+ run_id: str,
+ notes: List[str],
+ phases: List[PhaseMetadata],
+ summary_rows: List[Dict[str, Any]],
+ task_rows: List[Dict[str, Any]],
+ adapter_rows: List[Dict[str, Any]],
+ error_rows: List[Dict[str, Any]],
+ output_dir: str,
+) -> str:
+ lines: List[str] = []
+ lines.append("# OpenViking Server 压测报告")
+ lines.append("")
+ lines.append(f"- 运行 ID: `{run_id}`")
+ lines.append(f"- Server: `{config.server_url}`")
+ lines.append(f"- 调用路径: `{', '.join(config.adapters)}`")
+ lines.append(f"- 测试数据根目录: `{config.data_root_uri}`")
+ lines.append(f"- 输出目录: `{output_dir}`")
+ if notes:
+ lines.append("")
+ lines.append("## 说明")
+ for note in notes:
+ lines.append(f"- {note}")
+ lines.append("")
+ lines.append("## 核心结论")
+ total_requests = sum(row["requests"] for row in summary_rows if row["scenario"] != "ALL")
+ total_failures = sum(row["failures"] for row in summary_rows if row["scenario"] != "ALL")
+ success_rate = (
+ (total_requests - total_failures) / total_requests * 100.0 if total_requests else 0.0
+ )
+ lines.append(
+ f"- 本次共记录 {total_requests} 次请求,失败 {total_failures} 次,整体成功率 {success_rate:.2f}%。"
+ )
+ for adapter in config.adapters:
+ mixed_find = find_summary(summary_rows, adapter=adapter, scenario="mixed", operation="find")
+ retrieval_find = find_summary(
+ summary_rows, adapter=adapter, scenario="retrieval", operation="find"
)
- return rows
+ if mixed_find and retrieval_find:
+ lines.append(
+ "- "
+ f"`{adapter}` find p95: retrieval {fmt_ms(retrieval_find['p95_ms'])} -> "
+ f"mixed {fmt_ms(mixed_find['p95_ms'])},变化 "
+ f"{fmt_delta_percent(percent_change(retrieval_find['p95_ms'], mixed_find['p95_ms']))}。"
+ )
+ incomplete = [row for row in task_rows if row["status"] == "incomplete"]
+ if incomplete:
+ total_incomplete = sum(row["tasks"] for row in incomplete)
+ lines.append(f"- 后台任务存在积压:{total_incomplete} 个任务在 drain 后仍未完成。")
+ if error_rows:
+ top = error_rows[0]
+ lines.append(
+ f"- Top 错误: `{top['adapter']}/{top['scenario']}/{top['operation']}` "
+ f"`{top['error']}` 共 {top['count']} 次。"
+ )
+ lines.append("")
+ lines.append("## 阶段耗时")
+ lines.append(markdown_table([phase.to_dict() for phase in phases]))
+ lines.append("")
+ lines.append("## 请求汇总")
+ lines.append(
+ markdown_table(
+ [
+ pick(
+ row,
+ [
+ "adapter",
+ "scenario",
+ "operation",
+ "requests",
+ "success_rate",
+ "qps",
+ "p50_ms",
+ "p95_ms",
+ "p99_ms",
+ "max_ms",
+ ],
+ )
+ for row in summary_rows
+ if row["scenario"] != "ALL"
+ ]
+ )
+ )
+ lines.append("")
+ lines.append("## SDK/CLI 对比")
+ lines.append(
+ markdown_table(
+ [
+ pick(
+ row,
+ ["adapter", "operation", "requests", "success_rate", "qps", "p95_ms", "p99_ms"],
+ )
+ for row in adapter_rows
+ ]
+ )
+ )
+ lines.append("")
+ lines.append("## 后台任务")
+ lines.append(markdown_table(task_rows) if task_rows else "无后台任务记录。")
+ lines.append("")
+ lines.append("## 错误 Top")
+ lines.append(markdown_table(error_rows[:20]) if error_rows else "无请求错误。")
+ lines.append("")
+ lines.append("## 输出文件")
+ lines.append("- `run_summary.json`: 汇总数据")
+ lines.append("- `request_events.jsonl`: 每次请求明细")
+ lines.append("- `task_events.jsonl`: 后台任务明细")
+ lines.append("- `request_summary.csv`: 请求聚合")
+ lines.append("- `request_windows.csv`: 时间窗口聚合")
+ lines.append("- `adapter_comparison.csv`: SDK/CLI 对比")
+ return "\n".join(lines) + "\n"
-def build_api_error_breakdown(
- events: List[RequestEvent],
- *,
- api: str,
- phase: Optional[str] = None,
-) -> Dict[str, Any]:
- filtered = [
- event for event in events if event.api == api and (phase is None or event.phase == phase)
+def markdown_table(rows: List[Dict[str, Any]]) -> str:
+ if not rows:
+ return "_无数据_"
+ columns = list(rows[0].keys())
+ output = [
+ "| " + " | ".join(columns) + " |",
+ "| " + " | ".join("---" for _ in columns) + " |",
]
- exception_counts: Dict[str, int] = {}
- error_counts: Dict[str, int] = {}
- for event in filtered:
- if event.exception_type:
- exception_counts[event.exception_type] = (
- exception_counts.get(event.exception_type, 0) + 1
- )
- key = event.error_code or event.exception_type
- if key:
- error_counts[key] = error_counts.get(key, 0) + 1
- return {
- "requests": len(filtered),
- "successes": sum(1 for event in filtered if event.success),
- "failures": sum(1 for event in filtered if not event.success),
- "timeouts": sum(1 for event in filtered if event.timeout),
- "exception_counts": exception_counts,
- "error_counts": error_counts,
- }
+ for row in rows:
+ output.append("| " + " | ".join(format_cell(row.get(col)) for col in columns) + " |")
+ return "\n".join(output)
-def format_phase_name_cn(phase: str) -> str:
- mapping = {
- "setup": "预热",
- "baseline": "基线阶段",
- "mixed_load": "混合压测阶段",
- "recovery": "恢复阶段",
- "drain": "收尾等待阶段",
- "cleanup": "清理阶段",
- "ALL": "全程",
- }
- return mapping.get(phase, phase)
+def pick(row: Dict[str, Any], columns: List[str]) -> Dict[str, Any]:
+ return {column: row.get(column) for column in columns}
-def format_seconds(value: Optional[float]) -> str:
+def format_cell(value: Any) -> str:
if value is None:
return "n/a"
- return f"{value:.1f}s"
+ text = str(value)
+ return text.replace("|", "\\|").replace("\n", " ")
-def format_percent(value: Optional[float]) -> str:
- if value is None:
- return "n/a"
- return f"{value:.2f}%"
+def find_summary(
+ rows: List[Dict[str, Any]], *, adapter: str, scenario: str, operation: str
+) -> Optional[Dict[str, Any]]:
+ return next(
+ (
+ row
+ for row in rows
+ if row["adapter"] == adapter
+ and row["scenario"] == scenario
+ and row["operation"] == operation
+ ),
+ None,
+ )
-def format_delta_percent(value: Optional[float]) -> str:
- if value is None:
- return "n/a"
- sign = "+" if value >= 0 else ""
- return f"{sign}{value:.2f}%"
+def estimate_event_span_seconds(events: List[RequestEvent]) -> float:
+ if len(events) < 2:
+ return 1e-9
+ starts = [event.elapsed_ms_since_run_start for event in events]
+ ends = [event.elapsed_ms_since_run_start + event.latency_ms for event in events]
+ return max((max(ends) - min(starts)) / 1000.0, 1e-9)
-def format_delta_seconds(value: Optional[float]) -> str:
- if value is None:
- return "n/a"
- sign = "+" if value >= 0 else ""
- return f"{sign}{value:.1f}s"
+def total_span(events: List[RequestEvent]) -> float:
+ if not events:
+ return 1e-9
+ starts = [event.elapsed_ms_since_run_start for event in events]
+ ends = [event.elapsed_ms_since_run_start + event.latency_ms for event in events]
+ return max((max(ends) - min(starts)) / 1000.0, 1e-9)
-def format_change(old: Optional[float], new: Optional[float], *, unit: str = "ms") -> str:
- if old is None or new is None:
- return "n/a"
- if unit == "ms":
- return (
- f"{old:.2f}{unit} -> {new:.2f}{unit} ({format_delta_percent(percent_change(old, new))})"
- )
- return f"{old:.2f} -> {new:.2f} ({format_delta_percent(percent_change(old, new))})"
+def percent_change(old: Optional[float], new: Optional[float]) -> Optional[float]:
+ if old is None or new is None or old == 0:
+ return None
+ return ((new - old) / old) * 100.0
-def format_qps_change(old: Optional[float], new: Optional[float]) -> str:
- if old is None or new is None:
+def fmt_delta_percent(value: Optional[float]) -> str:
+ if value is None:
return "n/a"
- return f"{old:.2f} -> {new:.2f} ({format_delta_percent(percent_change(old, new))})"
+ sign = "+" if value >= 0 else ""
+ return f"{sign}{value:.2f}%"
-def render_human_summary_zh(
- *,
- config: BenchmarkConfig,
- output_dir: str,
- notes: List[str],
- phase_metadata: List[PhaseMetadata],
- request_summary_rows: List[Dict[str, Any]],
- request_events: List[RequestEvent],
- task_summary_rows: List[Dict[str, Any]],
- task_events: List[CommitTaskEvent],
-) -> str:
- lines: List[str] = []
- lines.append("=== OpenViking Session 竞争压测摘要 ===")
- lines.append(f"结果目录: {output_dir}")
+def fmt_ms(value: Optional[float]) -> str:
+ return "n/a" if value is None else f"{value:.2f}ms"
- if notes:
- lines.append("")
- lines.append("说明:")
- for note in notes:
- lines.append(f"- {note}")
- phase_rows = build_phase_overview_rows(config, phase_metadata)
- baseline_find = find_request_summary_row(request_summary_rows, api="find", phase="baseline")
- mixed_find = find_request_summary_row(request_summary_rows, api="find", phase="mixed_load")
- recovery_find = find_request_summary_row(request_summary_rows, api="find", phase="recovery")
- mixed_add = find_request_summary_row(
- request_summary_rows, api="add_message", phase="mixed_load"
- )
- mixed_commit = find_request_summary_row(request_summary_rows, api="commit", phase="mixed_load")
- mixed_extract = find_request_summary_row(
- request_summary_rows, api="extract", phase="mixed_load"
- )
- baseline_status = find_request_summary_row(
- request_summary_rows, api="system_status", phase="baseline"
- )
- mixed_status = find_request_summary_row(
- request_summary_rows, api="system_status", phase="mixed_load"
- )
- baseline_queue = find_request_summary_row(
- request_summary_rows, api="observer_queue", phase="baseline"
- )
- mixed_queue = find_request_summary_row(
- request_summary_rows, api="observer_queue", phase="mixed_load"
- )
- find_delta = build_find_phase_delta(request_summary_rows)
- extract_breakdown = build_api_error_breakdown(request_events, api="extract", phase="mixed_load")
- completed_tasks = next(
- (row for row in task_summary_rows if row["status"] == "completed"),
- None,
- )
- incomplete_tasks = next(
- (row for row in task_summary_rows if row["status"] == "incomplete"),
- None,
- )
- total_task_count = len(task_events)
+def round_optional(value: Optional[float], digits: int = 4) -> Optional[float]:
+ if value is None:
+ return None
+ return round(float(value), digits)
- lines.append("")
- lines.append("一、核心结论")
- if baseline_find and mixed_find and find_delta:
- lines.append(
- "- 已明确复现读接口退化:`find` 在混合压测阶段的 p95 从 "
- f"{baseline_find['p95_ms']:.2f}ms 升到 {mixed_find['p95_ms']:.2f}ms,"
- f"增幅 {find_delta['p95_delta_percent']:.2f}%;p99 从 "
- f"{baseline_find['p99_ms']:.2f}ms 升到 {mixed_find['p99_ms']:.2f}ms,"
- f"增幅 {find_delta['p99_delta_percent']:.2f}%。"
- )
- lines.append(
- "- `find` 吞吐也下降了:QPS 从 "
- f"{baseline_find['qps']:.2f} 降到 {mixed_find['qps']:.2f},"
- f"变化 {format_delta_percent(percent_change(baseline_find['qps'], mixed_find['qps']))}。"
- )
- if recovery_find and baseline_find and mixed_find:
- lines.append(
- "- 恢复阶段没有完全回到基线:`find` p95 为 "
- f"{recovery_find['p95_ms']:.2f}ms,仍高于基线 "
- f"{format_delta_percent(percent_change(baseline_find['p95_ms'], recovery_find['p95_ms']))};"
- "但相比混合压测阶段已经有明显回落。"
- )
- if mixed_extract:
- lines.append(
- "- 长尾压力主要来自 `extract`:混合压测阶段共 "
- f"{mixed_extract['requests']} 次调用,成功率 {mixed_extract['success_rate']:.2f}%,"
- f"p95 {mixed_extract['p95_ms']:.2f}ms。"
- )
- if extract_breakdown["timeouts"] > 0:
- lines.append(
- "- `extract` 失败几乎全是客户端超时:"
- f"{extract_breakdown['timeouts']}/{extract_breakdown['requests']} 次超时,"
- f"主异常是 {format_top_counts(extract_breakdown['exception_counts'])}。"
- )
- if mixed_commit and completed_tasks:
- lines.append(
- "- `commit` 接口本身不是最重的部分:前台 `commit` p95 只有 "
- f"{mixed_commit['p95_ms']:.2f}ms;真正重的是后台任务,已完成任务的后台 p95 达 "
- f"{completed_tasks['p95_server_duration_ms']:.2f}ms。"
- )
- if incomplete_tasks:
- lines.append(
- "- 后台积压明显:本次共跟踪到 "
- f"{total_task_count} 个 `commit` 背景任务,其中 {incomplete_tasks['tasks']} 个在压测结束"
- "并等待 drain 后仍未完成。"
- )
- lines.append("")
- lines.append("二、阶段时长")
- for row in phase_rows:
- extra = ""
- if row["delta_seconds"] is not None and row["delta_seconds"] > 1:
- extra = ",实际时长明显长于目标值,通常说明脚本在等待 in-flight 会话周期收尾"
- lines.append(
- f"- {format_phase_name_cn(row['phase'])}: 目标 {format_seconds(row['target_seconds'])},"
- f"实际 {format_seconds(row['actual_seconds'])},偏差 {format_delta_seconds(row['delta_seconds'])}{extra}"
- )
+def to_float(value: Any) -> Optional[float]:
+ try:
+ if value is None:
+ return None
+ return float(value)
+ except (TypeError, ValueError):
+ return None
- lines.append("")
- lines.append("三、关键指标对比")
- if baseline_find and mixed_find and recovery_find:
- lines.append(
- "- `find`:"
- f" 基线 p95={baseline_find['p95_ms']:.2f}ms / p99={baseline_find['p99_ms']:.2f}ms / qps={baseline_find['qps']:.2f};"
- f" 压测中 p95={mixed_find['p95_ms']:.2f}ms / p99={mixed_find['p99_ms']:.2f}ms / qps={mixed_find['qps']:.2f};"
- f" 恢复期 p95={recovery_find['p95_ms']:.2f}ms / p99={recovery_find['p99_ms']:.2f}ms / qps={recovery_find['qps']:.2f}。"
- )
- if mixed_add:
- lines.append(
- "- `add_message`: 混合压测阶段 "
- f"requests={mixed_add['requests']},p50={mixed_add['p50_ms']:.2f}ms,"
- f"p95={mixed_add['p95_ms']:.2f}ms,p99={mixed_add['p99_ms']:.2f}ms。"
- )
- if mixed_commit:
- lines.append(
- "- `commit`: 混合压测阶段 "
- f"requests={mixed_commit['requests']},p50={mixed_commit['p50_ms']:.2f}ms,"
- f"p95={mixed_commit['p95_ms']:.2f}ms,p99={mixed_commit['p99_ms']:.2f}ms。"
- )
- if mixed_extract:
- lines.append(
- "- `extract`: 混合压测阶段 "
- f"requests={mixed_extract['requests']},success_rate={mixed_extract['success_rate']:.2f}%,"
- f"timeouts={extract_breakdown['timeouts']},p95={mixed_extract['p95_ms']:.2f}ms。"
- )
- if completed_tasks:
- lines.append(
- "- `commit` 背景任务(completed):"
- f" tasks={completed_tasks['tasks']},p50={format_metric(completed_tasks['p50_server_duration_ms'])},"
- f" p95={format_metric(completed_tasks['p95_server_duration_ms'])},"
- f" p99={format_metric(completed_tasks['p99_server_duration_ms'])}。"
- )
- if incomplete_tasks:
- lines.append(
- "- `commit` 背景任务(incomplete):"
- f" tasks={incomplete_tasks['tasks']},本地等待 p95={format_metric(incomplete_tasks['p95_local_duration_ms'])}。"
- )
- if baseline_status and mixed_status:
- lines.append(
- "- `system_status`: p95 "
- f"{format_change(baseline_status['p95_ms'], mixed_status['p95_ms'])}。"
- )
- if baseline_queue and mixed_queue:
- lines.append(
- "- `observer_queue`: p95 "
- f"{format_change(baseline_queue['p95_ms'], mixed_queue['p95_ms'])}。"
- )
- lines.append("")
- lines.append("四、怎么理解这次结果")
- lines.append(
- "- `find` 没有报错,但延迟和吞吐同时变差,这比“报错”更说明问题:读请求被明显挤压了。"
- )
- lines.append("- `extract` 的大量 30 秒超时说明长尾请求已经被稳定制造出来了,压测目标基本达成。")
- lines.append(
- "- `commit` 前台接口看起来还好,但后台任务非常慢,说明资源竞争更可能发生在后续提取/索引阶段,而不是 HTTP 返回这一步。"
- )
- lines.append(
- "- 如果你要拿这次结果给别人看,最应该盯的是三组数字:"
- "`find` 基线 vs 压测 p95/p99、`extract` 超时比例、`commit` 背景任务完成时长。"
- )
+def truncate_error_message(message: Optional[str]) -> Optional[str]:
+ if message is None:
+ return None
+ if len(message) <= MAX_ERROR_MESSAGE_LEN:
+ return message
+ return message[:MAX_ERROR_MESSAGE_LEN] + "...[truncated]"
- return "\n".join(lines)
+def extract_error_message(value: Any) -> Optional[str]:
+ if not isinstance(value, dict):
+ return None
+ error = value.get("error")
+ if isinstance(error, dict):
+ message = error.get("message") or error.get("code")
+ return str(message) if message is not None else None
+ if error:
+ return str(error)
+ return None
-def format_top_counts(counts: Dict[str, int], limit: int = 3) -> str:
- if not counts:
- return "无"
- ordered = sorted(counts.items(), key=lambda item: (-item[1], item[0]))
- return ", ".join(f"{key}={value}" for key, value in ordered[:limit])
+def is_not_found_error(exc: Exception) -> bool:
+ text = str(exc).lower()
+ return "not_found" in text or "not found" in text
-def percent_change(old: float, new: float) -> float:
- if old == 0:
- return 0.0 if new == 0 else 100.0
- return ((new - old) / old) * 100.0
+def is_already_exists_error(exc: Exception) -> bool:
+ text = str(exc).lower()
+ return "already_exists" in text or "already exists" in text or "exist" in text
-def round_optional(value: Optional[float], ndigits: int = 4) -> Optional[float]:
- if value is None:
- return None
- return round(value, ndigits)
+def utc_now() -> str:
+ return datetime.now(UTC).isoformat(timespec="milliseconds").replace("+00:00", "Z")
-def write_json(path: Path, data: Any) -> None:
- with path.open("w", encoding="utf-8") as handle:
- json.dump(data, handle, indent=2, ensure_ascii=False)
+def write_json(path: Path, value: Any) -> None:
+ path.write_text(json.dumps(value, ensure_ascii=False, indent=2), encoding="utf-8")
-def write_text(path: Path, content: str) -> None:
- path.write_text(content, encoding="utf-8")
+
+def write_text(path: Path, value: str) -> None:
+ path.write_text(value, encoding="utf-8")
-def write_jsonl(path: Path, rows: Iterable[Any]) -> None:
+def write_jsonl(path: Path, values: Iterable[Any]) -> None:
with path.open("w", encoding="utf-8") as handle:
- for row in rows:
- if hasattr(row, "to_dict"):
- row = row.to_dict()
- handle.write(json.dumps(row, ensure_ascii=False) + "\n")
+ for value in values:
+ payload = value.to_dict() if hasattr(value, "to_dict") else value
+ handle.write(json.dumps(payload, ensure_ascii=False) + "\n")
def write_csv(path: Path, rows: List[Dict[str, Any]]) -> None:
if not rows:
path.write_text("", encoding="utf-8")
return
- fieldnames = list(rows[0].keys())
with path.open("w", encoding="utf-8", newline="") as handle:
- writer = csv.DictWriter(handle, fieldnames=fieldnames)
+ writer = csv.DictWriter(handle, fieldnames=list(rows[0].keys()))
writer.writeheader()
writer.writerows(rows)
-def extract_boolean(body: Optional[Dict[str, Any]], *keys: str) -> Optional[bool]:
- current: Any = body
- for key in keys:
- if not isinstance(current, dict):
- return None
- current = current.get(key)
- return current if isinstance(current, bool) else None
+def parse_args(argv: Optional[List[str]] = None) -> BenchmarkConfig:
+ parser = argparse.ArgumentParser(
+ description="Run SDK/CLI mixed load benchmark against a running OpenViking server."
+ )
+ parser.add_argument(
+ "--server-url",
+ default=os.getenv("OPENVIKING_SERVER_URL", "http://127.0.0.1:1935"),
+ )
+ parser.add_argument("--api-key", default=os.getenv("OPENVIKING_API_KEY", "test-root-api-key"))
+ parser.add_argument("--account", default=os.getenv("OPENVIKING_ACCOUNT", "default"))
+ parser.add_argument("--user", default=os.getenv("OPENVIKING_USER", "default"))
+ parser.add_argument("--agent-id", default=os.getenv("OPENVIKING_AGENT_ID"))
+ parser.add_argument("--timeout", type=float, default=60.0)
+ parser.add_argument(
+ "--adapters",
+ default="sdk,cli-http,cli-subprocess",
+ help="Comma separated: sdk,cli-http,cli-subprocess",
+ )
+ parser.add_argument("--profile", choices=sorted(PROFILE_DEFAULTS), default="standard")
+ parser.add_argument("--resource-count", type=int)
+ parser.add_argument("--session-count", type=int)
+ parser.add_argument("--messages-per-session", type=int)
+ parser.add_argument("--resource-concurrency", type=int)
+ parser.add_argument("--search-concurrency", type=int)
+ parser.add_argument("--session-concurrency", type=int)
+ parser.add_argument("--commit-concurrency", type=int)
+ parser.add_argument("--mixed-concurrency", type=int)
+ parser.add_argument("--phase-seconds", type=float)
+ parser.add_argument("--mixed-seconds", type=float)
+ parser.add_argument("--warmup-seconds", type=float)
+ parser.add_argument("--drain-timeout", type=float, default=60.0)
+ parser.add_argument("--task-poll-interval", type=float, default=1.0)
+ parser.add_argument("--request-window-seconds", type=float, default=5.0)
+ parser.add_argument("--data-root-uri", default=DEFAULT_DATA_ROOT_URI)
+ default_results_root = (
+ Path(__file__).resolve().parents[1] / "results" / "openviking_server_load"
+ )
+ parser.add_argument("--local-data-dir", default=str(default_results_root / "data"))
+ parser.add_argument(
+ "--output-dir",
+ default=str(default_results_root / datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")),
+ )
+ parser.add_argument("--clear-before-run", action=argparse.BooleanOptionalAction, default=True)
+ parser.add_argument("--cleanup-at-end", action="store_true")
+ parser.add_argument("--ov-bin", default=os.getenv("OV_BIN", "ov"))
+ parser.add_argument("--seed", type=int, default=42)
+ parser.add_argument("--find-limit", type=int, default=10)
+ args = parser.parse_args(argv)
+ profile = PROFILE_DEFAULTS[args.profile]
-def to_float(value: Any) -> Optional[float]:
- if isinstance(value, (float, int)):
- return float(value)
- return None
+ def get_int(name: str) -> int:
+ return max(
+ 0, int(getattr(args, name) if getattr(args, name) is not None else profile[name])
+ )
+ def get_float(name: str) -> float:
+ return max(
+ 0.0, float(getattr(args, name) if getattr(args, name) is not None else profile[name])
+ )
-def format_metric(value: Optional[float]) -> str:
- if value is None:
- return "n/a"
- return f"{value:.2f}ms"
+ adapters = [item.strip() for item in args.adapters.split(",") if item.strip()]
+ allowed = {"sdk", "cli-http", "cli-subprocess"}
+ invalid = sorted(set(adapters) - allowed)
+ if invalid:
+ parser.error(f"Unsupported adapters: {', '.join(invalid)}")
+ if not adapters:
+ parser.error("--adapters must not be empty")
+
+ return BenchmarkConfig(
+ server_url=args.server_url.rstrip("/"),
+ api_key=args.api_key or None,
+ account=args.account or None,
+ user=args.user or None,
+ agent_id=args.agent_id or None,
+ timeout=max(0.1, args.timeout),
+ adapters=adapters,
+ profile=args.profile,
+ resource_count=get_int("resource_count"),
+ session_count=max(1, get_int("session_count")),
+ messages_per_session=max(1, get_int("messages_per_session")),
+ resource_concurrency=max(1, get_int("resource_concurrency")),
+ search_concurrency=max(1, get_int("search_concurrency")),
+ session_concurrency=max(1, get_int("session_concurrency")),
+ commit_concurrency=max(1, get_int("commit_concurrency")),
+ mixed_concurrency=max(1, get_int("mixed_concurrency")),
+ phase_seconds=get_float("phase_seconds"),
+ mixed_seconds=get_float("mixed_seconds"),
+ warmup_seconds=get_float("warmup_seconds"),
+ drain_timeout=max(0.0, args.drain_timeout),
+ task_poll_interval=max(0.1, args.task_poll_interval),
+ request_window_seconds=max(1.0, args.request_window_seconds),
+ local_data_dir=args.local_data_dir,
+ output_dir=args.output_dir,
+ data_root_uri=args.data_root_uri.rstrip("/"),
+ session_prefix=DEFAULT_SESSION_PREFIX,
+ clear_before_run=bool(args.clear_before_run),
+ cleanup_at_end=bool(args.cleanup_at_end),
+ ov_bin=args.ov_bin,
+ seed=args.seed,
+ find_limit=max(1, args.find_limit),
+ )
async def async_main(argv: Optional[List[str]] = None) -> int:
@@ -1661,11 +1958,7 @@ async def async_main(argv: Optional[List[str]] = None) -> int:
def main(argv: Optional[List[str]] = None) -> int:
- try:
- return asyncio.run(async_main(argv))
- except KeyboardInterrupt:
- print("\n[stopped] benchmark interrupted by user", file=sys.stderr)
- return 130
+ return asyncio.run(async_main(argv))
if __name__ == "__main__":