Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
feat(eval): Locomo bot eval add check (#1629)
yeshion23333ce4238955838
diff --git a/benchmark/locomo/README.md b/benchmark/locomo/README.mdindex 2912accc7..ed3c676c4 100644--- a/benchmark/locomo/README.md+++ b/benchmark/locomo/README.md@@ -29,6 +29,29 @@ benchmark/locomo/## VikingBot 评测流程+### 前置配置说明+- vikingbot评测须确保 OpenViking 服务端已配置 root_api_key,即开启多租户模式。每个sample数据都会使用sample_id如`conv-26`作为user_id,存储在OpenViking中。+```json+{+ "server": {+ "root_api_key": "your-key"+ }+}+```+- OpenViking数据导入account会优先使用`ovcli.conf`中的account值,若未配置默认使用`default`;+- vikingbot必须配置OpenViking 的root级别API KEY,默认使用上述的server.root_api_key,也可单独配置;+- vikingbot查询数据account默认为`default`,如更改必须与导入OpenViking的account一致,即`ovcli.conf`中的account值,可通过`ov.conf`如下配置:+```json+{+ "bot": {+ "ov_server": {+ "root_api_key": "your-root-key",+ "account_id": "默认default,必须和ovcli.conf中的account_id一致"+ }+ }+}+```+### 完整一键评测使用 `run_full_eval.sh` 可以一键运行完整评测流程:@@ -39,8 +62,6 @@ bash run_full_eval.sh # 完整流程bash run_full_eval.sh --skip-import # 跳过导入,仅评测```-该脚本会依次执行以下四个步骤:-### 单题/批量测试使用 `import_and_eval_one.sh` 可以快速测试单个问题或批量测试某个 sample:@@ -80,6 +101,7 @@ python import_to_ov.py --input <数据文件路径> [选项]- `--force-ingest`: 强制重新导入,即使已导入过- `--clear-ingest-record`: 清除所有导入记录- `--openviking-url`: OpenViking 服务地址,默认 `http://localhost:1933`+- `--account`: 导入时使用的 account,默认 `default`**示例:**```bash@@ -404,6 +426,9 @@ A: 使用 `--force-ingest`(导入)或删除结果 CSV 文件。### Q: 评测速度慢怎么办?A: 增加 `--threads`(run_eval.py)或 `--parallel`(其他脚本)参数值。+### Q: 评测效果低,怎么排查 OpenViking 导入与评测查询的 account/user 是否一致?+A: 先核对三处是否对齐:`ovcli.conf.account`(导入 account)、`ov.conf.bot.ov_server.account_id`(Vikingbot 查询 account)、评测脚本使用的 user(Vikingbot 按 `sample_id`,OpenClaw 默认 `eval-1`)。这几项不一致时,常见现象是“导入看起来成功,但评测回答质量明显下降或查不到上下文”。+---## 常见问题排查@@ -461,7 +486,23 @@ grep -i "error" result/qa_results.csv**预期结果**:在用户提问(query)之前,应该有从 OpenViking 加载的记忆内容。-### 4. Token 统计异常+### 4. 评测效果低时,先口语化排查 account/user++如果你感觉“明明导入了,回答还是不对劲”,先按这个顺序看:++1. 打开 `~/.openviking/ovcli.conf`,看 `account` 是不是你这次要用的账号。+2. 打开 `~/.openviking/ov.conf`,重点看:+ - `bot.ov_server.account_id`+ - `server.host` / `server.port`+3. 跑 Vikingbot 脚本时留意 preflight 日志里打印的 `account` 和 `OpenViking URL`,确认和你配置里看到的一致。+4. 记住查询侧是谁在查:+ - Vikingbot 评测默认用 `sample_id` 当 user。+ - OpenClaw QA 默认是 `--user eval-1 --agent-id locomo-eval`。+ - 你如果改过 OpenClaw 的 `--user` 或 `--agent-id`,要保证 ingest 和 qa 两边用的是同一套值。++一句话:导入时的 account/user、评测时的 account/user、以及连接的服务地址,这三件事只要有一个没对齐,就很容易出现效果低或“查不到上下文”。++### 5. Token 统计异常如果 `stat_judge_result.py` 输出的 token 数量异常:diff --git a/benchmark/locomo/vikingbot/import_and_eval_one.sh b/benchmark/locomo/vikingbot/import_and_eval_one.shindex 3289fcb14..dcc3a4a1e 100755--- a/benchmark/locomo/vikingbot/import_and_eval_one.sh+++ b/benchmark/locomo/vikingbot/import_and_eval_one.sh@@ -4,15 +4,80 @@# Usage:# ./import_and_eval_one.sh 0 2 # sample 0, question 2 (单题)# ./import_and_eval_one.sh conv-26 2 # sample_id conv-26, question 2 (单题)-# ./import_and_eval_one.sh conv-26 # sample_id conv-26, 所有问题 (批量)-# ./import_and_eval_one.sh conv-26 2 --skip-import # 跳过导入,直接评测-# ./import_and_eval_one.sh conv-26 --skip-import # 跳过导入,批量评测+# ./import_and_eval_one.sh conv-26 # sample_id conv-26, 所有问题 (批量)+# ./import_and_eval_one.sh conv-26 2 --skip-import # 跳过导入,直接评测+# ./import_and_eval_one.sh conv-26 --skip-import # 跳过导入,批量评测set -eSCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"SKIP_IMPORT=false+if command -v python3 >/dev/null 2>&1; then+ PYTHON_BIN="python3"+elif command -v python >/dev/null 2>&1; then+ PYTHON_BIN="python"+else+ echo "未找到 python3/python,请先安装 Python。" >&2+ exit 1+fi++DEFAULT_OV_CONF_PATH="$($PYTHON_BIN - <<'PY'+from pathlib import Path++from openviking_cli.utils.config.config_loader import resolve_config_path+from openviking_cli.utils.config.consts import DEFAULT_OV_CONF, OPENVIKING_CONFIG_ENV++path = resolve_config_path(None, OPENVIKING_CONFIG_ENV, DEFAULT_OV_CONF)+print(str(path) if path is not None else str(Path.home() / ".openviking" / "ov.conf"))+PY+)"++if [ -t 0 ] && [ -t 1 ]; then+ echo "[preflight] OpenViking 配置默认路径: $DEFAULT_OV_CONF_PATH"+ printf "[preflight] 直接回车使用默认,或输入新路径 [%s]: " "$DEFAULT_OV_CONF_PATH"+ if ! read -r OV_CONF_PATH < /dev/tty; then+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"+ fi+ if [ -z "$OV_CONF_PATH" ]; then+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"+ fi+else+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"+fi++if [ "$OV_CONF_PATH" = "~" ]; then+ OV_CONF_PATH="$HOME"+elif [[ "$OV_CONF_PATH" == ~/* ]]; then+ OV_CONF_PATH="$HOME/${OV_CONF_PATH#~/}"+fi++export OPENVIKING_CONFIG_FILE="$OV_CONF_PATH"+echo "[preflight] 本次使用 ov.conf: $OPENVIKING_CONFIG_FILE"++# 评测前预检配置+PRECHECK_STATUS=0+"$PYTHON_BIN" "$SCRIPT_DIR/preflight_eval_config.py" || PRECHECK_STATUS=$?+if [ "$PRECHECK_STATUS" -ne 0 ]; then+ if [ "$PRECHECK_STATUS" -eq 2 ]; then+ echo "[preflight] 已完成 root_api_key 初始化,请先重启 openviking-server,再重新执行评测脚本。" >&2+ fi+ exit "$PRECHECK_STATUS"+fi++RUNTIME_ENV_FILE="$(mktemp "${TMPDIR:-/tmp}/ov_eval_runtime.XXXXXX")"+trap 'rm -f "$RUNTIME_ENV_FILE"' EXIT++if [ -t 0 ] && [ -t 1 ]; then+ INTERACTIVE=1+else+ INTERACTIVE=0+fi++INTERACTIVE="$INTERACTIVE" "$PYTHON_BIN" "$SCRIPT_DIR/preflight_eval_runtime.py" --output-env-file "$RUNTIME_ENV_FILE"+# shellcheck disable=SC1090+source "$RUNTIME_ENV_FILE"+# 解析参数for arg in "$@"; doif [ "$arg" = "--skip-import" ]; then@@ -47,16 +112,24 @@ if [[ "$SAMPLE" =~ ^-?[0-9]+$ ]]; thenecho "Using sample index: $SAMPLE_INDEX"else# 通过 sample_id 查找索引- SAMPLE_INDEX=$(python3 -c "+ SAMPLE_INDEX=$(SAMPLE="$SAMPLE" INPUT_FILE="$INPUT_FILE" "$PYTHON_BIN" - <<'PY'import json-data = json.load(open('$INPUT_FILE'))+import os++sample = os.environ["SAMPLE"]+input_file = os.environ["INPUT_FILE"]++with open(input_file, "r", encoding="utf-8") as f:+ data = json.load(f)+for i, s in enumerate(data):- if s.get('sample_id') == '$SAMPLE':+ if s.get("sample_id") == sample:print(i)breakelse:- print('NOT_FOUND')-")+ print("NOT_FOUND")+PY+)if [ "$SAMPLE_INDEX" = "NOT_FOUND" ]; thenecho "Error: sample_id '$SAMPLE' not found"exit 1@@ -75,11 +148,13 @@ if [ -n "$QUESTION_INDEX" ]; thenecho "[1/3] Skipping import (--skip-import)"elseecho "[1/3] Importing sample $SAMPLE_INDEX, question $QUESTION_INDEX..."- python benchmark/locomo/vikingbot/import_to_ov.py \+ "$PYTHON_BIN" "$SCRIPT_DIR/import_to_ov.py" \--input "$INPUT_FILE" \--sample "$SAMPLE_INDEX" \--question-index "$QUESTION_INDEX" \- --force-ingest+ --force-ingest \+ --account "$ACCOUNT" \+ --openviking-url "$OPENVIKING_URL"echo "Waiting for data processing..."sleep 3@@ -94,7 +169,7 @@ if [ -n "$QUESTION_INDEX" ]; thenif [[ "$SAMPLE" =~ ^-?[0-9]+$ ]]; then# 数字索引用默认输出文件OUTPUT_FILE=./result/locomo_qa_result.csv- python benchmark/locomo/vikingbot/run_eval.py \+ "$PYTHON_BIN" "$SCRIPT_DIR/run_eval.py" \"$INPUT_FILE" \--sample "$SAMPLE_ID_FOR_CMD" \--question-index "$QUESTION_INDEX" \@@ -102,7 +177,7 @@ if [ -n "$QUESTION_INDEX" ]; thenelse# sample_id 模式直接更新批量结果文件OUTPUT_FILE=./result/locomo_${SAMPLE}_result.csv- python benchmark/locomo/vikingbot/run_eval.py \+ "$PYTHON_BIN" "$SCRIPT_DIR/run_eval.py" \"$INPUT_FILE" \--sample "$SAMPLE_ID_FOR_CMD" \--question-index "$QUESTION_INDEX" \@@ -117,55 +192,62 @@ if [ -n "$QUESTION_INDEX" ]; thenelseecho "[3/3] Running judge..."fi- python benchmark/locomo/vikingbot/judge.py --input "$OUTPUT_FILE" --parallel 1+ "$PYTHON_BIN" "$SCRIPT_DIR/judge.py" --input "$OUTPUT_FILE" --parallel 1# 输出结果echo ""echo "=== 评测结果 ==="- python3 -c "+ OUTPUT_FILE="$OUTPUT_FILE" QUESTION_INDEX="$QUESTION_INDEX" "$PYTHON_BIN" - <<'PY'import csvimport json+import os-question_index = $QUESTION_INDEX+question_index = int(os.environ["QUESTION_INDEX"])+output_file = os.environ["OUTPUT_FILE"]-with open('$OUTPUT_FILE') as f:+with open(output_file, "r", encoding="utf-8") as f:reader = csv.DictReader(f)rows = list(reader)-# 找到指定 question_index 的结果row = Nonefor r in rows:- if int(r.get('question_index', -1)) == question_index:+ if int(r.get("question_index", -1)) == question_index:row = rbreakif row is None:- # 没找到则用最后一条row = rows[-1]-# 解析 evidence_text-evidence_text = json.loads(row.get('evidence_text', '[]'))-evidence_str = '\\n'.join(evidence_text) if evidence_text else ''+evidence_text = json.loads(row.get("evidence_text", "[]"))+evidence_str = "\n".join(evidence_text) if evidence_text else ""-print(f\"问题: {row['question']}\")-print(f\"期望答案: {row['answer']}\")-print(f\"模型回答: {row['response']}\")-print(f\"证据原文:\\n{evidence_str}\")-print(f\"结果: {row.get('result', 'N/A')}\")-print(f\"原因: {row.get('reasoning', 'N/A')}\")-"+print(f"问题: {row['question']}")+print(f"期望答案: {row['answer']}")+print(f"模型回答: {row['response']}")+print(f"证据原文:\n{evidence_str}")+print(f"结果: {row.get('result', 'N/A')}")+print(f"原因: {row.get('reasoning', 'N/A')}")+PYelse# ========== 批量模式 ==========echo "=== 批量模式: sample $SAMPLE, 所有问题 ==="# 获取该 sample 的问题数量- QUESTION_COUNT=$(python3 -c "+ QUESTION_COUNT=$(SAMPLE_INDEX="$SAMPLE_INDEX" INPUT_FILE="$INPUT_FILE" "$PYTHON_BIN" - <<'PY'import json-data = json.load(open('$INPUT_FILE'))-sample = data[$SAMPLE_INDEX]-print(len(sample.get('qa', [])))-")+import os++sample_index = int(os.environ["SAMPLE_INDEX"])+input_file = os.environ["INPUT_FILE"]++with open(input_file, "r", encoding="utf-8") as f:+ data = json.load(f)++sample = data[sample_index]+print(len(sample.get("qa", [])))+PY+)echo "Found $QUESTION_COUNT questions for sample $SAMPLE"# 导入所有 sessions@@ -173,10 +255,12 @@ print(len(sample.get('qa', [])))echo "[1/4] Skipping import (--skip-import)"elseecho "[1/4] Importing all sessions for sample $SAMPLE_INDEX..."- python benchmark/locomo/vikingbot/import_to_ov.py \+ "$PYTHON_BIN" "$SCRIPT_DIR/import_to_ov.py" \--input "$INPUT_FILE" \--sample "$SAMPLE_INDEX" \- --force-ingest+ --force-ingest \+ --account "$ACCOUNT" \+ --openviking-url "$OPENVIKING_URL"echo "Waiting for data processing..."sleep 10@@ -189,7 +273,7 @@ print(len(sample.get('qa', [])))echo "[2/4] Running evaluation for all questions..."fiOUTPUT_FILE=./result/locomo_${SAMPLE}_result.csv- python benchmark/locomo/vikingbot/run_eval.py \+ "$PYTHON_BIN" "$SCRIPT_DIR/run_eval.py" \"$INPUT_FILE" \--sample "$SAMPLE_ID_FOR_CMD" \--output "$OUTPUT_FILE" \@@ -201,7 +285,7 @@ print(len(sample.get('qa', [])))elseecho "[3/4] Running judge..."fi- python benchmark/locomo/vikingbot/judge.py --input "$OUTPUT_FILE" --parallel 5+ "$PYTHON_BIN" "$SCRIPT_DIR/judge.py" --input "$OUTPUT_FILE" --parallel 5# 输出统计结果if [ "$SKIP_IMPORT" = "true" ]; then@@ -209,9 +293,9 @@ print(len(sample.get('qa', [])))elseecho "[4/4] Calculating statistics..."fi- python benchmark/locomo/vikingbot/stat_judge_result.py --input "$OUTPUT_FILE"+ "$PYTHON_BIN" "$SCRIPT_DIR/stat_judge_result.py" --input "$OUTPUT_FILE"echo ""echo "=== 批量评测完成 ==="echo "结果文件: $OUTPUT_FILE"-fi\ No newline at end of file+fidiff --git a/benchmark/locomo/vikingbot/import_to_ov.py b/benchmark/locomo/vikingbot/import_to_ov.pyindex a6b23c461..c537186b1 100644--- a/benchmark/locomo/vikingbot/import_to_ov.py+++ b/benchmark/locomo/vikingbot/import_to_ov.py@@ -279,6 +279,7 @@ async def viking_ingest(session_time: Optional[str] = None,user_id: Optional[str] = None,agent_id: Optional[str] = None,+ account: str = "default",) -> Dict[str, int]:"""Save messages to OpenViking via OpenViking SDK client.Returns token usage dict with embedding and vlm token counts.@@ -289,6 +290,7 @@ async def viking_ingest(session_time: Session time string (e.g., "9:36 am on 2 April, 2023")user_id: User identifier for separate userspace (e.g., "conv-26")agent_id: Agent identifier for separate agentspace (e.g., "conv-26")+ account: OpenViking account identifier"""# 解析 session_time - 为每条消息计算递增的时间戳base_datetime = None@@ -303,6 +305,7 @@ async def viking_ingest(url=openviking_url,user=user_id,agent_id=agent_id,+ account=account,)await client.initialize()@@ -387,6 +390,7 @@ async def process_single_session(meta.get("date_time"),user_id=str(sample_id),agent_id=str(sample_id),+ account=args.account,)token_usage = result["token_usage"]task_id = result.get("task_id")@@ -668,6 +672,11 @@ def main():default="http://localhost:1933",help="OpenViking service URL (default: http://localhost:1933)",)+ parser.add_argument(+ "--account",+ default="default",+ help="OpenViking account identifier (default: default)",+ )parser.add_argument("--sample",type=int,diff --git a/benchmark/locomo/vikingbot/preflight_eval_config.py b/benchmark/locomo/vikingbot/preflight_eval_config.pynew file mode 100644index 000000000..10cc52213--- /dev/null+++ b/benchmark/locomo/vikingbot/preflight_eval_config.py@@ -0,0 +1,304 @@+#!/usr/bin/env python3++import json+import os+import sys+from pathlib import Path++from openviking_cli.utils.config.config_loader import resolve_config_path+from openviking_cli.utils.config.consts import (+ DEFAULT_OVCLI_CONF,+ DEFAULT_OV_CONF,+ OPENVIKING_CLI_CONFIG_ENV,+ OPENVIKING_CONFIG_ENV,+)++_USE_COLOR = hasattr(sys.stdout, "isatty") and sys.stdout.isatty()+++def _color(text: str, code: str) -> str:+ if not _USE_COLOR:+ return text+ return f"\033[{code}m{text}\033[0m"+++def _prefix() -> str:+ return _color("[preflight]", "36")+++def _log_info(message: str) -> None:+ print(f"{_prefix()} {_color('[INFO]', '34')} {message}")+++def _log_warn(message: str) -> None:+ print(f"{_prefix()} {_color('[WARN]', '33')} {message}")+++def _log_ok(message: str) -> None:+ print(f"{_prefix()} {_color('[OK]', '32')} {message}")+++def _log_error(message: str) -> None:+ print(f"{_prefix()} {_color('[ERROR]', '31')} {message}", file=sys.stderr)+++def _is_interactive() -> bool:+ return sys.stdin.isatty() and sys.stdout.isatty()+++def _prompt_text(prompt: str, default: str | None = None) -> str:+ suffix = f" [{default}]" if default else ""+ raw = input(f"{_prefix()} {prompt}{suffix}: ").strip()+ if not raw and default is not None:+ return default+ return raw+++def _prompt_confirm(prompt: str, default: bool = True) -> bool:+ hint = "Y/n" if default else "y/N"+ raw = input(f"{_prefix()} {prompt} [{hint}]: ").strip().lower()+ if not raw:+ return default+ return raw in {"y", "yes"}+++def _load_json(path: Path) -> dict:+ with open(path, "r", encoding="utf-8-sig") as f:+ raw = f.read()+ return json.loads(raw)+++def _backup_and_write_json(path: Path, data: dict) -> None:+ path.parent.mkdir(parents=True, exist_ok=True)+ if path.exists():+ backup = path.with_suffix(path.suffix + ".bak")+ with open(path, "r", encoding="utf-8") as src:+ original = src.read()+ with open(backup, "w", encoding="utf-8") as bak:+ bak.write(original)+ with open(path, "w", encoding="utf-8") as f:+ json.dump(data, f, indent=2, ensure_ascii=False)+ f.write("\n")+++def _resolve_ov_conf_path() -> Path:+ configured_path = os.environ.get(OPENVIKING_CONFIG_ENV, "").strip()+ if configured_path:+ return Path(configured_path).expanduser()++ resolved = resolve_config_path(None, OPENVIKING_CONFIG_ENV, DEFAULT_OV_CONF)+ default_path = str(resolved) if resolved is not None else str(Path.home() / ".openviking" / "ov.conf")++ if _is_interactive():+ _log_info(f"OpenViking 配置默认路径: {default_path}")+ chosen = _prompt_text("直接回车使用默认,或输入新路径", default=default_path)+ else:+ chosen = default_path+ return Path(chosen).expanduser()+++def _ensure_server_root_api_key(ov_data: dict) -> tuple[bool, bool]:+ server = ov_data.setdefault("server", {})+ root_key = str(server.get("root_api_key") or "").strip()+ if root_key:+ return True, False++ _log_warn("server.root_api_key 未配置。")+ if not _is_interactive():+ _log_error("非交互模式下无法录入 root_api_key,请先配置后重试。")+ return False, False++ while True:+ key = _prompt_text("请输入 OpenViking root_api_key")+ if key:+ server["root_api_key"] = key+ return True, True+ _log_warn("root_api_key 不能为空,请重新输入。")+++def _check_bot_root_key(ov_data: dict) -> tuple[bool, bool]:+ server_root_key = str(ov_data.get("server", {}).get("root_api_key") or "").strip()+ bot = ov_data.setdefault("bot", {})+ ov_server = bot.setdefault("ov_server", {})+ bot_root_key = str(ov_server.get("root_api_key") or "").strip()++ if not bot_root_key or bot_root_key == server_root_key:+ return True, False++ _log_warn("bot.ov_server.root_api_key 与 server.root_api_key 不一致。")+ _log_warn("vikingbot 必须使用 OpenViking 的 root 级 API key。")++ if not _is_interactive():+ _log_error("非交互模式下不会自动修改,请先修复配置后重试。")+ return False, False++ if _prompt_confirm("是否自动调整为 server.root_api_key", default=True):+ ov_server["root_api_key"] = server_root_key+ return True, True++ _log_error("已取消自动调整。")+ return False, False+++def _resolve_ovcli_path() -> Path:+ resolved = resolve_config_path(None, OPENVIKING_CLI_CONFIG_ENV, DEFAULT_OVCLI_CONF)+ if resolved is not None:+ return resolved+ return Path.home() / ".openviking" / "ovcli.conf"+++def _set_bot_account_id(ov_data: dict, account_id: str) -> bool:+ bot = ov_data.setdefault("bot", {})+ ov_server = bot.setdefault("ov_server", {})+ current = str(ov_server.get("account_id") or "").strip()+ if current == account_id:+ return False+ ov_server["account_id"] = account_id+ return True+++def _check_ovcli_keys_and_account(server_root_key: str, ov_data: dict) -> tuple[bool, bool, str]:+ ovcli_path = _resolve_ovcli_path()++ changed_ovcli = False+ changed_ov = False++ if not ovcli_path.exists():+ _log_warn(f"未找到 ovcli.conf: {ovcli_path}")+ if not _is_interactive():+ _log_error("非交互模式下不会创建 ovcli.conf。")+ return False, False, "default"+ if not _prompt_confirm("是否创建并写入 root key", default=True):+ _log_error("已取消创建 ovcli.conf。")+ return False, False, "default"+ ovcli_data = {+ "url": "http://localhost:1933",+ "api_key": server_root_key,+ "root_api_key": server_root_key,+ "account": "",+ "timeout": 60.0,+ }+ changed_ovcli = True+ else:+ try:+ ovcli_data = _load_json(ovcli_path)+ except Exception as exc:+ _log_error(f"读取 ovcli.conf 失败: {exc}")+ return False, False, "default"++ needs_key_update = False+ api_key = str(ovcli_data.get("api_key") or "").strip()+ if api_key != server_root_key:+ needs_key_update = True++ has_root_key_field = "root_api_key" in ovcli_data+ root_api_key = str(ovcli_data.get("root_api_key") or "").strip() if has_root_key_field else ""+ if has_root_key_field and root_api_key != server_root_key:+ needs_key_update = True++ if needs_key_update:+ _log_warn("ovcli.conf 的 key 与 server.root_api_key 不一致。")+ if not _is_interactive():+ _log_error("非交互模式下不会自动修改 ovcli.conf。")+ return False, False, "default"+ if not _prompt_confirm("是否自动更改为 server.root_api_key", default=True):+ _log_error("已取消自动更改 ovcli.conf。")+ return False, False, "default"+ ovcli_data["api_key"] = server_root_key+ if has_root_key_field:+ ovcli_data["root_api_key"] = server_root_key+ changed_ovcli = True+ _log_ok("ovcli.conf key 已同步为 server.root_api_key")++ selected_account = str(ovcli_data.get("account") or "").strip()+ if selected_account:+ if _is_interactive():+ use_current = _prompt_confirm(+ f"检测到 ovcli.conf.account={selected_account},是否使用该 account",+ default=True,+ )+ if not use_current:+ candidate = _prompt_text("请输入新的 account", default=selected_account).strip()+ if candidate:+ selected_account = candidate+ ovcli_data["account"] = selected_account+ changed_ovcli = True+ _log_ok(f"ovcli.conf account 已更新为: {selected_account}")+ _log_info(f"导入与评测将使用 account: {selected_account}")+ else:+ _log_warn("ovcli.conf 的 account 为空。")+ if _is_interactive():+ selected_account = _prompt_text(+ "将使用默认 Account=default;回车确认,或输入新 account",+ default="default",+ ).strip()+ if not selected_account:+ selected_account = "default"+ else:+ selected_account = "default"+ _log_info("将使用默认 Account=default 导入 OpenViking 数据和 Vikingbot 评测。")++ if selected_account != "default":+ ovcli_data["account"] = selected_account+ changed_ovcli = True+ _log_ok(f"ovcli.conf account 已更新为: {selected_account}")++ if _set_bot_account_id(ov_data, selected_account):+ changed_ov = True+ _log_ok(f"bot.ov_server.account_id 已同步为: {selected_account}")++ if changed_ovcli:+ _backup_and_write_json(ovcli_path, ovcli_data)+ _log_ok(f"已更新 {ovcli_path}")++ return True, changed_ov, selected_account+++def main() -> int:+ try:+ ov_conf_path = _resolve_ov_conf_path()+ if not ov_conf_path.exists():+ _log_error(f"ov.conf 不存在: {ov_conf_path}")+ return 1++ try:+ ov_data = _load_json(ov_conf_path)+ except Exception as exc:+ _log_error(f"读取 ov.conf 失败: {exc}")+ return 1++ ok, changed_server = _ensure_server_root_api_key(ov_data)+ if not ok:+ return 1++ ok, changed_bot = _check_bot_root_key(ov_data)+ if not ok:+ return 1++ if changed_server or changed_bot:+ _backup_and_write_json(ov_conf_path, ov_data)+ _log_ok(f"已更新 {ov_conf_path}")++ if changed_server:+ _log_warn("检测到 server.root_api_key 刚完成配置,请先重启 openviking-server。")+ _log_info("重启完成后请重新执行一键评测脚本。")+ return 2++ server_root_key = str(ov_data.get("server", {}).get("root_api_key") or "").strip()+ ok, changed_ov_account, _ = _check_ovcli_keys_and_account(server_root_key, ov_data)+ if not ok:+ return 1++ if changed_ov_account:+ _backup_and_write_json(ov_conf_path, ov_data)+ _log_ok(f"已更新 {ov_conf_path}")++ _log_ok("配置检查通过。")+ return 0+ except KeyboardInterrupt:+ _log_error("用户取消。")+ return 1+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmark/locomo/vikingbot/preflight_eval_runtime.py b/benchmark/locomo/vikingbot/preflight_eval_runtime.pynew file mode 100644index 000000000..cdf2516b8--- /dev/null+++ b/benchmark/locomo/vikingbot/preflight_eval_runtime.py@@ -0,0 +1,227 @@+#!/usr/bin/env python3++import argparse+import json+import os+import shlex+import sys+import urllib.error+import urllib.request+from pathlib import Path++from openviking_cli.utils.config.config_loader import resolve_config_path+from openviking_cli.utils.config.consts import (+ DEFAULT_OVCLI_CONF,+ DEFAULT_OV_CONF,+ OPENVIKING_CLI_CONFIG_ENV,+ OPENVIKING_CONFIG_ENV,+)+++def _log(message: str) -> None:+ print(f"[preflight] {message}")+++def _error(message: str) -> None:+ print(f"[preflight] {message}", file=sys.stderr)+++def _load_json(path: Path) -> dict:+ with open(path, "r", encoding="utf-8-sig") as f:+ return json.load(f)+++def _resolve_account() -> str:+ path = resolve_config_path(None, OPENVIKING_CLI_CONFIG_ENV, DEFAULT_OVCLI_CONF)+ if path is None:+ return "default"+ try:+ data = _load_json(Path(path))+ except Exception:+ return "default"+ account = str(data.get("account") or "").strip()+ return account or "default"+++def _resolve_openviking_url() -> str:+ host = "127.0.0.1"+ port = 1933++ path = resolve_config_path(None, OPENVIKING_CONFIG_ENV, DEFAULT_OV_CONF)+ if path is not None:+ try:+ data = _load_json(Path(path))+ server = data.get("server") or {}+ parsed_host = str(server.get("host") or "").strip()+ parsed_port = server.get("port")+ if parsed_host:+ host = parsed_host+ if isinstance(parsed_port, int):+ port = parsed_port+ elif isinstance(parsed_port, str) and parsed_port.strip().isdigit():+ port = int(parsed_port.strip())+ except Exception:+ pass++ return f"http://{host}:{port}"+++def _load_ov_conf() -> dict:+ ov_conf_path = resolve_config_path(None, OPENVIKING_CONFIG_ENV, DEFAULT_OV_CONF)+ if ov_conf_path is None:+ _error("未找到 ov.conf,无法读取 root_api_key。")+ raise SystemExit(1)++ try:+ return _load_json(Path(ov_conf_path))+ except Exception as exc:+ _error(f"读取 ov.conf 失败: {exc}")+ raise SystemExit(1)+++def _parse_accounts(body: str) -> list:+ try:+ payload = json.loads(body)+ except Exception as exc:+ _error(f"/api/v1/admin/accounts 返回非 JSON: {exc}")+ raise SystemExit(1)++ accounts = payload.get("result", payload)+ if not isinstance(accounts, list):+ _error("/api/v1/admin/accounts 返回格式异常。")+ raise SystemExit(1)+ return accounts+++def _account_exists(accounts: list, account: str) -> bool:+ for item in accounts:+ if isinstance(item, dict):+ account_id = str(item.get("account_id") or item.get("id") or "").strip()+ if account_id == account:+ return True+ elif isinstance(item, str) and item == account:+ return True+ return False+++def _prompt_confirm_create(account: str) -> bool:+ prompt = f"[preflight] account '{account}' 不存在,是否自动创建该 account? [Y/n]: "+ try:+ with open("/dev/tty", "r", encoding="utf-8") as tty_in:+ print(prompt, end="", flush=True)+ answer = tty_in.readline().strip().lower()+ except Exception as exc:+ _error(f"无法读取终端输入: {exc}")+ raise SystemExit(1)+ return answer in ("", "y", "yes")+++def _ensure_server_and_account_ready(url: str, account: str, interactive: bool) -> None:+ ov_data = _load_ov_conf()++ root_key = str((ov_data.get("server") or {}).get("root_api_key") or "").strip()+ if not root_key:+ _error("server.root_api_key 为空,无法执行服务连通性检查。")+ raise SystemExit(1)++ admin_user_id = str(+ ((ov_data.get("bot") or {}).get("ov_server") or {}).get("admin_user_id") or "default"+ ).strip() or "default"++ req = urllib.request.Request(+ f"{url}/api/v1/admin/accounts",+ headers={+ "X-API-Key": root_key,+ "Content-Type": "application/json",+ },+ method="GET",+ )++ try:+ with urllib.request.urlopen(req, timeout=10) as resp:+ body = resp.read().decode("utf-8", errors="replace")+ except urllib.error.HTTPError as e:+ detail = e.read().decode("utf-8", errors="replace")+ _error(f"OpenViking server 检查失败(HTTP {e.code}): {detail}")+ raise SystemExit(1)+ except Exception as exc:+ _error(f"OpenViking server 不可用: {exc}")+ raise SystemExit(1)++ accounts = _parse_accounts(body)+ if _account_exists(accounts, account):+ _log(f"OpenViking server 可用,account '{account}' 已就绪。")+ return++ if not interactive:+ _error(f"account '{account}' 不存在,非交互模式下不会自动创建。")+ raise SystemExit(1)++ if not _prompt_confirm_create(account):+ _error("已取消自动创建 account。")+ raise SystemExit(1)++ create_req = urllib.request.Request(+ f"{url}/api/v1/admin/accounts",+ headers={+ "X-API-Key": root_key,+ "Content-Type": "application/json",+ },+ data=json.dumps({"account_id": account, "admin_user_id": admin_user_id}).encode("utf-8"),+ method="POST",+ )++ try:+ with urllib.request.urlopen(create_req, timeout=10) as create_resp:+ create_body = create_resp.read().decode("utf-8", errors="replace")+ except urllib.error.HTTPError as e:+ detail = e.read().decode("utf-8", errors="replace")+ _error(f"创建 account 失败(HTTP {e.code}): {detail}")+ raise SystemExit(1)+ except Exception as exc:+ _error(f"创建 account 失败: {exc}")+ raise SystemExit(1)++ try:+ create_payload = json.loads(create_body)+ except Exception:+ create_payload = {}++ if isinstance(create_payload, dict) and create_payload.get("status") == "error":+ _error(f"创建 account 失败: {create_payload}")+ raise SystemExit(1)++ _log(f"已创建 account '{account}'(admin_user_id={admin_user_id})。")+ _log(f"OpenViking server 可用,account '{account}' 已就绪。")+++def _write_env_file(path: Path, account: str, openviking_url: str) -> None:+ with open(path, "w", encoding="utf-8") as f:+ f.write(f"ACCOUNT={shlex.quote(account)}\n")+ f.write(f"OPENVIKING_URL={shlex.quote(openviking_url)}\n")+++def main() -> int:+ parser = argparse.ArgumentParser(description="Resolve runtime eval account/url and validate OpenViking readiness")+ parser.add_argument("--output-env-file", required=True, help="File path to write ACCOUNT/OPENVIKING_URL exports")+ args = parser.parse_args()++ interactive_env = os.environ.get("INTERACTIVE", "").strip()+ if interactive_env:+ interactive = interactive_env == "1"+ else:+ interactive = sys.stdin.isatty() and sys.stdout.isatty()++ account = _resolve_account()+ openviking_url = _resolve_openviking_url()++ _log(f"本次导入与评测使用 account: {account}")+ _log(f"本次导入使用 OpenViking URL: {openviking_url}")++ _ensure_server_and_account_ready(openviking_url, account, interactive)+ _write_env_file(Path(args.output_env_file), account, openviking_url)+ return 0+++if __name__ == "__main__":+ raise SystemExit(main())diff --git a/benchmark/locomo/vikingbot/run_full_eval.sh b/benchmark/locomo/vikingbot/run_full_eval.shindex 08746e774..c9f0c9196 100755--- a/benchmark/locomo/vikingbot/run_full_eval.sh+++ b/benchmark/locomo/vikingbot/run_full_eval.sh@@ -2,14 +2,80 @@set -e-# 基于脚本所在目录计算数据文件路径+# 基于脚本所在目录计算路径SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"INPUT_FILE="$SCRIPT_DIR/../data/locomo10.json"+RESULT_FILE="./result/locomo_result_multi_read_all.csv"++if command -v python3 >/dev/null 2>&1; then+ PYTHON_BIN="python3"+elif command -v python >/dev/null 2>&1; then+ PYTHON_BIN="python"+else+ echo "未找到 python3/python,请先安装 Python。" >&2+ exit 1+fi++DEFAULT_OV_CONF_PATH="$($PYTHON_BIN - <<'PY'+from pathlib import Path++from openviking_cli.utils.config.config_loader import resolve_config_path+from openviking_cli.utils.config.consts import DEFAULT_OV_CONF, OPENVIKING_CONFIG_ENV++path = resolve_config_path(None, OPENVIKING_CONFIG_ENV, DEFAULT_OV_CONF)+print(str(path) if path is not None else str(Path.home() / ".openviking" / "ov.conf"))+PY+)"++if [ -t 0 ] && [ -t 1 ]; then+ echo "[preflight] OpenViking 配置默认路径: $DEFAULT_OV_CONF_PATH"+ printf "[preflight] 直接回车使用默认,或输入新路径 [%s]: " "$DEFAULT_OV_CONF_PATH"+ if ! read -r OV_CONF_PATH < /dev/tty; then+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"+ fi+ if [ -z "$OV_CONF_PATH" ]; then+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"+ fi+else+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"+fi++if [ "$OV_CONF_PATH" = "~" ]; then+ OV_CONF_PATH="$HOME"+elif [[ "$OV_CONF_PATH" == ~/* ]]; then+ OV_CONF_PATH="$HOME/${OV_CONF_PATH#~/}"+fi++export OPENVIKING_CONFIG_FILE="$OV_CONF_PATH"+echo "[preflight] 本次使用 ov.conf: $OPENVIKING_CONFIG_FILE"++# 评测前预检配置+PRECHECK_STATUS=0+"$PYTHON_BIN" "$SCRIPT_DIR/preflight_eval_config.py" || PRECHECK_STATUS=$?+if [ "$PRECHECK_STATUS" -ne 0 ]; then+ if [ "$PRECHECK_STATUS" -eq 2 ]; then+ echo "[preflight] 已完成 root_api_key 初始化,请先重启 openviking-server,再重新执行评测脚本。" >&2+ fi+ exit "$PRECHECK_STATUS"+fi++RUNTIME_ENV_FILE="$(mktemp "${TMPDIR:-/tmp}/ov_eval_runtime.XXXXXX")"+trap 'rm -f "$RUNTIME_ENV_FILE"' EXIT++if [ -t 0 ] && [ -t 1 ]; then+ INTERACTIVE=1+else+ INTERACTIVE=0+fi++INTERACTIVE="$INTERACTIVE" "$PYTHON_BIN" "$SCRIPT_DIR/preflight_eval_runtime.py" --output-env-file "$RUNTIME_ENV_FILE"+# shellcheck disable=SC1090+source "$RUNTIME_ENV_FILE"# Step 1: 导入数据(可跳过)if [ "$1" != "--skip-import" ]; thenecho "[1/4] 导入数据..."- python benchmark/locomo/vikingbot/import_to_ov.py --input $INPUT_FILE --force-ingest+ "$PYTHON_BIN" "$SCRIPT_DIR/import_to_ov.py" --input "$INPUT_FILE" --force-ingest --account "$ACCOUNT" --openviking-url "$OPENVIKING_URL"echo "等待 1 分钟..."sleep 60else@@ -18,15 +84,14 @@ fi# Step 2: 评估echo "[2/4] 评估..."-python benchmark/locomo/vikingbot/run_eval.py $INPUT_FILE --output ./result/locomo_result_multi_read_all.csv-+"$PYTHON_BIN" "$SCRIPT_DIR/run_eval.py" "$INPUT_FILE" --output "$RESULT_FILE"# Step 3: 裁判打分echo "[3/4] 裁判打分..."-python benchmark/locomo/vikingbot/judge.py --input ./result/locomo_result_multi_read_all.csv --parallel 40+"$PYTHON_BIN" "$SCRIPT_DIR/judge.py" --input "$RESULT_FILE" --parallel 40# Step 4: 计算结果echo "[4/4] 计算结果..."-python benchmark/locomo/vikingbot/stat_judge_result.py --input ./result/locomo_result_multi_read_all.csv+"$PYTHON_BIN" "$SCRIPT_DIR/stat_judge_result.py" --input "$RESULT_FILE"-echo "完成!"\ No newline at end of file+echo "完成!"