Skip to content

Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.

feat(eval): Locomo bot eval add check (#1629)

yeshion23333ce4238955838
diff --git a/benchmark/locomo/README.md b/benchmark/locomo/README.md
index 2912accc7..ed3c676c4 100644
--- a/benchmark/locomo/README.md
+++ b/benchmark/locomo/README.md
@@ -29,6 +29,29 @@ benchmark/locomo/
## VikingBot 评测流程
+### 前置配置说明
+- vikingbot评测须确保 OpenViking 服务端已配置 root_api_key,即开启多租户模式。每个sample数据都会使用sample_id如`conv-26`作为user_id,存储在OpenViking中。
+```json
+{
+ "server": {
+ "root_api_key": "your-key"
+ }
+}
+```
+- OpenViking数据导入account会优先使用`ovcli.conf`中的account值,若未配置默认使用`default`;
+- vikingbot必须配置OpenViking 的root级别API KEY,默认使用上述的server.root_api_key,也可单独配置;
+- vikingbot查询数据account默认为`default`,如更改必须与导入OpenViking的account一致,即`ovcli.conf`中的account值,可通过`ov.conf`如下配置:
+```json
+{
+ "bot": {
+ "ov_server": {
+ "root_api_key": "your-root-key",
+ "account_id": "默认default,必须和ovcli.conf中的account_id一致"
+ }
+ }
+}
+```
+
### 完整一键评测
使用 `run_full_eval.sh` 可以一键运行完整评测流程:
@@ -39,8 +62,6 @@ bash run_full_eval.sh # 完整流程
bash run_full_eval.sh --skip-import # 跳过导入,仅评测
```
-该脚本会依次执行以下四个步骤:
-
### 单题/批量测试
使用 `import_and_eval_one.sh` 可以快速测试单个问题或批量测试某个 sample:
@@ -80,6 +101,7 @@ python import_to_ov.py --input <数据文件路径> [选项]
- `--force-ingest`: 强制重新导入,即使已导入过
- `--clear-ingest-record`: 清除所有导入记录
- `--openviking-url`: OpenViking 服务地址,默认 `http://localhost:1933`
+- `--account`: 导入时使用的 account,默认 `default`
**示例:**
```bash
@@ -404,6 +426,9 @@ A: 使用 `--force-ingest`(导入)或删除结果 CSV 文件。
### Q: 评测速度慢怎么办?
A: 增加 `--threads`(run_eval.py)或 `--parallel`(其他脚本)参数值。
+### Q: 评测效果低,怎么排查 OpenViking 导入与评测查询的 account/user 是否一致?
+A: 先核对三处是否对齐:`ovcli.conf.account`(导入 account)、`ov.conf.bot.ov_server.account_id`(Vikingbot 查询 account)、评测脚本使用的 user(Vikingbot 按 `sample_id`,OpenClaw 默认 `eval-1`)。这几项不一致时,常见现象是“导入看起来成功,但评测回答质量明显下降或查不到上下文”。
+
---
## 常见问题排查
@@ -461,7 +486,23 @@ grep -i "error" result/qa_results.csv
**预期结果**:在用户提问(query)之前,应该有从 OpenViking 加载的记忆内容。
-### 4. Token 统计异常
+### 4. 评测效果低时,先口语化排查 account/user
+
+如果你感觉“明明导入了,回答还是不对劲”,先按这个顺序看:
+
+1. 打开 `~/.openviking/ovcli.conf`,看 `account` 是不是你这次要用的账号。
+2. 打开 `~/.openviking/ov.conf`,重点看:
+ - `bot.ov_server.account_id`
+ - `server.host` / `server.port`
+3. 跑 Vikingbot 脚本时留意 preflight 日志里打印的 `account` 和 `OpenViking URL`,确认和你配置里看到的一致。
+4. 记住查询侧是谁在查:
+ - Vikingbot 评测默认用 `sample_id` 当 user。
+ - OpenClaw QA 默认是 `--user eval-1 --agent-id locomo-eval`。
+ - 你如果改过 OpenClaw 的 `--user` 或 `--agent-id`,要保证 ingest 和 qa 两边用的是同一套值。
+
+一句话:导入时的 account/user、评测时的 account/user、以及连接的服务地址,这三件事只要有一个没对齐,就很容易出现效果低或“查不到上下文”。
+
+### 5. Token 统计异常
如果 `stat_judge_result.py` 输出的 token 数量异常:
diff --git a/benchmark/locomo/vikingbot/import_and_eval_one.sh b/benchmark/locomo/vikingbot/import_and_eval_one.sh
index 3289fcb14..dcc3a4a1e 100755
--- a/benchmark/locomo/vikingbot/import_and_eval_one.sh
+++ b/benchmark/locomo/vikingbot/import_and_eval_one.sh
@@ -4,15 +4,80 @@
# Usage:
# ./import_and_eval_one.sh 0 2 # sample 0, question 2 (单题)
# ./import_and_eval_one.sh conv-26 2 # sample_id conv-26, question 2 (单题)
-# ./import_and_eval_one.sh conv-26 # sample_id conv-26, 所有问题 (批量)
-# ./import_and_eval_one.sh conv-26 2 --skip-import # 跳过导入,直接评测
-# ./import_and_eval_one.sh conv-26 --skip-import # 跳过导入,批量评测
+# ./import_and_eval_one.sh conv-26 # sample_id conv-26, 所有问题 (批量)
+# ./import_and_eval_one.sh conv-26 2 --skip-import # 跳过导入,直接评测
+# ./import_and_eval_one.sh conv-26 --skip-import # 跳过导入,批量评测
set -e
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SKIP_IMPORT=false
+if command -v python3 >/dev/null 2>&1; then
+ PYTHON_BIN="python3"
+elif command -v python >/dev/null 2>&1; then
+ PYTHON_BIN="python"
+else
+ echo "未找到 python3/python,请先安装 Python。" >&2
+ exit 1
+fi
+
+DEFAULT_OV_CONF_PATH="$($PYTHON_BIN - <<'PY'
+from pathlib import Path
+
+from openviking_cli.utils.config.config_loader import resolve_config_path
+from openviking_cli.utils.config.consts import DEFAULT_OV_CONF, OPENVIKING_CONFIG_ENV
+
+path = resolve_config_path(None, OPENVIKING_CONFIG_ENV, DEFAULT_OV_CONF)
+print(str(path) if path is not None else str(Path.home() / ".openviking" / "ov.conf"))
+PY
+)"
+
+if [ -t 0 ] && [ -t 1 ]; then
+ echo "[preflight] OpenViking 配置默认路径: $DEFAULT_OV_CONF_PATH"
+ printf "[preflight] 直接回车使用默认,或输入新路径 [%s]: " "$DEFAULT_OV_CONF_PATH"
+ if ! read -r OV_CONF_PATH < /dev/tty; then
+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"
+ fi
+ if [ -z "$OV_CONF_PATH" ]; then
+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"
+ fi
+else
+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"
+fi
+
+if [ "$OV_CONF_PATH" = "~" ]; then
+ OV_CONF_PATH="$HOME"
+elif [[ "$OV_CONF_PATH" == ~/* ]]; then
+ OV_CONF_PATH="$HOME/${OV_CONF_PATH#~/}"
+fi
+
+export OPENVIKING_CONFIG_FILE="$OV_CONF_PATH"
+echo "[preflight] 本次使用 ov.conf: $OPENVIKING_CONFIG_FILE"
+
+# 评测前预检配置
+PRECHECK_STATUS=0
+"$PYTHON_BIN" "$SCRIPT_DIR/preflight_eval_config.py" || PRECHECK_STATUS=$?
+if [ "$PRECHECK_STATUS" -ne 0 ]; then
+ if [ "$PRECHECK_STATUS" -eq 2 ]; then
+ echo "[preflight] 已完成 root_api_key 初始化,请先重启 openviking-server,再重新执行评测脚本。" >&2
+ fi
+ exit "$PRECHECK_STATUS"
+fi
+
+RUNTIME_ENV_FILE="$(mktemp "${TMPDIR:-/tmp}/ov_eval_runtime.XXXXXX")"
+trap 'rm -f "$RUNTIME_ENV_FILE"' EXIT
+
+if [ -t 0 ] && [ -t 1 ]; then
+ INTERACTIVE=1
+else
+ INTERACTIVE=0
+fi
+
+INTERACTIVE="$INTERACTIVE" "$PYTHON_BIN" "$SCRIPT_DIR/preflight_eval_runtime.py" --output-env-file "$RUNTIME_ENV_FILE"
+# shellcheck disable=SC1090
+source "$RUNTIME_ENV_FILE"
+
# 解析参数
for arg in "$@"; do
if [ "$arg" = "--skip-import" ]; then
@@ -47,16 +112,24 @@ if [[ "$SAMPLE" =~ ^-?[0-9]+$ ]]; then
echo "Using sample index: $SAMPLE_INDEX"
else
# 通过 sample_id 查找索引
- SAMPLE_INDEX=$(python3 -c "
+ SAMPLE_INDEX=$(SAMPLE="$SAMPLE" INPUT_FILE="$INPUT_FILE" "$PYTHON_BIN" - <<'PY'
import json
-data = json.load(open('$INPUT_FILE'))
+import os
+
+sample = os.environ["SAMPLE"]
+input_file = os.environ["INPUT_FILE"]
+
+with open(input_file, "r", encoding="utf-8") as f:
+ data = json.load(f)
+
for i, s in enumerate(data):
- if s.get('sample_id') == '$SAMPLE':
+ if s.get("sample_id") == sample:
print(i)
break
else:
- print('NOT_FOUND')
-")
+ print("NOT_FOUND")
+PY
+)
if [ "$SAMPLE_INDEX" = "NOT_FOUND" ]; then
echo "Error: sample_id '$SAMPLE' not found"
exit 1
@@ -75,11 +148,13 @@ if [ -n "$QUESTION_INDEX" ]; then
echo "[1/3] Skipping import (--skip-import)"
else
echo "[1/3] Importing sample $SAMPLE_INDEX, question $QUESTION_INDEX..."
- python benchmark/locomo/vikingbot/import_to_ov.py \
+ "$PYTHON_BIN" "$SCRIPT_DIR/import_to_ov.py" \
--input "$INPUT_FILE" \
--sample "$SAMPLE_INDEX" \
--question-index "$QUESTION_INDEX" \
- --force-ingest
+ --force-ingest \
+ --account "$ACCOUNT" \
+ --openviking-url "$OPENVIKING_URL"
echo "Waiting for data processing..."
sleep 3
@@ -94,7 +169,7 @@ if [ -n "$QUESTION_INDEX" ]; then
if [[ "$SAMPLE" =~ ^-?[0-9]+$ ]]; then
# 数字索引用默认输出文件
OUTPUT_FILE=./result/locomo_qa_result.csv
- python benchmark/locomo/vikingbot/run_eval.py \
+ "$PYTHON_BIN" "$SCRIPT_DIR/run_eval.py" \
"$INPUT_FILE" \
--sample "$SAMPLE_ID_FOR_CMD" \
--question-index "$QUESTION_INDEX" \
@@ -102,7 +177,7 @@ if [ -n "$QUESTION_INDEX" ]; then
else
# sample_id 模式直接更新批量结果文件
OUTPUT_FILE=./result/locomo_${SAMPLE}_result.csv
- python benchmark/locomo/vikingbot/run_eval.py \
+ "$PYTHON_BIN" "$SCRIPT_DIR/run_eval.py" \
"$INPUT_FILE" \
--sample "$SAMPLE_ID_FOR_CMD" \
--question-index "$QUESTION_INDEX" \
@@ -117,55 +192,62 @@ if [ -n "$QUESTION_INDEX" ]; then
else
echo "[3/3] Running judge..."
fi
- python benchmark/locomo/vikingbot/judge.py --input "$OUTPUT_FILE" --parallel 1
+ "$PYTHON_BIN" "$SCRIPT_DIR/judge.py" --input "$OUTPUT_FILE" --parallel 1
# 输出结果
echo ""
echo "=== 评测结果 ==="
- python3 -c "
+ OUTPUT_FILE="$OUTPUT_FILE" QUESTION_INDEX="$QUESTION_INDEX" "$PYTHON_BIN" - <<'PY'
import csv
import json
+import os
-question_index = $QUESTION_INDEX
+question_index = int(os.environ["QUESTION_INDEX"])
+output_file = os.environ["OUTPUT_FILE"]
-with open('$OUTPUT_FILE') as f:
+with open(output_file, "r", encoding="utf-8") as f:
reader = csv.DictReader(f)
rows = list(reader)
-# 找到指定 question_index 的结果
row = None
for r in rows:
- if int(r.get('question_index', -1)) == question_index:
+ if int(r.get("question_index", -1)) == question_index:
row = r
break
if row is None:
- # 没找到则用最后一条
row = rows[-1]
-# 解析 evidence_text
-evidence_text = json.loads(row.get('evidence_text', '[]'))
-evidence_str = '\\n'.join(evidence_text) if evidence_text else ''
+evidence_text = json.loads(row.get("evidence_text", "[]"))
+evidence_str = "\n".join(evidence_text) if evidence_text else ""
-print(f\"问题: {row['question']}\")
-print(f\"期望答案: {row['answer']}\")
-print(f\"模型回答: {row['response']}\")
-print(f\"证据原文:\\n{evidence_str}\")
-print(f\"结果: {row.get('result', 'N/A')}\")
-print(f\"原因: {row.get('reasoning', 'N/A')}\")
-"
+print(f"问题: {row['question']}")
+print(f"期望答案: {row['answer']}")
+print(f"模型回答: {row['response']}")
+print(f"证据原文:\n{evidence_str}")
+print(f"结果: {row.get('result', 'N/A')}")
+print(f"原因: {row.get('reasoning', 'N/A')}")
+PY
else
# ========== 批量模式 ==========
echo "=== 批量模式: sample $SAMPLE, 所有问题 ==="
# 获取该 sample 的问题数量
- QUESTION_COUNT=$(python3 -c "
+ QUESTION_COUNT=$(SAMPLE_INDEX="$SAMPLE_INDEX" INPUT_FILE="$INPUT_FILE" "$PYTHON_BIN" - <<'PY'
import json
-data = json.load(open('$INPUT_FILE'))
-sample = data[$SAMPLE_INDEX]
-print(len(sample.get('qa', [])))
-")
+import os
+
+sample_index = int(os.environ["SAMPLE_INDEX"])
+input_file = os.environ["INPUT_FILE"]
+
+with open(input_file, "r", encoding="utf-8") as f:
+ data = json.load(f)
+
+sample = data[sample_index]
+print(len(sample.get("qa", [])))
+PY
+)
echo "Found $QUESTION_COUNT questions for sample $SAMPLE"
# 导入所有 sessions
@@ -173,10 +255,12 @@ print(len(sample.get('qa', [])))
echo "[1/4] Skipping import (--skip-import)"
else
echo "[1/4] Importing all sessions for sample $SAMPLE_INDEX..."
- python benchmark/locomo/vikingbot/import_to_ov.py \
+ "$PYTHON_BIN" "$SCRIPT_DIR/import_to_ov.py" \
--input "$INPUT_FILE" \
--sample "$SAMPLE_INDEX" \
- --force-ingest
+ --force-ingest \
+ --account "$ACCOUNT" \
+ --openviking-url "$OPENVIKING_URL"
echo "Waiting for data processing..."
sleep 10
@@ -189,7 +273,7 @@ print(len(sample.get('qa', [])))
echo "[2/4] Running evaluation for all questions..."
fi
OUTPUT_FILE=./result/locomo_${SAMPLE}_result.csv
- python benchmark/locomo/vikingbot/run_eval.py \
+ "$PYTHON_BIN" "$SCRIPT_DIR/run_eval.py" \
"$INPUT_FILE" \
--sample "$SAMPLE_ID_FOR_CMD" \
--output "$OUTPUT_FILE" \
@@ -201,7 +285,7 @@ print(len(sample.get('qa', [])))
else
echo "[3/4] Running judge..."
fi
- python benchmark/locomo/vikingbot/judge.py --input "$OUTPUT_FILE" --parallel 5
+ "$PYTHON_BIN" "$SCRIPT_DIR/judge.py" --input "$OUTPUT_FILE" --parallel 5
# 输出统计结果
if [ "$SKIP_IMPORT" = "true" ]; then
@@ -209,9 +293,9 @@ print(len(sample.get('qa', [])))
else
echo "[4/4] Calculating statistics..."
fi
- python benchmark/locomo/vikingbot/stat_judge_result.py --input "$OUTPUT_FILE"
+ "$PYTHON_BIN" "$SCRIPT_DIR/stat_judge_result.py" --input "$OUTPUT_FILE"
echo ""
echo "=== 批量评测完成 ==="
echo "结果文件: $OUTPUT_FILE"
-fi
\ No newline at end of file
+fi
diff --git a/benchmark/locomo/vikingbot/import_to_ov.py b/benchmark/locomo/vikingbot/import_to_ov.py
index a6b23c461..c537186b1 100644
--- a/benchmark/locomo/vikingbot/import_to_ov.py
+++ b/benchmark/locomo/vikingbot/import_to_ov.py
@@ -279,6 +279,7 @@ async def viking_ingest(
session_time: Optional[str] = None,
user_id: Optional[str] = None,
agent_id: Optional[str] = None,
+ account: str = "default",
) -> Dict[str, int]:
"""Save messages to OpenViking via OpenViking SDK client.
Returns token usage dict with embedding and vlm token counts.
@@ -289,6 +290,7 @@ async def viking_ingest(
session_time: Session time string (e.g., "9:36 am on 2 April, 2023")
user_id: User identifier for separate userspace (e.g., "conv-26")
agent_id: Agent identifier for separate agentspace (e.g., "conv-26")
+ account: OpenViking account identifier
"""
# 解析 session_time - 为每条消息计算递增的时间戳
base_datetime = None
@@ -303,6 +305,7 @@ async def viking_ingest(
url=openviking_url,
user=user_id,
agent_id=agent_id,
+ account=account,
)
await client.initialize()
@@ -387,6 +390,7 @@ async def process_single_session(
meta.get("date_time"),
user_id=str(sample_id),
agent_id=str(sample_id),
+ account=args.account,
)
token_usage = result["token_usage"]
task_id = result.get("task_id")
@@ -668,6 +672,11 @@ def main():
default="http://localhost:1933",
help="OpenViking service URL (default: http://localhost:1933)",
)
+ parser.add_argument(
+ "--account",
+ default="default",
+ help="OpenViking account identifier (default: default)",
+ )
parser.add_argument(
"--sample",
type=int,
diff --git a/benchmark/locomo/vikingbot/preflight_eval_config.py b/benchmark/locomo/vikingbot/preflight_eval_config.py
new file mode 100644
index 000000000..10cc52213
--- /dev/null
+++ b/benchmark/locomo/vikingbot/preflight_eval_config.py
@@ -0,0 +1,304 @@
+#!/usr/bin/env python3
+
+import json
+import os
+import sys
+from pathlib import Path
+
+from openviking_cli.utils.config.config_loader import resolve_config_path
+from openviking_cli.utils.config.consts import (
+ DEFAULT_OVCLI_CONF,
+ DEFAULT_OV_CONF,
+ OPENVIKING_CLI_CONFIG_ENV,
+ OPENVIKING_CONFIG_ENV,
+)
+
+_USE_COLOR = hasattr(sys.stdout, "isatty") and sys.stdout.isatty()
+
+
+def _color(text: str, code: str) -> str:
+ if not _USE_COLOR:
+ return text
+ return f"\033[{code}m{text}\033[0m"
+
+
+def _prefix() -> str:
+ return _color("[preflight]", "36")
+
+
+def _log_info(message: str) -> None:
+ print(f"{_prefix()} {_color('[INFO]', '34')} {message}")
+
+
+def _log_warn(message: str) -> None:
+ print(f"{_prefix()} {_color('[WARN]', '33')} {message}")
+
+
+def _log_ok(message: str) -> None:
+ print(f"{_prefix()} {_color('[OK]', '32')} {message}")
+
+
+def _log_error(message: str) -> None:
+ print(f"{_prefix()} {_color('[ERROR]', '31')} {message}", file=sys.stderr)
+
+
+def _is_interactive() -> bool:
+ return sys.stdin.isatty() and sys.stdout.isatty()
+
+
+def _prompt_text(prompt: str, default: str | None = None) -> str:
+ suffix = f" [{default}]" if default else ""
+ raw = input(f"{_prefix()} {prompt}{suffix}: ").strip()
+ if not raw and default is not None:
+ return default
+ return raw
+
+
+def _prompt_confirm(prompt: str, default: bool = True) -> bool:
+ hint = "Y/n" if default else "y/N"
+ raw = input(f"{_prefix()} {prompt} [{hint}]: ").strip().lower()
+ if not raw:
+ return default
+ return raw in {"y", "yes"}
+
+
+def _load_json(path: Path) -> dict:
+ with open(path, "r", encoding="utf-8-sig") as f:
+ raw = f.read()
+ return json.loads(raw)
+
+
+def _backup_and_write_json(path: Path, data: dict) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ if path.exists():
+ backup = path.with_suffix(path.suffix + ".bak")
+ with open(path, "r", encoding="utf-8") as src:
+ original = src.read()
+ with open(backup, "w", encoding="utf-8") as bak:
+ bak.write(original)
+ with open(path, "w", encoding="utf-8") as f:
+ json.dump(data, f, indent=2, ensure_ascii=False)
+ f.write("\n")
+
+
+def _resolve_ov_conf_path() -> Path:
+ configured_path = os.environ.get(OPENVIKING_CONFIG_ENV, "").strip()
+ if configured_path:
+ return Path(configured_path).expanduser()
+
+ resolved = resolve_config_path(None, OPENVIKING_CONFIG_ENV, DEFAULT_OV_CONF)
+ default_path = str(resolved) if resolved is not None else str(Path.home() / ".openviking" / "ov.conf")
+
+ if _is_interactive():
+ _log_info(f"OpenViking 配置默认路径: {default_path}")
+ chosen = _prompt_text("直接回车使用默认,或输入新路径", default=default_path)
+ else:
+ chosen = default_path
+ return Path(chosen).expanduser()
+
+
+def _ensure_server_root_api_key(ov_data: dict) -> tuple[bool, bool]:
+ server = ov_data.setdefault("server", {})
+ root_key = str(server.get("root_api_key") or "").strip()
+ if root_key:
+ return True, False
+
+ _log_warn("server.root_api_key 未配置。")
+ if not _is_interactive():
+ _log_error("非交互模式下无法录入 root_api_key,请先配置后重试。")
+ return False, False
+
+ while True:
+ key = _prompt_text("请输入 OpenViking root_api_key")
+ if key:
+ server["root_api_key"] = key
+ return True, True
+ _log_warn("root_api_key 不能为空,请重新输入。")
+
+
+def _check_bot_root_key(ov_data: dict) -> tuple[bool, bool]:
+ server_root_key = str(ov_data.get("server", {}).get("root_api_key") or "").strip()
+ bot = ov_data.setdefault("bot", {})
+ ov_server = bot.setdefault("ov_server", {})
+ bot_root_key = str(ov_server.get("root_api_key") or "").strip()
+
+ if not bot_root_key or bot_root_key == server_root_key:
+ return True, False
+
+ _log_warn("bot.ov_server.root_api_key 与 server.root_api_key 不一致。")
+ _log_warn("vikingbot 必须使用 OpenViking 的 root 级 API key。")
+
+ if not _is_interactive():
+ _log_error("非交互模式下不会自动修改,请先修复配置后重试。")
+ return False, False
+
+ if _prompt_confirm("是否自动调整为 server.root_api_key", default=True):
+ ov_server["root_api_key"] = server_root_key
+ return True, True
+
+ _log_error("已取消自动调整。")
+ return False, False
+
+
+def _resolve_ovcli_path() -> Path:
+ resolved = resolve_config_path(None, OPENVIKING_CLI_CONFIG_ENV, DEFAULT_OVCLI_CONF)
+ if resolved is not None:
+ return resolved
+ return Path.home() / ".openviking" / "ovcli.conf"
+
+
+def _set_bot_account_id(ov_data: dict, account_id: str) -> bool:
+ bot = ov_data.setdefault("bot", {})
+ ov_server = bot.setdefault("ov_server", {})
+ current = str(ov_server.get("account_id") or "").strip()
+ if current == account_id:
+ return False
+ ov_server["account_id"] = account_id
+ return True
+
+
+def _check_ovcli_keys_and_account(server_root_key: str, ov_data: dict) -> tuple[bool, bool, str]:
+ ovcli_path = _resolve_ovcli_path()
+
+ changed_ovcli = False
+ changed_ov = False
+
+ if not ovcli_path.exists():
+ _log_warn(f"未找到 ovcli.conf: {ovcli_path}")
+ if not _is_interactive():
+ _log_error("非交互模式下不会创建 ovcli.conf。")
+ return False, False, "default"
+ if not _prompt_confirm("是否创建并写入 root key", default=True):
+ _log_error("已取消创建 ovcli.conf。")
+ return False, False, "default"
+ ovcli_data = {
+ "url": "http://localhost:1933",
+ "api_key": server_root_key,
+ "root_api_key": server_root_key,
+ "account": "",
+ "timeout": 60.0,
+ }
+ changed_ovcli = True
+ else:
+ try:
+ ovcli_data = _load_json(ovcli_path)
+ except Exception as exc:
+ _log_error(f"读取 ovcli.conf 失败: {exc}")
+ return False, False, "default"
+
+ needs_key_update = False
+ api_key = str(ovcli_data.get("api_key") or "").strip()
+ if api_key != server_root_key:
+ needs_key_update = True
+
+ has_root_key_field = "root_api_key" in ovcli_data
+ root_api_key = str(ovcli_data.get("root_api_key") or "").strip() if has_root_key_field else ""
+ if has_root_key_field and root_api_key != server_root_key:
+ needs_key_update = True
+
+ if needs_key_update:
+ _log_warn("ovcli.conf 的 key 与 server.root_api_key 不一致。")
+ if not _is_interactive():
+ _log_error("非交互模式下不会自动修改 ovcli.conf。")
+ return False, False, "default"
+ if not _prompt_confirm("是否自动更改为 server.root_api_key", default=True):
+ _log_error("已取消自动更改 ovcli.conf。")
+ return False, False, "default"
+ ovcli_data["api_key"] = server_root_key
+ if has_root_key_field:
+ ovcli_data["root_api_key"] = server_root_key
+ changed_ovcli = True
+ _log_ok("ovcli.conf key 已同步为 server.root_api_key")
+
+ selected_account = str(ovcli_data.get("account") or "").strip()
+ if selected_account:
+ if _is_interactive():
+ use_current = _prompt_confirm(
+ f"检测到 ovcli.conf.account={selected_account},是否使用该 account",
+ default=True,
+ )
+ if not use_current:
+ candidate = _prompt_text("请输入新的 account", default=selected_account).strip()
+ if candidate:
+ selected_account = candidate
+ ovcli_data["account"] = selected_account
+ changed_ovcli = True
+ _log_ok(f"ovcli.conf account 已更新为: {selected_account}")
+ _log_info(f"导入与评测将使用 account: {selected_account}")
+ else:
+ _log_warn("ovcli.conf 的 account 为空。")
+ if _is_interactive():
+ selected_account = _prompt_text(
+ "将使用默认 Account=default;回车确认,或输入新 account",
+ default="default",
+ ).strip()
+ if not selected_account:
+ selected_account = "default"
+ else:
+ selected_account = "default"
+ _log_info("将使用默认 Account=default 导入 OpenViking 数据和 Vikingbot 评测。")
+
+ if selected_account != "default":
+ ovcli_data["account"] = selected_account
+ changed_ovcli = True
+ _log_ok(f"ovcli.conf account 已更新为: {selected_account}")
+
+ if _set_bot_account_id(ov_data, selected_account):
+ changed_ov = True
+ _log_ok(f"bot.ov_server.account_id 已同步为: {selected_account}")
+
+ if changed_ovcli:
+ _backup_and_write_json(ovcli_path, ovcli_data)
+ _log_ok(f"已更新 {ovcli_path}")
+
+ return True, changed_ov, selected_account
+
+
+def main() -> int:
+ try:
+ ov_conf_path = _resolve_ov_conf_path()
+ if not ov_conf_path.exists():
+ _log_error(f"ov.conf 不存在: {ov_conf_path}")
+ return 1
+
+ try:
+ ov_data = _load_json(ov_conf_path)
+ except Exception as exc:
+ _log_error(f"读取 ov.conf 失败: {exc}")
+ return 1
+
+ ok, changed_server = _ensure_server_root_api_key(ov_data)
+ if not ok:
+ return 1
+
+ ok, changed_bot = _check_bot_root_key(ov_data)
+ if not ok:
+ return 1
+
+ if changed_server or changed_bot:
+ _backup_and_write_json(ov_conf_path, ov_data)
+ _log_ok(f"已更新 {ov_conf_path}")
+
+ if changed_server:
+ _log_warn("检测到 server.root_api_key 刚完成配置,请先重启 openviking-server。")
+ _log_info("重启完成后请重新执行一键评测脚本。")
+ return 2
+
+ server_root_key = str(ov_data.get("server", {}).get("root_api_key") or "").strip()
+ ok, changed_ov_account, _ = _check_ovcli_keys_and_account(server_root_key, ov_data)
+ if not ok:
+ return 1
+
+ if changed_ov_account:
+ _backup_and_write_json(ov_conf_path, ov_data)
+ _log_ok(f"已更新 {ov_conf_path}")
+
+ _log_ok("配置检查通过。")
+ return 0
+ except KeyboardInterrupt:
+ _log_error("用户取消。")
+ return 1
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmark/locomo/vikingbot/preflight_eval_runtime.py b/benchmark/locomo/vikingbot/preflight_eval_runtime.py
new file mode 100644
index 000000000..cdf2516b8
--- /dev/null
+++ b/benchmark/locomo/vikingbot/preflight_eval_runtime.py
@@ -0,0 +1,227 @@
+#!/usr/bin/env python3
+
+import argparse
+import json
+import os
+import shlex
+import sys
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+from openviking_cli.utils.config.config_loader import resolve_config_path
+from openviking_cli.utils.config.consts import (
+ DEFAULT_OVCLI_CONF,
+ DEFAULT_OV_CONF,
+ OPENVIKING_CLI_CONFIG_ENV,
+ OPENVIKING_CONFIG_ENV,
+)
+
+
+def _log(message: str) -> None:
+ print(f"[preflight] {message}")
+
+
+def _error(message: str) -> None:
+ print(f"[preflight] {message}", file=sys.stderr)
+
+
+def _load_json(path: Path) -> dict:
+ with open(path, "r", encoding="utf-8-sig") as f:
+ return json.load(f)
+
+
+def _resolve_account() -> str:
+ path = resolve_config_path(None, OPENVIKING_CLI_CONFIG_ENV, DEFAULT_OVCLI_CONF)
+ if path is None:
+ return "default"
+ try:
+ data = _load_json(Path(path))
+ except Exception:
+ return "default"
+ account = str(data.get("account") or "").strip()
+ return account or "default"
+
+
+def _resolve_openviking_url() -> str:
+ host = "127.0.0.1"
+ port = 1933
+
+ path = resolve_config_path(None, OPENVIKING_CONFIG_ENV, DEFAULT_OV_CONF)
+ if path is not None:
+ try:
+ data = _load_json(Path(path))
+ server = data.get("server") or {}
+ parsed_host = str(server.get("host") or "").strip()
+ parsed_port = server.get("port")
+ if parsed_host:
+ host = parsed_host
+ if isinstance(parsed_port, int):
+ port = parsed_port
+ elif isinstance(parsed_port, str) and parsed_port.strip().isdigit():
+ port = int(parsed_port.strip())
+ except Exception:
+ pass
+
+ return f"http://{host}:{port}"
+
+
+def _load_ov_conf() -> dict:
+ ov_conf_path = resolve_config_path(None, OPENVIKING_CONFIG_ENV, DEFAULT_OV_CONF)
+ if ov_conf_path is None:
+ _error("未找到 ov.conf,无法读取 root_api_key。")
+ raise SystemExit(1)
+
+ try:
+ return _load_json(Path(ov_conf_path))
+ except Exception as exc:
+ _error(f"读取 ov.conf 失败: {exc}")
+ raise SystemExit(1)
+
+
+def _parse_accounts(body: str) -> list:
+ try:
+ payload = json.loads(body)
+ except Exception as exc:
+ _error(f"/api/v1/admin/accounts 返回非 JSON: {exc}")
+ raise SystemExit(1)
+
+ accounts = payload.get("result", payload)
+ if not isinstance(accounts, list):
+ _error("/api/v1/admin/accounts 返回格式异常。")
+ raise SystemExit(1)
+ return accounts
+
+
+def _account_exists(accounts: list, account: str) -> bool:
+ for item in accounts:
+ if isinstance(item, dict):
+ account_id = str(item.get("account_id") or item.get("id") or "").strip()
+ if account_id == account:
+ return True
+ elif isinstance(item, str) and item == account:
+ return True
+ return False
+
+
+def _prompt_confirm_create(account: str) -> bool:
+ prompt = f"[preflight] account '{account}' 不存在,是否自动创建该 account? [Y/n]: "
+ try:
+ with open("/dev/tty", "r", encoding="utf-8") as tty_in:
+ print(prompt, end="", flush=True)
+ answer = tty_in.readline().strip().lower()
+ except Exception as exc:
+ _error(f"无法读取终端输入: {exc}")
+ raise SystemExit(1)
+ return answer in ("", "y", "yes")
+
+
+def _ensure_server_and_account_ready(url: str, account: str, interactive: bool) -> None:
+ ov_data = _load_ov_conf()
+
+ root_key = str((ov_data.get("server") or {}).get("root_api_key") or "").strip()
+ if not root_key:
+ _error("server.root_api_key 为空,无法执行服务连通性检查。")
+ raise SystemExit(1)
+
+ admin_user_id = str(
+ ((ov_data.get("bot") or {}).get("ov_server") or {}).get("admin_user_id") or "default"
+ ).strip() or "default"
+
+ req = urllib.request.Request(
+ f"{url}/api/v1/admin/accounts",
+ headers={
+ "X-API-Key": root_key,
+ "Content-Type": "application/json",
+ },
+ method="GET",
+ )
+
+ try:
+ with urllib.request.urlopen(req, timeout=10) as resp:
+ body = resp.read().decode("utf-8", errors="replace")
+ except urllib.error.HTTPError as e:
+ detail = e.read().decode("utf-8", errors="replace")
+ _error(f"OpenViking server 检查失败(HTTP {e.code}): {detail}")
+ raise SystemExit(1)
+ except Exception as exc:
+ _error(f"OpenViking server 不可用: {exc}")
+ raise SystemExit(1)
+
+ accounts = _parse_accounts(body)
+ if _account_exists(accounts, account):
+ _log(f"OpenViking server 可用,account '{account}' 已就绪。")
+ return
+
+ if not interactive:
+ _error(f"account '{account}' 不存在,非交互模式下不会自动创建。")
+ raise SystemExit(1)
+
+ if not _prompt_confirm_create(account):
+ _error("已取消自动创建 account。")
+ raise SystemExit(1)
+
+ create_req = urllib.request.Request(
+ f"{url}/api/v1/admin/accounts",
+ headers={
+ "X-API-Key": root_key,
+ "Content-Type": "application/json",
+ },
+ data=json.dumps({"account_id": account, "admin_user_id": admin_user_id}).encode("utf-8"),
+ method="POST",
+ )
+
+ try:
+ with urllib.request.urlopen(create_req, timeout=10) as create_resp:
+ create_body = create_resp.read().decode("utf-8", errors="replace")
+ except urllib.error.HTTPError as e:
+ detail = e.read().decode("utf-8", errors="replace")
+ _error(f"创建 account 失败(HTTP {e.code}): {detail}")
+ raise SystemExit(1)
+ except Exception as exc:
+ _error(f"创建 account 失败: {exc}")
+ raise SystemExit(1)
+
+ try:
+ create_payload = json.loads(create_body)
+ except Exception:
+ create_payload = {}
+
+ if isinstance(create_payload, dict) and create_payload.get("status") == "error":
+ _error(f"创建 account 失败: {create_payload}")
+ raise SystemExit(1)
+
+ _log(f"已创建 account '{account}'(admin_user_id={admin_user_id})。")
+ _log(f"OpenViking server 可用,account '{account}' 已就绪。")
+
+
+def _write_env_file(path: Path, account: str, openviking_url: str) -> None:
+ with open(path, "w", encoding="utf-8") as f:
+ f.write(f"ACCOUNT={shlex.quote(account)}\n")
+ f.write(f"OPENVIKING_URL={shlex.quote(openviking_url)}\n")
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="Resolve runtime eval account/url and validate OpenViking readiness")
+ parser.add_argument("--output-env-file", required=True, help="File path to write ACCOUNT/OPENVIKING_URL exports")
+ args = parser.parse_args()
+
+ interactive_env = os.environ.get("INTERACTIVE", "").strip()
+ if interactive_env:
+ interactive = interactive_env == "1"
+ else:
+ interactive = sys.stdin.isatty() and sys.stdout.isatty()
+
+ account = _resolve_account()
+ openviking_url = _resolve_openviking_url()
+
+ _log(f"本次导入与评测使用 account: {account}")
+ _log(f"本次导入使用 OpenViking URL: {openviking_url}")
+
+ _ensure_server_and_account_ready(openviking_url, account, interactive)
+ _write_env_file(Path(args.output_env_file), account, openviking_url)
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main())
diff --git a/benchmark/locomo/vikingbot/run_full_eval.sh b/benchmark/locomo/vikingbot/run_full_eval.sh
index 08746e774..c9f0c9196 100755
--- a/benchmark/locomo/vikingbot/run_full_eval.sh
+++ b/benchmark/locomo/vikingbot/run_full_eval.sh
@@ -2,14 +2,80 @@
set -e
-# 基于脚本所在目录计算数据文件路径
+# 基于脚本所在目录计算路径
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
INPUT_FILE="$SCRIPT_DIR/../data/locomo10.json"
+RESULT_FILE="./result/locomo_result_multi_read_all.csv"
+
+if command -v python3 >/dev/null 2>&1; then
+ PYTHON_BIN="python3"
+elif command -v python >/dev/null 2>&1; then
+ PYTHON_BIN="python"
+else
+ echo "未找到 python3/python,请先安装 Python。" >&2
+ exit 1
+fi
+
+DEFAULT_OV_CONF_PATH="$($PYTHON_BIN - <<'PY'
+from pathlib import Path
+
+from openviking_cli.utils.config.config_loader import resolve_config_path
+from openviking_cli.utils.config.consts import DEFAULT_OV_CONF, OPENVIKING_CONFIG_ENV
+
+path = resolve_config_path(None, OPENVIKING_CONFIG_ENV, DEFAULT_OV_CONF)
+print(str(path) if path is not None else str(Path.home() / ".openviking" / "ov.conf"))
+PY
+)"
+
+if [ -t 0 ] && [ -t 1 ]; then
+ echo "[preflight] OpenViking 配置默认路径: $DEFAULT_OV_CONF_PATH"
+ printf "[preflight] 直接回车使用默认,或输入新路径 [%s]: " "$DEFAULT_OV_CONF_PATH"
+ if ! read -r OV_CONF_PATH < /dev/tty; then
+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"
+ fi
+ if [ -z "$OV_CONF_PATH" ]; then
+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"
+ fi
+else
+ OV_CONF_PATH="$DEFAULT_OV_CONF_PATH"
+fi
+
+if [ "$OV_CONF_PATH" = "~" ]; then
+ OV_CONF_PATH="$HOME"
+elif [[ "$OV_CONF_PATH" == ~/* ]]; then
+ OV_CONF_PATH="$HOME/${OV_CONF_PATH#~/}"
+fi
+
+export OPENVIKING_CONFIG_FILE="$OV_CONF_PATH"
+echo "[preflight] 本次使用 ov.conf: $OPENVIKING_CONFIG_FILE"
+
+# 评测前预检配置
+PRECHECK_STATUS=0
+"$PYTHON_BIN" "$SCRIPT_DIR/preflight_eval_config.py" || PRECHECK_STATUS=$?
+if [ "$PRECHECK_STATUS" -ne 0 ]; then
+ if [ "$PRECHECK_STATUS" -eq 2 ]; then
+ echo "[preflight] 已完成 root_api_key 初始化,请先重启 openviking-server,再重新执行评测脚本。" >&2
+ fi
+ exit "$PRECHECK_STATUS"
+fi
+
+RUNTIME_ENV_FILE="$(mktemp "${TMPDIR:-/tmp}/ov_eval_runtime.XXXXXX")"
+trap 'rm -f "$RUNTIME_ENV_FILE"' EXIT
+
+if [ -t 0 ] && [ -t 1 ]; then
+ INTERACTIVE=1
+else
+ INTERACTIVE=0
+fi
+
+INTERACTIVE="$INTERACTIVE" "$PYTHON_BIN" "$SCRIPT_DIR/preflight_eval_runtime.py" --output-env-file "$RUNTIME_ENV_FILE"
+# shellcheck disable=SC1090
+source "$RUNTIME_ENV_FILE"
# Step 1: 导入数据(可跳过)
if [ "$1" != "--skip-import" ]; then
echo "[1/4] 导入数据..."
- python benchmark/locomo/vikingbot/import_to_ov.py --input $INPUT_FILE --force-ingest
+ "$PYTHON_BIN" "$SCRIPT_DIR/import_to_ov.py" --input "$INPUT_FILE" --force-ingest --account "$ACCOUNT" --openviking-url "$OPENVIKING_URL"
echo "等待 1 分钟..."
sleep 60
else
@@ -18,15 +84,14 @@ fi
# Step 2: 评估
echo "[2/4] 评估..."
-python benchmark/locomo/vikingbot/run_eval.py $INPUT_FILE --output ./result/locomo_result_multi_read_all.csv
-
+"$PYTHON_BIN" "$SCRIPT_DIR/run_eval.py" "$INPUT_FILE" --output "$RESULT_FILE"
# Step 3: 裁判打分
echo "[3/4] 裁判打分..."
-python benchmark/locomo/vikingbot/judge.py --input ./result/locomo_result_multi_read_all.csv --parallel 40
+"$PYTHON_BIN" "$SCRIPT_DIR/judge.py" --input "$RESULT_FILE" --parallel 40
# Step 4: 计算结果
echo "[4/4] 计算结果..."
-python benchmark/locomo/vikingbot/stat_judge_result.py --input ./result/locomo_result_multi_read_all.csv
+"$PYTHON_BIN" "$SCRIPT_DIR/stat_judge_result.py" --input "$RESULT_FILE"
-echo "完成!"
\ No newline at end of file
+echo "完成!"