Self-evolving Context Database for AI Agents. Unify Agent Memory, Knowledge RAG and Skills.
Python agent-contextagent-memoryagent-pluginsagentic-ragcontext-databasedsh-pluginknowledge-basememorymemory-managementself-evolvingvector-database
fix(eval): OpenClaw eval, import to ov use default user (#1305)
yeshion23333b05ef1d02578
diff --git a/benchmark/locomo/README.md b/benchmark/locomo/README.mdindex 9b85563b2..8a0737745 100644--- a/benchmark/locomo/README.md+++ b/benchmark/locomo/README.md@@ -16,9 +16,12 @@ benchmark/locomo/│ ├── data/ # 测试数据目录│ └── result/ # 评测结果目录└── openclaw/ # OpenClaw 评测脚本- ├── eval.py # OpenClaw 评估脚本+ ├── import_to_ov.py # 导入数据到 OpenViking+ ├── eval.py # OpenClaw 评估脚本 (ingest/qa)├── judge.py # LLM 裁判打分(适配 OpenClaw)+ ├── stat_judge_result.py # 统计评分结果和 token 使用├── run_full_eval.sh # 一键运行完整评测流程+ ├── data/ # 测试数据目录└── result/ # 评测结果目录```@@ -158,19 +161,28 @@ python stat_judge_result.py --input <评分结果文件>```bashcd benchmark/locomo/openclaw-bash run_full_eval.sh # 只导入 OpenViking+bash run_full_eval.sh # 只导入 OpenViking(跳过已导入的)bash run_full_eval.sh --with-claw-import # 同时导入 OpenViking 和 OpenClaw(并行执行)bash run_full_eval.sh --skip-import # 跳过导入步骤,直接运行 QA 评估+bash run_full_eval.sh --force-ingest # 强制重新导入所有数据+bash run_full_eval.sh --sample 0 # 只处理第 0 个 sample```**脚本参数说明:**+| 参数 | 说明 |+|------|------|+| `--skip-import` | 跳过导入步骤,直接运行 QA 评估 |+| `--with-claw-import` | 同时导入 OpenViking 和 OpenClaw(并行执行) |+| `--force-ingest` | 强制重新导入所有数据(忽略已导入记录) |+| `--sample <index>` | 只处理指定的 sample(0-based) |+**脚本执行流程:**1. 导入数据到 OpenViking(可选同时导入 OpenClaw)2. 等待 60 秒确保数据导入完成3. 运行 QA 评估(`eval.py qa`,输出到 `result/qa_results.csv`)4. 裁判打分(`judge.py`,并行度 40)-5. 统计结果(`stat_judge_result.py`)+5. 统计结果(`stat_judge_result.py`,同时统计 QA 和 Import 的 token 使用)**脚本内部配置参数:**@@ -184,9 +196,50 @@ bash run_full_eval.sh --skip-import # 跳过导入步骤,直接运行 Q### 分步使用说明-使用 `openclaw/eval.py` 进行 OpenClaw 评测,该脚本有两种模式:+OpenClaw 评测包含以下脚本:+- `import_to_ov.py`: 导入数据到 OpenViking+- `eval.py`: OpenClaw 评估脚本(ingest/qa 两种模式)+- `judge.py`: LLM 裁判打分+- `stat_judge_result.py`: 统计评分结果和 token 使用++---++#### import_to_ov.py - 导入对话数据到 OpenViking++```bash+python import_to_ov.py [选项]+```-### 模式 1: ingest - 导入对话数据到OpenClaw+**参数说明:**+- `--input`: 输入文件路径(JSON 或 TXT),默认 `../data/locomo10.json`+- `--sample`: 指定样本索引(0-based)+- `--sessions`: 指定会话范围,如 `1-4`+- `--question-index`: 根据 question 的 evidence 自动推断需要的 session+- `--force-ingest`: 强制重新导入+- `--no-user-agent-id`: 不传入 user_id 和 agent_id 给 OpenViking 客户端+- `--openviking-url`: OpenViking 服务地址,默认 `http://localhost:1933`+- `--success-csv`: 成功记录 CSV 路径,默认 `./result/import_success.csv`+- `--error-log`: 错误日志路径,默认 `./result/import_errors.log`++**示例:**+```bash+# 导入所有数据(跳过已导入的)+python import_to_ov.py++# 强制重新导入,不使用 user/agent id+python import_to_ov.py --force-ingest --no-user-agent-id++# 只导入第 0 个 sample+python import_to_ov.py --sample 0+```++---++#### eval.py - OpenClaw 评估脚本++该脚本有两种模式:++##### 模式 1: ingest - 导入对话数据到 OpenClaw```bashpython eval.py ingest <输入文件> [选项]@@ -195,37 +248,83 @@ python eval.py ingest <输入文件> [选项]**参数说明:**- `--sample`: 指定样本索引- `--sessions`: 指定会话范围,如 `1-4`-- `--viking`: 使用 OpenViking 而非 OpenClaw 导入- `--force-ingest`: 强制重新导入- `--agent-id`: Agent ID,默认 `locomo-eval`+- `--token`: OpenClaw Gateway Token**示例:**```bash# 导入第一个样本的 1-4 会话到 OpenClaw-python eval.py ingest locomo10.json --sample 0 --sessions 1-4--# 导入到 OpenViking-python eval.py ingest locomo10.json --sample 0 --viking+python eval.py ingest locomo10.json --sample 0 --sessions 1-4 --token <token>```-### 模式 2: qa - 运行 QA 评估-- 该评测制定了指定了`X-OpenClaw-Session-Key`,确保每次openclaw使用相同的session_id。Token计算将统计`session.jsonl`文件中的所有assistant轮次的Token消耗。每道题目执行完后会清空session.jsonl文件。-- 该评测仅支持单线程运行,不支持并发。-- 需先执行一次,查看`.openclaw/agents/{your_agent_id}/sessions/`下的session文件ID,作为`--session-id`参数的值开始完整评测。+##### 模式 2: qa - 运行 QA 评估++- 该评测指定了 `X-OpenClaw-Session-Key`,确保每次 OpenClaw 使用相同的 session_id+- Token 计算统计 `session.jsonl` 文件中的所有 assistant 轮次的 Token 消耗+- 每道题目执行完后会归档 session 文件+- 支持并发运行(`--parallel` 参数)+- 问题会自动添加时间上下文(从最后一个 session 提取)+```bashpython eval.py qa <输入文件> [选项]```**参数说明:**-- `--output`: 输出文件路径+- `--output`: 输出文件路径(不含 .csv 后缀)- `--sample`: 指定样本索引- `--count`: 运行的 QA 问题数量- `--user`: 用户 ID,默认 `eval-1`+- `--parallel`: 并发数,默认 10,最大 40- `--token`: OpenClaw Gateway Token(或设置 `OPENCLAW_GATEWAY_TOKEN` 环境变量)**示例:**```bash-python eval.py qa locomo10.json --sample 0 --output qa_results.txt+# 运行所有 sample 的 QA 评估+python eval.py qa locomo10.json --token <token> --parallel 15++# 只运行第 0 个 sample+python eval.py qa locomo10.json --sample 0 --output qa_results_sample0+```++---++#### judge.py - LLM 裁判打分++```bash+python judge.py [选项]+```++**参数说明:**+- `--input`: QA 结果 CSV 文件路径+- `--parallel`: 并发请求数,默认 40++**示例:**+```bash+python judge.py --input ./result/qa_results.csv --parallel 40+```++---++#### stat_judge_result.py - 统计结果++同时统计 QA 结果和 OpenViking Import 的 token 使用:++```bash+python stat_judge_result.py [选项]+```++**参数说明:**+- `--input`: QA 结果 CSV 文件路径,默认 `./result/qa_results_sample0.csv`+- `--import-csv`: Import 成功 CSV 文件路径,默认 `./result/import_success.csv`++**输出统计包括:**+- QA 结果统计:正确率、token 使用(no-cache、cacheRead、output)+- OpenViking Import 统计:embedding_tokens、vlm_tokens、total_tokens++**示例:**+```bash+python stat_judge_result.py --input ./result/qa_results_sample0.csv --import-csv ./result/import_success.csv```---diff --git a/benchmark/locomo/openclaw/eval.py b/benchmark/locomo/openclaw/eval.pyindex 22a565710..3e6c92c5f 100644--- a/benchmark/locomo/openclaw/eval.py+++ b/benchmark/locomo/openclaw/eval.py@@ -22,16 +22,20 @@ import jsonimport osimport sysimport time+from concurrent.futures import ThreadPoolExecutor, as_completed+from datetime import datetimefrom pathlib import Path+from threading import Lockimport requests# Configuration constantsDEFAULT_BASE_URL = "http://127.0.0.1:18789"-DEFAULT_SESSION_KEY = "eval-test-2"-DEFAULT_AGENT_ID = "locomo-eval"+DEFAULT_AGENT_ID = "main"DEFAULT_INGEST_RECORD_PATH = ".ingest_record.json"-DEFAULT_OV_COMMAND = ["ov", "add-memory"]++# CSV write lock for thread safety+csv_lock = Lock()# ---------------------------------------------------------------------------@@ -181,6 +185,56 @@ def build_session_messages(return sessions+# ---------------------------------------------------------------------------+# Question time helpers+# ---------------------------------------------------------------------------++def parse_locomo_datetime(date_str: str) -> datetime | None:+ """解析 LoCoMo 时间格式,如 '1:56 pm on 8 May, 2023'"""+ try:+ # 移除时间部分,只保留日期 "8 May, 2023"+ if " on " in date_str:+ date_part = date_str.split(" on ")[-1]+ return datetime.strptime(date_part.strip(), "%d %B, %Y")+ except ValueError:+ pass+ return None+++def get_sample_question_time(sample: dict) -> str | None:+ """从 sample 的 conversation 中提取最后一个有内容 session 的时间,返回 ISO 格式日期"""+ conversation = sample.get("conversation", {})++ # 找所有 session_N 字段(非 date_time)+ session_keys = [+ k for k in conversation.keys() if k.startswith("session_") and "date_time" not in k+ ]+ if not session_keys:+ return None++ # 按 session 编号排序,找到最后一个有内容的+ def get_session_num(key):+ try:+ return int(key.replace("session_", ""))+ except ValueError:+ return 0++ session_keys.sort(key=get_session_num, reverse=True)++ for session_key in session_keys:+ if conversation.get(session_key): # 有内容+ # 找到对应的 date_time+ session_num = get_session_num(session_key)+ dt_key = f"session_{session_num}_date_time"+ date_str = conversation.get(dt_key)+ if date_str:+ dt = parse_locomo_datetime(date_str)+ if dt:+ return dt.strftime("%Y-%m-%d")++ return None++# ---------------------------------------------------------------------------# Ingest record helpers (avoid duplicate ingestion)# ---------------------------------------------------------------------------@@ -261,6 +315,51 @@ def extract_response_text(response_json: dict) -> str:return f"[ERROR: could not extract text from response: {response_json}]"+def get_session_id_from_key(session_key: str, user: str, agent_id: str = "main") -> str | None:+ """Search all agents' sessions.json files for the session_key and return sessionFile path.+ Returns the full path to the session JSONL file if found, None otherwise.+ """+ agents_base_dir = os.path.expanduser("~/.openclaw/agents")++ if not os.path.exists(agents_base_dir):+ print(f" [session] Agents directory not found: {agents_base_dir}", file=sys.stderr)+ return None++ # Iterate through all agent directories+ for agent_name in os.listdir(agents_base_dir):+ agent_dir = os.path.join(agents_base_dir, agent_name)+ if not os.path.isdir(agent_dir):+ continue++ sessions_dir = os.path.join(agent_dir, "sessions")+ sessions_file = os.path.join(sessions_dir, "sessions.json")++ if not os.path.exists(sessions_file):+ continue++ try:+ with open(sessions_file, "r") as f:+ data = json.load(f)++ # Search for the session_key in this sessions.json+ for key, value in data.items():+ if session_key in key and isinstance(value, dict):+ session_file = value.get("sessionFile")+ if session_file:+ print(f" [session] Found sessionFile in agent '{agent_name}': {session_file}", file=sys.stderr)+ return session_file++ except json.JSONDecodeError as e:+ print(f" [session] Error parsing {sessions_file}: {e}", file=sys.stderr)+ continue+ except IOError as e:+ print(f" [session] Error reading {sessions_file}: {e}", file=sys.stderr)+ continue++ print(f" [session] session_key '{session_key}' not found in any agent's sessions.json", file=sys.stderr)+ return None++def get_session_id(user: str, agent_id: str = "main") -> str | None:"""Read the current session ID for the given user from sessions.json."""sessions_file = os.path.expanduser(f"~/.openclaw/agents/{agent_id}/sessions/sessions.json")@@ -280,46 +379,85 @@ def get_session_id(user: str, agent_id: str = "main") -> str | None:return None-def reset_session(session_id: str, agent_id: str = "main") -> str | None:- """Archive the session .jsonl file by renaming it with a timestamp suffix.+def reset_session(session_path: str, agent_id: str = "main") -> str | None:+ """Rename the session .jsonl file with a timestamp suffix.+ Accepts either a session_id or a full path to the session file.Returns the new filename if successful, None otherwise."""- sessions_dir = os.path.expanduser(f"~/.openclaw/agents/{agent_id}/sessions")- src = os.path.join(sessions_dir, f"{session_id}.jsonl")- dst = f"{src}.{int(time.time())}"+ # Check if session_path is already a full path+ if os.path.isabs(session_path) and os.path.exists(session_path):+ src = session_path+ else:+ # Treat as session_id+ sessions_dir = os.path.expanduser(f"~/.openclaw/agents/{agent_id}/sessions")+ src = os.path.join(sessions_dir, f"{session_path}.jsonl")++ if not os.path.exists(src):+ print(f" [backup] Session file not found: {src}", file=sys.stderr)+ return None++ timestamp = time.strftime("%Y%m%d_%H%M%S")+ dst = f"{src}.{timestamp}"try:os.rename(src, dst)new_filename = os.path.basename(dst)- print(f" [reset] archived {session_id}.jsonl -> {new_filename}", file=sys.stderr)+ print(f" [backup] renamed {os.path.basename(src)} -> {new_filename}", file=sys.stderr)return new_filename- except FileNotFoundError:- print(f" [reset] Session file not found: {src}", file=sys.stderr)- return Noneexcept IOError as e:- print(f" [reset] could not archive session file: {e}", file=sys.stderr)+ print(f" [backup] could not rename session file: {e}", file=sys.stderr)return None-def viking_ingest(msg: str) -> None:- """Save a message to OpenViking via `ov add-memory`."""- import subprocess- result = subprocess.run(- DEFAULT_OV_COMMAND + [msg],- capture_output=True,- text=True,- )- if result.returncode != 0:- raise RuntimeError(result.stderr.strip() or f"ov exited with code {result.returncode}")+def calculate_usage_from_jsonl(jsonl_filename: str, agent_id: str = "main") -> dict:+ """Calculate token usage from archived JSONL file."""+ # Check if jsonl_filename is already a full path+ if os.path.isabs(jsonl_filename) and os.path.exists(jsonl_filename):+ jsonl_full_path = jsonl_filename+ else:+ sessions_dir = os.path.expanduser(f"~/.openclaw/agents/{agent_id}/sessions")+ jsonl_full_path = os.path.join(sessions_dir, jsonl_filename)++ usage = {+ "input_tokens": 0,+ "output_tokens": 0,+ "cacheRead": 0,+ "cacheWrite": 0,+ "total_tokens": 0,+ }++ if not os.path.exists(jsonl_full_path):+ return usage++ try:+ with open(jsonl_full_path, "r", encoding="utf-8") as f:+ for line in f:+ if not line.strip():+ continue+ entry = json.loads(line)+ if entry.get("type") == "message" and entry.get("message", {}).get("role") == "assistant":+ entry_usage = entry.get("message", {}).get("usage", {})+ usage["input_tokens"] += entry_usage.get("input", 0)+ usage["output_tokens"] += entry_usage.get("output", 0)+ usage["cacheRead"] += entry_usage.get("cacheRead", 0)+ usage["cacheWrite"] += entry_usage.get("cacheWrite", 0)+ usage["total_tokens"] += entry_usage.get("totalTokens", 0)+ except json.JSONDecodeError as e:+ print(f" [usage] Error parsing JSONL file: {e}", file=sys.stderr)+ except IOError as e:+ print(f" [usage] Error reading JSONL file: {e}", file=sys.stderr)++ return usagedef send_message_with_retry(- base_url: str, token: str, user: str, message: str, retries: int = 2, agent_id: str = DEFAULT_AGENT_ID+ base_url: str, token: str, user: str, message: str, retries: int = 2,+ agent_id: str = DEFAULT_AGENT_ID, session_key: str | None = None) -> tuple[str, dict]:"""Call send_message with up to `retries` retries on failure."""last_exc = Nonefor attempt in range(retries + 1):try:- return send_message(base_url, token, user, message, agent_id)+ return send_message(base_url, token, user, message, agent_id, session_key)except Exception as e:last_exc = eif attempt < retries:@@ -328,7 +466,8 @@ def send_message_with_retry(def send_message(- base_url: str, token: str, user: str, message: str, agent_id: str = DEFAULT_AGENT_ID+ base_url: str, token: str, user: str, message: str,+ agent_id: str = DEFAULT_AGENT_ID, session_key: str | None = None) -> tuple[str, dict]:"""Send a single message to the OpenClaw responses API.@@ -338,9 +477,10 @@ def send_message(headers = {"Content-Type": "application/json","Authorization": f"Bearer {token}",- "X-OpenClaw-Agent-ID": agent_id,- "X-OpenClaw-Session-Key": DEFAULT_SESSION_KEY+ "X-OpenClaw-Agent-ID": agent_id}+ if session_key:+ headers["X-OpenClaw-Session-Key"] = session_keypayload = {"model": "openclaw","input": message,@@ -414,62 +554,36 @@ def run_ingest(preview = msg.replace("\n", " | ")[:80]print(f" [{label}] {preview}...", file=sys.stderr)- if args.viking:- try:- viking_ingest(msg)- print(f" -> [viking] saved", file=sys.stderr)- results.append({- "sample_id": sample_id,- "session": meta["session_key"],- "user": user_key,- "reply": "[viking] saved",- "usage": {},- })- # Mark as successfully ingested- mark_ingested(args.agent_id, user_key, sample_id, meta['session_key'], ingest_record, {- "mode": "viking",- "date_time": meta['date_time']- })- except Exception as e:- print(f" -> [ERROR] {e}", file=sys.stderr)- results.append({- "sample_id": sample_id,- "session": meta["session_key"],- "user": user_key,- "reply": f"[ERROR] {e}",- "usage": {},- })- else:- try:- reply, usage = send_message(args.base_url, args.token, user_key, msg, args.agent_id)- print(f" -> {reply[:80]}{'...' if len(reply) > 80 else ''}", file=sys.stderr)- results.append({- "sample_id": sample_id,- "session": meta["session_key"],- "user": user_key,- "reply": reply,- "usage": usage,- })- # Mark as successfully ingested- mark_ingested(args.agent_id, user_key, sample_id, meta['session_key'], ingest_record, {- "mode": "openclaw",- "date_time": meta['date_time'],- "usage": usage- })- except Exception as e:- print(f" -> [ERROR] {e}", file=sys.stderr)- results.append({- "sample_id": sample_id,- "session": meta["session_key"],- "user": user_key,- "reply": f"[ERROR] {e}",- "usage": {},- })-- if session_id is None:- session_id = get_session_id(user_key, args.agent_id)- if session_id:- reset_session(session_id, args.agent_id)+ try:+ reply, usage = send_message(args.base_url, args.token, user_key, msg, args.agent_id)+ print(f" -> {reply[:80]}{'...' if len(reply) > 80 else ''}", file=sys.stderr)+ results.append({+ "sample_id": sample_id,+ "session": meta["session_key"],+ "user": user_key,+ "reply": reply,+ "usage": usage,+ })+ # Mark as successfully ingested+ mark_ingested(args.agent_id, user_key, sample_id, meta['session_key'], ingest_record, {+ "mode": "openclaw",+ "date_time": meta['date_time'],+ "usage": usage+ })+ except Exception as e:+ print(f" -> [ERROR] {e}", file=sys.stderr)+ results.append({+ "sample_id": sample_id,+ "session": meta["session_key"],+ "user": user_key,+ "reply": f"[ERROR] {e}",+ "usage": {},+ })++ if session_id is None:+ session_id = get_session_id(user_key, args.agent_id)+ if session_id:+ reset_session(session_id, args.agent_id)if args.output:try:@@ -545,6 +659,89 @@ def run_ingest(# QA: run QA questions and compare with expected answers# ---------------------------------------------------------------------------+def process_single_question(+ sample_id: str,+ sample_idx: int,+ original_qi: int,+ qa: dict,+ args: argparse.Namespace,+ csv_path: str,+ question_time: str | None = None,+) -> dict:+ """Process a single QA question. Returns the record."""+ question = qa["question"]+ expected = str(qa["answer"])+ category = qa.get("category", "")+ evidence = qa.get("evidence", [])++ # Generate unique session_key based on sample_id + question_index+ session_key = f"qa-{sample_id}-q{original_qi}"+ user_key = args.user or f"eval-{sample_idx}"++ print(f" [{sample_idx}] Q{original_qi}: {question[:60]}{'...' if len(question) > 60 else ''}", file=sys.stderr)+ # 如果有 question_time,注入到 prompt 中+ if question_time:+ input_msg = f"Current date: {question_time}. Answer the question directly: {question}"+ else:+ input_msg = f"Answer the question directly: {question}"++ jsonl_filename = ""+ try:+ response, api_usage = send_message_with_retry(+ args.base_url, args.token, sample_id, input_msg, 2, args.agent_id, session_key+ )+ print(f" [{sample_idx}] A: {response[:60]}{'...' if len(response) > 60 else ''}", file=sys.stderr)++ # Get sessionFile path from sessions.json using session_key+ session_file_path = get_session_id_from_key(session_key, user_key, args.agent_id)+ jsonl_filename = ""++ # Archive the session file if we found it+ if session_file_path:+ jsonl_filename = reset_session(session_file_path, args.agent_id)++ # Calculate usage from JSONL file if available, otherwise use API usage+ if jsonl_filename and session_file_path:+ # Use the directory from session_file_path and the archived filename+ usage = calculate_usage_from_jsonl(os.path.join(os.path.dirname(session_file_path), jsonl_filename), args.agent_id)+ print(f" [{sample_idx}] tokens (from JSONL): in={usage['input_tokens']} out={usage['output_tokens']} cacheRead={usage['cacheRead']} cacheWrite={usage['cacheWrite']} total={usage['total_tokens']}", file=sys.stderr)+ else:+ usage = {+ "input_tokens": api_usage.get("input_tokens", 0),+ "output_tokens": api_usage.get("output_tokens", 0),+ "cacheRead": api_usage.get("cacheRead", 0),+ "cacheWrite": api_usage.get("cacheWrite", 0),+ "total_tokens": api_usage.get("total_tokens", 0),+ }+ print(f" [{sample_idx}] tokens (from API): in={usage['input_tokens']} out={usage['output_tokens']} cacheRead={usage['cacheRead']} cacheWrite={usage['cacheWrite']} total={usage['total_tokens']}", file=sys.stderr)++ except Exception as e:+ response = f"[ERROR] {e}"+ usage = {}+ jsonl_filename = ""+ print(f" [{sample_idx}] A: {response}", file=sys.stderr)++ record = {+ "sample_id": sample_id,+ "sample_idx": sample_idx,+ "qi": original_qi,+ "question": question,+ "expected": expected,+ "response": response,+ "category": category,+ "evidence": evidence,+ "usage": usage,+ "jsonl_filename": jsonl_filename,+ }++ # Save to CSV with lock for thread safety+ with csv_lock:+ save_record_to_csv(csv_path, record)+ print(f" [{sample_idx}] Saved to CSV: Q{original_qi}", file=sys.stderr)++ return record++def run_sample_qa(item: dict,sample_idx: int,@@ -552,9 +749,10 @@ def run_sample_qa(executed_records: set,csv_path: str,) -> tuple[list[dict], dict]:- """Process QA for a single sample. Returns (records, sample_usage)."""+ """Process QA for a single sample with concurrent question execution. Returns (records, sample_usage)."""sample_id = item["sample_id"]user_key = args.user or f"eval-{sample_idx}"+ question_time = get_sample_question_time(item)qas = [q for q in item.get("qa", []) if str(q.get("category", "")) != "5"]if args.count is not None:qas = qas[:args.count]@@ -573,131 +771,35 @@ def run_sample_qa(print(f" All QA questions already executed, skipping sample.", file=sys.stderr)return [], {"input_tokens": 0, "output_tokens": 0, "cacheRead": 0, "cacheWrite": 0, "total_tokens": 0}- jsonl_path = f"{args.output}.{sample_idx}.jsonl" if args.output else None-- sample_usage = {"input_tokens": 0, "output_tokens": 0, "cacheRead": 0, "cacheWrite": 0, "total_tokens": 0}- records = []- session_id = None-print(f"\n=== Sample {sample_id} [{sample_idx}] (user={user_key}) ===", file=sys.stderr)- print(f" Running {len(qas)} QA question(s)...", file=sys.stderr)+ if question_time:+ print(f" Question time context: {question_time}", file=sys.stderr)+ print(f" Running {len(qas)} QA question(s) with max {args.parallel} workers...", file=sys.stderr)- jsonl_file = None- if jsonl_path:- try:- jsonl_file = open(jsonl_path, "w", encoding="utf-8")- except IOError as e:- print(f"Warning: Could not open JSONL file {jsonl_path}: {e}", file=sys.stderr)+ records = []+ sample_usage = {"input_tokens": 0, "output_tokens": 0, "cacheRead": 0, "cacheWrite": 0, "total_tokens": 0}- try:+ # Use ThreadPoolExecutor for concurrent question execution+ with ThreadPoolExecutor(max_workers=args.parallel) as executor:+ futures = []for original_qi, qa in qas:- question = qa["question"]- expected = str(qa["answer"])- category = qa.get("category", "")- evidence = qa.get("evidence", [])-- print(f" [{sample_idx}] Q{original_qi}: {question[:60]}{'...' if len(question) > 60 else ''}", file=sys.stderr)-- jsonl_filename = ""+ future = executor.submit(+ process_single_question,+ sample_id, sample_idx, original_qi, qa, args, csv_path, question_time+ )+ futures.append(future)++ # Collect results+ for future in as_completed(futures):try:- response, api_usage = send_message_with_retry(- args.base_url, args.token, user_key, question, 2, args.agent_id,- )- print(f" [{sample_idx}] A: {response[:60]}{'...' if len(response) > 60 else ''}", file=sys.stderr)-- # Use provided session_id if available, otherwise get from system- if args.session_id:- session_id = args.session_id- elif session_id is None:- session_id = get_session_id(user_key, args.agent_id)-- # Reset session and get archived filename- if session_id:- jsonl_filename = reset_session(session_id, args.agent_id)-- # Use API usage by default- usage = api_usage- # Calculate usage from JSONL file if session_id is provided and we have the archived file- if args.session_id and jsonl_filename:- # Parse the archived JSONL file to calculate usage- sessions_dir = os.path.expanduser(f"~/.openclaw/agents/{args.agent_id}/sessions")- jsonl_full_path = os.path.join(sessions_dir, jsonl_filename)- if os.path.exists(jsonl_full_path):- total_input = 0- total_output = 0- total_cache_read = 0- total_cache_write = 0- total_total_tokens = 0- try:- with open(jsonl_full_path, "r", encoding="utf-8") as f:- for line in f:- if not line.strip():- continue- entry = json.loads(line)- if entry.get("type") == "message" and entry.get("message", {}).get("role") == "assistant":- entry_usage = entry.get("message", {}).get("usage", {})- total_input += entry_usage.get("input", 0)- total_output += entry_usage.get("output", 0)- total_cache_read += entry_usage.get("cacheRead", 0)- total_cache_write += entry_usage.get("cacheWrite", 0)- total_total_tokens += entry_usage.get("totalTokens", 0)- usage = {- "input_tokens": total_input,- "output_tokens": total_output,- "cacheRead": total_cache_read,- "cacheWrite": total_cache_write,- "total_tokens": total_total_tokens,- }- print(f" [{sample_idx}] tokens (from JSONL): in={total_input} out={total_output} cacheRead={total_cache_read} cacheWrite={total_cache_write} total={total_total_tokens}", file=sys.stderr)- except json.JSONDecodeError as e:- print(f" [{sample_idx}] Error parsing JSONL file: {e}, using API usage", file=sys.stderr)- print(f" [{sample_idx}] tokens (from API): in={usage.get('input_tokens',0)} out={usage.get('output_tokens',0)} cacheRead={usage.get('cacheRead',0)} cacheWrite={usage.get('cacheWrite',0)} total={usage.get('total_tokens',0)}", file=sys.stderr)- except IOError as e:- print(f" [{sample_idx}] Error reading JSONL file: {e}, using API usage", file=sys.stderr)- print(f" [{sample_idx}] tokens (from API): in={usage.get('input_tokens',0)} out={usage.get('output_tokens',0)} cacheRead={usage.get('cacheRead',0)} cacheWrite={usage.get('cacheWrite',0)} total={usage.get('total_tokens',0)}", file=sys.stderr)- else:- print(f" [{sample_idx}] JSONL file not found: {jsonl_full_path}, using API usage", file=sys.stderr)- print(f" [{sample_idx}] tokens (from API): in={usage.get('input_tokens',0)} out={usage.get('output_tokens',0)} cacheRead={usage.get('cacheRead',0)} cacheWrite={usage.get('cacheWrite',0)} total={usage.get('total_tokens',0)}", file=sys.stderr)- else:- print(f" [{sample_idx}] tokens (from API): in={usage.get('input_tokens',0)} out={usage.get('output_tokens',0)} cacheRead={usage.get('cacheRead',0)} cacheWrite={usage.get('cacheWrite',0)} total={usage.get('total_tokens',0)}", file=sys.stderr)-+ record = future.result()+ records.append(record)+ # Accumulate usage+ usage = record.get("usage", {})for k in sample_usage:sample_usage[k] += usage.get(k, 0)except Exception as e:- response = f"[ERROR] {e}"- usage = {}- jsonl_filename = ""- print(f" [{sample_idx}] A: {response}", file=sys.stderr)-- record = {- "sample_id": sample_id,- "sample_idx": sample_idx,- "qi": original_qi,- "question": question,- "expected": expected,- "response": response,- "category": category,- "evidence": evidence,- "usage": usage,- "jsonl_filename": jsonl_filename,- }- records.append(record)-- # Save to CSV immediately after successful execution- save_record_to_csv(csv_path, record)- print(f" [{sample_idx}] Saved to CSV: Q{original_qi}", file=sys.stderr)-- if jsonl_file:- try:- jsonl_file.write(json.dumps(record, ensure_ascii=False) + "\n")- jsonl_file.flush()- except IOError as e:- print(f"Warning: Error writing to JSONL file: {e}", file=sys.stderr)-- finally:- if jsonl_file:- jsonl_file.close()- print(f" [{sample_idx}] written to {jsonl_path}", file=sys.stderr)+ print(f" [{sample_idx}] Error in question task: {e}", file=sys.stderr)return records, sample_usage@@ -763,9 +865,12 @@ def run_qa(print("Error: QA mode only works with LoCoMo JSON files", file=sys.stderr)sys.exit(1)+ # Ensure parallel is within reasonable bounds (1-40)+ args.parallel = max(1, min(40, args.parallel))+samples = load_locomo_data(args.input, args.sample)print(f" user: {args.user or 'eval-{sample_idx}'}", file=sys.stderr)- print(f" running in single-thread mode", file=sys.stderr)+ print(f" running with {args.parallel} concurrent workers", file=sys.stderr)# Load already executed records from CSVcsv_path = f"{args.output}.csv" if args.output else args.default_csv_path@@ -774,17 +879,6 @@ def run_qa(executed_records = load_executed_records(csv_path)print(f" Loaded {len(executed_records)} already executed records from {csv_path}", file=sys.stderr)- # Clean up existing session file if session_id is provided- if args.session_id:- sessions_dir = os.path.expanduser(f"~/.openclaw/agents/{args.agent_id}/sessions")- session_file = os.path.join(sessions_dir, f"{args.session_id}.jsonl")- if os.path.exists(session_file):- try:- os.remove(session_file)- print(f" Cleaned up existing session file: {os.path.basename(session_file)}", file=sys.stderr)- except Exception as e:- print(f" Warning: Could not remove existing session file: {e}", file=sys.stderr)-results_list = []for idx, item in enumerate(samples):result = run_sample_qa(item, idx + 1, args, executed_records, csv_path)@@ -797,7 +891,31 @@ def run_qa(print(f"\n total tokens: in={total_usage['input_tokens']} out={total_usage['output_tokens']} total={total_usage['total_tokens']}", file=sys.stderr)+ # Generate timestamp once for all backups+ timestamp = time.strftime("%Y%m%d_%H%M%S")+ import shutil++ # Backup CSV file with timestamp+ if os.path.exists(csv_path):+ csv_path_obj = Path(csv_path)+ backup_csv_path = csv_path_obj.parent / f"{csv_path_obj.stem}_{timestamp}{csv_path_obj.suffix}"+ try:+ shutil.copy2(csv_path, backup_csv_path)+ print(f" CSV backed up to: {backup_csv_path}", file=sys.stderr)+ except Exception as e:+ print(f"Warning: Failed to backup CSV file: {e}", file=sys.stderr)+if args.output:+ # Backup output summary file too+ if os.path.exists(args.output):+ output_path_obj = Path(args.output)+ backup_output_path = output_path_obj.parent / f"{output_path_obj.stem}_{timestamp}{output_path_obj.suffix}"+ try:+ shutil.copy2(args.output, backup_output_path)+ print(f" Summary backed up to: {backup_output_path}", file=sys.stderr)+ except Exception as e:+ print(f"Warning: Failed to backup summary file: {e}", file=sys.stderr)+try:with open(args.output, "w", encoding="utf-8") as f:f.write("=== TOTAL USAGE ===\n")@@ -877,15 +995,9 @@ def main():parser.add_argument("-p", "--parallel",type=int,- default=1,+ default=10,metavar="N",- help="QA mode: number of samples to process concurrently (max 10, default 1).",- )- parser.add_argument(- "--viking",- action="store_true",- default=False,- help="Ingest mode: save to OpenViking via `ov add-memory` instead of OpenClaw.",+ help="QA mode: number of questions to process concurrently (max 40, default 10).",)parser.add_argument("--agent-id",@@ -895,7 +1007,7 @@ def main():parser.add_argument("--session-id",default=None,- help="Session ID for API requests. If provided, will use this session ID and calculate token usage from corresponding JSONL file.",+ help="Session ID for API requests (ingest mode only).",)parser.add_argument("--force-ingest",@@ -913,7 +1025,7 @@ def main():# 添加默认 CSV 路径到 argsargs.default_csv_path = default_csv_path- if not args.token and not getattr(args, "viking", False):+ if not args.token:print("Error: --token or OPENCLAW_GATEWAY_TOKEN env var is required", file=sys.stderr)sys.exit(1)diff --git a/benchmark/locomo/openclaw/import_to_ov.py b/benchmark/locomo/openclaw/import_to_ov.pynew file mode 100644index 000000000..14a51cb21--- /dev/null+++ b/benchmark/locomo/openclaw/import_to_ov.py@@ -0,0 +1,669 @@+"""+OpenViking data import tool.++Import conversations from LoCoMo JSON or plain text files into OpenViking memory.++Usage:+ # Import LoCoMo JSON conversations+ uv run python import_to_ov.py locomo10.json --sample 0 --sessions 1-4++ # Import plain text conversations+ uv run python import_to_ov.py example.txt+"""++import argparse+import asyncio+import csv+import json+import sys+import time+import traceback+from datetime import datetime, timedelta+from pathlib import Path+from typing import List, Dict, Any, Tuple, Optional++import openviking as ov+++def _get_session_number(session_key: str) -> int:+ """Extract session number from session key."""+ return int(session_key.split("_")[1])+++def parse_test_file(path: str) -> List[Dict[str, Any]]:+ """Parse txt test file into sessions.++ Each session is a dict with:+ - messages: list of user message strings+ """+ with open(path, "r", encoding="utf-8") as f:+ content = f.read()++ raw_sessions = content.split("---\n")+ sessions = []++ for raw in raw_sessions:+ lines = [line for line in raw.strip().splitlines() if line.strip()]+ if not lines:+ continue++ messages = []+ for line in lines:+ if not line.startswith("eval:"): # Skip eval lines+ messages.append(line)++ if messages:+ sessions.append({"messages": messages})++ return sessions+++def load_locomo_data(+ path: str,+ sample_index: Optional[int] = None,+) -> List[Dict[str, Any]]:+ """Load LoCoMo JSON and optionally filter to one sample."""+ with open(path, "r", encoding="utf-8") as f:+ data = json.load(f)++ if sample_index is not None:+ if sample_index < 0 or sample_index >= len(data):+ raise ValueError(f"Sample index {sample_index} out of range (0-{len(data) - 1})")+ return [data[sample_index]]+ return data+++def build_session_messages(+ item: Dict[str, Any],+ session_range: Optional[Tuple[int, int]] = None,+) -> List[Dict[str, Any]]:+ """Build session messages for one LoCoMo sample.++ Returns list of dicts with keys: messages, meta.+ Each dict represents a session with multiple messages (user/assistant role).+ """+ conv = item["conversation"]+ speakers = f"{conv['speaker_a']} & {conv['speaker_b']}"++ session_keys = sorted(+ [k for k in conv if k.startswith("session_") and not k.endswith("_date_time")],+ key=_get_session_number,+ )++ sessions = []+ for sk in session_keys:+ sess_num = _get_session_number(sk)+ if session_range:+ lo, hi = session_range+ if sess_num < lo or sess_num > hi:+ continue++ dt_key = f"{sk}_date_time"+ date_time = conv.get(dt_key, "")++ # Extract messages with all as user role, including speaker in content+ messages = []+ for idx, msg in enumerate(conv[sk]):+ speaker = msg.get("speaker", "unknown")+ text = msg.get("text", "")+ messages.append(+ {"role": "user", "text": f"[{speaker}]: {text}", "speaker": speaker, "index": idx}+ )++ sessions.append(+ {+ "messages": messages,+ "meta": {+ "sample_id": item["sample_id"],+ "session_key": sk,+ "date_time": date_time,+ "speakers": speakers,+ },+ }+ )++ return sessions+++# ---------------------------------------------------------------------------+# Ingest record helpers (avoid duplicate ingestion)+# ---------------------------------------------------------------------------+++def load_success_csv(csv_path: str = "./result/import_success.csv") -> set:+ """加载成功导入的CSV记录,返回已成功的键集合"""+ success_keys = set()+ if Path(csv_path).exists():+ with open(csv_path, "r", encoding="utf-8") as f:+ reader = csv.DictReader(f)+ for row in reader:+ key = f"viking:{row['sample_id']}:{row['session']}"+ success_keys.add(key)+ return success_keys+++def write_success_record(+ record: Dict[str, Any], csv_path: str = "./result/import_success.csv"+) -> None:+ """写入成功记录到CSV文件"""+ file_exists = Path(csv_path).exists()+ fieldnames = [+ "timestamp",+ "sample_id",+ "session",+ "date_time",+ "speakers",+ "embedding_tokens",+ "vlm_tokens",+ "llm_input_tokens",+ "llm_output_tokens",+ "total_tokens",+ ]++ with open(csv_path, "a", encoding="utf-8", newline="") as f:+ writer = csv.DictWriter(f, fieldnames=fieldnames)+ if not file_exists:+ writer.writeheader()++ writer.writerow(+ {+ "timestamp": record["timestamp"],+ "sample_id": record["sample_id"],+ "session": record["session"],+ "date_time": record.get("meta", {}).get("date_time", ""),+ "speakers": record.get("meta", {}).get("speakers", ""),+ "embedding_tokens": record["token_usage"].get("embedding", 0),+ "vlm_tokens": record["token_usage"].get("vlm", 0),+ "llm_input_tokens": record["token_usage"].get("llm_input", 0),+ "llm_output_tokens": record["token_usage"].get("llm_output", 0),+ "total_tokens": record["token_usage"].get("total", 0),+ }+ )+++def write_error_record(+ record: Dict[str, Any], error_path: str = "./result/import_errors.log"+) -> None:+ """写入错误记录到日志文件"""+ with open(error_path, "a", encoding="utf-8") as f:+ timestamp = record["timestamp"]+ sample_id = record["sample_id"]+ session = record["session"]+ error = record["error"]+ f.write(f"[{timestamp}] ERROR [{sample_id}/{session}]: {error}\n")+++def is_already_ingested(+ sample_id: str | int,+ session_key: str,+ success_keys: Optional[set] = None,+) -> bool:+ """Check if a specific session has already been successfully ingested."""+ key = f"viking:{sample_id}:{session_key}"+ return success_keys is not None and key in success_keys+++# ---------------------------------------------------------------------------+# OpenViking import+# ---------------------------------------------------------------------------+def _parse_token_usage(commit_result: Dict[str, Any]) -> Dict[str, int]:+ """解析Token使用数据(从commit返回的telemetry或task result中提取)"""+ # 尝试从 task result 中提取(task 完成后包含完整 token_usage)+ if "result" in commit_result:+ result = commit_result["result"]+ if "token_usage" in result:+ tu = result["token_usage"]+ embedding = tu.get("embedding", {})+ llm = tu.get("llm", {})+ # embedding 格式可能是 {"total": N} 或 {"total_tokens": N}+ embed_total = embedding.get("total", embedding.get("total_tokens", 0))+ llm_total = llm.get("total", llm.get("total_tokens", 0))+ return {+ "embedding": embed_total,+ "vlm": llm_total,+ "llm_input": llm.get("input", 0),+ "llm_output": llm.get("output", 0),+ "total": tu.get("total", {}).get("total_tokens", embed_total + llm_total),+ }++ # 从 commit 响应的 telemetry 中提取+ telemetry = commit_result.get("telemetry", {}).get("summary", {})+ tokens = telemetry.get("tokens", {})+ return {+ "embedding": tokens.get("embedding", {}).get("total", 0),+ "vlm": tokens.get("llm", {}).get("total", 0),+ "llm_input": tokens.get("llm", {}).get("input", 0),+ "llm_output": tokens.get("llm", {}).get("output", 0),+ "total": tokens.get("total", 0),+ }+++async def viking_ingest(+ messages: List[Dict[str, Any]],+ openviking_url: str,+ session_time: Optional[str] = None,+ user_id: Optional[str] = None,+ agent_id: Optional[str] = None,+) -> Dict[str, int]:+ """Save messages to OpenViking via OpenViking SDK client.+ Returns token usage dict with embedding and vlm token counts.++ Args:+ messages: List of message dicts with role and text+ openviking_url: OpenViking service URL+ session_time: Session time string (e.g., "9:36 am on 2 April, 2023")+ user_id: User identifier for separate userspace (e.g., "conv-26")+ agent_id: Agent identifier for separate agentspace (e.g., "conv-26")+ """+ # 解析 session_time - 为每条消息计算递增的时间戳+ base_datetime = None+ if session_time:+ try:+ base_datetime = datetime.strptime(session_time, "%I:%M %p on %d %B, %Y")+ except ValueError:+ print(f"Warning: Failed to parse session_time: {session_time}", file=sys.stderr)++ # Create client+ client_kwargs = {"url": openviking_url}+ if user_id is not None:+ client_kwargs["user"] = user_id+ if agent_id is not None:+ client_kwargs["agent_id"] = agent_id+ client = ov.AsyncHTTPClient(**client_kwargs)+ await client.initialize()++ try:+ # Create session+ create_res = await client.create_session()+ session_id = create_res["session_id"]++ # Add messages one by one with created_at+ for idx, msg in enumerate(messages):+ msg_created_at = None+ if base_datetime:+ # 每条消息递增1秒,确保时间顺序+ msg_dt = base_datetime + timedelta(seconds=idx)+ msg_created_at = msg_dt.isoformat()++ await client.add_message(+ session_id=session_id,+ role=msg["role"],+ parts=[{"type": "text", "text": msg["text"]}],+ created_at=msg_created_at,+ )++ # Commit+ result = await client.commit_session(session_id, telemetry=True)++ # Accept both "committed" and "accepted" as success - accepted means the session was archived+ if result.get("status") not in ("committed", "accepted"):+ raise RuntimeError(f"Commit failed: {result}")++ # 等待 task 完成以获取准确 token 消耗+ task_id = result.get("task_id")+ if task_id:+ # 轮询任务状态直到完成+ max_attempts = 1200 # 最多等待20分钟+ for attempt in range(max_attempts):+ task = await client.get_task(task_id)+ status = task.get("status") if task else "unknown"+ if status == "completed":+ token_usage = _parse_token_usage(task)+ break+ elif status in ("failed", "cancelled", "unknown"):+ raise RuntimeError(f"Task {task_id} {status}: {task}")+ await asyncio.sleep(1)+ else:+ raise RuntimeError(f"Task {task_id} timed out after {max_attempts} attempts")+ else:+ token_usage = {"embedding": 0, "vlm": 0, "total": 0}++ # Get trace_id from commit result+ trace_id = result.get("trace_id", "")+ return {"token_usage": token_usage, "task_id": task_id, "trace_id": trace_id}++ finally:+ await client.close()+++def parse_session_range(s: str) -> Tuple[int, int]:+ """Parse '1-4' or '3' into (lo, hi) inclusive tuple."""+ if "-" in s:+ lo, hi = s.split("-", 1)+ return int(lo), int(hi)+ n = int(s)+ return n, n+++async def process_single_session(+ messages: List[Dict[str, Any]],+ sample_id: str | int,+ session_key: str,+ meta: Dict[str, Any],+ run_time: str,+ args: argparse.Namespace,+) -> Dict[str, Any]:+ """处理单个会话的导入任务"""+ try:+ # 根据参数决定是否使用 sample_id 作为 user_id 和 agent_id+ user_id = str(sample_id) if not args.no_user_agent_id else None+ agent_id = str(sample_id) if not args.no_user_agent_id else None+ result = await viking_ingest(+ messages,+ args.openviking_url,+ meta.get("date_time"),+ user_id=user_id,+ agent_id=agent_id,+ )+ token_usage = result["token_usage"]+ task_id = result.get("task_id")+ trace_id = result.get("trace_id", "")+ embedding_tokens = token_usage.get("embedding", 0)+ vlm_tokens = token_usage.get("vlm", 0)+ print(+ f" -> [COMPLETED] [{sample_id}/{session_key}] embed={embedding_tokens}, vlm={vlm_tokens}, task_id={task_id}, trace_id={trace_id}",+ file=sys.stderr,+ )++ # Write success record+ result = {+ "timestamp": run_time,+ "sample_id": sample_id,+ "session": session_key,+ "status": "success",+ "meta": meta,+ "token_usage": token_usage,+ "embedding_tokens": embedding_tokens,+ "vlm_tokens": vlm_tokens,+ "task_id": task_id,+ "trace_id": trace_id,+ }++ # 写入成功CSV+ write_success_record(result, args.success_csv)++ return result++ except Exception as e:+ print(f" -> [ERROR] [{sample_id}/{session_key}] {e}", file=sys.stderr)+ traceback.print_exc(file=sys.stderr)++ # Write error record+ result = {+ "timestamp": run_time,+ "sample_id": sample_id,+ "session": session_key,+ "status": "error",+ "error": str(e),+ }++ # 写入错误日志+ write_error_record(result, args.error_log)++ return result+++async def run_import(args: argparse.Namespace) -> None:+ session_range = parse_session_range(args.sessions) if args.sessions else None++ # 如果指定了 question-index,自动从 evidence 推断需要的 session+ if args.question_index is not None and not args.sessions:+ # 加载数据获取 question 的 evidence+ with open(args.input, "r", encoding="utf-8") as f:+ data = json.load(f)++ # 获取 sample+ sample_idx = args.sample if args.sample is not None else 0+ if sample_idx < 0 or sample_idx >= len(data):+ raise ValueError(f"sample index {sample_idx} out of range")+ sample = data[sample_idx]++ # 获取 question 的 evidence+ qa_items = sample.get("qa", [])+ if args.question_index < 0 or args.question_index >= len(qa_items):+ raise ValueError(f"question index {args.question_index} out of range")+ qa = qa_items[args.question_index]+ evidence_list = qa.get("evidence", [])++ # 从 evidence 提取 session 号 (D1:3 -> session 1)+ session_nums = set()+ for ev in evidence_list:+ try:+ # D1:3 -> session 1+ sess_num = int(ev.split(":")[0][1:])+ session_nums.add(sess_num)+ except (ValueError, IndexError):+ pass++ if session_nums:+ min_sess = min(session_nums)+ max_sess = max(session_nums)+ session_range = (min_sess, max_sess)+ print(+ f"[INFO] Auto-detected sessions from evidence: {min_sess}-{max_sess}",+ file=sys.stderr,+ )++ # 加载成功CSV记录用于去重+ success_keys = set()+ if not args.force_ingest:+ success_keys = load_success_csv(args.success_csv)+ print(+ f"[INFO] Loaded {len(success_keys)} existing success records from {args.success_csv}",+ file=sys.stderr,+ )++ # Write run header+ run_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")++ skipped_count = 0+ success_count = 0+ error_count = 0+ total_embedding_tokens = 0+ total_vlm_tokens = 0++ if args.input.endswith(".json"):+ # LoCoMo JSON format+ samples = load_locomo_data(args.input, args.sample)++ # 为每个 sample 创建独立的处理协程+ async def process_sample(item):+ sample_id = item["sample_id"]+ sessions = build_session_messages(item, session_range)++ print(f"\n=== Sample {sample_id} ===", file=sys.stderr)+ print(f" {len(sessions)} session(s) to import", file=sys.stderr)++ # 同一 sample 内串行处理所有 sessions+ for sess in sessions:+ meta = sess["meta"]+ messages = sess["messages"]+ session_key = meta["session_key"]+ label = f"{session_key} ({meta['date_time']})"++ # Skip already ingested sessions unless force-ingest is enabled+ if not args.force_ingest and is_already_ingested(+ sample_id, session_key, success_keys+ ):+ print(+ f" [{label}] [SKIP] already imported (use --force-ingest to reprocess)",+ file=sys.stderr,+ )+ nonlocal skipped_count+ skipped_count += 1+ continue++ # Preview messages+ preview = " | ".join(+ [f"{msg['role']}: {msg['text'][:30]}..." for msg in messages[:3]]+ )+ print(f" [{label}] {preview}", file=sys.stderr)++ # 串行执行(等待完成后再处理下一个 session)+ await process_single_session(+ messages=messages,+ sample_id=sample_id,+ session_key=session_key,+ meta=meta,+ run_time=run_time,+ args=args,+ )++ # 不同 sample 之间并行执行+ tasks = [asyncio.create_task(process_sample(item)) for item in samples]+ results = await asyncio.gather(*tasks, return_exceptions=True)++ else:+ # Plain text format+ sessions = parse_test_file(args.input)+ print(f"Found {len(sessions)} session(s) in text file", file=sys.stderr)++ for idx, session in enumerate(sessions, start=1):+ session_key = f"txt-session-{idx}"+ print(f"\n=== Text Session {idx} ===", file=sys.stderr)++ # Skip already ingested sessions unless force-ingest is enabled+ if not args.force_ingest and is_already_ingested(+ "txt", session_key, success_keys+ ):+ print(+ f" [SKIP] already imported (use --force-ingest to reprocess)", file=sys.stderr+ )+ skipped_count += 1+ continue++ # For plain text, all messages as user role+ messages = []+ for i, text in enumerate(session["messages"]):+ messages.append(+ {"role": "user", "text": text.strip(), "speaker": "user", "index": i}+ )++ preview = " | ".join([f"{msg['role']}: {msg['text'][:30]}..." for msg in messages[:3]])+ print(f" {preview}", file=sys.stderr)++ # 创建异步任务+ task = asyncio.create_task(+ process_single_session(+ messages=messages,+ sample_id="txt",+ session_key=session_key,+ meta={"session_index": idx},+ run_time=run_time,+ args=args,+ )+ )+ tasks.append(task)++ # 等待所有 sample 处理完成+ print(+ f"\n[INFO] Starting import with {len(tasks)} tasks to process",+ file=sys.stderr,+ )+ await asyncio.gather(*tasks, return_exceptions=True)++ # 从成功 CSV 统计结果+ if Path(args.success_csv).exists():+ with open(args.success_csv, "r", encoding="utf-8") as f:+ reader = csv.DictReader(f)+ for row in reader:+ success_count += 1+ total_embedding_tokens += int(row.get("embedding_tokens", 0) or 0)+ total_vlm_tokens += int(row.get("vlm_tokens", 0) or 0)++ # Final summary+ total_processed = success_count + error_count + skipped_count+ print(f"\n=== Import summary ===", file=sys.stderr)+ print(f"Total sessions: {total_processed}", file=sys.stderr)+ print(f"Successfully imported: {success_count}", file=sys.stderr)+ print(f"Failed: {error_count}", file=sys.stderr)+ print(f"Skipped (already imported): {skipped_count}", file=sys.stderr)+ print(f"\n=== Token usage summary ===", file=sys.stderr)+ print(f"Total Embedding tokens: {total_embedding_tokens}", file=sys.stderr)+ print(f"Total VLM tokens: {total_vlm_tokens}", file=sys.stderr)+ if success_count > 0:+ print(+ f"Average Embedding per session: {total_embedding_tokens // success_count}",+ file=sys.stderr,+ )+ print(f"Average VLM per session: {total_vlm_tokens // success_count}", file=sys.stderr)+ print(f"\nResults saved to:", file=sys.stderr)+ print(f" - Success records: {args.success_csv}", file=sys.stderr)+ print(f" - Error logs: {args.error_log}", file=sys.stderr)+++# ---------------------------------------------------------------------------+# CLI+# ---------------------------------------------------------------------------+++def main():+ # 基于脚本所在目录计算默认数据文件路径+ script_dir = Path(__file__).parent.resolve()+ default_input = str(script_dir / ".." / "data" / "locomo10.json")++ parser = argparse.ArgumentParser(description="Import conversations into OpenViking")+ parser.add_argument(+ "--input",+ default=default_input,+ help="Path to input file (.txt or LoCoMo .json)",+ )+ parser.add_argument(+ "--success-csv",+ default="./result/import_success.csv",+ help="Path to success records CSV file (default: import_success.csv)",+ )+ parser.add_argument(+ "--error-log",+ default="./result/import_errors.log",+ help="Path to error log file (default: import_errors.log)",+ )+ parser.add_argument(+ "--openviking-url",+ default="http://localhost:1933",+ help="OpenViking service URL (default: http://localhost:1933)",+ )+ parser.add_argument(+ "--sample",+ type=int,+ default=None,+ help="LoCoMo JSON: sample index (0-based). Default: all samples.",+ )+ parser.add_argument(+ "--sessions",+ default=None,+ help="LoCoMo JSON: session range, e.g. '1-4' or '3'. Default: all sessions.",+ )+ parser.add_argument(+ "--question-index",+ type=int,+ default=None,+ help="LoCoMo JSON: question index (0-based). When specified, auto-detect required sessions from question's evidence.",+ )+ parser.add_argument(+ "--force-ingest",+ action="store_true",+ default=False,+ help="Force re-import even if already recorded as completed",+ )+ parser.add_argument(+ "--no-user-agent-id",+ action="store_true",+ default=False,+ help="Do not pass user_id and agent_id to OpenViking client",+ )+ args = parser.parse_args()++ # 确保输出目录存在+ Path(args.success_csv).parent.mkdir(parents=True, exist_ok=True)+ Path(args.error_log).parent.mkdir(parents=True, exist_ok=True)++ try:+ asyncio.run(run_import(args))+ except ValueError as e:+ print(f"Error: {e}", file=sys.stderr)+ sys.exit(1)+++if __name__ == "__main__":+ main()diff --git a/benchmark/locomo/openclaw/run_full_eval.sh b/benchmark/locomo/openclaw/run_full_eval.shindex a14cfa413..3f1edff11 100755--- a/benchmark/locomo/openclaw/run_full_eval.sh+++ b/benchmark/locomo/openclaw/run_full_eval.sh@@ -6,9 +6,12 @@ set -eOpenClaw 完整评估流程脚本用法:- ./run_full_eval.sh # 只导入 OpenViking- ./run_full_eval.sh --with-claw-import # 同时导入 OpenViking 和 OpenClaw- ./run_full_eval.sh --skip-import # 跳过导入步骤+ ./run_full_eval.sh # 只导入 OpenViking (所有 samples)+ ./run_full_eval.sh --with-claw-import # 同时导入 OpenViking 和 OpenClaw (所有 samples)+ ./run_full_eval.sh --skip-import # 跳过导入步骤 (所有 samples)+ ./run_full_eval.sh --sample 0 # 只处理第 0 个 sample+ ./run_full_eval.sh --sample 1 --with-claw-import # 只处理第 1 个 sample,同时导入 OpenClaw+ ./run_full_eval.sh --force-ingest # 强制重新导入所有数据'# 基于脚本所在目录计算数据文件路径@@ -16,20 +19,58 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"INPUT_FILE="$SCRIPT_DIR/../data/locomo10.json"RESULT_DIR="$SCRIPT_DIR/result"OUTPUT_CSV="$RESULT_DIR/qa_results.csv"-GATEWAY_TOKEN="your_gateway_token"+GATEWAY_TOKEN="90f2d2dc2f7b4d50cb943d3d3345e667bb3e9bcb7ec3a1fb"# 解析参数SKIP_IMPORT=falseWITH_CLAW_IMPORT=false-for arg in "$@"; do- if [ "$arg" = "--skip-import" ]; then- SKIP_IMPORT=true- elif [ "$arg" = "--with-claw-import" ]; then- WITH_CLAW_IMPORT=true- fi+FORCE_INGEST=false+SAMPLE_IDX=""++while [[ $# -gt 0 ]]; do+ case $1 in+ --skip-import)+ SKIP_IMPORT=true+ shift+ ;;+ --with-claw-import)+ WITH_CLAW_IMPORT=true+ shift+ ;;+ --force-ingest)+ FORCE_INGEST=true+ shift+ ;;+ --sample)+ if [ -z "$2" ] || [[ "$2" == --* ]]; then+ echo "错误: --sample 需要一个参数 (sample index, 0-based)"+ exit 1+ fi+ SAMPLE_IDX="$2"+ shift 2+ ;;+ *)+ echo "警告: 未知参数 $1"+ shift+ ;;+ esacdone+# 构建 sample 参数+SAMPLE_ARG=""+if [ -n "$SAMPLE_IDX" ]; then+ SAMPLE_ARG="--sample $SAMPLE_IDX"+ # 如果指定了 sample,修改输出文件名以避免覆盖+ OUTPUT_CSV="$RESULT_DIR/qa_results_sample${SAMPLE_IDX}.csv"+fi++# 构建 force-ingest 参数+FORCE_INGEST_ARG=""+if [ "$FORCE_INGEST" = true ]; then+ FORCE_INGEST_ARG="--force-ingest"+fi+# 确保结果目录存在mkdir -p "$RESULT_DIR"@@ -39,18 +80,18 @@ if [ "$SKIP_IMPORT" = false ]; thenecho "[1/5] 导入数据到 OpenViking 和 OpenClaw..."# 后台运行 OpenViking 导入- python "$SCRIPT_DIR/../vikingbot/import_to_ov.py" --input "$INPUT_FILE" --force-ingest > "$RESULT_DIR/import_ov.log" 2>&1 &+ python "$SCRIPT_DIR/../vikingbot/import_to_ov.py" --input "$INPUT_FILE" $FORCE_INGEST_ARG $SAMPLE_ARG > "$RESULT_DIR/import_ov.log" 2>&1 &PID_OV=$!# 后台运行 OpenClaw 导入- python "$SCRIPT_DIR/eval.py" ingest "$INPUT_FILE" --force-ingest --token "$GATEWAY_TOKEN" > "$RESULT_DIR/import_claw.log" 2>&1 &+ python "$SCRIPT_DIR/eval.py" ingest "$INPUT_FILE" $FORCE_INGEST_ARG --token "$GATEWAY_TOKEN" $SAMPLE_ARG > "$RESULT_DIR/import_claw.log" 2>&1 &PID_CLAW=$!# 等待两个导入任务完成wait $PID_OV $PID_CLAWelseecho "[1/5] 导入数据到 OpenViking..."- python "$SCRIPT_DIR/../vikingbot/import_to_ov.py" --input "$INPUT_FILE" --force-ingest+ python "$SCRIPT_DIR/import_to_ov.py" --no-user-agent-id --input "$INPUT_FILE" $FORCE_INGEST_ARG $SAMPLE_ARGfiecho "导入完成,等待 1 分钟..."@@ -61,7 +102,7 @@ fi# Step 2: 运行 QA 模型(默认输出到 result/qa_results.csv)echo "[2/5] 运行 QA 评估..."-python "$SCRIPT_DIR/eval.py" qa "$INPUT_FILE" --token "$GATEWAY_TOKEN"+python "$SCRIPT_DIR/eval.py" qa "$INPUT_FILE" --token "$GATEWAY_TOKEN" $SAMPLE_ARG --parallel 15 --output "${OUTPUT_CSV%.csv}"# Step 3: 裁判打分echo "[3/5] 裁判打分..."@@ -69,7 +110,7 @@ python "$SCRIPT_DIR/judge.py" --input "$OUTPUT_CSV" --parallel 40# Step 4: 计算结果echo "[4/5] 计算结果..."-python "$SCRIPT_DIR/../vikingbot/stat_judge_result.py" --input "$OUTPUT_CSV"+python "$SCRIPT_DIR/stat_judge_result.py" --input "$OUTPUT_CSV"echo "[5/5] 完成!"echo "结果文件: $OUTPUT_CSV"diff --git a/benchmark/locomo/openclaw/stat_judge_result.py b/benchmark/locomo/openclaw/stat_judge_result.pynew file mode 100644index 000000000..63816e004--- /dev/null+++ b/benchmark/locomo/openclaw/stat_judge_result.py@@ -0,0 +1,161 @@+import argparse+import csv+import os+++def main():+ parser = argparse.ArgumentParser(description="Statistics for judge result csv")+ parser.add_argument(+ "--input",+ default="./result/qa_results_sample0.csv",+ help="Path to judge result csv file, default: ./result/qa_results_sample0.csv",+ )+ parser.add_argument(+ "--import-csv",+ default="./result/import_success.csv",+ help="Path to import_success.csv file for OpenViking token stats, default: ./result/import_success.csv",+ )+ args = parser.parse_args()++ output_lines = []++ # 统计 QA 结果+ if os.path.exists(args.input):+ qa_stats = process_qa_results(args.input)+ output_lines.extend(qa_stats)+ else:+ output_lines.append(f"Warning: QA result file not found: {args.input}")++ # 统计 Import token+ if os.path.exists(args.import_csv):+ if output_lines:+ output_lines.append("")+ import_stats = process_import_csv(args.import_csv)+ output_lines.extend(import_stats)+ else:+ output_lines.append(f"Warning: Import CSV file not found: {args.import_csv}")++ # 打印到控制台+ for line in output_lines:+ print(line)++ # 写入summary.txt+ if args.input:+ summary_path = os.path.join(os.path.dirname(args.input), "summary.txt")+ elif args.import_csv:+ summary_path = os.path.join(os.path.dirname(args.import_csv), "summary.txt")+ else:+ summary_path = "./result/summary.txt"++ os.makedirs(os.path.dirname(summary_path), exist_ok=True)+ with open(summary_path, "w", encoding="utf-8") as f:+ f.write("\n".join(output_lines) + "\n")+ print(f"\nSummary saved to {summary_path}")+++def process_qa_results(input_path: str) -> list[str]:+ """处理 QA 结果 CSV"""+ # 统计所有题目 (排除 category=5)+ correct = 0+ wrong = 0+ total_no_cache_tokens = 0 # input_tokens+ total_cache_read_tokens = 0 # cacheRead+ total_output_tokens = 0 # output_tokens+ total_input_tokens = 0 # no_cache + cacheRead+ valid_rows = 0++ with open(input_path, "r", encoding="utf-8", newline="") as f:+ reader = csv.DictReader(f)+ for row in reader:+ # 检查 category 是否为 5,跳过+ category = row.get("category", "")+ if category == "5":+ continue++ valid_rows += 1++ # 统计结果+ result = row.get("result", "").strip().upper()+ if result == "CORRECT":+ correct += 1+ elif result == "WRONG":+ wrong += 1++ # 统计token+ try:+ no_cache = int(row.get("input_tokens", 0))+ cache_read = int(row.get("cacheRead", 0))+ output = int(row.get("output_tokens", 0))++ total_no_cache_tokens += no_cache+ total_cache_read_tokens += cache_read+ total_output_tokens += output+ total_input_tokens += no_cache + cache_read+ except (ValueError, TypeError):+ pass++ total_graded = correct + wrong+ accuracy = correct / total_graded if total_graded > 0 else 0.0++ # 平均 token 消耗+ avg_no_cache = total_no_cache_tokens / valid_rows if valid_rows > 0 else 0.0+ avg_cache_read = total_cache_read_tokens / valid_rows if valid_rows > 0 else 0.0+ avg_output = total_output_tokens / valid_rows if valid_rows > 0 else 0.0+ avg_total_input = total_input_tokens / valid_rows if valid_rows > 0 else 0.0++ return [+ "=== Judge Result Statistics (excluding category=5) ===",+ f"Total rows: {valid_rows:,}",+ f"Graded rows: {total_graded:,}",+ f"Correct: {correct:,}",+ f"Wrong: {wrong:,}",+ f"Accuracy: {accuracy:.2%}",+ f"\nToken usage (QA):",+ f" Total no-cache tokens (input_tokens): {total_no_cache_tokens:,}",+ f" Total cacheRead tokens: {total_cache_read_tokens:,}",+ f" Total output tokens: {total_output_tokens:,}",+ f" Total input tokens (no-cache + cacheRead): {total_input_tokens:,}",+ f" Avg no-cache tokens: {avg_no_cache:,.2f}",+ f" Avg cacheRead tokens: {avg_cache_read:,.2f}",+ f" Avg output tokens: {avg_output:,.2f}",+ f" Avg total input tokens: {avg_total_input:,.2f}",+ ]+++def process_import_csv(input_path: str) -> list[str]:+ """处理 import_success.csv 的 token 统计"""+ total_embedding = 0+ total_vlm = 0+ total_total = 0+ valid_rows = 0++ with open(input_path, "r", encoding="utf-8", newline="") as f:+ reader = csv.DictReader(f)+ for row in reader:+ valid_rows += 1+ try:+ total_embedding += int(row.get("embedding_tokens", 0))+ total_vlm += int(row.get("vlm_tokens", 0))+ total_total += int(row.get("total_tokens", 0))+ except (ValueError, TypeError):+ pass++ avg_embedding = total_embedding / valid_rows if valid_rows > 0 else 0.0+ avg_vlm = total_vlm / valid_rows if valid_rows > 0 else 0.0+ avg_total = total_total / valid_rows if valid_rows > 0 else 0.0++ return [+ "=== OpenViking Import Token Statistics ===",+ f"Total sessions: {valid_rows:,}",+ f"\nToken usage (Import):",+ f" Total embedding tokens: {total_embedding:,}",+ f" Total VLM tokens: {total_vlm:,}",+ f" Total tokens: {total_total:,}",+ f" Avg embedding tokens: {avg_embedding:,.2f}",+ f" Avg VLM tokens: {avg_vlm:,.2f}",+ f" Avg total tokens: {avg_total:,.2f}",+ ]+++if __name__ == "__main__":+ main()